How we do inspections on websites

The authority’s internal preparations

Selection of organisations and websites

The authority selects which organisations and websites will be inspected based on an overall assessment that takes particular account of a number of considerations. We make use of the authority’s knowledge of where failures to comply with the regulations are likely to exist, which has been gained from previous status evaluations, surveys, and other measures.

Our supervision strategy (in Norwegian) provides comprehensive guidelines for the inspections and indicates our current priority sectors. At the same time, we assess organisations in sectors other than the priority ones if the solution in question:

• Has many users.
• Is important for the individual’s right to participate equally in society.

Selection of requirements and test pages

All inspections we carry out are based on samples and are not complete examinations of the website. We always select a sample of user tasks, individual pages, and requirements to test. The sample is selected based on an overall assessment of:

• Important user tasks on the website.
• Risk of breaches of the requirements.
• Different page templates and content types.
• Knowledge about common errors.
• Initial visual checks and spot checks.

Notice of inspection

We never conduct inspections without the prior knowledge of the organisation. We inform an organisation at least 14 days before we start testing the website.

We send out notices of inspection and other documents digitally via eDelivery (in Norwegian). The organisation receives the document in its case management system or message box in Altinn (in Norwegian). Notices of inspection are marked with the name of the organisation’s chief executive officer.

The aim of the notice of inspection is to:

• Notify the organisation that their website is going to be inspected.
• Ask for documentation and a contact person in the organisation.
• Provide an overview of important deadlines, dates, and steps in the inspection process.
• Provide information about what the inspection involves and what requirements will be tested.
• Provide information about the Authority for Universal Design of ICT in Norway.

The organisation must confirm that the notice of inspection has been received. Many do this at the same time as they specify who the organisation’s contact person is for the inspection. The contact person varies but it is often the web editor or director of communications.

Dialogue with the organisation

We always maintain a dialogue with the organisation during the inspection.

We are usually in contact with the organisation in connection with them receiving the notice of inspection, submitting documentation, receiving the preliminary inspection report, and while the organisation is working on correcting errors.

Many organisations and their service providers want more guidance on the content of the requirements, information on how we conduct tests, and assessments of their suggested approach to correcting the errors. In our opinion, organisations should be proactive, ask questions, and request clarifications. This is enlightening, both for the authority and the organisation. In our experience, how proactive organisations are during supervision can vary greatly.

Documentation

The notice of inspection asks the organisation to send us selected documents. The documentation we normally ask for includes:

• The contract or specification of requirements for the website.
• An overview of important changes and updates to the website.
• Important user tasks.
• An overview of the most frequently used pages.
• An overview of known errors related to universal design.

We need the documentation so we can select relevant user tasks, processes, and individual pages to test, and to gain an impression of whether the organisation has an overview of the website’s universal design status.

However, the website itself is the most important documentation of whether the organisation is actually complying with universal design requirements or not.

New ICT

The distinction between new and existing ICT ended on 1 January 2021. Every website that meets the regulation’s basic terms and conditions (in Norwegian) is covered by the requirements regardless of the website’s age.

Cancellation and postponement of inspections

Occasionally, an announced inspection has to be cancelled or postponed.

An inspection might be postponed if the organisation’s documentation shows that they are in the process of procuring a new website or of making significant changes to their current website. In such circumstances, we will normally deem it appropriate to postpone the inspection until the new or upgraded website has been launched. This is more sensible than both the organisation and us expending resources on testing and correcting errors in a website that is about to be replaced or significantly changed.

Testing websites and documenting results

Testing the organisation’s website is the most important activity in an inspection.

Testing reveals and documents the status of the website’s universal design as it was at the time of the testing.

We test selected parts of the website. Given today’s technology, requirements, and test methods it is not practically possible to conduct a full test of a website. We choose what pages will be tested based on the chosen theme for the inspection, combined with an overview of which pages and services are used most frequently.

Conducting tests

The testing is carried out by employees of the authority. We usually check important user tasks and the functionality of the website against a set of test rules. Sometimes we test a random selection of individual pages on the website.

The testing is a combination of manual, semi-automated, and automated testing. The majority of the requirements in the regulations still require manual testing. All test results are registered and documented. In addition to the test registration itself, the results are, where practically possible, documented with a screenshot of the error.

The test results are presented to the organisation in the inspection report. The organisations receive the screenshots of the errors at the same time as the inspection report.

The authority’s interpretation of WCAG and test rules

The test rules are the authority’s interpretation of the 35 minimum requirements in the regulations (in Norwegian).

We have produced a set of test rules to measure compliance with or breaches of the regulations. Each test rule is based on:

• A documented interpretation of WCAG 2.0.
• Information about what it takes to fulfil the specific requirement.
• A standardised and detailed test procedure.
• A specific template for registering test data.
• Pre-defined test results that are generated automatically based on the registered test data.

Assessment and collation of test results

After completing the tests we collate the various test results that show breaches of the regulations.

We assess the severity of the breaches and categorise the information, which provides the basis for the results presented in the inspection report and the division into non-conformance and remarks.

We document which requirements the breaches are in violation of and back this up with concrete test results.

Possible results after an inspection

An inspection can have three possible results. It is important to remember that the results provide an impression of the status of the website at the time of testing.

Non-conformance

Non-conformance means that the organisation is not complying with the requirements of the regulations. Errors that contribute to the non-conformance must subsequently be corrected by the organisation and are followed up by the authority.

Non-conformance can be followed up with sanctions if necessary.

Remarks

Remarks mean that while the Authority has found grounds for pointing out potential for improvement in the organisation the situation is not contrary to the requirements of the regulations.

but that condition of the solution is not in violation with the requirements.

It is up to the organisation itself to assess whether and how it wants to deal with remarks. Remarks cannot be followed up with sanctions.

Compliance with the requirements

If no breaches of the requirements that have been tested in the inspection are identified, it means that the website complies with the regulations as far as the requirements, individual pages and user tasks that were tested are concerned.

Inspection report

All of the checks are summarised in an inspection report. The purpose of the inspection report is to summarise and document the completed inspection. It provides, among other things:

• The results and associated justification for them, as well as an overview of which requirements the breaches are in violation of.
• What organisation and what website has been inspected.
• What parts of the website have been investigated and what requirements have been tested.
• How the inspection was conducted.

Preliminary report

First we send a preliminary report which gives the organisation a chance to look at the results and the reasoning behind them. The organisation has an opportunity to refute the report’s factual basis, ask questions and provide comments.

A preliminary report will be sent to the organisation within three weeks of the tests being completed.

Feedback from the organisation

The organisation has two weeks to provide written feedback on the factual basis and assessments in the report. This allows any errors or misunderstandings to be corrected in the final report.

Final report

The final inspection report provides a final summary of the inspection and includes an overview of the results and activities. The final report is updated in line with relevant feedback from the organisation.

Inspections without breaches of the regulations end with the final report.

Publication

Inspection reports are public documents and available from eInnsyn – Electronic Public Records (in Norwegian). We also publish all of our inspection reports on our website (in Norwegian).

Inspection reports also make useful reading for other organisations that have to comply with the regulations and everyone who works with ICT solutions. The same errors are often seen repeatedly on other websites. Familiarising yourself with results that show where others have gone wrong therefore contributes to learning, quality improvements and a better understanding of the regulations and the content of the requirements.

Follow-up of breaches of the regulations

Inspections in which we identify breaches of the regulations that are non-conformance are followed up by the authority. The organisation is followed up to ensure that the errors found when the website was tested are corrected.

Usually, a number of approaches can be taken to meet the requirements. As long as the results will comply with the requirements, the organisation is free to choose how to correct the errors.

Sanctions can be used as a mean of ensuring organisations comply with the regulations. Sanctions are always used where an inspection identifies non-conformance.

The authority can apply two types of sanction:

• Correction orders.
• Enforcement fines (daily fines).

Decisions regarding sanctions follow the rules for individual decisions in chapters 4 and 5 of the Public Administration Act.

Correction orders

The organisation receives a separate decision containing an order to correct something, which is sent at the same time as the final report. The contents of the order point to the non-conformance based on the reasons in chapter 2 of the inspection report.

Correction will result in the website going from being in breach of the regulations to being in compliance with the requirements as far as the topics and pages that were tested during the inspection are concerned.

Correction orders also contain a warning about the use of enforcement fines if the organisation does not correct the errors on the website by the deadline specified in the order.

Deadline for correcting non-conformance

Once the final inspection report has been sent, the organisation has 12 weeks to correct the breaches documented in the inspection report. The deadline is calculated from the date the final report was sent. Example: Final supervision report is sent on 26 February 2020. Deadline for correction is set for 12:00 on 20 May 2020.

The non-conformance will not be marked corrected and closed until the error correction have been put into production and the end users are no longer experiencing the errors on the website. The end users are the public, customers, and other users of the organisation’s website. Therefore, error corrections must be put into production by the deadline.

Follow-up testing of the website after orders

Once the deadline for correction passes, the authority will carry out follow-up testing of the website. The follow-up testing is limited to the breaches documented in the inspection report. This is done so we can obtain a documented, updated status after the corrections implemented by the organisation.

If the follow-up testing concludes that the errors have been corrected in a manner that meets the requirements of the regulations, the inspection ends with a letter to the organisation.

Enforcement fines (daily fines)

Enforcement fines are relevant for organisations that fail to show a willingness to correct errors on a website, even after the organisation has received an order to remedy breaches of the requirements.

Organisations receive a separate decision on enforcement fines.

In the decision the organisation is given a new deadline by which it needs to correct the remaining errors. This deadline is usually 14 days.

If an error is not corrected before the deadline expires, the organisation will be subject to an enforcement fine. The daily fine must be big enough to ensure that it does not pay off for an organization to not correct the errors.

It is set following a total assessment of several factors:

• Number of users of a website.
• The severity of the errors – the consequences for users.
• How much it costs to correct the errors.
• How difficult it is to correct the errors.
• The organisations’ turnover and financial situation.
• The organisation’s size.
• What amount of time should be deemed appropriate to correct the errors.

Follow-up testing of websites after enforcement fines

After the deadline in the decision on enforcement fines expires, we conduct a new follow-up test. We do this to obtain a documented, updated status for the website after a decision on enforcement fines has been made.

The organisation will receive a specific letter about the implementation and collection of enforcement fines. The enforcement fine is a daily fine and accrues until the error has been corrected. We have an agreement with the Norwegian National Collection Agency regarding the collection of enforcement fines.

Organisations have the right to appeal the imposition of sanctions

Organisations can appeal the imposition of correction orders and decisions regarding enforcement fines. The rules of the appeals procedure are set out in chapter 6 of the Public Administration Act.

The Ministry of Local Government and Modernisation (KDD) is the appeals body.