Website Accessibility Evaluation Methodology for WCAG 2.0

Status of this document

This clause describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 9 November 2011 Editors Draft of Website Accessibility Evaluation Methodology for WCAG 2.0 is an initial outline for future refinement. This document is intended to be published and maintained as a W3C Working Group Note after review and refinement.

The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback about this document by developers, evaluators, researchers, and other practitioners who have interest in web accessibility evaluation. In particular, Eval TF is looking for feedback on the completeness of this outline.

Please send comments on this Website Accessibility Evaluation Methodology for WCAG 2.0 document to public-wai-evaltf@w3.org (publicly visible mailing list archive).

Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

1. Introduction

This WCAG 2.0 Evaluation Methodology (WCAG-EM) proposes an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0. The Methodology supports evaluation in different contexts, such as self-assessment and third-party evaluation of websites. It is applicable to all websites (including web applications) regardless of size and it is independent of any particular evaluation tools, browsers, and assistive technology.

2. Scope of this document

Editor note: This clause defines without ambiguity the subject of the document and the aspects covered, thereby indicating the limits of applicability of the document or particular parts of it. It does not contain requirements.

The WCAG 2.0 Evaluation Methodology describes a formal and harmonized way to evaluate the conformance of websites to WCAG 2.0. The Methodology defines manual and computer assisted methods for selecting representative samples of web pages from websites that include complete processes. It provides guidance for evaluating the selected web pages to WCAG 2.0, and defines methods for integration and aggregation of the evaluation results into structured reports and conformance claims.

The Methodology is an editors draft that provides guidance on evaluation throughout the development process but it is specifically applicable to conformance evaluation of existing websites. It extends the existing WAI resource Conformance Evaluation of Websites for Accessibility. Complementary WAI resources such as Preliminary Review of Websites for Accessibility and Involving Users in Evaluating Web Accessibility provide further advice on evaluation during other stages of the development process.

3. Target audience

Editor note: Description of the target audience of the Methodology. We did some work for this in the requirements document.

The primary target audience of the Methodology is anyone, including individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Users of the Methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technology, and of how people with different disabilities use the Web.

4. References

Editor note: A list of the referenced documents cited in this document as far as they are indispensable for the application.

The following documents, in whole or in part, are (normatively) referenced in this document and are indispensable for its application. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

5. Terms and definitions

Editor note: Terminology that is really important for the understanding of the Methodology. General words and terms could be placed in the glossary at the end of the document. We already did some work in the requirements document like for website etc.

6. Expertise for evaluating accessibility

The W3C Evaluation suite extensively describes the expertise necessary for the evaluation of the accessibility of Web content for people with disabilities. Evaluation activities require diverse kinds of expertise and perspectives. Individuals evaluating the accessibility of web content require training and experience across a broad range of disciplines. A collaborative approach can bring better results for individuals evaluating web content. The W3C/WAI evaluation suite writes: "Effective evaluation of Web accessibility requires more than simply running an evaluation tool on a Web site. Comprehensive and effective evaluations require evaluators with an understanding of Web technologies, evaluation tools, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and accessibility guidelines and techniques." This document describes evaluation methodology. Automatic evaluation can significantly reduce the time and effort needed for an evaluation but please note that many accessibility checks require human judgement and must be evaluated manually. This is described in this document.

6.1 Involving People with Disabilities in the process

Besides evaluating the conformance to a set of standards like WCAG 2.0, there are many benefits to evaluating with real people. They can show how a website really works for users and in that way help to better understand the accessibility issues. Evaluating with users with disabilities and with older users can identify additional usability issues that are not discovered by conformance evaluation alone. user testing is not covered in this Methodology, but we strongly advise that you involve users in your evaluation process. Involving users in evaluation of websites is described in more detail in the W3C WAI Evaluation Suite section about users.

7. Procedure to express the Scope of the evaluation (based on:)

Editor note: How can an evaluator express the scope of a website. What is in and what can be left out? Below are some possible subclauses that cover things that look necessary to describe to pinpoint the exact scope of what is in and what can be left outside the scope of a website:

7.1 Technologies used on the webpages

While the WCAG 2.0 Recommendation focus on webpages, the Evaluation Methodology focuses on evaluating the conformance of a website. This means that it is important to define what is part of the website and what is not. To be more precise, to define what is part of the coherent collection of one or more related web pages that together provide common use or functionality and can include static web pages, dynamically generated web pages, and web applications.
Web pages are not limited to html. It includes PDF, Office documents, audio, video and photos. All web pages that are part of the website fall within the scope of the evaluation.

The scope of a website can include subdomains and parts of external domains. The setting of the scope delimites the pages and applications to be included into the sample.

A webpage is part of the scope of an evaluation if one or more of the following apply:

7.2 Base URI of the evaluation

First determine the main URI of the website. For most websites this will consist of a domain name and top level domain (like .com or .eu). The base URI can also be more specific, for example a directory added to the URI or a subdomain.

The base URI does not determine whether a page is inside or outside the scope. It is only the starting point from which the rest of the scope is determined. All pages that use the base URI, both because they have an extensive directory structure or use of sub-(sub-) domains fall within the scope unless there is a clearly described exception.

Some websites have multiple domains linked for the same content and dynamics, in that case the scope is limited to the primary domain. A conformity declaration would in that case also be valid for the other websites.

7.3 Perception and function

Perception and function focuses on 3 criteria, namely Graphical User Interface, general design and navigation. This includes pictograms used for certain functionalities on the website, interaction elements for users to navigate the website and other interaction functionalities on the website.

7.4 Complete processes

The definition of complete processes can be found in the subclause terms and definitions. Complete processes can include parts of a website into the scope of an evaluation. In case of a series of steps that need to be completed in order to accomplish an activity, all Web pages in the process are part of the scope of the evaluation and should conform at the specified level or better.

7.5 Key functionalities

7.6 Webpages behind authorization

Editor note:Parts of a website can be in or outside the scope of the evaluation if they cannot be viewed without the user having proper authorization or authentication by the organization that manages the website.

7.7 Dividing the scope into multiple evaluations

Editor note: Imagine a website is large and the would like to divide the evaluation over different parts for which different people are responsible. If all parts are in the scope of the website, then the scope could be divided into multiple parts that all have to be evaluated.

8. Sampling of pages

Editor note: An evaluator can manually evaluate all pages, but on website with millions of webpages that is a lot of work. How to select a representative sample of a website is described in this clause. How many pages and how do you choose them?

8.1 Sample selection (random and targeted sampling)

Editor note: This subclause describes how to select a sample. We will describe both random and targeted sampling as output, not describing in detail for instance how to select a random sample but more the size in relation to the website. Also this subclause will describe in a first round the relation between the sample, the technologies used on the websites and accessibility supported.

8.2 Size of evaluation samples (related to barriers)

Editor note: This section describes the size of the sample of a website. This could be by defining the number of webpages per technology related to the overall size of the website.

9. Evaluation

Editor note: This is the clause describing step by step how to do the evaluation. The evaluation is depending on many factors, like technologies used, technologies evaluated, Accessibility supported etc. Important factor in the evaluation is formed by the accessibility barriers that are encountered and their impact. When are they a real problem? how does this relate to the size of the sample. Also this clause describes a possible margin for errors on a website in relation to the barrierimpact.

9.1 Manual and machine evaluation

Editor note: This subclause describes how to relate manual and machine evaluation. It describes the both evaluation types and their relation to accessibility conformance evaluation.

9.2 Technologies

Editor note: How do you choose technologies for evaluation? This relates to the clause about sampling and scope and is dependant on the use of accessibility supported technologies and the availability of accessible and conformant alternatives.

9.3 Procedure for evaluation

Editor note: Step by step description of the evaluation of the website (sample). This does not include going into the guidelines, success criteria etc from WCAG 2.0 or related techniques. It could be possible to propose different ways to evaluate the guidelines: one by one, per theme, per technology, etc. The Methodology will not prescribe one of those ways as necessary.

9.4 Barrier recognition

Editor note: How to recognize the accessibility barriers and ascertain the impact. Also this subclause describes the relation to the sample. If a barrier has been recognized multiple times, it is necessary to keep on searching for it? This influences the sampling and reporting.

9.5 Error margin

Editor note: Define an error margin. If one out of 1 million images on a website fails the alt-attribute and the impact is very low. Does this constitute a fail for the complete website. We will define an error margin to cover these non-structural errors that have a low impact.

10. Conformity

Editor note: This clause is largely from WCAG 2.0 with additional steps related to scope, sample, barriers and reporting. It includes explanation and implementation of conformity requirements etc. (see subclauses below).

10.1 Conformity requirements

10.2 Accessibility supported

Editor note: Describe accessibility supported and the impact on the website. What to do if you find technologies that are not accessibility supported?

10.3 Partial conformance

Editor note: Description of how to claim partial conformance. Also relation to the scope that is set at the start of the eveluation.

10.4 Conformance claims

Editor note: Mostly already inside WCAG 2.0. Additions related to scope, sample, accessibility supported etc.

10.5 Score function (barrier change)

Editor note: Explanation of how to score both with manual and with machine evaluation of websites. Also depending on the way that the evaluation is done (one by one, thematic, etc.).

11. Reporting

Editor note: How to write a report from the evaluation that is human readable and one that is machine readable. And what should be in the report. Templates are included in the annexes.

11.1 Text based report

Editor note: Explanation of the manual report in human understandable language. What should be in the report minimally. The template is in the annex.

11.2 Machine readable report using EARL

Editor note: Explanation of the machine readable report. The template is in the annex.

12. Limitations and underlying assumptions

Editor note: This clause will be filled during the process of making the evaluation methodology. We will place here the limitations and underlying assumptions that are important for readers to understand the methodology.

13. Acknowledgements

Editor note: Names of all persons who worked on the WCAG 2.0 Evaluation Methodology, members of the Eval Taskforce.

14. Glossary

Editor note: Glossary for this document and link to general glossary for WCAG 2.0.

15. References

Annex 1: Template for manual evaluation report

Annex 2: Template for EARL report

Editor note: A complete example template for evaluation reporting using EARL.