[contents]

Abstract

This document provides guidance on evaluating how well websites conform to the Web Content Accessibility Guidelines (WCAG) 2.0. It describes a procedure to evaluate websites and includes considerations to guide evaluators and to promote good practice. It does not provide instructions for evaluating web content feature by feature, which is addressed by WCAG 2.0 success criteria. This document is one of a series of informative W3C/WAI resources about Evaluating Websites for Accessibility that complements the WCAG 2.0 Documents. It does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

The methodology described in this document is intended for people who are experienced in evaluating accessibility using WCAG 2.0 and its supporting resources. It provides guidance on good practice in defining the evaluation scope, exploring the target website, selecting representative samples from websites where it is not feasible to evaluate all content, auditing the selected samples, and recording the evaluation findings. It is primarily designed for evaluating existing websites, for example, to learn about them and to monitor their level of accessibility. It can also be useful during earlier development stages of websites. It applies to static and dynamically generated websites, mobile websites and applications, and other types of websites. It does not specify particular web technologies, evaluation tools, web browsers, assistive technologies, or other software to use for evaluation. It is suitable for use in different evaluation contexts, including self-assessment and third-party evaluation.

Introduction

Evaluating the extent to which a website conforms to the Web Content Accessibility Guidelines (WCAG) 2.0 is a process involving several steps. The activities carried out within these steps are influenced by many aspects such as the type of website (e.g. static, dynamic, responsive, mobile, etc.), its size, complexity, and the technologies used to create it (e.g. HTML, PDF, etc.), how much knowledge the evaluators have of how the website was or is being developed, and the main purpose for the evaluation (e.g. to issue an accessibility statement, to plan a redesign process, to perform research, etc.).

This methodology, describes the steps that are common to website conformance evaluation processes. It highlights considerations for evaluators to apply these steps in the context of a particular website. It does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. Following this methodology will help evaluators apply good practice, avoid commonly made mistakes, and achieve more comparable results. It supports common approaches and understanding for evaluating the extent of conformance of websites to WCAG 2.0, though in the majority of use cases it does not directly result in conformance claims.

This methodology does not in any way add to or change the requirements defined by the normative WCAG 2.0 standard, nor does it provide instructions on feature-by-feature evaluation of web content. The methodolody can be used in conjunction with techniques for meeting WCAG 2.0 success criteria, such as the Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or any other specific set of techniques.

Purposes for this Methodology

In many situations it is necessary to evaluate the accessibility of a website, for example before releasing, acquiring, or redesigning the website. Periodic evaluation is also useful for monitoring the accessibility performance of websites over time. This methodology is designed for anyone who wants to follow a common procedure for evaluating the conformance of websites to WCAG 2.0. This includes:

• Web consultants who want to analyze and report the accessibility conformance of websites, to inform website owners.
• Web accessibility evaluation service providers who want to evaluate websites to validate accessibility conformance.
• Website developers who want to evaluate the accessibility conformance of their websites to monitor or improve them.
• Website owners, procurers, and suppliers who want to learn about the accessibility conformance of their websites.
• Web accessibility monitoring activities who want to benchmark or compare the accessibility conformance over time.
• Web accessibility researchers and disability advocates who want to explore accessibility conformance practices.
• Web accessibility trainers and educators who want to teach approaches for evaluating the accessibility of websites.
• Web masters, content authors, designers, and others who want to learn more about web accessibility and evaluation.

Relation to WCAG 2.0 Conformance Claims

WCAG 2.0 defines conformance requirements for individual web pages (and in some cases, sets of web pages), but does not describe how to evaluate entire websites. It also defines how optional conformance claims can be made to cover individual web pages, a series of web pages such as a multi-page form, and multiple related web pages such as a website. This is applicable when all web pages that are in the scope of a conformance claim have each been evaluated or created in a process that ensures that they each satisfy all the conformance requirements.

WCAG 2.0 conformance claims cannot be made for entire websites based upon the evaluation of a selected sub-set of web pages and functionality alone, as it is always possible that there will be unidentified conformance errors on these websites. However, in the majority of uses of this methodology only a sample of web pages and functionality from a website is selected for evaluation. Thus in the majority of situations, using this methodology alone does not result into WCAG 2.0 conformance claims for the target websites. Guidance on making statements about the outcomes from using this methodology is provided in Step 5.c: Provide an Evaluation Statement (Optional).

Background Reading

The information below, related to web accessibility essentials, evaluation, and WCAG 2.0 is important for using this methodology:

Web Accessibility Essentials

The following documents introduce the essential components of web accessibility and explain how people with disabilities use the Web. They are critical for understanding the broader context of web accessibility evaluation:

• Essential Components of Web Accessibility
• How People with Disabilities Use the Web

Evaluating Websites for Accessibility

These are particularly important resources that outline different approaches for evaluating websites for accessibility:

• Easy Checks - A First Review of Web Accessibility
• Involving Users in Web Accessibility Evaluation
• Selecting Web Accessibility Evaluation Tools
• Using Combined Expertise to Evaluate Web Accessibility

Web Content Accessibility Guidelines (WCAG) 2.0

This is the internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important for accessibility evaluation of websites:

• WCAG 2.0 Overview
• WCAG 2.0 Technical Specification
• How to Meet WCAG 2.0 (Quick Reference)
• Understanding WCAG 2.0
• Techniques for WCAG 2.0

Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete processes

From WCAG 2.0 Conformance Requirement for Complete Processes:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)

Conformance

From WCAG 2.0 definition for "conformance":
Satisfying all the requirements of a given standard, guideline or specification

Common web pages

Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all other web pages (usually from the header, footer, or navigation menu of a web page). Common web pages may also be web page states in web applications.

Essential functionality

Functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information that users of a website refer to and tasks that they carry out to perform this functionality.

Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

Note: Other functionality is not excluded from the scope of evaluation. The term "essential functionality" is intended to help identify critical web pages and include them among others in an evaluation.

Evaluator

The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.

Evaluation commissioner

The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.

Note: In many cases the evaluation commissioner may be the website owner or website developer, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.

Relied upon (Technologies)

From WCAG 2.0 definition for "relied upon":
The content would not conform if that technology is turned off or is not supported

Templates

From ATAG 2.0 definition for "templates":
Content patterns that are filled in by authors or the authoring tool to produce web content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.

[Editor Note: This definition is from the 7 November 2013 Candidate Recommendation (CR) Working Draft of ATAG 2.0. While this is a stable Working Draft, this definition might change in future drafts depending on how ATAG 2.0 evolves.]

Website

A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and mobile websites and applications.

Note: The focus of this methodology is on full, self-enclosed websites. Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website. For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website.

Website developer

The person, team of people, organization, in-house department, or other entity that is involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.

Website owner

The person, team of people, organization, in-house department, or other entity that is responsible for the website.

Web page

From WCAG 2.0 definition for "web page":
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent

Note: Web pages may include multimedia content, interactive components, and rich and mobile web applications. Web pages are not limited to HTML and can be PDF documents and any other format.

Web page states

Web pages with dynamic content can have different states (changes to the Document Object Model - DOM); for example, they might generate different content and provide different presentation or functionality depending on the particular user and on actions initiated by the user. In the context of this methodology, web page states can be treated as ancillary to web pages (i.e., recorded as an additional state of a web page in a web page sample) or as individual web pages.

Note: Examples of web page states are the individual pages of a multi-part online form that are dynamically generated depending on users input. These individual states may not have unique URIs and may need to be identified by describing the settings, input, and actions required to generate them.

Using This Methodology

This methodology is used for thorough evaluation of websites using WCAG 2.0. Before evaluating an entire website it is usually good to do a preliminary evaluation of different web pages from the target website to identify obvious accessibility barriers and develop an overall understanding of the accessibility performance of the website. Easy Checks - A First Review of Web Accessibility describes an approach for such preliminary evaluation that is complementary to this methodology.

Required Expertise

Users of this methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of relevant web technologies, accessibility barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques, tools, and methods to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section Background Reading.

Evaluation Tools (Optional)

This methodology is independent of any particular web accessibility evaluation tool, web browser, and other software tool. Many checks are not automatable, however, web accessibility evaluation tools can significantly assist evaluators during the evaluation process and contribute to more effective evaluation. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation. Tools can also assist in manually evaluating the many checks that are not automatable. Selecting Web Accessibility Evaluation Tools provides further guidance beyond the scope of this document.

Review Teams (Optional)

This methodology can be carried out by an individual evaluator with the skills described in the section Required Expertise. However, using the combined expertise of review teams provides broader coverage of the required skills and helps identify accessibility barriers more effectively. While not required for using this methodology, the use of review teams is recommended when performing an evaluation of a website. Using Combined Expertise to Evaluate Web Accessibility provides further guidance beyond the scope of this document.

Involving Users (Optional)

Involving people with disabilities including people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by expert evaluation alone. While not required for using this methodology, it is recommended that evaluators involve real people with a wide range of abilities during the evaluation process. Involving Users in Web Accessibility Evaluation provides further guidance beyond the scope of this document.

Scope of Applicability

This methodology is designed for evaluating full, self-enclosed websites. This includes websites of organizations, entities, persons, events, products, and services.

Principle of Website Enclosure

When a target website is defined for evaluation, it is essential that all web pages, web page states, and functionality within the scope of this definition are considered for evaluation. Excluding such aspects of a website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements for full pages and complete processes, or otherwise distort the evaluation results.

In the preceding example, if the university website in its entirety is defined as the target for evaluation then all of the depicted areas are within the scope of the evaluation. This includes any aggregated and embedded content such as maps of the campus, forms for online payments, and discussion boards, including when such parts originate from third-party sources. If only a specific website area, such as the "Courseware Application", is defined as the target for evaluation then all the parts of this area are within the scope of the evaluation. In this case, the scope of evaluation would include all depicted courses, as well as the individual web pages that are common to all areas of the university. See also the definition for Common Web Pages.

Particular Types of Websites

This methodology is applicable to the broad variety of websites. The following considerations apply to particular types of websites:

Small Websites

For websites with few web pages the sampling procedure defined in Step 3: Select a Representative Sample will likely result in selecting most or all of the web pages from the target website. In cases where all web pages can be evaluated, the sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

Web Applications

Web applications are generally composed of dynamically generated content and functionality (see web page states). Web applications tend to be more complex and interactive. Some examples of web applications include webmail clients, document editors, and online shops. Web applications may be part of a larger website but can also constitute a website of their own in the context of this methodology. That is, an individual and separable entity for evaluation.

Note: Due to the many possibilities of generating content and functionality in web applications it is sometimes not feasible to exhaustively identify every possible web page, web page state, and functionality. Web applications will typically require more time and effort to evaluate, and larger web page samples to reflect the different types of content, functionality, and processes.

Website with Separable Areas

In some cases websites may have clearly separable areas where using one area does not require or depend on using another area of the website. For example, an organization might provide an extranet for its employees only that is linked from the public website but is otherwise separate, or it might have sub-sites for individual departments of the organization that are each clearly distinct from one another. Such separable areas can be considered as individual websites each for evaluation. In some cases there may be common web pages, such as legal notices, that need to be considered as part of each website area.

Note: Some websites provide additional or different content and functionality depending on the user (typically after a log-in). This additional content and functionality is generally part of the essential purpose and functionality of the website and is thus not considered to be a separable website area.

Website in Multiple Versions

Some websites are available in multiple versions that are independent of one another in use, that is, using one version does not require or depend on using another version of the website. For example, a website may have a mobile version and there may be versions of a website in different languages that meet this characteristic. Usually each such website version has a different set of URIs. Such website versions can be considered as individual websites for evaluation.

Note: Websites using responsive design techniques (i.e. adapting the presentation according to user hardware, software, and preferences) as opposed to redirecting the user to a different location are not considered to be independent website versions.

Website Using Responsive Design

[Editor Note: Brief explanation of how to evaluate websites using responsive design techniques to be added here in future revisions.]

Particular Evaluation Contexts

This methodology is designed to be flexible to facilitate its applicability in different situations and contexts. The following considerations apply to particular situations and contexts for an evaluation:

Self-Assessment of Conformance

In this context evaluators often have easier access to the website developers and maintainers, the development and hosting environments, the authoring tools, and the materials used for development and maintenance. Particularly use cases, design analysis, technical specifications and documentation, and testing resources can make evaluation more effective and should be leveraged where possible.

Third-Party Assessment of Conformance

Third-party evaluators typically have little information about how a website was developed, its internal software, and all its areas and functionality. Therefore it is more difficult, often impossible, to make conformance claims for entire websites based on such an evaluation alone. However, such evaluation can be effective for validating conformance claims and statements about websites.

Evaluating During Development

It is critical to evaluate accessibility throughout the design and implementation stages of a website to manage its conformance. This methodology provides guidance that can be useful during these earlier stages of the development process, though some adaptation may be needed. However, it is important to be aware that evaluations carried out during these development stages can quickly become obsolete by implementing even minor changes. Consequently evaluations carried out during these stages should not be used for making statements nor conformance claims about the finalized website.

Evaluating Composite Websites

When evaluating websites that are composed of separable areas, such as online shop, blog area, and other sub-sites, it can be useful to first evaluate each website area separately according to this methodology, followed by an overall evaluation with samples from each website area and any common web pages. This would ensure more complete coverage of the website in its entirety as well as provide insights about how each website area performed, which may differ from one area to another.

Evaluating Aggregated Websites

Websites that are generated using content that is combined from different sources, such as portals with portlets, are usually much more challenging to evaluate because of the many different content instances that can be generated. Generally it is not possible to evaluate the content from their sources separately but rather as displayed to the users when they are combined.

Evaluating Third-Party Content

Third-party content is not under the control of the website or web service providers; for example content generated by website users in an online forum. When such content is regularly monitored and repaired (within two business days), then the content can be deemed as conforming according to WCAG 2.0. Otherwise the non-conforming content needs to be clearly identified in the web pages in which it appears. WCAG 2.0 provides more guidance on providing a Statement of Partial Conformance.

Re-Running Website Evaluation

Website evaluation, according to this methodology, may be re-run after a short period; for example, when issues are identified and repaired by the website owner or website developer, or periodically to monitor progress. In such cases, for the barriers that were identified, include at least:

• A portion of web pages that were in the original sample to facilitate comparability between the results;
• Additional web pages that were not in the original sample to improve coverage of the website; and,
• Where possible, a different set of exemplar web page instances (Step 3.c) to facilitate sufficient variance.

Large-Scale Evaluation

Carrying out mass evaluation of many websites, for example for national or international surveying, is typically carried out using primarily automated evaluation tools. Relatively few web pages undergo full manual inspection. Such evaluations do not usually address the necessary qualitative depth of review per website for which this methodology is primarily designed.

Evaluation Procedure

This section defines the stages and activities of an evaluation procedure. The stages are not necessarily sequential. Also the exact sequence of the activities carried out during the evaluation stages depends on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the activities overlap or may be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

The workflow diagram depicts each of the five steps defined in this section with an arrow to the next step and arrows back to all of the previous steps. This way evaluators can proceed from one step to the next and return to any preceding step in the process whenever the evaluation reveals a need to return to any one of the previous steps: 1. Define the evaluation scope; 2. Explore the target website; 3. Select a representative sample; 4. Audit the selected sample and 5. Report the findings.

Note: See also considerations for Particular Evaluation Contexts that may influence how an evaluation procedure is carried out.

Step 1: Define the Evaluation Scope

Methodology Requirement 1: Define the evaluation scope according to Methodology Requirement 1.a, Methodology Requirement 1.b, and Methodology Requirement 1.c, and optionally Methodology Requirement 1.d and Methodology Requirement 1.e.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure. It is ideally carried out in consultation with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation. Initial exploration of the target website during this step may be necessary to better know specifics of the website and the required evaluation. Detailed exploration of the website is carried out in Step 2: Explore the Target Website.

Step 1.a: Define the Scope of the Website

Methodology Requirement 1.a: Define the target website according to Scope of Applicability, so that for each web page it is unambiguous whether it is within the scope of evaluation or not.

During this step the target website (the web pages and states of web pages that are in scope of the evaluation) is defined. This scope of the website is defined according to the terms established in the section Scope of Applicability.

To avoid later mismatches of expectations between the evaluator, evaluation commissioner, and readers of the resulting evaluation report, it is important to define the target website in such a way that for each web page it is unambiguous whether it is within scope or not. Using formalizations including regular expressions and listings of web addresses (URIs) is recommended where possible.

It is also important to document any particular aspects of the target website to support its identification. This includes:

• Use of third-party content and services;
• Mobile and language versions of the website;
• Parts of the website, especially those that may not be easily identifiable as such, for example, an online shop that has a different web address but is still considered to be part of the target website.

Step 1.b: Define the Conformance Target

Methodology Requirement 1.b: Select a target WCAG 2.0 conformance level ("A", "AA", or "AAA") for the evaluation.

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for. WCAG 2.0 Level AA is the generally accepted and recommended target.

Note: It is often useful to evaluate beyond the conformance target of the website to get a more complete picture of its accessibility performance. For example, while a website might not fully meet a particular conformance level, it might meet individual requirements from a higher conformance level. Having this information can help plan future improvements more effectively.

Step 1.c: Define an Accessibility Support Baseline

Methodology Requirement 1.c: Define the web browser, assistive technologies and other user agents for which features provided on the website are to be accessibility supported.

Particularly for new web technologies it is usually not possible that every accessibility feature, such as a 'show captions' function in a media player, is supported by every possible combination of web browser, assistive technology, and other user agents. WCAG 2.0 does not pre-define which combinations of features and technologies must be supported as this depends on the particular context of the website, including its language, the web technologies that are used to create the content, and the user agents currently available. Understanding Accessibility Support provides more guidance.

During this step a list of web browsers, assistive technologies, and other user agents that support the accessibility features on the website is defined. For some websites in closed networks, such as an intranet website where the computers used to access it are known, this baseline may be limited to the web browsers and assistive technologies used within this closed network. However, in most cases this baseline is ideally as broad as possible to cover the majority of user agents used by people with disabilities.

Step 1.d: Define Evaluation Methods to be Used (Optional)

Methodology Requirement 1.d: Define any initial evaluation methods to be used to help evaluate conformance to WCAG 2.0 Success Criteria (Optional).

Determining conformance to WCAG 2.0 Success Criteria can be typically done in several ways. W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some evaluators might use other methods (inline with the requirements for custom techniques) to evaluate conformance to WCAG 2.0.

During this step, particular evaluation methods, if any are to be used, are defined. This does not limit the evaluator from using other additional evaluation methods at a later point, for example to evaluate particular content that was not identified at this early stage of the evaluation process. However, it is good practice to specify the initial evaluation methods to be used during the evaluation, as it helps to ensure consistent expectation between the evaluator and the evaluation commissioner.

Step 1.e: Define Additional Evaluation Requirements (Optional)

Methodology Requirement 1.e: Define any additional evaluation requirements agreed on between the evaluator and evaluation commissioner (Optional).

An evaluation commissioner may be interested in additional information beyond what is needed to evalaute the extent of conformance of the target website to WCAG 2.0. For example, an evaluation commissioner might be interested in:

• Evaluation of additional web pages beyond what is needed to form a representative sample from the target website;
• Reports of all occurrences of issues rather than representative examples of the types of issues on the target website;
• Analysis of particular use cases, situations, and user groups for interacting with the target website;
• Explanation of possible solutions to the encountered issues beyond what would be in the scope of website evaluation;
• Adherence to specific documentation or reporting templates.

Such additional evaluation requirements that are agreed on with the evaluator need to be clarified early on and documented. This also needs to be reflected in the resulting report, for example, to clarify how the selection of the sample was carried out.

Step 2: Explore the Target Website

Methodology Requirement 2: Explore the website to be evaluated according to Methodology Requirement 2.a, Methodology Requirement 2.b, Methodology Requirement 2.c, Methodology Requirement 2.d, and Methodology Requirement 2.e.

During this step the evaluator explores the target website to be evaluated, to develop an initial understanding of the website and its use, purpose, and functionality. Much of this will not be immediately apparent to evaluators, in particular to those from outside the development team. In some cases it is also not possible to exhaustively identify and list all functionality, types of web pages, and technologies used to realize the website and its applications. The initial exploration carried out in this step is typically refined in the later steps Step 3: Select a Representative Sample and Step 4: Audit the Selected Sample, as the evaluator learns more about the target website. Involvement of website owners and website developers can help evaluators make their explorations more effective.

Note: Carrying out initial cursory checks during this step helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, document structure, or consistent navigation, and note them down for more detailed evaluation later on.

Note: To carry out this step it is critical that the evaluator has access to all the relevant parts of the website. For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation. Granting evaluators such access may require particular security and privacy precautions.

Step 2.a: Identify Common Web Pages of the Website

Methodology Requirement 2.a: Identify the common web pages, which may be web page states, of the target website.

Explore the target website to identify its common web pages, which may also be web page states in web applications. Typically these are linked directly from the main entry point (home page) of the target website, and often linked from the header, navigation, and footer sections of other web pages. The outcome of this step is a list of all common web pages.

Step 2.b: Identify Essential Functionality of the Website

Methodology Requirement 2.b: Identify an initial list of essential functionality of the target website.

Explore the target website to identify its essential functionality. While some functionality will be easy to identify, others will need more deliberate discovery. For example, it may be easier to identify the functionality for purchasing products in an online shop than the functionality provided for vendors to sell products through the shop. The outcome of this step is a list of functionality that users can perform on the website.

Note: The purpose of this step is not to exhaustively identify all functionality of a website but to determine those that are essential to the purpose and goal of the target website. This will inform later selection of web pages and their evaluation. Other functionality will also be included in the evaluation but through other selection mechanisms.

Step 2.c: Identify the Variety of Web Page Types

Methodology Requirement 2.c: Identify the types of web pages and web page states.

Web pages and web page states with varying styles, layouts, structures, and functionality often have varying support for accessibility. They are also often generated by different templates and scripts, or authored by different people. They may also appear differently, behave differently, and contain different content depending on the particular website user and context. The outcome of this step is a list of the different types (as opposed to specific instances) of web pages and web page states that appear on the target website.

Content to look for to identify different types of web page and web page states include:

• Content with varying styles, layout, structure, navigation, interaction, and visual design;
• Content with varying types of content such as forms, tables, lists, headings, multimedia, and scripting;
• Content with varying functional components such as date picker, lightbox, slider, and others;
• Content using varying technologies such as HTML, CSS, JavaScript, WAI-ARIA, PDF, etc.;
• Content from the different website areas (home page, web shop, departments, etc.) including any applications;
• Content that are created using different templates (if this is known to the evaluator);
• Content that are authored by different people (if this is known to the evaluator);
• Content that are created using different coding styles;
• Content that change appearance, behavior, and content depending on the user, device, browser, context, and settings;
• Content with dynamic content, error messages, dialog-boxes, pop-up windows, and other interaction;

Step 2.d: Identify Web Technologies Relied Upon

Methodology Requirement 2.d: Identify the web technologies relied upon to provide the website.

During this step the web technologies relied upon (for conformance) to provide the website are identified. This includes base web technologies such as HTML and CSS, auxiliary web technologies such as JavaScript and WAI-ARIA, as well as specific web technologies such as SMIL, SVG and PDF. The outcome of this step is a list of technologies that are relied upon according to WCAG 2.0.

Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.

Step 2.e: Identify Other Relevant Web Pages

Methodology Requirement 3.c: Identify other web pages and web page states that are relevant to people with disabilities and to accessibility of the website.

Some websites include web pages and web page states that are specifically relevant for people with disabilities and accessibility of the website. The outcome of this step is a list of such web pages. This includes:

• Web pages explaining accessibility features of the websites;
• Web pages with information and help on the use of the website;
• Web pages explaining settings, preferences, options, shortcuts, etc.;
• Web pages with contact information, directions, and support instructions.

Step 3: Select a Representative Sample

Methodology Requirement 3: Select a representative sample of web pages from the website according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, and Methodology Requirement 3.e.

During this step the evaluator selects a sample of web pages and web page states that is representative of the target website to be evaluated. The purpose of this selection is to ensure that the evaluation results reflect the accessibility performance of the website with reasonable confidence. In cases where it is feasible to evaluate all web pages, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps of the conformance evaluation procedure.

The actual size of sample web pages and web page states needed to evaluate a website depends on many factors including:

• Size of the website — websites with more web pages typically require a larger sample to evaluate;
• Age of the website — older websites tend to have more (often not easy to find) content with different levels of complexity, consistency, and development processes, so that a larger sample is typically required to evaluate;
• Complexity of the website — higher complexity requires a larger sample to evaluate; consider the following:
  • How interactive the content is —websites with content that is rich in interaction require larger samples to cover the functions provided by a website and the different states that individual web pages can have;
  • How the content is generated — websites with content that is aggregated from different sources or that is processed as it is served (at runtime) typically require larger samples to cover the different combinations of content that can be generated;
  • How the content is implemented — websites that are available in different versions, are served according to users and their preferences, or adapt to access devices require larger samples to cover these different situations;
• Consistency of the website — lower consistency requires a larger sample to evaluate; consider the following:
  • Variety of web page types — websites with a broader variety of web page types (see Step 2.c: Identify the Variety of Web Page Types) require larger samples to evaluate;
  • Variety of functionality — websites with a broader variety of functionality (see Step 2.b: Identify Essential Functionality of the Website), in particular different types of applications, require larger samples to evaluate;
  • Variety of technologies — websites with a broader variety of web technologies in use (see Step 2.d: Identify Web Technologies Relied Upon) require larger samples to evaluate;
  • Variety of coding styles — websites with a broader variety of coding styles (typically these are from different scripts that generate the code, templates, and web page authors) require larger samples to evaluate;
• Website development process — lower adherence requires a larger sample to evaluate; consider the following:
  • Formalization of the process — websites with formalized development and quality assurance processes tend to show more consistency in the coding and quality of the web pages so that they typically require smaller samples to evaluate;
  • Training for the developers — websites with developers and content authors that receive regular training tend to have web pages with more consistent accessibility performance so that they typically require smaller samples to evaluate;
  • Development tools being used — websites that are developed and maintained using a consistent set of tools such as a content management system (CMS) also tend to be more consistent and require smaller samples to evaluate;
  • Number of web page authors — websites that are developed and maintained by a more confined set of web page authors, including content editors, tend to be more consistent and require smaller samples to evaluate;
• Required level of confidence — higher confidence in the evaluation results requires evaluation of a larger sample.

This step relies initially on the exploration carried out in Step 2: Explore the Target Website and is continually refined during the later Step 4: Audit the Selected Sample, as the evaluator learns more about the particular implementation aspects of the target website.

Step 3.a: Include Common Web Pages of the Website

Methodology Requirement 3.a: Include all common web pages, including web page states, into the selected sample.

Include all common web pages and web page states that were identified in Step 2.a: Identify Common Web Pages of the Website into the selected sample for evaluation.

Step 3.b: Include Other Relevant Web Pages

Methodology Requirement 3.b: Include all other web pages and web page states that are relevant to people with disabilities and accessibility of the website into the selected sample.

Include all other relevant web pages and web page states that were identified in Step 2.e: Identify Other Relevant Web Pages into the selected sample for evaluation.

Step 3.c: Include Exemplar Instances of Web Pages

Methodology Requirement 3.c: Include instances of web pages and web page states (where applicable and available) to cover the (1) essential functionality, (2) types of web pages, and (3) web technologies relied upon into the selected sample.

Select additional instances of web pages and web pages states to ensure that the sample includes:

• Content from each essential functionality identified in Step 2.b: Identify Essential Functionality of the Website;
• Content from the different types of web pages identified in Step 2.c: Identify the Variety of Web Page Types;
• Content provided using the web technologies identified in Step 2.d: Identify Web Technologies Relied Upon.

Some of the content might already be reflected in the web pages and web page states selected in Step 3.a: Include Common Web Pages of the Website and Step 3.b: Include Other Relevant Web Pages. For example, the common web pages might use the same web technologies and design as the majority of other web pages on the website. The number of required instances of web pages and web page states depends on the particular aspects of the website (see factors influencing the sample size).

Note: An individual web page or web page state may reflect more than one of each of these listed criteria. For example, a single web page may be representative of a particular design layout, functionality, and web technologies used. The purpose of this step is to have sufficient representation of the different types of web pages and web page states, functionality, and web technologies that occur on the website. Careful selection of these representative instances can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Step 3.d: Include Complete Processes in the Sample

Methodology Requirement 3.d: Include all web pages and web page states that are part of a complete process into the selected sample.

The selected sample has to include all web pages and web page states that belong to a series presenting a complete process. No web page or web page state in the selected sample may be part of a process without all other web pages and web page states that are part of that process to be also included into the selected sample.

Use the following steps to include the necessary web pages and web page states into the sample:

1. For any web page and web page state selected through Step 3.c: Include Exemplar Instances of Web Pages and that is part of a process, locate the starting point (web page or web page state) for the process and replace them in the selected sample;
2. For each essential functionality identified in Step 2.b: Identify Essential Functionality of the Website, locate the starting points (web page or web page state) for the processes associated with each functionality and include them into the selected sample;
3. For each starting point for a process, identify and record at least the default sequence of web pages and web page states to complete the process. Add these web pages and web page states into the selected sample.
   Note: The default sequence follows the standard use case, describing the default path through the complete process. It assumes that there are no user input errors and no selection of additional options. For example, for a web shop application, the user would proceed to checkout, confirm the default payment option, provide all required payment details correctly, and complete the purchase, without changing contents of the shopping cart, using a stored user profile, selecting alternative options for payment or shipping address, providing erroneous input, and so forth.
4. For each process, identify and record the branch sequences of web pages and web page states that are commonly accessed and critical for the successful completion of the process. Add these web pages and web page states into the selected sample.
   Note: Branch sequences may terminate where they re-enter the default branch of the process. For example, adding a new shipping address will be registered as a critical alternative branch that leads back to the default branch of the process.

Note: In most cases it is necessary to record and specify the actions needed to proceed from one web page and web page state to the next in a sequence to complete a process so that they can be replicated later. An example of such action could be "fill out name and address, and select the 'Submit' button". In most cases the web address (URI will not be sufficient to identify the web page and web page state in a complete process. It is also useful to clearly record when web pages and web page states are part of a process so that evaluators can focus their effort on the relevant changes such as elements that were added, modified, or made visible.

Step 3.e: Include a Randomly Selected Sample

Methodology Requirement 3.e: Select a random sample of web pages and web page states, and include them for auditing.

A randomly selected sample of web pages and web page states acts as an indicator to verify that the structured sample selected through the previous steps truly reflects the accessibility performance of the website. Confidence in the overall evaluation outcome increases when the evaluation results from both selection approachs correlate.

The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5 instances of web pages and web page states. For example, if the structured sample resulted in 80 web pages and web page states, then the random sample size is 8 web pages and web page states. For structured samples of size 50 and below the random sample size is always 5 web pages and web page states (if the website is not smaller than that).

To perform this selection, randomly select unique instances of web pages and web page states from the target website and that are not already part of the structured sample selected through the previous steps. Depending on the type of website and the access that an evaluator has for it there are different techniques that may need to be used for this selection. This may include:

• Using a tool that will traverse the website and propose a list of randomly selected web pages and web page states;
• Using a script that will generate a list of all web pages and web page states available on a website, to select from;
• Using server logs, search engines, blind-folded colleagues, and other creative methods to get to a random sample.

Document the web pages and web page states that were randomly selected as these will need to be compared to the remaining structured sample in Step 4.e: Compare Structured and Random Samples.

Note: While the random sample need not be selected according to strictly scientific criteria, the scope of the selection needs to span the entire scope of the website (any web page and web page state on the website may be selected with similarly equivalent likelihood), and the selection of individual web page and web page states does not follow a predictable pattern. Recording the method used to generate the random sample is important for replicability and reliability of the results.

Step 4: Audit the Selected Sample

Methodology Requirement 4: Audit the selected sample of web pages according to Methodology Requirement 4.a, Methodology Requirement 4.b, Methodology Requirement 4.c, Methodology Requirement 4.d and Methodology Requirement 4.e.

During this step the evaluator audits (detailed evaluation) the sample selected in Step 3: Select a Representative Sample according to the five WCAG 2.0 conformance requirements at the conformance level from Step 1.b: Define the Conformance Target. Many web pages and web page states in the sample will have components, such as the header, navigation bars, search form, and others that are occur repeatedly. Typically these components do not need to be re-evaluated on each occurrence unless they appear or behave differently, or when additional requirements are defined in Step 1.d: Define Additional Requirements for Evaluation (Optional).

Important: Carrying out this step requires deep understanding of the WCAG 2.0 conformance requirements and the WCAG 2.0 Layers of Guidance. Understanding Conformance provides more guidance.

Note: If there is no content related to Success Criteria presented to the user (for example, no video on the web page), then the Success Criteria are "satisfied" according to WCAG 2.0. Optionally, an evaluation report can specifically indicate Success Criteria for which there is no relevant content, for example, with "not present".

Step 4.a: Check for Each Success Criterion

Methodology Requirement 4.a: Check that each full page (web page and web page state) in the selected sample satisfies each of the WCAG 2.0 Success Criteria of the target conformance level.

For each web page and web page state in the sample selected in Step 3: Select a Representative Sample that is not within or the end of a complete process, check its conformance with each WCAG 2.0 Success Criterion within the target conformance level set in Step 1.b: Define the Conformance Target. This includes all components of the web page or web page state without activating any functions, entering any data, or otherwise interacting with the content (this will be evaluated in the subsequent steps).

Consider the following approach for evaluating the conformance of all content to each WCAG 2.0 Success Criterion:

• Check for any occurrences of failures to meet WCAG 2.0 (W3C/WAI documents some Failures to Meet WCAG 2.0);
• Check that all content is supported by accessibility features (W3C/WAI documents some Techniques for WCAG 2.0);
• Check that all features are supported by the web browsers and assistive technologies defined in Step 1.c: Define an Accessibility Support Baseline;

Note: W3C/WAI continually documents (non-normative) failures and techniques for WCAG 2.0 but these can never be exhaustive to cover every possible situation. They are also not the only ways to meet or to fail to meet WCAG 2.0 Success Criteria, and evaluators typically use additional methods to evaluate conformance to WCAG 2.0 (see also Step 1.d: Define Evaluation Methods to be Used).

Note: While it is important to check the conformance of each web page and web page state in the sample to each WCAG 2.0 Success Criterion, repeated components such as the header, navigation bars, search form, and others usually do not need to be re-evaluated on each occurrence. For example, if the same header code is used on two web pages then it is evaluated on the first only in most situations.

Step 4.b: Check for Complete Processes

Methodology Requirement 4.b: Check that all interaction of each web page and web page state along a complete process satisfies each of the WCAG 2.0 Success Criteria of the target conformance level.

For each complete process selected in Step 3.d: Include Complete Processes in the Sample, follow the identified default and branch sequences of web pages and web page states, and evaluate each according to Step 4.a: Check WCAG 2.0 Success Criteria. However, in this case it is not necessary to evaluate all content but only the content that changes along the process.

Functionality, entering data, notifications, and other interaction is part of this check. In particular it includes:

• Interaction with forms, input elements, dialog boxes, and other web page components;
• Confirmations for input, error messages, and other feedback from user interaction;
• Behavior using different settings, preferences, devices, and interaction parameters;

Step 4.c: Check for Accessible Alternatives

Methodology Requirement 4.c: Check that each web page and web page state in the selected sample that does not conform to WCAG 2.0 at the target conformance level has a conforming alternate version.

For each of the web pages and web page states in the sample (selected in Step 3: Select a Representative Sample) that does not satisfy any of the five WCAG 2.0 conformance requirements at the conformance target defined in Step 1.b: Define the Conformance Target, check that it has at least one conforming alternate version as defined in WCAG 2.0. The WCAG 2.0 guidance on Understanding Conforming Alternate Versions provides more background on accessible alternatives.

Step 4.d: Check for Non-Interference

Methodology Requirement 4.d: Check that each web page and web page state in the selected sample satisfies the WCAG 2.0 conformance requirement non-interference.

For each of the web pages and web page states in the sample (selected in Step 3: Select a Representative Sample), check that it satisfies the WCAG 2.0 conformance requirement non-interference. The WCAG 2.0 guidance on Understanding Requirement 5 provides more background on non-interference.

Step 4.e: Compare Structured and Random Samples

Methodology Requirement 4.e: Check that each web page and web page state in the randomly selected sample does not show types of content and outcomes that are not represented in the structured sample.

While the individual occurences of WCAG 2.0 Success Criteria will vary between the structured and randomly selected samples, the randomly selected sample should not show new types of content not present in the structured sample. Also the outcomes from evaluating the randomly selected sample should not show new findings to those of the structured sample. If the randomly selected sample shows new types of content or new evaluation findings then it is an indication that the structured sample was not sufficiently representative of the content provided on the website. In this case evaluators need to go back to Step 3: Select a Representative Sample) to select additional web pages and web page states that reflect the newly identified types of content and findings. Also the findings of Step 2: Explore the Target Website might need to be adjusted accordingly. This step is repeated until the structured sample is adequately representative of the content provided on the website.

Step 5: Record the Evaluation Findings

Methodology Requirement 5: Document the evaluation findings according to Methodology Requirement 5.a and optionally Methodology Requirement 5.b, Methodology Requirement 5.c, Methodology Requirement 5.d, and Methodology Requirement 5.e.

While evaluation findings are reported at the end of the process, recording them is carried out throughout the evaluation process to ensure reliable outcomes. The recordings typically have varying levels of confidentiality. For example, recording the specific methods used to evaluate individual requirements might remain limited to the evaluator while reports about the outcomes from these checks are typically made available to the evaluation commissioner. Website owners might further choose to make public statements about the outcomes from evaluation according to this methodology.

Step 5.a: Document the Outcomes of Each Step

[Review Note: Feedback on this section is particularly welcome. Specifically, how elaborate does this documentation need to be to ensure transparency yet practicality of the methodology?]

Methodology Requirement 5.a: Document each outcome of the steps defined in Step 1: Define the Evaluation Scope, Step 2: Explore the Target Website, Step 3: Select a Representative Sample, and Step 4: Audit the Selected Sample.

Documenting the outcomes for each of the previous steps (including all sub-sections) is essential to ensure transparency of the evaluation process, replicability of the evaluation results, and justification for any statements made based on this evaluation. This documentation does not need to be public (the level of confidentiality is usually determined by the evaluation commissioner).

Documenting the outcomes for each step includes at least the following:

• About the Evaluation
  • Name of the evaluator
  • Name of the evaluation commissioner
  • Date for the evaluation (completion date or duration period)
• Evaluation Scope
  • Scope of the website defined in Step 1.a: Define the Scope of the Website
  • Conformance target defined in Step 1.b. Define the Conformance Target
  • Accessibility support baseline defined in Step 1.c: Define an Accessibility Support Baseline
  • Evaluation methods to be used, if any, defined in Step 1.d: Define Evaluation Methods to be Used (Optional)
  • Additional requirements, if any, defined in Step 1.e: Define Additional Evaluation Requirements (Optional)
• Website Exploration
  • Web technologies relied upon identified in Step 2.d: Identify Web Technologies Relied Upon
  • Optional: Common web pages identified in Step 2.a: Identify Common Web Pages of the Website
  • Optional: Essential functionality identified in Step 2.b: Identify Essential Functionality of the Website
  • Optional: Variety of web page types identified in Step 2.c: Identify the Variety of Web Page Types
  • Optional: Other relevant web pages identified in Step 2.e: Identify Other Relevant Web Pages
• Representative Sample
  • Common web pages selected in Step 3.a: Include Common Web Pages of the Website
  • Other relevant web pages selected in Step 3.b: Include Other Relevant Web Pages
  • Exemplar instances of web pages selected in Step 3.c: Include Exemplar Instances of Web Pages
  • Complete processes selected in Step 3.d: Include Complete Processes in the Sample
  • Randomly selected web pages and selection method used in Step 3.e: Include a Randomly Selected Sample
• Sample Audited
  • Evaluation outcomes from Step 4.a: Check for Each Success Criterion
  • Evaluation outcomes from Step 4.b: Check for Complete Procceses
  • Evaluation outcomes from Step 4.c: Check for Accessible Alternatives
  • Evaluation outcomes from Step 4.d: Check for Non-Interference
  • Evaluation outcomes from Step 4.e: Compare Structured and Random Samples

Note: Depending on the desired granularity of the report documentation, the outcomes of Step 4: Audit the Selected Sample can be aggregated over the entire sample. That is, for every WCAG 2.0 Success Criterion applicable (according to Step 1.b. Define the Conformance Target) and the remaining four WCAG 2.0 conformance requirements, the evaluator indicates whether each is met on every web page and web page state in the sample (selected in Step 3: Select a Representative Sample), or provides at least one example for each conformance requirement and WCAG 2.0 Success Criterion not met (without having an accessible alternative).

Depending on any additional evaluation requirements established in Step 1.e: Define Additional Evaluation Requirements (Optional) there may be additional evaluation outcomes to report. For example, an evaluation commissioner may request a report indicating every failure occurrence for every web page and web page state in the selected sample, more information about the nature and the causes of the identified failures, or repair suggestions to remedy the failures.

Step 5.b: Record the Evaluation Specifics (Optional)

Methodology Requirement 5.b: Archive the web pages and web page states audited, and record the evaluation tools, web browsers, assistive technologies, other software, and methods used to audit them (Optional).

While optional, it is good practice for evaluators to keep record of the evaluation specifics, for example to support conflict resolution in the case of dispute. This includes archiving the web pages and web page states audited, and recording the evaluation tools, web browsers, assistive technologies, other software, and methods used to audit them. This recording is typically kept internal and not shared by the evaluator unless otherwise agreed on in Step 1.e: Define Additional Evaluation Requirements (Optional).

Records of the evaluation specifics could include any of the following:

• Copies of the files and resources of the web pages and web page states;
  Note: Some tools can save the dynamically generated or modified content (DOM) as displayed during the evaluation rather than the initial content of the files and resources, which is often different;
• Screenshots (screen grabs) of the web pages and web page states;
• Description of the path to locate the web pages and web page states, especially when they are part of a process;
• Description of the settings, input, and actions used to generate or arrive to the web pages and web page states. Specific test credentials (useids, etc.) required to replicate a unique data set or workflow;
• Names and versions of the evaluation tools, web browsers and add-ons, assistive technology, and other software used;
• The methods, procedures, and techniques used to evaluate conformance to WCAG 2.0.

This recording may apply globally for the entire evaluation, to individual web pages, or to individual checks carried out within the audited web pages and web page states. A table or grid may be useful to record what was used for the different web pages and web page states audited.

Note: Records of the evaluation specifics may include sensitive information such as internal code, passwords, and copies of data. They may need particular security and privacy precautions.

Step 5.c: Provide an Evaluation Statement (Optional)

Methodology Requirement 5.c: Provide an statement describing the outcomes of the conformance evaluation (Optional).

Reminder: In the majority of situations, using this methodology alone does not result into WCAG 2.0 conformance claims for the target websites; see Relation to WCAG 2.0 Conformance Claims for more background.

Website owners may wish to make public statements about the outcomes from evaluations following this methodology. This can be done when at least every non-optional methodology requirement is satisifed, the conformance target defined in Step 1.b. Define the Conformance Target is satisfied by all web pages and web page states audited (in Step 4: Audit the Selected Sample), and the website owner commits to ensuring the validty and maintaining the accuracy of the evaluation statement made.

An evaluation statement according to this methodology includes at least the following information:

1. Date of when the evaluation statement was issued;
2. Guidelines title, version and URI: "Web Content Accessibility Guidelines 2.0 at http://www.w3.org/TR/WCAG20/";
3. Conformance level evaluated: Level A, AA or AAA, as defined in Step 1.b. Define the Conformance Target;
4. Definition of the website as defined in Step 1.a: Define the Scope of the Website;
5. Web technologies relied upon as identified in Step 2.d: Identify Web Technologies Relied Upon;
6. Accessibility support baseline as defined in Step 1.c: Define an Accessibility Support Baseline.

Evaluation statements according to this methodology can also be made when only partial conformance to WCAG 2.0 has been achieved. In such cases the evaluation statements have to also include the following information:

7. Website areas that do not conform to WCAG 2.0;
8. Reason for not conforming to WCAG 2.0: "third-party content" or "lack of accessibility support for languages";

Step 5.d: Provide an Aggregated Score (Optional)

[Review Note: Feedback on this section is particularly welcome. Specifically, how useful is the aggregated score for you, for example, to help communicate progress over time to website owners? What suggestions do you have for improving the concept of a score in this methodology.]

Methodology Requirement 5.d: Provide an Aggregated score (Optional).

While aggregated scores provide a numerical indicator to help communicate progress over time, there is currently no single widely recognized metric that reflects the required reliability, accuracy, and practicality. In fact, aggregated scores can be misleading and do not provide sufficient context and information to understand the actual accessibility of a website. A [Draft] W3C Research Report on Web Accessibility Metrics provides more background for different approaches and limitations of scoring metrics.

This methodology proposes two simple scoring approaches to select from. Other scoring mechanisms can be used as well. In any case, the scoring approach used has to be clearly indicated along with the score whenever such a score is provided. It is also recommended that the scoring approach is documented and made available to the evaluation commissioner along with the report to facilitate transparency and repeatability.

Aggregated score per website

This score calculates a ratio over the collective sample. That is, WCAG 2.0 Success Criteria are counted once for the entire sample as follows:

1. List all WCAG 2.0 Success Criteria relevant to the evaluation as defined in Step 1.b. Define the Conformance Target
2. Mark WCAG 2.0 Success Criteria that have not been satisfied on any web page or web page state in the sample selected in Step 3: Select a Representative Sample
3. Mark WCAG 2.0 Success Criteria that have been satisfied on all web pages and web page states in the sample selected in Step 3: Select a Representative Sample
4. Mark the remaining WCAG 2.0 Success Criteria as not present in any web page or web page state in the sample selected in Step 3: Select a Representative Sample
5. Divide the number of WCAG 2.0 Success Criteria marked satisfied by the number of WCAG 2.0 Success Criteria marked satisfied and not satisfied

This particular score sensitive towards failures but insensitive towards their relative frequency. Specifically, a failure to satisfy any WCAG 2.0 Success Criterion on any web page or web page state in the sample is directly reflected in the count. However, recurring failures do not influence the count any further.

Aggregated score per web page

This score calculates a ratio over the individual web pages. That is, WCAG 2.0 Success Criteria are counted on every web page in the sample as follows:

1. List all WCAG 2.0 Success Criteria relevant to the evaluation as defined in Step 1.b. Define the Conformance Target
2. For every web page and web page state in the sample selected in Step 3: Select a Representative Sample:
   • Mark the WCAG 2.0 Success Criteria that have not been satisfied
   • Mark the WCAG 2.0 Success Criteria that have been satisfied
   • Mark the WCAG 2.0 Success Criteria that are not present
3. Divide the total number of WCAG 2.0 Success Criteria marked satisfied (on all web pages and web page states) by the total number of WCAG 2.0 Success Criteria marked satisfied and not satisfied (on all web pages and web page states)

This particular score is more sensitive towards the frequency of failures but requires more effort to calculate. For example, a single occurence of a failure will have smaller effect on the score while repeated failures will have stronger effect. However, the score does not consider the relative impact of the failures on accessibility. For example, a single failure will yield a high score even if that one failire is an inaccessible CAPTCHA that prevents access to the entire website.

Step 5.e: Provide Machine-Readable Reports (Optional)

Methodology Requirement 5.e: Provide machine-readable reports of the evaluation results (Optional).

Machine-readable reports facilitate processing the evaluation results by authoring, web accessibility evaluation tools, and quality assurance tools. The Evaluation and Report Language (EARL) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata from WCAG 2.0 to learn more about uses of metadata, including machine-readable reports, such as EARL.

Appendix A: Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Gavin Evans; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; David MacDonald; Mary-Jo Mueller; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

QA-FRAMEWORK

Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/

RFC3986

Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt

RFC2616

Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt

UWEM

Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/

WCAG20

Caldwell B, Cooper M, Guarino Reid L, Vanderheiden G, eds (2008). Web Content Accessibility Guidelines 2.0. W3C. Available at: http://www.w3.org/TR/WCAG20/

Appendix C: Document Changes

Changes since the Public Working Draft of 26 February 2013 are documented in:

• Disposition of comments from 26 February 2013 Working Draft
• Disposition of comments from 29 November 2013 Editor Draft