Accessibility Conformance Testing for W3C

Use the discussion page for comments.

Related pages:

• (Proposed) Accessibility Conformance Testing (ACT) Task Force Work Statement
• ACT Deliverables
• ACT project plan

Introduction

There are currently many products available which aid their users in testing for conformance of accessibility standards such as WCAG 2.0. As the web develops and grows in both size and complexity, these tools are essential for managing the accessibility of resources available on the web. The volume of information and services organizations provide on the web make it often impractical to manually test for accessibility.

Experts often disagree on the exact way WCAG should be interpreted, which leads to confusion and frustration among those responsible for the quality of the website. This problem becomes even bigger when it comes to automated and assistive accessibility test tools, where the developer of those tools attempt to make conformance claims about web content, without knowing the context in which it appears.

Problem Description

Accessibility test tools give a variety of often contradictory results. These contradictions cause confusion and disagreement about what exactly is and what is not an accessibility error. These varying results in tools have a number of causes, such as:

• WCAG is not interpreted in a consistent way
• Tools fail to anticipate the many ways in which web technologies are used to cause or overcome accessibility issues
• Accessibility support in user agents and assistive technologies is not taken into account
• Results are reported in ways that make them difficult to compare

This can result in wildly different outcomes, depending on which tool is used in accessibility testing. This leads to distrust of accessibility test tools all together, which could result in organizations being less inclined to use such tools. Organizations using different tools can cause friction and mistrust of a web developer's expertise. This reflects badly on the field of web accessibility as a whole. All of this hinders in making content and services more accessible to people with disabilities.

Goals

1. Create greater consensus on how conformance to accessibility standards should be tested. By working out those parts that are generally agreed upon, and identifying areas which require further discussion, such as by the WCAG Working group.
2. Improve consistency between accessibility testers, be it expert testing, or testing through accessibility test tools.
3. Develop a set or rules that can identify specific non-conformance issues according to accessibility standards.
4. Ensure continued development of these rules in landscape of continuously changing technology standards, tools and assistive technologies.

Scope

The ACT Task Force will develop an extensible ruleset, with rules for WCAG Level AA testing for HTML, CSS and ARIA at the core. Because the framework is extensible, organizations can build on top of this core ruleset, with rules that they may have a particular interest in.

For example, the EPUB standard has it's own accessibility guidelines. Applicable rules from the WCAG / HTML rules can be used and new rules specific for EPUB can be added for an EPUB ruleset. Similarly, companies with their own accessibility policies can write custom rules, such as requiring a single h1 on every page.

In short, ACT rules can be created for:

• Multiple accessibility standards (such as WCAG 2.0, ARIA Authoring Practices), including advanced versions as these standards evolve
• Different web technologies and digital publishing technologies (such as HTML, ARIA, EPUB, PWP)
• Fully automated and semi-automated test tools; The framework would also be extensible, allowing for integrate of results from manual accessibility tests. (Such as through the WCAG-EM Report Tool.)

Audiences

The primary audience of ACT rules are developers of accessibility test tools. They implement rules into their products. The ACT rules serve as documentation and requirements for these tools.

A secondary audience of these rules are accessibility experts. They often assist in setting an organization's accessibility policy, so knowing what rules do, and being able to write rules will be important to them.

Users of accessibility tools (web developers, content authors, QA testers, etc.) are not expected to read the ACT rules. These rules add a layer of complexity to accessibility testing that is likely to make accessibility more complicated for them. It is up to accessibility tool vendors to find fitting solutions to communicate issues to their users.

Initial Rules

Rule development for ACT will initially focus on automated testing for WCAG 2.0, level A + AA using HTML, CSS and ARIA. Such rules are already being developed by Auto-WCAG, and they will continue to do so with the support of the ACT Task Force.

Use Existing Rulesets

Many organizations already have their own rulesets, either as policy or as part of their tools. The ACT Task Force will work with organizations interested in contributing their rules to the ACT Rule Suite. These can be WCAG rules, but also accessibility best practice, and rules designed to catch potential accessibility issues. Specific acceptance requirements for this will be worked out later.

Some organizations may not find it beneficial to share the rules in their accessibility tools. However, companies that compete on features other than rule coverage, benefit from showing their customers they conform and develop standards for accessibility testing. It also benefits users of these tools to know that what the tool does, is well thought out and maintained by a community of experts.

ACT Deliverables

The result of this initiative will be the development of five products, in three areas:

1. ACT Framework: A W3C Recommendation defining how to write test rules for accessibility conformance testing. The framework on top of which conformance rules can be developed.
2. ACT Benchmark
   1. ACT Benchmark Method: A description of how to avoid (unexpected) false positives.
   2. ACT Benchmark Tool: An implementation of the benchmark method
3. ACT Rule Suite
   1. ACT Rule Suite Repository: A collection of rules that are proven to return results sufficiently accurate for (non) conformance testing to WCAG.
   2. ACT Rule Suite Frontend: A website or web app where documentation can be found about exactly what the ACT rules in the repository do.

A detailed description is available in ACT Deliverables.

Organization

ACT Task Force

To run this project we propose the creation of a new Task Force within the WCAG Working Group. The work will be done by members of this task force, the Accessibility Conformance Testing (ACT) Task Force. The ACT Task Force will work under the direction of the WCAG Working Group.

See the (Proposed) Accessibility Conformance Testing (ACT) Task Force Work Statement for more information.

Auto-WCAG CG

Part of the success of this project depends on the continued work of Auto-WCAG (or a similar initiative). We aim to keep Auto-WCAG running in a way similar to how it has been running for the past two years, iteratively developing rules. But with the addition of having a stronger foundation and control through the ACT Task Force.

Project Outline

This project will be developed in three phases. Each phase will be completed in one year. The phases are designed to solve parts of the goal outlined in this document by developing certain products. Each phase will therefore result in concrete products ready for public use.

Phase 1: Definition

The first phase of this plan will be focused on the following activities:

• Create requirements for ACT Framework
• Develop the first draft of ACT Framework
• Get feedback from stakeholders on this draft
• Use this feedback to get the draft ready for test implementations
• Work out requirements for the ACT Benchmark and the ACT Rule Suite

Milestone 1:

1. Last Call ACT Framework
2. Requirements for ACT Benchmark
3. Requirements for ACT Suite

Phase 2: Implementation

• Call for public feedback on ACT Framework
• Work with rule designers to develop (test) implementations of the ACT Framework
• Update ACT Framework based on feedback from the implementation
• Develop a benchmark method to ensure the quality of ACT rules
• Make the first ACT ruleset of rules available

Milestone 2:

1. Enter Candidate Rec stage for ACT Framework
2. Complete ACT Benchmark (standalone Note or part of the ACT Framework)
3. Listing of first set of rules for ACT Suite

Phase 3: Consolidation

• Develop an implementation ACT Rule benchmark
• Test the quality of the ACT Rules developed in phase 2 using the benchmark
• Work with rule designers to ensure the ACT Rules can pass the benchmark
• Refine the ACT Framework based on lessons learned and public feedback
• Organize a group to maintain the ACT Rule Suite for the future

Milestone 3:

1. Recommendation ACT Framework
2. ACT Rule Suite on the W3C website
3. Organization to maintain the ACT Rule Suite

Relation to other Technologies and Standards

WCAG 2.0 and beyond

Consistent and accurate interpretation is one of the important goals of this project. For this work we will have to work closely with the WCAG Working Group to ensure the way WCAG is intended to be interpreted.

The primary focus of this project will be to create a framework, on top of which ACT Rules can be developed. An ACT Rule will likely have to be tied to a specific technology and accessibility standard. But it is important for the framework to be independent of this. This ensures that the work can be maintained for future versions of WCAG, as well as for other digital accessibility standards.

WCAG Techniques

At first glance, ACT Rules might seem like the techniques that were created for WCAG 2.0. The failure techniques in particular have a few things in common with ACT Rules. There are however a few important aspects to techniques that make them unsuitable for a foundation of this project:

• The language is not written for unambiguous interpretation
• There is insufficient details in techniques for consistent implementation
• They have a strong reliance on human analyses
• Techniques often go beyond setting the minimal requirements for WCAG

All of these stem from a difference in goals. Techniques are designed to teach developers about accessibility, not as a method of conformance testing.