[Draft] Conformance Evaluation of Web Sites for Accessibility

Note: This document is a draft [see change log in progress] and should not be referenced or quoted under any circumstances. This document is under development by the Education and Outreach Working Group (EOWG), and will be offered to other W3C groups and the public for review.

Introduction

A conformance evaluation combines automatic, semi-automatic, and manual testing of accessibility features. It is a technical assessment of the conformance to the Web Content Accessibility Guidelines (WCAG) 1.0 and does not include evaluation with users which is described in a separate section. Conformance evaluations require familiarity with Web mark-up languages; initial downloading and/or training on a variety of evaluation tools and approaches; configuration of browser settings; and experience in Web accessibility.

Scope of Conformance Evaluations

A properly conducted conformance evaluation can identify potentially major problems during the development phase for a new site; determine what level of accessibility a Web site meets; and/or provide assurance that a Web site meets a required level of accessibility. However, it does not include evaluation with users which is important to identify problems in how the technical solutions are being applied. Comprehensive evaluations of Web sites combine technical assessments and usability testing of accessibility features.

Steps To Conduct a Conformance Evaluation

A conformance evaluation includes all of the steps below except those that are explicitly identified as alternatives or optional. Note: while identifying the page selection is a key first step, and summarizing and reporting the results of evaluation is the logical conclusion, the order of the intervening steps is not crucial.

1. Identify and Disclose the Scope of the Conformance Evaluation

Identify and disclose scope of site to be evaluated and the targeted conformance level for the evaluation. Note: disclosure should be internal to the organization during evaluation; if conformance is claimed publicly, disclose externally (e.g. on the Web site).

1. identify and disclose the target conformance level of WCAG 1.0.
2. identify and disclose a page selection for technical testing which includes at least one of each different type of page on the site, and all pages on which people are more likely to enter your site. Note: there are special considerations for web sites with database driven dynamically generated web content
3. identify and disclose the entire Web site including all pages at a base URL for automatic and semi-automatic evaluation; Note: if testing of the entire site is not feasible (e.g. because of its unusual size or dynamic nature) identify an expanded page selection, to be clearly explained and disclosed on the Web site. Suggestions for inclusions in this expanded page selection: pages from different sections of the Web site; pages representing different "look & feel"; pages representing different development tools and processes including those generated from databases; pages produced under different guidelines; "contact us" pages; pages critical to your business; etc. If any area of a site is excluded from evaluation, be sure to disclose this information.

2. Use Automated or Semi-Automated Web Accessibility Evaluation Tools

1. Validate markup including syntax and style sheets, using all applicable validators, on page selection. Run at least one validation tool across entire Web site
   • HTML Validation service
   • HTML Tidy
   • CSS Validation service
   • MathML Validator
2. Use at least two accessibility evaluation tools on page selection and run at least one tool across entire Web site. Note any problems indicated by the tools. A list of tools is available at Evaluation, Repair, and Transformation Tools for Web Content Accessibility.

3. Carry Out Manual Evaluation of the Selected Web Page Sample

1. Examine page selection using relevant checkpoints from the Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0. Note: Relevant can mean checkpoints that cannot be evaluated by automatic or semi-automatic tools; checkpoints that actually apply to the site (e.g. if site contains no audio content, skip those); and, as a minimum, those checkpoints that apply to the level of conformance you are evaluating.
2. Examine page selection with graphical user interface (GUI) browsers: select at least three different configurations from among the following variables: different graphical user interface browsers (such as Internet Explorer, Mozilla FireFox, Netscape Navigator, Opera, Safari, or others), in different versions (latest, older), running on different platforms (Windows, Linux, Mac) and making the following adjustments. Note: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability:
   1. turn off images, and check whether appropriate alternative text is available.
   2. turn off the sound, and make sure audio content is still available through text equivalents.
   3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
   4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
   5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
   6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
   7. also examine page with scripts, style sheets, and applets not loaded
3. Examine page selection with one text browser (such as Lynx) AND one voice browser (such as Home Page Reader), and answer the following questions:
   1. With text browser
      • is equivalent information and function (e.g. links and scripted events) available through the text browser as is available through the GUI browser?
      • is the information presented in a meaningful order when read serially?
   2. With voice browser (Note: For settings where there is limited choice of assistive technologies, also perform a manual evaluation of the Web site with those assistive technologies; for instance, JAWS is the only screen reader translated into Danish, and therefore in Denmark a trained evaluator should evaluate the Web site using JAWS.)
      • is equivalent information available through the voice browser as is available through the GUI browser?
      • is the information presented in a meaningful order when spoken serially?
4. Read over the pages: is the text clear and simple to the extent appropriate for the purpose of the Web site? (For English sites, consider using Clear and Appropriate Language and Design (CLAD) test.)

4. Summarize and Reporte Evaluation Findings

Summarize any problems and best practices identified for each page type and a representative URL, and method by which they were identified. Recommend follow-up steps, potentially including:

• repair of accessibility barriers identified through conformance evaluation process. Note: for evaluations using "expanded page selection" instead of "entire Web site", apply what you've learned to other pages
• expanding positive aspects on site
• ongoing maintenance and monitoring of site