skip navigation barsW3C logoWeb Accessibility Initiative (WAI) logo
W3C Index - W3C Search - W3C Translations
WAI Resources - WAI Site Map - About WAI

you are here Evaluation Suite:
Evaluation Template
Evaluation Tools
Review Teams
Implementation Suite +
Training Suite +

Evaluating Web Sites for Accessibility

introduction - preliminary review - conformance evaluation - specific contexts - during development - ongoing monitoring - legacy sites

1. Introduction

Note: This document may be updated periodically to reflect clarifications or improvements in evaluation approaches, including increasing incorporation of semi-automated evaluation tools as the capability of such tools improves, and corresponding reduction of manual evaluation steps; as well as incorporation of more precise checkpoint-by-checkpoint testing techniques as these become available. Future versions of this document may include an evaluation checklist and/or be published as a W3C Note. Version date and information on where to send comments is available at the end of the document. Mention of specific evaluation tools in this document does not constitute endorsement by W3C, and use of either the preliminary review or conformance evaluation method does not guarantee conformance under the laws or regulations of any specific government.�

This document outlines approaches for preliminary review Web site accessibility, and for evaluation of conformance to the Web Content Accessibility Guidelines 1.0. While it does not provide checkpoint-by-checkpoint testing techniques it does include general procedures and tips for evaluation during development of Web sites, and for monitoring of established Web sites. Other resources will be developed for in-depth compliance testing. The measures described here are intended to supplement an organization's existing procedures for content management and quality assurance on their Web sites. For information about why making Web sites accessible is important read the Introductions on the WAI Resources page.

There are a variety of tools and approaches for evaluating Web site accessibility. No single evaluation tool yet provides comprehensive information or captures all problems with regard to the accessibility of a site; therefore evaluation involves a combination of approaches. Goals for evaluating Web sites vary, and require different approaches to meet those goals:

2. Preliminary Review

A preliminary review may help to quickly identify the scope of problems on a Web site. However, the preliminary review will not catch all of the problems on a site and should not be used to determine conformance level. A preliminary review does not include perspectives from a variety of users with disabilities nor does it touch or test every aspect of a site.

A preliminary review combines some manual checking of representative pages on a Web site, along with the use of several semi-automatic accessibility checkers. Reviewers do not need to know Web mark-up languages, but should be able to download software and familiarize themselves with some online tools, and change certain settings on their browser.

To conduct a preliminary review, complete all five steps below.

  1. Select a representative sampling of different kinds of pages from the Web site to be reviewed; must include all pages on which people are more likely to enter your site. ("welcome page" etc.) NOTE: on web sites with database driven dynamically generated web content, generate broadly representative samples, freeze, and test the output.
  2. Use a graphical user interface (GUI) browser (such as Internet Explorer, Netscape Navigator, or Opera) and examine the selection of pages while adjusting the browser settings as follows (NOTE: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.) (NOTE: Some of these manual checks can be performed by changing settings or preferences in the browser, some may require changes to operating system settings, and some may require additional software.)
    1. turn off images, and check whether appropriate alternative text is available.
    2. turn off the sound, and make sure audio content is still available through text equivalents.
    3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
    4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
    5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
    6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
  3. Use a voice browser (such as Home Page Reader) or a text browser (such as Lynx) and examine the Web site while answering these questions (NOTE: experienced users of screen readers may substitute a screen reader for a voice or text browser, but if blind, may need a sighted partner to compare information available visually; if sighted, listen to it with eyes closed, then open eyes and confirm whether the information is equivalent)
    1. is equivalent information available through the voice or text browser as is available through the GUI browser?
    2. is the information presented in a meaningful order if read serially?
  4. Use two accessibility evaluation tools and note any problems indicated by the tools, for example:
  5. Summarize results
    1. summarize the types of problems encountered, as well as best practices that should be continued or expanded on the site
    2. indicate the method by which problems were identified, and clearly state that this was not a full conformance evaluation
    3. recommend follow-up steps, including full conformance evaluation which includes validation of markup and other tests, and ways to address any problems identified.

3. Conformance Evaluation to WCAG 1.0

A comprehensive evaluation combines semi-automatic, manual, and user testing of accessibility features. Comprehensive evaluations require familiarity with Web mark-up languages; initial downloading and/or training on a variety of evaluation tools and approaches; configuration of browser settings; and coordination with reviewers with a variety of disabilities. Evaluation with users is important as it helps to identify problems in how the technical solutions are being applied.

A properly conducted comprehensive evaluation can identify potentially major problems during the development phase for a new site; determine what level of accessibility a Web site meets; and/or provide assurance that a Web site meets a required level of accessibility.

A comprehensive evaluation includes all of the steps below except those that are explicitly identified as alternatives or optional (NOTE: while identifying the page selection is a key first step, and summarizing and reporting the results of evaluation is the logical conclusion, the order of the intervening steps is not crucial).

  1. Identify and disclose scope of site to be evaluated and the targeted conformance level for the evaluation (NOTE: disclosure should be internal to the organization during evaluation; if conformance is claimed publicly, disclose externally (e.g. on the Web site)):
    1. identify and disclose the target conformance level of WCAG 1.0.
    2. identify and disclose a page selection for manual and user testing which includes at least one of each different type of page on the site, and all pages on which people are more likely to enter your site. NOTE: there are special considerations for web sites with database driven dynamically generated web content
    3. identify and disclose the entire Web site including all pages at a base URL for automatic and semi-automatic evaluation; NOTE: if testing of the entire site is not feasible (e.g. because of its unusual size or dynamic nature) identify an expanded page selection, to be clearly explained and disclosed on the Web site. Suggestions for inclusions in this expanded page selection: pages from different sections of the Web site; pages representing different "look & feel"; pages representing different development tools and processes including those generated from databases; pages produced under different guidelines; "contact us" pages; pages critical to your business; etc. If any area of a site is excluded from evaluation, be sure to disclose this information.
  2. Semi-automatic and automatic evaluation
    1. Validate markup including syntax and style sheets, using all applicable validators, on page selection. Run at least one validation tool across entire Web site
    2. Use at least two accessibility evaluation tools on page selection and run at least one tool across entire Web site . Note any problems indicated by the tools. The following are examples of evaluation tools, and a more extensive list is available at Evaluation, Repair, and Transformation Tools for Web Content Accessibility.
      • WAVE (an online accessibility assessment tool that flags any items on a Web page which should be examined for potential accessibility problems, and provides a description of what the problem might be)
      • Bobby (an online or downloadable accessibility checker which provides a semi-automated assessment of accessibility problems on a Web page or group of Web pages; it can identify many problems on sites, and lists problems which it is not able to evaluate automatically and which require manual review)
      • A-Prompt (a tool which identifies potential accessibility problems and provides guided editing to correct the problems)
  3. Manual evaluation
    1. Examine page selection using relevant checkpoints from the Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0. NOTE Relevant can mean: checkpoints that cannot be evaluated by automatic or semiautomatic tools; checkpoints that actually apply to the site (e.g. if site contains no audio content, skip those); and, as a minimum, those checkpoints that apply to the level of conformance you are evaluating.
    2. Examine page selection with graphical user interface (GUI) browsers: select at least three different configurations from among the following variables: different graphical user interface browsers (Internet Explorer, Netscape, Opera), in different versions (latest, older), running on different platforms (Windows, Linux, Mac) and making the following adjustments (NOTE: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.) :
      1. turn off images, and check whether appropriate alternative text is available.
      2. turn off the sound, and make sure audio content is still available through text equivalents.
      3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
      4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
      5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
      6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
      7. also examine page with scripts, style sheets, and applets not loaded
    3. Examine page selection with one text browser (such as Lynx) AND one voice browser (such as Home Page Reader), and answer the following questions:
      1. With text browser
        • is equivalent information and function (e.g. links and scripted events) available through the text browser as is available through the GUI browser?
        • is the information presented in a meaningful order when read serially?
      2. With voice browser
        (NOTE: For settings where there is limited choice of assistive technologies, also perform a manual evaluation of the Web site with those assistive technologies; for instance, JAWS is the only screen reader translated into Danish, and therefore in Denmark a trained evaluator should evaluate the Web site using JAWS.)
        • is equivalent information available through the voice browser as is available through the GUI browser?
        • is the information presented in a meaningful order when spoken serially?
    4. Read over the pages: is the text clear and simple to the extent appropriate for the purpose of the Web site? (For English sites, consider using Clear and Appropriate Language and Design (CLAD) test.) [NOTE: this is also an appropriate question to ask people participating in usability testing of accessibility features].
  4. Usability testing of accessibility features
  5. Summarize and follow-up
    1. Summarize any problems and best practices identified for each page type and a representative URL, and method by which they were identified
    2. Recommend follow-up steps, potentially including:
      • repair of accessibility barriers identified through conformance evaluation process. NOTE: for evaluations using "expanded page selection" instead of "entire Web site," apply what you've learned to other pages
      • expanding best practices on site
      • ongoing maintenance and monitoring of site

4. Considerations for Specific Contexts

Evaluation during the development process

Evaluation during the development process is essential. It can sometimes be difficult, as both in-house and subcontracted Web developers sometimes prefer to establish the site design and demonstrate their progress before getting feedback. However, accessibility issues identified early are easier to correct and avoid. Effective evaluation during the design period can include:

Ongoing monitoring

To maximize likelihood that a Web site will maintain a given conformance level in the future, the following provisions should be in place:

NOTE: A full conformance evaluation is not necessarily required at each milestone in an ongoing review process. Steps like repeated user testing may only be required after major template or content changes.

Evaluation of legacy sites

Occasionally Web sites that are "frozen" (legacy; no longer actively maintained) are found to have substantial accessibility problems. It can be difficult to determine how to address these. It is helpful to:

Evaluation of dynamically generated Web pages

Dynamically generated pages are usually assembled from one or more templates that provide common layout and navigation features, and content provided automatically from a database or other content management system. To achieve full conformance the accessibility of both templates and generated content must be evaluated. It is not sufficient to evaluate only templates: content may also contain markup, or be required to contain markup in order to be accessible. Things to consider:

  1. Templates
  2. Content
  3. Templates and content combined

Note: This draft WAI Resource developed by W3C/WAI's Education and Outreach Working Group (EOWG). We invite review and discussion. Please address your feedback to wai-eo-editors@w3.org, a mailing list with a public archive. Change log available.

Level Double-A conformance icon, Last updated 7 November 2002 by Judy Brewer. Editors: Judy Brewer and Chuck Letourneau, with assistance from participants of the EOWG.

Copyright� �� 2001-2002 W3C (MIT, INRIA, Keio ), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.