W3C logoWeb Accessibility Initiative (WAI) logo

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

Evaluating Web Sites for Accessibility

Page Contents

1. Introduction

This document outlines approaches for preliminary review Web site accessibility, and for evaluation of conformance to the Web Content Accessibility Guidelines 1.0. While it does not provide checkpoint-by-checkpoint testing techniques it does include general procedures and tips for evaluation during development of Web sites, and for monitoring of established Web sites. Other resources will be developed for in-depth compliance testing. The measures described here are intended to supplement an organization's existing procedures for content management and quality assurance on their Web sites. For information about why making Web sites accessible is important read the Introductions on the WAI Resources page.

There are a variety of tools and approaches for evaluating Web site accessibility. No single evaluation tool yet provides comprehensive information or captures all problems with regard to the accessibility of a site; therefore evaluation involves a combination of approaches. Goals for evaluating Web sites vary, and require different approaches to meet those goals:

Note: This document may be updated periodically to reflect clarifications or improvements in evaluation approaches, including increasing incorporation of semi-automated evaluation tools as the capability of such tools improves, and corresponding reduction of manual evaluation steps; as well as incorporation of more precise checkpoint-by-checkpoint testing techniques as these become available. Future versions of this document may include an evaluation checklist and/or be published as a W3C Note. Version date and information on where to send comments is available at the end of the document. Mention of specific evaluation tools in this document does not constitute endorsement by W3C, and use of either the preliminary review or conformance evaluation method does not guarantee conformance under the laws or regulations of any specific government.

2. Preliminary Review

A preliminary review may help to quickly identify the scope of problems on a Web site. However, the preliminary review will not catch all of the problems on a site and should not be used to determine conformance level. A preliminary review does not include perspectives from a variety of users with disabilities nor does it touch or test every aspect of a site.

A preliminary review combines some manual checking of representative pages on a Web site, along with the use of several semi-automatic accessibility checkers. Reviewers do not need to know Web mark-up languages, but should be able to download software and familiarize themselves with some online tools, and change certain settings on their browser.

To conduct a preliminary review, complete all five steps below.

  1. Select a representative sampling of different kinds of pages from the Web site to be reviewed; must include all pages on which people are more likely to enter your site. ("welcome page" etc.) NOTE: on web sites with database driven dynamically generated web content, generate broadly representative samples, freeze, and test the output.
  2. Use a graphical user interface (GUI) browser(such as Internet Explorer, Netscape Navigator, or Opera) and examine the selection of pages while adjusting the browser settings as follows (NOTE: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.) (NOTE: Some of these manual checks can be performed by changing settings or preferences in the browser, some may require changes to operating system settings, and some may require additional software.)
    1. turn off images, and check whether appropriate alternative text is available.
    2. turn off the sound, and make sure audio content is still available through text equivalents.
    3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
    4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
    5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
    6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
  3. Use a voice browser (such as Home Page Reader) or a text browser (such as Lynx) and examine the Web site while answering these questions (NOTE: experienced users of screen readers may substitute a screen reader for a voice or text browser, but if blind, may need a sighted partner to compare information available visually; if sighted, listen to it with eyes closed, then open eyes and confirm whether the information is equivalent)
    1. is equivalent information available through the voice or text browser as is available through the GUI browser?
    2. is the information presented in a meaningful order if read serially?
  4. Use two general accessibility evaluation tools and note any problems indicated by the tools.
  5. Summarize results
    1. summarize the types of problems encountered, as well as best practices that should be continued or expanded on the site
    2. indicate the method by which problems were identified, and clearly state that this was not a full conformance evaluation
    3. recommend follow-up steps, including full conformance evaluation which includes validation of markup and other tests, and ways to address any problems identified.

3. Conformance Evaluation to WCAG 1.0

A comprehensive evaluation combines semi-automatic, manual, and user testing of accessibility features. Comprehensive evaluations require familiarity with Web mark-up languages; initial downloading and/or training on a variety of evaluation tools and approaches; configuration of browser settings; and coordination with reviewers with a variety of disabilities. Evaluation with users is important as it helps to identify problems in how the technical solutions are being applied.

A properly conducted comprehensive evaluation can identify potentially major problems during the development phase for a new site; determine what level of accessibility a Web site meets; and/or provide assurance that a Web site meets a required level of accessibility.

A comprehensive evaluation includes all of the steps below except those that are explicitly identified as alternatives or optional (NOTE: while identifying the page selection is a key first step, and summarizing and reporting the results of evaluation is the logical conclusion, the order of the intervening steps is not crucial).

  1. Identify and disclose scope of site to be evaluated and the targeted conformance level for the evaluation (NOTE: disclosure should be internal to the organization during evaluation; if conformance is claimed publicly, disclose externally (e.g. on the Web site)):
    1. identify and disclose the target conformance level of WCAG 1.0.
    2. identify and disclose a page selection for manual and user testing which includes at least one of each different type of page on the site, and all pages on which people are more likely to enter your site. NOTE: there are special considerations for web sites with database driven dynamically generated web content
    3. identify and disclose the entire Web site including all pages at a base URL for automatic and semi-automatic evaluation; NOTE: if testing of the entire site is not feasible (e.g. because of its unusual size or dynamic nature) identify an expanded page selection, to be clearly explained and disclosed on the Web site. Suggestions for inclusions in this expanded page selection: pages from different sections of the Web site; pages representing different "look & feel"; pages representing different development tools and processes including those generated from databases; pages produced under different guidelines; "contact us" pages; pages critical to your business; etc. If any area of a site is excluded from evaluation, be sure to disclose this information.
  2. Semi-automatic and automatic evaluation
    1. Validate markup including syntax and style sheets, using all applicable validators, on page selection. Run at least one validation tool across entire Web site
    2. Use at least two accessibility evaluation tools on page selection and run at least one tool across entire Web site. Note any problems indicated by the tools. A list of tools is available at Evaluation, Repair, and Transformation Tools for Web Content Accessibility.
  3. Manual evaluation
    1. Examine page selection using relevant checkpoints from the Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0. NOTE Relevant can mean: checkpoints that cannot be evaluated by automatic or semiautomatic tools; checkpoints that actually apply to the site (e.g. if site contains no audio content, skip those); and, as a minimum, those checkpoints that apply to the level of conformance you are evaluating.
    2. Examine page selection with graphical user interface (GUI) browsers: select at least three different configurations from among the following variables: different graphical user interface browsers (Internet Explorer, Netscape, Opera), in different versions (latest, older), running on different platforms (Windows, Linux, Mac) and making the following adjustments (NOTE: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.) :
      1. turn off images, and check whether appropriate alternative text is available.
      2. turn off the sound, and make sure audio content is still available through text equivalents.
      3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
      4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
      5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
      6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
      7. also examine page with scripts, style sheets, and applets not loaded
    3. Examine page selection with one text browser (such as Lynx) AND one voice browser (such as Home Page Reader), and answer the following questions:
      1. With text browser
        • is equivalent information and function (e.g. links and scripted events) available through the text browser as is available through the GUI browser?
        • is the information presented in a meaningful order when read serially?
      2. With voice browser
        (NOTE: For settings where there is limited choice of assistive technologies,also perform a manual evaluation of the Web site with those assistive technologies; for instance, JAWS is the only screen reader translated into Danish, and therefore in Denmark a trained evaluator should evaluate the Web site using JAWS.)
        • is equivalent information available through the voice browser as is available through the GUI browser?
        • is the information presented in a meaningful order when spoken serially?
    4. Read over the pages: is the text clear and simple to the extent appropriate for the purpose of the Web site? (For English sites, consider using Clear and Appropriate Language and Design (CLAD) test.) [NOTE: this is also an appropriate question to ask people participating in usability testing of accessibility features].
  4. Usability testing of accessibility features
    • Have people with different disabilities, different levels of technical expertise, and different levels of familiarity with the site, using a variety of assistive technologies and adaptive strategies, review page selection and explore freely across entire Web site [NOTE: Sometimes this is done in the context of a testing laboratory, but it is also valuable for people to evaluate the site in their typical Web environment. Different levels of user technical experience and familiarity with the site are important, and note that these will change over time, as people participate in reviewing the site.]
    • Ask testers to try to find answers to the most common questions for which people visit the Web site. Note areas where it is difficult or impossible the use the Web site.
  5. Summarize and follow-up
    1. Summarize any problems and best practices identified for each page type and a representative URL, and method by which they were identified
    2. Recommend follow-up steps, potentially including:
      • repair of accessibility barriers identified through conformance evaluation process. NOTE: for evaluations using "expanded page selection" instead of "entire Web site," apply what you've learned to other pages
      • expanding best practices on site
      • ongoing maintenance and monitoring of site

4. Considerations for Specific Contexts

Evaluation during the development process

Evaluation during the development process is essential. It can sometimes be difficult, as both in-house and subcontracted Web developers sometimes prefer to establish the site design and demonstrate their progress before getting feedback. However, accessibility issues identified early are easier to correct and avoid. Effective evaluation during the design period can include:

Ongoing monitoring

To maximize likelihood that a Web site will maintain a given conformance level in the future, the following provisions should be in place:

NOTE: A full conformance evaluation is not necessarily required at each milestone in an ongoing review process. Steps like repeated user testing may only be required after major template or content changes.

Evaluation of legacy sites

Occasionally Web sites that are "frozen" (legacy; no longer actively maintained) are found to have substantial accessibility problems. It can be difficult to determine how to address these. It is helpful to:

Evaluation of dynamically generated Web pages

Dynamically generated pages are usually assembled from one or more templates that provide common layout and navigation features, and content provided automatically from a database or other content management system. To achieve full conformance the accessibility of both templates and generated content must be evaluated. It is not sufficient to evaluate only templates: content may also contain markup, or be required to contain markup in order to be accessible. Things to consider:

  1. Templates
    • evaluate all templates
    • if templates are generated by authoring software evaluate the capability of the authoring tool to include accessible features (see Selecting Authoring Tools), e.g.:
      • does the generated tab order from the template allow getting to the generated text content effectively?
  2. Content
    • if all dynamic content cannot be evaluated, generate broadly representative samples, capture content, and test the output;
    • evaluate the capability of the content management system to store and generate accessibility information (see Selecting Authoring Tools);
      • are images supplied with alt texts, and if needed, the longdesc?
      • do generated data tables have accessibility aids (e.g captions, id on <th> header cells, etc.)?
      • if generated video, is it captioned?
      • if generated audio narrative, is textual equivalent available?
  3. Templates and content combined
    • for pages that are generated as a result of a query to a database, the source generated as the page is rendered should be captured and fed back into the evaluation tool -- [may require operator intervention] ;