Eval Analysis

From Education & Outreach

Nav: EOWG wiki main page
Related pages: Web Accessibility Preliminary Evaluation, Eval in process

Tasks & Use Cases

The following table lists tasks, example user cases with audience, and focus for each document. (Note that we haven't decided whether or not to do a new doc on evaluation in the process.)


task example use cases prelim conform
overview		 WCAG-EM process  
Get general idea of accessibility of a webpage (major problems, some issues, maybe OK)
  1. Contractor: I've developed a prototype and I've been asked "is it accessible?" What can I do quickly to check?
  2. Website manager: I got a complaint from a user and want to get an idea of the scope of the accessibility problems on my website.
  3. Website manager: I'm checking out design companies to commission to re-do my website and I want to see if their website is accessible.
  4. Website manager: I commissioned a website that was suppose to meet WCAG. I've just received the first prototype and I want to do a quick check to see how it is on accessibility.
  5. Advocate: I encountered an accessibility barrier trying to use a website and I want to know if there are many other problems or not.
  6. Web content writer, journalist, blogger: I want my pages to be accessible in general so that I can reach the widest audience.
 mostly maybe some      
Get an overview of WCAG-EM
  1. Evaluation procurer: We are commissioning a WCAG conformance evaluation and want on overview of WCAG-EM, which we plan to reference in the Request for Tender/Proposals.
  2. @@
 maybe lots maybe just a little    
Thorough, authoratative evaluation of WCAG conformance
  1. Web development agency: Our services include providing a full accessibility audit of the new templates (CMS) and all webpages prior to launch.
  2. Accessibility champion or Quality assurance (in a large organization): I need to regularly evaluate new sections and applications before launch.
  3. Independent certification: We need to show that our expert audit service and user testing by people with disabilities meet recognised international standards.
  4. Accessibility Maintenance: We periodically check the accessibility of sections of our website to monitor that we continue to meet our certification status.
  5. Developer: I want to certify that my business website conforms to WCAG 2.0.
  6. Multi-site benchmark study: Our accessibility consulting firm has been hired to evaluate and report on the conformance of 25 governmental websites.
  7. Researcher: I research accessibility, and investigate accessibility issues on multiple sites
  8. Education: I teach students to become web designers, developers or accredited accessibility assessors
  9. Advocate: I need to be able to direct people (eg businesses, information providers) to an authorative resource that they can use to evaluate the conformance of their website to WCAG 2.0
   maybe at first mostly    
Evaluate throughout design and development process
  1. Design specification: We want an independent assessment of high level objects during early stage design planning. We can provide sample use cases, early page mock-ups, wireframes and simple prototypes.
  2. Development Support: Our company provides independent evaluation including from disabled people. This supports development of specific content, new applications and mobile. Reference to WCAG 2.0 AA and agreed elements of AAA.
  3. Developer: I want my sites to be professionally designed and to meet the standards baseline, including WCAG 2.0 AA. To do that effectively I want to review new sections and applications as I go along.
  4. Project manager, designer, developer: I see all these new resources about evaluating websites after they're done — but I need something that applies to a complete redesign.
  5. Developer: I'm developing a new feature (or adding new content) and want to assess progress on specific functions
 some     lots  

Notes

  • I have added some extra use cases. I wanted to have different types of websites: large and small websites, whole websites and sets of pages or new sections, and different types of developers/evaluators: inhouse and agency developers, third party evaluation, and also maintenance (I think its important!) I would like to discuss them and any other variations that anyone wants to add. To help discussion I turned the bullets into numbers. Also, I fixed a table format problem that I caused, but not the alignment issue. {Suzette}

Use case for preliminary evaluation

{@@dboudreau - 20120928} First draft - proposal

  • Vendor is commissioned to develop a WCAG 2.0 or Section 508 compliant website
  • Client wants to know if the vendor actually delivered an accessible website, but doesn't have the skill set to really measure it.
  • Client uses the Preliminary Evaluation process (simple tests) to check for basic things like alts, keyboard nav and headings on random pages.
  • Client determines if the website passes those tests or not, based on the tests results.
    • If the website passes those tests, then client or designated technical person digs further with additional informal tests.
    • If the website doesn't pass those tests, then client commissions a technical profile to integrally follow the WCAG-EM methodology.
  • Find and fix - developer, or content editor tests newly added content and fixes commonly occuring missed elements eg text alternatives, form elements, link labels, text resizing and colour contrast prior to release or more in depth evaluation. {Suzette}
  • From Website Accessibility Conformance Evaluation Methodology 1.0. Preliminary is mentioned three times including 1.3, 2.0 and this one: 4.1:"Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website."{Suzette}

Preliminary Evaluation Analysis

Audience

Primary audience:

  • Non-technical people with low accessibility knowledge who want a general idea of the level of accessibility of a web page

Secondary audiences:

  • Newbie accessibility evaluators who want to dive in progressively
  • Knowledgeable evaluators who want to run very high level evaluation only

Criteria for checks

  • Severity of barriers - impact on accessibility
  • Common accessibility barriers (what we see often as mistakes in web pages)
  • Easy to understand
  • Not complicated issues (not if lots of debate on forums)
  • Pass-fail not complex;
    Not likely to give false positive or false negative
  • Checks that clearly related to specific WCAG success criteria
  • ? easy of fixing
  • ? has good example in BAD

Note:

  • Give more weight to checks that do not require a specific browser or download tools
  • Might give more weight to some checks that don't require seeing visual rendering of page.

Misc

  • One web page — Focus on one web page, as opposed to broader issues of selecting representative pages, etc., which is addressed in WCAG-EM
  • Tool-specific info — For this draft we have some tool-specific guidance. However, there are potential issues with vendor-neutrality and we might need to address this a different way — for example, moving tool-specific guidance to Web Platform Docs or the WAI-Engage wiki where people can easily add tool-specific info.
  • Check intros/explanation — Include only brief explanation. Link liberally to relevant info in Accessibility Requirements section or Understanding WCAG page and others in the "Learn more from:" sections.
    Old discussion points on this:
    • Reasons to have more information on this page:
      • Newbies can get the basics from this one page without having to follow links elsewhere. {Shawn}
      • Can have very basic, focused description, whereas other places might be more complex. {Shawn}
    • Reasons to have less information on this page:
      • Once people understand the basics, it will be clutter to have to wade through all the information each time they want to use this as a checklist. {Shawn}
      • Duplicating information increases maintenance. {Shawn}
      • The title will probably contain "quick" so my gut feeling is to aim at being short, clear and easy. In this way, people will use this reference. Pointers to more complete information could be at the end under "Resources". {Vicki}
    • Perhaps if we had the expand-collapse functionality, it would work to have more explanation on this page? {Shawn}
  • Issue, quick versus thorough — Some things you can check partly in a quick check, but will need much more effort to do thorough check. Need to make this very clear so someone doesn't think passing the quick check means it's A-OK.
  • ...

Images of menus

[OPEN]

Should we have images of the menu selections for the toolbars? e.g., where we say things like:
In the toolbar, select "Structure", then "Heading structure".

For example, see To check alt text - with FF toolbar section

Would having images here help?
Or add more clutter than usefulness?

  • Tempting but on balance I think this would begin to look like a long tutorial and would change the character of the document. Maybe just offer one instance of a toolbar. Some images might help to confirm for instance what a page title, heading or a 'focus' etc look like and might be useful? {Suzette}
  • I find a screenshot helpful as it sometimes makes the explanation a lot easier- if possible, all the images should have the same width. {Vicki; +1 Andrew}
  • comment {name}


[end Images of menus section]

Flow

RESOLVED: Use order in option 2 for now.
OPEN: Need to discuss headings.

What's the story we're telling?

  1. Story option: Here's a process you can go through to check a web page.
  2. Story option: Here are the main sections of a web page and how to check them.

Some order options:

  • Should it be organized by topics, like it currently is? Or should it be significantly re-organized to flow the steps a person would take? (e.g., first open Validator; check x, y, z. then go to color analyzer; check color. then go to ...)
  • Should we put the most important and common first, like alt text?
  • Should we put the easiest first, like maybe page title and headings?

Thoughts:

  • @@comment {name}

Easy Checks Flow option 2 draft:

  • I suggest by page elements [as in this flow option draft] {Suzette}
  • Yes! I very much like this organization. It makes sense to a novice tester and can include all of the items we have mentioned. Most of the information can easily be ported over. I will give it a shot with Forms since I still have work to do on content for that. {Sharron}
  • Looks good to me, too! {Vicki}
  • Great addition to the example by Sharron to forms and links (keyboard)Thanks {Suzette}
  • I'm fine with the order of the topics. I think we want simpler, clearer headings. For example: "Page identity" doesn't seem a common term. I think just "Page title" is good. "Links and navigation" are likely to be understood as the navigation area (e.g., tabs across top, menu on left); but this section is about a lot more than "navigation". {Shawn}
  • EO 8/Feb - make sure first check is one anyone can do (done not require vision)

Two levels of checks?

resolved: EOWG 25 January 2013 decided on one level of checks. (old notes below)

Originally we planned to have two levels of checks:

  1. easiest ("5 minute") - does not require tools or a specific browser; very easy
  2. deeper look ("15 minute") - requires more tools, skills, time, etc.

(#Criteria for checks is below.)

The 23 January version is looking like it might be best to have just one level. For one thing, we've pared it down to a fairly small number of checks (6-8 depending on what stays). Another issue is that it would be redundant to have checks for one thing in two different places (e.g., alt text both in 5-minute list and in 15-minute list). Also, it seems that often the test-with-tools is better and easier than the no-download-tools-test, and in those cases, it would be best to present the better & easier check first and have the no-download-tools-test as a backup option. For example, see the current draft of the Check for alt text section.

How detailed for BAD?

discussed in EOWG 25 January 2013

Current plan is to have some tips on using BAD to help learn how to do each check. Some options on how to do this:

1. Just a sentence or two to say which page(s) to use and examples of what to look for — like the BAD Example for page titles

  • pro: littlest amount of text {Shawn}
  • con: doesn't walk newbies through the steps {Shawn}

2. Minimal instructions and notes on what steps to take and what to look for — e.g., like in the 23 January version of "To try checking alt text with BAD" and "To try checking headings with BAD" in the main page

  • pro: mid amount of text {Shawn}
  • con: newbies might not be able to follow it as well {Shawn}

3. Detailed step-by-step instructions — like in 18 January cut from main page

  • pro: easier for newbies to follow {Shawn}
  • con: lots and lots of text that is repetitive with the instructions above {Shawn}

Title ideas

Previous page was titled Preliminary Review of Websites for Accessibility. Do we want to change it?
Should we keep "Preliminary" somehow in the title for old links (i.e., places that link using the old title)?

  • remove the word "preliminary" - it's a "stop-to-think" word, leave alone "cannot pronounce" ;) {Vicki}
  • I agree with Vicki, this word is really difficult to pronounce. I like the idea of replacing it with quick check. - Sylvie}
  • In other contexts (like exams) the word gets contracted to prelims. Semantically it is closer to the idea of doing some 'first attempt' checks, that are not final or complete but give a general idea of how well you are doing. It reminds me of hospital emeregency facilities in the UK using 'triage' to sort out who is critical and who can wait around for a couple of hours! Possibly alternatives could include 'initial', 'first pass',or 'pre-checks' Suzette}
  • Suggestion: Could a server-side redirect be used to prevent broken links and lost favorites? {Bim} -- Shawn replies: We will not break the URI! We most likely will put this new page at the same URI, even if the title is different. --
    For discussion of whether to put at same URI or not, see minutes from 11 Jan

Brainstorms

[comment template: summary - Comment {name}]

See EOWG minutes from 8 February and 11 Jan for title discussions.

  • Easy Checks - A First Look at Web Accessibility
    • "A First Look at Web Accessibility" is misleading - it's just a first look at testing/checking {Andrew}
  • Quick Checks: Preliminary Evaluation of Web Accessibility
    • not - Difficult to pronounce, would like to find something that says the same thing but is easier. - Sylvie}
    • no - For the same "preliminary" reasons given above - Vicki}
  • Quick Checks: Preliminary Web Accessibility Evaluation
    • I'm not sure that 'Quick' is the right sales pitch for the document. Its more about doing some easy checks that indicate if there are possible problems that need further indepth investigation, or could be quickly fixed before organising a full inspection. - Suzette}
    • no - For the same "preliminary" reasons given above - Vicki}
  • Quick Checks for Accessibility (Preliminary Review of Web Accessibility)
    • part - I like this one but we could remove the brackets. What about: "quick and easy checks for accessibility" - Sylvie}
  • Quick and Easy Checks for Accessibility
    • like - {Sylvie}
    • like - {Vicki}
  • Quick Check to Evaluate Accessibility
    • maybe - "evaluate" might be good for SEO? {shawn}
    • like - like this one too - Sylvie}
    • like - {Vicki}
    • like - {Bim}
  • Quick Check to Test for Accessibility
    • maybe - "test" might be good for SEO? {shawn}
    • like - {Vicki}
  • Accessibility Quick Check
    • like - I can see people using this as a search term. {Bim}
  • Simple Evaluation
    • not - Maybe this one is too simple. - Sylvie}
    • Don't like - Evaluation of what? {Bim}
  • Evaluating accessibility quickly and efficiently
    • like - {Sylvie}
  • Checks to perform a quick web accessibility evaluation
    • like - {Sylvie}
  • Getting a quick idea of the accessibility level of a Website
    • like - {Sylvie}
    • like - {Vicki}
  • 10 easy tests for accessibility conformance of your website
    • or whatever number we come up with {Suzette}
    • no - think we don't want "conformance" in the title of this since it won't get you to conformance, and the other doc focuses on conformance {Shawn}
  • Basic evaluation....
  • Initial....
  • Accessibility Quick Checks: getting started with web accessibility evaluation

Usability Testing - notes

  • Do users imply an importance order to the list of checks?
    • maybe we need to explicitly state (in the Introduction?) that the order of the checks does not reflect any importance, but rather relates (sort of) to their complexity
  • How do users handle if they can't do a test due to disability?
    • do we need to indicate which tests anyone can do vs those that need vision, hearing, etc?
  • is Intro complete enough?
    • does it need to say that some tests won't work in all circumstances/situations (e.g. images placed on page via CSS)
  • any insight on the title?
  • views on images - helpful or cluttering?


Retrieved from "https://www.w3.org/WAI/EO/wiki/index.php?title=Eval_Analysis&oldid=13362"