Analysis/Requirements and Changelog for "Involving Users in Web Accessibility Evaluation"
Page Contents
About Involving Users in Web Accessibility Evaluation
Purpose, Goals, Objectives
- Introduce evaluating accessibility with users with
disabilities
- Briefly clarify levels and types of evaluating with users (some people have misconception that "usability testing" is only formal, task-based, site-wide evaluation)
- Encourage people to do whatever they can
- don't need to do formal, thorough usability testing — can get lots from cheaper, quicker stuff
- (perhaps have benefits or encouraging bit as a lead in)
- Address a few of the most serious common problems or cautions (see Archive below)
- Not be a comprehensive resource for usability testing with participants with disabilities (scope way too big)
Audience
- Same as whole Evaluation
Suite:
- Primary audience: Web developers (designers, content authors, etc.) who want to comply with Web accessibility standards
- Secondary audiences: decision makers, professional evaluators, and accessibility researchers
- Note: usability testing professionals might also "land" on this page
Notes
- Size: less than 3 printed pages
- Part of Evaluation Resource Suite (draft of new outline)
- see Archive below
References
- [note] wai-eo-editors list includes some replies to e-mail comments (to limit clutter in the main EOWG mailing list)
- [EARLY CONCEPT DRAFT] Involving Users in Web Accessibility Evaluation
- [done] 2005-Aug-19 e-mail from Alan Chuter
- [done] 2005-Aug-19 EOWG teleconference minutes
- [done] 2005-Aug-30 e-mail EOWG: Review this week: Evaluating with Users
- [done] 2005-Sept-01 e-mail from Pasquale
- [done] 2005-Sept-02 e-mail from Roberto
- [recorded in changes below] 2005-Sept-02 EOWG teleconference minutes
- [done] 2005-Sept-06 e-mail from Fraunhofer FIT
- [done] EOWG: Agenda and Readings for teleconference 16 September 2005 (includes detailed review questions)
- [done] 2005-Sept-14 e-mail from Wayne
- [done] 2005-Sept-16 e-mail from Pasquale
- [recorded in changes below] 2005-Sept-16 EOWG teleconference minutes
- [EARLY ROUGH ROUGH DRAFT] Including Users in Web Accessibility Evaluation
- [done] 2005-Sept 16 e-mail: for content review: Involving Users in Evaluating Web Accessibility
- [done] 2005-Sept-16 e-mail from Sailesh
- [done] 2005-Sept-24 e-mail from Pasquale
- [done] 2005-Sept-27 e-mail from Barry
- [done] EOWG: Agenda and Readings for teleconference 30 September 2005
- [done] 2005-Sept-29 e-mail from Roberto
- [recorded in changes below] 2005-Sept-30 EOWG teleconference minutes
- [done] 2005-Oct 1 EOWG: reply requested, wrapping up Involving Users doc
- [done] 2005-Oct 2 e-mail from Wayne
- [done] 2005-Oct 3 e-mail from Sailesh
- [done] 2005-Oct 3 e-mail from Helle
- [done] 2005-Oct 4 e-mail from Pasquale
- [done] 2005-Oct 4 e-mail from Henk(forwarded from Shawn)
- [DRAFT] Involving Users in Web Accessibility Evaluation (6 Oct 2005)
- [recorded in changes below] 2005-Oct-14 EOWG teleconference minutes
- [recorded in changes below] 2005-Oct-21 EOWG teleconference minutes
- [done] Results from Call for Review
- 2005-1-Nov e-mail to EOWG with final edits
Consider for next version or related documents or other
- Reduce spacing before top of list with previous paragraph
- Add image of green magnifying glass & look for other graphic oppotunities
- consider change:
location: "Introduction" section, second paragraph
current wording: example of screen reader user
suggested revision: example of screen magnification
rationale: screen readers are mentioned elsewhere in the document (for example in "Involving Users Effectively") and often not an obvious AT to Web developers. - Link to when done:
- in For More Info section: [selecting consultants] with the blurb organizations that can help
- in For More Info section: How People with Disabilities Use the Web describes how different disabilities affect Web use and accessibility requirements for people with different kinds of disabilities, and includes scenarios of people with disabilities using the Web.
- in Terminology section: Basic Glossary (formerly known as (fka) Lexicon)
- document addressing issues of usability & accessibility & "basic accessibility" and "usable accessibility"
- consider if want to tweak to meet additional "requirements"
- side goal: creating evangelists
- audience: includes tool developers
- scope: including throughout design & development, not just eval [already cover that some...]
Changes
Note: See "References" section above for meeting minutes and e-mail comments.
24 Oct - 1 Nov 2005
- [done] added link to acknowledgments
- [done] from 21 Oct teleconference: within the main text, include the word "Formal" in the link "usability tests can be optimized". leave Optimising for UT as <h3> with current wording
- [done] in Drawing Conclusions and Reporting section, changed "even large-scale usability studies cannot cover the diversity of disabilities, individuals, and situations" to "even large-scale usability studies cannot cover the diversity of disabilities, adaptive strategies, and assistive technologies"
- [done] to teminology section, assistive technologies, changed "... such as screen readers that read aloud Web pages for people who cannot see or read text, and voice recognition software..." to: "... such as screen readers that read aloud Web pages for people who cannot read text, screen magnifiers for people with some types of low vision, and voice recognition software..."
- [done] updated links in For More Information section from ittatc.org to uiaccess.com/accessucd/ (as ittatc.org likley to be taken offline before too long)
- [done] typos and link fixes
- [done] fixed broken link in Page Contents to Analyzing Accessibility
Problems - [done] fixed broken internal link in "even large-scale usability studies cannot cover the diversity of disabilities,"
- [done] corrected capitzation of "if" in "For example, If you are developing a Web application to be used"
- [done] fixed broken link in Page Contents to Analyzing Accessibility
14 Oct 2005
- [done] In the Introduction section, second paragrpah, added first sentence: "One of the benefits to including people with disabilities is that Web developers can learn how people with disabilities interact with the Web and with assisitive technologies." also considered:
- One of the benefits is that Web developers can learn how people with disabilities interact with the Web and with assisitive technologies.
- One of the benefits to including people with disabilities is that Web developers can learn how they interact with the Web and with assisitive technologies.
- [done] changed "People with disabilities are as diverse as any people. They have different experiences, different expectations, and different preferences. They use different interaction techniques, different adaptive strategies, and different assistive technology configurations." to "People with disabilities are as diverse as any people. They have diverse experiences, expectations, and preferences. They use diverse interaction techniques, adaptive strategies, and assistive technology configurations."
- [done] minor editing from issues identified during teleconference
- [done] changed "Assistive technologies are software or equipment that is used by people with disabilities to improve interaction with the Web" to "Assistive technologies are software or equipment that people with disabilities use to improve interaction with the Web"
For EOWG discussion 14 Oct
- [done] review of content changes since last version: listed under 12-13 Oct 2005
- [done] heading "Optimizing User Involvement": some suggestions (from Wayne) and comments (from Shawn):
- Involving Users Effectively
[one reaction: There are *many* more considerations for involving users more effectively, this section just touches on part -- so I'm a little uncomfortable with that broad section title. (although throughout the whole document we are just touching on little bits...) Also, we use "effectively" elsewhere in talking about implementing accessibility solutions] - Getting the Most from Users
[one reaction: Yes, I guess that's what this section is about; however, I wonder if this can be misconstrued (or translated) negatively?]
- Involving Users Effectively
- [done] clarifying benefit of involving PWDs: "I think the document needs to highlight that the developers get a
realistic feel of how content is rendered to PWD and how the AT devices most
likely being used in the target-population render the content." [from Sailesh]
Question: is this implicitly covered already (see current wording below), or should we add more about this (probably to an already long Introduction)?- current wording: "... involve people with disabilities helps better understand accessibility issues and implement more effective accessibility solutions.
For example, take a Web developer who does not know what it is like to use a screen reader... observing a person use the site with a screen reader will clearly show the developer that...
When Web developers, managers, and other project stakeholders see people with disabilities use their Web site, most are highly motivated by a new understanding of accessibility issues. Collaborating with people with disabilities who are target "users" of a Web site early in a project helps Web developers be more efficient in addressing accessibility, thus maximizing the results from investment in accessibility."
- current wording: "... involve people with disabilities helps better understand accessibility issues and implement more effective accessibility solutions.
- [done] minor formatting:
- bold: Bolding helps online skimming for most people (particularly helps emphasize important points). However, it adds visual complexity and might slow some careful reading. Which way do we want to err? How is the bolding in the current version (e.g., places we can cut it out)?
- alt <code> markup resulting in different font for most configurations: Does the different font face increase readability by differentiating the example alt text from the rest of the text, or decrease readability by complicating the visual presentation with another font face?
- with code markup see the: previous version, 2nd paragraph starting with "For example, take a Web developer who..."
- without code markup: see the current version, 2nd paragraph starting with "For example, take a Web developer who..."
- [done] adding user "needs" (or "requirements") to "Including Diverse Users" section [Shawn will provide summary for previous teleconference discussion and e-mails. notes from 30 Sept minutes: "need to stop people thinking of PWD as "needy" - prefer not to reference "needs. stressing the diversity of users is what we are trying to do. concern is that there is a myth that PWD have an incredibly wide range of "requirements" that can never be met. happy to leave as is."]
- [done] minor style guide question: contractions would make it easier reading for this native English speaker (-ed.). what is the trade off in understandability for native speakers of other languages, translators, and formality of the document?
- example without contractions: For example, take a Web developer who does not know what it is like to use a screen reader.
- example with contractions: For example, take a Web developer who doesn't know what it's like to use a screen reader.
12-13 Oct 2005
- [done] took out "comprehensive evaluation" and avoided "technical" and "basic" accessibility because it's too big of an issue to address right now. want to get this version of this document done ASAP, and can add back in as appropriate later. content changes from the previous draft included:
- Introduction, 4th paragraph
- Optimizing User Involvement section, second paragraph & bullets underneath it
- [done] took
<code>
off text in 2nd paragraph so fewer fonts - [done] changed "Diagnosing Accessibility Problems" to "Analyzing Accessibility Problems" to avoid medical connotations (potentially uncomfortable for some PWDs)
- [done] a few minor copyedits and changes from e-mails (recorded in wai-eo-editors)
5 Oct 2005
- [done] put @@ and [brackets] around some issues for discussion
- [done] edited terminology words & definitions
30 Sept 2005 Teleconference
- [note for reviewers] took out "When the evaluation is being used to improve a Web site, it is rarely necessary to distinguish between usability and accessibility issues. However, when the evaluation data will be used to make public statements about accessibility, distinguish between usability and accessibility issues as much as possible."
- [note for reviewers] took out "Avoid making decisions based only on the recommendations of one person with a disability. What works for one person might not work for everyone with that disability or other disabilities. Be careful not to assume that all users use browsers, assistive technologies, or your Web site the same way." and left: "It is especially important when you are able to involve only a few users to carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the Web, nor know enough about other disabilities to provide valid guidance on other accessibility issues."
from action items in minutes:
- [done - think this content is just not well suited for skimming; however editor is very open to specific suggestions for improvement in readable &/or skimmability!] when copyediting - look at skimmability
- [done - didn't come up with a smooth way to do this differently and not sure it's a major issue, especially considering that the browser Back feature should work in either case. note that we did this same thing this way in http://www.w3.org/WAI/intro/accessibility.php. anyway, editor is open to ideas on addressing.] consider how to address that one of the links go to the other document and some go to a terminology at the bottom of the page
- [done -- that is, not comfortable linking to either because http://en.wikipedia.org/wiki/Usability_testing does have some good definition but very poor (almost disturbing) list of external links, http://search.dmoz.org/cgi-bin/search?search=usability+testing mostly links to companies who do it as opposed to resources to help understand what it is and the issues surrounding it, and http://usableweb.com/topics/000878-0-0.html too limited and outdated. however, editor is open to more discussion about this.] Shawn to check DMOZ, Wikipedia, or other for usability resources
- [done] add into the for more info about using consultants...recruiting from disability organizations and specialization organizatinos
- [done] in different types section - look for watching and changing to interacting
- [done] in drawing conclusions....change testing methodology to testing method
- [done] in drawing conclusions...add some of the benefits of what it can do
- [done] information 4th paragraph second sentence change the bullets and change usability testing for accessibility issues to user evaluation
- [done] intro 3rd paragraph...change the idea of watching pwd to the idea of interacting with pwd
- [done] editors discretion about how best to handle this para - leave here, rewrite, etc
- [done, that is, editor's changes should have taken care of it] William to suggest changes to sentence following the bullets. William to suggest some wording not to include PWD just as "lab rats", rather as collaborators for para 4 or 5 and incorporate the preceding discussion
- [done] other changes were also made based on the EOWG discussion
28 Sept 2005 Revision Notes
- At the 16 September teleconference, we discussed linking to outside resources. This has been added as a topic to discuss at an upcoming EOWG meeting (probably will be 7 or 14 October). For now, I've added what I thought we might agree on (based on initial disucssions of what would be the parameters for linking to a specific outside resource) - basically the links to the direct resource on including accessibility in UCD & usability testing under http://www.w3.org/WAI/EO/Drafts/eval/users.html#fmi. I did not add links on general usability testing resources for several reasons. We should discuss this *after* the general discussion on linking to outside resources
- [note for reviewers] revised content: "Involving users with disabilities is an integral part of a comprehensive evaluation of Web accessibility that includes assessing conformance to WCAG, heuristic evaluation by an accessibility expert with first hand experience of how people with different disabilities interact with the Web, and usability testing for accessibility issues."
- [note for reviewers] edited to further clarify that usability testing does *not* ensure accessibility, and to avoid "technical conformance" or "technical accessibility" terminology
- [note for reviewers] deleted from previous version because deemed out of scope: <h2>Diagnosing Accessibility Problems</h2>
Here are some points related to identifying and addressing issues in different components:
- browser, assistive technology - different browsers and ATs have different features and interactions. {If so: do you need/want to do a work-around? please inform the vendor!}
- user knowledge/experience - as described above, an issue might be due to the users lack of experience. {If so: is this common in your target user group? do you want to provide additional instructions?}
- Web site
- most problems will probably be things that you can fix in the Web site.
{If so: then you can fix!}
[@@barry comment: "Re-reading this, I found I no longer understood what
we were trying to get across with this point. As it stands
might be read as saying that, of the various components
involved, the web site is most commonly ("most problems") the
one at fault. I'm not sure I could agree with that anyway; but
if that were the intention, it needs more clear expression."]
16 Sept 2005 Teleconference
from action item list in minutes:
- [done, that is, editor decided not to link to this so not necessary] Justin look at UCD notes page and adding usability testing
- [done] make more explicit and maybe in the intro doing prelim eval
- [done, modeled after http://www.w3.org/WAI/intro/accessibility.php#terms] add a section on terminology with screen reader
- [done] addressing pointing or including how do i do this
- [done] consider adding a sentence about adaptive strategies with mainstream browsers
- [done] formal usability testing sentence be broken up
- [done, decided too big and complex of an issue to try to address in this document] how we might want to cover the issue of usable accessibility
- [done] in diverse users section emphasize extreme variation
- [done] in the intro first sentence doesn't understand the experience of a screenreader
- [done] look at range of user involvement heading
- [done] look at referencing getting outside firms with evaluating with users
- [done - edited in first paragraph and strengthened in "Drawing Conclusions and Reporting" section] reconsider putting back in that its not required
- [done] relook at including what to do if you find browser AT issue or user issue
- [done] usability testing discipline sentence is deleted
- [done] put on future agenda referencing outside resources (hot topic for invluding users in eval doc)
[done, added to agenda planning] for EOWG discussion: referencing other documents (instead of how is in the For More Informatino section)
[done, for now - will re-address] from e-mail: for discussion: "About title, please consider also: "User involvement in evaluating web accessibility" or "Involving users in web accessibility evaluation". I like concept of "involving" more than "including"."
14 Sept 2005 Revision Notes
notes about things that were in the previous version or the comments that are not in this version:
- "usability testing not a requirement to ensure comply with WCAG"
- subtly addressed. don't feel will be a confusion as doc is currently shaping up. - Mention tasks and expected outcomes. Don't just sit the user down
and say "what do you think of that?"
- example is task based. is that enough? - bottom line: {Large-scale formal usability testing is not necessary to evaluate usable accessibility. Short informal evaluation can gather valuable feedback from people with disabilities without the rigor of formal usability testing.}
- think subtly addressed sufficiently - screening techniques... - covered in preliminary evaluation, point to that
- PWDS will find usability issues, {Usability testing with participants with disabilities will identify usability problems that impact all users, regardless of disability}... so instead of doing usual number of non-disabled + additional disabled, can do variety of disabled!
- decided not key issue & need to cut content - if you find an accessibility issue with a browser or AT: do you need/want to do a work-around? please inform the vendor!
- determined out of scope - if you find an accessibility issue that's really just user's lack of knoweldge/experience: is this common in your target user group? do you want to provide additional instructions?
- determined out of scope - Because usability and accessibility overlap, it is sometimes difficult to distinguish an issue as usability or accessibility. {Usability problems impact all users equally, regardless of ability. That is, a person with a disability is not disadvantaged to a greater extent by usability issues than a person without a disability.}
{Accessibility problems impact people with disabilities, and not people without disabilities. When a person with a disability is at a disadvantage relative to a person without a disability, that is an accessibility issue.}
- difficult to agree on & out of scope - comments from Fraunhofer FIT and others above
- many of the ideas for additional information are out of scope; however, I think this draft makes them unnecessary.
2 Sept 2005 Teleconference
- [done] define "users" up front
- [done, decided out of scope for now] consider adding to requirements doc:
- side goal: creating evangelists
- audience: includes tool developers
- [done] introduction:
- [not done for now. will keep eye open to see if fits later] consider adding in a point roughly along these lines: involvement of users in eval is a good first step towards establishing ongoing constructive feedback cycle
- [done] clearer focus on accessibility & ppts with disabilities
- [no done expliictly. thin kinnecessary with current draft] perhaps include up front the difference between involving users in evaluating accessibility & usability testing
- [done] considering including the "why" up front
- [done] set this doc/evaluation in the context of the eval suite & overall evalaution -- readers might not know either usability evaluation or accessibility evaluation :-/
- [concern that people will have different ideas in their head about what we are talking about here. woudl rather not try to explain it in the Introduction, but instead leave it as the first section...] consider swapping "Types of User Involvement" & "The User Aspect" sections (assuming clearly got rid of usability testing assumptions in introduction)
- [done, needs review. added placeholder for HowPWDS at end (to add after the doc is in place)] "more than screen readers users!" - consider mentioning a few assistive technologies, adaptive needs & strategies, perhaps here link to How PWDs Use Web
- [done] in "types of user involvement" add "for example, ..." for each approach (for people not familiar with user evaluation)
- [done] point to additional references of user evaluation (generally or specifically)
- [done] consider listing some of the considerations -- e.g., recruiting, compensating, ... consider a checklist
- [done] "*vital* to distinguish between issues of:" section - link to the Essential Components Document & not repeat
- [done] (note there is a little title discussion in minutes)
Archive
- one user not representative, don't do everything one user says
- distinguish between usability vs. accessibility issues
- distinguish between issues of user agent, Web site, assistive technology, user knowledge, ...
- more than just people who are blind
- Since this is part of the Evaluation Suite, focus on user involvement in accessibility testing; however, since ideally users are incorporated throughout the process, mention getting them involved from the start and throughout
- Carefully clarify that usability testing is not a requirement to ensure comply with WCAG
- level of user expertise (depends on target audience) - too advanced might know uncommon work-arounds, not advanced enoughed may not know thinks like links lists, headings nav, etc.) - people often mis-state their own level
- issues with non-PWDs using AT - false negatives, 'cause don't know how to use screen reader, think it's hard 'cause they don't know how to use them...
- stakeholders - live observation!!! (or at least highlight tapes), rather than dry report...
- issue: terminology. does "usability testing" conjure up formal testing through complete tasks too much? if so, should we use something else, such as "user involvement in accessibility testing" or "testing accessibility features with users" or ?? that is not too awkward?