Brainstorming Table Summary Usability Evaluation

Statement of Purpose

This document is an attempt to gather concerns (general and specific) from the public-html-a11y list that may be relevant to any usability evaluation that we possibly may undertake to investigate and resolve Table Summary HTML Issue 32.

General Usability Evaluation Questions Drawn From Discussion

Matt

• How can study data be fairly analyzed by humans who are frustratingly analog?

John

• Can findings be based on publicly available data?

Ian

• Is formal usability testing of authors followed by double-blind studies of AT users a good evaluation technique? (Questioning during testing seems to be problematic.)
• Is having an independent researcher who is not aware of the testing "grade" the tests a good idea? (Ian "might" be able to get resources to do testing early in 2010.)

Laura

• Would it be in-scope for this task force to do surveys of users? (Like the WebAIM surveys geared to the table summary Issue with questions like Roger Johansson asked on his blog.)
• Could the W3C WBS survey tool be used to conduct surveys of users?
• Do we have the resources and does the group want to do full blown usability testing?
• Are there any accessibility task force members willing to collaborate on doing usability evaluations?

Leif

• What is the feasibility of the task force doing a usability test? Do we have the needed resources?
• Can we include authors as well as assistive technology users as participants in any studies?
• Can goals be specified so everyone agrees upon them?
• How can evaluations be fairly designed, written, conducted, and measured so that this task force can accept the findings?

Cynthia

• Is it possible to include business concerns in a usability study? Can we include VPs of marketing in the user group? Change the authoring task to say that every addition pixel costs $1 million? (true story) Would that be valid?

Usability Testing Definitions

• A Concern is a question about usability.
• A Goal is a declarative statement resulting from a question of concern. A goal establishes an objective to measure.

Specific Table Summary Concerns (questions about usability)

Matt

• Will a table summary replacement that defaults to visible not be used by authors because it's presented visually?
• What is the best way to reconcile aesthetics and data with one another?
• If people are taught to write good table summary values will they write bad summary values?
• Will people think to add a summary paragraph, if they are they're not prompted to do so by a tool?

John

• What evidence is there that @summary causes harm to users?
• Is meta-data better than no data?

Ian

• Would a visible table summary help accessibility?
• Does a hidden table summary attribute harm accessibility?
• Does a hidden table summary attribute help accessibility?

Roger

• Do assistive technology users who use shortcuts to jump straight to the table miss any information if the table summary information is provided in paragraphs not explicitly associated with a table?
• What is the best way for authors to satisfy stakeholders aesthetics of not having a table summary information visible while still making a table accessible?

Leif

• What kind of extra table summary information is needed for accessibility?
• What is the best way to add that information?

Laura

• How can table summary be improved?

Specific Evaluation Goals

To be determined...agreed upon goals of any evaluation would be a first step (if in fact the group wants to do evaluations). Everything in any evaluation should stem from what the group wants to investigate without preconceptions or bias. Also criterion, methodology, etc could be a huge bone of contention if not agreed upon by the group in advance.

In usability testing, goals are declarative statements resulting from a question of concern. A goal establishes an objective to measure in an evaluation.

References

• HTML Issue 32
• Re: CHANGE PROPOSAL: Table Summary Thread
• Involving Users in Web Projects for Better, Easier Accessibility
• 2002 Usability Testing Planning Documents
• WebAIM Survey 1
• WebAIM Survey 2
• Roger Johansson Blog Post
• Types of Usability Evaluations Include:
  1. Inquiry (Contextual Inquiry, Field Study, Focus Groups, Interviews, Observation, Questionnaires, Surveys)
  2. Inspection (Heuristic Evaluation, Walkthrough)
  3. Testing