W3C

Site Navigation

Nearby

Results of Questionnaire Follow On from 29 April Face to Face Meeting

The results of this questionnaire are available to anybody.

This questionnaire was open from 2021-04-29 to 2021-05-04.

17 answers have been received.

Jump to results for question:

One-line description of question
How should we handle scope in 3.0?

1. One-line description of question

Based on the following discussion points from the meeting, can we agree to use a rating scale (adjectival rating) at the outcome (testable statement) level?

Simplicity and transparency matter especially to regulatory and legal specialists.
Transparency adds granularity so increases the number of checkpoints. Simplicity will most likely come from consistency in how we handle scoring. We should ideally pick a single approach to scoring outcomes.
We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this.
Assuming we want a single approach, percentages do not work across all outcomes so shouldn't be our choice. Percentages can be a threshold on some rating scales but not the final scoring approach.
Binary and Rating scales are the same at the outcomes level since we have to define, even in binary, where the cutoffs are.
The question is how many levels should be on a rating scale. If we approach the question this way, we can run tests and aggregate test results against 2, 3, and 5 point scales (or any other number) to compare the balance between simplicity and flexibility and make a decision.

Summary

Choice	All responders
Choice	Results
Yes, use a rating scale at the outcome level and test how many levels works well	8
Something else (add to comments)	8

(1 response didn't contain an answer to this question)

Details

Responder	One-line description of question	Comments
Jennifer Delisi	Something else (add to comments)	Regarding list item 4 - I'm unclear of why "but not the final scoring approach" is listed here, because I believe this is about the outcome (testable statement) level. Apologies if I am misunderstanding the question. Regarding list item 6 - I do not feel qualified without reviewing research about rating scales and levels, to answer this question. My assumption is that there is research that has tested different scoring models, in a variety of settings. If levels were selected based on this research, along with a researched-method for writing the scoping for levels (helps testers decide where their score falls when it is more ambiguous), then I would find it easier to vote on this question. If there is research the group has reviewed and is considering while answering this response, it would be helpful if a link could be shared.
Sarah Horton	Something else (add to comments)	Continue to research, discuss, and prototype approaches.
Stefan Schnabel	Yes, use a rating scale at the outcome level and test how many levels works well
Oliver Keim	Yes, use a rating scale at the outcome level and test how many levels works well
Jake Abma	Something else (add to comments)	Continue to research, discuss, and prototype approaches.
Justine Pascalides	Yes, use a rating scale at the outcome level and test how many levels works well	At a conceptual level, yes.
Makoto Ueki	Something else (add to comments)	Re: 4 - Percentages can be more repeatable and reliable than any other things to present a more nuanced way of showing how accessible their website, app or product is. There will be a number of guidelines (used to be "SC" in WCAG 2) where we can calculate percentages. If it is not the "final" scoring approach, I'd like to confirm how the final approach in this context would be, before I say "Yes". And +1 to John's "concerns over subjective determinations". We should check if multiple testers will get the same results/scores at the end. It maybe at later timing, not now. But the results of multiple assessments should be repeatable.
Michael Gower	Something else (add to comments)	Maybe you don't want any feedback on your list, but here are a few comments: "Simplicity will most likely come from consistency in how we handle scoring." I'd rather you used "Clarity" than 'simplicity', in almost all cases where you've used the word. "Transparency adds granularity so increases the number of checkpoints" I'm not sure I understand this, or agree with it :) "We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this." I'd say "consistent results" rather than "reliability" This is kind of a radical departure from this, and I apologize for just stating it in a survey question, but I wonder if allowing teams to report on design and dev process (its existence and adherence to) isn't at least as important a metric to capture in regard to pursuing accessibility. The ISO approach, if you will.
Marc Johlic
Jeanne F Spellman	Yes, use a rating scale at the outcome level and test how many levels works well	We have a group working on testing WCAG3 that can test this. We need more people with testing experience to help so we can get better data.
Alastair Campbell	Yes, use a rating scale at the outcome level and test how many levels works well
Andrew Kirkpatrick	Something else (add to comments)	Without more specific examples I can't answer this concretely. Am concerned about subjectivity (e.g., "process").
Rachael Bradley Montgomery	Yes, use a rating scale at the outcome level and test how many levels works well
David MacDonald	Yes, use a rating scale at the outcome level and test how many levels works well
Bruce Bailey	Yes, use a rating scale at the outcome level and test how many levels works well
Laura Carlson	Something else (add to comments)	Continue to research, discuss, and prototype approaches.
John Foliot	Something else (add to comments)	Re: Bullet 4 - I reject the assertion that "percentages do not work across all outcomes so shouldn't be our choice" as pretty much anything can be measured as a percentage. (xx/100) Re: Bullet 6 - "The question is how many levels should be on a rating scale." - Again, this can be handled by using percentages (for granularity), with minimum percentage values then mapping to Bronze, Silver and Gold. I also continue to have grave concerns over subjective determinations, as no two experts will always agree, and we're setting up a point of disagreement between evaluators which will be problematic for regulators. For Question 3 (Scope) the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".

2. How should we handle scope in 3.0?

WCAG 2 defines conformance against web pages. The first public working draft of WCAG 3 defines scope in terms of views and processes. There have been comments about the need for a better definition of process.

How do you think we should define scope within conformance?

Summary

Choice	All responders
Choice	Results
Use WCAG 2 scope of web page adjusted for non-html.
Use the WCAG 3 FPWD scope to views and processes and better define both.	15
Something else (add to comments)	2

Details

Responder	How should we handle scope in 3.0?
Jennifer Delisi	Use the WCAG 3 FPWD scope to views and processes and better define both.
Sarah Horton	Use the WCAG 3 FPWD scope to views and processes and better define both.
Stefan Schnabel	Use the WCAG 3 FPWD scope to views and processes and better define both.
Oliver Keim	Something else (add to comments)
Jake Abma	Use the WCAG 3 FPWD scope to views and processes and better define both.
Justine Pascalides	Use the WCAG 3 FPWD scope to views and processes and better define both.
Makoto Ueki	Use the WCAG 3 FPWD scope to views and processes and better define both.
Michael Gower	Use the WCAG 3 FPWD scope to views and processes and better define both.
Marc Johlic	Use the WCAG 3 FPWD scope to views and processes and better define both.
Jeanne F Spellman	Use the WCAG 3 FPWD scope to views and processes and better define both.
Alastair Campbell	Use the WCAG 3 FPWD scope to views and processes and better define both.	I think a key thing to also agree is that (like WCAG 2.x) it is up to the person claiming conformance that determines which views/processes to include. And then it's up to regulators (etc) to determine for certain industries/sectors what type of views/processes they must include.
Andrew Kirkpatrick	Use the WCAG 3 FPWD scope to views and processes and better define both.
Rachael Bradley Montgomery	Use the WCAG 3 FPWD scope to views and processes and better define both.
David MacDonald	Something else (add to comments)	It seems a little premature, because I think of this question like an exective summary. Lets get further into the actual methods before we decide what will be most appropriate
Bruce Bailey	Use the WCAG 3 FPWD scope to views and processes and better define both.
Laura Carlson	Use the WCAG 3 FPWD scope to views and processes and better define both.
John Foliot	Use the WCAG 3 FPWD scope to views and processes and better define both.	the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".