Disposition of Comments

Website Accessibility Conformance Evaluation Methodology (WCAG-EM),
W3C First Public Working Draft

This is a disposition of comments received on the Website Accessibility Conformance Evaluation Methodology (WCAG-EM), W3C First Public Working Draft 27 March 2012. This page is intended for internal discussion by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF).

Table of Contents

Open Comments

All comments are currently open. These include the comments that were left open from before the publication of the Public Working Draft.

Comments to Focus on

All. Please note that the status of some decisions is proposed. This means that the decision has to be discussed in the EvalTF first.

Comments for Section 1 - Introduction

Listing of the comments for Section 1 - Introduction and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
5 François Junique 1.1, 2.1 and 5c Open Scores better defined Size-independence can be tricky for a very large portal, modularity/truncatibility should be better supported, and aggregation of split scores should be explained (or to present it differently, scores - in 5c - should be defined in such way to be aggregatable according to a defined computation recipe)

There could be sites with different scopes.

1. What if a website is evaluated in website parts. If portlets evaluated are part of a portal and all checked. Is the result then ok for the total? How about if one piece is A and rest is AA

2. Can you include or exclude parts. This is currently covered in 3.1 scope

Proposed decision: Add to section 3.1 note about website parts and short clarification of including or excluding parts. Also add in Section 5c clarification of aggregating performance scores.
20 Peter Korn 1.2 Open For example, the four bullets under "Other audiences..." includes policy makers and project managers - who will also need a way to evaluate not only the extent to which a website / web application is meeting WCAG 2.0, but the extent to which support for WCAG 2.0 is improving from one evaluation to another - to assess the "progress toward done" that a website / web application is making. It may be useful to recognize some of the needs of these disparate audience members.

Currently, the comparison of evaluations over time is not part of the WCAG-EM.

WG participants indicate that they like the concise nature of 1.2. Proposal is to add a link to the business resource in section 1.3 to clarify the businesscase:

Decision to add in 1.3: "The different aspects of a business case for Web accessibility are presented in the pages 'Developing a Web Accessibility Business Case for Your Organization' with guidance on developing a customized business case."
21 Peter Korn 1.4 Open Add definiton of "Ancillary Functionality" Terms and Definitions defines the term "Key Functionality". Might it be useful to also define "Ancillary Functionality"?

Proposed decision: As a concept, Ancillary Functionality may be usefull. We follow the proposal by Peter Korn during the Telco to hold this thought but currently not add it to the document

Comments for Section 2 - Using this Methodology

Listing of the comments for Section 2 - Using this Methodology and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
27 Aurélien Levy 2.1 Open Add Partial Conformance The definition of the scope exclude the ability to remove part of a website. Why aren't you considering the same exception as in the WCAG 2.0 for partial conformance?

Partial conformance does not have to be in here as it is already inside WCAG2.0. We can clarify this here and refer to WCAG2.0.

Proposed decision: Clarify in section 3.5b and 3.5c the possibility of statement of partial conformance for third party content and for language

Comments for Section 3 - Evalution Procedure

Listing of the comments for Section 3 - Evaluation Procedure and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
1 Greg Gay 3.1b Open Add a “common elements” review as a goal of evaluation. This review will focus on issues associated with the accessibility of templates. Given sites now-a-days are typically template driven, in addition to detailed and indepth reviews, I would suggest a “common elements” review be included as another type or goal of evaluation. Because the issues associated with template accessibility tend to differ from issues typically identified for content, I believe a review of templates should be separate from a review of content, and be handled apart from the “Representative Sample.” The vast majority of sites implement only 2 or 3 templates, so a review is typically inexpensive and attractive to organizations that want to address accessibility as best they can, but don’t have a big budget.

The focus of this methodology is on website evaluation. The method proposed here would be a more limited evaluation like in the case of the preliminary evaluation inside the evaluation suite

Proposed decision: No change
2 Greg Gay 3.1b Open "Basic Conformance" "Basic Review" Replace “Basic Conformance” and instead use “Basic Review” since conformance cannot be determined from a sampling of pages. (read more in comments from Greg Gay)

We tried to avoid the impression that this is not a full conformance evaluation. This is a full conformance evaluation, only the report only covers Fail/Pass. This is rather basic. No details or analysis.

Proposed decision: No change
3 Greg Gay 3.1c Open “Reviewer should also evaluate Level AAA….” “In many situations it is useful…” Though we do suggest a conformance target that realistically reflects the effort that might be required to reach a level of conformance (Typically Level AA), or we make target level recommendations based on legal requirements, we always provide a review of Level AAA items and encourage developers to address as many of them as is feasible, some of which are relatively easy to implement. Step 1.c reads as if review of AAA items is optional. I might suggest a little stronger language here. Proposed decision: Try to clarify the positive impact of adding AAA SC in section 3.1.c
30 Aurélien Levy 3.1c Open Add something here (or in the reporting section) about the conformance to each WCAG level Making an AAA evaluation can be useful but the reporting must be adapted to give conformance for each WCAG Level. Furthermore using a AAA target will tend to decrease the conformity score and discourage the commissioner (or his development team)

This is the idea of the text: A, AA OR AAA. Will try to clarify further

Proposed decision: Clarify that there is a choice: OR
28 Aurélien Levy 3.1 Open Sometimes defining the scope with the evaluation commissioner can be not as good as this section suggest it because most of the time he doesn't know anything about accessibility and about what is important for user to be evaluated. Furthermore, he can also include in the scope some specific pages where the development team has concentrated his effort to improve accessibility

In that case, the evaluator could propose a scope for the evaluation to the evaluation commissioner. Just cherry picking pages is not the intention. If the evaluator notices that, he can point to the methodology and propose other pages.

Proposed decision: Clarify the role of the evaluator as responsible for an independant evaluation
32 Aurélien Levy 3.2a Open Maybe this step needs to be optional Not every evaluation commissioner has access to or control over templates

They are more or less optional because it is only for the templates that are available to the evaluator. If they are not available, this can be reported.

Proposed decision: Add a Note to section 5 saying that if pages or templates are missing, this should be reported.
33 Aurélien Levy 3.2b Open Add the help of the commissioner In this section I think we need to do it with the help of the commissioner

The evaluation commissioner could indicate key functionalities, but this is the responsibility of the evaluator

Proposed decision: No change

40 Jonathan Avila 3.2c Open include states of pages Under the section "identify the variety of page types" it is worthwhile to strength or include different states of pages such as error states for forms, dynamically added content, simulated dialogs or pop-ups, or page renderings based on different user settings and permissions including whether an accessibility enhancement settings are enabled.

We added this to 3.4.a.

Proposed decision: Clarify states of pages in the document. See if it should be in more places than 3.4.a
22 Peter Korn 3.2d Open Add the suggestion to note UI component sets & versions, if any, used (e.g. "JQuery UI version 1.9") Particularly for web applications, much of the accessibility support is built into the UI component sets.

The tools used in the development process can be important. Many errors are in the toolkit that is used by developers. We could prompt the evaluators to record this information if they know it (optional). It may help them diagnose accessibility issues.

Proposed decision: We will add this to 5.a as optional bullet under 'As documentation for each Step, describe': "Optional: Add information about the tools used in the development process if they are of relevance for the accessibility findings and can help to better understand possible accessibility issues found during the evaluation"
35 Aurélien Levy 3.3b Open give a minimum number of pages to include in the requirement 3 text as this section is defining a minimum number ofpages per features we can give a minimum number of pages to include in the requirement 3 text. Furthermore, I'm not really sure that taking two pages with the same kind of content (pages with data table for example) is useful if those pages are generated with the same template

The overall number is depending on the website, this is why we propose two pages as a minimum (if available)

Proposed decision: Consider adding minimum number of pages to include in section 3. For now: No change
34 Aurélien Levy 3.3 Open consider website composed by an unique webpage (where part of the page is updated with ajax somewhere Proposed decision: This will be clarified more in the introduction of section 3.
44 Gian Wild 3.3 Open Add an option for testing all pages. There are some automated testing tools available now that will test an entire web site. In addition to this, there are some companies (including our own) that complete manual and automated testing on *all* pages, and this should be represented in the evaluation methodology. It is true that template issues will be found by selecting a representative sample, however content issues - such as use of headings, coded field labels etc, can only be found if all pages are tested. Although this might be outside of the capability of most testers, it is something that should be covered in this evaluation methodology. Proposed decision: It will be clarified in section 3.3 that optionally, testing can include all pages of a website and does not have to be limited to the sample if that is the wish of the evaluation commissioner (optional)
41 Jonathan Avila 3.3 Open Choose reused portions of pages In "Step 3: Select a Representative Sample", it may be useful to choose portions of pages that are reused such as header sections, navigation structures, etc. While these portions are not full web pages and full web page conformance must general be met, capturing sections of the page can be useful in targeting violations, limiting repetitive evaluation of the same structure on each page, etc. An XPath or other identifier could be used to identify these sections within a given page. For all the pages in which this section or widget appears violations could then be applied as pattern violations thus saving time during the evaluation process. Proposed decision: Clarify this in section 3.2.c. Already 'header' is there, clarify with header sections, navigation structure. Also check if this is used in the reporting section. Also check section 3 introduction last lines for coverage.
42 Jonathan Avila 3.4d Open Step 4d calls for the archival of the pages and capturing of screenshots. This step is very important and is something that SSB does during the sample collection of the site. In addition, it may be important to record the steps (path) used to reach a page including any data entered, user rights, etc.

Recording of the paths used including any data, user rights etc is done in section 3.4.a and returns in 3.5.a (second bulletpoint)

Proposed decision: No change
16 François Junique 3.5c Open A special score should be available when only automatic measurements have been possible

This would then not be a full conformance evaluation as meant by this methodology

Proposed decision: No change
43 Jonathan Avila 3.5c Open add a performance metric and a unified process for its calculation Regarding providing conformance scores Step 5c, the current wording indicates that a score of non-conformant is likely to occur and that score is important. This is indeed the case as many sites support WCAG but are not fully conformant to a given level. WCAG requires conformance on a per level basis with only a few exceptions. In reality the accessibility of the site may be experience differently to different user groups. For example, a site may be accessible to users with hearing impairments via captions, visual equivalents for sound, etc. but may not be accessible or fully conformant to WCAG success criteria related to access by persons who are blind. Knowing the relative score may assist users in making decisions to use a particular site based on their needs. While a goal of full conformance to all criteria is key -- conformance may be an evolving process within an organization and may change with updates to a site. Thus, a performance metric and a unified process for its calculation is important to this field. Proposed decision: We will investigate the concept of a performance metric and unified process for its calculation for section 3.5.c.
45 Sharon Litchfield 3 Open I would like to see more discussion/specifics around the quantity of the sample pages selected to evaluate. I think some categories could be applied to this such as: 1) small sample of website - less than 25% of pages; 2) medium sample of website - between 25 - 50% of pages; 3) large sample of website - between 50 - 75% of pages; 4) extensive sample of website - greater than 75% of pages; 5) full evaluation - 100% of pages. I think having some categories like this would allow greater transparency of how much (quantity of pages) of a website is actually being assessed and evaluated. Selecting a sample size has different implications for websites of different sizes, and the category selected may vary depending accordingly.

Currently the outcome of the discussion was that the sample depends on the website as described in the introduction of section 3. We will look into the need to clarify the number more in depth.

Proposed decision: No change

Comments for Section 4 - Conformance with this Methodology

Listing of comments for Section 4 - Conformance with this Methodology and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
17 François Junique 4 Open what is this? If we were to have one day a EU legislation of web-accessibility based on the usual EU "new approach" on regulation, we would have some mandatory essential requirements and a presumption of conformity would be achieved by conforming with the M376-created standard using this WCAG-EM (if part of it...). Already the reliance for testing this conforming on agilely evolving techniques docs is tricky but if in addition the WCAG-EM is only one option for checking the conforming to the std (not talking about conforming to the essential requirements), this would be even more opening fuzziness

This section could provide clearer indication of the requirements to conform with the Guidelines by specifying the requirements to claim conformance using WCAG-EM. This section needs more work and coordination with other activities

Proposed decision: Clarify section
38 Jonathan Avila 4 Open The goal of "Basic conformance" as stated in the EM appears to be offer an option to essentially self certify without having to perform real evaluation of the site. This level of conformance is likely to be incorrectly applied and would likely mean that the site would in fact not be conformant. In cases where some national legislation references WCAG this could allow someone to argue conformance without defensible claims. making an assumption on conformance, while often well intentioned, is not safe for sites that have not previously been evaluated. The working group should consider allowing this level of conformance only for conformance claims that are made after updates are made to an already conforming site. Open

Comments for Section 5 - Limitations of this Methodology

Listing of comments for Section 5 - Limitations of this Methodology and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
4 Greg Gay 5 Open Change language on the requirements for a conformance statement Only detailed reviews can potentially include a conformance statement, I believe there should be strong language around the issuance of conformance statements, and when they are appropriate to make. Inevitably suits will arise, and evaluators are putting their necks on the line if they start using the word “conformance.” Proposed decision: clarify the section 3.5.b with this comment in our head.
18 François Junique 5 Open seems to contradict some of the options in 3 step 1.b Proposed decision: Clarify this

Comments for Appendices

Listing of comments for Appendices and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
25 Peter Korn Appendix C Open the examples all list "Person who did the evaluation". This will not always be appropriate - e.g. in a self-assessment from a corporation of its own website or web application, this should often be the corporation's name and not an individual name.

Decision 1: We will change it into "Person or organization"

Decision 2: Add "Tools used by Evaluator (optional)" and "Evaluation Commissioner (optional)"

Decision 3: Add "Contact/Address"

Clarification Edits

Listing of comments with suggested clarification edits and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
19 Peter Korn Abstract, 3rd bullit objective Open "Aggregating individual results into an overall conformance statement; this includes defining approaches for assessing the relative impact on failures, potentially through incorporating tolerance metrics." This can be helpful Open
6 François Junique 1.4 Open "web-site-part" unclear clarity Proposed decision: Clarify the term using section 3.1a Scope.
7 François Junique 2.3 and 3.1b Open Note: more should be investigated in particular in relation to 2nd option in 3 step 1.b Proposed decision: More discussion in EvalTF about the three goals and the naming of them. How do they relate. Also see previous comments.
8 François Junique 2.5 Open should be clearer if the intention is to verify website accessibility or conformance to WCAG2.0 clarity Proposed decision: Clarify the text to make clear that users are included to verify WCAG 2.0.
29 Aurélien Levy 3.1b Open The detailed review and in-depth analysis spoke about "errors". Is it success criteria errors or specific dom node errors? Clarity. In case it's dom node errors I don't think it's useful to have information about every identified errors including counting the number of errors and their locations within the web pages specially if the error is a repeatable one. Proposed decision: Clarify SC errors
9 François Junique 3.1d Open the implicit/interpretable-from-reading concept of for-specific-disability might be dangerous clarity Proposed decision: This is not the idea behind the text. Clarify this.
10 François Junique 3.1e and 4b Open both essential for agile adaptation to technology evolution and rather tricky as the techniques are said to unstable non-standardisable docs, nevertheless not all that clear regarding how to handle the passing and failing ones (plus the many "other advices" type ones) clarity Proposed decision: Clarify in text
31 Aurélien Levy 3.2a Open Definition of "template" Clarity. Is it the CMS template without any content ? If it's corresponding to CMS template, sometimes the commissioner can't have access to this kind of information (proprietary CMS, CMS without template mechanism).

Yes, it is CMS template, can be with or without content. Can be lore ipsum, but also pages with live content.The idea is that only templates available to the evaluator are defined.

Proposed decision: No change
23 Peter Korn 3.2d Open It would also be appropriate to include Java in the list of auxiliary web technologies. Clarity Proposed decision: Add Java to the list of auxiliary web technologies in 3.2.d.
11 François Junique 3.2 and 3.3 Open not so clear how relevant for each of the 3 options in 3 step 1.b clarity Proposed decision: Clarify relevance for different goals
12 François Junique 3.4 Open use-case: what is this: disability-specific clarity Proposed decision: Add definition of Use Cases in section 1.4 Terms and Definitions and clarify in 3.4
13 François Junique 3.5b Open the term "accessibility statement" seems to often used for some thing wide than a declaration of conformity and more for user informationt clarity Proposed decision: Limit to one term, use accessibility statement OR conformance statement
14 François Junique 3.5b and 3.5c Open it might be beneficial to use a richer score than a "one-figure" score, e.g. for each technology used a matrix of wcag2.0-principles versus use-case scoring Proposed decision: Has been changed in editor versions reflecting this comment. No further change for the moment
15 François Junique 3.5b and 3.5c Open where will be defined for 1st option of 3 step 1.b, what are the tolerable number of errors to still conform clarity

The tolerable number of errors is described in the introduction of section 3.4. There is no tolerance for the sample. All fails lead to non-conformance of the total sample.

Proposed decision: No change
26 Peter Korn Appendix C Open more example information would be helpful with respect to what the results should look like (in the penultimate bullet "Results: per guidelines, checkpoint...") Clarity Proposed decision: Clarify what the results should look like

Purely Typograhpical

Listing of purely typographical comments received and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
24 Peter Korn 3.2d Open "auxillary" "auxiliary" Typo Change to auxiliary

Additional Comments

Listing of additional comments received and their proposed resolutions
ID Commenter Location Nature Current Text Suggested Change Rationale Resolution
36 François Junique Open I don’t see where you plan to discuss handling/evaluating the different versions of a web site (based on different available style-sheets or dynamically generated content (UA or server side - non only js based) based on user profile/preference or device type or natural-language.

We cover this in 3.1.d Define the context of website use. This section requires that the target users of the website and minimum set of web browsers and assistive technology to evaluate for shall be defined. Additionally, in section 3.2.b Identify Key Functionalities of the Website. section 3.4.a requires to check for the broadest variety of use cases.

Proposed decision: Clarify handling different version of a website in sections above and also in 3.5.a.
37 Jonathan Avila Open The scope of the methodology is indicated to apply to full websites. While we applaud this effort and agree that sites should not be cherry-picked for accessible pages or features -- it may not always be practical for entire websites. Thus, the idea of applying the scope to departments or smaller portions of a site will likely be important. For example, a particular purpose of the site such as the online forum or reservation section of the site and whole pages within may be targets of the evaluation. Similarly allowing websites to report on conformance while still have non-accessible third party content such as social media links/log-ins/ads is a good idea. While in principle third party content should also conform sites should be credited with their efforts to be conformant of what they have control over.

This could be covered by partial conformance as described in WCAG2.0.

Proposed decision: The link to partial conformance will be clarified in section 3.5.b and 3.5.c
39 Jonathan Avila Open The goal of "Basic conformance" as stated in the EM appears to be offer an option to essentially self certify without having to perform real evaluation of the site. This level of conformance is likely to be incorrectly applied and would likely mean that the site would in fact not be conformant. In cases where some national legislation references WCAG this could allow someone to argue conformance without defensible claims. making an assumption on conformance, while often well intentioned, is not safe for sites that have not previously been evaluated. The working group should consider allowing this level of conformance only for conformance claims that are made after updates are made to an already conforming site. Open
47 Greg Gay Appendix C Open Dates are also important in a conformance statement.

The date is already in the appendix C for the report for conformance and for date of evaluation

Proposed decision: Add date to 3.5.b for the accessibility statement

Proposed decision: in 3.5.b use of Accessibility statement and of Conformance statement. Choose best term and use that consistently

Previous Commments

Please note that we also have a few comments left open from before the publication of the Public Working Draft. These can be found at http://www.w3.org/WAI/ER/conformance/comments-20120306