QAWG Last Call Issues

Specification reference.

Any spec SHOULD have a Security Consideration section. Protocol or behavioral specs MUST have a Security Consideration section.

Security sections make spec authors think about potential vulnerabilities and address at least some of them before the bad guys can exploit them. These sections are also a great place to warn implementors and users about most security-sensitive areas of the spec and, perhaps, common exploits.

IETF's Internet Architecture Board has published the following Internet Draft that may be of use to SpecGL authors: http://www.ietf.org/internet-drafts/draft-iab-sec-cons-03.txt

Discussed at 20030331 telecon. QAWG believes that the proposal has merit, but that specifying such requirements is outside of the scope of the QA Framework. Will be referred to appropriate W3C group or team.

Specification reference.

It seems as if either 'defines' or 'explains' is meant, not both. The first sentence is grammatically incorrect, thus the intended meaning is unclear. The second sentence might seem superfluous, depending on the intended meaning of the first.

Specification reference.

Remove the Committment Table and CP1.1, 1.2, 1.3. These items do not add information and are redundant. To satisfy Level 3 (5 or 7), you must satisfy other checkpoints in the OpsGL. In fact, if we did our jobs right, Level 3 should equal Conformance Level A (satifying all P1), Level 5 should equal Level Double A, and Level 7 = Triple A. Isn't it a goal to get WGs to conform to the OpsGL? That would mean that they must satisfy all the P1 checkpoints. Thus, they must have a committment level to P1. So, why do we use a new term - Level 3.

The Table introduces a DOV - called committment level. The OpsGL does not conform to the SpecGL's CPs that require DoVs to address the relationship to conformance, to other DoVs, etc.

Email discussion , including proposed resolution.

Discussed and resolved at 20030501 telecon and subsequent email . Table will be eliminated, and CP1.1 - 1.3 will be replaced. CP1.1 will require choice of A-AAA for OpsGL, choice of A-AAA for SpecGL, and choice of A-AAA for TestGL. Plus 2 additional CPs preserving the commitment to have some test materials.

Related LC issues: 3, 60.2, 72.2, 72.3, 83, 107.

[LH] See email.

Specification reference.

See subsequent email proposal, to remove empty (unused) definition.

Related issues: 4, 32.

Specification reference.

Discussed at 20030331 telecon, classified as editorial.

Specification reference.

Specification reference.

Discussed at 20030407 telecon. Resolved "yes", they can be in a separate document, referenced from the main document.

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Specification reference.

Previous email: ISO 'scope' definition; QAWG discussion thread.

Discussed at 20030331 telecon, classified as editorial. Action. (Someone) request clarification from originator.

Specification reference.

Discussed at 20030331 telecon, classified as editorial.

Specification reference.

Specification reference.

Discussed at 20030331 telecon, classified as editorial.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

(LH clarification.) The example does not correspond to the requirements that are listed immediately before it -- apparently the latter were updated without updating the example.

Specification reference.

Discussed at 20030331 telecon. QAWG agrees that SpecGL should be AAA-conformant to SpecGL, and commits to AAA conformance by Candidate Recommendation.

Discussed and resolved at June QAWG face-to-face. QAWG (MS) will develop a set of test assertions for SpecGL.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

Related discussion about whether testability is an inherent part of "conformance requirement" in this email sub-thread (of a thread about some OpsGL issues).

See mail thread (20030606), which includes suggested resolutions.

See also these directly related QAWG issues: issue #98, issue #99, issue #110.AR-001. These were marked as closed at Tokyo and after, however later email and telecon decisions, including new agreed definitions and some samples, suggest reviewing the earlier closures.

Discussed and resolution re-affirmed at 6/2003 QAWG face-to-face.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

Extension-group processing plan and subsequent email discussion.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Discussed at 20030331 telecon, classified as editorial.

Discussed at 20030410 telecon. AI given to DM (LH & DH assisting) to draft clarification.

New proposal circulated. Discussed and re-resolved at 20030505 telecon. 8.4 will be kept, broadened to "items" instead of "choices", and final text (derived from the new proposal) will be revised and clarified by SpecGL editors.

Discussed again at 20030908 telecon.

Related LC issues: 16, 27, 39.

Specification reference.

(See subsequent email discussion.)

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: there must be a single starting point to find all deprecated features. This could for example be a subsection of the Conformance Clause. It could consist of a table or list of links to individual parts of the specification.

Specification reference.

(See subsequent email discussion.)

Discussed at 20030331 telecon, classified as editorial.

Specification reference.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: The wording has been changed to eliminate this confusion.

Specification reference.

See related issue LC-96.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: one section. As CP3.1 has now been clarified in other issue discussion, its nature looks somewhat different. It is a summation of information that could be deduced from other individual requirements (although the process could be difficult and error prone). Therefore only "single section" makes sense.

Specification reference.

Email comments, including proposal (20030418).

Discussed and resolved at 20030421 telecon.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Specification reference.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: [provisional resolution -- revisit] Yes, a part of CP2.3 deals with where the information needs to be placed. Due to many changes, this intro text may need to be rewritten. For now, we have deleted the mention of which Guidelines fall into which general type. It may be that a guideline falls into both categories.

Specification reference.

I think this checkpoint is too vague. Is it possible to break it down into a few more concrete requirements that are more verifiable? For instance:

• For markup specifications, provide at least one example of each markup construct.
• For protocol specifications, provide at least one example of ...
• For transformation specifications, illustrate each transformation capability with an example showing input and output.
• For UI specifications, provide an example of each construct, and illustrate the desired output using at least one mechanism other than the specification itself (e.g., the SVG specification should not rely on SVG rendering alone to explain what something should look like).

While you may miss some cases, I think spec editors will find this more helpful than the general goal to "provide examples."

Email discussion and discussed in 20030307 telecon. Alternatives:

• Alt.1: Implement the proposal: make more specific requirements and tie it to specification categories.
• Alt.2: Implement the proposal in the ExTech document, but leave CP as is.
• Alt.3: Implement the proposal in the ExTech document, and make CP more specific without mentioning categories. (I'm not sure how we do this).

Resolved in favor of a variation on Alt.2 (which might be what was meant by Alt.3?) -- the discussion and breakdown by category will be done in ExTech. But the Discussion of CP1.4 will mention the concept raised by comment originator (that different kinds of specification categories require different particular associated kinds of examples), and maybe even give a single brief "for example."

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Specification reference.

Specification reference.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: Agreed, the text has been deleted as suggested.

Specification reference.

Adopt the UAAG 1.0 approach of separating requirements from applicability exclusions (called "normative inclusions/exclusions" in UAAG 1.0). Suggest that the statment "It is not applicable if..." be labeled as "Normative inclusion/exclusion" as in UAAG 1.0

Discussed at 20030331 telecon, classified as editorial.

Email discussion, including proposal.

Discussed and closed at 20030602 telecon (DRAFT). QAWG appreciates the utility of the "normative exclusion" concept in the complex conformance landscape of UAAG 1.0. For the few and simple occurrences of the "not applicable" usages in SpecGL, QAWG considers that it is cleaner and clearer not to introduce a new concept and terminology such as "normative exclusion".

Related LC applicability/normative-exclusion issues: 26, 73.2, 80.

Specification reference.

CP 7.1, 7.4, 7.5 ConfReq, last sentence: change 'is' to 'are' in "...if there is no deprecated feature"

CP8.4 Rationale: change 'identifying' to 'identify'

CP9.1, last para: "This is strict conformance" is a repeat of text in intro of GL9

CP 11.4, last para: modify "proper use of the conformance icons" to "proper use of any conformance icons"

GL13, last para: suggest switching the order of Manual of Style and PubRules

Related LC issues: 16, 27, 39.

Specification reference.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: Agreed, the stylesheets have been modified as suggested.

Specification reference.

1. In general, specifications make functional requirements. In some cases, it may be necessary to make a performance requirement. How should that be handled?

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: if performance requirements are within the scope of the standard, and they presented the same as any other conformance requirement -- e.g., using RFC2119 keywords -- then there would not seem to be any issue. For conformance purposes, they would not need any special handling. (No changes to SpecGL.)

2. In UAAG 1.0, we realized that for some of our configuration requirements, it didn't matter whether the user agent offered the desired configuration or didn't implement the functionality in the first place. E.g., we decided that it was ok for a user agent to not support blinking at all, or, if blinking is implemented, to offer a configuration to turn it off. However, in other cases, the configurability was "just as important" as the functionality to be configured. Authors should consider these when they specify configuration options.

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: Since SpecGL 1.0 has not gone to such a level of detail as addressing configuration options, we are unsure how this comment would affect SpecGL. Note that we (QAWG) have decided that the scope of SpecGL 1.0 should be limited to the most critical and basic topics. This could be added to requirements for SpecGL 2.0. It could also be mentioned, possibly, in SpecET (Examples & Techniques). If Originator thinks that this topic fits under one of SpecGL's existing checkpoints, then perhaps he would be interested to make a contribution, to be put into SpecET. The SpecGL scope has been clarified to indicate the "basic and critical" scope of SpecGL 1.0. See draft revised text.

3. It might be valuable for a specification to explain how to include its requirements in another specification. This is not the same as explaining how to reference the spec, or how to claim conformance to it.

   Discussed at 20030707 telecon. We (QAWG) have decided that the scope of SpecGL 1.0 should be limited to the most critical and basic topics. We think that this topic has merit, but is more advanced and detailed than we are dealing with in SpecGL 1.0. It could be added to requirements for SpecGL 2.0. Alternately or in addition, it could also be mentioned in SpecET (Examples & Techniques). If Originator thinks that this topic fits under one of SpecGL's existing checkpoints, then perhaps he would be interested to make a contribution, to be put into SpecET. The SpecGL scope has been clarified to indicate the "basic and critical" scope of SpecGL 1.0. See draft revised text.

4. It might be valuable to explain some desirable characteristics of a specified technical requirement:
   • Mutual independence from other requirements
   • Expresses a minimal requirement
   • Distinguish and label: requirements, exceptions to those requirements, necessary and/or sufficient techniques for satisfying those requirements.

   Related discussion about whether testability is an inherent part of "conformance requirement" in this email sub-thread (of a thread about some OpsGL issues).

   See mail thread (20030606), which includes suggested resolutions.

   For test-assertion topics, see also these TA-related QAWG issues: issue #98, issue #99, issue #110.AR-001.

   Discussed at June QAWG face-to-face. Further clarification needed from originator, sought and discussed in email thread.

   Discussed and resolved at 20030623 telecon. Agreed that it is useful to discuss some attributes of well-written conformance requirements. Some verbiage has been added to the introductory text of GL7 (new number). More extensive material may be added to SpecET (contributions would be welcome!). See draft revised text.

5. I think that more could be stated about useful ways of allowing extensibility. See, for example, how CSS (forward-compatible parsing), XML, and HTTP handle this. What should be avoided? Is the general practice of "ignore what you don't know" a good idea or a big mistake?

   Extension-group processing plan, including proposals, and subsequent email discussion.

   Resolved at 20030505 telecon, as part of the extensibility issue group resolution. This should not be covered in SpecGL, but it ought to be covered in the SpecET material on extensions. SpecET should illustrate numerous kinds of extensions -- some that have serious interoperability impacts, some that mitigate the impacts, some approaches that might be considered "useful", etc. Also, QAWG discussion sees some problems with examples such as CSS "forward compatible parsing" -- it opens up the ill-defined topic of what is an unknown extension versus what is corrupted or erroneous content. This has been added to the SpecET queue.

6. A spec should clearly indicate which illustrations (e.g., images) and examples are normative, if any.

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: agreed. The wording of CP13.2 has been adjusted to include all sorts of normative content. See draft revised text.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

Discussed at 20030410 telecon. Disagreement about keeping or merging the concepts. AIs given to: draft a proposal merge the concepts; survey the usage of the terms and concepts in W3C specs.

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. QAWG agrees that clarification is needed. The solution for the whole profile/module/level issue group is:

• the concepts of profiles, modules, and levels are kept separate, each a separate DoV -- although the concepts are sometimes misused, and although other ways to subdivide the specification could be postulated, the QAWG believes that these three are heavily used in current practice, and that SpecGL can help to improve consistent;
• a subsection (sec 2.3?) to clarify the definitions and concepts of profiles, modules, and levels will be put into the new Ch.2, "Concepts";
• the three guidelines GL4,5,6 will be combined into a single guideline, with reduced verbiage, mostly pointing back to section 2.3;
• however the checkpoints will be kept substantially as they are, each dealing individually with one of the concepts of profiles or modules or levels -- while there is interest in combining checkpoints, on the other hand it was tried in the 1st PWD of SpecGL (May 2002) and caused too many problems;
• the exception is that the three separate "relationship to other DoV" checkpoints will be combined into one.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

General comments/conformance model I think the document is well-organized, clearly written, and very helpful. Having spent a lot of time thinking about conformance issues (notably for UAAG 1.0), I thought it covered a lot of ground and did so well.

I like the distinction between requirements that relate to the conformance model and those that involve implementing the model in the spec.

Specification reference.

See subsequent email proposal, to remove empty (unused) definition.

Related issues: 4, 32.

Specification reference.

Hmm, "used" may not be a specific enough term. A spec may encourage a UA to support a feature, but discourage an author from producing it, and may be removed in some future version.

See email discussion.

Discussed at 20030410 telecon. Resolved: agree with the suggested clarifications, will make appropriate changes.

Related LC deprecation issues: 33, 40, 99.

Specification reference.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: agreed (and "mandatory" is added to the first sentence).

Specification reference.

See extension-group processing plan, including proposals, and subsequent email discussion.

Resolved at 20030505 telecon, as part of the extensibility issue group resolution. Although CP9.3 may be redundant, the redundancy doesn't hurt here. Not every reader is going to logically connect the dots, as has the commentor. Therefore CP9.3 (new CP6.2) should remain as is.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Discussed at 20030410 telecon. See also subsequent email thread.

SpecGL uses a focused definition of normative -- normative stuff is directly connected to conformance requirements -- that is narrowly applied. By this definition and its application, QAWG has considered that things like Glossary definitions ("5. Definitions") are not normative. SpecGL WD text (20030912) clarified what's normative in the subsection "Normative parts" in "4. Conformance". Also, in "1.6 Usage of terminology", it clarified that defined terms, when used in SpecGL, have the meanings given in the definitions. Because of the latter, it is actually irrelevant from a SpecGL conformance standpoint whether definitions are labelled "normative" or "informative" -- the conformance requirements for SpecGL are clear and unambiguous. This solution was illustrated in WG-only draft revised text.

Discussed again at 20030908 telecon, at which QAWG decided that the most expeditious (and harmless) way to close the issue pre-CR was to call definitions normative.

Related LC whats-normative issues: 36, 65, 106, 108.

Specification reference.

See extension-group processing plan, including proposals, and subsequent email discussion.

Resolved at 20030505 telecon, as part of the extensibility issue group resolution. Although CP9.3 may be redundant, the redundancy doesn't hurt here. Not every reader is going to logically connect the dots, as has the commentor. Therefore CP9.3 should remain as is.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Discussed at 20030407 telecon. Agreed that they should be combined. Details tbd by SpecGL editors.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Discussed at 20030331 telecon, classified as editorial.

Discussed at 20030410 telecon. AI given to DM (LH & DH assisting) to draft clarification.

New proposal circulated. Discussed and re-resolved at 20030505 telecon. 8.4 will be kept, broadened to "items" instead of "choices", and final text (derived from the new proposal) will be revised and clarified by SpecGL editors.

Discussed again at 20030908 telecon.

Related LC issues: 16, 27, 39.

Specification reference.

See email thread.

Discussed at 20030410 telecon. No general consensus yet. There were differing views on what obsolete means (is it closer to deprecation or to removal)? The nature of the HTML example needs to be looked at. LR takes action to discuss with originator, look at HTML, draft proposal for QAWG discussion.

Related LC deprecation issues: 33, 40, 99.

Specification reference.

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. See the description of the solution for the profile/module/level issue group, in LC issue 30.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

Discussed and resolved at 20030602 telecon (DRAFT). While the verbiage in GL3 is informative, versus the normative requirements of CP1.4, nevertheless the SpecGL editors will look at the text in question and fix it if MUST does indeed apply in the scope and context of the text in question.

Related LC GL3-policy issues: 42, 96.

Specification reference.

The following comments about the QA Process Document (QAPD), which is linked from the Ops Extech, are addressed in draft revised QAPD text. At 20030512 telecon, proposed and resolved that the new text resolves the issue.

1. - "test material" is referred to in multiple ways in the document including but not limited to "test material" and "test material type". We could not determine a difference in concept from context nor could we come up with an example that needed both.
2. - The language in Rec 1 seems odd. Especially the second sentence. "The Checklist should be developed in a public framework. The development of this framework itself is a central area of interest. The QAWG welcomes the participation of interested parties in developing the Checklist."
3. - The third sentence in Rec 3 also seems a bit odd
4. - We found a number of uses of "test materials" and "test suite" that did not use the "fill-in" markup.
5. - DocumentFramework should be Document Framework
6. - In Test materials contrabuition section in the a/b choice rather than saying tests or series.. say contributions.
7. - In Test Materials contribution section there is a missing 'span' element around address. above]'/span' mailing list: [address].'p'
8. - We found the first item in the list under Test Materials Contribution to be some what redundant. At least in our case as any submissions would already go to the list as it is. The test or series of tests, get submitted to the framework. Submitter should also send a notification to the mailing list that will be set up for the Checklist to indicate that he/she has submitted a test to the class="fill-this-in">[test material name] framework.
9. - There is a typo in section Receipt and review of test contributions "according tot he" should be "according to the"
10. - In Receipt and Review section list item 1 it currently says: "since it tests a feature that is not in the specification" should it say something more like: "since it tests a recommendation that is not in the specification" as 'feature' is not a universally applicable term.
11. - In Receipt and Review section list item 3: This item makes the assumption that the test cases will be written in some xml based language. How ever can we make this assumption? What about languages like rdf that don't use an xml syntax?
12. - this same section (item 3) also has a hanging ']'/span'' on it.
13. - In Receipt and Review section list item 4: "'li''em'Scenarios'/em' that underlay the particular test layout and its intended scope.'/li'" We found the use of "test layout" to be out of place in the document. it is the only place that this term is used. We believe that it could just as easily use language that's more standard to the document.
14. - In Receipt and Review section list item 5: The point of this checkpoint is valid, in that if the submitted test doesn't follow the Development Guide it should be rejected, how ever the notion of the Development Guide is introduced here (in this document). We feel that it should be listed earlier in the document as well. perhaps in the requirements, that the test materials be developed in accordance with the Development Guide.
15. - Typo in last sentence of Receipt and Review section: is "publication arereurned to" should be: "publication are returned to"
16. - In Test status and review procedures Section: In the second paragraph it reads: "the status of the test is changed to reflect the fact that its validity has been disputed" Earlier we say that "status is changed to "accepted" we should use the same verbiage here like: "state changed to "disputed""
17. - In Test status and review procedures Section: The last sentence of the second paragraph has very odd wording.
18. - In Test status and review procedures Section: second paragraph uses term "Task Force" in terms of the maintainers of the test materials. This is another concept that is first introduced in this context. This too would work better if the concept of appointing a "Test Task Force" was done at an earlier step.
19. - In Test status and review procedures Section in the first item of the list. The label "stable" may not be the best word for case 1 as it doesn't really reflect what is meant. "Error Free" or "Valid" would work better.
20. - In Test status and review procedures Section in the third item of the list. It reads: "questioned the correctness, consistency, clarity, etc" we think that correctness does not belong.
21. - In Test status and review procedures Section in the second and third item of the list. "A consensus exists in the community" What community? The w3c, the 'whole internet' the WG (as this is specifically talking about the task force.) We are also fairly uncomfortable with the notion that the task force can be over ridden by the (not clearly defined) community. Either the taskforce agrees or disagrees and the case accepted or not. Unless "community" means WG in which case this might make sense...
22. - In "Section Status of the test suite according to the above" We feel that the wording of this section heading could be improved quite a lot. We assume this means that if you conform to all of the above. or you've done/are doing all of the above.
23. - In "Section Status of the test suite according to the above" second paragraph It reads: "It is proposed that the W3C WG representative act as moderator and controller for incoming tests. The moderator is chosen by the [WG name] WG. All tests should be kept for archive purposes, whether they get published or not." Much of this is repeated from above. The new information is that a moderator should be chosen. This is another case where it may be beneficial to introduce the concept of a Moderator earlier in the document. This is already covered above to some degree. But with moderator.
24. - Both the Documentation and "See above" sections do not have references back to a GL document.

Specification reference.

This document is applied and conformance (to this document) achieved as new TRs are being written." (grammatically incorrect, intended meaning unclear) This document applies to new TRs and conformance (to this document) is achieved as they are being written.

As for legacy specification, they may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints. (grammatical/spelling error) "legacy specifications"

Within this Specification Guidelines document, the term "specifications' is specifically limited to W3C Technical Reports, even though these guidelines could be applied to other documents. (unbalanced quote marks)

Specification reference.

Specification reference.

I'm finding it difficult to follow which terms are reserved for 'classes of products'. The word 'consumer' is used to describe the classes of products, and itself is listed within the classes of products. For example, I find the following sentence semantically confusing: "For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification." "For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way." "Play" refers to a media player, or play refers to "process" ?

Divide this (enumerated) list into processor, consumer, or content? Make the terminology in this area unique, so that there will be no ambiguity? (It could be that the terminology is already unique, but in its current format, I can't be sure.)

Email discussion.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon. Rewrite the GL verbiage for clarification. The detail about sub-dividing the list of classes is to be decided by SpecGL editors during rewrite of the verbiage.

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

Specification reference.

"Checkpoint 2.3. Identify which of the categories of object are specified in the document as a whole. [Priority 3]"

Reader should be able to understand what 'categories of object' are upon reference - this is the first instance that this phrase is used in this document. Even though a URI is provided to the applicable definition of 'categories of object', the definition itself should introduce the list with this phrase. Eg: "Most specifications can be classified into one of the following categories of object ..."

Discussed at 20030418 telecon.

Discussed and resolved at 20030418 telecon. Rewrite the GL verbiage for clarification. "Categories of object" will be replaced by the standard terminology, "specification category".

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

Discussed at 20030410 telecon.

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. See the description of the solution for the profile/module/level issue group, in LC issue 30.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

Ambiguity or error? (I count four 'of's in the second clause! :D)

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. See the description of the solution for the profile/module/level issue group, in LC issue 30.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

Discussed at 20030410 telecon.

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. See the description of the solution for the profile/module/level issue group, in LC issue 30. Note that the specific suggestion to use diagrams was not addressed (SpecGL editors?).

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

Extension-group processing plan and subsequent email discussion.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Specification reference.

Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Specification reference.

Discussed at 20030331 telecon. QAWG believes that the proposal has merit, but that specifying accessibility requirements is outside of the scope of the QA Framework. It more appropriately belongs in pubrules. Will be referred to Comm team.

Discussed with commenter in a couple of QAWG email threads.

Specification reference.

Discussion. See previous QAWG issue 12.

Email discussion, including proposed resolution.

[LH] See email.

If, on the other hand, accessibility requirements are not integrated into the target specification as conformance requirements, then the problem is beyond the scope of the QA Framework as currently defined (for more about this scope point, see email discussion.) In this case, the issue is larger than OpsGL (or TestGL or SpecGL). QAWG issue #12 explored this -- the relationship amongst the QA, WAI, and I18N horizontals.

Specification reference.

Email discussion, including proposals.

Discussed at 20030512 telecon, and resolved similar to email proposal. There is a strong connection between some guidelines/checkpoints (GL/CP) and the stages in a WG's life. However, the link is ideal, not absolute, and the GL/CP are written in recognition of this. There is already a close association of GL/CP order with WG life cycle. For this reason, and to avoid disruptions to the OpsGL structure and text that don't fix serious problems, QAWG resolved not to attempt a major re-organization of GL/CP based on chronology. However, an informative section will be added to chapter 1 (Introduction), explaining the chronological connections with a graphics or a table.

See also related issue LC-60.1.

[LH] See email (essentially same as Originator's).

Specification reference.

[LH] See email closure proposal.

Specification reference.

The following seven items discussed in email thread, including proposals, and at 20030512 telecon, and resolved similar to email proposals. Resolutions are illustrated in draft revised text.

1. "QA deliverables" is very broad and not defined (GL3)

   Per email proposal, the ConfReq is made more precise, the "For example..." in Discussion is replaced with minimal content of "QA deliverables", and broader examples are also presented. See draft revised text.

2. "ensure" is not testable in a conformance requirement (at least CP 3.2, 6.1)

   Per email proposal, the ConfReq is reworded (in both CP3.2 & 6.1) to replace "ensure" with more testable language. See draft revised text of CP3.2 and CP6.1.

3. "commensurate" isn't either (CP 4.2)

   Per email proposal, the ConfReq is reworded to eliminate "commensurate" and make it more testable, and a new clarifying sentence is added to Discussion. See draft revised text of CP4.2 and CP2.2.

4. using the future makes a CP untestable (CP 6.3)

   Per email proposal, the ConfReq is reworded to eliminate the problem. See draft revised text of CP6.3.

5. "a quality assessment" is not well defined (CP 7.1)

   Per email proposal, a handful of examples are added to Discussion, to illustrate what goes into a TM quality assessment, and a more exhaustive list/template will be put into OpsET. See draft revised text of CP7.1.

6. "sufficient" is not testable (CP 7.2)

   The comment is moot for the occurrence of "sufficient", as it is in the statement of the CP, which is merely its "title" (not normative). However "commensurate" in the ConfReq is fixed, per email proposal, and a clarifying sentence is added to Discussion. See draft revised text of CP7.2.

7. 'all' is not testable (CP 7.3)

   Per email proposal, delete the occurrence of "all" in ConfReq. See draft revised text of CP7.3.

Specification reference.

1. Overall, I think we have the right checkpoints, but I think they would be easier to understand if they were grouped chronologically. Start with what needs to be done when the WG is formed, and proceed through the spec and test development cycle to the maintenance phase, as we have tried to do with the restructuring of TestGL.

   Email discussion, including proposals.

   Discussed at 20030512 telecon, and resolved similar to email proposal. QAWG resolved to minimize large changes to the OpsGL structure and text that don't fix serious problems. Therefore, OpsGL will not be reorganized based on chronology. Further, there is already an approximate association of guidelines and checkpoints (GL/CP) with the ideal point of applicability in a WG life, and this is partially reflected in the GL/CP ordering. An informative section is added to chapter 1 (Introduction), explaining these chronological connections with a graphic or a table. See also related issue LC-57.

2. Guideline 1: I find checkpoints 1.1 - 1.3 very confusing. These are "compound checkpoints" that incorporate multiple other checkpoints. We don't use this structure anywhere else in our docs - why here? The document states "This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA Workshop", but how we got here is not really interesting to the reader.

   The 'sub-checkpoints' on the 'left hand side' of the table are all spec-related. Either these duplicate checkpoints from SpecGL, or they should be incorporated into that document. Those on the 'right hand side' are operations-related, but they seem to overlap with other checkpoints specified in this document.

   The remaining checkpoints under guideline 1 are different kinds of beasts (not compound) and as such, the transition to them seems somewhat abrupt.

   Recommendation: drop the compound structure, make sure that all the spec-related sub-checkpoints are covered by SpecGL, and move the 'test materials' sub-checkpoints into the body of this doc.

   Email discussion , including proposed resolution.

   Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3).

3. Guideline 2: I'd say "Allocate resources..." rather than "Define resources..."

   See email proposal. Resolved per this proposal at 20030516 telecon. The wording of GL1 and GL2 is changed consistently, to emphasize that these are about commitment, and GL2 now starts with "Commit to resource level..." See draft revised text.

4. Guideline 3 begins with some text ("The benefits of....") that seems to be a rationale. Other Guidelines jump straight into the Conformance requirements - this one should too.

   Discussed in email. Withdrawn by originator while discussing clarification of the request.

5. How is checkpoint 3.1 different from 1.5?

   See email proposal. Resolved per this proposal at 20030516 telecon. Add "Related checkpoints" to each of CP1.5 and CP3.1, pointing to the other and explaining the distinction. See draft revised text.

6. Checkpoint 3.2 doesn't seem to be directly related to the Guideline under which it's classified. The Note for 3.2 points out that checkpoint 8.2 is related - doesn't this suggest that the overall structure should be re-worked (if they're related, why are they so far apart numerically?)

   See email proposal. Resolved per this proposal at 20030516 telecon. This and several other aspects of CP3.2 and 8.2 are clarified and resolved all at once in the proposal. See draft revised text.

7. Guideline 4: Probably should be #1 (it's the first chronologically). When we summarized this document in our outreach presentation we made checkpoint 4.1 the first bullet item...

   See email proposal. Resolved per this proposal at 20030516 telecon. GL1/2 are about early *commitment* to QA and QA deliverables. Possibly even in the Charter, before the WG is rolling. GL4 is about process and operations within the functioning WG. It is equally arguable that this is the best order. Per the 20030512 resolution of LC-60.1, we will avoid any re-ordering or re-grouping of GL/CP except to fix something that is really broken. Accordingly, it is resolved to keep the current order here.

8. Checkpoints 4.1 and 4.2 would seem to belong in Guideline 2 (define/allocate resources) rather than here?

   See email proposal. Resolved per this proposal at 20030516 telecon. GL1 and 2 are about early commitment (ideally Charter) to QA levels, staffing levels, etc. GL4 is about performance -- specific staff assignments -- going forward. To each of CP2.2, CP4.1 and CP4.2, added reciprocal clarifications about the others, and disambiguate the similarity between them. See draft revised text.

9. The Discussion for checkpoint 4.3 says "To summarize...". This implies that somewhere there's a more detailed description of what the QAPD must address, but I don't think there is. This checkpoint really seems to amount to "document how you meet these other checkpoints", yet the list of "other checkpoints" that must be implemented is incomplete. Why doesn't this simply require that *all* checkpoints be documented?

   See email proposal. Resolved per this proposal at 20030516 telecon. This is clarified, in conjunction with solution to LC-58, by adding a new paragraph at the beginning of CP4.3 "Discussion". The discussion is also expanded to specifically mention the other QAPD-related checkpoints, and a "Related checkpoints" is added. See draft revised text.

10. Checkpoint 4.5 uses the ambiguous term "framework" (we tried to avoid this in our re-write of TestGL).

    Discussed in 20030516 telecon. See subsequent email proposal. Resolved per email proposal by email (default). LC-72.7 also raises this issue. Other comments indicate confusion why CP4.5 is here at all, instead of TestGL. All of these are addressed in the proposal, for complete replacement text. The text clarifies why the topic is dealt with in OpsGL (instead of or in addition to TestGL), and changes the wording "QA framework" to "framework for test materials development". ("Framework" is still used, in lieu of specific proposals for a better phrasing.) The checkpoint is also moved, to become the first checkpoint of Guideline 5. See draft revised text.

11. Checkpoint 4.6 addresses branding - another argument for a chronological rather than a 'logical' grouping (this should be at the end).

    See email proposal. Resolved per this proposal at 20030516 telecon. Because branding policy decision has potential knock-on effects on other aspects of the WGs QA life, process, TM, and operations, it is equally arguable that its early consideration is beneficial. Given the resolution that we are not going to do a major chronological re-org (LC-60.1) except to fix really broken things, it is resolved to keep it as is.

12. Guideline 5: Checkpoint 5.2 - Define a contribution process. Why only priority 2? Without a contribution process you have nothing, surely?

    See email proposal. Resolved per this proposal at 20030516 telecon. There are test suites that have been built entirely without any formal or defined contribution process -- mostly by a single person, with ad-hoc acceptance procedures for internal contributions. Certainly this is not optimal, but also not critical in all cases (perhaps in some). If possible, we want to avoid increasing the number of P1 checkpoints unless we feel they are really critical/essential. Therefore, it is resolved to keep it as is.

13. Guideline 6: Checkpoint 6.1 contains a mixture of stuff. The guideline addresses "publication" but much of this checkpoint addresses "management". Moreover, the bullet items in the Discussion section don't seem to relate to repositories at all.

    See email proposal. Resolved per this proposal by email (default). Take the bullet list out of CP6.1, fine tune the wording, and put it in the verbiage of GL6. Add comment to the GL6 verbiage explaining the preprequisite relationship of some management aspects to publishing, and therefore their appropriate treatment of those here in GL6. See draft revised text.

14. Checkpoint 6.2 (defining the license for published test materials) is closely related to 5.3 (defining the license for submissions). Should these be grouped together (under a guideline that addresses submission processes)?

    See email proposal. Resolved per this proposal at 20030516 telecon. Consistent with the LC-60.1 resolution of "minimize improvements that don't fix something really broken", it was resolved that we should clarify and cross link them, instead of trying to combine them. Part of the clarification includes wording changes approved by W3C Legal, and other interested parties in the TM license issue. See draft revised text.

15. Guideline 7: This guideline is labelled "plan the transfer of test materials to W3C if needed", and explicitly states "all of the checkpoints... are not applicable if the WG does not transfer..." (should be "none of the checkpoints are applicable if..."). However, all the checkpoints seem to apply whether or not the materials are "transferred". It's obviously important to review the quality of submitted tests, to ensure that we have the staffing to deal with submissions, and to resolve IPR issues.

    See email proposal. Resolved per this proposal by email (default). All of the submission stuff does get addressed in GL5. GL7 is about the wholesale transfer of an entire externally written test suite. While this could be considered as a "submission" (therefore redundant with some of GL5), on the other hand the sense of GL5 is more about piecemeal submission during the building of test materials. QAWG has had experience moderating such a transfer, which is why GL7 exists separately -- it's useful to have it in one place, even if there are parallels elsewhere in OpsGL for piecemeal planning, development, staffing, submission, etc. Resolved to fine tune the wording of the GL7 verbiage and add some more rationale/explanation. See draft revised text.

For LC-60.3 through LC-60.15, see email, including some proposals.

Related LC QA-commit issues: 3, 60.2, 72.2, 72.3, 83, 107.

• Charter
• Allocation of resources
• Planning & synchronization
• Test submission/development
• Test management
• Test publication
• Conformance testing (test usage)
• Maintenance

Specification reference.

Email comments (20030417).

Discussed at 20030418 telecon.

Discussed and resolved at 20030418 telecon. We don't see documents and resources as examples of 'content' class (perhaps some better definition of 'content' class is needed?). We agreed that a 'specification' class should be added.

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

In the guideline title and table of contents entry for Guideline 12 and in 3.4 last par., "pro forma" is two words.

The table of contents link to section 3.1 is broken.

(These probably have been reported by now but just in case.)

Specification reference.

Discussed at 20030407 telecon. Agreed to implement suggested change -- subsections in TOC will be numbered N.M (M=1,2...), instead of just M.

Related LC Section/GL/CP numbering issues: 63, 77.ET-3, 91.

Specification reference.

Email discussion, including proposal for closure.

Specification reference.

Discussed at 20030410 telecon. See also subsequent email thread. Agreed that section 3.1 has some problems that will be fixed: it is not comprehensive (omits some normative stuff); and has misleading characterization of the normative units of text.

See mail thread (20030606), which includes suggested resolutions.

See also these directly related QAWG issues: issue #98, issue #99, issue #110.AR-001. These were marked as closed at Tokyo and after, however later email and telecon decisions, including new agreed definitions and some samples, suggest reviewing the earlier closures.

Related LC whats-normative issues: 36, 65, 106, 108.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

Email discussion (20030331).

Email discussion (20030417).

Email comments, including proposal (20030418).

Discussed at 20030421 telecon, resolved per email proposal.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Specification reference.

1. [Part 1]: "the specification MUST use RFC 2119 keywords to denote whether or not requirements are mandatory, optional, or suggested" is Priority 1. Must all testable statements and or test assertions (sorry I'm not clear on the difference between them and maybe that needs to be clarified, too) contain RFC 2119 key words?

   Discussion. Note counter example in UAAG 1.0: "This document demands substantially more conformance flexibility than can be achieved using the terms "must", "should", and "may" alone, as defined in RFC 2119 [RFC2119]. Where "must", "should", "required", and "may" appear in this document, they are used consistently with RFC 2119 for a chosen conformance profile. The imperative voice (e.g., "Allow configuration ...") used in the checkpoint provisions implies "must", but a user agent is only obligated to satisfy the requirements of a chosen conformance profile."

   Related discussion about whether testability is an inherent part of "conformance requirement" in this email sub-thread (of a thread about some OpsGL issues).

   See mail thread (20030606), which includes suggested resolutions.

   See also these directly related QAWG issues: issue #98, issue #99, issue #110.AR-001. These were marked as closed at Tokyo and after, however later email and telecon decisions, including new agreed definitions and some samples, suggest reviewing the earlier closures.

   Discussed at 6/2003 QAWG face-to-face -- clarification solicited about UAAG10 approach, in this email thread. Discussion resumed on email list, with copious comments generated.

   Discussed again and resolved at 20030707 telecon. Resolution is: SpecGL (old CP13.1) will strongly encourage RFC2119 keywords as the most widely applicable and usually the best method, but exemptions are allowed, provided that: the spec unambiguously defines how its conformance requirements are identified (even if RFC2119 keywords are used); and, the rationale for not using RFC2119 keywords is explained in the spec. See draft revised text.

2. [Part 2]: RFC 2119 [1] section "6. Guidance in the use of these Imperatives" puts limits [on] their use. (I mentioned this to the QAWG about a year ago.) They musn't be used only to ask implementers to do something a Working Group would like to see.

   Email discussion (20030418).

   Discussed and resolved 2nd part -- guidance about RFC keyword usage -- at 20030421 telecon. Resolved: SpecGL won't try to legislate correct use of RFC2119 keywords. However, SpecET (Examples & Techniques) will mention that correct-use guidance is given in Section 6 of RFC2119, and will point to it. QAWG is concerned that quoting the particular wording of RFC2119 -- with reflects a particular perspective (communications & protocols) and orientation -- creates confusion rather than clarity, in some conformance contexts. There is also considerable disagreement (see the email dialogs about this issue) within W3C about the "correct use" assertions, and there is a significant body of current practice that is contrary to them. QAWG thinks that it would be useful to consider a joint Comm-QAWG project to draft a W3C guidance Note on this topic.

Subsequent to resolution of 2nd part, additional comments & discussion occurred on QAIG list.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

"Imperatives of the type defined in this memo must be used with care and sparingly. In particular, they MUST only be used where it is actually required for interoperation or to limit behavior which has potential for causing harm (e.g., limiting retransmisssions) For example, they must not be used to try to impose a particular method on implementors where the method is not required for interoperability."

[LH] See email proposal: don't legislate it in SpecGL; develop a Note about it and reference that from SpecET.

[1] http://www.rfc-editor.org/rfc/rfc2119.txt

Specification reference.

• [1] http://lists.w3.org/Archives/Public/www-qa/2003Feb/0021.html
• [2] http://www.w3.org/2003/03/14-qa-intro-lesch.html

Specification reference.

Review comments:

1. In general it provides useful guidance on how to use the Framework, defining audience and WG activity applicability for each document.

   [No action required.] Thank you for the positive feedback -- it is useful to know that it is fulfilling its purpose.

2. Is this a Working Draft or Last Call Working Draft? Title indicates the former.

   W3C process and pubrules do not distinguish between WD and Last Call WD in the dated subtitle. The distinction is required in the "Status of this document" section.

3. Interspersed usage of the terms 'document', 'guideline', and 'specification' to describe the QA Framework documents could be confusing. E.g., paragraph 4 of Status: "... It is anticipated that this specification will eventually progress, along with its Operational Guidelines and Specification Guidelines companions, to Candidate Recommendation (CR) and beyond. The timing of progression of this specification will be determined by the progression of the companion guidelines documents." Similarly, 'TR', 'standard', 'specification', and 'recommendation' are used interchangeably to describe the output of WGs.

   This has been fixed as suggested -- document is used uniformly when "Introduction" refers to itself.

4. Section 1.3, first sentence: "The last underscores a key reality of improved quality practices associated with W3C technical reports". Not clear what 'the last' is (previous section?).

   This has been fixed as a result of the substantial rewrite of "QA Framework: Introduction" from issue #68 -- as a result of consolidation, that text is gone.

5. Section 1.4, paragraph 3, first sentence incomplete? "While some might perceive QA projects as a regrettable drain on WG resources, there is ample experience, both within W3C as well as other standards venues, that shows significant improvement to the products of the WGs."

   This has been fixed as a result the substantial rewrite of "QA Framework: Introduction" from issue #68 -- as a result of consolidation, that text is gone.

6. Sections 1.3 (paragraph 1) and 1.4 (paragraphs 2 and 3) contain general justification arguments for QA efforts in WGs - may be more appropriate content for Section 1.2.

   This has changed as a result of the substantial rewrite of "QA Framework: Introduction" from issue #68.

7. Section 3.1 Application Domain - does this belong in Section 3 (Structure and content of Framework documents)? Seems more like Section 1 (Overview) content where target audience is covered.

   This has changed as a result of the substantial rewrite of "QA Framework: Introduction" from issue #68.

8. Sections 3.5.2 - 3.5.5 describe each document - information on content, audience, and objective. It may help readability to use a consistent order for presenting this information across sections.

   It is our intent that the order be uniform where the framework family is discussed, unless driven by another criterion such as roles or scenarios. We are not aware of any non-uniformities in the revised text.

9. Section 3.5.4 Single item bullet list?

   This has changed as a result of the substantial rewrite of "QA Framework: Introduction" from issue #68.

10. Section 4.1.3 Useful breakout of document relevance by role within a WG.

    [No action required.] Thank you for the positive feedback.

11. Section 4.2 Provides a good life cycle view of the relationship between Framework documents and WG activities. A table summary might be useful as well.

    We will consider this for a future version of "QA Framework: Introduction" (which is now being published as a "Note"). We have put a "chronology diagram" into "QA Framework: Specification Guidelines." One lesson from there is that the chronology is only approximate, and there can be wide variation without substantially diluting the benefits. A submission of a draft table would be welcomed and considered for inclusion.

12. Section 4.1.3 "WG-TS moderator" - Section 4.2.2 "test materials (QA) moderator". Same role?

    Yes. The 4.2.2 wording has been changed to simply, "QA moderator".

13. Section 4.2.3, paragraph 4, second sentence is unclear: "Normally, this should not be considered as a good time to bring a specification for 'Specification Guidelines' conformance, as the latter could significantly disrupt and restructure the specification.". Is 'the latter' referring to bringing a spec to specification guidelines conformance, or something in a previous sentence?

    The first choice (bringing spec to conformance). The sentence has been revised to remove the ambiguity.

14. Section 4.2.5. Intra-WG build of test materials calls for an acceptance procedure for the individual bits. Import and assemble only call for quality assessment and assessment criteria - is an acceptance procedure required / implied?

    This has been revised to clarify.

15. (Editorial) Usage of Working Group vs. WG inconsistent throughout

    As a general rule, we have used "WG" a lot for brevity. We have tried to have a full "Working Group" spelled out nearby, e.g., somewhere early in the same section. We have used markup extensively to define the WG acronym.

16. Inconsistent bullet list punctuation (';' vs. ',' vs. nothing at line end, etc.)

    This has been made consistent (";").

17. Section 1.2, paragraph 1 "...." at end of first sentence

    Fixed.

18. Section 4.2.5, paragraph 2 "? -- as " in middle of second sentence

    Fixed.

Specification reference.

I have noticed a number of "checklists" being proliferated on the W3C pages. I have a strong recommendation for improving the adaptability of the checklist format - namely the number of column in the layout. For example, today most have a column for the number of the checkpoint, the description of the checkpoint, and then a number of columns for YES, NO, N/A. Please consider adapting the following format:

• Top matter: Version, date, owner, etc
• Column 1 10% Checkpoint number
• Column 2 40% Description
• Column 3 10% YES, NO, N/A, Planned
• Column 4 40% Comments
• Bottom matter: Footnotes, exceptions, references, etc.

an example we find useful is at http://www-3.ibm.com/able/accesssoftware.html .

[[Following is (unarchived) response comment from Ian Jacobs: We redesigned the UAAG 1.0 checklist [1] based on earlier comments from you [1]. We don't specify fixed widths for table columns.

• [1] http://www.w3.org/TR/2002/REC-UAAG10-20021217/uaag10-chktable.html
• [2] http://lists.w3.org/Archives/Public/w3c-wai-ua/2000AprJun/0137.html ]]

Discussed and resolved at 20030707 telecon. We need to clarify the purpose of the checklists. They are intended to be implementation conformance reporting forms. Accordingly, the category of "planned" is not considered appropriate, nor is the provision of a "Comments" field. Without the 40% "comments" field, the remainder of the reformatting proposal would need re-adjustment. QAWG has identified and intends to produce another sort of form where a comments field would indeed be appropriate -- a "test material" for the Guidelines documents, in the form of a questionnaire, which has a question/item per-conformance-requirement (as opposed to per-checkpoint).

Specification reference.

Six substantive and editorial issues,

1. [IN-1]: Define "conformance" and "interoperability".

   Agreed, the terms should be defined. Note that "conformance" is defined in the "QA Glossary", but "interoperability" is not. An action item has been given to the glossary owners to add this. A note has been added to "Terminology", to refer to "QA Glossary" for definitions.

2. [IN-2]: Add "coverage measurement tools" to list in "Technical Assets".

   Agreed, this has been added.

3. [IN-3]: [editorial] Define the "WAI" acronym in section 2.

   Agreed, the first occurrence of WAI is defined (in the text substantially revised per LC-68.)

4. [IN-4]: [editorial] Fix last bullet ("assets") in the resource list in section 2.

   Done, "assets" is now "technical assets" (per the 2nd following subsection).

5. [IN-5]: Since Intro is entirely non-normative, consistently refer to it as "document" instead of "specification".

   Agreed, "Introduction" is uniformly referred to as "document" now.

6. [IN-6]: Clarify what are the items in the bullet list of 4.2.3.

   This has been clarified by identifying them as SpecGL topics, and adding a Guideline # reference to each list item.)

on "QA Framework: Introduction" are included in the document at

http://www.w3.org/QA/WG/2003/03/qareview20030314.html ,

which was submitted to QA by XML Schema on 14 Mar 2003:

http://lists.w3.org/Archives/Public/www-qa/2003Mar/0079.html .

Specification reference.

The following thirteen substantive and editorial issues on "QA Framework: Operational Guidelines" are found in the (linked) document:

• http://www.w3.org/QA/WG/2003/03/qareview20030314.html ,

which was submitted to QA by XML Schema on 14 Mar 2003:

• http://lists.w3.org/Archives/Public/www-qa/2003Mar/0079.html .

1. [OG-1]: Editorial -- unclear wording in section 1.1.

   Proposal. Accept Originator's proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text (4th bullet).

2. [OG-2]: 7-level table creates confusing conformance requirements that contradict the priorities/degrees system.

   Email discussion , including proposed resolution.

   Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3).

3. [OG-3]: CP1.1 is priority 1, but requires checkpoints of other priorities.

   Email discussion , including proposed resolution.

   Discussed at 20030501 telecon. Becomes moot as a result of the resolution of LC-3 (and previous sub-issue.) See draft revised text of CP1.1 (as well as CP1.2, CP1.3).

4. [OG-4]: GL3 wording emphasizes interoperable implementations, should instead emphasize conformant implementations.

   Proposal. Accept Originator's proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text (3rd bullet).

5. [OG-5]: CP3.2, "support specification versioning/errata" should be priority 1 instead of 3.

   Proposal. Accept originators request -- change p3 to p1. Discussed at 20030512 telecon and 20030516 telecon. Resolved to change priority to P1, and also to rephrase requirements in terms of capability to unambiguosly associate spec and TM versions (for which TM infrastructure support is one method of achieving.) Relationship to CP8.2 also documented. See draft revised text.

6. [OG-6]: Editorial -- poor wording in CP4.3.

   Proposal. Agree that it's poor wording, change "minimally addresses all of the topics" to "addresses at least all of the topics". Resolved, proposal endorsed at 20030512 telecon. See draft revised text.

7. [OG-7]: Editorial -- CP4.5 uses "QA Framework" in a different (but undefined) sense that the OpsGL title.

   Proposal. Agree, the same phrase is used with a different meaning. Change the CP wording from "Define the QA framework for test materials development" to "Define a framework for test materials development." Resolved, proposal endorsed at 20030512 telecon. (The checkpoint is also moved, to become the first checkpoint of Guideline 5.) See draft revised text.

8. [OG-8]: CP4.5 should be priority 2 instead of priority 1.

   Email proposal [LH]: Accept originator comment -- defining the framework for TM development is not critical enough to be Priority 1. Lower to priority 2. Discussed and resolved as proposed at 20030512 telecon. (The checkpoint is also moved, to become the first checkpoint of Guideline 5.) See draft revised text.

9. [OG-9]: Editorial -- ambiguous wording in CP5.4.

   Proposal. Agree with the intent of the comment. Change "..the procedure MUST minimally address criteria for.." to "..the procedure MUST address at least criteria for.." Resolved, proposal endorsed at 20030512 telecon. See draft revised text.

10. [OG-10]: CP6.2 -- "address TM licenses" -- should be substantively modified.

    Email discussion & proposal, and discussion at 20030501 telecon, which provided an interim resolution. Interim Resolution -- describe the attributes of the licenses, and include a caveat about open issues involving the Document License. See interim draft revised text. It was also resolved that a task team of interested parties will be convened to close the remaining disagreements and possibly design a Test Materials license. The task team will either be formed and operate in a broader venue than QAWG, or will operate under QAWG if a' priori commitments of adequate resources are made by the interested parties. These interim results will be updated with the task team results.

    License task team results. Extensive discussion by TM-license task team at a 20030611 telecon, involving all identified principal stakeholders (member companies, QAWG, W3C Legal, etc): see public summary and detailed minutes (for now, member-only). Resolution: W3C will continue to pursue the Document and Software licenses as the mechanisms for governing test suites. A new TM license is not seen as needed, and there are no major defects that compromise the usability and applicability the Software License or the Document License for W3C Test Materials, or of both together applied to different components of a complete Test Materials. W3C Legal will continue to investigate a number of possible (relatively minor) improvements. Guidance in the licenses FAQ will be investigated. QAWG will continue to look at both adjustments to its Guidelines documents (e.g., TestGL guidance on partitioning the TM components suitably for different license application), and possibly some further FAQ-like guidance in "Operational Examples & Techniques" (specifically for Checkpoint 6.2). See draft revised text.

11. [OG-11]: CP6.3 arguments against publishing TM in the TR space are not convincing, should be improved (proposal included).

    Proposal. Accept the proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text.

12. [OG-12]: Problems with CP6.5 ("publish test results"), discussion of test harnesses, should be fixed.

    Email proposal [LH]: Agree with intent of Originator. In Discussion, change the sentence, "Such publication should include or describe a test harness that would allow anyone to reproduce the results" to "Such publication should include or describe a test harness that allows reproduction of the results." Resolved, proposal endorsed at 20030512 telecon. See draft revised text.

13. [OG-13]: CP8.2 (versioning/errata in maintenance) should be priority 1 instead of 2.

    Proposal to accept (change priority 2 to priority 1 in CP8.2 [versioning/errata in maintenance]) was discussed at 20030512 telecon and 20030516 telecon. Resolved to change priority to P1, and also to clarify scope of "versions" (minimally, "editions"). Relationship to CP3.2 also documented. See draft revised text.

Related LC QA-commit issues: 3, 60.2, 72.2, 72.3, 83, 107.

Specification reference.

The following nine substantive and editorial issues on "QA Framework: Specification Guidelines" are found in the (linked) document:

• http://www.w3.org/QA/WG/2003/03/qareview20030314.html ,

which was submitted to QA by XML Schema on 14 Mar 2003:

• http://lists.w3.org/Archives/Public/www-qa/2003Mar/0079.html .

1. [SG-1]: Editorial (SoTD).

   Resolved, it has been fixed as suggested. See draft revised text.

2. [SG-2]: Remove all "not applicable if..." statements.

   Email discussion, including proposal.

   Discussed and resolved at 20030602 telecon (DRAFT). QAWG agrees that there are some editorial problems in these clauses. As pointed out, "It" is undefined (should be "This checkpoint"). There is also inconsistent usage, e.g., some checkpoints in Guideline 9 have applicability exclusions, and some do not. In general, however, it is thought useful that there be an explicit applicability exclusion, rather than leaving the reader to interpret this (*only* if there is an applicability exclusion can the n/a box in the ICS be checked). SpecGL editors will improve the language, clarify the ambiguous statements, and apply consistent structuring, for all checkpoints that have applicability exclusions. See draft revised text.

3. [SG-3]: "Enumerate all CoP" is unreasonable requirement.

   Email comments (20030417), including proposals.

   Discussed and resolved at 20030418 telecon. We think that perhaps originator is confusing "all classes of product" with "all products". Nevertheless, the wording will be improved by: remove "all" from the checkpoint statement; and, change 1st bullet from "MUST list the classes of product it addresses" to "MUST list the classes of product for which it defines conformance requirements". See draft revised text.

4. [SG-4]: Editorial (8.5, ConfReq, change "profiles").

   Resolved, this has been fixed as suggested. See draft revised text.

5. [SG-5]: Editorial (Questions "NOT!" in GL9.)

   Resolved, changed "NOT!" to "not." See draft revised text.

6. [SG-6]: Editorial (Typo in CP9.1)

   Resolved. The typo has been fixed during the merger of 9.1 and 9.2 (which was the resolution of another, substantive issue.) See draft revised text.

7. [SG-7]: "Normative refs" requirement should be priority 1.

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: agreed, the priority of CP10.2 (now CP7.5) has been changed to 1. See draft revised text.

8. [SG-8]: Add Rationale and Discussion to CP13.3 ("Use consistent terminology").

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: Agreed, a rationale has been added. See draft revised text.

9. [SG-9]: Improve definition of "test assertions" (GL14).

   Related discussion about whether testability is an inherent part of "conformance requirement" in this email sub-thread (of a thread about some OpsGL issues).

   See mail thread (20030606), which includes suggested resolutions.

   Discussed and resolved at 6/2003 QAWG face-to-face. QAWG will make the various definitions consistent across the GL documents and the QA Glossary, but otherwise will leave the definition more or less as it was in the initial verbiage of (Last Call) SpecGL GL14.

See also these related closed QAWG issues: issue #98, issue #99, issue #110.AR-001.

Extension-group processing plan and subsequent email discussion.

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Related LC applicability/normative-exclusion issues: 26, 73.2, 80.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

If the 14 specification guidelines can be clearly summarized in 6 bullets, as we claim is done in the QA Outreach Kit, http://www.w3.org/2003/Talks/qa-outreach/slide5-0.html , then perhaps the guidelines themselves ought to be consolidated and reduced in number. I believe that most of the checkpoints are appropriate, but perhaps should be repackaged.

Six bullets:

• Scope - define the scope, identifying what needs to conform and how
• Conformance Clause - specify the conformance policy and requirements
• If you must subset - use profiles, modules, functional levels
• Extensions - specify whether and how extensions are allowed
• Tag for later use - identify testable assertions, discretionary items
• Claims - specify how conformance claims and statements are made

I think the first 4 are almost right for guidelines. The later ones don't seem to me to be quite right.

• new GL1: define and illustrate scope (includes old GL1, GL2)
• new GL2: define conformance policy and requirements (old GL3, GL10)
• new GL3: subset as needed (old GL4, GL5, GL6)
• new GL4: extensions (old GL9)

  (It gets a little rougher from here on...)

• new GL5: get a handle on other conformance variability
• new GL6: is there a nice umbrella statement that would accomodate GL13 and GL14?
• new GL7: claims -- specify how and provide an ICS [old GL11, GL12]

So at best, we would have 7 guidelines. At worst, 9 (if the new GL5 wouldn't stick together, and we couldn't find a new GL6 that was satisfactory).

Email comments (20030417), including proposals.

Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Specification reference.

1. Guideline 1: Checkpoint 1.4 is priority 2, yet it's listed after 1.3 which is priority 3. I suggest listing checkpoints in priority order.

   Discussed at 20030407 telecon. Agreed to switch the order in this case. However, this is NOT a general endorsement of "order by priority". Generally agreed that logical flow should have precedence, and then priority order if there is no determinant based on logical flow.

   As part of the resolution of the larger issue group, CP1.3 has been merged into CP1.2 and the single checkpoint has been reworded as "Illustrate scope", so the issue becomes moot. CP1.4 (now CP1.3) is reworded as "Illustrate technical details". See draft revised text.

2. Guideline 3: The Conformance Policy is likely to appear towards the end of a Spec. I suggest ordering the guidelines as they are likely to appear in the spec. (Besides, it belongs with, and maybe could even be combined with, guidelines 10, 11, 12, and 13.)

   Discussed and resolved at 6/2003 QAWG face-to-face. Considering all related guideline (GL) consolidation and reordering issues together, it was resolved to: merge (old numbers) GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12; and reorder as GL13, GL10+GL3, GL11+GL12. SeeSee new GL7, GL8, GL9. See draft revised text.

3. Guidelines 4-9: This is a large number of guidelines and checkpoints to deal with something that's important, but that we wish to discourage (DOVs). Could we combine some or all of these guidelines?

   Email comments, including proposal (20030418).

   Discussed and resolved at 20030421 telecon. It was decided to consolidate GL4-6 (profiles/modules/levels), into one guideline, while preserving the identities of profiles, modules, and levels as DoV. But otherwise, the other DoV checkpoints don't seem amenable to consolidation. See resolution summary of this and related DoV issues. See draft revised text.

4. The "or NOT!" language in guideline 9 seems a little too informal,

   Resolved at 20030505 telecon. Agreed, "NOT!" is replaced with "not." See draft revised text.

5. but more substantively, why do we wait until this guideline [GL9] to express our opinion that DOVs are undesirable.

   Email discussion.

   Email comments, including proposal (20030418).

   Discussed and resolved at 20030421 telecon. A modified version of the previous per-GL DoV "excessive variability" caveat has been restored. See also resolution summary of this and related DoV issues. See draft revised text.

6. Guideline 10: Checkpoint 10.2 doesn't seem to be directly related to the guideline.

   Discussed and resolved at 6/2003 QAWG face-to-face. Considering all related guideline (GL) consolidation and reordering issues together, it was resolved to: merge (old numbers) GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12; and reorder as GL13, GL10+GL3, GL11+GL12. See new GL7, GL8, GL9. See draft revised text.

7. Guidelines 10 - 13: As suggested above, perhaps these could be combined.

   Discussed and resolved at 6/2003 QAWG face-to-face. Considering all related guideline (GL) consolidation and reordering issues together, it was resolved to: merge (old numbers) GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12; and reorder as GL13, GL10+GL3, GL11+GL12. See new GL7, GL8, GL9. See draft revised text.

8. Guideline 14: This is a key guideline - I'd like to see it earlier in the list.

   Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12. While test assertions are pivotal and early in the process in *Test* Guidelines, it is resolved that this is an appropriate placement in the *SpecGL* context.

9. Section 3.3: "This... document does not enumerate a list of test assertions". Haven't we agreed that our checklist is (the closest thing to) a list of assertions?

   Related discussion about whether testability is an inherent part of "conformance requirement" in this email sub-thread (of a thread about some OpsGL issues).

   See mail thread (20030606), which includes suggested resolutions.

   See also these related QAWG issues: issue #98, issue #99, issue #110.AR-001.

   Discussed and closed at June QAWG face-to-face. The checklists are a test results reporting form. Recognizing that there are possibly multiple conformance requirements (CR) per checkpoint, a test material for the Guidelines documents would be something like a per-CR questionnaire, "How is this CR satisfied?".

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Four substantive and editorial issues (IN-1, .., IN-4) on "QA Framework: Introduction" are included in the document at

http://www.w3.org/QA/WG/2003/03/DIWGcomments.html ,

which was submitted to QA on 14 Mar 2003:

http://lists.w3.org/Archives/Member/w3c-di-wg/2003Mar/0074.html .

1. [IN-1]: Rephrase "ultimate mission of W3C".

   Accepted, "ultimate" is replaced with "ongoing".

2. [IN-2]: Meaningless phrase, "unimplementable language".

   Agreed, the phrase has been changed to read, "early detection and correction of unintentionally vague or contradictory language".

3. [IN-3]: More detail in 3.5.2 audience list than in "Audience" section.

   This has been fixed during the substantial rewrite of QA Framework: Introduction that resulted from issue #68. As a result of consolidation of similar parts, the 3.5.2 list is gone.

4. [IN-4]: Combine target audience sections -- 4.1 and 1.3.

   Agreed, the disparate sections been consolidated during the substantial rewrite of QA Framework: Introduction that resulted from issue #68.

Also there are three issues applicable to all Framework, (FR-1, ..), and six general comments to QA (GC-1, .., GC-6).

Specification reference.

• http://www.w3.org/QA/WG/2003/03/DIWGcomments.html ,

which was submitted by DI WG to QA on 14 Mar 2003:

• http://lists.w3.org/Archives/Member/w3c-di-wg/2003Mar/0074.html .

Two substantive and editorial issues on "QA Framework: Specification Guidelines", plus four related comments about the Specification Examples & Techniques documents,

1. [SG-1] More discussion of several DoV and their relationships.

   Email comments, including proposal (20030418).

   Discussed and resolved at 20030421 telecon. Subsection 2.1 of the new chapter "2. Concepts" provides additional discussion about relationships amongst DoV. See draft revised text.

2. [SG-2] Standard headings for (Framework GL?) documents.

   20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: Agree that this is a good principle and we will adhere to it, as best we can, recognizing that each Framework document may have some unique chapters. All documents will have the same core headings and in the same order: Intro, Guidelines, Conformance, Definitions (if applicable), Acknowledgements, References, Change History. No changes to SpecGL needed. See draft revised text.

3. [ET-1] (Affects Intro, and all GL-ET pairs) Explain GL-ET pairing better.

   Proposal. See similar OpsGL text.

   Discussed at 20030714 telecon. Agreed, this is now handled by additional clarifying text in the GL document, and/or references to "QA Framework: Introduction". See draft revised text.

4. [ET-2] (Affects SpecET and the other ETs). More E&T, in order to conform to SpecGL (GL1)!

   Resolved: it is the QAWG intention that SpecET (Examples & Techniques) will have at least one example per checkpoint. This is queued for SpecET editors, but may lag slightly behind SpecGL publication (after the structure and checkpoint collection of SpecGL stabilizes.)

5. [ET-3] (Affects SpecGL/ET, and the other GL/ET) Change GL, CP, and section numbering scheme.

   Discussed at 20030407 telecon. Closed with resolution to leave numbering scheme as is. Reasons: given that the context is usually clear (or easily made clear), the shorter GL/CP numbers are considered both more convenient and sufficient; and, the numbering scheme is familiar from WAI specifications and has been successfully used there.

6. [ET-4] (Affects SpecET) When will SpecET be released? (!?)

   The QAWG policy is that each new publication of SpecGL will be accompanied by a new version of SpecET (possibly lagging slightly behind the SpecGL publication). In this sense, it is already "released". SpecET may be updated and improved more frequently than SpecGL publications. It is for this reason that QAWG resolved to keep SpecET (as well as OpsET and TestET) in /QA/WG/ space as a Note. When SpecGL 1.0 has finally stabilized and stopped changing in /TR/, and when active work on SpecET is substantially over, then SpecET will be published in /TR/ (as a Note, since it is purely informative.)

An additional three comments/issues applicable to all Framework -- [FR-1]..[FR-3] -- can now be found at Last Call issue #111.

An additional six general comments/issues to the QA activity -- [GC-1]..[GC-3] -- can now be found at Last Call issue #112.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Related LC Section/GL/CP numbering issues: 63, 77.ET-3, 91.

Specification reference.

When I visited the joint XSL/XQ WG session (March 6 in Cambridge), there was quite a bit of discussion about the claims and disclaimers suggested in Guideline 11 of SpecGL. WG members attending wanted to have something more concrete to start with, like a "boilerplate" paragraph or two. In particular, we talked about number of test cases passed as a bad metric for conformance, because it implies that each case has equal weight. The current checkpoints don't call out this practice as warranting discouragement.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: good idea. Volunteers are sought, to help make the templates.

I think that for 11.2, you already know that you intend a template resembling: This [class of product] was tested for conformance to [name of spec] version [N.nn], [Nth] Edition, dated [date], and all errata issued through [date], ... more specs cited same way .... The testing occurred on [date] using [test suite identifier] and [name of product and version] was found to conform at [level] level except for [enumeration of failing tests]. A full report of the parameter settings for the test harness and all results is posted at [URL] for open, public review.

For 11.3, the template could look like this: While the test suite provides [hundreds, thousands] of test cases, not all cases should be considered to carry equal weight. A product that passes all the tests may still not conform in some untested area. The W3C hereby states that claims of passage of a certain number of test cases or a certain percentage of the test cases, but not all, are invalid as relative measurements of conformance or worthiness, and that the only valid data that can be derived from such a result is that the product being tested does not pass all the tests. [Optional: More tests may be added to the suite in the future, and existing tests may be changed when errata are discovered. Failing some test cases cannot be interpreted as failing to conform without corroboration.] [Optional: The test suite can be tailored to suit permissible variability in product behavior. The W3C encourages implementers to provide information in their Implementation Conformance Statement that will lead to accurate configuration of the test suite, but holds the test lab responsible for obtaining the information and tailoring the suite accordingly, or else reporting which pieces of information were undetermined and indicating that some test failures may in fact be due to configuration problems.]

Specification reference.

Alternatives:

• Alt.1: Add examples and/or use cases, and provide usage scenarios;
• Alt.2: No need to, these are "not applicable" to SpecGL;
• Alt.3: No need to, SpecGL only needs to conform to itself at level "A-Conforming" (only priority 1 checkpoints).

While I believe that alternative #1 is the correct alternative, it might be helpful for SpecGL to explain its intentions about self-conformance (p1 or p2 or p3?). I.e., should SpecGL contain a SpecGL conformance claim?

Discussed at 20030407 telecon. Resolved (and action item assigned) to add use cases and/or usage scenarios and/or examples to illustrate scope.

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Specification reference.

It is arguable that some SpecGL checkpoints -- applied to SpecGL itself, or to any other specification for that matter -- might be "not applicable", because of the nature of the specification. For some checkpoints (CP), SpecGL states conditions under which the CP is not applicable. It is unlikely that SpecGL has anticipated all of the circumstances under which a CP might be "not applicable".

Email discussion, including proposal.

Discussed and closed at 20030602 telecon (DRAFT). The suggested change -- instructions in "3. Conformance" about what one should do if one considers a checkpoint to be not applicable -- are considered to be unnecessary and possibly more confusing than useful. Confusing because they might be interpreted as relieving of the obligation to conform to the checkpoint.

Related LC applicability/normative-exclusion issues: 26, 73.2, 80.

Specification reference.

Checkpoint 9.6 conformance requirements talk about an operating mode under which only strict-conforming content may be produced. I think that this may be too narrow a requirement. Would we consider the intent of the checkpoint to be satisfied if an implementation generated strict-conforming 'alt' content (to use the HTML analogy) to a private extension? In my opinion, a no-extensions mode is the best way to satisfy this CP. But is it the only way?

Note. There is some question why someone would include the extension at all, if the 'alt' content is "equivalent". Perhaps it is a way of round-tripping private functions of an implementation, while providing an alternative formulation that other implementations can use.

Extension-group processing plan and subsequent email discussion.

Discussed at 20030505 telecon, without resolution.

Discussed and resolved at 20030602 telecon (DRAFT). CP9.6 will be kept, but the requirements on target specifications will be revised. Rather than requiring that target specifications place the current specific conformance requirement on extension-generating implementations (for a no-extensions mode), CP9.6 will require that target specifications explicitly define a policy about implementation requirements for mitigation of the interoperability impacts of extensions. Such policy could, to cite three examples, include a requirement that implementations have a no-extensions mode; or could include a requirement that implementations include equivalent alternative (standard) content with any extensions; or could explicitly state that there are in fact no implementation requirements for mitigation of interoperability impacts of extensions. [PC volunteered to draft text.]

Further discussion at 20030623 telecon, resolved to clarify that this applies only to producers of content.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

Alternatives:

• Alt.1: Add examples and/or use cases, and provide usage scenarios;
• Alt.2: No need to, these are "not applicable" to OpsGL;
• Alt.3: No need to, OpsGL only needs to conform to itself at level "A-Conforming" (only priority 1 checkpoints).

While I believe that alternative #1 is the correct alternative, it might be helpful for OpsGL to explain its intentions about conformance (A or AA or AAA?) to SpecGL. I.e., should OpsGL contain a SpecGL conformance claim?

Resolved at 20030501 telecon, per prior email proposal.

Specification reference.

Ed note. Submitted as SpecGL issue, reclassified as OpsGL, as QAWG believes that the problem originates with the table of CP1.1 of OpsGL.

Email discussion , including proposed resolution.

Discussed and resolved at 20030428 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3).

Related LC issues: 3, 60.2, 72.2, 72.3, 83, 107.

[LH] See email.

Specification reference.

Email comments (20030417), including proposals.

Email comments, including proposal (20030418).

Discussed and resolved at 20030421 telecon.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Specification reference.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Specification reference.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Specification reference.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Specification reference.

Note. We assume that the reference to 1.3 is wrong, and this actually refers to the 8-item list in the verbiage of GL2.

20030714 telecon directed that it be resolved as indicated in ungrouped-issue proposals: it should be left to the editors' discretion to research any W3C style requirements, and chose a style that they consider appropriate, while conforming to any existing style guidelines.

Specification reference.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Specification reference.

Email comments (20030417), including proposals.

Email comments, including proposal (20030418).

Discussed and resolved at 20030421 telecon.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Specification reference.

Discussed at 20030407 telecon. Closed with resolution to leave numbering scheme as is. Reasons: given that the context is usually clear (or easily made clear by the usual prefix of "guideline" or "checkpoint" [x.y]), the current GL/CP numbers are considered both more convenient and sufficient; the numbering scheme is familiar from WAI specifications and has been successfully used there; and, the proposed numbering scheme would have the undersireable characteristic that every guideline would start with "2." (e.g., 2.8) and every checkpoint would start with "2." (e.g., 2.8.4). The QAWG believes that the GL number should be the most significant component, and believes it is easier to quickly recognize "checkpoint 8.4" than "checkpoint 2.8.4".

Related LC Section/GL/CP numbering issues: 63, 77.ET-3, 91.

Specification reference.

Discussed at 20030407 telecon: CP1.2 requires examples of what is in scope; CP 1.4 requires examples of specific functionality, concepts, behavior. Need to make this clearer, perhaps remove 'concepts' from CP 1.4 will help. Alternatives:

• Alt.1: Clarify the difference that 1.2 is broader examples, whereas 1.4 is very specific to a function or behavior.
• Alt.2: Remove CP 1.4
• Alt.3: Remove CP 1.2

Proposed resolution: re-word 1.2 as "Illustrate scope.", reword 1.4 as "Illustrate technical details", and clarify difference further in the ConfReqs, Rationale, and Discussion. Also, examples (use cases) need not be in the specification itself, they can be in companion document(s). Language should change to use "provide" rather than "include", to avoid implication of "within specification".

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Specification reference.

Email comments (20030417), including proposals.

Discussed at 20030418 telecon.

Discussed and resolved at 20030418 telecon. Rewrite the GL verbiage for clarification. Implementation of the TOC suggestion depends on whether a sub-section on 'class' and 'category' is set up in a new "Ch.2 Concepts".

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

Email comments (20030417), including proposals.

Discussed at 20030418 telecon.

Related LC cat-class issues: 11, 46, 48, 61, 73.3, 93, 94.

Specification reference.

Email comments, including proposal (20030418).

Discussed at 20030421 telecon, and in email thread.

Discussed at 20030428 telecon, and further analyzed in another email thread. Resolution postponed -- there are related issues in the "re-org" group, for consolidating various amongst GL3/10, GL11/12, etc.

Related previous QAWG issue 95 (which appears more "Postponed" than "Closed"), additional email proposal (20030429), email suggestion (20030429) of additional checkpoint.

Discussed at 6/2003 QAWG face-to-face, during the discussion of the "major-restructure issues" group. All of the bits of GL3 which would have qualified it as a DoV have been changed or moved elsewhere, and GL3 is to be merged into GL10.

Related LC DoV issues: 21, 66, 75.3, 75.5, 84, 90, 95, 77.SG-1.

Specification reference.

Discussed and resolved at 20030602 telecon (DRAFT). CP3.1 is priority 2 because it is implicit (but not explicitly spelled out now in SpecGL) that the requirements that comprise the universal- minimum set (across CoP) all occur elsewhere, in the definition of the conformance requirements for each individual Classes of Product (CoP). Thus, the enumeration of CP3.1 could be derived. Because the enumeration could be difficult or obscure, it is considered important/desirable (P2), but not critical/essential (P1). SpecGL editors will ensure that this is clarified.

Related LC GL3-policy issues: 42, 96.

Specification reference.

Email comments (20030417), including proposal to seek clarification.

Discussed (for clarification) at 20030418 telecon, and in subsequent email thread (20030418).

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. During discussion, a key distinction between extension modules and from Rules for Profiles was established -- new profiles are not *extensions* to the specification, but rather different ways to subdivide the (typically) standard functionality of the specification for different conformance targets. Agreed that the problem should be resolved along the lines in LR email proposal -- a combination of 1.) clarification to Originator about how existing checkpoints actually cover this case; and, 2.) some additional clarifying verbiage (to be drafted, for either section 2.3, or the modules GL verbiage, or GL9/extensibility -- tbd). For further context, see also the solution for the profile/module/level issue group.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Specification reference.

There is email discussion of the issue group here, and see also numerous related messages nearby on the same list.

Discussed and resolved at 20030428 telecon. That "levels create a hierarchy of conformance" is part of the definition of levels. It was thought unnecessary and redundant to try to bind the definition into a Checkpoint and test requirements. For further context, see also the solution for the profile/module/level issue group.

Previous profile/module/level QAWG issues: 73, 74, 75, 76.

Related LC profile/module/level issues: 30, 41, 49, 50, 51, 97, 98.

Specification reference.

See email thread.

Discussed at 20030410 telecon. The analysis of the email thread was basically agreed -- it is not necessary to fully define the deprecated functionality, in order to meet the intent of the checkpoint. That intent is that the impact of deprecating the feature on other DoV needs to be discussed. E.g., if the spec is modularized or profiled, how does a feature's deprecation impact those DoV. DM agree to draft clarifying discussion, including at least one "for example", to prevent the confusion.

Related LC deprecation issues: 33, 40, 99.

Specification reference.

Extension-group processing plan and subsequent email discussion.

Resolved at 20030505 telecon.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Subsequent email discussion, including proposed resolution.

Specification reference.

Extension-group processing plan and subsequent email discussion.

Discussed at 20030505 telecon, without resolution.

Discussed and resolved at 20030602 telecon (DRAFT) -- the Originator has misunderstood what Checkpoint 9.6 actually says. Nevertheless, some changes will be made to CP9.6, according to the resolution of LC-81.

Related LC extensibility issues: 15, 29.5, 35, 37, 38, 52, 73.5, 73.6, 75.4, 81, [97?, ] 100, 101.

Subsequent email discussion, including proposed resolution. Extension-group processing plan and subsequent email discussion.

Specification reference.

Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge (old numbers) GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12; and reorder as GL13, GL10+GL3, GL11+GL12. See new GL7, GL8, GL9.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Specification reference.

Email comments (20030417), including proposals.

Discussed and resolved at 20030418 telecon.

Specification reference.

Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Specification reference.

Email comments (20030417), including proposals.

Email discussion (20030418).

Discussed and resolved at 6/2003 QAWG face-to-face. Taking the whole group of related guideline (GL) consolidation and reordering issues together, it was resolved to: merge GL3 into GL10; move current CP10.2 into GL13; move CP11.1 into GL10; merge GL11 and GL12.

Related LC major-restructure issues: 54, 74, 75.2, 75.3, 75.7, 75.8, 102, 104, 105.

Specification reference.

See email thread, with proposal: SpecGL's definition of normative is okay in glossary, but is more focused than originator assumes (so that it includes only direct conformance statements such as conformance requirements and test assertions). Per 20030410 telecon section 3.1 should be clarified and improved.

Discussed again and resolved at 20030414 telecon.

Related LC whats-normative issues: 36, 65, 106, 108.

Related LC TA-requirements issues: 13, 14, 29.4, 65, 67, 73.9, 75.9, [106?].

Specification reference.

Email comments (20030417), including proposals.

Discussed and closed (as SpecGL issue) in email.

Email discussion , including proposed resolution (of OpsGL issue).

Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution.

Related LC issues: 3, 60.2, 72.2, 72.3, 83, 107.

Specification reference.

See email thread.

Discussed and resolved at 20030414 telecon.

Related LC whats-normative issues: 36, 65, 106, 108.

Specification reference.

Originated during 20030407 telecon, during discussion of issue #92. Discussion: It is unclear what is the difference between CP 1.2 and CP 1.3, according to their respective stated rationales. Is CP 1.3 a superset of CP1.2? Can they be combined? Do we need both? If we need both, then what purposes are the "usage scenarios" of CP1.3 serving, that are distinct from the purposes of CP1.2?

Discussed at 20030410 telecon. Resolved that CP1.2 and 1.3 should be combined, and that use case and/or usage scenario should be emphasized as valuable technique in discussion (so that it is somewhat distinguished from "examples").

See previous QAWG (non-LC) issues #72, #84, and #86.

Related LC example/use-case issues: 7, 23, 75.1, 77.ET-2, 79, 92, 109.

Specification reference.

[email]

Ed Note -- DH and KD organized a project review of the QA OpsGL for the W3C Team. This is an informal summary of the comments they got. Note that the presentation was about OpsGL, but most of the comments apply equally to SpecGL.

The following seven items discussed in email thread, including proposals, and resolved at 20030512 telecon.

1. QA is very important, and the QA WG has the right goals. [No issue.]

   Thanks for the comment -- the positive affirmation of the QAWG goals is useful to us.

2. There were a lot of discussions regarding the writing style we adopted for the framework, namely the fact that we use RFC keywords in conformance requirements. Some people thought it was too "aggressive", other felt it was the right thing to do.

   QAWG reaffirms that it considers the RFC keywords appropriate for OpsGL and the other Framework documents. See also Last Call issue 67.

3. [OpsGL only] The table in OpsGL GL 1 caused much confusion and was deemed as not-understandable, which I think we already more or less agree with.

   Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3).

4. The intents of the priorities/degrees is not always clear. Proposal [DH]: we should probably emphasize somewhere that the minimal recommended degree is to be AA conformant (or that is the intention of the WG to request it to be the lowest level for work in W3C).

   Discussed and resolved at 20030512 telecon. First, it is our intent that Level A be the minimum. This will somehow be worked into the description of the semantics of the levels, in the Conformance clause ("Conformance definition"). Second, in the re-write of the introduction, we will make generic reference to checkpoint priorities, and move the details into the Conformance clause. See draft revised text in the Introduction, and the "Conformance definition".

5. Generally speaking, the distinction between GL and ExTech was not always clear. Proposal [DH]: we probably need to rework the introductory sections to clarify that.

   Resolved per email proposals -- it will be clarified and emphasized. In order to comply with the "better readability" request of point #7 in this issue, the detail will be put into QA Framework: Introduction, and linked from the brief treatment in the OpsGL introduction. See draft revised text.

6. The summarized view (ICS/Checklists) are not easy enough to find, and are not explicitly recommended enough. Proposal [DH]: again, that means some re-working of the introduction.

   Resolved per email proposals -- they will be given more exposure and emphasis. See draft revised text.

7. The introduction needs to be much more efficient to read. Proposal [DH?]: some kind of an executive summary rather than the long prose we currently have.

   Discussed and resolved at 20030512 telecon. It is not so much "executive summary", as writing style and organization that needs attention: more concise, more use of emphasis, use bullets instead of long prose, etc. See the clarifications (and useful Web-writing-style reference) in the email thread. See extensively rewritten introduction in the draft revised text.

Specification reference.

• http://www.w3.org/QA/WG/2003/03/DIWGcomments.html ,

which was submitted by DI WG to QA on 14 Mar 2003:

• http://lists.w3.org/Archives/Member/w3c-di-wg/2003Mar/0074.html .

Three issues applicable to all-Framework (which have been classified as comments against "Introduction", for processing purposes):

1. [FR-1] (Affects SpecGL and other GLs) Can ISO9000 be applied?

   After preliminary discussion, QAWG does not think that ISO9000 could replace QA Framework. However an action item to generate a comparison has been assigned and is in progress. It will be circulated to QA distribution when finished. A future version of the "Introduction" part of the QA Framework (now a W3C Working Group Note) could mention or point to the results.

2. [FR-2] (Affects Intro, and maybe all GLs). Give a cost/benefit analysis for conforming to the GLs.

   An action item has been assigned and is in progress, to generate a some cost-benefit data. It will be circulated to QA distribution when finished. A future version of the "Introduction" part of the QA Framework (now a W3C Working Group Note) could mention or point to the results. Some preliminary results have been incorporated into the new (for renewal) QA Activity statements and charters.

3. [FR-3] (Affects Intro, and maybe all GLs). Explain/highlight the relationship of QA Framework to W3C processes.

   Related issues: QAWG issue 16, QAWG issue 71.

   QA Framework: Introduction has been substantially revised, and in the process the motivational material and discussion of QA within W3C has been substantially condensed and focused. Regarding any formal relationship of QA to W3C processes as described in "W3C Process" document itself -- there is no defined relationship now. As the status of any such relationship is likely to be changeable and volatile, QAWG considers it inadvisable to address that in any of the Framework documents.

(Other comments/issues from DIWG: LC-77, LC-112.)

Specification reference.

• http://www.w3.org/QA/WG/2003/03/DIWGcomments.html ,

which was submitted by DI WG to QA on 14 Mar 2003:

• http://lists.w3.org/Archives/Member/w3c-di-wg/2003Mar/0074.html .

Six general comments to the QA activity (which have been classified as comments against "Introduction", for processing purposes):

1. [GC-1]: QA work appreciated, but please "author good tools" to minimize cost to WGs.

   Resolution. It is QAWG's intention to shift its resources to tools now, as well as finishing Test Guidelines. How much we can achieve is ultimately limited QAWG resources (number of active participants). Tool suggestions are welcome.

2. [GC-2]: A DIFF application for subsequent (WG) spec versions would be useful.

   Resolution. A tool like this might be considered outside of the scope of QAWG/IG charters (more in-scope for Comm, for example). That notwithstanding, a number of such tools exist. A summary of diff tools was a topic on the W3C spec-prod list.

3. [GC-3]: A set of standard universal templates to help WGs write their specs.

   Resolution. Templates to facilitate the QA-related aspects of operations, specifications, and test materials is definitely within the QAWG/IG scope and future agenda. The suggested "universal template" might actually fall in the domain of the Comm team, or possible joint Comm-QAWG project. In recent Chairs email, a new homepage for editors was announced. This suggestion will be forwarded to Comm for further discussion.

4. [GC-4]: Can QA suggest review procedures/guidelines that will ensure early involvement of non-WG reviewers?

   QAWG recognizes the value of early involvement of non-WG members in QA deliverables. It will be added to QAWG's agenda, to consider where and how this might best be addressed. It is unclear whether it should be addressed in one of the QA Framework documents (which one?), in a QA Note, or some other way.

5. [GC-5]: [This is a comment with no apparent action needed -- it is a forward (now-current) reference to their CC/PP work.]

   Resolution. Subsequent to this comment, QAWG did a review of CC/PP according to Last Call WD of Specification Guidelines, and a discussion followed. Thanks for your consideration of QAWG's review comments in your CC/PP progression.

6. [GC-6]: Questions about how to manage multiple glossaries and what's happening with the Glossary Project (3/2003 Technical Plenary).

   The glossary project have been through 5 months of development and is being tested at the moment, It can scrape, manipulate and index glossaries from various sources (including multiple, dated versions of a document), as well as translations. We are planning to send some news to the public list public-glossary@w3.org (to which the commenter could subscribe to get updates on the project, by the way) very soon now.

(Other comments/issues from DIWG: LC-77, LC-111.)