QAWG Last Call Issues

Any spec SHOULD have a Security Consideration section. Protocol or behavioral specs MUST have a Security Consideration section.

Security sections make spec authors think about potential vulnerabilities and address at least some of them before the bad guys can exploit them. These sections are also a great place to warn implementors and users about most security-sensitive areas of the spec and, perhaps, common exploits.

IETF's Internet Architecture Board has published the following Internet Draft that may be of use to SpecGL authors: http://www.ietf.org/internet-drafts/draft-iab-sec-cons-03.txt

It seems as if either 'defines' or 'explains' is meant, not both. The first sentence is grammatically incorrect, thus the intended meaning is unclear. The second sentence might seem superfluous, depending on the intended meaning of the first.

Remove the Committment Table and CP1.1, 1.2, 1.3. These items do not add information and are redundant. To satisfy Level 3 (5 or 7), you must satisfy other checkpoints in the OpsGL. In fact, if we did our jobs right, Level 3 should equal Conformance Level A (satifying all P1), Level 5 should equal Level Double A, and Level 7 = Triple A. Isn't it a goal to get WGs to conform to the OpsGL? That would mean that they must satisfy all the P1 checkpoints. Thus, they must have a committment level to P1. So, why do we use a new term - Level 3.

The Table introduces a DOV - called committment level. The OpsGL does not conform to the SpecGL's CPs that require DoVs to address the relationship to conformance, to other DoVs, etc.

(See subsequent email discussion.)

(See subsequent email discussion.)

I think this checkpoint is too vague. Is it possible to break it down into a few more concrete requirements that are more verifiable? For instance:

1. For markup specifications, provide at least one example of each markup construct.
2. For protocol specifications, provide at least one example of ...
3. For transformation specifications, illustrate each transformation capability with an example showing input and output.
4. For UI specifications, provide an example of each construct, and illustrate the desired output using at least one mechanism other than the specification itself (e.g., the SVG specification should not rely on SVG rendering alone to explain what something should look like).

While you may miss some cases, I think spec editors will find this more helpful than the general goal to "provide examples."

I think the previous paragraph doesn't belong here; it could be deleted or moved to the status section (after some editorial fixes).

Suggest that the statment "It is not applicable if..." be labeled as "Normative inclusion/exclusion" as in UAAG 1.0

CP 7.1, 7.4, 7.5 ConfReq, last sentence: change 'is' to 'are' in "...if there is no deprecated feature"

CP8.4 Rationale: change 'identifying' to 'identify'

CP9.1, last para: "This is strict conformance" is a repeat of text in intro of GL9

CP 11.4, last para: modify "proper use of the conformance icons" to "proper use of any conformance icons"

GL13, last para: suggest switching the order of Manual of Style and PubRules

1. In general, specifications make functional requirements. In some cases, it may be necessary to make a performance requirement. How should that be handled?
2. In UAAG 1.0, we realized that for some of our configuration requirements, it didn't matter whether the user agent offered the desired configuration or didn't implement the functionality in the first place. E.g., we decided that it was ok for a user agent to not support blinking at all, or, if blinking is implemented, to offer a configuration to turn it off. However, in other cases, the configurability was "just as important" as the functionality to be configured. Authors should consider these when they specify configuration options.
3. It might be valuable for a specification to explain how to include its requirements in another specification. This is not the same as explaining how to reference the spec, or how to claim conformance to it.
4. It might be valuable to explain some desirable characteristics of a specified technical requirement:
   1. Mutual independence from other requirements
   2. Expresses a minimal requirement
   3. Distinguish and label: requirements, exceptions to those requirements, necessary and/or sufficient techniques for satisfying those requirements.
5. I think that more could be stated about useful ways of allowing extensibility. See, for example, how CSS (forward-compatible parsing), XML, and HTTP handle this. What should be avoided? Is the general practice of "ignore what you don't know" a good idea or a big mistake?
6. A spec should clearly indicate which illustrations (e.g., images) and examples are normative, if any.

General comments/conformance model I think the document is well-organized, clearly written, and very helpful. Having spent a lot of time thinking about conformance issues (notably for UAAG 1.0), I thought it covered a lot of ground and did so well.

I like the distinction between requirements that relate to the conformance model and those that involve implementing the model in the spec.

Hmm, "used" may not be a specific enough term. A spec may encourage a UA to support a feature, but discourage an author from producing it. and may be removed in some future version.

1. - "test material" is referred to in multiple ways in the document including but not limited to "test material" and "test material type". We could not determine a difference in concept from context nor could we come up with an example that needed both.
2. - The language in Rec 1 seems odd. Especially the second sentence. "The Checklist should be developed in a public framework. The development of this framework itself is a central area of interest. The QAWG welcomes the participation of interested parties in developing the Checklist."
3. - The third sentence in Rec 3 also seems a bit odd
4. - We found a number of uses of "test materials" and "test suite" that did not use the "fill-in" markup.
5. - DocumentFramework should be Document Framework
6. - In Test materials contrabuition section in the a/b choice rather than saying tests or series.. say contributions.
7. - In Test Materials contribution section there is a missing 'span' element around address. above]'/span' mailing list: [address].'p'
8. - We found the first item in the list under Test Materials Contribution to be some what redundant. At least in our case as any submissions would already go to the list as it is. The test or series of tests, get submitted to the framework. Submitter should also send a notification to the mailing list that will be set up for the Checklist to indicate that he/she has submitted a test to the class="fill-this-in">[test material name] framework.
9. - There is a typo in section Receipt and review of test contributions "according tot he" should be "according to the"
10. - In Receipt and Review section list item 1 it currently says: "since it tests a feature that is not in the specification" should it say something more like: "since it tests a recommendation that is not in the specification" as 'feature' is not a universally applicable term.
11. - In Receipt and Review section list item 3: This item makes the assumption that the test cases will be written in some xml based language. How ever can we make this assumption? What about languages like rdf that don't use an xml syntax?
12. - this same section (item 3) also has a hanging ']'/span'' on it.
13. - In Receipt and Review section list item 4: "'li''em'Scenarios'/em' that underlay the particular test layout and its intended scope.'/li'" We found the use of "test layout" to be out of place in the document. it is the only place that this term is used. We believe that it could just as easily use language that's more standard to the document.
14. - In Receipt and Review section list item 5: The point of this checkpoint is valid, in that if the submitted test doesn't follow the Development Guide it should be rejected, how ever the notion of the Development Guide is introduced here (in this document). We feel that it should be listed earlier in the document as well. perhaps in the requirements, that the test materials be developed in accordance with the Development Guide.
15. - Typo in last sentence of Receipt and Review section: is "publication arereurned to" should be: "publication are returned to"
16. - In Test status and review procedures Section: In the second paragraph it reads: "the status of the test is changed to reflect the fact that its validity has been disputed" Earlier we say that "status is changed to "accepted" we should use the same verbiage here like: "state changed to "disputed""
17. - In Test status and review procedures Section: The last sentence of the second paragraph has very odd wording.
18. - In Test status and review procedures Section: second paragraph uses term "Task Force" in terms of the maintainers of the test materials. This is another concept that is first introduced in this context. This too would work better if the concept of appointing a "Test Task Force" was done at an earlier step.
19. - In Test status and review procedures Section in the first item of the list. The label "stable" may not be the best word for case 1 as it doesn't really reflect what is meant. "Error Free" or "Valid" would work better.
20. - In Test status and review procedures Section in the third item of the list. It reads: "questioned the correctness, consistency, clarity, etc" we think that correctness does not belong.
21. - In Test status and review procedures Section in the second and third item of the list. "A consensus exists in the community" What community? The w3c, the 'whole internet' the WG (as this is specifically talking about the task force.) We are also fairly uncomfortable with the notion that the task force can be over ridden by the (not clearly defined) community. Either the taskforce agrees or disagrees and the case accepted or not. Unless "community" means WG in which case this might make sense...
22. - In "Section Status of the test suite according to the above" We feel that the wording of this section heading could be improved quite a lot. We assume this means that if you conform to all of the above. or you've done/are doing all of the above.
23. - In "Section Status of the test suite according to the above" second paragraph It reads: "It is proposed that the W3C WG representative act as moderator and controller for incoming tests. The moderator is chosen by the [WG name] WG. All tests should be kept for archive purposes, whether they get published or not." Much of this is repeated from above. The new information is that a moderator should be chosen. This is another case where it may be beneficial to introduce the concept of a Moderator earlier in the document. This is already covered above to some degree. But with moderator.
24. - Both the Documentation and "See above" sections do not have references back to a GL document.

This document is applied and conformance (to this document) achieved as new TRs are being written." (grammatically incorrect, intended meaning unclear) This document applies to new TRs and conformance (to this document) is achieved as they are being written.

As for legacy specification, they may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints. (grammatical/spelling error) "legacy specifications"

Within this Specification Guidelines document, the term "specifications' is specifically limited to W3C Technical Reports, even though these guidelines could be applied to other documents. (unbalanced quote marks)

I'm finding it difficult to follow which terms are reserved for 'classes of products'. The word 'consumer' is used to describe the classes of products, and itself is listed within the classes of products. For example, I find the following sentence semantically confusing: "For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification." "For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way." "Play" refers to a media player, or play refers to "process" ?

Divide this (enumerated) list into processor, consumer, or content? Make the terminology in this area unique, so that there will be no ambiguity? (It could be that the terminology is already unique, but in its current format, I can't be sure.)

Ambiguity or error? (I count four 'of's in the second clause! :D)

1. - "QA deliverables" is very broad and not defined (GL3)
2. - "ensure" is not testable in a conformance requirement (at least CP 3.2, 6.1)
3. - "commensurate" isn't either (CP 4.2)
4. - using the future makes a CP untestable (CP 6.3)
5. - "a quality assessment" is not well defined (CP 7.1)
6. - "sufficient" is not testable (CP 7.2)
7. - 'all' is not testable (CP 7.3)

Overall, I think we have the right checkpoints, but I think they would be easier to understand if they were grouped chronologically. Start with what needs to be done when the WG is formed, and proceed through the spec and test development cycle to the maintenance phase, as we have tried to do with the restructuring of TestGL.

Guideline 1:

I find checkpoints 1.1 - 1.3 very confusing. These are "compound checkpoints" that incorporate multiple other checkpoints. We don't use this structure anywhere else in our docs - why here? The document states "This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA Workshop", but how we got here is not really interesting to the reader.

The 'sub-checkpoints' on the 'left hand side' of the table are all spec-related. Either these duplicate checkpoints from SpecGL, or they should be incorporated into that document. Those on the 'right hand side' are operations-related, but they seem to overlap with other checkpoints specified in this document.

Recommendation: drop the compound structure, make sure that all the spec-related sub-checkpoints are covered by SpecGL, and move the 'test materials' sub-checkpoints into the body of this doc.

The remaining checkpoints under guideline 1 are different kinds of beasts (not compound) and as such, the transition to them seems somewhat abrupt.

Guideline 2: I'd say "Allocate resources..." rather than "Define resources..."

Guideline 3 begins with some text ("The benefits of....") that seems to be a rationale. Other Guidelines jump straight into the Conformance requirements - this one should too.

How is checkpoint 3.1 different from 1.5?

Checkpoint 3.2 doesn't seem to be directly related to the Guideline under which it's classified. The Note for 3.2 points out that checkpoint 8.2 is related - doesn't this suggest that the overall structure should be re-worked (if they're related, why are they so far apart numerically?)

Guideline 4

Probably should be #1 (it's the first chronologically). When we summarized this document in our outreach presentation we made checkpoint 4.1 the first bullet item...

Checkpoints 4.1 and 4.2 would seem to belong in Guideline 2 (define/allocate resources) rather than here?

The Discussion for checkpoint 4.3 says "To summarize...". This implies that somewhere there's a more detailed description of what the QAPD must address, but I don't think there is. This checkpoint really seems to amount to "document how you meet these other checkpoints", yet the list of "other checkpoints" that must be implemented is incomplete. Why doesn't this simply require that *all* checkpoints be documented?

Checkpoint 4.5 uses the ambiguous term "framework" (we tried to avoid this in our re-write of OpsGL).

Checkpoint 4.6 addresses branding - another argument for a chronological rather than a 'logical' grouping (this should be at the end).

Guideline 5

Checkpoint 5.2 - Define a contribution process. Why only priority 2? Without a contribution process you have nothing, surely?

Guideline 6

Checkpoint 6.1 contains a mixture of stuff. The guideline addresses "publication" but much of this checkpoint addresses "management". Moreover, the bullet items in the Discussion section don't seem to relate to repositories at all.

Checkpoint 6.2 (defining the license for published test materials) is closely related to 5.3 (defining the license for submissions). Should these be grouped together (under a guideline that addresses submission processes)?

Guideline 7

This guideline is labelled "plan the transfer of test materials to W3C if needed", and explicitly states "all of the checkpoints... are not applicable if the WG does not transfer..." (should be "none of the checkpoints are applicable if..."). However, all the checkpoints seem to apply whether or not the materials are "transferred". It's obviously important to review the quality of submitted tests, to ensure that we have the staffing to deal with submissions, and to resolve IPR issues.

• Charter
• Allocation of resources
• Planning & synchronization
• Test submission/development
• Test management
• Test publication
• Conformance testing (test usage)
• Maintenance

In the guideline title and table of contents entry for Guideline 12 and in 3.4 last par., "pro forma" is two words.

The table of contents link to section 3.1 is broken.

(These probably have been reported by now but just in case.)

RFC 2119 [1] section "6. Guidance in the use of these Imperatives" puts limits their use. (I mentioned this to the QAWG about a year ago.) They musn't be used only to ask implementers to do something a Working Group would like to see.

"Imperatives of the type defined in this memo must be used with care and sparingly. In particular, they MUST only be used where it is actually required for interoperation or to limit behavior which has potential for causing harm (e.g., limiting retransmisssions) For example, they must not be used to try to impose a particular method on implementors where the method is not required for interoperability."

[1] http://www.rfc-editor.org/rfc/rfc2119.txt

• [1] http://lists.w3.org/Archives/Public/www-qa/2003Feb/0021.html
• [2] http://www.w3.org/2003/03/14-qa-intro-lesch.html

If the 14 specification guidelines can be clearly summarized in 6 bullets, as we claim is done in the QA Outreach Kit, http://www.w3.org/2003/Talks/qa-outreach/slide5-0.html , then perhaps the guidelines themselves ought to be consolidated and reduced in number. I believe that most of the checkpoints are appropriate, but perhaps should be repackaged.

Six bullets:

• Scope - define the scope, identifying what needs to conform and how
• Conformance Clause - specify the conformance policy and requirements
• If you must subset - use profiles, modules, functional levels
• Extensions - specify whether and how extensions are allowed
• Tag for later use - identify testable assertions, discretionary items
• Claims - specify how conformance claims and statements are made

I think the first 4 are almost right for guidelines. The later ones don't seem to me to be quite right.

1. new GL1: define and illustrate scope (includes old GL1, GL2)
2. new GL2: define conformance policy and requirements (old GL3, GL10)
3. new GL3: subset as needed (old GL4, GL5, GL6)
4. new GL4: extensions (old GL9)

   (It gets a little rougher from here on...)

5. new GL5: get a handle on other conformance variability
6. new GL6: is there a nice umbrella statement that would accomodate GL13 and GL14?
7. new GL7: claims -- specify how and provide an ICS [old GL11, GL12]

So at best, we would have 7 guidelines. At worst, 9 (if the new GL5 wouldn't stick together, and we couldn't find a new GL6 that was satisfactory).