This document records the QA Working Group's (QAWG) response to comments on the Last Call Working Draft of QA Framework: Operational Guidelines Please note that the table below is for Operational Guidelines only. We will generate separate tables for the Specification Guidelines and Introduction specifications.
The full statement of each issue and its full processing history can be found in QAWG's Last Call Issues List. The issue number in the first column of the table links to that. The Resolution column has links to specific text in a Operational Guidelines editor's draft, whose purpose is to illustrate the Disposition.
The last column indicates QAWG's disposition of the issue:
The deadline for replies to this DoC document is 12pm (noon) EDT, Monday, 18 August 2003. Your reply at your earliest convenience — whether you accept QAWG's disposition of your comment(s) or not — will help us to stay on schedule for progression of Operational Guidelines. If we do not hear from you (originator) by the deadline, your default reply is "accept QAWG's disposition of comments".
If you do not accept QAWG's disposition of your comments, please provide details, including your reasons, and also what change to the disposition would be required to satisfy you.
Last update: $Date: 2005/06/23 12:22:40 $
| Issue # | Originator | Description | Resolution | Status |
|---|---|---|---|---|
| LC-57 | Dominique Hazael-Massieux | GL and timeline of a document | There is already an approximate association of GL/CP order with WG life cycle (although the linkage cannot always pertain). For this reason, and to minimize disruptions to the OpsGL structure and text, OpsGL will not further re-organize based on chronology. However, per originator's (alternate) proposal, an informative section will be added to chapter 1 (Introduction), explaining the chronological connections with a graphic or a table. See draft revised text. | accepted |
| LC-58 | Dominique Hazael-Massieux | Process Document requirement is too specific | Thirteen other checkpoints specificy the minimal content of a QAPD. The OpsGL text will be changed to indicate that the QAPD can be a separate standalone document, or a TOC to distributed bits of required QAPD information, or a hybrid. See draft revised text. | accepted with modifications |
| LC-59.1 | Dominique Hazael-Massieux | Testability concerns: "QA deliverables" is very broad and not defined (GL3) |
Per email proposal, the ConfReq is made more precise, the "For example..." in Discussion is replaced with minimal content of "QA deliverables", and broader examples are also presented. See draft revised text. |
accepted |
| LC-59.2 | Dominique Hazael-Massieux | Testability concerns: "ensure" is not testable in a conformance requirement (at least CP 3.2, 6.1) |
Per email proposal, the ConfReq is reworded (in both CP3.2 & 6.1) to replace "ensure" with more testable language. See draft revised text of CP3.2 and CP6.1. |
accepted |
| LC-59.3 | Dominique Hazael-Massieux | Testability concerns: "commensurate" isn't either (CP 4.2) |
Per email proposal, the ConfReq is reworded to eliminate "commensurate" and make it more testable, and a new clarifying sentence is added to Discussion. See draft revised text of CP4.2 and CP2.2. |
accepted |
| LC-59.4 | Dominique Hazael-Massieux | Testability concerns: using the future makes a CP untestable (CP 6.3) |
Per email proposal, the ConfReq is reworded to eliminate the problem. See draft revised text of CP6.3. |
accepted |
| LC-59.5 | Dominique Hazael-Massieux | Testability concerns: "a quality assessment" is not well defined (CP 7.1) |
Per email proposal, a handful of examples are added to Discussion, to illustrate what goes into a TM quality assessment, and a more exhaustive list/template will be put into OpsET. See draft revised text of CP7.1. |
accepted |
| LC-59.6 | Dominique Hazael-Massieux | Testability concerns: "sufficient" is not testable (CP 7.2) |
The comment is moot for the occurrence of "sufficient", as it is in the statement of the CP, which is merely its "title" (not normative). However "commensurate" in the ConfReq is fixed, per email proposal, and a clarifying sentence is added to Discussion. See draft revised text of CP7.2. |
accepted with modification |
| LC-59.7 | Dominique Hazael-Massieux | Testability concerns: 'all' is not testable (CP 7.3) |
Per email proposal, delete the occurrence of "all" in ConfReq. See draft revised text of CP7.3. |
accepted |
| LC-110.1 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: QA is very important, and the QA WG has the right goals. [No issue.] |
Thanks for the comment -- the positive affirmation of the QAWG goals is useful to us. |
accepted |
| LC-110.2 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: There were a lot of discussions regarding the writing style we adopted for the framework, namely the fact that we use RFC keywords in conformance requirements. Some people thought it was too "aggressive", other felt it was the right thing to do. |
QAWG reaffirms that it considers the RFC keywords appropriate for OpsGL and the other Framework documents. See also Last Call issue 67. |
accepted |
| LC-110.3 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: [OpsGL only] The table in OpsGL GL 1 caused much confusion and was deemed as not-understandable, which I think we already more or less agree with. |
Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3). |
accepted |
| LC-110.4 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: The intents of the priorities/degrees is not always clear. Proposal [DH]: we should probably emphasize somewhere that the minimal recommended degree is to be AA conformant (or that is the intention of the WG to request it to be the lowest level for work in W3C). |
Discussed and resolved at 20030512 telecon. First, it is our intent that Level A be the minimum. This will somehow be worked into the description of the semantics of the levels, in the Conformance clause ("Conformance definition"). Second, in the re-write of the introduction, we will make generic reference to checkpoint priorities, and move the details into the Conformance clause. See draft revised text in the Introduction, and the "Conformance definition". |
accepted with modification |
| LC-110.5 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: Generally speaking, the distinction between GL and ExTech was not always clear. Proposal [DH]: we probably need to rework the introductory sections to clarify that. |
Resolved per email proposals -- it will be clarified and emphasized. In order to comply with the "better readability" request of point #7 in this issue, the detail will be put into QA Framework: Introduction, and linked from the brief treatment in the OpsGL introduction. See draft revised text. |
accepted with modification |
| LC-110.6 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: The summarized view (ICS/Checklists) are not easy enough to find, and are not explicitly recommended enough. Proposal [DH]: again, that means some re-working of the introduction. |
Resolved per email proposals -- they will be given more exposure and emphasis. See draft revised text. |
accepted |
| LC-110.7 | Dominique Hazael-Massieux | Collected OpsGL comments from Team.: The introduction needs to be much more efficient to read. Proposal [DH?]: some kind of an executive summary rather than the long prose we currently have. |
Discussed and resolved at 20030512 telecon. It is not so much "executive summary", as writing style and organization that needs attention: more concise, more use of emphasis, use bullets instead of long prose, etc. See the clarifications (and useful Web-writing-style reference) in the email thread. See extensively rewritten introduction in the draft revised text. |
accepted with modification |
| LC-56 | Jon Gunderson | Accessibility |
To the extent that accessibility requirements are written into
the target specification as conformance requirements
of that specification, then they are adequately covered by the QA Framework
guidelines family, specifically by SpecGL and TestGL. No additional
QA Framework requirements are needed.
In this case, tests of accessibility requirements are within the
domain of the (OpsGL-required) Test Moderator's job, and therefore a
special accessibility test coordinator is not needed.
If, on the other hand, accessibility requirements are not integrated into the target specification as conformance requirements, then the problem is beyond the scope of the QA Framework as currently defined (for more about this scope point, see email discussion.) In this case, the issue is larger than OpsGL (or TestGL or SpecGL). QAWG issue #12 explored this -- the relationship amongst the QA, WAI, and I18N horizontals. |
not accepted |
| LC-83 | Jonathan Marsh | Seven levels vs. Three | The 7-level commitment table has been eliminated, and the commitment requirements have been written into new CP1.1 - CP1.3 that are consistent with the 3-level conformance model of the QA Framework family. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3). | accepted with modifications |
| LC-72.1 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-1]: Editorial -- unclear wording in section 1.1. |
Proposal. Accept Originator's proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text (4th bullet). |
accepted |
| LC-72.2 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-2]: 7-level table creates confusing conformance requirements that contradict the priorities/degrees system. |
Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3). |
accepted |
| LC-72.3 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-3]: CP1.1 is priority 1, but requires checkpoints of other priorities. |
Discussed at 20030501 telecon. Becomes moot as a result of the resolution of LC-3 (and previous sub-issue.) See draft revised text of CP1.1 (as well as CP1.2, CP1.3). |
accepted with modification |
| LC-72.4 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-4]: GL3 wording emphasizes interoperable implementations, should instead emphasize conformant implementations. |
Proposal. Accept Originator's proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text (3rd bullet). |
accepted |
| LC-72.5 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-5]: CP3.2, "support specification versioning/errata" should be priority 1 instead of 3. |
Proposal. Accept originators request -- change p3 to p1. Discussed at 20030512 telecon and 20030516 telecon. Resolved to change priority to P1, and also to rephrase requirements in terms of capability to unambiguosly associate spec and TM versions (for which TM infrastructure support is one method of achieving.) Relationship to CP8.2 also documented. See draft revised text. |
accepted with modification |
| LC-72.6 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-6]: Editorial -- poor wording in CP4.3. |
Proposal. Agree that it's poor wording, change "minimally addresses all of the topics" to "addresses at least all of the topics". Resolved, proposal endorsed at 20030512 telecon. See draft revised text. |
accepted with modification |
| LC-72.7 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-7]: Editorial -- CP4.5 uses "QA Framework" in a different (but undefined) sense that the OpsGL title. |
Proposal. Agree, the same phrase is used with a different meaning. Change the CP wording from "Define the QA framework for test materials development" to "Define a framework for test materials development." Resolved, proposal endorsed at 20030512 telecon. (The checkpoint is also moved, to become the first checkpoint of Guideline 5.) See draft revised text. |
accepted with modification |
| LC-72.8 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-8]: CP4.5 should be priority 2 instead of priority 1. |
Email proposal [LH]: Accept originator comment -- defining the framework for TM development is not critical enough to be Priority 1. Lower to priority 2. Discussed and resolved as proposed at 20030512 telecon. (The checkpoint is also moved, to become the first checkpoint of Guideline 5.) See draft revised text. |
accepted |
| LC-72.9 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-9]: Editorial -- ambiguous wording in CP5.4. |
Proposal. Agree with the intent of the comment. Change "..the procedure MUST minimally address criteria for.." to "..the procedure MUST address at least criteria for.." Resolved, proposal endorsed at 20030512 telecon. See draft revised text. |
accepted with modification |
| LC-72.10 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-10]: CP6.2 -- "address TM licenses" -- should be substantively modified. |
License task team results. Extensive discussion by TM-license task team at a 20030611 telecon, involving all identified principal stakeholders (member companies, QAWG, W3C Legal, etc): see public summary and detailed minutes (for now, member-only). Resolution: W3C will continue to pursue the Document and Software licenses as the mechanisms for governing test suites. A new TM license is not seen as needed, and there are no major defects that compromise the usability and applicability the Software License or the Document License for W3C Test Materials, or of both together applied to different components of a complete Test Materials. W3C Legal will continue to investigate a number of possible (relatively minor) improvements. Guidance in the licenses FAQ will be investigated. QAWG will continue to look at both adjustments to its Guidelines documents (e.g., TestGL guidance on partitioning the TM components suitably for different license application), and possibly some further FAQ-like guidance in "Operational Examples & Techniques" (specifically for Checkpoint 6.2). See draft revised text. |
accepted with modification |
| LC-72.11 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-11]: CP6.3 arguments against publishing TM in the TR space are not convincing, should be improved (proposal included). |
Proposal. Accept the proposed rewording. Resolved, proposal endorsed at 20030512 telecon. See draft revised text. |
accepted |
| LC-72.12 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-12]: Problems with CP6.5 ("publish test results"), discussion of test harnesses, should be fixed. |
Email proposal [LH]: Agree with intent of Originator. In Discussion, change the sentence, "Such publication should include or describe a test harness that would allow anyone to reproduce the results" to "Such publication should include or describe a test harness that allows reproduction of the results." Resolved, proposal endorsed at 20030512 telecon. See draft revised text. |
accepted with modification |
| LC-72.13 | Leonid Arbouzov | Collected substantive & editorial comments: [OG-13]: CP8.2 (versioning/errata in maintenance) should be priority 1 instead of 2. |
Proposal to accept (change priority 2 to priority 1 in CP8.2 [versioning/errata in maintenance]) was discussed at 20030512 telecon and 20030516 telecon. Resolved to change priority to P1, and also to clarify scope of "versions" (minimally, "editions"). Relationship to CP3.2 also documented. See draft revised text. |
accepted |
| LC-82 | Lofton Henderson | OpsGL fails SpecGL checkpoints 1.2 and 1.3 | Use cases are added to OpsGL to illustrate its scope. See draft revised text. | accepted |
| LC-3 | lynne rosenthal | Committment Table and its CPs | The table is eliminated, and CP1.1 - 1.3 are replaced. CP1.1 will require choice of A, AA, or AAA for OpsGL, choice of A, AA, or AAA for SpecGL, and choice of A, AA, or AAA for TestGL. New CP1.2 and CP1.3 will require commitment to (respectively) "some" or "complete" test materials, preserving that normative aspect of the previous table. All previous normative requirements are preserved, while the redundant conformance scheme is eliminated. See draft revised text of CP1.1 (as well as CP1.2, CP1.3). | accepted |
| LC-60.1 | Patrick Curran | Structure/Organization of Guidelines: Overall, I think we have the right checkpoints, but I think they would be easier to understand if they were grouped chronologically. Start with what needs to be done when the WG is formed, and proceed through the spec and test development cycle to the maintenance phase, as we have tried to do with the restructuring of TestGL. |
Discussed at 20030512 telecon, and resolved similar to email proposal. QAWG resolved to minimize large changes to the OpsGL structure and text that don't fix serious problems. Therefore, OpsGL will not be reorganized based on chronology. Further, there is already an approximate association of guidelines and checkpoints (GL/CP) with the ideal point of applicability in a WG life, and this is partially reflected in the GL/CP ordering. An informative section is added to chapter 1 (Introduction), explaining these chronological connections with a graphic or a table. See also related issue LC-57. |
accepted with modification |
| LC-60.2 | Patrick Curran | Structure/Organization of Guidelines: Guideline 1: I find checkpoints 1.1 - 1.3 very confusing. These are "compound checkpoints" that incorporate multiple other checkpoints. We don't use this structure anywhere else in our docs - why here? The document states "This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA Workshop", but how we got here is not really interesting to the reader. |
Discussed and resolved at 20030501 telecon. See LC-3 for description and resolution. See draft revised text of CP1.1 (as well as CP1.2, CP1.3). |
accepted |
| LC-60.3 | Patrick Curran | Structure/Organization of Guidelines: Guideline 2: I'd say "Allocate resources..." rather than "Define resources..." |
See email proposal. Resolved per this proposal at 20030516 telecon. The wording of GL1 and GL2 is changed consistently, to emphasize that these are about commitment, and GL2 now starts with "Commit to resource level..." See draft revised text. |
accepted with modification |
| LC-60.4 | Patrick Curran | Structure/Organization of Guidelines: Guideline 3 begins with some text ("The benefits of....") that seems to be a rationale. Other Guidelines jump straight into the Conformance requirements - this one should too. |
Discussed in email. Withdrawn by originator while discussing clarification of the request. |
accepted with modification |
| LC-60.5 | Patrick Curran | Structure/Organization of Guidelines: How is checkpoint 3.1 different from 1.5? |
See email proposal. Resolved per this proposal at 20030516 telecon. Add "Related checkpoints" to each of CP1.5 and CP3.1, pointing to the other and explaining the distinction. See draft revised text. |
accepted with modification |
| LC-60.6 | Patrick Curran | Structure/Organization of Guidelines: Checkpoint 3.2 doesn't seem to be directly related to the Guideline under which it's classified. The Note for 3.2 points out that checkpoint 8.2 is related - doesn't this suggest that the overall structure should be re-worked (if they're related, why are they so far apart numerically?) |
See email proposal. Resolved per this proposal at 20030516 telecon. This and several other aspects of CP3.2 and 8.2 are clarified and resolved all at once in the proposal. See draft revised text. |
accepted with modification |
| LC-60.7 | Patrick Curran | Structure/Organization of Guidelines: Guideline 4: Probably should be #1 (it's the first chronologically). When we summarized this document in our outreach presentation we made checkpoint 4.1 the first bullet item... |
See email proposal. Resolved per this proposal at 20030516 telecon. GL1/2 are about early *commitment* to QA and QA deliverables. Possibly even in the Charter, before the WG is rolling. GL4 is about process and operations within the functioning WG. It is equally arguable that this is the best order. Per the 20030512 resolution of LC-60.1, we will avoid any re-ordering or re-grouping of GL/CP except to fix something that is really broken. Accordingly, it is resolved to keep the current order here. |
not accepted |
| LC-60.8 | Patrick Curran | Structure/Organization of Guidelines: Checkpoints 4.1 and 4.2 would seem to belong in Guideline 2 (define/allocate resources) rather than here? |
See email proposal. Resolved per this proposal at 20030516 telecon. GL1 and 2 are about early commitment (ideally Charter) to QA levels, staffing levels, etc. GL4 is about performance -- specific staff assignments -- going forward. To each of CP2.2, CP4.1 and CP4.2, added reciprocal clarifications about the others, and disambiguate the similarity between them. See draft revised text. |
accepted with modification |
| LC-60.9 | Patrick Curran | Structure/Organization of Guidelines: The Discussion for checkpoint 4.3 says "To summarize...". This implies that somewhere there's a more detailed description of what the QAPD must address, but I don't think there is. This checkpoint really seems to amount to "document how you meet these other checkpoints", yet the list of "other checkpoints" that must be implemented is incomplete. Why doesn't this simply require that *all* checkpoints be documented? |
See email proposal. Resolved per this proposal at 20030516 telecon. This is clarified, in conjunction with solution to LC-58, by adding a new paragraph at the beginning of CP4.3 "Discussion". The discussion is also expanded to specifically mention the other QAPD-related checkpoints, and a "Related checkpoints" is added. See draft revised text. |
accepted with modification |
| LC-60.10 | Patrick Curran | Structure/Organization of Guidelines: Checkpoint 4.5 uses the ambiguous term "framework" (we tried to avoid this in our re-write of TestGL). |
Discussed in 20030516 telecon. See subsequent email proposal. Resolved per email proposal by email (default). LC-72.7 also raises this issue. Other comments indicate confusion why CP4.5 is here at all, instead of TestGL. All of these are addressed in the proposal, for complete replacement text. The text clarifies why the topic is dealt with in OpsGL (instead of or in addition to TestGL), and changes the wording "QA framework" to "framework for test materials development". ("Framework" is still used, in lieu of specific proposals for a better phrasing.) The checkpoint is also moved, to become the first checkpoint of Guideline 5. See draft revised text. |
accepted with modification |
| LC-60.11 | Patrick Curran | Structure/Organization of Guidelines: Checkpoint 4.6 addresses branding - another argument for a chronological rather than a 'logical' grouping (this should be at the end). |
See email proposal. Resolved per this proposal at 20030516 telecon. Because branding policy decision has potential knock-on effects on other aspects of the WGs QA life, process, TM, and operations, it is equally arguable that its early consideration is beneficial. Given the resolution that we are not going to do a major chronological re-org (LC-60.1) except to fix really broken things, it is resolved to keep it as is. |
not accepted |
| LC-60.12 | Patrick Curran | Structure/Organization of Guidelines: Guideline 5: Checkpoint 5.2 - Define a contribution process. Why only priority 2? Without a contribution process you have nothing, surely? |
See email proposal. Resolved per this proposal at 20030516 telecon. There are test suites that have been built entirely without any formal or defined contribution process -- mostly by a single person, with ad-hoc acceptance procedures for internal contributions. Certainly this is not optimal, but also not critical in all cases (perhaps in some). If possible, we want to avoid increasing the number of P1 checkpoints unless we feel they are really critical/essential. Therefore, it is resolved to keep it as is. |
not accepted |
| LC-60.13 | Patrick Curran | Structure/Organization of Guidelines: Guideline 6: Checkpoint 6.1 contains a mixture of stuff. The guideline addresses "publication" but much of this checkpoint addresses "management". Moreover, the bullet items in the Discussion section don't seem to relate to repositories at all. |
See email proposal. Resolved per this proposal by email (default). Take the bullet list out of CP6.1, fine tune the wording, and put it in the verbiage of GL6. Add comment to the GL6 verbiage explaining the preprequisite relationship of some management aspects to publishing, and therefore their appropriate treatment of those here in GL6. See draft revised text. |
accepted with modification |
| LC-60.14 | Patrick Curran | Structure/Organization of Guidelines: Checkpoint 6.2 (defining the license for published test materials) is closely related to 5.3 (defining the license for submissions). Should these be grouped together (under a guideline that addresses submission processes)? |
See email proposal. Resolved per this proposal at 20030516 telecon. Consistent with the LC-60.1 resolution of "minimize improvements that don't fix something really broken", it was resolved that we should clarify and cross link them, instead of trying to combine them. Part of the clarification includes wording changes approved by W3C Legal, and other interested parties in the TM license issue. See draft revised text. |
accepted with modification |
| LC-60.15 | Patrick Curran | Structure/Organization of Guidelines: Guideline 7: This guideline is labelled "plan the transfer of test materials to W3C if needed", and explicitly states "all of the checkpoints... are not applicable if the WG does not transfer..." (should be "none of the checkpoints are applicable if..."). However, all the checkpoints seem to apply whether or not the materials are "transferred". It's obviously important to review the quality of submitted tests, to ensure that we have the staffing to deal with submissions, and to resolve IPR issues. |
See email proposal. Resolved per this proposal by email (default). All of the submission stuff does get addressed in GL5. GL7 is about the wholesale transfer of an entire externally written test suite. While this could be considered as a "submission" (therefore redundant with some of GL5), on the other hand the sense of GL5 is more about piecemeal submission during the building of test materials. QAWG has had experience moderating such a transfer, which is why GL7 exists separately -- it's useful to have it in one place, even if there are parallels elsewhere in OpsGL for piecemeal planning, development, staffing, submission, etc. Resolved to fine tune the wording of the GL7 verbiage and add some more rationale/explanation. See draft revised text. |
accepted with modification |
| LC-43 | Peter Fawcett | OpsGL Appendix 1 - Process Document Template | Per updated (20030706) draft revised QAPD text. | accepted |
| LC-70 | Phill Jenkins | Checklist format issue | The QA Guidelines checklists are intended to be implementation conformance reporting forms. Accordingly, the category of "planned" is not considered appropriate, nor is the provision of a "Comments" field. Without the 40% "comments" field, the remainder of the reformatting proposal would need significant re-adjustment. QAWG has identified and intends to produce another sort of form where a comments field would indeed be appropriate -- a "test material" for the Guidelines documents, in the form of a questionnaire, which has a question/item per-conformance-requirement (as opposed to per-checkpoint). See clarification of checklist purpose in draft revised text. | not accepted |