QA Framework: Process & Operational Guidelines

W3C Working Draft 01 February 2002

This version:

http://www.w3.org/QA/WG/2002/framework-20020201/qaframe-ops

Latest version:

http://www.w3.org/QA/WG/qaframe-ops

Previous version:

http://www.w3.org/QA/WG/2002/framework-20020118/qaframe-ops

Editors:

Kirill Gavrylyuk (kirillg@microsoft.com)

Dimitris Dimitriadis (dimitris@ontologicon.com)

Lofton Henderson (lofton@rockynet.com)

Lynne Rosenthal (lsr@nist.gov)

Contributors:

See Acknowledgments.

Copyright ©2002 W3C ^® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

---

Abstract

This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Materials Guidelines.

Status of this document

(Ed note. This is almost the same as the first public WD of the same date in /TR/, except for some minor formatting differences in "References" and in link-anchor style.)

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first public Working Draft, and incorporates the discussions at the first QA face-to-face, plus several subsequent QA Working Group [QAWG] teleconferences, and supersedes all previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

This part of the Framework document family will eventually have an accompanying "Process & Operational Examples and Techniques" (in progress). Some of the lengthier examples and "how to" text of this current guidelines document version will be moved to the "Examples and Techniques" document.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.

Table of contents

1. Introduction
    1.1 Motivation for this Guidelines document
    1.2 Navigating through this document.
    1.3 Terminology
2. Guidelines
        G 1. In charters, describe QA goals, criteria, deliverables and resources
        G 2. In the Working Group charter, address where and how conformance test materials will be produced.
        G 3. Ensure that the goals and scope of QA development are clear
        G 4. Allocate staff for the QA process.
        G 5. Synchronize specification and test materials development
        G 6. Produce QA process document.
        G 7. Plan the transfer of test materials to W3C if needed
        G 8. Plan for test materials maintenance
3. Working Group relationship to QA Activity
    3.1 Liaison and consultation.
    3.2 Active reviews.
    3.3 Adjudicate entry and exit criteria.
    3.4 QA resources supplement.
    3.5 Resolution of external QA requests
4. Conformance
5. Acknowledgments
6. References

1.Introduction

This document describes the process and operational activities for building conformance test suites and tools for W3C specifications. The primary goal is to assist the W3C Working Groups (WGs) by providing guidelines for the planning, development and deployment of conformance test materials. In this guideline, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

• Introduction
• Process & Operational Guidelines (this document)
• Specification Guidelines
• Test Materials Guidelines

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

1.1Motivation for this Guidelines document

As the complexity of W3C specifications and their interdependencies increases, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials.

Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.

"[...] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (see Process Document, section 3.)

"W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (see Process Document, section 5.2.5, "Ongoing work".)

In an effort to meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their CR-exit and PR-entrance criteria. Examples include Scalable Vector Graphics (SVG), User Agent Accessibility Guidelines (UAAG), and Extensible Style Language - Formatting Objects (XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.

There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix identifies more than a score of test suites and validators, in various states of development. Moreover, many Working Groups have already established procedures, techniques and tools for developing test materials (e.g., Document Object Model - DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.

1.2Navigating through this document.

The Guidelines in the document are organized chronologically starting with those that apply at the formation of a Working Group (e.g., charter considerations) and through a progression of guidelines leads the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all Working Groups, including those that are being rechartered or already exist. Working Groups may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:

• The guideline number
• The statement of the guideline
• The rationale behind the guideline
• A list of checkpoint definitions.

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

• The checkpoint number
• The statement of the checkpoint
• The priority of the checkpoint. Priority 1 checkpoints are highlighted through the use of style sheets.
• Optional informative notes, clarifying examples, and cross references to related guidelines or checkpoints.

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

1.3Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

2. Guidelines

Guideline 1. In charters, describe QA goals, criteria, deliverables and resources

In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials (TM). Therefore, conformance test materials -- test suites and tools -- must be considered as Working Group deliverables the same as the standards themselves. Accordingly, they must be addressed in the Working Group charter.

Checkpoints:

Checkpoint 1.1. In the Working Group charter include conformance test materials plans. Plan to have at least a commitment to "QA level three". [Priority 1]

Detailed planning and specification of test materials is inappropriate in the charter, as it binds the Working Group inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a Working Group could make, based on anticipated Working Group resources, existence of outside efforts, and the Working Group's current stage in the process [REC-TRACK].

Based on level of testability of the specification and extent of test materials, a range of possible commitments has been identified.

This 10-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the Working Group commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

Checkpoint 1.2. In the Working Group charter, specify completion and publication of test materials to be a criterion for CR-exit and PR-entrance. [Priority 2]

A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG), UAAG), ...) This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

"Advancement of a technical report to Candidate Recommendation is an explicit call for implementation experience to those outside of the related Working Groups or the W3C itself." (see Process Document, section 5.2, about "Candidate Recommendation".)

"Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature." (see Process Document, section 5.2.4, about "Entrance Criteria".)

Checkpoint 1.3. In the Working Group charter, address QA staffing commitments. [Priority 2]

There will be at least some staffing required from the Working Group, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques".

Guideline 2. In the Working Group charter, address where and how conformance test materials will be produced.

There are several possibilities for producing conformance test materials:

• build them entirely within W3C (normally, the Working Group). Example: SVG test suite.
• build them externally and transfer to W3C. Example: XML test suite.
• build procedures for assembling contributions from multiple sources. Example: XSLT test suite (a work in progress, this is actually being done in OASIS (Organization for Structured Information Standards) XSLT/Xpath Conformance Technical Committee, but it could just as well work within a W3C Working Group).
• hybrids of the above.

Checkpoints:

Checkpoint 2.1. In the Working Group charter, specify the WG's level of involvement in the ongoing evolution of test materials. [Priority 1]

There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the level to which the Working Group committed in its charter.

Checkpoint 2.2. Ensure a secure and reliable repository location for future test materials. [Priority 2]

This doesn't necessarily mean that test suite/tools should physically reside in the W3C, although that is a recommended way of meeting the checkpoint, because:

1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications normally increase, and the Working Group is the best venue for initial processing and adjudication of such queries against both.
2. Further to #1, it is more likely that technical report (Recommendation) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the Working Group.

• the test materials have proper maintenance and support
• tests materials are freely available for download. If license is applied, it does not preclude the test suite from being used by all interested parties
• the test materials' quality and usability is appropriate

If the Working Group ceases to operate, then other responsible for ongoing interpretation and maintenance (errata) of the standard should be designated to ensure continued availability of the test suite.

Guideline 3. Ensure that the goals and scope of QA development are clear

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

• to enable W3C Web standards to achieve full interoperability and access to all, by addressing the quality and correctness of implementations.

Specific goals of test materials development include:

• provide self-assessment tools and materials for implementation builders;
• help implementation builders to achieve interoperability of implementations;
• help users to verify compliance of implementations with the stated conformance requirements of their respective specifications;

The goals of QA work within Working Group do not encompass:

• measurement of performance, scalability of implementations, or their assessment against any other "goodness" criteria, except what is stated as a conformance requirement of their respective specification.
• certification of implementations

Checkpoints:

Checkpoint 3.1. Ensure that test materials are documented, self contained and reusable by accredited third parties for testing purposes. [Priority 2]

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, developed test materials can be used by external parties including certification services. Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Checkpoint 3.2. Specify a policy for branding materials if applicable. [Priority 3]

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

Checkpoint 3.3. Provide disclaimer regarding use of the test materials for compliance verification. [Priority 2]

Although tests suites may be used for compliance verification, users must be aware that:

1. passing all of the tests does not guarantee full compliance of an implementation to the specification
2. failing test suite means failing tests for specific feature they target

The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid:

• for conformance testing of the (web) content: statement 1 is false, but statement 2 is true
• for content/driven conformance testing of the tools/product: both statements are true
• for interfaces (API) conformance testing of the tools or libraries: both statements are true

Guideline 4. Allocate staff for the QA process.

Before starting test materials development, a Working Group has to appoint several key participants for its QA process.

Checkpoints:

Checkpoint 4.1. Appoint a test QA moderator. [Priority 2]

• Working Group members (and other W3C members)
• W3C QA staff

When a test suite is developed outside of the Working Group, a moderator often already exists in the external group that is the source of the test suite. It is recommended to invite the moderator to become a Working Group member, since he/she needs access to all Working Group materials and resources. See also section on transferring test suite from outside W3C.

Checkpoint 4.2. Designate a task force or another entity to maintain the test suite, as well as maintenance procedures. [Priority 2]

It is recommended that the Working Group's errata task force be involved in test suite maintenance, since this will help to keep test suite and specification synchronized. The maintenance group shall establish a tracking system to track issues and changes to the test suite.

Guideline 5. Synchronize specification and test materials development

There are several strong benefits from starting synchronizing of specification and test materials developments as early as possible:

• it gives extensive list of use cases
• it identifies ambiguities in the specification at the early stages, which saves cycles in the late phases
• it provides clear set of testable assertions -- skeleton of the specification -- which in its turn facilitates development of interoperable implementations. The latter is a W3C process criterion for entering Candidate Recommendation phase

Checkpoints:

Checkpoint 5.1. Introduce intermediate milestones for test materials completion. [Priority 3]

These milestones can be naturally bound to Working Draft and/or Candidate Recommendation publications.

Checkpoint 5.2. Synchronize the latest published version of the specification and the latest published version of the test materials. [Priority 3]

Checkpoint 5.3. Support specification versioning/errata in the test materials. [Priority 2]

The structure of the test materials should allow versioning/errata to enable verification of compliance with different versions of the specification.

Guideline 6. Produce QA process document.

A Working Group QA process document describes how the QA-related processes are set up within the Working Group. Examples of such a document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of the QA process in a Working Group, and the following requirements apply to it.:

Checkpoints:

Checkpoint 6.1. Review Test Materials Guidelines before writing the QA Process document and starting test materials development. [Priority 2]

Taking into account the nature of the future standard (Recommendation), before starting test suite development, the test suite implementer (Working Group or 3rd party) should look at Test Materials Guidelines and decide what approach to take for test materials organization, test criteria, etc.

Checkpoint 6.2. In the QA Process document, define a contribution process. [Priority 2]

The process document must describe where to submit test materials and whom to notify (e.g., moderator) of a submission. The contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.

Checkpoint 6.3. In the QA Process document, define the licenses applicable to submitted and published test materials. [Priority 2]

If test materials are produced within the W3C, the Working Group can choose either the W3C Document or the W3C Software license. If applicable, W3C Document license is recommended. For publishing test materials acquired from an external group, other licenses may be applicable (see transferring test suite).

Checkpoint 6.4. In the QA Process document, define a means for communication regarding test suites. [Priority 2]

Working Group needs to choose mailing list to send all comments and announcements to.

Checkpoint 6.5. In the QA Process document, define review procedures for submitted test materials. [Priority 2]

The following example may serve as a guideline for tests review process

1. Each contribution should be reviewed for its eligibility by at least one vendor other than the submitter. Any vendor involved in the creation of the Working Group test collection other than the submitter may review contributed tests.
2. Eligibility of tests should be judged by:
   • The accuracy of the test. The accuracy of a test is determined by a judgment of the reviewer whether the test case actually tests what the submitter states that the test case tests. The accuracy is the extent to which the test matches the cited parts of the specification. If it does not match, or only partially matches, the test should be considered inaccurate.
   • The scope of the test. Test should target MUST provisions of the specification and not MAY/SHOULD provisions. The test should target only conformance requirements. Data values limits, memory or performance limits should not be targets to the test unless those are specified in the conformance requirements.
   • The clarity of the test. The clarity of a test is a determination of whether the aspect being tested is clearly described with the anticipated results acceptably explained.
   • The clarity of aspect of the Specification being tested. The test suite aims to test parts of the Specification and errata that aren't vague.
3. After review, results should be sent back to the submitter for feedback.
4. In case of disagreement, a request for clarification should go to Working Group. Tests in question should be left out of the test suite until the W3C Working Group clarifies the issue.
5. Only those tests that are agreed to be correct by reviewers and the submitter, or that are approved by W3C Working Group clarification, should be published.

Checkpoint 6.6. In the QA Process document, describe the way test materials will be published and point to the corresponding web page. [Priority 2]

If the test materials are to be published on the W3C site, it is recommended to locate them within the corresponding activity domain.

Checkpoint 6.7. In the QA Process document, describe how vendors can publish test results for their products, if applicable. [Priority 3]

It is recommended to use one of the practices for publishing test materials, described in Test Materials Guidelines. The Working Group can encourage vendors to publish results for their implementations, such publication ideally including or describing a test harness that would allow anyone to reproduce the results.

Guideline 7. Plan the transfer of test materials to W3C if needed

As stated before in the notes to the checkpoint for test materials repository, it is recommended that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group (WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define the requirements for the Working Group.:

Checkpoints:

Checkpoint 7.1. If transfer of the test materials to W3C is planned, perform assessment of their quality. [Priority 2]

Checkpoint 7.2. If transfer of the test materials to W3C is planned, identify staff resources to meet requirements. [Priority 2]

Checkpoint 7.3. If transfer of the test materials to W3C is planned, resolve IPR questions and reach agreement with external party that produced test materials. [Priority 1]

Checkpoint 7.4. If transfer of the test materials to W3C is planned, update QA commitments in the Working Group charter if necessary. [Priority 2]

The following is a recommended procedure for meeting these requirements.

Legend for the procedure::

EG

the external group or entity

QAWG

the QA Working Group

TM

the test materials

WG

the subject Working Group

Before the transfer, WG with the help of QAWG:

• performs an assessment of the quality of candidate TM (by WG, QAWG)
• identifies and commits to derive a set of QA deliverables from the candidate TM. These can be: releases, appeals/challenges, maintenance, submissions, Web site, mail list, etc. (by WG)
• identifies staff resources to meet the requirements, including at least a TM moderator. It is recommended to invite the TM moderator from the EG (if one exists) to become a WG member should the transfer happen.(by WG)
• takes the decision to seek/accept the transfer. (by WG)
• initiates charter revisions for QA commitments (by WG, QAWG may consult)

During transfer:

• EG and W3C reach agreement to transfer the TM (by WG, QAWG, EG)
• WG and EG perform the actual transfer of the TM, WG creates an initial repository (by WG, EG)

After transfer, initial process setup:

• WG appoints a TM moderator. The TM moderator creates a QA process document for WG (by WG, TM moderator, QAWG may consult)
• Logistics setup: the TM home page, a TM issues mailing list (by WG, TM moderator)
• Determine the appropriate W3C IPR license (by WG, QAWG)

First W3C public release of new TM:

• Make any necessary enhancements prior to the first public release: fix known/reported errors, produce documentation (by WG)
• Announce the first public release of TM (by TM moderator, Communications Group)
• Joint W3C/EG press release (by WG, QAWG, Communications Group, EG)

After the first public release, the TM enter maintenance phase which is described below.

Guideline 8. Plan for test materials maintenance

Once test materials development is initiated, its maintenance requires an equally significant amount of resources. Resources allocation is needed as prescribed by the maintenance checkpoint above.

Checkpoints:

Checkpoint 8.1. Maintain contribution and review procedures throughout test materials' and standard's entire life cycles. [Priority 3]

Checkpoint 8.2. In the Working Group's QA process document, specify a procedure for updates of the test materials to track new errata/specification versions. [Priority 2]

The following procedure is an example that meets the requirements:

• Initiating test suite change/update. As soon as errata are accepted or a new version/edition of the Recommendation is published, the errata task force (or test suite maintenance group) identifies affected test cases and initiates updates to the test suite by adding entries to the tracking system with status "ACTIVE"
• Providing test suite changes for review. Preferably, the test case submitter or initiator of the change provides updated versions for the affected tests. If no changes are thus provided, then the test maintenance group finds a way to provide it. Once changes are ready, maintenance group changes the corresponding status in the tracking system to "RESOLVED"
• Reviewing/accepting the change. Test changes go through the same tests review process as presented earlier. If the test changes are accepted, they are added to the test suite with an indicator of errata/version. Old tests stay in the test suite marked with last errata/version level they comply with. The status changes to "CLOSED". If tests are rejected, they go back to the provider of the change with that determination, and the status is reverted back to "ACTIVE"

Checkpoint 8.3. In the Working Group's QA process document, identify the communication channel and procedure for appeals of tests validity. [Priority 2]

The following procedure is an example that meets the requirements:

• Initiating an appeal
• Review of the appeal
• Providing change to the test suite for review, in case of appeal acceptance
• Review and acceptance of the change

[Active issue] If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

• it may assigned to a Coordination Group;
• it may be assigned to a related Working Group;
• the Team could take care of minor items.

3.Working Group relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track. This section describes relationships that may exist between a Working Groups and the QA Working Group, as well as procedures that the Working Group and QA Working Group should follow. It is more informative than normative, and does not contain any guidelines and checkpoints.

Potential relationships between Working Groups and the QA Activity include:

• Liaison and consultation.
• Active reviews.
• Adjudicate entry and exit criteria.
• QA resources supplement.
• Resolution of external QA requests.

Key determinants of the Working Group/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

3.1Liaison and consultation.

In an assistive role, the QA Activity provides consultation to Working Groups in the planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to www-qa@w3.org. In order to make assistance quick and accurate the following process is defined for formal requests:

• A Working Group may submit to the QA Working Group questions or requests of the following types:
  • request for clarification on documents published by the QA Working Group
  • requests for review of the Working Group charter
  • requests for review of the Working Group specification
  • requests for review of the Working Group QA process document
  • requests for consultation on specific QA techniques and/or test suite architecture
  • requests for help on coordination with an external organization of joint QA related activity or test materials transfer
  Other requests will be considered by the QA Working Group on a case-by-case basis. They may be accepted, or may be considered out of scope and rejected (subject to appeal) by vote of the QA Working Group members.
• Within 1 week QA Working Group chair should assign the question to a task team of QA Working Group members and give an estimated time for resolution.
• The assigned QA Working Group task team should respond to the question (request) with appropriate resolution within the term allocated by the QA Working Group chair.

3.2Active reviews.

• Any member of the QA Working Group may participate in the review.
• The QA Working Group Chair appoints at least 2 reviewers.
• Reviewers verify that:
  • A Working Group QA process document is in place and addresses all checkpoints from QA Process section of this document.
  • Available test materials are produced consistently with best practices described in the QA Framework: Test Materials guidelines
  • For specification reviews, the specification conforms to QA Framework: Specification Guidelines
  • For charter reviews, the guidelines and checkpoints from the Charter considerations section are satisfied.

3.3Adjudicate entry and exit criteria.

3.4QA resources supplement.

3.5Resolution of external QA requests

• Escalates the request to corresponding the Working Group(s) and coordinates resolution if the matter involves more than one Working Group. TAG may be involved as well.
• If no active Working Group exists for the standard in question, the request is escalated according to maintenance guideline.

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

This section defines three levels of conformance to this specification:

• Conformance Level "A": all Priority 1 checkpoints are satisfied;
• Conformance Level "AA": all Priority 1 and Priority 2 checkpoints are satisfied;
• Conformance Level "AAA": all Priority 1, Priority 2, and Priority 3 checkpoints are satisfied;

A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

• The guidelines title: "QA Framework: Process & Operational Guidelines"
• The guidelines URI: http://www.w3.org/TR/2002/qaframe-ops/
• The conformance level satisfied: "Level A", "Level AA", or "Level AAA".

Example:

"This QA processes and operations of this Working Group conform to W3C's 'QA Framework: Process & Operational Guidelines', available at http://www.w3.org/TR/2002/qaframe-ops/, Level AA."

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

• Daniel Dardailler (W3C),
• Dimitris Dimitriadis (Ontologicon)
• Karl DuBost (W3C),
• Peter Fawcett (RealNetworks),
• Kirill Gavrylyuk (Microsoft),
• Dominique Hazael-Massieux (W3C)
• Lofton Henderson (CGM Open),
• Ian Jacobs (W3C)
• David Marston (Lotus Development Corp),
• Olivier Thereaux (W3C),
• Lynne Rosenthal (NIST).
• Mark Skall (NIST)
• Andrew Thackrah (Open Group)

6. References

EXTERN-TA

QA activity email thread about third-party participation in test materials production.

MATRIX

W3C-wide conformance activity survey covering all the Working Groups, "The Matrix".

PROCESS

W3C Process Document, 19 July 2001.

TAXONOMY

QA Activity test taxonomy, a classification scheme for conformance test materials.

QA-GLOSSARY

A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)

QAIG

Quality Assurance Interest Group of the W3C QA Activity.

QAWG

Quality Assurance Working Group of the W3C QA Activity.

DOM Working Group TS

Process document for DOM Working Group Test suite.

REC-TRACK

Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).

RFC2119

Key words for use in RFCs to Indicate Requirement Levels, March 1997.

SVGTEST

SVG Working Group's test suite resource page.

WCAG10

Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.

WG-QA-RANGE

Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production.

XMLTEST

OASIS XML Conformance TC's XML test suite resource page.

XSLT-TEST

OASIS XML Conformance TC's XSLT/Xpath test suite resource page.

QAF-TEST

QA Framework: Test Materials Guidelines (not yet published).

QAF-SPEC

QA Framework: Specification Guidelines (not yet published).