QA Framework: Introduction

W3C Working Group Note 12 September 2003

This version:: http://www.w3.org/TR/2003/NOTE-qaframe-intro-20030912/
Latest version:: http://www.w3.org/TR/qaframe-intro/
Previous version:: http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/

Editors:: Lofton Henderson (lofton@rockynet.com); Lynne Rosenthal (lsr@nist.gov)
Contributors:: See Acknowledgments.

Abstract

An introduction, roadmap, and primer are presented, for a common framework for enhancing the quality practices of the W3C Working Groups in the areas of specification editing, production of test materials, and coordination of QA efforts with internal and external groups. This is the first of the family of QA Framework family of documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Operational Guidelines, Specification Guidelines and Test Guidelines.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Working Group Note, made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version incorporates changes resulting from Last Call review. For more details about the revision history of this document, see "Change history."

The previous version of this Working Group Note was published as a Working Draft. Because it contains no conformance requirements and is a purely informative companion to the normative members of the QA Framework family, the QA Working Group resolved to publish henceforth as a Working Group Note. It will continue to be published concurrently with other specifications in the QA Framework family, specifically Operational Guidelines and Specification Guidelines.

The QA Working Group Patent Disclosure page contains details on known patents related to this document, in conformance with W3C policy requirements.

Comments on this document are welcome. You may email comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data.

Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Introduction

1.1. Scope and goals

The scope of this document is to introduce a common framework for promoting and facilitating the quality practices of the W3C Working Groups (WGs). It presents a roadmap and orientation for the Quality Assurance (QA) Activity and its QA Framework document family.

The QA Framework documents aim to provide W3C Working Groups with resources and tools for all phases and aspects of their quality and conformance activities, including:

planning and process setup;
writing better, more testable specifications;
coordination with internal and external groups;
and ultimately to building or acquiring conformance test materials.

QA is properly an integral part of the efforts of each W3C Working Group. This excerpt from the QA Working Group (QAWG) charter succinctly summarizes the point that has been made from the earliest W3C conformance papers:

QA is to be considered a 'natural overhead' of any WG.... QA will succeed only if every person inside W3C participates in it.

Foremost amongst the purposes of defining a common QA Framework is the principal goal of the QA Activity itself: to improve the quality of W3C standards' implementations in the field. For those undertaking quality assurance projects, the QA Framework should give:

a step-by-step guide to QA process and operational setup, as well as to development;
and, access to resources that the QA Activity makes available to assist the WGs in all aspects of their quality practices.

More specific goals include:

to encourage the employment of quality practices in the development of the specifications from their inception through their deployment;
to encourage development and use of a common, shared set of tools and methods for building test materials;
to foster a common look-and-feel for test suites, validation tools, harnesses, and results reporting.

1.2. About QA at W3C

The ongoing mission of W3C is Leading the Web to its full potential..., and W3C's many active working groups contribute towards this goal, in large part, by building a family of high-quality Web standards. These standards are the building blocks of an architecture to maximize the Web's potential.

Most would agree that the writing of the standard is not, in itself, the end purpose of the Working Groups (WGs). It is a means to achieve the real end: the success of the standardized technology in practice and practical application. One measure of the success that can be applied to most Web standards is a set of complete, correct, high-quality, interoperable implementations that actually realize the functionality of the standard.

In the past few years, several WGs have discovered that the early production of test materials (TM) has been a major contributor to the success of their standards. The benefits of test materials work are several-fold, but a couple of the big benefits are: early detection and correction of unintentionally vague or contradictory language in the technical reports (TRs) [W3C-TR]; and, a solid basis for the development of the implementation interoperability requirements of the W3C Process Document.

Better specifications are inherently more implementable;
and, better implementations are our ultimate goal.

The individual conformance test materials efforts of the Working Groups (WGs) have in several cases been inspired and on target, in others have been rudimentary or incomplete. Even amongst the best efforts, however, there is little uniformity in the quality-control principles, methodologies, or even such basics as terminology. Moreover, these efforts are distributed throughout W3C, making it difficult or at least time consuming for WGs pursuing their QA goals to find and take advantage of what has already been done. Each WG has started from scratch, researching the numerous existing TS activities and defining their own processes, operational framework and deliverables.

This defines the mission of our QA Activity. QA aims assist the WGs by assembling and organizing the best of a body of good practice, defining a framework, and providing a roadmap for the working groups. These best-practice guides should dramatically ease the job of planning and implementing QA projects within the Working Groups.

QA Framework Document Family Roadmap

2.1. Overview

The QA Framework document family includes:

QA Framework: Introduction (this document);
QA Framework: Operational Guidelines [QAF-OPS];
- QA Framework: Operational Examples & Techniques [OPS-EXTECH];
QA Framework: Specification Guidelines [QAF-SPEC];
- QA Framework: Specification Examples & Techniques [SPEC-EXTECH];
QA Framework: Test Guidelines [QAF-TEST];
- QA Framework: Test Examples and Techniques [TEST-EXTECH].

The QA Framework: Introduction (this document) is for everyone interested in QA within the W3C. It serves as a first read and a starter for most QA-related activities.

The QA Framework: Operational Guidelines [QAF-OPS] is written for people explicitly involved in QA activities, both those within the W3C Working Groups as well as others from organizations external to the W3C involved in developing test materials. The goal of this document is to present operational and process guidelines for groups undertaking conformance materials development or acquisition. The document contains information about process, intellectual property rights, logistical setup, resource considerations for staffing QA effort, and interaction between Working Groups, the QA Activity and external organizations.

The QA Framework: Specification Guidelines [QAF-SPEC] is written for people who are involved with either writing or reviewing the specifications of a Working Group, as well as those explicitly involved in QA activities. The goal of this document is to present guidelines for clearer, more implementable, and better testable technical reports.

The companion documents Operational Examples & Techniques [OPS-EXTECH] and Specification Examples & Techniques [SPEC-EXTECH] provide examples and pointers to existing QA work, illustrating the principles and guidelines set forth in their respective parts of the QA Framework.

The QA Framework: Test Guidelines [QAF-TEST] and its companion Test Examples & Techniques [TEST-EXTECH] are written for people who are actively involved with either building or assessing and acquiring conformance test materials. They provide detailed information for building test suites and test tools. They cover:

principles and requirements for general test suite design, test case content, and reporting;
principles and requirements for issue tracking systems in TS production, matrices of existing, expected and wanted tests and so forth;
tools used to produce conformance test suites;
technologies used in doing so (RDF, XSLT, DTD/Schema etc.);
building or using harnesses for running the TS and generating relevant results.

2.2. Target Audience -- the Working Groups

The following list gives the parts of the QA Framework that will interest those who fill the roles in a Working Group's (WG's) activities.

all WG participants: For any (potential) WG participant, the charter and QA-commitment parts of Operational Guidelines ([QAF-OPS], Guideline 1) should be helpful in understanding what the WG has committed to deliver. Familiarity with the Specification Guidelines [QAF-SPEC] will be helpful to any participant who works on the advancement of the WG's specifications to Recommendation.
WG spec editors and authors: As for all WG participants, the operational guidelines [QAF-OPS] are useful. A good working understanding of specification guidelines [QAF-SPEC] will be needed in order to satisfy the specification guidelines and checkpoints, and Specification Examples & Techniques [SPEC-EXTECH] should be a valuable resource in choosing document structure, formats, and techniques that will facilitate satisfying the requirements.
WG chair: As the person ultimately responsible for both the advancement of the WG's specifications and the WG's QA projects, a familiarity with the guidelines for operations and process [QAF-OPS], for specifications [QAF-SPEC], and for test materials [QAF-TEST] will be useful.
WG-TS participant: Those who are active in building the conformance test materials of the WG will need a good working understanding of both the guidelines for test materials [QAF-TEST] and Test Examples & Techniques [TEST-EXTECH]. Because of the close dependency of test materials on the functional specifications, a good familiarity with the specification guidelines [QAF-SPEC] as well as examples and techniques [SPEC-EXTECH] will also be helpful.
WG-TS moderator: The person who manages the WG's QA projects should have good working understandings of guidelines and techniques for specifications ([QAF-SPEC] and [SPEC-EXTECH]), as well as the guidelines and techniques for test materials ([QAF-TEST] and [TEST-EXTECH]). In addition, knowledge of all of the process and logistical setup guidelines [QAF-OPS] and techniques [OPS-EXTECH] are needed.
non-WG spec reviewers: Whether from other WGs, or the public at large, the specification guidelines [QAF-SPEC] will be helpful to those who review a WG's specifications.
non-WG TM reviewers: Whether from other WGs, or the public at large, reviewers of the test materials of a WG should be familiar with guidelines for test materials [QAF-TEST], and familiarity with the techniques [TEST-EXTECH] as well would facilitate a critical review.
Reviewers of Activity proposals & charters: For those W3C Members who will be reviewing Activity proposals and proposed WG charters, and helping to form their Advisory Committee Representative's positions, the commitment and deliverable requirements defined in the operational guidelines [QAF-OPS] are helpful.
QA Activity participants: Participants in the QAWG are an expert resource for the W3C Working Groups, and accordingly should be expert on all parts of the Framework; participants in the QA Interest Group (QAIG) need thorough familiarity with all parts as well, to effectively render some of the QAIG's chartered deliverables.

2.3. Structure and Content

Applicable Domain

Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:

Working Groups (WGs) at all stages of maturity, ranging from newly created (i.e., just chartered), to mature, to extended (i.e., re-chartered);
Specifications at all maturity levels and stages of document progression (i.e., Recommendation-track), ranging from first Working Draft through Recommendation, post-Recommendation errata processing and subsequent-edition publication;
Conformance test materials of all types (i.e., described in [TAXONOMY]), ranging from content validators to test suites and tools for products, interfaces, and APIs;
QA experience in developing test materials, ranging from minimal to significant;
Resources for developing test materials, ranging from insufficient staffing levels to staffing levels commensurate for the QA deliverables;
Venue for development of test materials, ranging from intra-WG development to combination of WG and external organization to external organization development of complete test material.

These framework documents should be applicable and useful throughout this diverse context.

Document Orientation and Structure

The QA Framework documents utilize structured guidelines and verifiable checkpoints similar to the W3C Web Accessibility Initiative (WAI) standards such as Web Content Accessibility Guidelines 1.0 [WCAG10]. The degree to which a simple cookbook guide is achievable is complicated by the factors enumerated previously -- diversity of potential test materials, as well as organizational diversity. Complexity and diversity will be managed in the QA Framework document family by:

stating the guidelines and checkpoints as much as possible with general, cross-taxonomy and all-WG applicability;
and, for each of the QA Framework's guideline documents, having an accompanying Examples & Techniques document.

The guideline documents are closely linked to their respective "Examples & Techniques" documents. The linked material explores, defines, and presents examples of taxonomy-dependent implementations of checkpoints. This model from the WAI standards has served well to manage diversity and complexity.

These checkpoint technique documents are not intended to be a comprehensive survey of existing W3C practice. That is the purpose of the QA Matrix [MATRIX]. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.

Normative Guidelines

The QA Framework guidelines -- or more precisely, the checkpoints associated with the guidelines -- contain the individual conformance requirements of these QA Framework documents. Each guidelines document contains a conformance chapter which unambiguously defines conformance to the guidelines and checkpoints.

The relationship of the QA Framework documents to the W3C Process is undefined at this time of publication. It is not precluded that all or parts of these guidelines may ultimately be incorporated (by reference) into the W3C Process Document [PROCESS].

Terminology

In the normative parts of the QA Framework guidelines documents, the keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119]. When used with their RFC2119 meanings, they will be all uppercase. Occurrences of these words in lowercase comprise normal prose usage, with no normative implications.

This document contains no normative requirements.

Unusual terms in the framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

Please consult the QA Glossary, [QA-GLOSSARY], for the definitions of terms used in this document.

QA Resources

A variety of resources are provided by the QA Activity, including:

The Matrix [MATRIX], which is a definitive survey of existing QA practice;
this QA Framework document family;
common QA Glossary [QA-GLOSSARY];
QA Taxonomy description [TAXONOMY];
Notes (white papers) on QA-related topics (see [QA-LIBRARY]);
the Issues List of the QAWG [ISSUES-LIST];
expertise and consultancy;
technical assets.

A comprehensive survey of the resources of QA Activity is beyond the scope of this document, and may be located on the QA Working Group [QAWG] and QA Interest Group [QAIG] Web pages. However the last two items above are pertinent to other parts of these Framework documents.

3.1. Expertise and Consultancy

The QA Working Group expects to have access to considerable expertise on a spectrum of QA topics, methods, and techniques. One key aim of the QA Activity is to make this expertise available to the Working Groups, at least on an as-needed (request) basis, as they plan and implement their conformance and quality projects. The degree of participation of QA experts in WG projects will likely be limited by QAWG staff resource constraints, but a consultancy role should almost always be possible.

The QA Framework document, QA Framework: Operational Guidelines [QAF-OPS], explores this topic in some detail, including:

the nature of QAWG relationships to other Working Groups (WGs);
level and degree of participation of QAWG experts in other WG projects;
and, means and mechanisms for managing WG requests for QA assistance.

3.2. Technical Assets

While the initial resources that the QA Working Group offers to other WGs focuses heavily on the Framework documents and available consulting expertise, it is intended that ongoing development within QA Activity will result in a collection of commonly useful assets. Some anticipated offerings include:

XML-based test case description languages (TCDL) that can be applied or adapted to diverse test suite projects;
scripts and transformation stylesheets to generate commonly needed TS materials, harnesses, and documents from TCDLs;
grammars and methods for granular construction of specifications, as well as tools and methods for management of testable assertions and even (in some cases) automatic generation of tests;
methods and tools for assessment of test coverage of specifications;
templates for setting up project TS process documents, ../Test/ Web sites, and other conformance project logistics;
standardized and routine procedures (including forms and scripts) for WGs to manage the process of (external) test case submission, review, and integration;
results reporting languages (e.g., EARL), plus methodologies and specific tools for working with them.

Most of these will be discussed in QA Framework: Test Examples & Techniques [TEST-EXTECH], to illustrate how to meet the QA Framework: Test Guidelines [QAF-TEST] with common tools.

QA Framework Primer

This chapter provides an introduction to the use of the QA Framework document family in the form of a primer. It will help Working Groups (WGs) get started on QA activities through references to guidelines and techniques. It examines significant milestones in a WG's activities, from writing a charter through publishing Recommendations, and building or acquiring test suites and tools.

4.1. First Step -- QA Commitment

Because QA is considered to be an integral part of the activities of each WG, each WG has to consider and commit to a set of QA deliverables. A spectrum of possible commitments, and associated guidelines, is found in the QA-commitment checkpoint(s) of the operational guidelines ([QAF-OPS], Checkpoint 1.1).

If a WG is being newly formed, then the WG charter needs to document its QA deliverables, just as it does all other WG deliverables. Again, see the QA-commitment checkpoint(s) of the operational guidelines ([QAF-OPS], Checkpoint 1.1). If a WG is being re-chartered, this case should be considered the same as a newly formed WG -- the charter should document the WG's committed set of conformance and test related deliverables.

For an already-chartered Working Group undertaking new QA projects, if these deliverables are not documented in the charter already, then the W3C Process Document [PROCESS] describes how the charter can be amended to accommodate significant new deliverables. Other methods of satisfying the commitment requirements are also described. Again, see the QA-commitment checkpoint(s) of the operational guidelines ([QAF-OPS], Checkpoint 1.1).

4.2. Set Up Processes and Logistics

Once the Working Group (WG) commitments to QA deliverables are made and documented, the next step is to set up the processes and logistics that the WG will use for managing its QA activities. These include, among other things:

QA moderator;
"/Test/" home page in the WG's Web space;
-ts email discussion list;
WG process document for QA;
decisions about IPR and test materials.

See the sections about staffing WG QA efforts ([QAF-OPS], Guideline 2) and producing the WG's QA process document ([QAF-OPS], Guideline 4) in the operational guidelines, about how to do this.

4.3. Planning and Writing the Specification

There is a very tight relationship between how the specification (Recommendation) is written on the one hand, and on the other hand its testability and its suitability as the foundation for interoperable implementations.

New specification. QA Framework: Specification Guidelines [QAF-SPEC] should be applied from very beginning. Among the key topics that it addresses are:

conformance policy & clause (Guideline 8);
normative language throughout (Guideline 7);
test assertions (Guideline 10);
optional features, discretion, extensions (Guidelines 5 & 6);
profiles, levels, conformance claims (Guidelines 3 & 9).

Consider the guidelines and checkpoints of QA Framework: Specification Guidelines [QAF-SPEC] even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" (the Member-only rules document for W3C editors who are publishing technical reports) and W3C Manual of Style [STYLE-MAN], document authors and editors should refer to the spec guidelines throughout, about testable language, clarity, conciseness.

New Edition of specification. A new Edition of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered a good time to bring a specification to QA Framework: Specification Guidelines conformance -- the requirements associated with achieving such conformance could significantly disrupt and restructure the specification.

New Version of specification. A new Version of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.

4.4. Reviewing and Progressing the Specification

This stage in the specifications life has two significant aspects:

the advancement of the specification to and beyond the first public working draft (FPWD);
and, the need to synchronize production of test suites and tools more closely with the spec progression.

When the specification is published in TR space, then non-WG W3C Members and the general public begin to review and comment. Such reviewers should consult and understand the material in Specification Guidelines [QAF-SPEC], in order to have an informed set of evaluation criteria for the conformance, testability, and interoperability aspects of the specification.

Working Group (WG) participants and especially WG-TS participants should refer to the sections (checkpoints) of the operational guidelines, regarding specification exit criteria ([QAF-OPS], checkpoint 1.5) and synchronization of the test materials with specification progression ([QAF-OPS], Guideline 3). The project enters The Matrix [MATRIX] at Last Call Working Draft (if not sooner). Additionally, a de-facto process convention is emerging, that there should be significant conformance test materials when exiting Candidate Recommendation and commencing Proposed Recommendation. This is the same timing as the explicit process requirement of two interoperable implementations.

4.5. Designing and Building Test Materials

There are several scenarios for how the Working Group (WG) "builds" its conformance test materials:

design and build test cases and test tools in the WG, from scratch;
import completed test materials from an outside group or organization;
assemble the test materials from contributed test cases and materials.

Intra-WG build. Before starting the development, the WG-TS participants should be thoroughly familiar with the material in Test Guidelines [QAF-TEST]. There is useful information for both high-level planning -- e.g., breadth-first Basic Effectivity versus fully detailed suite? -- as well as specific detail for the building of individual test cases. Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the review-procedures requirements ([QAF-OPS], checkpoint 5.4) of the operational guidelines.

Import completed test materials. Several high-quality test suites have been developed outside of the relevant W3C WG, and then transferred to the WG. WGs which are considering such a transfer should refer to the test materials transfer guideline ([QAF-OPS], Guideline 7) of the operational guidelines. Clearly, the quality of the candidate test materials should be carefully assessed, and for this the Test Guidelines [QAF-TEST] can provide useful assessment criteria.

Assemble contributions. Some test suites have been built by implementing processes to assemble significant chunks of material from outside (or internal member) contributions. Operational Guidelines [QAF-OPS] addresses the steps needed to complete such a transfer -- they are the same as the preceding paragraph about transferring completed test materials. In addition, there should be careful quality assessment of contributions, for which Test Guidelines [QAF-TEST] can be useful. Finally, there must be procedures for submission, review, and integration of contributions ([QAF-OPS], checkpoints 5.2, 5.4), which are described in several related checkpoints of the operational guidelines.

4.6. Publication of Test Materials

Typically, a Working Group TS group will want to publish releases of test materials, particularly as the specification advances through its final maturity levels (e.g., Proposed Recommendation) towards Recommendation. Test material publication is addressed in TM publication checkpoint ([QAF-OPS], checkpoint 6.3) of the operational guidelines.

As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions. This is addressed in a test appeal checkpoint ([QAF-OPS], checkpoint 8.3) of the operational guidelines.

Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Assemble Contributions" are equally applicable here.

4.7. Specification Publication and Beyond

When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned according to one of the mechanisms discussed above. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and validity or applicability of particular tests -- mechanisms for this are discussed in Test Guidelines [QAF-TEST], and the general process overview is discussed in the errata update checkpoint ([QAF-OPS], checkpoint 8.2) of the operational guidelines. Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if Working Group TS active development of test materials ceases, it may be desired to continue to consider submissions, and review and integrate them per the requirements of the test-contribution checkpoint ([QAF-OPS], checkpoint 8.1) of the operational guidelines.

4.8. Life after WG

It is possible that the Working Group (WG) and WG-TS may disband after its charter expires. This situation, and what to do about test materials, is discussed in a "secure repository" checkpoint ([QAF-OPS], checkpoint 6.1) of the operational guidelines.

Conformance

This introductory document to the QA Framework documents family is entirely informative. It contains no conformance requirements. Therefore the concept of conformance to this document is not applicable.

The guidelines documents of this Framework document family do contain conformance requirements, and each such document contains a conformance chapter with an unambiguous definition of conformance to the documents guidelines and checkpoints.

Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

Patrick Curran (Sun Microsystems)
Daniel Dardailler (W3C),
Dimitris Dimitriadis (Ontologicon)
Karl Dubost (W3C),
Peter Fawcett (RealNetworks),
Kirill Gavrylyuk (Microsoft),
Dominique Hazael-Massieux (W3C)
Lofton Henderson (CGM Open)
Ian Jacobs (W3C)
Susan Lesch (W3C)
David Marston (Lotus Development Corp)
Lynne Rosenthal (NIST)
Mark Skall (NIST)
Andrew Thackrah (Open Group)
Olivier Thereaux (W3C)

References

[ISSUES-LIST]: QA Activity Issues List, maintained by the QA Working Group, available at http://www.w3.org/QA/WG/#issues.
[MATRIX]: W3C-wide conformance activity survey covering all the WGs, The Matrix, available at http://www.w3.org/QA/TheMatrix.
[OPS-EXTECH]: QA Framework: Operational Examples & Techniques, L. Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Note, (initially) 12 September 2003, companion version to this document, latest 12-sept-companion version available at http://www.w3.org/QA/WG/2003/09/qaframe-ops-extech.
[PROCESS]: World Wide Web Consortium Process Document, I. Jacobs, Ed., 18 June 2003, available at http://www.w3.org/2003/06/Process-20030618/ .
[QA-GLOSSARY]: Comprehensive glossary of QA terminology. (Under construction at http://www.w3.org/QA/glossary.)
[QA-LIBRARY]: Comprehensive bibliography of documents, talks, Notes, etc., that are published by the QA Activity. Available at http://www.w3.org/QA/Library/.)
[QAF-OPS]: QA Framework: Operational Guidelines, L. Henderson, D. Hazaël-Massieux, L. Rosenthal, K. Gavrylyuk, Eds., W3C Candidate Recommendation, 12 September 2003, companion version to this document, available at http://www.w3.org/TR/2003/CR-qaframe-ops-20030912/.
[QAF-SPEC]: QA Framework: Specification Guidelines, D. Hazaël-Massieux, L. Henderson, L. Rosenthal, Eds., W3C Working Draft, 12 September 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/
[QAF-TEST]: QA Framework: Test Guidelines, P. Curran, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, P. Fawcett, Eds. W3C Working Draft, 16 May 2003, current latest companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-test-20030516/
[QAIG]: Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
[QAWG]: Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
[RFC2119]: Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
[SPEC-EXTECH]: QA Framework: Specification Examples & Techniques, K. Dubost et al, Eds., W3C Note, (initially) 12 September 2003, companion version to this document, latest 10-feb-companion version available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech.
[STYLE-MAN]: W3C Manual of Style, summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
[TAXONOMY]: QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
[TEST-EXTECH]: QA Framework: Test Examples & Techniques, not yet published.
[W3C-TR]: Location of all published W3C technical reports, see http://www.w3.org/TR/.
[WCAG10]: Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, and G. Vanderheiden, Eds, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.

Change history

2003-09-12, W3C Working Group Note (to accompany the CR of Operational Guidelines and Specification Guidelines).

Changed from Working Draft to "W3C Working Group Note".
Substantial rewrite, per suggestions from S.Lesch in Last Call issue #68.
Folded in the resolutions, where applicable, of Last Call issues LC-69, LC-71, LC-76, LC-111, and LC-112.

2003-02-10, Last Call Working Draft.

Editorial cleanup for Last Call

Add reference to QA Library.

2002-11-08, third published WD

Revised "Introduction" (now "Overview", and subsections parallel intro sections of the guidelines documents). Revised references to Examples & Techniques documents, per their new permanent status as W3C Note. Minor editoral changes throughout.

fix lots of incorrect text descriptions on links to (moved) Operational Guidelines checkpoints.
editorial changes to realign with re-phrasing of the checkpoints in Operational Guidelines.
clarify in "Terminology" that this document has no normative requirements.
revise "schemata" discussion in Specification Guidelines description, per 20021108 version.

2002-0515, second published WD

Table of contents