QA Framework: Specification Guidelines

W3C Working Draft 24 June 2002

This version:: http://www.w3.org/QA/WG/2002/framework-20020507/qaframe-spec
Latest version:: http://www.w3.org/QA/WG/qaframe-spec
Previous version:: http://www.w3.org/QA/WG/2002/framework-20020405/qaframe-spec
Editors:: Lynne Rosenthal (lynne.rosenthal@nist.gov); Dimitris Dimitriadis (dimitris@ontologicon.com); Kirill Gavrylyuk (kirillg@microsoft.com); Lofton Henderson (lofton@rockynet.com)
Contributors:: See Acknowledgments.

Abstract

The principal goal of this document is to help W3C Working Groups in writing clearer, more implementatable, and better testable technical reports. It both provides a common framework for specifying conformance requirements and definitions, and also addresses the representation of specifications (technical reports) as schemata, both of which facilitate the generation of test materials. The material is presented as a set of organizing guidelines and verifiable checkpoints. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Operational Guidelines; and, Test Guidelines.

Status of this document

This is an Editors draft. It is meant for QAWG Editors and members internal review only. Its contents have not been endorsed or approved by the QAWG, or by any other entity of W3C.

This is a major reordering and revision of the previous version, which is the FPWD version of 2002-05-15. The reordering and revision was done according to the email proposal (by DM), which was discussed and endorsed at the Montreal face-to-face. All guidelines (GL) are re-ordered accordingly. New GL.5 ("Conformance policy", formerly GL.3) is rewritten. Old GL.5 is split into new GL.3, GL.4, GL.7, and checkpoints are rewritten. The text in the checkpoints of the latter is rough, and there are numerous open issues. There is a new checkpoint 5.6, "Minimize variability."

The goal of this WG draft is to start to organize and get a handle on the dimensions of variability, as a basis for re-opening discussion on the IG list about how to maximize interoperability.

The following SOTD is from FPWD...

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first public Working Draft, and supersedes all previous WG-only drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

In this version, a complete set of guidelines and prioritized checkpoints for W3C specifications (technical reports) is presented and explained. At this time, the QAWG (QA Working Group) has addressed the contents of the document at a high level, agreeing to the concepts and principles as well the coverage of the guidelines. The QAWG has not, at this time, addressed and achieved consensus of the priority levels. A future version of this document will be accompanied by a "Specification Examples & Techniques" document, which will illustrate the guidelines and checkpoints with case studies, and explain how to satisfy the checkpoints.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

1. Introduction
    1.1 Motivation for this guidelines document
    1.2 Navigating through this document
    1.3 Terminology
    1.4 Major Themes
    1.5 Variability, complexity, and progression of guidelines
2. Guidelines
         G 1. Define user scenarios.
         G 2. Identify what needs to conform and how.
         G 3. Address the use of profiles to divide the specification.
         G 4. Address the use of modules to divide the specification.
         G 5. Specify conformance policy.
         G 6. Clarify the relation between deprecated features and conformance.
         G 7. Address the use of levels to divide the specification.
         G 8. Define discretionary behaviors.
         G 9. Allow extensions or NOT!
         G 10. Provide a conformance clause.
         G 11. Specify how to make conformance claims.
         G 12. Publish an Implementation Conformance Statement proforma.
         G 13. Support general document conformance conventions.
         G 14. Use granular grammars to author the specification.
         G 15. Include test assertions.
3. Conformance
    3.1 Motivation for this Guidelines Document
    3.2 Conformance disclaimer
4. Acknowledgments
5. References (non-normative)
6. Change history

An appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.

1. Introduction

The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbaised manner.

This document defines a common framework for specifying conformance requirements and definitions, and for representing the structure of the document as schemata, both of which facilitate the generation of test materials. The primary goal is to assist W3C Working Groups (WGs) by providing guidelines and verifiable checkpoints for writing clearer, more implementable, and better testable specifications (technical reports). Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.

These guidelines were developed so that WGs can apply them in a common-sense and workable manner. This document identifies the conformance requirements and statements to be included or addressed in specifications as well as addressing the anatomy of the specification. Conformance requirements are the expressions that convey the criteria to be fulfilled in an implementation of a specification. The conformance requirements are stated in a conformance clause or statements within the specification. The anatomy of the specification pertains to the method for writing the specification using schemata. A specification represented by an XML grammar with sufficient granularity of its information facilitates the generation of test materials by providing the ability to point to, extract, query, manipulate and/or automatically generate test materials. Given the symbiotic connection between specification and test materials, this document also addresses the intermixed quality practices of specification authoring and test material production and maintenance within the W3C process.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementations by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

QA Framework: Introduction [QAF-INTRO ]

QA Framework: Operational Guidelines [QAF-OPS ]

QA Framework: Specification Guidelines (this document)

QA Framework: Test Guidelines [QAF-TEST ]

The process for developing testable technical reports and specifications is affected by QA activities beyond those that are explicitly provided in this document. Specifically, the QA Framework documents are interrelated and complement each other. For example, the anatomy of a specification is dependent on the type of test materials that will be built - hence there is interrelationship between this document and the Test Guidelines. Links between applicable guidelines in this document and the other Framework documents will be given.

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

1.1 Motivation for this guidelines document

The ultimate goal of a specification is to be implemented. In order to be implemented, a specification needs to be written such that developers and users can comprehend what is expected of them and build correct, robust, interoperable software and content. Correct utilization of specifications leads to portability and interoperability. In particular, it is easier to implement clear specifications, where the possibility of misinterpretation has been eliminated, safeguarding the quality of implementations. In addition, if care is given to the specifications, interpretability between W3C technologies is easier to track and assert.

Given that the W3C invests time and resources into producing specifications that are eventually implemented, and especially given that some of those specifications are interrelated, it makes sense to enhance and streamline the specification authoring process. This should be done for three main reasons:

More straightforward specifications are easier to understand and implement
They foster the development of conformance materials
It is easier to construct W3C-generic testing frameworks if both the test framework and what is being tested, i.e. implementations and specifications, follow generally accepted and implemented rules.

The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable requirements and that can be used to easily generate Quality Assurance materials (tests and suites). A specification is testable if there exists some finite cost-effective process with which a person or software can check that the requirement has been met.

The benefits of producing clear, testable specifications include:

Straightforward to assess and assert conformance
Easy to generate tests and test suites
Given that specification authoring is standardized, it will be straightforward to test interdependencies between specifications (for example it is easier to point to relevant parts of other specifications with regard to technical details where specifications share common features) as well as test interoperability between implementations of different W3C specifications.

The benefits derived by writing granular specifications that provide detailed control over the information in the specification, include

Information extraction directly from the specification
Generating a test harness directly from specifications
Traceability
Generating graphical representations of and interfaces to specifications
Efficient and effective test development

In particular, information from the specifications is easier to extract if specification authoring involves a standardized set of specification schemas where both technology and intended behaviour is specified. Tests can be generated directly from the specifications, where needed, to generate primers for test suites. In this way, a fairly broad coverage of test cases can be achieved in a straightforward way. An added bonus is that a graphical representation of the specification can be generated in parallel to the normative HTML version. Specification granularity allows for more concise reporting when running test suites by pointing directly to the relevant part of the specification. Specification granularity allows for generating tests that can be validated against the specification itself, thus making it easier in those cases where different interpretations of the normative text are given.

Use of a formal (or formal-like) specification language may enable 'validation' of the' specification as well as auto generation of tests or test components.

Synchronizing writing specifications with building test materials provides a feedback loop. Testing can help find ambiguities, inconsistencies, and holes in a specification.

In particular

Ambiguities and misinterpretations are more easily caught and can be resolved earlier in the specification writing process
Given the possibility to automatically generate primers for test suites, coverage can be tracked fairly easily and therefore enhanced. Corner cases are easy to identify and write tests for.

Interaction between WGs and other bodies in specification authoring is related to test suite production and maintenance. The reasons to develop specifications and test assertions/test materials together are:

help to uncover inconsistencies, ambiguities, gaps, and non-testable statements in the specification by developing the test assertions in parallel with the specification.
ensure consistency between the specification and assertions,
allow test assertions to be reviewed and accepted by the specifications developers and the public,
provide a common set of assertions from which test developers can develop test materials
encourage the early development of conformance tests that can be used by implementers during the development of their implementation,
achieve confidence in the resulting test materials as a measure of conformance.

1.2 Navigating through this document

Since this document is part of a family of documents, the reader is strongly encouraged to be familiar with the other documents in the family. There is an interrelationship between the Guidelines in this document and the Guidelines in the other Framework documents. In particular, Guidelines 1, 2, 3 and 5 in the QA Framework: Operational Guidelines [QAF-OPS ] are especially relevant and interrelated to the Guidelines in this document.

This document describes what goes into the specification with respect to conformance and conformance topics, followed by the anatomy of the specification to enable test development as well as better, more testable specifications. It does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES ] (member-only).

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:

The guideline number
The statement of the guideline
The rationale behind the guideline
A list of checkpoint definitions.

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

The checkpoint number
The statement of the checkpoint
The priority of the checkpoint.
Optional informative notes, clarifying examples, and cross references to related guidelines or checkpoints.

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

A separate appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.

1.3 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

1.4 Major Themes

The guidelines address two general themes: specifying conformance requirements in the specification; and addressing the anatomy of the specification.

Theme 1, Conformance content: general requirements and definitions concerning conformance and related issues. It is intended to contribute towards mutual understanding amongst developers of specifications and conformance test suites and tools. It does not define specific conformance requirements for any specific specification - this is the responsibility of the WG chartered to develop the specification. For the developers of specifications, it helps them to develop conformance requirements and to create testable, unambiguous specifications. A conformance clause:

promotes a common understanding of conformance and what is required to claim conformance to a specification
facilitates consistent application of conformance within a specification
facilitates consistent application of conformance across related specifications,
promotes interoperability and open interchange,
encourages the use of applicable conformance test suites,
promotes uniformity in the development of conformance test suites

Theme 2, Anatomy: invest in writing granular specification with careful modeling of the information conveyed. This allows for:

detailed control over information extraction from the spec
generating tests directly from the specs
easy graphic interface to specification

1.5 Variability, complexity, and progression of guidelines

In dealing with Theme 1, considerable complexity is introduced by a basic reality of diverse W3C specifications: multiple dimensions of variability can be accommodated in a specification. The WG may determine that conformant products are allowed to vary in one or more ways. This document recognizes the following dimensions:

product classes (Guideline 2)
profiles (Guideline 3)
modules (Guideline 4)
conformance policy (Guideline 5)
deprecation (Guideline 6)
levels (Guideline 7)
discretionary choices (Guideline 8)
extensability (Guideline 9)

The above are not necessarily all orthogonal to one another; possible associations are noted in the individual guidelines (@@Ed note. Not yet, at least not very extensively).

There are 15 guidelines that are presented in a sequence that runs approximately from the general to the specific. The later (higher-numbered) guidelines are likely to depend upon decisions made in accordance with earlier guidelines.

The first four guide the establishment of strategic objectives and address the overall motivation for the creation of the specification. The next two bring the QA concerns into the plan. Guidelines 7 through 9 are about additional specification tools to adjust the boundaries between conformant and non-conformant implementations. The final six guidelines pertain to particular documentation features.

Guidelines 2 through 9 cover the eight "dimensions of variability" that a specification may permit.

Scope and use cases (often in Requirements Doc)
Specification categories and product classes
Profiles (hardware, environment, etc.)
Modules
Conformance policy
Deprecation policy
Levels
Discretionary items
Extensibility
Conformance clause
Claims of conformance
Proforma
Document conventions
Granular grammar
Test assertions

2. Guidelines

Guideline 1. Define user scenarios.

When writing specifications it is critical to understand their primary purpose and scope. If the specification describes a protocol or Application Programmer Interface (API) -- examples XML Protocol, DOM-- there should be a clear understanding of the primary user scenarios. If the specification describes content requirements (for example, [WCAG10], [UAAG10]), understanding of their purpose is the key to defining the minimal sufficient set. Clearly defined scope helps to keep the specification content focused and unambiguous. When reading the specification, we face similar difficulties. To understand what the document says, the reader needs to know the context of the author, and the scenarios the author had in mind. In case of protocols and APIs specifications developers try to assess whether the specifications covers the user scenarios the product needs to cover. Having no user scenarios described in the specification generates misuses of the spec itself.

Checkpoint 1.1. Define the scope of the specification. [Priority 1]

This is very basic but powerful requirement. The specification must clearly define what scenarios are in scope and what are out of scope in order to be interpreted, implemented, and tested correctly.

Checkpoint 1.2. Include Use Cases. [Priority 2]

A Use Case is a description of the user scenario given in formally defined terms. Once included in the specification Use Cases become Normative, unless otherwise specified. The specification should have extensive list of the orthogonal Use Cases the authors have in mind. Priorities MAY be assigned to the Use Cases, describing how important the particular scenario is for the specification. Use Cases in their turn may help to assess what features are missing and what features are superfluous in the specification.

Checkpoint 1.3. Include examples. [Priority 1]

It is good practice to illustrate each behavior or requirement in the specification by short and precise examples.

Checkpoint 1.4. Include an interpretation section. [Priority 2]

It is hard to understand the formal descriptions content without the informative interpretation. The recent complex specifications like XML Schema [XML-SCHEMA] and XML Protocol have shown an absolute necessity to have a "Primer" part or section to illustrate how to use the specification. Such a section does not substitute for the numerous examples needed for each requirement or behavior described in the specification, but illustrates how the specification fulfills the User Scenarios.

Guideline 2. Identify what needs to conform and how.

Catagorizing the specification provides a basis for classifying the software that may be affected by the specification - and thus, provides the answer to 'what needs to conform?'. To help answer this question, it helps to look at the nature of the specification and categorize it. Most specifications can be classified into one of the following catagories:

foundation or abstract (e.g., Infoset),
content/data (e.g., MathML, SVG),
protocols (e.g., SOAP),
processors (e.g., XSLT, XML Query),
APIs (e.g., DOM),
notation/syntax (e.g., XPath),
set of events (e.g., one part of XForms).

One specification could potentially fall into more than one catagory.

From this categorization of specifications, the WG can identify the "class of products" that are affected by the specification. Classes of products can be generalized as either producers or consumers. Identifying which are producers and consumers is very clear for a protocol-type specification, the two parties to the dialog are the targets. For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification. For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way.

The following is a non-exhaustive list of classes of products for W3C specifications.

producer of content (may be divided into initiators and modifiers)
player (read-only consumer, conveys content in non-XML way)
consumer in a one-way pipeline
server/responding agent/implementer of API (consumer and producer)
processor (consumer of its vocabulary/instructions)
module that hosts the processor
producer of instructions/commands to processor

One specification could define more than one player. For example, MathML could address the behavior of display of math notation and also a consumer that parses the MathML as a formula and uses it for mathematical processing.

The conformance clause identifies the "class of products" (i.e., object of the claim) that is the target of the specification. In addition to indentifying what conforms (i.e., class of products), it describes how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.

Checkpoints

Checkpoint 2.1. Identify all classes of product. [Priority 1]

For example, within the SMIL 2.0 Language Profile ([SMIL20], chapter 13), there are 2 classes of products: documents and basic user agents and within Mathematical Markup Language (MathML) 2.0 [MATHML20] there are output-compliant, authoring tools and input-compliant, rendering/reading tools. For specifications that are used as components of other specifications and rely on these other specifications to define conformance, indicate that conformance criteria for independent implementations are not defined within the specification. For example, Xpath ( XML Path Language 1.0 , [XPATH10]) relies on specifications that use XPath -- such as XPointer ( XML Pointer Language , [XPOINTER]) and XSLT ( XSL Transformations 1.0 , [XSLT10]) -- to specify criteria for conformance of implementations of XPath.

Checkpoint 2.2. For each class of product, define the conformance requirements. [Priority 1]

The conformance requirements indicate the conditions to be satisfied in order to claim conformance. In addition to specifying the requirements for claiming conformance, it may be appropriate to specify that which is not a requirement. It is likely that these conformance definitions will reference normative text within the specification or in other related specifications.

Checkpoint 2.3. Indicate minimal support requirements. [Priority 3]

Describe the minimum set of elements, attributes, etc., that are required to claim conformance for a specific module and/or class of product. This information is often presented in tabular form (e.g., SMIL 2.0 section 2.4, table 5). If applicable, indicate that there are no minimum support requirements. This is helpful in providing the reader a full understanding of the conformance requirements, especially when some of the classes of products have minimal requirements whereas others do not.

Guideline 3. Address the use of profiles to divide the specification.

Dividing a specification into functional groupings can facilitate its implementation by enabling a developer to build to portions of the specification rather than implementing the entire specification. Some of the dimensions of variability, but not all, can be used to divide the specification: product classes, profiles, modules, and (to some extent) levels divide the specification, while extensibility separates specified behavior from other product features.

A profile is a subset of the specification that supports a particular functional objective. Profiles can also be used to group a set of standards and define how they are required to operate together. Profiles may be based on hardware considerations associated with target product classes (e.g., SVG Tiny is aimed at mobile phones), or they may be driven by other functional requirements of their target constituencies. The union of all the profiles does not necessarily constitute the entire specification (@@is this a useful statement?).

Modules are independent divisions or functional groupings of the specification, that can be implemented independently (e.g., audio vs. video module). See Guideline 4 for the full discussion of the topic. A profile can gather several modules.

Levels are used to group functionality into nested subsets. See Guideline 7 for the full discussion of the topic. A profile can require a particular level. Furthermore, profiles can be hierarchical in the sense that Profile B can be defined as all of Profile A plus additional capabilities.

The SMIL Recommendations (e.g., [SMIL20]) illustrate these concepts. Modularization is an approach in which markup functionality is specified as a set of modules that contain semantically-related XML elements, attributes and attribute values. Profiling is a method for defining subsets of a specification by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. In SMIL, it is the creation of an XML-based language through combining these modules, in order to provide the functionality required by a particular application. SMIL 2.0 defines all modules, levels and profiles -- three modules (timing, ...), levels (regular, extended animation), and profiles are defined at the end of the specification.

Exercise caution - having too much variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it.

Checkpoints

Checkpoint 3.1. Choose whether or not to have profiles. [Priority 1]

It is possible for a specification to have one or more of profiles, modules, levels, or none of these. If profiles are defined and supported in the specification, explicitly document it. Otherwise, explicitly state that profiles are not supported.

Checkpoint 3.2. If profiles are chosen, ensure that a table of contents entry is generated. [Priority 1]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Checkpoint 3.3. If profiles are chosen, indicate any mandatory conditions or constraints on usage . [Priority 1]

[@@Ed. note: was "...whether their use is mandatory". Does this lose any intended meaning of original checkpoint?] If there are additional conditions associated with a particular profile, they must also be described. For example: if one and only one profile can be implemented at a time.

Checkpoint 3.4. If profiles are chosen, define any minimal requirements. [Priority 2]

[@@Ed note. Can this fold into previous ckpt now?]

Checkpoint 3.5. If profiles are selected, define their relationships and interaction with other dimensions of variability. [Priority 2]

[@@Ed note. Does this capture the intended sense of the original checkpoint?]

Often there is a kind of layered dependency or interrelationship among profiles, on the one hand, and modules and/or levels on the other.

[@@Ed note. Original wording was... "Consider whether or not a claim of conformance to a particular module/profile/level can include functionality or features of another module/profile/level. For example, can implementations that purport to conform to a specific module include functionality defined in another module?"@@]

Guideline 4. Address the use of modules to divide the specification.

Modules are non-hierarchical, discrete divisions or functional groupings of the specification, that can be implemented independently (e.g., audio vs. video module). It is possible and common to implement multiple modules. The union of all the modules does not necessarily equal the entire specification.

See Guideline 3 for full discussion of profiles, and Guideline 7 for full discussion of levels.

Checkpoints

Checkpoint 4.1. Choose whether or not to have modules. [Priority 1]

It is possible for a specification to have profiles, modules, levels, or none of these. If modules are defined and supported in the specification, explicitly document it. Otherwise, explicitly state that modules are not supported.

Checkpoint 4.2. If modules are chosen, ensure that a table of contents entry is generated. [Priority 1]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Checkpoint 4.3. If modules are chosen, indicate any mandatory conditions or constraints on usage. [Priority 1]

[@@Ed. note: was "...whether their use is mandatory". Does this lose any intended meaning of original checkpoint?] If there are additional conditions associated with a particular module, they must also be described. For example: any restrictions or constraints on the number or types of modules that can be implemented, or if specific modules can not be used in combination.

Checkpoint 4.4. If modules are chosen, define any minimal requirements. [Priority 2]

[@@Ed note. Can this fold into previous ckpt now?]

Checkpoint 4.5. If modules are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

[@@Ed note. Does this capture the intended sense of the original checkpoint?]

Often there is a kind of layered dependency or interrelationship among modules, on the one hand, and profiles and/or levels on the other.

Guideline 5. Specify conformance policy.

A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and where the product is allowed to have more or less functionality than specified. If the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Often, the specification will allow discretionary choices, such as the choice of one or more natural languages for verbal content and messages, but require a conforming product to make a choice only within the allowable range. (See Guideline 8 for more discussion.)

Sometimes a product developer can choose to implement certain modules. There may be per-module conformance requirements that apply if and only if the developer chooses to implement a particular module.

Where the products must be exactly alike, it should be clear that a "strict conformance" policy is in effect. Strict conformance is defined as conformance of an implementation that employs only the requirements of the specification and no more (e.g., no extensions).

Sometimes a product developer can choose to implement extensions. There may be conformance requirements for non-interference of extensions. (See Guideline 9 for more discussion.)

Overall, the intent of the WG should be clear. In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and possibly provide examples.

Checkpoint 5.1. Make it clear where there are universal requirements for minimum functionality. [Priority 1]

The reader should be able to recognize a minimum that applies to all conforming products. If levels are used (see Guideline 7), the lowest level is the minimum set of requirements. If profiles or modules are used, there may be different minima for each one.

Checkpoint 5.2. Make it clear when conformance requirements are strict. [Priority 1]

The reader should be able to recognize when a policy of "strict conformance" applies. As defined above, this implies that all conformant products behave the same way. If profiles or modules are used, each may have its own conformance boundaries.

Either use the definition provided above (or in the QA Glossary [QA-GLOSSARY]), modify it, or provide your own definition. It is strongly recommended that the definition given above be used to ensure consistency across WGs and promote a common understanding of conformance requirements.

Checkpoint 5.3. Make it clear where requirements stop and product-specific extra features begin. [Priority 1]

The reader should be able to recognize conformance requirements as distinct from allowable extra functionality. If profiles are used (see Guideline 3), make it clear whether extra capabilities of the platform may be exploited. If modules are used (see Guideline 4), there may be different provisions for extra features applying to each. If deprecation applies (see Guideline 6), make it clear whether support of the obsolete features is optional, part of a level, or required. If levels are used (see Guideline 7), make it clear whether the highest level may be exceeded with additional features or functionality. If discretionary choices are allowed (see Guideline 8), make it clear if more than one may be implemented, when it is technically possible to do so.

Checkpoint 5.4. The conformance clause should contain or refer to the conformance policy. [Priority 1]

The reader should be able to easily identify and locate this information. See Guideline 10 for further discussion of conformance clauses. It should be possible for the reader to start at the table of contents and find all aspects of the conformance requirements, including the overall policy.

Checkpoint 5.5. If special conformance terms are used, include a definition in the specification. [Priority 1]

It is necessary to define terms that govern application of the conformance provisions. Ideally, all terms are from QA documents and other existing literature and need only be cited. If special terms are constructed, such as to combine modules and levels or modules and discretionary choices, these terms should be defined in the specification. The conformance clause should reference all locally-defined terms that affect conformance.

Checkpoint 5.6. Minimize variability. [Priority 1]

Excessive variability (dimensions of variability) is inimical to interoperability.

Application in a specification of any particular dimension of variability should be documented and justified by reference to the project requirements, use cases, etc.

[@@Ed note. Original proposals were for: "Don't exceed 3 dimensions in a first-edition specification, 4 dimensions in ...". "Index of variability based on weighted combination of dimensions ..."]

Guideline 6. Clarify the relation between deprecated features and conformance.

After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. A deprecated feature is a feature whose use is discouraged because it has been outdated by newer constructs or is no longer viable. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.

Checkpoint 6.1. Identify and clearly indicate each deprecated feature. [Priority 1]

Checkpoint 6.2. For each class of product, specify the level of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]

Define what it means for a feature to be deprecated and how this affects conformance. For example, a deprecated-features section of MathML 2.0 ([MATHML20], section 7.2.1.2) describes, about deprecated MathML 1.x features, that MathML-output-compliant authoring tools may not generate MathML markup containing deprecated features; whereas MathML-input-compliant rendering/reading tools must support deprecated features.

Checkpoint 6.3. Include an explanation for the deprecation. [Priority 3]

Providing the rationale for deprecating a feature helps implementers and users to understand the motivation for the deprecation, the impact and consequences on current and future implementations, and perhaps insight into its eventual disappearance from the specification.

Checkpoint 6.4. Include examples to illustrate how to avoid using deprecated features. [Priority 3]

Examples are helpful in providing alternatives or better ways to get the same results. By showing what can be done in place of the deprecated feature will help to get implementers to discontinue use of the deprecated feature.

Checkpoint 6.5. Generate a table of contents entry. [Priority 2]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Guideline 7. Address the use of levels to divide the specification.

Levels are used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionally. Typically, Level 1 is the minimal or core of the specification. Level 2 includes all of level 1 and also additional functionality. This nesting continues until level n, which consists of the entire specification. Levels may result from progressive historical development and enrichment of the specification, as in the case of CSS and DOM. Levels may also be defined explicitly in a single edition of the specification, including the first edition or version.

See Guideline 3 for full discussion of profiles, and Guideline 4 for a full discussion of modules.

Checkpoints

Checkpoint 7.1. Choose whether or not to have levels. [Priority 1]

It is possible for a specification to have profiles, modules, levels, or none of these. If levels are defined and supported in the specification, explicitly document it. Otherwise, explicitly state that levels are not supported. Levels may be defined explicitly in a single edition of the specification, or they may be planned as a mechanism associated with future, richer editions of the specification.

Checkpoint 7.2. If levels are chosen, ensure that a table of contents entry is generated. [Priority 1]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Checkpoint 7.3. If levels are chosen, indicate any mandatory conditions or constraints on usage. [Priority 1]

@@If there are additional conditions associated with a particular level, they should also be described. For example: any restrictions or constraints on the number or types of levels that can be implemented, if one and only one module/profile/level can be implemented at a time, or if specific @@levels can not be used in combination.@@

Checkpoint 7.4. If levels are chosen, define any minimal requirements. [Priority 2]

[@@Ed note. Can this fold into previous ckpt now?]

Checkpoint 7.5. If levels are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

[@@Ed note. Does this capture the intended sense of the original checkpoint?]

Guideline 8. Define discretionary behaviors.

Discretionary choices are often made available in specifications, giving vendors of technology the opportunities to decide from alternatives when implementating applications and tools. Discretionary behavior is when a specification deliberately and explicity grants discretion to the implementation - that is, describe or allows optionality of behavior, functionality, parameter values, error handling, etc. Discretionary items may be warranted because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc. Discretionary items may be enumerated choices, where the implementation must choose from a set of prescribed choices, or it may be describe open-ended discretion, such as locales.

One type of discretionary item is implementation dependent values. Implementation dependent values are used when it is not possible to define the behavior or values of a function. Implementation dependent means that an implementation may determine the effect, rather than having the effect mandated by the specification. Details in a specification may deliberately be omitted (i.e., not specified), so as to provide freedom to adapt implementations to different environments and different requirements. In general this is not a recommended practice.

Checkpoints

Checkpoint 8.1. Explicitly state the cases and conditions where discretion is allowed and/or expected. [Priority 2]

Although there may be many individual discretionary choices sprinkled throughout the specification, they often arise from broader ideas about how the specified software should operate. To help readers, implementers and testers to find these discretionary items, it is recommended that where ever possible these items be collected and presented as a group. For example, in XSLT [XSLT10] most discretionary items offer two choices, representing design philosophies to escalate an error or to continue processing. Thus, XSLT could address discretion as a choice between two groups of items.

Checkpoint 8.2. Indicate implementation dependencies and where applicable address, allowable differences between implementations. [Priority 1]

Examples of allowable differences to be addressed include:

implementation dependent ranges, data, minimum or maximum values,
environmental resources (e.g., memory or disk limitations),
environmental values (i.e., language and local settings).

Checkpoint 8.3. Describe alternative approaches and the conditions under which an implementation is considered to be conforming. [Priority 1]

Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats, protocols, or encodings). In such a case, enumerate the approaches and specify if there are limitations on the number of approaches or combination of approaches that can be implemented. Some possible ways to define conformance when allowing alternative approaches include mandating that an implementation:

implement only one approach,
implement every approach,
be allowed to implement none of the approaches.

Note that if the specification does not include the different approaches, this becomes an implementation detail.

Checkpoint 8.4. Include a statement regarding consistent handling of a discretionary item within an implementation. [Priority 2]

The effect of a discretionary item should be consistent within a single implementation. For example, a browser's rendering of a XSL-FO (XSL Formatting Object) should be the same for every invocation regardless of the document instance.

Checkpoint 8.5. Generate a table of contents entry. [Priority 2]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Guideline 9. Allow extensions or NOT!

An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Care should be exercised in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed.

Extensions may be private (often vendor specific) or may be public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.

Specifications allow extensions for various reasons. Extensions allow implementers to include features that are in demand by their customers. Also, extensions, often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods. For example, a specification that allows private extensions (e.g., proprietary) is more likely to impede interoperability that a specification than requires extensions to be registered.

Checkpoints

Checkpoint 9.1. If extensions are disallowed, explicitly state it. [Priority 3]

If extensions are not allowed, then make sure it is clear to the reader that not only are extensions not allowed, but the circumstances under which they are not allowed. The implementations of the specification precisely implement the specification. This is strict conformance. Strict conformance is often imposed on applications or content (e.g., a software program or document instance). This prohibition of extensions could be applied to a specific profile or module, rather than to the entire specification.

Checkpoint 9.2. If extensions are allowed, explicitly state it. [Priority 1]

State the conditions under which extensions are allowed, the applicability of the extensions, their effect on conformance claims, and any limitations or restrictions on the use of the extension.

Checkpoint 9.3. If extensions are allowed, make it clear that the extensions do not negate support for required functionality. [Priority 1]

An extension does not change the fact that an implementation needs to support all required functionality in the specifications exactly as specified; nor does it cause the non-conformance of functionality defined in the specification. The specification can include statements such as:

Each implementation shall fully support all required functionality of the specification exactly as specified.
Extensions shall not re-define semantics for existing functions.
The use of extensions shall not contradict nor cause the non-conformance of functionality defined in the specification.

Checkpoint 9.4. If extensions are allowed, use a standard mechanism to define the extension. [Priority 3]

One mechanism to allow extensions within a specification is to provide a standard way of defining the extension or a "standard way of being non-standard". This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate actions (i.e., to ignore or to implement). The nature of the extension dictates the method for defining the extension. It may be possible to define a generic function or mechanism that indicates external functionality. This external function may take the form of an escape or control character or may be an identifier, the presence of indicates that an extension follows. Another method, especially when extending a list of numeric parameters is to use a scheme where positive values represent standardized values and negative values are reserved for private extensions.

Checkpoint 9.5. If extensions are allowed, register or publish them. [Priority 3]

Registration is a procedure that allows extensions to be acknowledged and made available to the public. Registration provides for a degree of rigor and technical review for any proposed extension. Typically the WG would be responsible for processing the registration of an extension, thus ensuring adequate quality of a proposed extension and a technical description sufficient to be uniformly implementable. Often registered extensions may migrate into a later version of the specification.

Checkpoint 9.6. If extensions are allowed, require that implementations include a way to operate without the extension. [Priority 3]

If an implementation contains extensions, require it to have a mode under which the implementation can be directed to produce only conforming files (documents) or to operate in a strictly conforming manner.

Checkpoint 9.7. Generate a table of contents entry. [Priority 2]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Guideline 10. Provide a conformance clause.

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.

Checkpoint 10.1. Include a conformance clause. [Priority 1]

Checkpoint 10.2. Create a separate conformance section. [Priority 2]

Having the conformance clause exist as a separate section within the specification makes it is clearly identifiable, allowing a reader to find all conformance provisions from a single starting point.

Checkpoint 10.3. Generate a conformance clause entry in the table of contents. [Priority 2]

Make normative reference to specifications on which the current specification depends.

Often a specification is dependent on other specifications or portions of specifications. For example requiring that the class of product called "user agent" be consistent with the XML 1.0 Recommendation [XML10 ] and support Cascading Style Sheets, level 1 [CSS1]. The inclusion or reference to related specifications can affect conformance of the current specification. To ensure clarity and understanding of these implications, provide a description of the relationship between the specifications and any conformance implications. Linking from the prose to the reference is described in the linking parts of the Manual of Style ([STYLE-MAN], section 11.5.1).

Guideline 11. Specify how to make conformance claims.

A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The WG includes in the specification the way they want people to claim their conformance.

Checkpoints

Checkpoint 11.1. Identify and define all conformance levels or designations. [Priority 1]

The naming convention used to label the conformance can provide useful information, such as imply incremental importance or difficulty. Functional specifications use functionality to differentiate between conformance levels - the lowest level of conformance is defined by a core set or minimally supported features, the next level of conformance would include the core plus additionally features, etc. until the entire specification was covered. For example, the modularization description of SVG 1.1 (Scalable Vector Graphics 1.1, [SVG11], section 1.1.2) identifies three conformance levels (i.e., basic, tiny, and full) where each level subsumes the lower level, providing additional functionality and/or complexity. In contrast, the WAI Guidelines specifications (e.g., Web Content Accessibility Guidelines 1.0 [WCAG10]) and the QA Framework Guidelines specifications (e.g., this document) use a 3-level conformance format (A, AA, AAA) to convey priorities based on a checkpoint's assessment.

Checkpoint 11.2. Provide specific wording of the claim. [Priority 3]

A well-formed conformance claim includes: date of the claim, specification name, date and version, URI of the specification, conformance level satisfied, and information about the subject (that which is claiming conformance). Information about the subject refers to information such as, the name of the software or software component, version information, and operating environment.

Checkpoint 11.3. Provide a conformance disclaimer. [Priority 3]

Although it is possible to prove with certainty when something does not conform, the reverse is not necessarily true. Especially for functional specifications, where a claim goes beyond syntax testing, a claim of conformance is not a guaruntee that the claimant is 100% conforming with the specification. A disclaimer can help clarify the meansing of a conformance claim as well as its limitations. For example, this document contains a conformance disclaimer.

Checkpoint 11.4. Impose no restrictions about who can make a claim or where claims can be published. [Priority 1]

Claimants (or relevant assuring parties) are solely responsible for the validity of their claims, keeping claims up to date, and proper use of the conformance icons. As of the publication of this document, W3C does not act as an assuring party, but it may do so in the future, or it may establish recommendations for assuring parties. Claimants are expected to modify or retract a claim if it may be demonstrated that the claim is not valid.

Checkpoint 11.5. Generate a table of contents entry. [Priority 2]

The reader should be able to easily identify and locate this information. A link from the table of contents provides this ability.

Guideline 12. Publish an Implementation Conformance Statement proforma.

An Implementation Conformance Statement (ICS) or questionnaire is useful in clarifying and declaring optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements. The ICS is also helpful in describing the expected interoperabliity to be achieved with other implementations or applications of the specifications.

Checkpoints

Checkpoint 12.1. Include an Implementation Conformance Statement proforma as part of the specification. [Priority 3]

If an ICS is included as part of the specification, indicate whether it is a normative or informative part of the specification.

Checkpoint 12.2. Require the ICS be completed as part of the conformance claim. [Priority 3]

An ICS provides specific information about the implementation and can be helpful in substantiating the conformance claim.

Guideline 13. Support general document conformance conventions.

There is a lot to be said about consistency and clarity within a document - it leads to the understanding and comprehensiveness of the document. Authors and editors of specifications should already be familiar with the W3C Manual of Style [STYLE-MAN] and Publication Rules [PUBRULES ], which help to achieve this. With respect to conformance, it is important to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional. To achieve this objective, throughout the document, employ uniformity of structure and style and consistency of terminology and phraseology.

Checkpoints

Checkpoint 13.1. Use conformance key words. [Priority 1]

Use RFC 2119 key words to denote whether or not requirements are mandatory, optional, or suggested. Using these keywords helps to identify the testable statements in a specification.

Checkpoint 13.2. Distinguish normative and informative text. [Priority 2]

Normative statements are the prescriptive parts of the specification whereas informative statements are for informational purposes and assist in the understanding or use of the specification. It is important that the reader be able to distinguish between normative and informative statements in order to know what is required to claim conformance and what is optional.

Checkpoint 13.3. Follow Web Accessibility Initiative and Internationalization Guidelines. [Priority 1]

Applying the principles of the Web Content Accessibility Guidelines 1.0 [WCAG10] and Internationalization Guidelines (@@reference? what do you mean to point to here?) not only ensures accessibility and internationalization, but also contributes to the principles of conformance - clear, unambiguous, testable statements. For example, use simpler words to express your ideas, markup text with structural elements, add markup to distinguish common words from keywords.

Checkpoint 13.4. Use the same words to express the same ideas. [Priority 1]

Use identical wording to express identical provisions and analogous wording to express analogous provisions.

Guideline 14. Use granular grammars to author the specification.

Using a W3C endorsed grammar language (DTD or XML Schema) provides control over the information conveyed in the specification. This allows for and facilitates automatic generation of test materials, more detailed reporting of test results as well as specification coverage.

Examples include the XML and DOM Working Groups, that both use the xmlspec DTD .

Using a granular grammar to author a specification draws further on separation of content and presentation and simplifies the task of generating readable versions of the specification, maintaining the possibility to generate test materials from the granular version of the specification. In addition, using a grammar to author the specification makes it easier to logically group the constituents of the specification, thus adding control over interpretation and implementation.

Using a grammar greatly simplifies controlling conformance with W3C publication rules and guidelines/checkpoints. Also, having written the specification using a grammar, automated validation with regard to these aspects is simplified.

Finally, using an advanced schema for specification authoring allows for greater control over coverage of the specification in the Test Suite.

Checkpoint 14.1. Use W3C endorsed grammar where applicable. [Priority 1]

One example is the DTD noted above. Other examples will be given.

Checkpoint 14.2. Specify intended behavior in the specification using markup. [Priority 1]

There are different types of technologies that are specified, and this checkpoint applies in different ways to those. For API specifications, such as DOM, the intended behavior is the algorithm output; the specification contains wording on what the expected result of applying a particular method on an interface is. On user-centric specifications, such as WAI guidelines specifications, the intended behavior is not so much a question of testing algorithms, but rather how user agents should behave given different input or how markup should be designed. In this case, intended behavior is more a question of parsing documents. On processor specifications, such as XML, the intended behavior is, for example, preserving or expanding entity references, and the intended behavior is a particular state after processing a document.

Checkpoint 14.3. Supply prose description of intended behavior together with each test assertion. [Priority 1]

Following the guideline below, group test assertions and statements of intended behavior in the specification, or group the pointer to the test assertions together with the statement of intended behavior in the specification (in case the test assertion is, for example, given in an appendix).

Guideline 15. Include test assertions.

Some specifications include test assertions as part of the specification, and more should. A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behaviour. Including test assertions as part of the specification facilitates and promotes the development of test materials. Tests can point directly to the test assertion in the specification. Specific benefits include:

Test assertions (that are part of a specification) are publicly reviewed and agreed to;
Detailed control over what parts of a specification an implementation supports, grouped by behaviour, technical aspect (method/interface) or other conceptual grouping;
Detailed reporting when testing an implementation using the test;

Checkpoints

Checkpoint 15.1. Supply test assertions in the markup of the specification, if applicable using a set of predefined tags used in the specification markup language. [Priority 1]

Checkpoint 15.2. Tag test assertions according to the above. [Priority 1]

3. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

3.1 Motivation for this Guidelines Document

This section defines three levels of conformance to this specification:

Conformance Level "A": all Priority 1 checkpoints are satisfied;
Conformance Level "AA": all Priority 1 and Priority 2 checkpoints are satisfied;
Conformance Level "AAA": all Priority 1, Priority 2, and Priority 3 checkpoints are satisfied;

A specification conforms to the QA Framework: Specification Guidelines at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

The guidelines title: QA Framework: Specification Guidelines
The guidelines URI: http://www.w3.org/TR/qaframe-spec/
The conformance level satisfied: "Level A", "Level AA", or "Level AAA".

Example:

This specification conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/qaframe-spec/, Level AA.

3.2 Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable test requirements, it is also true of these specification requirements that:

Passing all of the requirements to achieve a given conformance level -- A, AA, or AAA -- does not guarantee that the subject specification is well-suited to or will achieve its intended purposes, nor does it guarantee the quality or suitability of test materials produced from the specification.
Failing to achieve level A conformance does not mean that the subject specification is necessarily deficient to its intended purposes, nor does it mean that it is an unacceptable basis for the development of quality test materials. It means that the specification has failed one or more checkpoints that best-practice experience has shown to improve the testability and usability of specifications, and to facilitate the timely and successful development and maintenance of quality test materials based on the specification.

4. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

Daniel Dardailler (W3C)
Dimitris Dimitriadis (Ontologicon)
Karl DuBost (W3C)
Peter Fawcett (RealNetworks)
Kirill Gavrylyuk (Microsoft)
Dominique Hazael-Massieux (W3C)
Lofton Henderson (CGM Open)
Ian Jacobs (W3C)
David Marston (IBM Research)
Jack Morrison (Sun Microsystems)
Lynne Rosenthal (NIST)
Mark Skall (NIST)
Andrew Thackrah (Open Group)
Olivier Thereaux (W3C)

5. References (non-normative)

CSS1: Cascading Style Sheets, Level 1 , W3C Recommendation, 11 Jan 1999, available at http://www.w3.org/TR/REC-CSS1.
MATHML20: Mathematical Markup Language (MathML) Version 2.0 , W3C Recommendation, 21 February 2001, available at http://www.w3.org/TR/MathML2/.
PUBRULES: W3C Publication Rules, available at http://www.w3.org/Guide/pubrules.html (member-only).
QA-GLOSSARY: Comprehensive glossary of QA terminology. (Under construction.)
QAF-INTRO: QA Framework: Introduction , L. Henderson, D. Dimitriadis, K. Gavrylyuk, L. Rosenthal, Eds., W3C Working Draft, May 2002, companion version to this document, available at [...]
QAF-OPS: QA Framework: Operational Guidelines , K. Gavrylyuk, D. Dimitriadis, L. Henderson, L. Rosenthal, Eds., W3C Working Draft, May 2002, companion version to this document, available at [...].
QAF-TEST: QA Framework: Test Guidelines, not yet published.
QAIG: Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG: Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
RFC2119: Key words for use in RFCs to Indicate Requirement Levels , March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
SMIL20: Synchronized Multi media Integration Language (SMIL 2.0) , W3C Recommendation, 07 August 2001, available at http://www.w3.org/TR/smil20/.
SPEC-CHECKLIST: An appendix to this specification guidelines document presents all checkpoints in tabular form. Available at http://www.w3.org/TR/2002/WD-qaframe-spec-20020515/qaframe-spec-checklist.html
SPEC-EXTECH: QA Framework: Specification Examples & Techniques, not yet published.
STYLE-MAN: W3C Manual of Style , summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
SVG11: Scalable Vector Graphics (SVG) 1.1 Specification , W3C (last call) Working Draft, 15 February 2002, available at http://www.w3.org/TR/SVG11/.
TEST-EXTECH: "QA Framework: Test Examples and Techniques", not yet published.
UAAG10: User Agent Accessibility Guidelines 1.0 , I. Jacobs, J. Gunderson, E. Hansen, Eds., W3C Candidate Recommendation, 12 September 2001, available at http://www.w3.org/TR/UAAG10/.
W3C-TR: Location of all published W3C technical reports, see http://www.w3.org/TR/.
WCAG10: Web Content Accessibility Guidelines 1.0 , W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
XML10: Extensible Markup Language (XML) 1.0 (Second Edition) , W3C Recommendation, 6 October 2000, available at http://www.w3.org/TR/REC-xml.
XML-SCHEMA: XML Schema Part 1: Structures , W3C Recommendation, 2 May 2001, available at http://www.w3.org/TR/xmlschema-1/.
XPATH10: XML Path Language (XPath) Version 1.0 , W3C Recommendation, 16 November 1999, available at http://www.w3.org/TR/xpath.
XPOINTER: XML Pointer Language (XPointer) Version 1.0 , W3C Candidate Recommendation, 11 September 2001, available at http://www.w3.org/TR/2001/CR-xptr-20010911/
XSLT10: XSL Transformations (XSLT) Version 1.0 , W3C Recommendation, 16 November 1999, available at http://www.w3.org/TR/xslt.

6. Change history

2002-06-xx version

Significantly reorganized and revised QAWG draft.

Introduced "dimensions of variability" in sec 1.4
Reordered guidelines (GL), split GL.5 into three new ones, GL.3, GL.4, and GL.7
Rewrote (old) GL.3, "conformance flavors", in terms of "specify conformance policy, (new) GL.5.

2002-05-15 version

First public working draft.