About this QA Process Document (QAPD) template
This is an Appendix to QA
Framework: Operational Examples & Techniques. It contains a
template for a Working Group QA Process Document (QAPD). It is complete in
outline form -- all necessary sections and topics, as required by the
QA Framework: Operational
Guidelines (OpsGL), are represented.
This template should help implementers of QA Framework: Operational Guidelines
(OpsGL) to satisfy those
OpsGL checkpoints that require a QAPD or place requirements on its content --
start with this template and fill in the designated QA-related information.
This will produce a complete QAPD that satisfies the OpsGL requirements.
This template contains prototype statements and content as required of a
QAPD. The prototype statements contain fill-in-the-blank sections, which
appear like this, "[specification name]". A
WG would supply the appropriate information here to complete the prototype
statement and make it an actual QAPD statement.
This template makes reference to an Authoring Guideline or Test Material
manual. This is typically a separate WG document that describes the format of
the test materials and is used to ensure that test material contributions are
in an appropriate format and include the requisite information to facilitate
consistency, assessment, and inclusion of the test materials. If a WG does
not have an Authoring Guideline or Test material manual, then it is
recommended that authoring information be included in its QA Process
Document.
Each subsection relates to a particular Checkpoint of OpsGL. The
appropriate back reference and link immediately follows the subsection title,
and looks like this:
[For OpsGL checkpoint CPx.y]
These may be deleted from the final QAPD text that the WG produces, at the
discretion of the QAPD author.
The following are some fill-in-the-blank items that are commonly used or
repeated in this template:
- [WG name]: the Working Group's
name
- [specification name]: name of the
WG's standard or specification
- [test materials type]: generic type
of the test materials, e.g., test suite, validator, etc.
- [test materials name]: name and/or
acronym of test materials
- [choice of: choice-1 | choice-2 | choice-3 |
other(specify)]: items such these require the QAPD author to
choose one of the "or" choices, or specify another value for the
item.
- [optional]: item is optional.
Status
This is the second revision, and incorporates comments and corrections
that resulted from the Last Call review (20030210) of the QA Framework
document family.
Caveat: This version is a QAWG approval draft. Its
links do not necessarily connect properly to lastest versions of QA
Framework: Operational Examples & Techniques or QA
Framework: Operational Guidelines.
The QA Process Document (QAPD) template follows this line.
This is the [DATE] QA Test Material
Process Document for [WG Name].
Comments on this document are invited and are to be sent to the [name of mail list] mailing list [address of list]. An archive is available at
[address of archive].
This document has been produced by the W3C [WG
Name]. The authors of this document are the [WG Name] WG members.
Table of contents
This document describes the QA process for ensuring availability of
comprehensive conformance [test material type -
choice of: test suite(s) | validator tool(s) | checklist |
other(specify)] for the [specification name
| (W3C-[WG Name]-specifications)]. It provides information about the
WG's quality-related logistical and communications setup, contribution
mechanism, licensing policies, publication and maintenance plans, as well as
any other critical processes and operational details needed to produce and
maintain the [test material type].
The [test material type] produced by the
W3C [WG name] will be known as [test material name].
- The [test material type] should be
developed in a public forum such that participation and contributions
from parties interested in developing the [test materials type] is possible and
welcome.
- A test material framework should be developed to provide a uniform
method for producing and managing the [test
material type].
- The [test material type] should be as
comprehensive, functional and complete as possible.
- A standard approach to [test material
type] IPR which meets the needs of both contributors and users
should be established.
- The [test material type] should aid
implementors in developing [specification
name] implementations. Validation and certification of these
implementations are outside the scope of this work.
- The [test material type] will be
hosted on the W3C site once developed. It may also after development be
mirrored at various sites in order to simplify access and enhance
availability to the community.
The [test material name] will be
produced in a way that provides for contributions from developers and
companies in the community. The [test material
name] will be coordinated and supervised by the [WG Name and/or other names]. The WG appoints a
moderator to lead the effort.
[For OpsGL checkpoint CP4.4]
[Choose
one: a) if you use the WGs established list or b) if you set up a dedicated
list]
a) The main channel of communication for the [test material name] will be the existing [WG name] mailing list [address]. An archive is available at [address]
b)The main channel of communication for the [test material name] will be a dedicated mailing
list, called [test material name] mailing
list [address]. An archive is available at
[address].
The public web page for QA information is named /Test/ and is available
at: [http://www.w3.org/WGname/Test/]. This
page includes links to this process document, the [test material name] and information about the
[test material name] including:
- an overview or introduction to the [test
material name]
- releases
- how to participate
- how to use the [test material
name]
- [test material name] principles
- acknowledgements
[For OpsGL checkpoint CP4.5]
The [test material type] will be
produced in a public framework with contributions from members of the [WG name], W3C, or the community at large. The
framework will provide a stable mechanism for submission of test materials,
saving information about those tests, version handling, use of the tests, and
so forth.
[For OpsGL checkpoint CP5.2]
[Delete this paragraph, if the WG has no Authoring
Guideline or Test Material Manual.] Contributions should be submitted
in accordance with the [Choose: Authoring
Guidelines | Test Material Manual].The [Choose: AuthoringGuidelines | Test Material
Manual] explain how to write tests for the [test material name]. It defines the [choose one or more: format | test description file | schema
| other] for the contribution along with any additional required
documenation.
[Choose one of a) or b)]:
a) Contributions of test materials should be submitted to the [test material name] via the [WG name or test material name -- this is a or b from above] mailing list: [address].
b) Contributions of test materials should be submitted to the [test material name] test submission mailing
list: [name-testsuite]-submit@w3.org. An
archive is available at: [address]
The process for test materials contributions is:
- Contributions of test materials are submitted to the framework.
- Receipt of the contribution is acknowledged. This is done via email
directly to the submitter, with a copy sent to the mailing list. The test
receives the "Received"; status.
- The contributed test materials together with the necessary
documentation is saved for future reference.
[For OpsGL checkpoint CP5.4]
Contributions to the [test material
name] are evaluated according to the following criteria.
- Relevance for the particular requirements of [specification name]. It can be decided here
that the particular test is irrelevant to the [test material name], e.g., because it does
not test a normative requirement of the specification.
- Part of the specification being tested.
- Quality of code and documentation. The code must be stable and
not contain any errors; and if written in an XML based language, must be
well-formed. Documentation must be written in accordance with prescribed
information used to formulate and support the [test material name].
- [optional item, applicable if there is a schema
for test submissions; delete otherwise.] Each contribution must
conform to the schema for test submissions, which is available at the
[test material name] website.
- Scenarios that underly the particular test methodology and its
intended scope.
- [optional item, applicable if there is a
Guidelines/Manual, delete otherwise]Adherence to the [choice of: Authoring Guidelines | [test material name] Manual].
If a contributed test is judged suitable for inclusion in the [test material type] according to these criteria,
the test is added to the [test material
type] with status "Accepted".
Tests that are judged to be inappropriate for publication are returned to
the contributor. Such tests are archived and not included in the [test material type].
[For OpsGL checkpoint CP8.3 ]
At any time, any member of the community or WG may dispute the validity of
any test in the [test material name]. This
is done by raising the issue on the [test material
name] mailing list. If the dispute is felt to be valid by members of
the WG, the status of the test is changed to "Disputed", and the test is
referred to the WG for detailed review. The possible outcomes of the review
are:
- Error Free. The test and its accompanying documentation have
undergone review by the WG and have been found to be free of errors and
relevant to the specification.
- Disputed: test error. A consensus exists in the WG that the
test is in error. A resolution is formulated and the test is changed
accordingly.
- Disputed: specification error. The the consistency, clarity,
etc. of the part(s) of the specification addressed by the test is
questioned. The WG agrees that a problem exists with the specification
and submits the test and the associated dispute to the WG's erratum
process.
[optional section follows, probably redundant if
Authoring Guideline or Test Material manual exists.]
Each contribution to the [test material
name] must be fully documented. The documentation provides such
information as:
- Name of submitter, date of creation, and, if appropriate,
organization;
- Suggested ID for each test (this may be changed by the moderator to
comply with the identification mechanism of the technical platform);
- Part(s) of the specification being tested. Areas in the specification
being tested should be suitably indicated by either a link to the
relevant part of the specification or a text annotation to the relevant
part of the specification;
- The expected validity of the test;
- If appropriate, a textual annotation containing remarks on the test in
human readable format, for example any scenario which prompted the
writing of the test.
[For OpsGL checkpoints CP4.6, CP6.4]
The [test material name] aims to help
implementers write applications that support [specification name]. In no way are these
conformance tests to be construed as providing certification or branding of
[specification name] implementations. The
only claim that could be made is that a particular implementation is
conformant to a particular version of the [test
material name]. There are two cases of results of running the [test material type]:
- The implementation fails at least one aspect of the test materials that
is associated with normative requirements of [specification name]. In this case, it can be
asserted that the implementation fails to conform to [specification name].
- The implementation passes the test materials. In this case, all that
can be asserted is that the implementation is conformant to that
particular version of the [test material
name].
[For OpsGL checkpoints CP5.3, CP6.2]
The Contributor hereby makes certain materials, as described below (the
"Materials";), available for use in supporting the World Wide Web Consortium
(W3C). The Contributor hereby grants to MIT, ERCIM, and Keio (the "W3C
Hosts";) a perpetual, nonexclusive, royalty-free, world-wide right and
license to use, copy, modify, create derivatives of and distribute the
Materials, in whole or in part, solely in connection with the W3C, and to
allow others to do the same. W3C Hosts will publish and distribute the
Materials, or any modifications or derivatives thereof, pursuant to the W3C
Software License (20021231), and/or the W3C
Document License (20021231) as modified from time to time.
The Contributor represents that she/he has all rights necessary to
contribute the Materials, and that use of the Materials as contemplated
herein by W3C Hosts does not violate any copyright, patent, trademark, or
contractual obligations.
The Contributor agrees that any derivatives of the Materials created in
connection with the W3C shall be owned by the W3C Hosts. Any publication or
distribution of the Materials or any derivative thereof, will retain
attribution of authorship to the Contributor. Whenever modifications are made
to the Materials, this fact, and the nature of the modifications, will be
clearly identified in the distributed version thereof. The W3C Hosts make no
commitment to support or distribute the Materials.
[Choose one of a) or b) depending on whether the
same license is applied to all components, or different licenses are applied
to different components]:
a.) The [test material name] produced by
the [WG name] Working Group will be
distributed under the W3C [License type - choice
of: Document License | Software License] License.
b.) The [test material name] produced by
the [WG name] Working Group will be
distributed with different W3C licenses applied to different components, as
follows:
- Test Cases component: under the W3C [License
type - choice of: Document | Software] License.
- Test Software component: under the W3C [License type - choice of: Document |
Software] License.
- Test Documentation component: under the W3C [License type - choice of: Document |
Software] License.
[For OpsGL checkpoints CP6.1, CP6.3]
The [test material name] will be
published on the W3C [WG name] website,
available from: [http://www.w3.org/[WGname]/Test/].
The [test material name] will be
published after thorough review by the WG and the community, and with the
consensus of the WG.
The [test material name] will contain
the conformance disclaimer as stated above.
[For OpsGL checkpoint CP6.5]
The [WG name] WG encourages the
publication of test results. To facilitate publication of results, the WG
will make available on its webspace, a place to publish test results. Only
test results that have been supplied or approved by the developer of the
tested implementation will be considered for publication.
The WG will also consider a 'members only' webspace to publish developer
submitted test results. This restricted publication of results may be used
for situations in which the specification and/or tests are being debugged.
For example, when the specification is in CR and a set of tests are being
used to assess the features of the specification by determining what is being
implemented.
[For OpsGL checkpoints CP8.2, CP8.3]
The WG is committed to maintaining the quality and relevance of the [test material name]. This will be done by
ensuring that the test materials are consistent with any erratum and are
updated as needed to reflect the current versions and status of the
specification. The procedures describe above will be used to accept new
contributions as well as to handle tests that are in dispute.