This document is part of the of the Quality Assurance (QA) Activity. It presents examples and describes the
techniques of operational aspects of quality practices within the W3C's
Working Groups. It complements "QA Framework:
Operational Guidelines"[QAF-OPS], by specifying or
illustrating how to meet the operational and process-related checkpoints of
that document.
This version is a Editors-only draft. SOTD for FPWD
follows...
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest status
of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the first public Working Draft, and supersedes all
previous WG-only drafts. It is expected that updated WD versions of this document will be produced
regularly, along with other members of the Framework documents family. Future
progression of this document beyond Working Draft is possible, but has not
yet been determined.
In this version, several example case studies are presented in some detail
-- how the QA projects of three existing Working Groups relate to the
checkpoints of the operational guidelines [QAF-OPS].
It is expected that future versions will add at least one example to the
current three (SVG, DOM, and XML). This version contains relatively little
development of "techniques" -- what Working Groups should do to satisfy the
operational and process-related checkpoints, and what constitutes conformance
to the checkpoints. This is expected to be a significant addition in future
versions.
Please send comments to www-qa@w3.org,
the publicly
archived list of the QA Interest
Group [QAIG]. Please note
that any mail sent to this list will be publicly archived
and available, do not send information you wouldn't want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR.
1. Introduction
2. Anatomy of this document
3. Conformance test suite projects
3.1 DOM
3.2 SVG
3.3 XML
3.4 XSLT
4. Operational guidelines in action
G 1. Integrate Quality Assurance into Working Group
activities.
G 2. Define
resources for Working Group QA activities.
G 3. Synchronize QA activities with the milestones for
WG deliverables.
G 4. Define the QA process.
G 5. QA
Process: Plan test materials development.
G 6. QA
Process: Plan test materials publication.
G 7. Plan the transfer of test materials to W3C if
needed.
G 8. Plan
for test materials maintenance.
5. General remarks
6. Conformance
7. Acknowledgments
8. References
1. Introduction
This document contains examples and techniques of QA-related activities
within the W3C as well as guidance as to how the quality processes of the
Working Groups (WGs) should be carried out. It complements "QA Framework: Operational Guidelines" [QAF-OPS], by specifying or illustrating how to meet the
operational and process-related checkpoints of that document.
The operational guidelines document should be considered a prerequisite to
this one. In addition to containing all of the checkpoints to which the
contents of this document pertain, it explains principles underlying the
conduct of WG quality processes.
Four test suite (TS) efforts related to W3C Recommendations are used as
examples. Some of them were written in-house (within the responsible WG),
some externally (outside of W3C), and some are hybrid (external plus in-house
effort). All of them have succeeded in producing significant and valuable TS
deliverables.
- SVG: intra-WG TS development (both SVG1 and SVG2)
- DOM: hybrid (WG/NIST) development.
- XML: external development and import TS into WG
- XSLT: external development (no import into W3C).
In all four cases, the associated WGs have existed and were well advanced
on the relevant Recommendations before the operational guidelines [QAF-OPS] were first published. Because these projects
preceded the guidelines, it makes no sense to score them against checkpoints
-- it is unreasonable to expect them to meet all of the checkpoints. Instead,
the various aspects of the different projects are used to illustrate the
meaning and intent of the checkpoints. In a sense, the checkpoints are the
union of the best practice and techniques from all of projects (these and
others).
This points to a timing-based reality of conformance to the operational
guidelines. The conformance criteria ("techniques", to be developed and added
in the future), will be different for new Working Groups versus existing
Working Groups.
This document is heavily weighted towards case studies, examples and
illustrations. It has relatively little specification of techniques for
meeting checkpoints, or criteria for judging conformance to the
checkpoints.
2. Anatomy of this document
The structure of this document parallels the structure of the operational
guidelines [QAF-OPS] document, particularly its
Chapter 3, "Guidelines". It
should be realized that all of the given examples of W3C Working Groups'
QA-related activities were started before the applicable parts of the W3C QA
Activity. The useful milestone for the latter is the first publication of the
Operational Guidelines, February 2002.
Relevant common features in these example frameworks will be pointed out,
and they will be compared against the statements of the checkpoints of the
operational guidelines. For each of the given example QA-related frameworks,
a short description of each will be presented. The subsequent parts of the
document will elaborate on the checkpoints -- including links to relevant
process-related documents and resources -- and how they have been realized in
various efforts.
3. Conformance test suite projects
Herein is a summary description of each of a small set of case studies of
W3C QA-related activities, as well as some external to the W3C.
Each subsection covers the following topics:
- General outline
- Internal/External/hybrid and staffing requirements
- Charter considerations
- When TS is produced -- during or after specification phase?
- Pointer to process document
- Test material storage, maintenance, tracking
3.1.1 General overview
The DOM TS was jointly launched by the W3C and NIST and is produced in
public. It uses coordination from the W3C DOM WG, and is produced using
communication through a W3C-hosted mailing list as well as W3C-supplied
storage for files, documents and so forth. The TS was to be developed in
public, and the WG would endorse two bindings, Java and ECMAscript, the same
as the official DOM bindings. Other bindings were welcomed but would not form
part of the official TS.
3.1.2 Internal/external/hybrid and staffing requirements
The DOM TS was launched by the W3C DOM WG and NIST in an effort to invite
the public to participate in the production of quality testing material for
implementations of the DOM specifications. Staffing requirements were
limited, so an expert was invited to the DOM WG in order to produce and edit
the DOM TS Process Document. The DOM TS was in its initial phase very similar
to a previously launched DOM Test Suite launched by NIST, which was used as a
primer for the forthcoming work.[@@include links to relevant emails]
3.1.3 Charter considerations
Since the DOM charter did not include sections on producing test
materials, this was included in the DOM TS process, which outlined the DOM TS
deliverables in detail.
3.1.4 When TS is produced -- during or after specification phase?
The DOM TS was produced after specification for the first two levels (DOM
Level 1 & 2) and is produced in parallel with specifying DOM Level 3. DOM
Level 1 was released in late 1998, so it was not possible to synchronise the
specification of the firsl level of the ODM with the Test Suite.
Starting with the current level of the DOM, the TS is produced in parallel
with the specification. Tests are being produced to allow for implementors to
check their implenentations of DOM level 3 while contributing to relevant
parts of the specification. Also, members of the DOM WG have commited to
produce tests for those modules of the ODM that are particularly relevant to
their implementations.
3.1.5 Pointer to process documents
There is a separate DOM TS
Process Document, and also a Test
Web page where you can find more information and pointers to primers,
FAQ's and tutorials, maintained by the DOM WG representative to the DOM
TS.
3.1.6 Test material storage, maintenance, tracking
The DOM TS uses W3C machinery to store test files, using CVS for easy
tracking, and an issue tracking system at Sourceforge, in order to simplify
issues and maintenance.
3.2.1 General overview
As used in this case study, "SVG" refers collectively to two consecutively
chartered Working Groups. SVG1 refers to the originally chartered group
(first meeting, Sept 1998) that produced SVG 1.0 (Recommendation, Sept 2001).
SVG2 is the successor to SVG1, (re-)chartered in 2001, and charged with
developing SVG 1.1 (modularization and profiles), and SVG 2.0. SVG 1.1 is
presently at the stage of CR.
SVG 1.0 was released with a comprehensive "Basic Effectivity" (BE) test
suite, that was developed entirely within the working group. The test suite
(TS) comprises 127 test cases that span the functionality of SVG, exercising
each major functional feature of SVG 1.0 at least lightly, but not probing
any features in detail. SVG2 is adapting and subsetting the tests of the SVG1
TS for smaller and less capable mobile devices that are the target
constituency of the Tiny and Basic profiles.
3.2.2 Internal/external/hybrid and staffing requirements
The SVG1 test suite was developed entirely by the working group. A pilot
study sponsored by one of the members became the basis for design. A "TS
Editor" (moderator) managed the project from inception to completion. All WG
members are expected to participate in the writing and review of test cases.
The SVG2 approach is identical -- a WG member is the TS moderator, and all WG
members contribute.
3.2.3 Charter considerations
Both SVG1 & SVG2 charters specify a "comprehensive test suite" as a
deliverable, and coordinate the dates of first TS delivery (first
publication) and the specification CR milestone. There is no further detailed
breakdown of milestones or deliverables, as this was worked out (for SVG1) as
the specification development progressed.
3.2.4 When TS is produced -- during or after specification phase?
The test suites were (are) produced during the specification-writing
phase. The target is substantial completion at the CR milestone, so that the
TS could be used to facilitate the "interoperable implementations" W3C
Process requirements of CR.
3.2.5 Pointer to process documents
SVG has no QA process document per se, but much of the content that these
operational checkpoints anticipate and require may be found in the extensive
reference document, "SVG Conformance
Test Suite -- Test Builder's Manual". Unless otherwise specified in that
document, the normal WG processes of the SVG Working Group are used to
conduct the TS work.
3.2.6 Test material storage, maintenance, tracking
The SVG WG keeps the SVG TS on a CVS server of one of the WG members.
Revision tracking is therefore handled by CVS. Maintenance procedures and
issue tracking for SVG TS are not separately defined, but rather SVG TS uses
the normal SVG WG processes for these.
3.3.1 General overview
The XML Conformance Test Suite, v1.0, Second Edition contains over 2000
test files and an associated test report. The test report contains background
information on conformance testing for XML as well as test descriptions for
each of the test files included in the TS.
The OASIS/NIST XML Test Suite forms the basis for the W3C XML TS. It was
transferred to W3C where it is being augmented with new functionality,
corrected to align with the Recommendation and its errata, and maintained.
Prior to accepting the OASIS/NIST Test Suite, the XML Core WG assessed the
level of effort necessary for TS development and maintenance and ensured that
staff resources were interested and available to work on the TS.
Additionally, the WG enlisted the assistance of NIST to lead the effort. The
TS is being developed as a deliverable of the WG, following the WG's Process.
There is a W3C hosted XML TS mailing list, Issues list, and W3C supplied
storage (CVS) for the test-suite related files.
3.3.2 Internal/external/hybrid and staffing requirements
A commitment from NIST to take editor-lead responsibility for the TS was
obtained prior to the commencement of the work. Consequently, NIST joined the
WG. Members of the XML Core WG contribute to the development and quality
assessment of the TS. When the XML TS is released by the WG, the general
public will be invited to review the tests as well as participate in
contributing tests.
3.3.3 Charter considerations
The W3C XML Core WG charter does not include sections on producing test
materials. A new charter provides for the possibility of working on the
OASIS/NIST XML Test Suite. The WG does not have a separate QA Process, but
follows the WG Process for the development, assessment and maintenance of the
XML TS. This is not explicitly documented in a separate QA process document,
but the individual decisions are on record in WG mail archives and minutes as
part of the normal WG process.
3.3.4 When TS is produced -- during or after specification phase?
The work on the OASIS/NIST XML Test Suite was conducted after the XML 1.0
Recommendation was released. A revised edition of the OASIS/NIST Test Suite
1.0 (Second Edition) Test Suite was released to complement the W3C XML 1.0
Second Edition Recommendation (October 6, 2000). Within the XML Core WG, the
XML TS is being updated to reflect the current Recommendation and all
applicable Errata.
3.3.5 Pointer to process document
There is no separate QA Process document. The XML Core WG charter uses its
established WG Process to handle test development/assessment activities,
communication means and test materials related functions.
3.3.6 Test material storage, maintenance, tracking
The XML WG uses the W3C CVS repository to maintain the XML TS. The XML TS
issues are maintained utilizing an internal assessment process that consists
of a running log of test issues, XML TS mailing lists, XML Core WG
discussions and the XML Test Suite Issues document that keeps track of the
running log of issues.
[@@ to be done. Probably must wait till next version after FPWD.]
4. Operational guidelines in action
This section looks at the guidelines and checkpoints in the operational
guidelines [QAF-OPS
], and how their
requirements are reflected in the various existing frameworks. Where
appropriate, pointers to relevant documents and materials are provided.
The structure of this section exactly parallels the structure of the its
Chapter 3, "Guidelines", of
the operational guidelines [QAF-OPS] document. Each
checkpoint links back to the corresponding checkpoint of "QA Framework:
Operational Guidelines" [QAF-OPS].
Under each checkpoint, there is discussion of how each of the example QA
or TS projects relate to the checkpoint. Finally, there will be a summary
enumeration of ways in which the each checkpoint may be satisfactorily
satisfied [@@this synthesis will be done in a future version].
Guideline 1. Integrate Quality Assurance into Working Group
activities.
DOM discussion
In the DOM TS, the QA activities were integrated into Working Group
activities by inviting a field expert to do the coordination between WG and
TS Group. Coordination was necessary as the DOM WG QA activity was launched
in coordination with an external party, and, furthermore, was to be produced
externally.
SVG discussion
QA was integrated into the SVG WG activities by charter commitment to
produce test suites, and subsequent involvement of most WG members in the
writing, reviewing, and maintenance of the test materials. The charters
committed, in the cases of both SVG1 and SVG2, that a "basic test suite"
(i.e., Basic Effectivity) would exist at or before Candidate Recommendation.
The details of deliverables, milestones, and publication were generally
worked out by the WG as the specifications developed.
XML discussion
The QA activities are integrated into WG activities by using the WG
Process. Although the activities are not reflected in the WG Charter,
deliverables are identified, expected release dates determined, staff
allocated, and there is a clear commitment to developing and maintaining the
TS.
Checkpoints
Checkpoint 1.1. In the Charter, specify the level of commitment
to QA. Plan to have at least a commitment to "QA level three". [Priority 1]
- SVG. The SVG1
charter (member-only) addressed the topic with "Publish a
comprehensive test suite to aid developers in achieving SVG conformance".
Together with the milestones schedule, it would satisfy the TS portion of
at least level 3, although the details of this commitment were not
spelled out in the charter (or any other single location). The SVG2
charter (member-only) makes a similar commitment.
- DOM. As the W3C QA framework did not exist when the DOM TS Process was
written, several of the checkpoints below do not apply. In particular, no
explicit commitments to QA related deliverables are made in the DOM
charter. Any such commitments have been made in the DOM TS Process document or
in email on the DOM TS mailing
list (archives).
- XML. The initial work on the test suite was done by OASIS/NIST, and it
was transferred to the XML Core WG. Level 3 was already built into the TS
by the OASIS/NIST creators. Because the (original) XML Charter far
pre-dated these events, no commitment exists in the Charter. However the
transfer was discussed -- and commitments to transfer and support the TS
are on record -- on archived XML mail list (member-only) and a
teleconference, between W3C (XML WG and QAWG), NIST, and OASIS.
Guidelines document
description of checkpoint.
Checkpoint 1.2. In the Charter, plan at least for the
"QA level five". [Priority 2]
- SVG. Both SVG1 and SVG2 could be considered to conform in fact to level
5 in the TS column, although the details of this commitment were not
spelled out in the charter (or any other single location). The position
in the first (spec testability) column is uncertain. It depends on the
definition of "normative use cases" -- the SVG specs have thorough
coverage with examples.
- DOM. No such plans were made for the DOM TS.
- XML. As with the previous
checkpoint -- level 5 was built into the TS prior to transfer, and
the agreements and/or commitments are on record in mail archives.
Guidelines document
description of checkpoint.
Checkpoint 1.3. In the Charter, plan for the
"QA level seven". [Priority 3]
- SVG. Both SVG1 and SVG2 might be considered to conform in fact to level
7 in the TS column, although the details of this commitment are not
spelled out in the charter (or any other single location). This is
dependent on the definition of "complete". The SVG test suite is
"comprehensive" in the sense that it has complete breadth first
"lightweight" coverage (Basic Effectivity test suite), but lacks complete
detailed or drill-down coverage.
- DOM. No such plans were made for the DOM TS.
- XML. An upgrade of the TS materials to level 7 is being done now. Per
comments in preceding checkpoints,
this is not documented in the charter.
Guidelines document
description of checkpoint.
Checkpoint 1.4. In the Charter, enumerate QA deliverables and
expected milestones.
[Priority 1]
- SVG. Both the SVG1 and SVG2 charters specify delivery of a
comprehensive test suite, and relate delivery to document milestones.
There is no further detailed breakdown of milestones or deliverables.
This was worked out (for SVG1) as the specification development
progressed.
- DOM. See the DOM TS Process
document, where a timeframe is indicated.
- XML. Not quite. A new XML
WG Charter (member-only), in the approval process in April 2002, says
"..may do work in related supporting areas such as: overseeing
coordination with the OASIS/NIST XML Conformance Testing test suites."
Deliverables, tentative schedules, commitments, milestones were discussed
and agreed in (member-only) meetings and mail lists.
Guidelines document description
of checkpoint.
Checkpoint 1.5. In the Charter, associate QA criteria with
WG milestones.
[Priority 2]
- SVG. The SVG1
charter (member-only) scheduled a BE TS and Candidate Recommendation
(CR) for the same date. The SVG2
charter (member-only) commits to a BE TS date that is earlier than
the date for CR.
- DOM. See the DOM TS Process
document, where a connection to WG deliverables is indicated. Please
note that the QA deliverables statement is valid only for DOM Level 3, as
the previous two levels had already been specified (in part) before the
DOM WG decided to provide a DOM TS.
- XML. Not quite, see comments in previous checkpoint.
Guidelines document
description of checkpoint.
Guideline 2. Define
resources for Working Group QA activities.
DOM discussion
The DOM WG appointed an invited expert to monitor and coordinate the work
of the DOM TS. Furthermore, the members of the DOM TS have looked into
producing tests fot that part of the specification which is of particular
interest to their implementations.
SVG discussion
The SVG1 and SVG2 charters explicitly charge the WG with producing test
materials, but do not further address specific staffing. In fact, the
expectation has evolved that all WG members should contribute to the building
of the TS. In both cases, a "TS Editor" has been assigned to lead and
coordinate the TS production.
XML discussion
The WG appointed NIST to lead the TS effort and coordinate the work of the
XML TS. Furthermore, members of the WG have reviewed and assessed the current
set of tests, corrected tests, and contributed tests.
Checkpoints
Checkpoint 2.1. In the Charter, address where and how conformance
test materials will be produced.
[Priority 1]
- SVG. In both SVG1 and SVG2, the test suites were defined as a WG
deliverables -- it was explicit that the WG was to produce the TS, and
explicit that "basic" completion was to occur at or before CR.
- DOM. This information is given in the DOM TS Process document
and not in the DOM WG charter. Please see relevant parts of the DOM TS
Process.
- XML. The new XML
Charter (member-only, in process) indicates that the WG scope may
include, "..overseeing coordination with the OASIS/NIST XML Conformance
Testing test suites." The details of the "where and how" are in meeting
minutes and mail archives, as described in an earlier checkpoint.
Guidelines document description
of checkpoint.
Checkpoint 2.2. In the Charter, address QA staffing
commitments.
[Priority 1]
- SVG. While the charters do not explicitly address staffing commitments
(nor do any other standing documents), they do charge the WG with
producing the TS. In fact, all WG members are expected to participate in
the TS work. This was explicitly documented in the minutes
of a SVG2 face-to-face (member-only).
- DOM. There are no QA staffing commitments made, but the way the TS is
to be produced and coordinated is outlined in the DOM TS
Process.
- XML. QA staffing commitments are not addressed in the Charter, and no
specific staffing levels are documented or committed to elsewhere.
Staffing resource was addressed and agreed in multi-way discussions
between W3C (XML and QAWG), NIST, and OASIS prior to the TS transfer to
W3C, and accordingly should be on record in the XML mail archives.
Several WG members do most of the TS work, but the whole WG participates
in the sense that TS topics are discussed as a regular part of WG
meetings and teleconferences.
Guidelines document
description of checkpoint.
Checkpoint 2.3. In the Call For Participation, request
allocation of QA resources to the Working Group.
[Priority 1]
- SVG. (@@unavailable -- where is CfP?).
- DOM. Not applicable.
- XML. No Call For Participation was done. The participation happened
when the TS was transferred from OASIS to W3C.
Guidelines document
description of checkpoint.
Guideline 3. Synchronize QA activities with the milestones for
WG deliverables.
DOM discussion
The milestones were given in the DOM TS Process document and were
monitored by the DOM WG in order to assure the satisfactory continuation of
the DOM TS group work. In those cases where the TS Group needed more time to
achieve its goals, the DOM WG representative reported to the WG in order to
appeal for an extension.
Applicable in so far as the TS releases are (for levels currently being
specified) synchronized with the relevant modules of the DOM
specifications.
SVG discussion
The SVG1 and SVG2 charters establish the basic synchronization, which is
that the first publications of basic test suites (TS) should be at or before
the beginning of CR stage. Beyond that, there are no explicit (i.e., written
down) synchronizations. This happens in fact as part of the WG process and
the TS building. TS publications are synchronized with selected late-stage
document versions (CR, PR, Rec). There is no explicit scheme for
labelling/linking test cases according to errata, although this would be
reconstructable from the CVS repository for the TS.
XML discussion
The TS work is being conducted in close coordination with the WG's other
deliverables. The TS is being updated and augmented to reflect the current
work of the WG, including resolved issues related to the Recommendation and
published Errata. Tests for new functional areas are also being explored as
the deliverables for those Recommendations progress (i.e., Xinclude and
Namespace).
Checkpoints
Checkpoint 3.1. Synchronize the publication of QA deliverables and
the specification's drafts.
[Priority 2]
- SVG. The SVG1 and SVG2 charters link a basic TS with the start date of
CR, as described above. In the SVG1 process -- in fact, but not specified
in any document -- three publications of the TS were coordinated with
three late-stage specification drafts (CR, PR, Rec texts), as explained
on the SVG TS page.
Not every draft resulted in a new TS, but every TS was clearly linked to
its specification version.
- DOM. There is a DOM WG policy to do so (not commited to in print).
- XML. Not directly. The Test Suite is amended to incorporate Errata.
Guidelines document description of
checkpoint.
Checkpoint 3.2. Support specification versioning/errata in the
QA deliverables.
[Priority 3]
- SVG. The TS is maintained to keep current with the specification.
However, there is apparently no explicit link between test cases and
errata. There is implicit linkage with specification versions (1.0, 1.1,
...) via the SVG DOCTYPE declaration and external DTD reference in each
test case.
- DOM. The DOM TS keeps its source files in W3C CVS, therefore versioning
of source files is simplified. Also, there are guidelines on
versioning of TS in the DOM TS Process document.
- XML. The Test Suite has versions. [@@linked to ...? Is there any
explicit usage of TS Version to tie TS to: 1.) new XML versions (e.g.,
1.0 2nd ed; 1.1; ...); 2.) XML errata level/date?]
Guidelines document
description of checkpoint.
Guideline 4. Define the QA process.
DOM discussion
DOM TS Process document
SVG discussion
SVG support the process requirements of a QA moderator (manager of TS
development) and a "task force" (it is the whole WG, by explicit choice).
TS-related communication is handled internally on the whole-WG list, and
externally via a public TS comment list. SVG does not have a Process
document, per se, that spells out these details. It does have a TS manual
that defines the TS framework in some detail.
XML discussion
The WG Process is used.
Checkpoints
Checkpoint 4.1. Appoint a QA moderator.
[Priority 2]
- SVG. Both SVG1 and SVG2 TS projects have a moderator (referred to as
"TS editor" in SVG1).
- DOM. An invited expert to the DOM WG was appointed contact between the
DOM WG and the DOM TS group. He also edited the DOM TS Process document.
Relevant announcements were made on the public
DOM mailing list by the DOM WG chair.
- XML. A moderator from NIST was appointed as part of the TS transfer
agreements and process.
Guidelines document description
of checkpoint.
Checkpoint 4.2. Appoint a QA task force.
[Priority 2]
- SVG. Per se, there is no QA task force in the SVG TS projects -- the
whole WG is expected to participate and contribute. In SVG2 this was made
explicit. The minutes
of the first SVG2 face-to-face (member-only), say: "Consensus that
developing and maintaining tests for the test suite is a requirement for
participation in the working group. (ie. each working group member will
be responsible for creating and maintaining tests on a particular part of
the spec)"
- DOM. Since the DOM TS was launched by the W3C in coordination with NIST
(see announcement above) but wat to be developed in public, no QA taks
force was appointed.
- XML. There is no appointed task force, but the moderator and a few
others regularly work on TS actions, and the TS topic is a regular item
on WG meetings and teleconferences.
Guidelines document description
of checkpoint.
Checkpoint 4.3. Produce the QA Process document.
[Priority 1]
- SVG. SVG1 had no TS Process document per se, but there is a "SVG
Conformance Test Suite --Test Builder's Manual". Its content is
wide-ranging, and includes some of the material that would be in a TS
Process document.
- DOM. The DOM TS Process
Document was produced by the DOM WG coordinator and the NIST
representative.
- XML. There is no explicit QA Process document per se. The TS work is
conducted as a normal piece of WG business, and therefore follows the WG
Process.
Guidelines document
description of checkpoint.
Checkpoint 4.4. In the QA Process document, define means for
QA-related communication.
[Priority 2]
Guidelines document description
of checkpoint.
Checkpoint 4.5. In the QA Process document, specify the
framework for test materials development.
[Priority 2]
Guidelines document
description of checkpoint.
Checkpoint 4.6. Specify a policy for branding
materials.
[Priority 3]
- SVG. N/a -- SVG does support branding or assigning conformance
icons.
- DOM. Not applicable.
- XML. Not applicable.
Guidelines document
description of checkpoint.
Guideline 5. QA
Process: Plan test materials development.
DOM discussion
As the DOM TS drew on experiences made by an external party (NIST), the
work initially drew heavily on those experiences. Where the subsequent work
indicated digression, the future version of the DOM TS was branched off and
formed the official W3C DOM TS. The test material planning was made by the
DOM WG representative to the DOM TS in coordination with NIST and the public,
after the DOM TS had been officially launched.
Several design issues were discussed by the DOM TS Group itself, then
brought to the DOM WG for acceptance. Such issues include the automatic
generation of the schema language for the DOM TS, using the same test
description file to generate the official bindings (and the unofficial ones
using a different transform) as well as using particular build tools to build
working and official releases of the DOM TS.
Most discussion was done on the DOM Group mailing
list and coordinated with the DOM WG. More information can be found in
the DOM TS page, which serves as a
complete reference for DOM TS related information.
SVG discussion
The planning for SVG's test materials development started with a study and
prototype design, sponsored by one of the SVG WG members. This work was
brought into the WG as the starting point of the TS development, and was
further developed, refined, and applied by the moderator ("TS Editor"). Both
user documentation (for TS releases) and contributors documentation were
developed. The latter described the TS framework in some detail, and gave
both technical criteria for the contributed test cases plus review/acceptance
criteria. The actually processes and procedures for contribution and review
were not explicitly defined, as all materials came from WG members (and
therefore followed WG process).
XML discussion
The XML TS drew on the experiences and expertise of NIST, the original
developer of the test suite framework. The original work was revised to
included test requirements with a mapping to the test cases and to the
specification. The planning of improvements and augmentation to the original
work was discussed and decided by the WG. Techniques and methods were
borrowed from other test suite development efforts, such as DOM and
XSL-FO.
Checkpoints
Checkpoint 5.1. In the test framework, ensure test materials
are documented and usable for the intended purpose.
[Priority 2]
- SVG. SVG1 published good release
notes with TS releases, as well as the Manual.
The TS home page
describes and points to all the materials.
- DOM. All of the tests used in the DOM TS are validated using a DOM TS
ML schema generated directly form the relevant DOM specification using
XLST transforms that are part of the DOM TS framework. In addition, each
test is required to contain information in prose on intended testing
purpose and part of specification being tested. For more information, see
the DOM TS ML
document.
- XML. Yes (@@pointer tbd@@)
Guidelines document
description of checkpoint.
Checkpoint 5.2. In the QA Process document, define a contribution
process.
[Priority 2]
- SVG. in the Manual,
there are contribution criteria, and test review criteria, but no
contribution process per se. Contributions are described as potential
source of tests.
- DOM. Defined in the DOM TS Process
document.
- XML. Contributions are encouraged and should be in the format of the
Test Framework.
Guidelines document description
of checkpoint.
Checkpoint 5.3. In the QA Process document, define the
licenses applicable to submitted test materials.
[Priority 2]
- SVG. This has not been directly addressed by the SVG TS project.
However, because all materials have so far come from WG members, in fact
it is somewhat covered by the WG patent and IP policies.
- DOM. Defined in the DOM TS Process
document, with a pointer to the relevant grant of license wording.
This wording is also sent to any party submitting tests using the
required mailing mechanism for submitting tests, as outlined in the DOM TS Process
document.
- XML. There is no QA Process document per se. It is unclear whether the
topic has been explicitly dealt with.
Guidelines document
description of checkpoint.
Checkpoint 5.4. In QA Process document, define review
procedures for submitted test materials.
[Priority 2]
- SVG. Chapter
6 of the Manual gives review criteria for and and all tests. But
there are no procedures, just criteria.
- DOM. Defined in the DOM TS Process
document.
- XML. There is no QA Process document per se. All tests are reviewed by
the WG members, who run the tests.
Guidelines document
description of checkpoint.
Checkpoint 5.5. Test materials review process includes but not
limited to checking for accuracy, scope and clarity of the tests.
[Priority 1]
- SVG. Chapter
6 of the Manual addresses exactly these topics. Additionally,
checklist has been generated, for use in reviewing test cases.
- DOM. Defined in the DOM TS Process
document.
- XML. The review process and criteria are not defined in any TS
documents, but the criteria of this checkpoint are, de facto, what is
applied during test review.
Guidelines document description of
checkpoint.
Guideline 6. QA
Process: Plan test materials publication.
DOM discussion
Plans for test material publication are discussed internally in the DOM TS
group, and then decided by the DOM TS Group in coordination with the DOM
WG.
SVG discussion
The SVG TS project has explicitly discussed and planned the key aspects of
TS publication. CVS repository is used for the ongoing development of the TS,
and at agreed milestones, a snapshot is taken and published under the W3C
Document License. The TS home page links all of the materials and
documentation, and also points to an implementation matrix that publishes the
TS results of a half-dozen or so implementations.
XML discussion
Publication of the TS is being discussed by the WG. The TS will reside on
the W3C web space and will be freely and publicly available. In general, the
publication details of the XML TS are relatively invisible outside of the WG,
because there has not been any publication yet.
Checkpoints
Checkpoint 6.1. Ensure a secure and reliable repository location
for future test materials.
[Priority 2]
- SVG. SVG1 TS is maintained in a CVS repository on a server of a member
company. SVG2 TS is still early in organization and assembly, and not
committed to CVS yet.
- DOM. The DOM TS uses W3C CVS to store
all files.
- XML. The XML TS is kept in an XML-TS section of
the W3C CVS.
Guidelines document description
of checkpoint.
Checkpoint 6.2. In the QA Process document, define the
licenses applicable to published test materials.
[Priority 2]
- SVG. SVG has no QA Process document per se, but the license was
deliberated carefully (in WG email) and the W3C Document License chosen.
The license is linked prominently throughout the test materials.
- DOM. The grant of license which is included in the published tests is
not given in the DOM TS Process document.
- XML. The publication license details are unknown. Test materials in the
CVS repository contain the copyright provisions of the contributors.
Guidelines document
description of checkpoint.
Checkpoint 6.3. In the QA Process document, describe how the
test materials will be published and point to the corresponding
webpage.
[Priority 2]
- SVG. SVG has no QA Process document per se, but the publication process
has been explicitly discussed and decided (in email and meetings). The
CVS archive is WG-member only, and snapshots are taken and published at
agreed document milestones. The published TS location and documention are
linked from the TS home
page.
- DOM. There is a web space for the DOM TS which is pointed to from the
DOM TS Process
document.
- XML. Logistical details of publication are under discussion within the
XML WG.
Guidelines document
description of checkpoint.
Checkpoint 6.4. Provide a disclaimer regarding the use of the
test materials for compliance verification.
[Priority 2]
- SVG. Although such a disclaimer is not attached to the test materials,
on the other hand the WG has recognized and articulated the principle in
the Status section of SVG
1.0 (see fourth paragraph).
- DOM. Relevant wording is included in the DOM TS Process
document.
- XML. There is such a disclaimer in the test suite materials (@@need
link, or need quote).
Guidelines document description
of checkpoint.
Checkpoint 6.5. In the QA Process document, describe how test
results for the products can be published.
[Priority 3]
- SVG. SVG TS has not addressed this explicitly (there is no QA Process
document per se), however it does maintain a selective implementation
matrix linked from TS
home page.
- DOM. Not done in the process document.
- XML. This detail is under discussion within the XML WG.
Guidelines document
description of checkpoint.
Guideline 7. Plan the transfer of test materials to W3C if
needed.
DOM discussion
This issue was addressed early on in the specification phase of the DOM TS
work, as a choice had to be made between integrating an existing TS or just
using the existing one as a primer for its succesor. The latter was
preferred.
This is addressed in the DOM TS Process
document.
SVG discussion
This guideline is not applicable to the SVG TS -- all TS development so
far has occurred within the SVG WG.
XML discussion
The issue was addressed prior to the transfer of the TS to XML WG, and
prior to the start of TS development and maintenance there. Since that phase
is finished, it is no longer applicable. Included in the discussions were the
quality of the OASIS/NIST XML Test Suite, amount of work needed to correct
and improve it, as well as IPR questions.
Checkpoints
Checkpoint 7.1. If transfer of the test materials to W3C is
planned, perform an assessment of their quality.
[Priority 2]
- SVG. This and the other checkpoints are not applicable -- all TS
development took place within the SVG WG.
- DOM. Not in process document, done by the DOM TS Group.
- XML. This is being done with the OASIS/NIST test suite, to complete the
transfer process and prepare for first W3C publication.
Guidelines document
description of checkpoint.
Checkpoint 7.2. If transfer of the test materials to W3C is
planned, identify sufficient staff resources to meet the requirements.
[Priority 2]
- SVG. This and the other checkpoints are not applicable -- all TS
development took place within the SVG WG.
- DOM. Not applicable
- XML. This was done prior to the TS transfer from OASIS to W3C, in
discussions involving W3C (XML and QAWG), OASIS, and QAWG. It is
accordingly on record in XML WG mail archive (member only).
Guidelines document
description of checkpoint.
Checkpoint 7.3. If transfer of the test materials to W3C is
planned, resolve IPR questions and reach agreement with the external party
that produced test materials.
[Priority 1]
- SVG. This and the other checkpoints are not applicable -- all TS
development took place within the SVG WG.
- DOM. Not applicable, the DOM TS was launched in coordination between
the W3C and NIST, NIST donated the
first bulk of tests.
- XML. This was done in discussions involving OASIS, NIST, W3C, and is
accordingly on record in the XML WG mail archive.
Guidelines document description
of checkpoint.
Guideline 8. Plan
for test materials maintenance.
DOM discussion
No clear plan for test material maintenance has been made, other than
indicating in the process document that these issues will be adressed by the
DOM TS group.
Not applicable
SVG discussion
[@@ to be done]
XML discussion
No clear plan for test material maintenance has been made yet. The need to
address this has been acknowledged by the WG, with plans to address it in the
near future.
Checkpoints
Checkpoint 8.1. Maintain contribution and review
procedures throughout test materials' and standard's entire life
cycles.
[Priority 3]
- SVG. Because the TS process is basically the WG process, this happens
de facto -- there is no explicit plan, but it happens as a natural part
of the WG's specification development and ongoing TS development.
- DOM. Done by the DOM TS Group.
- XML. It is the XML WG intention to do this.
Guidelines document
description of checkpoint.
Checkpoint 8.2. In the Working Group's QA process document,
specify a procedure for updates of the test materials to track new
errata/specification versions.
[Priority 2]
- SVG. SVG has no QA Process document per se, nor is this explicitly
described in any of the WG or TS standing documents. It is informally
handled as part of normal WG process concurrent with the evolution of
specification versions and the processing of specification errata.
- DOM. Relevant wording in the DOM TS Process
document.
- XML. XML has no QA Process document per se, nor is this documented in
WG standing documents or archives. It will de facto be handled as part of
the WG process.
Guidelines document
description of checkpoint.
Checkpoint 8.3. In the Working Group's QA process document,
identify the communication channel and procedure for appeals of tests
validity.
[Priority 2]
- SVG. SVG has no QA Process document per se, but disputes of a tests
validity are handled as part of the normal WG consensus process. There is
a public mail list, svg-testsuite-comments@w3.org, for public comment on
the tests.
- DOM. Relevant wording in the DOM TS Process
document.
- XML. XML has no QA Process document per se, nor is this documented in
WG standing documents or archives. Any disputes, disagreements are
brought to the attention of the WG and put on the Issues List for
discussion.
Guidelines document
description of checkpoint.
5. General remarks
[@@Issue. Should we have a section like this? "Assessment" or "Lessons
learned" or "..."? It would draw general lessons from the three examples,
assess the effectiveness and even *goodness* of the approaches, compare and
contrast, critique, etc. If "yes" ... can we postpone it to next publication,
and just go w/ examples for FPWD? (Don't think there's time to develop it
properly, and it could be sensitive, particularly if it contains
"critique".)]
[@@As just proposed, the third bullet below would be in the scope of the
section, but the first two probably wouldn't.]
Experiences drawn from previous efforts, obvious pitfalls, good examples,
open issues, and so forth.
DOM discussion
There are a series of issues with the DOM TS:
- No proprietary mechanism is used for producing the tests. They are
represented using an XML grammar, then generated using transforms (one
per binding)
- Tests are validated against the schema which is generated directly from
the specification (using transforms). Thus, sanity and coverage is easy
to track.
- Producing the DOM TS could have been made quicker if there was a clear
schedule for submitting tests to the framework. Now an initial bulk of
tests was used and translated into the language-neutral description
format.
SVG discussion
[@@to be done@@]
XML discussion
[@@to be done@@]
6. Conformance
There is no concept of conformance to this document. Rather, conformance
is measured relative to the checkpoints of the companion document, "QA Framework: Operational Guidelines" [QAF-OPS]. This document relates real-world examples to
the checkpoints' requirements, and also presents techniques by which the
checkpoints may be satisfied. In that sense, it defines conformance criteria
for the conformance requirements (checkpoints) of the operational guidelines
document.
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C)
- Dimitris Dimitriadis (Ontologicon)
- Karl DuBost (W3C)
- Peter Fawcett (RealNetworks)
- Kirill Gavrylyuk (Microsoft)
- Dominique Hazael-Massieux (W3C)
- Lofton Henderson (CGM Open)
- Ian Jacobs (W3C)
- David Marston (Lotus Development Corp)
- Jack Morrison (Sun Microsystems)
- Lynne Rosenthal (NIST)
- Mark Skall (NIST)
- Andrew Thackrah (Open Group)
- Olivier Thereaux (W3C)
- ISSUES-LIST
- QA Activity Issues List, maintained by the QA Working Group, available at http://www.w3.org/QA/WG/#issues.
- MATRIX
- W3C-wide conformance activity survey covering all the WGs, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
- PROCESS
- W3C Process Document, 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
- QA-GLOSSARY
- Comprehensive glossary of QA terminology. (Under construction.)
- >QAF-OPS
- "QA Framework: Operational Guidelines", Working Draft companion version to this document, available at [...].
- QAF-SPEC
- "QA Framework: Specification Guidelines", Working Draft companion version to this document, available at [...].
- QAF-TEST
- "QA Framework: Test Guidelines", not yet published.
- QAIG
- Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
- QAWG
- Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
- RFC2119
- Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
- SPEC-EXTECH
- "QA Framework: Specification Examples & Techniques", not yet published.
- STYLE-MAN
- "W3C Manual of Style", summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
- TAXONOMY
- QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
- TEST-EXTECH
- "QA Framework: Test Examples & Techniques", not yet published.
- W3C-TR
- Location of all published W3C technical reports, see http://www.w3.org/TR/.
- WCAG10
- Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.