The QA Handbook (QAH) is a non-normative handbook about the
process and operational aspects of the quality
practices of W3C's Working Groups (WG). It is intended for Working Group chairs
and team contacts, to help them to avoid known pitfalls and to benefit from
experiences gathered from the W3C Working Groups themselves. It provides
techniques, tools, and templates that should facilitate and accelerate the
work of the WGs. This document is
one of the QA Framework family of documents of the Quality Assurance (QA)
Activity, which includes the other existing or in-progress
specifications: Specification Guidelines; and, Test
Guidelines.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. A list of current
W3C publications and the latest revision of this technical report can be
found in the W3C technical
reports index at http://www.w3.org/TR/.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity
statement.
This is the First Public Working Draft of this document. This draft
accurately reflects the redesign of the QA Framework (QAF) resolved by the QA Working Group (QAWG) at its 2004 Technical Plenary
face-to-face. The QA Handbook (QAH) replaces and incorporates the best
features of the former QA Framework:
Introduction and QA Framework:
Operational Guidelines.
This draft is accurate in overall content. However it is incomplete in
some details (generally indicated by "@@" or "TBD"), especially in the examples -- lots more
examples are intended. Also, some links to planned references are yet to be
done. Because this is a significant departure from the last-published
QAF, QAWG wants to get it out for review and
comment on the overall direction as soon as possible. The just-mentioned
incomplete details will be fleshed out in the next publication.
The QA Handbook will be coordinated with the other parts of
the QA Framework, QA Framework: Specification Guidelines and
QA Framework: Test Guidelines. Synchronization at uniform level
of maturity will occur no later than Last Call. The first publications of the
various parts will occur somewhat more independently.
The QAWG does not expect this
document to become a Recommendation. Rather, after further development,
review and refinement, it will finally be published and maintained as a
WG Note.
Publication as a Working Draft does not imply endorsement by the W3C
Membership. This is a draft document and may be updated, replaced or
obsoleted by other documents at any time. It is inappropriate to cite this
document as other than work in progress.
The QA
Working Group Patent Disclosure page contains details on known patents
related to this specification, in conformance with W3C policy
requirements.
Comments on this document may be emailed to www-qa@w3.org, the publicly archived list
of the QA Interest Group
(QAIG).
Commenters please note that your comments will be publicly
archived and available, do not send information that should not be
distributed, such as private data. Please give any feedback by 12 June 2004,
so that it may be considered by QAWG at its next face-to-face meeting.
- Introduction & roadmap
- Early planning and commitment
- Day-to-day QA operations
- Licensing & branding
- Acquiring test materials
- Appendix -- Orientation to QA Framework
- Acknowledgments
- References
- Change History
This document has two separate external appendices, QAPD
template and Charter
template, which contain templates to help Working Group chairs and staff
contacts to implement the document's good practice guidelines.
Here are five use cases for The QA Handbook (QAH). Told as stories, they show when and
how QAH could be helpful to chairs
and staff contacts. They cover five different situations that might typically
arise over the life of the WG.
The QA Handbook could be useful to chairs and staff contacts
at Charter time...
Story 1: Think about quality early
The Charter-drafting team of a new Working Group, led by staff contacts and
co-Chairs, looks through The QA Handbook. On the advice of the
early-commitment Good Practice
bits, the team debates how much it can realistically commit to at this
early stage. After deciding on some test materials deliverables, milestones,
and spec synchronization that it considers to be safe commitments, it uses
the provided Charter Template to write it into the draft Charter.
Or it could be a big help to a chair who is trying to jump-start a test
suite project...
Story 2: Jump-starting a testing
effort
An existing Working Group has just finished writing its Requirements and Use
Cases documents, and has begun to draft its specification. At the same time,
it is taking a first look at its test suite (TS) plans. As recommended in the The QA
Handbook, the Chair jump starts the TS project by appointing a QA Moderator and part-time TS team, whose first output is a quick QA
Process Document (QAPD) using
the provided QAPD template.
Or maybe the chairs/staff contacts are pondering the steps to transfer a
test suite from an external entity...
Story 3: Test-suite transfer
A Working Group has re-chartered to finish a Second Edition of its
specification, and to develop the next functional version, 2.0. The group did
not develop a test suite (TS) during
its first charter, but a collaboration of outside organizations and an
industry consortium has developed one. The Chair and staff contacts have
negotiated agreement in principle to transfer it to a stable home in W3C. In
The QA Handbook, they find a handy checklist of preliminary steps, requirements, and
specific activities for a smooth transfer.
Or maybe they need to take pre-emptive action due to looming possible
license-issue hassles...
Story 4: Test suite licensing
A Working Group is almost ready to request Candidate Recommendation (CR), and has gotten a
comprehensive test suite (TS) together
for CR's trial
implementation period. As the Chair starts to arrange for publication of the
TS, she finds the WG split on issues around TS distribution licenses to use. Consulting
The QA Handbook, she finds discussion of the pros and cons of the W3C licenses (the Software
License and the Document
License), and advice on resolving an optimal licensing strategy.
Or maybe they can borrow the experience of other W3C WGs for various useful and common test suite
processes...
Story 5: Test development processes
A Working Group has built and released a basic test suite (TS) for its specification. A staff contact has
been given the Action Item to plan its expansion to a more comprehensive
TS, by leveraging and integrating the
large test collections of several early implementors. Rather than figure out
the issues and write a Test Contribution & Review process from scratch,
he looks at the summary advice in The
QA Handbook. QAH points him
both to some useful templates, and to more detailed stuff in QA
Framework: Test Guidelines [QAF-TEST] significantly shortening completion of his
action item.
WG Chairs and staff contacts are
overworked. They share a lot of the same concerns and pressures as specification
editors.
Common problems are:
- never enough motivated people to do the work;
- deliverables delivered late;
- licenses and legal hassles;
- too much to do, so little time
A lot of groups have faced these issues, and there is a growing body of
experience in W3C about what works and what doesn't. QAH aims to pull this together.
The QA Handbook (QAH) is intended for: chairs and staff
contacts of W3C Working Groups. These include the process-savvy and novices
alike.
QAH is a non-normative handbook
about the process and operational aspects of the quality practices of WG life -- mostly test suite stuff, but also
some specification quality assurance. Guidance in the QAH is supported by real-world stories and
examples. It aims to flag avoidable pitfalls, and to provide techniques,
tools, and templates that will facilitate and accelerate the work of the
WGs.
Because there is no normative content in QAH, conformance to QAH is not an issue. QAH takes this approach for several
reasons, chiefly:
- because of the diversity amongst W3C's working groups, it is difficult
(but not impossible) to craft simple normative rules for all;
- as past experience has shown, imposing such rules in diverse,
people-centric processes is often not well received -- too authoritarian
and fierce.
Advice is presented in the imperative voice in Principle and
Good Practice sections. The aim is to present helpful guidance
gathered from the experience of W3C WGs. If it looks helpful, follow it. If not,
don't -- there's no consequence.
The QA Handbook (QAH) is a practical guide for integrating
quality practices into the WG. The
main focus is on things that influence the development and implementation of
specifications and tests, especially:
- people resources;
- deliverables and their schedules;
- interaction with W3C and the world;
- W3C process, licenses, IPR, etc.
However, the scope is not limited to specifications and tests, but
includes the total environment in which the WG operates. The goal of QAH is to capture and share current good
practices, point out hazards along the way, and suggest new modes of
operation to help the WGs pursue
their missions in the most timely and effective ways.
Editors, test builders, and general WG membership will find advice specific to
their roles in the other parts of the QA Framework:
- QA Framework: Specification Guidelines [QAF-SPEC]
- QA Framework: Test Guidelines [QAF-TEST]
Chairs and staff contacts will likely also get involved with these
eventually. A brief orientation to the rest of the QA Framework
is provided in an appendix to
this document.
After this orientation section, QAH contains four topical modules:
Each presents advice and guidance about some part of the
WG's test
and quality activities. These tend to have a chronological
correspondence to phases (rec-track stages) of the WG's activities. Earlier is better, but
later is better than never. Here's a graphic depiction of when, in a WG's life, the various QAH topics are ideally dealt with and
applicable:
[TBD. Here
is planned to be embedded a chopped down version of the OpsGLchronology diagram]
Why care? It's not always easy to
anticipate Working Group (WG) needs
during the rush and pressure of Chartering, or in the busyness of early
WG life. But the earlier the
WG can accurately identify and
commit to its test and other quality related deliverables, the less likely
that the WG get big surprises later
on, or run into problems that will delay its progress towards
Recommendation.
Additional reason -- binding decisions about WG membership (e.g., numbers per company)
often are made at Charter time.
What does this mean? In practice, this
means attention to a handful of points, which we enumerate as Good
Practices.
Good Practice: Decide ASAP -- will the Working Group build
test materials or acquire them?
Clearly, it is going to make a big difference in a Working Group's
(WG) staffing requirements --
building test materials tends to be labor intensive (extremely so for some
types of specifications). Even if the WG imports them, some staff resources will
have to be applied (see last module). The
particular test-related activities and milestones in the WG's programme of work will in general be
completely different for develop versus acquire options.
Good Practice: Think about and
enumerate the QA activities &
deliverables that might help the Working Group through the
Recommendation track. Minimally, commit to assuring the timely existence of
test materials.
Different kinds of QA deliverables might
include:
- basic test suites
- validators
- test assertions enumeration
- one or more spec quality reviews (using Specification
Guidelines [QAF-SPEC])
- comprehensive test materials (beyond basic coverage)
- [other suggestions?]
Test materials are by far the most common QA
deliverable, and likely will be a key to quick and painless passage
through Candidate Recommendation (CR).
Good Practice: Synchronize QA deliverables with specification milestones, and
for the bigger QA deliverables, define and schedule intermediate milestones
if possible.
This advice echoes that in Tips for Getting to Recommendation
Faster [REC-TIPS] for example
see item #6 in section 2.
Good Practice: Consider whether the
Working Group should tie any quality criteria
to Rec-track advancement.
For example, finishing test materials deliverables before requesting
Candidate Recommendation (CR) is common, in order to
facilitate CR's
Implementation Assessment. (Counter-example: as in the above Story, if the WG is still building them during CR, it is likely to uncover things
that will oblige it to cycle in CR.)
This good practice takes the synchronization of the previous one a step
further -- the specification cannot advance unless the committed test or other quality criteria are met.
How
to Organize a Recommendation Track Transition [TRANSITION] talks some about the role of test
suites in advancement decisions.
Good Practice: Put some thought into
how to staff the WG's QA plans.
The earlier this is done, the more options will be available. Some options
include:
- dedicated team from Working Group (WG) membership;
- part of everyone's time;
- Consider: asking in Call for Participation (CfP) for extra people who
would specifically be tasked to work on QA
deliverables.
The third option isn't really different from the first. It's just a way of
doing it. But notice that it's an option that is only available by looking
into these questions at Charter time.
By the way, there has been confusion [TBD: find the chairs archive reference] about
"W3C Process only allows each company to have two members on the WG". In fact, that is not from W3C Process.
W3C Process gives considerable freedom for this to be tailored to WG needs -- W3C
Process says it may be specified in the Charter. So, for example, the
WG could decide on these rules:
allow two per company in technical discussion, issue resolution, voting, etc;
and, allow additional dedicated test suite builders.
By the way, this is another good reason to put some thought into test
suite plans and other quality-related deliverables as early as possible.
Tips for
Getting to Recommendation Faster [REC-TIPS] (section 3) also talks some about the value of
(early) staffing decisions.
Examples:
- [TBD. initially get some existing examples
from OpsGL/ET and Intro]
- [TBD. during reviews, explicitly solicit good
examples and counter-examples]
How can I do it?
- The Charter template
provides for a superset of the things that one might want to specify and
commit to early (at Charter time). Use it for those items that are deemed
appropriate to write it into the WG's Charter.
- Having advocated "earlier is better", now is a good time for QAH to acknowledge that Charter time
may not be appropriate for each and every of the above Good-Practice
recommendations, for every WG.
In some cases, it might be premature to commit to that level of detail at
that early stage of the WGs
life.
Why care? Over and over we hear the
message from successful test suite projects -- define the work process early:
test dev't framework, issues list, email list, faq, contribution & review
process, etc. [Examples TBD: DOM, Xquery, maybe SVG...] Conversely, there
are examples of disorganized collections of test cases with no one apparently
in charge, no way to find their status, correctness, [Example TBD: find
KD's counter-example of "can't find anyone who's responsible for amorphous
undocumented heap of "tests"?]
The W3C
Process Document clearly organizes the way that a WG's specs are progressed through the Rec
track. But there is no similar template for the other aspects of a WG's life, particularly aspects related to
all-important test and quality processes --
logistical setup, communications with the outside, licensing and branding
policy, test development process, etc.
Good Practice: Put all of the Working
Group's important test and QA-related
information in one place in a QA Process Document (QAPD).
How can I do it? Simple: Use the QAPD template. It will guide the
user through everything needed, and then some. It is not only a template, but
also a checklist of sorts, for the sort of things that the Working Group
(WG) should consider having in its
QA Process Document. The QAPD
template has been assembled as the union of good practices seen in real
QAPDs of W3C Working
Groups.
In the past, before the construction of the QAPD template, test efforts would often
copy-edit a QAPD from another
effort. There isn't really any reason to do this anymore. Here are some
examples from which the QAPD
template has been assembled:
- [TBD:
example1]
- [TBD:
example2]
- ...etc...
What does it mean? In practice, it means (easiest
solution) producing a QAPD
recording at least those details of the WG's test- and
QA-related work processes that are outlined in the QAPD template and briefly discussed in
the following subsections.
See the corresponding
section in QAPD
template.
Good Practice: Identify a Working
Group point-of-contact about test materials or other QA-related business.
This can be a special appointed "QA Moderator". Or it might be the
WG Chair (or a co-chair), or staff
contact. The important part is that there be a single identified
point-of-contact to whom others can address questions, contributions, etc,
and who will be accountable for responding.
See corresponding
QAPD subsection.
Here are a couple of TBD: Example(s).
Good Practice: Specify an archived
email list to use for QA-related
communications (test suite questions, bug reports, contributions,
etc).
It can be a dedicated "Test" list. Or it can be a public Interest Group
(IG) list. Or it can be a public
comment list separate from the WG
list, in the case that there isn't an IG. Because of the variety of public/private
mixes amongst W3C's WGs, "one size
fits all" does not work well here.
Here are a couple of TBD: Example(s).
See corresponding
QAPD subsection.
Good Practice: Identify Web page(s)
for test suites, announcements, and any other QA-related deliverables.
Obviously, this needs to be publicly accessible. Doing it all in the
public portion of the WG's Web space
is one way to achieve that. In particular, that provides a good, secure
repository location for test materials.
Some WGs [Example: @@SVG] have two locations
actually -- a private CVS repository for the in-development test materials,
and a simplified public repository for periodic public releases of test
materials.
Here are a couple of TBD: Example(s).
See corresponding
QAPD subsection.
The Working Group's QAPD is
also a handy central location to record its licensing policies and (any)
branding policies. These are sufficiently important that they have their own QAH
module, which addresses:
- license terms applicable to test materials, both submitted and
published;
- any branding policy -- logos, conformance icons, etc -- and associated
rules and restrictions.
[Ed TBD. The QAPD subsection references in this
section need to be refined more precisely.]
If the Working Group (WG) is
developing test materials itself, then there are several topics associated
with TM development process are
going to impact its operations and management. These are described, along
with Principle and Good Practice guidance, in some detail
in QA Framework: Test Guidelines [QAF-TEST]. These topics, which might conveniently be
documented in the WG's QAPD, include definitions of:
Finally, as a part of
planning about "life after WG", the
Working Group (WG) will need to
decide what happens to both its test materials and associated processes.
These maintenance-related topics are described in some detail in QA
Framework: Test Guidelines [QAF-TEST] , but again the information might conveniently
be located in the WG's QAPD:
- what happens to the test materials themselves?
- what happens to the WG's
contribution and review processes? [QAPD sub-section]
- how to track errata and (for example) new specification editions? [QAPD sub-section]
- what happens with bug reports about the test materials? [QAPD sub-section]
Success criteria:
TBD:
Cartoon?
Why care? Get it right early, or it may
stall the Working Group's Rec-track progress indefinitely. While this might
seem to be a routine piece of Day-to-day
operations, it has proved to be sufficiently troublesome within W3C that
it deserves to be a Principle by itself.
What does this mean? There are two kinds
of licensing issues: submission license, and publication license. Both of
them can be problematic and can interrupt the WG's progress on Rec-track if not early
addressed and carefully handled. Test materials license and other IPR issues are the subject of
ongoing debate and discussion within W3C, but there are some tactics to
minimize potential problems.
Good Practice: As early as possible,
get WG consensus and define
acceptable license terms for submission of test materials'.
W3C currently has a more or less routine submission license, Contribution
of Software or Test Materials to W3C [CONTRIB]. By early attention, it should become clear
whether any potential sources require custom terms, before possible disputes
can impact the WG's deliverables and
schedules. Cases are known where potential submitters would not accept the
standard terms, and custom terms had to be negotiated.
- [TBD
Example: need someone to contribute one. JR is gone.]
Good Practice: As soon as the nature
of the Working Group's test materials becomes clear, get consensus and define
license terms for publication of the test materials.
Any W3C-hosted materials must have approved license and use terms.
Experience has shown that there is no single license that is appropriate for
all test materials, so the WG needs
to address this after it has come to an understanding of the structure,
content, and intended use of its test materials.
Currently approved W3C licenses that may be applied to test materials are
the Document
License and the Software
License. The Document License has two characteristics that are attractive
for test materials:
- It prohibits the test materials from being modified by the user upon
download, therefore guaranteeing the integrity of the test materials;
- It requires that if the test materials are copied or redistributed,
they must contain the link to the original test materials and their
status.
On the other hand, there are situations in which the Document
License is inappropriate, because (for example) some Test Materials
require modification or completion in order to apply them.
In some cases, one license may seem appropriate for some parts of Test
Materials, but the other license better for different parts. Test Materials
might contain some mixture of these components: test software, test
documentation, and test cases (see Test Guidelines[QAF-TEST]?). A careful look at contents and
use cases may reveal that different licenses to different components is the
best solution.
If the WG considers that neither
the Document
License nor the Software
License is applicable, not even piecewise as just described, it should
consult with W3C Legal.
Good Practice: Consider whether to
have brands, logos, or conformance icons associated with the Working Group's
test materials. Define associated policies.
[TBD] Points to
develop about logos policy: terms for valid application, conformance claims,
challenge procedures, policing & enforcement policy, etc. Some of these
start to border on significant legal questions.
[TBD] Reference to
and discussion about W3C Logo
and Icon Usage, and maybe also SpecLite material about Conformance
Claims.
- The decision ultimately belongs to the Working Group. [TBD. Actually, we can go through a lot of
the things about Doc vs. SW, applying different licenses to different
bits, etc]
- There are places in QAPD template to document the
WG's decisions, about licenses
as well as branding/logos.
- Don't forget to make license stuff obvious and visible in all
deliverables (and all their bits and pieces)!
Examples:
[TBD. Should
be easy to come up with a handful of good examples. Especially for
brands/logos. WAI, W3C content validators, etc]
The big-three problems are:
- IPR issues
- staff resources
- quality of the candidate materials
The IPR and staff
issues are similar in concept to what the Working Group faces if builds build
the TM, but probably lesser in
degree. A pre-transfer quality assessment might seem unique to the acquire
option, but the actual steps will probably look similar to a test case
contribution/review process in a build-your-own scenario (see QA
Framework: Test Guidelines [QAF-TEST]).
Good Practice: Do a quality assessment
of candidate test materials before going any further.
Basic things that a good quality assessment might cover would for example
include: clarity of scope, traceability, atomicity, user documentation,
etc.
A more comprehensive list of things that an assessment process could cover
might include: correctness, traceability, atomicity, user documentation,
maintainer documentation, declaration of scope, completeness (vis a' vis
declared scope), harnesses or interfaces for application of the TM, reconfigurability, results assessment,
results recording & reporting, automation features, versioning/errata
support, declaration of publication licenses, integrated submission
procedures, etc."
QA Framework: Test Guidelines [QAF-TEST] deals with this topic in much more detail,
including (planned) templates and assessment aids.
Good Practice: Ensure there are
adequate staff to support the transferred test materials.
A Test Materials (TM) manager is
still needed, but total staff resources ought to be considerably less than
build-your-own scenarios. With luck, the original TM manager of the external TM source might become a WG member after the test materials are
transferred.
Good Practice: Sort out IPR issues with the external
party that produced the test materials.
How can I do it?
This is a virtualization of an actual
transfer scenario that QA helped to moderate. It could serve as a
checklist of steps to consider for Working Group's taking the
acquire route.
Legend: EG the
external group or entity; QAWG the QA Working Group;
TM the test
materials; WG the
Working Group acquiring the test materials.
- Before the transfer, WG with
the help of QAWG:
- performs an assessment of the quality of candidate TM (by WG, QAWG)
- identifies and commits to a set of test-related deliverables from
the candidate TM. These
could be: releases, appeal/challenge processes, maintenance plan,
submission/review process, Web site, mail list, etc. (by WG)
- identifies sufficient staffing, including at least a TM manager. Recommendation:
recruit the TM manager from
the EG (if one exists) to
become a WG member after the
transfer. (by WG)
- makes the decision to seek/accept the transfer. (by WG)
- (potentially) initiates Charter amendment (by WG, QAWG may consult), if the TM acquisition doesn't fit within
the current Charter.
- During the transfer:
- EG and W3C reach
agreement to transfer the TM (by WG, QAWG, EG)
- WG and EG perform the actual transfer of
the TM, WG creates an initial repository (by
WG, EG)
- After transfer, initial test development/framework process setup:
- WG appoints a TM manager.
- The TM manager creates a
QA Process Document for WG
(by WG, TM manager, QAWG may consult)
- set up the TM home page,
a TM issues mailing list
(by WG, TM manager)
- determine the appropriate W3C IPR license (by
WG, QAWG)
- First W3C public release of the new TM:
- make any needed enhancements prior to the first public release: fix
known/reported errors, produce documentation (by WG), W3C license labelling, etc.
- announce the first public release of TM (by TM moderator, Communications
Group)
- joint W3C/EG press
release (by WG, QAWG, Communications Group,
EG)
- After the first public release, the TM enter the maintenance phase (see
QA Framework: Test Guidelines [QAF-TEST].
The following list identifies the parts of the The QA
Framework (QAF) that might
interest those who fill the various roles within a Working Group (WG). While each part of QAF is targeted itself at a specific principal
audience, the various parts might have somewhat broader interest and
applicability.
- all WG participants
- For any (potential) WG
participant, the early planning and
commitment parts of The QA Handbook might provide
helpful context for understanding what the WG has committed to deliver.
Familiarity with the Specification Guidelines [QAF-SPEC] will be helpful to any
participant who actively participates in the advancement of the
WG's specifications to
Recommendation.
- WG spec editors and
authors
- As for all WG participants,
The QA Handbook might be interesting, for shedding some
light on the context in which the WG is operating. A good working
understanding the Principles and Good Practices of Specification
Guidelines [QAF-SPEC],
together with its collected examples, tools, and templates, should be a
valuable resource in choosing document structure, formats, and
techniques that will facilitate production of a high-quality
specification.
- WG chair
- As the person ultimately responsible for all aspects of the WG's work, a familiarity with the
guidance for operations and process of The QA Handbook
(QAH) should be helpful --
Chairs and staff contacts are the principal intended audience of
QAH. Some familiarity with
the advice and guidance of Specification Guidelines [QAF-SPEC] and Test
Guidelines [QAF-TEST]
should be helpful as well, as the Chairs ultimately oversees both the
advancement of the WG's
specifications and the WG's
test materials projects.
- WG-TS
participant
- Those who are active in building the test materials of the WG should benefit from reading the
guidance for test materials in Test Guidelines [QAF-TEST], and from its associated
examples, techniques, and tools. Because of the close dependency of
test materials on the functional specifications, a familiarity with the
Specification Guidelines [QAF-SPEC] could be useful as well.
- WG-TS
moderator
- The person who manages the WG's QA projects should have working
understandings of guidance and techniques for specifications of
Specification Guidelines [QAF-SPEC], as well as the test materials
guidelines, techniques and examples of Test Guidelines [QAF-TEST]. In addition, the
test-related process and logistical advice of Test
Guidelines [QAF-TEST]
should prove useful.
- non-WG spec reviewers
- Whether from other WGs, or
the public at large, the Specification Guidelines [QAF-SPEC] will be helpful to those
who review a WG's
specifications, by providing some objective metrics by which to measure
the specifications.
- non-WG TM reviewers
- Whether from other WGs, or
the public at large, reviewers of a WG's test materials of a WG would benefit from a familiarity
with Test Guidelines [QAF-TEST]. Its guidance and examples should
facilitate a critical review of a WG's test materials.
- Reviewers of Activity proposals & charters
- For those W3C Members who will be reviewing Activity proposals and
proposed WG charters, and
helping to form their Advisory Committee Representative's positions,
the early planning and commitment
parts of The QA Handbook might be helpful in
evaluating whether or not the WG's attention test and quality
deliverables is appropriate and consistent with the WG's overall programme of work.
- QA Activity participants
- Participants in the QAWG
are an expert resource for the W3C Working Groups, and accordingly
should be expert on all parts of the QA Framework;
participants in the QA Interest Group (QAIG) need familiarity with all
parts as well, in order to effectively render some of the QAIG's
chartered deliverables.
[TBD. Could
this section maybe be simplified further, turned into a checklist of sorts?
]
[TBD. This
section will need close coordination especially withTest Guidelines, when that
is first published.]
This chapter, in the style of a primer, introduces the use of the QA
Framework document family. It progresses through significant milestones in a
WG's life, from writing a Charter
through publishing Recommendations, and looks at associated test suite and
other quality practice activities.
Because QA is properly an integral part of the activities of each WG, each WG has to consider and commit to a set of QA
deliverables appropriate to its work. A spectrum of possibilities are
discussed and illustrated in the early planning
and commitment module of The QA Handbook.
If a WG is being newly formed,
and if the WG is able to anticipate
and agree at Charter time on deliverables like test suites, then it should
consider documenting those QA deliverables in its Charter, just as it does
all other WG deliverables. Again,
see the early planning and commitment
section of The QA Handbook. A WG being re-chartered is a similar case to a
newly formed WG, although the scope
and direction of its work might be clearer.
For an already-chartered Working Group undertaking new QA projects, if
these deliverables are not documented in the Charter already, then there are
a couple of options. The W3C Process describes how to amend a Charter to
accommodate significant new deliverables, if the WG wishes to take this route.
Once the Working Group (WG) is
off and running, and assuming that it has planned on some test- or other
quality-related deliverables, the next step is to chose and document the
processes and logistics that it will use for its QA activities. These include
such typical details as:
- QA moderator (single point-of-contact);
- "/Test/" home page in the WG's Web
space;
- -ts email discussion list;
- WG process document for
QA;
- decisions about IPR and test
materials.
The sections within the Day-to-Day
operations module of The QA Handbook give good-practice
advice about how to do this, plus examples and a handy template for writing a
QA Process Document.
There is a tight bond between how the specification (Recommendation) is
written on the one hand, and on the other hand its testability and its
suitability as a basis for interoperable implementations.
New specification. QA Framework: Specification Guidelines [QAF-SPEC] should be applied from very
beginning. Among the key topics that it addresses are:
- conformance policy & clause;
- normative language throughout;
- writing with test assertions;
- optional features, discretion, extensions;
- profiles, levels, conformance claims.
Consider the advice of QA Framework: Specification Guidelines
[QAF-SPEC] even at the stage of planning
the structure and presentation style of the spec. Along with W3C "pubrules"
and W3C Manual of Style, spec authors and editors should refer
to the spec guidelines throughout their work, on topics like testable
language, clarity, conciseness.
New Edition of specification. A new Edition of the same functional level
of a specification is typically used for incorporation of errata (e.g.,
XML 1.0 Second Edition). Normally, this should not be considered
a good time to align a specification to QA Framework: Specification
Guidelines [QAF-SPEC] -- the
changes associated with such alignment could significantly disrupt and
restructure the specification.
New Version of specification. A new Version of the specification refers to
a significant functional change and enhancement. This presents a good
opportunity to improve the testability and implementability of the
specification, as just described for new specifications.
This stage in the specifications life has two significant aspects:
- the advancement of the specification to first public working draft
(FPWD) and
beyond;
- and, the need to synchronize production of test suites and tools more
closely with the spec progression.
When the specification is published in TR space, then non-WG W3C Members and the general public begin
to review and comment. It would be valuable that such reviewers consult and
understand the material in QA Framework: Specification
Guidelines [QAF-SPEC] -- it gives
and informed set of evaluation criteria about the conformance, testability,
and interoperability aspects of the specification.
Working Group (WG) participants
and especially WG-TS
participants should refer to the good-practice pieces about advancement criteria and synchronization (between
specs and test materials) of The QA Handbook. Projects enter
The Matrix [MATRIX] at Last
Call Working Draft (if not sooner). A de-facto process convention is
emerging, that there should be significant conformance test materials before
finishing Candidate Recommendation phase. This timing coordinates with the
explicit process requirement of two interoperable implementations.
There are several scenarios for how the Working Group (WG) "builds" its conformance test
materials:
- design and build test cases and test tools in the WG, from scratch;
- import completed test materials from an outside group or
organization;
- assemble the test materials from contributed test cases and
materials.
Intra-WG build. Before starting
the development, the WG-TS participants would
benefit from a familiarity with the material in QA Framework: Test
Guidelines [QAF-TEST]. There is
useful information for both high-level planning -- e.g., does the WG want breadth-first Basic Effectivity or a
fully detailed suite? -- as well as specific details for building the
individual test cases. Another aspect of building test materials is an
acceptance procedure for the individual bits, as they are built. This is
addressed in the review-procedures guidance of QA
Framework: Test Guidelines [QAF-TEST].
Import completed test materials. Several high-quality test suites have
been developed outside of the relevant W3C WG, and then transferred to the WG. WGs which are considering such a transfer
should refer to test materials
acquisition module of The QA Handbook. Clearly, the quality
of the candidate test materials should be carefully assessed, and for this
the Test Guidelines can provide useful assessment criteria.
Assemble contributions. Some test suites have been built by
implementing processes to assemble significant chunks of material from
outside (or internal member) contributions. Again, The QA
Handbook test materials
acquisition module addresses the steps needed to complete such a transfer
-- they are the same as the preceding paragraph about transferring completed
test materials. In addition, there should be careful quality assessment of
contributions, for which QA Framework: Test Guidelines [QAF-TEST] can be helpful.
Typically, a Working Group
TS group will want to publish releases of test materials,
particularly as the specification advances through its final maturity levels
(e.g., Proposed Recommendation) towards Recommendation. Test material
publication is addressed in QA Framework: Test Guidelines [QAF-TEST].
One hurdle on the way to publication is legal -- deciding and agreeing on
suitable publication licenses. Advice on navigating this potential quagmire
is presented in the licensing module of The
QA Handbook.
As soon as the test materials become public, then there is definite need
for a procedure to process challenges to correctness, make determinations,
and appeal decisions. Test appeal process advice is presented in QA
Framework: Test Guidelines [QAF-TEST].
Publication of test materials often comprises an implicit (or explicit)
invitation for contributions. The considerations described in "Assemble Contributions" are
equally applicable here.
When the specification reaches Recommendation, there is typically a
concurrent publication of the test materials. This might be considered a
"final" publication, or ongoing development may still be planned according to
one of the mechanisms discussed above. In any case, a maintenance procedure
must be in place for the test materials. Firstly, there are tie-ins between
approved specification errata and applicability of particular tests --
mechanisms for this are discussed in QA Framework: Test
Guidelines [QAF-TEST]. Secondly,
there is the above-discussed need for both challenge/review/appeal processes.
Finally, even if the Working Group ceases active TS development of test materials, it may want to
continue to consider submissions, and review and integrate them per the
test-contribution advice of QA Framework: Test Guidelines [QAF-TEST].
It is possible that the Working Group (WG) and WG-TS may disband after its
charter expires. The various aspects of this situation are introduced in The QA Handbook, and dealt with in somewhat
more detail in QA Framework: Test Guidelines [QAF-TEST].
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Dimitris Dimitriadis (Ontologicon),
- Karl DuBost (W3C),
- Dominique Hazael-Massieux (W3C),
- Lofton Henderson (CGM Open),
- David Marston (Lotus Development Corp),
- Patrick Curran (Sun Microsystems),
- Lynne Rosenthal (NIST),
- Mark Skall (NIST),
- Andrew Thackrah (Open Group),
- Olivier Thereaux (W3C).
- [PROCESS]
- World Wide Web
Consortium Process Document, I. Jacobs, Ed., 05 February
2004, available at http://www.w3.org/2004/02/Process-20040205/ .
- [CONTRIB]
- Contribution
of Software or Test Materials to W3C, which defines
W3C-approved procedures and terms for submission of Software and Test
Materials, available at
http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231
.
- [DOM
Working Group TS]
- DOM
TS Process Document, D.
Dimitriadis, Ed., 15 January 2002, available at
http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115 .
- [MATRIX]
- QA Matrix...blah...blah...
- [PROTOCOL-WG-TS]
- The XML Protocol WG has a TS process document, available at
http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a contribution/submission
license (example of a submission legal notice), available at
http://www.w3.org/2001/10/test-materials-license.html .
- [REC-TIPS]
- Tips
for Getting to Recommendation Faster, a public part of the
(member-only) W3C
Guidebook.
- [QA-GLOSSARY]
- A comprehensive glossary of QA terms, maintained by the QA Working
Group. (Incomplete
version under construction at
http://www.w3.org/QA/glossary .)
- [QAF-SPEC]
- QA Framework: Specification Guidelines, L. Rosenthal,
Ed., future W3C Working Draft companion to this document, May-June 2004
anticipated first publication. When published, it will be a
light-weight revision of the previous (Candidate Recommendation)
Specification Guidelines collection of documents.
- [QAF-TEST]
- QA Framework: Test Guidelines, P. Curran, D.
Dimitriadis, Eds, future W3C Working Draft companion to this document,
May-June 2004 anticipated first publication. When published, it will be
a light-weight revision of the previous (Candidate Recommendation)
Test Guidelines collection of documents.
- [QAWG]
- Quality Assurance
Working Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/WG/ .
- [SCHEMA-WG-TS]
- The XML Schema WG has a TS process document, available at
http://www.w3.org/2001/05/xmlschema-test-collection.html, and a contribution/submission
license (example of a submission legal notice), available at
http://www.w3.org/2001/05/test-materials-license.html .
- [TRANSITION]
- How to Organize a
Recommendation Track Transition, a part of the Member Guide
(member-only), is available at http://www.w3.org/2004/02/02-transitions
.
- 2004-05-10, First Published Working Draft
- Combines the best features of the former QA Framework:
Introduction and QA Framework: Operational
Guidelines into a lightweight, non-normative handbook for W3C
Working Group chairs and staff contacts.