The QA Handbook (QAH) is a non-normative handbook about the
process and operational aspects of the quality
practices of W3C's Working Groups (WG). It is intended for Working
Group chairs and team contacts, to help them to avoid known pitfalls and to
benefit from experiences gathered from the W3C Working Groups themselves. It
provides techniques, tools, and templates that should facilitate and
accelerate the work of the WGs. This document is one of the QA Framework
family of documents of the Quality Assurance
(QA) Activity, which
includes the other existing or in-progress specifications:
Specification Guidelines; and, Test Guidelines.
Warning ... this is an Editor's draft of QAH. It has had some QAWG
review & discussion. Lots in it (but not everything yet) reflects QAWG
consensus. There are open issues. There are many incomplete (tbd)
bits.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. A list of current
W3C publications and the latest revision of this technical report can be
found in the W3C technical reports index
at http://www.w3.org/TR/.
This document is a W3C Working Draft (WD), made available by the W3C
Quality Assurance (QA) Activity for discussion by W3C members and other
interested parties. For more information about the QA Activity, please see
the QA Activity
statement.
This is the first published Working Draft of this document. This draft
accurately reflects the redesign of the QA Framework (QAF) resolved by QA
Working Group (QAWG) at its 2004 Tech Plenary
face-to-face. The QA Handbook (QAH) replaces and
incorporates the best features of the former QA Framework:
Introduction and QA Framework:
Operational Guidelines.
Although accurate in overall content, this draft is incomplete in some
details, especially in the examples -- lots more examples are intended. Also,
some links to intended references are yet to be done (generally indicated by
"@@"). Because this is a significant departure from the last-published QAF,
QAWG wants public review and comment on the overall direction as soon as
possible. Details will be fleshed out in future publications.
The QA Handbook will be coordinated with the other parts of the QA
Framework, Specification Guidelines and Test
Guidelines. Synchronization at uniform level of maturity will occur no
later than Last Call. The first publications of the various parts will be
somewhat more independent.
The QAWG does not expect this document to become a Recommendation. Rather,
after further development, review and refinement, it will finally be
published and maintained as a WG Note.
Publication as a Working Draft does not imply endorsement by the W3C
Membership. This is a draft document and may be updated, replaced or
obsoleted by other documents at any time. It is inappropriate to cite this
document as other than work in progress.
The QA
Working Group Patent Disclosure page contains details on known patents
related to this specification, in conformance with W3C policy
requirements.
Comments on this document may be emailed to www-qa@w3.org, the publicly archived list
of the QA Interest Group
[QAIG]. Commenters
please note that your comments will be publicly archived and
available, do not send information that should not be distributed, such as
private data.
[Ed note. As before, it will be
auto-generated. Manual skeleton looks like:]
- Introduction & roadmap
- [...several subsections...]
- Early planning and commitment
- General modus operandi
- Test development framework and processes
- Life after WG -- maintenance
- Day-to-day QA operations
- Licensing & branding
- Acquiring test materials
- Appendix -- Orientation to QA
Framework
Two separate external appendices to this document ... QAPD template and Charter
template.
Key to highlighted styles in this draft
- "Issue:" or "Question" or "Action",
and styled like this means an issue to be discussed.
- [in square brackets and
styled like this, means editor's notes about the outline.]
- "vague-flag" [suppressed now] highlights class of phrases like "QA process", "quality
practices", etc
Here are five use cases for The QA Handbook (QAH). Told as
stories, they show when and how QAH could be helpful to chairs and staff
contacts. They cover five different situations that might typically arise
over the life of the WG.
The QA Handbook could be useful to chairs and staff contacts
at Charter time...
Story 1: Think about quality early
The Charter-drafting team of a new Working Group, led by staff contacts and
co-Chairs, looks through The QA Handbook. On the advice of the
early-commitment Good Practice bits, the team
debates how much it can realistically commit to at this early stage. After
deciding on some test materials deliverables, milestones, and spec
synchronization that it considers to be safe commitments, it uses the
provided Charter Template to write it into the draft Charter.
Or it could be a big help to a chair who is trying to jump-start a test
suite project...
Story 2: Jump-starting a testing
effort
An existing Working Group has just finished writing its Requirements and Use
Cases documents, and has begun to draft its specification. At the same time,
it is taking a first look at its test suite (TS) plans. As recommended in the The QA
Handbook, the Chair jump starts the TS project by appointing a
QA Moderator and part-time TS team,
whose first output is a quick QA Process Document (QAPD) using the provided
QAPD template.
Or maybe the chairs/staff contacts are pondering the steps to transfer a
test suite from an external entity...
Story 3: Test-suite transfer
A Working Group has re-chartered to finish a Second Edition of its
specification, and to develop the next functional version, 2.0. The group did
not develop a test suite (TS) during
its first charter, but a collaboration of outside organizations and an
industry consortium has developed one. The Chair and staff contacts have
negotiated agreement in principle to transfer it to a stable home in W3C. In
The QA Handbook, they find a handy
checklist of preliminary steps, requirements, and specific activities for
a smooth transfer.
Or maybe they need to take pre-emptive action due to looming possible
license-issue hassles...
Story 4: Test suite licensing
A Working Group is almost ready to request Candidate Recommendation (CR), and has gotten a
comprehensive test suite (TS) together
for CR's trial implementation period. As the Chair starts to arrange for
publication of the TS, she finds the WG split on issues around TS distribution licenses to use. Consulting
The QA Handbook, she finds discussion of the pros and cons of the W3C licenses (Software License and
Document License), and advice on resolving an optimal licensing strategy.
Or maybe they can borrow the experience of other W3C WGs for various
useful and common test suite processes...
Story 5: Test development processes
A Working Group has built and released a basic test suite (TS) for its specification. A staff contact has
been given the Action Item to plan its expansion to a more comprehensive TS,
by leveraging and integrating the large test collections of several early
implementors. Rather than figure out the issues and write a Test Contribution
& Review process from scratch, he looks at the summary
advice in The QA Handbook. QAH points him both
to some useful templates, and to more detailed stuff in QA Framework:
Test Guidelines, significantly shortening completion of his action
item.
WG Chairs and staff contacts are overworked. They share a lot of the same
concerns and pressures as specification
editors.
Common problems are:
- not enough (motivated) staff;
- stuff being done too late;
- licenses and legal hassles.
A lot of groups have faced these issues, and there is a growing body of
experience in W3C about what works and what doesn't. QAH aims to pull this
together.
The QA Handbook (QAH) is intended for: chairs and staff
contacts of W3C Working Groups. These include the process-savvy and novices
alike.
QAH is a non-normative handbook about the process and operational aspects
of the quality practices of WG life -- mostly
test suite stuff, but also some specification quality assurance. QAH aims to
flag avoidable pitfalls, and to provide techniques, tools, and templates that
will facilitate and accelerate the work of the WGs.
Because there is no normative content in QAH, conformance to QAH is not an
issue. QAH takes this approach for several reasons, chiefly:
- because of the diversity amongst W3C's working groups, it is difficult
(but not impossible) to craft simple normative rules for all;
- as past experience has shown, imposing such rules in diverse,
people-centric processes is often not well received -- too authoritarian
and fierce.
Advice is presented in the imperative voice in Principle and
Good Practice sections. The aim is to present helpful guidance
gathered from the experience of W3C WGs. If it looks helpful, follow it. If
not, don't -- there's no consequence.
Therefore, the QAH deals with things that impact these aspects of WGs'
test activities and other quality practices:
- people resources;
- deliverables and their schedules;
- interaction with W3C and the world;
- W3C process, licenses, IPR, etc.
Editors, test builders, and general WG membership will find advice
specific to their roles in the other parts of the QA
Framework:
- @@QA Framework: Specification Guidelines@@
- @@QA Framework: Test Guidelines@@
Chairs and staff contacts will likely also get involved with these
eventually. A brief orientation to the rest of the QA Framework
is provided in an appendix to this
document.
After this orientation section, QAH contains four topical modules:
Each presents advice and guidance about some part of the WG's test and quality activities. These tend to have a
chronological correspondence to phases (rec-track stages) of the WG's
activities. Earlier is better, but later is better than never. Here's a
graphic depiction of when, in a WG's life, the various QAH topics are ideally
dealt with and applicable:
[TBD. here is planned to be embedded a chopped
down version of the OpsGLchronology diagram]
Why care? It's not always easy to
anticipate Working Group (WG) needs during the rush and pressure of
Chartering, or in the busyness of early WG life. But the earlier the WG can
accurately identify and commit to its test and other quality related
deliverables, the less likely that the WG get big surprises later on, or run
into problems that will delay its progress towards Recommendation.
Additional reason -- binding decisions about WG membership (e.g., numbers
per company) often are made at Charter time.
What does this mean? In practice, this
means attention to a handful of points, which we enumerate as Good
Practices.
Good Practice: Decide ASAP -- will the
Working Group build test materials or acquire them?
Clearly, it is going to make a big difference in a Working Group's (WG)
staffing requirements -- building test materials tends to be labor intensive
(extremely so for some types of specifications). Even if the WG imports them,
some staff resources will have to be applied (see last
module). The particular test-related activities and milestones in the
WG's programme of work will in general be completely different for develop
versus acquire options.
Good Practice: Think about and
enumerate the QA activities &
deliverables that might help the Working Group through the
Recommendation track. Minimally, commit to assuring the timely existence of
test materials.
Different kinds of QA deliverables might
include:
- basic test suites
- validators
- test assertions enumeration
- one or more spec (SpecGL) quality reviews
- comprehensive test materials (beyond basic coverage)
- [other suggestions?]
Test materials are by far the most common QA
deliverable, and likely will be a key to quick and painless passage
through Candidate Recommendation (CR).
Good Practice: Synchronize QA deliverables with specification milestones, and
for the bigger QA deliverables, define and schedule intermediate milestones
if possible.
This advice echoes that in Tips for Getting to Recommendation
Faster, for example see item #6 in section 2.
Good Practice: Consider whether the
Working Group should tie any quality criteria
to Rec-track advancement.
For example, finishing test materials deliverables before requesting
Candidate Recommendation (CR) is common, in order to facilitate CR's
Implementation Assessment. (Counter-example: as in the above Story, if the WG is still building them during
CR, it is likely to uncover things that will oblige it to cycle in CR.)
This good practice takes the synchronization of the previous one a step
further -- the specification cannot advance unless the committed test or other quality criteria are met.
How
to Organize a Recommendation Track Transition talks some about the
role of test suites in advancement decisions.
Good Practice: Put some thought into
how to staff the WG's QA plans.
The earlier this is done, the more options will be available. Some options
include:
- dedicated team from Working Group (WG) membership;
- part of everyone's time;
- Consider: asking in Call for Participation (CfP) for extra people who
would specifically be tasked to work on QA
deliverables.
The third option isn't really different from the first. It's just a way of
doing it. But notice that it's an option that is only available by looking
into these questions at Charter time.
By the way, there has been confusion (@@find the chairs archive
reference@@) about "W3C Process only allows each company to have two members
on the WG". In fact, that is not from W3C Process. W3C Process gives
considerable freedom for this to be tailored to WG needs -- W3C
Process says it may be specified in the Charter. So, for example: two per
company in technical discussion, issue resolution, voting, etc; and,
additional dedicated test suite builders allowed.
By the way, this is another good reason to put some thought into test
suite plans and other quality-related deliverables as early as possible.
Tips for Getting to
Recommendation Faster (section 3) also talks some about the value
of (early) staffing decisions.
Examples:
- [Ed note. initially get stuff
from OpsGL/ET and Intro]
- [Ed note. during reviews,
explicitly solicit good examples and counter-examples]
How can I do it?
- The Charter template provides for a
superset of the things that one might want to specify and commit to early
(at Charter time). Use it for those items that are deemed appropriate to
write it into the WG's Charter.
- Having advocated "earlier is better", now is a good time for QAH to
acknowledge that Charter time may not be appropriate for each and every
of the above Good-Practice recommendations, for every WG. In some cases,
it might be premature to commit to that level of detail at that early
stage of the WGs life.
- [@@other ideas?@@]
[Editor's overview of what went
into this module: implemented QAWG's general consensus (see f2f minutes) that
old-CP1.4
was the important one -- it becomes the Principle -- and the other CPs of GL1
& 2 are details (become Good Practices). CP1.5
(Rec-track criteria), CP2.1
(internal or external?), CP2.3
(request alloc of people for QA) are worth mentioning and survived as details
-- "Good practice" or "Recommended practice" or ..." CP2.2
(staff commitments) survives. Dropped CP1.1
(A/AA/AAA to Ops, Spec, Test). CP1.2
(commit to TM) and CP1.3
(commit to complete TM) seem like details -- should become minimal bottom
line of some good practice or whatever. See also checklist.]
Why care? Over and over we hear the
message from successful test suite projects -- define the work process early:
test dev't framework, issues list, email list, faq, contribution & review
process, etc. [Examples TBD: DOM, Xquery, maybe
SVG...] Conversely, there are examples of disorganized collections of
test cases with no one apparently in charge, no way to find their status,
correctness, [@@Example: what was KD's example of
"can't find anyone who's responsible for amorphous undocumented heap of
"tests"?]
The W3C Process
Document clearly organizes the way that a WG's specs are progressed
through the Rec track. But there is no similar template for the other aspects
of a WG's life, particularly aspects related to all-important test and quality processes -- logistical setup,
communications with the outside, licensing and branding policy, test
development process, etc.
Good Practice: Put all of the Working
Group's important test and QA-related
information in one place in a QA Process Document (QAPD).
How can I do it? Simple: Use the QAPD template. It will guide the user through
everything needed, and then some. It is not only a template, but also a
checklist of sorts, for the sort of things that the Working Group (WG) should
consider having in its QA Process Document. The QAPD template has been
assembled as the union of good practices seen in real QAPDs of W3C Working
Groups.
In the past, before the construction of the QAPD template, test efforts
would often copy-edit a QAPD from another effort. There isn't really any
reason to do this anymore. Here are some examples from which the QAPD
template has been assembled:
- @@example1@@
- @@example2@@
- ...etc...
What does it mean? In practice, it means (easiest
solution) producing a QAPD recording at least those details of the WG's test- and QA-related work processes that are
outlined in the QAPD template and briefly discussed in the following
subsections.
[TBD. This section should have
lots of examples]
See the corresponding section in
QAPD template.
Good Practice: Identify a Working
Group point-of-contact about test materials or other QA-related business [was
CP4.1]
This can be a special appointed "QA Moderator". Or it might be the WG
Chair (or a co-chair), or staff contact. The important part is that there be
a single identified point-of-contact to whom others can address questions,
contributions, etc, and who will be accountable for responding.
See @@QAPD subsection@@.
Here are a couple @@Example(s)@@.
Good Practice: Specify an archived
email list to use for QA-related
communications (test suite questions, bug reports, contributions,
etc). [was CP4.4]
It can be a dedicated "Test" list. Or it can be a public IG list. Or it
can be a public comment list separate from the WG list, in the case that
there isn't an IG. Because of the variety of public/private mixes amongst
W3C's WGs, "one size fits all" does not work well here.
Here are a couple @@Example(s)@@.
See @@QAPD subsection@@.
Good Practice: Identify Web page(s)
for test suites, announcements, and other QA
deliverables. [was CP4.4]
Obviously, this needs to be publicly accessible. Doing it all in the
public portion of the WGs Web space is one way to achieve that. In
particular, that provides a good, secure repository location for test
materials [was CPx.y].
Some WGs [@@Ex: SVG] have two locations actually -- a private CVS
repository for the in-development test materials, and a simplified public
repository for periodic public releases of test materials.
Here are a couple @@Example(s)@@.
See @@QAPD subsection@@.
The Working Group's QAPD is also a handy central location to record its
licensing policies and (any) branding policies. These are sufficiently
important that they have their own QAH module, which
addresses:
- license terms applicable to test materials, both submitted and
published [was CP5.4, CP6.2];
- any branding policy -- logos, conformance icons, etc -- and associated
rules and restrictions [was CP4.5]
If the Working Group (WG) is developing test materials itself, then there
are several topics associated with TM development process are going to impact
its operations and management. These are described, along with
Principle and Good Practice guidance, in some detail in
@@QA Framework: Test Guidelines@@. These topics, which might
conveniently be documented in the WG's QAPD, include definitions of:
Finally, as a part of planning about
"life after WG", the Working Group (WG) will need to decide what happens to
both its test materials and associated processes. These maintenance-related
topics are described in some detail in @@TestLite@@, but again the
information might conveniently be located in the WG's QAPD:
Success criteria:
[Overview of what appears in this module:old-CP4.1
(qa moderator); dropped CP4.2
(appoint a task force); *** CP4.3
(QAPD) is central principle***; CP4.4
(email & web); CP5.4,
CP6.2
(license stuff) and CP4.5
(branding) are recommended here to be documented in QAPD, but otherwise there
is a forward reference to IANAL/license module for details; CP6.1
(repository) rolled into Web site stuff as a detail; CP5.1,
CP5.3,
CP5.5
(TM-related) are recommended here to be documented in QAPD, but otherwise
@@TestLite@@ is referenced for details; CP6.3,
CP6.5
(TM-publication-related) are recommended here to be documented in QAPD, but
otherwise @@TestLite@@ is referenced for details. CP8.1,
CP8.2,
CP8.3
are about TM maintenance -- are recommended here to be documented in QAPD,
but otherwise @@TestLite@@ is referenced for details. For reference, see the
old OpsGL checklist.]
Cartoon?[Anyone have any good
lawyer cartoons/jokes?]
[TBD. in CR? (find out from MB who this is --
e.g., Xquery? -- and what the details are].
Why care? Get it right early, or it may
stall the Working Group's Rec-track progress indefinitely. While this might
seem to be a routine piece of Day-to-day
operations, it has proved to be sufficiently troublesome within W3C that
it deserves to be a Principle by itself.
What does this mean? There are two kinds
of licensing issues: submission license, and publication license. Both of
them can be problematic and can interrupt the WG's progress on Rec-track if
not early addressed and carefully handled. Test materials license and other
IPR issues are the subject of ongoing debate and discussion within W3C, but
there are some tactics to minimize potential problems.
Good Practice: As early as possible,
get WG consensus and define acceptable license terms for submission of test
materials'. [wasCP5.4]
W3C currently has a more or less routine submission license, Contribution
of Software or Test Materials to W3C. By early attention, it
should become clear whether any potential sources require custom terms,
before possible disputes can impact the WG's deliverables and schedules.
Cases are known where potential submitters would not accept the standard
terms, and custom terms had to be negotiated.
- @@example@@ [need someone to contribute one. JR
is gone.]
Good Practice: As soon as the nature
of the Working Group's test materials becomes clear, get consensus and define
license terms for publication of the test materials. [wasCP6.2]
Any W3C-hosted materials must have approved license and use terms.
Experience has shown that there is no single license that is appropriate for
all test materials, so the WG needs to address this after it has come to an
understanding of the structure, content, and intended use of its test
materials.
Currently approved W3C licenses that may be applied to test materials are
the @@Document License@@ and the @@Software License@@. The Document license
has two characteristics that are attractive for test materials:
- It prohibits the test materials from being modified by the user upon
download, therefore guaranteeing the integrity of the test materials;
- It requires that if the test materials are copied or redistributed,
they must contain the link to the original test materials and their
status.
On the other hand, there are situations in which the Document License is
inappropriate, because (for example) some Test Materials require modification
or completion in order to apply them.
In some cases, one license may seem appropriate for some parts of Test
Materials, but the other license better for different parts. Test Materials
might contain some mixture of these components: test software, test
documentation, and test cases (see @@TestLite?@@). A careful look at contents
and use cases may reveal that different licenses to different components is
the best solution.
If the WG considers that neither the Document nor the Software License is
applicable, not even piecewise as just described, it should consult with W3C
Legal.
Good Practice: Consider whether to
have brands, logos, or conformance icons associated with the Working Group's
test materials. Define associated policies. [was CP4.5]
[TBD] Points to develop about logos policy: terms for valid application,
conformance claims, challenge procedures, policing & enforcement policy,
etc. Some of these start to border on significant legal questions.
[TBD] Reference to and discussion about W3C Logo
and Icon Usage, and maybe also SpecLite material about Conformance
Claims.
- The decision ultimately belongs to the Working Group. [TBD. Actually,
we can go through a lot of the things about Doc vs. SW, applying
different licenses to different bits, etc]
- There are places in QAPD template
to document the WG's decisions, about licenses as well as
branding/logos.
- Don't forget to make license stuff obvious and visible in all
deliverables (and all their bits and pieces)!
Examples:
[TBD. Should be easy to come up with a handful of
good examples. Especially for brands/logos.]
[Overview of what went into this module: all the
useful stuff associated with CP5.4
(submission licenses) and CP6.2
(publication licenses); CP4.5
(branding policy) and references to W3C branding policy. See also checklist.]
The big-three problems are:
- IPR issues [was CP7.3]
- staff resources [was CP7.2]
- quality of the candidate materials [was
CP7.1]
The IPR and staff issues are similar in concept to what the Working Group
faces if builds build the TM, but probably lesser in degree. A pre-transfer
quality assessment might seem unique to the acquire option, but the actual
steps will probably look similar to a test case contribution/review process
in a build-your-own scenario (see @@QA Framework: Test
Guidelines@@).
Good Practice: Do a quality assessment
of candidate test materials before going any further.
Basic things that a good quality assessment might cover would for example
include: clarity of scope, traceability, atomicity, user documentation,
etc.
A more comprehensive list of things that an assessment process could cover
might include: correctness, traceability, atomicity, user documentation,
maintainer documentation, declaration of scope, completeness (vis a' vis
declared scope), harnesses or interfaces for application of the TM,
reconfigurability, results assessment, results recording & reporting,
automation features, versioning/errata support, declaration of publication
licenses, integrated submission procedures, etc."
QA Framework: Test Guidelines deals with this topic in much
more detail, including (planned) templates and assessment aids.
Good Practice: Ensure there are
adequate staff to support the transferred test materials.
A Test Materials (TM) manager is still needed, but total staff resources
ought to be considerably less than build-your-own scenarios. With luck, the
original TM manager of the external TM source might become a WG member after
the test materials are transferred.
Good Practice: Sort out IPR issues
with the external party that produced the test materials.
How can I do it?
This is a virtualization of an actual transfer
scenario that QA helped to moderate. It could serve as a checklist of
steps to consider for Working Group's taking the acquire route.
Legend: EG the external group or entity;
QAWG the QA Working Group; TM the test
materials; WG the Working Group acquiring the test
materials.
- Before the transfer, WG with the help of QAWG:
- performs an assessment of the quality of candidate TM (by WG,
QAWG)
- identifies and commits to a set of test-related deliverables from
the candidate TM. These could be: releases, appeal/challenge
processes, maintenance plan, submission/review process, Web site,
mail list, etc. (by WG)
- identifies sufficient staffing, including at least a TM manager.
Recommendation: recruit the TM manager from the EG (if one
exists) to become a WG member after the transfer. (by WG)
- makes the decision to seek/accept the transfer. (by WG)
- (potentially) initiates Charter amendment (by WG, QAWG may
consult), if the TM acquisition doesn't fit within the current
Charter.
- During the transfer:
- EG and W3C reach agreement to transfer the TM (by WG, QAWG, EG)
- WG and EG perform the actual transfer of the TM, WG creates an
initial repository (by WG, EG)
- After transfer, initial test development/framework process setup:
- WG appoints a TM manager.
- The TM manager creates a QA Process Document for WG (by WG, TM
manager, QAWG may consult)
- set up the TM home page, a TM issues mailing list (by WG, TM
manager)
- determine the appropriate W3C IPR license (by WG, QAWG)
- First W3C public release of the new TM:
- make any needed enhancements prior to the first public release: fix
known/reported errors, produce documentation (by WG), W3C license
labelling, etc.
- announce the first public release of TM (by TM moderator,
Communications Group)
- joint W3C/EG press release (by WG, QAWG, Communications Group,
EG)
- After the first public release, the TM enter the maintenance phase (see
@@QA Framework: Test Guidelines@@).
[Overview of what went into this module:CP7.1,CP7.2,CP7.3(transferring a
suite). See alsochecklist. [tbd] Flesh it out
more? per the OpsGL & OpsET content for those checkpoints.]
Audiences for the QA Framework parts
The following list identifies the parts of the The QA
Framework (QAF) that might interest those who fill the various roles
within a Working Group (WG). While each part of
QAF is targets itself at a specific principal audience, the various parts
might have somewhat broader interest and applicability.
- all WG participants
- For any (potential) WG
participant, the early planning and
commitment parts of The QA Handbook might provide
helpful context for understanding what the WG has committed to deliver.
Familiarity with the Specification Guidelines will be
helpful to any participant who actively participates in the advancement
of the WG's specifications to
Recommendation.
- WG spec editors and
authors
- As for all WG participants,
The QA Handbook might be interesting, for shedding some
light on the context in which the WG is operating. A good working
understanding the Principles and Good Practices of Specification
Guidelines, together with its collected examples, tools, and
templates, should be a valuable resource in choosing document
structure, formats, and techniques that will facilitate production of a
high-quality specification.
- WG chair
- As the person ultimately responsible for all aspects of the WG's
work, a familiarity with the guidance for operations and process of
The QA Handbook (QAH) should be helpful -- Chairs and
staff contacts are the principal intended audience of QAH. Some
familiarity with the advice and guidance of Specification
Guidelines and Test Guidelines should be helpful as
well, as the Chairs ultimately oversees both the advancement of the
WG's specifications and the
WG's test materials
projects.
- WG-TS
participant
- Those who are active in building the test materials of the WG should benefit from reading the
guidance for test materials in Test Guidelines, and from
its associated examples, techniques, and tools. Because of the close
dependency of test materials on the functional specifications, a
familiarity with the Specification Guidelines could be
useful as well.
- WG-TS
moderator
- The person who manages the WG's QA projects should have working
understandings of guidance and techniques for specifications of
Specification Guidelines, as well as the test materials
guidelines, techniques and examples of Test Guidelines. In
addition, the test-related process and logistical advice of Test
Guidelines should prove useful.
- non-WG spec reviewers
- Whether from other WGs, or
the public at large, the Specification Guidelines will be
helpful to those who review a WG's specifications, by providing some
objective metrics by which to measure the specifications.
- non-WG TM reviewers
- Whether from other WGs, or
the public at large, reviewers of a WG's test materials of a WG would benefit from a familiarity
with Test Guidelines. Its guidance and examples should facilitate a
critical review of a WG's test materials.
- Reviewers of Activity proposals & charters
- For those W3C Members who will be reviewing Activity proposals and
proposed WG charters, and
helping to form their Advisory Committee Representative's positions,
the early planning and commitment
parts of The QA Handbook might be helpful in
evaluating whether or not the WG's attention test and quality
deliverables is appropriate and consistent with the WG's overall
programme of work.
- QA Activity participants
- Participants in the QAWG
are an expert resource for the W3C Working Groups, and accordingly
should be expert on all parts of the QA Framework;
participants in the QA Interest Group (QAIG) need familiarity with all
parts as well, in order to effectively render some of the QAIG's
chartered deliverables.
QA Framework Primer
Issue: we decided in 20040414 telecon to transfer this
from QAF-Intro to an appendix. Its editing is in-progress.
Simplified & edited to appropriately relate to QAF-Lite, do we reaffirm
that this should be here? 20040426 -- tentatively yes. Maybe simplify to a
checklist of sorts?
This chapter, in the style of a primer, introduces the use of the QA
Framework document family. It progresses through significant milestones in a
WG's life, from writing a Charter
through publishing Recommendations, and looks at associated test suite and
other quality practice activities.
First Step -- QA Commitment
Because QA is properly an integral part of the activities of each WG, each WG has to consider and commit to a set of QA
deliverables appropriate to its work. A spectrum of possibilities are
discussed and illustrated in the early planning and
commitment module of The QA Handbook.
If a WG is being newly formed,
and if the WG is able to anticipate and agree at Charter time on deliverables
like test suites, then it should consider documenting those QA deliverables
in its Charter, just as it does all other WG deliverables. Again, see the early planning and commitment section of The QA
Handbook. A WG being
re-chartered is a similar case to a newly formed WG, although the scope and direction of its
work might be clearer.
For an already-chartered Working Group undertaking new QA projects, if
these deliverables are not documented in the Charter already, then there are
a couple of options. The W3C Process describes how to amend a Charter to
accommodate significant new deliverables, if the WG wishes to take this
route.
Set Up Processes and Logistics
Once the Working Group (WG) is
off and running, and assuming that it has planned on some test- or other
quality-related deliverables, the next step is to chose and document the
processes and logistics that it will use for its QA activities. These include
such typical details as:
- QA moderator (single point-of-contact);
- "/Test/" home page in the WG's Web
space;
- -ts email discussion list;
- WG process document for
QA;
- decisions about IPR and test
materials.
The sections within the Day-to-Day operations
module of The QA Handbook give good-practice advice about how to
do this, plus examples and a handy template for writing a QA Process
Document.
Planning and Writing the Specification
There is a tight bond between how the specification (Recommendation) is
written on the one hand, and on the other hand its testability and its
suitability as a basis for interoperable implementations.
New specification. QA Framework: Specification Guidelines
should be applied from very beginning. Among the key topics that it addresses
are:
- conformance policy & clause;
- normative language throughout;
- writing with test assertions;
- optional features, discretion, extensions;
- profiles, levels, conformance claims.
Consider the advice of QA Framework: Specification Guidelines
even at the stage of planning the structure and presentation style of the
spec. Along with W3C "pubrules" and W3C Manual of Style, spec
authors and editors should refer to the spec guidelines throughout their
work, on topics like testable language, clarity, conciseness.
New Edition of specification. A new Edition of the same functional level
of a specification is typically used for incorporation of errata (e.g.,
XML 1.0 Second Edition). Normally, this should not be considered
a good time to align a specification to QA Framework: Specification
Guidelines -- the changes associated with such alignment could
significantly disrupt and restructure the specification.
New Version of specification. A new Version of the specification refers to
a significant functional change and enhancement. This presents a good
opportunity to improve the testability and implementability of the
specification, as just described for new specifications.
Reviewing and Progressing the Specification
This stage in the specifications life has two significant aspects:
- the advancement of the specification to and beyond the first public
working draft (FPWD);
- and, the need to synchronize production of test suites and tools more
closely with the spec progression.
When the specification is published in TR space, then non-WG W3C Members and the general public begin
to review and comment. Such reviewers should consult and understand the
material in Specification Guidelines [QAF-SPEC], in order to have an informed set of evaluation
criteria for the conformance, testability, and interoperability aspects of
the specification.
Working Group (WG) participants
and especially WG-TS
participants should refer to the sections (checkpoints) of the operational
guidelines, regarding specification exit criteria ([QAF-OPS], checkpoint 1.5) and synchronization of the test materials with specification
progression ([QAF-OPS], Guideline 3).
The project enters The Matrix [MATRIX] at Last Call Working Draft (if not sooner).
Additionally, a de-facto process convention is emerging, that there should be
significant conformance test materials when exiting Candidate Recommendation
and commencing Proposed Recommendation. This is the same timing as the
explicit process requirement of two interoperable implementations.
Designing and Building Test Materials
There are several scenarios for how the Working Group (WG) "builds" its conformance test
materials:
- design and build test cases and test tools in the WG, from scratch;
- import completed test materials from an outside group or
organization;
- assemble the test materials from contributed test cases and
materials.
Intra-WG build. Before starting
the development, the WG-TS participants should be
thoroughly familiar with the material in Test Guidelines [QAF-TEST]. There is useful information for
both high-level planning -- e.g., breadth-first Basic Effectivity versus
fully detailed suite? -- as well as specific detail for the building of
individual test cases. Another aspect of building test materials is an
acceptance procedure for the individual bits, as they are built. This is
addressed in the review-procedures requirements ([QAF-OPS], checkpoint 5.4) of the
operational guidelines.
Import completed test materials. Several high-quality test suites have
been developed outside of the relevant W3C WG, and then transferred to the WG. WGs which are considering such a transfer
should refer to the test materials transfer guideline ([QAF-OPS], Guideline 7) of the operational guidelines.
Clearly, the quality of the candidate test materials should be carefully
assessed, and for this the Test Guidelines [QAF-TEST] can provide useful assessment criteria.
Assemble contributions. Some test suites have been built by
implementing processes to assemble significant chunks of material from
outside (or internal member) contributions. Operational
Guidelines [QAF-OPS] addresses the
steps needed to complete such a transfer -- they are the same as the
preceding paragraph about transferring completed test materials. In addition,
there should be careful quality assessment of contributions, for which
Test Guidelines [QAF-TEST] can be useful.
Finally, there must be procedures for submission, review, and integration of
contributions ([QAF-OPS], checkpoints
5.2, 5.4), which are described in several related checkpoints of the
operational guidelines.
Publication of Test Materials
Typically, a Working Group
TS group will want to publish releases of test materials,
particularly as the specification advances through its final maturity levels
(e.g., Proposed Recommendation) towards Recommendation. Test material
publication is addressed in TM publication
checkpoint ([QAF-OPS], checkpoint
6.3) of the operational guidelines.
As soon as the test materials become public, then there is definite need
for a procedure to process challenges to correctness, make determinations,
and appeal decisions. This is addressed in a test appeal checkpoint ([QAF-OPS], checkpoint 8.3) of the operational guidelines.
Publication of test materials often comprises an implicit (or explicit)
invitation for contributions. The considerations described in "Assemble Contributions" are
equally applicable here.
Specification Publication and Beyond
When the specification reaches Recommendation, there is typically a
concurrent publication of the test materials. This might be considered a
"final" publication, or ongoing development may still be planned according to
one of the mechanisms discussed above. In any case, a maintenance procedure
must be in place for the test materials. Firstly, there are tie-ins between
approved specification errata and validity or applicability of particular
tests -- mechanisms for this are discussed in Test Guidelines [QAF-TEST], and the general process
overview is discussed in the errata update checkpoint ([QAF-OPS], checkpoint 8.2) of the operational guidelines.
Secondly, there is the above-discussed need for both challenge/review/appeal
processes. Finally, even if Working Group TS active
development of test materials ceases, it may be desired to continue to
consider submissions, and review and integrate them per the requirements of
the test-contribution checkpoint ([QAF-OPS], checkpoint 8.1) of the operational guidelines.
Life after WG
It is possible that the Working Group (WG) and WG-TS may disband after its
charter expires. This situation, and what to do about test materials, is
discussed in a "secure repository" checkpoint ([QAF-OPS], checkpoint 6.1) of the operational guidelines.
[...tbd...]
[...tbd...]
[...tbd...]