Introduction
- Two graphical test suites
- SVG (at Candidate Recommendation, 4/2001):
- a basic but comprehensive set of viewer tests
- built from scratch within SVG Working Group
- started at "stable Working Draft" (1/2000)
- WebCGM (Recommendation, 1/1999):
- basic plus some detailed viewer tests
- built from adapted (NIST-ATA) plus new materials
- started 2 years after Recommendation
Components of the Suites
Both test suites (SVG/WebCGM) consist of:
- XML test case description database
- set of test case instances (.svg/.cgm files)
- matching set of raster reference images (expected results)
- set of "operator scripts" -- instructions, pass/fail, test
descriptions
- test harness to organize, coalesce, and present suite & results
- Miscellaneous production and managment tools
(Refer to test suites for examples of each. To be presented with
slides.)
Goals of the Two Suites
Top Problems and Issues
- Both projects short-cut traceability.
- Specs (Recommendations) not always conducive to testability.
- normative language; optional features; organization;
complexity.
- Visual comparison (w/ reference image) is imprecise and error
prone.
- Uniformity and accountability difficult working within WG.
- Overall, these projects (in several aspects) are very labor
intensive.
See the workshop
paper for discussion of additional problems and issues.
Main Conclusions
- Improve specifications (Recs) themselves -- guidelines and standards
needed.
- Early conformance work confers big benefit to quality of specifications
(Recs)
- Start some conformance work (e.g., TR/TP, some BE, etc) in WG early in
spec life.
- Realistic plan (acknowledging resource needs) for finishing work.
- Address in WG charter: what, how, when, by whom conformance will be
done.
- Attention to more automation methods and tools, especially:
- TR/TP extraction and synthesis; traceability implementation.
- comparison of actual and expected results
See the workshop
paper for additional issues, lessons, observations.
Note about Accessibility
- Harnesses
- the dynamic JavaScript harness (WebCGM) is more powerful and
flexible
- the static-linked page set of SVG is more accessible
- both are driven off of similar XML databases, so both can be
provided
- Operator Scripts
- terse pass/fail criteria of existing (NIST-ATA) WebCGM tests --
designed for certification operators -- are not very accessible
- the style of the SVG BE conformance suite provides, essentially,
detailed alternative text to the TC pictures and reference
images.