Editor's draft: Last edited $Date: 2011/12/16 02:59:23 $
This document defines a minimal set of metadata elements that can usefully be applied to test cases that are intended for publication within a conformance test suite. Defining and providing metadata has proved helpful in a variety of areas during the test development process. It also helps those who run the tests to verify the conformance of their implementations. For example, metadata can be used to:
If the use of standard metadata elements is adopted within the W3C it is likely that standardized tools will be developed to facilitate many of these tasks.
Please note that these metadata elements are simply suggestions. It is likely that additional metadata elements may be appropriate for your implementation. It may even make sense to define metadata about metadata (for example, the Dublin Core elements Date and Language may usefully be applied to several of the elements defined below.)
Name | Identifier |
Description | An unambiguous reference to the test case. |
Rationale | Enables the test case to be referenced and identified. |
Required? | Yes. |
Syntax | Freeform (URI recommended). |
Example | http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm |
See Also | Dublin Core:
identifier Testcase Metadata: Title |
Comments | Most W3C test suites use URIs as identifiers. |
Name | Title |
Description | The name by which the test case is formally known. |
Rationale | It may be advantageous to provide a more human-oriented alternative to the Identifier. |
Required? | No. |
Syntax | Text. |
Example | The CSS Test Suite uses the term Description for human readable names that more properly match this Title element. (For example, CSS 2.1 Test Suite: Co>.) |
See Also | Dublin Core: title Testcase Metadata: Identifier |
Comments | This element is optional because the Identifier will often be sufficient. Note also that in very large test suites it may be impractical to define unambiguous Titles. |
Name | Purpose |
Description | A brief explanation of the reason the test case was developed. |
Rationale | Provides additional context for the test case. |
Required? | Yes. |
Syntax | Free-form (hypertext). |
Example | The CSS1 test suite provides such explanations. For example: "This page tests the 'text-transform' property of CSS1". |
See Also | Testcase Metadata: Description Testcase Metadata: SpecRef |
Comments | The Purpose should typically be kept short enough (a "one liner") to be displayed in tablular format. More detailed explanation can be provided in a Description element. |
Name | Description |
Description | A representation in words of the nature and characteristics of the test case. |
Rationale | A detailed explanation can help reviewers to understand the purpose of
a test and may also provide those executing the test with information that will be helpful if the test fails. |
Required? | No. |
Syntax | Free-form (hypertext). |
Example | The WebCGM test suite incorporates detailed descriptions. For example: "Test basic CGM-to-HTML link, from a Application Structure with linkuri APS Attribute within the CGM, to a whole HTML document, with no behaviors or fragments associated with the link." |
See Also | Dublin Core:
description Testcase Metadata: Purpose |
Comments | This element can be used to provide more detailed information than can be provided in the relatively short Purpose element. |
Name | Status |
Description | One of an enumerated list of values that can be used to track the state of a test at a given time. |
Rationale | It can be helpful to track the status of a test case during the development process. |
Required? | No. |
Syntax | Constrained choice from an enumerated list |
Example | One of: unconfirmed, new, assigned, pending, accepted, rejected, holding (see Conformance Test Process For WCAG 2.0 and their Test Case Status document.). |
See Also | |
Comments | This metadata element may also be useful after test suite publication, when it could be used to record the state of challenges to the validity of the test case. |
Name | SpecRef |
Description | Identification of the portion of the specification tested by this test case. |
Rationale | Traceability back to the specification. |
Required? | Yes. |
Syntax | Freeform hypertext or (ideally) an XPath expression. |
Example | The HTML 4.1 test suite provides a list of Testable Assertions that associates individual test cases with assertions in the specification. |
See Also | |
Comments | The more specific this reference is the better. (Simply pointing to the beginning of a large sub-section of the spec is not helpful. Identifying the exact string containing the requirement to be tested is ideal.) |
Name | Preconditions |
Description | Conditions that must be met before this test is executed. |
Rationale | Any such conditions must be understood and met before the test case can be successfully executed. |
Required? | No. |
Syntax | Freeform (hypertext). |
Example |
It might be necessary that a network connection be available or a server process be running before the test case is executed. |
See Also | Testcase Metadata: Inputs |
Comments | This element does not refer to parameters or data required for execution (see Inputs). |
Name | Inputs |
Description | Parameters or data that are needed for the test execution. |
Rationale | Must be understood and supplied for test execution. |
Required? | No. |
Syntax | Implementation dependent. |
Example | The XML Query Test Suite metadata defines an input_file element. |
See Also | Testcase Metadata: Preconditions |
Comments |
Name | ExpectedResults |
Description | The results that a conformant implementation is expected to produce when this test case is executed. |
Rationale | If these are not defined, it will be impossible to determine whether the test case passed or failed. |
Required? | Yes. |
Syntax | Implementation dependent. |
Example | Test cases within the CSS Test Suite embed their expected results within the test's output. See this example. The XML Query Test Suite metadata defines an output_file element. |
See Also | |
Comments |
Name | Version |
Description | An identifier thatallows one to distinguish between different revisions of test case. |
Rationale | Test cases often evolve over time, and it is important to maintain a history and to be able to identify and distinguish between different revisions. |
Required? | Yes. |
Syntax | Implementation dependent. |
Example | This will often be generated by a source-code control system such as CVS. |
See Also | |
Comments |
Name | Contributor |
Description | The individual or organization that contributed this test case. |
Rationale | It may be necessary to contact the contributor to ask for information about the test case or to request an update. |
Required? | Yes. |
Syntax | Freeform. |
Example | The XML Conformance Test Suite encodes the contributor in the directory structure used to store the test cases. |
See Also | Dublin Core: contributor |
Comments |
Name | Rights |
Description | Information about rights held in and over the test case. |
Rationale | Publishers and users of the test case need to understand the associated Intellectual Property Rights (IPR). |
Required? | Yes. |
Syntax | Ask your lawyer. |
Example | |
See Also | Dublin Core: rights |
Comments | This will often be simply a pointer to a copyright notice contained within the source code. |
Name | Grouping |
Description | A mechanism for classifying test cases into groups. |
Rationale | To enable the selection of subsets of test cases that share certain characteristics. |
Required? | No. |
Syntax | Implementation dependent. Possibilities include naming conventions and enumerated lists. |
Example | Tests may be classified as interactive or automated, positive or negative, voice or dtmf. |
See Also | |
Comments | An important use of grouping techniques is to classify tests that belong to particular profiles, modules, or other discretionary groupings (see Variability in Specifications for a detailed discussion of these issues). The SVG test suite, for example, uses naming conventions to distinguish between test cases targeted at different profiles (full, basic, or tiny). |
Name | SeeAlso |
Description | A list of references to relevant materials. |
Rationale | Can help to clarify the intent or usefulness of the test case. |
Required? | No. |
Syntax | Freeform (hypertext). |
Example | A pointer to an item in an issue-tracking system such as Bugzilla or to a mailing list thread in which the justification for this test case is discussed. |
See Also | Dublin Core: relation |
Comments |
See the Dublin Core Metadata Initiative for an introduction to the use of metadata. (Note that this document re-uses Dublin Core elements where this seems appropriate.)
For background information on the test development process and additional rationale for the the use of metadata see the QA Working Group's Test FAQ.
For a practical implementation of some of these principles see the the XML Query and XSL Working Groups' use of metadata in the XML Query Test Suite. (Their Guidelines for Test Submission and Guidelines for Test Execution provide explanations and examples.)