The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any thing tested against any set of criteria. The test subject might be a Web site, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.
EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.
A companion document Evaluation and Report Language (EARL) 1.0 Guide [Guide], to this specification provides more introductory material and explanation of the use cases for EARL. The companion document also highlights specific considerations, such as security and privacy.
[Editor's note: Does this overview of a test result really belong here prior to other introductory material? Suggest moving this to beginning of Section 2, just prior to section 2.1, or to end of section 1.]
Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:
- Assertor
- This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
- Test Subject
- This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
- Test Criterion
- What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
- Test Result
- What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.
Examples
Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.
- Assertor
- Bob B. Bobbington
- Test Subject
- A Web page located at
http://www.example.org/page.html
- Test Criterion
- Success Criteria 1.1.1 of the Web Content Accessibility Guidelines 2.0
- Test Result
- Passed
Example 2: A software carries out automated validation of a Web page against a technical specification.
- Assertor
- The W3C Markup Validator located at
http://validator.w3.org/
- Test Subject
- The XHTML returned from a GET request to the URI
http://www.example.org/page.html at 2004-04-14T14:00:04+1000
- Test Criterion
- The validity of the XHTML code
- Test Result
- Failed, the
<li> element on line 53, char 7 was not closed.
The assumed audience of this specification is developers who are implementing EARL in software or processes, or those who are seeking to understand the ideas, models, or properties and classes used in the EARL vocabulary. Readers who would like more introductory material for the language with explanation of its foreseen use cases are referred to the Evaluation and Report Language (EARL) 1.0 Guide [Guide].
This document assumes that the reader is familiar with the ideas of Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].
The keywords must, required, recommended, should, may, and optional are used in accordance with RFC 2119 [RFC2119].
Where an RDF term is used in its abbreviated form (e.g. Assertion or foaf:Person), that term is in the EARL namespace if no namespace is provided. The following prefixes are used in examples throughout this document:
earl- The EARL namespace
http://www.w3.org/ns/earl# which is described in this document
dc- The Dublin Core (DC) elements namespace
http://purl.org/dc/elements/1.1/ whose terms are described in [DC]
dct- The Dublin Core terms namespace
http://purl.org/dc/terms/ described at [DCT]
foaf- The Friend of a Friend (FOAF) namespace
http://xmlns.com/foaf/0.1/ which is also where the terms are described [FOAF]
http- The HTTP Vocabulary in RDF namespace
http://www.w3.org/2006/http# described in [HTTP]
owl- The OWL namespace
http://www.w3.org/2002/07/owl# described in [OWL]
ptr- The Pointer Methods in RDF namespace
@@@@ described in [Pointers]
rdf- The RDF namespace
http://www.w3.org/1999/02/22-rdf-syntax-ns# described in [RDF]
rdfs- The RDF Schema namespace
http://www.w3.org/2000/01/rdf-schema# described in [RDFS]
xsd- The XMLS namespace
http://www.w3.org/2001/XMLSchema# described in [XMLS]
Assertion - a statement that embodies the results of a test.
Related Properties
Properties defined by this document:
[Editor's note: Recommend adopting definition lists ratehr than tables for readability with ATs; also consistent with lists of external properties.]
- in domain of:
- in range of:
- none
An EARL Assertion is a statement about a particular test result that typically denotes:
- who or what performed or executed the test
- the content, application, or other entity that is being tested
- the criteria upon which an evaluation or result is based
- the test result itself
An Assertion is the fundamental unit of an EARL report.
Examples
Example 3: Instance of an assertion expressed as an RDF/XML fragment.
<earl:Assertion rdf:about="#assertion">
<earl:assertedBy rdf:resource="#assertor"/>
<earl:subject rdf:resource="#subject"/>
<earl:test rdf:resource="#testcase"/>
<earl:result rdf:resource="#result"/>
</earl:Assertion>
[Editor's note: this section uses the 
icons to highlight external links. Are these useful, and should they be added throughout the remainder of the doucment?]
Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.
Related Properties
Properties defined by this document:
Domain earl:Assertor |
Range earl:Assertor |
|
|
Properties not defined by this document:
dc:title - Human readable title for the assertor.
dc:description - Human readable description of the assertor.
foaf:name - Name of the assertor, which could be supplemented with further refinements such as
foaf:firstName , foaf:surname , or foaf:nick .
foaf:mbox - E-mail address of the assertor, which is preferably provided in an encrypted format using the
foaf:mbox_sha1sum property.
foaf:homepage - Homepage of the assertor, which could be supplemented with further refinements such as
foaf:workplaceHomepage , foaf:workInfoHomepage , or foaf:schoolHomepage .
Related Classes
earl:Software- The assertor is a piece of software.
foaf:Agent - The assertor is an agent, as defined in [FOAF].
foaf:Person - The assertor is a person, as defined in [FOAF].
foaf:Organization - The assertor is an organization, as defined in [FOAF].
foaf:Group - The assertor is a group of agents, as defined in [FOAF]
Conformance
An Assertor must have a name provided by the foaf:name or dc:title properties. In addition, an Assertor should have identifying information such as a description, e-mail address, homepage, or version number as applicable.
Examples
Example 4: An Assertor that is a person called Bob B. Bobbington.
<foaf:Person rdf:about="#bob">
<foaf:name>Bob B. Bobbington</foaf:name>
<foaf:mbox rdf:resource="mailto:bob@example.org"/>
<foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>
Example 5: An Assertor that is a piece of software called Cool Tool.
<earl:Software rdf:about="#tool">
<dc:title xml:lang="en">Cool Tool</dc:title>
<dc:description xml:lang="en">My favorite tool!</dc:description>
<foaf:homepage rdf:resource="http://example.org/tools/#cool"/>
<dc:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>
Example 6: An Assertor this is the person from example 4 using the software tool from example 5.
<earl:compoundAssertor rdf:about="#assertor">
<dc:title xml:lang="en">Bob using Cool Tool</dc:title>
<dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
<earl:mainAssertor rdf:resource="#bob"/>
</earl:compoundAssertor>
Test Subject - the class of things that have been tested against some test criterion. This class is intentionally generic to serve a wide variety of usages.
Related Properties
Properties defined by this document:
Domain earl:TestSubject |
Range earl:TestSubject |
| - |
|
Properties not defined by this document:
dc:title- Human readable title for the subject.
dc:description- Human readable description of the subject.
dc:date- Date on which the subject was tested (or when it was identified if more appropriate).
dc:hasPart- Relationship to other subjects that are part of this subject.
dc:isPartOf- Relationship to other subjects of which this subject is a part of.
Related Classes
earl:Software- The test subject is a piece of software being tested.
http:Connection- An HTTP connection, as defined in [HTTP-in-RDF].
repr:Content- Representation of the content, as defined in [Content-in-RDF].
[Editor's note: should we also adopt the foaf:Document as a possible Test Subject?]
Conformance
A Test Subject should have a title provided by the dc:title property. In addition, a Test Subject should have identifying information such as a description, date, and version numbers for software tools or other products. In particular, information about the HTTP connection or a representation of the actual content tested should be considered for Web resources and other types of electronic content.
[Editor's note: should we add dc:title to http:Connection and repr:Content for the sake of completeness?]
Examples
Example 7: A group of pages that have been tested together as a single subject.
<earl:TestSubject rdf:about="http://www.example.org/">
<dc:title xml:lang="en">example.org Web site</dc:title>
<dc:description xml:lang="en">Each page on the example.org Web site</dc:description>
<dct:hasPart rdf:resource="http://www.example.org/style.css"/>
<dct:hasPart rdf:resource="http://www.example.org/page1.html"/>
<dct:hasPart rdf:resource="http://www.example.org/page2.html"/>
<dct:hasPart rdf:resource="http://www.example.org/page3.html"/>
</earl:TestSubject>
Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.
Related Properties
Properties defined by this document:
Domain earl:TestCriterion |
Range earl:TestCriterion |
| - |
|
Properties not defined by this document:
dc:title- Human readable title for the test criterion.
dc:description- Human readable description of the test criterion.
dc:hasPart- Relationship to other test criteria that are part of this criterion.
dc:isPartOf- Relationship to other test criteria of which this criterion is a part of.
Related Classes
While the generic earl:TestCriterion class can be used directly, one of the following more specifc refinements should be used instead:
earl:TestRequirement- A higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criteria 1.1.1, which is evaluated using several Techniques for Success Criteria 1.1.1 and combining the results.
earl:TestCase- An atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criteria 1.1.1.
Conformance
A Test Criterion should have a title and a description provided by the dc:title and dc:description properties. In addition, a Test Criterion may be related to other criteria using the dc:hasPart and dc:isPartOf properties.
Examples
Example 8: Instance of a test case that is described with a title and its relationship to a test suite.
<earl:TestCase rdf:about="#testcase">
<dc:title xml:lang="en">HTML Test 282</dc:title>
<dc:description xml:lang="en">Test 282 of the HTML test suite</dc:description>
<dct:isPartOf rdf:resource="http://example.org/tests/html/"/>
<dct:hasPart rdf:resource="http://example.org/tests/html/#617"/>
</earl:TestCase>
Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).
Related Properties
Properties defined by this document:
Domain earl:TestResult |
Range earl:TestResult |
|
- |
Properties not defined by this document:
dc:title- Human readable title for the subject.
dc:description- Human readable description of the subject.
[Editor's note: should we have a dc:date for the testing date (and make the date of the Test Subject only the fetch/locating/production date)?]
Conformance
A Test Result must have an Outcome provided by the earl:outcome property. In addition, a Test Result should have a title and a description provided by the dc:title and dc:description properties. A Test Result may also have pointers to the relevant locations in the test result provided by the earl:pointer property, or additional information such as warnings or informative messages provided by the earl:info property.
Examples
Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/2007/earl#fail"/>
<dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
<dc:description rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dc:description>
<earl:pointer rdf:resource="#xpointer"/>
<earl:info rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>It seems the <code>p</code> element has not been closed</p>
</div>
</earl:info>
</earl:TestResult>
The Test Mode described how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries. The generic earl:TestMode class should not be used, but instead one of the following refinements:
earl:Automatic- Automatic - where the test was carried out automatically by the software tool and without any human intervention.
earl:Manual- Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
earl:SemiAuto- Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
earl:Undisclosed- Undisclosed - where the exact testing process is undisclosed or unknown.
[Editor's note: notice the capitalization since these are now class names -- needs confirmation by the group.]
Related Properties
Properties defined by this document:
Domain earl:TestMode |
Range earl:TestMode |
| - |
|
Conformance
A Test Mode must have a label and a comment provided by the rdfs:label and rdfs:comment properties. The Test Mode should be one of the four type refinements provided above, or sub-classes of these.
Examples
Example 10: The assertion from example 3 was carried out in semi-automatic mode.
<earl:Assertion rdf:about="#assertion">
<earl:mode rdf:resource="http://www.w3.org/2007/earl#semiauto"/>
</earl:Assertion>
[Editor's note: class name is OutcomeValue to differentiate from the outcome property (difference was only the letter case, which is hard to perceive in many situations -- needs group resolution.]
Outcome Value - a discrete value that describes a resulting condition from carrying out the test. The generic earl:OutcomeValue class should not be used, but instead one of the following refinements:
earl:Pass- Passed - the subject passed the test.
earl:Fail- Failed - the subject failed the test.
earl:CannotTell- Can not tell - it is unclear if the subject passed or failed the test. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:NotApplicable- Not applicable - the test is not applicable to the subject.
earl:NotTested- Not tested - the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.
[Editor's note: notice the capitalization on the class refinements since these are now class names -- needs confirmation by the group.]
Related Properties
Properties defined by this document:
Domain earl:OutcomeValue |
Range earl:OutcomeValue |
| - |
|
Conformance
An Outcome must have a label and a comment provided by the rdfs:label and rdfs:comment properties. The Outcome should be one of the five type refinements provided above, or sub-classes of these.
[Editor's note: ERT WG is considering the use of DOAP terms to describe Software. Feedback on this consideration is welcome.]
A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to the Authoring Tool Accessibility Guidelines or of a browser to the User Agent Accessibility Guidelines).
Related Properties
Properties not defined by this document:
dc:title- Human readable title for the software.
dc:description- Human readable description of the software.
dc:hasPart- Relationship to other software components that are part of this software.
dc:isPartOf- Relationship to other software of which this software component is a part of.
[Editor's note: should we have a dc:date for the production date)?]
Conformance
A Software should have a title using the dc:title property. In addition, a Software should have identifying information such as a description, homepage, or version number where available. A Software may also be related to other Software components by using the dc:hasPart and dc:isPartOf properties.
Examples
Example 11: The software which was an Assertor in example 5 could also be used as a Test Subject.
<earl:Assertion rdf:about="#assertion">
<earl:subject>
<earl:Software rdf:about="#tool">
<--// Note: the dc:title, dc:description, foaf:homepage, and dct:hasVersion
properties are already defined by the RDF code in example 5 //-->
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date">2005-06-25</dc:date>
<dct:isPartOf rdf:resource="http://example.org/tools/"/>
<dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
</earl:Software>
</earl:subject>
</earl:Assertion>
Asserted By - assertor of an assertion.
Subject - test subject of an assertion.
Test - test criterion of an assertion.
Result - result of an assertion.
Mode - mode of an assertion.
[Editor's note: this wording needs to be checked by he group.]
Main Assertor - assertor who is primarily responsible for performing the test.
Outcome - outcome value of a test result.
Pointer - relevant locations within the test subject.
Info - additional warnings or error messages.
[Editor's note: this section needs revision. Previous content is commented out within the source code. The following is a suggestion for further discussion:]
EARL 1.0 Reports
A valid EARL 1.0 report must validate against the formal RDF grammar, and must adhere to the conformance requirements defined by this document. To support XML based applications, EARL 1.0 reports should be represented in (a compact) RDF/XML format when being exported or published outside an application.
EARL 1.0 Processors
EARL 1.0 processors must process EARL 1.0 reports in RDF/XML format and should be able to parse EARL 1.0 reports in other RDF formats such as N3. EARL 1.0 processors are expected to process the following information in conforming EARL 1.0 reports:
Assertion
- Equivalent Classes
- none
- Required Properties
- Optional Properties
- none
Assertor
- Equivalent Classes
- Required Properties
- Optional Properties
- none
[Editor's note: this section needs revision.]
- [DC]
- The Dublin Core Metadata Element Set - DC Recommendation, 20 December 2004.
http://www.dublincore.org/documents/dces/
- [DCT]
- The Dublin Core Metadata Terms - DC Recommendation, 13 June 2005.
http://www.dublincore.org/documents/dcmi-terms/
- [FOAF]
- FOAF Vocabulary Specification
- Working Draft, 3 June 2005.
http://xmlns.com/foaf/0.1/
- [Guide]
- Evaluation and Report Language (EARL) 1.0 Guide - Editors' Draft 14 December 2005. C. Velasco, J. Koch, S. Abou-Zahra eds.
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Guide-20051214
- [HTTP]
- HTTP Vocabulary in RDF - W3C Working Draft 20 December 2006. J. Koch, C. Velasco, S. Abou-Zahra eds.
http://www.w3.org/TR/HTTP-in-RDF/
- [OWL]
- OWL Web Ontology Language - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/owl-features/
- [Pointers]
- Pointer Methods in RDF - Editors' Draft 22 February 2007. J. Ley eds.
http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222
- [RDF]
- RDF/XML Syntax Specification (Revised) - W3C Recommendation, 10 February 2004. D. Beckett ed.
http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/
- [RDF-PRIMER]
- RDF Primer - W3C Recommendation, 10 February 2004. F. Manola, E.Miller eds.
http://www.w3.org/TR/rdf-primer/
- [RDFS]
- RDF Vocabulary Description Language 1.0: RDF Schema - W3C Recommendation, 10 February 2004. D. Brickley, R.V. Guha eds.
http://www.w3.org/TR/rdf-schema/
- [RDF-XML-DIFFS]
- Why RDF model is different from the XML model - Paper, September 1998. T. Berners-Lee.
http://www.w3.org/DesignIssues/RDF-XML
- [RFC2119]
- Key words for use in RFCs to Indicate Requirement Levels - IETF RFC, March 1997.
http://www.ietf.org/rfc/rfc2119.txt
- [WCAG10]
- Web Content Accessibility Guidelines 1.0 - W3C Recommendation, 5 May 1999. W. Chisholm, I. Jacobs, G. Vanderheiden eds.
http://www.w3.org/TR/WCAG10/
- [XMLS]
- XML Schema Part 0: Primer Second Edition - W3C Recommendation, 28 October 2004. D. Fallside, P. Walmsley erds.
http://www.w3.org/TR/xmlschema-0/
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.