Please refer to the glossary for this document.
Copyright © 2009 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
Every day, governments and government agencies publish more data on the Internet. Sharing this data enables greater transparency; delivers more efficient public services; and encourages greater public and commercial use and re-use of government information. Some governments have even created catalogs or portals (such as data.gov) to make it easy for the public to find and use this data.
Although the reasons may vary, the logistics and practicalities of opening government data are the same. To help governments open and share their data, the W3C eGov Interest Group has developed the following guidelines. These straightforward steps emphasize standards and methodologies to encourage publication of government data, allowing the public to use this data in new and innovative ways.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
The W3C eGovernment Interest Groupis pleased to offer this First Public Working Draft which briefly explains how to publish government data on the Web, based on the our gathered experience. We encourage anyone with additional experience to join the group or send written comment to help with the next version.
The W3C eGovernment Interest Group views this as an evolving document and seeks public feedback on this Working Draft. Please send your comments to public-egov-ig@w3.org (archives). If possible, please offer specific changes to the text that would address your concern. You may also wish to check the Wiki Version of this document and see if the relevant text has already been updated.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. The group does not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
Step 1: The quickest and easiest way to make data available on the Internet is to publish the data in its raw form (e.g., an XML file of polling data from past elections). However, the data should be well-structured. Structure allows others to successfully make automated use of the data. Well-known formats or structures include XML, RDF and CSV. Formats that only allow the data to be seen, rather than extracted (for example, pictures of the data), are not useful and should be avoided.
Step 2: Create an online catalog of the raw data (complete with documentation) so people can discover what has been posted.
These raw datasets should be reliably structured and documented, otherwise their usefulness is negligible. Most governments already have mechanisms in place to create and store data (e.g., Excel, Word, and other software-specific file formats).
Posting raw data, with an online catalog, is a great starting point, and reflects the next-step evolution of the Internet - "website as fileserver".
Step 3: Make the data both human- and machine-readable:
These steps will help the public to easily find, use, cite and understand the data. The data catalog should explain any rules or regulations that must be followed in the use of the dataset. Also, the data catalog itself is considered "data" and should be published as structured data, so that third parties can extract data about the datasets. Thoroughly document the parts of the web page, using valid XHTML, and choose easily patterned and discoverable URLs for the pages. Also syndicate the data for the catalog (using formats such as RSS) to quickly and easily advertise new datasets upon publication.
The ability to identify things by a URI/URL is the foundation of the Internet. Best practices for "open government data" depend on applying the architecture described in Architecture of the World Wide Web, Volume 1 [WEBARCH]. If you give permanent, patterned and/or discoverable URI/URLs to your data, processes and people will be able to find and use it more easily. URI/URLs can be used in databases and metadata as universal, unique identifiers (e.g. by appending a serial number or other internal naming system to a domain: http://www.example.gov/objects/optional-hierarchy/serial12345678.html).
Use internal IDs to identify specific data for machine re-use (e.g., http://www.census.gov/main/www/popclock.html). By using URIs, and incorporating internal links inside large files or documents, people can authoritatively cite the information on their own websites.
Without documentation, data is not very useful. When possible, you should use industry standards such as those based on XML/RDF, as they tend to be self-documenting. Create a web page with a plain language description of the dataset to help search engines find the data, so people can use it. The description should explain how to find the pages and/or files, and describe the contents of the files. At a minimum, document the title, description, publication date and authoritative source for the data. Use clear language, and link to related information to help put the data into context. When possible, also fully document the description of each datum and the data standard and/or data validation (e.g., if datum is a temperature, also explain the related standard, such as Celsius, Fahrenheit or Kelvin). As standards emerge for describing the datasets, take advantage of them. Document search tools and ReSTful methods of obtaining the data when applicable.
The next step beyond raw data is fully linked data. In other words, raw data can only be linked to, there are no links back out - but linked data contains links out to other data and documentation. There is a spectrum from raw to fully linked data, and standards such as RDF that can help you link your data.
Here are four expectations for the linked data web:
Using URIs as described provides for a rich and easily connected web of data that is easier for the public to search and use. However, many databases do not build URIs, but instead use internally derived unique identifiers. To link this data into the larger web, we need to use systems that can anticipate or build the URIs prior to publishing the data.
Preserving the data we publish is a top concern for governments. How can we ensure that data is findable and can be referenced for as long as people need it? People may continue to cite older data at the original URI decades after it was initially published.
To maintain the history of a dataset, think about how to integrate new URIs for upgraded and new datasets, and structure your URIs accordingly. Use versioning of datasets so people can cite and link to present and past versions. New and upgraded datasets can refer back to original datasets. If needed, provide tools to translate older data. Carefully document the changes between versions, and embed the version number/indicator within the data if possible. Consider the file format carefully, and opt for open standards that are not software specific.
See the References section for more information on standards and best practices for data preservation.
If you really want to help people discover and explore the data you are publishing, there are some useful W3C standards which can help. It is possible to make data human-readable by using XSLT for XML- and RDF-based formats. Make web pages machine-readable by embedding semantic information in the HTML, using RDFa and/or Microformats. (Note that search engines can take advantage of embedded semantic information, helping more people to find your data.) If it is too difficult to append XSL or CSS styles to old data, it is still possible to document how to transform the data with a specific or generic XSLT engines (for example, see http://www.xmldatasets.net).
External parties can create new and exciting interfaces that may not be obvious to the data publishers. For that reason, do not compromise the integrity of the data to create flashy interfaces. If you must create an interface, then publish the data separate from the interface, and ensure external parties have direct access to the raw data, so they can build their own interfaces if they wish.
Additionally, since all web documents using (X)HTML, XML and RDF can be used as an object database or ReSTful API, the public can create software, Web applications and mash-ups that use the datasets, and link back to the authoritative version on the government web site. Two W3C standard languages that make that possible are XQuery and SPARQL. Once your data is published, governments themselves can also create a user-friendly, transparent interface that allows users to play with and refine the information in which they are interested.
Having a unique identifier for every registered company or school (object) is as important as having information about those companies or schools. In addition to using open industry standards, an effort should be made to catalog all reference-able government entities, officers and objects. The URIs can then be used within all of the data. This aids in discoverability, improves metadata, and ensures authenticity.
All data that can be shared with the public should be opened for public dissemination. Data should be published in compliance with applicable laws and regulations, and only after addressing issues of security and privacy.
First, publish all data that is already available in other formats (such as print), or already being collected and shared with the public in other ways. Publish both highly structured data and text documents. Publish laws, regulations and codes, and public documents related to government agency mission, vision, values and goals.
There are many standards for such documents (such as AIIM's emerging Strategy Markup Language (StratML), XBRL and others). Use of standards enables stakeholders to easily discover and provide feedback on items of interest to them. Creating and publishing new documents in various standard formats is easier than fixing after the fact.
As you create your documentation for the data, include and reference XML Schemas, DTDs and taxonomies. Explain how to discover all of the files/documents in a collection. Use syndication standards and tools to announce additions or changes. Provide complete documentation to enable automatically-generated data dictionaries and provide query services, which will make it easier for the public to find your documents and datasets.
There are many different data formats, but which will work best with your data? The primary format for human-readable data is (X)HTML.
Raw data is more likely to be produced using formats customized to the specific data, the tools used, or industry standards. The W3C has pioneered XML and RDF, which allow for excellent manipulation and standardized tool sets. RDF and XML files can be accessed like databases, using SPARQL, XQuery, JavaScript and many other computer languages. When possible, use established open standards, and tools that allow easy and efficient production and publishing of the data. See the References section for a list of current tools. Also keep in mind the power of linked data.
Once you've made your data available, be sure to document clearly any legal or regulatory restrictions on the use of the data. In many cases, there are defined standards to insert copyright and/or licensing information right into the data (see metadata standards, such as Dublin Core). It is very important to be clear regarding the expected uses, credits, limitations and responsibilities that particular parties have in regard to working with and providing the data.
We still have much to learn about the best ways for governments to open up their data. The W3C eGov Interest Group plans to publish relevant use cases to illustrate working examples of current thinking and best practices. The W3C technologies and approaches described in this document can be implemented quickly, and often with relatively little cost. If you want to learn more, there are a number of communities that can help. If you are a government agency, think about joining the W3C and getting involved in e-Government activities. You can connect with people from other governments who are facing similar issues and sharing their experiences. Even if you don't join the W3C, please let us know how you are getting on. We can help.