Toward Open Data

I want my data back.

Jon Bosak circa 1997
I've long believed that customers of any application own the data they enter into it.

Jeffrey Veen 2 November 2005

Is this what Web 2.0 is all about? If so, maybe it's not such a bad thing.

Outline

A history of (X)HTML and the Web
Toward data integration, Web style
Semantic Web * iCalendar = RDF Calendar
GRDDL: Semantic Web data in XHTML
hCalendar and microformats
hCalendar * GRDDL = RDF Calendar
Microformats + SPARQL
Microformats + OWL, rules

Objective: making it cost-effective to record and share knowledge formally, i.e. so that computers can manipulate it.

Getting into the Web

What was the tipping point— the killer app—for you?

for Tim Berners-Lee's colleagues at CERN: a cross-platform phone directory

for me at Convex in 1991: a nifty hypertext system design:

Newsgroups: alt.hypertext
Subject: WorldWideWeb: Summary
Date: 6 Aug 91 16:00:12 GMT

		WorldWideWeb - Executive Summary


The WWW project merges the techniques of information retrieval and hypertext to  
make an easy but powerful global information system. ...

in the early 1990s: NCSA What's New
these days: maps, auctions, wikipedia, webmail, ...

Getting into the Web: downhill steps

To grow, start with some actual value plus lots of potential value and a downhill path for contributors:

for a little investment
Note: licensing and legal agreements are not little.
a big return
due to network effects
Scaling
- yes, scalability to 10^9 nodes and up is important
- but so is scalability down to families and scout troops

Architect for participation -- Tim O'Reilly @ W3C's 10 anniversary

Web Basics: URIs, HTTP, ...

So what makes all that work? Much of it is a story for another day.

See Architecture of the World Wide Web, Volume One
W3C Recommendation 15 December 2004

Web Basics: HTML

In the "garage" at CERN:
NextStep RTF object meets SGML
IETF HTML Working Group:
Is <p> a container or a separator?
HTML Validation Service at Halsoft
W3C HTML Working Group:
Netscape trades <blink> for Microsoft's <marquee>
HTML 3.2 is born

Aside: CSS deployment

... and the importance of test suites

Dec 1996: Original CSS Rec
Goodbye <FONT>, right?
At least <FONT> didn't crash my browser
The experience of the CSS working group, with the CSS1 Test Suite, and the many compliant and interoperable implementations that followed, demonstrated fairly clearly that a simple test suite is far far better than none at all.

CSS Test Suite Documentation

HTML / SGML * XML = XHTML

along with CSS, of course

HTML 4.0: a vocabulary we can understand and agree on
XML 1.0: a syntax we can understand and agree on
CSS: a style sheet mechanism with graceful degradation
After ~5 years of single-pixel-gifs and other black magic...
XHTML + CSS arrives
Designing With Web Standards by Zeldman, May 2003

The Personal Information Disaster

For data, we are pre-web:

Airline sends me email from their database

I copy/paste each of the data into my PDA

Soccer coach distributes a schedule

each players with an online calendar re-keys the data
The bane of my existence is doing things I know the computer could do for me.

The XML Revolution, Nature Web Matters Oct 1998
Let's find ways to make it cost-effective record and share knowledge formally, i.e. so that computers can manipulate it.

RDF Calendar for travel itineraries

travel admin mails text itinerary
- dumped from database, probably
nasty perl script scrapes RDF statements in travel terms
logical rules convert to iCalendar vocabulary
- mixing in airport timezone info
toIcal.py converts RDF calendar to iCalendar .ics syntax

But why not just spit out .ics from the nasty perl script?

Travel from place to place

for details, see Integration Example: travel tools in Semantic Web Tutorial Using N3 given at WWW2003 in Budapest

iCalendar does have location/geo fields, but only one per event. Flights have a departure and an arrival.

TODO list views of email archives

lists.w3.org gives message headers in RDF
convention: [closed] marks closed threads in public-cwm-bugs
some rules relate these conventions to RDF/iCalendar
toIcal.py converts to .ics format
developers subscribe to TODO list

details: bug status in .ics of Jan 2004

Semantic Web Basics

What is the Semantic Web?
- Data integration across application, organizational boundaries
How does it work?
1. Apply power of URIs to concepts of relational data
  - Don't say "colour" say <http://example.com/2002/std6#col>
2. Model real things, not just documents or database tables

The element of the Semantic Web

The Resource Description Framework (RDF)
- abstract syntax, formal semantics
- standard encoding in XML
- emerging programmer-friendly short-hand encoding: N3/turtle

      @prefix foaf: <http://xmlns.com/foaf/0.1/>.
      <people.rdf#dan> foaf:name "Dan Connolly".
      <people.rdf#dan> foaf:interest <http://www.w3.org/XML/>.

Note the relationship to HTML links, especially with the re-discovery of the rel attribute.

Semantic web includes tables, trees...

Arrows can make a table, an arrow from each row to each value

... and tangly messes

Arrows can make a table, an arrow from each row to each value

RDFS and OWL

RDF Schema (RDFS) and the Web Ontology Language (OWL) correspond to UML notions such as subclass, domain, range, cardinality, ...

cyc terms in red
DAML airport ontology terms in purple
custom travelTerms in green.
RDF standard terms in blue
XML Schema terms in orange

RDF Calendar in a nutshell

In iCalendar terms (RFC2445), an event is a component with various properties:

BEGIN:VEVENT
UID:20020630T230445Z-3895-69-1-7@jammer
DTSTART;VALUE=DATE:20020703
DTEND;VALUE=DATE:20020706
SUMMARY:Scooby Conference
LOCATION:San Francisco
END:VEVENT

and RDF/XML has analogous class and property constructs:

      <Vevent>
        <uid>20020630T230445Z-3895-69-1-7@jammer</uid>
        <dtstart>2002-07-03</dtstart>
        <dtend>2002-07-06</date>
        <summary>Scooby Conference</summary>
        <location>San Francisco</location>
      </Vevent>

For details, see RDF Calendar - an application of the Resource Description Framework to iCalendar Data, Connolly and Miller September 2005

Round-trip testing

Comparing .ics files is tricky, so...

to test .ics -> RDF:
1. convert $testcase.ics to temporary $testcase-actual.rdf
2. Use RDF graph comparison to check actual results vs. expected results
to test RDF -> .ics
w.r.t a known-good .ics -> RDF tool:
1. convert $testcase.rdf to $testcase-temp.ics
2. Use the known-good tool get $testcase-actual.rdf
3. Use RDF graph comparison to check $test-case-actual.rdf against expected results

On RDF/XML Syntax

Too constrained for some
- Tries to look like Book/author/title metadata but has subtle constraints with striping, rdf:about,rdf:resource, etc.
Not constrained enough for others
- doesn't work with DTDs nor W3C XML Schemas; works awkwardly with XPath and XSLT

At least the issues in the 1998 spec have all been resolved, complete with test cases. There are plenty of interoperable parsers. And it works great with Relax-NG and nxml-mode :)

Observation: lots of structured data in XHTML dialects

bibliographies
schedules
issues lists
news items
blog rolls

Data in Documents

I believe that one of the best ways to transition into RDF, if not a long-term deployment strategy for RDF, is to manage the information in human-consumable form (XHTML) annotated with just enough info to extract the RDF statements that the human info is intended to convey. In other words: using a relational database or some sort of native RDF data store, and spitting out HTML dynamically, is a lot of infrastructure to operate and probably not worth it for lots of interesting cases.

We all know that we have to produce a human-readable version of the thing... why not use that as the primary source?

XSLT for screen-scraping RDF out of real-world data
Dan Connolly to www-rdf-interest March 2000

Case Study: News Syndication at W3C

W3C Communications Team knows HTML well; picked up XHTML quickly
RSS emerged as a syndication mechanism
W3C's news content management system is just CVS; no database
But by adding just a class attribute here and a rel attribute there, we get enough structure to transform it to RSS using XSLT

Site Summaries in XHTML is a cost-effective way to formalize our news metadata.

GRDDL Semantics: explicit, grounded in the Web

Screen scraping: at your own risk
GRDDL: author explicitly agrees to data extraction algorithm
rel="transformation" is grounded in URI space:
http://www.w3.org/2003/g/data-view#transformation
output of transformation is expressed in RDF, using URIs for property names

A person (or a machine) can "follow your nose" from the document to the transformation algorithm, to the data, to the definitions of the terms used in the data.

GRDDL Syntax: author's choice

works with DTD-happy XHTML
via a link with rel="transformation", qualified by the GRDDL profile
works with XML in general
via grdd:transformation attribute on the root element
the link refers to a transformation, typically XSLT, that maps to RDF/XML
the link may be indirect via a profile or namespace document

GRDDL drawback: turing completeness

XSLT is a big hammer
- much less so than in 2000
Trust issues: consumer runs XSLT from who knows where
- well-known transformations should cover 80%
  - short-cut local implementation
  - known-good cached copies

GRDDL Details

In addition to structured XHTML, add the GRDDL profile and a typed link:

<html xmlns="http://www.w3.org/1999/xhtml">
<head profile="http://www.w3.org/2003/g/data-view">
  <link rel="transformation"
     href="http://www.w3.org/2000/08/w3c-synd/home2rss.xsl" />
  ...
</head>
<body>
  ...
<div id="x200501110b" class="item">
<h3><img alt="" width="17" height="11" src="/Icons/right" />Policy for
Authorized W3C Translations Announced</h3>

<p><span class="date">2005-11-10:</span> W3C is pleased to announce...

The home2rss transformation turns that into RDF/XML:

...
<item rdf:about="http://www.w3.org/News/2005#item158">
<title>Policy for Authorized W3C Translations Announced</title>
<description>2005-11-10: W3C is pleased to announce ...

GRDDL: multiple dialects allowed

GRDDL: one profile for lots of documents

For details, see Gleaning Resource Descriptions from Dialects of Languages (GRDDL), Hazaël-Massieux and Connolly May 2005

GRDDL for any XML, not just XHTML

<java version="1.5.0_04" class="java.beans.XMLDecoder"
   xmlns:grddl="http://www.w3.org/2003/g/data-view#"
   grddl:transformation="grokVioletUML.xsl">
 <object class="com.horstmann.violet.ClassDiagramGraph"> 
  <void method="addNode"> 
   ...

Origins of hCalendar and Microformats

Borrowing from Microformats: Evolving the Web by Tantek Çelik Sep 2005

Very common "structures" on weblogs
Create explicit structures
- in order to easily publish, index, aggregate
Minimize impact on authors (and developers)
Avoid duplicating all content
- Avoid requiring file uploads

Microformats Principles

solve a specific problem
simple as possible
- evolutionary improvements
humans first, machines second
- presentable and parsable
- adapt to current behaviors
reuse from widely adopted standards
- semantic (X)HTML, schemas from interoperable RFCs
modularity / embeddability
decentralized development, content, services
- explicitly encourage "spirit of the Web"

XHTML * iCalendar = hCalendar

Map iCalendar constructs 1:1 into XHTML
- hCalendar specification, issues, examples... wiki

example


<ol class="vcalendar">
 <li class="vevent">
  <a href="http://tantek.com/presentations/..." 
     class="url">

   <span class="summary">Microformats: Evolving the Web</span
   <abbr class="dtstart" title="20050930T1530+1000">
    September 30th, 2005
   </abbr>
  </a>
 </li>

</ol>

more hCalendar

hCalendar + GRDDL = RDF Calendar

hCalendar + glean-hcal.xsl GRDDL transformation:
<a href="e1.html">event 1</a>

RDF:

@prefix : <http://www.w3.org/2002/12/cal/icaltzd#> .
@prefix XML: <http://www.w3.org/2001/XMLSchema#> .

[       a :Vevent;
           :attendee  [ :cn "Hoopy Frood";
             :calAddress <mailto:frood@example> ];
           :dtend "2005-10-07"^^XML:date;
           :dtstart "2005-10-05"^^XML:date;
           :geo  (37.0625 -95.677068 );
           :location "Argent Hotel, San Francisco, CA";
           :summary "Web 2.0 Conference";
           :url <http://www.web2con.com/> ].

Upcoming events on blogs

From Tantek's Thoughts:

10/5-7 Web 2.0 - at The Argent, San Francisco

10/17-19 W3C CSS Working Group f2f at San Francisco

Mix with glean-hcal.xsl and we get:

<r:RDF xmlns:r="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:c="http://www
.w3.org/2002/12/cal/icaltzd#">
...
      <c:Vevent>
        <c:summary xml:lang="en-us">Web 2.0 </c:summary>
        <c:dtstart r:datatype="http://www.w3.org/2001/XMLSchema#date">2005-10-05</c:dtstart>
        <c:dtend r:datatype="http://www.w3.org/2001/XMLSchema#date">2005-10-08</c:dtend>
        <c:url r:resource="http://web2con.com/"/>
        <c:location xml:lang="en-us">The Argent, San Francisco </c:location>
      </c:Vevent>

SQL * URIs = SPARQL

Use GRDDL to aggregate data from friends etc, then...

  PREFIX foaf: <http://xmlns.com/foaf/0.1/>
  PREFIX c: <http://www.w3.org/2002/12/cal/icaltzd#>
  SELECT ?name, ?summary, ?when
   FROM <myFriendsBlogsData>
   WHERE { ?somebody foaf:name ?name; foaf:mbox ?mbox.
           ?event c:summary ?summary;
                  c:dtstart ?ymd;
                  c:attendee [ c:calAddress ?mbox ]
         }.

?name	?summary	?when
Tantek Çelik	Web 2.0	2005-10-05
Norm Walsh	XML 2005	2005-11-13
Dan Connolly	W3C tech plenary	2006-02-27

See SPARQL Query Language for RDF W3C Working Draft 21 July 2005, plus Norm's travel this year

Fill in blanks with rules

Mix with some logical rules:

Infer the place implicit in the iCalendar geo and location properties.

{  ?E c:geo (?LAT ?LONG) }
 => { ?E k:eventOccursAt [ geo:lat ?LAT; geo:long ?LONG] }.

{  ?E c:location ?PLACENAME; k:eventOccursAt ?WHERE
} => {
  ?WHERE k:nameOfAgent ?PLACENAME;
    s:label ?PLACENAME;
    k:inRegion [ a k:City ]. # Assume each event is in some city
}.

Cities containing points far from each other are distinct

{
  ?C1 a k:City; is k:inRegion of [geo:lat ?X1; geo:long ?Y1 ].
  ?C2 a k:City; is k:inRegion of [geo:lat ?X2; geo:long ?Y2 ].

  # DSQ = (X1-X2)^2 + (Y1-Y2)^2
  (((?X2 ?X1).m:difference 2).m:exponentiation
   ((?Y2 ?Y1).m:difference 2).m:exponentiation) m:sum ?DSQ.
  ?DSQ m:greaterThan 0.2.
} => { ?C1 owl:differentFrom ?C2 }.

Full details: calendar background rules

A model behind hCalendar

Can we do this event too?

RDF merges trivially

Partial Understanding

RDF statements* are independent. RDF semantics are monotonic.

	RDF	XML
Premise	<Book rdf:ID="book1"> <dc:title>The Grapes of Wrath</title> <dc:creator>Steinbeck</author> </Book>	<xsd:simpleType name="myInteger"> <xsd:restriction base="xsd:integer"> <xsd:minInclusive value="10000"/> <xsd:maxInclusive value="99999"/> </xsd:restriction> </xsd:simpleType>
Conclusion	<Book rdf:ID="book1"> <dc:title>The Grapes of Wrath</title> </Book>	<!-- no, this does not follow --> <xsd:simpleType name="myInteger"> <xsd:restriction base="xsd:integer"> <xsd:maxInclusive value="99999"/> </xsd:restriction> </xsd:simpleType>

RDF

XML

Premise

<Book rdf:ID="book1">
 <dc:title>The Grapes of Wrath</title>
 <dc:creator>Steinbeck</author>
</Book>

<xsd:simpleType name="myInteger">
  <xsd:restriction base="xsd:integer">
    <xsd:minInclusive value="10000"/>
    <xsd:maxInclusive value="99999"/>
  </xsd:restriction>
</xsd:simpleType>

Conclusion

<Book rdf:ID="book1">
 <dc:title>The Grapes of Wrath</title>
</Book>

<!-- no, this does not follow -->
<xsd:simpleType name="myInteger">
  <xsd:restriction base="xsd:integer">
    <xsd:maxInclusive value="99999"/>
  </xsd:restriction>
</xsd:simpleType>

*RDF/XML does have a rdf:parseType="Collection" syntax, which expands to a lisp style binary tree in the abstract syntax. This erasure property works not on XML elements, but on RDF statements.

Merging hCalendar data

OWL identity reasoning

Premise	# one-to-many foaf:mbox a owl:InverseFunctionalProperty. :dan foaf:mbox <mailto:connolly@w3.org>. :dan foaf:name "Dan Connolly". :daniel foaf:mbox <mailto:connolly@w3.org>. :daniel foaf:name "Daniel W. Connolly".
Conclusion:	:daniel owl:sameAs :dan. :daniel foaf:name "Dan Connolly". :daniel foaf:name "Daniel W. Connolly".