1 Introduction
[RFC 3986] defines the character string following the ? sign in
a URI as the "query component". The character string follwing the # sign is
known as the "fragment identifier" and used to address specific locations in a
document. Nearly 20 years later, the Web has built a strong set of conventions around how URI parameters are used. As transactional applications began moving on to the Web in the late 1990's, query parameters formed a core building block for how application state was communicated between client and server. In this phase of Web evolution, clients were still comparatively simple, and client-side URI parameters did not move beyond the use of fragment identifiers. But with Web 2.0 applications increasingly moving traditional client-side applications to the Web and highly interactive
applications being built using Web parts (HTML, CSS and JavaScript component
resources
that are themselves Web addressible, see [TVRaman CACM 2009]), fragment identifiers are being used in a variety of ways quite different from their originally intended use. The Web is beginning to discover and codify design patterns based on fragment identifiers for many of these use cases.
This document explores the issues that arise in this context, and attempts to define best practices that help:
- Create URIs for intermediate states/pages in a Web application and manage the browser history so that the back button does the right thing
- Enable clients to address into specific points in a stream of content, e.g., video.
This document discusses only the situations where fragment identifiers are used with HTTP. Fragment identifiers can be used with other protocols such as FTP where they
may have different semantics. These situations are not discussed here.
In this document we use the term "URI" to indicate both URLs and URIs.
2 Use Case Scenarios
This section discusses several usage scenarios that make innovative use of fragment identifiers as client-side parameters.
2.1 Addressing Into Multimedia Streams -- CNN
When publishing multimedia streams, there is often a need to address into specific points in the multimedia stream, e.g., by using a time-index. The simplest means of doing this is to pass in the start-time as a query parameter in the URI to the server, e.g. http://www.example.com/media.stream?start=03:06:09 and have the server start streaming the content starting at 3 hours, 6 minutes and 9 seconds into the content. This has the additional side-benefit of creating distinct URIs for each point in the media stream and such URIs can be used to bookmark locations of interest.
It is also possible to leverage client-side parameters encoded as part of the URI (using a # ), where this pseudo fragment identifier is used by client-side scripts as an argument to be passed to an appropriate locator function. Consider the following example taken from cnn.com:
<a href="http://www.cnn.com/video/#/video/tech/2008/02/19/vo.aus.sea.spider.ap">
Giant sea spider filmed deep underwater
</a> CNN uses links like the above for all the topical video
segments that are published on its site. The URI in this case has
the following components:
| Component
| Value
|
|---|
| Protocol
| http
|
| Host
| www.cnn.com
|
| Path
| video
|
| Client
Param
| #/video/tech/2008/02/19/vo.aus.sea.spider.ap
|
2.1.1 Things to Note
The browser is expected to do a GET of the URI leading up to the # sign, and the processing application, in this case, the JavaScript embedded in the HTML Response processes the portion of the URI following the # .
Note that in the general case, the JavaScript function that eventually processes the client parameter may not have been present in the original HTTP Response. It may come from a JavaScript library that was loaded as the result of a subsequent HTTP GET request or as a result of a script in the text/html response.
The fragment identifier in this pattern is intentionally refrerred to as a client parameter.
Treating it as a regular fragment identifier in this usage would result in one incorrectly inferring that the URI for the video resource being addressed is http://www.cnn.com/video.
This would result in all the video links on the CNN site getting the same URI.
Thus, the entire URI in this case is http://www.cnn.com/video/#/video/tech/2008/02/19/vo.aus.sea.spider.ap.
A consumer of this URI who goes looking for an id within the Response that matches the #-suffix of this URI will fail.
The reported Content-Type for the resource is text/html. However, the behavior of the #-suffix in this case is not defined by the HTML specification.
As used, the #-suffix is a first-class client parameter in that it gets consumed by a script that is served as part of the HTML document returned by the server upon receiving a GET request.
This embedded script examines the URI available to it as script variable content.location, strips off the # and uses the rest of the prefix as an argument to a function that generates the actual URI.
Having constructed this content URI, the script then proceeds to instruct the browser to play the media at the newly constructed location.
The content URI is constructed purely for processing by the JavaScript code.
Thus, there is no conflict if the content URI contains a fragment identifier that
is the same as the fragment identifier in the original URI. Errors in processing the content URI would be handled by the JavaScript code.
2.1.2 Extrapolating From This Pattern
The CNN example cited above is not unique with respect to its use of a
fragment identifier
within the URI for encoding parameters to the receiving application. It shows that in a world of dynamic documents, the traditional fragment identifier need no longer be an idref value that addresses an existing node in the serialized HTML making up the HTTP Response. In addition to possibly being a static idref, the fragment identifier in the URI, the pattern demonstrated here and in other uses cases
discussed in this document generalizes to the following:
- An
idref to a dynamically generated node. - A parameter to be consumed by the application that is delivered as the HTTP Response to the original GET request.
2.2 Creating URIs for Media Fragments
The Media Fragments Working Group at W3C is developing a specification to address
spatial and temporal media fragments on the Web: [Media Fragments URI 1.0].
As the title of the specification states, the objective is to create URIs for
media fragments. This is done by using fragment identifiers. For example:
http://www.example.org/video.ogv#t=60,100. In this case, the user agent knows that the primary resource is http://www.example.org/video.ogv and that it is expected to display the portion of the primary resource that relates to the fragment #t=60,100, i.e. seconds 60 to 100. The syntax is very rich and allows media fragments to be identified along a number of different dimensions.
The fragment identifiers are constructed as name-value pairs separated by the & sign.
The relationship between the primary resource and the secondary resource is, in
these cases, quite complicated. Selection of the fragment may involve decompression,
mapping fragment ranges to byte offsets and other complex calculations. Depending on
the format and the capabilities of the user agent, some of this may be done at the
user agent and some at the server. Retrieval of the primary resource may be
accomplished using several requests to conserve bandwidth and adapt to
network conditions.
Retrieving fragments using time ranges (for time fragments) may be done heuristically. For example, if the href is in a video tag. Also, the time range will not apply to anything other than a video stream, which is the desired behaviour. If there is a conflict, the response headers will indicate this -- not the right content-type.
In some cases, for optimization reasons or because the user agent cannot perform the
fragment to byte mapping, only the bytes required for the fragment are retrieved
directly from the server. HTTP header extensions are used to accomplish this.
In summary: the [Media Fragments URI 1.0] specification does create URIs
for media fragments using fragment identifiers but the derivation of the
secondary resource from the primary resource is complex and specialized to the
type of the media stream and the storage format.
This is discussed further in 6 Do These Patterns Violate Existing Specifications?.
2.3 Interaction State and Browser History
A variety of methods are available in Web Architecture to save application state. Cookies store information on the client-side that is sent along with the GET request. Similarly, data can be stored on the server-side -- in a database, for example, identified by a cookie -- and can be used to change the details of the GET request.
There are also specifications under development (See [Web Storage]) that extend the cookie mechanism
in several directions. These specifications allow large amounts of data to be stored
on the client and can also be used to encode application state.
These mechanisms, however, encode private applications states.
In some cases, an application may want to allow selected states to be made public
and shareable. For this we require a URI, appropriately decorated with client-side and server-side parameters. The challenge in designing a mechanism to encode state
is to preserve the familiar user experience especially to
make the back button do the right thing. For live examples of this design pattern, see GMail and Google Maps both of which take extreme care to ensure that the user's expectations of
Web interaction are preserved. These applications use iframe proxies to achieve the desired effect.
A very early interactive Web application was the [Xerox Parc Map Viewer].
When you bring up the application it shows you, by default, a map of the world.
If you select a spot on the map it changes to show you a map centered on the
selected spot. Users can interact with the map in various ways: pan, zoom, select
degree of detail, etc. Each interaction is encoded as a parameter in a URL which is
sent back to the server who generates a new map and refreshes the page. The format
of the URL is:
http://hostname/map/option=value/.../option=value
This application was created before JavaScript or AJAX Toolkits.
Essentially, it creates maps as documents, one for each URL and retains the Web
paradigm of displaying a document for each URL.
Google Maps, though many times more functional
than the simple Map Viewer attempts to retain the same paradigm.
Maps are displayed as documents and each interaction with a map or request for more
information, such as nearby hotels, generates and displays a different map/document.
If you point your browser at http://maps.google.com an an HTML page is returned.
This page loads the maps API which fetches the JavaScript. The JavaScript then
determines the default location, depending on whether Geolocation is enabled, etc.
Based on this, an XHR request is made to the server to pull down the tiles for the default map
and these are used to generate the display.
If you work with Google Maps you
will notice that even after you have customized the map by enetering a different
location, adding nearby attractions or scrolling, panning a or zooming, the
address bar has not changed - it still says http://maps.google.com/. If you want a link to the displayed map, you click the "Link" button on the right and it gives you a URI to the map displayed. For example:
http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=212+Hessian+Hills+Rd,+Croton-on-Hudson,+NY+10520&sll=37.0625,-95.677068&sspn=32.527387,51.679688&ie=UTF8&hq=&hnear=212+Hessian+Hills+Rd,+Croton-on-Hudson,+Westchester,+New+York+10520&z=16
Notice the structure of this URI: it includes the address as well as other parameters.
The URI that Google Maps creates for the customized map has a long query parameter
string but no fragment identifier. These parameters are used by the JavaScript to
fetch the tiles
for the customized map. Some client-side processing may also be performed to
generate the display.
Thus, the Web paradigm is preserved: maps are displayed as documents,
the back button works, each map has a distinct URI that can be transmitted in email or an instant message and used to regenerate the map.
If new information is requested to embellish the map, such as traffic, this
requires additional information
from the server and is correctly encoded as query parameters. Scrolling, panning and
zooming
are performed on the client and could have been encoded as fragment identifiers
but are not.
GMail, on the other hand, uses fragment identifiers to encode the state. For example,
https://mail.google.com/mail/?shva=1#inbox/12c7e6abbc328af4
identifies the inbox and a specific piece of mail in the inbox. If a piece of mail
is not selected, the fragment identifier merely identifies the inbox:
https://mail.google.com/mail/?shva=1#inbox
These mechanisms behind this are discussed below in more detail.
2.4 AJAX Libraries And State Management
AJAX applications use features of Dynamic HTML (DHTML) to create highly reactive user experiences. Updates to the Web user interface in response to user actions no longer require a full page reload. Consequently, the user can perform a sequence of interaction steps while remaining on the same page at least as seen from the browser's perspective of content.location. This makes for a good user experience, except for the following:
- Recording key points in the interaction flow, e.g., for bookmarking.
- Providing intuitive behavior for the browser's history mechanism.
- Snapshotting interaction state to enable returning to a partially completed task at a later time.
Today, many of the details of AJAX programming have been abstracted away by higher level toolkits such as [Dojo AJAX Toolkit] and [google-gwt]. Management of interaction state and browser history is one of the key affordances implemented in these libraries. History mechanisms in AJAX libraries like GWT and Dojo share a lot in common, and the approach can be traced back to Really Simple History (RSH). In addition, the mechanism described here has also been adopted by a recent update to GMail.
The basic premise is to keep track of the application's internal state in the URI fragment identifier. This works because updating the fragment doesn't typically cause the page to be reloaded. This approach has several benefits:
- It is about the only way to control the browser's history reliably.
- It provides good feedback to the user.
- It is
bookmarkable i.e., the user can create a bookmark to the current state and save it, email it, or whatever.
One of the techniques that is used to provide this functionality is to open a number
of frames within a browser window. In such an architecture, parent and child frames
are allowed to change each others' location URI as long as the frames
display
information from the same domain or have agreed to collaborate by some other
means. Otherwise, changing a frame's location URI opens up a cross-site scripting hole.
If the frames can collaborate, then one of the frames, say the parent, passes data to
the child via a fragment identifier by reseting the child's
location URI. Thus, given a parent frame P and a child frame
C , where the location URIs U_P and U_C may come from different domains, the parent frame might pass data to the child by resetting its location URI to U_C#data; the child picks up this data by polling for changes in its location URI. This technique is used in Comet Programming. As an example, the [Dojo AJAX Toolkit] uses an IFrame proxy to enable cross-domain XML HTTP Requests. this is a useful technique when writing cross-site mashups. As an example, see XKCD and AxsJAX a cross-site mashup that mashes together XKCD comics with their associated transcripts to create a speech-friendly XKCD experience.
2.5 Web Command Lines
When applications are built from Web parts, there is often a need to configure them when the application is launched. Traditional applications would call these default start-up or command-line options. We see the equivalent emerging for configuring desktop gadgets and widgets where command-line options are passed in via URI parameters in this context, the URI is the Web command-line. For one sample implementation and its associated usage, see Using URIs To Pass Parameters To The Web. Dave Raggett's HTMLSlidy uses URIs of the form ...#(nn) to address into a deck of slides.
Similarly, [Superfeedr] allows you to subscribe to a fragment of a
document using a fragment identifier, for example, http://www.nytimes.com/weather#.wCurrent%20.summary. [Addrable] uses a fragment identifier to
address into a dataset consisting of comma-separated values.
2.6 The Naked Hash-Ref
In some situations a single # sign is used as the value of the href attribute on HTML anchors. This can be thought of as a relative URI with a null fragment identifier. Web sites wishing to override the default-target behavior of anchors used this for compatibility with older browsers when attaching a JavaScript mouse-click event-handler to anchor elements. This at one time best practice is now considered outdated, though still commonly used. Note that this idiom also creates significant hurdles for non-mouse users of the Web.
3 Manipulating Browser History
As discussed above, when an application moves from one reproducible state to
another, it is desirable to capture the state by creating an appropriate fragment
identifier. It is also desirable to change the URI in the address bar to reflect the
new fragment identifier.
HTML5 introduced the history.pushState() and history.replaceState() methods,
which allow you to add and modify history entries, respectively.
These methods work in conjunction with the window.onpopstate. event. These facilities
allow for changing URIs in the history so that the URI exposed to copy-and-paste still makes sense in contexts without scripting. It can also be used to address the back button concern. See, for example, Manipulating Browser History.
In JavaScript the location.hash and the window.location.hash properties can be used to get and set the characters following the #.
Tracking the onhashchange event enables code that reacts to
changes in the fragment identifier.
6 Do These Patterns Violate Existing Specifications?
Not surprisingly, some of these new uses of fragment identifiers
violate existing specifications.
[RFC 3986] says: "The fragment identifier component of a
URI allows indirect identification of a secondary resource by reference to a
primary resource and additional identifying information.
...
The semantics of a fragment identifier are defined by the set of
representations that might result from a retrieval action on the
primary resource. The fragment's format and resolution is therefore
dependent on the media type [RFC2046] of a potentially retrieved
representation ..."
For HTML and XML the fragment identifier processing rules are defined in
[RFC 2854] and [XPointer] respectively. Essentially,
the fragment identifier is a pointer into a document. If the value of the fragment
identifier equals the id of an element in the document, then it identifies that
element.
In the CNN example, the fragment identifiers designate no element, and, thus, strictly
speaking,
its fragment processing violates [RFC 2854]. But this works in
practice. If the fragment identifier identified an element, it would generate a conflict between the element identified, and the JavaScript code handling the fragment.
Also, still for the CNN example, modifying the behaviour using only the fragment allows caching, and pushing content through CDNs, which is a nice property.
The media fragments case is a bit different. There are no media types
defined for audio or video and, as they point out, many media type specifications
do not specify fragment identifier processing rules. Thus, they do not
explicitly violate
[RFC 3986] although the relationship between primary and secondary
resources is quite a bit more complex than [RFC 3986] describes.
In some cases they fetch the fragment directly from the server rather
than selecting it on the client.
This document discusses highly interactive applications built using JavaScript and Ajax libraries that run on the client and
use fragment identifiers as arguments to functions and in other ways that were not envisaged by [RFC 3986].
Should we condemn such applications as non-conformant?
Or do we need to revise [RFC 3986] to take into account these new usage
patterns and emerging functionality? And should there be different standards
for browsers that support/run JavaScript and those that don't?
RECOMMENDATIONS FOR MEDIA TYPE REGISTRATIONS:
Extend the definition in the media types that accept "active content", like HTML and SVG, to acknowledge the fact that fragment identifiers might also be used (if not in contradiction with the 'static' use of those fragment identifiers) for programmatic purposes.
The media type registration needs to say (for active content) when and how
fragment identifier parameters are passed to the active content and whether or how
they may be used
to identify the portion of the state that is reproducible and can be referenced externally.
All media types, especially new types, must specify fragment identifier semantics
for both static use and use in active content.