Content Transformation Guidelines 1.0

1 Introduction (Non-Normative)

1.1 Purpose

From the point of view of this document, Content Transformation is the manipulation in various ways, by proxies, of requests made to and content delivered by an origin server with a view to making it more suitable for mobile presentation.

The W3C Mobile Web Best Practices Working Group neither approves nor disapproves of Content Transformation, but recognizes that is being deployed widely across mobile data access networks. The deployments are widely divergent to each other, with many non-standard HTTP implications, and no well-understood means either of identifying the presence of such transforming proxies, nor of controlling their actions. This document establishes a framework to allow that to happen.

The overall objective of this document is to provide a means, as far as is practical, for users to be provided with at least a "functional user experience" [Device Independence Glossary] of the Web, when mobile, taking into account the fact that an increasing number of content providers create experiences specially tailored to the mobile context which they do not wish to be altered by third parties. Equally it takes into account the fact that there remain a very large number of Web sites that do not provide a functional user experience when perceived on many mobile devices.

The user agent needs to be able to tell the Content Transformation proxy and the origin server:
1. what type of mobile device and which user agent is being used;
2. that all Content Transformation should be avoided.
The Content Transformation proxy needs to be able to tell the origin server:
1. that some degree of Content Transformation (restructuring and recoding) can be performed;
2. that content is being requested on behalf of something else and what that something else is;
3. that the request headers have been altered and what the original ones were.
The origin server needs to be able to tell the Content Transformation proxy:
1. that it varies the representation of its responses according to device type and other factors;
2. that it is not permissible to perform Content Transformation;
3. that it has media-specific representations;
4. that is unable or unwilling to deal with the request in its present form.
The Content Transformation proxy needs to be able to tell the user agent:
1. that it has applied transformations of various kinds to the content.
The Content Transformation proxy needs to be able to interact with the user:
1. to allow the user to disable its features;
2. to alert the user to the fact that it has transformed content and to allow access to an untransformed representation of the content.

Note:

A more extensive discussion of the requirements for these guidelines can be found in "Content Transformation Landscape" [CT Landscape].

2 Terminology (Normative)

2.1 Types of Proxy

Alteration of HTTP requests and responses is not prohibited by HTTP other than in the circumstances referred to in [RFC 2616 HTTP] Section 13.5.2.

HTTP defines two types of proxy: transparent proxies and non-transparent proxies. As discussed in [RFC 2616 HTTP] Section 1.3, Terminology:

"A transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification. A non-transparent proxy is a proxy that modifies the request or response in order to provide some added service to the user agent, such as group annotation services, media type transformation, protocol reduction, or anonymity filtering. Except where either transparent or non-transparent behavior is explicitly stated, the HTTP proxy requirements apply to both types of proxies."

This document elaborates the behavior of non-transparent proxies, when used for Content Transformation in the context discussed in [CT Landscape].

2.2 Types of Transformation

Transforming proxies can carry out a wide variety of operations. In this document we categorize these operations as follows:

Alteration of Requests
Transforming proxies process requests in a number of ways, especially replacement of various request headers to avoid HTTP 406 Status responses (if a server can not provide content that is compatible with the original HTTP request headers) and at user request.
Alteration of Responses
There are three classes of operation on responses:

3 Conformance (Normative)

3.1 Classes of Product

The Content Transformation Guidelines specification has two classes of products:

Content Deployment: A Content Deployment is the provision of resources intended for retrieval by user agents. Provisions that are applicable to a Content Deployment are identified in this document by use of the terms "origin server", "server" and "Web site" in the singular or plural.
Transformation Deployment: A Transformation Deployment is the provision of non-transparent components in the path of HTTP requests and responses. Provisions that are applicable to a Transformation Deployment are identified in this document by use of the term "transforming proxy" or "proxy" in the singular or plural.

3.2 Normative and Informative Parts

Normative parts of this document are identified by the use of "(Normative)" following the section name. Informative parts are identified by use of "(Non-Normative)" following the section name.

3.3 Normative Language for Conformance Requirements

The key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional in this Recommendation have the meaning defined in [RFC 2119].

3.4 Content Deployment Conformance

A Content Deployment conforms to these guidelines if it follows the statements in 4.2 Server Response to Proxy.

3.5 Transformation Deployment Conformance

A Transformation Deployment conforms to these guidelines if it follows the statements in 4.1 Proxy Forwarding of Request, 4.3 Proxy Forwarding of Response to User Agent and 5 Testing (Normative).

4 Behavior of Components (Normative)

4.1 Proxy Forwarding of Request

4.1.1 Applicable HTTP Methods

Proxies should not intervene in methods other than GET, POST, HEAD and PUT.

User agents sometimes issue HTTP HEAD requests in order to determine if a resource is of a type and/or size that they are capable of handling. A transforming proxy may convert a HEAD request into a GET request (in order to determine the characteristics of a transformed response that it would return if the user agent subsequently issued a GET request for the same resource).

If the HTTP method is altered from HEAD to GET, proxies should (providing such action is in accordance with normal HTTP caching rules) cache the response so that a second GET request for the same content is not required (see also 4.1.4 Serving Cached Responses).

4.1.2 `no-transform` directive in Request

If the request contains a Cache-Control: no-transform directive proxies must forward the request unaltered to the server, other than to comply with transparent HTTP behavior and as noted below (see 4.1.6 Additional HTTP Headers).

Note:

An example of the use of Cache-Control: no-transform is the issuing of asynchronous HTTP requests, perhaps by means of XMLHTTPRequest [XHR], which may include such a directive in order to prevent transformation of both the request and the response.

4.1.3 Treatment of Requesters that are not Web browsers

Proxies must act as though a no-transform directive is present (see 4.1.2 no-transform directive in Request) unless they are able positively to determine that the user agent is a Web browser. The mechanism by which a proxy recognizes the user agent as a Web browser should use evidence from the HTTP request, in particular the User-Agent and Accept headers.

4.1.4 Serving Cached Responses

Proxies should follow standard HTTP procedures in respect of caching and should use cached copies of resources where this is in accordance with those procedures.

In some circumstances, proxies may paginate responses and where this is the case a request may be for a subsequent page of a previously requested resource. In this case proxies may for the sake of consistency of representation serve stale data but when doing so should notify the user that this is the case and should provide a simple means of retrieving a fresh copy.

4.1.5 Alteration of HTTP Header Values

Other than to comply with transparent HTTP operation, proxies should not modify request headers unless:

the user would be prohibited from accessing content as a result of the server responding that the request is "unacceptable" (see 4.3.3 Server Rejection of HTTP Request);
the user has specifically requested a restructured desktop experience;
the request is part of a sequence of requests to the same Web site and either it is technically infeasible not to adjust the request because of earlier interaction, or because doing so preserves consistency of user experience.

These circumstances are detailed in the following sections.

Note:

In this section, the concept of "Web site" is used (rather than "origin server") as some origin servers host many different Web sites. Since the concept of "Web site" is not strictly defined, proxies should use heuristics including comparisons of domain name to assess whether resources form part of the same "Web site".

4.1.5.1 Content Tasting

The theoretical idempotency of GET requests is not always respected by servers. In order, as far as possible, to avoid mis-operation of such content, proxies should avoid issuing duplicate requests and specifically should not issue duplicate requests for comparison purposes.

4.1.5.2 Avoiding "Request Unacceptable" Responses

A proxy may reissue a request with altered HTTP header values if a previous request with unaltered values resulted in the origin server rejecting the request as "unacceptable" (see 4.3.3 Server Rejection of HTTP Request). A proxy may apply heuristics of various kinds to assess, in advance of sending unaltered header values, whether the request is likely to cause a "request unacceptable" response. If it determines that this is likely then it may alter header values without sending unaltered values in advance, providing that it subsequently assesses the response as described under 4.3.4 Receipt of Vary HTTP Header below, and is prepared to reissue the request with unaltered headers, and alter its subsequent behavior in respect of the Web site so that unaltered headers are sent.

A proxy must not re-issue a POST/PUT request with altered headers when the response to the unaltered POST/PUT request has HTTP status code 200 (in other words, it may only send the altered request for a POST/PUT request when the unaltered one resulted in an HTTP 406 response, and not a "request unacceptable" response).

4.1.5.3 User Selection of Restructured Experience

Proxies may offer users an option to choose to view a restructured experience even when a Web site offers a choice of user experience. If a user has made such a choice then proxies may alter header values when requesting resources in order to reflect that choice, but must, on receipt of an indication from a Web site that it offers alternative representations (see 4.2.3.2 Indication of Intended Presentation Media Type of Representation), inform the user of that and allow them to select an alternative representation.

Proxies should assume that by default users will wish to receive a representation prepared by the Web site. Proxies must assess whether a user's expressed preference for a restructured representation is still valid if a Web site changes its choice of representations (see 4.3.4 Receipt of Vary HTTP Header).

4.1.5.4 Sequence of Requests

When requesting resources that form part of the representation of a resource (e.g. style sheets, images), proxies should make the request for such resources with the same headers as the request for the resource from which they are referenced.

For the purpose of consistency of representation, proxies may request linked resources (e.g. those referenced using the a element) that form part of the same Web site as a previously requested resource with the same headers as the resource from which they are referenced.

When requesting linked resources that do not form part of the same Web site as the resource from which they are linked, proxies should not base their choice of headers on a consistency of presentation premise.

4.1.5.5 Original Headers

When forwarding an HTTP request with altered HTTP headers proxies must include in the request copies of the unaltered header values in the form "X-Device-"<original header name>. For example, the User-Agent header has been altered, an X-Device-User-Agent header must be added with the value of the received User-Agent header.

Note:

The X-Device- prefix was chosen primarily on the basis that this is a already existing convention. It is noted that the values encoded in such header may not ultimately derive from a device, they are merely received headers. The treatment of received X-Device headers, which may happen where there are multiple transforming proxies, is undefined (see D Scope for Future Work).

4.1.6 Additional HTTP Headers

Irrespective of the presence of a no-transform directive:

proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header;
proxies must include a Via HTTP header (see 4.1.6.1 Proxy Treatment of Via Header).

4.1.6.1 Proxy Treatment of `Via` Header

Proxies must (in accordance with compliance to RFC 2616) include a Via HTTP header indicating their presence and should indicate their conformance to this Recommendation by including a comment in the Via HTTP header consisting of the URI "http://www.w3.org/ns/ct".

When forwarding Via headers proxies should not alter them in any way.

Note:

According to [RFC 2616 HTTP] Section 14.45 Via header comments "may be removed by any recipient prior to forwarding the message". However, the justification for removing such comments is based on memory limitations of early proxies, most modern proxies do not suffer such limitations.

4.2 Server Response to Proxy

4.2.1 Use of HTTP 406 Status

Servers should respond with an HTTP 406 Status (and not an HTTP 200 Status) if a request cannot be satisfied with content that meets the criteria specified by values of the HTTP request headers.

4.2.2 Server Origination of `Cache-Control: no-transform`

Servers must include a Cache-Control: no-transform directive if one is received in the HTTP request.

Servers should include a Cache-Control: no-transform directive if, for any reason, they wish to inhibit transformation of the response.

Note:

Including a Cache-Control: no-transform directive can disrupt the behavior of WAP/WML proxies, because it can inhibit such proxies from converting WML to WMLC.

4.2.3 Varying Representations

Servers should take account of user agent capabilities and formulate an appropriate experience according to those capabilities. Servers should provide a means for users to select among available representations, should default to the last selected representation and should provide a means of changing the selection.

4.2.3.1 Use of `Vary` HTTP Header

If a server varies its representation according to examination of received HTTP headers then it must include a Vary HTTP header indicating this to be the case. If, in addition to, or instead of HTTP headers, a server varies its representation based on other factors (e.g. source IP Address) then it must, in accordance with [RFC 2616 HTTP], include a Vary header containing the value '*'.

Servers may base their actions on knowledge of behavior of specific transforming proxies, as identified in a Via header, but should not choose an Internet content type for a response based on an assumption or heuristics about behavior of any intermediaries. (e.g. a server should not choose Content-Type: application/vnd.wap.xhtml+xml solely on the basis that it suspects that proxies will not transform content of this type).

4.2.3.2 Indication of Intended Presentation Media Type of Representation

If a server has distinct representations that vary according to the target presentation media type, it should inhibit transformation of the response by including a Cache-Control: no-transform directive (see 4.2.2 Server Origination of Cache-Control: no-transform).

In HTML content it should indicate the medium for which the representation is intended by including a link element identifying in its media attribute the target presentation media types of this representation and setting the href attribute to a valid local reference (i.e. use the fragment identifier (see [RFC 3986] section 3.5) added to the URI of the document being served to point to a valid target within the document).

In addition it should include link elements identifying the target presentation media types of other available representations by setting the media attribute to indicate those representations and the href attribute to a URI without a fragment identifier.

Note:

The presence of link elements which do not contain a valid local reference does not indicate one way or another whether this representation is formatted for the presentation media types listed.

Note:

Some examples of the use of the link element are included below in B Example Transformation Interactions.

4.3 Proxy Forwarding of Response to User Agent

4.3.1 Receipt of `Cache-Control: no-transform`

If the response includes a Cache-Control: no-transform directive then proxies must not alter it other than to comply with transparent HTTP behavior and other than as follows.

If a proxy determines that a resource as currently represented is likely to cause serious mis-operation of the user agent then it may advise the user that this is the case and must provide the option for the user to continue with unaltered content.

4.3.2 Receipt of `Warning: 214 Transformation Applied`

If the response includes a Warning: 214 Transformation Applied HTTP header, proxies must not apply further transformation.

4.3.3 Server Rejection of HTTP Request

For compatibility with servers that do not implement this Recommendation (see 4.2.1 Use of HTTP 406 Status), a proxy may treat responses with an HTTP 200 Status as though they were responses with an HTTP 406 Status if it has determined that the content (e.g. "Your browser is not supported") is equivalent to a response with an HTTP 406 Status.

4.3.4 Receipt of `Vary` HTTP Header

If, in response to an HTTP request with altered headers that was not preceded by an HTTP request with unaltered headers, a proxy receives a response containing a Vary header referring to one of the altered headers then it should request the resource again with unaltered headers, it should update whatever heuristics it uses so that unaltered headers are presented first in subsequent requests for this resource and it should resume the behavior described under 4.1.5.2 Avoiding "Request Unacceptable" Responses to avoid rejection of subsequent requests.

4.3.5 Link to "handheld" Representation

If the response is an HTML response and it contains a <link rel="alternate" media="handheld" /> element, the CT-proxy should request and process the referenced resource, unless the resource referenced is the current resource as determined by the presence of link elements as discussed under 4.2.3.2 Indication of Intended Presentation Media Type of Representation.

4.3.6 Proxy Decision to Transform

In the absence of a Vary or no-transform directive (or a meta HTTP-Equiv element containing Cache-Control: no-transform) proxies should apply heuristics to the response to determine whether it is appropriate to restructure or recode it (in the presence of such directives, heuristics should not be used.)

Examples of heuristics:

The Web site (see note) has previously shown that it is contextually aware, even if the present response does not indicate this;
a claim of mobileOK Basic [mobileOK Basic Tests] conformance is indicated;
the Content-Type or other aspects of the response (such as the DOCTYPE) are known to be specific to the device or class of device;
Examples of mobile specific DOCTYPEs:
```
-//OMA//DTD XHTML Mobile 1.2//EN
```
```
-//WAPFORUM//DTD XHTML Mobile 1.1//EN 
```
```
-//WAPFORUM//DTD XHTML Mobile 1.0//EN
```
```
-//W3C//DTD XHTML Basic 1.1//EN
```
```
-//W3C//DTD XHTML Basic 1.0//EN)
```
the user agent has linearization or zoom capabilities or other features which allow it to present the content unaltered;
the URI of the response (following redirection or as indicated by the Content-Location HTTP header) indicates that the resource is intended for mobile use (e.g. the domain is *.mobi, wap.*, m.*, mobile.* or the leading portion of the path is /m/ or /mobile/);
the response contains client-side scripts that may mis-operate if the resource is restructured;
the response is an HTML response and it includes <link> elements specifying alternatives according to presentation media type.

4.3.6.1 Alteration of Response

A proxy should strive for the best possible user experience that the user agent supports. It should only alter the format, layout, dimensions etc. to match the specific capabilities of the user agent. For example, when resizing images, they should only be reduced so that they are suitable for the specific user agent, and this should not be done on a generic basis.

If a proxy alters the response then:

It must add a Warning 214 Transformation Applied HTTP header;
The altered content should validate according to an appropriate published formal grammar;
It should indicate to the user that the content has been transformed for mobile presentation and provide an option to view the original, unmodified content.

4.3.6.2 HTTPS Link Re-writing

If the response contains links whose URIs have the scheme https proxies may only rewrite them so that they can transform the content of linked resources, if the following provision is met. If a proxy does rewrite such links, it must advise the user of the security implications of doing so and must provide the option to avoid decryption and transformation of the resources the links refer to.

If a proxy re-writes HTTPS links, replacement links must have the scheme https.

Note:

For clarity it is emphasized that it is not possible for a transforming proxy to transform content accessed via an HTTPS link without breaking end to end security.

5 Testing (Normative)

Operators of transforming proxies should make available interfaces that facilitate testing of Web sites accessed through them and should make such interfaces available through normal Internet access paths.