Guidelines for Web Content Transformation Proxies 1s

1 Introduction (Non-Normative)

1.1 Purpose

Within this document Content Transformation refers to the manipulation of requests to, and responses from, an origin server. This manipulation is carried out by proxies in order to provide a better user experience of content that would otherwise result in an unsatisfactory experience on the device making the request.

The W3C Mobile Web Best Practices Working Group neither approves nor disapproves of Content Transformation, but recognizes that is being deployed widely across mobile data access networks. The deployments are widely divergent to each other, with many non-standard HTTP implications, and no well-understood means either of identifying the presence of such transforming proxies, nor of controlling their actions. This document establishes a framework to allow that to happen.

The overall objective of this document is to provide a means, as far as is practical, for users to be provided with at least a "functional user experience" [Device Independence Glossary] of the Web, when mobile, taking into account the fact that an increasing number of content providers create experiences specially tailored to the mobile context which they do not wish to be altered by third parties. Equally it takes into account the fact that there remain a very large number of Web sites that do not provide a functional user experience when perceived on many mobile devices.

1.2 Audience

The audience for this document is creators of Content Transformation proxies and purchasers and operators of such proxies. The document also contains non-normative guidance for content providers whose services may be accessed by means of such proxies.

1.3 Scope

The recommendations in this document refer only to "Web browsing" - i.e. access by user agents that are intended primarily for interaction by users with HTML Web pages (Web browsers) using HTTP. Clients that interact with proxies using mechanisms other than HTTP (and that typically involve the download of a special client) are out of scope, and are considered to be a distributed user agent. Proxies which are operated in the control of or under the direction of the operator of an origin server are similarly considered to be a distributed origin server and hence out of scope.

The W3C Mobile Web Best Practices Working Group (BPWG) is not chartered to create new technology - its role is to advise on best practice for use of existing technology. In satisfying Content Transformation requirements, existing HTTP header fields, directives and behaviors must be respected, and as far as is practical, no extensions to [RFC 2616 HTTP] are to be used.

The recommendations in this document refer to interactions of a proxy and do not refer to any presumed aspects of the internal operation of the proxy. For this reason, the document does not discuss use of "allow" and "disallow" lists (though it does discuss behavior that is induced by the implementation of such lists). In addition it does not discuss details of how transformation is carried out except if this is reflected in interoperability. For this reason, it does not discuss the insertion or insertion of headers and footers or any other specific behaviors (though it does discuss the need for essential user interaction of some form).

1.4 Principles

1.4.1 IAB Considerations

The BPWG made reference to Internet Architecture Board (IAB) work on "Open Pluggable Edge Services" [RFC 3238 OPES] for various principles that underlie behavior of proxies. In this work the IAB expressed its concerns about privacy, control, monitoring, and accountability of such services.

1.4.2 Priority of Intention

The Web allows users considerable flexibility in respect of the representation of content. At the same time, Content Providers may have a preferred manner in which they wish their content to be represented. Content Transformation must reconcile these contrasting factors. In creating this Recommendation the BPWG has determined that Content Transformation proxies should respect Content Providers intentions, where they are expressed, but may allow users to choose other representations, except where Content Providers specifically prohibit this.

The BPWG recognizes that there is neither a systematic vocabulary for Content Provider Intentions, nor a systematic means of expression of such intentions. There is scope for further work in this area (see I Scope for Future Work).

2 Terminology (Normative)

2.1 Types of Proxy

Alteration of HTTP requests and responses is not prohibited by HTTP other than in the circumstances referred to in [RFC 2616 HTTP] Section 13.5.2 and Section 14.9.5.

HTTP defines two types of proxy: transparent proxies and non-transparent proxies. As discussed in [RFC 2616 HTTP] Section 1.3, Terminology:

"A transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification. A non-transparent proxy is a proxy that modifies the request or response in order to provide some added service to the user agent, such as group annotation services, media type transformation, protocol reduction, or anonymity filtering. Except where either transparent or non-transparent behavior is explicitly stated, the HTTP proxy requirements apply to both types of proxies."

This document elaborates the behavior of non-transparent proxies, when used for Content Transformation in the context discussed in [CT Landscape].

2.2 Types of Transformation

Transforming proxies can carry out a wide variety of operations. In this document we categorize these operations as follows:

Alteration of Requests
Transforming proxies process requests in a number of ways, especially replacement of various request header fields to avoid HTTP 406 Status responses (if a server can not provide content that is compatible with the original HTTP request header fields) and at user request.
Alteration of Responses
There are three classes of operation on responses:

3 Conformance (Normative)

3.1 Classes of Product

The Content Transformation Guidelines specification has one class of products:

Transformation Deployment: A Transformation Deployment is the provision of non-transparent components in the path of HTTP requests and responses. Provisions that are applicable to a Transformation Deployment are identified in this document by use of the term "transforming proxy" or "proxy" in the singular or plural.

A Transformation Deployment that wishes to claim conformance must make available a conformance statement B Conformance Statement that specifies the reasons for non-compliance with any clauses containing the key words should and should not, recommended and not recommended.

4 Behavior of Components (Normative)

4.1 Proxy Forwarding of Request

4.1.1 Applicable HTTP Methods

Proxies should not intervene in methods other than GET, POST, HEAD.

User agents sometimes issue HTTP HEAD requests in order to determine if a resource is of a type and/or size that they are capable of handling. A transforming proxy may convert a HEAD request into a GET request (in order to determine the characteristics of a transformed response that it would return if the user agent subsequently issued a GET request for the same resource).

If the HTTP method is altered from HEAD to GET, proxies should (providing such action is in accordance with normal HTTP caching rules) cache the response so that a second GET request for the same content is not required (see also 4.1.4 Serving Cached Responses).

Other than to convert between HEAD and GET proxies must not alter request methods.

4.1.2 `no-transform` directive in Request

If the request contains a Cache-Control: no-transform directive, proxies must not alter the request other than to comply with transparent HTTP behavior defined in [RFC 2616 HTTP] sections section 14.9.5 and section 13.5.2 and to add header fields as described in 4.1.6 Additional HTTP Header Fields below.

Note:

An example of the use of Cache-Control: no-transform is the issuing of asynchronous HTTP requests, perhaps by means of XMLHttpRequest [XHR], which may include such a directive in order to prevent transformation of both the request and the response.

4.1.3 Treatment of Requesters that are not Web browsers

Before altering aspects of an HTTP request proxies need to take account of the fact that HTTP is used as a transport mechanism for many applications other than "Traditional Browsing". Alteration of HTTP requests for those applications can cause serious misoperation.

4.1.4 Serving Cached Responses

Aside from the usual caching procedures defined in [RFC 2616 HTTP], in some circumstances, proxies may paginate responses and where this is the case a request may be for a subsequent page of a previously requested resource. In this case proxies may for the sake of consistency of representation serve stale data but when doing so should notify the user that this is the case and must provide a simple means of retrieving a fresh copy.

4.1.5 Alteration of HTTP Header Field Values

Aside from the usual procedures defined in [RFC 2616 HTTP] proxies should not modify the values of header fields other than the User-Agent, Accept, Accept-Charset, Accept-Encoding, and Accept-Language header fields and must not delete header fields. It must be possible for the server to reconstruct the original User Agent originated header fields by copying directly from the corresponding X-Device header field values (see 4.1.5.5 Original Header Fields).

Other than to comply with transparent HTTP operation, proxies should not modify any request header fields unless:

the user would be prohibited from accessing content as a result of the server responding that the request is "unacceptable" (see 4.2.5 Server Rejection of HTTP Request);
the user has specifically requested a restructured desktop experience (see 4.1.5.3 User Selection of Restructured Experience);
the request is part of a sequence of requests to the same Web site and either it is technically infeasible not to adjust the request because of earlier interaction, or because doing so preserves consistency of user experience.

These circumstances are detailed in the following sections.

Note:

In this section, the concept of "Web site" is used (rather than "origin server") as some origin servers host many different Web sites. Since the concept of "Web site" is not strictly defined, proxies should use heuristics including comparisons of domain name to assess whether resources form part of the same "Web site".

Note:

The heuristics discussed in 4.2.9 Proxy Decision to Transform relating to URI patterns are not part of the decision to alter HTTP Header Field values.

4.1.5.1 Content Tasting

The theoretical idempotency of GET requests is not always respected by servers. In order, as far as possible, to avoid misoperation of such content, proxies should avoid issuing duplicate requests and specifically should not issue duplicate requests for comparison purposes.

4.1.5.2 Avoiding "Request Unacceptable" Responses

A proxy may reissue a request with altered HTTP header field values if a previous request with unaltered values resulted in the origin server rejecting the request as "unacceptable" (see 4.2.5 Server Rejection of HTTP Request). A proxy may apply heuristics of various kinds to assess, in advance of sending unaltered header field values, whether the request is likely to cause a "request unacceptable" response. If it determines that this is likely then it may alter header field values without sending unaltered values in advance, providing that it subsequently assesses the response as described under 4.2.6 Receipt of Vary HTTP Header Field below, and is prepared to reissue the request with unaltered header fields, and alter its subsequent behavior in respect of the Web site so that unaltered header fields are sent.

A proxy must not reissue a POST request with altered header fields when the response to the unaltered POST request has HTTP status code 200 (in other words, it may only send the altered request for a POST/PUT request when the unaltered one resulted in an HTTP 406 response, and not a "request unacceptable" response).

4.1.5.3 User Selection of Restructured Experience

Proxies may offer users an option to choose to view a restructured experience even when a Web site offers a choice of user experience. If a user has made such a choice then proxies may alter header field values when requesting resources in order to reflect that choice, but must, on receipt of an indication from a Web site that it offers alternative representations (see G.1.4.2 Indication of Intended Presentation Media Type of Representation), inform the user of that and allow them to select an alternative representation.

Proxies should assume that by default users will wish to receive a representation prepared by the Web site. Proxies must assess whether a user's expressed preference for a restructured representation is still valid if a Web site changes its choice of representations (see 4.2.6 Receipt of Vary HTTP Header Field).

4.1.5.4 Sequence of Requests

When requesting resources that are included resources (e.g. style sheets, images), proxies should make the request for such resources with the same User-Agent header field as the request for the resource from which they are referenced.

For the purpose of consistency of representation, proxies may request linked resources (e.g. those referenced using the a element) that form part of the same Web site as a previously requested resource with the same header fields as the resource from which they are referenced.

When requesting linked resources that do not form part of the same Web site as the resource from which they are linked, proxies should not base their choice of header fields on a consistency of presentation premise.

4.1.5.5 Original Header Fields

When forwarding an HTTP request with altered HTTP header fields proxies must include in the request copies of the unaltered header field values in the form "X-Device-"<original header name>. For example, if the User-Agent header field has been altered, an X-Device-User-Agent header field must be added with the value of the received User-Agent header field.

Specifically the following mapping must be used:

Original	Replacement	Ref
`User-Agent`	`X-Device-User-Agent`	RFC2616 Section 14.43
`Accept`	`X-Device-Accept`	RFC2616 Section 14.1
`Accept-Charset`	`X-Device-Accept-Charset`	RFC2616 Section 14.2
`Accept-Encoding`	`X-Device-Accept-Encoding`	RFC2616 Section 14.3
`Accept-Language`	`X-Device-Accept-Language`	RFC2616 Section 14.4

Note:

The X-Device- prefix was chosen primarily on the basis that this is a already existing convention. It is noted that the values encoded in such header fields may not ultimately derive from a device, they are merely received fields. The treatment of received X-Device header fields, which may happen where there are multiple transforming proxies, is undefined (see I Scope for Future Work).

4.1.6 Additional HTTP Header Fields

Irrespective of the presence of a no-transform directive:

proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header field;
proxies must include a Via HTTP header field (see 4.1.6.1 Proxy Treatment of Via Header Field).

4.1.6.1 Proxy Treatment of `Via` Header Field

Proxies must (in accordance with RFC 2616) include a Via HTTP header field indicating their presence and should indicate their ability to transform content by including a comment in the Via HTTP header field consisting of the URI "http://www.w3.org/ns/ct".

When forwarding Via header fields, proxies should not alter them by removing comments from them.

Note:

According to [RFC 2616 HTTP] Section 14.45 Via header field comments "may be removed by any recipient prior to forwarding the message". However, the justification for removing such comments is based on memory limitations of early proxies. Most modern proxies do not suffer such limitations.

4.2 Proxy Forwarding of Response to User Agent

In the following, proxies must check for the presence of equivalent <meta http-equiv> elements in HTML content, if the relevant HTTP header field is not present.

4.2.1 Applicable Responses

Proxies should not intervene in the response if the request method was not HEAD, GET or POST.

4.2.2 User Preferences

Proxies must provide a means for users to express preferences for inhibiting content transformation. Those preferences must be maintained on a user by user and Web site by Web site basis. Proxies must solicit re-expression of preferences in respect of a server if the server starts to indicate that it offers varying responses as discussed under 4.2.6 Receipt of Vary HTTP Header Field.

4.2.3 Receipt of `Cache-Control: no-transform`

If the response includes a Cache-Control: no-transform directive then proxies must not alter it other than to comply with transparent HTTP behavior as described in [RFC 2616 HTTP] Section 13.5.2 and Section 14.9.5 and other than as follows.

If a proxy determines that a resource as currently represented is likely to cause serious misoperation of the user agent then it may advise the user that this is the case and must provide the option for the user to continue with unaltered content.

4.2.4 Use of `Cache-Control: no-transform`

Proxies may use Cache-Control: no-transform to inhibit transformation by further proxies.

4.2.5 Server Rejection of HTTP Request

Proxies may treat responses with an HTTP 200 Status as though they were responses with an HTTP 406 Status if it has determined that the content (e.g. "Your browser is not supported") is equivalent to a response with an HTTP 406 Status.

4.2.6 Receipt of `Vary` HTTP Header Field

A proxy may not be carrying out content tasting as described under 4.1.5.2 Avoiding "Request Unacceptable" Responses if it anticipates receiving a "request unacceptable" response. However, if it makes a request with altered header fields in these circumstances, and receives a response containing a Vary header field referring to one of the altered header fields then it should request the resource again with unaltered header fields. It should also update whatever heuristics it uses so that unaltered header fields are presented first in subsequent requests for this resource.

4.2.7 Link to "handheld" Representation

If the response is an HTML response and it contains a <link rel="alternate" media="handheld" /> element, a proxy should request and process the referenced resource, unless the resource referenced is the current resource as determined by the presence of link elements.

Note:

Note that an empty href attribute indicates a "same document reference", whereas the URI from which the document was retriecved does not.Need confirmation or otherwsie of this under Francois's ACTION-983

4.2.8 WML Content

If the content is WML proxies should act in a transparent manner.

Note:

This does not affect the operation of proxies that are also WAP Gateways.

4.2.9 Proxy Decision to Transform

In the absence of a Vary or no-transform directive (or a meta HTTP-Equiv element containing Cache-Control: no-transform) proxies should not transform content matching the following rules unless the user has specifically requested transformation:

the content is HTML and contains <link rel="alternate" media="handheld" href=""/>
the DOCTYPE of the content (if it has one) indicates XHTML-MP, XHTML Basic, WML or iMode as listed in D DOCTYPEs Associated with Mobile Content.
the Content-Type has a value listed in C Internet Content Types associated with Mobile Content.
the URI of the response (following redirection or as indicated by the Content-Location HTTP header field) matches a pattern listed in E URI Patterns Associated with Mobile Web Sites.
a claim of mobileOK Basic [mobileOK Basic Tests] conformance is indicated (see [mobileOK Scheme] for how such a claim may be indicated);

Other factors that a proxy may take into account:

The Web site (see note) has previously shown that it is contextually aware, even if the present response does not indicate this;
the user agent has features (such as linearization or zoom) that allow it to present the content unaltered;
the response contains client side scripts that may misoperate if the resource is restructured;
the response is an HTML response and it includes <link> elements specifying alternatives according to presentation media type.

4.2.9.1 Alteration of Response

Note:

Other than as noted in this section the nature of restructuring that is carried out, any character encoding alterations and what is omitted and what is inserted is, as discussed in 1.3 Scope, out of scope of this document.

If a proxy alters the response then:

It must add a Warning 214 Transformation Applied HTTP header field;
The altered content should validate according to an appropriate published formal grammar and if XML must be well-formed;
It should indicate to the user that the content has been transformed for mobile presentation and provide an option to view the original, unmodified content.