Content Transformation Guidelines 1d

1.1 Purpose

From the point of view of this document Content Transformation is the manipulation in various ways, by proxies, of content as delivered by an origin server with a view to making it more suitable for mobile presentation.

The W3C MWI BPWG neither approves nor disapproves of Content Transformation, but recognizes that is being deployed widely across mobile data access networks. The deployments are widely divergent to each other, with many non-standard HTTP implications, and no well-understood means either of identifying the presence of such transforming proxies, nor of controlling their actions.

This document establishes a framework to allow that to happen.

A more extensive discussion of the requirements for these guidelines is discussed in "Content Transformation Landscape" [CT-Landscape].

1.2 Scope

Note that we are not chartered to create new technology.

And that we are talking only about browsing.

1.3 Audience

Browser Vendors, Transforming Proxy Vendors and Operators, Content Providers.

2.1 Summary of Requirements

Needs more work.

The purpose of this section is to summarize the communication requirements of actors (users, user agents, transforming proxies, and origin servers) to communicate with each other. The relevant scenario involving a content transformation proxy is as follows:

user agent <---HTTP---> one or more content transformation proxies <---HTTP---> origin server

The needs of these actors are as follows:

1. The user agent needs to be able to tell the content transformation proxy [@@and the origin server]:

   1. what type of mobile device and what user agent is being used;

   2. what media-type (presentation format e.g. desktop, handheld) the user desires;

   3. that all content transformation should be avoided, or that reformatting is allowed/desired.

2. The content transformation proxy needs to be able to tell the origin server:

   1. that some degree of content transformation (re-coding and reformatting) can be performed;

   2. that content transformation will be carried out unless instructed not to;

   3. that content is being requested on behalf of something else [@@?? and what that something else is];

   4. that the request headers have been altered (e.g. additional content types inserted).

3. The origin server needs to be able to tell the content transformation proxy:

   1. that it varies its presentation according to device type and other factors;

   2. that it's permissible or otherwise to perform content transformation of various kinds;

   3. that it has media-specific representations;

   4. that is unable or unwilling to deal with the request in its present form.

4. The content transformation proxy needs to be able to tell the client browser:

   1. that it has applied transformations of various kinds to the content;

   2. how to access the untransformed representation of the content.

5. The content transformation proxy needs to be able to interact with the user:

   1. to allow the user to disable its features;

   2. to alert the user to the fact that it has transformed content and to allow access to an untransformed representation of the content.

2.2 Objectives

In satisfying these requirements existing HTTP headers and directives and behaviors must be respected. However, not all of the features required can be achieved without extensions to the behaviors defined in [RFC 2616]. Knowing that many actors will be unaware of any HTTP extensions, special consideration needs to go into making sure that the fall-back behavior - i.e. strict adherence to HTTP/1.1 - is "safe". For example, if there is no standard way for a client browser to specify that all content transformation should be avoided in a request, then we must define a default behavior for a well-behaved content transformation proxy that receives a request from such a client.

[@@ other principles behind what we are trying to do - e.g. noting Sean's point that there is a wide diversity of different devices that all fall under the simple appellation of "handheld".]

2.3 Types of Proxy

Alteration of HTTP requests and responses is not prohibited by HTTP other than in the circumstances referred to in [HTTP] section 13.5.2. This document describes how the Client and the Destination Server may require conforming transforming proxies not to alter HTTP requests and responses.

HTTP defines two types of proxy: transparent proxies and non-transparent proxies. As discussed in Section 1.3 [HTTP], Terminology:

"A transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification. A non transparent proxy is a proxy that modifies the request or response in order to provide some added service to the user agent, such as group annotation services, media type transformation, protocol reduction, or anonymity filtering. Except where either transparent or non-transparent behavior is explicitly stated, the HTTP proxy requirements apply to both types of proxies."

This document elaborates the behavior of non-transparent proxies, when used for content transformation in the context discussed in [Content Transformation Landscape] and henceforward referred to as transforming proxies.

2.4 Proxy States

A transforming proxy is viewed as being in one of two states in respect to a client and a server. In the active state it may transform content and manipulate HTTP headers. In the passive state it behaves like a transparent proxy and behaves as though a Cache-Control: no-transform directive were present on every request and every response, with the possible exception that - only with the consent of both the user and the content provider - content which it has been determined would cause serious mis-operation of the client, such as causing it to crash, may be minimally transformed to prevent that mis-operation.

This must steer clear of recommending a deviation from the HTTP spec which I don't think is acceptable. However there are parallels with the operation of e.g. child protection mechanisms?

2.5 Types of Transformation

Transforming proxies can carry out a wide variety of operations. To carry out an exhaustive survey of those operations and to discuss means of server or client side control of them is beyond the scope of this document. In this document we categorize this rich vocabulary of possible operation into two types:

1. Alteration of Request Headers;

2. Alteration of Responses.

Alteration of responses is further sub-categorized into

1. Restructuring content including rewriting URIs;

2. Recoding content;

3. Optimizing content.

Restructuring content is a process whereby the original layout is altered so that content is added or removed or where the spatial or navigational relationship of parts of content is altered, e.g. by linearization or pagination. It includes also rewriting of URIs to route via the proxy.

Recoding content is a process whereby the layout of the content remains the same, but details of its encoding may be altered. Examples include re-encoding HTML as XHTML, correcting invalid markup in HTML, conversion of images between formats (but not, for example, reducing animations to static images).

Optimizing content means removing redundant white space, re-compressing images (without loss of fidelity), zipping for transfer ...

2.6 Control by User and Client

A transforming proxy as described in this document must offers a level of control to users irrespective of whether their browser is aware of the conventions of this document. It interacts at least in a limited way to turn itself off, and may offer other features via interaction. Interaction of this kind is essential where the users client does not implement the HTTP conventions of this document. The interaction may supplement those HTTP conventions.

In addition to allowing control by means of HTTP conventions and interaction with the user, transforming proxies may also allow control by other means such as administrative arrangements.

2.7 Control by Server

A transforming proxy gains knowledge of whether a server permits alteration of requests and responses in a number of ways. For requests, by having previously received from the origin server [for a resource on the path that this request is in scope of] an indication of what degree of content transformation is permissible. For responses as a result of the response containing indications as to the servers intentions - such indications include use of HTTP conventions, and mobileOK labels.

3.1 Client Origination of Request

The client may request that the Content-Type and Content-Encoding must not be altered in the response by setting the Cache-Control: no-transform directive.

The client may add an [@@allow-recode directive] to the Cache-Control: no-transform directive, indicating that the proxy may change the format of the response but not restructure the content.

The client may add a [@@allow-https-rewrite=yes] parameter to the [@@allow-recode] directive to allow the rewriting of the https scheme portion of any URIs - the default value is "no" in the absence of this parameter or in the absence of the [@@allow-recode directive].

The client may add an [@@allow-compress] to the Cache-Control: no-transform directive, meaning that a proxy may remove redundant white space, re-compress images or change the Content-Encoding (to use gzip, from identity, for example).

The client may also add [@@preferred-medium directive] indicating that a preference for a presentation style. The [@@preferred-medium directive] has the form media=presentation-format (as described in RFC ..., current values of the presentation format-directive are taken from IANA ... and include "screen" and "handheld").

In the absence any of the previous directives elaborating the no-transform directive, the client should indicate that it understands the conventions of this document by including a [@@ct-proxy-aware directive].

[It would also be useful if the client were able to turn off the User Interaction features of the proxy via HTTP]

3.2 Proxy Receipt, Forwarding or response to a Request

If the incoming request contains neither a no-transform nor an [@@allow-transform] directive and the proxy intends to enter the active mode, the proxy must provide the ability for the user to interact with it in some way in order to instruct it to enter the passive mode. Whether the proxy does this by responding to the incoming request, decorating the response or by some other means is beyond the scope of this document.[@@see 4.2.3 below for requirements of this interaction]

Irrespective of the presence of the no-transform directive, the proxy must behave transparently (q.v.) if it detects that the user agent is not a browser [@@open question as to how it does that].

If the request contains a no-transform directive for a resource that has already been served to the client, it may respond with a cached untransformed copy of the resource, providing that serving that response is in accordance with the cache-control directives that the server attached to the untransformed response.

If the request contains a [@@reload-untransformed] directive the server must not forward the request to the server and must respond with an untransformed cached copy of the resource, irrespective of cache-control directives attached to the resource. [@@this needs careful consideration but I think is probably OK. A number of justifications come to mind. A related issue is the reuse by browsers of images that appear multiple times in a page, but the images prohibit caching - in practice the same image appears to be re-used.] If the proxy is unable to respond with the untransformed resource it must respond with an HTTP 406 response to indicate this.

If the request contains a Cache-Control: no-transform directive [@@or any of the other directives specified in previous section] the proxy must forward the request unaltered to the server, other than to comply with transparent HTTP behavior and in particular to add a Via HTTP header.

The proxy must (in accordance with compliance to RFC 2616) include a Via HTTP header indicating its presence, and must indicate its capabilities using the [@@transforming-proxy-capability] mechanism.

If there are no [@@ transformation related directives] present in the request from the client, and there is no indication from a downstream proxy that it intends to transform [@@ see I will transform below] the proxy should analyze whether it intends to offer transformation services by referring to any administrative arrangements that are in place with the user of the client, or the server, and any a priori knowledge it has of client capabilities [@@ from a DDR and so on]. Knowing that the client has available a linearization or zoom capability and/or supports a broad range of content formats the proxy should not offer to recode content.

If as a result of this deliberation it intends to (restructure / reformat / compress) the proxy must indicate this by including a [@@@ I will transform (restructure / reformat / compress)] - [@@ and even if it doesn't it may indicate its potential for restructuring or recoding or compressing content [@@by means of ...].

Proxies must not intervene in HTTPS requests and should not intervene in methods other than [@@we have an open question here as to which methods are applicable].

A proxy should not alter HTTP requests unless not doing so would result in the users request being rejected by the origin server (this includes HTTP 406 status as well as HTTP 200 status, saying that the request cannot be handled - e.g. "Your browser is not supported").

e.g. Accept: image/jpeg, image/gif, image/png;q=0.7;[@@transform] 

e.g. Accept: image/jpeg, image/gif, image/png;q=0.7 

3.3 Server Response to Proxy

Servers should distinguish URIs that are intended for access only by HTTPS from those that are intended for insecure access in order to be able to detect and reject requests that should have been submitted by HTTPS but have been re-written in contravention of its directives.

If the server varies its presentation according to examination of received HTTP Headers then it must include a Vary HTTP header indicating this to be the case. If, in addition to, or instead of HTTP headers, the server varies its presentation on other factors (source IP Address ...) then it must include a * as one of the fields in the Vary response.

The server must include a no-transform directive if one is received from the client. If it is capable of varying its presentation it should take account of client capabilities [@@as derived from a DDR etc.] and formulate an appropriate experience according to those criteria.

If the server has distinct presentations according to its perception of the presentation media, then the medium for which the presentation is intended should be indicated [@@using the ...]

If the client has requested a specific presentation using the [@@ directive] the server should provide a presentation of that kind. e.g. if the server would ordinarily provide a handheld experience but the client requests a screen experience the screen experience should be provided. And vice versa, of course.

If the server creates a specific user experience for certain presentation media types it should inhibit transformation of the response by including a no-transform directive. The server should not prohibit recoding or compression of its content unless it has specific reasons not to allow it [including that this has been requested by the client] and hence should in general add a [@@allow-recoding or allow-compression] directive when adding a no-transform directive.

If the response contains URIs with the scheme https and the server is content to allow the scheme to be re-written as the http scheme then it must indicate this using the [@@allow-https-rewrite] directive, otherwise rewriting is inhibited.

Note that including a no-transform directive may [@@should actually] disrupt the behavior of WAP/WML proxies, because this inhibits such proxies from converting WML to WMLC (because this is a content-encoding behavior). Adding [@@allow-recoding] or [@@allow-compression] is unlikely to be recognized in the short-term by such proxies which predate these guidelines.

Servers may base their actions on a priori knowledge of behavior of transforming proxies, when they are identified in a Via header.

The server should not choose a Content-Type for its response based on its assumptions about the heuristic behavior of any intermediaries. (e.g. it should not choose content-type: application/vnd.wap.xhtml+xml solely on the basis that it suspects that transforming proxies will apply heuristics that make them not restructure it).

If servers provide only limited variants of presentation they should consider providing a rich presentation and allowing a transforming proxy to reduce this - which may result in a richer experience for the user than providing a basic handheld experience only, say.

406 Response - Note that some clients (MSIE for instance) don't display the body of a 406 response, this is in contravention of HTTP/1.1 as far as I can see.

Vary headers in 406 response - restrict to the one(s) that have caused the 406.

In general, successful responses should are done with 200 OK Vary: User-Agent, Accept, Accept-Language etc. e.g. MS doesn't want you to do updates except with IE. so they should say 406 Vary: User-Agent (but note that IE doesn't display the body of 406 responses)

Servers should respond with a 406 not a 200 if they can't handle the request and should indicate that they permit header alteration in that 406. Servers should provide information about alternative representations by using the Vary header (if the alternatives are available from the same URI) or using link information if alternative representations are handled by different URIs. [This restricts to HTML for now. If link headers a reinstated in HTTP then this becomes a more universal mechanism. Open question as to whether it SVG or WICD etc. support any such notion]

[@@300 Response - could this be used as a signal from the server to say that it understands the protocol? A la RFC 2295]

3.4 Proxy Receipt and Forwarding of Response from Server

If the proxy has altered any of the HTTP request headers, and it receives a Vary response from the server it should re-make the request with the original headers and forward the subsequent response without restructuring it, irrespective of the contents of the subsequent response. The proxy should take note of this and should not vary headers for subsequent requests, unless requests are subsequently received with no Vary header [@@ + note on back off below]

[@@note that loop detection and elimination is needed here]

3.5 Proxy Response to Client

If the response includes a Warning: 214 Transformation Applied the proxy must not apply further transformation.

If the response includes a Cache-Control: no-transform directive that is not modified by [@@ other directives on recoding] then the response must be forwarded to the client unaltered other than in the respects noted for transparent operation of HTTP proxies as specified in RFC2616, and in particular the addition of a Via HTTP header [@@which includes, or is in addition to a [@@transforming-proxy-capability] ...].

In the absence of a Vary or no-transform directive the proxy should apply heuristics to the content to determine whether it is appropriate to restructure or recode it (in the presence of such directives, heuristics should not be used.)

If the proxy alters the content then it must add a Warning: 214 Transformation Applied HTTP Header. [@@ should this be elaborated to say what kind of transformation?]

If the proxy has transformed (reformatted) the content but not rewritten https links it should annotate those links to indicate that transformation service is not available on them.

A proxy should strive for the best possible user experience that the client supports. It should only alter the format, layout, dimensions etc. to match the specific capabilities of the client. For example, when resizing images, they should only be reduced so that they are suitable for the specific client, and this should not be done on a generic basis.

In the passive mode (as well as in the active mode), if the proxy determines that the resource as currently represented is likely to cause serious mis-operation of the client then the proxy may transform the resource but only sufficiently to alter the specific aspect of the content that is likely to cause mis-operation. Proxies must not exhibit this behavior unless this has been specifically allowed by both the server and the user. [@@ either by persistent registration of preferences, or by use of the [@@correct dangerous content] directive.]

3.6 Client Action on Receipt of Response

[@@ discussion of what to do on receipt of Warnings etc.]

3.7 Summary of Proposed Features

[@@allow-recode directive] to the Cache-Control: no-transform directive, indicating that the proxy may change the format of the response but not restructure the content

[@@allow-https-rewrite=yes] parameter to the [@@allow-recode] directive to allow the rewriting of the https scheme portion of any URIs - the default value is "no" in the absence of this parameter or in the absence of the [@@allow-recode directive]

[@@allow-compress] to the Cache-Control: no-transform directive, meaning that a proxy may remove redundant white space, re-compress images or change the Content-Encoding (to use gzip, from identity, for example).

[@@preferred-medium directive] indicating that a preference for a presentation style. The [@@preferred-medium directive] has the form media=presentation-format (as described in RFC ..., current values of the presentation format-directive are taken from IANA ... and include "screen" and "handheld")

[@@ct-proxy-aware directive]

[@@reload-untransformed] directive the server must not forward the request to the server and must respond with an untransformed cached copy of the resource,

[It would also be useful if the client were able to turn off the User Interaction features of the proxy via HTTP]

[@@@ I will transform (restructure / reformat / compress)]

[@@ and even if it doesn't it may indicate its potential for restructuring or recoding or compressing content [@@by means of ...][@@transforming-proxy-capability]

in addition,[@@extension] add a further transform parameter indicating that the format is not natively supported by the client.

e.g. Accept: image/jpeg, image/gif, image/png;q=0.7;[@@transform] 

[@@ User Agent Modified indication with the Original User-Agent indicated]