This document highlights our experiences with HTTP compliance testing and proposes models for including third-party HTTP compliance tests in W3C QA activity. We intend to discuss these issues at the W3C Workshop on Quality Assurance.

Table of Contents

1. Background
2. Co-Advisor project
    2.1 Basic principles
    2.2 Current status
    2.3 Plans
3. Cooperation with W3C
    3.1 Status quo
    3.2 TMF proposal
4. Conclusions

1. Background

HTTP (RFC 2616) is the core protocol of the Web. Nearly all Web transactions rely on HTTP. The Measurement Factory (TMF) team has been working on various HTTP tests since 1998. Our first project dealt with HTTP performance issues. We have developed Web Polygraph, a sophisticated performance benchmark for caching proxies. Web Polygraph is now used to test and tune most of the caching proxies on the market, as well as Web accelerators, L4/7 load balancers, filtering products, and other HTTP intermediaries. Polygraph is available to anybody in source code form at no charge.

HTTP is a complex protocol. It is practically impossible to verify compliance by code analysis or passive traffic monitoring. Many vendors claim HTTP compliance, but it is a common belief, confirmed by several research studies, that most of the HTTP implementations are not fully compliant. On the other hand, compliance and compatibility of HTTP agents are very important because of the scope of HTTP deployment and because implementation bugs are often costly to fix or work around (e.g., replacing a popular broken browser on all workstations of a large corporation is very expensive and cumbersome). Many environments contain mission-critical HTTP devices that cannot be bypassed by users or turned off by sysadmins. New HTTP agent implementations still appear at a high rate, increasing the number of variables in the equation.

HTTP differs from many of the W3C specifications: it is a communication protocol and not just a format or ``language'' specification. The complexity of the protocol and the dynamic nature of the required tests probably explain why no decent HTTP compliance tests have been developed.

The following sections discuss our approach to HTTP compliance testing and propose models for W3C-TMF cooperation.

2. Co-Advisor project

The majority of TMF clients and collaborators needed an HTTP compliance test suite. This demand resulted in the ``Co-Advisor'' project that we describe in this section.

2.1 Basic principles

Co-Advisor is a collection of tests and a driver program to execute the tests. The tool primarily targets Web intermediaries, but can be tuned to work with HTTP servers or even clients given sufficient demand. At the early stages of the project, it became clear that Co-Advisor has to go beyond formal HTTP compliance to be useful in practice because most developers have to deal with (and check for) HTTP bugs in the software that is beyond their control (e.g., a buggy browser communicating with a compliant origin server). Thus, they need to make sure that not only their HTTP implementation is compliant (nice for marketing materials), but that it compatible with existing, possibly non-compliant implementations (essential for real-world operation).

Early user input also indicated that it is impossible to design a single set of tests that will satisfy the majority of Co-Advisor users. Several, partially overlapping, test sets must be developed to cover major application areas: HTTP client/proxy/balancer/server tests, formal HTTP compliance, compatibility with existing agent implementations, ICAP and other HTTP-enabled protocol compliance, and others.

The Co-Advisor design (driver software plus a collection of test sets) is the result of these observations. Co-Advisor checks for compatibility with existing Web products, compliance with protocol specs, agreement with good-practice standards, susceptibility to vendor-specific bugs, and other requirements. New tests are added based on user demand/interest; the test set is determined by actual user needs. Test groups can be formed (e.g., ``HTTP/1.1 compliance test suite'' or ``Netscape Navigator compatibility test suite''). Co-Advisor supplies users with client, server, and other agent implementations so that the tests do not rely on or require third-party software/hardware besides the device under the test itself.

While technical details about HTTP compliance testing are beyond the scope of this position paper, we would be happy to discuss them during the W3C QA workshop. After developing Squid caching proxy and Web Polygraph benchmark, we did not expect many surprises from the HTTP compliance testing project. We were wrong.

The Co-Advisor project is seeded by membership fees. Project members get certain privileges. Non-paying users get restricted free access to the tool. Details about the project can be found at http://www.measurement-factory.com/products/co-advisor/.

2.2 Current status

At the time of writing, the following companies expressed their interest: Akamai, Cisco, F5, InterX, Lucent, LodBroker, Mirror Image, and Novell. We have also discussed possible cooperation with the folks from AT&T and HP research labs.

TMF is currently evaluating a suite of compliance tests provided to us by one of the caching vendors. The tool contains 256 test cases and is designed to work with caching proxies. The prototype has been used with five products. Several HTTP implementation bugs have already been found and many potential problems await further inspection. The prototype was also extremely useful for polishing the design of the future Co-Advisor tool.

Here are the ``totals'' statistics of the prototype application to Squid caching proxy:

	Passed Tests    : 157
	Failed Tests    :  98
	Aborted Tests   :   1
	Total Tests     : 256

2.3 Plans

Our immediate plans include three major items:

1. Attract more project members to get sufficient seed funding.
2. Develop the first version of the tool, keeping in mind the lessons learned from prototype applications.
3. Attempt to cooperate with W3C on the tool's functionality and distribution.

3. Cooperation with W3C

3.1 Status quo

We would like to make Co-Advisor tools widely available. TMF does not spend resources on advertising and marketing while making testing tools freely available. This means that we can benefit from W3C distributing the tool while W3C can benefit from having a quality HTTP compliance test under the belt.

Next come the politics and the questions of control. TMF has always worked in cooperation with interested parties, allowing them to contribute to design and the feature set of the benchmarks, but we want to have the overall control of the development of our tools. Unfortunately, current W3C policies seem to disallow adoption of the tools not owned by W3C. We believe that this restriction is artificial and rigid. In our opinion, it does not make much sense to spend membership fees of both organizations on developing two very similar tools. A wiser approach would be to join the efforts, not double the costs. However, the cooperation model must give W3C sufficient rights and control over the tool to be supported by W3C members and management.

It appears that W3C may benefit from more flexible cooperation models in areas other than HTTP compliance. More open software developers are likely to contribute their tools to W3C collection if the ownership restrictions are adjusted. This observation makes the discussion even more useful as it may create a precedent that will lead to a long-term positive change.

3.2 TMF proposal

TMF wants to control the tool development. W3C needs to control the tools it distributes or endorses. The conflict can be resolved by dividing the tool into two parts. Specifically, TMF retains control over the Co-Advisor driver program while W3C owns a collection of tests. The driver can be used with other collections. The W3C collection can be owned and developed by W3C, independently from the tool itself.

Clearly, some amount of glue is needed to protect both parties from arbitrary unilateral actions such as vast modifications of the interfaces or distribution practices. This glue can be provided by a simple contract or other form of legal agreement where both parties pledge to cooperate with each-other.

It may be important to note that similar arrangements already exist in practice: companies may own C programs without owning a C compiler; Qwest Communications owns its Yellow Pages without owning phone numbers or company names; FAQ and book authors often own the collection of the tool-related resources without owning the tool itself.

We will be happy to discuss alternatives.

4. Conclusions

TMF is developing an HTTP compliance test suite. We have a working prototype that already found bugs in several production HTTP agents. We would like to cooperate with W3C on test suite design, development, and distribution. TMF proposes a way how this cooperation can be arranged to benefit both parties and the Web community in general.