Propagation, Caching and Replication on the Web

This area which is part of the W3C technical areas addresses the definition of advanced protocols performing distribution of information in order to decrease response time, network load, and user management. As the Web has grown and continues to grow at exponential rates, the demand for efficient caching and replication has become a clear MUST in the Web architecture. However, it has also become clear that caching and replication affects all layers of the Internet - not only the application layer where HTTP and the Web normally belongs.

W3C Position Statement

It is time for the Web to become an industrial strength resource available anywhere, anytime. Making the Web more robust is a very complex problem involving resource naming, discovery, replication, and propagation. Many of the problems associated with building a robust Web have been subject to research for a long time and real-world solutions are already deployed. Now it is time for technologies such as multicast, directory services, and caching and replication to come together and to collaborate in order to move the Web forward. Read more about the scalability problems of the Web and W3C's position statement on how to proceed.

Specifications, Drafts, Papers, and Reports

Check out our list of interesting papers related to propagation, caching and replication. Other papers of interest include

• HTTP/1.1 specification - it's here!
• Papers published by the Advanced Network Architecture group at MIT's Laboratory for Computer Science - a MUST read!

Reference Software

W3C produces a set of Web applications in order to demonstrate and experiment with the development of new Web recommendations and specifications. The W3C Jigsaw server written in Java provides full HTTP/1.1 functionality including caching and persistent connections.

Talks and Presentations

• Meeting on Web Efficiency and Robustness april 1996 and the notes from the meeting
• LBL Network Research Group Talks - many very good papers/talks about scalability!

Mailing lists

There are not that many mailing lists dedicated to discussing caching and replication. Most often the discussions are part of the HTTP mailing lists or the generic Web lists.

www-talk@w3.org (Archive)

This is the primary public mailing list for technical discussion among those developing World Wide Web software. It is explicitly intended for the collaborative design of new systems, software, protocols, and documentation which may be useful to the WWW developer community. General questions from non-developers should go to one of the many newsgroups.

MBone related mailing lists (Archives)

MBone and multicast technology has a lot of experience on high scalable propagation of information

IETF Related Information

IETF has traditionally been deeply involved in propagation and caching of information as this is an important part of the Internet architecture. There are many examples of caching and propagation from DNS, directory services, and at the lower level of routing protocols. Check out the current list of working groups for more information.

Web Caching Projects

This is a list of caching projects related to the Web. If you have additions then please mail them!

• Cooperative Hierarchical Object Indexing and Caching for Europe (CHOICE)
• The DESIRE Cache Project as part of the DESIRE project (Development of a European Service for Information on Research and Education)
• The Harvest Project provides a set of tools to gather, extract, organize, search, cache, and replicate relevant information across the Internet
• The Squid Object cache which is derived from the Harvest Project
• The National information caching prototype project from National Laboratory for Applied Network Research (NLANR)
• The UK Academic National Web Cache at HENSA Unix

Information Needed!

If you have information on any of these or related topics then please send me a note!

• Disconnected operation and cache fill for off-line browsing - statistics or papers?
• Caching schemes for low bandwidth. low performance devices like for example cordless PDAs etc. - statistics or papers?
• Integration of resource replication and Quality of Service (QoS) - any attempts of doing this?

Related Areas of Interest

• Addressing - URIs, or Uniform Resource Identifiers, are the technology for addressing documents on the web.
• Demographics - There is a constant tension, in all commercial media, between a desire by the public for privacy and a need for information about the viewing audience.
• What is MBone? This is a really good information site. Also check out our Real Time Area
• Web/News/Mail Integration - how does these technologies relate?
• The Caching entry in the Yahoo Index

Tim Berners-Lee and Henrik Frystyk Nielsen
@(#) $Id: Overview.html,v 1.18 2002/04/17 23:32:54 kdubost Exp $

Copyright  ©  1997 W3C (MIT, INRIA, Keio ), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.