Webbot - The Libwww Mini Robot

The Basics

Author(s)

Henrik Frystyk Nielsen

Status

An example application for libwww. The robot is the primary tool used for the HTTP performance measurements.

Plans

New releases follow the releases of libwww with no exception.

Platforms

It runs on the same platforms as libwww

Getting the Source Code

The webbot comes as an integral part of the libwww distribution package.

Forums

Discussions on libwww and its example applications take place on <www-lib@w3.org> (archives). See the documentation for how to subscribe.

Features

• Really fast libwww HTTP/1.1 engine - uses HTTP/1.1 pipelining which speeds up performance a lot.
• Smart use of GET and HEAD - only downloads the strictly necessary
• Check of inlined images as well as hyperlinks
• Support for regular expressions allows detailed Web traversal constraints
• Support for SQL based logging which allows for an open-ended set of thongs you can do with the logs
• Support for traditional file based logging using Common Log File formats and Referer log file formats
• A wealth of logging facilities including:
  • Missing or empty ALT tags in inlined images
  • Broken links
  • Document titles based on the TITLE HTML tag and the Title HTTP header field
  • Link relationships between documents based on the LINK HTML tag
  • Modification dates based on the Last-Modified HTTP header field
  • Distributions of content-types and charsets encountered in the traversal

How to Run It

Be careful - this is a robot and hence can be used to traverse many links - it should not be used for anything but HTTP/1.1 testing. Especially, it doesn't follow directions indicated in robot.txt files and hence should not be used on the Internet at large.

The robot has a large set of command line options that can be used in a large set of different combinations. You can try and see this simple script in order to see an example of how it can be run.