See also:

• Computers: Internet: Abuse: Spam: Preventing  (65)
• Computers: Internet: Web Design and Development: Authoring: Online Tools: robots.txt  (6)
• Computers: Software: Internet: Site Management: Log Analysis  (280)

All About Search Indexing Robots and Spiders - - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

HTTP User Agent Index - - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.

List of Robot Agent Strings - - A list from PGTS of Web robots with the identifying data they leave in Web site logs.

Psychedelix List of User-Agents - - Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.

Robot IP Address - - Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.

Robots, Spiders and Other User Agents: a Resource for WebMasters - - José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.

Search Engine IP Addresses - - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.

Search Engine Robots and Other User Agents - - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.

Spider Track - - Tracks and displays real time hits and IP addresses of MSNbot, Yahoo's Slurp, and Googlebot. Provides aggregate hits of each spider over the last 30 days.