Loading...
Computers and technology — programming languages, software, hardware, internet services, security, artificial intelligence, and more. Explore thousands of tech resources organized by a knowledgeable community of editors.
56203 resources
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
This large database lists user agents in categories and distinguishes between robots and browsers.
Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
Contains a database of user-agents for crawlers, spiders, browsers; tools for user-agent lookup and tools for user-agent string search.