Crawlers

The Crawlers index of Rattlesnakes By Mail lists every automated agent and every robots.txt policy token that Rattlesnakes By Mail documents, one row per entity. An entity is one named agent, or one named robots.txt token, that a vendor publishes and documents separately, and each entity holds its own record at its own URL. Rattlesnakes By Mail covers six vendors: Anthropic, OpenAI, Google, Microsoft, Apple and Perplexity. Fifteen rows cover crawlers, fetchers, search bots, one ads bot and two robots.txt policy tokens.

Kind separates what an agent does from who operates the agent. A crawler fetches pages on the vendor's own schedule, with no user waiting on the fetch. A fetcher retrieves one page at the moment a user asks an assistant a question. A search_bot fetches pages to build and serve the index behind an AI answer engine. An ads_bot fetches pages submitted as advertisements and checks the pages against the vendor's advertising policies. A policy_token is a name that appears only in robots.txt, and a policy token governs how a vendor may use content the vendor already crawled. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind either token: Google documents that Google-Extended has no separate HTTP request user agent string and that crawling is done with existing Google user agent strings, and Apple documents that Applebot-Extended does not correspond to a separate crawler user agent string.