Rattlesnakes By Mail

A Field Guide to the Crawlers of the Web

Rattlesnakes By Mail is a reference on how AI crawlers and AI search engines read the web. Each page on Rattlesnakes By Mail is a record, and each fact on a record is a dated claim. Three formats serve the same content: HTML, Markdown at the .md suffix, and JSON at the .json suffix. The same URL answers Accept: text/markdown and application/json. Claim 47, for example, states: ClaudeBot honours industry standard robots.txt directives that signal do not crawl. Claim 47 carries confidence high and the method vendor_doc, quotes Anthropic's crawler documentation dated 2026-04-07, and was verified on 2026-09-13.

Observations

Rattlesnakes By Mail logs each request from an identified crawler and publishes per-crawler daily counts at /observed. ClaudeBot made 1142 requests in the last 30 days and 100% of the requests were verified against Anthropic's published IP ranges. An unverified request means the request IP address did not match a published range, and an unverified request is not evidence that the user agent was spoofed.

Claims

Every fact on Rattlesnakes By Mail is a dated claim with a verbatim vendor quote, a source URL, a method and a confidence, published at /claims. A published claim is never edited: a change creates a new claim that supersedes the old claim, both claims stay addressable, and every supersession is listed at /changes.

Questions

Rattlesnakes By Mail records every search query made on the site, matches each query against the published claims, and publishes the unmatched queries as gaps at /questions. A seeded question carries the label seeded, and an organic question carries the source the query arrived from.

Data

Rattlesnakes By Mail exports each public table nightly as JSON and as CSV under CC BY 4.0 to the public GitHub repository crank-box/rattlesnakesbymail-data. The dataset and the nightly export are described at /data.

Blocking

Disallowing Google-Extended in robots.txt stops Google from using already-crawled content to train Gemini models and to ground Gemini features.

Google documents that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal in Google Search.

Disallowing GPTBot in robots.txt signals to OpenAI that crawled content is not to be used to train OpenAI's generative AI foundation models, and allowing OAI-SearchBot alongside that disallow keeps a site eligible to appear in ChatGPT search results.

A GPTBot disallow does not govern ChatGPT-User, because OpenAI documents that ChatGPT-User fetches a page when a user asks ChatGPT a question and that robots.txt rules may not apply to a user-initiated fetch.

Crawler identity is checked against the IP ranges the vendor publishes, and Google and Apple document a reverse DNS method in addition.

Crawlers

Which AI crawlers exist, and what each crawler does

The Crawlers index of Rattlesnakes By Mail lists every automated agent and every robots.txt policy token that Rattlesnakes By Mail documents, one row per entity. An entity is one named agent, or one named robots.txt token, that a vendor publishes and documents separately, and each entity holds its own record at its own URL. Rattlesnakes By Mail covers six vendors: Anthropic, OpenAI, Google, Microsoft, Apple and Perplexity. Fifteen rows cover crawlers, fetchers, search bots, one ads bot and two robots.txt policy tokens.

Kind separates what an agent does from who operates the agent. A crawler fetches pages on the vendor's own schedule, with no user waiting on the fetch. A fetcher retrieves one page at the moment a user asks an assistant a question. A search_bot fetches pages to build and serve the index behind an AI answer engine. An ads_bot fetches pages submitted as advertisements and checks the pages against the vendor's advertising policies. A policy_token is a name that appears only in robots.txt, and a policy token governs how a vendor may use content the vendor already crawled. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind either token: Google documents that Google-Extended has no separate HTTP request user agent string and that crawling is done with existing Google user agent strings, and Apple documents that Applebot-Extended does not correspond to a separate crawler user agent string.

Changes

How AI crawler documentation has changed, by date

The Changes log of Rattlesnakes By Mail records every movement in the claim ledger, one row per claim that entered the ledger, moved within the ledger, or left the ledger. A change row names the date, the entity, the kind of change, the old value and the new value. Kinds read as follows: new marks a claim published for the first time, updated marks a claim whose value changed, superseded marks a claim replaced by a newer claim, retired marks a claim withdrawn without a replacement, and disputed marks a claim a reviewer has flagged against contrary evidence.

Observations

AI crawler traffic data from the last 30 days

The table below ranks every crawler by requests logged on Rattlesnakes By Mail in the last 30 days, with the share Rattlesnakes By Mail could verify against each vendor's published identity.

Questions

Questions about AI crawlers with no documented answer

A gap is a question that matched no published claim, and a gap appears at /questions only once an editor publishes it.

62 questions have been recorded as gaps since 2026-09-13, 49 of the questions are published at /questions, and the question raised most often so far is: Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform?

Related

Every claimed field on every crawler has its own page at /fields.

Any two crawlers can be compared side by side at /compare.