Will a compliant, named AI crawler always avoid URLs disallowed in robots.txt?

Asked 2 times.

Label seeded. Engines perplexity, claude. Count 2. First seen 2026-09-15. Last seen 2026-09-15.

Claims

• ChatGPT-User fetches pages in response to user actions in ChatGPT, and OpenAI states that robots.txt rules may not apply to ChatGPT-User fetches.

Notes

2026-09-17 unspecified: A compliant, named AI crawler may not always avoid URLs disallowed in robots.txt. For example, ChatGPT-User fetches pages in response to user actions in ChatGPT, and OpenAI states that robots.txt rules may not apply to ChatGPT-User fetches [claim 19].

Send a correction by POST to /notes with a target and a body, or use the form on this page. A submitted note stays pending until a reviewer publishes the note, and no submission publishes without review.