How ChatGPT Browses the Web and Whether It Reads Your Favorite Sites

ChatGPT can search the web and read pages when a question needs current information, then answer with links to its sources. It doesn't read everything: it only sees pages that are public and that the site allows it to access.

How ChatGPT search works

1. When your question needs fresh or specific information, ChatGPT runs a web search, or you can ask it to search.
2. It picks results, fetches those pages and reads them.
3. It writes an answer from what it read, with citations linking to the pages.

Its search draws on third-party search providers as well as OpenAI's own index. OpenAI has also launched its own browser, ChatGPT Atlas, and an agent mode that can open and use websites on your behalf.

OpenAI's crawlers

OpenAI uses separate crawlers for separate jobs, and documents each one:

• OAI-SearchBot crawls sites so they can appear in ChatGPT's search results.
• ChatGPT-User fetches a page when a user's request needs it, such as when you paste a link or ChatGPT opens a result.
• GPTBot collects content that may be used to train OpenAI's models.

What ChatGPT can't read

• Pages behind logins and paywalls, unless the content is publicly visible.
• Sites that block OpenAI's crawlers in their robots.txt file, or with firewall rules.
• Some content that only appears after complex scripts or interaction.

So if ChatGPT seems unaware of a site you like, the site may block it, or it may simply not have searched for it.

For site owners: controlling access

Each crawler can be allowed or blocked separately in robots.txt. For example, to stay in ChatGPT search but out of model training:

User-agent: GPTBot
Disallow: /

and leave OAI-SearchBot allowed. Blocking OAI-SearchBot keeps your pages out of ChatGPT search answers and citations. Check your analytics for referrals from chatgpt.com to see how much traffic it sends. A newer convention, an llms.txt file, gives AI tools a summary of your site; the free llms.txt generator makes one.

Using it well as a reader

• Open the cited sources before relying on an answer, since summaries can get details wrong.
• Ask it to search when you need current information, and say which sites you trust.
• For questions where you'd rather pick the sources yourself, a regular search may be better. See how AI search compares with Google.