Why it matters
One line in a file you never open can hide you from an assistant.
Every website can publish a small file called robots.txt that tells visiting programs where they may go. AI assistants send their own programs, each with its own name, and their search crawlers read that file before they read your pages.
Some of those rules were written years ago, by a theme, a plugin or a well-meaning setting, to keep out every program the site did not recognise. The assistants your customers use today were not around to be recognised.
Not every block is a mistake. Keeping your pages out of model training is a fair choice, and it does not hide you from anybody's search. So the answer splits the two: the crawlers that decide whether an assistant can show you, and the ones that only collect pages for training.
What we check
8 crawlers, by the name each one answers to.
- OAI-SearchBot (OpenAI). Finds pages to show and link in ChatGPT's search answers.
- ChatGPT-User (OpenAI). Opens a page when somebody asks ChatGPT about it.
- PerplexityBot (Perplexity). Finds pages to show and link in Perplexity's answers.
- Bingbot (Microsoft). Reads pages for Bing search.
- GPTBot (OpenAI). Collects pages to train OpenAI's future models.
- ClaudeBot (Anthropic). Collects pages that may be used to train Anthropic's models.
- Google-Extended (Google). Lets Google use your pages to train Gemini and to ground Gemini's answers.
- Applebot-Extended (Apple). Lets Apple use your pages to train the models behind Apple Intelligence.
We also look for llms.txt, a short plain-text guide to a site written for AI assistants. It is optional, and not every assistant reads it yet.
We read the two files the way a crawler does and nothing else. A firewall or bot protection setting at your host can also turn these crawlers away, and that never shows in either file. The full free check opens your pages too.
The long version, with how to check it yourself: can AI read your website.
One question answered. The full check asks sixteen.
The free check reads your whole website the way an AI assistant does and tells you, in the same plain words, what it can and cannot do with your business today.