Let AI crawlers in, or go invisible in AI answers

robots.txt often blocks GPTBot, PerplexityBot, and ClaudeBot by accident. Audit it with one prompt, then serve real HTML.

Blocking AI crawlers in robots.txt is usually accidental (a security plugin, a broad block-bad-bots rule). If the crawler that reads your page can't get in, no AI engine can ever cite you. This is the cheapest, highest-leverage GEO fix, and most teams have never checked it.

Most sites invisible to ChatGPT, Perplexity, and Google AI Overviews aren't losing on content. They block the crawler before it reads a single page.

AI can read your robots.txt and tell you exactly which bots are shut out. But it can't see your live file unless you paste it or let it browse, so give it the real thing.

In this lesson

Fastest path: one prompt, end to end
Or do it in 5 steps
What blocking actually costs, and two traps

What you'll learn

Pull your live robots.txt
Match each AI bot to allow or block
Never confuse Google-Extended with Googlebot
Edit and re-deploy
Re-verify with curl