Should I Block AI Bots From Crawling My Site?

For most businesses, the answer is no. AI tools can only recommend or cite your website if their bots can read it. If you block them, you can disappear from AI answers, and more and more of your customers are searching there.

Not all AI bots do the same job

• Answer and search bots read your pages so an AI tool can cite you when someone asks a question. Examples: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), and PerplexityBot (Perplexity).
• Training bots collect content to train AI models. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Gemini models) and CCBot (Common Crawl).

Will blocking AI bots hurt my Google rankings?

Blocking training bots won't change your Google or Bing rankings. Google says blocking Google-Extended doesn't affect Google Search, and that includes AI Overviews. Blocking answer and search bots, though, can keep you out of AI answers. Never block Googlebot, or you'll drop out of Google Search entirely.

If you only want to stop AI training

Add the lines below to your robots.txt file. Your site stays open to AI answer tools, so they can still cite you.

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: meta-externalagent
Disallow: /

Check your current setting

Your diib dashboard's AI Scan Permission widget shows whether AI bots can reach your site. Learn more: What is the AI Scan Permission widget?

Not sure how to edit robots.txt? Ask your web developer or hosting provider. They can usually change it in a few minutes.