What is the AI Scan Permission widget?

The AI Scan Permission widget in diib® lets you know if LLM agents can crawl your website and display your website as a resource for queries. 

If your result on the widget says "Blocked," it means that your website cannot be accessed by LLM bots.

If your result on the widget says, "Allowed," it means that your website can be accessed by LLM bots and your site could be displayed in LLM results. 

When should you discourage LLM agents from accessing your site?

If you don't want your content to show in ChatGPT, Claude, etc., you should discourage LLM agents from crawling your site.

How do you block LLMs from accessing your website. 

Here are the components of a robots.txt that will discourage LLMS from accessing your website:

User-Agent: *
Content-signal: search=yes,ai-train=no
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

Should you block LLM agents from accessing your website?

If you want your website to be found, you should not block LLM agents from accessing your website.