AI Bot Directory / Claude-Web

What is Claude-Web?

Claude-Web is Anthropic's on-demand fetcher: it hits a page in real time when a person asks the assistant about it. Every hit is a human, right now, asking an AI about your content.

How to spot Claude-Web in your logs

Look for this token in the request User-Agent (case-insensitive):

claude-web

Should you allow or block Claude-Web?

Blocking it breaks real-time page reads when users ask the AI about your site. Almost nobody benefits from blocking assistant fetchers.

To block it anyway, add this to your robots.txt:

User-agent: Claude-Web
Disallow: /

Note: robots.txt is a request, not a lock — well-behaved crawlers honor it; verify with your own traffic data.