Free tool·Keep your work out of AI training sets·200 crawlers · list refreshed each build

A robots.txt for the bots that scrape for AI.

Every crawler below identifies itself by name when it visits a site. This builds the block list for yours from the community-maintained ai.robots.txt list, grouped by who operates each bot. Untick anything you want to allow (AI search crawlers send some sites traffic; training crawlers send nothing back), then download or copy the file.

Your robots.txt

Also: the machine-readable opt-out

The W3C’s TDM Reservation Protocol lets a site state that text-and-data mining rights are reserved, which the EU’s copyright rules recognise. Two files: /.well-known/tdmrep.json and a tag in each page’s head.

[
  { "location": "/", "tdm-reservation": 1 }
]
<meta name="tdm-reservation" content="1">

What this does and doesn’t do

  1. robots.txt is a request, not a wall. Operators that say they respect it (noted per bot above where known) will stop; the ones that don’t need a firewall rule or your host’s bot controls. Your web host or CDN usually has a one-click “block AI crawlers” setting that enforces the same list at the network edge.
  2. It only stops future crawls. Work already collected stays in the datasets it was collected into. The reservation file above is the legal signal; the block list is the technical one. Use both.
  3. Check it worked. After uploading, open yoursite.com/robots.txt in a browser and look for the names. Server logs will show which bots still visit.
  4. Re-run this every few months. New crawlers appear constantly; the list here is rebuilt from the maintained source each time the site refreshes (200 entries today).

Sources

Crawler names, operators and behaviour notes: the ai.robots.txt project (MIT licence), fetched 2026-10-24. The reservation format: the W3C TDM Reservation Protocol community group. Nothing you select here is sent anywhere; the file is assembled in your browser.