Robots.txt AI training signal: one line to refuse them all

Remember robots.txt? That tiny text file from 1994 that politely asks crawlers to stay out of your basement, and that every crawler is free to ignore? Well, it just got a promotion. Google, Microsoft, Anthropic and OpenAI have agreed on a shared robots.txt AI training signal: one line that tells all four of them, at once, “don’t train your models on this.”

Yes, all four. The companies that spent the last three years hoovering up the web and explaining that it was all fair use are now offering you an off switch. I’ll let you decide how touched you feel.

How the robots.txt AI training signal works

The idea is almost embarrassingly simple. Until now, if you wanted to opt out of AI training, you had to chase each bot by name: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on, with a new one popping up every time someone launched a startup. The new proposal introduces a single policy token, __usage_ai-training, that you put in the User-Agent line exactly like a bot name. A Disallow means no training, an Allow means go ahead, and it works path by path like any other robots.txt rule.

So blocking training on your whole site, except for your press kit, looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That’s it. No new directives, no new grammar, no JSON manifest to maintain. The spec is an IETF draft called Machine-Readable Training Preference Signal, written by Kevin Kelley from Anthropic, and the four companies announced on October 30 that they’d all honor it “in the coming months”, each on its own schedule. If nobody publishes a new version, the draft expires on May 3, 2027.

Credit where it’s due: this is the first time the big AI labs agree on anything that makes life easier for website owners rather than for themselves. One line to refuse them all is a real improvement over the whack-a-mole we’ve been playing.

The catch (there’s always a catch)

Now for the cold shower, and Next.ink lays it out pretty mercilessly. First, the token doesn’t block crawling at all. Your pages still get fetched; you’re just expressing a wish about what happens to them afterwards. Second, it’s purely declarative. There’s no verification, no audit, no penalty. You will never be able to prove that your Disallow was respected, because you can’t open a model and check what’s inside. It’s a pinky promise, formalized as an RFC.

Third, it only applies to content fetched after you add the line. Everything already scraped since 2021 stays in the training sets. Your blog is in GPT-4, Gemini and Claude whether you like it or not; this signal is about the next model, not the last one.

And fourth, who’s missing from the list? Meta, Apple, Amazon, xAI, plus every Chinese lab and every anonymous scraper running from a datacenter in nowhere-land. The companies that ignore robots.txt today will ignore this token tomorrow, and the ones that signed are the ones with lawyers and reputations to protect. In other words, it binds the people who were already the easiest to deal with.

Should you add it anyway?

Honestly, yes. It costs you thirty seconds and it establishes a paper trail. If a lawsuit or a regulator ever comes asking whether publishers clearly expressed their preferences, a dated robots.txt with that line in it is a much better answer than “well, we blocked GPTBot in 2024 and hoped for the best”. It’s the same logic as locking your bike: it won’t stop a determined thief, but it does settle who’s in the wrong.

What I’d do: keep your existing per-bot rules (they still control crawling, which this token doesn’t), add the __usage_ai-training group on top, and don’t expect miracles. And if you run a site that’s been quietly walling itself off from bots lately, like Reddit did with its login wall, this is the gentler alternative, at least on paper.

The web finally has a “do not train” sign. Now we just need someone to enforce it.