Meta recently backed away from controversial website-crawling rules after publishers and regulators pushed back. But here’s the thing: Meta’s AI crawlers are still out there. Their hunger is endless. They creep. They crawl. THEY’RE UNDER THE BED!!!!!!
Maybe not, but they are everywhere. And if you’re not thrilled about your content potentially helping train someone else’s AI, there are ways to make those bots unwelcome.
1. Put Up a Digital No Trespassing Sign
Most websites have a file called robots.txt. It tells web crawlers where they’re allowed to go.
Add these instructions to discourage Meta’s AI crawlers:
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-WebIndexer
Disallow: /The first targets Meta’s AI training and product-improvement crawler. The second targets its AI search indexer. It’s free and takes minutes to implement.
The catch? robots.txt relies on crawlers following the rules. It’s a polite request, not a locked door.
2. Bring Out the Bouncer
Want stronger protection? Cloudflare offers free tools that can actively block identified AI crawlers before they reach your website.
Its AI Crawl Control dashboard lets you identify and block specific bots. You can also distinguish AI training crawlers from AI search crawlers, helping protect your content without unnecessarily sacrificing discoverability.
See Cloudflare’s AI Crawl Control guide.
3. Send Bad Bots Into a Maze
Here’s the fun one. Cloudflare has a free feature called AI Labyrinth. It plants invisible links that can lure misbehaving bots into a maze of AI-generated pages.
The bots waste crawling effort following useless links, while ordinary visitors never see them. It’s basically giving an unwanted scraper an endless supply of homework.
AI Labyrinth doesn’t itself block every crawler, but it’s an entertaining addition to your defensive toolkit.
The Bottom Line
You can’t guarantee that publicly accessible content will never be copied. But you can make it considerably harder for unwanted AI crawlers to collect it, and reduce the server resources they consume.
Meta and the rest of the AI industry can learn to knock first.
Wondering what started the controversy? Read how Meta backed down from its proposed website-crawling rules after publishers pushed back.
For more practical AI guides, news and tools, bookmark MonkeyButtAI.com.
