Cloudflare's AI crawler wall went live, and Googlebot is caught in it
Future Technology
New article published
AI
Cloudflare's AI crawler wall went live, and Googlebot is caught in it
Cloudflare's crawler policy, announced back in July, took effect on 15 September. New domains, new sites from existing customers, and every existing free tier customer now block Training and Agent category crawlers by default on any page that displays ads. Search crawling stays allowed.
Key Takeaways
- From 15 September, Cloudflare blocks Training and Agent category crawlers by default on any page showing ads, for new domains and all existing free customers.
- Googlebot crawls for both search indexing and AI training under one user agent, so blocking Training can also block it unless you opt out.
- Search crawling stays allowed by default, and site owners can override the defaults in either direction.
You received this because you subscribe to Future Technology.