AI Agents Behind Cloudflare: How to Stop reCAPTCHA Loops on Your Site

Why AI Agents Behind Cloudflare Hit reCAPTCHA Loops

We’ve been running a fleet of AI agents against our own site — content audits, SEO scans, page generators, draft pushes. They all hit sengo.com over HTTPS, exactly like a regular visitor would. That worked fine for months. Then one morning, mobile visitors started reporting a Cloudflare-style overlay on the homepage with the message “This site is exceeding reCAPTCHA Enterprise free quota.”

Login still worked. Wp-admin was fine. But the public marketing pages were broken for a slice of real visitors. After ruling out our security plugin and our contact form’s CAPTCHA, we found the real source: our agents themselves. Every script we ran was, from Cloudflare’s perspective, a robot — and a robot that came back hundreds of times a day from a single residential IP.

That’s the first lesson about AI agents behind Cloudflare: your own automation is hostile traffic by default. Cloudflare doesn’t know your laptop’s curl call is friendly. It sees the same fingerprint patterns it sees from credential stuffing tools and scrapers — and it treats them the same way.

How Cloudflare’s Bot Protection Sees Agent Traffic

Cloudflare’s bot management runs on layered signals — User-Agent strings, JA3/JA4 TLS fingerprints, header order, behavioral patterns, and IP reputation. The naive defenses you might reach for fail predictably.

• Spoofing User-Agent isn’t enough. Cloudflare correlates UA with TLS fingerprint. A request that claims to be Chrome but uses curl’s TLS profile is flagged instantly.
• Rotating IPs doesn’t help small teams. Residential proxies are easy to detect, and a single developer hammering one IP gets reputation-flagged within hours.
• Even legitimate libraries are obvious. Python requests, Node fetch, Go’s http.Client — each ships a recognizable TLS handshake.

On the Pro plan, this enforcement is called Super Bot Fight Mode. It classifies traffic into Definitely automated, Likely automated, and Verified bots, and lets you choose an action for each. By default, “Definitely automated” gets a Managed Challenge — which means your agents are blocked behind a CAPTCHA they can’t solve.

The Cached-Challenge Cascade — Why It Broke Real Visitors

Here’s the part most teams miss. When Cloudflare issues a Managed Challenge, the challenge page uses Google reCAPTCHA Enterprise behind the scenes — but it doesn’t use your Google Cloud project. It uses Cloudflare’s own shared reCAPTCHA key, which is pooled across all their customers.

That pooled key has a free-tier quota. When agent traffic across many sites adds up, the quota gets exhausted, and Google starts refusing to issue tokens. The challenge page then renders with the message “This site is exceeding reCAPTCHA Enterprise free quota.”

So far, just a degraded challenge experience for bots. The real damage comes next: on the Pro plan, Cloudflare’s Automatic Platform Optimization (APO) for WordPress caches HTML responses at the edge — including, in some edge cases, the broken challenge page itself. A real human visitor, hitting the same Cloudflare PoP, gets served that cached broken page. Suddenly your homepage is unusable, and the symptom looks unrelated to anything you’ve done.

This is why agent hygiene matters even if you don’t care about scraper protection. As a result, the actions you take to keep AI agents behind Cloudflare working cleanly have a direct effect on the experience of every real visitor to your marketing site.

The Fix: A WAF Skip Rule and a Secret Header

The clean solution is a private credential that agents present on every request, and a Cloudflare WAF rule that recognizes it and bypasses the bot management layer. It’s the same pattern Cloudflare itself recommends for trusted-source automation.

The mechanics are simple:

1. Generate a long random secret token. Treat it like an API key — store it in a gitignored env file, never commit it.
2. Make every agent script send that token in a custom header like X-Agent-Token.
3. Configure a Cloudflare WAF custom rule: if the header matches the secret value, Skip all security features (WAF managed rules, Super Bot Fight Mode, rate limiting).

Because the rule depends on a private secret rather than an IP or User-Agent, it survives laptop changes, VPN rotation, and remote work. And because it Skips security rather than disabling it, real visitors still get the full protection envelope.

For a fuller treatment of where this sits in the broader Cloudflare expression language, the Cloudflare WAF custom rules documentation is the canonical reference.

Implementing the AI Agents Cloudflare Allowlist

Here’s the concrete five-step recipe, drawn from how we ship this on sengo.com.

1. Generate the secret. 32 random bytes is plenty. In PowerShell: [System.BitConverter]::ToString([System.Security.Cryptography.RandomNumberGenerator]::GetBytes(32)).Replace('-','').ToLower(). Save the output as AGENT_TOKEN in your .env.local (gitignored).
2. Add the Cloudflare WAF rule. Dashboard → your site → Security → WAF → Custom Rules → Create. Expression: (http.request.headers["x-agent-token"][0] eq "<your-secret-token>"). Action: Skip. Check every box: WAF managed rules, rate limiting rules, Super Bot Fight Mode, all remaining custom rules. Deploy.
3. Wire the header into your agent scripts. Any script that hits the site needs to send the header. For shell: curl -H "X-Agent-Token: $AGENT_TOKEN" …. For Python: add {"X-Agent-Token": os.environ.get("AGENT_TOKEN", "")} to the request headers dict. For a Node script using fetch: include it in the headers option.
4. Document the convention in your repo. A short note in CLAUDE.md, README, or your agent’s system prompt instructing future scripts to include the header. Without it, new automation will silently get challenged again.
5. Optionally, add a verified-bot allowlist for legitimate AI crawlers. If you want ClaudeBot, GPTBot, PerplexityBot, and similar to reach your content for citation purposes, add a second rule that Skips when cf.client.bot is true and the User-Agent matches one of those names. The cf.client.bot field is Cloudflare’s cryptographically verified bot signal — it can’t be spoofed.

Verifying the AI Agents Cloudflare Rule Works

We’ve found a simple two-call test catches every misconfiguration. From any machine, with a bot-like User-Agent:

• Request 1 — without the token: curl -A "curl-test/1.0" https://<your-site>/. Expected: HTTP 403, response title <title>Just a moment...</title>, and a response header cf-mitigated: challenge.
• Request 2 — with the token: curl -A "curl-test/1.0" -H "X-Agent-Token: $AGENT_TOKEN" https://<your-site>/. Expected: HTTP 200, full page body, no cf-mitigated header.

If both return 200, your bot mode might be set to Allow somewhere — check Super Bot Fight Mode in the Security → Bots dashboard. Specifically, if both return 403, your rule isn’t deployed yet or the header name is mismatched.

After 24 to 48 hours of normal traffic, the Security → Events log will show the rule firing with each agent request. Filter by your rule name and confirm that real automation is matching it. In short, if you see your script’s calls in the Skip log, you’re done.

Lessons for Running AI Agents in Production

This incident taught us a few things we’d apply to any team running AI agents behind Cloudflare or any comparable edge layer.

• Don’t evade detection — work with the protection layer. User-Agent spoofing and IP rotation are short-lived workarounds. A private credential allowlist is durable and auditable.
• Plan for cache cascade effects. Edge caching (APO, LiteSpeed, CDN) can take a one-off bot incident and turn it into a real-visitor outage. Always consider the path from upstream challenge to downstream cached delivery.
• Use Cloudflare’s verified-bot signal for real crawlers. Spoofable UA matching gives anyone a free pass. The verified-bot field cf.client.bot is what you want when allowlisting ClaudeBot, GPTBot, Bingbot, and the rest.
• Rotate the agent token like any other secret. Quarterly is reasonable. Update the WAF rule and your env file in lockstep.
• Tell your agents about the convention. Whether it’s a CLAUDE.md note or a documented coding standard, the rule has to outlive the developer who set it up.

The broader pattern here is that AI agents behind Cloudflare — or any production edge layer — need first-class operational treatment, not retrofit hacks. Treat your automation as part of your infrastructure, give it credentials, and audit its access just like you would a human team member.

How Sengo Helps Teams Run AI Agents Safely

We’ve been on both sides of this problem — building AI agents that work against client sites, and protecting client sites against bad automation. The pattern in this article comes straight from a real incident on our own site, and it’s the same blueprint we deploy for clients integrating Claude Code, custom AI agents, or third-party AI search crawlers into their Cloudflare-protected infrastructure.

If your team is starting to run AI agents against your own production site — or you’re worried that AI search crawlers are getting challenged and your content isn’t being cited — that’s a problem we can size up quickly. Our enabling teams with AI solution covers exactly this kind of operational hardening for sites already in production. We’ve helped clients across financial services, higher education, and retail get their agent and crawler traffic on the right footing without weakening their security posture.