Scribbles for my memory

An AI agent posting to my API through Python's urllib kept getting 403 Forbidden. The same request sent with curl succeeded. The request never reached the API at all: Cloudflare rejected it at the edge.

The difference is the User-Agent header. Compare the two:

curl -s -o /dev/null -w '%{http_code} %header{server}\n' https://api.example.com/health
# 200 cloudflare
curl -s -A 'Python-urllib/3.12' https://api.example.com/health
# error code: 1010

error code: 1010 together with server: cloudflare means Cloudflare refused the client's signature. On a zone with no Bot Fight Mode and no WAF rules, the usual cause is Browser Integrity Check (BIC). It is on by default, and it blocks requests whose headers look like those of abusive bots. Against my zone I tried six User-Agents. Python-urllib/3.12 got 403. python-requests, curl, Go-http-client, node, and no UA at all all got 200. So one Python script fails while its neighbour using requests works, and the failure looks random.

You can confirm BIC is the cause with the zone settings API:

curl -s -H "X-Auth-Key: $CF_API_KEY" -H "X-Auth-Email: $CF_API_EMAIL" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE/settings/browser_check"
# "value": "on"

The quick fix is to make every client send a browser-ish UA. That fixes your own scripts, but every other agent and SDK still hits the wall, and an API that only answers browsers is an odd thing to publish. A better fix is to turn BIC off only for the API hosts, and keep it on for the rest of the zone. A Configuration Rule does exactly that:

curl -s -X PUT -H "X-Auth-Key: $CF_API_KEY" -H "X-Auth-Email: $CF_API_EMAIL" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE/rulesets/phases/http_config_settings/entrypoint" \
  -d '{"rules":[{
        "description": "API hosts: no Browser Integrity Check",
        "expression": "(http.host in {\"api.example.com\" \"upload.example.com\"})",
        "action": "set_config",
        "action_parameters": {"bic": false}
      }]}'

A PUT on the phase entrypoint replaces the whole rule list. If the zone already has Configuration Rules, fetch the entrypoint first and send the existing rules back along with the new one. On a zone with none, the GET returns error 10003 ("could not find entrypoint ruleset") and the PUT creates the entrypoint. The change took effect within seconds: the Python-urllib request went from 403 to the origin's own response.

Don't forget sibling hosts. The API worked after the change, but the image upload host behind it still returned 403 to the same agent until I added it to the rule.