403 Forbidden in Python requests: which bot wall sent it

If Chrome opens a page and your Python script gets 403 Forbidden, it’s usually a bot wall in front of the site that refused you: Cloudflare, DataDome, Akamai, PerimeterX or AWS WAF. Walls judge how your client connects (the TLS handshake, the HTTP version and its settings, the order of your headers) and compare that with the name in your User-Agent. requests gets every one of those wrong for a browser, which is why pasting in Chrome’s User-Agent so rarely helps.

The 403 also says which wall sent it and whether it’s a block or a challenge, and each has a different fix.

Send Chrome’s handshake, not just its User-Agent

requests introduces itself as python-requests/ plus a version number, so swapping the User-Agent feels like the obvious fix. Put Chrome’s User-Agent on a plain HTTP client, though, and Kickstarter and Priceline still answer 403 with the same block page as before. The rest of the request still says “script”: requests negotiates TLS through OpenSSL with its own ciphers and extensions, and it only speaks HTTP/1.1, where Chrome talks to these sites over HTTP/2. Cloudflare calls the result a JA3 or JA4 fingerprint, and a Chrome name over a non-Chrome fingerprint is an easy catch.

Send the same request with Chrome’s handshake, HTTP/2 settings and header order, and both pages load, and so do six more walled sites. In Python, curl_cffi gives you that handshake with one argument:

from curl_cffi import requests

r = requests.get(url,
                 impersonate="chrome")

Read the 403: each wall signs its response

Each vendor marks its response:

Don’t read server: cloudflare as a Cloudflare block: Priceline’s 403 travels through Cloudflare, but the page is PerimeterX’s. And Akamai’s _abck and bm_sz cookies show up on pages that load fine; they mean Akamai is there, not that you’re blocked.

To check in code, pass the status, headers and body to this function. It looks for the challenge headers and Akamai’s page before it looks at the status, because those can arrive with a 202 or even a 200.

import re
import requests

AKAMAI_PAGE = re.compile(
    r"errors\.edgesuite\.net|Reference #18")

def which_wall(status, headers, body):
    h = {k.lower(): v for k, v in headers.items()}
    via = h.get("via", "").lower()
    if h.get("cf-mitigated") == "challenge":
        return "Cloudflare challenge"
    if h.get("x-amzn-waf-action"):
        return f"AWS WAF {h['x-amzn-waf-action']}"
    if AKAMAI_PAGE.search(body):
        return "Akamai block"
    if status not in (401, 403, 429, 503):
        return None
    if "x-datadome-cid" in h or "captcha-delivery.com" in body:
        return "DataDome block"
    if "px-captcha" in body or "_pxAppId" in body:
        return "PerimeterX block"
    if h.get("server", "").lower() == "akamaighost":
        return "Akamai block"
    if "cf-ray" in h:
        return f"Cloudflare {status}"
    if "Request blocked" in body and "cloudfront" in via:
        return "AWS WAF block"
    return f"Unbranded {status}"

r = requests.get(url)  # your request
print(r.status_code, which_wall(r.status_code, r.headers, r.text))

In axios, a 403 throws AxiosError: Request failed with status code 403, and the same three inputs are err.response.status, err.response.headers and err.response.data.

HTTP/2 403
server: cloudflare
cf-mitigated: challenge
cf-ray: a480d8d37e259aac-IAD
set-cookie: __cf_bm=…

<title>Just a moment...</title>
<noscript>… Enable JavaScript and cookies to continue …</noscript>
<script>(function(){window._cf_chl_opt = {… cType: 'managed', …};
… a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/…'; …

HTTP/2 401
server: CloudFront
x-datadome: protected
x-datadome-cid: …
x-dd-b: 2
set-cookie: datadome=…; Max-Age=31536000; Domain=.marketwatch.com; …

<title>marketwatch.com</title>
<p id="cmsg">Please enable JS and disable any ad blocker</p>
<script>var dd={'rt':'c','cid':'…','hsh':'…','t':'bv',…,
'host':'geo.captcha-delivery.com','cookie':'…'}</script>
<script src="https://ct.captcha-delivery.com/c.js"></script>

HTTP/2 403
server: AkamaiGHost
akamai-grn: 0.c424c317.1791585316.3a9f823

<TITLE>Access Denied</TITLE>
<H1>Access Denied</H1>
You don't have permission to access "http://www.sephora.com/"
on this server.<P>
Reference #18.c424c317.1791585316.3a9f823
<P>https://errors.edgesuite.net/18.c424c317.1791585316.3a9f823</P>

HTTP/2 403
server: cloudflare
cf-cache-status: DYNAMIC
set-cookie: _pxhd=…

<meta name="description" content="px-captcha">
<title>Access to this page has been denied</title>
window._pxAppId = 'PX9aTjSd0n';
script.src = 'https://captcha.px-cloud.net/PX9aTjSd0n/captcha.js?…';
Press & Hold to confirm you are<br>a human (and not a bot).

HTTP/2 202
x-amzn-waf-action: challenge
via: 1.1 ….cloudfront.net (CloudFront)

<title></title>
window.gokuProps = { "key":"…", "iv":"…", "context":"…" };
<script src="https://….token.awswaf.com/…/challenge.js"></script>
AwsWafIntegration.getToken().then(() => { window.location.reload(true); });

Some blocks don’t look like errors

raise_for_status() only fires on a 4xx or 5xx, and axios only throws outside 2xx. Walls don’t always answer that way:

• Lowe’s sends Akamai’s Access Denied page with a 200. A scraper that trusts the status saves the block page as data.
• IMDb’s AWS WAF challenge, which it sends once your handshake looks like Chrome’s, comes back as a 202. Both libraries treat that as success.
• MarketWatch’s DataDome block is a 401, which reads like a login problem.
• Best Buy doesn’t answer at all. It drops the connection, so you get a connection error instead of a 403.
• Kickstarter gives Chrome’s handshake a 200, but its homepage is a JavaScript app: the content appears only once a browser runs it.

So run the check on every response before you parse it, 200s included, and always set a timeout.

A block needs a better client, a challenge needs a browser

A block refuses your client or your IP. Fix the client first with Chrome’s handshake. If the block survives that, the wall is judging your IP (DataDome scores IP reputation alongside the client), and the fix is a different network.

A challenge is a page that runs JavaScript to check that a browser is there. A good handshake often avoids it: Kickstarter challenges a plain client and serves Chrome’s handshake the page. When the challenge comes anyway, as IMDb’s does, only a browser that runs the script gets through, which means Playwright or Puppeteer and a Chrome to keep running.

Or let one call climb for you

LurkAPI’s scrape endpoint climbs that ladder for you: a quick direct request, then Chrome’s handshake from a static ISP IP, then a browser, then other networks for walls that judge the IP. It stops at the first step that gets the page and charges for that step: 1 credit when either of the first two gets it, 4 credits when the browser renders it. A call that every step fails costs nothing.

Priceline needed one step up:

curl -G "https://api.lurkapi.com/v1/web/scrape" \
  --data-urlencode "url=https://www.priceline.com/" \
  -d format=markdown \
  -H "x-api-key: $LURK_KEY"

{
  "success": true,
  "credits_charged": 1,
  "status": 200,
  "url": "https://www.priceline.com/",
  "content": "Skip to main content\n…\n- [Hotels](https://www.priceline.com/hotels)\n- [Cars](…",
  "via": "static",
  "attempts": [
    { "via": "direct", "ms": 116, "status": 403,
      "wall": "perimeterx challenge" },
    { "via": "static", "ms": 565, "status": 200,
      "wall": null }
  ],
  "billing": { "mode": "auto", "credits": 1,
    "breakdown": { "base": 1 } }
}

attempts lists every step and the wall it met, in plain labels like perimeterx challenge or datadome block, so you can see why a page cost what it did.

Kickstarter went one step further: Chrome’s handshake got a 200, the endpoint saw a page that JavaScript still had to build, and the browser came back with the full page in about 7 seconds for 4 credits. Most pages take 1 to 3 seconds and browser renders 5 to 15, so give the call a 90-second timeout for the slow ones. The web scraping guide lists every step and its price.

Paste the URL that’s giving you the 403 and see which step gets it: