LurkAPI
Docs · Scrape a page (Auto)

Scrape a page (Auto)

Any public page, trying the cheapest step first and stepping up only if the site blocks it. 1–24 credits.

1 credit per call; +3 if a browser on separate IPs gets it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solvedMCP tool: web_scrape
GEThttps://api.lurkapi.com/v1/web/scrape

Example response

When to use this

Pass a url and get the page. Use it when you don't know what a site needs. Auto tries these steps in order, and you pay only for the one that got the page:

1. direct and static: plain requests, made the way a browser makes them. 1 credit.

2. browser: a real Chrome loads the page. For JavaScript challenges (Cloudflare, AWS WAF, Akamai, PerimeterX) and pages built entirely with JavaScript. No extra credits.

3. unblocker: a browser on separate IPs, for sites that block our usual ones (DataDome and the like). It returns the HTML the site sent, before JavaScript runs. +3.

4. residential: a residential IP, the kind a home internet connection uses. A last resort. +4.

5. browser_residential: our browser on a residential IP. The last resort for pages that need a browser. +13.

6. captcha: if a captcha still blocks our browser (Cloudflare Turnstile, DataDome's slider, reCAPTCHA), we solve it. +10, only when one is solved.

The price depends on the site, from 1 to 24 credits. Every answer's billing says what the call cost and why.

To control the price, call one step's own endpoint (/v1/web/fetch, /v1/web/render and so on) for a fixed price. To cap it, set max_credits, list the steps you accept in /v1/web/custom, or set render and proxy here. render=false&proxy=static never costs more than 1 credit. render=true always returns the page after its JavaScript ran, never through the unblocker.

What comes back. via names the step that got the page, and attempts lists every step tried. status is the site's own HTTP status; a 404 page is charged like any other page. Use format=markdown for AI models: links, headings, lists and tables survive, scripts and styles don't. Add main=true to keep only the page's own content. Pages over 2 MB are cut off (truncated).

Repeat calls. If a site needed a stronger step recently, Auto starts there for 15 minutes. Once we get past a site's wall, later calls reuse that pass, often as a plain request in a few seconds. Nothing is cached: every call fetches the page fresh.

Failures are free. If every allowed step is blocked, or the site is down or answers with a 5xx error, the call fails with 502 upstream_error and costs nothing. The message says why. After a call is blocked at every allowed step, the same call fails at once, still free, for 5 minutes. Change render or proxy to try other steps.

Speed. 1–3 seconds on most sites, 5–15 when our browser renders the page, 15–35 through the unblocker, and up to 85 on the slowest (a captcha solved, or a blocked first visit retried).

Example request

Copy one and replace YOUR_API_KEY with your API key (get one free). The code versions read the key from the LURKAPI_API_KEY environment variable instead.

Language

Parameters

Add them to the URL after a ?, as in ?format=markdown. Join several with &.

ParameterDescription
url
stringrequired
The page to fetch, as a full URL starting with http:// or https://. It must be a public site on the standard ports (80 or 443).
Example
https://books.toscrape.com/
format
stringoptional
How content comes back: html, markdown (best for AI models) or plain text. JSON, XML and plain-text pages come back unchanged.
One of
htmlmarkdowntext
Default
html
Example
markdown
main
booleanoptional
true keeps only the page's own content: its <h1> and its <main> element (or its articles if it has no <main>, else the whole page). Navigation, sidebars, headers, footers, hidden parts, skip links and cookie banners are left out. Works with format=markdown or text; format=html ignores it.
One of
truefalse
Default
false
render
stringoptional
auto uses a browser only when the page needs one: ours at no extra cost, or the unblocker (+3). true always uses our browser and returns the page after its JavaScript ran. false sticks to plain requests.
One of
autotruefalse
Default
auto
proxy
stringoptional
Which IPs to fetch from. auto uses our usual IPs and the unblocker, and a residential IP only when those are blocked (+4 when one is used, +13 with our browser). static never uses a residential IP. residential uses only residential IPs.
One of
autostaticresidential
Default
auto
country
stringoptional
Fetch from a residential IP in this country, as a two-letter code like us, gb or de. Setting it means proxy=residential (+4).
captcha
booleanoptional
true solves a captcha that blocks the browser (+10, only when one is solved). false never pays for a solve.
One of
truefalse
Default
true
wait_for
stringoptional
A CSS selector to wait for before returning, like .price or #reviews. Setting it renders the page in our browser.
wait
integeroptional
Milliseconds to let the page settle after it loads, up to 10000. Setting it renders the page in our browser.
max_credits
integeroptional
The most this call may cost, in credits. A paid step that would go past it doesn't run; a call that can't be done within it fails with 402 over_max_credits, free. For example, 1 allows plain requests and our browser, and 4 the unblocker too, but no residential IP or captcha solve.

Example response

A real response, captured from the live API. Lists are trimmed to a couple of items, and long text such as media links is cut short with ….

Show the example response (1 KB)
{
  "success": true,
  "credits_remaining": 9999,
  "credits_charged": 1,
  "status": 200,
  "url": "https://books.toscrape.com/",
  "title": "All products | Books to Scrape - Sandbox",
  "content_type": "text/html",
  "content": "[Books to Scrape](https://books.toscrape.com/index.html) We love being s…",
  "truncated": false,
  "via": "direct",
  "captcha": null,
  "attempts": [
    {
      "via": "direct",
      "ms": 504,
      "status": 200,
      "wall": null,
      "error": null
    }
  ],
  "billing": {
    "mode": "auto",
    "credits": 1,
    "breakdown": {
      "base": 1
    },
    "note": "Auto: the price follows the strategy that got the page: 1 for a plain re…"
  }
}

Response fields

Every field in a successful response, and what it means. A name with [] belongs to each item of a list: attempts[].via is the via of every item in attempts. null means there's no value.

Show all 22 fields
FieldTypeDescription
successtrueAlways true here; errors have success: false.
credits_remainingnumberYour balance after this call. On anonymous playground calls: free tries left today.
credits_chargednumberCredits this call cost; 0 on free endpoints. On anonymous playground calls: tries used (1).
statusintegerThe HTTP status the site gave the page: 200, or its own 404, 410 and so on.
urlstringThe page's URL after any redirects. When via is unblocker, it's the URL you asked for, since that step doesn't report redirects.
titlestringcan be nullThe page's title, from its <title>; null when it has none or isn't HTML.
content_typestringcan be nullThe page's Content-Type, like text/html; charset=utf-8; null if the site sent none.
contentstringcan be nullThe page in the format you asked for: HTML, markdown or plain text. Null for binary files such as images, PDFs and archives.
truncatedbooleanThe page was over 2 MB, so it was cut off there.
viastringThe step that got the page: direct or static (plain requests), browser (our browser), unblocker (a browser on separate IPs), residential (a residential IP) or browser_residential (our browser on a residential IP).
captchastringcan be nullThe captcha solved to get the page (turnstile, datadome or recaptcha), which is charged. Null if none was solved.
attemptsobject[]Every step tried, in order. The last one got the page.
attempts[].viastringThe step tried.
attempts[].msintegerHow long it took, in milliseconds.
attempts[].statusintegercan be nullThe HTTP status it got; null if no answer came back.
attempts[].wallstringcan be nullWhat blocked it, like cloudflare challenge or datadome block; null if nothing did.
attempts[].errorstringcan be nullWhy it failed when no answer came back; null if one did.
billingobjectWhat this call cost and why.
billing.modestringfixed: this endpoint's set price. auto and custom: the price of the step that got the page.
billing.creditsintegerWhat this call cost, the sum of breakdown.
billing.breakdownobjectCredits by what they paid for: base, plus on Auto or Custom unblock (a browser on separate IPs), premium (a residential IP) or premium_render (a browser on a residential IP), and captcha when one was solved.
billing.notestringcan be nullOn Auto and Custom, why the price varies; null on fixed-price endpoints.

Cost and freshness

Each call costs 1 credit; +3 if a browser on separate IPs gets it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solved. Calls turned away before they reach the platform (invalid parameters, a key or balance problem, a rate limit) are free, and so are 5xx errors. A not_found is charged, since the platform was checked. How charging works.

Not cached. Every call fetches fresh data.

Errors

A failed call returns success: false, an error message, a code your program can check, and a docs link. These are the errors this endpoint can return. All error codes.

StatusCodeMeaning
401missing_api_keyNo API key was sent. Put it in the x-api-key header.
401invalid_api_keyThis key doesn't exist or was revoked.
402insufficient_creditsNot enough credits for this call. Buy a pack, or wait for tomorrow's free top-up.
402over_max_creditsThis call would cost more than the max_credits you set. The error says what it needs; raise max_credits to allow it. Not charged.
403account_suspendedThis account is suspended. Email support@lurkapi.com. Not charged.
405method_not_allowedEndpoints only take GET, with parameters in the URL.
429rate_limitedToo many calls in a short time. Wait as many seconds as the retry-after header says, then retry.
500internal_errorSomething broke on our side. Retry. Not charged.
502upstream_errorThe platform didn't send a usable answer. Retry. Not charged.
503upstream_busyAll our connections to the platform are busy. Retry in a few seconds. Not charged.
504upstream_timeoutThe platform took over 30 seconds to answer (90 for web pages). Retry. Not charged.

Use it in Claude

Connect LurkAPI to Claude and ask in plain English. This endpoint is the web_scrape tool. For example:

Ask Claude
Use LurkAPI's web_scrape tool with url "https://books.toscrape.com/", format "markdown", then summarize what you find.

Claude calls web_scrape with arguments like these, and each call costs 1 credit; +3 if a browser on separate IPs gets it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solved:

Tool arguments
{"url": "https://books.toscrape.com/","format": "markdown"}

To save tokens, tool results leave out empty fields, keep one URL per image or video, and give transcripts as plain text unless you pass timestamps=true. A result over 45,000 characters comes in parts. Its _note gives the offset for the next part, and later parts are free for 15 minutes. Pass trim=false for the full response.

From the blog

All posts