Docs · Scrape a page (Auto)
Scrape a page (Auto)
Any public page, trying the cheapest step first and stepping up only if the site blocks it. 1–24 credits.
https://api.lurkapi.com /v1/web/scrapeWhen to use this
Pass a url and get the page. Use it when you don't know what a site needs. Auto tries these steps in order, and you pay only for the one that got the page:
1. direct and static: plain requests, made the way a browser makes them. 1 credit.
2. browser: a real Chrome loads the page. For JavaScript challenges (Cloudflare, AWS WAF, Akamai, PerimeterX) and pages built entirely with JavaScript. No extra credits.
3. unblocker: a browser on separate IPs, for sites that block our usual ones (DataDome and the like). It returns the HTML the site sent, before JavaScript runs. +3.
4. residential: a residential IP, the kind a home internet connection uses. A last resort. +4.
5. browser_residential: our browser on a residential IP. The last resort for pages that need a browser. +13.
6. captcha: if a captcha still blocks our browser (Cloudflare Turnstile, DataDome's slider, reCAPTCHA), we solve it. +10, only when one is solved.
The price depends on the site, from 1 to 24 credits. Every answer's billing says what the call cost and why.
To control the price, call one step's own endpoint (/v1/web/fetch, /v1/web/render and so on) for a fixed price. To cap it, set max_credits, list the steps you accept in /v1/web/custom, or set render and proxy here. render=false&proxy=static never costs more than 1 credit. render=true always returns the page after its JavaScript ran, never through the unblocker.
What comes back. via names the step that got the page, and attempts lists every step tried. status is the site's own HTTP status; a 404 page is charged like any other page. Use format=markdown for AI models: links, headings, lists and tables survive, scripts and styles don't. Add main=true to keep only the page's own content. Pages over 2 MB are cut off (truncated).
Repeat calls. If a site needed a stronger step recently, Auto starts there for 15 minutes. Once we get past a site's wall, later calls reuse that pass, often as a plain request in a few seconds. Nothing is cached: every call fetches the page fresh.
Failures are free. If every allowed step is blocked, or the site is down or answers with a 5xx error, the call fails with 502 upstream_error and costs nothing. The message says why. After a call is blocked at every allowed step, the same call fails at once, still free, for 5 minutes. Change render or proxy to try other steps.
Speed. 1–3 seconds on most sites, 5–15 when our browser renders the page, 15–35 through the unblocker, and up to 85 on the slowest (a captcha solved, or a blocked first visit retried).
Example request
Copy one and replace YOUR_API_KEY with your API key (get one free). The code versions read the key from the LURKAPI_API_KEY environment variable instead.
curl "https://api.lurkapi.com/v1/web/scrape?url=https%3A%2F%2Fbooks.toscrape.com%2F&format=markdown" \
-H "x-api-key: YOUR_API_KEY"Paste it into a terminal, with your key in place of YOUR_API_KEY.
const params = new URLSearchParams({
url: "https://books.toscrape.com/",
format: "markdown",
});
const res = await fetch(`https://api.lurkapi.com/v1/web/scrape?${params}`, {
headers: { "x-api-key": process.env.LURKAPI_API_KEY },
});
const data = await res.json();
if (!data.success) throw new Error(`${data.code}: ${data.error}`);
console.log(data.content);Node 18+. Save it as lurk.mjs and run node lurk.mjs. It reads your key from LURKAPI_API_KEY.
import os
import requests
res = requests.get(
"https://api.lurkapi.com/v1/web/scrape",
params={
"url": "https://books.toscrape.com/",
"format": "markdown",
},
headers={"x-api-key": os.environ["LURKAPI_API_KEY"]},
timeout=95,
)
data = res.json()
if not data["success"]:
raise RuntimeError(f"{data['code']}: {data['error']}")
print(data["content"])Run pip install requests, save it as lurk.py and run python lurk.py. It reads your key from LURKAPI_API_KEY.
package main
import (
"encoding/json"
"fmt"
"net/http"
"net/url"
"os"
"time"
)
func main() {
params := url.Values{}
params.Set("url", "https://books.toscrape.com/")
params.Set("format", "markdown")
req, _ := http.NewRequest("GET", "https://api.lurkapi.com/v1/web/scrape?"+params.Encode(), nil)
req.Header.Set("x-api-key", os.Getenv("LURKAPI_API_KEY"))
client := &http.Client{Timeout: 95 * time.Second}
res, err := client.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
var data map[string]any
if err := json.NewDecoder(res.Body).Decode(&data); err != nil {
panic(err)
}
if data["success"] != true {
panic(fmt.Sprintf("%v: %v", data["code"], data["error"]))
}
fmt.Println(data["content"])
}Save it as main.go and run go run main.go. It reads your key from LURKAPI_API_KEY.
<?php
$params = http_build_query([
'url' => 'https://books.toscrape.com/',
'format' => 'markdown',
]);
$context = stream_context_create(['http' => [
'header' => 'x-api-key: ' . getenv('LURKAPI_API_KEY'),
'timeout' => 95,
'ignore_errors' => true,
]]);
$body = file_get_contents('https://api.lurkapi.com/v1/web/scrape?' . $params, false, $context);
$data = json_decode($body, true);
if (!$data['success']) {
throw new Exception("{$data['code']}: {$data['error']}");
}
print_r($data['content']);Save it as lurk.php and run php lurk.php. It reads your key from LURKAPI_API_KEY.
require 'json'
require 'net/http'
uri = URI('https://api.lurkapi.com/v1/web/scrape')
uri.query = URI.encode_www_form(
'url' => 'https://books.toscrape.com/',
'format' => 'markdown',
)
req = Net::HTTP::Get.new(uri)
req['x-api-key'] = ENV.fetch('LURKAPI_API_KEY')
res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https', read_timeout: 95) do |http|
http.request(req)
end
data = JSON.parse(res.body)
raise "#{data['code']}: #{data['error']}" unless data['success']
puts data['content']Save it as lurk.rb and run ruby lurk.rb. It reads your key from LURKAPI_API_KEY.
Parameters
Add them to the URL after a ?, as in ?format=markdown. Join several with &.
| Parameter | Description |
|---|---|
urlstringrequired | The page to fetch, as a full URL starting with http:// or https://. It must be a public site on the standard ports (80 or 443).
|
formatstringoptional | How content comes back: html, markdown (best for AI models) or plain text. JSON, XML and plain-text pages come back unchanged.
|
mainbooleanoptional | true keeps only the page's own content: its <h1> and its <main> element (or its articles if it has no <main>, else the whole page). Navigation, sidebars, headers, footers, hidden parts, skip links and cookie banners are left out. Works with format=markdown or text; format=html ignores it.
|
renderstringoptional | auto uses a browser only when the page needs one: ours at no extra cost, or the unblocker (+3). true always uses our browser and returns the page after its JavaScript ran. false sticks to plain requests.
|
proxystringoptional | Which IPs to fetch from. auto uses our usual IPs and the unblocker, and a residential IP only when those are blocked (+4 when one is used, +13 with our browser). static never uses a residential IP. residential uses only residential IPs.
|
countrystringoptional | Fetch from a residential IP in this country, as a two-letter code like us, gb or de. Setting it means proxy=residential (+4). |
captchabooleanoptional | true solves a captcha that blocks the browser (+10, only when one is solved). false never pays for a solve.
|
wait_forstringoptional | A CSS selector to wait for before returning, like .price or #reviews. Setting it renders the page in our browser. |
waitintegeroptional | Milliseconds to let the page settle after it loads, up to 10000. Setting it renders the page in our browser. |
max_creditsintegeroptional | The most this call may cost, in credits. A paid step that would go past it doesn't run; a call that can't be done within it fails with 402 over_max_credits, free. For example, 1 allows plain requests and our browser, and 4 the unblocker too, but no residential IP or captcha solve. |
Example response
A real response, captured from the live API. Lists are trimmed to a couple of items, and long text such as media links is cut short with ….
Show the example response (1 KB)Hide
{
"success": true,
"credits_remaining": 9999,
"credits_charged": 1,
"status": 200,
"url": "https://books.toscrape.com/",
"title": "All products | Books to Scrape - Sandbox",
"content_type": "text/html",
"content": "[Books to Scrape](https://books.toscrape.com/index.html) We love being s…",
"truncated": false,
"via": "direct",
"captcha": null,
"attempts": [
{
"via": "direct",
"ms": 504,
"status": 200,
"wall": null,
"error": null
}
],
"billing": {
"mode": "auto",
"credits": 1,
"breakdown": {
"base": 1
},
"note": "Auto: the price follows the strategy that got the page: 1 for a plain re…"
}
}Response fields
Every field in a successful response, and what it means. A name with [] belongs to each item of a list: attempts[].via is the via of every item in attempts. null means there's no value.
Show all 22 fieldsHide
| Field | Type | Description |
|---|---|---|
| success | true | Always true here; errors have success: false. |
| credits_remaining | number | Your balance after this call. On anonymous playground calls: free tries left today. |
| credits_charged | number | Credits this call cost; 0 on free endpoints. On anonymous playground calls: tries used (1). |
| status | integer | The HTTP status the site gave the page: 200, or its own 404, 410 and so on. |
| url | string | The page's URL after any redirects. When via is unblocker, it's the URL you asked for, since that step doesn't report redirects. |
| title | stringcan be null | The page's title, from its <title>; null when it has none or isn't HTML. |
| content_type | stringcan be null | The page's Content-Type, like text/html; charset=utf-8; null if the site sent none. |
| content | stringcan be null | The page in the format you asked for: HTML, markdown or plain text. Null for binary files such as images, PDFs and archives. |
| truncated | boolean | The page was over 2 MB, so it was cut off there. |
| via | string | The step that got the page: direct or static (plain requests), browser (our browser), unblocker (a browser on separate IPs), residential (a residential IP) or browser_residential (our browser on a residential IP). |
| captcha | stringcan be null | The captcha solved to get the page (turnstile, datadome or recaptcha), which is charged. Null if none was solved. |
| attempts | object[] | Every step tried, in order. The last one got the page. |
| attempts[].via | string | The step tried. |
| attempts[].ms | integer | How long it took, in milliseconds. |
| attempts[].status | integercan be null | The HTTP status it got; null if no answer came back. |
| attempts[].wall | stringcan be null | What blocked it, like cloudflare challenge or datadome block; null if nothing did. |
| attempts[].error | stringcan be null | Why it failed when no answer came back; null if one did. |
| billing | object | What this call cost and why. |
| billing.mode | string | fixed: this endpoint's set price. auto and custom: the price of the step that got the page. |
| billing.credits | integer | What this call cost, the sum of breakdown. |
| billing.breakdown | object | Credits by what they paid for: base, plus on Auto or Custom unblock (a browser on separate IPs), premium (a residential IP) or premium_render (a browser on a residential IP), and captcha when one was solved. |
| billing.note | stringcan be null | On Auto and Custom, why the price varies; null on fixed-price endpoints. |
Cost and freshness
Each call costs 1 credit; +3 if a browser on separate IPs gets it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solved. Calls turned away before they reach the platform (invalid parameters, a key or balance problem, a rate limit) are free, and so are 5xx errors. A not_found is charged, since the platform was checked. How charging works.
Not cached. Every call fetches fresh data.
Errors
A failed call returns success: false, an error message, a code your program can check, and a docs link. These are the errors this endpoint can return. All error codes.
| Status | Code | Meaning |
|---|---|---|
| 401 | missing_api_key | No API key was sent. Put it in the x-api-key header. |
| 401 | invalid_api_key | This key doesn't exist or was revoked. |
| 402 | insufficient_credits | Not enough credits for this call. Buy a pack, or wait for tomorrow's free top-up. |
| 402 | over_max_credits | This call would cost more than the max_credits you set. The error says what it needs; raise max_credits to allow it. Not charged. |
| 403 | account_suspended | This account is suspended. Email support@lurkapi.com. Not charged. |
| 405 | method_not_allowed | Endpoints only take GET, with parameters in the URL. |
| 429 | rate_limited | Too many calls in a short time. Wait as many seconds as the retry-after header says, then retry. |
| 500 | internal_error | Something broke on our side. Retry. Not charged. |
| 502 | upstream_error | The platform didn't send a usable answer. Retry. Not charged. |
| 503 | upstream_busy | All our connections to the platform are busy. Retry in a few seconds. Not charged. |
| 504 | upstream_timeout | The platform took over 30 seconds to answer (90 for web pages). Retry. Not charged. |
Use it in Claude
Connect LurkAPI to Claude and ask in plain English. This endpoint is the web_scrape tool. For example:
Use LurkAPI's web_scrape tool with url "https://books.toscrape.com/", format "markdown", then summarize what you find.Claude calls web_scrape with arguments like these, and each call costs 1 credit; +3 if a browser on separate IPs gets it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solved:
{"url": "https://books.toscrape.com/","format": "markdown"}To save tokens, tool results leave out empty fields, keep one URL per image or video, and give transcripts as plain text unless you pass timestamps=true. A result over 45,000 characters comes in parts. Its _note gives the offset for the next part, and later parts are free for 15 minutes. Pass trim=false for the full response.
From the blog
All posts
· 5 min read
Reddit RSS feeds end November 13: how to keep yours working
Reddit turns off every RSS feed on November 13, 2026. Keep your feed URLs: a small Cloudflare Worker serves the same paths from LurkAPI. Change only the host.

· 4 min read
Best Facebook scrapers in 2026: prices and speed compared
LurkAPI is the best Facebook scraper: public Pages' posts at $0.19 per 1,000, about 1 s a call, with contacts, comments and events on the same key.

· 4 min read
TikTok Research API: who can get it, and what businesses use instead
Non-commercial researchers in the US, Europe, Canada and Brazil, after a review of about 4 weeks. Businesses can’t apply; they use a public-data API instead.