Scrape a page with your own steps

When to use this

Pass a url and steps, the strategies to try in order, comma-separated: fetch (1), render (4), unblock (4), residential (5), render-residential (14). We stop at the first that gets the page and you pay for that one.

We try them in turn and skip any a wall has already ruled out, so the most you can pay is the dearest step you listed (+10 when you allow a captcha solve and one happens). billing in the answer says what this call cost. For example steps=fetch,unblock never costs more than 4 credits and never runs JavaScript; steps=render,render-residential always returns the page as rendered. country applies to the residential steps; the others go out from the US. wait_for and wait apply to the rendering steps. A call walled at every step fails with 502 upstream_error and costs nothing. The web scraping guide explains each strategy.

Parameters

All parameters go in the query string.

Example request

Replace YOUR_API_KEY with your key, or set LURKAPI_KEY for the code. Get a free key.

curl "https://api.lurkapi.com/v1/web/custom?url=https%3A%2F%2Fbooks.toscrape.com%2F&steps=fetch%2Crender&format=markdown" \
  -H "x-api-key: YOUR_API_KEY"

const params = new URLSearchParams({
  url: "https://books.toscrape.com/",
  steps: "fetch,render",
  format: "markdown",
});
const res = await fetch(`https://api.lurkapi.com/v1/web/custom?${params}`, {
  headers: { "x-api-key": process.env.LURKAPI_KEY },
});
const data = await res.json();
if (!data.success) throw new Error(`${data.code}: ${data.error}`);
console.log(data.content);

import os
import requests

res = requests.get(
    "https://api.lurkapi.com/v1/web/custom",
    params={
        "url": "https://books.toscrape.com/",
        "steps": "fetch,render",
        "format": "markdown",
    },
    headers={"x-api-key": os.environ["LURKAPI_KEY"]},
    timeout=95,
)
data = res.json()
if not data["success"]:
    raise RuntimeError(f"{data['code']}: {data['error']}")
print(data["content"])

Example response

A real response from this endpoint, captured from the live API and trimmed to a couple of items. Strings over 96 characters (mostly signed media URLs) are cut short and end in ….

{
  "success": true,
  "credits_remaining": 9999,
  "credits_charged": 1,
  "status": 200,
  "url": "https://books.toscrape.com/",
  "content_type": "text/html",
  "content": "[Books to Scrape](https://books.toscrape.com/index.html) We love being s…",
  "truncated": false,
  "via": "direct",
  "captcha": null,
  "attempts": [
    {
      "via": "direct",
      "ms": 369,
      "status": 200,
      "wall": null,
      "error": null
    }
  ],
  "billing": {
    "mode": "custom",
    "credits": 1,
    "breakdown": {
      "base": 1
    },
    "note": "Custom: you pay for the step that got the page (fetch 1, render 4, unblo…"
  }
}

Response fields

Every field of a successful response. [] marks a list: a[].b is the b of each item in a. nullable fields can be null; optional fields can be missing.

Caching and freshness

Not cached: every call is answered fresh.

Errors

Errors return { success: false, error, code, docs } with the HTTP status below. Validation errors, 401/402/429 rejections and 5xx failures are free; not_found is charged (how charging works). All error codes.

Use it in Claude

Once LurkAPI is connected to Claude, this endpoint is the web_custom tool. Ask in plain English, for example:

Use LurkAPI's web_custom with url "https://books.toscrape.com/", steps "fetch,render", format "markdown" and summarize what you find.

Claude calls web_custom with arguments like these, and each call costs 1 credit; +3 if a browser renders it, +4 if it needs a residential IP, +13 if a browser renders it on a residential IP, +10 if a captcha is solved:

{
  "url": "https://books.toscrape.com/",
  "steps": "fetch,render",
  "format": "markdown"
}

Tool results skip nulls and empty lists to save tokens.