Skip to main content
POST
Scrape URL

​
Request Body

​
string
required
The URL to scrape. Must be a valid HTTP or HTTPS URL. Private IPs and cloud metadata endpoints are blocked for security.
​
string
default:"GET"
HTTP method for the request: GET, POST, PUT, PATCH, DELETE, HEAD or OPTIONS. POST, PUT and PATCH require request_body.
​
string
Raw body to send with a POST/PUT/PATCH. Pass it as a string, and set content_type to match — there is no form_data parameter.
Submit a form
Call a JSON API
​
string
Content-Type header for request_body. Defaults to application/json when a body is present.
​
string
default:"auto"
Scraping engine tier to use. ScrapeBadger automatically selects the best approach.In auto mode, simple pages use fast HTTP (1 credit) and JavaScript-heavy pages use a browser (5 credits).
The engine cost is not the total. Every request also pays the proxy_tier surcharge, so the cheapest possible scrape is 2 credits, not 1, and a browser scrape is 6, not 5. See Credit costs for the full arithmetic.
​
string
default:"simple"
Proxy quality to route the request through. This surcharge is added to every request, on top of the engine cost.Stay on simple unless the target actually blocks datacenter IPs — moving to premium makes a browser scrape 10 credits instead of 7. Use max_cost if you want a hard ceiling.
​
string
default:"html"
Output format for the scraped content. Applies to text responses only — a binary target (image, PDF, archive) ignores it and is returned untouched.
  • html — Raw HTML of the page
  • markdown — Converted to clean Markdown
  • text — Plain text with HTML tags stripped
These are the only three values. There is no raw format — to get an unwrapped body, use raw_content.
​
boolean
default:false
Force JavaScript rendering before extracting content. Automatically switches to the browser engine. Use this for single-page applications or pages that load content dynamically.
​
string
CSS selector or XPath expression to wait for before extracting content. Only works with the browser engine. If render_js is false and this is set, JS rendering is forced automatically.The wait runs after js_scenario, so a login flow can fill the form, submit, and then wait for an element that only exists on the post-login page. The wait is a precondition, not a hint: if the selector has not appeared within wait_timeout, the request fails with 422 wait_for_timeout, nothing is charged, and the response carries js_scenario_report so you can see how far the flow got. A successful response reports wait_for_found: true. For a best-effort dwell that never fails, use wait_after_load instead.
render_js alone is often not enough. It returns the DOM once the page has loaded — which for anything drawn by a third-party widget (an embedded login form, a consent gate, a chat panel, a payment iframe) is the empty container, not the content. Those widgets fetch and paint seconds after load.If a form or list is missing from your result, name it in wait_for rather than raising wait_after_load; the selector returns as soon as the element exists instead of always paying a fixed dwell.
​
integer
default:30000
Maximum time in milliseconds to wait for the wait_for selector to appear. Range: 1000 – 120000.
​
integer
Additional milliseconds to wait after the page has finished loading, before extracting content. Useful for pages with animations or delayed rendering. Only works with browser engines. Range: 0 – 30000.
​
array
A list of browser actions to perform, in order, after the page loads and before wait_for is awaited and content is extracted. Forces the browser engine. Each step is an object with an action and action-specific parameters.Supported actions:
Steps run one after another and stop at the first one that fails (a selector that never becomes actionable, a script that throws). A failed step fails the request with 422 js_scenario_failed and nothing is charged. Every response that ran a scenario includes js_scenario_report, one entry per executed step with its outcome and the page URL after it — the log to read when a flow does not end where you expected. An unknown action is rejected up front with a validation error rather than skipped. (type and milliseconds are accepted as aliases of action and timeout.)
​
string
A unique identifier to persist cookies, fingerprint, and browser storage across multiple requests. Use the same session_id to maintain login state or continue a browsing session.
​
integer
default:3
Maximum number of retry attempts when a blocking page is detected. Retries are free — you only pay for the final successful engine. Range: 0 – 10.
​
boolean
default:true
Whether to automatically retry when a blocking page is detected. Set to false to get the blocked response immediately.
​
string
ISO 3166-1 alpha-2 country code for proxy geo-targeting. Routes the request through a proxy in the specified country.
Common values: US, GB, DE, FR, JP, BR, AU.
​
object
Additional HTTP headers to include in the request to the target URL. Accepts a key-value object.
​
string
JavaScript to evaluate in the loaded page after navigation and all waits complete. Its return value replaces the page HTML in content (objects are JSON-serialised). May be an async arrow function. Browser engines only — forces the browser engine. No extra credits.Use it to pull something out of the live page that the HTML does not carry:
Read the page's cookies
Read a value the page computed
document.cookie returns only non-HttpOnly cookies — that is a browser rule, not a ScrapeBadger limit. Anti-bot vendors’ session cookies are also bound to the IP and TLS fingerprint that minted them, so replaying them from a different machine does not carry the session. To keep a session alive across several scrapes, use session_id instead and keep the whole flow on our side.
​
boolean
default:false
Capture a PNG screenshot of the rendered page. Forces the browser engine. Returned in the screenshot_url response field as a data:image/png;base64,… URI. Captures the viewport unless screenshot_full_page is set. For a screenshot-only call, POST /v1/web/screenshot returns the bare base64 PNG.
​
boolean
default:false
With screenshot: true, capture the whole scrollable page instead of the viewport.
​
integer
Browser viewport width in pixels for the render (320 – 3840).
​
integer
Browser viewport height in pixels for the render (240 – 4320).
​
boolean
default:false
Record the browser session, from navigation to the final page, as an animated GIF. Forces the browser engine. Returned in the video_url response field as a data:image/gif;base64,… URI. No extra charge — you pay the browser engine cost you were already paying. Useful for debugging, visual verification, or monitoring how a page loads.The recording is bounded so it stays small enough to return inline: up to 5 frames per second, at most 640 px on its longest side, idle stretches (a wait_for or a dwell) shortened to 3 s, and at most 2 MB. A recording over 2 MB is downscaled, then thinned to fewer frames. Recording never fails the scrape: if nothing could be recorded you still get the page, with video_url: null.
​
boolean
default:false
Attempt to bypass detected anti-bot protection using registered solvers. Adds +5 credits to the request cost when a solver is invoked. Only triggered when blocking is actually detected.
​
boolean
default:false
Allow automatic escalation to more powerful engines when the initial engine is blocked.Escalation path: http → browserYou only pay for the engine that succeeds — costs are not cumulative. Without this flag, only the selected engine is tried.
​
integer
Maximum credits to spend on this request. The request fails with a 400 error if the estimated cost would exceed this budget. Useful for controlling costs when using escalate or anti_bot. Minimum: 1.
​
boolean
default:false
Return the body directly as the HTTP response instead of wrapping it in JSON. Metadata comes back in X-Scrape-* response headers.Use it for two things: skipping the JSON encode/decode on large HTML payloads (saves 300–1000 ms on 1 MB+ responses), and downloading binary files without the ~33% base64 overhead. See Binary files and raw bodies.Cannot be combined with ai_extract, screenshot or video — those need the JSON envelope, so the request falls back to it automatically.
​
boolean
default:false
Run AI-powered extraction on the scraped content using the instruction in ai_prompt. Adds +2 credits to the request cost. The scrape result is still returned even if AI extraction fails, and a failed extraction is not charged.
​
string
Natural language instruction for AI data extraction. Required when ai_extract is true. Maximum 2000 characters.

​
Credit costs

Every request is billed as engine + proxy tier + options. The proxy tier is charged on every request, which is why nothing costs 1 credit in practice. Common totals: The exact amount charged is always in the X-Credits-Used response header and the credits_used body field — trust those over any estimate. A request rejected before scraping (400, 422) is charged 0.
Set max_cost to put a ceiling on a request. It is checked against the engine’s cost, plus anything a per-host override adds (a higher proxy tier than you asked for, or a browser engine). The proxy tier you chose yourself is not part of the check, so compare max_cost with the engine price.

​
Per-host overrides and automatic unblocking

Some sites only answer a specific setup: a particular proxy tier or country, a warm-up visit, or a real browser. ScrapeBadger keeps a working configuration per host and applies it for you:
  • On a host with an override, its access settings (engine chain, proxy tier/type/country, warm-up and injected headers) are used instead of yours. Your output settings (render_js, wait_for, js_scenario, format, screenshot, …) are still honoured. You are billed for the configuration actually used; the X-Proxy-Tier-Effective response header names the tier.
  • On a host without one, your parameters are used as sent. If the host keeps blocking requests, ScrapeBadger starts searching in the background for the cheapest configuration that reliably gets through, and applies it to later requests automatically. The blocked request still returns its 422 straight away, with the search state in the X-Unblock header and the unblock body field: discovering (a search is running), cooldown (a recent search found nothing; retried later) or none.
  • When an override stops working, the search runs again and replaces it only if the new configuration does measurably better.

​
Response

​
boolean
Whether the scrape completed successfully. false when all retries are exhausted and the page is still blocked.
​
string
The final URL after any redirects.
​
integer
HTTP status code from the target URL.
​
string
The scraped content in the requested format. null when success is false, and null for binary targets — those come back in content_base64.
​
string
Base64-encoded response body, returned instead of content when the target serves a binary payload. Only present when is_binary is true. Bodies above 25 MB are not base64-encoded — re-request those with raw_content: true.
​
boolean
Whether the target returned a binary (non-text) body.
​
string
The target’s response Content-Type, normalised to the bare media type (e.g. image/jpeg).
​
string
The output format used: html, markdown, or text.
​
string
The engine tier that produced the final result.
​
integer
Total credits charged for this request: engine cost + proxy_tier surcharge, plus solver and AI extraction when used. See Credit costs. This is the authoritative figure and matches the X-Credits-Used header.
​
integer
Total request processing time in milliseconds.
​
integer
Number of retry attempts performed. 0 if the first attempt succeeded.
​
integer
Size of the returned content in bytes.
​
string
The PNG screenshot as a data:image/png;base64,… URI. Only present when screenshot: true was requested.
​
string
The browser session as an animated GIF data:image/gif;base64,… URI. Only present when video: true was requested; null if the session could not be recorded.
​
object
HTTP response headers from the target URL.
​
boolean
Whether a blocking page was detected during scraping.
​
object
Details about the detected blocking page. Only present when blocking_detected is true.
​
array
List of anti-bot systems detected on the page.
​
array
List of CAPTCHA systems detected on the page.
​
boolean
Whether the anti-bot solver successfully bypassed the protection.
​
string
Name of the solver that successfully bypassed the block. null if no solver was used.
​
boolean
Whether the wait_for selector appeared within wait_timeout. null when no wait_for was requested. A miss never reaches you as a 200 — it is a 422 wait_for_timeout.
​
array
One entry per executed js_scenario step, in order. null when no scenario ran. Execution stops at the first failed step, so a report shorter than the scenario means a step broke.
​
object | string | array
Structured data extracted by the LLM based on ai_prompt. The shape depends on your prompt. null when ai_extract is false or extraction failed.
​
string
The LLM model used for extraction (e.g., gpt-4o-mini). null when AI extraction was not used.
​
string
Error message if AI extraction failed. The scrape result is still returned. null on success.

​
Binary files and raw bodies

/v1/web/scrape handles binary targets — images, PDFs, archives, fonts, audio and video — as well as HTML. The bytes are returned exactly as the origin sent them; nothing is decoded, parsed or re-encoded on the way through. This works through the same anti-bot machinery as a page scrape, so a file behind Cloudflare, DataDome or Imperva is fetched with the same engine, proxy tier and session you would use for its parent page. Pass the same session_id you used to scrape the page the file was linked from and the file download reuses that session’s cookies and fingerprint. There are two ways to get the bytes. raw_content: true — the body itself. No base64 overhead. Best for large files and for piping straight to disk.
Read the response as bytes, not text. response.text in Python or response.text() in JavaScript will decode the payload as a string and corrupt it. Use response.content / response.arrayBuffer().
JSON with content_base64 — one request, metadata included. Use this when you also want credits_used, engine_used or the protection detections alongside the file.
Python
Binary responses served with raw_content always carry Content-Disposition: attachment and X-Content-Type-Options: nosniff, and their Content-Type is restricted to a known-safe set — scraped bytes are never labelled in a way that would let a browser execute them. Bodies larger than 25 MB are not base64-encoded into JSON. Those return content_base64: null with a detail telling you to re-request using raw_content: true, which returns them without the base64 expansion. Very large files are still held in memory end to end, so treat a few hundred MB as the practical ceiling for a single request.

​
Examples

​
Basic scrape

​
JavaScript rendering with wait

​
AI extraction

​
Full anti-bot bypass with budget

​
Browser automation scenario

Log in, then wait for an element that only exists once the login succeeded. If the flow never gets there you receive a free 422 with the step-by-step report instead of a billed copy of the login page.
Infinite scroll:

​
Error Responses

​
Hostname does not exist

If two independent DNS resolvers confirm that your URL’s hostname does not exist (NXDOMAIN), the request returns 400 with error: "Invalid URL" before scraping. You are charged 0 credits (X-Credits-Used: 0). Check the hostname for typos and confirm that the domain exists before retrying. A hostname with no address records (NODATA) is not enough to trigger this rejection. DNS timeouts, SERVFAIL, other resolver errors, and disagreement between resolvers also do not trigger it; the request continues through the normal scraping and retry path.

​
Unknown request fields

A field this endpoint does not define is rejected rather than ignored, so a request never looks like it succeeded at something it did not do. The response names the offending field and lists everything the endpoint accepts, and nothing is charged.
422 — Unknown request field
There is no return_cookies or form_data parameter. To read cookies use eval_js; to send a request body use method with request_body.

Authorizations

​
x-api-key
string
header
required

Body

application/json
​
url
string
required

The URL to scrape. Must be a valid HTTP or HTTPS URL.

​
method
enum<string>
default:GET

HTTP method. POST/PUT/PATCH require request_body.

Available options:
GET,
POST,
PUT,
PATCH,
DELETE,
HEAD,
OPTIONS
​
request_body
string

Raw body for POST/PUT/PATCH, as a string. There is no form_data parameter — set content_type to match.

​
content_type
string

Content-Type for request_body. Defaults to application/json when a body is present.

​
proxy_tier
enum<string>
default:simple

Proxy pool to route through. Its surcharge is added to EVERY request: simple +1, premium +8, ultra +8 credits.

Available options:
simple,
premium,
ultra
​
eval_js
string

JavaScript evaluated in the loaded page after navigation and waits; its return value replaces the page HTML in content. Browser engines only. Use it to read what the HTML does not carry, e.g. document.cookie (non-HttpOnly cookies only, and anti-bot cookies are bound to the IP and TLS fingerprint that minted them).

​
engine
enum<string>
default:auto

Scraping engine tier to use.

Available options:
auto,
browser
​
format
enum<string>
default:html

Output format for the scraped content.

Available options:
html,
markdown,
text
​
render_js
boolean
default:false

Force JavaScript rendering.

​
wait_for
string

CSS selector or XPath to wait for before extracting.

​
wait_timeout
integer
default:30000

Max wait time in ms for wait_for selector.

​
wait_after_load
integer

Additional ms to wait after page load.

​
js_scenario
object[]

Browser actions to perform before extracting.

​
session_id
string

Persist cookies and state across requests.

​
retry_count
integer
default:3

Max retry attempts on blocking detection.

​
retry_on_block
boolean
default:true

Auto-retry on blocking page detection.

​
country
string

ISO 3166-1 alpha-2 country code for proxy geo-targeting.

​
custom_headers
object

Additional HTTP headers for the target request.

​
screenshot
boolean
default:false

Capture a full-page PNG screenshot.

​
video
boolean
default:false

Record browser session as animated GIF. No extra charge.

​
anti_bot
boolean
default:false

Attempt anti-bot bypass when blocking detected.

​
escalate
boolean
default:false

Allow auto-escalation to stronger engines.

​
max_cost
integer

Maximum credits budget for this request.

​
ai_extract
boolean
default:false

Run AI extraction on scraped content.

​
ai_prompt
string

Natural language instruction for AI extraction.

Response

Successful scrape

​
success
boolean
​
url
string
​
status_code
integer
​
content
string
​
format
string
​
engine_used
string
​
credits_used
integer
​
duration_ms
integer
​
retries_used
integer
​
content_length
integer
​
screenshot_url
string | null
​
video_url
string | null
​
headers
object
​
blocking_detected
boolean
​
blocking_details
object | null
​
antibot_systems
array
​
captcha_systems
array
​
anti_bot_solved
boolean
​
solver_used
string | null
​
ai_extraction
unknown
​
ai_model
string | null
​
ai_error
string | null