Skip to main content

​
Web Scraping API

Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.

​
Key Features

Smart Engine Selection

Automatically picks the best scraping engine for each site — from fast HTTP requests to full browser rendering.

Anti-Bot Bypass

Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.

AI Extraction

Extract structured data from any page using natural language prompts powered by LLMs.

​
How It Works

  1. Send a URL to the /v1/web/scrape endpoint
  2. ScrapeBadger selects the best engine — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
  3. Anti-bot detection runs automatically — if a block is detected, the request retries with escalated engines
  4. Get clean content back in your preferred format (HTML, Markdown, or plain text)

​
Scraping Engines

ScrapeBadger uses a tiered engine system. When engine is set to "auto" (the default), the cheapest engine is tried first and escalation happens automatically if blocking is detected.
EngineDescriptionCostBest For
curl_cffiHTTP client with Chrome TLS fingerprint impersonation1 creditStatic pages, APIs, most websites
browserHeadless browser (Patchright/Camoufox)5 creditsJavaScript-rendered pages, SPAs
windows_chromeReal Chrome on a remote Windows VPS via CDP10 creditsHeavily protected sites with advanced fingerprinting
rnet is a backward-compatible alias for curl_cffi. patchright is a backward-compatible alias for browser.

​
Auto-Escalation

When escalate is enabled (default: false), ScrapeBadger automatically tries more powerful engines if the initial one is blocked: 
curl_cffi (1 credit) → browser (5 credits) → windows_chrome (10 credits)
You only pay for the engine that succeeds — escalation costs are not cumulative.

​
Credit Costs

ComponentCost
curl_cffi / rnet engine1 credit
browser / patchright engine5 credits
windows_chrome engine10 credits
Anti-bot solver (anti_bot: true)+5 credits
AI extraction (ai_extract: true)+2 credits
RetriesFree
Failed requests0 credits
Use the max_cost parameter to set a credit budget per request. The request will fail rather than exceed your budget.

​
Quick Example

curl -X POST "https://scrapebadger.com/v1/web/scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "markdown"
  }'

​
Endpoints

EndpointMethodDescription
/v1/web/scrapePOSTScrape a URL and return content
/v1/web/detectPOSTDetect anti-bot and CAPTCHA systems on a URL

​
Next Steps

Scrape Endpoint

Full API reference for the scrape endpoint

Detect Endpoint

Analyze a URL for anti-bot protection