Skip to main content

​
Web Scraping API

Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.

​
Key Features

Smart Engine Selection

Automatically picks the best scraping approach for each site — from fast HTTP requests to full browser rendering.

Anti-Bot Bypass

Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.

AI Extraction

Extract structured data from any page using natural language prompts powered by LLMs.

Video Recording

Record browser sessions as animated GIFs for debugging, visual verification, and monitoring.

Screenshots

Capture full-page PNG screenshots of any webpage.

Geo-Targeting

Route requests through proxies in 37+ countries to access location-specific content.

​
How It Works

  1. Send a URL to the /v1/web/scrape endpoint
  2. ScrapeBadger picks the best approach — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
  3. Anti-bot detection runs automatically — if a block is detected, the request retries with more powerful methods
  4. Get clean content back in your preferred format (HTML, Markdown, or plain text)

​
Scraping Tiers

ScrapeBadger uses a tiered system. When engine is set to "auto" (the default), the most cost-effective method is tried first and the system escalates automatically if needed.
TierDescriptionCost
HTTPFast HTTP request with Chrome TLS fingerprint — works for most websites1 credit
BrowserFull headless browser with JavaScript rendering — for SPAs and dynamic pages5 credits
Premium BrowserReal browser with advanced fingerprinting — for heavily protected sites10 credits

​
Auto-Escalation

When escalate is enabled, ScrapeBadger automatically tries more powerful methods if the initial one is blocked: 
HTTP (1 credit) → Browser (5 credits) → Premium Browser (10 credits)
You only pay for the method that succeeds — escalation costs are not cumulative.

​
Credit Costs

ComponentCost
Basic HTTP scrape1 credit
Browser rendering (render_js: true)5 credits
Premium browser (via escalation)10 credits
Anti-bot solver (anti_bot: true)+5 credits
AI extraction (ai_extract: true)+2 credits
Video recording (video: true)+3 credits
RetriesFree
Failed requests0 credits
Use the max_cost parameter to set a credit budget per request. The request will fail rather than exceed your budget.

​
Quick Example

curl -X POST "https://scrapebadger.com/v1/web/scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapebadger.com",
    "format": "markdown"
  }'

​
Endpoints

EndpointMethodDescription
/v1/web/scrapePOSTScrape a URL and return content
/v1/web/detectPOSTDetect anti-bot and CAPTCHA systems on a URL

​
Next Steps

Scrape Endpoint

Full API reference for the scrape endpoint

Detect Endpoint

Analyze a URL for anti-bot protection