Web Scraping API
Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.Key Features
Smart Engine Selection
Automatically picks the best scraping approach for each site — from fast HTTP requests to full browser rendering.
Anti-Bot Bypass
Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.
AI Extraction
Extract structured data from any page using natural language prompts powered by LLMs.
Video Recording
Record browser sessions as animated GIFs for debugging, visual verification, and monitoring.
Screenshots
Capture full-page PNG screenshots of any webpage.
Geo-Targeting
Route requests through proxies in 37+ countries to access location-specific content.
How It Works
- Send a URL to the
/v1/web/scrapeendpoint - ScrapeBadger picks the best approach — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
- Anti-bot detection runs automatically — if a block is detected, the request retries with more powerful methods
- Get clean content back in your preferred format (HTML, Markdown, or plain text)
Scraping Tiers
ScrapeBadger uses a tiered system. Whenengine is set to "auto" (the default), the most cost-effective method is tried first and the system escalates automatically if needed.
Auto-Escalation
Whenescalate is enabled, ScrapeBadger automatically tries more powerful methods if the initial one is blocked:
Credit Costs
Caching
/v1/web/detect caches its result per domain for 5 minutes. A cached detect response is returned with status 200 and costs 0 credits — handy when you probe the same domain repeatedly. To get a fresh detection within that window, wait for the 5-minute TTL to lapse.
/v1/web/scrape does not cache page content: every scrape performs a fresh crawl and is billed for the engine that produced the result. There is no stale-content window to work around.
Unsupported Sites
Some sites with aggressive, account-gated bot detection are not supported and may return errors. In particular, walled social platforms such as Facebook, Instagram, and LinkedIn block automated access behind login and are out of scope for/v1/web/scrape. For supported social and marketplace data, use the dedicated source APIs (Twitter/X, Reddit, TikTok, YouTube, etc.).
Quick Example
Endpoints
Next Steps
Scrape Endpoint
Full API reference for the scrape endpoint
Detect Endpoint
Analyze a URL for anti-bot protection

