Skip to main content

Web Scraping API

Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.

Key Features

Smart Engine Selection

Automatically picks the best scraping approach for each site — from fast HTTP requests to full browser rendering.

Anti-Bot Bypass

Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.

AI Extraction

Extract structured data from any page using natural language prompts powered by LLMs.

Video Recording

Record browser sessions as animated GIFs for debugging, visual verification, and monitoring.

Screenshots

Capture full-page PNG screenshots of any webpage.

Geo-Targeting

Route requests through proxies in 37+ countries to access location-specific content.

How It Works

  1. Send a URL to the /v1/web/scrape endpoint
  2. ScrapeBadger picks the best approach — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
  3. Anti-bot detection runs automatically — if a block is detected, the request retries with more powerful methods
  4. Get clean content back in your preferred format (HTML, Markdown, or plain text)

Scraping Tiers

ScrapeBadger uses a tiered system. When engine is set to "auto" (the default), the most cost-effective method is tried first and the system escalates automatically if needed.

Auto-Escalation

When escalate is enabled, ScrapeBadger automatically tries more powerful methods if the initial one is blocked:
You only pay for the method that succeeds — escalation costs are not cumulative.

Credit Costs

Use the max_cost parameter to set a credit budget per request. The request will fail rather than exceed your budget.

Caching

/v1/web/detect caches its result per domain for 5 minutes. A cached detect response is returned with status 200 and costs 0 credits — handy when you probe the same domain repeatedly. To get a fresh detection within that window, wait for the 5-minute TTL to lapse. /v1/web/scrape does not cache page content: every scrape performs a fresh crawl and is billed for the engine that produced the result. There is no stale-content window to work around.

Unsupported Sites

Some sites with aggressive, account-gated bot detection are not supported and may return errors. In particular, walled social platforms such as Facebook, Instagram, and LinkedIn block automated access behind login and are out of scope for /v1/web/scrape. For supported social and marketplace data, use the dedicated source APIs (Twitter/X, Reddit, TikTok, YouTube, etc.).

Quick Example

Endpoints

Next Steps

Scrape Endpoint

Full API reference for the scrape endpoint

Detect Endpoint

Analyze a URL for anti-bot protection