> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapebadger.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping Overview

> Scrape any website with automatic anti-bot bypass, JavaScript rendering, and AI-powered data extraction.

# Web Scraping API

Scrape any public webpage and get back clean HTML, Markdown, or plain text. The API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically.

## Key Features

<CardGroup cols={3}>
  <Card title="Smart Engine Selection" icon="gear">
    Automatically picks the best scraping approach for each site — from fast HTTP requests to full browser rendering.
  </Card>

  <Card title="Anti-Bot Bypass" icon="shield-halved">
    Detects and bypasses Cloudflare, DataDome, Akamai, Kasada, and other protection systems.
  </Card>

  <Card title="AI Extraction" icon="brain">
    Extract structured data from any page using natural language prompts powered by LLMs.
  </Card>

  <Card title="Video Recording" icon="video">
    Record browser sessions as animated GIFs for debugging, visual verification, and monitoring.
  </Card>

  <Card title="Screenshots" icon="camera">
    Capture full-page PNG screenshots of any webpage.
  </Card>

  <Card title="Geo-Targeting" icon="globe">
    Route requests through proxies in 37+ countries to access location-specific content.
  </Card>
</CardGroup>

## How It Works

1. **Send a URL** to the `/v1/web/scrape` endpoint
2. **ScrapeBadger picks the best approach** — fast HTTP for simple pages, headless browser for JavaScript-heavy sites
3. **Anti-bot detection runs automatically** — if a block is detected, the request retries with more powerful methods
4. **Get clean content back** in your preferred format (HTML, Markdown, or plain text)

## Scraping Tiers

ScrapeBadger uses a tiered system. When `engine` is set to `"auto"` (the default), the most cost-effective method is tried first and the system escalates automatically if needed.

| Tier                | Description                                                                  | Cost       |
| ------------------- | ---------------------------------------------------------------------------- | ---------- |
| **HTTP**            | Fast HTTP request with Chrome TLS fingerprint — works for most websites      | 1 credit   |
| **Browser**         | Full headless browser with JavaScript rendering — for SPAs and dynamic pages | 5 credits  |
| **Premium Browser** | Real browser with advanced fingerprinting — for heavily protected sites      | 10 credits |

### Auto-Escalation

When `escalate` is enabled, ScrapeBadger automatically tries more powerful methods if the initial one is blocked:

```
HTTP (1 credit) → Browser (5 credits) → Premium Browser (10 credits)
```

You only pay for the method that succeeds — escalation costs are **not cumulative**.

## Credit Costs

| Component                             | Cost       |
| ------------------------------------- | ---------- |
| Basic HTTP scrape                     | 1 credit   |
| Browser rendering (`render_js: true`) | 5 credits  |
| Premium browser (via escalation)      | 10 credits |
| Anti-bot solver (`anti_bot: true`)    | +5 credits |
| AI extraction (`ai_extract: true`)    | +2 credits |
| Video recording (`video: true`)       | +3 credits |
| Retries                               | Free       |
| Failed requests                       | 0 credits  |

<Tip>
  Use the `max_cost` parameter to set a credit budget per request. The request will fail rather than exceed your budget.
</Tip>

## Quick Example

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://scrapebadger.com/v1/web/scrape" \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://scrapebadger.com",
      "format": "markdown"
    }'
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      "https://scrapebadger.com/v1/web/scrape",
      headers={"x-api-key": "YOUR_API_KEY"},
      json={
          "url": "https://scrapebadger.com",
          "format": "markdown"
      }
  )

  data = response.json()
  print(data["content"])
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch("https://scrapebadger.com/v1/web/scrape", {
    method: "POST",
    headers: {
      "x-api-key": "YOUR_API_KEY",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      url: "https://scrapebadger.com",
      format: "markdown"
    })
  });

  const data = await response.json();
  console.log(data.content);
  ```
</CodeGroup>

## Endpoints

| Endpoint                                                        | Method | Description                                  |
| --------------------------------------------------------------- | ------ | -------------------------------------------- |
| [`/v1/web/scrape`](/api-reference/endpoint/web-scraping/scrape) | POST   | Scrape a URL and return content              |
| [`/v1/web/detect`](/api-reference/endpoint/web-scraping/detect) | POST   | Detect anti-bot and CAPTCHA systems on a URL |

## Next Steps

<CardGroup cols={2}>
  <Card title="Scrape Endpoint" icon="globe" href="/api-reference/endpoint/web-scraping/scrape">
    Full API reference for the scrape endpoint
  </Card>

  <Card title="Detect Endpoint" icon="radar" href="/api-reference/endpoint/web-scraping/detect">
    Analyze a URL for anti-bot protection
  </Card>
</CardGroup>