> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapebadger.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Long-running and async requests

> How long a synchronous call can run, what happens when it is cut off, and how to run slow calls such as ChatGPT, Gemini and AI Overviews asynchronously at production volume.

Most calls answer in a few seconds. Some render a live page or wait for an AI answer: ChatGPT, Gemini and Perplexity asks usually take 15–40 s and sometimes over a minute. At volume, run these asynchronously.

## Synchronous calls have about two minutes

A normal (synchronous) call has about **115 s** to return. Long-running endpoints such as the AI asks fit their retries into that window: if they can't finish, you get an error response instead of a dropped connection. Failed calls are not billed.

* Set your HTTP client timeout to **130 s** for AI endpoints. A shorter timeout aborts answers that would have arrived.
* If your connection drops before the response arrives, the call is **not billed**.

## Run slow calls asynchronously

Add `Prefer: respond-async, wait=25` to any read request. Async calls are not bound by that window.

* **If the call finishes within `wait` seconds** (1–120), you get the normal response.
* **Otherwise** you get `202 Accepted` with a `request_id`:

  ```json theme={null}
  {"request_id": "…", "status": "running", "retry_after_seconds": 5,
   "next_action": {"method": "GET", "url": "/v1/requests/…"}}
  ```

Poll `GET /v1/requests/{request_id}`. Polling is free. It returns `202` while the call runs, then the original status, headers and body. Results stay available for about 15 minutes.

The call is billed once, when it completes. A call that never completes is not billed. Don't re-send the original request while it runs: that starts, and bills, a second call.

```bash cURL theme={null}
curl -s -i "https://scrapebadger.com/v1/gemini/ask?prompt=best+crm+for+startups" \
  -H "X-API-Key: $SCRAPEBADGER_API_KEY" \
  -H "Prefer: respond-async, wait=25"
# 202 → poll the Location header:
curl -s "https://scrapebadger.com/v1/requests/$REQUEST_ID" -H "X-API-Key: $SCRAPEBADGER_API_KEY"
```

```python Python theme={null}
import time, httpx

def ask_async(client: httpx.Client, path: str, params: dict) -> httpx.Response:
    r = client.get(path, params=params, headers={"Prefer": "respond-async, wait=25"})
    while r.status_code == 202:
        time.sleep(int(r.headers.get("Retry-After", "5")))
        r = client.get(r.json()["next_action"]["url"])
    return r

with httpx.Client(base_url="https://scrapebadger.com",
                  headers={"X-API-Key": "YOUR_API_KEY"}, timeout=130) as client:
    print(ask_async(client, "/v1/chatgpt/ask", {"prompt": "best crm for startups"}).json())
```

```javascript JavaScript theme={null}
async function askAsync(path, params) {
  const headers = { 'X-API-Key': process.env.SCRAPEBADGER_API_KEY };
  let r = await fetch(`https://scrapebadger.com${path}?${new URLSearchParams(params)}`, {
    headers: { ...headers, Prefer: 'respond-async, wait=25' },
  });
  while (r.status === 202) {
    const { next_action, retry_after_seconds } = await r.json();
    await new Promise((ok) => setTimeout(ok, retry_after_seconds * 1000));
    r = await fetch(`https://scrapebadger.com${next_action.url}`, { headers });
  }
  return r.json();
}
```

## Get the result by webhook

Add `X-Callback-URL: https://your-server.example/hook` to an async call (it needs `Prefer: respond-async`). When the call completes, ScrapeBadger POSTs the result there as JSON. A call that finishes within `wait` still gets its webhook, as well as the normal response.

```json theme={null}
{
  "event": "request.completed",
  "request_id": "6ca4ab7a-…",
  "batch_id": null,
  "status": 200,
  "headers": {"content-type": "application/json", "x-credits-used": "20"},
  "body": {"answer": "…", "sources": ["…"]},
  "body_size": 4812,
  "result_url": "https://scrapebadger.com/v1/requests/6ca4ab7a-…",
  "result_expires_at": 1793139751,
  "attempt": 1,
  "sent_at": 1793138851
}
```

* `status` and `body` are the call's own, errors included. A body over 1 MB is left out (`"body": null, "body_omitted": true`): fetch it from `result_url` within about 15 minutes.
* The URL must be `https` on a public address. Private, loopback and cloud-metadata addresses are refused, and redirects are not followed. A bad URL gets `400 invalid_callback_url` before the call runs, and nothing is billed.
* Answer with any `2xx`. On a timeout, a connection error, `408`, `429` or `5xx`, the delivery is retried after 10 s, 60 s and 5 min (4 attempts in total). Other statuses are final. Deliveries can repeat, so deduplicate on `request_id`.
* `GET /v1/requests/{request_id}` reports the delivery: as a `callback` object (`status`: `pending` / `delivered` / `failed`, `attempts`, `last_error`) while the call runs, and as `X-Callback-Status` and `X-Callback-Attempts` headers on the final response.
* Calls refused before they run (invalid key, no credits, rate limit) answer synchronously and send no webhook.
* Deliveries in progress don't survive a restart of our gateway. If a restart interrupts retries, the poll reports the delivery as `failed` with `last_error: "interrupted (gateway restart)"`. Fetch the result from `result_url` while it is still kept.

### Verify the signature

Every delivery is signed with your account's webhook secret. Get the secret from `GET /v1/account/webhook-secret`, and rotate it with `POST /v1/account/webhook-secret/rotate`. Both need a full-access API key: a key restricted to some APIs gets `403 full_access_required`. `X-ScrapeBadger-Signature` is `sha256=` followed by the hex HMAC-SHA256 of the **raw** request body. Check it before you parse the body, and reject stale `sent_at` values to stop replays. This is the same scheme stream monitors use.

```python Python theme={null}
import hashlib, hmac

def verify(secret: str, raw_body: bytes, signature: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

```javascript JavaScript theme={null}
import { createHmac, timingSafeEqual } from 'node:crypto';

function verify(secret, rawBody, signature) {
  const expected = 'sha256=' + createHmac('sha256', secret).update(rawBody).digest('hex');
  return expected.length === signature.length &&
    timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}
```

## Submit a batch

`POST /v1/batch` takes up to **100** GET reads and returns a `request_id` for each, at once:

```bash theme={null}
curl -s https://scrapebadger.com/v1/batch \
  -H "X-API-Key: $SCRAPEBADGER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"method": "GET", "path": "/v1/chatgpt/ask", "query": {"prompt": "best crm for startups"}},
      {"method": "GET", "path": "/v1/gemini/ask", "query": {"prompt": "best crm for startups"}}
    ],
    "callback_url": "https://your-server.example/hook"
  }'
```

```json theme={null}
{"batch_id": "efaa0fe6-…", "count": 2, "accepted": 2, "requests": [
  {"index": 0, "accepted": true, "status": 202, "request_id": "3cb3…", "poll_url": "/v1/requests/3cb3…"},
  {"index": 1, "accepted": true, "status": 202, "request_id": "8a73…", "poll_url": "/v1/requests/8a73…"}
]}
```

* Each item is an ordinary async request under your key. It has the same authentication, permissions, credit check, [rate limit](https://docs.scrapebadger.com/rate-limits.md) and price as if you had sent it alone. Submitting the batch is free.
* Items are checked against your balance before any of them start, at each endpoint's base price (what `POST /v1/quote` reports). Items whose price no longer fits, after the items before them and your other batch items still running, come back as `402 insufficient_credits` and are not started.
* An item that can't start (for example `402` as above, `429` over your rate limit, or `404` unknown endpoint) comes back with `"accepted": false`, its `status` and `error`. It is not billed, and you can resubmit just that item later.
* Poll each `poll_url`, or pass `callback_url` to get one webhook per item, each with `batch_id` set.
* At most 10 items of one batch run at a time. The others wait their turn and show as `running` when polled.
* A batch over 100 items gets `413 batch_too_large`.
* Items still waiting for a slot when our gateway restarts are dropped. They are not billed, and polling them returns `404 request_not_found` within about 5 minutes. Resubmit those.
* Unclaimed trial keys can't batch (`403 trial_not_allowed`) or use webhooks. They can still use `Prefer: respond-async`, one call at a time: a second call while the first is still running gets `429 trial_concurrency`.

## Concurrency

There is no per-key concurrency limit beyond your plan's [rate limit](https://docs.scrapebadger.com/rate-limits.md). AI asks render a live answer, so latency rises when many run at once. Start around 10 in parallel per engine and scale up while you watch the error rate. A `502` on an AI ask means the engine produced no usable answer on that attempt. It is not billed, and retrying it is safe.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.