# Web Scraper Crawl API Documentation

Scrappa's `POST /api/web-scraper/crawl` follows links from a start URL and returns the pages in one request.

## Request

Send JSON: `{"url": "https://example.com", "limit": 10, "max_depth": 1, "include_paths": ["/docs"], "exclude_paths": ["/docs/old"], "format": "markdown"}`.

## Rules

The crawl stays on the host of the start URL (`include_subdomains` adds subdomains). `max_depth` 0 returns only the start page. Paths are prefixes and `*` matches anything. `exclude_paths` wins over `include_paths`. A request has a time budget of 80 seconds. If the budget or `limit` ends the crawl early, `truncated` is `true`. This is one bounded request, not a background job.

## Result

`pages` has `url`, `depth`, `status`, `title`, `content` (markdown or HTML) and `links`. `failed` lists pages that could not be fetched. Page content is capped at 300000 characters (`content_truncated` is `true` then). The whole response is capped at 5000000 characters; later pages are listed in `failed` and `truncated` is `true`.

## Credits

1 credit per returned page. 3 credits for a page that needed a browser (at most 5 per request). `meta.credits` is the total. Failed pages are free. If no page works, the request answers `503` and costs nothing.

- **Documentation:** [https://scrappa.co/docs/web-scraper-api/web_scraper_crawl](https://scrappa.co/docs/web-scraper-api/web_scraper_crawl)
- **API group:** Web Scraper API
- **Endpoint:** `POST https://scrappa.co/api/web-scraper/crawl`

## Authentication

Send your Scrappa API key in the `X-API-KEY` request header. Paid endpoints also support accountless x402 payments when called without an API key.

## Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | Yes | Start URL. |
| `limit` | integer | No | Maximum pages, 1-100. Default 10. |
| `max_depth` | integer | No | Link depth, 0-3. Default 1. |
| `include_paths` | array | No | JSON array of path prefixes to crawl. |
| `exclude_paths` | array | No | JSON array of path prefixes to skip. |
| `format` | string | No | markdown (default) or html. |
| `include_subdomains` | boolean | No | true to follow subdomains. |

## Example response

```json
{
    "data": {
        "pages": [
            {
                "url": "https://example.com",
                "depth": 0,
                "links": [
                    "https://www.iana.org/domains/example"
                ],
                "title": "Example Domain",
                "status": 200,
                "content": "# Example Domain"
            }
        ],
        "failed": [],
        "start_url": "https://example.com",
        "truncated": false,
        "pages_returned": 1
    },
    "meta": {
        "credits": 1
    },
    "success": true
}
```

## Errors

| Status | Error | Description |
| --- | --- | --- |
| 422 | Validation Error | A parameter is missing or invalid, or the url points to a private, local or non-public address. Not charged. |
| 503 | Site Unreachable Or Blocked | The site could not be fetched after several tries, or it blocked the request. Not charged. |

## More Scrappa resources

- [API documentation](https://scrappa.co/docs)
- [Full LLM-readable API reference](https://scrappa.co/llms-full.txt)
- [OpenAPI specification](https://scrappa.co/docs/api.json)
- [Pricing](https://scrappa.co/pricing)