# Web Scraper Site Map API Documentation

Scrappa's `GET /api/web-scraper/map` lists the URLs of a public website.

## Sources

The API reads the `Sitemap:` lines of `robots.txt`, then `/sitemap.xml` and `/sitemap_index.xml`, and follows sitemap indexes two levels deep. If the site has no sitemap, it lists the links of the home page (`source` is `link`). Compressed `.xml.gz` sitemaps are not read. `robots.txt` Disallow rules are not applied.

## Filters

`search` keeps URLs that contain the text. `include_subdomains=true` adds subdomains. Both filters apply before `limit`, so `returned` never contains a URL that breaks a filter. `total_found` is the count after the filters and `truncated` tells you whether `limit` cut the list.

## Result

Each URL has `url`, `lastmod`, `changefreq`, `priority` and `source`. Missing values are `null`.

## Credits

1 credit per successful request. A site without URLs answers `404` and costs nothing.

- **Documentation:** [https://scrappa.co/docs/web-scraper-api/web_scraper_map](https://scrappa.co/docs/web-scraper-api/web_scraper_map)
- **API group:** Web Scraper API
- **Endpoint:** `GET https://scrappa.co/api/web-scraper/map`

## Authentication

Send your Scrappa API key in the `X-API-KEY` request header. Paid endpoints also support accountless x402 payments when called without an API key.

## Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | Yes | Website address, for example https://www.sitemaps.org. |
| `limit` | integer | No | Maximum URLs returned, 1-50000. Default 5000. |
| `search` | string | No | Return only URLs that contain this text. |
| `include_subdomains` | boolean | No | true to include subdomains. Default false. |

## Example response

```json
{
    "data": {
        "url": "https://www.sitemaps.org",
        "urls": [
            {
                "url": "https://www.sitemaps.org/protocol.html",
                "source": "sitemap",
                "lastmod": "2026-10-01",
                "priority": 0.8,
                "changefreq": null
            },
            {
                "url": "https://www.sitemaps.org/faq.html",
                "source": "sitemap",
                "lastmod": null,
                "priority": null,
                "changefreq": null
            }
        ],
        "returned": 2,
        "truncated": true,
        "total_found": 40
    },
    "meta": {
        "credits": 1
    },
    "success": true
}
```

## Errors

| Status | Error | Description |
| --- | --- | --- |
| 422 | Validation Error | A parameter is missing or invalid, or the url points to a private, local or non-public address. Not charged. |
| 503 | Site Unreachable Or Blocked | The site could not be fetched after several tries, or it blocked the request. Not charged. |
| 404 | No URLs Found | The site has no sitemap, no links, or no URL matches the filters. Not charged. |

## More Scrappa resources

- [API documentation](https://scrappa.co/docs)
- [Full LLM-readable API reference](https://scrappa.co/llms-full.txt)
- [OpenAPI specification](https://scrappa.co/docs/api.json)
- [Pricing](https://scrappa.co/pricing)