Scrappa's GET /api/web-scraper/map lists the URLs of a public website.
Sources
The API reads the Sitemap: lines of robots.txt, then /sitemap.xml and /sitemap_index.xml, and follows sitemap indexes two levels deep. If the site has no sitemap, it lists the links of the home page (source is link). Compressed .xml.gz sitemaps are not read. robots.txt Disallow rules are not applied.
Filters
search keeps URLs that contain the text. include_subdomains=true adds subdomains. Both filters apply before limit, so returned never contains a URL that breaks a filter. total_found is the count after the filters and truncated tells you whether limit cut the list.
Result
Each URL has url, lastmod, changefreq, priority and source. Missing values are null.
Credits
1 credit per successful request. A site without URLs answers 404 and costs nothing.
Run this endpoint
Endpoint
Parameters
Start with the required fields, then add optional filters only when your use case needs them.
Runnable path
1 required parameter needed before sending a request.
3 optional filters available.
string
Required
Website address, for example https://www.sitemaps.org.
https://www.sitemaps.org
integer
Optional
Maximum URLs returned, 1-50000. Default 5000.
2
string
Optional
Return only URLs that contain this text.
example
boolean
Optional
true to include subdomains. Default false.
true
Response Schema
Example response fields are illustrative; inspect the JSON before integrating.
Example response fields
Scan these fields before integrating.
data
meta
success
{
"data": {
"url": "https://www.sitemaps.org",
"urls": [
{
"url": "https://www.sitemaps.org/protocol.html",
"source": "sitemap",
"lastmod": "2026-10-01",
"priority": 0.8,
"changefreq": null
},
{
"url": "https://www.sitemaps.org/faq.html",
"source": "sitemap",
"lastmod": null,
"priority": null,
"changefreq": null
}
],
"returned": 2,
"truncated": true,
"total_found": 40
},
"meta": {
"credits": 1
},
"success": true
}Errors
Handle these documented responses before retrying or showing customer-facing failures.
Validation Error
A parameter is missing or invalid, or the url points to a private, local or non-public address. Not charged.
{
"error": {
"code": "VALIDATION_ERROR",
"message": "The request validation failed"
},
"success": false
}Site Unreachable Or Blocked
The site could not be fetched after several tries, or it blocked the request. Not charged.
{
"error": {
"code": "UPSTREAM_UNAVAILABLE",
"message": "The site could not be reached. Please retry shortly."
},
"success": false
}No URLs Found
The site has no sitemap, no links, or no URL matches the filters. Not charged.
{
"error": {
"code": "NO_URLS_FOUND",
"message": "No URLs were found for this site and filter."
},
"success": false
}Generate Code with AI
Copy a ready-made prompt with all the endpoint details, parameters, and example responses. Paste it into ChatGPT, Claude, or any AI assistant to instantly generate working code.
Try It Live
Test this endpoint in our interactive playground with real data.