Scrappa's POST /api/web-scraper/crawl follows links from a start URL and returns the pages in one request.
Request
Send JSON: {"url": "https://example.com", "limit": 10, "max_depth": 1, "include_paths": ["/docs"], "exclude_paths": ["/docs/old"], "format": "markdown"}.
Rules
The crawl stays on the host of the start URL (include_subdomains adds subdomains). max_depth 0 returns only the start page. Paths are prefixes and * matches anything. exclude_paths wins over include_paths. A request has a time budget of 80 seconds. If the budget or limit ends the crawl early, truncated is true. This is one bounded request, not a background job.
Result
pages has url, depth, status, title, content (markdown or HTML) and links. failed lists pages that could not be fetched. Page content is capped at 300000 characters (content_truncated is true then). The whole response is capped at 5000000 characters; later pages are listed in failed and truncated is true.
Credits
1 credit per returned page. 3 credits for a page that needed a browser (at most 5 per request). meta.credits is the total. Failed pages are free. If no page works, the request answers 503 and costs nothing.
Run this endpoint
Endpoint
Parameters
Start with the required fields, then add optional filters only when your use case needs them.
Runnable path
1 required parameter needed before sending a request.
6 optional filters available.
integer
Optional
Maximum pages, 1-100. Default 10.
3
integer
Optional
Link depth, 0-3. Default 1.
1
array
Optional
JSON array of path prefixes to crawl.
["example"]
array
Optional
JSON array of path prefixes to skip.
["example"]
string
Optional
markdown (default) or html.
markdown
boolean
Optional
true to follow subdomains.
true
Response Schema
Example response fields are illustrative; inspect the JSON before integrating.
Example response fields
Scan these fields before integrating.
data
meta
success
{
"data": {
"pages": [
{
"url": "https://example.com",
"depth": 0,
"links": [
"https://www.iana.org/domains/example"
],
"title": "Example Domain",
"status": 200,
"content": "# Example Domain"
}
],
"failed": [],
"start_url": "https://example.com",
"truncated": false,
"pages_returned": 1
},
"meta": {
"credits": 1
},
"success": true
}Errors
Handle these documented responses before retrying or showing customer-facing failures.
Validation Error
A parameter is missing or invalid, or the url points to a private, local or non-public address. Not charged.
{
"error": {
"code": "VALIDATION_ERROR",
"message": "The request validation failed"
},
"success": false
}Site Unreachable Or Blocked
The site could not be fetched after several tries, or it blocked the request. Not charged.
{
"error": {
"code": "UPSTREAM_UNAVAILABLE",
"message": "The site could not be reached. Please retry shortly."
},
"success": false
}Generate Code with AI
Copy a ready-made prompt with all the endpoint details, parameters, and example responses. Paste it into ChatGPT, Claude, or any AI assistant to instantly generate working code.
Try It Live
Test this endpoint in our interactive playground with real data.