# Web Scraper

Fetch any URL and extract structured data including title, meta description, keywords, favicon, social links, all links on the page, emails, phone numbers, images, body text, and detected languages. Set javascript=true for client-rendered pages. Returns the site's HTTP status code separately so you can distinguish site errors (404, 500) from infrastructure errors. Use response_type=markdown to get only the page content as clean markdown.

- **Documentation:** [https://scrappa.co/docs/web-scraper/web_scraper](https://scrappa.co/docs/web-scraper/web_scraper)
- **API group:** Web Scraper API
- **Endpoint:** `GET https://scrappa.co/api/web-scraper`

## Authentication

Send your Scrappa API key in the `X-API-KEY` request header. Paid endpoints also support accountless x402 payments when called without an API key.

## Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | Yes | The URL to scrape (must include protocol, e.g., https://example.com) |
| `include_html` | boolean | No | Set to true to include the raw HTML in the response. Default: false. Ignored when response_type=markdown |
| `javascript` | boolean | No | Set to true to execute page JavaScript before extracting content. This mode uses the Obscura browser engine through Scrappa's proxy layer. Default: false |
| `response_type` | string | No | Set to "markdown" to return only the page content as clean markdown text. When set to markdown, only the markdown content is returned (not JSON). Default: json |

## Example request

```bash
#!/bin/bash

curl -X GET \
    -H "x-api-key: YOUR_API_KEY_HERE" \
    "https://scrappa.co/api/web-scraper?url=https%3A%2F%2Fexample.com&include_html=0&javascript=0"
```

## Example response

```json
{
    "success": true,
    "site_status_code": 200,
    "url": "https://example.com",
    "final_url": "https://example.com",
    "data": {
        "title": "Example Domain",
        "description": "This domain is for use in illustrative examples.",
        "keywords": [],
        "favicon": "https://example.com/favicon.ico",
        "social_links": {
            "linkedin": null,
            "twitter": null,
            "facebook": null,
            "instagram": null,
            "youtube": null,
            "tiktok": null
        },
        "extracted_keywords": [
            "domain",
            "example",
            "illustrative"
        ],
        "links": [
            "https://www.iana.org/domains/example"
        ],
        "emails": [],
        "phone_numbers": [],
        "images": [],
        "body_text": "Example Domain This domain is for use in illustrative examples in documents.",
        "languages_detected": [
            "en"
        ],
        "html": null
    }
}
```

## Errors

| Status | Error | Description |
| --- | --- | --- |
| 400 | Invalid URL | The url parameter is not a valid absolute URL or domain. |
| 422 | Validation Error | Required parameters failed request validation. |
| 502 | Proxy Or SSL Failure | The proxy pool was exhausted or the target SSL connection failed. This is separate from the target site HTTP status code. |
| 504 | Connection Timeout | The target could not be fetched through the proxy layer before the scraper timeout. |
| 503 | JavaScript Renderer Unavailable | JavaScript rendering is temporarily unavailable or its bounded queue is full. Retry later; failed requests are not billed. |

## More Scrappa resources

- [API documentation](https://scrappa.co/docs)
- [Full LLM-readable API reference](https://scrappa.co/llms-full.txt)
- [OpenAPI specification](https://scrappa.co/docs/api.json)
- [Pricing](https://scrappa.co/pricing)