# Web Scraping Start Crawl API (https://www.socialcrawl.dev/platforms/web/crawl)

> Starts an async crawl job and returns a SocialCrawl job id (job_...). Poll or cancel it via GET/DELETE /v1/web/jobs/{job_id}; polling is the authoritative status.

TL;DR: `POST /v1/web/crawl` costs 1 credit per page crawled. Submitting holds `limit` credits up front (default limit 10, max 10,000) and the unused portion is refunded when the job settles and returns SocialCrawl's unified JSON schema. Single x-api-key auth, 100 free credits on signup.

## Parameters

| Parameter | Required | Description |
| --- | --- | --- |
| `url` | yes | Root URL to crawl. |
| `limit` | no | Maximum pages to crawl. |
| `max_depth` | no | How many link hops from the root URL to follow. 1 is the root page and the pages it links to. Leave unset to let `limit` alone bound the crawl. |
| `allow_backward_links` | no | Crawl the whole domain rather than only URLs under the starting path. Send it when the root URL is a deep page but you want the entire site. |
| `allow_external_links` | no | Follow links off the domain. Off by default, and easy to make expensive: `limit` is the only thing bounding it. |
| `include_paths` | no | CSV of URL path patterns to crawl, e.g. '/blog/.*,/docs/.*'. Everything else is skipped, which is the cheapest way to narrow a crawl. |
| `exclude_paths` | no | CSV of URL path patterns to skip, e.g. '/tag/.*'. Applied after include_paths. |
| `webhook_url` | no | Optional URL the crawl backend calls directly with its own progress events. SocialCrawl does not sign or filter that callback, and its payload carries the backend's own job id, not your job_... id; poll GET /v1/web/jobs/{job_id} for the authoritative status. |
| `formats` | no | CSV of output formats for each crawled page, e.g. 'markdown' (default) or 'markdown,html'. |
| `goal` | no | Optional. What you are looking for on the site, in your own words. SocialCrawl maps the site, scrapes the URLs that likely hold it (always at least two), and lists every skipped URL with p. Without this param the crawl follows links by depth only. |

## Code example

```bash
curl -X POST "https://www.socialcrawl.dev/v1/web/crawl" \
  -H "x-api-key: sc_YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'
```

## FAQ

### What does the Web Crawl API return?

Starts an async crawl job that walks a site and scrapes every page it finds, returning a job id to poll. The response uses the same JSON envelope as every SocialCrawl endpoint, with the data, credits_used and request_id fields.

### When should I use the Web Crawl endpoint?

Use it when you need many pages from one site; check progress with GET jobs/{job_id}.

### How much does the Web Crawl API cost?

The price depends on the request, from 1 to 10000 credits per call. Every new account gets 100 free credits. Credits for failed or empty calls are refunded automatically, and there is no subscription.

### What parameters does the Web Crawl API need?

The one required parameter is url (root URL to crawl). It also accepts 9 optional parameters, all listed on this page.

### How do I call the Web Crawl API?

Send a POST request to https://www.socialcrawl.dev/v1/web/crawl with a JSON body and your API key in the x-api-key header. An Authorization: Bearer header works too.

See the full Web Scraping API: https://www.socialcrawl.dev/platforms/web

## Pricing

- Standard endpoints: 1 credit per call
- Advanced endpoints: 5 credits per call
- Premium endpoints: 10 credits per call
- 100 free credits on signup, no credit card required. Cached responses cost 0 credits. Credit packs never expire.
- Full pricing: https://www.socialcrawl.dev/pricing

## Explore with AI

Questions this API answers, phrased for an AI assistant:

- What does the SocialCrawl Web Crawl API return?
- How do I get Web Crawl data with the SocialCrawl API?