# Web Scraping Map API (https://www.socialcrawl.dev/platforms/web/map)

> Discovers URLs for a public site and returns them as a normalized WebPageList.

TL;DR: `GET /v1/web/map` costs 1 credit per call and returns SocialCrawl's unified JSON schema. Single x-api-key auth, 100 free credits on signup.

## Parameters

| Parameter | Required | Description |
| --- | --- | --- |
| `url` | yes | Site URL to map. |
| `search` | no | Optional path or keyword filter. |
| `limit` | no | Maximum URLs to return, up to 5000. |
| `sitemap` | no | How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it. (include \| skip \| only) |
| `include_subdomains` | no | Follow links onto subdomains of the root URL. Default true. |
| `ignore_query_parameters` | no | Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true. |
| `fresh` | no | Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale. |
| `goal` | no | Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged. |

## Code example

```bash
curl -G "https://www.socialcrawl.dev/v1/web/map" \
  --data-urlencode "url=https://example.com" \
  -H "x-api-key: sc_YOUR_API_KEY"
```

## FAQ

### What does the Web Map API return?

Returns a list of URLs that exist on a site, without fetching the content of each page. The response uses the same JSON envelope as every SocialCrawl endpoint, with the data, credits_used and request_id fields.

### When should I use the Web Map endpoint?

Use it to discover what pages a site has before choosing which ones to scrape or crawl.

### How much does the Web Map API cost?

Each call costs 1 credit, so the 100 free credits every new account gets cover 100 calls. Credits for failed or empty calls are refunded automatically, and there is no subscription.

### What parameters does the Web Map API need?

The one required parameter is url (site URL to map). It also accepts 7 optional parameters, all listed on this page.

### How do I call the Web Map API?

Send a GET request to https://www.socialcrawl.dev/v1/web/map?url=https%3A%2F%2Fexample.com with your API key in the x-api-key header. An Authorization: Bearer header works too.

See the full Web Scraping API: https://www.socialcrawl.dev/platforms/web

## Pricing

- Standard endpoints: 1 credit per call
- Advanced endpoints: 5 credits per call
- Premium endpoints: 10 credits per call
- 100 free credits on signup, no credit card required. Cached responses cost 0 credits. Credit packs never expire.
- Full pricing: https://www.socialcrawl.dev/pricing

## Explore with AI

Questions this API answers, phrased for an AI assistant:

- What does the SocialCrawl Web Map API return?
- How do I get Web Map data with the SocialCrawl API?