# Web Scraping (/docs/web) Scrape, search, map, extract, crawl, and monitor any public web page. Clean markdown and structured data in the unified WebPage schema Scrape, search, map, extract, crawl, and monitor any public web page. Content-bearing responses return the unified [WebPage schema](/docs/schema/web-page.md): clean markdown or HTML, metadata, structured extraction, and fetch provenance under `data.page`, while search and map return a `WebPageList` under `data.items[]`. This is the only platform on the API that sells **stateful** work: long-running jobs you poll, monitors that keep checking on a schedule, and browser sessions you drive step by step. Everything else here is one request, one answer. Base URL: `/v1/web/...` The crawl hold is taken against `limit`, not against how big the site turns out to be. Submitting `limit=1000` reserves 1000 credits the moment the job is accepted, even if the site only has 30 pages. Run `GET /v1/web/map` first and set `limit` from what you find. ## Quickstart ### 1. Scrape a page to markdown ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com" \ -H "x-api-key: YOUR_API_KEY" ``` The page content arrives at `data.page.content.markdown`. Add `formats=markdown,screenshot` for a screenshot URL under `data.page.media`. ### 2. Pull typed fields instead of prose ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/web/extract?url=https://example.com&prompt=the%20page%20title%20and%20any%20pricing%20mentioned" \ -H "x-api-key: YOUR_API_KEY" ``` `extract` needs either a `schema` (JSON) or a `prompt`. Sending neither is a `400` before billing. Typed fields land under `data.page.extraction`. Every value is checked against the page's own text. A placeholder (`"Not specified"`, `"/"`, a bare link path in a field that is not a link) or a value the page does not state comes back as `null`, and `data.page.extraction_check.removed` lists each one with the value it had and why (`placeholder` or `not_on_page`). A value stated in another language is kept. A value that could not be checked is returned as extracted and named in `extraction_check.unverified` and `_warnings`. ## Synchronous endpoints These four answer immediately and cover most integrations. | Endpoint | Credits | What it returns | Key parameters | | --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `GET /v1/web/scrape` | 1, or 5 for heavy work. A social platform URL the scrapers cannot read costs that platform endpoint's price ([see below](#social-platform-urls)) | One known URL as clean markdown or HTML, with the resolved URL, status code, and fetch metadata | `url`, `formats`, `only_main_content`, `wait_for`, `mobile`, `timeout`, `max_age`, `location_country`, `include_tags`, `exclude_tags`, `proxy`, `pdf_parse`, `block_ads` | | `GET /v1/web/search` | 2 per 10 results | Ranked web, news, and image results as one normalised list, each row tagged with `source_type` | `query`, `sources`, `categories`, `limit`, `country`, `location`, `time_range`, `sort_by_date`, `include_domains`, `exclude_domains`, `include_content` | | `GET /v1/web/map` | 1 | Every URL a site exposes, without fetching any page bodies | `url`, `search`, `limit`, `sitemap`, `include_subdomains`, `ignore_query_parameters`, `fresh` | | `GET /v1/web/extract` | 5 flat | Structured fields from one page, shaped by your `schema` or `prompt` | `url`, plus `schema` or `prompt`, `only_main_content`, `timeout`, `max_age`, `proxy` | `scrape` holds 5 credits when you ask for work the cheap path may not be able to do (`proxy=auto`, `proxy=enhanced`, or `pdf_parse=true`) and settles to what the fetch actually used, so a page a plain request could read still costs 1. The `data.page.fetch.proxy_tier` field tells you which path actually ran. `search` is 2 credits per 10 results, so `limit=50` is 10 credits. Adding `include_content=true` scrapes each result inline and adds 1 credit per result on top, which is how a `limit=50` content search reaches 60 credits. Filter with `include_domains` / `exclude_domains`, `country`, `time_range`, and `categories` before you widen `limit`. ### Social platform URLs LinkedIn, Reddit and Threads block page scrapers, and so do most Facebook and Instagram pages, so a scrape of a profile or post on those sites usually fails. When `/v1/web/scrape` gets one of the URLs below and the scrapers cannot open it, SocialCrawl reads the page through that platform's own endpoint and returns the result in the usual `data.page` shape. | URL | Read through | Price when it is used | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------- | | `linkedin.com/in/` (any country host, such as `de.linkedin.com`) | `/v1/linkedin/profile` | 5 credits | | `linkedin.com/company/` | `/v1/linkedin/company` | 5 credits | | `linkedin.com/posts/...-activity--...`, `linkedin.com/feed/update/urn:li:activity:` | `/v1/linkedin/post` | 5 credits | | `instagram.com/`, `instagram.com/p/`, `/reel/` | `/v1/instagram/profile`, `/v1/instagram/post` | 1 credit | | `facebook.com/`, `facebook.com//posts/`, `/reel/` | `/v1/facebook/profile`, `/v1/facebook/post` | 1 credit | | `reddit.com/r/`, `reddit.com/r//comments/`, `reddit.com/user/` | `/v1/reddit/subreddit`, `/v1/reddit/post`, `/v1/reddit/profile` | 1 credit | | `tiktok.com/@`, `tiktok.com/@/video/` | `/v1/tiktok/profile`, `/v1/tiktok/post` | 1 credit | | `youtube.com/watch?v=`, `youtu.be/`, `/shorts/`, `youtube.com/@` | `/v1/youtube/video`, `/v1/youtube/channel` | 1 credit | | `x.com/`, `x.com//status/` (and `twitter.com`) | `/v1/twitter/profile`, `/v1/twitter/tweet` | 1 credit | | `threads.com/@`, `threads.com/@/post/` | `/v1/threads/profile`, `/v1/threads/post` | 1 credit | A LinkedIn profile asked for as HTML, which the scrapers cannot open: ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://de.linkedin.com/in/williamhgates&formats=html" \ -H "x-api-key: YOUR_API_KEY" ``` ```json title="Response (trimmed)" { "success": true, "credits_used": 5, "data": { "page": { "url": "https://de.linkedin.com/in/williamhgates", "final_url": "https://www.linkedin.com/in/williamhgates/", "title": "Bill Gates (@williamhgates)", "content": { "markdown": null, "html": "

Bill Gates (@williamhgates)

Chair, Gates Foundation and Founder, Breakthrough Energy

  • Followers: 40,768,450
  • Following: 8
" }, "page_state": { "value": "content", "confidence": 1, "skipped_judge": true }, "native": { "endpoint": "/v1/linkedin/profile", "credits_used": 5, "data": { "author": { "username": "williamhgates", "display_name": "Bill Gates", "followers": 40768450, "url": "https://www.linkedin.com/in/williamhgates/" } } } }, "_warnings": [ "The page scrapers could not open this page, so it was read through /v1/linkedin/profile and charged at that endpoint's price (5 credits) instead of the scrape price. That call is its own row in your Activity Logs. The full record is in page.native.data." ] } } ``` How it works and how it is billed: - **The scrapers go first.** If they can read the page, you get the scraped page at the scrape price, as before. The platform endpoint runs only when they cannot. - **You pay the platform endpoint's price, once.** That is 5 credits for a LinkedIn profile, company page or post, and 1 credit for every other URL in the table. The scrape's own charge is refunded to 0, so `credits_used` and `X-Credits-Used` show the platform price alone. Your Activity Logs list two rows: `/v1/web/scrape` at 0 credits and the platform endpoint at its price. A cached platform answer is free, as it is on a direct call. - **`data.page.native` says which endpoint answered.** It appears only on these pages. `endpoint` is the endpoint that read the page, `credits_used` is what it charged, and `data` is its full structured record, the same one a direct call returns. `data._warnings` carries one line with the same facts, and `data.page.page_state.value` is `content`. - **Markdown and HTML are written from the record.** For `formats=markdown`, `html` and `rawHtml`, `data.page.content` holds a readable rendering of the profile or post, built from the record rather than taken from the site's HTML. Other formats (`screenshot`, `json`, `links`, `summary`) and `pdf_parse=true` keep the scrapers' answer, because the record has no screenshot, link list or page text to give back. `only_main_content`, `include_tags`, `exclude_tags`, `wait_for`, `mobile` and `location_country` have no effect on a rendered record. - **Only the URL shapes in the table are read this way.** Other pages on these sites, such as LinkedIn job pages, YouTube RSS feeds and Reddit search pages, stay with the scrapers. - **For the structured record on every call, use the platform endpoint directly.** It returns the same record with every field documented, and it skips the scrape attempt. The usual chain is `map` to find the pages, then `scrape` or `extract` on the ones that matter. Reach for `crawl` only when you want the site walked for you. ```bash title="cURL" # 1. What pages does this site have? curl "https://www.socialcrawl.dev/v1/web/map?url=https://example.com&search=pricing" \ -H "x-api-key: YOUR_API_KEY" # 2. Pull the one that matters curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com/pricing&formats=markdown" \ -H "x-api-key: YOUR_API_KEY" ``` ## Async jobs: submit, poll, settle `crawl`, `batch-scrape`, and `agent` are too slow to answer inline, so they return a job id and you poll one unified surface. All three ids are prefixed `job_`, and all three read through the same four routes. There is no per-kind status path. | Endpoint | Credits | What it does | Key parameters | | ------------------------------ | ------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | | `POST /v1/web/crawl` | 1 per page crawled | Walks a site from a root URL and scrapes what it finds | `url`, `limit`, `max_depth`, `allow_backward_links`, `allow_external_links`, `include_paths`, `exclude_paths`, `formats`, `webhook_url` | | `POST /v1/web/batch-scrape` | 1 per URL submitted | Scrapes a `urls` array you already know | `urls`, `ignore_invalid_urls`, `formats`, `only_main_content`, `proxy`, `webhook_url` | | `POST /v1/web/agent` | Holds 25, floor 5 | Drives a browser toward an answer from a plain-language `prompt` | `url`, `prompt`, `model` | | `GET /v1/web/jobs` | 0 | Lists this key's jobs, cursor-paginated | `limit`, `cursor` | | `GET /v1/web/jobs/{job_id}` | 0 | Status, progress, credits held and charged, and the result once terminal | `job_id` | | `DELETE /v1/web/jobs/{job_id}` | 0 | Cancels the job and refunds the unspent hold | `job_id` | `limit` caps pages on a crawl and `max_depth` caps how far from the root it goes, with `include_paths` / `exclude_paths` keeping it in the section you care about. `ignore_invalid_urls` keeps one bad entry from rejecting a whole batch submission. Use `agent` when the data is behind clicking and navigating rather than at a URL you can name. ```bash title="cURL" # 1. Submit, returns data.job_id, e.g. "job_9f3k2n8d1" curl -X POST "https://www.socialcrawl.dev/v1/web/crawl" \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"url":"https://example.com","limit":25,"max_depth":2}' # 2. Poll until data.status is terminal curl "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \ -H "x-api-key: YOUR_API_KEY" # 3. Lost the id? List everything this key has running curl "https://www.socialcrawl.dev/v1/web/jobs?limit=20" \ -H "x-api-key: YOUR_API_KEY" # 4. Stop it early and get the unspent hold back curl -X DELETE "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \ -H "x-api-key: YOUR_API_KEY" ``` ### How async jobs are billed **Credits are held, then settled.** A crawl holds 1 credit per page of `limit` (so `limit=25` holds 25) and charges only for the pages actually fetched. The default `limit` is 10, and the maximum is 10,000. A batch scrape holds one credit per URL you submit. An agent job holds a flat 25 and settles against the work it did, with a 5-credit floor. Whatever is held and not spent comes back automatically, including on cancel. Reading and cancelling a job are free. `POST /v1/web/crawl` and `POST /v1/web/batch-scrape` also accept a `webhook_url` if you would rather be told than poll. That call carries no SocialCrawl signature, so read the job back with your key before you act on it; see [Web jobs and web monitors](/docs/webhooks-and-events.md#web-jobs-and-web-monitors). **Per-page failures.** Append `/errors` to a job read (`GET /v1/web/jobs/{job_id}`, then the same path with `/errors` on the end) to list the individual pages that failed inside an otherwise successful job. That sub-route is a convenience mount rather than a registry entry, so it does not appear in the roster below or in the machine-readable contracts. It works, but treat it as unversioned: do not generate an SDK method for it. ## Monitors: watch a page on a schedule A monitor re-runs a scrape or a search on a cadence and records every result, so you can ask what changed rather than diffing it yourself. | Endpoint | Credits | What it does | Key parameters | | ------------------------------------------ | ----------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /v1/web/monitors` | 0 to create | Creates a monitor | `url`, `name`, `mode`, `cadence_minutes`, `schedule_text`, `schedule_cron`, `timezone`, `query`, `goal`, `retention_days`, `judge_enabled`, `webhook_url` | | `GET /v1/web/monitors` | 0 | Lists them, cursor-paginated | `cursor`, `limit` | | `GET /v1/web/monitors/{monitor_id}` | 0 | One monitor's settings plus `last_check_at` and `next_check_at` | `monitor_id` | | `GET /v1/web/monitors/{monitor_id}/checks` | 0 | The check history: the results, not the configuration | `monitor_id`, `limit` | | `PATCH /v1/web/monitors/{monitor_id}` | 0 | Changes `status` (pause/resume) or the schedule, keeping the check history | `monitor_id`, `status`, `cadence_minutes`, `schedule_text`, `schedule_cron`, `timezone` | | `DELETE /v1/web/monitors/{monitor_id}` | 0 | Stops future checks; past checks stay readable | `monitor_id` | ```bash title="cURL" # Create one, returns data.monitor_id, e.g. "wm_5d1p8s3k7" curl -X POST "https://www.socialcrawl.dev/v1/web/monitors" \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"url":"https://example.com/pricing","name":"Pricing page","mode":"scrape","cadence_minutes":15}' # Read what it has found curl "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7/checks?limit=20" \ -H "x-api-key: YOUR_API_KEY" # Pause it without losing history curl -X PATCH "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7" \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"status":"paused"}' ``` - `mode` is `scrape` (watch a page) or `search` (watch a query, via `query`). Schedule it with `cadence_minutes`, `schedule_text` ("every 15 minutes"), or `schedule_cron`. The last two are mutually exclusive. `retention_days` controls how long checks are kept, `webhook_url` pushes each result, and `judge_enabled` turns on the change assessment against `goal`. - **`cadence_minutes` is 5-60, or a whole number of hours up to 1440** (120, 180, and so on). Hour cadences are scheduled as cron under the hood. Values between those two bands are rejected. - **Creating, listing, reading, updating, and deleting a monitor are all free.** You pay when a check runs: the underlying scrape or search cost plus 1 credit of orchestration. A paused monitor bills nothing. - `GET /v1/web/monitors/{monitor_id}/checks` is the path a webhook's `run_id` refers back to. ## Browser sessions: drive a live page When content only appears after clicking, typing, or logging in, open a session and send it instructions. | Endpoint | Credits | What it does | Key parameters | | -------------------------------------------- | -------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------- | | `POST /v1/web/sessions` | 20 per browser-hour, min 5 | Opens a short-lived browser at `url` and returns a `ws_`-prefixed id | `url`, `ttl_seconds`, `activity_ttl_seconds`, `stream_web_view` | | `POST /v1/web/sessions/{session_id}/execute` | 0 | Runs code inside the session | `session_id`, `code`, `language`, `timeout` | | `GET /v1/web/sessions` | 0 | What is still open, cursor-paginated | `limit`, `cursor` | | `GET /v1/web/sessions/{session_id}` | 0 | One session and when it expires | `session_id` | | `DELETE /v1/web/sessions/{session_id}` | 0 | Closes it and settles the hold | `session_id` | ```bash title="cURL" # 1. Open, returns data.session_id, e.g. "ws_3g7x1v5m2" curl -X POST "https://www.socialcrawl.dev/v1/web/sessions" \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"url":"https://example.com","ttl_seconds":120}' # 2. Drive it curl -X POST "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2/execute" \ -H "x-api-key: YOUR_API_KEY" \ -H "content-type: application/json" \ -d '{"code":"await page.click(\"#load-more\"); console.log(await page.title());","language":"node"}' # 3. Close it curl -X DELETE "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2" \ -H "x-api-key: YOUR_API_KEY" ``` - **A session holds 20 credits per browser-hour of `ttl_seconds`, with a minimum of 5.** `ttl_seconds` defaults to 60 and clamps to 30-3600, so a two-minute session holds the 5-credit floor and the one-hour maximum holds 20. Close it as soon as you are done rather than waiting for expiry. The unused hold refunds on close. - `ttl_seconds` sets the lifetime and `activity_ttl_seconds` the idle timeout. `stream_web_view=true` gives you a viewable stream. - `language` on `execute` is `node`, `python`, or `bash`. With `node` you get the Playwright `page`, `browser`, and `chromium` globals and the final expression lands in `result`. `bash` and `python` report through `stdout`. The response carries `success`, `stdout`, `stderr`, `exitCode`, and a `killed` flag. Executing is free: you already paid for the session. ## Document parse | Endpoint | Credits | What it returns | Key parameters | | -------------------- | ------- | ------------------------------------------------------------------------------------ | -------------------------------------------------- | | `POST /v1/web/parse` | 1 | The same `data.page` shape as a scrape, with `page_count` and `total_page_count` set | `file` (multipart), `filename`, `mime_type`, `url` | PDFs and office documents both work. Use it when you hold the file itself. For a document sitting at a URL, `scrape` with `pdf_parse=true` gets there in one call. ```bash title="cURL" curl -X POST "https://www.socialcrawl.dev/v1/web/parse" \ -H "x-api-key: YOUR_API_KEY" \ -F "file=@report.pdf" \ -F "filename=report.pdf" ``` ## All endpoints ## Platform notes - **Method-aware surface.** The sync data endpoints (`scrape`, `search`, `map`, `extract`) and all reads are `GET`. Async jobs, monitors, sessions, and parse use `POST`/`PATCH`/`DELETE` as listed above. - **Variable-cost work is metered honestly.** Crawl pages, PDF parsing, enhanced proxy, agent runs, and session time are held, settled to actual usage, and refunded when unused. - **Social platform URLs fall back to their own endpoints.** When the scrapers cannot open a LinkedIn, Instagram, Facebook, Reddit, TikTok, YouTube, X or Threads profile or post, `/v1/web/scrape` reads it through that platform's endpoint at that endpoint's price (5 credits for LinkedIn, 1 for the others) and puts the record under `data.page.native`. See [Social platform URLs](#social-platform-urls). - **Some sites cannot be scraped.** When `/v1/web/scrape` cannot fetch a site at all, it returns `404 RESOURCE_NOT_FOUND` with `error.details.reason: "site_not_supported"` and `retry_will_succeed: false`, and charges nothing. Retrying returns the same answer. When the URL is on a platform SocialCrawl reads directly, `error.details.suggestion` names the endpoint to use instead, for example `/v1/instagram/profile` or `/v1/reddit/post`. - **A page that is not the page is not charged.** When the site answers `/v1/web/scrape` with its own error or block page (HTTP 400 or more), a CAPTCHA, a sign-in wall, a rate-limit notice, or an empty app shell, you still get the page back, with `data.page.status_code` and `data.page.page_state` saying what it is, `credits_used: 0`, and one line in `data._warnings` with what to try next. A desktop Naver cafe article link loads the article in a frame that cannot be scraped, so it comes back as `page_state.value: "empty_shell"`, free; use `/v1/naver/cafearticle/search` for the article's title and excerpt. - **`proxy=auto` and `proxy=enhanced` read the page live.** They skip the cached copy unless you set `max_age` yourself, so a retry with a stronger proxy never gets the block page it is retrying. A cached copy of a failed page is never served on a plain scrape either. - **A markdown scrape can be served by a second extractor.** When the primary scraper cannot fetch a page, a markdown-only request is retried on a second extractor at the same 1-credit price. That extractor returns the whole page, so `only_main_content` does not apply to it. - **Job ids are `job_…`, monitor ids `wm_…`, session ids `ws_…`.** All three are scoped to the API key that created them, so listing only ever shows your own. - **`POST /v1/web/crawl` and `POST /v1/web/batch-scrape` honour an `Idempotency-Key` header.** Resubmitting the same key returns the original job rather than starting a second one. The other web `POST` calls ignore the key; see [Idempotency](/docs/idempotency.md). - Content responses follow the unified [WebPage schema](/docs/schema/web-page.md). Job, monitor, and session lists return the top-level `pagination` block. Send its opaque `next_cursor` back verbatim as `cursor`; the public token preserves the database keyset, and a malformed or stale token safely restarts at the first page. ## Next steps - [Job schema](/docs/schema/job.md): The status shape every async job and its result come back in. - [Webhooks and events](/docs/webhooks-and-events.md): Every way to be told when a job settles or a check runs. - [Credits](/docs/credits.md): Holds, settlements, and automatic refunds.