# SocialCrawl API: web endpoints
# Base URL: https://www.socialcrawl.dev
# Auth: x-api-key header
# Full docs: https://www.socialcrawl.dev/docs/web

## GET /v1/web/scrape
Scrape a web page
Credit cost: 1-5 (metered): 1 credit for a standard scrape. `proxy=auto`, `proxy=enhanced` and `pdf_parse=true` hold 5 credits and settle to what the fetch actually used, so a page a plain request could read still costs 1. Nothing else moves the price: screenshots, tag filters, `wait_for` and a mobile viewport are included. A page that comes back as the site's own block or error page (HTTP 400 or more), a CAPTCHA, a sign-in wall or an empty shell is returned with `page_state` saying so and a `_warnings` line, and is not charged. When the page scrapers cannot open a LinkedIn, Instagram, Facebook, Reddit, TikTok, YouTube, X or Threads profile or post, the page is read through that platform's own endpoint and costs that endpoint's price instead: 5 credits for a LinkedIn profile, company page or post, 1 credit for the others. The scrape's own charge is refunded to 0, so you pay the platform price once, and Activity Logs show the platform call as its own row
Parameters:
- url (required): Public URL to fetch.
- formats (optional, string): Comma-separated output formats such as markdown,screenshot. A social platform URL is read through its own endpoint only for markdown, html and rawHtml.
- only_main_content (optional, boolean): Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page.
- wait_for (optional, integer): Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built.
- mobile (optional, boolean): Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones.
- timeout (optional, integer): Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture.
- max_age (optional, integer): Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours), or 0 (a live fetch) with `proxy=auto` or `proxy=enhanced`; send 0 to force a live fetch. A cached copy of a page that failed (HTTP 400 or more) is never served: it is fetched again, live.
- location_country (optional, string): ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability.
- screenshot_full_page (optional, boolean): When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport.
- include_tags (optional, string): CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container.
- exclude_tags (optional, string): CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags.
- proxy (optional, enum: basic | auto | enhanced): Proxy tier: basic (the default), auto (a plain request first, then a stronger one if the site blocks it), or enhanced. Use auto when a plain scrape returns a block or CAPTCHA page. auto and enhanced read the page live unless you set max_age.
- pdf_parse (optional, boolean): Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page.
- block_ads (optional, boolean): Block ad and tracker requests, which is faster and quieter. Default true.
- remove_base64_images (optional, boolean): Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true.
curl -G "https://www.socialcrawl.dev/v1/web/scrape" \
  --data-urlencode "url=https://example.com" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/search
Search the web
Credit cost: 2-120 (metered): 2 credits per 10 results, so `limit` up to 10 costs 2, 11-20 costs 4, and the 100 maximum costs 20. `include_content=true` adds 1 credit per result, because it scrapes each result page as well as reading the SERP row - that is what takes a 100-result content search to the 120-credit ceiling. The hold settles down to the work actually done
Parameters:
- query (required): Search query.
- sources (optional, string): Comma-separated sources: web,news,images.
- categories (optional, string): CSV of source categories to restrict the search to, e.g. 'github,research'. Omit for an unrestricted web search.
- limit (optional, integer): Results per source, from 1 to 100.
- country (optional, string): ISO 3166-1 alpha-2 country code the search is run from, e.g. 'US' (default) or 'GB'. Changes which results rank.
- location (optional, string): Free-text locality to bias results towards, e.g. 'London, United Kingdom'. Finer-grained than country.
- time_range (optional, string): Recency window as a Google `tbs` value: 'qdr:d' past day, 'qdr:w' past week, 'qdr:m' past month, 'qdr:y' past year.
- sort_by_date (optional, boolean): Sort newest-first instead of by relevance. Combines with time_range rather than replacing it.
- include_domains (optional, string): CSV of domains to restrict results to, e.g. 'nytimes.com,ft.com'. Cannot be combined with exclude_domains - sending both is a 400 before billing.
- exclude_domains (optional, string): CSV of domains to drop from the results. Cannot be combined with include_domains.
- include_content (optional, boolean): Also scrape each result and return its page content as markdown, not just the SERP row. This is what makes the call expensive: it scrapes one page per result, which is why the range runs to 120 credits.
curl -G "https://www.socialcrawl.dev/v1/web/search" \
  --data-urlencode "query=social media data api" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/map
Map URLs on a site
Credit cost: 1 (override; tier standard)
Parameters:
- url (required): Site URL to map.
- search (optional, string): Optional path or keyword filter.
- limit (optional, integer): Maximum URLs to return, up to 5000.
- sitemap (optional, enum: include | skip | only): How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it.
- include_subdomains (optional, boolean): Follow links onto subdomains of the root URL. Default true.
- ignore_query_parameters (optional, boolean): Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true.
- fresh (optional, boolean): Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale.
- goal (optional, string): Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged.
curl -G "https://www.socialcrawl.dev/v1/web/map" \
  --data-urlencode "url=https://example.com" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/extract
Extract structured data from a web page
Credit cost: 5 (metered): A flat 5 credits. It runs through the metered pricer for consistency with the rest of the web surface, but nothing in your query changes the charge
Parameters:
- url (required): Public URL to extract from.
- schema (optional, string): JSON object schema describing the data to return.
- prompt (optional, string): Plain-language extraction instruction.
- only_main_content (optional, boolean): Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome.
- timeout (optional, integer): Hard cap on each page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture.
- max_age (optional, integer): Accept an upstream-cached copy up to this many milliseconds old, which is faster and costs the same. Default 172800000 (48 hours); send 0 to force a live fetch.
- proxy (optional, enum: basic | auto | enhanced): Proxy tier: basic, auto, or enhanced. Raise it only when a site blocks the default tier - enhanced is slower.
- Constraint: one of schema, prompt (at least one required)
curl -G "https://www.socialcrawl.dev/v1/web/extract" \
  --data-urlencode "url=https://example.com/pricing" \
  --data-urlencode "prompt=Extract plan names and prices." \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/crawl
Start an async web crawl
Credit cost: 1-10000 (metered): 1 credit per page crawled. Submitting holds `limit` credits up front (default limit 10, max 10,000) and the unused portion is refunded when the job settles
Parameters:
- url (required): Root URL to crawl.
- limit (optional, integer): Maximum pages to crawl.
- max_depth (optional, integer): How many link hops from the root URL to follow. 1 is the root page and the pages it links to. Leave unset to let `limit` alone bound the crawl.
- allow_backward_links (optional, boolean): Crawl the whole domain rather than only URLs under the starting path. Send it when the root URL is a deep page but you want the entire site.
- allow_external_links (optional, boolean): Follow links off the domain. Off by default, and easy to make expensive: `limit` is the only thing bounding it.
- include_paths (optional, string): CSV of URL path patterns to crawl, e.g. '/blog/.*,/docs/.*'. Everything else is skipped, which is the cheapest way to narrow a crawl.
- exclude_paths (optional, string): CSV of URL path patterns to skip, e.g. '/tag/.*'. Applied after include_paths.
- webhook_url (optional, string): Optional URL the crawl backend calls directly with its own progress events. SocialCrawl does not sign or filter that callback, and its payload carries the backend's own job id, not your job_... id; poll GET /v1/web/jobs/{job_id} for the authoritative status.
- formats (optional, string): CSV of output formats for each crawled page, e.g. 'markdown' (default) or 'markdown,html'.
- goal (optional, string): Optional. What you are looking for on the site, in your own words. SocialCrawl maps the site, scrapes the URLs that likely hold it (always at least two), and lists every skipped URL with p. Without this param the crawl follows links by depth only.
curl -X POST "https://www.socialcrawl.dev/v1/web/crawl" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'

## POST /v1/web/batch-scrape
Start an async batch scrape
Credit cost: 1 credit per URL submitted, held up front and refunded down to the URLs actually scraped when the job settles
Parameters:
- urls (required): Array or comma-separated list of public URLs to scrape.
- ignore_invalid_urls (optional, boolean): Skip invalid URLs instead of failing the job.
- formats (optional, string): CSV of output formats for each URL, e.g. 'markdown' (default) or 'markdown,html'.
- only_main_content (optional, boolean): Strip nav, headers, footers and sidebars on every URL and keep the article body. Default true.
- proxy (optional, enum: basic | auto | enhanced): Proxy tier applied to every URL: basic, auto, or enhanced.
- webhook_url (optional, string): Accepted but not used: a batch-scrape job sends no webhook. Poll GET /v1/web/jobs/{job_id} for its status.
curl -X POST "https://www.socialcrawl.dev/v1/web/batch-scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"urls":["https://example.com/a","https://example.com/b"]}'

## GET /v1/web/jobs
List async web jobs
Credit cost: 0 (override; tier standard)
Parameters:
- limit (optional, integer): Page size, capped at 100.
- cursor (optional, string): Previous response cursor.
curl "https://www.socialcrawl.dev/v1/web/jobs" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/jobs/{job_id}
Get an async web job
Credit cost: 0 (override; tier standard)
Parameters:
- job_id (required): Job id (job_...) returned by POST /v1/web/crawl, POST /v1/web/batch-scrape, or POST /v1/web/agent as data.job_id.
curl "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \
  -H "x-api-key: YOUR_API_KEY"

## DELETE /v1/web/jobs/{job_id}
Cancel an async web job
Credit cost: 0 (override; tier standard)
Parameters:
- job_id (required): Job id (job_...) returned by POST /v1/web/crawl, POST /v1/web/batch-scrape, or POST /v1/web/agent as data.job_id.
curl -X DELETE "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/agent
Start an async web agent job
Credit cost: 25 (override; tier premium)
Parameters:
- url (required): Starting URL folded into the agent instruction.
- prompt (required): Task instruction.
- model (optional, enum: spark-1-mini | spark-1-pro): Agent model: spark-1-mini or spark-1-pro.
curl -X POST "https://www.socialcrawl.dev/v1/web/agent" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","prompt":"Find the pricing page."}'

## POST /v1/web/monitors
Create a web monitor
Credit cost: 0 credits to create. Each scheduled check then bills its upstream cost plus 1 credit for orchestration, repeating on `cadence_minutes` until you pause or delete the monitor (a 5-minute cadence is 288 checks a day)
Parameters:
- url (required): URL or root target to monitor.
- name (optional, string): Human-readable monitor name.
- mode (optional, enum: scrape | search): Monitor target type.
- cadence_minutes (optional, integer): Check interval in minutes: 5-60, or a whole number of hours up to 1440 (120, 180, ...). Hour cadences are scheduled as cron.
- schedule_text (optional, string): Plain-language schedule, such as every 15 minutes.
- schedule_cron (optional, string): Cron schedule. Mutually exclusive with schedule_text.
- timezone (optional, string): IANA timezone for the schedule.
- webhook_url (optional, string): Optional URL the monitoring backend calls directly with its own check payloads. SocialCrawl does not sign or deliver that callback, and the ids in it are the backend's own, not your wm_... id; read GET /v1/web/monitors/{monitor_id}/checks for the authoritative record.
- query (optional, string): Search query the monitor re-runs, used only when mode=search. On a scrape monitor the url is the target instead.
- goal (optional, string): Plain-language description of what a meaningful change looks like, used by the judge to decide whether a check is worth notifying about. A search monitor needs this unless judge_enabled is false.
- retention_days (optional, integer): How long to keep each check's captured result, 1 to 365 days. Older checks are dropped.
- judge_enabled (optional, boolean): Run the LLM judge over each check so you are notified on meaningful changes rather than every byte-level diff. Defaults on, except on a search monitor with no goal.
curl -X POST "https://www.socialcrawl.dev/v1/web/monitors" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'

## GET /v1/web/monitors
List web monitors
Credit cost: 0 (override; tier standard)
Parameters:
- cursor (optional, string): Pagination cursor.
- limit (optional, integer): Page size.
curl "https://www.socialcrawl.dev/v1/web/monitors" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/monitors/{monitor_id}
Get a web monitor
Credit cost: 0 (override; tier standard)
Parameters:
- monitor_id (required): Monitor id (wm_...) returned by POST /v1/web/monitors as data.monitor_id.
curl "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7" \
  -H "x-api-key: YOUR_API_KEY"

## PATCH /v1/web/monitors/{monitor_id}
Update a web monitor
Credit cost: 0 (override; tier standard)
Parameters:
- monitor_id (required): Monitor id (wm_...) returned by POST /v1/web/monitors as data.monitor_id.
- status (optional, enum: active | paused): Set to paused to stop the schedule without losing history, or active to resume it.
- cadence_minutes (optional, integer): New check interval in minutes: 5-60, or a whole number of hours up to 1440 (120, 180, ...). Hour cadences are scheduled as cron.
- schedule_text (optional, string): New plain-language schedule, such as every 15 minutes. Upstream accepts 1-60 minutes in this form; use schedule_cron for anything longer.
- schedule_cron (optional, string): New cron schedule. Mutually exclusive with schedule_text.
- timezone (optional, string): IANA timezone the new schedule is interpreted in, e.g. 'Europe/London'. Defaults to UTC.
curl -X PATCH "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"status":"paused"}'

## DELETE /v1/web/monitors/{monitor_id}
Delete a web monitor
Credit cost: 0 (override; tier standard)
Parameters:
- monitor_id (required): Monitor id (wm_...) returned by POST /v1/web/monitors as data.monitor_id.
curl -X DELETE "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/monitors/{monitor_id}/checks
List web monitor checks
Credit cost: 0 (override; tier standard)
Parameters:
- monitor_id (required): Monitor id (wm_...) returned by POST /v1/web/monitors as data.monitor_id.
- limit (optional, integer): How many checks to return, newest first. Page size, capped at 100.
curl "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7/checks" \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/sessions
Create an interactive web session
Credit cost: 20 credits per browser-hour, minimum 5. The hold is taken from `ttl_seconds` when the session is created (60s default = 5, the 3,600s maximum = 20) and settled when it closes
Parameters:
- url (required): URL to open in the session after creation.
- ttl_seconds (optional, integer): Session TTL in seconds.
- activity_ttl_seconds (optional, integer): Idle timeout in seconds: the session closes this long after the last interaction, even if ttl_seconds has not elapsed. Closing early settles the hold down, so it lowers the bill.
- stream_web_view (optional, boolean): Return a live view URL in the response so a human can watch the session drive the page.
curl -X POST "https://www.socialcrawl.dev/v1/web/sessions" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'

## GET /v1/web/sessions
List interactive web sessions
Credit cost: 0 (override; tier standard)
Parameters:
- limit (optional, integer): Page size.
curl "https://www.socialcrawl.dev/v1/web/sessions" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/sessions/{session_id}
Get an interactive web session
Credit cost: 0 (override; tier standard)
Parameters:
- session_id (required): Session id (ws_...) returned by POST /v1/web/sessions as data.session_id.
curl "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2" \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/sessions/{session_id}/execute
Execute an interaction in a web session
Credit cost: 0 (override; tier standard)
Parameters:
- session_id (required): SocialCrawl session id.
- code (required): Code to execute in the browser sandbox.
- language (optional, enum: node | python | bash): Execution language: node, python, or bash.
- timeout (optional, integer): Execution timeout in seconds, from 1 to 40 (default 30). Larger values are capped at 40 so the call always returns inside the request window; a script still running at the cap returns a 504 UPSTREAM_ERROR envelope. Split longer work across several calls to the same session.
curl -X POST "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2/execute" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"code":"await page.goto(\"https://example.com\"); console.log(await page.title());"}'

## DELETE /v1/web/sessions/{session_id}
Close an interactive web session
Credit cost: 0 (override; tier standard)
Parameters:
- session_id (required): Session id (ws_...) returned by POST /v1/web/sessions as data.session_id.
curl -X DELETE "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2" \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/parse
Parse an uploaded document
Credit cost: 1 (override; tier standard)
Parameters:
- file (required): Multipart file field.
- filename (optional, string): Name to record for the uploaded file, e.g. 'report.pdf'. Used for the parsed document's title when the file carries none.
- mime_type (optional, string): Optional MIME type override.
- url (optional, string): Public URL to fetch and parse instead of uploading a file. Send either this or the file field.
curl -X POST "https://www.socialcrawl.dev/v1/web/parse" \
  -H "x-api-key: YOUR_API_KEY" \
  -F "file=@document.pdf"