Web Scraping

Scrape, search, map, extract, crawl, and monitor any public web page. Content-bearing responses return the unified WebPage schema: clean markdown or HTML, metadata, structured extraction, and fetch provenance under data.page, while search and map return a WebPageList under data.items[].

This is the only platform on the API that sells stateful work: long-running jobs you poll, monitors that keep checking on a schedule, and browser sessions you drive step by step. Everything else here is one request, one answer.

Base URL: /v1/web/...

Quickstart

curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com" \
  -H "x-api-key: YOUR_API_KEY"

curl "https://www.socialcrawl.dev/v1/web/extract?url=https://example.com&prompt=the%20page%20title%20and%20any%20pricing%20mentioned" \
  -H "x-api-key: YOUR_API_KEY"

Synchronous endpoints

These four answer immediately and cover most integrations.

scrape holds 5 credits when you ask for work the cheap path may not be able to do (proxy=auto, proxy=enhanced, or pdf_parse=true) and settles to what the fetch actually used, so a page a plain request could read still costs 1. The data.page.fetch.proxy_tier field tells you which path actually ran.

search is 2 credits per 10 results, so limit=50 is 10 credits. Adding include_content=true scrapes each result inline and adds 1 credit per result on top, which is how a limit=50 content search reaches 60 credits. Filter with include_domains / exclude_domains, country, time_range, and categories before you widen limit.

Social platform URLs

LinkedIn, Reddit and Threads block page scrapers, and so do most Facebook and Instagram pages, so a scrape of a profile or post on those sites usually fails. When /v1/web/scrape gets one of the URLs below and the scrapers cannot open it, SocialCrawl reads the page through that platform's own endpoint and returns the result in the usual data.page shape.

A LinkedIn profile asked for as HTML, which the scrapers cannot open:

curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://de.linkedin.com/in/williamhgates&formats=html" \
  -H "x-api-key: YOUR_API_KEY"

{
  "success": true,
  "credits_used": 5,
  "data": {
    "page": {
      "url": "https://de.linkedin.com/in/williamhgates",
      "final_url": "https://www.linkedin.com/in/williamhgates/",
      "title": "Bill Gates (@williamhgates)",
      "content": {
        "markdown": null,
        "html": "<article><h1>Bill Gates (@williamhgates)</h1><p>Chair, Gates Foundation and Founder, Breakthrough Energy</p><ul><li>Followers: 40,768,450</li><li>Following: 8</li></ul></article>"
      },
      "page_state": { "value": "content", "confidence": 1, "skipped_judge": true },
      "native": {
        "endpoint": "/v1/linkedin/profile",
        "credits_used": 5,
        "data": {
          "author": {
            "username": "williamhgates",
            "display_name": "Bill Gates",
            "followers": 40768450,
            "url": "https://www.linkedin.com/in/williamhgates/"
          }
        }
      }
    },
    "_warnings": [
      "The page scrapers could not open this page, so it was read through /v1/linkedin/profile and charged at that endpoint's price (5 credits) instead of the scrape price. That call is its own row in your Activity Logs. The full record is in page.native.data."
    ]
  }
}

How it works and how it is billed:

• The scrapers go first. If they can read the page, you get the scraped page at the scrape price, as before. The platform endpoint runs only when they cannot.
• You pay the platform endpoint's price, once. That is 5 credits for a LinkedIn profile, company page or post, and 1 credit for every other URL in the table. The scrape's own charge is refunded to 0, so credits_used and X-Credits-Used show the platform price alone. Your Activity Logs list two rows: /v1/web/scrape at 0 credits and the platform endpoint at its price. A cached platform answer is free, as it is on a direct call.
• data.page.native says which endpoint answered. It appears only on these pages. endpoint is the endpoint that read the page, credits_used is what it charged, and data is its full structured record, the same one a direct call returns. data._warnings carries one line with the same facts, and data.page.page_state.value is content.
• Markdown and HTML are written from the record. For formats=markdown, html and rawHtml, data.page.content holds a readable rendering of the profile or post, built from the record rather than taken from the site's HTML. Other formats (screenshot, json, links, summary) and pdf_parse=true keep the scrapers' answer, because the record has no screenshot, link list or page text to give back. only_main_content, include_tags, exclude_tags, wait_for, mobile and location_country have no effect on a rendered record.
• Only the URL shapes in the table are read this way. Other pages on these sites, such as LinkedIn job pages, YouTube RSS feeds and Reddit search pages, stay with the scrapers.
• For the structured record on every call, use the platform endpoint directly. It returns the same record with every field documented, and it skips the scrape attempt.

The usual chain is map to find the pages, then scrape or extract on the ones that matter. Reach for crawl only when you want the site walked for you.

# 1. What pages does this site have?
curl "https://www.socialcrawl.dev/v1/web/map?url=https://example.com&search=pricing" \
  -H "x-api-key: YOUR_API_KEY"

# 2. Pull the one that matters
curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com/pricing&formats=markdown" \
  -H "x-api-key: YOUR_API_KEY"

Async jobs: submit, poll, settle

crawl, batch-scrape, and agent are too slow to answer inline, so they return a job id and you poll one unified surface. All three ids are prefixed job_, and all three read through the same four routes. There is no per-kind status path.

limit caps pages on a crawl and max_depth caps how far from the root it goes, with include_paths / exclude_paths keeping it in the section you care about. ignore_invalid_urls keeps one bad entry from rejecting a whole batch submission. Use agent when the data is behind clicking and navigating rather than at a URL you can name.

# 1. Submit, returns data.job_id, e.g. "job_9f3k2n8d1"
curl -X POST "https://www.socialcrawl.dev/v1/web/crawl" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","limit":25,"max_depth":2}'

# 2. Poll until data.status is terminal
curl "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \
  -H "x-api-key: YOUR_API_KEY"

# 3. Lost the id? List everything this key has running
curl "https://www.socialcrawl.dev/v1/web/jobs?limit=20" \
  -H "x-api-key: YOUR_API_KEY"

# 4. Stop it early and get the unspent hold back
curl -X DELETE "https://www.socialcrawl.dev/v1/web/jobs/job_9f3k2n8d1" \
  -H "x-api-key: YOUR_API_KEY"

How async jobs are billed

Credits are held, then settled. A crawl holds 1 credit per page of limit (so limit=25 holds 25) and charges only for the pages actually fetched. The default limit is 10, and the maximum is 10,000. A batch scrape holds one credit per URL you submit. An agent job holds a flat 25 and settles against the work it did, with a 5-credit floor. Whatever is held and not spent comes back automatically, including on cancel.

Reading and cancelling a job are free. POST /v1/web/crawl and POST /v1/web/batch-scrape also accept a webhook_url if you would rather be told than poll. That call carries no SocialCrawl signature, so read the job back with your key before you act on it; see Web jobs and web monitors.

Per-page failures. Append /errors to a job read (GET /v1/web/jobs/{job_id}, then the same path with /errors on the end) to list the individual pages that failed inside an otherwise successful job. That sub-route is a convenience mount rather than a registry entry, so it does not appear in the roster below or in the machine-readable contracts. It works, but treat it as unversioned: do not generate an SDK method for it.

Monitors: watch a page on a schedule

A monitor re-runs a scrape or a search on a cadence and records every result, so you can ask what changed rather than diffing it yourself.

# Create one, returns data.monitor_id, e.g. "wm_5d1p8s3k7"
curl -X POST "https://www.socialcrawl.dev/v1/web/monitors" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com/pricing","name":"Pricing page","mode":"scrape","cadence_minutes":15}'

# Read what it has found
curl "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7/checks?limit=20" \
  -H "x-api-key: YOUR_API_KEY"

# Pause it without losing history
curl -X PATCH "https://www.socialcrawl.dev/v1/web/monitors/wm_5d1p8s3k7" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"status":"paused"}'

• mode is scrape (watch a page) or search (watch a query, via query). Schedule it with cadence_minutes, schedule_text ("every 15 minutes"), or schedule_cron. The last two are mutually exclusive. retention_days controls how long checks are kept, webhook_url pushes each result, and judge_enabled turns on the change assessment against goal.
• cadence_minutes is 5-60, or a whole number of hours up to 1440 (120, 180, and so on). Hour cadences are scheduled as cron under the hood. Values between those two bands are rejected.
• Creating, listing, reading, updating, and deleting a monitor are all free. You pay when a check runs: the underlying scrape or search cost plus 1 credit of orchestration. A paused monitor bills nothing.
• GET /v1/web/monitors/{monitor_id}/checks is the path a webhook's run_id refers back to.

Browser sessions: drive a live page

When content only appears after clicking, typing, or logging in, open a session and send it instructions.

# 1. Open, returns data.session_id, e.g. "ws_3g7x1v5m2"
curl -X POST "https://www.socialcrawl.dev/v1/web/sessions" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","ttl_seconds":120}'

# 2. Drive it
curl -X POST "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2/execute" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{"code":"await page.click(\"#load-more\"); console.log(await page.title());","language":"node"}'

# 3. Close it
curl -X DELETE "https://www.socialcrawl.dev/v1/web/sessions/ws_3g7x1v5m2" \
  -H "x-api-key: YOUR_API_KEY"

• A session holds 20 credits per browser-hour of ttl_seconds, with a minimum of 5. ttl_seconds defaults to 60 and clamps to 30-3600, so a two-minute session holds the 5-credit floor and the one-hour maximum holds 20. Close it as soon as you are done rather than waiting for expiry. The unused hold refunds on close.
• ttl_seconds sets the lifetime and activity_ttl_seconds the idle timeout. stream_web_view=true gives you a viewable stream.
• language on execute is node, python, or bash. With node you get the Playwright page, browser, and chromium globals and the final expression lands in result. bash and python report through stdout. The response carries success, stdout, stderr, exitCode, and a killed flag. Executing is free: you already paid for the session.

Document parse

PDFs and office documents both work. Use it when you hold the file itself. For a document sitting at a URL, scrape with pdf_parse=true gets there in one call.

curl -X POST "https://www.socialcrawl.dev/v1/web/parse" \
  -H "x-api-key: YOUR_API_KEY" \
  -F "file=@report.pdf" \
  -F "filename=report.pdf"

All endpoints

22 endpoints available.

Platform notes

• Method-aware surface. The sync data endpoints (scrape, search, map, extract) and all reads are GET. Async jobs, monitors, sessions, and parse use POST/PATCH/DELETE as listed above.
• Variable-cost work is metered honestly. Crawl pages, PDF parsing, enhanced proxy, agent runs, and session time are held, settled to actual usage, and refunded when unused.
• Social platform URLs fall back to their own endpoints. When the scrapers cannot open a LinkedIn, Instagram, Facebook, Reddit, TikTok, YouTube, X or Threads profile or post, /v1/web/scrape reads it through that platform's endpoint at that endpoint's price (5 credits for LinkedIn, 1 for the others) and puts the record under data.page.native. See Social platform URLs.
• Some sites cannot be scraped. When /v1/web/scrape cannot fetch a site at all, it returns 404 RESOURCE_NOT_FOUND with error.details.reason: "site_not_supported" and retry_will_succeed: false, and charges nothing. Retrying returns the same answer. When the URL is on a platform SocialCrawl reads directly, error.details.suggestion names the endpoint to use instead, for example /v1/instagram/profile or /v1/reddit/post.
• A page that is not the page is not charged. When the site answers /v1/web/scrape with its own error or block page (HTTP 400 or more), a CAPTCHA, a sign-in wall, a rate-limit notice, or an empty app shell, you still get the page back, with data.page.status_code and data.page.page_state saying what it is, credits_used: 0, and one line in data._warnings with what to try next. A desktop Naver cafe article link loads the article in a frame that cannot be scraped, so it comes back as page_state.value: "empty_shell", free; use /v1/naver/cafearticle/search for the article's title and excerpt.
• proxy=auto and proxy=enhanced read the page live. They skip the cached copy unless you set max_age yourself, so a retry with a stronger proxy never gets the block page it is retrying. A cached copy of a failed page is never served on a plain scrape either.
• A markdown scrape can be served by a second extractor. When the primary scraper cannot fetch a page, a markdown-only request is retried on a second extractor at the same 1-credit price. That extractor returns the whole page, so only_main_content does not apply to it.
• Job ids are job_…, monitor ids wm_…, session ids ws_…. All three are scoped to the API key that created them, so listing only ever shows your own.
• POST /v1/web/crawl and POST /v1/web/batch-scrape honour an Idempotency-Key header. Resubmitting the same key returns the original job rather than starting a second one. The other web POST calls ignore the key; see Idempotency.
• Content responses follow the unified WebPage schema. Job, monitor, and session lists return the top-level pagination block. Send its opaque next_cursor back verbatim as cursor; the public token preserves the database keyset, and a malformed or stale token safely restarts at the first page.

Next steps