# Event webhooks (/docs/webhook-endpoints) Register one HTTPS endpoint and receive signed batches of account events, such as a discover run settling, with retries and a test send Register one HTTPS endpoint with `POST /v1/webhook-endpoints` and SocialCrawl sends it every event your API key's work produces: a discover run that completes, pauses or finds new rows on a schedule, or a listener that finds a new post. Each delivery is a signed JSON batch of events, retried for up to 41 hours until your endpoint answers `2xx`. Every webhook endpoint route is free. Monitor runs and billing events use their own webhooks; [Webhooks and events](/docs/webhooks-and-events.md) compares every channel. ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints' \ -H 'x-api-key: YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://example.com/hooks/socialcrawl", "types": ["discover.*"], "description": "discover alerts" }' ``` ```json title="Response" { "success": true, "data": { "endpoint": { "id": "we_V1StGXR8Z5jdHi6BmyT", "url": "https://example.com/hooks/socialcrawl", "description": "discover alerts", "types": ["discover.*"], "subject_ids": null, "status": "enabled", "disabled_reason": null, "failing_since": null, "api_version": "2026-10-01", "format": "evidence", "include": null, "secret_rotation": null, "created_at": "2026-10-19T10:00:00.000Z", "updated_at": "2026-10-19T10:00:00.000Z", "next": { "self_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT", "test_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test", "attempts_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/attempts", "rotate_secret_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret", "events_url": "/v1/events" } }, "secret": "whsec_...", "next_step": "Store data.secret now: it is shown once and never again. ..." }, "credits_used": 0, "request_id": "req_..." } ``` `data.secret` is the signing secret. It is shown in this answer only; SocialCrawl stores it encrypted and never returns it again. Leave `types` out to receive every event type, now and as new products start sending events. `format` (`evidence`, the default, `compact` or `text`) and `include` (`["raw"]` adds each post's original platform item) set how listener posts arrive; both are fixed when the endpoint is created, so create a new endpoint to change them. The endpoint belongs to the API key that created it: it receives the events of that key's runs and schedules and the events about the whole account, and only that key can read, change or delete it. ## Send a test ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test' \ -H 'x-api-key: YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "type": "discover.completed" }' ``` A test sends one signed sample of the type you name (default: the first type the endpoint subscribes to) and answers with what your receiver did: ```json title="Response" { "success": true, "data": { "type": "discover.completed", "attempt": { "id": "wat_...", "delivery_id": "msg_...", "test": true, "status": 200, "outcome": "delivered", "duration_ms": 142, "response_excerpt": "ok", "error": null, "created_at": "2026-10-19T10:00:01.000Z" }, "next_step": "The receiver answered 2xx. Real events are delivered the same way, in batches, signed per attempt." }, "credits_used": 0, "request_id": "req_..." } ``` The sample event carries `"test": true` beside its `type` and `"cursor": null`. A `mention.created` sample arrives in the endpoint's `format`, so the test proves your receiver parses a post. A test is sent once, never retried, never enters the event log and never changes the endpoint's health, so a test answered `410` does not disable it. ## What a delivery looks like Every delivery is an `event.batch`, also when it holds one event. A batch holds at most 100 events or 256 KB. ```json title="JSON" { "id": "bat_V1StGXR8Z5jdHi6BmyT", "type": "event.batch", "data": { "events": [ { "id": "evt_k3m7x2q7v4b6n5c5z6w2a2s4d3", "type": "discover.completed", "api_version": "2026-10-01", "created_at": "2026-10-19T10:14:52.118Z", "cursor": "c_...", "subject": { "kind": "discover_run", "id": "run_8Hq2LmNp4Rt6" }, "data": { "run_id": "run_8Hq2LmNp4Rt6", "status": "completed", "stopped_because": null, "counts": { "legs": { "planned": 12, "done": 11, "empty": 1, "failed": 0, "skipped": 0, "pending": 0 }, "rows_kept": 84 }, "credits": { "spent": 42 }, "version": 7, "schedule_id": null }, "links": { "run": "https://www.socialcrawl.dev/v1/prism/discover/runs/run_8Hq2LmNp4Rt6", "results": "https://www.socialcrawl.dev/v1/prism/discover/runs/run_8Hq2LmNp4Rt6/results.jsonl", "changes": "https://www.socialcrawl.dev/v1/prism/discover/runs/run_8Hq2LmNp4Rt6?changes_since=0" } } ] } } ``` Discover events carry counts and ids only, never your keywords, context or row text. `counts.legs` counts the run's searches by state and `counts.rows_kept` the rows it kept. Fetch the rows from the URLs in `links`; every URL there is absolute, so you can call it as it is, with your API key. Listening events carry the post they found: each is stored with ids only and filled in when the batch is frozen, in the endpoint's `format`. A post suppressed after its event is never sent; if the privacy check cannot run, only those posts wait for the next batch and every other event is sent. A post text longer than 32,768 bytes is cut and the post marked `truncated: true`; read it whole from the listener's mentions (`links.mentions`). Test vector 2 below is such an event. Each event keeps its own `id`. Deliveries are at least once and unordered, so deduplicate on each inner event `id`. Every event also carries `cursor`, its position in your account's [event feed](/docs/events.md): to fill a gap after an outage, read `GET /v1/events?cursor=`. ## Event types | Type | When | `data` | | ---- | ---- | ------ | | `discover.completed`, `discover.partial`, `discover.paused` | A background or scheduled discover run settles in that status | `run_id`, `status`, `stopped_because`, `counts`, `credits`, `version`, `schedule_id`. A `discover.paused` for `service_unavailable` or `capacity` adds `auto_resumes_left`: the automatic resumes still to come. At 0 the run stays paused until you resume it | | `discover.failed` | A background or scheduled discover run settles `failed`; a run you cancel sends nothing | `run_id`, `status`, `stopped_because`, `counts`, `credits`, `version`, `schedule_id` | | `discover.progress` | A background run passes each tenth of its planned searches | `run_id`, `status`, `progress` (`percent`, `legs_finished`, `legs_planned`), `counts`, `credits`, `version` | | `discover.accounts_ready` | A `view=accounts` run has enriched and judged its accounts | `run_id`, `accounts` (`kept`, `enriched`, `deferred`), `credits`, `version` | | `discover.new_rows`, `discover.new_accounts`, `discover.new_items` | A scheduled run settles with new results for its view | `schedule_id`, `run_id`, `count`, `ids` (the first 100), `more_ids`, `version` | | `discover.run_skipped` | A schedule occurrence is skipped | `schedule_id`, `reason`, `previous_run_id`, `scheduled_for` | | `discover.run_failed` | A scheduled run fails, or three skips in a row pause the schedule | `schedule_id`, `run_id`, `reason` | | `endpoint.disabled` | One of your webhook endpoints was disabled (below) | `endpoint_id`, `reason`, `last_error` | | `mention.created` | A listener kept a new post after its first check | `platform`, `methodology_version`, `mention` (`mention_id`, `evidence` in the endpoint's `format`, `first_seen_at`) | | `mention.deleted` | A kept post was suppressed: delete every copy you stored | `mention_id`, `evidence_id`, `reason` | | `stream.baseline.established` | A listener's first check finished on every lane | `lanes` (`platform`, `lane`, `kept`, `dropped`) | | `stream.coverage_gap` | A check stopped before it reached posts it had already seen | `platform`, `lane`, `slot`, `reason`, `pages_read`, `new_ids` | | `stream.paused` | A listener paused or stopped | `status`, `paused_because`, `resumes`, `resumes_at`, `next_step` | | `stream.resumed`, `stream.deleted` | A listener resumed or was deleted | `reason` | | `stream.budget.warning` | A listener reached 80% of its monthly credit cap | `credits_net`, `max_credits_per_month` | | `stream.health` | A lane or a listener changed health | `kind`, `platform`, and `lane`, `skipped_because` or `group` where they apply | `types` takes exact types, a namespace wildcard such as `discover.*`, or `*`. A run that answered inside the original `GET` call sends no event: you already have its result. ## How do I verify a delivery? Every attempt carries three headers, in the [Standard Webhooks](https://www.standardwebhooks.com) format: ```text title="Header" webhook-id: msg_V1StGXR8Z5jdHi6BmyT webhook-timestamp: 1792231200 webhook-signature: v1,n4OBPwOJOeyYgw9sOfwuvt4WnxtW6CX30IHU5WLqf5Y= ``` The signature is the base64 HMAC-SHA256 of `{webhook-id}.{webhook-timestamp}.{raw body}`, keyed with the base64 decoding of your secret after its `whsec_` prefix. `webhook-id` is the same on every retry of a batch; `webhook-timestamp` is fresh on each attempt. Verify over the raw body before you parse it, and reject a timestamp more than 300 seconds from your clock. **Node.js** ```js title="JavaScript" import { createHmac, timingSafeEqual } from "node:crypto"; // True if the Standard Webhooks signature over the raw body is valid and fresh. export function verify(secret, headers, rawBody, nowSeconds = Date.now() / 1000) { const id = headers["webhook-id"], ts = headers["webhook-timestamp"]; if (Math.abs(nowSeconds - Number(ts)) > 300) return false; const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64"); const expected = createHmac("sha256", key).update(`${id}.${ts}.`).update(rawBody).digest(); return headers["webhook-signature"].split(" ").some((entry) => { const [version, sig] = entry.split(","); const got = Buffer.from(sig ?? "", "base64"); return version === "v1" && got.length === expected.length && timingSafeEqual(got, expected); }); } ``` **Python** ```python title="Python" import base64, hashlib, hmac, time def verify(secret: str, headers: dict, body: bytes, now: float | None = None) -> bool: """True if the Standard Webhooks signature over the raw body is valid and fresh.""" msg_id, ts, sigs = headers["webhook-id"], headers["webhook-timestamp"], headers["webhook-signature"] if abs((now or time.time()) - int(ts)) > 300: return False key = base64.b64decode(secret.removeprefix("whsec_")) signed = f"{msg_id}.{ts}.".encode() + body expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode() for entry in sigs.split(" "): version, _, sig = entry.partition(",") if version == "v1" and hmac.compare_digest(sig, expected): return True return False ``` `webhook-signature` can hold several space-separated `v1,` entries: during a secret rotation each delivery is signed with the old secret and the new one, and a match on any entry is valid. ### Test vectors Check your verifier against these before you go live. Both use the made-up secret `whsec_bGlzdGVuaW5nLXRlc3QtdmVjdG9yLWtleS0wMDAwMDE=` and `webhook-timestamp: 1792231200` (17/10/2026 10:00:00 UTC), so pass that time as "now". The body is the exact bytes shown, with no trailing newline. | Vector | `webhook-id` | Body | Expected `webhook-signature` | | ------ | ------------ | ---- | ---------------------------- | | 1, an empty batch | `msg_test_0001` | `{"id":"bat_test_0001","type":"event.batch","data":{"events":[]}}` | `v1,n4OBPwOJOeyYgw9sOfwuvt4WnxtW6CX30IHU5WLqf5Y=` | | 2, a one-event test send | `msg_test_0002` | the line below, 664 bytes | `v1,7p7cIiickEFY7ppxKoCdaHAVxoWojbHPo4+AzKLweSY=` | ```json title="Vector 2 body" {"id":"bat_test_0002","type":"event.batch","data":{"events":[{"id":"evt_test_0001","type":"mention.created","test":true,"api_version":"2026-10-01","created_at":"2026-10-17T10:00:00Z","cursor":"c_test_0001","subject":{"kind":"listening_stream","id":"lst_test","name":"Client B: Globex Mobile"},"data":{"platform":"twitter","methodology_version":"1","mention":{"mention_id":"men_test_0001","evidence":{"id":"tw_000000000000","platform":"twitter","url":"https://x.com/example/status/1","published_at":"2026-10-17","text":"Globex Mobile is down again","author":"example","group":"outage","group_tags":{"client":"client-b"}},"first_seen_at":"2026-10-17T09:59:30Z"}}}]}} ``` Both verifiers above accept vectors 1 and 2, and reject vector 2 with one byte added to the body and vector 1 checked 301 seconds after its timestamp. ## What your receiver answers, and what SocialCrawl does | Your receiver answers | SocialCrawl | | --------------------- | ----------- | | `2xx` within 15 seconds | The delivery is done | | `401` for a bad signature or a stale timestamp | A failed attempt, retried, so a clock or secret mix-up can be fixed without losing events | | `410` | The endpoint is disabled at once (`endpoint.disabled`, reason `gone`). Answer it only to stop deliveries for good | | `429`, `502`, `503` or `504` with `Retry-After` | The next attempt waits for the later of the next retry and `Retry-After`, at most 24 hours | | Any other status, no answer within 15 seconds, or a redirect | A failed attempt, retried. Redirects are never followed | | Failed attempts for 5 days in a row, with no delivery in between | The endpoint is disabled (`endpoint.disabled`, reason `failing`). A day with no attempt at all ends the run, so one failure after a quiet week starts a new count instead | Retries follow this ladder after the first attempt: 30 seconds, 2 minutes, 10 minutes, 1 hour, 4 hours, 12 hours and 24 hours. After the eighth attempt the batch is dropped; its events stay readable from `GET /v1/events` for 30 days. Each retry sends the identical body under the same `webhook-id`. A receiver checklist: verify over the raw body before parsing, reject a timestamp more than 300 seconds from now, deduplicate on each inner event `id`, answer `2xx` within 15 seconds and do slow work afterwards. On `mention.deleted`, delete every copy of that post you stored. ## Managing endpoints | Call | What it does | | ---- | ------------ | | `GET /v1/webhook-endpoints` | The key's endpoints, newest first. `limit` 1 to 100 (default 50), `cursor`, `status` | | `GET /v1/webhook-endpoints/{id}` | One endpoint. The secret is never in it | | `PATCH /v1/webhook-endpoints/{id}` | Change `url` (checked again, and the account owner is emailed), `types`, `description`, `subject_ids` or `status` (`enabled` or `disabled`). `format` and `include` cannot change | | `DELETE /v1/webhook-endpoints/{id}` | Delete the endpoint and its delivery history | | `POST /v1/webhook-endpoints/{id}/rotate-secret` | A new secret, shown once; the old one keeps signing for 24 hours | | `POST /v1/webhook-endpoints/{id}/test` | A signed sample, sent once | | `GET /v1/webhook-endpoints/{id}/attempts` | Delivery attempts, newest first: status, outcome, duration and the first 1 KB of your receiver's answer | `subject_ids` narrows an endpoint to events about the listed runs, schedules or listeners, for example `["dsch_..."]` for one discover schedule or `["lst_..."]` for one listener (`stream_ids` is not a field; it answers `400` naming `subject_ids`). A key holds at most 20 endpoints. The account owner gets a security email whenever an endpoint is added, and whenever a `PATCH` changes an endpoint's URL, naming the old and the new host, the API key and the time. A leaked key cannot quietly point your events somewhere else. The account owner is also emailed when an endpoint is disabled, with the reason and the last error. An endpoint is disabled with a `disabled_reason`: `gone` (it answered `410`), `failing` (failed attempts for 5 days in a row), `key_revoked` (the API key that created it was revoked) or `user` (you sent `status: "disabled"`). To re-enable it, fix the receiver and send `PATCH` with `{ "status": "enabled" }`: batches held while it was disabled are sent again at once. Events created while it was disabled are not delivered; read them from `GET /v1/events`. ### Rotating the secret ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret' \ -H 'x-api-key: YOUR_API_KEY' ``` The answer carries the new `data.secret` (shown once) and `secret_rotation.previous_secret_expires_at`. Until then every delivery carries two signatures, one per secret, so deploy the new secret to your receiver within 24 hours and nothing is rejected in between. ## Which URLs are accepted? The URL must be HTTPS on port 443, with no user name or password in it, at most 2,048 characters, on a host whose every DNS address is public. Private, loopback and link-local addresses are refused, and so are SocialCrawl's own hosts. A refused URL is a free `400 INVALID_REQUEST` with `details.reason: "invalid_webhook_url"`, a `why` and a `next_step`: | `why` | Meaning | | ----- | ------- | | `not_https`, `port_not_443`, `credentials_in_url`, `invalid_url`, `url_too_long` | The URL itself | | `private_host`, `private_dns`, `dns_failed` | Where the host points | | `own_host` | A SocialCrawl host or preview deployment | | `chat_incoming_webhook` | A chat service's incoming-webhook URL (below) | The host is checked again on every attempt, so a name that later points at a private address is refused then too. ## Send events to a chat channel Chat incoming-webhook URLs (Slack, Microsoft Teams, Discord, Google Chat) are refused: they cannot check a signature and expect their own message format. Run this short relay instead and register its URL. It verifies each delivery with the `verify` function above (saved as `verify.mjs`), posts one line per event, and answers `503` with `Retry-After` when the chat service fails, so SocialCrawl retries the batch. ```js title="JavaScript" import { createServer } from "node:http"; import { verify } from "./verify.mjs"; const { SOCIALCRAWL_WEBHOOK_SECRET: SECRET, CHAT_WEBHOOK_URL: CHAT, PORT = 8080 } = process.env; createServer(async (req, res) => { const chunks = []; for await (const chunk of req) chunks.push(chunk); const raw = Buffer.concat(chunks); if (!verify(SECRET, req.headers, raw)) return res.writeHead(401).end(); const lines = JSON.parse(raw).data.events.map(({ type, subject, data, links }) => `${type} ${subject?.id ?? ""}: ${data.status ?? data.reason ?? `${data.count ?? 0} new`} ${links?.run ?? ""}`.trim()); if (lines.length > 0) { const r = await fetch(CHAT, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ text: lines.join("\n") }) }); if (!r.ok) return res.writeHead(503, { "retry-after": "60" }).end(); } res.writeHead(204).end(); }).listen(PORT); ``` It does not deduplicate retried batches. If each event must be posted once, keep the inner event ids you have posted and skip those. ## Next steps - [Event feed](/docs/events.md): Read the same events by polling, with a cursor or a named consumer. - [Discover](/docs/prism/discover.md): Background and scheduled runs that send these events. - [Errors](/docs/errors.md): Every error type and whether to retry. - [Monitor webhooks](/docs/webhooks.md): The per-Monitor webhook, signed with its own header.