# Response Schema (/docs/response-schema) The unified SocialCrawl response envelope, headers, and partial-data warnings Every SocialCrawl response uses the same envelope, on success and on error. Write one parser and it handles every platform. ## Successful response ```json title="Response" { "success": true, "platform": "tiktok", "endpoint": "/v1/tiktok/profile", "data": { "author": { "username": "charlidamelio", "followers": 156800000 }, "computed": { "engagement_rate": null, "language": null, "content_category": null, "estimated_reach": null } }, "credits_used": 1, "credits_remaining": 4999, "request_id": "req-XXXXX", "cached": false } ``` The `computed` leaves are `null` here because this abridged author carries no `likes_count` and no `bio`, and because `estimated_reach` is always `null` on a profile. Every computed field is a real value or an honest `null`, never a filled-in guess, so type your parser for `T | null` on all four. [Computed fields](/docs/computed-fields.md) states the exact rule per field. ### Envelope fields | Field | Type | Required | Description | | --- | --- | --- | --- | | `success` | `boolean` | yes | true for successful responses, false for errors. | | `platform` | `string` | yes | Platform identifier such as tiktok or instagram. Account endpoints like /v1/credits/balance report meta. | | `endpoint` | `string` | yes | The full endpoint path that was called. | | `data` | `object` | yes | Platform-specific, normalised payload. Lists are returned as items plus optional next_cursor and total. The computed sub-object is attached to every Author and Post, and judged lists add labels to their rows and a report such as data.labels. | | `credits_used` | `integer` | yes | Credits consumed by this request. 0 on cache hits and idempotent replays. | | `credits_remaining` | `integer \| null` | yes | Your current committed balance when known. null when a response cannot establish it without an auxiliary balance lookup. | | `request_id` | `string` | yes | Unique identifier in the form req-XXXXX, for debugging and log lookup. | | `cached` | `boolean` | yes | true when served from cache. Cache hits cost 0 credits. | | `age_s` | `integer \| null` | no | Cache hits only: whole seconds since the copy was fetched, also sent as the X-Cache-Age header. null for a copy stored before ages were recorded. Absent on a live call. Send max_age_s to set how old a copy you accept. | | `credits_remaining_note` | `string` | no | Cache hits only: why credits_remaining is null there (a free hit does not read the balance) and where to read it. | | `pagination` | `object` | no | List endpoints only. The uniform pagination contract next_cursor, has_more, page_size, identical on every platform, plus typical_page_size (rows a full page holds) and next_page_credits (what the next page costs). A creator post list called with since or stop_at_id also carries stopped_at. | | `meta` | `object` | no | Present only when the response carries a hint. meta.hint names one call that does what your recent calls did in fewer steps. Absent on almost every response. | | `idempotent_replay` | `boolean` | no | Present and true only when this response is an idempotency replay. Absent on a normal response. | A few more fields live inside `data` rather than at the envelope root: `data.dropped`, `data.empty_reason` with `data.empty_reason_note`, and `data._warnings[]`. ### `data.dropped`: Discarded-row counter On list endpoints, `data.dropped` counts upstream items omitted because they could not be repaired to the endpoint schema. A healthy list response carries `"dropped": 0`. It is the only signal that a page came back short, so read it on every page of a drain rather than inferring completeness from `items.length`. `dropped` sits next to `items` inside `data`, while `pagination` sits at the envelope root. Reading `response.dropped` returns `undefined` and silently looks like zero. ### `data.empty_reason`: Why a list is empty Every list response whose `data.items` is empty says why, so "nothing new since your last poll" never looks like "the source broke". `data.empty_reason` is one code from the table below, and `data.empty_reason_note` is one or two sentences saying what the code means and what to do next, written to be shown to a person or read by an agent. Both sit next to `items`, and a page with rows carries neither. ```json title="Response" { "success": true, "platform": "instagram", "endpoint": "/v1/instagram/profile/posts", "data": { "items": [], "dropped": 0, "empty_reason": "nothing_new_since", "empty_reason_note": "The walk reached your since or stop_at_id boundary before any newer row, so nothing was published after it. Poll again later with the same boundary." }, "credits_used": 1, "credits_remaining": 4998, "request_id": "req-XXXXX", "cached": false, "pagination": { "next_cursor": null, "has_more": false, "page_size": 0, "stopped_at": "known_id" } } ``` | `empty_reason` | What happened | What to do | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `no_matches` | The source answered and nothing matches: zero search results, or a list that two sources agree is empty | Change the query or the filters. The same call returns the same answer | | `end_of_list` | A page requested with a cursor came back empty and there is no next page | Stop paging | | `nothing_new_since` | A walk with `since` or `stop_at_id` reached your boundary before any newer row | Nothing was published since your last poll. Poll again later with the same boundary | | `window_filtered` | Every row on the page fell outside `since`, `until` or `recent_days`, or was a pinned post you excluded | Keep paging while `pagination.has_more` is `true`, or widen the window | | `relevance_filtered` | `relevance=filter` judged every row off-topic. Their ids are in `data.relevance.dropped_ids` | Keep paging, or send `relevance=score` to keep every row with its score | | `row_filtered` | A row filter you sent, such as `country` or `min_views`, kept no row. The report next to `items` counts what it dropped | Keep paging, or loosen the filter | | `already_seen` | Every row on the page was one already returned on this walk, by cursor de-duplication or `seen=` | Keep paging while `pagination.has_more` is `true` | | `source_empty_unverified` | The source returned no rows and nothing confirmed it: only one source answered a list that should have rows, a source failed first, or a page mid-list came back empty | Retry later or with `Cache-Control: no-cache`. Keep paging while `pagination.has_more` is `true` | Branch on `empty_reason`, never on the note, whose wording can improve. Read it together with `pagination.has_more`, which still decides whether to keep paging: a filtered page in the middle of a list is empty and has more pages after it. A private account is never an empty list: it answers `404` with `error.details.reason` set to `account_private`. ### `data._warnings[]`: Partial-data channel When the field-map or computed-field pipeline hits ambiguous upstream data, it attaches a human-readable notice to `data._warnings: string[]`: ```json title="Response" { "data": { "author": { "username": "..." }, "computed": { "engagement_rate": 1.0 }, "_warnings": [ "computed.engagement_rate: value exceeded 1.0 (raw: 1.42); clamped" ] } } ``` Treat entries as advisory. The response is still valid. Empty arrays are omitted, so `_warnings` is present only when there is something to report. A query parameter the endpoint does not declare is never forwarded under that name. The house names `sort`, `country`, `limit`, `since` and `until` work on every endpoint that takes the same idea under another name (`sort_by`, `order`, `region`, `location`, `page_size`, `max_results`, `start_date`, `from`), and so do the common other spellings (`order`, `sort_by`, `region`, `geo`, `market`, `count`, `date_from`). The value is applied under the endpoint's own name and `_warnings` says what was applied: | Request | `_warnings` line | | ------- | ---------------- | | `tiktok/profile/videos?handle=nasa&sort=popular` | `sort was applied as sort_by, this endpoint's name for it.` | | `youtube/video/comments?url=...&sort=recent` | `sort was applied as order=newest, this endpoint's name and value for it.` | | `tiktok/search/users?query=nasa®ion=US` | `region was applied as country, this endpoint's name for it.` | | `sephora/search?query=lipstick&limit=500` | `limit was applied as page_size=60, the most this endpoint returns in one page.` | | `us_congress_trades/trades?since=2026-09-01` | `since was applied as start_date, this endpoint's name for it.` | A value moves only when the endpoint's own parameter accepts it, so a request that worked never becomes an error. `newest`, `latest` and `recent` reach the endpoint's own word for newest first, and `popular` and `top` its word for most popular. `country` never lands on a `region` that only picks the country the request is routed through, because that would not filter the rows. Any other undeclared parameter is ignored and named in `_warnings` with what the endpoint does support, for example `"days was ignored: did you mean recent_days?"`, `"sentiment was ignored: this endpoint has no sentiment parameter; label=sentiment is free here."`, or, on a list whose page size is fixed, `"limit was ignored: this endpoint's page size is fixed. To get more rows in one call, send max_items with the number of rows you want (up to 5,000)."`. An identity alias (`username` for `handle` on `tiktok/profile`, `video_id` for `url` on `youtube/video`) is applied with no line. ### `profile/posts` on every platform `GET /v1/{platform}/profile/posts` is each platform's posts feed. Where the feed has another name, that path is served by it: `/v1/tiktok/profile/posts` answers as `/v1/tiktok/profile/videos`, `/v1/twitter/profile/posts` as `/v1/twitter/user/tweets`, and `/v1/threads/profile/posts`, `/v1/bluesky/profile/posts` and `/v1/truthsocial/profile/posts` as their `user/posts`. It is the same call at the same price, served from the same cache entry. The response's `endpoint` and its `Content-Location` header name the path that answered, and `_warnings` says `"GET /v1/tiktok/profile/posts was served as GET /v1/tiktok/profile/videos, this platform's name for it."`. `/v1/youtube/profile/posts` answers as `/v1/youtube/channel/videos` too, with the same `_warnings` line. Use the native path in new code. Some entries start with a stable token before the colon, so code can match on it. `session_priced:` appears on `tiktokshop/search` outside `region=US`: TikTok prices each shopper session separately there, so the same product can return a different `price.current` on each call, and `price.original` appears only when that session was offered a promotion. Product prices (`product.price.current`, `product.price.original`) are always decimal amounts in the major unit of `product.price.currency`, on every platform and in every region: `10.90` with `EUR` means 10.90 euros, never 1,090 cents. ### Data-level reports Lists that carry SocialCrawl [labels](/docs/labels.md) add a report next to `items`, so you can tell a judged page from an unjudged one without reading every row. All of them are on by default, cost nothing, and disappear with `judgments=off`. | Key | On | What it reports | | ---------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data.labels` | comment, post and review lists | The labels on the rows: `presets`, `mode` (`default` or `requested`), `status`, `rows`, `labelled`, `cached`, `pending`, `pending_ids`, `unjudged`, `dropped_ids`, `extra_credits`. When you name labels with `label=`, `data.labels.default` carries the report of the default ones | | `data.relevance` | post search lists, `search/everywhere`, `search/forums` | How many rows were scored against your query and, with `relevance=filter`, which were dropped (`dropped_ids`). On the post search lists it also carries `origin`, `status`, `pending` and `pending_ids` | | `data.stance_split` | `search/everywhere` | Positive, negative, neutral and no-opinion counts and shares, overall and per platform, with the result ids behind every count | | `data.stories`, `data.story_grouping` | `search/news` (JSON only) | Articles about the same event grouped into stories across editions and languages, and how many articles were judged. `data.stories` is `null` when grouping did not finish | | `data.account_kinds` | `search/creators` | How many creators were judged for account kind | `status` is one of three values: `complete` (every row that can be judged carries its judgment), `partial` (some rows are `null`, either still being judged, counted in `pending`, or not judged this time), or `skipped` (judging was switched off for this request because capacity was short, and the page is exactly what `judgments=off` returns plus the report). A `pending` row is filled on your next call for the same page, including a free cache hit. See [Labels](/docs/labels.md#null-and-pending) for the details, including the search endpoints, which do not fill late rows. ### `meta.hint`: A shorter path for what you just did When a run of your calls repeats what one other call already does, the response to the call that completes the run can carry a top-level `meta.hint` naming that call. It appears only on a successful `GET`: ```json title="Response" { "success": true, "platform": "youtube", "endpoint": "/v1/youtube/search", "data": { "items": [ /* ... */ ], "dropped": 0 }, "credits_used": 1, "credits_remaining": 4998, "request_id": "req-XXXXX", "cached": false, "meta": { "hint": { "code": "search_multi", "message": "You searched \"matcha latte\" on 3 platforms one call at a time; /v1/search/multi returns each platform's own search rows in one call at the same price per platform, and a platform with no rows is not charged.", "method": "GET", "path": "/v1/search/multi?query=matcha%20latte&platforms=tiktok,instagram,youtube" } } } ``` | `code` | When it appears | Points to | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | `creator_card` | You looked up the same handle on the profile endpoints of three platforms the creator card covers, within 30 minutes | `GET /v1/prism/creator-card` | | `batch_profiles` | You looked up 50 different handles on one platform's profile endpoint within 30 minutes | `POST /v1/prism/profiles` | | `search_multi` | You searched the same query on the search endpoints of three platforms that `search/multi` covers, within 30 minutes | `GET /v1/search/multi` | | `incremental_sync` | You fetched page 1 of the same creator post list again within 24 hours, without `since` or `stop_at_id` | The same list with `stop_at_id` ([Pagination](/docs/pagination.md#incremental-sync)) | | `comment_labels` | After a search, you opened the comments on five or more posts from the same platform within 30 minutes, without `label=` | Filtering on the `computed.labels` every comment row already carries at no extra credits, or `GET /v1/prism/comments`, with its price in the message | | `search_repeat` | You ran page 1 of the same search on the same endpoint on an earlier visit (more than 30 minutes ago, within two days), without `seen=` or a date filter | The same search with `seen=` (or the endpoint's date filter) | | `batch_comments` | You read the comments under 10 or more different posts one call at a time within a minute, on any platform | `POST /v1/prism/post-comments` (25 post URLs a call), or `POST /v1/prism/jobs` for up to 1,000 in the background | | `batch_jobs` | You looked up 10 or more different items on the same endpoint one call at a time within a minute (for example `youtube/channel/about` by handle) | That endpoint's own batch where it has one (`POST /v1/youtube/videos`, `POST /v1/youtube/transcripts`), otherwise `POST /v1/prism/jobs` with that endpoint and the items | `method` and `path` are the call to make, and `message` says why in one sentence. When the suggested call costs more than the calls it replaces, the message states both prices. A key receives each hint at most once per UTC day, and a response carries at most one. A hint never appears on an error or on an idempotent replay, it costs nothing, and the rest of the response is unchanged. Send the request header `X-SocialCrawl-Hints: off` to never receive one. If your parser rejects unknown keys, allow an optional `meta` object at the envelope root. ## Error response ```json title="Response" { "success": false, "error": { "type": "INSUFFICIENT_CREDITS", "message": "Your account has 0 credits remaining. This endpoint requires 1 credits.", "status": 402, "doc_url": "https://www.socialcrawl.dev/docs/errors#insufficient-credits" }, "credits_used": 0, "credits_remaining": 0, "request_id": "req-abc123" } ``` `credits_used` is the **net committed** charge for the failed request. It is `0` when no deduction occurred or the refund committed. If an immediate refund fails, it remains non-zero while exact compensation is queued; keep the `request_id` for reconciliation. `doc_url` is always the anchor form `/docs/errors#`, never a per-code sub-path. See [Errors](/docs/errors.md) for the full error-code table and refund rules. ## Response headers | Header | Value | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X-Request-Id` | Matches `request_id` in the body. On an idempotent replay the body keeps the original `request_id` and this header is the current delivery. Use it to correlate logs | | `X-Credits-Used` | Credits consumed. `0` on cache hits, empty-upstream 404s, 405/409/422, idempotent replays, and a background run's POST (its searches bill later: read `data.run.credits.spent`) | | `X-Credits-Remaining` | Current balance when known. On an idempotency replay, omitted when the balance lookup fails; body `credits_remaining` is `null` instead | | `X-Cache` | `HIT` (served from cache, 0 credits) or `MISS`. Force a `MISS` with `Cache-Control: no-cache` or `max_age_s=0` | | `X-Cache-Age` | On a cache hit, seconds since the copy was fetched (the same number as body `age_s`). Absent on a live call and when the age is not recorded. The standard `Age` header is our CDN's own and is not the age of the copy | | `X-Idempotent-Replay` | `"true"` on idempotent replays. Only present when the response was replayed | | `Idempotent-Replayed` | `"true"` on the same replay. Same meaning as `X-Idempotent-Replay`. Absent otherwise | | `X-RateLimit-Limit` / `-Remaining` / `-Reset` | Request-window headroom, on every authenticated response. See [Rate limits](/docs/rate-limits.md) | | `X-Concurrency-Limit` / `-Remaining` | In-flight headroom, on every authenticated response. Not an alias of `X-RateLimit-*` | | `X-Upstream-Retries` | How many upstream retries this call needed. Omitted when there were none | | `Retry-After` | Seconds to wait. Sent on both `429`s (seconds until the window resets for `RATE_LIMITED`, a static `1` for `CONCURRENCY_LIMIT`), and on a transient `503` (`30`). Never on a `402`. Absent otherwise | | `Allow` | Accepted method set on 405 responses: `"GET, HEAD"` for registry reads or `"POST"` for POST-only routes | ## List endpoints List archetypes (`PostList`, `CommentList`, `SearchResult`) are normalised to a consistent shape regardless of upstream key names: ```json title="Response" { "data": { "items": [ /* ... */ ], "next_cursor": "eyJwYWdlIjoyfQ==", "total": 1247 } } ``` The `data`-level `next_cursor` and `total` are legacy per-payload fields, included only when upstream provides them. **Use the top-level `pagination` block for pagination.** Every list response also carries a uniform `pagination` object at the envelope root. It is identical on all platforms and all three pagination styles (cursor, page, offset): ```json title="Response" { "data": { "items": [ /* ... */ ], "dropped": 0 }, "pagination": { "next_cursor": "sc.eyJ2IjoyLCJjIjoiMTc3NTM5NDk1MDAwMCJ9", "has_more": true, "page_size": 30, "typical_page_size": 30, "next_page_credits": 1 } } ``` The loop is the same everywhere. Read `pagination.next_cursor`, send it back as the `cursor` query parameter, and stop when `pagination.has_more` is `false`. You never handle a per-platform input-param name yourself. The API rewrites the `sc.` cursor to the endpoint's native param, `max_cursor` or `after` for example, on the way in. For the full drain recipe, see [Pagination](/docs/pagination.md). Two fields help you budget before you send the cursor back. `pagination.typical_page_size` is how many items a full page of this endpoint usually holds for your request (`null` when we have not verified it, as on a source whose page changes with your parameters; an endpoint that merges several sources can return more), so a smaller `page_size` with `has_more: true` reads as a short page from the source rather than the end. `pagination.next_page_credits` is what the next page costs, priced from that next request by the same pricer that charges it, and `null` when there is no next page. A metered endpoint refunds what the page does not use, and a cached page is free. See [What the next page costs](/docs/pagination.md#what-the-next-page-costs). On a list in date order, a call that sends `since` or `stop_at_id` gets one more field, `pagination.stopped_at`. It is `known_id` or `since` when the walk reached that boundary, `end` when the list ran out first, and `null` while there is more to walk. A call that sends neither, or a list that is not in date order, has no `stopped_at` key. See [Incremental sync](/docs/pagination.md#incremental-sync) and [Date window on every list](/docs/pagination.md#date-window-on-every-list). ## Idempotent requests Send an `Idempotency-Key` header, a UUIDv4 for example, and a retry with the same key returns the stored answer instead of running again. A replay keeps the first call's body except for the billing fields: `credits_used` is `0`, `idempotent_replay` is `true`, `credits_remaining` is your balance now (or `null` when it cannot be read, with `X-Credits-Remaining` left out), and `request_id` stays the first call's. ```bash title="cURL" curl 'https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio' \ -H "x-api-key: YOUR_API_KEY" \ -H "Idempotency-Key: 7a5e1b4c-2d8f-4a3b-9c1e-6e8b4d2a1f3c" ``` Not every call takes a key: seven `POST` calls ignore it and cohort writes require it. A `GET` that streams with `Accept: text/event-stream` cannot carry one. [Idempotency](/docs/idempotency.md) lists which calls take a key, how long it is kept, and the errors it can produce. ## Force a fresh fetch Responses from cache-enabled endpoints are cached and the cache is shared, so a repeat call returns instantly and costs 0 credits (`X-Cache: HIT`). To bypass the cache for a single call, send the standard `Cache-Control: no-cache` request header: ```bash title="cURL" curl 'https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio' \ -H "x-api-key: YOUR_API_KEY" \ -H "Cache-Control: no-cache" ``` The request fetches live from the source and is billed at the normal endpoint cost, a forced `MISS`, so `X-Cache: MISS` and a non-zero `X-Credits-Used`. The fresh result is written back, so your next plain call gets a free `HIT`. Only the `no-cache` directive triggers it. `Cache-Control: no-store` on its own does not. `max_age_s=0` in the query does the same for a client that cannot set headers. Registry-marked uncached endpoints skip this cache contract. To choose how old a copy you accept instead, send `max_age_s` in seconds: `max_age_s=3600` serves a profile or post copy up to an hour old for 0 credits, and `max_age_s=60` fetches fresh when the copy is older than a minute. The hit says its age in `age_s`. See [Choose how old a copy you accept](/docs/caching.md#choose-how-old-a-copy-you-accept). For the freshness windows per data type, the shared-cache model, and guidance on when to bypass, see [Caching](/docs/caching.md). ## Next steps - [Pagination](/docs/pagination.md): The cursor drain loop, identical on every list endpoint. - [Computed fields](/docs/computed-fields.md): How engagement_rate, language, category, and reach are derived. - [Errors](/docs/errors.md): Every error type, its status, and whether it refunds. - [Caching](/docs/caching.md): Freshness windows, free hits, and forcing a live fetch.