# Universal social search (/docs/search/everywhere) One query sent to many social platforms in parallel, then ranked, clustered and enriched with top comments in one answer, as JSON or SSE Send one query to `GET /v1/search/everywhere` and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 20 sources across 15 platforms. An LLM plans the query, results are fused and reranked, near-duplicates are clustered, and every result carries the top comments from its source. A cited web answer, and on request a summary of X posts, come back beside the ranked results in `data.synthesis`. Returns one JSON envelope by default, or a live SSE stream when you ask for one. Flat **20 credits** per call. ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews" \ -H "x-api-key: YOUR_API_KEY" ``` This is the only endpoint that bypasses the per-source 1/5/10 tier ladder. The 20 credits are flat, however many sources fire and however many comments are enriched. ## How the fan-out works The pipeline runs in five stages, and every one of them is visible in the response. 1. **Plan.** A plan is ready at once (intent, freshness and source weights), and an LLM refines it in parallel; your query itself is what every source is sent. A fast planner scores per-source relevance at the same time, and a source it judges irrelevant for the query is **pruned**, so its results are not ranked and it leaves the coverage denominator. The planner fires an abort on a pruned source, but a source that had already answered before the abort landed still reports its `items`, so treat a prune as "this source was not ranked" rather than as proof that no request was made. A source you name in `sources` is never pruned. 2. **Fan out.** Every surviving source runs in parallel through the endpoint you could call yourself (the table below names it), with that endpoint's own fallbacks. A source whose endpoint keeps failing is skipped for a short while instead of being called, and is reported as `skipped: breaker_open`. Beside other sources each source has 12 seconds (6 seconds for `twitter`, 20 seconds for `facebook`, which runs only when you ask for it); when you search one platform it has 25 seconds. The X summary, `twitter-ai-search`, has up to 45 seconds because writing it is slow: the stream waits for it before `done`, and the JSON answer includes it only when it is ready by the time the rest of the answer is. 3. **Fuse and dedupe.** Results are merged, and the same item found by two sources becomes one candidate with multi-source provenance and a higher fused score. 4. **Enrich and rerank.** Top comments are fetched in parallel with a judged rerank that has 3 seconds. Scores already judged are kept when it runs out. Each result's stance toward your query is judged beside the rerank, and its relevance score comes from the rerank's own answers. 5. **Cluster.** Near-duplicates and same-story groupings are collapsed into `clusters`, when the resolved intent is clusterable. ### Sources The default fan-out hits every source below except `naver`, which joins when the query contains Hangul, `facebook`, which runs only when you ask for it with `include=facebook` (added to the default sources) or name it in `sources`, and `twitter-ai-search`, the X summary, which runs only with `include=synthesis`. Use `sources` / `exclude` to narrow it. | Source name | Platform or kind | Reads | Comment-enriched | | ------------------- | --------------------------------------- | ----------------------------- | ---------------- | | `reddit` | Reddit | `/v1/reddit/search` | yes | | `twitter` | X posts (`x` is another name for it) | `/v1/twitter/search/tweets` | no | | `twitter-ai-search` | X (a written summary of X posts, only with `include=synthesis`) | `/v1/twitter/ai-search` | no | | `youtube` | YouTube | `/v1/youtube/search` | yes | | `tiktok` | TikTok | `/v1/tiktok/search` | yes | | `instagram` | Instagram | `/v1/instagram/search/reels` | yes | | `hackernews` | Hacker News | `/v1/hackernews/search` | yes | | `polymarket` | Polymarket prediction markets | `/v1/polymarket/search` | no | | `github` | GitHub | `/v1/github/search` | yes | | `threads` | Threads | `/v1/threads/search` | no | | `pinterest` | Pinterest | `/v1/pinterest/search` | no | | `perplexity` | The web (a cited web answer) | `/v1/perplexity/research` | no | | `web` | The web (search results; `tavily` is another name for it) | `/v1/web/search` | no | | `linkedin` | LinkedIn | `/v1/linkedin/search/posts` | no | | `rumble` | Rumble | `/v1/rumble/search` | no | | `tiktok-hashtag` | TikTok hashtag search (hashtag mode) | `/v1/tiktok/search/hashtag` | no | | `instagram-hashtag` | Instagram hashtag search (hashtag mode) | `/v1/instagram/search/hashtag` | no | | `youtube-hashtag` | YouTube hashtag search (hashtag mode) | `/v1/youtube/search/hashtag` | no | | `naver` | Naver blog, cafe and KnowledgeiN (Korean queries) | `/v1/naver/blog/search`, `/v1/naver/cafearticle/search` and `/v1/naver/kin/search` | no | | `facebook` | Facebook public posts (only with `include=facebook` or when named in `sources`) | `/v1/facebook/search/posts` | no | The three hashtag lanes fire only in hashtag mode. `naver` fires when the query contains Hangul, or on any query when you name it in `sources`. X and the web each have a second source (`twitter-ai-search` and `perplexity`), and TikTok, Instagram and YouTube each have a hashtag lane, so the 15 platforms make up to 20 sources on one call. **Each source is its own endpoint.** A source calls the endpoint in the Reads column with your query, through that endpoint's own fallbacks, so a source answers whenever a direct call to the endpoint would. The calls a search makes to these endpoints are part of its 20 credits: they are not billed separately and do not count against your rate limit. They are paced per caller, and a source call past that pace is skipped and reported as `skipped: leg_pace`. [Polymarket](/docs/polymarket.md) has the market search the `polymarket` source reads. **Facebook.** `facebook` searches public Facebook posts by keyword, newest first inside your date window (`lookback_days`, or `from_date` / `to_date`), and returns about 5 posts. Each row has the post text in `body`, the permalink in `url`, the page or person who posted it in `author`, the publish time in `published_at` and like, comment and share counts in `engagement`; the account URL is in `metadata.author_url`, so you can follow it to `/v1/facebook/profile`. It is not in the default fan-out. `include=facebook` adds it to the default sources; `sources=facebook` (alone) or `sources=facebook,reddit` runs only the sources you name. Beside other sources it has up to 20 seconds instead of 12, so a request that includes it can take longer. It stays inside the flat 20 credits. **Korean queries.** `naver` searches Naver blog posts, cafe communities and KnowledgeiN questions in one lane. Each Naver row carries its corpus in `metadata.corpus`. Blog rows have the posting day; KnowledgeiN rows have an estimated date, so their `date_confidence` is `low`; cafe rows have no date. **Dates.** Check each row's `date_confidence` before you filter on `published_at`. `low` means the date is an estimate: a KnowledgeiN estimate, or a YouTube row dated from a "3 years ago" or "2 months ago" label. **X posts and the X summary.** `twitter` returns X posts as ranked rows, like any other source, and is in the default fan-out. `twitter-ai-search` writes a summary of X posts with its citations, which takes 15 to 45 seconds, so it runs only when you ask for it with `include=synthesis`. The summary arrives in `data.synthesis.x`, never as ranked rows. The stream waits for it. The JSON answer includes it only when it is ready by the time the rest of the answer is; otherwise `sources_failed` lists `twitter-ai-search` with the reason `not ready when the rest of the answer was`. For the X summary beside other sources, use the stream. **The web results and the web answer.** `web` returns web search results as ranked rows; their `body` is plain text of up to 1,500 characters. `perplexity` writes a cited web answer, which arrives in `data.synthesis.web`. Its rows stay in `items_by_source.perplexity` and are not ranked. **Other names and platform shorthands.** `x` is another name for `twitter`, and `tavily` for `web`, so `sources=x` and `sources=twitter` return X posts. `twitter-ai-search` is still accepted in `sources`, where it means `twitter` plus `include=synthesis`; in `exclude` it leaves out only the summary. A bare platform name resolves to its whole group: `youtube`, `instagram` and `tiktok` each cover their `-hashtag` lane too, so `exclude=youtube` cannot leak YouTube content back in through `youtube-hashtag`. Exact lane names stay individually addressable. An unknown name in `sources` or `exclude` is a `400` that enumerates the valid names, never a silent drop. A parameter this endpoint does not take is ignored, never refused: the answer carries a `data._warnings` line naming it, with a did-you-mean when a real parameter is close. ### Keyword mode and hashtag mode There is no user-facing mode switch. The planner decides: - **Keyword mode (default).** A natural-language query fans out to the base keyword-search sources. - **Hashtag mode.** The three hashtag siblings are added automatically in either of two cases. **One tag or one word:** the query is a single token, for example `query=%23thrifting` or `query=thrifting`, and a leading `#` is stripped before dispatch. **Topic term:** for a multi-word topic query the planner extracts a topic term (`makeup tutorials` yields the tag `makeup`) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query. When the same TikTok, Instagram or YouTube post appears in both streams, fusion merges the two into one candidate. If the planner times out or fails, explicit `#tag` queries still route to hashtag mode deterministically, and multi-word topic queries degrade to keyword-only. The `query` value is passed to the planner as-is. There are no special search operators: no `site:`, no boolean `AND`/`OR`, no quoting. The single input with special meaning is a leading `#`. ## Request parameters | Param | Required | Type | Default | Description | | --------------- | -------- | ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | yes | string (1 to 512 chars) | none | The natural-language search query. A leading `#` puts the query into hashtag mode. | | `lookback_days` | no | integer (minimum 1) | `30` | Recency window in days. Zero or negative is rejected before billing; there is no upper bound, because this is a ranking signal rather than a hard filter and a very wide window is a degraded answer rather than an invalid request. It sets the freshness curve, so an older item can still rank when nothing recent matches, and the date window sent to `facebook`. When `from_date` is sent too, `from_date` applies. On the stream, windows over 90 days add a `warning` chunk. For hard bounds use `since`, `until`, `from_date` and `to_date`. | | `from_date` | no | ISO `YYYY-MM-DD` or timestamp | none | Lower bound of an explicit date window, from the start of that UTC day. A hard bound: rows dated before it are dropped from `items`, rows with no date are kept, and `data.since` reports it. Sent with `since`, the later of the two applies and `data._warnings` says so. An invalid date is a free `400`. When `lookback_days` is sent too, `from_date` applies. | | `to_date` | no | ISO `YYYY-MM-DD` or timestamp | today | Upper bound of an explicit date window, to the end of that UTC day (inclusive). A hard bound: rows dated after it are dropped from `items`, rows with no date are kept. Sent with `until`, the earlier of the two applies and `data._warnings` says so. Defaults to today when `from_date` is set alone. An invalid date is a free `400`. | | `since` | no | ISO 8601 timestamp or date | none | Only rows published at or after this time, for polling: pass the previous run's time and `items` holds only what is new. A hard filter, unlike `lookback_days`: a row dated before `since` is dropped from `items` and `items_by_source`, rows with no date are kept, and `data.since` reports `{ since, until, dropped, kept_undated }`. The call is billed as usual. As a Monitor with `since=now-2h` and `webhook_rows: "new"`, each run delivers the rows not seen on the previous run (`row_key` defaults to `candidate_id`). | | `until` | no | ISO 8601 timestamp or date | none | Only rows published at or before this time; a bare date includes that whole UTC day. A hard filter like `since`: rows with no date are kept and counted in `data.since`. Billed as usual. An invalid date is a free `400`. | | `period` | no | `Nd` (1 to 3650), `yesterday`, `this_week`, `last_week`, `this_month`, `last_month`, `last_quarter`, `this_year`, `ytd` | none | A window that holds, by name. Each platform is sent its own date filter and every row is checked against the window again, so rows outside it are dropped whatever the platform did. `data.request.window` echoes the resolved window with the rows dropped for it, and `data.platforms..window` reports the date filter that platform was sent and its `rows_outside`. `since`, `until`, `from_date`, `to_date` and `lookback_days` win over `period` when sent with it, and `data._warnings` says so. Billed as usual. An unknown period is a free `400`. | | `tz` | no | IANA time zone name | `UTC` | The zone `period` and bare `since`, `until`, `from_date` and `to_date` dates are read in, for example `Asia/Seoul`. An unknown zone is a free `400`. | | `undated` | no | `keep` or `drop` | `keep` | What a row with no date does inside a window: `keep` keeps it and counts it in `data.request.window.undated_kept`; `drop` removes it and counts it in `undated_dropped`. Applies only when a window is set. | | `sources` | no | CSV string | all | Allowlist of source or platform names, for example `reddit,youtube,hackernews`. Mutually exclusive with `exclude`. | | `exclude` | no | CSV string | none | Blocklist of source or platform names to skip. Mutually exclusive with `sources`. | | `platforms` | no | CSV string | none | Another name for `sources`, with the same values. When both are sent and differ, `sources` applies and `data._warnings` says so. | | `limit` | no | integer 1 to 40 | `40` | How many ranked results `items` returns. Above 40 is clamped to 40 with a `data._warnings` line. It trims the answer only: the search, its price and its cache entry are the same whatever the limit, so `limit=10` then `limit=20` on one query is one billed search. | | `include_transcripts` | no | `true` / `1` | `false` | Opt-in spoken-word transcripts on the top 3 video results, so a video whose title never mentions your query but whose narration does still surfaces. | | `judgments` | no | `on` \| `off` | `on` | `on` adds a stance (`computed.stance`) and a relevance score (`computed.relevance`) to every result, with `data.stance_split` and `data.relevance` counting them. Nothing is dropped or reordered and the price does not change. `off` returns the response without these blocks. | | `relevance` | no | `filter` | none | Drops the results the ranking step judged off-topic for your query and lists their ids in `data.relevance.dropped_ids`. A result that could not be judged is never dropped, and `items_by_source` is not filtered. Included in the flat 20 credits. | | `include` | no | CSV: `synthesis`, `facebook`, `stance` | none | `synthesis` adds the X summary (`twitter-ai-search`) to the sources and returns it in `data.synthesis.x`. `facebook` adds Facebook public post search to the default sources. `stance` is kept for compatibility: stance is on by default now, and naming it gives the stance judgment priority when capacity is short. An unknown value is a free `400`. | | `switch_from` | no | string (a brand) | none | `data.switch_receipts` lists the results where the author says they personally left that brand, with the verbatim text, the reason, where they went and the link. There is no percentage. Rows that are not a switch stay in `items`. Included in the flat 20 credits. | `from` and `to` are accepted as aliases for `from_date` and `to_date`, so a date window carried over from `/v1/search/news` (which spells them `from` / `to`) is honoured here rather than silently dropped. See [One date vocabulary across the four lanes](/docs/search.md#one-date-vocabulary-across-the-four-lanes). Passing both `sources` and `exclude` returns `400 INVALID_REQUEST`. Check `data.sources_called` (or the SSE `done` summary) against what you asked for. ## Filter examples ```bash title="cURL" # Narrow to specific sources curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews" \ -H "x-api-key: YOUR_API_KEY" # Exclude noisy sources curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket" \ -H "x-api-key: YOUR_API_KEY" # Explicit date window curl "https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30" \ -H "x-api-key: YOUR_API_KEY" # Recency window curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7" \ -H "x-api-key: YOUR_API_KEY" # Hashtag mode (leading #, URL-encoded as %23) curl "https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting" \ -H "x-api-key: YOUR_API_KEY" ``` In a real `#thrifting` call the plan intent resolves to `how_to`, and `sources_called` includes `tiktok-hashtag`, `instagram-hashtag` and `youtube-hashtag` alongside the base sources. ## What you get back By default (`Accept: application/json`) you get one envelope. `data` carries: | Field | Type | Description | | ------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | `query` | string | The query as the planner saw it (`#` stripped for explicit-tag queries). | | `plan` | object | The resolved plan: `intent`, `freshness_mode`, `cluster_mode`, `subqueries[]`, `source_weights`, `notes[]`. | | `items` | array | The ranked, deduped candidates (up to 40). Each carries scoring plus its `source_items[]`. | | `items_by_source` | object | Every row each source returned, grouped by source name, including rows that were not ranked into `items` and the rows of the web answer and the X summary. | | `clusters` | array | Near-duplicate and same-story groupings over `items`. | | `sources_called` | array | Every source the pipeline attempted this request. | | `sources_failed` | object (`{ source: reason }`) | Sources that were searched and failed, and sources that were skipped without a call (`skipped: breaker_open`, `skipped: leg_pace`), each with a reason. Pruned sources are in `sources_pruned`. | | `sources_succeeded` | array | Sources that returned without failing. | | `coverage` | number (0 to 1) | The share of the searched sources that answered: pruned sources are left out of the count, and a skipped source counts as one that did not answer. Drives the partial-refund decision. | | `platforms` | object | What each searched platform did, keyed by platform (`reddit`, `twitter`, `web` and so on, with a platform's hashtag lane folded in): `{ status, rows, latency_ms, error }`. `status` is `ok`, `empty`, `failed`, `skipped` or `pruned`; `error` is `null`, or `{ code, retryable }` with a code such as `platform_failure`, `rate_limited` or `breaker_open`. | | `synthesis` | object | `{ web, x }`: the cited web answer and, with `include=synthesis`, the X summary. Each is `{ text, citations }`, or `null` when that source did not run or wrote nothing. Never ranked rows. | | `partial_failure` | boolean | `true` when a partial-coverage refund fired this request. | | `request_id` | string | The same id as the envelope's `request_id`. The pipeline's own id is `trace_id`. | | `judgments` | object | `{ relevance, stance }`, each `judged`, `skipped` or `unavailable`, so a missing judgment always says why. | | `stance_split` | object | Stance counts and shares over the results, overall and per platform, with the ids behind every count. Absent with `judgments=off`. | | `relevance` | object | How many results were judged and how many are off-topic (`off_topic`, `off_topic_ids`); with `relevance=filter`, the `dropped_ids`. Absent with `judgments=off`. | | `_warnings` | array | Present only when something needs saying, such as `stance_partial`, `stance_unavailable`, `stance_skipped`, `relevance_partial` or `relevance_unavailable`. | ### The web answer and the X summary `data.synthesis` holds the two written answers, so you can show them above the ranked results without parsing rows: ```ts title="TypeScript" { web: { text: string, citations: Citation[] } | null, // the perplexity answer x: { text: string, citations: Citation[] } | null // the X summary, with include=synthesis } // Citation { url: string, // the page the answer cites, in citation order also_on?: string // the id of the `web` row for the same page, when the web results found it too } ``` `also_on` lets you link a citation to the matching web result instead of showing the page twice. ### Stance and relevance on every result Each result in `items` carries two judgments by default, free: ```json title="JSON" "computed": { "stance": { "label": "neutral", "probabilities": { "positive": 0.01, "negative": 0, "neutral": 0.92, "no_opinion": 0.07 }, "confidence": 0.9 }, "relevance": { "score": 0.93, "on_topic": true, "entity": 0.97 } } ``` `stance.label` is what the post, with any replies shown, says about your query: `positive`, `negative`, `neutral`, `no_opinion`, or `null` when the judgment is under the confidence floor, which counts as uncertain in `data.stance_split`, never as a stance. `relevance.score` runs from 0 to 1, and `on_topic: false` marks exactly the results `relevance=filter` would drop. Both are judged inside the request under a deadline that runs beside the ranking step, so they add no wait; a result not judged in time is `null` for the life of the cached response. [Labels](/docs/labels.md#how-we-measure) states how stance was measured. ### Real-people comments For every result whose source exposes a comments endpoint, the pipeline fetches the top-scoring comments and attaches them at `data.items[i].source_items[0].metadata.top_comments[]`. No second API call. Each `TopComment` has this shape: ```ts title="TypeScript" { score: number | null, // upvote / like / points count, source-specific excerpt: string, // up to 300 chars, HTML stripped author: string | null, // null when [deleted] / [removed] url: string | null, // direct comment URL when the source exposes one date: string | null // ISO timestamp when available } ``` Comments are sorted by `score` descending: up to 15 on each of the top 10 results and 5 after, 300 characters per excerpt. Sources without a comments endpoint (`twitter`, `twitter-ai-search`, `threads`, `pinterest`, `polymarket`, `perplexity`, `web`, `linkedin`, `rumble`, `naver`) render no `top_comments`. `facebook` rows carry none either: the lane returns the comment count in `engagement.comments`, and `/v1/facebook/post/comments` reads the comments themselves. **Where each source's comments come from** | Source | Comments come from | | ------------ | --------------------------------------------------- | | `reddit` | `/v1/reddit/post/comments` | | `tiktok` | `/v1/tiktok/post/comments` | | `instagram` | `/v1/instagram/post/comments` | | `youtube` | `/v1/youtube/video/comments` | | `hackernews` | Algolia HN thread tree | | `github` | `/repos/{o}/{r}/issues/{n}/comments?sort=reactions` | **A trimmed response** For `query=apple vision pro reviews`, wrapped by the standard `{ success, data }` envelope: ```json title="Response" { "success": true, "data": { "query": "apple vision pro reviews", "plan": { "intent": "opinion", "freshness_mode": "balanced_recent", "cluster_mode": "debate", "subqueries": [ { "label": "reviews", "search_query": "apple vision pro review", "ranking_query": "hands-on apple vision pro review verdict", "sources": ["reddit", "youtube", "hackernews"], "weight": 1.0 } ], "source_weights": { "hackernews": 0.8, "youtube": 0.85, "reddit": 0.6 }, "notes": [] }, "items": [ { "candidate_id": "https://news.ycombinator.com/item?id=...", "source": "hackernews", "title": "Apple Vision Pro review: magic, until it's not", "url": "https://news.ycombinator.com/item?id=...", "rerank_score": 93, "final_score": 64.88, "sources": ["hackernews"], "cluster_id": "c1", "source_items": [ { "engagement": { "points": 412, "num_comments": 380 }, "media": { "thumbnail_url": null }, "metadata": { "top_comments": [ { "score": null, "excerpt": "This strikes me as the best review I've seen if you actually want to know what daily use is like...", "author": "some_real_user", "url": "https://news.ycombinator.com/item?id=...", "date": "2026-02-05T12:34:56Z" } ] } } ] } ], "items_by_source": { "hackernews": [], "youtube": [] }, "clusters": [ { "cluster_id": "c1", "title": "Vision Pro daily-use verdicts", "candidate_ids": ["..."], "representative_ids": ["..."], "sources": ["hackernews", "youtube"], "score": 0.71 } ], "sources_called": ["hackernews", "youtube", "reddit"], "sources_failed": { "reddit": "the source did not answer within its 12 s slot" }, "sources_succeeded": ["hackernews", "youtube"], "coverage": 0.6666666666666666, "partial_failure": false, "platforms": { "hackernews": { "status": "ok", "rows": 12, "latency_ms": 2140, "error": null }, "youtube": { "status": "ok", "rows": 12, "latency_ms": 3810, "error": null }, "reddit": { "status": "failed", "rows": 0, "latency_ms": 12000, "error": { "code": "platform_failure", "retryable": true } } }, "synthesis": { "web": null, "x": null } } } ``` In this call `reddit` ran out of its 12-second slot, so it lands in `sources_failed` and `data.platforms.reddit.status` is `failed`, `coverage` is `0.67`, and because that is still at or above the coverage floor the full 20 credits are charged (`partial_failure: false`). That is the normal, healthy shape: partial results are billable. ## Streaming Add `Accept: text/event-stream` and you get a live stream instead of one envelope. First byte lands in well under a second, typically around 300ms on the `meta` chunk, and fast sources stream their items while slow ones are still running. ```bash title="cURL" curl -N "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings" \ -H "x-api-key: YOUR_API_KEY" \ -H "Accept: text/event-stream" ``` Each event is one `data: \n\n` frame, discriminated by `type`. The order guarantees: - **`meta` is first**, emitted as soon as the request is parsed. It carries the deterministic plan and the full `sources_planned` list. - **`source_started` fires once per source** before that source's adapter runs. Pruned or failing sources still emit `source_started`, then a `source_pruned` or `source_failed`. - **`source_pruned`** fires when the fast planner judged a source irrelevant for the query, and the source appears in `sources_pruned` (never in `sources_failed`, and never in the coverage denominator). The planner fires an abort on the source, but a source that had already answered before the abort landed still emits its `items` chunk, so treat `source_pruned` as "do not rank this source" rather than as proof that no request was made. - **`items` yield in completion order**, so fast sources land first. The web answer and the X summary arrive as `items` chunks for `perplexity` and `twitter-ai-search`: the answer is the row whose `metadata.role` is `synthesis`, and each cited page is a row whose `metadata.role` is `citation`. The stream has no `data.synthesis` block. - **`plan_refined` is conditional**: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged. - **`ranked_partial` streams repeatedly** as the reranker scores batches. Each chunk carries only the scores that settled since the previous one, so merge by `candidate_id` rather than replacing your map. `scored` and `total` carry cumulative progress for a percentage indicator. - **`comments_enriched` is per-candidate.** Key on `candidate_id` to merge comments into the row already on screen. These always land before `done`, and may interleave with `ranked_partial` / `ranked_final`. - **`done` is always last**, carrying the run summary. On a thrown error you get a terminal `error` followed by `done` with `refunded: true`. JSON answers are cached for 120 seconds, and a cache hit is free. Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache. The two are keyed separately. **Full chunk vocabulary** ```text title="Stream" { "type": "meta", "request_id": "...", "query": "...", "plan": {...}, "sources_planned": [...] } { "type": "source_started", "source": "reddit" } { "type": "source_pruned", "source": "github", "reason": "low_affinity", "affinity": 0.1 } { "type": "items", "source": "youtube", "items": [...], "duration_ms": 1234 } { "type": "source_failed", "source": "linkedin", "error": { "code": "platform_failure", "message": "the source did not answer within its 12 s slot" } } { "type": "plan_refined", "plan": {...} } { "type": "ranked_partial", "scores": [{ "candidate_id": "...", "rerank_score": 88 }], "scored": 12, "total": 40 } { "type": "ranked_final", "items": [...] } { "type": "comments_enriched", "candidate_id": "...", "source": "reddit", "comments": [{ "score": 4823, "excerpt": "...", "truncated": false, "author": "...", "url": "...", "date": "..." }] } { "type": "transcript_enriched","candidate_id": "...", "source": "youtube", "transcript": {...} } { "type": "clusters", "clusters": [...] } { "type": "warning", "message": "lookback_days > 90 degrades coverage on some sources" } { "type": "done", "summary": {...} } { "type": "error", "code": "INTERNAL_ERROR", "message": "..." } ``` `clusters` is emitted when the intent is clusterable. `transcript_enriched` fires only when `include_transcripts=true`. The items on `ranked_final` carry the same `computed.stance` and `computed.relevance` blocks as the JSON response; the terminal `done` summary carries the page-level blocks of the JSON answer (`relevance`, `stance_split`, `topic_coverage`, `since`, `judgments`, `_warnings`, `switch_receipts`) and, on an empty answer, its `empty_reason`. **A real done summary** ```json title="JSON" { "type": "done", "summary": { "sources_called": ["polymarket", "hackernews", "web", "twitter", "youtube", "perplexity", "reddit", "twitter-ai-search", "..."], "sources_failed": {}, "sources_pruned": { "hackernews": 0.1, "polymarket": 0.03, "github": 0.04, "rumble": 0.05 }, "sources_succeeded": ["web", "twitter", "youtube", "perplexity", "reddit", "twitter-ai-search", "..."], "coverage": 1, "total_items": 213, "credits_used": 20, "cached": false, "refunded": false, "partial_failure": false } } ``` A search with `include=synthesis`: `twitter` and `twitter-ai-search` both answered, and the four pruned sources left the coverage count. When a source fails, its reason is in our own words, for example `"reddit": "the source did not answer within its 12 s slot"`. **Transcript enrichment (opt-in)** Set `include_transcripts=true` to attach spoken-word transcripts to the top video results. It is off by default. - Accepts `true` or `1`, case-insensitive. Anything else is treated as `false`. - Only the top few video-shaped candidates by rank get a transcript. The rest are unaffected. - Transcripts populate at `source_items[0].metadata.transcript` in JSON mode and stream as `transcript_enriched` chunks over SSE. - The endpoint stays flat 20 credits with the flag on. - `youtube` and `rumble` return timestamped segments. `tiktok` and `instagram` return a flat caption string with no timing. ## Billing Flat **20 credits** per call, deducted up front. Two automatic refund paths: | Situation | What you pay | | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Normal run, including one where some sources failed but coverage stayed at or above the floor | 20 credits | | Some results, but a strict majority of the called sources failed | 10 credits (half refunded), and `partial_failure` is `true` | | No result anywhere, and at least one source answered (or every source was pruned) | 0 credits (fully refunded). Sync returns `200` with `items: []` and an `empty_reason`; SSE sends a terminal `done` with `refunded: true` | | No source could be searched (every searched source failed) | 0 credits (fully refunded). Sync returns `502 ZERO_RESULTS` with the failures in `sources_failed`; SSE sends a terminal `done` with `refunded: true` | Stance, relevance and `relevance=filter` are included in the 20 credits: a call costs the same with `judgments=off`. So are the web answer, the X summary with `include=synthesis`, and every call the search makes to a source's endpoint: those calls add no charge of their own and do not count against your rate limit. The terminal `done` summary's `credits_used` and `refunded` fields are the source of truth for what you were actually charged. See [Credits](/docs/credits.md) and [Endpoint pricing](/docs/endpoint-pricing.md). ## Latency and failure modes Streaming consumers see progress within about a second of the first source finishing. Sync consumers wait for the whole pipeline, typically 20 to 30 seconds: allow a 120-second client timeout. - **A source times out or errors:** the pipeline continues and the source lands in `sources_failed` with `data.platforms..status` `failed`. SSE gets a `source_failed` chunk. - **A source's endpoint is failing repeatedly:** it is skipped without a call. `sources_failed` says `skipped: breaker_open`, and `data.platforms..status` is `skipped`. This holds for a source you named in `sources` too. - **A comment fetch fails for one result:** non-fatal. That row just has no `top_comments`. - **An author has asked not to be served:** rows by an account on our suppression list are never delivered, whether the answer is a fresh search, a cache hit or a replay, and that account's comments are left out of `top_comments`. If the list cannot be read, the rows are served without the check and `data._warnings` says so. The [Public Data Notice](/legal/public-data-notice) explains how a person asks to be suppressed. - **`lookback_days` over 90:** date-limited sources (`tiktok`, `instagram`, `threads`, `pinterest`, `reddit`) can under-cover the window, and SSE emits a `warning`. - **Pinterest** may return empty even on broad queries, because of an upstream vendor issue. ## Next steps - [Universal news search](/docs/search/news.md): The same idea for news, planned into angles and localized per country. - [Streaming](/docs/streaming.md): Consuming SSE across the API. - [Prism](/docs/prism.md): Cross-platform composites for brand, creator, and commerce questions.