Universal social search

Send one query to GET /v1/search/everywhere and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 20 sources across 15 platforms. An LLM plans the query, results are fused and reranked, near-duplicates are clustered, and every result carries the top comments from its source. A cited web answer, and on request a summary of X posts, come back beside the ranked results in data.synthesis.

Returns one JSON envelope by default, or a live SSE stream when you ask for one. Flat 20 credits per call.

curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews" \
  -H "x-api-key: YOUR_API_KEY"

How the fan-out works

The pipeline runs in five stages, and every one of them is visible in the response.

1. Plan. A plan is ready at once (intent, freshness and source weights), and an LLM refines it in parallel; your query itself is what every source is sent. A fast planner scores per-source relevance at the same time, and a source it judges irrelevant for the query is pruned, so its results are not ranked and it leaves the coverage denominator. The planner fires an abort on a pruned source, but a source that had already answered before the abort landed still reports its items, so treat a prune as "this source was not ranked" rather than as proof that no request was made. A source you name in sources is never pruned.
2. Fan out. Every surviving source runs in parallel through the endpoint you could call yourself (the table below names it), with that endpoint's own fallbacks. A source whose endpoint keeps failing is skipped for a short while instead of being called, and is reported as skipped: breaker_open. Beside other sources each source has 12 seconds (6 seconds for twitter, 20 seconds for facebook, which runs only when you ask for it); when you search one platform it has 25 seconds. The X summary, twitter-ai-search, has up to 45 seconds because writing it is slow: the stream waits for it before done, and the JSON answer includes it only when it is ready by the time the rest of the answer is.
3. Fuse and dedupe. Results are merged, and the same item found by two sources becomes one candidate with multi-source provenance and a higher fused score.
4. Enrich and rerank. Top comments are fetched in parallel with a judged rerank that has 3 seconds. Scores already judged are kept when it runs out. Each result's stance toward your query is judged beside the rerank, and its relevance score comes from the rerank's own answers.
5. Cluster. Near-duplicates and same-story groupings are collapsed into clusters, when the resolved intent is clusterable.

Sources

The default fan-out hits every source below except naver, which joins when the query contains Hangul, facebook, which runs only when you ask for it with include=facebook (added to the default sources) or name it in sources, and twitter-ai-search, the X summary, which runs only with include=synthesis. Use sources / exclude to narrow it.

The three hashtag lanes fire only in hashtag mode. naver fires when the query contains Hangul, or on any query when you name it in sources. X and the web each have a second source (twitter-ai-search and perplexity), and TikTok, Instagram and YouTube each have a hashtag lane, so the 15 platforms make up to 20 sources on one call.

Each source is its own endpoint. A source calls the endpoint in the Reads column with your query, through that endpoint's own fallbacks, so a source answers whenever a direct call to the endpoint would. The calls a search makes to these endpoints are part of its 20 credits: they are not billed separately and do not count against your rate limit. They are paced per caller, and a source call past that pace is skipped and reported as skipped: leg_pace. Polymarket has the market search the polymarket source reads.

Facebook. facebook searches public Facebook posts by keyword, newest first inside your date window (lookback_days, or from_date / to_date), and returns about 5 posts. Each row has the post text in body, the permalink in url, the page or person who posted it in author, the publish time in published_at and like, comment and share counts in engagement; the account URL is in metadata.author_url, so you can follow it to /v1/facebook/profile. It is not in the default fan-out. include=facebook adds it to the default sources; sources=facebook (alone) or sources=facebook,reddit runs only the sources you name. Beside other sources it has up to 20 seconds instead of 12, so a request that includes it can take longer. It stays inside the flat 20 credits.

Korean queries. naver searches Naver blog posts, cafe communities and KnowledgeiN questions in one lane. Each Naver row carries its corpus in metadata.corpus. Blog rows have the posting day; KnowledgeiN rows have an estimated date, so their date_confidence is low; cafe rows have no date.

Dates. Check each row's date_confidence before you filter on published_at. low means the date is an estimate: a KnowledgeiN estimate, or a YouTube row dated from a "3 years ago" or "2 months ago" label.

X posts and the X summary. twitter returns X posts as ranked rows, like any other source, and is in the default fan-out. twitter-ai-search writes a summary of X posts with its citations, which takes 15 to 45 seconds, so it runs only when you ask for it with include=synthesis. The summary arrives in data.synthesis.x, never as ranked rows. The stream waits for it. The JSON answer includes it only when it is ready by the time the rest of the answer is; otherwise sources_failed lists twitter-ai-search with the reason not ready when the rest of the answer was. For the X summary beside other sources, use the stream.

The web results and the web answer. web returns web search results as ranked rows; their body is plain text of up to 1,500 characters. perplexity writes a cited web answer, which arrives in data.synthesis.web. Its rows stay in items_by_source.perplexity and are not ranked.

Other names and platform shorthands. x is another name for twitter, and tavily for web, so sources=x and sources=twitter return X posts. twitter-ai-search is still accepted in sources, where it means twitter plus include=synthesis; in exclude it leaves out only the summary. A bare platform name resolves to its whole group: youtube, instagram and tiktok each cover their -hashtag lane too, so exclude=youtube cannot leak YouTube content back in through youtube-hashtag. Exact lane names stay individually addressable. An unknown name in sources or exclude is a 400 that enumerates the valid names, never a silent drop. A parameter this endpoint does not take is ignored, never refused: the answer carries a data._warnings line naming it, with a did-you-mean when a real parameter is close.

Keyword mode and hashtag mode

There is no user-facing mode switch. The planner decides:

• Keyword mode (default). A natural-language query fans out to the base keyword-search sources.
• Hashtag mode. The three hashtag siblings are added automatically in either of two cases. One tag or one word: the query is a single token, for example query=%23thrifting or query=thrifting, and a leading # is stripped before dispatch. Topic term: for a multi-word topic query the planner extracts a topic term (makeup tutorials yields the tag makeup) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query.

When the same TikTok, Instagram or YouTube post appears in both streams, fusion merges the two into one candidate.

If the planner times out or fails, explicit #tag queries still route to hashtag mode deterministically, and multi-word topic queries degrade to keyword-only.

The query value is passed to the planner as-is. There are no special search operators: no site:, no boolean AND/OR, no quoting. The single input with special meaning is a leading #.

Request parameters

from and to are accepted as aliases for from_date and to_date, so a date window carried over from /v1/search/news (which spells them from / to) is honoured here rather than silently dropped. See One date vocabulary across the four lanes.

Passing both sources and exclude returns 400 INVALID_REQUEST. Check data.sources_called (or the SSE done summary) against what you asked for.

Filter examples

# Narrow to specific sources
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews" \
  -H "x-api-key: YOUR_API_KEY"

# Exclude noisy sources
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket" \
  -H "x-api-key: YOUR_API_KEY"

# Explicit date window
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30" \
  -H "x-api-key: YOUR_API_KEY"

# Recency window
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7" \
  -H "x-api-key: YOUR_API_KEY"

# Hashtag mode (leading #, URL-encoded as %23)
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting" \
  -H "x-api-key: YOUR_API_KEY"

In a real #thrifting call the plan intent resolves to how_to, and sources_called includes tiktok-hashtag, instagram-hashtag and youtube-hashtag alongside the base sources.

What you get back

By default (Accept: application/json) you get one envelope. data carries:

The web answer and the X summary

data.synthesis holds the two written answers, so you can show them above the ranked results without parsing rows:

{
  web: { text: string, citations: Citation[] } | null, // the perplexity answer
  x:   { text: string, citations: Citation[] } | null  // the X summary, with include=synthesis
}
// Citation
{
  url: string,     // the page the answer cites, in citation order
  also_on?: string // the id of the `web` row for the same page, when the web results found it too
}

also_on lets you link a citation to the matching web result instead of showing the page twice.

Stance and relevance on every result

Each result in items carries two judgments by default, free:

"computed": {
  "stance": {
    "label": "neutral",
    "probabilities": { "positive": 0.01, "negative": 0, "neutral": 0.92, "no_opinion": 0.07 },
    "confidence": 0.9
  },
  "relevance": { "score": 0.93, "on_topic": true, "entity": 0.97 }
}

stance.label is what the post, with any replies shown, says about your query: positive, negative, neutral, no_opinion, or null when the judgment is under the confidence floor, which counts as uncertain in data.stance_split, never as a stance. relevance.score runs from 0 to 1, and on_topic: false marks exactly the results relevance=filter would drop. Both are judged inside the request under a deadline that runs beside the ranking step, so they add no wait; a result not judged in time is null for the life of the cached response. Labels states how stance was measured.

Real-people comments

For every result whose source exposes a comments endpoint, the pipeline fetches the top-scoring comments and attaches them at data.items[i].source_items[0].metadata.top_comments[]. No second API call.

Each TopComment has this shape:

{
  score: number | null,    // upvote / like / points count, source-specific
  excerpt: string,         // up to 300 chars, HTML stripped
  author: string | null,   // null when [deleted] / [removed]
  url: string | null,      // direct comment URL when the source exposes one
  date: string | null      // ISO timestamp when available
}

Comments are sorted by score descending: up to 15 on each of the top 10 results and 5 after, 300 characters per excerpt. Sources without a comments endpoint (twitter, twitter-ai-search, threads, pinterest, polymarket, perplexity, web, linkedin, rumble, naver) render no top_comments. facebook rows carry none either: the lane returns the comment count in engagement.comments, and /v1/facebook/post/comments reads the comments themselves.

Streaming

Add Accept: text/event-stream and you get a live stream instead of one envelope. First byte lands in well under a second, typically around 300ms on the meta chunk, and fast sources stream their items while slow ones are still running.

curl -N "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Accept: text/event-stream"

Each event is one data: <json>\n\n frame, discriminated by type. The order guarantees:

• meta is first, emitted as soon as the request is parsed. It carries the deterministic plan and the full sources_planned list.
• source_started fires once per source before that source's adapter runs. Pruned or failing sources still emit source_started, then a source_pruned or source_failed.
• source_pruned fires when the fast planner judged a source irrelevant for the query, and the source appears in sources_pruned (never in sources_failed, and never in the coverage denominator). The planner fires an abort on the source, but a source that had already answered before the abort landed still emits its items chunk, so treat source_pruned as "do not rank this source" rather than as proof that no request was made.
• items yield in completion order, so fast sources land first. The web answer and the X summary arrive as items chunks for perplexity and twitter-ai-search: the answer is the row whose metadata.role is synthesis, and each cited page is a row whose metadata.role is citation. The stream has no data.synthesis block.
• plan_refined is conditional: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged.
• ranked_partial streams repeatedly as the reranker scores batches. Each chunk carries only the scores that settled since the previous one, so merge by candidate_id rather than replacing your map. scored and total carry cumulative progress for a percentage indicator.
• comments_enriched is per-candidate. Key on candidate_id to merge comments into the row already on screen. These always land before done, and may interleave with ranked_partial / ranked_final.
• done is always last, carrying the run summary. On a thrown error you get a terminal error followed by done with refunded: true.

JSON answers are cached for 120 seconds, and a cache hit is free. Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache. The two are keyed separately.

Billing

Flat 20 credits per call, deducted up front. Two automatic refund paths:

Stance, relevance and relevance=filter are included in the 20 credits: a call costs the same with judgments=off. So are the web answer, the X summary with include=synthesis, and every call the search makes to a source's endpoint: those calls add no charge of their own and do not count against your rate limit. The terminal done summary's credits_used and refunded fields are the source of truth for what you were actually charged. See Credits and Endpoint pricing.

Latency and failure modes

Streaming consumers see progress within about a second of the first source finishing. Sync consumers wait for the whole pipeline, typically 20 to 30 seconds: allow a 120-second client timeout.

• A source times out or errors: the pipeline continues and the source lands in sources_failed with data.platforms.<platform>.status failed. SSE gets a source_failed chunk.
• A source's endpoint is failing repeatedly: it is skipped without a call. sources_failed says skipped: breaker_open, and data.platforms.<platform>.status is skipped. This holds for a source you named in sources too.
• A comment fetch fails for one result: non-fatal. That row just has no top_comments.
• An author has asked not to be served: rows by an account on our suppression list are never delivered, whether the answer is a fresh search, a cache hit or a replay, and that account's comments are left out of top_comments. If the list cannot be read, the rows are served without the check and data._warnings says so. The Public Data Notice explains how a person asks to be suppressed.
• lookback_days over 90: date-limited sources (tiktok, instagram, threads, pinterest, reddit) can under-cover the window, and SSE emits a warning.
• Pinterest may return empty even on broad queries, because of an upstream vendor issue.

Next steps