# Universal news search (/docs/search/news) One query, planned into angles, localized into up to 12 country editions and searched across two independent news indexes in parallel, merged into a single deduplicated article list with honest per-leg provenance. JSON or SSE. Send one query to `GET /v1/search/news` and get back a planned, localized, multi-country news search in a single call. An LLM plans your query into search angles and localizes each angle into the language of every requested country edition, the lane fans out up to 12 parallel legs per engine, and the articles merge into one deduplicated list. Two independent news indexes are available through `engines`: `google` (the default, 50 country editions) and `bing` (49 of them). **They barely overlap.** On measured fixtures across four locales, none of the articles the `bing` engine returned appeared anywhere in the `google` results for the same query, so asking for both roughly doubles the distinct articles and the distinct publishers a query returns. It is also how you get article text: the `google` engine rarely carries a snippet, the `bing` engine carries one on effectively every row. Every leg reports truthful provenance: `query_source` tells you whether its keyword was `translated`, confirmed `original`, or a `fallback_original` after a localization failure. Every article carries `engine` and a `published_at_precision` of `exact` or `approximate`, because the two indexes do not report publish times with the same fidelity. Returns one JSON envelope by default, or a live SSE stream when you ask for one. ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/search/news?query=samsung+galaxy+launch&countries=KR,US,JP&engines=google,bing" \ -H "x-api-key: YOUR_API_KEY" ``` This lane is metered, **2-62 credits**, not flat. On the default engine a 2-credit base covers planning and localization, then 1 credit per leg that returned at least one article, with an upfront hold of `2 + min(5 x countries, max_legs, 12)` and a ceiling of 14. Adding the `bing` engine adds 1 credit per 5 articles that engine returns, because that index is priced per article rather than per call. The unused remainder of the hold is refunded automatically, so the `credits_used` you see in the response is what you actually paid. Read `credits_used` rather than assuming a price. On a stream, the terminal `done` summary is the only place the net charge appears. ## How a request becomes legs 1. **Plan.** One combined LLM call turns your query into up to 5 weighted search angles and localizes each angle into the default language of every requested country. The call is fail-open: on timeout or error the lane degrades to a single identity angle (your query, clipped to a keyword) and every leg is stamped `fallback_original`, never silently. 2. **Expand.** Angles cross with countries into legs, capped at `min(5 x countries, max_legs, 12)`. Identical (country, language, keyword) legs collapse before dispatch, so two angles that localize to the same keyword never bill twice. 3. **Fan out.** Every leg runs in parallel, once per engine you asked for. A `google` leg is an in-process Google News call using the primary source with automatic fallback and empty-result rescue, unless the request is pinned. A `bing` leg is a single call to the second index, paced at 6 concurrent legs. 4. **Assemble.** Articles dedupe across legs and across engines on `canonical_url ?? url`. The first leg's copy wins the body, and every later leg that returned the same article is appended to that item's `legs_returned` provenance list. `sort` orders the merged list. ### The `query_source` vocabulary Every leg carries one of three values, and they are part of the public contract: | Value | Meaning | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `translated` | The localizer produced a different keyword for that language. | | `original` | The localizer confirmed the planned keyword is already correct for that language. This value is never stamped on a failure path. | | `fallback_original` | Localization was unavailable for that language, so the leg searched the planned keyword as-is. | Because `original` is never stamped on a failure path, a total localizer outage is always distinguishable from "keyword already correct". ## Request parameters | Param | Required | Type | Default | Description | | ------------ | -------- | ------------------------------------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | yes | string (1 to 500 chars) | none | News topic or question. Boolean AND/OR/NOT and quoted phrases are supported; Google advanced operators (`site:`, `intitle:`, `before:`) are rejected with a 400. | | `countries` | no | CSV of ISO alpha-2 codes | `US` | 1 to 12 of the 50 supported country editions (e.g. `KR,US,JP`). Each country searches its news edition in that edition's default language; duplicates collapse. | | `engines` | no | CSV of `google` \| `bing` | `google` | News indexes to search. Naming both runs the same plan on each and merges the results. `bing` serves 49 of the 50 editions (not `CN`) and is priced per article. | | `sort` | no | `relevance` \| `date` | `relevance` | Ordering of the merged articles. `date` is newest first, with undated articles last. | | `time_range` | no | `day` \| `week` \| `month` \| `year` | `day` | Recency window. Ignored when `from`/`to` is present (explicit dates win). | | `from` | no | `YYYY-MM-DD` or Unix | none | Lower publish-date bound. Pins every leg to the primary news source. | | `to` | no | `YYYY-MM-DD` or Unix | none | Upper publish-date bound. Pins every leg to the primary news source. | | `publisher` | no | bare domain | none | Publisher filter, e.g. `bbc.com` (no scheme, no path). Pins every leg to the primary news source. | | `depth` | no | integer 10-100, steps of 10 | `10` | Articles requested per leg. The `google` engine is priced per call, so depth is free there; the `bing` engine is priced per article and caps its own legs at 20. | | `max_legs` | no | integer 1-12 | `12` | Hard cap on planned legs, applied **per engine**. The upfront hold shrinks with it, so it is your cost-control knob. | | `judgments` | no | `on` \| `off` | `on` | `on` groups the articles into same-event stories (`data.stories`) under a short time budget. `items[]`, their order and the price are unchanged. `off` returns the response without the grouping. | | `group` | no | `stories` | none | Asks for the story grouping explicitly, with a longer time budget, for when you need it on every response. Free. JSON responses only. | `publisher`, `from`, and `to` pin every `google` leg to the primary Google News source: the fallback source cannot express those filters, and silently dropping a filter would return unfiltered data. Pinned requests trade away the fallback's empty-result rescue for filter correctness. Those three filters are **google-only**. The `bing` index cannot express them, so a request that asks for them without `google` in `engines` is rejected with a 400 rather than quietly returning unfiltered results under a filter you believe was applied. ## What you get back By default (`Accept: application/json`) you get one envelope. `data` carries: | Field | Type | Description | | --------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------ | | `query` | string | Your query as parsed. | | `search_status` | string | `succeeded` (every leg returned), `partial` (some legs failed), or `failed` (every leg failed). | | `plan` | object | The resolved plan: `topic`, `source` (`llm` or `identity`), `degraded`, `subqueries[]`, `notes[]`. | | `legs` | array | One truthful outcome row per dispatched leg. | | `items` | array | Deduplicated articles across all legs, each with `legs_returned` provenance. | | `total_articles` | number | `items.length`. | | `countries_searched` | array | The resolved country codes, first-seen order. | | `credits_charged` | number | The net charge for this call (base plus what each leg settled at). Matches the envelope's own `credits_used` and the `X-Credits-Used` header. | | `methodology_version` | string | `search-news/1.0`. | | `stories` | array or null | Articles about the same event grouped across editions, outlets and languages. `null` when grouping did not finish in time or failed. Absent with `judgments=off`. | | `story_grouping` | object | How many articles were judged, how many stories came out, the largest story, and `extra_credits` (always `0`). | | `_warnings` | array | Present only when something needs saying: `stories_pending` (grouping ran out of time) or `stories_unavailable` (grouping failed). | ### Same-event stories A search across several editions returns the same event many times: the Fed's rate decision reported by US, Korean and Japanese outlets in three languages. `data.stories` groups those articles into one story each, without touching `items[]`: ```json title="JSON" "stories": [ { "id": "story_1", "head_id": "https://www.reuters.com/markets/...", "member_ids": ["https://www.reuters.com/markets/...", "https://www.yna.co.kr/view/..."], "size": 2, "outlets": ["Reuters", "Yonhap"], "outlet_count": 2, "countries": ["US", "KR"], "languages": ["en", "ko"], "first_seen": "2026-09-17T18:02:00Z", "last_seen": "2026-09-17T21:40:00Z", "judged": true } ] ``` Every id is an article's `canonical_url` (or `url`), so you can join a story back to `items[]`. An article that could not be judged is its own story with `judged: false`. By default the grouping runs under a short time budget after the legs return; when it does not finish, `stories` is `null` for the life of that cached response and `data._warnings` says `stories_pending`. Send `group=stories` when you need it every time. The grouping tends to join a preview of an event with its outcome; see [Labels](/docs/labels.md#how-we-measure) for how it was measured. The event stream carries no grouping. Two outcomes worth handling explicitly: - **Zero articles with succeeded legs is a 200 success**, with an empty `items` and base-only billing. It is not an error. - **All legs failed is a `502 ALL_LEGS_FAILED`** with `ok: false`, and the full hold is refunded automatically. The 502 body is the standard error envelope on its own: per-leg detail (`legs`, `plan`) is not returned on this path, so ask for the SSE stream when you need to see which legs failed and why. **Every field on a legs[] row** `leg_id` (`angleId|COUNTRY` on the default engine, `angleId|COUNTRY|bing` on the second, so ids stay unique and every id that worked before this engine existed still works), `engine` (`google` or `bing`), `country`, `language`, `angle_id`, `angle_label`, `effective_query` (the keyword actually searched), `query_source`, `status` (`succeeded` / `failed` / `skipped`), `article_count`, `quarantined_count` (rows dropped as schema-invalid or for having no usable http(s) URL), `credits_used` (on `google`, 1 when the leg returned at least one article, else 0; on `bing`, 1 per 5 articles returned, else 0), `latency_ms`, and `error`. A `bing` leg for a country that index does not serve is `skipped` with the reason in `error` and `credits_used: 0`. A skipped leg never ran, so it neither counts as a success nor drags `search_status` down to `partial`. **Every field on an items[] article** `id` (upstream article id, `null` when the source omits one), `title`, `url` (always http(s)), `canonical_url` (lane-derived: fragment dropped, tracking params stripped, host lowercased), `source_name`, `domain`, `snippet`, `image_url`, `published_at` (UTC ISO or `null`, never fabricated), `published_at_precision`, `engine`, `rank`, `placement`, plus the provenance stamps from the leg that found it (`country_code`, `language_code`, `angle_id`, `effective_query`, `query_source`) and `legs_returned` (every leg that returned it). `published_at_precision` is `exact` when the index reported an absolute instant (the `google` engine) and `approximate` when it reported only a relative age like "5h" or "3d" and the instant was derived from it (the `bing` engine, where the true resolution is hours to days). Sort or window on `published_at` accordingly: the field is never fabricated, but on `bing` rows it is inferred rather than reported. ## Streaming Add `Accept: text/event-stream` and you get a live stream instead of one envelope. ```text title="Stream" { "type": "meta", "query": "...", "plan": {...}, "sources_planned": ["KR", "US", ...] } { "type": "plan_refined", "plan": {...}, "legs": [{ "leg_id": "sq0|KR", "country": "KR", "language": "ko", "keyword": "...", "query_source": "translated", "angle_id": "sq0", "angle_label": "..." }] } { "type": "source_started", "source": "sq0|KR" } { "type": "items", "source": "sq0|KR", "items": [...], "duration_ms": 1234 } { "type": "source_failed", "source": "sq1|US", "error": { "code": "...", "message": "..." } } { "type": "done", "summary": { "credits_used": 4, "refunded": true, "coverage": 1, ... } } ``` Order and guarantees: - **`meta` is first**, carrying the deterministic identity plan before the LLM call resolves. - **`plan_refined` is always emitted on this lane**, unlike [universal social search](/docs/search/everywhere.md), where it is conditional on the LLM changing the plan. It carries the final plan plus the expanded `legs[]` with per-leg `query_source`, so a consumer can render localization honesty (translated versus fallback chips) at plan time, before any leg settles. A degraded run still emits it, with every leg stamped `fallback_original`. - **Legs are keyed as `source` strings** in the form `angleId|COUNTRY`, for example `sq0|KR`. Angle ids are positional with an `sq` prefix. These are per-request keys, not the fixed source ids of universal social search, so that endpoint's client reducer is not suitable for news streams. Skipped legs, aborted before dispatch, surface as `source_failed` with `error.code: "aborted"`. - **`items` yield in completion order**, so fast editions land first. The deduplicated cross-leg view is only available in sync mode, or by deduplicating client-side on `canonical_url ?? url`. - **`done` is always last**, with `credits_used` (net), `refunded`, `coverage`, `sources_called` / `sources_succeeded` / `sources_failed`, `total_items`, and `partial_failure`. On a stream, `done` is the **only** place the net charge appears. Response headers are flushed before the run settles, so a streaming response carries no `X-Credits-Used` or `X-Credits-Remaining` header. Sync mode does carry them. Streaming responses are never cached and never touch the sync cache. They also cannot carry an `Idempotency-Key`: replay applies to sync JSON responses only, so a streaming request with a key is a free `400 INVALID_REQUEST`. Ask for JSON when you need a replay; see [Idempotency](/docs/idempotency.md). ## Billing Metered, **2-62 credits**. A 2-credit base covers planning and localization. Each `google` leg that returned at least one article is 1 credit; each `bing` leg is 1 credit per 5 articles it returned, because that index is priced per article rather than per call. The upfront hold is the base plus each planned leg's per-engine cost, which is `2 + min(5 x countries, max_legs, 12)` on the default engine, and the unused remainder is refunded automatically. | Scenario | Charge | | ------------------------------------------- | ------------------------------------------------- | | 3 countries, google only, all legs return | 2 + productive legs (up to 12 with `max_legs=12`) | | 1 country, both engines, 10 articles each | 2 + 1 (google) + 2 (bing, 10 articles / 5) | | Zero articles anywhere, legs succeeded | 2 (base only), 200 response | | A `bing` leg for a country it does not serve | 0, `skipped` with the reason in `error` | | Every leg failed | 0 (full automatic refund), `502 ALL_LEGS_FAILED` | | Cache hit (sync, within 5 minutes) | 0, `X-Cache: HIT` | | Idempotency-Key replay (sync) | 0, `X-Idempotent-Replay: true`, `Idempotent-Replayed: true` | Story grouping, by default or with `group=stories`, is free: it never adds to the hold or the charge. `X-Credits-Used` on the response, and `done.summary.credits_used` on a stream, is the source of truth for what you paid. Use `max_legs` to cap the worst case: `max_legs=3` caps the hold at 5 credits regardless of how many countries you request. Billing on a stream is charge-for-delivered. If you disconnect mid-stream you pay the base plus the productive legs whose `items` chunks you received, and the rest of the hold is refunded. On an internal error the whole hold is refunded and the terminal `done` reports `credits_used: 0`, `refunded: true`. ## Latency and failure modes The lane plans before it fans out, so the combined plan-and-localize call is on the critical path. Its budget scales with the number of distinct languages (8s base plus 0.7s per extra language, capped at 18s). On timeout the run degrades to the identity plan with every leg stamped `fallback_original` and a note in `plan.notes`. It does not fail. Legs then run fully in parallel, each bounded by the remaining route budget. Typical end to end: a few seconds for one country, 10 to 30 seconds for a wide multi-language fan-out. - **One leg times out or errors:** its row reports `status: "failed"` with an error class, sibling legs continue, and the leg bills 0. - **The localizer drops a language** (operator-laden, overlong, or script-mismatched keyword): that leg searches the planned keyword stamped `fallback_original`. - **Client disconnect over SSE:** work stops, undelivered legs are refunded, and the charge is settled exactly once. Supported countries are the 50 editions in the lane's catalog, and an unknown `countries` code returns an error message carrying the up-to-date list. Notable exclusions: Russia and Lebanon are not servable by the primary news source and are not in the catalog. The `bing` engine serves 49 of the 50; `CN` is skipped on that engine, for free. Adding the `bing` engine adds latency: measured end to end, a two-country search went from roughly 7 seconds on the default engine to between 8 and 24 seconds with both, because the second index is slower and is paced at 6 concurrent legs. Both stay inside the route budget. If a `bing` leg is throttled or fails, the `google` legs still answer, the call is a 200 `partial`, and the failed leg bills nothing. ## Next steps - [Universal social search](/docs/search/everywhere.md): The social sibling: one query fanned across social platforms. - [Google News](/docs/google_news.md): The single-edition news search each leg rides on. - [Credits](/docs/credits.md): Holds, settlements, and automatic refunds.