# Hybrid search-then-enrich (/docs/recipes/search-then-enrich)

Find every relevant TikTok and Instagram post about a topic with universal search, then pull all their transcripts into one LLM-ready corpus. 120 credits per run.

You will build a deep-research pipeline that finds the most relevant TikTok and Instagram posts about a topic, pulls the transcript of every one of them, and concatenates the result into a single corpus ready for LLM analysis.

  **Cost per run:** 120 credits (20 for the search, plus 10 enriched posts at 10
  credits each). That is a deep-research budget, and it produces a real corpus
  rather than a feed. Tune `POSTS_PER_SOURCE` to trade depth for cost.

## How do you build a deep-research pipeline over social media?

Chain SocialCrawl's two big primitives. `GET /v1/search/everywhere` finds where the conversation is happening across 15 platforms, and the per-platform transcript endpoints pull what was actually said. Universal search returns the canonical post URL for every result, which feeds straight into the transcript endpoints with no string-massaging.

Step 1 passes `relevance=filter`, which drops the results the ranking step judged off-topic for the query, inside the flat 20 credits. Every off-topic post it drops is a 10-credit transcript you do not buy.

Search results tell you a conversation exists; they do not tell you what is in it. A snippet and a title are not enough for real research, and the full spoken content of the top posts is.

## What you need

Three endpoints, one pipeline:

| Endpoint                             | Role                    | Cost      |
| ------------------------------------ | ----------------------- | --------- |
| `GET /v1/search/everywhere`          | Universal social search | 20cr flat |
| `GET /v1/tiktok/post/transcript`     | TikTok transcript       | 10cr each |
| `GET /v1/instagram/media/transcript` | Instagram transcript    | 10cr each |

## The code

```typescript title="TypeScript"
// recipe-deep-research.ts
// Finds top posts on a topic, fetches their transcripts, returns the corpus.
// Run with: SOCIALCRAWL_KEY=sc_... npx tsx recipe-deep-research.ts

const KEY = process.env.SOCIALCRAWL_KEY;
if (!KEY) throw new Error("Set SOCIALCRAWL_KEY");

const BASE = "https://www.socialcrawl.dev/v1";
const topic = "ozempic side effects";
const POSTS_PER_SOURCE = 5;

// ── Step 1: Universal search, sync mode, scoped to TikTok + Instagram ─────
async function search(query: string) {
  const url = new URL(`${BASE}/search/everywhere`);
  url.searchParams.set("query", query);
  url.searchParams.set("sources", "tiktok,instagram");
  url.searchParams.set("lookback_days", "30");
  // Drop off-topic results before you pay 10 credits a transcript. Free.
  url.searchParams.set("relevance", "filter");

  const res = await fetch(url, {
    headers: { "x-api-key": KEY!, accept: "application/json" },
  });
  const json = (await res.json()) as {
    success: boolean;
    data: { items: Array<{ source: string; url: string; title: string }> };
    credits_used: number;
  };
  if (!json.success) throw new Error("search failed");
  return json.data.items;
}

// ── Step 2: Per-platform transcript fan-out ───────────────────────────────
type Platform = "tiktok" | "instagram";
const TRANSCRIPT_PATH: Record<Platform, string> = {
  tiktok: "tiktok/post/transcript",
  instagram: "instagram/media/transcript",
};

// See the video transcription recipe, transcript shapes differ per platform.
// This adapter covers the two used here; extend for the others as needed.
function extractText(
  platform: Platform,
  data: Record<string, unknown>,
): string {
  if (platform === "tiktok") return (data.transcript as string) ?? "";
  if (platform === "instagram") {
    const t = data.transcripts as Array<{ text?: string }> | undefined;
    return t?.[0]?.text ?? "";
  }
  return (data.transcript_only_text as string) ?? "";
}

async function getTranscript(platform: Platform, postUrl: string) {
  const url = new URL(`${BASE}/${TRANSCRIPT_PATH[platform]}`);
  url.searchParams.set("url", postUrl);
  const res = await fetch(url, { headers: { "x-api-key": KEY! } });
  const json = (await res.json()) as {
    success: boolean;
    data?: Record<string, unknown>;
    error?: { message: string };
  };
  if (!json.success) {
    console.warn(`skipped ${postUrl}: ${json.error?.message}`);
    return null;
  }
  return { text: extractText(platform, json.data!) };
}

// ── Step 3: Pipeline ──────────────────────────────────────────────────────
const items = await search(topic);

const tiktoks = items
  .filter((i) => i.source === "tiktok")
  .slice(0, POSTS_PER_SOURCE);
const instagrams = items
  .filter((i) => i.source === "instagram")
  .slice(0, POSTS_PER_SOURCE);

const transcripts = await Promise.all([
  ...tiktoks.map(async (item) => ({
    title: item.title,
    url: item.url,
    transcript: await getTranscript("tiktok", item.url),
  })),
  ...instagrams.map(async (item) => ({
    title: item.title,
    url: item.url,
    transcript: await getTranscript("instagram", item.url),
  })),
]);

const corpus = transcripts
  .filter((t) => t.transcript !== null && t.transcript.text.length > 0)
  .map((t) => `# ${t.title}\n${t.url}\n\n${t.transcript!.text}`)
  .join("\n\n---\n\n");

console.log(`corpus length: ${corpus.length} chars`);
console.log(corpus.slice(0, 500), "…");
```

## What you get back

```jsonc title="JSON"
// The final `corpus` string after Step 3, markdown-formatted, ready for an LLM:
// # I tried Ozempic for 90 days, here's what happened
// https://www.tiktok.com/@drmike/video/7314...
//
// So I want to talk about what nobody tells you about Ozempic...
//
// ---
//
// # The Ozempic side effect doctors won't mention
// https://www.instagram.com/reel/CzA1234abcd/
//
// Three weeks in I started experiencing what they call...
//
// ---
//
// (8 more transcripts concatenated)
```

## What to change

- **`topic`**: any subject with public video discussion.
- **`POSTS_PER_SOURCE`**: this is the price dial. Five per source is 10 transcripts and 100 credits of enrichment; drop it to 2 and the run costs 60 credits total.
- **The sources**: swap `sources=tiktok,instagram` for `sources=youtube,reddit` and update `TRANSCRIPT_PATH`. The same pipeline works for any combination of the seven transcript-capable platforms.
- **Narrow the candidate set**: layer `from_date`, `to_date`, or `exclude` onto Step 1, on top of `relevance=filter`, so you spend transcript credits on the right posts.
- **Take the comments too**: `GET /v1/prism/video-intel?url=...&include=transcript` returns a TikTok, Instagram, YouTube or Rumble video's detail, stats, top comments and transcript in one call. It costs 5 credits, plus 10 for the transcript, which is refunded when the video has none.
- **The last mile**: feed `corpus` to your LLM and ask for a structured summary, a sentiment breakdown, or a claim list. That is where the 120-credit budget pays off.

## Related

- [Video transcription](/docs/recipes/video-transcription-api.md): The transcript endpoints and shape adapter, in full.
- [Sentiment analysis](/docs/recipes/sentiment-analysis.md): The same LLM step, applied to comments instead of transcripts.
- [Social listening pulse-check](/docs/recipes/social-listening-pulse-check.md): The streaming version of the search that starts this pipeline.
- [Universal social search](/platforms/search/everywhere): Advanced filters you can layer onto step 1.
- [Quickstart](/docs/quickstart.md): New here? Get a key and make your first request.