# Caching (/docs/caching)

How SocialCrawl caches responses, why cache hits are free, freshness windows per data type, how to accept an older copy for free or require a younger one with max_age_s, the shared-cache model, and how to force a live fetch

Every successful response from a cache-enabled endpoint is cached, and the cache is shared. Cache hits cost 0 credits, which makes caching the main lever for controlling both spend and latency.

A minority of endpoints disable caching and run live on every request. Each endpoint's exact cache contract is stated on its own page and in [`llms.txt`](/llms.txt).

## Caching is on by default

There is nothing to enable or configure. When you call any `/v1/*` endpoint:

1. If a warm copy exists for the same endpoint and the same parameters, you get it instantly for **0 credits** (`X-Cache: HIT`).
2. If not, the source is fetched live, the endpoint's normal cost is billed (`X-Cache: MISS`), and the result is stored for the next caller.

You never pay twice for the same data within its freshness window.

## Reading the cache signals

Every response tells you whether it came from cache, in both the body and the headers:

| Signal                    | Where           | Meaning                                                |
| ------------------------- | --------------- | ------------------------------------------------------ |
| `"cached": true \| false` | response body   | Whether this response was served from cache            |
| `"age_s": 340`            | response body   | On a hit only: seconds since the copy was fetched      |
| `X-Cache-Age: 340`        | response header | The same age, on a hit only                            |
| `X-Cache: HIT \| MISS`    | response header | `HIT` = from cache (0 credits), `MISS` = live (billed) |
| `X-Credits-Used`          | response header | `0` on a hit, the endpoint cost on a miss              |

`age_s` is absent on a live call. It is `null` on the rare hit whose copy was stored before ages were recorded, and `X-Cache-Age` is then left out. The standard `Age` header on our responses is set by our CDN, not by the API, and is not the age of the copy.

A cache hit deducts **0 credits**. No transaction row is written and your balance is unchanged. The upstream provider was already paid when the original was fetched, so charging again for the same bytes would be double-billing. Idempotent replays are free for the same reason. See [Credits](/docs/credits.md).

## Freshness windows

How long an entry stays warm depends on the kind of data. After the window passes, the next call is a live `MISS` that re-warms the entry for the next caller:

| Data type                       | Cache window |
| ------------------------------- | ------------ |
| Profiles                        | ~15 minutes  |
| Posts                           | ~10 minutes  |
| Comments                        | ~5 minutes   |
| Search results                  | ~2 minutes   |
| Analytics / audience            | ~30 minutes  |
| Immutable content (transcripts) | ~30 days     |

Shorter windows on fast-moving data (comments, search) keep results current. Longer windows on slower-moving data (profiles, analytics) maximise free hits. Single-video transcripts are immutable once published, so they stay warm for 30 days. Empty transcripts are never cache-written.

Those are the category defaults, and a handful of endpoints override them where the data justifies it:

| Override               | Window     | Instead of                      |
| ---------------------- | ---------- | ------------------------------- |
| Instagram music search | 6 hours    | the 2-minute search default     |
| Google Finance quotes  | 60 seconds | the 30-minute analytics default |
| Google News search     | 5 minutes  | the 2-minute search default     |

Around 35 endpoints do not cache at all. Do not assume a category default when you are budgeting a specific job. The effective window for a given endpoint is stated on that endpoint's page, and in [`llms.txt`](/llms.txt) for machine consumers.

## Choose how old a copy you accept

The windows above are what a plain call accepts. Send `max_age_s` to set it yourself, in seconds, on any cached `GET`:

```bash title="cURL"
curl 'https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio&max_age_s=3600' \
  -H "x-api-key: YOUR_API_KEY"
```

- **Accept an older copy, for free.** Profile and post copies are kept for an hour, past their 15- and 10-minute windows. With `max_age_s=3600`, a copy up to an hour old is served as a cache hit for 0 credits, with its age in `age_s`. This is the one to use when you poll the same profiles or posts and an answer a few minutes older is fine.
- **Require a younger copy.** `max_age_s=60` refuses a copy older than a minute: the call fetches fresh, is billed as a miss, and writes the new copy back for the next caller.
- **Always fetch live.** `max_age_s=0` accepts no cached copy, the same as `Cache-Control: no-cache`.

Other categories keep their copies for exactly their window, so `max_age_s` can tighten them but cannot reach further back. A value past what an endpoint keeps is read as that limit, and `data._warnings` says so, for example `max_age_s=7200 was read as 3600: this endpoint keeps a cached copy for at most 3600 seconds.` On an endpoint that does not cache, `max_age_s` does nothing and `data._warnings` says that too.

`max_age_s` takes a whole number, 0 or more. Any other value is a free `400`. It is never part of the cache key, so `max_age_s=3600` and a plain call read the same copy. `max_age` on `web/scrape` and `web/extract` is a different parameter: the scraper's own window, in milliseconds.

## The cache is shared

The cache key is derived from the endpoint and its **sorted, normalised query parameters**, not from your API key and not from your account. Three consequences follow:

- **Shared hits.** Any identical request already served inside the freshness window comes back to you as a free hit, whether it was your own earlier call, another job on your account, or another customer's traffic. Public platform data is the same bytes for everyone, so the upstream is paid once and the fetch is billed once.
- **Freshness is a property of the entry, not of your traffic.** "Nobody on my team has called this handle recently" does not mean you are getting a live fetch. When you need a guaranteed-live read, send `Cache-Control: no-cache`.
- **Unknown params cannot bust it.** A query parameter the endpoint does not declare is dropped before cache lookup, billing, and upstream dispatch, so it never reaches the cache key. `data._warnings` names it, with the parameter you probably meant when there is one. Empty undeclared values are treated as omitted and get no line.

  A cache-buster like `?_cb=12345` is dropped and named in `data._warnings`;
  it does **not** force a fresh call or turn a free hit into a paid miss. The
  `Cache-Control: no-cache` header and `max_age_s=0` are the supported
  bypasses.

## Force a fresh fetch

When you need guaranteed-live data, send the standard `Cache-Control: no-cache` request header:

**cURL**

```bash title="cURL"
curl 'https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Cache-Control: no-cache"
```

**TypeScript**

```ts title="TypeScript"
const res = await fetch(
  "https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio",
  {
    headers: {
      "x-api-key": process.env.SOCIALCRAWL_API_KEY!,
      "Cache-Control": "no-cache",
    },
  },
);

console.log(res.headers.get("x-cache"), res.headers.get("x-credits-used"));
```

**Python**

```python title="Python"
import os

import requests

res = requests.get(
    "https://www.socialcrawl.dev/v1/tiktok/profile",
    params={"handle": "charlidamelio"},
    headers={
        "x-api-key": os.environ["SOCIALCRAWL_API_KEY"],
        "Cache-Control": "no-cache",
    },
)

print(res.headers["X-Cache"], res.headers["X-Credits-Used"])
```

The request skips the cache lookup and fetches live from the source. It is billed at the endpoint's normal cost, a forced `MISS`, so `X-Cache: MISS` and a non-zero `X-Credits-Used`. The fresh result is written back to the cache. The pattern is "pay once to refresh, then ride free hits until the window expires".

Details worth knowing:

- The header affects only the single request it rides on. It never disables caching for your key.
- Only the `no-cache` directive triggers it. `Cache-Control: no-store` on its own does not. A composite value like `no-store, no-cache` does, and matching is case-insensitive.
- For a client that cannot set headers (an agent's fetch tool, a no-code HTTP step), `max_age_s=0` does the same. Its name says what it does, so it cannot be sent by accident. A cache-buster such as `?fresh=1` or `?_cb=` is still dropped and does nothing.
- `Idempotency-Key` takes precedence. A replayed idempotent request returns the stored response even if it also carries `Cache-Control: no-cache`. See [Idempotent requests](/docs/idempotency.md).

## When should I bypass the cache?

Most workloads should leave caching on and let hits accrue. Reach for `Cache-Control: no-cache` only when staleness is a correctness problem, for example:

- Reconciliation or audit jobs that must reflect the source at the exact moment of the call.
- Reading data back immediately after a user action or webhook changed it upstream.
- Debugging a field-map or freshness question where you want to compare cached against live.

When "fresh within the last minute" is enough, send `max_age_s=60` instead: you pay only when the copy is older than that. For everything else, reusing identical requests within the freshness window is the cheapest and fastest path, and `max_age_s=3600` on profiles and posts makes it cheaper still.

`Cache-Control` and `Idempotency-Key` are listed as optional headers on every endpoint in the [API Reference](/docs/api-reference.md) "Try it" panel, so you can experiment against your own key and watch `X-Cache` and `X-Credits-Used` change.

## A row join reads the cache row by row

Some list endpoints take an opt-in join token, such as `include=profile` on the LinkedIn people lanes, which looks each row up separately behind your one request. Those lookups read the same shared cache a direct call would, so a row whose subject is already warm is filled for 0 extra credits and only the rows that had to be fetched are charged. Ask for the same joined page again inside the freshness window and the whole response is an ordinary cache hit at 0 credits. `data.hydration` reports how many rows came from cache. LinkedIn member and company URLs are normalised before the key is built, so a country subdomain, a trailing slash, and `?trk=` tracking parameters all reach the same entry. See [Credits](/docs/credits.md) for how the held credits settle.

## Labels on a cache hit

A cached page still carries its SocialCrawl [labels](/docs/labels.md), at 0 credits. Labels are attached on every read, hit or miss, from a separate per-row store: one entry per comment, post or review, kept for 30 days and keyed by the row's text, so an edited comment is judged again. A row that was still `pending` when the first response was sent is usually labelled by the time the next caller, or your own repeat call, reads the cached page.

The four universal search endpoints are the exception: their judgments are stored with the cached response, so a result that was not judged in time stays `null` until that cached response expires. `judgments=off` is part of the cache key, so a request with it never receives a labelled cached page.

## Next steps

- [Credits](/docs/credits.md): What a call costs, and when a charge is refunded.
- [Response schema](/docs/response-schema.md): Cache headers, the envelope, and idempotent replays.
- [Endpoint pricing](/docs/endpoint-pricing.md): Per-endpoint costs and cache contracts.
- [Rate limits](/docs/rate-limits.md): Request-per-minute and concurrency budgets.