# Web page schema (/docs/schema/web-page)

The unified SocialCrawl web page schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.

Every SocialCrawl endpoint that returns a web page gives you this exact shape, whatever the source platform. Write your parser once and the same code reads web page data from every platform below. That is the unified schema: one contract instead of a dozen raw upstream formats.

## Field reference

| Field | Type | Nullable | Description |
| ----- | ---- | -------- | ----------- |
| `url` | `string` | Yes | Requested URL. |
| `final_url` | `string` | Yes | Final resolved URL after redirects. |
| `status_code` | `number` | Yes | HTTP status code observed while fetching the page. |
| `scrape_id` | `string` | Yes | Opaque scrape identifier for follow-up interactions. |
| `fetched_at` | `string` | Yes | Fetch timestamp when reported. |
| `published_at` | `string` | Yes | When the page was published: an ISO 8601 instant, or a YYYY-MM-DD date when only the day is printed. From the page's own article:published_time meta tag on a scrape, and from the news date or that tag on a search row. Null when the page states none. |
| `content.markdown` | `string` | Yes | Markdown body. |
| `content.html` | `string` | Yes | HTML body. |
| `content.raw_html` | `string` | Yes | Raw HTML body. |
| `content.summary` | `string` | Yes | Generated summary. |
| `media.screenshot_url` | `string` | Yes | Screenshot URL when requested. |
| `media.audio_url` | `string` | Yes | Audio URL. |
| `media.video_url` | `string` | Yes | Video URL. |
| `extraction` | `object` | Yes | Structured extraction result when requested. On `web/extract` every value is checked against the page text: a placeholder such as 'Not specified', or a value the page does not state, comes back as null and is listed in `extraction_check.removed`. |
| `extraction_check.checked` | `integer` | No | Non-null values examined. |
| `extraction_check.found_in_text` | `integer` | No | Values kept because they appear word for word on the page. |
| `extraction_check.confirmed` | `integer` | No | Values kept because a check found the page states them, for example in another language. |
| `extraction_check.removed` | `object[]` | No | Fields set to null, with the value the extractor gave. |
| `extraction_check.unverified` | `string[]` | No | Fields returned as extracted without a check. |
| `fetch.cache_state` | `string` | Yes | Upstream cache state when reported. |
| `fetch.cached_at` | `string` | Yes | Upstream cache timestamp when reported. |
| `fetch.proxy_tier` | `string` | Yes | Proxy tier used for the fetch. |

## Platform availability

1 platform returns the `Web page` shape. `extraction_check.checked`, `extraction_check.found_in_text`, `extraction_check.confirmed`, `extraction_check.removed`, `extraction_check.unverified` are never null on any platform. The fields below vary: **yes** means the platform populates it (the value may still be null); a blank means the platform never provides it, so it is always null.

<div className="overflow-x-auto">

| Platform | `url` | `final_url` | `status_code` | `scrape_id` | `fetched_at` | `published_at` | `content.markdown` | `content.html` | `content.raw_html` | `content.summary` | `media.screenshot_url` | `media.audio_url` |
| -------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Web Scraping | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |

</div>

## Machine-readable schema

Validate responses programmatically against the JSON Schema (2020-12):

[`/schemas/web-page.json`](/schemas/web-page.json)

Point a validator (Ajv, `jsonschema`, or your framework's) at that URL, or hand it to an agent so it can check the shape without a live call.

## Returned by

Endpoints with the `WebPage` / `WebPageList` archetype return this shape:

[Web Scraping](/platforms/web)

See how the same fields map to each platform's raw upstream names in the [cross-platform field equivalence table](/docs/schema/field-equivalence.md).