Web page schema

The unified SocialCrawl web page schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.

Every SocialCrawl endpoint that returns a web page gives you this exact shape, whatever the source platform. Write your parser once and the same code reads web page data from every platform below. That is the unified schema: one contract instead of a dozen raw upstream formats.

Field reference

Field	Type	Nullable	Description
`url`	`string`	Yes	Requested URL.
`final_url`	`string`	Yes	Final resolved URL after redirects.
`status_code`	`number`	Yes	HTTP status code observed while fetching the page.
`scrape_id`	`string`	Yes	Opaque scrape identifier for follow-up interactions.
`fetched_at`	`string`	Yes	Fetch timestamp when reported.
`published_at`	`string`	Yes	When the page was published: an ISO 8601 instant, or a YYYY-MM-DD date when only the day is printed. From the page's own article:published_time meta tag on a scrape, and from the news date or that tag on a search row. Null when the page states none.
`content.markdown`	`string`	Yes	Markdown body.
`content.html`	`string`	Yes	HTML body.
`content.raw_html`	`string`	Yes	Raw HTML body.
`content.summary`	`string`	Yes	Generated summary.
`media.screenshot_url`	`string`	Yes	Screenshot URL when requested.
`media.audio_url`	`string`	Yes	Audio URL.
`media.video_url`	`string`	Yes	Video URL.
`extraction`	`object`	Yes	Structured extraction result when requested. On `web/extract` every value is checked against the page text: a placeholder such as 'Not specified', or a value the page does not state, comes back as null and is listed in `extraction_check.removed`.
`extraction_check.checked`	`integer`	No	Non-null values examined.
`extraction_check.found_in_text`	`integer`	No	Values kept because they appear word for word on the page.
`extraction_check.confirmed`	`integer`	No	Values kept because a check found the page states them, for example in another language.
`extraction_check.removed`	`object[]`	No	Fields set to null, with the value the extractor gave.
`extraction_check.unverified`	`string[]`	No	Fields returned as extracted without a check.
`fetch.cache_state`	`string`	Yes	Upstream cache state when reported.
`fetch.cached_at`	`string`	Yes	Upstream cache timestamp when reported.
`fetch.proxy_tier`	`string`	Yes	Proxy tier used for the fetch.

Platform availability

1 platform returns the Web page shape. extraction_check.checked, extraction_check.found_in_text, extraction_check.confirmed, extraction_check.removed, extraction_check.unverified are never null on any platform. The fields below vary: yes means the platform populates it (the value may still be null); a blank means the platform never provides it, so it is always null.

Platform	`url`	`final_url`	`status_code`	`scrape_id`	`fetched_at`	`published_at`	`content.markdown`	`content.html`	`content.raw_html`	`content.summary`	`media.screenshot_url`	`media.audio_url`
Web Scraping	yes	yes	yes	yes	yes	yes	yes	yes	yes	yes	yes	yes

Machine-readable schema

Validate responses programmatically against the JSON Schema (2020-12):

/schemas/web-page.json

Point a validator (Ajv, jsonschema, or your framework's) at that URL, or hand it to an agent so it can check the shape without a live call.

Returned by

Endpoints with the WebPage / WebPageList archetype return this shape:

Web Scraping

See how the same fields map to each platform's raw upstream names in the cross-platform field equivalence table.