WARC Compression & Byte-Range Fetching Specifications

Technical guidelines for efficiently retrieving historical web content using standardized compression and HTTP byte-range requests.

ⓘ Important: This specification applies to all WARC/WACZ files archived from 1990–1999. Ensure your client supports gzip and brotli decoding before making requests.

Overview

The 1990 Web Archive stores historical web content in WARC (Web ARChive) 1.1 format, compressed using modern algorithms to optimize storage and transfer speeds. DOC-041 defines the exact protocol for requesting partial content and handling compressed archives programmatically. This endpoint is designed for researchers, developers, and digital preservationists who need granular access to specific HTTP records without downloading entire multi-gigabyte archives.

Prerequisites

Valid API key with archive:read scope
HTTP/1.1 or HTTP/2 compatible client library
Support for Content-Encoding: gzip or br
Local WARC parsing library (e.g., warcprox, webrecorder-watcher)

API Endpoint

GET /api/v2/archives/{warc_id}/content/{record_index}

Path Parameters

Parameter	Type	Description
`warc_id`	string	Unique identifier for the WARC container (UUID v4)
`record_index`	integer	Zero-based index of the HTTP response record within the archive

Byte-Range Request Support

The archive server fully supports RFC 7233 byte-range requests. This is critical for large WARC files (>2GB) to avoid downloading entire archives when only specific records are needed. The server maintains a block-aligned index mapping logical record offsets to compressed byte boundaries.

Request Example

GET /api/v2/archives/7f9d2c1a-88b4-4e11-9c3a-001b2c3d4e5f/content/42
Host: api.1990webarchive.org
Authorization: Bearer <YOUR_API_KEY>
Range: bytes=1048576-2097151
Accept-Encoding: gzip, br

Response Headers

Header	Description
`Content-Range`	Specifies the byte range returned (e.g., `bytes 1048576-2097151/3221225472`)
`Content-Length`	Size of the returned byte range in bytes
`Accept-Ranges`	Always returns `bytes`
`WARC-Record-Offset`	Custom header mapping the byte range to the WARC record boundary for faster client-side parsing

⚠️ Warning: Byte-range requests are only supported on uncompressed (identity) or block-aligned compressed archives. Random access on stream-compressed files without block indexes will return 416 Range Not Satisfiable.

Compression Handling

Archives are stored with dual compression layers to balance storage efficiency and retrieval speed:

Block Compression: WARC records are compressed individually using brotli (preferred, window=22) or gzip (legacy, level=6).
Container Indexing: A companion .idx file maps record offsets to compressed block boundaries for O(1) lookup during range requests.

When requesting content, the server transparently decompresses only the requested range and returns it with the appropriate Content-Encoding header. Clients should not attempt to decompress partial streams without validating block boundaries.

Error Responses

Status Code	Meaning	Resolution
`206` Partial Content	Byte range successfully returned	Process payload and seek to record boundary
`400` Bad Request	Invalid range format or unsupported encoding	Verify `Range` header syntax
`403` Forbidden	Insufficient API permissions	Check scope includes `archive:read`
`404` Not Found	WARC ID or record index does not exist	Verify identifiers via list endpoint
`416` Range Not Satisfiable	Requested range exceeds archive size	Use `HEAD` to fetch `Content-Length` first

Rate Limits & Best Practices

Maximum 100 concurrent byte-range streams per API key
Recommended chunk size: 512KB - 2MB for optimal I/O throughput
Always cache companion index files (.idx) locally to reduce latency on repeated requests
Use conditional requests (If-None-Match) to avoid redundant downloads when archives haven't changed
Implement exponential backoff on 429 responses with jitter

Version History

Version	Date	Changes
`v2.4.1` Stable	2024-03-15	Added brotli block alignment support; updated error codes and retry logic
`v2.3.0`	2023-11-22	Introduced `WARC-Record-Offset` header for faster client-side parsing
`v2.1.0` Legacy	2023-06-08	Initial release of DOC-041 specification

Documentation generated for 1990 Web Archive • Last updated: March 2024

Report an issue • Edit this page • API Status