Preservation Standards

Technical specifications, metadata frameworks, and integrity protocols that ensure the long-term viability, authenticity, and accessibility of our 1990s web archive.

Core Format Standards

All digital assets captured by the 1990 Web Archive are stored and maintained according to internationally recognized preservation formats. We prioritize bit-level fidelity alongside forward-compatible normalization.

Format	Use Case	Status	Reference
WARC 1.1	Primary web capture container (HTTP/1.0 & 1.1)	Active	IETF RFC 8493
WARC-JSON	Structured metadata & request/response headers	Active	IIIF Presentation API
PDF/A-2b	Static visual snapshots of rendered pages	Active	ISO 19005-2
Byte-Sidecar (.html.raw)	Original 1990s HTML without normalization	Active	Internal Spec
GIF/PNG (Lossless)	Embedded graphics & animated assets	Active	ImageMagick / IIIF

archival-format warc | Primary storage. Contains HTTP requests, responses, headers, and payload. Compressed via GZIP (level 6). Indexed via WARC-IT.
normalization-format html5-utf8 | Sidecar conversion for modern rendering. Original encoding preserved in sidecar.

Byte-Level Preservation Policy Every captured URL maintains a 1:1 binary copy of the original response body, headers, and TCP stream metadata. Normalized versions are generated separately for access and never overwrite source files.

Metadata & Cataloging

Archival records are enriched using Dublin Core, MODS, and custom temporal tags to support scholarly research and temporal discovery.

Required Metadata Fields

warcinfo {}
  software: "1990-WA-Crawler/v4.2"
  format: "application/warc"
  conformsTo: "RFC8493"
  timestamp: "1995-08-14T12:33:09Z"
  dublinCore: { "creator", "date", "source", "subject" }
  1990wa_extensions: { "browser_agent", "server_stack", "visual_tags" }

We implement automated OCR for image-heavy pages and semantic tagging for era-specific technologies (e.g., `

` layouts, framesets, MIDI embeds, WebRings). All metadata is machine-readable and exported via JSON-LD.

Integrity & Verification

Data integrity is maintained through cryptographic hashing, periodic fixity checks, and distributed replication.

SHA-256 Checksums: Generated for every WARC record, sidecar file, and extracted asset. Stored in `.sha256sum` manifests.
Fixity Audits: Automated quarterly verification against stored hashes. Corrupted records trigger automatic repair from geographically distributed mirrors.
Cryptographic Signing: WARC files are signed using PGP (EdDSA) by the Preservation Authority. Signature blocks are embedded in WARC-JSON metadata.
Replication Strategy: 3-2-1 rule enforced. Primary cold storage + 2 geo-distributed tape archives + 1 cloud replica.

verify 1990wa-integrity
$ 1990wa verify --archive ./1995-geoCities.warc.gz --check fixity,signature,hash
[✓] 4,201 records verified
[✓] EdDSA signature valid (Preservation Authority)
[✓] SHA-256 manifests match

Access & Emulation Standards

While preservation prioritizes fidelity, access requires accurate rendering of legacy web technologies. We adhere to the following emulation frameworks:

Technology	Emulation Method	Accuracy Target
Netscape Navigator 3.0–4.7	JSDoit / Webrecorder Emulation Server	Pixel & DOM structure parity
Framesets & Table Layouts	Original CSS/HTML isolation mode	100% structural fidelity
Animated GIF / APNG	Frame-accurate decoding via libwebp/giflib	Lossless frame playback
MIDI / AU Audio	FluidSynth + SoundFont 2.0	Timing & channel mapping preserved

IIIF Integration All visual assets and page snapshots are served via IIIF Image API 3.0. Canvas representations include viewport dimensions, color profiles, and era-specific rendering hints.

Compliance & Open Standards

The 1990 Web Archive operates as a compliant OAIS (Open Archival Information System) repository per ISO 14721. All tools, schemas, and pipelines are open-source and documented under MIT/Apache 2.0 licenses.

OAIS Functional Model: Ingest, Archival Storage, Data Management, Access, Administration, and Preservation Planning components fully implemented.
LOCKSS / CLOCKSS: Participates in distributed preservation initiatives for redundant academic access.
API Standards: RESTful endpoints follow OpenAPI 3.1. Bulk data exports support WARC, JSON-LD, and CSV formats.
Provenance Tracking: Full audit trails maintained via PROV-O ontology. Every transformation, migration, or access event is logged immutably.

compliance-status verified
last-audit 2024-11-02
framework OAIS-ISO14721 / W3C-Digital-Preservation
license CC0-1.0 (Data) | MIT (Tools)