📦 1990 Web Archive // Technical Specification

1. System Overview & Scope

The 1990 Web Archive is a distributed digital preservation system engineered to capture, normalize, store, and provide authenticated access to early World Wide Web content (1990–1999). The platform operates strictly on the Open Archival Information System (OAIS) reference model, ensuring long-term integrity, authenticity, and accessibility of historically significant web resources.

Core Objectives:

• Recover and preserve legacy HTML, HTTP/1.0, and early web technologies
• Maintain cryptographic proof of content existence and precise capture timestamps
• Provide researcher-facing APIs for programmatic access and large-scale analysis
• Ensure full compliance with ISO 14721 (OAIS), W3C archival guidelines, and Memento protocol standards

2. Architecture & Data Flow

The system follows a microservices architecture with event-driven ingestion. Key components include:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐     ┌──────────────┐
│  Crawlers   │────▶│  Event Bus   │────▶│  Processing │────▶│  Object Store│
│  (Dist. Pool)│   │  (Kafka/ NATS)│    │  (Parse/    │    │  (S3/GCS Comp)│
└─────────────┘     └──────────────┘     │  Validate) │    └───────┬───────┘
                                          └─────┬─────┘            │
                                                │                    │
                                          ┌─────▼─────┐     ┌───────▼───────┐
                                          │  Metadata  │────▶│  Auth & API   │
                                          │  Indexer   │     │  Gateway      │
                                          │  (Postgres)│     └───────────────┘
                                          └───────────┘

Infrastructure Stack: Kubernetes orchestration, Apache Kafka for event streaming, PostgreSQL for relational metadata, S3-compatible object storage for binary payloads, and Elasticsearch for full-text/search indexing. All services communicate via gRPC or REST with strict schema validation.

3. Ingestion & Crawling Pipeline

Archival capture follows a multi-stage validation and normalization process to ensure bit-accurate preservation:

1. Discovery: URL seeds sourced from historical dumps, Memento TimeGate mirrors, and authenticated researcher submissions.
2. Capture: Raw HTTP/1.0 socket capture paired with legacy user-agent emulation (Netscape Navigator 4.x, Mosaic 3.0) to trigger era-appropriate server responses.
3. Normalization: HTML Tidy processing, character encoding detection (ISO-8859-1/Shift_JIS → UTF-8), and relative-to-absolute URL resolution.
4. Fingerprinting: SHA-256 hash generation for content deduplication and fixity verification.
5. Queueing: Published to ingestion topic with `ArchiveEvent` schema. Failed captures enter exponential backoff retry logic.

{
  "event_id": "evt_9a8b7c6d",
  "timestamp": "2024-10-15T08:30:00Z",
  "source_url": "http://www.geocities.com/SunsetStrip/7842/",
  "capture_method": "http1.0_socket_legacy",
  "status": "pending_validation",
  "payload": {
    "html_size_bytes": 14200,
    "detected_charset": "iso-8859-1",
    "http_status": 200
  }
}

4. Storage & Metadata Schema

All archived entities conform to a normalized relational structure optimized for archival retrieval and research querying:

Storage Layout: s3://1990-archive/{year}/{month}/{entity_id}/index.html + linked assets preserved in /assets/ with path rewriting for offline rendering.

5. Core API Reference

RESTful endpoints for programmatic access. Authentication required via API key in X-Archive-Key header. All responses return JSON unless Accept: application/json is overridden.

Returns full entity record, metadata, Memento links, and signed download URLs. Rate limit: 100 req/min.

Submit URL batch for archival processing. Accepts JSON array or raw newline-separated text. Returns job tracking ID.

Full-text and metadata search. Supports Lucene syntax. Pagination via cursor token. Max results: 500 per page.

Verify cryptographic integrity of an archived payload. Returns boolean status and fixity check timestamp.

6. Preservation Standards

• OAIS Compliance: Full adherence to ISO 14721 ingest, archive storage, and access functions.
• PREMIS Metadata: Used for events, agents, rights management, and object identification tracking.
• Format Sustainability: All HTML normalized to valid HTML5 for long-term rendering, with original byte-for-byte artifacts preserved in cold storage.
• Fixity Checks: Automated daily SHA-256 verification against storage references. PagerDuty alert triggers on checksum mismatch.
• Memento Protocol: TimeGate, TimeMap, and TimeCapsule endpoints fully implemented per RFC 7089.

7. Security & Access Control

Access is tiered based on researcher credentials, institutional affiliation, and data sensitivity classification:

All API traffic enforced over TLS 1.3. Payloads encrypted at rest (AES-256-GCM). Immutable audit logging enabled for all write/ingest operations. OAuth 2.0 client credentials flow supported for institutional access.

8. Version History