Under the Hood | 1990 Web Archive

// Capture Pipeline

From Seed to Static Archive

Our ingestion workflow is fully automated, deterministic, and compliant with the OAIS reference model. Every URL passes through six validated stages before becoming queryable.

🌐

Seed Intake

URL submission & priority queue

🕷️

Distributed Crawl

Parallel HTTP/1.0 & modern agents

⚖️

Deduplication

Content-addressable hashing

📦

WARC Packaging

Immutable archival format

🗄️

Cold Storage

Geo-replicated object tier

🔍

Index & Serve

Inverted index & public API

// Crawling Engine

Legacy-Aware Web Harvesting

⚡ Protocol Support

Modern crawlers often fail on early web stacks. Our agents natively support HTTP/1.0, FTP, Gopher, WAIS, and legacy TLS 1.0/1.1 handshakes without modern cipher negotiation.

🧠 Politeness & Throttling

We respect robots.txt (including legacy variants), implement domain-level rate limiting, and use randomized jitter to avoid DDoS-like behavior on fragile legacy servers.

🖼️ Asset Reconstruction

Automatic resolution of relative paths, broken frame src attributes, and inline JavaScript document.write() calls to preserve structural integrity.

📊 Scale Metrics

4.2M

Unique URLs

89TB

Raw WARC Data

Global Nodes

// Preservation Layer

Immutable Archival Storage

We follow the WARC-2.2 standard and OAIS compliance guidelines to ensure long-term readability and cryptographic verifiability of every captured artifact.

storage-verify.sh

archive@node-01$ warc-verify --integrity ./captures/1996/geo_8821.warc.gz

[INFO] Parsing WARC2.2 container...

[INFO] Checking SHA-256 digests...

[OK] 14,892 records verified (0 mismatches)

[INFO] Cross-referencing manifest index...

[OK] Storage tier: Cold (S3-Glacier) / Replication: 3x

archive@node-01$ █

Component	Technology	Purpose	Retention
Primary Object Store	S3-Compatible (MinIO + AWS)	Hot/Warm WARC access	Indefinite
Cold Archive	Glacier Deep Archive / Tape	Cost-optimized preservation	100+ Years
Index Database	PostgreSQL + pgvector	Metadata, search vectors	Real-time
Search Engine	Elasticsearch 8.x	Full-text & facet indexing	Incremental
Cache Layer	Redis Cluster	Query & render caching	TTL-based

// Rendering & Emulation

Faithful Historical Playback

Modern browsers strip away or misinterpret early web standards. Our rendering pipeline uses sandboxed emulation environments to reproduce the original user experience.

🖥️ Legacy Browser VMs

Lightweight containers running Netscape Navigator 4.x, Internet Explorer 5.5, and Mosaic 3.0 with exact CSS1/CSS2 parsing engines and table-layout algorithms.

🎵 Media & Interactivity

Automatic transpilation of MIDI/RA containers to modern formats, frame-based layout reconstruction, and client-side hit counter preservation without live server dependencies.

// Technology Stack

Built for Longevity & Scale

stack-summary.log

system$ cat /etc/1990archive/stack.conf

[CORE] Go 1.21, Rust 1.74, Node.js 20

[DATA] PostgreSQL 16, Elasticsearch 8, Redis 7

[INFRA] Kubernetes, Terraform, Prometheus, Grafana

[ARCHIVE] WARC-2.2, IIIF, OAIS-Compliant, SHA-256 Signatures

[NET] gRPC, GraphQL, WebSocket, HTTP/2

[SEC] Vault, OIDC, RBAC, Air-gapped Backups

// System Status

Live Infrastructure Health

Real-time monitoring of our preservation clusters. All critical paths operating within SLA thresholds.

Crawler Fleet 98.2% Active

Index Clusters Healthy

Storage Replication 3/3 Nodes

API Gateway 12ms Avg Latency

Render Engine Scaling (High Demand)

Integrity Checks Verified (24h)

// Developer Access

API & Integration

Access raw WARC streams, query the inverted index, or fetch rendered HTML snapshots programmatically. Full SDKs and GraphQL endpoints available.

api-example.sh

dev$ curl -X POST https://api.1990webarchive.io/v2/capture \\\

-H "Authorization: Bearer $TOKEN" \\\

-d '{"url": "http://example.geocities.com", "depth": 2, "render": true}'

{

"job_id": "cap_8x92m4p1",

"status": "queued",

"estimated_completion": "120s",

"webhook_url": "https://your-server.dev/archive/notify"

}