The Archive Pipeline
Our methodology spans six core phases: discovery, crawling, rendering, verification, storage, and access. Each phase employs specialized tools and quality controls.
Seed Lists
WARC Capture
Authenticๅ็ฐ
Hash Chains
Redundant Store
Every archived page is captured as a complete, immutable artifact โ including its original HTML source, associated stylesheets, embedded images, JavaScript, server response headers, and network timing data. Nothing is sanitized, nothing is modernized.
Seed Identification & Curation
Finding the right URLs to archive โ the foundation of a quality collection.
Seed List Curation
We begin with a manually curated seed list of over 12,000 URLs spanning the golden age of the web (1990โ2000). Our curators โ web historians, digital archaeologists, and longtime internet veterans โ identify sites across several critical categories:
๐ Personal Homepages
GeoCities, Angelfire, Tripod, and free-hosting sites that defined personal web publishing.
๐ข Early Commercial Sites
Pioneering businesses, dot-com era startups, and pre-social-media brand presences.
๐ฎ Niche Communities
Fan sites, web rings, hobby forums, and subcultural spaces from the pre-social-media era.
๐ Academic & Institutional
University pages, government portals, and library sites from the early WWW era.
๐ฌ BBS & Usenet
Archive of Bulletin Board System mirrors and Usenet newsgroup archives linked from early web pages.
๐ Web Rings & Directories
WebRing gateways, Open Directory Project (DMOZ) links, and other directory services.
Discovery Tools & Sources
We supplement curated seeds with automated discovery across multiple archival sources:
Authentic Capture & WARC Generation
Capturing web content as it existed โ using vintage browsers and modern archival standards.
Browser-Based Capture
Unlike conventional crawlers that fetch only HTML, our capture engine uses a headless vintage browser rendered through a specialized Chromium-based engine configured to emulate Netscape Navigator 4.x and Internet Explorer 5.x behavior. This ensures:
- Table-based layouts render identically to their original appearance
- JavaScript executes as it would have in a 1990s browser (including deprecated features)
- Plugin-dependent content (Flash, Java applets, QuickTime) is captured with fallback metadata
- Framesets, frames, and layered HTML are preserved with full document tree integrity
- CSS from the era renders with authentic font fallback chains
- Server response headers, cookies, and authentication states are captured
WARC File Generation
Every capture produces a WARC (Web Archive File Format) file โ the ISO-standard format for web archiving (ISO 28500:2015). Each WARC contains:
| WARC Record Type | Purpose |
|---|---|
| response | HTTP response from the server (HTML, CSS, images, etc.) |
| request | The exact HTTP request sent by the crawler |
| revisit | Record referencing an earlier capture of the same resource |
| conversion | Screenshot render of the page as visually rendered |
| metadata | Contextual metadata (capture timestamp, crawler version, IP geolocation) |
Over 60% of seed URLs are no longer live. Our crawler gracefully handles 404s, DNS failures, and TLS mismatches by falling back to previously cached snapshots from partner archives (Internet Archive, UK Web Archive) and generating detailed failure reports for curator review.
Authentic Visual Reconstruction
Reproducing pages exactly as they appeared on original hardware and browsers.
Multi-Browser Rendering Engine
We render every captured page across a matrix of vintage browser profiles to ensure faithful visual reconstruction:
๐ Netscape Navigator 3.0
The default browser of the mid-90s. Captures table layouts, tags, and Marquee elements authentically.
๐ช Internet Explorer 4.0
Covers ActiveX dependencies, conditional CSS hacks, and IE-specific rendering behaviors.
๐ Opera 3.6
Provides an alternative rendering baseline for cross-browser comparison and completeness.
๐ฅ๏ธ Mosaic 2.7
One of the earliest browsers. Used for pages designed for maximum compatibility.
Screenshot Pipeline
Each page is rendered to a 1200ร900 PNG screenshot at 800ร600 logical resolution with authentic pixel ratios. Screenshots are stored alongside the WARC files:
Integrity Verification & Quality Assurance
Ensuring every capture is complete, authentic, and cryptographically verifiable.
Content Completeness Checks
After each capture, our QA pipeline runs automated checks to verify completeness:
| Check | Method | Threshold |
|---|---|---|
| Resource completeness | Compare HTML src/href tags against captured resources | โฅ95% matched |
| Image integrity | SHA-256 hash verification of all .gif, .jpg, .png files | 100% match |
| JavaScript validity | Static analysis of JS for syntax errors in captured files | 0 errors |
| CSS completeness | Verify all linked stylesheets are present and non-empty | 100% present |
| DOM structure | Validate HTML structure against expected doctype | Valid HTML 3.2/4.0 |
| Screenshot fidelity | Pixel-diff against reference render of same page from IA | โค3% difference |
| HTTP header capture | Verify server response headers are fully captured in WARC | 100% complete |
Cryptographic Hash Chains
Every archived resource receives dual cryptographic hashes. These form an immutable chain linking the original capture to the archived state:
Our verification pipeline is aligned with NASA's STP-IP (Standard for Technical Information Preservation) and ISO 28500:2015 (WARC format) standards. Each archive unit includes a manifest file compliant with OAIS (Open Archival Information System) reference model (ISO 14721).
Redundant, Distributed Archive Storage
Multi-replica storage across geographically distributed data centers with checksum-based integrity monitoring.
Storage Architecture
Every WARC file is stored on 6 independent replicas across 3 geographically dispersed data centers:
Integrity Monitoring
Monthly checksum verification of every stored WARC file against its recorded digest:
Search, Browsing & API Access
Making the archive discoverable and usable through multiple access channels.
Search Index Architecture
We maintain a full-text search index built on inverted indices of all captured content, with era-specific filtering:
| Index Field | Indexed Content | Example Query |
|---|---|---|
page_text | Full HTML text content (stripped of tags) | query: "welcome to my homepage" |
capture_date | WARC timestamp of capture | date:[1995-01-01 TO 1999-12-31] |
domain | Original domain name | domain:geocities.com |
technology | Detected tech (JavaScript, Flash, frames, tables) | tech:frames AND tech:flash |
content_type | HTTP content type | type:text/html |
hosting_platform | GeoCities, Angelfire, Tripod, personal server, etc. | hosting:angelfire |
geo_cities_section | GeoCities Neighborhood (SunsetStrip, etc.) | neighborhood:sunsetstrip |
file_size | Total WARC file size | size:[1KB TO 50KB] |
Access Channels
๐ Web Portal
Full-featured search interface with era filters, visual browsing, and side-by-side comparison of captures.
๐ REST API
Programmatic access for researchers. Query by date range, technology, domain, or content. Returns WARC URLs or direct file downloads.
๐ก Memento URI-R
Compatible with the Memento framework (RFC 7089), enabling integration with other archival systems and time travel queries.
๐ฆ Bulk Download
Academic and institutional partners can request bulk WARC downloads for offline analysis and preservation.
Tools & Infrastructure
The proprietary and open-source tools that power our archive pipeline.
Ethical Considerations & Rights Management
Preserving history while respecting the creators and the law.
Rights & Takedown Process
We operate under a respect-first policy. All content is archived with the understanding that copyright and privacy remain in effect:
- All personal homepages are archived with explicit permission where contact information is available
- DMCA takedown requests are honored within 24 hours of receipt โ content is immediately removed from search indexes and access endpoints
- Content from deceased individuals is archived only with next-of-kin consent
- Personal information (addresses, phone numbers) is redacted in publicly accessible versions while preserved in WARC for research use only
- Academic researchers receive access to redacted-only datasets unless explicitly granted full access by rights holders
- Regular outreach campaigns remind content creators of their ability to opt out at any time
Since 1994, we have honored 100% of all takedown requests with a median response time of 4.2 hours. We have never been found non-compliant with DMCA, GDPR, or CCPA requirements. Our privacy board reviews every takedown request independently.
The 1990 Web Archive Methodology โ at a Glance
A summary of our end-to-end preservation approach.
Our methodology is built on six pillars: (1) Curated discovery ensuring comprehensive coverage of the early web, (2) Browser-emulated crawling that captures content authentically, (3) Multi-browser rendering for visual fidelity, (4) Cryptographic verification for integrity assurance, (5) Redundant distributed storage for long-term preservation, and (6) Multiple access channels for public and academic use. Every step complies with international archival standards (ISO 28500, OAIS/ISO 14721, Memento/RFC 7089) and ethical best practices for digital preservation.