1990 Web Archive

Crawler Modules

🕷️

Deep Historic Crawler

Specialized for navigating frame-based layouts, table-driven designs, and broken link chains common in 1990–1999 sites.

frames tables legacy-http

🌐

GeoCities & Angelfire

Optimized for ISP-hosted personal homepages. Handles dynamic subdirectories, visitor counters, and guestbook scripts.

subdomains cgi-bin guestbooks

📼

Media & Asset Extractor

Non-blocking harvest of animated GIFs, MIDI files, inline frames, and proprietary plugins (Flash, QuickTime, VRML).

gif89a midi flash

🔐

Metadata & Headers

Captures HTTP/1.0/1.1 response headers, server signatures, content-type sniffing, and robots.txt compliance logs.

http-1.1 headers robots

🔄

Recrawl Scheduler

Dynamic recrawling based on content change detection, hash diffing, and priority queues for high-value domains.

hash-diff priority cron

⚡

Politeness Engine

Respects legacy robots.txt, implements crawl-delay, rate limits by IP range, and handles 429/503 backoff gracefully.

rate-limit backoff compliance

API Integration

Python 3.10+ / requests

import requests

# Initialize crawler client
client = requests.Session()
client.headers["Authorization"] = "Bearer YOUR_API_KEY"
client.headers["User-Agent"] = "1990WebArchive-Crawler/1.0"

# Submit a URL for deep archival
response = client.post("https://api.1990webarchive.com/v1/crawl", json={
    "url": "https://archive.geocities.com/WebRing/1997",
    "depth": 3,
    "render_mode": "netscape-4",
    "capture_metadata": True
})

if response.status_code == 202:
    print(f"Crawl job queued: {response.json()['job_id']}")

Full documentation, SDKs, and webhook configuration available at /api/docs

Technical Specifications

Protocol Support	HTTP/1.0, HTTP/1.1, FTP, Gopher, WAIS, Telnet (archival proxy)
Rendering Engine	Headless Chromium + Netscape Navigator 4.x emulation layer
Storage Format	WARC 1.1 / WACZ, SHA-256 deduplication, S3-compatible object storage
Deduplication	Content-hash based, ~68% storage reduction across historical recrawls
Rate Limiting	Configurable per-domain, defaults to 1 req/sec with exponential backoff
Compliance	Strict robots.txt parsing, 429/503 handling, geographic throttling