Precision Legacy Web Crawling

Our specialized crawler engine handles deprecated protocols, table-based layouts, and ancient hosting architectures to faithfully capture and preserve early internet content.

Run Test Crawl → Read API Docs

ACTIVE NODES

REQUESTS / SEC

2.4k

LEGACY PROTOCOLS

HTTP/1.0, FTP, Gopher

AVG. PAGE SIZE

18.2 KB

// HOW IT WORKS

From URL submission to immutable storage, our pipeline is optimized for the architectural constraints of the 1990–1999 web.

STEP 01

Protocol Handshake

Detects era-specific server headers and downgrades request methods to HTTP/1.0 or FTP when necessary.

STEP 02

DOM & Frame Mapping

Traverses nested frames, framesets, and table-based navigation structures to build a complete site graph.

STEP 03

Asset Extraction

Captures inline images, .mid/.wav background music, animated GIFs, and client-side cursor trails.

STEP 04

Immutable Archival

Stores snapshots with cryptographic hashes, original metadata, and era-specific rendering flags.

// CRAWLER CAPABILITIES

Built for the quirks of the early web. No JavaScript execution required, just pure HTML recovery.

📡

HTTP/1.0 Compatible

Automatically negotiates legacy protocol versions for servers that dropped support in 2000+.

🖼️

Low-Res Asset Handling

Preserves 800x600 optimized images, tiled backgrounds, and dithered GIFs without compression.

🔗

WebRing & Link Tracking

Follows WebRing navigation, guestbook links, and external "under construction" banners.

📜

Robots.txt Compliance

Respects historical `robots.txt` rules and adds respectful crawl delays for aging infrastructure.

🎛️

Table Layout Parser

Reconstructs visual hierarchy from `<table>`-based layouts instead of relying on modern CSS.

💾

Metadata Injection

Tags each snapshot with creation date, browser era, resolution target, and original host headers.

// LIVE CRAWLER CONSOLE

Simulate a crawl session. Watch how our engine navigates a classic 1997 GeoCities-style domain.

crawler-node-04 ~ session.log

// API & TECHNICAL SPECS

Developer endpoints for automated archiving, batch submissions, and metadata querying.

POST /api/v1/crawl/initiate
curl -X POST https://api.1990archive.net/v1/crawl/initiate \\
  -H "Authorization: Bearer YOUR_API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{
    "url": "http://angelfire.lycos.com/~retro_user",
    "protocol_override": "http/1.0",
    "depth": 3,
    "capture_media": true,
    "render_flags": ["frames", "tables", "midi"]
  }'

Parameter	Type	Description
`protocol_override`	String	Force HTTP/1.0, FTP, or auto-detect for legacy servers
`depth`	Integer	Maximum link traversal depth (1–8 recommended)
`capture_media`	Boolean	Enable extraction of .gif, .mid, .wav, and inline images
`render_flags`	Array	Specify layout engines: `frames`, `tables`, `blinking_text`
`rate_limit`	Integer	Delay between requests in ms (default: 1500)

// FREQUENTLY ASKED QUESTIONS

How do you handle dead or redirected 90s domains?

Our crawler attempts multiple archival fallbacks including Wayback Machine mirrors, DNS history lookups, and cached ISP proxies. If a live response is impossible, we flag it with a `STATUS: UNREACHABLE` tag and preserve any cached fragments.

Can I schedule recurring crawls for historical sites?

Yes. Enterprise plans include cron-based crawl scheduling with delta-comparison. We only store new snapshots when content changes exceed a 15% threshold to save storage.

Do you execute client-side JavaScript?

No. The 1990s web was primarily static HTML with inline scripts that are now incompatible. Our engine parses raw DOM structure, form actions, and `` refresh tags to reconstruct navigation without modern JS runtimes.

How are large asset downloads handled?

Assets over 5MB (rare for the era) are chunked and stored in our S3-compatible archival tier. MIDI files and GIFs are always preserved losslessly with original EXIF/creation metadata.

Ready to Capture the Early Web?

Deploy your own crawling pipeline, access our REST API, or request a custom archival sprint for historical preservation projects.

Get API Access & Deploy →