WebCrawler Directories
Centralized index structures optimized for legacy and modern web crawlers. Submit sites, manage crawl queues, and access protocol-specific sitemap registries.
🔧 Crawler Configuration & Protocols
Our directories expose machine-readable endpoints for automated archival. Configure your crawler to target specific eras, protocols, or structural layouts.
HTTP/1.0 & 1.1 Legacy
Optimized for pre-1999 server responses, including non-standard headers and early CGI endpoints.
Gopher & WAIS
Protocol adapters for pre-web text menus and wide-area information servers still hosted on heritage infrastructure.
Frame & Table Layouts
DOM parsing rules that correctly resolve nested framesets and table-based navigation structures.
Sitemap Registries
Auto-generated `sitemap.xml` and `robots.txt` mirrors for historical sites that never implemented them.
# Configure crawler target for 1990s GeoCities sector
archive$ config set --target "https://archive.1990web.org/crawler/geocities/1998"
archive$ config set --protocol "http/1.0" --user-agent "NCSA_Mosaic/2.0"
archive$ config set --follow-frames true --respect-robots legacy
archive$ crawl start --depth 4 --concurrency 12 --output-format warc
[✓] Target verified. Queue initialized. Est. pages: 84,300