Dataset Download v3.2
Released: 2024-10-24 | Checksums verified | Mirrors active
Overview
The Core Collection v3.2 contains 14.2 million archived web pages, resources, and metadata snapshots spanning 1990–1999. Optimized for research, training, and digital preservation workflows. Includes reconstructed framesets, inline CSS/JS extraction, and temporal link graphs.
48.7 GB
Total Size (Compressed)
14.2M
Archived Pages
42.1M
Resource Files
1990-1999
Time Span
Download Files
| File | Size | Format | SHA-256 | Action |
|---|---|---|---|---|
| core_v3.2_pages.jsonl.gz | 28.4 GB | JSONL | a1b2c3d4e5f6789012345678901234567890abcdef | ↓ Download |
| core_v3.2_resources.parquet | 12.1 GB | Parquet | f8e7d6c5b4a321098765432109876543210fedcb | ↓ Download |
| core_v3.2_link_graph.csv.gz | 5.2 GB | CSV | 776655443322110099887766554433221100aa | ↓ Download |
| core_v3.2_sqlite_dump.db | 1.8 GB | SQLite | 998877665544332211009988776655443322bb | ↓ Download |
| manifest_v3.2.sha256 | 4.2 MB | TXT | 112233445566778899001122334455667788cc | ↓ Download |
All files are available via HTTP(S) and IPFS. Use the --split flag for chunked downloads.
Quick Start & CLI
$ wget https://archive.1990web.org/datasets/v3.2/core_v3.2_pages.jsonl.gz
$ sha256sum -c manifest_v3.2.sha256
# Verify integrity before extracting
$ gunzip core_v3.2_pages.jsonl.gz
$ head -n 5 core_v3.2_pages.jsonl
$ sha256sum -c manifest_v3.2.sha256
# Verify integrity before extracting
$ gunzip core_v3.2_pages.jsonl.gz
$ head -n 5 core_v3.2_pages.jsonl
# Python ingestion example
import json, gzip
with gzip.open('core_v3.2_pages.jsonl.gz', 'rt') as f:
for line in f:
page = json.loads(line)
print(page['url'], page['timestamp'])
import json, gzip
with gzip.open('core_v3.2_pages.jsonl.gz', 'rt') as f:
for line in f:
page = json.loads(line)
print(page['url'], page['timestamp'])
Version 3.2 Changelog
- ADDED Reconstructed GeoCities & Tripod subset (~2.1M pages) with original CSS/JS inline extraction
- ADDED Temporal link graph with cross-crawl session metadata
- FIXED Timestamp parsing for DST transition edge cases (1996 US zones)
- FIXED Frameset fallback generation now correctly maps frame borders to modern div layouts
- CHANGED JSONL schema updated: `mime_type` replaced with `content_type` for RFC 6838 compliance
- CHANGED Deprecated v3.1 legacy index format; migrate using `archive-index-convert --v3.1-to-v3.2`
Licensing & Usage
License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC BY-NC-SA 4.0)
This dataset is intended for academic research, historical preservation, and non-commercial development. Commercial usage requires a separate enterprise license. All archived content retains original copyright where applicable. Please cite "1990 Web Archive Core Collection v3.2" in publications.
This dataset is intended for academic research, historical preservation, and non-commercial development. Commercial usage requires a separate enterprise license. All archived content retains original copyright where applicable. Please cite "1990 Web Archive Core Collection v3.2" in publications.