NLP & Archives 22

Overview

NLP & Archives '22 is a flagship research initiative developed by the 1990 Web Archive to bridge natural language processing with historical web preservation. The project focuses on extracting, classifying, and analyzing textual content from the first decade of the World Wide Web (1990–1999), enabling computational linguists, historians, and developers to query early internet discourse at scale.

Objective: Build a pipeline that cleans, tokenizes, and entities-recognizes millions of raw HTML pages from the early web, transforming fragmented markup into structured, searchable linguistic corpora.

Technical Methodology

1. HTML-to-Text Normalization

Early web pages relied heavily on table-based layouts, inline styles, and deprecated tags. Our pipeline uses a hybrid approach combining tree-sitter HTML parsing with regex-based cleanup to extract semantic text while discarding navigation clutter, guestbook signatures, and "under construction" banners.

2. Temporal Language Modeling

We trained a custom Transformer variant on 14.2M extracted documents, fine-tuned for period-specific vocabulary. The model handles spelling variations, early internet slang (e.g., uucp, bbs, www), and domain-shifted entity references.

import archive_nlp
from archive_nlp.pipeline import EarlyWebProcessor

processor = EarlyWebProcessor(
    era="1990s",
    model="1990web-llama-7b-archived",
    tokenization="html-aware"
)

corpus = processor.load_dataset("geocities_recovery_1998")
results = processor.extract_entities(corpus)
print(f"Named entities found: {len(results)}")

Dataset Specifications

Metric	Value	Notes
Raw Pages	4.8M	Collected via distributed crawler fleet
Cleaned Documents	3.1M	Post-normalization & deduplication
Total Tokens	2.4B	HTML-aware tokenization applied
Language Distribution	92% EN, 4% DE, 2% JP, 2% Other	Auto-classified via fastText
Temporal Span	1991–1999	Peak density: 1997–1998

Key Findings

Lexical Drift: Tech terminology from 1995 shows a 34% mismatch with modern NLP stopword lists. Terms like frame, table, and guestbook dominated early web syntax.
Entity Decay: Personal URLs, ISP email domains (e.g., @aol.com, @compuserve.com), and dialup-era references require custom NER pipelines to resolve correctly.
Sentiment Shift: Early web text exhibits significantly higher optimism and community-focused language compared to post-2000 commercialized content.

Access & Usage

The NLP & Archives '22 corpus is available under a CC-BY-NC 4.0 license for academic and non-commercial research. API access requires a registered researcher key. Bulk download snapshots are updated quarterly.

For integration questions, dataset validation reports, or model checkpoints, visit the Developer Documentation or contact research@1990webarchive.org.