3. Data Preparation
Transforming raw, heterogeneous inputs into structured, machine-readable knowledge assets ready for graph ingestion and AI enrichment.
Stage Overview
Data preparation is the critical bridge between raw acquisition and structured knowledge modeling. At Aevum Encyclopedia, this stage standardizes formatting, resolves ambiguities, enforces taxonomic alignment, and prepares entities for seamless ingestion into our semantic graph.
Processing Workflow
The preparation pipeline operates as a series of deterministic and AI-assisted transformations. Each micro-service outputs versioned artifacts with full lineage tracking.
Ingestion & Parsing
Multi-format parsing (PDF, HTML, JSON, DOCX) with layout-aware text extraction and metadata harvesting.
Cleaning & Deduplication
Removal of boilerplate, normalization of whitespace, encoding fixes, and fuzzy entity deduplication.
Structuring & Tagging
Paragraph segmentation, heading hierarchy reconstruction, and automatic section classification.
Entity Resolution
Disambiguation of names, dates, and locations using cross-referenced authority files and context windows.
Schema & Normalization
All content is mapped to our internal Knowledge Markup Language (KML), a JSON-LD superset optimized for graph databases. Key transformations include:
- Temporal Normalization: All dates converted to ISO 8601 with uncertainty ranges
- Numerical Standardization: Unit conversion to SI, decimal formatting, and scale alignment
- Reference Mapping: Citations linked to DOI/ISBN/ISBN-13 with validity checksums
- Terminology Alignment: Synonyms mapped to preferred lexical forms via our controlled vocabulary
// KML Snippet: Normalized Entity Record
{
"@context": "https://schema.aevum.io/kml/v2",
"@type": "Concept",
"id": "urn:aevum:concept:482910",
"label": {
"preferred": "CRISPR-Cas9",
"aliases": ["CRISPR", "Clustered Regularly Interspaced Short Palindromic Repeats"]
},
"domain": "molecular_biology",
"created": "2024-11-02T14:30:00Z",
"quality_score": 0.98
}
Quality Gates
Before advancing to Stage 4, every dataset must pass automated quality checkpoints. Records failing validation are routed to the correction queue for manual or assisted review.
Pipeline Automation
The preparation stage runs on Apache Airflow with dynamic scaling via Kubernetes. Each DAG node emits OpenTelemetry traces for full observability.
Key automation features:
- Self-healing parsers that adapt to new document layouts
- Automated rollback on schema drift detection
- Parallelized NLP pipelines using spaCy and custom transformer models
- CI/CD integration for vocabulary and rule updates