2. Data Acquisition
The foundation of Aevum's knowledge graph lies in how we gather, normalize, and ingest information. Our acquisition pipeline operates across academic, institutional, and community channels, ensuring every data point enters the system with provenance, structure, and contextual metadata.
Acquisition Overview
Unlike traditional scrapers or static databases, Aevum employs a multi-modal ingestion architecture. Data arrives through scheduled API pulls, real-time webhooks, direct institutional partnerships, and vetted contributor submissions. Each stream is tagged with a confidence score, freshness timestamp, and source tier before entering the parsing layer.
No data is indexed without verifiable provenance. Every entity, fact, and relationship is mapped to its origin before storage.
Ingestion Pipeline
Raw data passes through five deterministic stages before reaching the knowledge graph:
Incoming streams are classified by source tier and routed to specialized parsers (PDF, JSON-LD, XML, Markdown, or structured DB exports).
NLP models identify persons, organizations, locations, concepts, and dates. Coreference resolution links aliases to canonical IDs.
Semantic classifiers assign predicates (e.g., author_of, part_of, contradicts) to build graph edges.
Translation models align terminology across 140+ languages, maintaining conceptual equivalence without literal drift.
Validated triples are committed to the graph database. Inverted indexes and vector embeddings are updated for search.
Source Categories
| Tier | Source Type | Examples | Refresh Cycle |
|---|---|---|---|
| Tier 1 | Academic & Institutional | Peer-reviewed journals, university repositories, government archives | Daily / On-publish |
| Tier 2 | Open Knowledge & Standards | Wikidata, DBpedia, ISO standards, public domain corpora | Weekly / Snapshot |
| Tier 3 | Expert Contributions | Verified scholar submissions, institutional partnerships, curated datasets | Continuous (Review-gated) |
| Tier 4 | Real-Time Feeds | News wires, preprint servers, patent filings, regulatory updates | Hourly / Webhook |
Quality & Validation Gates
Before any data reaches production, it must pass automated and human-in-the-loop checks:
- Consistency Check: Cross-references against existing graph nodes for contradictions or redundant definitions.
- Provenance Audit: Validates source reputation, publication date, and citation chain integrity.
- Bias & Neutrality Scan: Detects loaded language, unverified claims, or regional skew using sentiment and stance analysis.
- Expert Review Queue: High-impact or low-confidence entries are routed to domain specialists for manual verification.
Licensing & Ethical Compliance
All acquired data is cataloged with precise licensing metadata. Aevum strictly enforces:
- Attribution requirements per CC BY, CC BY-SA, and institutional policies
- GDPR & privacy compliance for personal data (PII is automatically redacted unless historically significant)
- Copyright-safe transformation for public domain and expired works
- Transparent contributor licensing agreements (default: CC BY 4.0 unless specified)
Technical Integration
For institutional partners or data providers wishing to feed directly into Aevum's acquisition layer, we expose a RESTful and GraphQL ingestion API with webhook callbacks for status tracking.
Rate limits: 500 req/min for T1 partners. Bulk upload support via S3-compatible endpoints. Detailed API documentation available in the developer portal.