Message Queue (MQ)

● Stable v2.4.1 Updated Oct 24, 2025

Aevum's Message Queue provides a high-throughput, low-latency event streaming backbone for decoupling microservices, processing real-time knowledge updates, and triggering AI inference pipelines asynchronously.

Architecture

The MQ system follows a partitioned, replicated broker architecture optimized for horizontal scaling. Messages are organized into topics, which are split into partitions and distributed across broker nodes. Consumers subscribe to partitions and process events at their own pace.

ℹ️ Architecture Note

Aevum MQ uses a custom zero-copy network layer with ring-buffer dispatch, reducing serialization overhead by ~40% compared to standard Kafka-compatible brokers.

Core Concepts

Concept	Description
Producer	Service that publishes events to a topic. Handles batching, compression, and exactly-once semantics.
Consumer	Service that subscribes to topics/partitions. Manages offsets, retries, and dead-letter routing.
Broker	Stateful node that stores partitions, handles replication, and serves read/write requests.
Topic	Logical category for event streams. Configurable retention, partitions, and schema validation.
Schema Registry	Centralized Avro/JSON Schema store enforcing backward/forward compatibility on publish.

Quick Start

Initialize the MQ client in your application using the official SDK. The client handles connection pooling, retry backoff, and schema validation automatically.

JavaScript / TypeScript

import { AevumMQ, ProducerConfig } from '@aevum/mq-client';

const config: ProducerConfig = {
  bootstrapServers: ['mq-01.aevum.internal:9092', 'mq-02.aevum.internal:9092'],
  topic: 'knowledge.ingestion.events',
  retries: 3,
  retryBackoffMs: 1000,
  enableIdempotence: true,
  schemaId: 'org.aevum.knowledge.v2'
};

const mq = new AevumMQ(config);

async function publishArticleUpdate(payload: any) {
  const response = await mq.publish({
    key: payload.articleId,
    value: payload,
    headers: {
      'x-trace-id': crypto.randomUUID(),
      'content-type': 'application/json'
    }
  });
  console.log('Offset:', response.offset, 'Partition:', response.partition);
}

Message Schema & Validation

All topics are schema-enforced by default. Messages must conform to the registered Avro or JSON Schema. Invalid payloads are rejected at the broker level with a 400 SCHEMA_VIOLATION error.

JSON Schema Example

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "articleId": { "type": "string", "format": "uuid" },
    "eventType": { "type": "string", "enum": ["created", "updated", "deleted", "reviewed"] },
    "timestamp": { "type": "string", "format": "date-time" },
    "payload": { "type": "object" },
    "metadata": {
      "type": "object",
      "properties": {
        "contributorId": { "type": "string" },
        "lang": { "type": "string", "pattern": "^[a-z]{2}$" }
      }
    }
  },
  "required": ["articleId", "eventType", "timestamp"]
}

Delivery Guarantees

Aevum MQ supports three delivery semantics. Choose based on your pipeline's consistency requirements.

Guarantee	Use Case	Configuration
At-Least-Once	Standard processing with idempotent consumers	`acks: 'all'`, `enableIdempotence: true`
Exactly-Once	Critical financial/audit streams	`transactionalId: 'txn-xxx'`, `isolationLevel: 'read_committed'`
At-Most-Once	High-throughput telemetry where drops are acceptable	`acks: '0'`

⚠️ Exactly-Once Requirement

Transactions require min.insync.replicas ≥ 2. Ensure your cluster replication factor matches your durability requirements.

HTTP API Reference

For service-to-service orchestration, MQ exposes a RESTful control plane alongside the native binary protocol.

Method	Endpoint	Description
GET	`/v2/topics`	List all accessible topics with partition counts
POST	`/v2/topics/{name}/messages`	Publish single or batch messages
PUT	`/v2/consumers/{group}/offsets`	Manually commit consumer group offsets
GET	`/v2/metrics/brokers`	Retrieve broker health, lag, and throughput

Error Handling & Retries

Consumers should implement exponential backoff with jitter. The SDK provides a built-in retry policy, but circuit breakers are recommended for external dependency failures.

Retry Configuration

// YAML / Helm values override
mq:
  consumer:
    maxPollRecords: 500
    sessionTimeoutMs: 30000
    heartbeatIntervalMs: 3000
    retryPolicy:
      maxAttempts: 5
      initialDelayMs: 200
      maxDelayMs: 5000
      multiplier: 2.0
      jitterFactor: 0.2
    deadLetterTopic: 'dlx.knowledge.ingestion'

Monitoring & Observability

MQ exposes Prometheus metrics on :9404/metrics. Key indicators include:

aevum_mq_broker_partition_lag - Consumer group lag per partition
aevum_mq_producer_send_rate - Messages/sec produced
aevum_mq_consumer_commit_failures - Offset commit errors
aevum_mq_schema_validation_errors - Rejected payloads

Integrate with Aevum's Grafana dashboards for real-time pipeline visibility. Alerts are pre-configured for lag > 10k messages and broker disk usage > 85%.

Was this page helpful?