akh-medu + Eleutherios Integration Guide

Overview

akh-medu serves as a multilingual pre-processor that sits between Eleutherios's document chunking stage and its LLM extraction pipeline. Where Eleutherios's 7-dimensional extraction is English-strong but degrades for Russian, Arabic, and historical mixed-language sources, akh-medu's GF-based grammar parser runs in sub-millisecond time per chunk and produces language-neutral structured data: entities, relations, and claims with confidence scores.

Documents
    |
    v
Eleutherios chunking (document -> text chunks)
    |
    v
akh-medu pre-processor (text -> entities + claims + AbsTrees)
    |
    v
Eleutherios LLM enrichment (Mistral Nemo 12B -> 7D extraction)
    |
    v
Neo4j / pgvector

The pre-processor gives Eleutherios cleaner starting data that reduces extraction noise, particularly for multilingual corpora (technical manuals, diplomatic correspondence, academic texts with mixed-language citations, etc.).

Table of Contents

• Quick Start
  • Build
  • CLI Pipeline (JSONL)
  • CLI Pipeline (JSON batch)
  • HTTP Server
• Running with Eleutherios Docker
  • Prerequisites
  • Step 1: Start Ollama on All Interfaces
  • Step 2: Start Eleutherios Services
  • Step 3: Build and Start akh-medu
  • Step 4: Load Documents
  • Step 5: Run the Pipeline
  • Step 6: Pre-Process with akh-medu
  • Benchmark: Real-World Performance
  • Memory Requirements
• Step-by-Step Tutorial
  • 1. Prepare Your Corpus
  • 2. Pre-Process via CLI
  • 3. Pre-Process via HTTP
  • 4. Interpret the Output
  • 5. Feed into Eleutherios
• What Meaning We Extract
  • Claim Types
  • Entity Types
  • Canonical Predicates
• Available Languages
  • Language Detection
  • Mixed-Language Corpora
• Cross-Language Entity Resolution
  • Static Equivalence Table
  • Dynamic Equivalence Learning
  • Managing Equivalences via CLI
  • Managing Equivalences via HTTP
  • Seeding Domain-Specific Equivalences
• VSA Similarity Algorithms
  • How Hypervectors Work
  • Encoding Strategies
  • Similarity Search
  • Similarity Thresholds
  • How VSA Is Used in the Pipeline
• Extending with New Languages
  • Step 1: Add the Language Variant
  • Step 2: Add the Language Lexicon
  • Step 3: Add Detection Support
  • Step 4: Add Equivalences
  • Step 5: Rebuild and Test
  • Checklist for New Languages
• CLI Reference
• HTTP API Reference
• Integration Patterns
  • Python Integration
  • Eleutherios Mapping
• Architecture Notes
• Relational Pattern Reference

Quick Start

Build

# Core binary (CLI pre-processor)
cargo build --release

# HTTP server (optional, for network integration)
cargo build --release --features server

CLI Pipeline (JSONL)

Eleutherios pipes chunked text through stdin and reads structured output from stdout:

# Auto-detect language per chunk
cat chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl > structured.jsonl

# Force a specific language
cat russian_chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl --language ru > structured.jsonl

Input format (one JSON object per line):

{"id": "doc1-p1", "text": "Protein folding is a fundamental process in molecular biology."}
{"id": "doc1-p2", "text": "Misfolded proteins are associated with neurodegenerative diseases."}

The id field is optional but recommended for traceability. The language field is optional; omit it to auto-detect.

Output format (one JSON object per line):

{
  "chunk_id": "doc1-p1",
  "source_language": "en",
  "detected_language_confidence": 0.80,
  "entities": [
    {
      "name": "protein folding",
      "entity_type": "CONCEPT",
      "canonical_name": "protein folding",
      "confidence": 0.83,
      "aliases": [],
      "source_language": "en"
    }
  ],
  "claims": [
    {
      "claim_text": "Protein folding is a fundamental process in molecular biology.",
      "claim_type": "FACTUAL",
      "confidence": 0.83,
      "subject": "protein folding",
      "predicate": "is-a",
      "object": "fundamental process in molecular biology",
      "source_language": "en"
    }
  ],
  "abs_trees": [...]
}

CLI Pipeline (JSON batch)

For batch processing, use --format json with a JSON array on stdin:

echo '[
  {"id": "1", "text": "Gravity is a fundamental force of nature."},
  {"id": "2", "text": "Гравитация является фундаментальной силой природы."}
]' | ./target/release/akh-medu preprocess --format json

Returns:

{
  "results": [...],
  "processing_time_ms": 0
}

HTTP Server

For network-accessible integration (e.g., Eleutherios calling akh-medu over HTTP):

# Start server on port 8200
./target/release/akh-medu-server

Endpoints summary:

POST /preprocess:

curl -X POST http://localhost:8200/preprocess \
  -H 'Content-Type: application/json' \
  -d '{
    "chunks": [
      {"id": "1", "text": "The cell membrane contains phospholipids."},
      {"id": "2", "text": "Клеточная мембрана содержит фосфолипиды."},
      {"id": "3", "text": "La membrane cellulaire contient des phospholipides."}
    ]
  }'

Running with Eleutherios Docker

This section covers running the full stack: Eleutherios Docker services, Ollama for LLM inference, and akh-medu as the pre-processing layer.

Prerequisites

Step 1: Start Ollama on All Interfaces

Ollama must listen on 0.0.0.0 (not just 127.0.0.1) so Docker containers can reach it via host.docker.internal:

# Start Ollama listening on all interfaces
OLLAMA_HOST=0.0.0.0:11434 ollama serve &

# Pull required models
ollama pull nomic-embed-text      # Embeddings (274 MB)
ollama pull mistral-nemo:12b      # Extraction LLM (7.1 GB)

# Verify
curl -s http://localhost:11434/api/tags | python3 -c "
import sys, json
for m in json.load(sys.stdin).get('models', []):
    print(f'  {m[\"name\"]}')
"

Common mistake: If Ollama is started without OLLAMA_HOST=0.0.0.0:11434, Docker containers will get "connection refused" when calling the LLM. You can verify the listening address with ss -tlnp | grep 11434 — it must show *:11434, not 127.0.0.1:11434.

Step 2: Start Eleutherios Services

# Clone Eleutherios Docker
git clone https://github.com/Eleutherios-project/Eleutherios-docker.git
cd Eleutherios-docker

# Create data directories
mkdir -p data/inbox data/processed data/calibration_profiles

# Start services (Neo4j + PostgreSQL + API)
docker compose up -d

# First startup takes several minutes (demo data seeding).
# Watch progress:
docker compose logs -f api

The Eleutherios API (port 8001) runs a demo data import on first start (SEED_ON_FIRST_RUN=true) with ~144K Cypher statements. This typically takes 5-15 minutes. Wait until the health check passes:

# Poll until healthy
until curl -sf http://localhost:8001/api/health/simple; do
    echo "Waiting for Eleutherios API..."
    sleep 10
done
echo "API is ready"

Step 3: Build and Start akh-medu

cd /path/to/akh-medu

# Build CLI and HTTP server
cargo build --release
cargo build --release --features server

# Start the pre-processing server
./target/release/akh-medu-server &

# Verify
curl -s http://localhost:8200/health

Step 4: Load Documents

Copy your PDF/EPUB corpus into the Eleutherios inbox:

# Copy files (do NOT symlink — Docker bind mounts can't follow host symlinks)
cp /path/to/your/corpus/*.pdf /path/to/Eleutherios-docker/data/inbox/

# Verify files are visible inside the container
curl -s http://localhost:8001/api/list-inbox-files | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'{d[\"total_count\"]} files ({d[\"total_size_mb\"]:.0f} MB)')
for f in d['files'][:5]:
    print(f'  {f[\"filename\"]} ({f[\"size_mb\"]:.1f} MB)')
"

Important: Do not use symlinks into the Docker inbox directory. Docker bind mounts expose the directory to the container, but symlinks pointing to paths outside the mount will appear as broken links inside the container.

Step 5: Run the Pipeline

# Start the Eleutherios load pipeline
JOB_ID=$(curl -s -X POST http://localhost:8001/api/load-pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "type": "pdfs",
    "path": "/app/data/inbox",
    "selected_files": ["your-document.pdf"]
  }' | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])")

echo "Job started: $JOB_ID"

# Monitor progress
watch -n 10 "curl -s http://localhost:8001/api/load-status/$JOB_ID | python3 -c \"
import sys,json
d=json.load(sys.stdin)
print(f'Status: {d.get(\"status\")} | Progress: {d.get(\"progress_percent\",0)}%')
s=d.get('stats',{})
print(f'Entities: {s.get(\"entities\",0)} | Claims: {s.get(\"claims\",0)}')
\""

Step 6: Pre-Process with akh-medu

After Eleutherios completes Step 1 (chunking), retrieve the JSONL output and run it through akh-medu for immediate structural extraction:

# Copy the JSONL chunks from the container
docker cp aegis-api:/tmp/aegis_imports/${JOB_ID}_jsonl/combined_chunks.jsonl /tmp/chunks.jsonl

# Pre-process through akh-medu (CLI)
cat /tmp/chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl > /tmp/structured.jsonl

# Or via HTTP (batched, for production)
python3 -c "
import json, urllib.request

chunks = []
with open('/tmp/chunks.jsonl') as f:
    for line in f:
        data = json.loads(line)
        chunks.append({'id': data['metadata']['doc_id'], 'text': data['text']})

# Process in batches of 20
for i in range(0, len(chunks), 20):
    batch = chunks[i:i+20]
    payload = json.dumps({'chunks': batch}).encode()
    req = urllib.request.Request(
        'http://localhost:8200/preprocess',
        data=payload,
        headers={'Content-Type': 'application/json'}
    )
    resp = urllib.request.urlopen(req)
    result = json.loads(resp.read())
    for r in result['results']:
        print(json.dumps(r))
" > /tmp/structured.jsonl

Benchmark: Real-World Performance

Tested with "Memories, Dreams, Reflections" by Carl Jung (1.8 MB PDF):

akh-medu provides near-instant structural pre-extraction that complements the deeper but slower LLM-based extraction. High-confidence akh-medu claims can be ingested directly while the LLM pipeline runs.

Memory Requirements

Running the full stack simultaneously:

If you hit "model requires more system memory" errors from Ollama, free memory by dropping filesystem caches (sync && echo 3 | sudo tee /proc/sys/vm/drop_caches) or by stopping unused services.

Step-by-Step Tutorial

This section walks through a complete end-to-end workflow that you can replicate with your own corpus.

1. Prepare Your Corpus

akh-medu expects text chunks as JSON objects. Each chunk has a text field and an optional id and language field.

Create a sample corpus file (chunks.jsonl):

cat > /tmp/chunks.jsonl << 'EOF'
{"id": "intro-en-1", "text": "The mitochondria is a membrane-bound organelle found in eukaryotic cells."}
{"id": "intro-en-2", "text": "ATP synthesis depends on the electron transport chain."}
{"id": "intro-ru-1", "text": "Митохондрия является мембранным органоидом эукариотических клеток."}
{"id": "intro-fr-1", "text": "La mitochondrie est un organite présent dans les cellules eucaryotes."}
{"id": "intro-es-1", "text": "La mitocondria es un orgánulo presente en las células eucariotas."}
{"id": "intro-ar-1", "text": "الميتوكوندريا هي عضية موجودة في الخلايا حقيقية النواة."}
EOF

You can also convert existing documents. For PDF/EPUB corpora, use your preferred text extraction tool first:

# Example: extract text from PDFs, then chunk
# (Use your own extraction tool — pdftotext, Apache Tika, etc.)
pdftotext /path/to/your/document.pdf - | \
  python3 -c "
import sys, json
text = sys.stdin.read()
# Simple paragraph-level chunking (replace with your own chunking strategy)
paragraphs = [p.strip() for p in text.split('\n\n') if p.strip()]
for i, para in enumerate(paragraphs):
    print(json.dumps({'id': f'doc-p{i}', 'text': para}))
" > /tmp/chunks.jsonl

2. Pre-Process via CLI

# Build the binary
cargo build --release

# Run pre-processing with language auto-detection
cat /tmp/chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl > /tmp/structured.jsonl

# Inspect the output
head -1 /tmp/structured.jsonl | python3 -m json.tool

Expected output (formatted for readability):

{
  "chunk_id": "intro-en-1",
  "source_language": "en",
  "detected_language_confidence": 0.80,
  "entities": [
    {
      "name": "mitochondria",
      "entity_type": "CONCEPT",
      "canonical_name": "mitochondria",
      "confidence": 0.90,
      "aliases": [],
      "source_language": "en"
    },
    {
      "name": "membrane-bound organelle",
      "entity_type": "CONCEPT",
      "canonical_name": "membrane-bound organelle",
      "confidence": 0.90,
      "aliases": [],
      "source_language": "en"
    }
  ],
  "claims": [
    {
      "claim_text": "The mitochondria is a membrane-bound organelle found in eukaryotic cells.",
      "claim_type": "FACTUAL",
      "confidence": 0.90,
      "subject": "mitochondria",
      "predicate": "is-a",
      "object": "membrane-bound organelle",
      "source_language": "en"
    }
  ],
  "abs_trees": [...]
}

Force a specific language when auto-detection isn't reliable (short text, mixed-script content, or unsupported language using English as fallback):

cat /tmp/german_chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl --language en > /tmp/structured.jsonl

3. Pre-Process via HTTP

# Build and start the server
cargo build --release --features server
./target/release/akh-medu-server &

# Wait for startup
sleep 1

# Check health
curl -s http://localhost:8200/health | python3 -m json.tool

# Pre-process chunks
curl -s -X POST http://localhost:8200/preprocess \
  -H 'Content-Type: application/json' \
  -d '{
    "chunks": [
      {"id": "en-1", "text": "The cell membrane contains phospholipids."},
      {"id": "ru-1", "text": "Клеточная мембрана содержит фосфолипиды."}
    ]
  }' | python3 -m json.tool

# Check supported languages
curl -s http://localhost:8200/languages | python3 -m json.tool

4. Interpret the Output

Each output object maps directly to Eleutherios's data model:

5. Feed into Eleutherios

Use the structured output to seed Eleutherios's extraction pipeline. The pre-extracted entities and claims give the LLM a head start:

import json

# Read akh-medu output
with open("/tmp/structured.jsonl") as f:
    for line in f:
        result = json.loads(line)

        # Use source_language to route to language-specific models
        lang = result["source_language"]

        # Pre-extracted entities → seed Neo4j / entity resolution
        for entity in result["entities"]:
            # entity["canonical_name"] is already cross-language resolved
            upsert_entity(entity["canonical_name"], entity["entity_type"])

        # Pre-extracted claims → skip LLM for high-confidence relations
        for claim in result["claims"]:
            if claim["confidence"] >= 0.85:
                # High confidence: ingest directly
                insert_triple(claim["subject"], claim["predicate"], claim["object"])
            else:
                # Lower confidence: send to LLM for validation
                queue_for_llm_validation(claim)

What Meaning We Extract

Claim Types

The pre-processor classifies every extracted relation into a claim type that Eleutherios can use to route into the appropriate dimension of its extraction pipeline:

Entity Types

Canonical Predicates

All languages map to the same 9 canonical predicates, ensuring Eleutherios receives uniform relation labels regardless of source language:

Available Languages

Language Detection

Detection uses a two-stage approach that requires no external NLP models:

Stage 1 — Script analysis (highest confidence):

Stage 2 — Latin disambiguation (word frequency + diacritics):

Each Latin-script text is scored against word frequency markers:

Winner is the language with the highest normalized score. Confidence ranges from 0.60–0.85.

Mixed-Language Corpora

For documents that contain multiple languages (e.g., English text with Russian quotations or Arabic transliterations), use auto-detection per chunk rather than forcing a single language:

# Each chunk is detected independently — this is the default behavior
cat mixed_corpus.jsonl | ./target/release/akh-medu preprocess --format jsonl > structured.jsonl

You can also split mixed-language documents at the sentence level before chunking:

# Pre-split mixed-language paragraphs into per-sentence chunks
import json

paragraph = "The experiment was conducted in Moscow. Результаты были опубликованы в журнале."

# Simple sentence-level splitting
sentences = [s.strip() for s in paragraph.replace('. ', '.\n').split('\n') if s.strip()]
for i, sentence in enumerate(sentences):
    print(json.dumps({"id": f"para1-s{i}", "text": sentence}))

akh-medu's internal detect_per_sentence() function handles this automatically when processing through the grammar module.

Cross-Language Entity Resolution

When the same entity appears in different languages, the resolver unifies them under a canonical English label:

"Moscow"  (EN) -> canonical: "Moscow"
"Москва"  (RU) -> canonical: "Moscow", aliases: ["Москва"]
"Moscou"  (FR) -> canonical: "Moscow", aliases: ["Moscou"]
"موسكو"   (AR) -> canonical: "Moscow", aliases: ["موسكو"]

Static Equivalence Table

The compiled-in static table covers ~120 entries across categories:

Dynamic Equivalence Learning

Beyond the static table, akh-medu can discover new equivalences dynamically using three learning strategies. Discovered mappings persist across sessions via the durable store (redb).

4-Tier Resolution Order

When resolving an entity, the resolver checks in this order:

1. Runtime aliases — hot in-memory mappings added during the current session
2. Learned equivalences — persisted mappings discovered by learning strategies
3. Static equivalence table — ~120 hand-curated entries compiled into the binary
4. Fallback — return the surface form unchanged

Learned equivalences override the static table, allowing domain-specific corrections.

Strategy 1: KG Structural Fingerprints

Two entities in different languages that share identical relational patterns are likely the same concept.

How it works:

1. For each unresolved entity e, collect its "relational fingerprint": the set of (predicate, resolved_object) tuples from the knowledge graph
2. For each already-resolved entity c, collect its fingerprint too
3. If e and c share >= 1 fingerprint tuple, propose e -> c with confidence based on the overlap ratio

Example: If the KG has ("собака", is-a, "млекопитающее") and ("Dog", is-a, "mammal"), and "млекопитающее" already resolves to "mammal", then "собака" structurally maps to "Dog".

Strategy 2: VSA Similarity

Hypervector encodings capture distributional similarity. For Latin-script near-matches (programme/program, organisation/organization) and transliterated names, VSA similarity catches what string matching misses. See VSA Similarity Algorithms for a full explanation.

How it works:

1. For each unresolved entity, encode its label as a hypervector
2. Search item memory for the 5 nearest neighbors
3. For results above the similarity threshold (>= 0.65), check if the matched symbol resolves to a known canonical
4. If yes, propose the equivalence with confidence = similarity score

Strategy 3: Parallel Chunk Co-occurrence

When Eleutherios sends parallel translations of the same content, entities at corresponding positions are likely equivalent.

How it works:

1. Group chunks by shared chunk_id prefix (e.g., "doc1_en" and "doc1_ru" share prefix "doc1")
2. For each group with different languages, align entities by extraction order
3. For entities at the same index across languages, propose equivalence

Chunk ID convention: Use {document_id}_{language_code} format:

{"id": "report-ch3_en", "text": "The experiment was conducted in Moscow.", "language": "en"}
{"id": "report-ch3_ru", "text": "Эксперимент был проведён в Москве.", "language": "ru"}

Equivalence Sources

Each learned equivalence records how it was discovered:

Managing Equivalences via CLI

# List all learned equivalences
./target/release/akh-medu equivalences list

# Show counts by source (kg-structural, vsa-similarity, co-occurrence, manual)
./target/release/akh-medu equivalences stats

# Run all learning strategies on current engine state
./target/release/akh-medu equivalences learn

# Export to JSON for manual curation
./target/release/akh-medu equivalences export > /tmp/equivalences.json

# Import curated equivalences
./target/release/akh-medu equivalences import < /tmp/equivalences.json

Managing Equivalences via HTTP

# List all learned equivalences
curl -s http://localhost:8200/equivalences | python3 -m json.tool

# Show statistics
curl -s http://localhost:8200/equivalences/stats | python3 -m json.tool
# => {"runtime_aliases": 0, "learned_total": 12, "kg_structural": 3, "vsa_similarity": 4, "co_occurrence": 2, "manual": 3}

# Trigger learning
curl -s -X POST http://localhost:8200/equivalences/learn | python3 -m json.tool
# => {"discovered": 5, "total_learned": 17}

# Import curated equivalences
curl -X POST http://localhost:8200/equivalences/import \
  -H 'Content-Type: application/json' \
  -d '[
    {"canonical": "mitochondria", "surface": "митохондрия", "source_language": "ru", "confidence": 0.95, "source": "Manual"},
    {"canonical": "cell membrane", "surface": "клеточная мембрана", "source_language": "ru", "confidence": 0.95, "source": "Manual"}
  ]'
# => {"imported": 2}

Seeding Domain-Specific Equivalences

For specialized corpora (medical terminology, legal terms, engineering jargon), seed the equivalence table with domain-specific terms before processing:

1. Create an equivalence file (/tmp/domain-terms.json):

[
  {"canonical": "mitochondria", "surface": "митохондрия", "source_language": "ru", "confidence": 1.0, "source": "Manual"},
  {"canonical": "mitochondria", "surface": "mitochondrie", "source_language": "fr", "confidence": 1.0, "source": "Manual"},
  {"canonical": "mitochondria", "surface": "mitocondria", "source_language": "es", "confidence": 1.0, "source": "Manual"},
  {"canonical": "photosynthesis", "surface": "фотосинтез", "source_language": "ru", "confidence": 1.0, "source": "Manual"},
  {"canonical": "photosynthesis", "surface": "photosynthèse", "source_language": "fr", "confidence": 1.0, "source": "Manual"},
  {"canonical": "photosynthesis", "surface": "fotosíntesis", "source_language": "es", "confidence": 1.0, "source": "Manual"},
  {"canonical": "enzyme", "surface": "фермент", "source_language": "ru", "confidence": 1.0, "source": "Manual"},
  {"canonical": "enzyme", "surface": "إنزيم", "source_language": "ar", "confidence": 1.0, "source": "Manual"}
]

2. Import before processing:

# Via CLI
./target/release/akh-medu equivalences import < /tmp/domain-terms.json

# Via HTTP
curl -X POST http://localhost:8200/equivalences/import \
  -H 'Content-Type: application/json' \
  -d @/tmp/domain-terms.json

3. Process your corpus — imported terms will be resolved:

echo '{"text": "Митохондрия является органоидом клетки."}' | \
  ./target/release/akh-medu preprocess --format jsonl
# entities[0].canonical_name will be "mitochondria" instead of "митохондрия"

4. Export/import workflow for iterative curation:

# Process a batch, let learning discover new equivalences
cat /tmp/corpus.jsonl | ./target/release/akh-medu preprocess --format jsonl > /dev/null
./target/release/akh-medu equivalences learn

# Export for review
./target/release/akh-medu equivalences export > /tmp/review.json

# Edit review.json manually (fix mistakes, add missing terms)
# Then re-import
./target/release/akh-medu equivalences import < /tmp/review.json

VSA Similarity Algorithms

akh-medu uses Vector Symbolic Architecture (VSA) — also known as Hyperdimensional Computing — to encode symbols, detect similar entities, and support fuzzy matching across the pipeline.

How Hypervectors Work

A hypervector is a high-dimensional binary vector (default: 10,000 dimensions) where each dimension is a single bit interpreted as +1 or -1 (bipolar encoding).

Key properties:

Encoding Strategies

akh-medu uses four encoding functions, each serving a different purpose:

1. Symbol Encoding (encode_symbol)

Maps a SymbolId to a deterministic hypervector using seeded random generation. The same ID always produces the same vector:

SymbolId(42) → deterministic random HyperVec (seeded with 42)
SymbolId(43) → different deterministic random HyperVec (seeded with 43)

This is the base encoding — every symbol in the system gets one.

2. Token Encoding (encode_token)

Maps a text string to a hypervector by hashing it to a synthetic SymbolId:

"dog"  → hash("dog")  → synthetic SymbolId → deterministic HyperVec
"dogs" → hash("dogs") → different SymbolId  → different HyperVec

Token encoding is deterministic: the same word always produces the same vector. However, "dog" and "dogs" produce unrelated vectors (no morphological awareness).

3. Label Encoding (encode_label)

For multi-word labels, encodes each word separately and bundles them:

"big red dog" → bundle(encode_token("big"), encode_token("red"), encode_token("dog"))

The resulting vector is similar to each component (similarity > 0.55) but identical to none. This captures set-like semantics: "big red dog" is similar to anything containing "big", "red", or "dog".

Single-word labels fall through to encode_token directly.

4. Role-Filler Encoding (encode_role_filler)

Encodes structured knowledge by binding a role vector with a filler vector:

bind(encode_symbol(color_id), encode_symbol(blue_id))
  → "the color is blue" as a single vector

Binding (XOR) produces a vector that is dissimilar to both inputs but can be decoded: unbind(bound, color_id) ≈ blue_id.

5. Sequence Encoding (encode_sequence)

Captures order information using permutation:

[A, B, C] → bundle(permute(A, 2), permute(B, 1), C)

Each element is shifted by its distance from the end, preserving positional information. [A, B, C] produces a different vector than [C, B, A].

Similarity Search

Item Memory provides fast approximate nearest-neighbor (ANN) search using the HNSW algorithm with Hamming distance:

Query: encode_token("programme")
Search: item_memory.search(&query_vec, k=5)
Results: [
  { symbol: "program",   similarity: 0.72 },  // near-match
  { symbol: "procedure", similarity: 0.53 },  // unrelated
  { symbol: "project",   similarity: 0.51 },  // unrelated
]

The search is sub-linear — HNSW provides O(log n) search time even with millions of vectors.

How ANN search works internally:

1. The query vector is encoded as packed u32 words for HNSW compatibility
2. HNSW navigates its layered graph using Hamming distance (bitwise XOR + popcount)
3. Raw Hamming distances are converted to similarity: similarity = 1.0 - (hamming / total_bits)
4. Results are sorted by descending similarity

Similarity Thresholds

How VSA Is Used in the Pipeline

VSA operates at three points in the akh-medu pipeline:

1. Lexer — Fuzzy Token Resolution

When the lexer encounters an unknown word, it encodes it as a hypervector and searches item memory for similar known symbols:

Input:  "programm" (misspelling)
Lookup: registry.lookup("programm") → None
Fuzzy:  item_memory.search(encode_token("programm"), k=3)
Match:  "program" with similarity 0.68 → Resolution::Fuzzy

The threshold is 0.60 (DEFAULT_FUZZY_THRESHOLD). Below this, the token stays Unresolved.

2. Dynamic Equivalence Learning — Strategy 2

When learn_equivalences() is called, the VSA strategy encodes unresolved entity labels and searches for similar resolved entities:

Unresolved: "organisation" (British spelling)
Search:     5 nearest neighbors in item memory
Match:      "organization" (American spelling), similarity 0.71
Result:     LearnedEquivalence { surface: "organisation", canonical: "organization", confidence: 0.71 }

The threshold is 0.65 (higher than lexer fuzzy matching for cross-lingual safety).

3. Knowledge Graph — Similarity Queries

The engine exposes similarity search for ad-hoc queries:

# Find symbols similar to a given symbol
# (Used internally by the agent module and available via Engine API)
engine.search_similar_to(symbol_id, top_k=10)

Extending with New Languages

Adding a new language requires changes in three files and takes ~30 minutes. Here is the complete procedure using German as an example.

Step 1: Add the Language Variant

File: src/grammar/lexer.rs

Add the new variant to the Language enum:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default, Serialize, Deserialize)]
pub enum Language {
    English,
    Russian,
    Arabic,
    French,
    Spanish,
    German,       // <-- add here
    #[default]
    Auto,
}
}

Update the three methods on Language:

#![allow(unused)]
fn main() {
impl Language {
    pub fn bcp47(&self) -> &'static str {
        match self {
            // ...existing arms...
            Language::German => "de",
        }
    }

    pub fn from_code(code: &str) -> Option<Self> {
        match code.to_lowercase().as_str() {
            // ...existing arms...
            "de" | "german" => Some(Language::German),
            _ => None,
        }
    }
}

impl std::fmt::Display for Language {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            // ...existing arms...
            Language::German => write!(f, "German"),
        }
    }
}
}

Step 2: Add the Language Lexicon

File: src/grammar/lexer.rs

A lexicon defines five components for each language:

Add an arm to Lexicon::for_language() and create the lexicon constructor:

#![allow(unused)]
fn main() {
impl Lexicon {
    pub fn for_language(lang: Language) -> Self {
        match lang {
            // ...existing arms...
            Language::German => Self::default_german(),
        }
    }

    /// Build the German lexicon.
    pub fn default_german() -> Self {
        let void_words = vec![
            "der".into(), "die".into(), "das".into(),
            "ein".into(), "eine".into(), "eines".into(),
            "dem".into(), "den".into(), "des".into(),
        ];

        // Map German surface forms to canonical predicates.
        // IMPORTANT: Sort longest patterns first for greedy matching.
        let relational_patterns = vec![
            // 4+ word patterns first
            rel("befindet sich in", "located-in", 0.90),
            rel("ist ähnlich wie", "similar-to", 0.85),
            rel("ist zusammengesetzt aus", "composed-of", 0.85),
            rel("ist Teil von", "part-of", 0.90),
            rel("hängt ab von", "depends-on", 0.85),
            // 2-word patterns
            rel("ist ein", "is-a", 0.90),
            rel("ist eine", "is-a", 0.90),
            rel("hat ein", "has-a", 0.85),
            rel("hat eine", "has-a", 0.85),
            // 1-word patterns last
            rel("enthält", "contains", 0.85),
            rel("verursacht", "causes", 0.85),
            rel("ist", "is-a", 0.80),
            rel("hat", "has-a", 0.80),
        ];

        let question_words = vec![
            "was".into(), "wer".into(), "wo".into(), "wann".into(),
            "wie".into(), "warum".into(), "welcher".into(), "welche".into(),
        ];

        let goal_verbs = vec![
            "finden".into(), "entdecken".into(), "erforschen".into(),
            "analysieren".into(), "bestimmen".into(), "identifizieren".into(),
        ];

        // Commands stay English (CLI is English)
        let commands = vec![
            ("help".into(), CommandKind::Help),
            ("?".into(), CommandKind::Help),
            ("status".into(), CommandKind::ShowStatus),
        ];

        Self { void_words, relational_patterns, question_words, goal_verbs, commands }
    }
}
}

Writing relational patterns — guidelines:

• Patterns are matched greedily, longest first. Always sort multi-word patterns before shorter ones.
• Map every pattern to one of the 9 canonical predicates: is-a, has-a, contains, located-in, causes, part-of, composed-of, similar-to, depends-on.
• Assign confidence scores between 0.80 and 0.90:
  • 0.90 for unambiguous patterns ("ist ein" is always classification)
  • 0.85 for patterns with occasional false positives
  • 0.80 for very short/ambiguous patterns (bare "ist" could be copula or identity)
• Include accent-stripped variants for languages with diacritics (the existing French and Spanish lexicons do this).

Step 3: Add Detection Support

File: src/grammar/detect.rs

This step makes --language auto work for the new language.

For non-Latin scripts (e.g., Chinese, Japanese, Korean, Hindi), add a codepoint range check in the detect_language() function:

#![allow(unused)]
fn main() {
// In the character-counting loop:
match c {
    // CJK Unified Ideographs
    '\u{4E00}'..='\u{9FFF}' => { cjk += 1; }
    // Devanagari
    '\u{0900}'..='\u{097F}' => { devanagari += 1; }
    // ...
}

// After the loop, add a block like the Cyrillic/Arabic ones:
if cjk_ratio > 0.5 {
    return DetectionResult {
        language: Language::Chinese,
        confidence: (0.70 + cjk_ratio * 0.25).min(0.95),
    };
}
}

For Latin-script languages (like German), add word frequency markers to detect_latin_language():

#![allow(unused)]
fn main() {
const GERMAN_MARKERS: &[&str] = &[
    "der", "die", "das", "ist", "und", "ein", "eine", "nicht",
    "mit", "auf", "für", "von", "sich", "den", "dem", "auch",
    "werden", "haben", "sind", "wird", "kann", "nach", "über",
];
}

Then add a german_score accumulator alongside english_score, french_score, spanish_score, and include German-specific diacritics (ä, ö, ü, ß):

#![allow(unused)]
fn main() {
let has_german_diacritics = lower.contains('ä')
    || lower.contains('ö')
    || lower.contains('ü')
    || lower.contains('ß');

if has_german_diacritics {
    german_score += 2.0;
}
}

Finally, include German in the winner selection:

#![allow(unused)]
fn main() {
let max_score = en_norm.max(fr_norm).max(es_norm).max(de_norm);
// ... pick the winner with the highest score
}

Step 4: Add Equivalences

File: src/grammar/equivalences.rs

Add the new language's surface forms to existing entries:

#![allow(unused)]
fn main() {
Equivalence { canonical: "Germany", aliases: &["Allemagne", "Alemania", "Германия", "ألمانيا", "Deutschland"] },
//                                                                                              ^^^^^^^^^^ add
}

And add new entries for language-specific terms:

#![allow(unused)]
fn main() {
Equivalence { canonical: "psyche", aliases: &["Psyche", "психика", "نفس", "psyché", "psique"] },
}

You can also import equivalences at runtime instead of modifying the source:

echo '[
  {"canonical": "Germany", "surface": "Deutschland", "source_language": "de", "confidence": 1.0, "source": "Manual"}
]' | ./target/release/akh-medu equivalences import

Step 5: Rebuild and Test

# Run tests
cargo test --lib

# Build release binary
cargo build --release

# Test with sample text (explicit language)
echo '{"text":"Der Archetyp ist ein universelles Muster."}' \
  | ./target/release/akh-medu preprocess --format jsonl --language de

# Test auto-detection (if Step 3 was implemented)
echo '{"text":"Die Zelle enthält Mitochondrien und andere Organellen."}' \
  | ./target/release/akh-medu preprocess --format jsonl

Checklist for New Languages

• Add variant to Language enum in lexer.rs
• Update bcp47(), from_code(), Display on Language
• Add arm to Lexicon::for_language()
• Create Lexicon::default_LANG() with:
  • Void words (articles, determiners)
  • Relational patterns (sorted longest-first, mapping to canonical predicates)
  • Question words
  • Goal verbs
  • Commands (usually keep English)
• (Optional) Add detection markers in detect.rs for Language::Auto support
• (Optional) Add cross-lingual entries in equivalences.rs
• Run cargo test --lib (must pass, zero warnings)
• Test with sample text: echo '{"text":"..."}' | akh-medu preprocess --format jsonl --language XX

CLI Reference

Global Options

Commands

init

Initialize a new akh-medu data directory:

./target/release/akh-medu --data-dir /tmp/akh-data init

preprocess

Pre-process text chunks from stdin:

# JSONL mode (streaming, one object per line)
cat chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl

# JSON mode (batch, array on stdin)
cat chunks.json | ./target/release/akh-medu preprocess --format json

# With explicit language
cat chunks.jsonl | ./target/release/akh-medu preprocess --format jsonl --language ru

ingest

Ingest structured data into the knowledge graph:

# JSON triples
./target/release/akh-medu --data-dir /tmp/akh-data ingest --file /path/to/triples.json

# CSV (subject, predicate, object format)
./target/release/akh-medu --data-dir /tmp/akh-data ingest --file /path/to/data.csv --format csv --csv-format spo

# CSV (entity format: column headers are predicates)
./target/release/akh-medu --data-dir /tmp/akh-data ingest --file /path/to/data.csv --format csv --csv-format entity

# Plain text
./target/release/akh-medu --data-dir /tmp/akh-data ingest --file /path/to/text.txt --format text --max-sentences 100

equivalences

Manage cross-lingual entity equivalences:

./target/release/akh-medu equivalences list      # Show all learned equivalences
./target/release/akh-medu equivalences stats     # Counts by source
./target/release/akh-medu equivalences learn     # Run learning strategies
./target/release/akh-medu equivalences export    # Export to JSON (stdout)
./target/release/akh-medu equivalences import    # Import from JSON (stdin)

grammar

Grammar system commands:

# List available grammar archetypes
./target/release/akh-medu grammar list

# Parse prose to abstract syntax
./target/release/akh-medu grammar parse "Dogs are mammals"

# Parse and ingest into knowledge graph
./target/release/akh-medu --data-dir /tmp/akh-data grammar parse "Dogs are mammals" --ingest

# Linearize a triple (generate prose from structured data)
./target/release/akh-medu grammar linearize --subject Dog --predicate is-a --object mammal

# Compare a triple to knowledge graph
./target/release/akh-medu --data-dir /tmp/akh-data grammar compare --subject Dog --predicate is-a --object mammal

# Load a custom TOML grammar
./target/release/akh-medu grammar load --file /path/to/grammar.toml

# Render an entity's knowledge graph neighborhood
./target/release/akh-medu --data-dir /tmp/akh-data grammar render --entity Dog

HTTP API Reference

Server: Listens on 0.0.0.0:8200

Build/Run:

cargo build --release --features server
./target/release/akh-medu-server

Environment:

• RUST_LOG — Logging level (default: info,egg=warn,hnsw_rs=warn)

GET /health

Health check.

Response:

{
  "status": "ok",
  "version": "0.1.0",
  "supported_languages": ["en", "ru", "ar", "fr", "es", "auto"]
}

GET /languages

List supported languages with pattern counts.

Response:

{
  "languages": [
    {"code": "en", "name": "English", "pattern_count": 21},
    {"code": "ru", "name": "Russian", "pattern_count": 13},
    {"code": "ar", "name": "Arabic", "pattern_count": 11},
    {"code": "fr", "name": "French", "pattern_count": 16},
    {"code": "es", "name": "Spanish", "pattern_count": 14}
  ]
}

POST /preprocess

Pre-process text chunks.

Request:

{
  "chunks": [
    {"id": "optional-id", "text": "Text to process.", "language": "en"}
  ]
}

Fields: id and language are optional. Omit language for auto-detection.

Response:

{
  "results": [
    {
      "chunk_id": "optional-id",
      "source_language": "en",
      "detected_language_confidence": 0.80,
      "entities": [...],
      "claims": [...],
      "abs_trees": [...]
    }
  ],
  "processing_time_ms": 0
}

GET /equivalences

List all learned equivalences.

Response:

[
  {
    "canonical": "Moscow",
    "surface": "москва",
    "source_language": "ru",
    "confidence": 0.95,
    "source": "Manual"
  }
]

GET /equivalences/stats

Equivalence statistics by source.

Response:

{
  "runtime_aliases": 0,
  "learned_total": 12,
  "kg_structural": 3,
  "vsa_similarity": 4,
  "co_occurrence": 2,
  "manual": 3
}

POST /equivalences/learn

Trigger all learning strategies.

Response:

{
  "discovered": 5,
  "total_learned": 17
}

POST /equivalences/import

Bulk import equivalences.

Request: Array of LearnedEquivalence objects (see GET /equivalences for format).

Response:

{
  "imported": 3
}

Integration Patterns

Python Integration

Three integration approaches, from simplest to most performant:

Subprocess (JSON batch)

Best for batch processing where latency per call isn't critical:

import subprocess
import json
import os

def preprocess_chunks(chunks: list[dict]) -> list[dict]:
    """Send chunks through akh-medu for multilingual pre-processing."""
    input_json = json.dumps(chunks)
    result = subprocess.run(
        ["./target/release/akh-medu", "preprocess", "--format", "json"],
        input=input_json,
        capture_output=True,
        text=True,
        env={**os.environ, "RUST_LOG": "error"},
    )
    if result.returncode != 0:
        raise RuntimeError(f"akh-medu failed: {result.stderr}")
    response = json.loads(result.stdout)
    return response["results"]

# Usage
results = preprocess_chunks([
    {"id": "1", "text": "The cell membrane contains phospholipids."},
    {"id": "2", "text": "Клеточная мембрана содержит фосфолипиды."},
])

Subprocess (JSONL streaming)

Best for large corpora where you want to process chunks as they arrive:

import subprocess
import json
import os

def preprocess_stream(chunks):
    """Stream chunks through akh-medu one at a time."""
    proc = subprocess.Popen(
        ["./target/release/akh-medu", "preprocess", "--format", "jsonl"],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        text=True,
        env={**os.environ, "RUST_LOG": "error"},
    )
    for chunk in chunks:
        proc.stdin.write(json.dumps(chunk) + "\n")
        proc.stdin.flush()
        line = proc.stdout.readline()
        if line:
            yield json.loads(line)
    proc.stdin.close()
    proc.wait()

# Usage
for result in preprocess_stream(chunks_iterator):
    process_result(result)

HTTP Client

Best for long-running services where the akh-medu server stays up:

import requests

AKH_MEDU_URL = "http://localhost:8200"

def preprocess_http(chunks: list[dict]) -> list[dict]:
    """Call the akh-medu HTTP server."""
    resp = requests.post(
        f"{AKH_MEDU_URL}/preprocess",
        json={"chunks": chunks},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["results"]

def learn_equivalences() -> dict:
    """Trigger equivalence learning."""
    resp = requests.post(f"{AKH_MEDU_URL}/equivalences/learn", timeout=60)
    resp.raise_for_status()
    return resp.json()

def import_equivalences(equivs: list[dict]) -> int:
    """Import domain-specific equivalences."""
    resp = requests.post(
        f"{AKH_MEDU_URL}/equivalences/import",
        json=equivs,
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["imported"]

# Usage
results = preprocess_http([
    {"id": "1", "text": "CERN is located in Geneva."},
    {"id": "2", "text": "Le CERN se trouve dans Genève."},
])

Eleutherios Mapping

How akh-medu output maps to Eleutherios concepts:

Architecture Notes

Performance

• Grammar parsing runs in < 1ms per chunk — 196-chunk batch in 0.8s (~300 chunks/sec)
• Batches of 20 chunks process in ~55ms (measured with HTTP endpoint)
• No external NLP dependencies, no model loading, no GPU required
• The HTTP server handles concurrent requests via tokio with a RwLock<Engine>
• Memory footprint: ~50MB for the engine with default 10,000-dimension hypervectors
• HNSW ANN search: O(log n) for similarity queries

What the Pre-Processor Does NOT Do

• Coreference resolution: "He studied in Zurich" — "He" is not resolved
• Complex clause parsing: Subordinate clauses, relative clauses, passives
• Morphological analysis: No lemmatization (Russian "психики" stays inflected)
• Named Entity Recognition: Entity types are inferred from predicate context only

These are deliberate scope limits. Eleutherios's LLM pipeline handles them in its enrichment stage. The pre-processor's job is to give it a clean head start with the structural relations that grammar can catch deterministically.

The abs_trees Field

The output includes full AbsTree abstract syntax trees for consumers that want the raw parse. This is useful for:

• Debugging parse quality
• Custom post-processing beyond the entity/claim extraction
• Feeding back into akh-medu's knowledge graph via Engine::commit_abs_tree()

Persistence

When started with --data-dir, the engine persists:

• Symbol registry (all known symbols and their metadata)
• Knowledge graph triples
• Learned equivalences (via equiv: prefix in redb)
• Agent session state (working memory, cycle count)

Data is stored in a 3-tier architecture:

• Hot tier: In-memory DashMap for fast concurrent access
• Warm tier: Memory-mapped files for large read-heavy data
• Durable tier: redb (ACID transactions) for data that must survive restarts

Relational Pattern Reference

English (21 patterns)

Russian (13 patterns)

Arabic (11 patterns)

French (16 patterns)

Spanish (14 patterns)