Architecture

Overview

Loomem is a Rust workspace with four crates:

All business logic lives in loomem-core. The server is a thin HTTP layer that maps routes to core functions.

Data model

Chunk

The fundamental unit of memory. Every piece of knowledge — raw event, compressed summary, semantic group — is a Chunk.

chunk:L{level}:{uuid}  →  Chunk (JSON in RocksDB)

Core fields:

Consolidation fields:

Contradiction / versioning fields:

Knowledge extraction metadata:

Memory tiers

L0 (Raw)          L1 (Compressed)
─────────         ─────────────
Verbatim input    LLM-summarized
Fast decay        Medium decay

Promotion flow:

Ingest → L0 (raw event)
            │
            ▼  [Consolidation worker, every 5 min]
         L1 (compressed observations)

A separate clustering worker periodically groups L1 chunks by embedding similarity; the clusters feed the associator (serendipity engine), not a separate storage tier.

Entity graph

Entities and their relationships form a knowledge graph stored in RocksDB:

EntityNode {
    canonical_name: "Cursor"
    entity_type: "TECHNOLOGY"
    aliases: ["cursor", "Cursor IDE"]
    chunk_ids: ["abc-123", "def-456"]   // evidence
}

Edge {
    source: "Alice"
    target: "Cursor"
    relation_type: "uses"
    chunk_ids: ["abc-123"]              // evidence
}

Per-stream isolation: Each entity and edge is scoped to a stream_id. The same entity name in two streams gets separate entity nodes. Name/alias indexes are prefixed by stream.

RocksDB key scheme for graph:

graph:entity:{id}                    → EntityNode (JSON, includes stream_id)
graph:s:{stream_id}:name:{lower()}   → entity_id
graph:s:{stream_id}:alias:{lower()}  → entity_id
graph:adj:{src_id}:{edge_id}         → target_id
graph:radj:{tgt_id}:{edge_id}        → source_id
graph:chunk:{chunk_id}               → [entity_ids]  (reverse index)

Storage layer

RocksDB

Primary persistent store. Holds chunks, embeddings, graph, users, cost tracking.

Column families:

Configuration: - Compression: LZ4 - Write buffer: 64 MB x 3 buffers - Max open files: 1000

Tantivy

Full-text search index. Mirrors chunk content with additional indexed fields.

Schema:

Polish stemming enabled for content field.

Vector store

Embeddings stored in RocksDB's embeddings column family.

Providers: - Local (default) — multilingual-e5-small (384 dimensions) via the tract ONNX runtime (pure Rust, no native dependencies, runs offline) - OpenAI — text-embedding-3-small (1536 dimensions), opt-in via embedding_provider = "openai"

Embedding queue: batch processing (50 items, 5s flush interval) to amortize API latency.

Intent log (WAL)

Write-ahead log ensures cross-store consistency between RocksDB and Tantivy.

1. Append: PENDING(op_type, chunk_id)
2. Write to RocksDB
3. Write to Tantivy
4. Append: COMMITTED(op_type, chunk_id)

On crash recovery: - PENDING without COMMITTED → rollback partial writes - COMMITTED → verify both stores have the data

Ingest pipeline

Content is sanitized before any storage:

Input content
    │
    ▼  [sanitizer.rs]
 1. HTML tag stripping + entity decoding
 2. Instruction injection detection (18 patterns — logs warning, does not block)
    │
    ▼  [pii_filter.rs]
 3. PII redaction (phones, emails, PESEL, blocklist words)
    │
    ▼  [persist_chunk]
 4. RocksDB store (sanitized content)
 5. Entity extraction + graph population (stream-scoped)
 6. Tantivy indexing
 7. Embedding queue
 8. Contradiction detection

The sanitizer detects but does not block injection attempts — content is stored after stripping. PII redaction replaces sensitive data with [PHONE], [EMAIL], [ID], [REDACTED] tokens before storage.

Source → fragments → index (opt-in, [fragments] enabled = true, off by default). On memory_ingest, the whole conversation is stored first as a canonical source chunk; the fragments step then reads that source back as persisted and cuts it into turn-level fragments (one per transcript turn, long turns split further at paragraph, sentence and whitespace boundaries) that fit the embedder's input window. Each fragment is a level-0 chunk ({source_id}:f{index}) carrying the author role, the conversation date and an exact byte range into the source, and goes through the same persist_chunk path as the source (RocksDB store, entity extraction, Tantivy indexing, embedding queue), so it is indexed and embedded on its own. Fragments are consolidated from birth (never consolidation or dream input), carry no event date of their own, and are tombstoned together with their source. Fact extraction runs after the fragments step and is never blocked by it. See Configuration.

Search pipeline

A query flows through multiple stages:

1. Query classification

"what do you know about me?"   → Profile
"how many projects do I run?"  → Aggregation (top_k boosted to 30)
"when did I change my IDE?"    → Temporal (date filtering)
"why did I choose Rust?"       → Complex (top_k = 20)
"my dog's name"                → Simple (top_k = 3, BM25 only)

2. Date filter extraction

Parses relative dates from query text: - "last week" → date_from: now - 7d - "in March" → date_from: 2026-03-01, date_to: 2026-03-31 - Explicit date_from/date_to params override

3. Parallel search

BM25 and vector search run concurrently:

BM25 (Tantivy):

QueryParser(content^1.0, entities^0.2, relations^0.2)
  + stream filter
  + date range filter
  → ranked results

Vector (cosine similarity):

query_embedding = embed(query_text)
for each stored embedding:
    score = cosine(query_embedding, chunk_embedding)
→ top-K by similarity

4. Fusion

normalized_bm25   = bm25_score / max_bm25
normalized_vector  = vector_score / max_vector
fusion_score       = 0.6 * normalized_vector + 0.4 * normalized_bm25

5. Time decay

age_days = (now - chunk.timestamp) / 86400
lambda   = { L0: search.decay.l0_lambda, L1: search.decay.l1_lambda }
decay    = e^(-lambda * age_days)
score    = fusion_score * decay

6. Graph enhancement

For top results, find related entities and add their connected chunks:

result "Cursor" → entity "Cursor" → edges → related entities
  → neighbor chunks added with score * boost_factor (0.3)

7. Deduplication

Collapse near-identical results (high cosine similarity between result contents).

8. Optional reranking

If enabled, top-20 candidates are re-scored by: - ONNX cross-encoder (local, ~97ms/pair) — or - Async LLM reranking with speculative cache

9. Implicit boost

Non-dry-run searches increment access_count and boost importance of returned chunks (capped at 1.5, 1-hour cooldown).

10. Response

{
  "results": [
    {
      "chunk_id": "abc-123",
      "content": "User prefers Cursor over VSCode",
      "score_final": 0.87,
      "trace_info": {
        "level": "L1",
        "source": "consolidation",
        "is_latest": true,
        "access_count": 5
      }
    }
  ],
  "trace_metadata": {
    "total_results_before_topk": 42,
    "search_latency_us": 1200
  }
}

Background workers

The scheduler orchestrates background jobs. All workers can be paused/resumed at runtime via POST /admin/workers/pause|resume (useful for eval runs). Pause state is an AtomicBool shared between AppState and Scheduler — does not survive restart.

Consolidation (L0 → L1)

Process: 1. Scan unconsolidated L0 chunks 2. Group by stream (user isolation) 3. Sub-group by topic similarity — greedy clustering on embeddings (cosine threshold). Prevents unrelated facts from merging into one L1 chunk. 4. PII redaction (per sub-group) 5. LLM compression (gpt-4.1-mini) — one call per sub-group 6. Create L1 chunk with source_ids linking back to L0 7. Index in Tantivy + embedding queue 8. Mark L0 as consolidated: true

Chunks without embeddings fall back to a single group (pre-clustering behavior). Single-topic streams produce one sub-group — zero regression risk.

Decay

Adaptive decay: chunks with high access_count decay slower (adaptive_dampening / adaptive_cap in [worker.decay_worker]).

Clustering

Cluster output feeds the associator (below); there is no separate storage tier for clusters.

Entity extraction queue

Async LLM NER runs in background for entities not in the dictionary.

Embedding queue

Associator (ECA — serendipity engine)

Components: - Clustering — k-means on chunk embeddings, per-stream - Graph walk — random walk with weak-tie preference (fewer shared chunks = more novel) - Temporal — find chunks near the same time period - Serendipity scoring — relevance × (1 - obviousness) × cluster distance

Hard-purge (retention worker)

Scans for soft-deleted chunks past their recovery window. Purge pipeline: graph references → Tantivy → RocksDB hard delete (chunk + embedding + entities + relations).

Dream (auto-consolidation)

Authentication

Loomem is single-user: one API key controls access to the whole instance.

• The key is read from the env var named by server.auth_token_env in config.toml (default LOOMEM_AUTH_TOKEN).
• All requests require Authorization: Bearer <key>; /health remains open.
• If no key is configured, the server runs in local passthrough mode — every request is accepted with admin privileges. Use only for local development.

Data written without an explicit stream lands in the default stream __user_default__. Additional streams (from [streams] / [namespaces] in config) partition data within the same instance — they are an organizational boundary, not separate identities.

OAuth 2.0

For MCP Remote Connector (claude.ai): - Dynamic Client Registration (RFC 7591) - Authorization Code flow with PKCE - User enters API key during authorization - Access token = API key (no extra token layer)

MCP integration

Loomem implements the Model Context Protocol (MCP) as a JSON-RPC 2.0 endpoint at POST /mcp.

Request flow:

Claude (MCP client)
    │
    ▼
POST /mcp (JSON-RPC)
    │
    ▼
mcp::handler → parse request → extract tool + args
    │
    ▼
mcp::dispatcher → match tool name → call internal handler
    │
    ▼
loomem-core / handlers → execute → return ToolResult
    │
    ▼
JSON-RPC response → Claude

Session management: OAuth tokens map to sessions, sessions map to stream_id for data isolation.

Crash recovery

1. Intent log replay — on startup, scan WAL for uncommitted operations
2. Partial write detection — check RocksDB and Tantivy for consistency
3. Orphan cleanup — remove chunks marked in_progress from failed consolidation
4. Tantivy rebuild — if schema version mismatch, rebuild index from RocksDB source of truth

Cost tracking

Every LLM call (consolidation, extraction, embedding) is tracked:

[cost]
daily_cap_usd = 15.00           # Hard stop
alert_threshold_usd = 10.00     # Warning
anomaly_multiplier = 3.0        # 3x typical = anomaly alert

Costs persisted in RocksDB column family. Workers check budget before each LLM call.