Vector memory on LanceDB: inside the Kireo memory MCP retrieval path

Short answer

A coding agent starts every session cold. You can paste the context back in each time, or you can give it a memory it can query. The second option is only worth it if retrieval is both fast enough to sit inside the tool loop and cheap enough in tokens to earn its place in the prompt. Those two constraints pull against each other, and everything on this page is a consequence of them.

The token constraint is settled by architecture, not by tuning: memory is pulled, not pushed. Nothing is injected into the system prompt. The agent calls memory_search when it judges prior context worth retrieving, and pays only for the hits it asked for — DEFAULT_LIMIT = 10, MAX_LIMIT = 50 (apps/api/src/search/service.ts:82-83). A per-turn injection scheme pays for memory on every message whether it helps or not; a top-k tool pays only when asked, with a ceiling you set.

The latency constraint is settled by keeping the retrieval path short — which is the rest of this page.

The stack, small on purpose

This is not a scale-bragging post. The whole thing runs on one modest VM, and the interesting part is how far that goes when the retrieval path stays lean.

• LanceDB — an embedded columnar vector store. No separate database server: it is a library reading Lance-format files off a disk volume mounted into the API and worker containers. Memory content lives here.
• Neon Postgres — bookkeeping only (row metadata, embedding status). It never stores the memory body. That split is load-bearing later.
• Self-hosted embeddings — HuggingFace Text Embeddings Inference (TEI) running intfloat/multilingual-e5-small on CPU, speaking an OpenAI-compatible /v1/embeddings endpoint on the internal network, with no published port (docker-compose.prod.yml:45-68).

The memories table has 21 top-level columns. Two of them explain most of the design: content_hash (which makes batch writes idempotent) and embedding, an Arrow FixedSizeList(dim) whose width is baked into the table schema — remember that one, it comes back to bite us.

Hybrid search and the real RRF numbers

Pure vector search misses exact matches — a specific error code, a function name. Pure keyword search misses paraphrase. So memory_search runs both arms and fuses them. The constants are not tuned per query; they are four numbers in one file:

The flow, in order:

1. Embed the query, with a 5-second Redis cache keyed on query + filters + model, so a user hammering the same search does not re-embed (apps/api/src/search/query-cache.ts).
2. In parallel (Promise.all), run a vector ANN search over the embedding column (top 50) and a native full-text search (top 50), both scoped to the caller's tenant.
3. Apply the non-namespace predicates — type, entities, tags, importance, occurred_from/to — to the candidate rows before ranking, so the fused top-N is computed over rows that actually match, not filtered after the fact.
4. Fuse both ranked lists with reciprocal rank fusion and slice to limit.

RRF is deliberately boring. It adds 1 / (k + rank) per list without requiring the two scoring systems to share a scale — cosine distance and BM25 do not. Robust and cheap, which is what you want in a hot path.

Here is the part nobody else can tell you, because it falls out of our constants. With k=60, ranks starting at 1, and each arm capped at 50, a document can contribute at most twice. So the entire achievable score range is arithmetic:

That is why min_score behaves the way it does, and why a threshold that looks conservative can silently return nothing — see Gotchas.

Why 384 dimensions, self-hosted, on CPU

The stack did not start here. The default in source is still EMBEDDING_DIM = 1536 (apps/api/src/config/env.ts:59), matching a hosted 1536-dim model; production overrides it via env, and the repo runbook records the production value as 384 — the native width of multilingual-e5-small (docs/superpowers/plans/2026-06-12-self-hosted-embedding.md:102-105).

The selection argument was written down before the switch — a two-model comparison table in the same document — and it was not "bigger is better":

• Memory snippets are short. A decision, a gotcha, a config note. You do not need a frontier-scale vector to separate "we chose row-level security over app-layer checks" from "the CI cache key needs the lockfile hash." Smaller vectors mean cheaper ANN and less storage.
• Multilingual beat English-only. The runner-up was bge-small-en-v1.5 — same 384 dimensions, smaller (33M vs 118M parameters), also MIT-licensed, and faster. It lost on one axis: English-only, against 100+ languages for e5. A memory layer that silently fails to recall a note you wrote in German is worse than one that is a few milliseconds slower. (Model comparison recorded in docs/superpowers/plans/2026-06-12-self-hosted-embedding.md:21-32.)
• Both are honestly a tier below a hosted frontier embedding model on generic benchmarks — the comparison table says so in as many words. The bet was that short, personal, highly specific text is the case where that gap matters least — and that hybrid retrieval covers the residue, because the full-text arm does not care about embedding quality at all.
• TEI's OpenAI-compatible endpoint made it a config change, not a rewrite. The same client code talks to the self-hosted model by pointing a base URL at an internal service (OPENAI_BASE_URL).

One catch worth knowing if you copy this: e5-family models want asymmetric prefixes — passage: for stored documents, query: for search queries. They are supplied from config (EMBED_DOC_PREFIX / EMBED_QUERY_PREFIX, both defaulting to empty for OpenAI-style models) and applied at request time only, so content_hash stays based on the raw text (apps/api/src/config/env.ts:66-70).

The container is pinned by digest, not by tag, with the reason written next to it: an earlier CPU tag of TEI bundled a dependency version that failed model download with a cryptic "relative URL without a base." Pinning means an innocent :latest pull cannot resurrect that bug. It is capped at mem_limit: 1g so it cannot starve the box, and it has no healthcheck — the minimal image ships neither curl nor wget, so an HTTP healthcheck literally cannot run inside it; on fatal error the container exits and restart: unless-stopped brings it back while the app degrades to the queue.

The graceful-degradation ladder

In a lean stack any dependency can be briefly unavailable, and search still has to return something. The ladder in apps/api/src/search/service.ts:98-136 has four rungs:

1. Both arms healthy — ANN + FTS, fused by RRF. The normal path.
2. Query embedding unavailable (provider hiccup, timeout) — drop the vector arm, search keyword-only, and emit a search.embedding.degraded warning so it alerts rather than silently getting worse.
3. No embedding and zero FTS hits (common in a fresh environment with no FTS index) — fall back to a bounded keyword scan.
4. Everything empty — return an empty result set, not a 500.

None of that is glamorous. It is the difference between "memory occasionally returns fewer hits" and "memory throws inside the agent's tool loop," which is the difference between a tool people keep and a tool people disable.

Tenant isolation sits underneath all four rungs: every row carries user_id, isolation is a predicate pushed into every LanceDB query, and results are re-checked after they come back — if any row's user_id does not match the caller, the code throws instead of returning it (tenant assertion failed in lance listForUser). The filter should never be wrong, so we check anyway, on every read path. Cross-tenant leakage is the one bug you never want to ship and a three-line assertion is cheap insurance.

Three war stories

1. The embedding-dimension migration

Moving from 1536 to the self-hosted 384-dim model meant every stored vector was suddenly the wrong length. This is where FixedSizeList(dim) stops being an implementation detail: the width is part of the table schema, and in the LanceDB version we run (0.21) there is no in-place column resize and no table rename. Once the dimension changes, every insert fails against the old table.

The migration is dump → drop → recreate → reload:

1. Read every row out of the memories table.
2. Write a durable JSON dump to disk before anything destructive. Memory content lives only in LanceDB — Postgres has metadata only — so if the recreate dies halfway, that dump is the only copy. Old vectors are dropped on reload anyway, so the dump excludes them.
3. Drop the table; recreate it empty at the target dimension.
4. Re-insert in batches of 500 with embedding = null and embedding_status = 'queued', then reset the Postgres status rows to queued too, so the backfill job re-embeds everything with the new model.

The script is idempotent — already at the target dimension is a no-op — which matters when you are running it by hand on a live box, unsure whether the last attempt finished (apps/api/scripts/migrate-embedding-dim.ts).

2. Making batch writes idempotent

Indexing a repository uploads symbols in batches of up to 100. Batches time out sometimes, and the obvious retry — re-send the batch — creates duplicates unless the write path is idempotent.

The fix is content-hash dedup as a single set query, not N point lookups: before inserting, one query fetches the active (deleted_at IS NULL) rows whose content_hash is in the batch's hashes — content_hash IN (...) for the whole batch — and skips them (lance/memories-table.ts:100-134). One query per batch, not one per symbol, so dedup never lands on the per-item hot path.

The part that matters to you is that this is a documented contract, not an internal nicety: if a batch upload times out, re-running the same command is safe. The person hitting the timeout is exactly the person who needs to trust the retry.

3. What "trash" actually means

Delete is where naive implementations quietly lose data or lie about counts. Deletes are soft: a delete stamps deleted_at and a 30-day expires_at restore window (30 * 24 * 3600 * 1000 ms, literally, in memories-table.ts:265). The row stays in the table, filtered out of normal reads by deleted_at IS NULL. Restore checks the window and refuses once it has expired; a TTL sweep physically removes rows past expires_at. Three follow-on requirements each needed explicit handling:

• A trash view needs the inverse filter. Listing deleted items is not "include deleted" — it is "only deleted," a different query, and it has to win when both flags are set. Get it subtly wrong and the user sees an empty or wrong trash.
• Counts have to match the view. The trashed count is its own query, not total − active. Off-by-a-little count arithmetic is exactly what users notice and stop trusting.
• Updates are in-place, never delete-then-add. LanceDB will happily let you delete and re-add a row, but a crash between those two calls permanently loses it while the Postgres metadata survives — a torn write. Every mutation (edit, soft-delete, restore, namespace rename) is an in-place table.update(), so there is no window in which the row does not exist.

One more storage-shaped wrinkle: a plain LanceDB scan has no ORDER BY. To list newest-first without materializing the whole table, the list path streams every matching batch, re-sorts as they arrive, and truncates to limit + 1 so only the current page stays in memory — with the extra row doubling as the has-next-page signal. More code than ORDER BY … LIMIT, but it is what the engine actually supports.

What repository indexing actually measures

kireo index walks a repository, extracts symbols with tree-sitter, and uploads them into a code-<repo> namespace. We ran the local half only — walk plus extraction plus assembly, the part with no network calls — against this product's own monorepo, three times, and the numbers were identical run to run.

Token figures for the same corpus, for anyone budgeting context: the 213 assembled symbols come to 43,626 tokens (mean 204.8, median 106, p90 428, max 2,248 per symbol), while feeding all 336 files into a prompt whole would be 382,209 tokens. Counted with OpenAI's tiktoken 0.13.0 / o200k_base, not a Claude tokenizer — treat these as order-of-magnitude, not exact. We are deliberately not turning that pair into a "saves N×" claim: it compares a retrieval budget against a hypothetical nobody actually runs, and we have not instrumented a real agent session.

Incremental runs are cheap because state is a file: .kireo/index-state.json holds a SHA-256 per file, and the next run only re-extracts files whose hash changed (packages/mcp-server/src/index/state.ts). Combined with the server-side content_hash dedup from war story 2, re-running the same index is safe rather than duplicative.

Config

All of the above sits behind one stdio MCP server. This is the generic mcpServers stanza — the same shape every MCP client uses, with the explicit "type": "stdio" that some hosts require and others merely tolerate:

And this is a memory_search call using the numbers derived above. min_score: 0.016 is a precise statement, not a round one: it keeps every document that appeared in both arms (minimum 2/110 ≈ 0.0182) plus single-arm hits ranked first or second (1/61 ≈ 0.0164, 1/62 ≈ 0.0161), and drops everything else:

Prerequisite: Node.js ≥ 20 on PATH for npx (packages/mcp-server/package.json engines). Debug logging is safe with stdio: pino writes only to rotating files under ~/.kireo/logs/ plus a warn-level stderr target, so KIREO_LOG_LEVEL=debug cannot corrupt the JSON-RPC stream on stdout.

The MCP tools, and what each one hits 8

Names and order below are the ALL_TOOLS array in packages/mcp-server/src/tools/index.ts:11-20, checked against the published npm build. Every input schema is strict (additionalProperties: false) — an unknown field is rejected as InvalidParams, not ignored.

Three gotchas you will only hit once

FAQ

Where this runs

This is the retrieval layer behind the Kireo memory MCP — an MCP server that gives Claude Code, Cursor and any other MCP client one shared long-term memory. Save decisions and gotchas as you work, recall them from any tool, and browse, edit or delete everything from a web dashboard. Read the quickstart for the per-client install commands, or the privacy policy for the storage and retention specifics referenced above.

Architecture details on this page verified against source on 2026-08-03.