Memory

Coexo is an exoskeleton, not an app. It runs across Stephan's phone, his servers, his coding sessions, and whatever new AI surface comes next. For that to work, all of those surfaces need to remember the same things — what's going on, who Stephan is, what was decided last week, which skills exist, which calls were already made.

Memory is where that lives. It is one place. One server. One database. One small front door — the operator — that every Coexo surface talks to. A message Stephan sends through Telegram in the morning is in scope for a coding session that starts on a different machine in the afternoon.

Three rules govern it.

Everything is kept. Facts that change are superseded with timestamps, never overwritten. The system can always reconstruct what it knew at any past moment. This is not perfectionism — it is what makes it safe to consolidate aggressively elsewhere, because the raw record is always available to fall back on.

Stephan owns it, not any specific AI. Today's main surface is Claude. Tomorrow's might be ChatGPT, or Cursor, or something that doesn't exist yet. Memory survives the swap because no model is the keeper of it. The substrate is plain Postgres on Stephan's own server; he has the keys and could open the database directly to read what it knows.

Every surface uses the same door. No agent has private memory the others can't see. The interface is simple enough that a new agent surface gets wired up in a few hours, and from then on it shares everything the others know.

The substrate isn't theoretical — it does specific work, today.

Every new session arrives oriented. When Claude Code starts on any host, it receives a short briefing — who Stephan is, what's on his fleet, his working preferences, the skills he's authored, and the most recent things he's been focused on. He doesn't have to re-explain himself. The briefing is built fresh from memory each time, so it's always current.

Skills appear automatically on the surfaces that can run them. A skill is a small recipe — deploy the coexo site, restart the operator, verify substrate health. Each one declares which hosts it applies to. The right surface sees the right skills; the wrong surface doesn't waste tokens on what it can't act on.

Every coding session writes itself back. When a session ends, it summarises itself and lands as a structured event in memory — what was done, what was decided, what files changed, what to pick up next. The next session, on any host, can read it. The data flywheel turns without anyone curating.

*Recall finds things by words or meaning. Ask for the deploy script's API token location* and it finds it whether the events use those words or different ones. Behind the scenes there are three signals running in parallel — keyword search, meaning search, and a learned reranker on top — but the caller just asks one question and gets a ranked answer. Measured against a 20-query eval set, the hybrid stack lifts ranking quality 31.5% over a vector-only baseline.

Memory connects related things even when the words are different. Ask for active forgetting and HNSW reindex and you get back the project, the relevant procedures, and the affected hosts — found through the entity graph, not through text match. This is the HippoRAG-style graph layer; it surfaces relevance you wouldn't have spelt out.

The substrate keeps itself tidy. Every night, low-importance untouched events get tombstoned. Every week, duplicate facts get merged (the system has already merged things like service:postgres into service:postgresql). Every month — planned — a full reconciliation will re-derive the projections from the raw event log. The live write path stays append-only; destructive operations only happen offline, behind safety gates, and even then only as tombstones — never hard deletes in v1, so every removal is reversible.

Everything is auditable. Every change to memory is appended to a hash-chained log. Tampering is detectable in code. The chain has stayed intact across the v0 build, the live PG17→PG18 upgrade, and every consolidation cycle since.

Memory has four interfaces in daily use.

Speaking to it. A message to @coexo_memory_bot on Telegram lands in the substrate as an event with full provenance — who sent it, when, where it came from. Stephan never has to mark something as remember this; anything he says into the channel is in.

Reading from it (passively). Every Claude Code session opens with the prelude — a short, dense briefing the operator builds from the most important pinned events plus current focus. It is pointers and identity facts, not bulk content; bulk arrives on demand. Stephan never has to attach context manually. End-to-end injection takes about 70 ms.

Reading from it (actively). Any agent can call recall with a question and get back the most relevant events. The substrate runs hybrid retrieval (words + meaning + reranker) and graph traversal (find-by-relationship) under the same interface. The caller doesn't need to know which signal answered.

Writing to it. Sessions write themselves back at the end of every conversation. Skills are authored as small markdown files via the same memory API any agent uses. The shape of what's written follows Anthropic's memory_20250818 tool contract, so any future Anthropic-compatible agent works as a drop-in.

Adding a new surface. Three steps. Mint a token, register it on the operator's token file (only the SHA256 hash leaves the local box, so plaintext tokens don't live on the operator host), and drop a SessionStart hook on the new host. From that point the new surface participates in everything — reading the prelude, recalling, writing back, authoring skills.

Three structural layers.

Events are the bedrock. Every signal, message, decision, and state change is an immutable row with an ID, timestamp, source, author type, and importance score. Events are never mutated. Anything else in the system can be regenerated from them.

Entities are the layer above — Stephan, the fleet hosts, the projects, the procedures, the concepts. They're derived from events by a background worker, with relationships to each other (project hosts on, person knows skill, service runs on host). When a fact about an entity changes, the entity gets updated; the event that caused it stays exactly as it was. Today the substrate holds roughly 120 entities and 60 entity-to-entity relationships — small, growing organically.

Cognitive types are the policy layer. Each event is tagged with its kind — episodic (something that happened), semantic (a learned fact), procedural (a skill), core (identity that always loads), resource (a pointer to a file), or vault (sensitive — never embedded into shared search). The type tag drives what shows up in the prelude, what gets retrieved, what gets forgotten when, and what stays out of the open index entirely.

Four cadences keep the substrate alive.

Live. Every new event is admitted instantly, with rule-based importance scoring. The hot path is append-only; nothing destructive runs while Stephan is waiting.

Light (continuous). A background worker reads new events and uses Haiku to extract entities and relationships from them. This is what populates the entity graph. Cheap, fast, idempotent — the same event can be reprocessed safely.

REM (weekly). A heavier worker walks the entity catalogue, finds duplicates, and merges them. It uses Haiku with a strict tool schema so the merge decision is auditable. Real merges happen — service:postgres got merged into service:postgresql, concept:hipporag-2 into concept:hippo-rag-2, and so on. Re-runs after the first pass correctly produce zero merges (the model treats PG17 and PG18 as legitimately distinct).

Deep (daily). Tombstones low-importance events that nothing has touched in a month, and reindexes the vector store when it gets too fragmented. No LLM needed. Tombstone-only — never a hard delete in v1, so every removal is reversible.

One door. The operator is a small service running on the memory host. It is the only thing allowed to touch the database. Every surface — Claude Code, the Telegram bot, future ChatGPT or Cursor sessions — talks to the operator over the private tailnet. The operator handles auth, enforces what each surface is allowed to read, and exposes the same simple verbs to all of them: read, write, recall, consolidate. No agent surface ever talks to the database directly.

Memory has its own machine. Not a shared box, not a managed service — a single-tenant server in Falkenstein, Germany, joined to Stephan's private tailnet. The CPU is the right shape for the kind of search memory does (lots of cache-sensitive pointer-chasing); the RAM is large enough that the entire vector index lives in memory for years; the disks are mirrored NVMe.

Why dedicated, not managed. The data is sovereign — Stephan owns the keys, no shared cache, no shared CPU. Performance has multi-year headroom — the substrate isn't going to outgrow this hardware on this timescale. And if the choice ever needs to be revisited, Postgres is fully portable; nothing about the substrate is tied to the provider.

Monitoring lives off the box, on purpose. Putting the dashboards on the same host they're monitoring is an anti-pattern — when the box dies, the dashboards die with it. So infrastructure metrics, logs, and alerts run on a separate ops server, watching the whole fleet. LLM call traces — every Haiku, Sonnet, Voyage call with input, output, latency, and cost — flow to a hosted observability service so they survive any local outage too. Seven alert rules cover the things that actually matter: host unreachable, disk near full, disk filling fast, memory pressure, sustained high load, Postgres down, connection-pool saturation.

Backups are belt-and-suspenders. Continuous incremental backups land on local NVMe in the same machine; every night, a mirror ships to a Hetzner Storage Box in a different datacenter. Both are encrypted; the keys live in 1Password. Recovery point objective is five minutes; recovery time is one hour. A monthly automated restore drill verifies they actually work.

The audit log is cryptographic. Every write produces an entry chained to the previous one's hash. The operator exposes a verify endpoint that walks the chain in code. It has stayed intact across every change since v0, including the live PG17→PG18 upgrade.

Sensitive content is gated separately. Credentials, intimate personal facts, and personality inferences are tagged vault and never embedded into the open vector index. They're addressable only by ID and only through explicit access; they don't show up in any retrieval or in the prelude.

Secrets always live in 1Password. Every credential the substrate uses — Postgres roles, API tokens, backup encryption passphrases — has a 1Password entry as the canonical copy, with the on-disk location noted in the entry's notes.

Coexo Memory ships in tiers. Each tier is end-to-end working; pausing at any tier boundary leaves something usable.

v0 — Substrate roundtrip. Live since 2026-05-01. The dedicated server, Postgres 18 with the locked extension stack, the operator service, the Telegram bot adapter, the embedding worker, encrypted backups in a second datacenter, off-box observability, the audit chain. The success signal — write an event through the operator → it gets embedded → recall returns it → audit verifies → off-host backup completes — is met.

v0.5 — First real surface. Live since 2026-05-01. The always-loaded prelude on Claude Code, gated by per-source bearer tokens, with surface identity bound to the token. End-to-end injection at ~70 ms.

v1 — Feature-complete substrate. Substantially shipped 2026-05-03. Eleven things landed in one day, in order:

• •Schema v1 — entity catalogue + triple graph, bitemporal-ready envelope
• •Anthropic memory-tool front door — the canonical write contract every surface uses
• •Light-phase consolidation worker — entity and relationship extraction, idempotent
• •Write-back hook on Claude Code — every session summarises itself into memory
• •Per-type retriever dispatch in recall
• •REM-phase entity dedup + recall log substrate — real merges executed; idempotent on re-run
• •Deep-phase active forgetting + HNSW drift reindex — tombstone-only; reversible
• •Procedural memory tier with surface-aware filtering — skills appear only on hosts they apply to
• •HippoRAG-2 graph recall — find by relationship, not just text
• •Hybrid retrieval — vector + BM25 + Reciprocal Rank Fusion + Voyage rerank-2.5
• •20-query eval harness with measured numbers committed to the repo

Total measured lift over the vector-only baseline: +20% recall@5, +31.5% MRR.

One item remains in v1. A monthly self-rewriting prelude that improves its own template from accumulated session-summary outcomes. Deferred until end-of-month so there are three to four weeks of write-back data to learn from.

v2 is well-scoped, not yet started. Plaud audio ingestion, WhatsApp / Gmail signal adapters, image embeddings, the per-section Vault encryption layer, optional native graph traversal (Apache AGE) only if the relational expansion proves insufficient, async streaming replication to a second datacenter for warm-standby disaster recovery, and self-hosted LLM workers if the economics ever shift. Each item has a specific trigger that would pull it forward; none are blocking today.

A handful of things are deliberately off the roadmap — personal LoRA adapters, latent-space episodic memory, unified multi-modal embeddings, productized agent-memory frameworks as substrate, heavy symbolic reasoning. Each is excluded for a specific reason documented in the deep dive below.

What's running today. A Hetzner AX102-U bare-metal server in Falkenstein-DC7 (`coexo-memory` on the tailnet) hosts PostgreSQL 18.3 with the full locked extension stack — pgvector 0.8.2 (halfvec), pgvectorscale 0.9.0, pg_search 0.23.2, pgcrypto, pgaudit. The database is tuned for the box (32 GB shared_buffers, 90 GB effective_cache_size, NVMe-aware costs, `data_checksums=on`, archive mode on). Five Postgres roles split owner / operator / audit / readonly / exporter; all credentials in 1Password.

The operator service runs as a hardened systemd unit at `/opt/coexo-substrate` (FastAPI + Telegram bot + embedding worker + three consolidation phases + REM loop + deep loop, single process). Telegram bot `@coexo_memory_bot` is locked to Stephan's user_id. The hash-chained audit log verifies in code (`/audit/verify` returns ok across every change including the PG17→PG18 dump-restore, byte-equal MD5). The embedding worker pulls events without an embedding row, calls Voyage's voyage-3-large at 1024-dim halfvec, indexes via HNSW cosine, and round-trips through similarity recall. The light-phase consolidation worker walks events not yet processed and calls Haiku 4.5 with a forced-tool-choice extraction schema; the REM loop dedupes near-duplicate entities weekly via Haiku merge decisions; the deep loop runs daily, tombstoning low-importance unread events and reindexing HNSW when dead-tuple drift crosses 20%.

Encrypted backups run via pgBackRest with AES-256-CBC + zstd compression to local NVMe (full + incremental + WAL archive). A nightly systemd timer rsyncs the local repo to a Hetzner Storage Box BX11 in a separate datacenter, giving off-host redundancy under simpler tooling than the SFTP-native pgBackRest backend (which is deferred — the rsync mirror has the same RPO).

Observability is off-box deliberately. Prometheus 3 + Loki 3 + Grafana 13 run as a Docker compose stack on dymontlabs at `/opt/observability`, bound to the tailnet only. Grafana Alloy is installed on every fleet host (six total, including coexo-memory) and ships node metrics + systemd-journal logs via remote_write / Loki push. Alloy's built-in postgres exporter scrapes coexo-memory's database directly, surfacing 388 `pg_*` series. Seven alert rules cover host scrape staleness, disk near-full and filling-fast, memory pressure, sustained high load, postgres unreachable, and connection-pool saturation. Langfuse Cloud (EU region) is wired into the embedding worker — every Voyage call produces a typed `EMBEDDING` observation visible in the `Coexo / coexo-memory` project.

v0 boundary. The success signal — Telegram message → operator stores it with embedding, importance, and provenance → recall returns it via similarity search → the audit chain verifies → off-host backup completes — is met. The pieces explicitly not in v0: full hybrid retrieval (BM25 seed plus vector seed plus rerank), Anthropic Contextual Retrieval at ingest, the consolidation engine (per-conversation A-MAC LLM scoring, nightly Auto-Dream, weekly rollups, monthly reconciliation), domain entity tables (cases, tasks, persons, documents) as first-class projections, the Knowledge Vault encryption layer, multi-modal ingestion (Plaud, WhatsApp, Gmail, image embeddings), and the always-loaded prelude generator. Those are v0.5 and v1 work.

1. Append-only event log as canonical substrate. Every signal, conversation turn, decision, and journal entry is an immutable event with a stable ID. This mirrors Anthropic's Managed Agents reframing: the durable substrate lives outside the model, and the context window is just a view materialized per turn. Compaction, the memory tool, and context editing become projections of one event stream rather than competing primitives.

2. MIRIX six-type taxonomy. Episodic, Semantic, Procedural, Core, Resource, Knowledge Vault. The taxonomy maps cleanly onto Coexo's heterogeneous content. Implemented as type tags on events and entities, not as separate stores or separate agents — the value is in the typology, not in multi-agent orchestration.

3. Hybrid retrieval — multiple seeds, learned rerank, structure expands. Three signals run in parallel: BM25 lexical search via pg_search, vector similarity via pgvector + pgvectorscale, and relational expansion over the entity graph. Results are fused via weighted Reciprocal Rank Fusion. A learned cross-encoder reranker (gte-reranker-modernbert-base, Apache 2.0, self-hosted on the same host) then re-reads the top candidates and produces a calibrated relevance score that replaces raw cosine similarity in the final ranking. At ingest time, every event also gets a Claude-generated 50–100 token contextual prefix (Anthropic's "contextual retrieval" pattern) — independently shown to drop retrieval failure rates by roughly half versus naive embeddings. Heavy symbolic reasoning (Prolog, Datalog) and graph-native traversal are deferred until a use case demands them.

4. Mutation-on-write, with audit log, only at the projection layer. Primary records — events, decisions, working notes — are immutable. Profiles, case state, and derived semantic facts are projections regenerated from event sources, with every regeneration written through an audit trail so it is reversible. New events trigger refinement of derived caches; the underlying truth never changes.

5. Nested consolidation cadences with write isolation and safety gates. Four cadences. Per-conversation is inline-fast and append-only — no LLM, no destructive operation on the live write path. New events pass a five-factor admission gate (utility, confidence, novelty, recency, type prior, after the A-MAC pattern) where the cognitive type tag does most of the work and an LLM call only fires for borderline cases. Nightly runs an Auto-Dream cleanup as Claude invoked headlessly with the operator's memory tools — same toolbelt as foreground agents — with a Sonnet-class contradiction gate before any episodic-to-semantic promotion. Weekly does hierarchical rollups; new versions append, never overwrite. Monthly runs random-walk associative passes plus a full reconciliation that re-derives projections from raw events, defending against multi-year drift. All offline cadences stack Anthropic's Batch API with prompt caching for roughly 95% cost reduction over naive pricing. Bi-temporal validity intervals on every projection make time-travel queries free.

6. Multi-modal via modality-specific embeddings joined by graph IDs. Text, image, and audio each get a strong specialist embedder. Audio is transcribed and embedded as text by default; raw audio remains addressable for future queries about tone, speaker, or non-speech content. Unified-embedding-space approaches like ImageBind are skipped — modality-specific embedders joined at the metadata and graph layer outperform them.

7. Always-loaded prelude is pointers, not content. The pinned context every session sees is hundreds of tokens of identifiers — current focus, open case IDs, person handles, top preferences — plus a tiny set of identity facts that must always be in scope (name, timezone, language preferences). Sensitive personality inferences and any Knowledge Vault content are never in the prelude. Bulk content is fetched on demand through the memory tool API.

8. Postgres on dedicated bare-metal, fronted by an operator service speaking an internal API. Postgres 18 with pgvector (halfvec embeddings), pgvectorscale (StreamingDiskANN for production-scale vector retrieval), pg_search (BM25 lexical seed for hybrid retrieval), pgcrypto (Vault encryption), and JSONB for flexible event bodies — all running on a single-tenant Hetzner AX102-U in Falkenstein chosen specifically for the workload (Ryzen 9 7950X3D with 3D V-Cache for HNSW pointer-chasing, 128 GB DDR5, NVMe Gen4 RAID1). Memory integrity at our one-user scale is defended in software rather than via ECC RAM (which is end-of-life as a managed-dedicated product on consumer Ryzen): Postgres `data_checksums=on` catches corrupt pages on disk, raw events are the canonical source so any derived projection is regenerable, monthly full reconciliation re-derives every projection from raw events, and encrypted off-host backups close the loop on physical loss. A small Coexo memory operator service is the only thing that touches the database; every agent surface is a thin client speaking an internal typed API (recall, event_log, write, consolidate). Anthropic's memory_20250818 tool shape is one adapter on top — OpenAI, Google, and Cursor adapters slot in alongside as those tools land. Apache AGE for native graph traversal is deferred — the relational relationships table holds the same data shape and AGE can layer on top later if multi-hop queries demonstrate need. Every piece is open-source and replaceable; lock-in is near-zero.

Memory has its own server. This is not co-location with another application; the substrate runs on dedicated hardware whose only job is to be the canonical memory store.

The host is a Hetzner AX102-U dedicated bare-metal server in Falkenstein, Germany, joined to the same private tailnet as Stephan's other servers. AMD Ryzen 9 7950X3D (16 cores / 32 threads, with 3D V-Cache providing 96 MB of stacked L3) is the right CPU shape for HNSW vector search — which is cache-bound pointer-chasing rather than SIMD-bound math. 128 GB DDR5 RAM holds the entire vector index plus working set in memory for years. Two 1.92 TB NVMe Datacenter Edition Gen4 drives in RAID1 give roughly 1.9 TB of fast local storage for multi-modal blobs, with mirrored redundancy.

Note on RAM and integrity. The original plan specified ECC RAM for silent-bit-flip protection. Market discovery in April 2026 showed the 7950X3D + full sideband ECC combination has been quietly retired across the dedicated-server market — Hetzner replaced AX102 with AX102-U (non-ECC), Bacloud and Evolushost are out of stock, OVH's 9950X3D line has only on-die ECC. The genuine 2026 frontier for ECC + V-Cache is server-grade EPYC Genoa-X, available only by custom enterprise quote at €600–1500+/mo. For a one-user personal-memory workload at GB scale (not Cloudflare-class trillion-vector), ECC is meaningful defense-in-depth but not load-bearing. Substrate integrity here is defended in software: Postgres `data_checksums=on` catches corrupt pages before reads, raw events are the canonical source so every derived projection (vector index, person profile, summary) is regenerable, the monthly full reconciliation re-derives projections from raw events as a routine integrity sweep, and encrypted off-host backups close the loop on physical loss. Expected silent-bit-flip rate at 128 GB DDR5 with on-die ECC is roughly one or two per year, most landing in places these safeguards already cover.

Three things still follow from the dedicated-bare-metal choice. Performance is not a constraint — the substrate has multi-year headroom before any spec becomes a bottleneck. Data sovereignty is real — the box is single-tenant with no shared cache, no shared CPU, and Stephan owns the keys; the data is not parked on someone else's managed service. Single point of failure is acknowledged and mitigated — automated incremental backups (pgBackRest) ship to a separate Hetzner Storage Box, and warm-standby replication can be added later when a real second user or hard real-time dependence makes high availability matter.

The cost is €124 per month (€122.30 server + €1.70 IPv4) plus a €500 one-time setup fee, with no minimum contract — the server can be cancelled any month. Postgres is fully portable; if a different provider becomes the right answer later, the migration is mechanical.

Three pieces run on the dedicated memory host, side by side.

The database (PostgreSQL) holds everything — events, entity tables, embeddings, audit log. It is the canonical store; nothing happens without it.

The operator service is a small program (Python / FastAPI) and the only thing allowed to read or write the database. Every agent surface — Claude Code, the OpenClaw agent, future ChatGPT or Cursor sessions — talks to the operator over the tailnet. The operator handles authentication, enforces ACLs (sensitive Vault content is gated here), and exposes a typed internal API: recall, event_log, write, consolidate. Adapters on top of the internal API translate to whatever tool shape each agent's model expects (Anthropic's memory_20250818 today, OpenAI / Google / Cursor adapters as those tools land).

The background workers are little programs that run on a schedule and do the AI work the memory needs to maintain itself. New events get scored for importance and given a contextual one-line prefix for findability. Every night the system tidies up — finds duplicates, refreshes profiles, promotes frequently-referenced items. Every week it rolls up summaries; every month it looks for non-obvious connections and reconciles projections against the raw event log. The orchestrated jobs are not separately-coded Python services that re-implement memory access — they invoke the same Claude CLI with the same memory tools the foreground agents use, so the consolidation worker has identical capabilities and the same operational behavior as anything else in the system.

The model the workers use is Anthropic's Claude family in tiers. Haiku for cheap frequent tasks (importance scoring, contextual prefixes, light entity extraction). Sonnet for safety-critical reasoning, especially the contradiction gate that runs before any new fact is promoted into the semantic layer. Opus for the highest-quality monthly work — regenerating person profile sections from event sources. Calls go to Anthropic's API; orchestrated jobs spawn the Claude CLI in headless mode. Anthropic's Batch API and prompt caching stack to roughly 95% off list pricing, which is what makes the per-cadence budgets ($0.50 per night, $2 per week, $10 per month) reachable.

Observability runs off-box, deliberately. Putting Prometheus, Grafana, and Loki on the same host you're monitoring is an anti-pattern: when the box dies, the dashboards that should tell you why die with it. So infrastructure metrics and logs are aggregated on dymontlabs (Stephan's existing ops box), which monitors the whole fleet. LLM traces — every Claude call's input, output, latency, and cost — flow to Langfuse Cloud (EU region) rather than a self-hosted Langfuse on the memory box, both for independent uptime and to remove the operational burden of running another stateful service. The hash-chained audit log inside Postgres stays on the memory host because it is part of the substrate's tamper-evidence story, not part of monitoring.

This choice is grounded in economics and tooling fit, not in vendor preference. The substrate is provider-portable; if a different provider becomes the right choice for a worker later, that worker swaps. The agents that call the memory can be on any model — that decision belongs to whoever builds the agent surface, not to the memory.

The architecture is deliberately surface-agnostic. Memory is owned by Stephan, not by any specific agent or model. Memory has an operator — a single service running on the dedicated memory host — and every agent surface is a thin client of that operator. Internally the operator exposes a typed Coexo Memory API; on top of it sit adapters that let each agent speak whatever tool shape its model expects (Anthropic's memory_20250818 today, OpenAI / Google / Cursor adapters as those tools land). The operator is the single writer to Postgres, owns the embedding workers and consolidation jobs, and enforces ACLs. No agent surface ever talks to the database directly.

Three implications follow. First, memory survives model upgrades. Today's surface is Claude Code on Opus 4.7; tomorrow's might be Claude 5, ChatGPT, Cursor, or something that does not exist yet. Memory persists across all of them because none of them owns it. This is the explicit reason personal LoRA adapters and weight-level memory are excluded from the design — they die with each base-model upgrade and lock memory to a specific model lineage.

Second, memory is multi-author by default. Claude Code on dymontlabs may write a feedback rule during a coding session; the OpenClaw agent on openclaw-vps reads it on the next interaction. Every event records its author type — human, system, sub-agent, or external — so that consolidation and access control can reason about provenance.

Third, memory remains legible. Beneath the tool API, the substrate is Postgres tables and on-disk artefacts. Stephan can inspect any record, run a SQL query, restore from backup, or migrate to a different substrate without depending on any vendor's runtime. The site documents the schema as canonical reference; the substrate documents itself through audit trail and structured queries.

Several state-of-the-art directions surveyed during the research phase are explicitly excluded from v1. Each has a specific reason.

Personal LoRA and weight-level memory are excluded for at least 24 months. The frontier labs that could most easily ship per-user weights are conspicuously not doing so. Catastrophic forgetting on continual updates remains unsolved at production quality. Adapters die with each base-model upgrade — a fatal flaw for a personal system meant to outlive any specific model. Apple ships task adapters, not user adapters; that choice is the tell. The one weight-level technique worth tracking is test-time training as a context-compression mechanism, because if it matures it becomes a transparent inference-layer optimization that does not require architectural change.

Latent-space episodic memory (Larimar-style encoded vectors as memory rather than text) is genuinely interesting research but two to three years from production. It requires custom model surgery, has not been demonstrated at scale, and trades legibility for compactness — the wrong trade for Coexo's transparent substrate principle.

Unified multi-modal embedding spaces (ImageBind, fully shared text-image-audio vector spaces) underperform per-modality encoders joined at the metadata layer. Coexo uses modality-specific embedders.

Productized agent-memory frameworks as primary substrate — Cognee, Graphiti, Letta, Mem0 — are excluded as the canonical store. Cognee's dependency on Kuzu (archived October 2025) is a live risk; Letta's framework is agent-shaped, not memory-shaped; Mem0 dropped graph mode in April 2026; Graphiti is well-engineered but locks the substrate to its episode-edge model. Graphiti remains a candidate as a thin temporal-entity layer on top of Postgres, but only if hot-path queries reveal a need.

Heavy symbolic reasoning — Prolog, Datalog, theorem provers as core retrieval — is deferred until a use case requires provable correctness. The graph-plus-vector hybrid handles relational and temporal queries at productized maturity; logic programming is layered on top only when Coexo needs verified rule enforcement (a future use case, not a current one).

Three implementation-level questions are explicitly deferred to prototype rather than resolved in design.

Hot-path retrieval latency at scale. Hybrid retrieval — BM25 lexical seed via pg_search, vector seed via pgvector with pgvectorscale, RRF fusion, relational expansion, and rerank — needs to land under 300 ms p95 for top_k ≤ 20. The hardware substantially de-risks this: the AX102's 3D V-Cache CPU is the right shape for HNSW's cache-bound access pattern, and 128 GB ECC RAM keeps the entire index in memory for years. Real performance still must be measured against representative data. Fallbacks if needed: pgvector iterative scans, materialized hot-path views, or moving hot-path retrieval to a graph cache tier (Memgraph or AGE on top of the same data). The original "Postgres-plus-AGE composition" risk is no longer in the v1 path because Apache AGE is deferred until multi-hop queries demonstrate need over relational expansion.

Chunking strategy and overlap for long content. Plaud transcripts, PDFs, and email threads need a chunking policy. The default is semantic-boundary chunking with overlap, plus a document-level summary embedding for top-down retrieval. The right chunk size and overlap depend on the embedder and the content; both are calibrated empirically.

Embedding model choice. Voyage-3-large, Gemini Embedding 2, and Cohere Embed v3 are all candidates for the text leg; Voyage-multimodal-3 and SigLIP-2 are candidates for images. The choice depends on cost-quality trade-off measured against real Coexo content, plus operational considerations (API dependency vs self-hosted).

Everything else — the three-layer model, the eight design choices, the consolidation cadences, the surface-agnostic API — can be built with confidence based on the research phase. These three questions are calibration, not architecture.

Memory is implemented in four tiers. Each tier ships an end-to-end working system; pausing at any tier boundary leaves something usable. The original ten-phase plan is restructured here to reflect the scope reduction surfaced by review and research.

v0 — Substrate roundtrip. Live as of 2026-05-01. AX102-U provisioned in Falkenstein-DC7 and joined to the tailnet. PostgreSQL 18.3 with the locked extension stack (pgvector 0.8.2 halfvec, pgvectorscale 0.9.0, pg_search 0.23.2, pgcrypto, pgaudit) running with `data_checksums=on` and a tuned config (upgraded from PG17.9 → PG18.3 on 2026-05-03 via dump+restore on the empty-ish cluster, ~7 min downtime, audit hash chain preserved byte-equal). Minimal schema applied: events with provenance envelope and tombstone primitives, event_embeddings (halfvec 1024-dim, HNSW cosine), hash-chained audit_log (INSERT-only via trigger), tombstones, schema_versions. Five Postgres roles (owner / operator / audit / readonly / exporter) with credentials in 1Password. Operator service running as a hardened systemd unit, exposing recall, event listing, and audit verification; Telegram bot `@coexo_memory_bot` locked to Stephan's user_id is the v0 source adapter. Importance scoring uses the A-MAC band assignment with a 0.5 utility placeholder (LLM-backed utility scorer for borderline events lands in v0.5). Embedding worker uses halfvec via Voyage voyage-3-large at 1024-dim. Encrypted backups: pgBackRest with AES-256-CBC + zstd to local NVMe; nightly rsync mirror of the local repo to a Hetzner Storage Box BX11 in a separate datacenter (the SFTP-native pgBackRest backend is deferred; the rsync mirror has the same RPO under simpler tooling). Observability is off-box: Prometheus + Loki + Grafana run as a Docker compose stack on dymontlabs, with Grafana Alloy on every fleet host (six total) shipping node metrics + journal logs and Alloy's built-in postgres exporter scraping coexo-memory directly; seven alert rules cover host scrape staleness, disk pressure, memory, load, and Postgres. Langfuse Cloud (EU, Hobby tier) is wired into the embedding worker — every Voyage call produces a typed EMBEDDING observation under the Coexo / coexo-memory project. Success signal met: test event written via the operator → embedded via Voyage → recalled via similarity search → audit chain verifies → off-host backup cycle completes nightly.

v0.5 — First real surface and write-back. The always-loaded prelude generator goes live on Claude Code on dymontlabs (operator exposes `GET /prelude` over bearer-gated tailnet HTTP; SessionStart hook injects the response into every session at ~70 ms end-to-end). Public-facing `POST /events` lands behind per-source bearer tokens with operator-side source-binding. The 14 `mirix_type='core'` seed events render the identity / fleet / preferences sections of the prelude. What v0.5 does NOT include (rescoped 2026-05-02): hybrid retrieval, Contextual Retrieval at ingest, RAGAS, Plaud audio ingestion, and second-surface wiring all move to v1, where they make sense once the substrate has more than seed content. Langfuse Cloud trace coverage extended to all worker LLM calls via OpenTelemetry GenAI semantic conventions remains in v0.5. Success signal met: prelude injection round-trip works end-to-end on Claude Code on dymontlabs; first real `POST /events` from a remote surface verified.

v1 — Feature-complete substrate. In flight; Sprints A + B complete 2026-05-03. Plan reshaped 2026-05-02 after deep research on OpenClaw memory-core internals, the early-2026 SOTA agent-memory landscape (MemGPT/Letta, MIRIX, Mem0, HippoRAG 2, A-MEM, Generative Agents, LightMem, Anthropic memory tool, Voyager/SKILL.md, Self-RAG), and architectural patterns for combining operational data with audit-chain memory. Postgres 18 active (upgraded 2026-05-03 while the substrate was essentially empty) — native SQL:2011 application-time temporal constraints are now available for any future projection tables.

Shipped 2026-05-03 (in order on the same day):

✓ Schema v1 (migration `0002_v1_envelope`): events envelope reserves well-known top-level JSONB keys (entity_id, entity_type, relations[], valid_from, valid_to) so any future entity is just an event with the right mirix_type — no schema change preempted; `event_embeddings` gains an `embedding_kind` discriminator (body_text | summary | entity_label | context); new `entities` and `triples` tables for the HippoRAG-2 graph layer.

✓ Anthropic-memory-tool-compatible front door (migration `0003_memory_paths` + `POST /memory` endpoint): drop-in compatible with Anthropic's `memory_20250818` six-verb FS API (view, create, str_replace, insert, delete, rename), now the canonical write contract every agent surface should use. The `memory_paths` projection tracks which event currently backs each `/memories/...` path; mutations INSERT a new event AND upsert the path in one transaction, with `provenance.derived_from` chaining str_replace and insert to their predecessor for forensic history. Mem0's 4-verb shim (ADD/UPDATE/DELETE/NOOP) is deferred — low priority, lands when batch ingestion needs it.

✓ Light-phase consolidation worker (migration `0004_consolidation_log` + module `consolidation.py`): Haiku 4.5 entity/triple extraction running as a background asyncio task in the operator. Forced `tool_choice` on a structured `record_extraction` schema constrains slugged entity IDs (`^[a-z][a-z0-9_]:[a-z0-9][a-z0-9_-]$`) and XOR object_id|object_literal on triples. The append-only `consolidation_log` table tracks per-event/per-phase runs (status: success | skipped | error) so reprocessing is naturally idempotent and the events table stays UPDATE-free. First-run snapshot: 17/17 eligible events processed cleanly, 47 entities and 42 triples populated, hash chain verified intact across 78 audit rows.

✓ Write-back hook on Claude Code dymontlabs (`POST /events/from-transcript` + module `session_summary.py`): SessionEnd hook posts the raw transcript JSONL to the operator, which runs Haiku 4.5 with forced tool_choice on a `record_session_summary` schema (headline, narrative_md, decisions[], key_files[], commits[], next_steps[]) and writes one `mirix_type='episodic'` event per session. Idempotent on (source, source_id=session_id), so re-running the hook against a transcript replaces nothing. Closes the data flywheel — every coding session now leaves a structured trace the consolidation worker can chew on.

✓ Per-type retriever dispatch in /recall (module `retrievers.py`): three strategies registered — `recall_vector` (cosine k-NN over event_embeddings, filtered by mirix_type, optional time_window_days + min_importance), `recall_core_list` (no vector lookup; non-tombstoned core events ordered by importance with optional body section filter), and `recall_dispatch` orchestrator that interleaves results round-robin from each retriever and dedupes on event id. Response shape changed from list to `{hits, by_type, mirix_types}`. Vault stays gated separately by policy.

✓ REM phase A-MEM dedup + recall_log Hebbian substrate (migration `0005_recall_log` + module `rem.py`): two-part. (1) Append-only `recall_log` table records every `/recall` hit (event_id, source, query_excerpt, mirix_type, rank_position, similarity, recalled_at) so future Hebbian promotion and active forgetting can read recall counts as a queryable signal — without ever UPDATE-ing the events table. (2) A-MEM-style entity deduplication: cheap heuristic candidate generation (same display_name within entity_type, slug Levenshtein ≥ 0.7) → Haiku 4.5 forced tool_choice on a `merge_decision` schema → execute merge (re-point triples on subject_id + object_id, tombstone newly-self-looped triples, union aliases/attributes onto the canonical, accumulate recall_count, tombstone the loser). Background loop on weekly cadence; manual trigger via `POST /consolidate/rem`. First pass merged 5 genuine duplicates the light phase had produced (`project:coexo` → `project:coexo-memory`, `service:postgres` → `service:postgresql`, `tool:pg-search` → `tool:pg_search`, etc.); idempotent re-run correctly produced 0 merges (Haiku flagged PG17 vs PG18 as distinct versions). Deferred within REM: Generative-Agents threshold reflection (synthesise semantic events when accumulated importance crosses threshold) and Hebbian episodic→semantic promotion via 6-component scorer — both depend on substrate scale we don't have yet (~70 entities post-dedup is too small to tune meaningfully; Hebbian needs weeks of recall traffic in `recall_log` to accumulate signal). They land once the substrate matures.

✓ Deep phase first cut (migration `0006_operator_maintain` + module `deep.py`): daily background loop with two no-LLM sub-actions. Active forgetting tombstones events meeting all of `importance < 0.2`, zero `recall_log` hits in 30d, age > 30d, mirix_type ≠ 'core', not flagged sensitive, and not referenced as `provenance.derived_from` by any non-tombstoned successor (preserves write-back hook chains and memory_tool str_replace/insert chains). NULL importance is treated as 1.0 so unscored events are never forgotten. Capped at 100 per pass. Hard-delete is intentionally NOT in v1 — tombstone-only matches every other "remove" path in the codebase (memory_paths.delete, REM merge losers) and stays reversible; hard-delete waits for actual disk pressure to justify giving up reversibility. HNSW drift check REINDEX CONCURRENTLY's `event_embeddings_hnsw_idx` when n_dead_tup/total > 20% AND total > 100; force-reindex bypasses both gates for testing. The PG17+ MAINTAIN privilege (granted to the operator role in 0006) lets REINDEX run from the worker without table ownership. Manual trigger `POST /consolidate/deep?dry_run=&force_reindex=` mirrors `/consolidate/rem`.

✓ Procedural memory tier — Skills section in prelude: SKILL.md-shaped events written under `/memories/procedural/` via `POST /memory` create. Convention is YAML-ish frontmatter at the top of body_text — `---\nname: ...\ndescription: ...\n---\n` — with the prelude builder parsing the frontmatter and rendering one line per skill (name + description + path) as a "Skills" section between Working preferences and Recent focus. Skills are sourced from the `memory_paths` projection joined to the live event so str_replace edits surface the new description without lag. Procedural events are now excluded from the "Recent focus" pointer list. Skills lacking frontmatter fall back to path basename for name and first non-blank line for description. Canonical write path is the memory_20250818 surface — raw `POST /events` writes don't appear in Skills by design (memory_paths projection is the source of truth).

✓ HippoRAG-2 graph recall (`POST /recall/graph` + module `graph.py`): vector-seed via existing `recall_vector` over event_embeddings → map seed events to seed entities through `triples.source_event_id` (each entity weighted by the sum of similarities of events that mention it, normalized to sum to 1.0) → run Personalized PageRank over the entities + triples graph (treated as undirected at the edge level for first cut, self-loops defensively skipped) → return top-k entities ranked by PPR score, hydrated with first_seen_event for context. Pure-Python iteration rather than recursive SQL — substrate is small (~100 entities / ~60 entity triples) and PPR is fundamentally iterative. Defaults are standard PageRank: alpha=0.15 reset probability, 15 iterations; both tunable per request. Each surfaced hit bumps `entities.recall_count` (Hebbian signal — entities table already has the column, REM uses the same convention for accumulation on merges). Vault excluded by policy. Smoke-tested: query "deep phase active forgetting and HNSW reindex" surfaces project:coexo-memory, procedure:deploy-coexo-site, host:coexo-memory, host:openclaw-vps, host:dymontlabs in top 7 — all relevant via the entity graph rather than direct text match.

✓ Surface-aware filtering for procedural skills: prelude builder now respects an optional `applies_to: [surface_a, surface_b]` field in skill frontmatter. Omitted = renders on every surface (default; back-compat for skills that genuinely apply everywhere). Explicit non-empty list = renders only when the bearer's surface matches. Explicit empty list `[]` = never renders (escape hatch for retiring a skill without tombstoning). First demo: `deploy-coexo-site` is dymontlabs-only because the rsync source path lives there; `coexo-memory-api`, `restart-coexo-operator`, `apply-coexo-migration`, `verify-coexo-substrate-health`, `query-coexo-recall` all render on every surface. Lets the procedural tier scale to many surfaces without each surface paying a token bill for skills it can't act on.

✓ Hybrid retrieval — vector + BM25 + RRF + Voyage rerank-2.5 (Sprint C step 1 + 2 + 3). Replaces the previous vector-only-plus-importance-listed-core dispatcher with a two-lane hybrid: pg_search BM25 over `events.body_text` (the events_search_idx was provisioned in migration 0001 but had been unwired) running in parallel with the existing Voyage voyage-3-large vector retriever, both over an over-fetched 75-candidate pool, fused via Reciprocal Rank Fusion (Cormack 2009, k=60), then reordered by Voyage rerank-2.5 over the full fused pool, finally cut to top-K. Rerank is config-toggleable and fail-soft (RRF order is the fallback when rerank is off, times out, or errors). Each hit carries optional `cosine_similarity`, `bm25_score`, `rrf_score`, and `rerank_score` fields for debug visibility; response carries a `rank_signal` field telling the caller which signal determined the final order. The `core` mirix_type now goes through the same hybrid path as everything else (the previous core-only listing was query-insensitive — the prelude has its own importance-ordered core path that doesn't go through this dispatcher). Pivot from the original Sprint C plan: research surfaced that Self-RAG-style retrieval gating is calibrated for substrates where retrieval is expensive and frequently the wrong choice; ours is sub-100ms and sub-1k events. Mem0 (arxiv 2504.19413, Apr 2025), MIRIX (2507.07957, Jul 2025), Graphiti, and Anthropic's own Contextual Retrieval guidance (Sept 2024) all converge on unconditional hybrid retrieval as the 2025-26 SOTA. The hybrid stack is a structurally bigger win for our use case and lands today; Self-RAG would have shipped a tax for negligible benefit.

✓ Eval harness — 20-query golden corpus + recall@k + MRR runner (Sprint C step 1). Hand-curated against the substrate snapshot; queries balanced across identifier-heavy (BM25 strength), conceptual (vector strength), preferences/identity (was query-insensitive, now BM25-driven), and mixed (fusion strength). Pure-stdlib Python runner outputs JSON reports under `eval/results/`. Three measured stages over the same corpus: - vector-only baseline: r@5=0.754, r@10=0.908, MRR=0.711 - +BM25+RRF: r@5=0.896, r@10=0.983, MRR=0.912 — preferences MRR jumped 0.32→0.81 (was random importance order; BM25 ranks query-relevant core events at #1) - +rerank-2.5: r@5=0.904, r@10=0.950, MRR=0.935 — rerank improves MRR and recall@5 but slightly trades off recall@10 by demoting near-duplicate session summaries; net win for UX where users read top-5

Total Sprint C lift over baseline: +20% recall@5, +4.6% recall@10, +31.5% MRR. The RAGAS/LOCOMO hookup into CI is the natural follow-on once the substrate has enough events that a representative ground-truth corpus is reasonable to assemble (today's 32-event substrate constrains corpus design).

Still to ship in v1 (one item remains).

○ Self-rewriting prelude — a Sonnet/Opus monthly task that rewrites the prelude template itself from accumulated session-summary outcomes (LangMem procedural-prompt pattern). Why deferred: requires weeks of write-back hook output to have meaningful signal to rewrite from. The write-back hook shipped 2026-05-03; the first run worth doing is end-of-month with three to four weeks of session summaries in the substrate. Bumped to Sprint D.

Success signal. Claude Code on dymontlabs writes back episodic events automatically (✓ as of 2026-05-03); three-phase consolidation produces semantic events nightly (✓ as of 2026-05-03 — light continuous, REM weekly, deep daily); the prelude renders identity + fleet + preferences + skills + recent focus with surface-aware skill filtering inside the 800-token cap (✓ as of 2026-05-03); recall@10 against the eval set holds at ≥95% across the hybrid retrieval stack (✓ as of 2026-05-03 with measured numbers committed under `eval/results/`); cross-surface read-write proven when a second surface (Lucas, Cursor, ChatGPT, or future) is wired up.

v2 — Deferred or capability-only. Plaud audio ingestion, WhatsApp / Gmail signal adapters, image embeddings via Voyage-multimodal-3 or SigLIP-2, Knowledge Vault encryption layer with pgcrypto. Apache AGE for native graph traversal, only if recursive-CTE PPR demonstrably underperforms on real Coexo data. ColPali / ColQwen2 lane for visual content (PDFs, screenshots) — kills the OCR pipeline for scanned content. Operational projection tables (`operations.*` typed projections with PG18 application-time temporal history) for any entity type whose JSONB query path proves too slow under real load — added on access-pattern evidence, not preempted. Async streaming replication to a second Hetzner datacenter for warm-standby disaster recovery, when warranted. High availability with automated failover, only when warranted by a real second user or hard real-time agent dependence. Federation and shared memory slices with foreign agents. Self-hosted LLM workers (Qwen3-32B class) if traffic patterns justify the operational cost over Anthropic's Batch + caching economics. Third backup repository (Hetzner Object Storage S3) added back if real-world incident analysis shows two repos was insufficient. Latent-space episodic compression and personal LoRA adapters remain on the watch list with explicit reassessment horizons (18–24 months and 24+ months respectively).