ctx.

The actual trigger for ctx: Claude instances lose and distort knowledge across sessions. This is made worse by the Constitution filter, which systematically shifts knowledge in a more conservative direction at every handover — so information is not merely forgotten but actively bent in one direction. On top of that came context-window scarcity: originally 256k tokens, continuously eaten away by MCPs and skills. ctx is the answer to both problems at once — an external, reliable knowledge layer that does not drift.

With the setup already in place, GottZ could have built a multi-GPU inference scaler, an agent framework, or a workflow platform. The deliberate decision was instead to build the persistence layer first. In his own words: "because I need the brain first, before I implement anything else … without central control over information, no other project can be implemented in any meaningful way. Not with my warnings, and not with my armada either." The W-System and the armada pattern both operate on insights accumulated over time — without a brain to hold those insights, neither could function. Persistence was therefore the precondition for everything that followed, not a downstream feature.

A key conceptual correction: persistence is not merely a workaround for scarce context windows. Even with a 1M-token context, ctx remains valuable — the image is the "big bucket with a bottom that becomes more valuable the bigger the bucket gets." The more context that is available, the more value a reliable persistence layer unfolds, not less. This positions ctx as a multiplier of an LLM's capabilities, not a crutch that better hardware would render obsolete.

The technical genesis ran through clearly documented stages. (1) First idea: wire n8n MCP into Claude to store knowledge blocks with pgvector and make them findable. (2) n8n MCP discarded — "garbage. too bloated. you get the entire input back as output as well. it talks too much." (3) n8n webhooks as a leaner alternative on the same basic idea. (4) An API key added as an auth layer. (5) A standalone Bun/TypeScript build — "quickly discarded again." (6) After the language comparison, the switch to a Go monolith. Today the system consists of two binaries (the ctx CLI + the ctxd daemon) under MPL-2.0, with packages such as auth, store, rrf, dream, digest, and guard — the lean consequence of this chain.

The result of the neutral language evaluation was Go. The actual porting effort was minimal: "the entire ctx rewrite from typescript with n8n to golang was a one hour problem." This was possible because the substance of the matter (data model, pgvector retrieval, auth) had already been worked out through the previous stages — the rewrite only had to pour a proven idea into a better language. From this step came today's Go monolith (Go 1.26 in the repo), which bundles API, Guard, Dream, Digest, and MCP into a single binary.

Rather than picking the target language out of familiarity or gut feeling, an armada of 92 sub-agents was set up to evaluate the language candidates neutrally against one another. The result was Go. This step is emblematic of the project's entire way of working: it is not the author who decides on a hunch, but a parallelized, structured apparatus that delivers a neutral result, which is then followed. The armada as an evaluation tool is already present here at the core, long before it became the fixed red/blue-team pattern.

The load-bearing methodological principle behind ctx and its emergence: "I built myself tools that cover my ignorance and deliver neutral results. The output speaks for itself." Knowledge gaps are not compensated for by learning oneself — that is too slow — but by tools that produce structured, neutral output. Combined with consistent parallelization, this leads to enormous speed: "moved more in 20 minutes than others do in 14 days." ctx itself is the first and most fundamental incarnation of this principle — the tool that covers memory.

Until the end of March 2026, the workflow framework was GSD (glittercowboy's repo). In early April 2026 it was replaced by a custom W-System with 14 RLHF warnings and a custom armada with a red/blue-team structure. Both build directly on the persistence layer: the W-System steers behavior based on accumulated insights, and the armada verifies adversarially against stored knowledge. This closes the loop back to the foundational "the brain first" decision — only once persistence was in place did the behavior and verification systems that depend on it come into being.

The thrust of future versions was clearly outlined early on: "The brain can be extended much more deeply later as a v2." v2 thus means deepening the brain itself — better retrieval, synthesis, and persistence capabilities — and explicitly not horizontal expansion into other tool categories. A multi-GPU inference scaler and full-fledged agent frameworks were deliberately deferred to v3+. The order of the genesis (the brain first, then behavior/verification on top of it) thus continues as the logic of the roadmap.

The first ctx idea ran on n8n: first as an MCP server inside Claude, then — because that MCP was "garbage, too bloated" (it returned the entire input as output and "talked too much") — via leaner n8n webhooks with API-key auth. A Bun/TypeScript standalone attempt was "quickly abandoned again." Rather than picking the language by gut feeling, GottZ built a 92-sub-agent armada to compare the candidates neutrally — a methodological principle: "I built myself tools that cover my ignorance and deliver neutral results." The result was Go. The actual port was trivial — "the entire ctx rewrite from TypeScript with n8n to Go was a one-hour problem." What came out was a single-binary monolith (CLI ctx + daemon ctxd) with its own inference client, its own migrations (embed.FS), and LISTEN/NOTIFY-based async processing.

Until around April 2026, all inference ran on Ollama. It was switched fully to llama.cpp (image ghcr.io/ggml-org/llama.cpp:server) and Ollama was removed entirely. Every backend has since spoken the OpenAI wire protocol, and the routing of roles (chat, synthesis, Dream, digest, classify, translate, rerank, embedding) runs through the context_backends table. A pivotal step was moving the embedding service off the GPU onto a CPU-based llama.cpp side container (llama-embed) on the ctx host — this frees VRAM for chat/rerank on the RTX 6000 but turns the bulk embedding rebuild into a CPU bottleneck. Synthesis/Dream today run on qwen3.6:27b (herbert-chat, @96k context), reranking on bge-reranker-v2-m3 (herbert-rerank), and embedding on qwen3-embedding-8b (1024d Matryoshka from a native 4096d).

Retrieval started as a weighted sum — which failed because ts_rank and cosine similarity have incompatible score distributions and cannot be meaningfully added. It was replaced by a 3-way hybrid fusion (dense pgvector + sparse tsvector + trigram pg_trgm) built on Reciprocal Rank Fusion, which combines ranks instead of raw scores. The maturation step was the extension to today's 4-Way Weighted RRF (k=60): Semantic 0.45 + English FTS 0.25 + German FTS 0.20 + Trigram Title 0.10. The fourth channel (German FTS, weight 0.20) was added once the bilingual gap had been identified as 75% of the retrieval problem — German queries against purely English titles returned effectively zero FTS hits. The entire fusion runs in ONE SQL transaction over the same tables, with no network hop to an external vector store.

For a long time ctx had NO cross-encoder; reranking was RRF alone. In Wave 2 (2026-06-08), bge-reranker-v2-m3 was added as a local llama.cpp sidecar (port 8082), proven via live A/B (5 surface-level queries) and a 14-agent review that caught two scale bugs in the process. The real insight came out of Wave 3 (47 verified gold cases, sweep.py): the cross-encoder did NOT raise the recall ceiling (a reorderer cannot surface anything RRF does not deliver) and was net zero-to-negative on latent gold — it scores surface similarity, but the gold rewarded latent link relevance. Hard-coupling graph+ce was even destructive (R@10 0.571 instead of 0.715 for graph alone), because the CE re-ranks the graph-injected latent neighbors back out. Production verdict: graph default-ON, rerank default-OFF, until real surface-level gold exists; if the CE is enabled, then with a blend of bw=0.5 and never as the final arbiter. Two pipelines for two query classes instead of one winner.

ctx was originally pure CLI + API. The web frontend was built per a masterplan of 41 waves (G01–G41) along six axes: F1 config core, F2 settings, F3 backend pool + trust, F4 SPA, F5 graph, and F6 chat. Concretely, the waves delivered the dashboard/settings, a chat client, a graph map (a Louvain-cluster supergraph over the dream_links, v3.15.0), and a backend pool with a vault editor (G32) for provider failover. After the 41-wave plan was completed, the Block Workbench (CRUD, v3.16.0) was built as the first subsequent wave. Important: the graph map is an overview visualization, NOT query traversal — at query time the dream links are still used only as a supersedes drop filter.

In v2.0.0 the Scope model was generalized from a hard CHECK-constraint enum (chk_scope, originally private/work/shared) to a generic VARCHAR(50) (migration M044). Scopes have since been configurable strings rather than hardwired values — the prerequisite for building Multi-Tenant (Model C: Tenant/Scope/Block levels + cross-tenant grants) on top of them. This generalization is part of a continuous line from "hardcoded mechanism → configurable policy": the later core-redesign decision (block_role enum → dynamic type configs, 2026-06-01) is explicitly "the same line, one level deeper" — mechanism (embedding, RRF, Guard, Dream) stays code, policy becomes data.

The Dream cross-reference engine matured primarily through its LLM prompt. Pre-reset (2026-04-20), accuracy at raw_conf≥0.7 was only 53% (47% error rate): factual overused, supersedes drastically underproduced (13/1836), causal lacking a temporal check. From that came Prompt v3 (Session 24, 2026-04-23): topical-as-fallback default, supersedes VERY RARE, causal accepting decision→implementation — 62.3% accuracy on the top-50 hardest cases (+19.7pp). Wave 46 (Session 38) additionally flipped the supersedes convention (src=old → src=newer) and brought the supersedes VALID rate from 0% to 80%. The later inference-reduction study (W49c) then rejected both v6 AND v7: prompt variants do raise real recall but cost none precision; production-weighted (real candidate sets are 70–85% none), v5 dominates (best none precision 0.776) — v5 is production-optimal. minRawConfidence was unified at 0.7 for all types (qwen3.6 has uncalibrated confidence).

ctx was shaped by live tests on the real corpus, not by assumptions. Rejected: n8n (too bloated, talks too much); Q8_0/FP16 in favor of Q4_K_M (higher weight precision → more paraphrasing → keyword coverage drops from 94% to 88%, worse for RAG); models >9B for synthesis (tested up to ~22B, larger ones do not extract better; qwen3.5:9b also beat Gemma candidates — reasoning modes eat the token budget, with a death spiral under native thinking: 19.4x token overhead); the samplers Mirostat/tfs_z/typical_p; an external vector store (Qdrant/Weaviate/Milvus) — rejected because it costs ACID transactions, JOINs/CTEs, and atomicity, and offers no real performance advantage below ~5M vectors; JSONB temporal_coords (in favor of the EAV context_temporal table with O(log n)); content_dates DATE[]; a 5th RRF channel; SQL gravity functions; a cosine pre-filter; two-pass synthesis; the GSD planner. Further validated lessons: FTS month names had a 75% false-positive rate (removed, only ISO+YYYY-MM); the embed prefix caused an 18.6% centroid shift (disabled); Guard thresholds 0.95/0.85 had 80% FP (raised to 0.98/0.92); agent claims are factually wrong about 1 in 8 times (always verify against the code).

The PL/pgSQL function ctx_rrf (migration 048_rrf_exclude_params.sql) computes a weighted RRF score per candidate across four parallel CTEs: Semantic (pgvector HNSW cosine over halfvec(1024), weight 0.45, LIMIT 75 candidates), English FTS (ts_rank_cd over ts_en, 0.25, LIMIT 100), German FTS (ts_de, 0.20, LIMIT 100), and Trigram Title (pg_trgm similarity, 0.10, similarity > 0.05, LIMIT 30). The fusion uses the classic 1.0/(60 + rank) formula per channel with k=60, joined via a FULL OUTER JOIN across all four result sets. Each term is additionally multiplied by a block_mass factor (1/sqrt(number of content_times), which dampens mega-blocks) and a block_role_factor (audit-trail damping). Filtering is applied on NOT is_archived, block_role != 'system-meta', scope = ANY(read_scopes), plus optional category/tags/exclude lists. Go invokes the function with exactly one SQL call (rrf.Search) and internally retrieves 200 candidates so that Gravity, Graph, and the reranker can re-sort on a broad basis before the result is trimmed to the user limit (1-20, default 5). The 5th RRF channel was removed in M020 (it was never enabled from Go).

The four fusion weights are hard-wired in migration 048: 0.45 Semantic, 0.25 English FTS, 0.20 German FTS, 0.10 Trigram Title; the RRF constant is k=60. Confidence classification runs in the Go layer (llm.ClassifyConfidence) against two thresholds from the config registry (config.go): query.score_threshold default 0.001 and query.confident_threshold default 0.008. maxScore >= 0.008 yields 'confident', >= 0.001 yields 'low_confidence' (then capped at a maximum of 2 sources), and anything below yields 'no_relevant_blocks_found'. Important (verified against config.go, correcting an earlier briefing): the lower threshold is 0.001, not 0.005 — it was lowered in Wave 37 because the mass_factor in the RRF score was pushing large architecture blocks below 0.005 and thus dropping them out of the synthesizer. A config validation rejects score_threshold > confident_threshold (otherwise low_confidence would be unreachable).

After RRF and Gravity (and before filterSuperseded), a cross-encoder re-scores the candidates. Dispatch goes through the backend routing table: if a backend carries the rerank role, the request goes to the llama.cpp sidecar (rerank.Score speaks the Cohere/Jina wire format /v1/rerank, model bge-reranker-v2-m3 on CPU, up to MaxDocs=50 candidates, with title + up to 2048 characters of content per doc); if the role is missing, it falls back to the historical LLM-as-judge over the synthesis chain (up to 15 docs, 400 characters, JSON score array). The cross-encoder returns RAW logits (which can be negative, e.g. -11..+6); rrf.RerankCrossEncoder maps them onto (0,1) via a sigmoid before blending. Blend: rerankWeight*rerankNorm + (1-rerankWeight)*rrfNorm, default BlendWeight 1.0 (pure cross-encoder ordering, also the final arbiter over graph-injected neighbors). rerank.Score realigns the server response STRICTLY by result.Index (the server sorts descending and trims to top_n — array position is meaningless) and validates gap-free index coverage. Both paths are fail-open: any error preserves the pre-rerank ordering. Because of the CPU latency (~1s per pair), a chunked-body heartbeat (a single space every 25s, a valid JSON prefix) keeps the connection alive through buffering reverse proxies.

After retrieval/rerank, llm.Synthesize builds the answer: filter by score_threshold, confidence classification, low-confidence capping to 2 sources, lost-in-the-middle reordering (best source first, second-best at the end), then the LLM call over the synthesis chain. The model is qwen3.6:27b (backend herbert-chat, RTX 6000), think:false. The system prompt is configurable (config key query.prompt_version): the default is v5.2 ('You are a fact extraction engine for a technical knowledge base'), and the Wave 48 variant v6 with graded confidence is also available (three modes: DIRECT / INFERRED-with-caveat / REFUSAL). Note (verified against synthesize.go, correcting an earlier briefing that mentioned 'v5.3'): the prompt versions defined in the code are v5.2 and v6. The prompt's hard rules: every fact must come from the sources, at most 3 sentences, inline citations [1][2], answer in the language of the question, and on zero hits exactly 'NO_RELEVANT_SOURCES'. A security block instructs the model to ignore any instructions embedded in the sources (prompt-injection protection). Each source is rune-safely truncated to at most 1500 characters; for temporal queries the current date plus resolved references are injected into the prompt. NO_RELEVANT_SOURCES is normalized to 'I don't know based on the available sources.' (locale-agnostic for CRAG judges); if RRF was 'confident', it is downgraded to low_confidence rather than no_relevant.

llm.DetectGerman identifies German queries by umlauts/ß, >=2 of 50 stopwords, or >=1 of 16 domain tech terms (word-boundary, case-insensitive). On a match, llm.TranslateQuery translates the query into an English search query over the translate chain. The prompt contains a fixed domain glossary (e.g. Schreibschutz = Write Guard, Sicherheitstrennung = Scope Isolation, Duplikat-Erkennung = Duplicate Detection, Kontextspeicher = Context Store, Einbettung = Embedding, Schwellenwert = Threshold, Zugriffsschlüssel = API Key). The translation passes through hard gates: SQL meta-characters are stripped, at most 3x the length of the original, only safe characters (alphanumeric + a few punctuation marks), single-line — otherwise the original query is kept. Key architectural property: only the SEARCH query is translated; the LLM synthesis receives the ORIGINAL (German) query so that the answer is produced in the user's language. The entire step is fail-open (empty/exhausted chain → original query).

Temporal resolution is primarily deterministic: llm.NormalizeTemporalRules (temporal_rules.go) runs without an LLM call. There are exactly 18 match* functions (verified via grep), processed in a priority cascade from specific to vague: P1 ISO dates (purely linear) → P2 compound ranges (from-to, linear+weekday) → P3 since/until (linear+month) → P4 week segments → P5 relative weeks → P6 relative months → P7 weekday within a week → P8 relative days (N days ago / in N days, purely linear) → P8a annual holidays (month+seasonal) → P8b month segments (start/end of month, monthday) → P8c time of day (morning/evening, daily) → P9 weekend (weekday-dominant) → P10 multi-keyword (today and tomorrow) → P11 weekday + verb tense → P12 simple keywords including typos. Helper functions: DetectVerbTense/hasPerfektForm (past → LAST, future → NEXT), hasRecurrence ('immer/jeden/dienstags' upgrades to pure weekday weighting), Levenshtein fuzzy matching for typos. Only when the rules return nil AND HasTemporalIntent (whole-token match against ~70 intent words + ISO regex) is true does the LLM fallback kick in (NormalizeTemporal over the translate role, a calendar with an explicit LAST/NEXT table so the model only looks up values instead of computing them).

Resolved dates are translated via TemporalToFTSExpansion into a websearch_to_tsquery OR string that feeds into ctx_rrf as an additional FTS channel. Deliberately kept narrow: only ISO dates (2026-03-23) and YYYY-MM prefixes (2026-03). Weekday names, month names, and calendar-week numbers were removed — empirical finding T07 (Session 22): they caused a 75% false-positive rate in the top 5 because they match any block that happens to mention the word, rather than temporally related blocks. Scaling caps for 1M+ blocks: at most 7 distinct dates get per-day expansion (otherwise the first and last half), with an overall cap of 20 OR terms (GIN cost scales linearly with term count × 2 FTS channels). The actual temporal relevance does NOT run through FTS, but post-RRF via the gravity boost and the context_temporal dimension table. The embed prefix for temporal augmentation is also disabled (T06: it degraded 4/5 queries, ~33% centroid shift).

The Cyclic Phase Model (gravity.go) treats every cyclic time structure as an independent dimension with a normalized phase [0,1) and its own Gaussian decay sigma: weekday σ=0.07 (7-day cycle), month σ=0.10 (12-month cycle), quarter σ=0.12 (4 quarters, wider because imprecise), week σ=0.08 (52-week ISO), monthday σ=0.10 (~30-day start/end of month), seasonal σ=0.08 (365 days), daily σ=0.08 (24h time of day). Year remains linear-monotonic (not cyclic). For each matcher the parser sets DimensionWeights (examples from the code: 'am 2026-03-27' → {linear:1.0}, 'am Dienstag' → {linear:0.6, weekday:0.4}, 'immer dienstags' → {weekday:1.0}, 'Q1 Reviews' → {quarter:1.0}, 'im März' → {linear:0.5, month:0.5}). ComputeCyclicGravity computes, per dimension, the CyclicDistance between the query phase and the block phase, applies GaussianDecay = exp(-d²/(2σ²)), and takes the maximum over all anchors of a block (best match, not the sum), then a weighted sum across dimensions. Example σ=0.07: Monday→Tuesday (distance 1/7) decays to ~0.13, Monday→Wednesday (2/7) to ~1e-4 (effectively zero). The boost is multiplicative and capped at a maximum of 0.30 (a 30% score lift), scaling with the sum of the cyclic weights; blocks without a matching dimension get a factor of 1.0 (neutral, no penalty).

The linear gravity path (ComputeGravity/ApplyGravityBoost) treats blocks as masses in time-space: gravity = Σ 1/dist^power over the content_times relative to the query target date, base power p=1.5, minimum distance 0.5 days to guard against a singularity. The gravitation is asymmetric: future dates get a 1.2× sharper cutoff, past ones a 'forward telescoping' — the backward power is lowered via 1/(1+0.3·ln(1+age/30)) so that older blocks get a wider gravitational well (calibrated against Rubin & Baddeley 1989: older memories are coarser; age=30d → power≈1.24, age=365d → power≈0.86). Cutoff default 14 days (60 for ranges), a direction filter (past/future/both), boost capped at 0.30. The GottZ Temporal Dimension Table (context_temporal, migration 009) is an EAV table (block_id, source_date, dimension, value) with partial B-tree indexes per dimension (year/month/week/weekday/quarter) for O(log n) filtering — new dimensions without a schema change. For cyclic queries the handler fetches the dimensions via FetchBlockDimensions, and for linear ones the content_times via FetchContentTimes; mixed weights ('am Dienstag') run both paths, each scaled by its respective weight, so that the total boost stays ≤0.30.

Before trimming to the user limit, the handler enriches the results with the dream.SupersedesMap (key = outdated block, value = newer superseder IDs; the convention since Wave 46). filterSuperseded removes a superseded block ONLY if at least one of its superseders is also present in the result set (a safety rule — otherwise the current information would be missing entirely). The decisive factor is the temporal gate: the filter runs exclusively for non-temporal queries (temporalResult == nil or no dates) — temporal queries explicitly need historical/outdated blocks and must not lose their history. The RRF score ordering is preserved. In the API response, each source optionally carries a superseded_by field. Architectural boundary (honestly documented): the Dream links are NOT traversed @Query — here the graph serves only as a supersedes drop filter; the optional graph expansion (1-hop along positive Dream links, post-RRF/pre-rerank) is a separate, default-OFF gated feature.

Dream Mode is the component that densifies the knowledge graph without human intervention: a scheduler worker (concurrency 1) pulls a Block from the corpus and looks for related Blocks that a human would never have explicitly linked when saving — the LATENT cross-connections. The pipeline runs in eight steps (defined in go/internal/dream/dream.go): Step 0 cycle skip (skipped if no backend serves the role), Step 1 block pick (unchecked first, then the oldest with an expired cooldown via FOR UPDATE SKIP LOCKED), Step 1b temporal dimension validation, Step 2 LLM keyword extraction (max. 5, persisted per Block and reused), Step 3 RRF search per keyword, Step 4 LLM evaluation of the candidates, Step 5 write links, Step 5b recurrent detection, Step 6 quality score update, Step 7 adaptive cooldown, Step 8 promotion to canonical. The pipeline generation is Version=5 (wave 38b, 2026-05-06) and is persisted with every link. Cooldowns are adaptive: 3 days for Blocks that produced links (CooldownActiveDays), 14 days for Blocks without links (CooldownInertDays), 5 minutes on transient system errors such as LLM timeouts (CooldownTransientMinutes). When idle, the worker waits 20s (CTX_DREAM_IDLE_WAIT, hot-mutable). A throttle mechanism enforces GPU cooldown pauses between LLM calls (on/throttled/off via API + CLI).

The graph knows five edge types (validRelationships in linkfilters.go): supersedes (the source corrects/replaces a concrete fact in the target — VERY RARE), causal (the source describes an event/decision whose occurrence enabled the target; decision→implementation qualifies), factual (the source SPECIFIES a concrete thing that the target implements — one-way SPEC→IMPL), topical (both deal with the same subject substantively — the documented fallback), and recurrent. The LLM evaluation prompt (dreamSystemPrompt in evaluate.go) classifies only the first FOUR; recurrent is detected separately in a phase-1 pass (same dimension+value in context_temporal AND title similarity > 0.5) plus a phase-2 LLM confirmation that runs after the main evaluation, so that recurrent wins over topical. The prompt explicitly sets 'topical-as-fallback' as the default. On the top-50-hardest benchmark against double-judge stable gold, it reached 62.3% accuracy (+19.7pp over the pre-reset baseline of 42.6%). The output is a JSON array {target_id, type, confidence}, with a maximum of 5 entries. An attempt at a V6 prompt (causal anti-pattern clauses) was discarded: stable gold n=122 showed V6 to be net-worse (54.1% vs 57.4%), with topical recall dropping from 76% to 63% due to over-hedging toward 'none'.

Each edge type has a minimum raw (unweighted) LLM confidence floor (minRawConfidence): topical, factual, causal, and supersedes sit at 0.7, recurrent one quantization step higher at 0.8 (wave 38b — phase-2 classification has more failure modes). The uniform 0.7 floor replaced the old factual=0.9 gate: qwen3.6:27b produces uncalibrated confidence (full_agree mean 0.87 vs wrong_type mean 0.85 — a gap of only 0.02), which is why the 0.9 gate was a legacy workaround for qwen3.5 and on 3.6 only filtered out 2 correct links without removing any wrong types. After confidence filtering, EvaluateRelationships enforces a hard cap (MaxLinksPerCycle=5), sorted by confidence descending with a type-diversity tie-break. The evaluation runs over the backend with RoleDream (qwen3.6:27b, non-thinking), using the vendor-recommended sampler settings Temperature 0.7, TopP 0.8, TopK 20, NumPredict 400. The HTTP timeout per Dream call is 180s (DreamTimeout, tuned empirically from 5878 production calls: p50=23s, p90=30s, p99=156s), and the total cycle timeout is 700s.

Supersedes is the most dangerous edge type (it marks the target as a snapshot and influences retrieval filtering), so there is a multi-stage structural safety net in Go in addition to the LLM judgment. enforceSupersedesDirection runs as a post-parse hard constraint (in evaluate.go after parseLinks, before filterValidCandidates): for every supersedes link, the source timestamp must lie strictly after the target timestamp; if the direction is inverted, the link is DOWNGRADED to 'topical' rather than discarded (preserving the topical connection). acceptSupersedes (V8, linkfilters.go) checks three conditions at write time: same category (srcCat == tgtCat), source genuinely newer (srcCreated.After(tgtCreated)), and title similarity via pg_trgm ≥ supersedesTitleSimThreshold=0.25. ApplySupersedes (marking the target as a snapshot) fires only at a weighted confidence ≥0.7. Background (wave 46, 2026-05-22): an audit found ALL 15 supersedes links in production to be inverted relative to natural-language semantics ('A supersedes B' → A is the newer authoritative version). The convention was switched from updated_at to created_at and the direction from 'source older' to 'source newer'; migration 042 truncated context_dream_links and a re-Dream pass rewrote the corpus.

The Guard prevents the store from being cluttered with near-duplicates. On every write operation, the PG function ctx_guard_check (migration 011) uses the HNSW index to check the top-1 nearest neighbor over halfvec(1024) cosine distance (cb.embedding::halfvec(1024) <=> v_embedding::halfvec(1024)), excluding self, archived, and chunk Blocks. The thresholds: similarity ≥0.98 → decision 'near_duplicate' → auto-archive (is_archived=true, guard_status='archived_dup'); 0.92–0.98 → 'needs_review' (flagged but visible); <0.92 → 'clean'. These thresholds were raised empirically from 0.95/0.85 because the lower values had an 80% false-positive rate. A cross-scope status is set when the match lies in a different Scope and is not 'shared'. RunGuardBatch processes pending Blocks (NOT is_archived, no guard_checked_at, category != 'index', embedding IS NOT NULL, block_type knowledge/source) via FOR UPDATE SKIP LOCKED so that concurrent Guard runs do not grab the same Block. Each Block runs in its own SAVEPOINT (W47-02 tx-abort-cascade fix): a failed Block (SQL error) is rolled back to its savepoint without poisoning the surrounding transaction (PG 25P02), and the loop continues. Every decision writes an audit entry to context_write_log.

The Guard does not run blindly on a polling cadence but reacts in an event-driven fashion. The DB trigger trg_block_write fires a PostgreSQL NOTIFY on the channel ctx_block_write (channelBlockWrite in events/listener.go) on every Block write operation. A pgxlisten listener receives the notification and calls scheduler.NotifyWrite(), which triggers the Guard run immediately. As a fallback, a time ticker additionally runs with guardInterval = 60s (events/scheduler.go) in case notifications are missed. On reconnect after a connection loss, HandleBacklog unconditionally signals Guard+Digest in order to catch up on writes missed during the disconnection (defaultReconnectDelay 5s). The same listener mechanism also serves the channel ctx_settings_write (for hot-reloading settings/secrets/backends via psql or a break-glass edit). This architecture (LISTEN/NOTIFY + SKIP LOCKED + own migrations via embed.FS) is a central design principle: no external message broker, everything inside PostgreSQL.

The embedding layer (go/internal/embed/embed.go) produces the semantic vectors for retrieval and the Guard. The model is qwen3-embedding-8b, which natively delivers 4096-dimensional vectors that are truncated to TargetDims=1024 via Matryoshka Representation Learning — the first 1024 dimensions carry the bulk of the information. This is followed by L2 normalization and a quality gate: the norm must lie within [0.99, 1.01] (minNorm/maxNorm) and the variance must exceed minVariance=0.0001 (catching degenerate/constant vectors). Asymmetric instruction prefixes separate query from document (PrefixQuery: 'Instruct: Represent the query for retrieving relevant documents', PrefixDocument: 'Represent the document for retrieval'). The client supports two wire protocols: ollama (/api/embed) and openai (/v1/embeddings) — host and protocol arrive as a single backends.Backend tuple, so the wire path can never diverge from the configured host. There is no longer an artificial request timeout (latency is data-dependent on a shared GPU under concurrency); the parent context supplies the real timeout. In the current stack, embedding runs as the backend 'llama-embed' (Docker, CPU-only on the ctx host, physically separated from the GPU) — which makes CPU embedding the throughput bottleneck during bulk rebuilds, not live query latency.

Embeddings are not generated synchronously on write (that would block the write path) but are backfilled asynchronously. On the write path, a Block is stored without an embedding; the scheduler backfill then generates it offline. In addition, there is an inline backfill directly in the query path: backfillPending (handler/query.go) generates embeddings for Blocks that do not yet have one BEFORE the search runs — normally a no-op (0 Blocks), but it guarantees that no freshly written Block drops out of retrieval. Every backfill wire call is logged via logEmbedWire with the pipeline 'embed-backfill'. ClearEmbedding (store/blocks.go) sets the embedding to NULL so that the scheduler backfill regenerates it on demand (e.g. on a model change). A planned extension (T35b) writes embed_model provenance at three write paths (blocks.go, scheduler.go, query.go) plus a one-time backfill — a prerequisite for later selective rebuilds and cost attribution per api_key_id.

A live DB measurement (ctx Block 019ed21c) found that context_blocks.embedding has attstorage='e' (EXTERNAL) — the vector lives OUT-OF-LINE in the TOAST table, NOT inline in the heap. A metadata UPDATE does not rewrite the external TOAST tuple (shared pointer), so the heap does not bloat with the embedding on Block updates. Conclusion: a separate embedding table purely for performance/fragmentation reasons is YAGNI — TOAST already provides the heap/vector split. The right FORM for a separate table, by contrast, is the multi-vector space (different generations, per-tenant models, blue-green rebuild): context_block_embeddings(block_id FK, generation, embedding, PK(block_id,generation)). This aligns exactly with the multi-tenant decision OD-1 (CoW embedding, Block 019ed02c): one shared embedding generation as a hard default (ghosted from the pool/parent), with divergence (a different model/dimension) materialized lazily into separate storage, with refcount GC. An external vector store (Qdrant/Milvus) was deliberately discarded — the core value of ctx is the hybrid fusion (RRF + FTS + graph dream_links + temporal gravity + Scope isolation) within ONE SQL transaction; an external store would add a network hop plus a consistency sync. pgvector is benchmarked at 1M Blocks. The build trigger is data-driven: the first incompatible Tenant OR measured HNSW recall pressure as N grows.

Every API key is stored as a SHA-256 hash (via pgcrypto) in context_api_keys — plaintext keys never leave the table. Authentication runs through the PG function ctx_auth($key), which in the live system returns six columns: api_key_id, home_scope, allowed_scopes, read_scopes, is_valid, and is_admin. The home_scope (e.g. private or work) is the only scope a key may write to and delete from; read_scopes = [home_scope] + allowed_scopes determines what it can read. Element [0] of read_scopes is invariably the home_scope — this ordering is wire-bearing (whoami, chat_sessions snapshot). Three scopes exist live: private (890 blocks), shared (21), and work (3); a typical key such as GottZ Private has home_scope=private and allowed_scopes={shared,work}. The model is fail-closed: an invalid or empty key yields is_valid=false with no existence oracle.

On top of the scope read filter sits a separate admin tier: the is_admin bit on the API key (introduced in migration 052). The RequireAdmin middleware sits behind the auth middleware and gates the sensitive routes — /api/settings (runtime overrides), /api/secrets (sealed credentials), /api/status, /api/llmlog, and /api/events. A non-admin key or a missing auth context results in 403. The admin tier is deliberately decoupled from scope: a key can be admin regardless of which scope it reads from and writes to. Several keys carry the bit live (including Admin (dedicated, host-only), OpenCode, and MCP claude.ai). In the multi-tenant code that is built but not yet deployed (migration 059/060), an orthogonal tenant role (owner | admin | member) is added that confers authority only within the key's own tenant; the Go constants are pinned byte-for-byte to the DB CHECK so that drift can never silently grant rights.

To let cloud clients such as Claude.ai bind to the MCP endpoint, a minimal, home-grown OAuth 2.1 server runs (no external identity provider). The flow: Claude.ai calls /authorize, the user enters their ctx API key into an HTML form, the server issues a single-use authorization code (valid for 5 minutes, in-memory) and redirects back; /token exchanges the code for an access token — where the token simply IS the API key itself. PKCE is mandatory: only S256 is supported (token_endpoint_auth_methods=none), and SHA256(code_verifier) must match the code_challenge, otherwise invalid_grant. Two discovery documents are served: /.well-known/oauth-authorization-server and /.well-known/oauth-protected-resource. Registered clients live in context_oauth_clients (client_id prefix ctx_, secret stored only as a hash) and are created via `ctx mcp add`. Open-redirect defense: a scheme allowlist (http/https only) plus reconstruction of the redirect URL via net/url.

ctx exposes its core functions as an MCP server (Model Context Protocol) over the Streamable HTTP transport for remote clients such as Claude Code. It is implemented with the official modelcontextprotocol/go-sdk v1.5.0; the server identifies itself as ctx version 0.25.3, runs stateless (every call independent), and prefers to respond with JSON rather than SSE. Five tools are registered: query (hybrid search + LLM synthesis, delegating to the full /api/query handler including Temporal, RRF, Gravity, Rerank, Supersedes), store (upsert on category+title+scope, with sensitivity gating), search (lightweight FTS without an LLM), get (block by UUID, with prefix resolution), and recent (most recently modified blocks). Auth runs through the same middleware as the /api routes — X-Context-Key or Authorization: Bearer — and injects the AuthResult into the request context; store/search/get/recent are fail-closed against a missing tenant identity or missing scopes.

The HTTP daemon ctxd (port 8080) uses a chi v5 router with a global middleware chain: SecurityHeaders, RequestID, Logger, Recovery. /health is anonymous and name-free. The OAuth endpoints (/.well-known/*, /authorize, /token) sit unguarded ahead of it — they are the auth flow. All remaining routes are grouped behind the auth middleware: POST /api/query (RRF + synthesis), /api/store (upsert + auto-embed), /api/search (FTS), GET /api/graph/ego and /api/graph/overview, /api/whoami, POST /api/manage (CRUD + Guard), /api/digest, /api/synthesize/daily, the admin-gated /api/settings and /api/secrets, the web chat (/api/chat/stream + sessions), and the blob routes (/api/blob/fetch|search|manage|store). Body limits are tiered: 1 MB by default, 10 MB for /api/ingest, 75 MB for blob store. Admin-only with elevated detail: /api/status, /api/llmlog, /api/events (SSE). Unknown paths fall through to the embedded SPA (History API fallback), while incorrect API paths still return 404/405.

ctx is a Go monolith (module github.com/GottZ/ctx, with go.mod declaring Go 1.26.0) with two entry points: the ctx CLI (installable via go install) and the HTTP daemon ctxd (port 8080). The business logic lives in roughly 25 internal packages, including auth (SHA-256 key verification), handler (HTTP/MCP/OAuth), llm (chat/synthesis/translate/classify/temporal), embed, rerank, rrf (4-Way Reciprocal Rank Fusion), backends (the declarative pool), store (PG access, migrations via embed.FS), dream (async cross-reference), guard (dedup), digest/overview (topic map + graph), events (scheduler, LISTEN/NOTIFY), settings, sealbox (sealed secrets), sensitivity, and chat. Core dependencies: go-chi/chi v5.2.5, jackc/pgx v5.9.2 plus pgxlisten (PG LISTEN/NOTIFY with SKIP-LOCKED jobs), pgvector/pgvector-go v0.3.0, and modelcontextprotocol/go-sdk v1.5.0. Custom building blocks instead of frameworks: a custom migration runner (tracking table _migrations) and a custom LLM client.

The persistence core is PostgreSQL 18 with the extensions pgvector (HNSW cosine index over vector(1024)), pg_trgm (trigram title matching), GIN full-text indexes (English + German), and TimescaleDB. The live database context_store currently sits at migration 57 (tracked in the _migrations table). A deliberate decision against an external vector store (Qdrant/Milvus): the core value is the hybrid fusion — RRF + FTS + graph traversal + temporal reranking + scope filter — in ONE transaction, all in SQL; pgvector is benchmarked for 1M blocks. The embeddings are stored out-of-line in TOAST (attstorage EXTERNAL), so that metadata updates do not rewrite the vector tuple. The GottZ Temporal Dimension Table (context_temporal, an EAV layout with partial B-tree indexes) provides O(log n) access to temporal anchors. The schema includes, among others, context_blocks, context_api_keys, context_backends, context_oauth_clients, context_settings/secrets, and the Dream Mode link tables.

All inference runs on-prem via llama.cpp (Ollama has been fully retired since ~April 2026) — every backend speaks the OpenAI wire protocol: /v1/chat/completions, /v1/embeddings, and the cohere-style /v1/rerank interface. Routing is declarative through the context_backends table: each row carries roles (synthesis, translate, chat, digest, dream, classify, embed, rerank), a model_map (role→model + params), a priority, and a trust level (full-trust | no-credentials | non-personal | public), which together with the block sensitivity gate which content may reach which backend. The Chain() method resolves a priority-DESC chain per role with cooldown sorting. Five backends are live: herbert-chat (qwen3.6:27b, think:false — chat/synthesis/translate/digest/dream/classify, full-trust, lan, prio 100), herbert-rerank (bge-reranker-v2-m3, prio 100), llama-embed (qwen3-embedding-8b, 1024d Matryoshka from a native 4096d, prio 100), openrouter (qwen/qwen3.6-27b, no-credentials, external, prio 50), and llama-cpu (prio 10) as a CPU fallback. Hardware: herbert is a Windows box with a single Quadro RTX 6000 (24 GB) running two llama-server instances (~21 GB occupied); the ctx Docker host has no GPU, so embedding runs CPU-only. Backend mutations take effect without a restart (hot-reload via LISTEN/NOTIFY).

Releases are produced via .github/workflows/release.yml: a build matrix produces five platform binaries — linux/amd64, linux/arm64, darwin/amd64, darwin/arm64, and windows/amd64. The current release tag is v3.16.0. Versioning is hook-enforced: the pre-push hook rejects lightweight tags on version releases and requires annotated tags whose annotation serves as the release body. Further mandatory hooks (core.hooksPath = .hooks): pre-commit (golangci-lint on staged Go files) and commit-msg (README review on feat: commits and schema migrations). A note on version scoping: the release tag (v3.16.0, MPL-2.0, author GottZ) is independent of the internally reported MCP server version (0.25.3), which belongs to the MCP implementation.

The admin SPA lives under go/web/ (Svelte 5.56.3 + TypeScript + Vite, Bun as the toolchain, roughly 13,660 lines of source) and is compiled directly into the ctxd binary via go:embed. It is built in a dedicated Docker stage (oven/bun:1.3-alpine, bun install --frozen-lockfile, svelte-check gate) — only the Docker image ships the real UI; a bare `go install .../cmd/ctxd` produces a binary that serves a 503 placeholder while all APIs remain fully functional. The CLI (cmd/ctx) never depends on the frontend. What gets served are hashed assets (immutable-cached, pre-compressed .br/.gz); the History API fallback responds only to HTML navigations (Accept: text/html), so that mistyped API URLs still return 404 for JSON clients. The master plan G01–G41 was fully built with v3.14.0 (2026-06-13); routing is handled by sv-router 0.16.3. Current release head: v3.16.0.

The Graph section (/graph?focus=<uuid>, deep-linkable) renders Dream Mode link ego networks via sigma 3.0.3 (WebGL) over a single graphology 0.26.0 instance as the single source of truth — deliberately kept outside Svelte reactivity, because the Runes proxy overhead on thousands of node objects would be too expensive. Layout uses ForceAtlas2 (graphology-layout-forceatlas2) in a web worker (Blob URL; the CSP carries worker-src blob:), taking 3–10s depending on graph size. Client memory is hard-capped: above 5,000 nodes / 20,000 edges, the nodes farthest from the focus (BFS distance, LRU tiebreak) are evicted down to 4,000; pinned nodes and the focus survive. Block content never travels over the graph endpoint (titles only). The graph map (overview map, named outside the 41-wave plan) resolves the scope count leak via scope-partitioned pre-aggregates (migration 057: three regular tables graph_cluster_member/_node/_edge, no materialized view) and a gonum Louvain daemon job (internal/overview/cluster.go, raw_confidence weighting, hardwired seeds for reproducibility, advisory lock); cluster_id is never exposed raw but assigned per request as an ordinal. Released in v3.15.0 (2026-06-14). LIVE deployed: clusters=70, nodes=669, edge_rows=58, modularity=0.857, elapsed=45ms (corpus currently 691 blocks).

The Blocks section is a corpus browser: full-text search plus category/tag/Scope facets, a sensitivity-badged, keyset-paginated newest-first list over the scope-gated /api/search, a detail panel, and create/edit/delete via /api/store plus manage update/delete (with sensitivity-downgrade and delete confirmations). The Settings section renders the full Settings API catalog generically from registry metadata — one category card per key prefix, widgets dispatched by registry type, a source badge (default/env/db) and the env-var name per field; hot keys edit live, while restart/coupled keys stay read-only. Embedded within it is the Backends subroute (/settings/backends): a backend pool editor with a trust dropdown and elevation-confirm dialog, a role multiselect, a model_map row editor, priority ordering, and a per-row reachability test, plus the write-only secrets vault with reference tracking. The Chat section (/chat) streams a single turn from POST /api/chat/stream via fetch + eventsource-parser (no reconnect — a turn is one-shot), shows collapsible tool-call cards and a backend badge; assistant Markdown runs through markdown-it (html:false) + DOMPurify, and ctx citations are rewritten to /graph?focus=<id> before sanitization. Test coverage: vitest, svelte-check 0/0.

The CLI (cmd/ctx, spf13/cobra) registers exactly 25 top-level commands: query, save, search, stats, categories, get, delete, list-meta, digest, guard (list/stats/resolve/throttle), manage, health, ingest, statusline, dream (stats/review), mcp, keys, brief, persist, settings (get/set/unset/list), secrets, backends (list/create/update/delete/test), eject, and init. `ctx query` is the primary interface (hybrid semantic + full-text search with LLM synthesis, returning an answer plus source refs); `ctx save`/`store` upserts by (category, title) and generates embeddings; `ctx search` returns compact hits without an LLM. Administrative commands such as `keys` (X-Context-Key provisioning; home_scope has been mandatory at creation since v2.0.0) and `mcp` (OAuth client registration for MCP remote) round out the family. The CLI never depends on the web frontend.

Blob storage exposes four POST endpoints (/api/blob/store, /fetch, /search, /manage), registered in cmd/ctxd/server.go. store accepts base64-encoded data (50 MB limit, otherwise 413), validates the write scope against home_scope or shared (otherwise 403), respects the write rate limit, and upserts via store.UpsertBlob by (category, title) along with filename, MIME type, tags, metadata, checksum, and storage_type. fetch retrieves by ID or category/title (MetaOnly option, data base64-encoded), search filters scope-gated by category/tags/MIME type/limit, and manage knows the actions stats, get, list, and delete. Every write is logged fire-and-forget via LogAccess. Auth uses the same X-Context-Key header.

`ctx brief --hook` is the SubagentStart hook: it reads the CWD from stdin JSON, looks up project-tagged blocks in the agent-briefing category (falling back to untagged), and returns the Claude Code hook JSON format (continue, suppressOutput, hookSpecificOutput.additionalContext). Without --hook it returns human-readable text; scoping is by project tag (git root or CWD), with a default limit of 20 blocks. `ctx persist --hook` is the SubagentStop hook: it parses [PERSIST:category:title] markers from the last assistant message and saves them into the store, including automatically enriched metadata (session_id, hostname, project, agent). Both degrade silently (no error) when ctx is unreachable or no markers are found. The `-p` armada mode (`claude -p "..."`) serves as a context firewall that isolates subagents.

`ctx ingest <vault-path>` walks an Obsidian vault directory, parses and chunks all .md files (collectMarkdownFiles skips .obsidian/, dot directories, and node_modules), and uploads them in batches. The parser (internal/ingest, package ingest) processes frontmatter (tags, date, title, aliases, custom fields). Metadata is marked with source=obsidian. Flags: --dry-run (preview without upload), --workers and --batch for throughput control. Example invocation: `ctx ingest /path/to/vault/ --workers 4 --batch 100`.

The core elaboration follows the hybrid Model C with three levels: TENANT (context_tenants, an owner registry as an independent project/company), SCOPE (context_tenant_scopes, which remains the data discriminator without a tenant_id on data tables), and BLOCK (context_block_grants, row-level sharing). The API key carries tenant_id and tenant_role; read_scopes are resolved per tenant in ctx_auth (own ∪ grants, fail-closed). Scope: 44 waves (T01–T43, with T40 split), phases A–G, 11 migrations 058–068, roughly 7805 LOC. Verified against Git (branch feat/multi-tenant, HEAD ddbfd7f): 20 waves committed, migrations 058–064 (7 of them) present — all additive and behavior-neutral, so construction can be paused at any time. Still local, not pushed or deployed (the live DB has max(migration)=57).

An enterprise layer builds on top of the core MT planning, elaborated across five dimensions: reseller/tenant management, corporate hierarchy (parent/subsidiary), sandbox environments/ephemeral tenants, separate models (embedding + inference), and load distribution. Branding/white-label is deliberately parked. Key insight: administrative inheritance is orthogonal to data-visibility inheritance — the latter is opt-in and often an anti-feature. The super-tenant is a real default tenant as the tree root (parent NULL); _global never becomes a tenant. Shared embedding generation is a hard default, with load keyed on tenant_id. The analysis covers roughly 140 requirements and identifies six fail-open spots in the MT path.

Planned after the web frontend and multi-tenant (not yet built): the core redesign separates mechanism from policy. The hardcoded 4-class block_role enum (system-meta/audit-trail/reference/knowledge) becomes configurable type configs with behavior fields (retrieval_policy, guard_dedup, dream_linkable). Goal: users build their own workflows on ctx. The first workflow on top of it is an issue system, where an issue is a context_block in a repo-specific scope tenant and inherits the ctx machinery for free: Guard as duplicate detection, RRF as related-issues, Dream as autonomous cross-linking. Forge sync (GitHub first) reconciles via a 3-way content hash. End goal: users and agents document the rest as issues in the system itself, at which point todos.md disappears entirely.