How the hermes-graph triage works

From raw GitHub history to guard-railed comments — every gear of an autonomous triage pipeline, end to end.

system snapshot · 2026-08-03 · NousResearch/hermes-agent the mechanics are current; the counts age gracefully.

Harvest pulls the full repo history into a private graph database; Derivation folds edges into duplicate/topic complexes; Triage runs a three-stage LLM pass (assess, verify, draft) per complex; Curation tries to refute every draft before it may leave; Execution posts through a server-side guardrail chain under a standing order whose hourly pace an adaptive AIMD governor tunes against GitHub's live limits; Clockwork paces it all with timers plus an event-driven dispatcher — including an expansion pass that seeds a first-post draft for every uncovered open node and an LLM-verdicted comment-mention lane that feeds the fold graph; Observe writes every run into a journal that feeds alarms, reports and a live dashboard — all standing on a least-privilege Postgres foundation.

Harvest

GitHub → graph truth

GitHub Client GraphQL ∪ REST · rate-window aware backoff + jitter · budget abort
  • Hand-built client — no octokit plugin; proving its own backoff behavior was a build gate.
  • Understands both Retry-After forms (seconds and HTTP date) and detects secondary limits from the response body.
  • Pauses on a remaining-quota floor instead of running the window to zero; a hard budget abort raises RateLimitBudgetExceeded.
  • Ships its own backoff selftest and writes per-run telemetry to harvest_telemetry.
1rate-windowed GraphQL ∪ REST
Hourly Loop (:17) delta + comment streams each slot prefetch · contributors · alarms
  • Runs as a compose service on a collision-free slot raster (phase 1020 s → :17 UTC); cadence is re-read from the database every cycle.
  • Each slot spawns delta and comment-stream subruns, then a diff prefetch for top complexes (250 by config).
  • Refreshes the repo-contributor table and runs a disk check plus alarm detection, both fail-safe wrapped.
  • Recent health: 24/24 slots green over the last day (snapshot).
diff prefetch · top complexes
Hydration Sweeps open-scoped backfill · resumable capped sub-connections
  • Ascending, cursor-resumable sweeps over open PRs and issues with idempotent upserts.
  • Sub-connections are capped per node (files 50 · commits 50 · comments 30 · reviews 20 · closing refs 20 · timeline 50); the overflow is closed later by outlier pagination.
  • Partial GraphQL answers never tombstone a node — a completeness guard blocks destructive writes.
  • Edge sets are replaced only through hg_resync_* functions, never by ad-hoc DELETE/INSERT.
Delta Cycle watermark · all states updated_at DESC · ~1 h lag
  • The freshness workhorse: fetches everything changed since the watermark, including closed and merged items.
  • The next watermark is taken BEFORE the first fetch — nothing can fall between two cycles.
  • Also carried a flagged one-time merged_at backfill for the existing corpus.
  • Live watermark lag at snapshot: about one hour for PRs and issues.
Comment Streams 2 REST since-streams → node_dirty with reason
  • Two separate streams with separate watermarks: issue comments and PR review comments.
  • Every harvested comment marks its target node dirty with a reason (new_issue_comment / new_review_comment) — the feed for incremental derivation.
  • An is_own filter keeps the system's own posts out of the discussion context (echo protection).
  • Also backfills gh_comment_id for posted drafts by joining on the comment URL.
Review Bodies top-level review verdicts APPROVE / REQUEST_CHANGES
  • A dedicated stream for top-level PR review bodies — added after 4 of 5 curation failures traced back to an invisible keep-open review.
  • Review state is tagged into the LLM working set as [PR review:STATE].
  • Before this stream existed the corpus held ~95k issue comments but zero review bodies — exactly the signal class earlier drafts violated most.
Deep Backfill ~68k nodes · full history one-time build phase — done
  • All-states full pass over the entire issue/PR number space; regression detection ("you remove X, introduced in PR Y") needs full history.
  • Adaptive fetch sizing and wall-clock-budgeted tranches; NUL scrubbing protects Postgres text columns.
  • Finished green 2026-07-22; the delta cycle carries all updates since.
  • Includes a built-in red-proof mode used to demonstrate failure paths before trusting the green runs.
Linked-Closed Batch referenced but unhydrated by-number GraphQL batch
  • Pulls numbers that edges point to but harvest never hydrated, via GraphQL by-number batches.
  • The candidate query is itself incremental; the nightly pass resets its state before running (single-statement reset + separate verify).
  • A run can legitimately process zero items when the delta cycle got there first.
Outlier Pagination closes capped connections sweep + backfill phases
  • Two-phase closure of the edge gap left by capped sub-connections: marker sweep, then paginated backfill.
  • Overflow markers are maintained inline by delta and sweeps, so the nightly pass runs only the backfill phase.
  • The backfill phase is deliberately safe to run parallel to a complex rebuild.
2idempotent upserts · hg_resync_*
3reason-tagged → node_dirty
Nightly Pass (03:03) linked-closed · outlier · dup fail-closed kill switch
  • Nightly incremental chain: linked-closed reset + run, outlier backfill, then ANN duplicate scoring.
  • Guarded by harvest.nightpass_enabled (anything but true means off).
  • The duplicate step checks the triage unit via systemd ActiveState — oneshot units report activating for their whole run, so is-active would lie.
03:03 · guarded
Diff Cache prefetched, never live bound to head_oid
  • PR diffs are fetched harvest-side into diff_cache; the triage executor never talks to GitHub.
  • Each diff is keyed to the PR's head_oid — a stale head means a refetch, not a silent mismatch.
  • The diff is the largest per-PR input of the assess stage; prefetch depth is a config knob.

Derivation

edges → complexes

1OpenRouter qwen3 · L2-norm
Embedding Pass (:43) qwen3-8b → 1024-d vectors pending set = the cursor
  • Embeds new node texts via OpenRouter (qwen3-embedding-8b), Matryoshka-truncated to 1024 dimensions, then L2-normalized.
  • The pending selection IS the cursor — no new nodes, no work, a clean no-op.
  • Provider pinning prevents silent model drift; a byte-stable seed formula makes text_hash idempotent.
  • Cost class: cents — about $0.01 per million input tokens.
2ANN k=10 · ≥ τ 0.59 · + deterministic competes_with
ANN Duplicates k=10 neighbors · ≥ τ 0.59 + deterministic competes_with
  • Embedding nearest-neighbor scoring produces duplicate_of edges; a deterministic pass derives competes_with edges.
  • Neighbors are same-kind restricted (issues match issues, PRs match PRs).
  • An optional cross-encoder rerank (bge-reranker-v2-m3, sigmoid on raw logits) exists but is not part of steady operation.
  • Runs in the nightly pass or supervised; ~670k embedding duplicate edges live.
3df-damped Jaccard ≥ 0.5
File Overlap df-damped Jaccard ≥ 0.5 → file_heuristic edges
  • Scores PR pairs by shared files with document-frequency damping — without it, everything glues to hot files.
  • Thresholds are policy data (tau_jac 0.5, file_df_ceiling 30), not code.
  • Every run prints its active threshold snapshot (W11 discipline).
  • ~12k file_overlap edges live at snapshot.
4union-find · ~0.6 s · closes ∪ pipeline ∪ file≥.75 ∪ embed≥.88
Complex Rebuild (:33) union-find over fold edges shadow staging → atomic swap
  • Global connected components over the fold set: closes always, pipeline edges always, file heuristic ≥ 0.75, embedding ≥ 0.88, LLM-verdicted comment mentions ≥ 0.9 (new 2026-08-03); raw reference edges stay deliberately excluded — only the verdict gate lets a mention fold.
  • App-side union-find takes ~0.6 s where SQL label propagation took >25 minutes.
  • Result is staged in shadow tables, then swapped atomically by RENAME with a 10 s lock timeout; a failed lock rolls back to a consistent old generation.
  • The slot first consumes dirty batches, then rebuilds, then runs both post-swap guards.
  • Snapshot: 12,447 complexes from 80,590 fold edges.
5atomic RENAME swap · lock_timeout 10 s
swappedlock busy → retryrollback → next slot or pace
6view re-bind + grant probe
guards passdriftunit fails loudly
comment → touch · structure → BFS ≤2
Dirty Neighborhoods comment → activity touch structure → local BFS ≤ 2
  • Consumes node_dirty in batches; the reason decides the reaction.
  • Comment reasons only touch last_activity (re-activates triage) — no rebuild.
  • Structural reasons (edge removed, force push, new edge) trigger a bounded local rebuild, breadth-first, at most 2 hops.
  • Dirty volume also feeds the pace dispatcher's rebuild signal.
Stand Hash sha256 over member states (signature, stand_hash) = key
  • stand_hash is a sha256 over the ordered member-state representation of a complex — the system's change detector.
  • The pair (signature, stand_hash) is the dedup unit of the whole triage: each state of each complex is worked exactly once.
  • A moved hash makes a complex triagable again; posted_stand freezes the state a comment was posted against.
  • The server-side freshness gate compares against the claimed hash before any post.
Read-Txn Discipline commit before every LLM call lock_timeout 10 s on swap
  • No database read transaction stays open across an LLM or network call — idle workers holding locks once covered the rename swap for hours.
  • The swap function sets its own 10 s lock timeout: fail fast and loud instead of queueing behind readers.
  • This discipline is enforced in every unit of the triage trio plus curation and drain.

Triage

assess → verify → draft

1enqueue · same-stand excluded
Queue & Enqueue each (sig, stand) exactly once done ∪ error excluded same-stand
  • Every :37 slot starts by enqueueing active complexes through a SECURITY DEFINER function — the queue is always fresh.
  • A complex finished on the same state stays out; an errored complex re-enters only after its state moves (symmetric exclusion).
  • A claim guard also skips complexes that already have a draft for the same state — no LLM work is spent on a deterministic dedup fail.
  • Old rows age into an archive table after 30 days.
Claim & Complete SKIP LOCKED · 30 min lease OPEN targets first
  • Multi-worker claiming: expired leases are reclaimed, then rows are taken oldest-first with FOR UPDATE SKIP LOCKED.
  • Complexes with an OPEN posting target are prioritized — posting yield first.
  • Disjointness under two concurrent connections is probe-proven, not assumed.
  • Signature-bound claiming exists for probe isolation.
2lease reclaim → OPEN first → SKIP LOCKED
Runner (:37 · ×8) 8 workers · 100 complexes/slot
  • The slot script launches 8 parallel workers, each with a slice of the cycle limit.
  • The script pins TRIAGE_LLM_BACKEND=codex — the backend env is the single most important line of the slot (the default is a test fixture).
  • Hard-killed workers are harmless: their leases expire and the work is reclaimed.
  • Runs on the host path by design; snapshot health 25/25 slots green.
Cycle Orchestrator claim → assess → verify → draft → complete · one run_id
  • Chains the full pipeline per complex under a single run_id in the run journal.
  • Errors are isolated per complex — one failure kills one run, never the slot.
  • Token usage is booked as a delta per complex; the backend trio (fixture / claude-cli / codex) is selected once per process.
  • A selftest mode forces uniform fixtures for deterministic probes.
3stage 1 · per-PR vs diff + discussion
Stage 1 · Assess per-PR · diff + discussion write-less executor
  • Analyzes each member PR against its cached diff and full discussion; results land in triage_pr_assessment.
  • The harness asserts at startup that NO GitHub write credential is in the environment — the executor cannot post even if prompted to.
  • Context chunking is budgeted from the live model window (measured 1.05 M tokens, not assumed), with a byte guard against runaway prompts.
  • Per-PR caps were deliberately removed — the model window is the only hard limit.
4stage 2 · adversarial pair verdicts
Stage 2 · Verify adversarial issue↔PR verdicts edges only via write_llm_edge()
  • An adversarial head issues one verdict per issue–PR pair (fixes / partial_fix / unrelated / duplicate_of / best_fix).
  • Verdicts are written exclusively through a SECURITY DEFINER function that hard-codes source='llm' — LLM output cannot masquerade as structural fact.
  • New verdicts supersede old ones instead of overwriting — the chain stays readable.
  • Snapshot: ~19k fixes, ~12k best_fix verdicts live.
flipped → retract
5stage 3 · one consolidation draft
Stage 3 · Draft one consolidation comment never posts · mermaid map
  • One LLM call writes the consolidation comment as a triage_draft; the draft unit never posts.
  • Drafts carry a mermaid graph: duplicate families as undirected subgraphs, best-fix edge labels, every complex issue, clickable nodes, top-N plus "+K more".
  • Draft validation includes the English-only gate — one German signal word fails the draft.
  • Unverified relations say "unverified" — no silent downgrade to a plausible-sounding default.
doneerrorre-entry only on stand moverestalestand moved → re-enqueue
LLM Backend gpt-5.6-sol · tool-less injection canary at start
  • Pluggable backend layer; production path is Codex through the Hermes runtime, invoked tool-less.
  • A toolset-inertness canary runs before the first real call of every process — the review path is provably static.
  • The default backend is a fixture (test double): every production entry point must pin the env, a documented operational trap.
  • The context window figure is measured live from the runtime, not taken from training data.
Prior Dossier own posts fed back structured delta re-assess on moved stand
  • When a posted complex moves, re-assessment gets a structured dossier: own past comments byte-exact, active and superseded verdicts, assessment notes — all source-attributed.
  • The dossier is labeled as "earlier OWN LLM judgment, to re-validate — not fact".
  • Echo protection: the own post reaches the prompt ONLY through this block, never through discussion lines.
  • Output is a verdict supersede plus a followup draft; the block caps at 40 rendered verdicts with the newest own post always complete.
prior dossier · delta re-assess → followup draft
Retraction flipped verdicts withdrawn one txn per complex
  • If a new verify round contradicts a posted verdict (fixes→unrelated, best-fix gone, duplicate resolved), the old claim is withdrawn in the books.
  • Detection compares active old edges against the new verdict set; each complex settles in a single transaction.
  • Invariant: no refuted posted claim is left standing silently.
Error Isolation one failure = one run re-entry only on stand move
  • A failing complex marks only its own queue row as error; the slot and its siblings continue.
  • Errored rows are excluded from re-enqueue until the complex state moves — no burn loops on permanently broken inputs.
  • The exclusion was built after a red proof showed repeat offenders doubling overnight.

Curation

refute before you post

19 deterministic checks a–i · no LLM
Stage 1 · Checks a–i 9 deterministic facts vs DB fail ⇒ no LLM spent
  • Nine fact checks against the primary source: PR mentions, state tags, counts, target open+member, mermaid consistency, footer, English-only, verdict-edge basis, consolidation consistency.
  • Any stage-1 fail ends curation immediately — no LLM call is spent on a factually broken draft.
  • Backend-independent and cheap; fail signatures are directly usable as drain excludes.
  • Content quality (diff vs. root cause) is deliberately NOT this stage's job — that is the adversarial stage.
cleanfact fail
2stage 2 · adversarial refutation
Stage 2 · Refutation "try to REFUTE this draft" fail-closed on LLM error
  • The LLM is instructed to refute the draft against the evidence; a pass is required, and it never replaces stage 1.
  • Fail-closed: LLM errors, timeouts and parse failures all yield fail with their own reason class — there is no bypass switch in the drain.
  • Injection hardening: instructions live outside; all GitHub/draft content is neutralized inside a nonce-guarded block.
  • The working set is hydrated fresh, including review verdicts.
pass→ pass poolrefutedcuration-errorLLM error ⇒ fail-closed
exactly one repair re-draft
Curation Slots (:22/:52) pre-curate the pool · ×6 in-drain path · same gates
  • The pre-curation slot works ahead: uncurated drafts run both stages with parallel LLM calls (batch 150, parallelism 20), so the drain mostly draws from a pass pool.
  • Second entry path: whatever reaches the drain uncurated runs the same run_curation inline — not a single gate changes between paths.
  • Failed candidates are steered out and journaled, never posted.
  • Snapshot example: 30 candidates → 17 pass / 13 fail / 14 repaired in one slot.
Repair Loop exactly one re-draft on fail burst unit drains backlog
  • After an LLM fail with substantive reasons, the reasons go back into the prompt for EXACTLY ONE re-draft, followed by a final re-curation.
  • The one-repair maximum is structural (selection is uncurated-only), not a counter.
  • A transient systemd burst unit exists to drain the historical fail backlog through the same curate_one path, batch-journaled with its own kill switch — fail drafts have no other re-entry.
  • First-pass repairs carry their weight: measured batches pass mostly without any repair at all.
English-Only Gate signal-word regex · no LLM enforced twice
  • All GitHub-bound text is strictly English, enforced deterministically by a signal-word regex.
  • The gate runs twice: at draft validation and again as curation check g.
  • The curation prompt itself is English — the whole outward-facing path speaks one language.

Execution

the vollzug layer — guarded posting

1issues · pass-pool first · slot limit 60
Auto-Drain (every minute) posts curated drafts, capped followups first · limit 30
  • The drain selects candidates issue-first and pass-pool-first, newest-first, up to the slot limit (60), and walks each through refresh → arm → guardrail chain → gh post → mark posted; a draft that failed a post attempt re-enters only after an exponential per-draft backoff (2·2ⁿ⁻¹ min, capped at 60).
  • The hourly cap is owned by the post governor — an AIMD state machine inside the pace tick: +25/h per clean window, ×0.6 on gh errors, a 6/h probe cadence while blocked, ceiling 480/h just under GitHub's content limit.
  • Steered-out branches (skipped_stale, curation fails, the blocked-* family) are all journaled per run.
  • Three start paths exist: the minute timer, the pace dispatcher, and the Hermes trigger layer — see Clockwork.
2one GraphQL call · all members
Freshness Gate live refresh before every post age ≤ 10 min, server-checked
  • Immediately before a post, ONE GraphQL call refreshes state, updated_at and head_oid of every complex member, last-write-wins, with a logged proof line.
  • The server then enforces a maximum refresh age (10 min policy) — a stale refresh means blocked-norefresh, not a hopeful post.
  • A failed refresh skips the draft fail-closed; state moved between claim and post counts as skipped_stale.
  • Origin is a verbatim user directive: quality before quantity.
freshrefresh error → skipskipped_stale
3arm_machine_run · provenance
Standing Order grade · caps · provenance revoke = kill arm №1
  • Every post series runs under an armed standing order carrying an autonomy grade and caps (steady and backfill).
  • Arming records trigger provenance: cron, event or manual — not hard-coded.
  • The active order at snapshot: grade 1; the daily steady cap is effectively unbounded since the hourly authority moved to the post governor (2026-08-03). Revoking the order is the first arm of the kill-switch triad.
  • Quota caps always follow the ACTIVE order — a lesson from a false cap alarm.
4vollzug_post_comment · 8-step chain
Guardrail Chain server-side · cheap → expensive grade → header → author → open → reply → fresh → dedup → quota
  • vollzug_post_comment (SECURITY DEFINER) walks a fixed order: grade gate, AI-disclosure header, author filter, target-open check, reply rules, freshness age, dedup claim, atomic quota reservation.
  • The function never does network I/O — the actual gh write is the module's job, after claimed.
  • Quota reservation is atomic across minute/hour/day windows and fail-closed: a reservation error counts as "cap reached".
  • Atomicity is probe-proven: three parallel posters against cap 2 yield exactly 2.
  • Dry runs do not poison the dedup history.
claimedblocked-*every refusal persisted
Refusal Verdicts blocked-* family · ≥8 exits every refusal persisted
  • Every refusal is a named verdict, never an exception: dry-run, blocked-header, blocked-target-closed, blocked-reply-disabled, blocked-reply-cap, blocked-reply-cooldown, blocked-norefresh, blocked-dedup, blocked-cap.
  • Each one writes a post_attempt_log row plus an immutable audit line — refusals are data, not silence.
  • The dedup claim uses INSERT … ON CONFLICT semantics: no row claimed means someone already posted here.
  • Completion is symmetric: vollzug_mark_posted or vollzug_mark_error, and errors are reclaimable.
5gh api · issue comments (review path retired 2026-08-03)
PR Reviews retired 2026-08-03 · PRs post as comments one policy row brings it back
  • Retired 2026-08-03 (post.pr_as_review=false): GitHub throttles review creation separately from issue comments (HTTP 422, only ~15% of reviews passed), so PR targets now post as ordinary issue comments through the freely flowing channel.
  • While active, the review event was mapped structurally from graph facts: APPROVE only if the target IS the complex's active best-fix edge; otherwise COMMENT. LLM text never chooses the event.
  • REQUEST_CHANGES is deliberately unused — there is no structural signal for it.
  • An echo guard filters the bot's own reviews out of its inputs.
6vollzug_mark_posted | vollzug_mark_error
Quota & Audit atomic rate windows append-only audit log
  • Three books under the chain: post_quota (atomic minute/hour/day reservations), post_attempt_log (every decision), vollzug_audit_log (append-only; delete and update are denied).
  • Reservation failure equals "cap reached" — fail-closed by construction.
  • Redaction patterns applied to journal details are shared with the alarm system.
postedreclaimable
review_issue_open · answer poll · byte-exact author
Human Review Issues external actions need a human byte-exact author check
  • Any action touching external resources requires a review issue in the operator's repo with answer options; only the authorized login arms the action.
  • The author check is byte-exact against config; a stranger's comment does not block a later authorized approval.
  • The whole external-resources path defaults to disabled (fail-closed).
armedrejecteddisabled (fail-closed)
Execution Module runs inside Hermes hg_vollzug: no table DML
  • The posting layer (code name vollzug) runs inside the host-unbounded Hermes instance against its container gh auth — no separate broker process.
  • Its database role has NO table DML: every mutation goes through SECURITY DEFINER functions; the denial is probe-proven per statement class.
  • Credentials resolve ONLY from the environment or the home .env — under cron, absent secrets resolve to None instead of a fallback path.
  • Adjacent container infrastructure (update deadman, approval broker) belongs to the Hermes instance, not to this pipeline.
Credential Custody token lives in hosts.yml only static gh binary
  • One token carrier: the container's gh auth store; the account is the operator's, the binary an official static release.
  • The token never leaves the auth store toward the database or any prompt.
  • The custody chain is documented and audited as part of the credential-path review.
Follow-up Replies new comment, never edit ≤2 per thread · 60 min cooldown
  • New events produce a NEW comment — no edits, so thread history stays honest.
  • Strictly gated: reply flag must be exactly true, a per-thread cap (2) and a cooldown (60 min) apply, plus substance rules.
  • Grounding: of 11 pre-rule follow-ups, 0 carried substantive human signal — the rules exist because the data said so.
  • Bot/AI-authored events alone never trigger a reply.
Poll Sweep (:47) open drafts re-walk the chain
  • A periodic sweep pushes open drafts through the exact same guardrail chain — no duplicated logic, it calls the module's functions.
  • Runs hourly in the Hermes/host path with its own journal cycle (vollzug_poll).
  • The hourly cadence is config data (vollzug.poll_interval_min = 60), not code — retiming the sweep is an UPDATE, not a deploy.
Trigger Layer read-only backlog probe fires the host drain unit
  • Inside the Hermes container, a strictly read-only role checks: postable backlog present AND last post older than the minimum gap.
  • On fire it starts the host drain unit — the write path stays on the host, the container only signals.
  • Double-run and backstop behavior are probe-covered.

posted comments flow back through Harvest ② — the system re-reads its own words only as a structured prior (Triage, prior dossier).

Clockwork

timers are the backstop — pace pulls work forward

systemd Clockwork 10 timers · collision-free raster + 1 compose loop
  • The minute raster: pace :01/5 · drain every minute · expansion :04/5 · ref-backfill :06/10 · diff-gap :08/10 · mentions :09/:39 · tokscale :12 · harvest continuous (PRs 60 s, issues 2 min) · curation :22/:52 · rebuild :33 · triage :37 · embed :43 · poll :47 · meta-feedback :57 · nightly 03:03 · report 06:53.
  • Every slot script documents its raster position; unit copies live in the repo, and /etc↔repo parity is a session close gate.
  • Oneshot lesson: a running oneshot reports ActiveState=activating the whole time — health checks must read ActiveState, is-active deceives.
Pace Dispatcher (:01/5) backlog signals pull work early timers stay the backstop
  • Every five minutes, four backlog signals (dirty nodes → rebuild, queued → triage, uncurated → curation, pass pool → drain) may start a slot EARLY; the timers remain the guaranteed floor.
  • The post governor (an AIMD state machine inside the pace tick, 2026-08-03) sets the posting caps itself: +25/h per clean 10-minute window, ×0.6 on gh errors, 6/h probe cadence while fully blocked, ceiling 480/h. Its error signal counts the issue channel only, and a window that was clean but quota-capped still counts as growth — self-throttling is not "no traffic".
  • A conflict matrix keeps mutually exclusive units apart (rebuild ⊥ triage ⊥ nightly pass); drain and curation run independently.
  • Minimum gaps per cycle are computed against the run journal; starts are non-blocking; busy detection reads ActiveState.
  • Fail-closed behind its own flag; stage 2 (repo activity as a signal) is deliberately a separate, open step.
Kill Switches 3 arms + 9 fail-closed flags
  • The triad: revoke the standing order · disable triage · drop the cap — three independent arms.
  • Nine enabled/gate flags in ops.config, one per cycle family; every slot script checks its flag BEFORE working and skips loudly.
  • Any value other than true means off — fail-closed, never fail-open.
Run Journal one row per cycle run counts · tokens · cost · git_sha
  • Every cycle execution writes one row: watermarks before/after, counters, tokens, cost, config snapshot, git SHA.
  • The git SHA makes deploy drift visible per run; status is ok/partial/failed.
  • The journal feeds everything downstream: alarm detection, pace gaps, the token ledger, the daily report and the dashboard chronicle.
Token Ledger Sync (:12) real usage → 1 session/day
  • Transfers real Codex usage from the journal onto a single ledger session per UTC day — instead of ~3,000 session entries.
  • Uses a byte-exact usage-file chain; a sync watermark tracks progress.
  • Exists because the incognito one-shot runtime no longer writes sessions the ledger could read.
Codex Re-Auth device-flow watchdog · 5 min
  • A watchdog detects exhausted Codex credentials, starts the device flow and messages code plus URL to the operator.
  • The only human step is entering the code; successful logins clean up exhausted credentials.
  • A file lock prevents double runs.
Nightly Backup graph dump · 3-day retention embeddings excluded (regen)
  • The graph database is dumped nightly within the general backup, with a shorter 3-day retention due to disk pressure.
  • Embedding DATA is excluded — hundreds of MB that are cheaply regenerable.
  • The restore path is a written runbook, not tribal knowledge.
OnCalendar raster · the backstop — starts the Derivation, Triage, Curation and Execution slots
enabledoff → skip (logged)
every run journaled → feeds Observe · conflict matrix: rebuild ⊥ triage ⊥ nightly

Observe

journal → alarms → dashboard

1journal → detect · dedup · auto-resolve
Alarm System one open alarm per class auto-resolve on detect
  • Alarm state is deduplicated: at most one open alarm per class, enforced by a partial unique index.
  • Classes cover heartbeat, staleness, watermark lag, failed runs and disk; state alarms auto-resolve when the detector sees green.
  • Staleness is throughput-relative: stale means older than a factor of the measured full-drain time — a fixed SLA was permanently red at this scale.
  • Alarm details pass the same redaction patterns as the journal.
206:53 · secret-grep fail-closed
Daily Report (06:53) ops truth → context store secret-grep fail-closed
  • Once a day the operational truth of the last 24 h is condensed into a text block in the knowledge store.
  • A secret grep runs over the ENTIRE text fail-closed — any hit aborts the save; nothing is ever masked-and-stored.
  • Stdlib only; the odd 06:53 slot dodges the fleet's timer synchronization spike.
3prices × usage per journal row
Cost Ledger prices × usage per run inflow-limited · ~19 drafts/h
  • Model prices live in an ops table; every journal row carries its cost; the dashboard aggregates today and total.
  • The same journal rows drive the public dashboard's cost tiles — the running totals live there, deliberately not in this static document.
  • The measured bottleneck is NOT capacity: the system is inflow-limited at roughly 19 postable drafts per hour — no internal knob is binding.
4read-only poll 10 s → SSE
Live Dashboard SSE · 10 s · vase-painting UI hermes-triage.gottz.de
  • A public Go dashboard, one embedded self-contained HTML file, updating every 10 s over SSE — black-figure in dark mode, red-figure in light.
  • The status event carries queue, throughput, postable drafts, repair queue, last slots, alarms, watermarks, cost and tokens.
  • The amphora medallion renders any complex: PR = diamond, issue = circle, open = filled, closed = hollow, root ring gold, edges colored by source — everything clickable.
  • Reads through a dedicated read-only role whose grants are probed after every rebuild swap.
  • This artifact borrows its visual language from that dashboard.

Foundation

the plinth every stage stands on

Graph Database PostgreSQL + pgvector 45 + 6 ops tables
  • A dedicated Postgres database (in a TimescaleDB container, plain tables) with pgvector, reached over TCP with per-role credentials.
  • 45 public tables plus 6 ops tables at snapshot.
  • Migrations run in a single transaction with checksum tracking — drift aborts.
Least-Privilege Roles 8 roles · one per layer writes via SECURITY DEFINER
  • One minimal role per pipeline layer: owner (DDL only), harvest, derive, triage, execution, trigger, operator, dashboard.
  • The execution role has no table DML at all; the trigger role is read-only by grants; the dashboard role defaults to read-only transactions.
  • Sensitive mutations everywhere go through SECURITY DEFINER functions, not grants.
Migration Rig plain psql · sha256-tracked single txn · drift = abort
  • A plain psql runner applies SQL files lexicographically, records version plus sha256, and stops on any error.
  • Only the owner role runs DDL; every run prints its active parameters.
  • Every migration file opens with its red finding and rationale as a header comment.
Node Tables PRs · issues · comments commits · files · reviews
  • The GitHub objects as tables; snapshot: ~55k PRs, ~18k issues, ~102k comments.
  • GraphQL global node IDs are the schema seam for comments and reviews.
  • Nodes removed on GitHub get a NOT_FOUND marker instead of eternal fetch errors; PR diffs live in their own cache table.
Edge Tables structural facts ≠ derived llm edges via one function
  • Structural facts (closes, references, commit/file links) and derived relations live in separate tables.
  • LLM verdicts enter exclusively through write_llm_edge() — the source column cannot lie.
  • Set replacement happens only through resync functions; edge source is also the color key of the dashboard medallion.
Complex Tables shadow staging + rename swap identity survives rebuilds
  • Complexes and their members are rebuilt into staging tables and swapped in atomically by RENAME.
  • Member-stable complexes keep their identity across rebuilds (earlier, every run re-stamped fresh IDs).
  • Two historic swap incident classes (OID-bound views, OID-bound grants) are structurally closed by post-swap guards.
Posting Ledger claimed → posted | error dedup (kind, number, author)
  • Every attempted post is a ledger row with a claimed→posted|error state machine.
  • The dedup anchor is (target kind, target number, author) — claiming happens by conditional upsert; no row means someone already posted.
  • GitHub comment/review IDs are backfilled by the comment harvest; posted_stand freezes the complex state at post time.
Embedding Store 1024-d vectors · norm-checked
  • Vectors are stored with a norm CHECK (0.99–1.01) — an unnormalized vector cannot enter.
  • A text hash makes the pending selection idempotent; ANN neighbors are same-kind by construction.
  • Backups exclude the vector data: regenerable for cents.
Policy Thresholds 14 knobs as data, not code
  • All scoring and folding thresholds are table rows — changing one is an UPDATE, not a deploy.
  • The 14 knobs include ANN k, fold thresholds, fanout caps, component-size limits and the freshness age.
  • Every derivation run prints the active snapshot; operational mutations run as single statements with a separate verify.
Ops Schema journal · config · alarms · prices
  • Operational truth is separated from graph truth in its own schema: run journal, run events, config, alarms, alarm watermarks, model prices.
  • The config table carries all kill switches and operating parameters (38 keys at snapshot).
  • Consumers mutate alarms only through definer functions.
Post-Swap Guards view re-bind + grant probe drift fails the unit, loudly
  • Views and grants bind to table OIDs — a rename swap once left both pointing at the old generation (a stale-read every second hour).
  • Guard 1 re-binds all dependent views after the swap and verifies view freshness equals table freshness.
  • Guard 2 probes a SELECT as the dashboard role against the live side.
  • Any drift fails the rebuild unit loudly instead of serving a silent stale hour.

Cross-cutting safeguards

Injection Hardening neutralize · nonce · canary
  • Hostile content flow is mapped end to end; three defenses: role/turn markers are broken, untrusted content sits in a content-secret nonce block, and an injection canary proves the backend inert.
  • The guardrail chain is body-blind — LLM text never reaches a scope decision; PR review events map from graph facts only.
  • The hardening was a mandatory gate before grade-1 autonomy.
Probe Suites ~30 red-first gates
  • About 30 standalone probe modules across all layers; every build wave is proven RED first, then green.
  • Probe fixtures never anchor on live data (a learned rule); close-out sweeps count their gates explicitly (e.g. 8/8, 25/25).
  • One suite enforces its own maintenance-window guard and refuses to run against a live flag without an explicit override.
Governance Corpus queue · boards · warnings
  • Build and operations decisions are versioned truth: a work queue with state, two decision boards, amendments, and a documented rule that re-deciding happens BEFORE a wave is built.
  • User directives are quoted verbatim and bind gates; stop conditions are fixed (a red gate without a plan-conforming fix stops the loop).
  • An RLHF-warnings framework (22 axes) is applied as working discipline across sessions.