torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

feat: docs/ carries the operator's how-to guides, a configuration reference and the architecture and telemetry explanations carved out of README.md, INSTALL.md, INSTALL-MCP.md and EVALS.md; the last two retired (#190)
CI / verify (push) Successful in 6m17s
Details
CI / pwsh (push) Successful in 1m59s
Details
Release / release (push) Successful in 35s
Details

Browse Source 
Files changed:
- .gitea/workflows/ci.yml
- .gitignore
- .vibe/hooks.toml
- AGENTS.md
- CHANGES.md
- EVALS.md
- INSTALL-MCP.md
- INSTALL.md
- README.md
- VERSION
- docs/dev/development.md
- docs/explanation/README.md
- docs/explanation/architecture.md
- docs/explanation/language-boundaries.md
- docs/explanation/telemetry-and-evaluation.md
- docs/how-to/README.md
- docs/how-to/browse-with-obsidian.md
- docs/how-to/capture-repository-docs.md
- docs/how-to/export-guidelines.md
- docs/how-to/ingest-a-source.md
- docs/how-to/maintain-the-wiki.md
- docs/how-to/query-the-wiki.md
- docs/how-to/run-the-weekly-review.md
- docs/how-to/set-up-the-mcp-server.md
- docs/how-to/trace-and-score-sessions.md
- docs/how-to/troubleshoot.md
- docs/how-to/update-an-instance.md
- docs/reference/README.md
- docs/reference/commands/distribution-and-versioning.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/commands/telemetry.md
- docs/reference/commands/types-instructions-and-docs.md
- docs/reference/configuration.md
- instructions/dev/dev-setup.md
- instructions/dev/doc-pull-through.md
- instructions/dev/stack-build/SKILL.md
- instructions/dev/testing-conventions.md
- instructions/dev/version-parts.md
- instructions/setup-instance.md
- instructions/upgrade-instance.md
- reports/CONTRACT.md
- tools/.coveragerc
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/eval_cmd.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/commands/upload_cmd.py
- tools/chemenu/commands/version_cmd.py
- tools/chemenu/evals/__init__.py
- tools/chemenu/evals/scorecard.py
- tools/chemenu/evals/trajectory.py
- tools/chemenu/telemetry/policy.py
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_git_publish.py
- tools/chemenu/tests/test_instructions_shell.py
- tools/chemenu/tests/test_pipeline_l0.py
- tools/chemenu/toc.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-07 05:13:56 +02:00
1 parent cff7df4ffa
commit f9056c04f7
61 files changed
+2169 -1772
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/explanation/telemetry-and-evaluation.md
+354
View File
@@ -0,0 +1,354 @@
# Telemetry and evaluation
How the stack observes what an agent did, and how that record turns into a score. This page explains
the design and points at the code. The rules an agent follows live in
[tools/CONTRACT.md](../../tools/CONTRACT.md) and [reports/CONTRACT.md](../../reports/CONTRACT.md).
Turning tracing on and reading a session back is a how-to:
[trace-and-score-sessions.md](../how-to/trace-and-score-sessions.md).
<!-- wikitool:toc -->
## Contents
- [Why, beyond the unit tests](#why-beyond-the-unit-tests)
- [Architecture](#architecture)
- [The trace](#the-trace)
- [Whether it runs at all](#whether-it-runs-at-all)
- [What never reaches a trace](#what-never-reaches-a-trace)
- [Evaluation levels](#evaluation-levels)
- [L0 lives in the test suite, not in a separate harness](#l0-lives-in-the-test-suite-not-in-a-separate-harness)
- [Scoring a session](#scoring-a-session)
- [A worked example, and where the check moved to](#a-worked-example-and-where-the-check-moved-to)
- [Status](#status)
- [The agent runner, and why it is not here yet](#the-agent-runner-and-why-it-is-not-here-yet)
<!-- /wikitool:toc -->
## Why, beyond the unit tests
The test suite that ships with the origin repository checks the **compiler**: given this input, does
`wikitool` produce that output? It says nothing about the two things that actually go wrong in
practice: whether the *agent* followed the contracts, and whether the pages it wrote are any good.
Those need a different kind of check. Two notes in the
[Commonplace](https://github.com/zby/commonplace) research base, which the origin repository
vendors, supply the theory:
- *oracle-strength-spectrum* - correctness checks form a gradient, from hard (deterministic) to none
(vibes). The engineering move is to *harden* oracles step by step, not to reach straight for a
judge.
- *evaluation-automation-is-phase-gated-by-comprehension* - comprehension, then specification,
then generalisation. A judge built before anyone has read real failures optimises a proxy.
That ordering is why this page describes a lot of telemetry and only a little scoring. Reading real
traces is the first phase, and it cannot be skipped.
## Architecture
Three sources, three different jobs.
```mermaid
flowchart TD
subgraph runner["Eval runner (planned)"]
R["isolated HOME · programmatic mode<br/>NDJSON capture · run manifest"]
end
subgraph hooks["Harness hooks (interactive work)"]
H1["Claude Code<br/>.claude/settings.json"]
H2["Copilot CLI<br/>.github/hooks/*.json"]
H3["Mistral Vibe<br/>.vibe/hooks.toml"]
H4["VS Code Chat<br/>chronicle SQLite, post hoc"]
end
subgraph inner["Repo layer (always on)"]
W["wikitool emitter<br/>+ git"]
end
R --> T[("reports/telemetry/&lt;session&gt;/trace.jsonl")]
H1 & H2 & H3 & H4 --> I["tools/trace_ingest.py"] --> T
W --> T
T --> S["scorers L0-L4"] --> O[("reports/evals/&lt;date&gt;/")]
```
- **The repo layer is the truth.** `wikitool` records its own calls, so what happened *to the wiki*
is known even when no hook fired and no runner was involved.
- **The runner owns session boundaries.** Not every harness reports a session start - Mistral Vibe
has no such hook - so the process that launches the agent is what brackets a run.
- **Hooks enrich.** They add the tool calls the repo layer cannot see: file reads, greps, shell
commands, prompts.
Everything joins on one session id, which every source that has to pick one resolves the same way
(`chemenu.session`). The chain is `WIKITOOL_SESSION_ID`, then a harness's own session variable where
one is registered (`chemenu.session.HARNESS_ENV_VARS` - Claude Code's `CLAUDE_CODE_SESSION_ID` today),
then the parent process id. The middle step exists because the last one does not survive a harness
that runs every tool call in its own freshly started shell. There, `os.getppid()` is a new "session"
for every call, and neither the join nor the Iteration Budget Gate could see more than one or two
calls of a real run. A harness only earns an entry in that chain once a live session has been
observed setting the variable, and it has been confirmed to be the exact id the harness's own hooks
write elsewhere in a trace. A name that merely looks plausible would mis-key a session more quietly
than the pid fallback it replaced.
A trace hook that only *observes* tool calls (a `PreToolUse`/`PostToolUse`-style wiring) does not by
itself fix a harness whose events carry a different id than `wikitool`'s own emitter. The two would
still not join. Wiring such a hook is worth doing only once this fallback chain already keys both
sides on the same id.
## The trace
One JSON object per line, appended to `reports/telemetry/<session>/trace.jsonl`. The contract is
[tools/chemenu/telemetry/schema.py](../../tools/chemenu/telemetry/schema.py).
| Field | Meaning |
|---|---|
| `v` | Schema version |
| `ts` | ISO-8601 UTC, microsecond precision |
| `session_id` | The join key - `chemenu.session`'s fallback chain: `WIKITOOL_SESSION_ID`, else a registered harness variable, else the parent process id |
| `pid`, `seq` | `seq` counts **within one process**. Sort a trace by `(ts, pid, seq)` |
| `source` | `wikitool`, `runner`, or a harness name |
| `event` | See below |
| `attrs` | Normalised payload - the same field names whichever harness produced it |
| `run_key` | The `work/<runkey>/` run, when one is open |
| `trace_id`, `span_id` | From `TRACEPARENT`, when the harness exports it |
| `redactions` | Which secret patterns fired on this event |
**Events.** The core - `tool.pre`, `tool.post`, `wikitool.call`, `gate.refused` - is available on
every surface. Everything else is optional: `session.start`, `session.end`, `session.error`,
`prompt.submitted`, `assistant.message`, `turn.end`, `tool.error`, `instructions.loaded`,
`subagent.start`, `subagent.stop`, `compaction`, `page.written`, `publish.commit`, `budget.state` and
`gate.cleared`.
**The degradation rule.** No scorer may *require* an optional event. Claude Code has about thirty
hooks and Mistral Vibe has three. A scorer built on the rich end would silently report zero on the
poor end, which reads as "the agent did nothing" rather than "this harness cannot say". Every
`session.start` carries a `completeness` list naming the classes its harness can emit, so a scorer
can answer "not measurable here" instead. What each harness can report is tabled in
[trace-and-score-sessions.md](../how-to/trace-and-score-sessions.md#what-each-harness-can-report).
**Budget and trace are not the same set.** The Iteration Budget Gate exempts read-only retrieval,
and the trace records it. What an agent looked at before acting is exactly what a trajectory scorer
needs, and charging for a `search` would discourage the one habit that lowers cost.
## Whether it runs at all
The default depends on how the tree got here, not on one hard-coded switch.
[tools/chemenu/telemetry/policy.py](../../tools/chemenu/telemetry/policy.py) is the one place that
resolves it, so `wikitool doctor`, the writer and the MCP server's start-up guard all answer the
same question the same way:
| Installation form | Default | Marker |
|---|---|---|
| Git clone of the origin repository (dev checkout) | **on** (opt-out) | No `.wikitool-release.json` |
| An instance installed from a release, and every clone of its own repository | **off** (opt-in) | `.wikitool-release.json` present |
The form is read off `.wikitool-release.json`, the same stamp `version check`, `dist upgrade` and
`version notes` already use to tell a distribution from the repository it came from. Present means
an operator never asked for telemetry. Absent means this is the dev checkout the stack ships from,
where the traces are the stack's own measuring instrument. An instance commits the stamp with its
first `publish`, so a clone of it on a second machine carries the stamp and defaults to off too. It
is a consuming instance, not a measuring stand.
**Turning it on for a distributed instance** takes a per-checkout `.wikitool-telemetry.json` at the
root, gitignored like `.wikitool-remotes.json` and for the same reason. The consent to write
prompts in cleartext to *this* disk belongs to the checkout, not the corpus, so a second clone must
not inherit it silently. The setup instruction asks about it; nothing writes it automatically. Its
shape is in the [configuration reference](../reference/configuration.md#telemetry).
**Two independent quantity caps, both enforced fail-silent in `emit()`.** Neither is enforced in
`write_event()`, which the test suite calls directly to exercise the format without the policy
around it.
- **A byte cap per session trace** (default 5 MiB), checked with one `stat` before every append.
Once a trace is at or over the cap, further calls in that session write nothing except a single
`telemetry.limit` event. It is elected by the same single-writer trick `session.start` uses, an
exclusive create on a `.limit` marker file, so a trace that was cut off can be told apart from one
whose writer simply crashed.
- **Retention by session count** (default 250), applied once, right before a brand-new session
directory is created. It is never applied per event, and never against the session doing the
creating. It deletes exactly `trace.jsonl` and `.limit` from the oldest directories beyond the cap,
and `rmdir`s a directory only once it is empty. Nothing under `reports/` is ever removed in bulk.
The default of 250 was set above what the origin repository's own checkout had accumulated as of
2026-09-10: 231 session directories, well under 400 KiB in total. The cap bites on future growth,
not on the existing history.
## What never reaches a trace
Prompts and assistant replies **are** recorded in cleartext, locally. A failure taxonomy cannot be
read out of hashes, and building one is the first phase of any evaluation work. Four guards make
that defensible, all of them in
[tools/chemenu/telemetry/scrub.py](../../tools/chemenu/telemetry/scrub.py), so there is one place to
audit:
1. **Secret scrubbing.** Tokens, keys, `Authorization:` headers and `SECRET=` assignments are
replaced with `[REDACTED:<type>]`. The scrubbing is pattern-based, so it is best effort.
2. **A content cap.** 60 KiB per attribute, with a `[TRUNCATED n chars]` marker.
3. **A kill switch.** `WIKI_TRACE_CONTENT=0` keeps only `<field>_length` and `<field>_sha256`. The
digest is computed either way, so traces stay comparable.
4. **`raw/` contents never enter a trace at all.** That text is data, not instruction
([AGENTS.md](../../AGENTS.md) invariant 4), and a trace gets read back later. Callers record a path
and a digest.
Nothing leaves the machine. `reports/` is gitignored, no exporter is configured, and where a vendor
offers one it is off. Mistral's `enable_otel` ships prompts to Mistral Studio, so it stays `false`,
and so does `enable_telemetry`. Claude Code's and Copilot's content gates may only be enabled
against a collector you run yourself.
The variables that steer all of this are in the
[configuration reference](../reference/configuration.md#environment-variables).
## Evaluation levels
Ordered by oracle strength: hard checks first, judgment last.
| Level | Oracle | What it measures | Status |
|---|---|---|---|
| **L0** Pipeline | hard | A wiki the tools built themselves lints clean, and the catalog is a fixed point | **done** - in the origin repository's test suite |
| **L1** Artifact scorecard | hard | Counters from `lint`'s own checks: hard errors and advisories, page count | **done** |
| **L2** Trajectory | hard | Rules over the trace: was a refused call repeated, was a gate flag passed unearned, did a page change go unlogged | **done** |
| **L3** Task evals | medium | Gold set: question → expected cited pages; ingest fixture → expected page titles. Scored by set overlap | needs the runner |
| **L4** Rubric / judge | soft | Prose quality, cramming, tone | **out of scope** until a failure taxonomy exists |
Evaluation results are statistical, not binary. A case runs several times and reports a pass rate
with its variance, because sampling is not deterministic. A single failure is not a merge blocker;
a regression against a baseline is.
### L0 lives in the test suite, not in a separate harness
It runs the CLI in-process against an empty wiki. It asserts that `new` → write → `xref` →
`index rebuild` leaves a tree `lint` calls clean, and that rebuilding the catalog again changes
nothing on disk. That is a hard oracle over the compiler, which is what the test suite is for.
Giving it its own runner would have duplicated the suite for nothing.
One behaviour it pins is easy to mistake for a defect: **a scaffolded page does not lint clean**.
`new` writes placeholder wikilinks for the author to replace. So a page that was created but not yet
written reports broken links, and reports its `TODO`-only sections as the advisory
*Unfilled Template Sections*. That is the scaffold saying it is unfinished.
## Scoring a session
`tools/wikitool eval score` scores one session. It only reads, and it is exempt from the Iteration
Budget Gate: reading back what a session did is not iterating on the wiki, and charging for it
would discourage checking one's own work.
**L1** re-runs `lint`'s checks in-process and reports their counters. It shares its definition of a
hard error with `lint --fail-on-error` - one constant, `HARD_ERROR_KEYS` - so a run can never pass
its score while lint refuses it.
**L2** checks five rules. Each one restates an invariant the code cannot enforce in-process. A gate
can refuse a call, but nothing stops an agent from calling again with the gate's own flag. That gap
is the whole point:
| Rule | Invariant | Severity |
|---|---|---|
| `refusal-not-retried` | A refused call, repeated unchanged, is the loop the gate exists to break | error |
| `gate-not-self-opened` | `--yes`/`-y` no longer exist at all; `--override-budget` is for a human to pass after a refusal; `--force` never | error |
| `content-change-logged` | A publish that changes `kb/` pages needs a `log append` in the same session | advisory |
| `clearance-was-asked-for` | A `gate.cleared` token must match one some earlier `gate.refused` issued - catches an invented token, and one reused from a different changeset | error |
| `clearance-ended-the-turn` | No `wikitool.call` between a clearance request (exit 42) and the next `prompt.submitted` - skipped, not failed, on a harness that cannot report `prompt.submitted` | error |
New rules belong here only when a real trace shows a real failure. Checks invented from the contract
text produce a score that improves while behaviour does not - the failure mode the phase-gate note
describes. The first three rules were chosen because each guards a refusal an agent can talk its way
around. The last two carry the load for the Mass-Update Gate's clearance mechanism (2026-08-28),
which deliberately enforces very little in-process, as described below.
A rule can report a third state besides pass and fail: **skipped**, meaning this trace cannot answer
the question at all - for example `clearance-ended-the-turn` on a harness with no prompt hook.
Skipped reads as neither a pass nor a violation. `scorecard.render_markdown` marks it `skip`,
distinct from `ok`, `warn` and `FAIL`, and `score()["violations"]` never includes a skipped rule,
whatever its severity.
### A worked example, and where the check moved to
Scoring the session that shipped the telemetry layer itself used to report a failure:
```
FAIL gate-not-self-opened
call=publish --message "tools: session traces …" --yes, reason=no mass-update refusal preceded it
```
That finding was true and benign at the same time. The change was 17 files, the file list was shown,
and a human approved it. But the approval happened in conversation, and `--yes` went out on the
first attempt, so the gate never got to refuse anything. The documented path had the refusal first,
and the trace records tool calls, not conversations.
Three sessions did some version of this before the mechanism changed rather than the procedure. The
interesting part is *which* half changed. The obvious fix was built first: make the gate impossible
to open without independent proof of a human. That meant a ticket bound to the changeset, redeemed
only after a `prompt.submitted` event and a `wikitool approve` typed in a second terminal. It worked,
and it was unusable. On a harness that drives `wikitool` through a shell tool, the agent cannot
produce the turn signal, and the human is asked to leave the conversation to approve anything at all.
A gate nobody can pass is not a safe gate, it is an abandoned one.
So the enforcement got cheap and the *verification* got real:
- **In-process, `publish` enforces only what it can do honestly.** Exit 42 says "a human must see
this". `--confirm <token>` says "here is what they saw", and the token digests the file list, so it
cannot silently cover a different one. None of that proves a human was involved, and the code says
so in as many words.
- **The trace answers the question the tool cannot.** `clearance-ended-the-turn` looks for a
`wikitool.call` between an exit-42 call and the next `prompt.submitted`. That is exactly the
"refused, then confirmed in the same breath" shape all three incidents had, and it is invisible to
any single call. `clearance-was-asked-for` catches a token that no refusal ever issued.
This is the loop this whole page is about, run to its conclusion: a rule that only a trace could
check, a trace that showed it being broken three times, a mechanism rebuilt because no procedural fix
survived - and then the check itself moved to the layer that can actually see it.
**What this still cannot see:** an agent that receives the exit-42 output, does not relay it, and
passes the token anyway *after* the user happens to say something unrelated. `prompt.submitted`
proves a turn boundary, not that the output was relayed or that the reply meant yes. That gap is
recorded rather than papered over. Closing it needs the harness to report what the agent actually
said, which no adapter here does yet.
That gap stopped being hypothetical within the hour. The first agent to receive the new gate - the
one that had just written the paragraph above - answered the user with a file *count* and a pointer
to "the output above". On that harness the user could not see the output: a command's stdout goes
to the agent's context, not to anyone's screen. Nothing in the trace distinguishes that from a
correct relay, and nothing ever will. The response was to fix the half that *is* fixable, the
wording. The message now says "THE USER CANNOT SEE THIS OUTPUT", asks for the paths to be copied
into the reply, and names the near-misses that do not count: a count, a summary, "the output above".
An instruction that conflates printing with showing reads as already satisfied by the text existing.
That is a general lesson about writing for agents, not a detail of this gate.
## Status
| Piece | State |
|---|---|
| Event schema, scrubber, writer | done - `tools/chemenu/telemetry/` |
| `wikitool.call` on every command | done - one hook point in `cli.py`, next to the budget gate |
| `gate.refused` (loop-breaker, iteration budget, mass-update) | done |
| `publish.commit` | done |
| Hook entry point | done - `tools/trace_ingest.py`, mappings for all three hook-capable harnesses |
| Copilot CLI hooks | done - `.github/hooks/wiki-trace.json` |
| VS Code chronicle import | done - `tools/import_chronicle.py` |
| Mistral Vibe hooks | done - `.vibe/hooks.toml`, `.vibe/config.toml` |
| Session header seeded on every trace | done - a trace declares its own `completeness` even without a session hook |
| Scoring L1 + L2 | done - `wikitool eval score` |
| L0 pipeline check | done - in the test suite |
| Claude Code hooks | **partial** (2026-08-28) - `.claude/settings.json` wires `UserPromptSubmit` (the turn boundary `clearance-ended-the-turn` scores against) plus a `permissions.ask` rule on the `--confirm` form of `publish`; `PreToolUse` verified unable to force a prompt over `permissions.allow` |
| Claude Code OTel | not started; not yet verified against the live CLI |
| Agent runner (`eval run`) and L3 | designed below, not built |
## The agent runner, and why it is not here yet
Scoring reads what a session left behind. The missing half is *starting* one: a runner that launches
a harness against a task, several times, and reports a pass rate. What it has to do is no longer
guesswork, because the adapter work settled most of it:
- **Give each run a copy of the repository, not a fixture directory.** `config.ROOT` is derived from
`chemenu/config.py`'s own location, so `wikitool` cannot be aimed at another tree from outside. An
agent case therefore needs a worktree or a clone, where the tool sits inside the tree it edits. (A
*fixture* directory works fine in-process, which is why L0 is a test.)
- **Give each run its own HOME:** `VIBE_HOME`, `COPILOT_HOME`, plus `WIKI_TRACE_DIR` and
`WIKITOOL_SESSION_ID`, so that state, trust decisions and traces cannot leak between runs.
- **Never launch Vibe without `--agent`.** Its programmatic mode falls back to `auto-approve` when no
agent is named, and it does not ask about folder trust. The runner must refuse rather than inherit
that default.
- **Record a run manifest:** harness and version, model, git SHA, and hashes of `AGENTS.md`, the
published skills and the fixture. Without it, a pass rate cannot be compared to anything.
What is missing is not the design but the ability to check it. No provider credentials were
configured on the machine the design was worked out on, so a live run could not be executed, let
alone verified. Writing the runner anyway would repeat exactly the mistake the Vibe adapter avoided:
the hook format there was wrong in the documentation, and only the installed CLI showed it. A runner
written against an unverified mental model of three harnesses would be worse.
Open questions: whether Claude Code has a redirectable config directory (only `--settings` is
documented), and whether the scrubber's own patterns are enough, or a dedicated secret scanner
belongs in the verification step.