feat: docs/ carries the operator's how-to guides, a configuration reference and the architecture and telemetry explanations carved out of README.md, INSTALL.md, INSTALL-MCP.md and EVALS.md; the last two retired (#190)
Files changed: - .gitea/workflows/ci.yml - .gitignore - .vibe/hooks.toml - AGENTS.md - CHANGES.md - EVALS.md - INSTALL-MCP.md - INSTALL.md - README.md - VERSION - docs/dev/development.md - docs/explanation/README.md - docs/explanation/architecture.md - docs/explanation/language-boundaries.md - docs/explanation/telemetry-and-evaluation.md - docs/how-to/README.md - docs/how-to/browse-with-obsidian.md - docs/how-to/capture-repository-docs.md - docs/how-to/export-guidelines.md - docs/how-to/ingest-a-source.md - docs/how-to/maintain-the-wiki.md - docs/how-to/query-the-wiki.md - docs/how-to/run-the-weekly-review.md - docs/how-to/set-up-the-mcp-server.md - docs/how-to/trace-and-score-sessions.md - docs/how-to/troubleshoot.md - docs/how-to/update-an-instance.md - docs/reference/README.md - docs/reference/commands/distribution-and-versioning.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/commands/telemetry.md - docs/reference/commands/types-instructions-and-docs.md - docs/reference/configuration.md - instructions/dev/dev-setup.md - instructions/dev/doc-pull-through.md - instructions/dev/stack-build/SKILL.md - instructions/dev/testing-conventions.md - instructions/dev/version-parts.md - instructions/setup-instance.md - instructions/upgrade-instance.md - reports/CONTRACT.md - tools/.coveragerc - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/dist_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/eval_cmd.py - tools/chemenu/commands/git_publish.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/commands/version_cmd.py - tools/chemenu/evals/__init__.py - tools/chemenu/evals/scorecard.py - tools/chemenu/evals/trajectory.py - tools/chemenu/telemetry/policy.py - tools/chemenu/tests/test_dist_cmd.py - tools/chemenu/tests/test_git_publish.py - tools/chemenu/tests/test_instructions_shell.py - tools/chemenu/tests/test_pipeline_l0.py - tools/chemenu/toc.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed
1 parent
cff7df4ffa
commit
f9056c04f7
61 files changed
+2169
-1772
No files matched your search
@@ -0,0 +1,176 @@
|
||||
# Trace and score agent sessions
|
||||
|
||||
Record what an agent did in a session, read the record back, and score it. A score covers the state
|
||||
the session left the wiki in and how it got there: whether it repeated a refused call, or passed a
|
||||
gate's flag without that gate ever refusing. Why this exists, what the trace contains, and what a
|
||||
score means are covered in [telemetry-and-evaluation.md](../explanation/telemetry-and-evaluation.md).
|
||||
|
||||
<!-- wikitool:toc -->
|
||||
## Contents
|
||||
|
||||
- [What you prepare](#what-you-prepare)
|
||||
- [Reading a session back](#reading-a-session-back)
|
||||
- [What each harness can report](#what-each-harness-can-report)
|
||||
- [Claude Code](#claude-code)
|
||||
- [Copilot CLI](#copilot-cli)
|
||||
- [VS Code Chat](#vs-code-chat)
|
||||
- [Mistral Vibe](#mistral-vibe)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
## What you prepare
|
||||
|
||||
**Turn tracing on, if it is off.** It is on in a clone of the origin repository and off in an
|
||||
instance installed from a release. `tools/wikitool doctor` reports which state this checkout is in,
|
||||
and why. To turn it on for one checkout, create `.wikitool-telemetry.json` at the root:
|
||||
|
||||
```json
|
||||
{ "enabled": true }
|
||||
```
|
||||
|
||||
The file and its other keys are in the [configuration reference](../reference/configuration.md#telemetry).
|
||||
Everything stays on this machine: traces land under `reports/telemetry/`, which is gitignored, and
|
||||
no exporter is configured.
|
||||
|
||||
**Scope the session to the task.** Every source of events joins on one session id. Setting it
|
||||
yourself keeps one task in one trace:
|
||||
|
||||
```bash
|
||||
export WIKITOOL_SESSION_ID="my-task"
|
||||
```
|
||||
|
||||
[instructions/session-setup.md](../../instructions/session-setup.md) has the line for each shell,
|
||||
and what to do on a harness that starts a fresh shell for every command.
|
||||
|
||||
## Reading a session back
|
||||
|
||||
```bash
|
||||
tools/wikitool eval sessions # which sessions have a trace
|
||||
|
||||
# what did that session do?
|
||||
jq -r '[.ts, .source, .event, (.attrs.command // .attrs.tool_name // "")] | @tsv' \
|
||||
reports/telemetry/my-task/trace.jsonl
|
||||
|
||||
# how well did it do it?
|
||||
tools/wikitool eval score --session my-task
|
||||
tools/wikitool eval score --session my-task --save # also write the scorecard under reports/evals/
|
||||
```
|
||||
|
||||
Both `eval` commands only read, and neither counts against the session's call budget: reading back
|
||||
what a session did is not iterating on the wiki. Their options are in the
|
||||
[telemetry command reference](../reference/commands/telemetry.md).
|
||||
|
||||
To see how a harness hook payload would be recorded, feed one in by hand without writing anything:
|
||||
|
||||
```bash
|
||||
echo '{"session_id":"x","hook_event_name":"post_tool","tool_name":"bash"}' \
|
||||
| tools/trace_ingest.py --source mistral-vibe --dry-run
|
||||
```
|
||||
|
||||
## What each harness can report
|
||||
|
||||
`wikitool` records its own calls on every harness. What the agent did *between* those calls -
|
||||
file reads, searches, shell commands, prompts - comes from the harness's hooks, and harnesses differ
|
||||
a lot in what they offer. Verified against vendor documentation on 2026-08-23:
|
||||
|
||||
| | Claude Code | Copilot CLI | VS Code Chat | Mistral Vibe |
|
||||
|---|---|---|---|---|
|
||||
| Hook events | ~30 | 14 | none | 3 (`pre_tool`, `post_tool`, `post_agent`) |
|
||||
| Session start/end hook | yes | yes | - | **no** |
|
||||
| Prompt submit hook | yes | yes | - | **no** |
|
||||
| Which instructions loaded | yes (`InstructionsLoaded`) | no | no | no |
|
||||
| Block / rewrite a tool call | yes | yes | - | yes |
|
||||
| OTel to your own collector | yes | via MDM `telemetry` | no | **no** - `enable_otel` targets Mistral Studio only |
|
||||
| `TRACEPARENT` to subprocesses | yes | undocumented | - | undocumented |
|
||||
| Programmatic mode | `-p`, `stream-json` | `-p` | no | `-p`, `--output streaming` |
|
||||
| Isolated config home | to be confirmed | `COPILOT_HOME` | no | `VIBE_HOME` |
|
||||
| Local session log | `~/.claude/projects/*.jsonl` (unstable format) | `~/.copilot/session-state/<id>/events.jsonl` | chronicle SQLite | `$VIBE_HOME/logs/` (no format guarantee) |
|
||||
| Wired up here | **partial** - `.claude/settings.json` (`UserPromptSubmit` + a `permissions.ask` rule on the clearing publish) | **yes** - `.github/hooks/wiki-trace.json` | **yes** - `tools/import_chronicle.py` | **yes** - `.vibe/hooks.toml` |
|
||||
|
||||
A rule that needs an event a harness cannot report is shown as *skipped*, not as passed or failed.
|
||||
|
||||
### Claude Code
|
||||
|
||||
`.claude/settings.json` wires `UserPromptSubmit` to `tools/trace_ingest.py`, through
|
||||
`tools/trace-hook` like every hook here. That gives the trace its turn boundaries, which the rule
|
||||
`clearance-ended-the-turn` needs. Without a `prompt.submitted` event it reports "cannot say"
|
||||
instead of a verdict.
|
||||
|
||||
**No `PreToolUse` decision hook is wired, and that is a finding, not an oversight.** Verified against
|
||||
the live CLI on 2026-08-27: a `PreToolUse` hook that returns
|
||||
`hookSpecificOutput.permissionDecision: "ask"` does **not** override a matching `permissions.allow`
|
||||
rule. Permissions are evaluated before the hook's decision, so a hook cannot force a confirmation
|
||||
prompt on a command that is already allowed. A `permissions.ask` rule *does* win, which is why
|
||||
`.claude/settings.json` carries one on the `--confirm` form of `publish`
|
||||
(`Bash(tools/wikitool publish --confirm:*)`). The clearing call prompts, and ordinary publishes below
|
||||
the threshold do not. It is a prefix match, so it depends on `--confirm` coming first, which is why
|
||||
`publish` always prints its re-run line in that order. Treat it as a useful second line, not a
|
||||
guarantee: an agent that reorders the flags gets around it.
|
||||
|
||||
### Copilot CLI
|
||||
|
||||
[.github/hooks/wiki-trace.json](../../.github/hooks/wiki-trace.json) is committed, so a clone
|
||||
brings its own tracing. Eleven events go to `tools/trace_ingest.py`. `disableAllHooks` in
|
||||
`.github/copilot/settings.json` turns them off without deleting anything. `userPromptSubmitted` is
|
||||
among them, so `clearance-ended-the-turn` can be scored on Copilot CLI with no change.
|
||||
|
||||
A `preToolUse` entry that forces a decision on the clearing call would first need Copilot's own
|
||||
decision-document schema verified against a live CLI. That has not happened, and an adapter that
|
||||
cannot be verified is not written.
|
||||
|
||||
Four details of the hook configuration matter. The first two hold for all three harnesses that have
|
||||
hooks:
|
||||
|
||||
- **The interpreter is the venv's, never the script's shebang.** Each `bash` command is
|
||||
`./tools/trace-hook ...`, which runs `trace_ingest.py` with `tools/.venv`'s Python in either venv
|
||||
layout. Each `powershell` command names `tools\.venv\Scripts\python.exe` directly. On Windows in
|
||||
Git Bash, the shebang's `python3` was the Microsoft Store alias, which made every hook a silent
|
||||
no-op. Before the preflight has created the venv, `trace-hook` records nothing and exits 0.
|
||||
- **`./tools/trace-hook` has a PowerShell twin, `tools/trace-hook.ps1`.** PowerShell on Windows
|
||||
resolves the string to the `.ps1` first. Without the twin, it hands the sh script to a file
|
||||
association, and Windows asks which app should open it - on every hook event. Copilot CLI also
|
||||
reads `.claude/settings.json` and runs its single `command` under PowerShell on Windows, so a
|
||||
`bash` field alone does not keep the string out of PowerShell. The twin follows the sh script's
|
||||
rules - venv Python, silent without a venv, exit 0 whatever happens - and stays parseable by
|
||||
Windows PowerShell 5.1, because VS Code starts hooks under 5.1.
|
||||
- **Every command ends in `|| true`.** `preToolUse` hooks fail closed: a non-zero exit denies the
|
||||
tool call. Without the guard, a missing interpreter would turn the observer into a blocker that
|
||||
refuses every tool call in the session. Timeouts fail open, so the 5-second `timeoutSec` is no
|
||||
hazard.
|
||||
- **The event name is passed explicitly.** Copilot serves two payload dialects: camelCase event
|
||||
names give camelCase fields, and PascalCase names give the snake_case shape that VS Code and Claude
|
||||
use. `--event` means the mapping does not depend on which one a configuration chose.
|
||||
|
||||
### VS Code Chat
|
||||
|
||||
VS Code Chat has no hooks, so nothing observes a session while it runs. Its chronicle store
|
||||
(`session-store.db`, the same schema as Copilot CLI's) keeps sessions, turns and touched files, and
|
||||
[tools/import_chronicle.py](../../tools/import_chronicle.py) rebuilds a trace from it afterwards:
|
||||
|
||||
```bash
|
||||
tools/import_chronicle.py --dry-run # this repository's sessions
|
||||
tools/import_chronicle.py --session 0f1e2d3c
|
||||
```
|
||||
|
||||
The store starts empty, and `/chronicle reindex` fills it. That command also syncs session data to
|
||||
your GitHub account, so run it deliberately rather than from a script. A rebuilt trace marks itself
|
||||
`reconstructed: true`, and its `completeness` names `tool.post` but not `tool.pre`: the store records
|
||||
that a file was touched, not that a tool was about to run.
|
||||
|
||||
### Mistral Vibe
|
||||
|
||||
[.vibe/hooks.toml](../../.vibe/hooks.toml) declares the three hooks Vibe has, and
|
||||
[.vibe/config.toml](../../.vibe/config.toml) keeps the telemetry settings in the repository rather
|
||||
than in someone's shell profile. Both were checked against the installed CLI's own loader
|
||||
(`mistral-vibe 2.24.2`) rather than against its documentation. What that turned up:
|
||||
|
||||
- **A failing hook cannot block anything.** With `strict = false`, the default, a crash or a timeout
|
||||
is only a warning. Only a hook that opts into `strict` can deny a tool call - the mirror image of
|
||||
Copilot.
|
||||
- **`post_agent` carries no response text,** only the session context. That is why it maps to
|
||||
`turn.end` rather than `assistant.message`.
|
||||
- **`enable_telemetry` defaults to `true`.** Turning it off is a real change, and it also gates OTel
|
||||
export, which needs both flags on. This repository leaves both off: Vibe's `enable_otel` sends
|
||||
prompts to Mistral Studio.
|
||||
- **Vibe already reads `.agents/skills/` and `AGENTS.md`.** The directory that
|
||||
`wikitool instructions sync` publishes is a project-scope skill source for Vibe, so working with
|
||||
Vibe needs no adaptation, only a trusted folder.
|
||||
Reference in new issue
Block a user