chemenu/docs/how-to/trace-and-score-sessions.md

Trace and score agent sessions

Record what an agent did in a session, read the record back, and score it. A score covers the state the session left the wiki in and how it got there: whether it repeated a refused call, or passed a gate's flag without that gate ever refusing. Why this exists, what the trace contains, and what a score means are covered in telemetry-and-evaluation.md.

Contents

• What you prepare
• Reading a session back
• What each harness can report
  • Claude Code
  • Copilot CLI
  • VS Code Chat
  • Mistral Vibe

What you prepare

Turn tracing on, if it is off. It is on in a clone of the origin repository and off in an instance installed from a release. tools/wikitool doctor reports which state this checkout is in, and why. To turn it on for one checkout, create .wikitool-telemetry.json at the root:

{ "enabled": true }

The file and its other keys are in the configuration reference. Everything stays on this machine: traces land under reports/telemetry/, which is gitignored, and no exporter is configured.

Scope the session to the task. Every source of events joins on one session id. Setting it yourself keeps one task in one trace:

export WIKITOOL_SESSION_ID="my-task"

instructions/session-setup.md has the line for each shell, and what to do on a harness that starts a fresh shell for every command.

Reading a session back

tools/wikitool eval sessions                       # which sessions have a trace

# what did that session do?
jq -r '[.ts, .source, .event, (.attrs.command // .attrs.tool_name // "")] | @tsv' \
  reports/telemetry/my-task/trace.jsonl

# how well did it do it?
tools/wikitool eval score --session my-task
tools/wikitool eval score --session my-task --save  # also write the scorecard under reports/evals/

Both eval commands only read, and neither counts against the session's call budget: reading back what a session did is not iterating on the wiki. Their options are in the telemetry command reference.

To see how a harness hook payload would be recorded, feed one in by hand without writing anything:

echo '{"session_id":"x","hook_event_name":"post_tool","tool_name":"bash"}' \
  | tools/trace_ingest.py --source mistral-vibe --dry-run

What each harness can report

wikitool records its own calls on every harness. What the agent did between those calls - file reads, searches, shell commands, prompts - comes from the harness's hooks, and harnesses differ a lot in what they offer. Verified against vendor documentation on 2026-08-23:

	Claude Code	Copilot CLI	VS Code Chat	Mistral Vibe
Hook events	~30	14	none	3 (pre_tool, post_tool, post_agent)
Session start/end hook	yes	yes	-	no
Prompt submit hook	yes	yes	-	no
Which instructions loaded	yes (InstructionsLoaded)	no	no	no
Block / rewrite a tool call	yes	yes	-	yes
OTel to your own collector	yes	via MDM telemetry	no	no - enable_otel targets Mistral Studio only
TRACEPARENT to subprocesses	yes	undocumented	-	undocumented
Programmatic mode	-p, stream-json	-p	no	-p, --output streaming
Isolated config home	to be confirmed	COPILOT_HOME	no	VIBE_HOME
Local session log	~/.claude/projects/*.jsonl (unstable format)	~/.copilot/session-state/<id>/events.jsonl	chronicle SQLite	$VIBE_HOME/logs/ (no format guarantee)
Wired up here	partial - .claude/settings.json (UserPromptSubmit + a permissions.ask rule on the clearing publish)	yes - .github/hooks/wiki-trace.json	yes - tools/import_chronicle.py	yes - .vibe/hooks.toml

A rule that needs an event a harness cannot report is shown as skipped, not as passed or failed.

Claude Code

.claude/settings.json wires UserPromptSubmit to tools/trace_ingest.py, through tools/trace-hook like every hook here. That gives the trace its turn boundaries, which the rule clearance-ended-the-turn needs. Without a prompt.submitted event it reports "cannot say" instead of a verdict.

No PreToolUse decision hook is wired, and that is a finding, not an oversight. Verified against the live CLI on 2026-08-27: a PreToolUse hook that returns hookSpecificOutput.permissionDecision: "ask" does not override a matching permissions.allow rule. Permissions are evaluated before the hook's decision, so a hook cannot force a confirmation prompt on a command that is already allowed. A permissions.ask rule does win, which is why .claude/settings.json carries one on the --confirm form of publish (Bash(tools/wikitool publish --confirm:*)). The clearing call prompts, and ordinary publishes below the threshold do not. It is a prefix match, so it depends on --confirm coming first, which is why publish always prints its re-run line in that order. Treat it as a useful second line, not a guarantee: an agent that reorders the flags gets around it.

Copilot CLI

.github/hooks/wiki-trace.json is committed, so a clone brings its own tracing. Eleven events go to tools/trace_ingest.py. disableAllHooks in .github/copilot/settings.json turns them off without deleting anything. userPromptSubmitted is among them, so clearance-ended-the-turn can be scored on Copilot CLI with no change.

A preToolUse entry that forces a decision on the clearing call would first need Copilot's own decision-document schema verified against a live CLI. That has not happened, and an adapter that cannot be verified is not written.

Four details of the hook configuration matter. The first two hold for all three harnesses that have hooks:

• The interpreter is the venv's, never the script's shebang. Each bash command is ./tools/trace-hook ..., which runs trace_ingest.py with tools/.venv's Python in either venv layout. Each powershell command names tools\.venv\Scripts\python.exe directly. On Windows in Git Bash, the shebang's python3 was the Microsoft Store alias, which made every hook a silent no-op. Before the preflight has created the venv, trace-hook records nothing and exits 0.
• ./tools/trace-hook has a PowerShell twin, tools/trace-hook.ps1. PowerShell on Windows resolves the string to the .ps1 first. Without the twin, it hands the sh script to a file association, and Windows asks which app should open it - on every hook event. Copilot CLI also reads .claude/settings.json and runs its single command under PowerShell on Windows, so a bash field alone does not keep the string out of PowerShell. The twin follows the sh script's rules - venv Python, silent without a venv, exit 0 whatever happens - and stays parseable by Windows PowerShell 5.1, because VS Code starts hooks under 5.1.
• Every command ends in || true. preToolUse hooks fail closed: a non-zero exit denies the tool call. Without the guard, a missing interpreter would turn the observer into a blocker that refuses every tool call in the session. Timeouts fail open, so the 5-second timeoutSec is no hazard.
• The event name is passed explicitly. Copilot serves two payload dialects: camelCase event names give camelCase fields, and PascalCase names give the snake_case shape that VS Code and Claude use. --event means the mapping does not depend on which one a configuration chose.

VS Code Chat

VS Code Chat has no hooks, so nothing observes a session while it runs. Its chronicle store (session-store.db, the same schema as Copilot CLI's) keeps sessions, turns and touched files, and tools/import_chronicle.py rebuilds a trace from it afterwards:

tools/import_chronicle.py --dry-run     # this repository's sessions
tools/import_chronicle.py --session 0f1e2d3c

The store starts empty, and /chronicle reindex fills it. That command also syncs session data to your GitHub account, so run it deliberately rather than from a script. A rebuilt trace marks itself reconstructed: true, and its completeness names tool.post but not tool.pre: the store records that a file was touched, not that a tool was about to run.

Mistral Vibe

.vibe/hooks.toml declares the three hooks Vibe has, and .vibe/config.toml keeps the telemetry settings in the repository rather than in someone's shell profile. Both were checked against the installed CLI's own loader (mistral-vibe 2.24.2) rather than against its documentation. What that turned up:

• A failing hook cannot block anything. With strict = false, the default, a crash or a timeout is only a warning. Only a hook that opts into strict can deny a tool call - the mirror image of Copilot.
• post_agent carries no response text, only the session context. That is why it maps to turn.end rather than assistant.message.
• enable_telemetry defaults to true. Turning it off is a real change, and it also gates OTel export, which needs both flags on. This repository leaves both off: Vibe's enable_otel sends prompts to Mistral Studio.
• Vibe already reads .agents/skills/ and AGENTS.md. The directory that wikitool instructions sync publishes is a project-scope skill source for Vibe, so working with Vibe needs no adaptation, only a trusted folder.