# Trace and score agent sessions

Record what an agent did in a session, read the record back, and score it. A score covers the state
the session left the wiki in and how it got there: whether it repeated a refused call, or passed a
gate's flag without that gate ever refusing. Why this exists, what the trace contains, and what a
score means are covered in [telemetry-and-evaluation.md](../explanation/telemetry-and-evaluation.md).

<!-- wikitool:toc -->
## Contents

- [What you prepare](#what-you-prepare)
- [Reading a session back](#reading-a-session-back)
- [What each harness can report](#what-each-harness-can-report)
  - [Claude Code](#claude-code)
  - [Copilot CLI](#copilot-cli)
  - [VS Code Chat](#vs-code-chat)
  - [Mistral Vibe](#mistral-vibe)
<!-- /wikitool:toc -->

## What you prepare

**Turn tracing on, if it is off.** It is on in a clone of the origin repository and off in an
instance installed from a release. `tools/wikitool doctor` reports which state this checkout is in,
and why. To turn it on for one checkout, set it in the per-checkout settings:

```bash
tools/wikitool config set telemetry.enabled true
```

This writes the `telemetry` section of `.wikitool-local.json`. The section and its other keys are in the [configuration reference](../reference/configuration.md#telemetry).
Everything stays on this machine: traces land under `reports/telemetry/`, which is gitignored, and
no exporter is configured.

**Scope the session to the task.** Every source of events joins on one session id. Setting it
yourself keeps one task in one trace:

```bash
export WIKITOOL_SESSION_ID="my-task"
```

[instructions/session-setup.md](../../instructions/session-setup.md) has the line for each shell,
and what to do on a harness that starts a fresh shell for every command.

## Reading a session back

```bash
tools/wikitool eval sessions                       # which sessions have a trace

# what did that session do?
jq -r '[.ts, .source, .event, (.attrs.command // .attrs.tool_name // "")] | @tsv' \
  reports/telemetry/my-task/trace.jsonl

# how well did it do it?
tools/wikitool eval score --session my-task
tools/wikitool eval score --session my-task --save  # also write the scorecard under reports/evals/
```

Both `eval` commands only read, and neither counts against the session's call budget: reading back
what a session did is not iterating on the wiki. Their options are in the
[telemetry command reference](../reference/commands/telemetry.md).

To see how a harness hook payload would be recorded, feed one in by hand without writing anything:

```bash
echo '{"session_id":"x","hook_event_name":"post_tool","tool_name":"bash"}' \
  | tools/trace_ingest.py --source mistral-vibe --dry-run
```

## What each harness can report

`wikitool` records its own calls on every harness. What the agent did *between* those calls -
file reads, searches, shell commands, prompts - comes from the harness's hooks, and harnesses differ
a lot in what they offer. Verified against vendor documentation on 2026-08-23:

| | Claude Code | Copilot CLI | VS Code Chat | Mistral Vibe |
|---|---|---|---|---|
| Hook events | ~30 | 14 | none | 3 (`pre_tool`, `post_tool`, `post_agent`) |
| Session start/end hook | yes | yes | - | **no** |
| Prompt submit hook | yes | yes | - | **no** |
| Which instructions loaded | yes (`InstructionsLoaded`) | no | no | no |
| Block / rewrite a tool call | yes | yes | - | yes |
| OTel to your own collector | yes | via MDM `telemetry` | no | **no** - `enable_otel` targets Mistral Studio only |
| `TRACEPARENT` to subprocesses | yes | undocumented | - | undocumented |
| Programmatic mode | `-p`, `stream-json` | `-p` | no | `-p`, `--output streaming` |
| Isolated config home | to be confirmed | `COPILOT_HOME` | no | `VIBE_HOME` |
| Local session log | `~/.claude/projects/*.jsonl` (unstable format) | `~/.copilot/session-state/<id>/events.jsonl` | chronicle SQLite | `$VIBE_HOME/logs/` (no format guarantee) |
| Wired up here | **partial** - `.claude/settings.json` (`UserPromptSubmit` + a `permissions.ask` rule on the clearing publish) | **yes** - `.github/hooks/wiki-trace.json` | **yes** - `tools/import_chronicle.py` | **yes** - `.vibe/hooks.toml` |

A rule that needs an event a harness cannot report is shown as *skipped*, not as passed or failed.

### Claude Code

`.claude/settings.json` wires `UserPromptSubmit` to `tools/trace_ingest.py`, through
`tools/trace-hook` like every hook here. That gives the trace its turn boundaries, which the rule
`clearance-ended-the-turn` needs. Without a `prompt.submitted` event it reports "cannot say"
instead of a verdict.

**No `PreToolUse` decision hook is wired, and that is a finding, not an oversight.** Verified against
the live CLI on 2026-08-27: a `PreToolUse` hook that returns
`hookSpecificOutput.permissionDecision: "ask"` does **not** override a matching `permissions.allow`
rule. Permissions are evaluated before the hook's decision, so a hook cannot force a confirmation
prompt on a command that is already allowed. A `permissions.ask` rule *does* win, which is why
`.claude/settings.json` carries one on the `--confirm` form of `publish`
(`Bash(tools/wikitool publish --confirm:*)`). The clearing call prompts, and ordinary publishes below
the threshold do not. It is a prefix match, so it depends on `--confirm` coming first, which is why
`publish` always prints its re-run line in that order. Treat it as a useful second line, not a
guarantee: an agent that reorders the flags gets around it.

### Copilot CLI

[.github/hooks/wiki-trace.json](../../.github/hooks/wiki-trace.json) is committed, so a clone
brings its own tracing. Eleven events go to `tools/trace_ingest.py`. `disableAllHooks` in
`.github/copilot/settings.json` turns them off without deleting anything. `userPromptSubmitted` is
among them, so `clearance-ended-the-turn` can be scored on Copilot CLI with no change.

A `preToolUse` entry that forces a decision on the clearing call would first need Copilot's own
decision-document schema verified against a live CLI. That has not happened, and an adapter that
cannot be verified is not written.

Four details of the hook configuration matter. The first two hold for all three harnesses that have
hooks:

- **The interpreter is the venv's, never the script's shebang.** Each `bash` command is
  `./tools/trace-hook ...`, which runs `trace_ingest.py` with `tools/.venv`'s Python in either venv
  layout. Each `powershell` command names `tools\.venv\Scripts\python.exe` directly. On Windows in
  Git Bash, the shebang's `python3` was the Microsoft Store alias, which made every hook a silent
  no-op. Before the preflight has created the venv, `trace-hook` records nothing and exits 0.
- **`./tools/trace-hook` has a PowerShell twin, `tools/trace-hook.ps1`.** PowerShell on Windows
  resolves the string to the `.ps1` first. Without the twin, it hands the sh script to a file
  association, and Windows asks which app should open it - on every hook event. Copilot CLI also
  reads `.claude/settings.json` and runs its single `command` under PowerShell on Windows, so a
  `bash` field alone does not keep the string out of PowerShell. The twin follows the sh script's
  rules - venv Python, silent without a venv, exit 0 whatever happens - and stays parseable by
  Windows PowerShell 5.1, because VS Code starts hooks under 5.1.
- **Every command ends in `|| true`.** `preToolUse` hooks fail closed: a non-zero exit denies the
  tool call. Without the guard, a missing interpreter would turn the observer into a blocker that
  refuses every tool call in the session. Timeouts fail open, so the 5-second `timeoutSec` is no
  hazard.
- **The event name is passed explicitly.** Copilot serves two payload dialects: camelCase event
  names give camelCase fields, and PascalCase names give the snake_case shape that VS Code and Claude
  use. `--event` means the mapping does not depend on which one a configuration chose.

### VS Code Chat

VS Code Chat has no hooks, so nothing observes a session while it runs. Its chronicle store
(`session-store.db`, the same schema as Copilot CLI's) keeps sessions, turns and touched files, and
[tools/import_chronicle.py](../../tools/import_chronicle.py) rebuilds a trace from it afterwards:

```bash
tools/import_chronicle.py --dry-run     # this repository's sessions
tools/import_chronicle.py --session 0f1e2d3c
```

The store starts empty, and `/chronicle reindex` fills it. That command also syncs session data to
your GitHub account, so run it deliberately rather than from a script. A rebuilt trace marks itself
`reconstructed: true`, and its `completeness` names `tool.post` but not `tool.pre`: the store records
that a file was touched, not that a tool was about to run.

### Mistral Vibe

[.vibe/hooks.toml](../../.vibe/hooks.toml) declares the three hooks Vibe has, and
[.vibe/config.toml](../../.vibe/config.toml) keeps the telemetry settings in the repository rather
than in someone's shell profile. Both were checked against the installed CLI's own loader
(`mistral-vibe 2.24.2`) rather than against its documentation. What that turned up:

- **A failing hook cannot block anything.** With `strict = false`, the default, a crash or a timeout
  is only a warning. Only a hook that opts into `strict` can deny a tool call - the mirror image of
  Copilot.
- **`post_agent` carries no response text,** only the session context. That is why it maps to
  `turn.end` rather than `assistant.message`.
- **`enable_telemetry` defaults to `true`.** Turning it off is a real change, and it also gates OTel
  export, which needs both flags on. This repository leaves both off: Vibe's `enable_otel` sends
  prompts to Mistral Studio.
- **Vibe already reads `.agents/skills/` and `AGENTS.md`.** The directory that
  `wikitool instructions sync` publishes is a project-scope skill source for Vibe, so working with
  Vibe needs no adaptation, only a trusted folder.