torbenandClaude Opus 5.5
f9056c04f7
Files changed: - .gitea/workflows/ci.yml - .gitignore - .vibe/hooks.toml - AGENTS.md - CHANGES.md - EVALS.md - INSTALL-MCP.md - INSTALL.md - README.md - VERSION - docs/dev/development.md - docs/explanation/README.md - docs/explanation/architecture.md - docs/explanation/language-boundaries.md - docs/explanation/telemetry-and-evaluation.md - docs/how-to/README.md - docs/how-to/browse-with-obsidian.md - docs/how-to/capture-repository-docs.md - docs/how-to/export-guidelines.md - docs/how-to/ingest-a-source.md - docs/how-to/maintain-the-wiki.md - docs/how-to/query-the-wiki.md - docs/how-to/run-the-weekly-review.md - docs/how-to/set-up-the-mcp-server.md - docs/how-to/trace-and-score-sessions.md - docs/how-to/troubleshoot.md - docs/how-to/update-an-instance.md - docs/reference/README.md - docs/reference/commands/distribution-and-versioning.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/commands/telemetry.md - docs/reference/commands/types-instructions-and-docs.md - docs/reference/configuration.md - instructions/dev/dev-setup.md - instructions/dev/doc-pull-through.md - instructions/dev/stack-build/SKILL.md - instructions/dev/testing-conventions.md - instructions/dev/version-parts.md - instructions/setup-instance.md - instructions/upgrade-instance.md - reports/CONTRACT.md - tools/.coveragerc - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/dist_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/eval_cmd.py - tools/chemenu/commands/git_publish.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/commands/version_cmd.py - tools/chemenu/evals/__init__.py - tools/chemenu/evals/scorecard.py - tools/chemenu/evals/trajectory.py - tools/chemenu/telemetry/policy.py - tools/chemenu/tests/test_dist_cmd.py - tools/chemenu/tests/test_git_publish.py - tools/chemenu/tests/test_instructions_shell.py - tools/chemenu/tests/test_pipeline_l0.py - tools/chemenu/toc.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
115 lines
3.2 KiB
Markdown
115 lines
3.2 KiB
Markdown
# Telemetry
|
|
|
|
Reading and scoring the traces that sessions leave behind.
|
|
|
|
<!-- wikitool:toc -->
|
|
## Contents
|
|
|
|
- [`eval sessions`](#eval-sessions)
|
|
- [`eval score`](#eval-score)
|
|
<!-- /wikitool:toc -->
|
|
|
|
<!-- wikitool:reference-commands -->
|
|
## `eval sessions`
|
|
|
|
List the sessions that have a trace under `reports/telemetry/`.
|
|
|
|
**Synopsis**
|
|
|
|
- `wikitool eval sessions [--json]`
|
|
|
|
**Options**
|
|
|
|
```text
|
|
--json Print the list as JSON
|
|
-h, --help Show this message and exit.
|
|
```
|
|
|
|
**Examples**
|
|
|
|
```bash
|
|
tools/wikitool eval sessions
|
|
tools/wikitool eval sessions --json
|
|
```
|
|
|
|
**Behaviour**
|
|
|
|
- Lists the sessions that have a trace under `reports/telemetry/`, most recent first.
|
|
- Never fails; an empty list is a valid answer.
|
|
- Read-only and exempt from the Iteration Budget Gate.
|
|
|
|
**Properties**
|
|
|
|
- Changes files: no
|
|
- Idempotent: yes
|
|
- Atomic: Read-only
|
|
- Counts toward the session call budget: no
|
|
- Can reach the network: no
|
|
|
|
**Exit codes**
|
|
|
|
- `0` - success
|
|
|
|
**See also**
|
|
|
|
- `wikitool eval score` - scores one of them
|
|
- `docs/explanation/telemetry-and-evaluation.md` - how telemetry and evaluation work
|
|
|
|
## `eval score`
|
|
|
|
Score one traced session.
|
|
|
|
**Synopsis**
|
|
|
|
- `wikitool eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]`
|
|
|
|
**Options**
|
|
|
|
```text
|
|
--session <str> Session to score. Defaults to this shell's session, the
|
|
same id the budget gate uses.
|
|
--json Print the scorecard as JSON
|
|
--markdown <path> Write a markdown scorecard here, conventionally under
|
|
reports/evals/.
|
|
--save Write the scorecard to
|
|
reports/evals/<date>/<session>.{json,md}.
|
|
--fail-on-error Exit non-zero when the tree has hard errors or an invariant
|
|
was violated.
|
|
-h, --help Show this message and exit.
|
|
```
|
|
|
|
**Examples**
|
|
|
|
```bash
|
|
tools/wikitool eval score
|
|
tools/wikitool eval score --session wiki-1727330000 --save
|
|
```
|
|
|
|
**Behaviour**
|
|
|
|
- Scores one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged.
|
|
- Defaults to the current session; `--session <id>` picks another.
|
|
- `--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the report to the named file.
|
|
- A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault.
|
|
- Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.
|
|
|
|
**Properties**
|
|
|
|
- Changes files: no
|
|
- Idempotent: yes
|
|
- Atomic: Read-only, apart from the files `--save`/`--markdown` write
|
|
- Counts toward the session call budget: no
|
|
- Can reach the network: no
|
|
|
|
**Exit codes**
|
|
|
|
- `0` - success
|
|
- `1` - No trace exists for the named session
|
|
- `1` - Only with `--fail-on-error`: the tree has hard errors or an invariant was violated
|
|
|
|
**See also**
|
|
|
|
- `wikitool eval sessions` - which session ids exist
|
|
- `docs/explanation/telemetry-and-evaluation.md` - the scoring levels
|
|
<!-- /wikitool:reference-commands -->
|