torbenandClaude Opus 5.5
53383e7e9b
Files changed: - CHANGES.md - README.md - VERSION - docs/reference/README.md - docs/reference/commands/README.md - docs/reference/commands/catalog-and-log.md - docs/reference/commands/content-migrations.md - docs/reference/commands/distribution-and-versioning.md - docs/reference/commands/finding-and-checking.md - docs/reference/commands/git.md - docs/reference/commands/guideline-export.md - docs/reference/commands/instance-health.md - docs/reference/commands/links-and-citations.md - docs/reference/commands/pages.md - docs/reference/commands/provenance.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/commands/telemetry.md - docs/reference/commands/types-instructions-and-docs.md - docs/reference/commands/workshop-runs-and-session-budget.md - docs/reference/exit-codes.md - instructions/dev/doc-pull-through.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_reference_doc.py - tools/chemenu/toc.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
115 lines
3.1 KiB
Markdown
115 lines
3.1 KiB
Markdown
# Telemetry
|
|
|
|
Reading and scoring the traces that sessions leave behind.
|
|
|
|
<!-- wikitool:toc -->
|
|
## Contents
|
|
|
|
- [`eval sessions`](#eval-sessions)
|
|
- [`eval score`](#eval-score)
|
|
<!-- /wikitool:toc -->
|
|
|
|
<!-- wikitool:reference-commands -->
|
|
## `eval sessions`
|
|
|
|
List the sessions that have a trace under `reports/telemetry/`.
|
|
|
|
**Synopsis**
|
|
|
|
- `wikitool eval sessions [--json]`
|
|
|
|
**Options**
|
|
|
|
```text
|
|
--json Print the list as JSON
|
|
-h, --help Show this message and exit.
|
|
```
|
|
|
|
**Examples**
|
|
|
|
```bash
|
|
tools/wikitool eval sessions
|
|
tools/wikitool eval sessions --json
|
|
```
|
|
|
|
**Behaviour**
|
|
|
|
- Lists the sessions that have a trace under `reports/telemetry/`, most recent first.
|
|
- Never fails; an empty list is a valid answer.
|
|
- Read-only and exempt from the Iteration Budget Gate.
|
|
|
|
**Properties**
|
|
|
|
- Changes files: no
|
|
- Idempotent: yes
|
|
- Atomic: Read-only
|
|
- Counts toward the session call budget: no
|
|
- Can reach the network: no
|
|
|
|
**Exit codes**
|
|
|
|
- `0` - success
|
|
|
|
**See also**
|
|
|
|
- `wikitool eval score` - scores one of them
|
|
- `EVALS.md` - how telemetry and evaluation work
|
|
|
|
## `eval score`
|
|
|
|
Score one traced session.
|
|
|
|
**Synopsis**
|
|
|
|
- `wikitool eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]`
|
|
|
|
**Options**
|
|
|
|
```text
|
|
--session <str> Session to score. Defaults to this shell's session, the
|
|
same id the budget gate uses.
|
|
--json Print the scorecard as JSON
|
|
--markdown <path> Write a markdown scorecard here, conventionally under
|
|
reports/evals/.
|
|
--save Write the scorecard to
|
|
reports/evals/<date>/<session>.{json,md}.
|
|
--fail-on-error Exit non-zero when the tree has hard errors or an invariant
|
|
was violated.
|
|
-h, --help Show this message and exit.
|
|
```
|
|
|
|
**Examples**
|
|
|
|
```bash
|
|
tools/wikitool eval score
|
|
tools/wikitool eval score --session wiki-1727330000 --save
|
|
```
|
|
|
|
**Behaviour**
|
|
|
|
- Scores one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged.
|
|
- Defaults to the current session; `--session <id>` picks another.
|
|
- `--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the report to the named file.
|
|
- A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault.
|
|
- Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.
|
|
|
|
**Properties**
|
|
|
|
- Changes files: no
|
|
- Idempotent: yes
|
|
- Atomic: Read-only, apart from the files `--save`/`--markdown` write
|
|
- Counts toward the session call budget: no
|
|
- Can reach the network: no
|
|
|
|
**Exit codes**
|
|
|
|
- `0` - success
|
|
- `1` - No trace exists for the named session
|
|
- `1` - Only with `--fail-on-error`: the tree has hard errors or an invariant was violated
|
|
|
|
**See also**
|
|
|
|
- `wikitool eval sessions` - which session ids exist
|
|
- `EVALS.md` - the scoring levels
|
|
<!-- /wikitool:reference-commands -->
|