tools: command records, Telemetry group - examples, missing --fail-on-error exit line (#142)
CI / verify (push) Successful in 1m14s
Release / release (push) Successful in 35s

Files changed:
- CHANGES.md
- VERSION
- tools/CONTRACT.md
- tools/chemenu/commands/eval_cmd.py
This commit is contained in:
torben committed 2026-09-26 09:24:15 +02:00
1 parent 9617d722de
commit 71efdbe01b
4 files changed
+85 -19

No files matched your search

+8 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 7.1.0-beta.16 - 2026-09-26 - Command records, Types, instructions and docs group: one line per cause, examples, prohibitions
## 7.1.0-beta.17 - 2026-09-26 - Command records, Telemetry group: examples, the missing --fail-on-error exit line
**Author:** Torben Nehmer
@@ -85,6 +85,7 @@ concern - readable here, never shipped as something to parse.
- Command records, Raw material and uploads group: one line per cause, examples, prohibitions
- Command records, Workshop runs and session budget group: examples, prohibitions
- Command records, Types, instructions and docs group: one line per cause, examples, prohibitions
- Command records, Telemetry group: examples, the missing --fail-on-error exit line
<!-- /wikitool:bumps -->
### CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
@@ -299,6 +300,12 @@ references, tables of contents, links), with six exit-1 causes and a reaction ea
always done. The reasoning `docs toc`'s record carried about its scope already lived in
`toc.py`'s module docstring and now lives only there.
### Command records, Telemetry group: examples, the missing --fail-on-error exit line
`eval sessions` and `eval score` rewritten the same way; text only. `eval score --fail-on-error`
has always exited 1 on a failed scorecard, but its record only listed the missing-trace case; it
now names both.
---
## 7.0.0 - 2026-09-22 - Task-Tracker-Anbindung: Vorhaben als Seitenart, Verpflichtungsschicht, Weekly Review als Read-Time-Join
+1 -1
View File
@@ -1 +1 @@
7.1.0-beta.16
7.1.0-beta.17
+31 -3
View File
@@ -2278,13 +2278,25 @@ List the sessions that have a trace under `reports/telemetry/`.
- budget: exempt
- network: no
**EXAMPLES**
- `tools/wikitool eval sessions`
- `tools/wikitool eval sessions --json`
**EXIT STATUS**
- 0 success
**NOTES**
Most recent first. Read-only and exempt from the Iteration Budget Gate. Never fails; an empty list is a valid answer.
- Lists the sessions that have a trace under `reports/telemetry/`, most recent first.
- Never fails; an empty list is a valid answer.
- Read-only and exempt from the Iteration Budget Gate.
**SEE ALSO**
- `wikitool eval score` - scores one of them
- `EVALS.md` - how telemetry and evaluation work
#### `eval score`
@@ -2302,18 +2314,34 @@ Score one traced session.
- budget: exempt
- network: no
**EXAMPLES**
- `tools/wikitool eval score`
- `tools/wikitool eval score --session wiki-1727330000 --save`
**EXIT STATUS**
- 0 success
- 1 No trace exists for the named session
- 1 Only with `--fail-on-error`: the tree has hard errors or an invariant was violated
**ON FAILURE**
- No trace exists for the named session -> Run `eval sessions` to see which ids exist. A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault. Safe to retry
- No trace exists for the named session -> Run `eval sessions` to see which ids exist; check with `doctor` whether telemetry is on
- Only with `--fail-on-error`: the tree has hard errors or an invariant was violated -> Act on the scorecard; re-run only to re-measure
**NOTES**
Structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see `EVALS.md`
- Scores one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged.
- Defaults to the current session; `--session <id>` picks another.
- `--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the report to the named file.
- A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault.
- Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.
**SEE ALSO**
- `wikitool eval sessions` - which session ids exist
- `EVALS.md` - the scoring levels
### Distribution and versioning
+45 -14
View File
@@ -36,9 +36,20 @@ EVALS_DIR = config.REPORTS_DIR / "evals"
atomic="Read-only",
budget=cli_contract.Budget.EXEMPT,
),
notes="Most recent first. Read-only and exempt from the Iteration Budget Gate. Never fails; "
"an empty list is a valid answer.",
notes=(
"Lists the sessions that have a trace under `reports/telemetry/`, most recent first.",
"Never fails; an empty list is a valid answer.",
"Read-only and exempt from the Iteration Budget Gate.",
),
failures=(),
examples=(
"tools/wikitool eval sessions",
"tools/wikitool eval sessions --json",
),
see_also=(
"`wikitool eval score` - scores one of them",
"`EVALS.md` - how telemetry and evaluation work",
),
))
def sessions_command(
json_out: bool = typer.Option(False, "--json", help="Print the list as JSON"),
@@ -71,19 +82,39 @@ def sessions_command(
atomic="Read-only, apart from the files `--save`/`--markdown` write",
budget=cli_contract.Budget.EXEMPT,
),
notes="Structural state from `lint`'s own checks (L1) plus trajectory rules over the trace "
"(L2) - was a refused call repeated unchanged, was a gate flag passed without that gate "
"having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current "
"session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` "
"and exempt from the budget; see `EVALS.md`",
failures=(cli_contract.Failure(
label="",
notes=(
"Scores one traced session: structural state from `lint`'s own checks (L1) plus "
"trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a "
"gate flag passed without that gate having refused anything, did a publish of `kb/` "
"pages go unlogged.",
"Defaults to the current session; `--session <id>` picks another.",
"`--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the "
"report to the named file.",
"A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed "
"instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not "
"necessarily a fault.",
"Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.",
),
failures=(
cli_contract.Failure(
cause="No trace exists for the named session",
reaction="Run `eval sessions` to see which ids exist. A session records nothing when "
"telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in "
"(`wikitool doctor` says which) - so an absent trace is not necessarily a fault. Safe "
"to retry",
),),
reaction="Run `eval sessions` to see which ids exist; check with `doctor` whether "
"telemetry is on",
),
cli_contract.Failure(
cause="Only with `--fail-on-error`: the tree has hard errors or an invariant was "
"violated",
reaction="Act on the scorecard; re-run only to re-measure",
),
),
examples=(
"tools/wikitool eval score",
"tools/wikitool eval score --session wiki-1727330000 --save",
),
see_also=(
"`wikitool eval sessions` - which session ids exist",
"`EVALS.md` - the scoring levels",
),
))
def score_command(
session: Optional[str] = typer.Option(