tools: command records, Telemetry group - examples, missing --fail-on-error exit line (#142)
Files changed: - CHANGES.md - VERSION - tools/CONTRACT.md - tools/chemenu/commands/eval_cmd.py
This commit is contained in:
1 parent
9617d722de
commit
71efdbe01b
4 files changed
+85
-19
No files matched your search
+8
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7.1.0-beta.16 - 2026-09-26 - Command records, Types, instructions and docs group: one line per cause, examples, prohibitions
|
## 7.1.0-beta.17 - 2026-09-26 - Command records, Telemetry group: examples, the missing --fail-on-error exit line
|
||||||
|
|
||||||
**Author:** Torben Nehmer
|
**Author:** Torben Nehmer
|
||||||
|
|
||||||
@@ -85,6 +85,7 @@ concern - readable here, never shipped as something to parse.
|
|||||||
- Command records, Raw material and uploads group: one line per cause, examples, prohibitions
|
- Command records, Raw material and uploads group: one line per cause, examples, prohibitions
|
||||||
- Command records, Workshop runs and session budget group: examples, prohibitions
|
- Command records, Workshop runs and session budget group: examples, prohibitions
|
||||||
- Command records, Types, instructions and docs group: one line per cause, examples, prohibitions
|
- Command records, Types, instructions and docs group: one line per cause, examples, prohibitions
|
||||||
|
- Command records, Telemetry group: examples, the missing --fail-on-error exit line
|
||||||
<!-- /wikitool:bumps -->
|
<!-- /wikitool:bumps -->
|
||||||
|
|
||||||
### CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
|
### CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
|
||||||
@@ -299,6 +300,12 @@ references, tables of contents, links), with six exit-1 causes and a reaction ea
|
|||||||
always done. The reasoning `docs toc`'s record carried about its scope already lived in
|
always done. The reasoning `docs toc`'s record carried about its scope already lived in
|
||||||
`toc.py`'s module docstring and now lives only there.
|
`toc.py`'s module docstring and now lives only there.
|
||||||
|
|
||||||
|
### Command records, Telemetry group: examples, the missing --fail-on-error exit line
|
||||||
|
|
||||||
|
`eval sessions` and `eval score` rewritten the same way; text only. `eval score --fail-on-error`
|
||||||
|
has always exited 1 on a failed scorecard, but its record only listed the missing-trace case; it
|
||||||
|
now names both.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7.0.0 - 2026-09-22 - Task-Tracker-Anbindung: Vorhaben als Seitenart, Verpflichtungsschicht, Weekly Review als Read-Time-Join
|
## 7.0.0 - 2026-09-22 - Task-Tracker-Anbindung: Vorhaben als Seitenart, Verpflichtungsschicht, Weekly Review als Read-Time-Join
|
||||||
|
|||||||
+31
-3
@@ -2278,13 +2278,25 @@ List the sessions that have a trace under `reports/telemetry/`.
|
|||||||
- budget: exempt
|
- budget: exempt
|
||||||
- network: no
|
- network: no
|
||||||
|
|
||||||
|
**EXAMPLES**
|
||||||
|
|
||||||
|
- `tools/wikitool eval sessions`
|
||||||
|
- `tools/wikitool eval sessions --json`
|
||||||
|
|
||||||
**EXIT STATUS**
|
**EXIT STATUS**
|
||||||
|
|
||||||
- 0 success
|
- 0 success
|
||||||
|
|
||||||
**NOTES**
|
**NOTES**
|
||||||
|
|
||||||
Most recent first. Read-only and exempt from the Iteration Budget Gate. Never fails; an empty list is a valid answer.
|
- Lists the sessions that have a trace under `reports/telemetry/`, most recent first.
|
||||||
|
- Never fails; an empty list is a valid answer.
|
||||||
|
- Read-only and exempt from the Iteration Budget Gate.
|
||||||
|
|
||||||
|
**SEE ALSO**
|
||||||
|
|
||||||
|
- `wikitool eval score` - scores one of them
|
||||||
|
- `EVALS.md` - how telemetry and evaluation work
|
||||||
|
|
||||||
#### `eval score`
|
#### `eval score`
|
||||||
|
|
||||||
@@ -2302,18 +2314,34 @@ Score one traced session.
|
|||||||
- budget: exempt
|
- budget: exempt
|
||||||
- network: no
|
- network: no
|
||||||
|
|
||||||
|
**EXAMPLES**
|
||||||
|
|
||||||
|
- `tools/wikitool eval score`
|
||||||
|
- `tools/wikitool eval score --session wiki-1727330000 --save`
|
||||||
|
|
||||||
**EXIT STATUS**
|
**EXIT STATUS**
|
||||||
|
|
||||||
- 0 success
|
- 0 success
|
||||||
- 1 No trace exists for the named session
|
- 1 No trace exists for the named session
|
||||||
|
- 1 Only with `--fail-on-error`: the tree has hard errors or an invariant was violated
|
||||||
|
|
||||||
**ON FAILURE**
|
**ON FAILURE**
|
||||||
|
|
||||||
- No trace exists for the named session -> Run `eval sessions` to see which ids exist. A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault. Safe to retry
|
- No trace exists for the named session -> Run `eval sessions` to see which ids exist; check with `doctor` whether telemetry is on
|
||||||
|
- Only with `--fail-on-error`: the tree has hard errors or an invariant was violated -> Act on the scorecard; re-run only to re-measure
|
||||||
|
|
||||||
**NOTES**
|
**NOTES**
|
||||||
|
|
||||||
Structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see `EVALS.md`
|
- Scores one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged.
|
||||||
|
- Defaults to the current session; `--session <id>` picks another.
|
||||||
|
- `--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the report to the named file.
|
||||||
|
- A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault.
|
||||||
|
- Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.
|
||||||
|
|
||||||
|
**SEE ALSO**
|
||||||
|
|
||||||
|
- `wikitool eval sessions` - which session ids exist
|
||||||
|
- `EVALS.md` - the scoring levels
|
||||||
|
|
||||||
### Distribution and versioning
|
### Distribution and versioning
|
||||||
|
|
||||||
|
|||||||
@@ -36,9 +36,20 @@ EVALS_DIR = config.REPORTS_DIR / "evals"
|
|||||||
atomic="Read-only",
|
atomic="Read-only",
|
||||||
budget=cli_contract.Budget.EXEMPT,
|
budget=cli_contract.Budget.EXEMPT,
|
||||||
),
|
),
|
||||||
notes="Most recent first. Read-only and exempt from the Iteration Budget Gate. Never fails; "
|
notes=(
|
||||||
"an empty list is a valid answer.",
|
"Lists the sessions that have a trace under `reports/telemetry/`, most recent first.",
|
||||||
|
"Never fails; an empty list is a valid answer.",
|
||||||
|
"Read-only and exempt from the Iteration Budget Gate.",
|
||||||
|
),
|
||||||
failures=(),
|
failures=(),
|
||||||
|
examples=(
|
||||||
|
"tools/wikitool eval sessions",
|
||||||
|
"tools/wikitool eval sessions --json",
|
||||||
|
),
|
||||||
|
see_also=(
|
||||||
|
"`wikitool eval score` - scores one of them",
|
||||||
|
"`EVALS.md` - how telemetry and evaluation work",
|
||||||
|
),
|
||||||
))
|
))
|
||||||
def sessions_command(
|
def sessions_command(
|
||||||
json_out: bool = typer.Option(False, "--json", help="Print the list as JSON"),
|
json_out: bool = typer.Option(False, "--json", help="Print the list as JSON"),
|
||||||
@@ -71,19 +82,39 @@ def sessions_command(
|
|||||||
atomic="Read-only, apart from the files `--save`/`--markdown` write",
|
atomic="Read-only, apart from the files `--save`/`--markdown` write",
|
||||||
budget=cli_contract.Budget.EXEMPT,
|
budget=cli_contract.Budget.EXEMPT,
|
||||||
),
|
),
|
||||||
notes="Structural state from `lint`'s own checks (L1) plus trajectory rules over the trace "
|
notes=(
|
||||||
"(L2) - was a refused call repeated unchanged, was a gate flag passed without that gate "
|
"Scores one traced session: structural state from `lint`'s own checks (L1) plus "
|
||||||
"having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current "
|
"trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a "
|
||||||
"session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` "
|
"gate flag passed without that gate having refused anything, did a publish of `kb/` "
|
||||||
"and exempt from the budget; see `EVALS.md`",
|
"pages go unlogged.",
|
||||||
failures=(cli_contract.Failure(
|
"Defaults to the current session; `--session <id>` picks another.",
|
||||||
label="",
|
"`--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the "
|
||||||
|
"report to the named file.",
|
||||||
|
"A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed "
|
||||||
|
"instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not "
|
||||||
|
"necessarily a fault.",
|
||||||
|
"Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.",
|
||||||
|
),
|
||||||
|
failures=(
|
||||||
|
cli_contract.Failure(
|
||||||
cause="No trace exists for the named session",
|
cause="No trace exists for the named session",
|
||||||
reaction="Run `eval sessions` to see which ids exist. A session records nothing when "
|
reaction="Run `eval sessions` to see which ids exist; check with `doctor` whether "
|
||||||
"telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in "
|
"telemetry is on",
|
||||||
"(`wikitool doctor` says which) - so an absent trace is not necessarily a fault. Safe "
|
),
|
||||||
"to retry",
|
cli_contract.Failure(
|
||||||
),),
|
cause="Only with `--fail-on-error`: the tree has hard errors or an invariant was "
|
||||||
|
"violated",
|
||||||
|
reaction="Act on the scorecard; re-run only to re-measure",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
examples=(
|
||||||
|
"tools/wikitool eval score",
|
||||||
|
"tools/wikitool eval score --session wiki-1727330000 --save",
|
||||||
|
),
|
||||||
|
see_also=(
|
||||||
|
"`wikitool eval sessions` - which session ids exist",
|
||||||
|
"`EVALS.md` - the scoring levels",
|
||||||
|
),
|
||||||
))
|
))
|
||||||
def score_command(
|
def score_command(
|
||||||
session: Optional[str] = typer.Option(
|
session: Optional[str] = typer.Option(
|
||||||
|
|||||||
Reference in new issue
Block a user