tools: command records, Telemetry group - examples, missing --fail-on-error exit line (#142)
Files changed: - CHANGES.md - VERSION - tools/CONTRACT.md - tools/chemenu/commands/eval_cmd.py
This commit is contained in:
1 parent
9617d722de
commit
71efdbe01b
4 files changed
+86
-20
No files matched your search
+31
-3
@@ -2278,13 +2278,25 @@ List the sessions that have a trace under `reports/telemetry/`.
|
||||
- budget: exempt
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool eval sessions`
|
||||
- `tools/wikitool eval sessions --json`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
|
||||
**NOTES**
|
||||
|
||||
Most recent first. Read-only and exempt from the Iteration Budget Gate. Never fails; an empty list is a valid answer.
|
||||
- Lists the sessions that have a trace under `reports/telemetry/`, most recent first.
|
||||
- Never fails; an empty list is a valid answer.
|
||||
- Read-only and exempt from the Iteration Budget Gate.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool eval score` - scores one of them
|
||||
- `EVALS.md` - how telemetry and evaluation work
|
||||
|
||||
#### `eval score`
|
||||
|
||||
@@ -2302,18 +2314,34 @@ Score one traced session.
|
||||
- budget: exempt
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool eval score`
|
||||
- `tools/wikitool eval score --session wiki-1727330000 --save`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 No trace exists for the named session
|
||||
- 1 Only with `--fail-on-error`: the tree has hard errors or an invariant was violated
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- No trace exists for the named session -> Run `eval sessions` to see which ids exist. A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault. Safe to retry
|
||||
- No trace exists for the named session -> Run `eval sessions` to see which ids exist; check with `doctor` whether telemetry is on
|
||||
- Only with `--fail-on-error`: the tree has hard errors or an invariant was violated -> Act on the scorecard; re-run only to re-measure
|
||||
|
||||
**NOTES**
|
||||
|
||||
Structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see `EVALS.md`
|
||||
- Scores one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged.
|
||||
- Defaults to the current session; `--session <id>` picks another.
|
||||
- `--save` writes `reports/evals/<date>/<session>.{json,md}`; `--markdown` writes the report to the named file.
|
||||
- A session records nothing when telemetry is off - `WIKI_TRACE=0`, or a distributed instance with no opt-in (`wikitool doctor` says which) - so an absent trace is not necessarily a fault.
|
||||
- Read-only over `kb/`, safe to retry, and exempt from the Iteration Budget Gate.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool eval sessions` - which session ids exist
|
||||
- `EVALS.md` - the scoring levels
|
||||
|
||||
### Distribution and versioning
|
||||
|
||||
|
||||
Reference in new issue
Block a user