torben
18ae28f918
Chemenu kompiliert Rohnotizen zu einem verlinkten, quellengebundenen Wiki: raw/ -> types/ + tools/ -> kb/ -> reports/. Was mechanisch ist, macht tools/wikitool; was Urteil braucht, macht ein Agent unter Contracts, deren Grenzen in Code durchgesetzt sind statt im Prompt. Dieser Commit ist der Startpunkt der oeffentlichen Historie. Die vorherige Entwicklung fand in einer privaten Instanz statt und ist nicht Teil dieses Repositorys; ihre Erzaehlung steht vollstaendig in CHANGES.md, das mit 44 Eintraegen von 0.1.0 bis 2.1.0 erhalten geblieben ist. Der mitgelieferte Korpus ist ein Testbett und eine Demo: 170 Seiten ueber den Stack selbst - Gates, Lint, Versionierung, Suche, das Wiki-Muster. Er dokumentiert das Werkzeug mit den eigenen Mitteln des Werkzeugs. Lizenz: AGPL-3.0 fuer den Stack (tools/, types/), CC-BY-4.0 fuer die Inhalte. Die Grenze zwischen beiden ist der Dateiplan, den dist export berechnet - siehe NOTICE.
135 lines
6.0 KiB
Markdown
135 lines
6.0 KiB
Markdown
# Conversation Transcript - Gate Counting and Measured Calibration Session
|
|
|
|
> Source: Claude Code session (`claude-opus-5`), llm-wiki-test1 workspace
|
|
> Collected: 2026-08-31
|
|
> Participant: Torben
|
|
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instruction is quoted
|
|
> verbatim; the call counts and the before/after file counts are real measurements taken during
|
|
> the session, not estimates; the assistant's reasoning is condensed.
|
|
> No second-hand material. No credentials appeared in the session.
|
|
> One of two transcripts cut from this stretch of the session; the other covers
|
|
> `touch --set/--add/--remove` and the write-once frontmatter gap.
|
|
|
|
Covers two calibrations of limits that had never been measured against real use: the
|
|
Mass-Update Gate's file count, and the iteration budget's guidance band. Stack `1.5.0`
|
|
(`3166c31`). No issue - Torben raised both directly from what he had watched happen.
|
|
|
|
---
|
|
|
|
## Turn 1 - the observation
|
|
|
|
Torben, after watching three ordinary ingests each stop at the Mass-Update Gate:
|
|
|
|
> Passe die approval gate publish an:
|
|
> - automatic erzeugte files wie Index.md können wir Raus nehmen. Wir hatten drei normale
|
|
> ingests und alle liegen ins Gate wo immer ein Haufen Datenbank files dazu kommen
|
|
> - schau sich mal auf. Die Anzahl der Tool ops. Von denen wir ausgehen. Ich habe den Eindruck,
|
|
> dass die Maßgabe 15-20/29-25 zu gering ist
|
|
|
|
Both halves turned out to be measurable rather than matters of taste, and the measurements were
|
|
already lying in the repository.
|
|
|
|
---
|
|
|
|
## Part 1 - generated files no longer count
|
|
|
|
### What was already there
|
|
|
|
`git_publish.py` already knew which files are generated - `is_generated()` covers
|
|
`kb/index.md`, `kb/log.md`, `kb/provenance.md` and every `INDEX.md` - and already had an
|
|
exemption mechanism: `GATE_EXEMPT_PREFIXES = ("work/",)`, with `counted_files()` filtering by
|
|
prefix. The two facts had simply never been connected. `is_generated` was used only to *group*
|
|
the file list for display, under the heading "rebuilt by wikitool - no review needed" - a note
|
|
that said the reviewer need not read them while the count still made them approve them.
|
|
|
|
### The change
|
|
|
|
Generated files are now exempt from the count for the same reason `work/` is: they carry no
|
|
decision. Each is recomputable from the tree by `index rebuild` / `sources rebuild-index`, so
|
|
approving one decides nothing - it only produces the review fatigue the threshold exists to
|
|
prevent. They are still staged, committed and pushed.
|
|
|
|
Measured against the three real changesets from earlier the same day:
|
|
|
|
| Ingest | Files | Counted before | Counted now |
|
|
|---|---|---|---|
|
|
| Comma Bug | 14 | 14 → gate | **9 → passes** |
|
|
| Issue Triage | 16 | 16 → gate | **9 → passes** |
|
|
| Auto Mode | 11 | 11 → gate | **5 → passes** |
|
|
|
|
None of the three was a mass update, and none would now stop.
|
|
|
|
The gate stays armed: ten real pages still trip it however much index churn rides along, and a
|
|
test asserts exactly that so the exemption cannot quietly become a disarming.
|
|
|
|
### Two consequences worth recording
|
|
|
|
**The refusal line accounts for both reasons separately** - "3 under work/ and 5 generated by
|
|
wikitool committed but not counted". A reviewer who sees "9 counted" against a 14-file commit
|
|
otherwise reads the difference as a bug. Keeping the reasons distinct also keeps them honest:
|
|
scratch state and derived output are not the same thing.
|
|
|
|
**The `--confirm` token now digests only what a human actually read.** A rebuilt `INDEX.md` no
|
|
longer invalidates a clearance that was already given.
|
|
|
|
### Test coverage found missing
|
|
|
|
All 67 gate tests passed *before* the tests for the new behaviour were written - meaning no test
|
|
had ever asserted that generated files were counted. The old behaviour was untested, which is
|
|
part of how it survived unexamined.
|
|
|
|
---
|
|
|
|
## Part 2 - the calibration band was demonstrably too low
|
|
|
|
### Where the evidence was
|
|
|
|
`tools/.wikitool_session/budget.json` holds the per-session `wikitool` call counts. It had been
|
|
recording them all along. Four real ingests:
|
|
|
|
| Session | Calls |
|
|
|---|---|
|
|
| `ingest-comma-bug-2026-08-31` | **30** |
|
|
| `ingest-transcript-personalization-plane` | **29** |
|
|
| `ingest-issue-triage-2026-08-31` | **26** |
|
|
| `ingest-auto-mode-2026-08-31` | **24** |
|
|
| `issue-14-2026-08-31` (stack work) | 9 |
|
|
|
|
The documented band for a complex multi-tool workflow was **15-25**. Every single ingest sat at
|
|
or above its ceiling while doing nothing unusual.
|
|
|
|
### Why that matters beyond the number
|
|
|
|
A guideline the normal case exceeds is not a guideline. It teaches an agent that the numbers are
|
|
decorative - which is precisely the failure the iteration budget was built to be immune to,
|
|
since a prompt-level limit is one an agent can talk itself past.
|
|
|
|
New: **~5-15** for a simple task (measured 5-9), **~20-35** for a complex multi-tool workflow.
|
|
Pulled through `run_budget.py`, `instructions/gates.md`, and the `wiki-ingest` and `wiki-lint`
|
|
skills. The ceiling of 60 was left alone - it is not a target but the point past which a session
|
|
is presumed stuck.
|
|
|
|
### The provenance distinction that was preserved
|
|
|
|
The old band was an inherited industry rule of thumb, and `kb/concepts/Iteration and Cost
|
|
Limits.md` cites it as exactly that, with a source. It was **not** rewritten: it is a sourced
|
|
claim about the state of the art, not about this instance. What this instance measures is a
|
|
different claim needing its own source, which is why it waited for this transcript rather than
|
|
being edited into the page directly.
|
|
|
|
`gates.md` now also records *where the number comes from and how to re-measure it*, naming
|
|
`tools/.wikitool_session/budget.json`. A guideline with no measurement procedure goes stale
|
|
silently - which is what had happened.
|
|
|
|
---
|
|
|
|
## Outcome
|
|
|
|
| Artifact | Result |
|
|
|---|---|
|
|
| Stack version | `1.5.0` (MINOR - no content has to migrate) |
|
|
| Commit | `3166c31` - 9 files |
|
|
| Tests | 678 passing, in the normal and the hardened environment |
|
|
| Gate | Generated files committed, not counted; threshold unchanged at 10 |
|
|
| Budget | Band 15-25 → 20-35 for complex workflows; ceiling unchanged at 60 |
|