--- type: types/instruction.md name: 5.0.0-confidence-removal description: Remove the confidence/confidence_base frontmatter fields from every entity and concept page - a shape change with no judgment call, since the machinery that read them is already gone. manual: true migrates_to: 5.0.0 migration_kind: mechanical --- # Remove `confidence`/`confidence_base` from every page (5.0.0) 5.0.0 removes the confidence mechanism from the stack: the two frontmatter fields, the `wikitool confidence decay`/`init-base` commands, the search sort/filter machinery, and the advisory lint check that compared it to source standing. `AGENTS.md` invariant 3 and `kb/CONVENTIONS.md`'s hedging rule take over the two jobs the number used to do - hedging follows what the sources carry, and the work list finds unreviewed pages by `!sources`/`provenance=general` rather than by a threshold. The full measurement behind the removal - why the mechanism never tracked anything auditable - is the changelog entry for this version, not repeated here. **No page learns anything new, and no body is touched.** This migration only drops two keys from frontmatter; it is `mechanical` because there is no per-page judgment to make; the removal itself was the judgment, made once, in the version that introduced this document. ## Contents - [When to run](#when-to-run) - [Steps](#steps) - [How to tell a migrated page from an unmigrated one](#how-to-tell-a-migrated-page-from-an-unmigrated-one) - [Decision points](#decision-points) - [Scope](#scope) ## When to run After installing 5.0.0 machinery over an instance on 4.x. `tools/wikitool migrate status` names it, and `tools/wikitool lint --fail-on-error` refuses with `schema_validation_errors` on every entity/concept page still carrying either field - `types/entity.schema.yaml` and `types/concept.schema.yaml` both declare `additionalProperties: false`, so the two keys are not merely unused after 5.0.0, they are invalid. **This is not a window to live in.** Unlike a labelled-edge migration, there is no advisory phase here: a page carrying the old fields is schema-invalid the moment the new schema lands, not merely outdated. Both schemas declare `additionalProperties: false`, so a `lint --fail-on-error` run between the two halves refuses the whole corpus. Install the 5.0.0 machinery and run this migration in the same sitting, publishing both together rather than the schema change on its own. ## Steps 1. **Confirm the scope.** Only `entity` and `concept` pages ever declared the fields; `source` and `comparison` pages never did and need no check: ```bash tools/wikitool search --field 'confidence:*' ``` Every hit is a page this migration must touch. (Once the fields are gone from the schema, the same predicate becomes a refusal rather than an empty result - that refusal is itself the signal that the migration finished; see step 4.) 2. **Strip both fields from each page's frontmatter, mechanically, not by hand.** A script that reads a page, deletes the `confidence`/`confidence_base` keys if present, and writes the frontmatter back through this stack's own YAML writer - never a hand-edit, and never a subagent guessing at formatting. The invariant a checked run must hold: - `modified:` is byte-for-byte unchanged. This is a shape change, not a content confirmation, and a bumped date would misstate 152 pages as freshly reviewed. - The body is byte-for-byte unchanged, including footnotes and every generated region. - Every page-reference array (`related:`, `sources:`, `entities:`, `concepts:`) is unchanged in both content and order. - The remaining frontmatter keys keep the schema's field order. - No untracked or gitignored file is touched. Cut the corpus into units against the iteration budget - `instructions/migrate-corpus.md`'s rule of thumb is 48 pages or fewer per unit, which for ~152 affected pages is four units. 3. **Check each unit mechanically before anything else:** ```bash tools/wikitool migrate verify --from --path kb/ --fail-on-error ``` It compares wikilink and citation counts, footnote definitions, H1, and structural frontmatter - `confidence_base` no longer among the fields it compares as of this same version, so the field's absence is not itself reported as a defect. Everything else it checks stays exactly as strict as it was. 4. **Record it. The checks tighten themselves:** ```bash tools/wikitool lint --fail-on-error # schema_validation_errors must be 0 tools/wikitool migrate done 5.0.0 --pages ``` Only once `lint --fail-on-error` passes clean is the run finished. From this point, `search --field 'confidence<0.6'` (or any predicate naming either field) is refused with an "unknown field" error rather than silently returning nothing - the search layer already refuses a predicate no page in the corpus carries, so the migration's own completion is what makes that refusal fire. ## How to tell a migrated page from an unmigrated one An unmigrated entity or concept page still has `confidence:` and `confidence_base:` lines in its frontmatter block; a migrated one has neither, and validates against the 5.0.0 schema without them. `tools/wikitool search --field 'confidence:*'` lists every page still waiting; an empty result (or, once the schema has landed, the predicate's own refusal) is the corpus-wide answer. ## Decision points - **A page's `confidence_base` sat far below its cited sources' standing, or far above?** Not this migration's question. The check that used to flag that mismatch (`confidence_exceeds_source_standing`) is gone with the field it read, and its replacement - if the corpus wants one - is a prose spot-check `wiki-lint` applies by hand, not a mechanical gate. Do not use this migration as an occasion to also rewrite a page's hedging; that is `wiki-manage`'s job, on its own schedule. - **A page has `confidence` but no `confidence_base`, or the reverse?** Both are pre-existing states this stack already tolerated (the decay formula skipped pages missing a base). Strip whichever key is present; there is nothing to reconcile between them first. - **Unsure whether a script wrote frontmatter correctly?** Diff one migrated page's frontmatter block against its pre-migration version by hand before running the rest of a unit - the two keys should be the only difference, in the position the schema declares. ## Scope The `confidence`/`confidence_base` keys on every `kb/entities/**` and `kb/concepts/**` page. Nothing under `raw/`, no body prose anywhere, and no other frontmatter field. Installing the 5.0.0 machinery itself - the schema change, the removed commands, the hedging rule in `kb/CONVENTIONS.md` - is a separate step that must land first; this document only carries the corpus across the boundary that step opens.