chemenu/instructions/migrations/5.0.0-confidence-removal.md

type, name, description, manual, migrates_to, migration_kind

type	name	description	manual	migrates_to	migration_kind
types/instruction.md	5.0.0-confidence-removal	Remove the confidence/confidence_base frontmatter fields from every entity and concept page - a shape change with no judgment call, since the machinery that read them is already gone.	true	5.0.0	mechanical

Remove confidence/confidence_base from every page (5.0.0)

5.0.0 removes the confidence mechanism from the stack: the two frontmatter fields, the wikitool confidence decay/init-base commands, the search sort/filter machinery, and the advisory lint check that compared it to source standing. AGENTS.md invariant 3 and kb/CONVENTIONS.md's hedging rule take over the two jobs the number used to do - hedging follows what the sources carry, and the work list finds unreviewed pages by !sources/provenance=general rather than by a threshold. The full measurement behind the removal - why the mechanism never tracked anything auditable - is the changelog entry for this version, not repeated here.

No page learns anything new, and no body is touched. This migration only drops two keys from frontmatter; it is mechanical because there is no per-page judgment to make; the removal itself was the judgment, made once, in the version that introduced this document.

Contents

• When to run
• Steps
• How to tell a migrated page from an unmigrated one
• Decision points
• Scope

When to run

After installing 5.0.0 machinery over an instance on 4.x. tools/wikitool migrate status names it, and tools/wikitool lint --fail-on-error refuses with schema_validation_errors on every entity/concept page still carrying either field - types/entity.schema.yaml and types/concept.schema.yaml both declare additionalProperties: false, so the two keys are not merely unused after 5.0.0, they are invalid.

This is not a window to live in. Unlike a labelled-edge migration, there is no advisory phase here: a page carrying the old fields is schema-invalid the moment the new schema lands, not merely outdated. Both schemas declare additionalProperties: false, so a lint --fail-on-error run between the two halves refuses the whole corpus. Install the 5.0.0 machinery and run this migration in the same sitting, publishing both together rather than the schema change on its own.

Steps

1. Confirm the scope. Only entity and concept pages ever declared the fields; source and comparison pages never did and need no check:

   tools/wikitool search --field 'confidence:*'
   

   Every hit is a page this migration must touch. (Once the fields are gone from the schema, the same predicate becomes a refusal rather than an empty result - that refusal is itself the signal that the migration finished; see step 4.)

2. Strip both fields from each page's frontmatter, mechanically, not by hand. A script that reads a page, deletes the confidence/confidence_base keys if present, and writes the frontmatter back through this stack's own YAML writer - never a hand-edit, and never a subagent guessing at formatting. The invariant a checked run must hold:

   • modified: is byte-for-byte unchanged. This is a shape change, not a content confirmation, and a bumped date would misstate 152 pages as freshly reviewed.
   • The body is byte-for-byte unchanged, including footnotes and every generated region.
   • Every page-reference array (related:, sources:, entities:, concepts:) is unchanged in both content and order.
   • The remaining frontmatter keys keep the schema's field order.
   • No untracked or gitignored file is touched.

   Cut the corpus into units against the iteration budget - instructions/migrate-corpus.md's rule of thumb is 48 pages or fewer per unit, which for ~152 affected pages is four units.

3. Check each unit mechanically before anything else:

   tools/wikitool migrate verify --from <pre-migration rev> --path kb/<area> --fail-on-error
   

   It compares wikilink and citation counts, footnote definitions, H1, and structural frontmatter - confidence_base no longer among the fields it compares as of this same version, so the field's absence is not itself reported as a defect. Everything else it checks stays exactly as strict as it was.

4. Record it. The checks tighten themselves:

   tools/wikitool lint --fail-on-error   # schema_validation_errors must be 0
   tools/wikitool migrate done 5.0.0 --pages <N>
   

   Only once lint --fail-on-error passes clean is the run finished. From this point, search --field 'confidence<0.6' (or any predicate naming either field) is refused with an "unknown field" error rather than silently returning nothing - the search layer already refuses a predicate no page in the corpus carries, so the migration's own completion is what makes that refusal fire.

How to tell a migrated page from an unmigrated one

An unmigrated entity or concept page still has confidence: and confidence_base: lines in its frontmatter block; a migrated one has neither, and validates against the 5.0.0 schema without them. tools/wikitool search --field 'confidence:*' lists every page still waiting; an empty result (or, once the schema has landed, the predicate's own refusal) is the corpus-wide answer.

Decision points

• A page's confidence_base sat far below its cited sources' standing, or far above? Not this migration's question. The check that used to flag that mismatch (confidence_exceeds_source_standing) is gone with the field it read, and its replacement - if the corpus wants one - is a prose spot-check wiki-lint applies by hand, not a mechanical gate. Do not use this migration as an occasion to also rewrite a page's hedging; that is wiki-manage's job, on its own schedule.
• A page has confidence but no confidence_base, or the reverse? Both are pre-existing states this stack already tolerated (the decay formula skipped pages missing a base). Strip whichever key is present; there is nothing to reconcile between them first.
• Unsure whether a script wrote frontmatter correctly? Diff one migrated page's frontmatter block against its pre-migration version by hand before running the rest of a unit - the two keys should be the only difference, in the position the schema declares.

Scope

The confidence/confidence_base keys on every kb/entities/** and kb/concepts/** page. Nothing under raw/, no body prose anywhere, and no other frontmatter field. Installing the 5.0.0 machinery itself - the schema change, the removed commands, the hedging rule in kb/CONVENTIONS.md - is a separate step that must land first; this document only carries the corpus across the boundary that step opens.