chemenu/instructions/migrations/5.0.0-confidence-removal.md

---
type: types/instruction.md
name: 5.0.0-confidence-removal
description: Remove the confidence/confidence_base frontmatter fields from every entity and concept page - a shape change with no judgment call, since the machinery that read them is already gone.
manual: true
migrates_to: 5.0.0
migration_kind: mechanical
---
# Remove `confidence`/`confidence_base` from every page (5.0.0)

5.0.0 removes the confidence mechanism from the stack: the two frontmatter fields, the
`wikitool confidence decay`/`init-base` commands, the search sort/filter machinery, and the
advisory lint check that compared it to source standing. `AGENTS.md` invariant 3 and
`kb/CONVENTIONS.md`'s hedging rule take over the two jobs the number used to do - hedging follows
what the sources carry, and the work list finds unreviewed pages by `!sources`/`provenance=general`
rather than by a threshold. The full measurement behind the removal - why the mechanism never
tracked anything auditable - is the changelog entry for this version, not repeated here.

**No page learns anything new, and no body is touched.** This migration only drops two keys from
frontmatter; it is `mechanical` because there is no per-page judgment to make; the removal itself
was the judgment, made once, in the version that introduced this document.

<!-- wikitool:toc -->
## Contents

- [When to run](#when-to-run)
- [Steps](#steps)
- [How to tell a migrated page from an unmigrated one](#how-to-tell-a-migrated-page-from-an-unmigrated-one)
- [Decision points](#decision-points)
- [Scope](#scope)
<!-- /wikitool:toc -->

## When to run

After installing 5.0.0 machinery over an instance on 4.x. `tools/wikitool migrate status` names
it, and `tools/wikitool lint --fail-on-error` refuses with `schema_validation_errors` on every
entity/concept page still carrying either field - `types/entity.schema.yaml` and
`types/concept.schema.yaml` both declare `additionalProperties: false`, so the two keys are not
merely unused after 5.0.0, they are invalid.

**This is not a window to live in.** Unlike a labelled-edge migration, there is no advisory phase
here: a page carrying the old fields is schema-invalid the moment the new schema lands, not
merely outdated. Both schemas declare `additionalProperties: false`, so a `lint --fail-on-error`
run between the two halves refuses the whole corpus. Install the 5.0.0 machinery and run this
migration in the same sitting, publishing both together rather than the schema change on its
own.

## Steps

1. **Confirm the scope.** Only `entity` and `concept` pages ever declared the fields; `source`
   and `comparison` pages never did and need no check:

   ```bash
   tools/wikitool search --field 'confidence:*'
   ```

   Every hit is a page this migration must touch. (Once the fields are gone from the schema,
   the same predicate becomes a refusal rather than an empty result - that refusal is itself
   the signal that the migration finished; see step 4.)

2. **Strip both fields from each page's frontmatter, mechanically, not by hand.** A script that
   reads a page, deletes the `confidence`/`confidence_base` keys if present, and writes the
   frontmatter back through this stack's own YAML writer - never a hand-edit, and never a
   subagent guessing at formatting. The invariant a checked run must hold:

   - `modified:` is byte-for-byte unchanged. This is a shape change, not a content
     confirmation, and a bumped date would misstate 152 pages as freshly reviewed.
   - The body is byte-for-byte unchanged, including footnotes and every generated region.
   - Every page-reference array (`related:`, `sources:`, `entities:`, `concepts:`) is unchanged
     in both content and order.
   - The remaining frontmatter keys keep the schema's field order.
   - No untracked or gitignored file is touched.

   Cut the corpus into units against the iteration budget - `instructions/migrate-corpus.md`'s
   rule of thumb is 48 pages or fewer per unit, which for ~152 affected pages is four units.

3. **Check each unit mechanically before anything else:**

   ```bash
   tools/wikitool migrate verify --from <pre-migration rev> --path kb/<area> --fail-on-error
   ```

   It compares wikilink and citation counts, footnote definitions, H1, and structural
   frontmatter - `confidence_base` no longer among the fields it compares as of this same
   version, so the field's absence is not itself reported as a defect. Everything else it
   checks stays exactly as strict as it was.

4. **Record it. The checks tighten themselves:**

   ```bash
   tools/wikitool lint --fail-on-error   # schema_validation_errors must be 0
   tools/wikitool migrate done 5.0.0 --pages <N>
   ```

   Only once `lint --fail-on-error` passes clean is the run finished. From this point, `search
   --field 'confidence<0.6'` (or any predicate naming either field) is refused with an "unknown
   field" error rather than silently returning nothing - the search layer already refuses a
   predicate no page in the corpus carries, so the migration's own completion is what makes
   that refusal fire.

## How to tell a migrated page from an unmigrated one

An unmigrated entity or concept page still has `confidence:` and `confidence_base:` lines in its
frontmatter block; a migrated one has neither, and validates against the 5.0.0 schema without
them. `tools/wikitool search --field 'confidence:*'` lists every page still waiting; an empty
result (or, once the schema has landed, the predicate's own refusal) is the corpus-wide answer.

## Decision points

- **A page's `confidence_base` sat far below its cited sources' standing, or far above?** Not
  this migration's question. The check that used to flag that mismatch
  (`confidence_exceeds_source_standing`) is gone with the field it read, and its replacement -
  if the corpus wants one - is a prose spot-check `wiki-lint` applies by hand, not a mechanical
  gate. Do not use this migration as an occasion to also rewrite a page's hedging; that is
  `wiki-manage`'s job, on its own schedule.
- **A page has `confidence` but no `confidence_base`, or the reverse?** Both are pre-existing
  states this stack already tolerated (the decay formula skipped pages missing a base). Strip
  whichever key is present; there is nothing to reconcile between them first.
- **Unsure whether a script wrote frontmatter correctly?** Diff one migrated page's frontmatter
  block against its pre-migration version by hand before running the rest of a unit - the two
  keys should be the only difference, in the position the schema declares.

## Scope

The `confidence`/`confidence_base` keys on every `kb/entities/**` and `kb/concepts/**` page.
Nothing under `raw/`, no body prose anywhere, and no other frontmatter field. Installing the
5.0.0 machinery itself - the schema change, the removed commands, the hedging rule in
`kb/CONVENTIONS.md` - is a separate step that must land first; this document only carries the
corpus across the boundary that step opens.