torben
18ae28f918
Chemenu kompiliert Rohnotizen zu einem verlinkten, quellengebundenen Wiki: raw/ -> types/ + tools/ -> kb/ -> reports/. Was mechanisch ist, macht tools/wikitool; was Urteil braucht, macht ein Agent unter Contracts, deren Grenzen in Code durchgesetzt sind statt im Prompt. Dieser Commit ist der Startpunkt der oeffentlichen Historie. Die vorherige Entwicklung fand in einer privaten Instanz statt und ist nicht Teil dieses Repositorys; ihre Erzaehlung steht vollstaendig in CHANGES.md, das mit 44 Eintraegen von 0.1.0 bis 2.1.0 erhalten geblieben ist. Der mitgelieferte Korpus ist ein Testbett und eine Demo: 170 Seiten ueber den Stack selbst - Gates, Lint, Versionierung, Suche, das Wiki-Muster. Er dokumentiert das Werkzeug mit den eigenen Mitteln des Werkzeugs. Lizenz: AGPL-3.0 fuer den Stack (tools/, types/), CC-BY-4.0 fuer die Inhalte. Die Grenze zwischen beiden ist der Dateiplan, den dist export berechnet - siehe NOTICE.
122 lines
6.5 KiB
Markdown
122 lines
6.5 KiB
Markdown
---
|
||
type: types/instruction.md
|
||
name: migrate-corpus
|
||
description: Change the shape of every kb/ page at once - a schema field, a vocabulary, a language - in planned units, with a mechanical check per unit and a recorded KB version at the end.
|
||
---
|
||
# Migrate the corpus
|
||
|
||
A change that touches the *shape* of pages rather than their content: a new required
|
||
frontmatter field, a renamed enum value, a section heading vocabulary, a language. It is not an
|
||
ingest and not a lint fix - nothing new is learned, the same knowledge is restated in a new
|
||
form. The failure mode is therefore specific and quiet: **something present before is missing
|
||
afterwards**, and the corpus is still internally consistent, so `lint` reports nothing.
|
||
|
||
Every rule below was paid for once already. The German translation of 248 pages found four
|
||
defects this way - a dropped citation that silently unsourced a claim, a dropped wikilink, an
|
||
invented one, and a translated H1 - and three of the four had unchanged link *sets* and only
|
||
changed counts.
|
||
|
||
## When to run
|
||
|
||
A change that would otherwise be applied to more than a handful of pages by hand, or any change
|
||
declared by a migration document under `instructions/migrations/`. A single page is
|
||
`wiki-manage`; a raw tree is [ingest-large-tree.md](ingest-large-tree.md).
|
||
|
||
## Steps
|
||
|
||
1. **Open a workshop.** `tools/wikitool work new --key <slug>` - not `ingest-`, which is
|
||
reserved for keys derived from a `raw/` path. `plan.md` cuts the corpus into units and says
|
||
why each cut falls where it does; `README.md` carries the closing condition and the
|
||
decisions made so far. See [work/CONTRACT.md](../work/CONTRACT.md).
|
||
|
||
2. **Size the units by the iteration budget, not by feel.** One unit costs roughly
|
||
`N × touch` + `index rebuild` + `log append` + `publish` (twice - the Mass-Update Gate
|
||
refuses once and publishes on the confirm), plus `sources rebuild-index` if it contains
|
||
source pages. Against the 60-call ceiling that puts the ceiling near 55 pages; aim for 48 or
|
||
fewer.
|
||
|
||
**Units and publishes are not the same boundary.** The budget is per session id; the gate is
|
||
per publish. Several units may run back to back, each with its own
|
||
`WIKITOOL_SESSION_ID="<slug>/u<N>"`, and publish once together - which is what the written
|
||
unit boundaries in `plan.md` make legitimate rather than a way around a gate refusal (see
|
||
[gates.md](gates.md)).
|
||
|
||
3. **Rewrite the unit's pages.** Bodies only. Frontmatter is written with `touch`, never by
|
||
hand, and never by a subagent.
|
||
|
||
4. **Check mechanically, before anything else:**
|
||
|
||
```bash
|
||
tools/wikitool migrate verify --from HEAD --path kb/<area> --fail-on-error
|
||
```
|
||
|
||
This is the step the whole procedure exists for. It compares wikilink and citation
|
||
**counts**, footnote definitions, H1 and structural frontmatter against the last commit.
|
||
Run it before the summaries, before `lint`, before anything - it is the cheapest place to
|
||
catch a subagent that helpfully translated a link target.
|
||
|
||
5. **Write the summaries yourself** with `touch --summary`, from the original. Never paste a
|
||
subagent's proposal unread: they embellish, and a summary is a claim about the page.
|
||
|
||
6. **`index rebuild`, then `lint` - and read the whole report**, not only the sections this
|
||
unit could plausibly have touched. The translation's first unit had a frontmatter
|
||
round-trip bug that surfaced as a schema error on a field nobody had edited.
|
||
|
||
7. **`log append`, then publish** through [publish-cycle.md](publish-cycle.md). Expect exit 42
|
||
on a corpus-sized change; reproduce the breakdown for the user and wait.
|
||
|
||
8. **Carry the vocabulary between units.** Terminology settled in unit 5 and re-decided in unit
|
||
9 is the failure a glossary file in the workshop exists to prevent. Add to it *before*
|
||
dispatching the next unit.
|
||
|
||
9. **Record the migration** once the last unit is published:
|
||
|
||
```bash
|
||
tools/wikitool migrate done <version> --pages <N>
|
||
```
|
||
|
||
This advances `kb_version` in `.wikitool-kb.json`. It refuses any version that is not the
|
||
next link in the chain, so a multi-step upgrade cannot silently skip one.
|
||
|
||
10. **Close the workshop** per [work/CONTRACT.md](../work/CONTRACT.md), after promoting whatever
|
||
outlives the run. The translation's glossary became
|
||
[german-terminology.md](german-terminology.md); its checklist and unit plan died with the
|
||
directory, correctly.
|
||
|
||
## Decision points
|
||
|
||
- **Can the change be made backwards-compatible instead?** Prefer it. A vocabulary migration
|
||
does not need a flag day: `tools/chemenu/sections.py` gives each heading one canonical name
|
||
and any number of aliases, so a page is found under the old name and takes the new one only
|
||
when it is rewritten. Removing an alias afterwards is a second breaking change, not a cleanup.
|
||
- **Mechanical or assisted?** A rename with a fixed rule is `mechanical` and wants a script; a
|
||
change needing a judgment call per page is `assisted` and wants this procedure. There is no
|
||
`migrate run` today - `migration_kind` describes the work, it does not perform it.
|
||
- **The check finds something mid-unit.** Fix it in that unit and re-run `verify`. Never carry
|
||
a finding into the next unit "to fix later": the next unit's diff baseline is this unit's
|
||
commit, so an uncorrected drop becomes invisible.
|
||
- **Contradiction with an existing page.** Never overwrite. Record both, ask the user, and pull
|
||
the confidence down with `touch --confidence-base` if it stays unresolved.
|
||
|
||
## Writing the migration document
|
||
|
||
A migration that a distributed instance must also run is a `manual: true` instruction under
|
||
`instructions/migrations/<version>-<slug>.md`, carrying `migrates_to:` and `migration_kind:`.
|
||
`tools/wikitool migrate status` builds the outstanding chain from those files, and `version
|
||
bump` refuses a compatibility-breaking release that has none.
|
||
|
||
Write it for a reader who has the new machinery and the old content, and who is not you: what
|
||
changed, which pages are affected, how to tell a migrated page from an unmigrated one, and what
|
||
`migrate verify` should report when it is done.
|
||
|
||
**Baseline: 1.0.0.** Migrations that predate it - the type-system move, the `confidence_base`
|
||
backfill, the German section headings, the translation itself - have no documents and will not
|
||
get any. An instance older than that is re-exported, not migrated.
|
||
|
||
## Scope
|
||
|
||
For `kb/` content. A single page is `wiki-manage`; a `raw/` tree is
|
||
[ingest-large-tree.md](ingest-large-tree.md). Changing the machinery that *causes* a migration
|
||
is a different job with its own rules, and in a distributed instance it is not done at all -
|
||
the stack is developed in the origin repo.
|