chemenu/tools/README.md

# tools/

Developer documentation for `wikitool` - how the CLI is built, how to change it,
and how to run its tests.

**This is not the command reference.** That is [CONTRACT.md](CONTRACT.md), which
`wikitool docs verify` checks against the registered commands. Copying the
command table here would create a second copy that drifts, so this file
deliberately has none - and `docs verify` now enforces that.

| Document | Audience |
|---|---|
| `README.md` (this file) | Humans working *on* wikitool |
| [`CONTRACT.md`](CONTRACT.md) | Agents working *with* wikitool - commands, error contracts, maintenance schedule |
| [`../AGENTS.md`](../AGENTS.md) | The invariants that say when using a command is mandatory |
| [`../CHANGES.md`](../CHANGES.md) | What changed in the stack, and when |

## Setup

```bash
cd tools
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

`jsonschema` and `PyYAML` are hard dependencies, not optional extras: schema
validation is the tool's whole safety net, so `cli.py` fails loudly with the
install command rather than degrading silently.

## Layout

```
tools/
  wikitool                 entry point
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate
    config.py              repo layout: root resolution and every path under it
    api.py                 the in-process entry point - point Chemenu at a corpus and read it
    errors.py              ChemenuError / ValidationError / BackendError
    corpus_cache.py        one parsed corpus per commit, never cached while the tree is dirty
    kb_scan.py             page iteration/loading over kb/
    blocks.py              generated regions in a page body, found by marker rather than by heading
    links.py               labelled edges in `related:` - the graph's semantics as data, not prose
    kb_collections.py      collection discovery (a directory with COLLECTION.md), and what one declares about itself
    conventions.py         kb/CONVENTIONS.md: what this instance decided about authoring, as opposed to what the stack enforces
    ownership.py           the stack-vs-instance boundary under a content stage - one predicate, read by `dist_cmd.py` and `commands/upstream_cmd.py` so the two cannot answer it differently
    type_resolver.py       type-spec loading and schema resolution
    catalog.py             how the corpus groups into collections and areas, and the shard threshold - with no CLI attached
    lint_core.py           the lint checks and the report, with no CLI attached
    types_core.py          type-spec listing/description, with no CLI attached
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    search/                pluggable search backends, plus service.py - the search core
    commands/              one module per command or command group: the terminal adapters
    tests/                 pytest suite
```

**Two consumers, one core.** The CLI is not the only caller any more. The cores
(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`) hold what
decides an answer and import no `typer` and no `rich`; the modules under `commands/` turn
those values into terminal output and those exceptions into exit codes.
`api.Corpus` is the in-process entry point over the same functions - it takes a
corpus root, returns exactly the structures the `--json` forms print, and
raises instead of exiting. A second consumer is therefore a second adapter
rather than a second implementation, and the write commands are unreachable
from `api` because nothing under `commands/` is imported there.

**Which corpus.** The root resolves by precedence: an explicit argument, then
`$CHEMENU_ROOT`, then a walk up from the package's own location. The walk-up is
the default, so the CLI is unaffected by any of this. Nothing under the root is
bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.

## Adding a command

1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
   app; a single command is a plain function.
2. Register it in `cli.py` (`app.add_typer(...)` or `app.command(...)`).
3. Add a row to [CONTRACT.md](CONTRACT.md)'s command table **and** to its error
   contract table. `docs verify` fails in both directions - an undocumented
   command and a documented non-command are equally reported. Write the rows
   without citing an issue number: `docs verify` also refuses any `.md` or
   `.template` `dist export` ships that carries one, because the tracker exists
   only in this repo (`instructions/dev/issue-tracking.md` § Citing an issue in
   the repo).
4. Add tests. Pure logic belongs in a function separate from the Typer callback
   (see `mass_update_gate_message`, `derive_run_key`, `run_export`), so a test
   does not need a CLI runner - and a Typer callback called directly from a test
   receives `OptionInfo` objects, not values, for any argument the test omits.
5. Raise the version: `wikitool version bump --minor --title "..."` for a new
   command (`--patch` for a fix, `--major` when the new version is **not a
   drop-in replacement** - a renamed flag or artefact, a stricter check that
   newly fails on **shipped content an instance already had** (a `kb/` page,
   an `instructions/*.md` file), anything needing hand-work after the copy).
   A new command that is merely pickier about its *own* fresh input - a flag
   it did not previously accept, a write it now refuses without more from the
   caller - is the ordinary MINOR case: nothing an instance already has stops
   validating, there is simply more to say when the command is next invoked.
   Content migration is one way to land in the MAJOR row, not the definition
   of it: a `--major` may well ship `--no-migration`, and one that does
   migrate also needs a document under `instructions/migrations/`. The full
   test is `instructions/dev/version-parts.md` - read it before choosing
   `--major`.
   A new command reaches every future instance, and CI's version gate refuses a
   stack change that moved no version.

Every command is counted against the iteration budget unless it is listed in
`run_budget.SKIP_COMMANDS` / `SKIP_COMMAND_PATHS`. Only read-only retrieval
belongs there.

## Design notes

**Deterministic by default.** Anything an LLM would otherwise re-derive -
frontmatter, index statistics, cross-reference bookkeeping, log formatting,
version arithmetic - is computed here so it comes out the same every time.

**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the
Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
prompt-level limit is one an agent can talk itself past. Exemption lists are
constants, never flags.

**Clearance is an exit code, not an instruction.** `EXIT_NEEDS_CLEARANCE`
(42, in `commands/_util.py`) is a third outcome beside success and validation
error, meaning "a human has to see this output first". The command's own
message carries the reason, the evidence and the exact `--confirm <token>`
re-run line; the instruction layer deliberately holds none of it, because a
procedure written down in advance is one an agent can complete alone. Whether
a human *actually* saw it is not enforced here - that question is answered in
the eval layer (`evals/trajectory.py`, `clearance-ended-the-turn`).

**Nothing locates a region by its prose.** `xref` owns the links region and `cite` the footnotes
region, and each is delimited by a `<!-- wikitool:<name> -->` marker pair (`blocks.py`). The
heading inside is rendered from `kb/CONVENTIONS.md` and is replaced along with the rest of the
region on every write - so no heading text exists in Python, and changing the declaration cannot
split a page.

Both halves of that mattered. Matching on the heading made the KB language a compiler constant;
*guessing* where the region ended - at the next heading, and before that at the end of the file -
silently deleted content sitting after it on eight pages. `migrate verify` compares marker-pair
counts for the same reason it compares wikilink counts: a dropped marker is invisible otherwise.

**A relationship label is data, not prose.** `related:` carries `- <label>: <target>`
(`links.py`), the label drawn from `instructions/link-taxonomy.md` and authorised per
destination by the *source* collection's `outbound:` block. The body bullet is a rendering of
that, which is what removed the need to parse a German phrase back into a relationship - and why
the vocabulary can be checked at all, after drifting to 152 distinct labels while it could not
be. Both readers accept a bare title as an unlabelled edge: that is the shape a page is in
between the machinery landing and the migration reaching it, and `lint` is what reports it.

**An edge is authored in one direction.** `xref add` writes one, on the asserting page. The
inbound view is rendered from the graph rather than stored, so navigation does not depend on
anyone writing a mirror - and per-collection authorisation stays coherent, which it cannot be if
the tool writes edges into a collection whose rules the author never read.

**Generated output is never committed.** `reports/`, `.agents/skills/` and
`.claude/skills/` are build output; `docs verify` carries canaries in both
directions so an ignore rule can neither swallow tracked content nor stop
ignoring generated copies.

## Tests

```bash
cd tools && .venv/bin/python -m pytest -q
```

`pytest.ini` sets the import path. Tests use `tmp_path` fixtures and monkeypatch
`config` paths rather than touching the real `kb/`.

Coverage is optional locally and measured on every CI run:

```bash
.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov
```

`pytest-cov` is deliberately not in `requirements.txt`: that file is what an *instance*
needs at runtime and ships with `dist export`, and an instance does not measure this
suite. Configuration is `.coveragerc` (coverage.py does not read `pytest.ini`), which
measures `chemenu/` without `chemenu/tests/` and sets no threshold - see EVALS.md
for the measured number and how to read it.

**Two exceptions, and they bite.** `test_types_cmd.py` and `test_index_build.py` resolve the real
`types/` through the shared `resolver`, because what they assert *is* that behaviour comes from the
type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look
unrelated to it - translating the wiki broke six that way. Anything those tests pin must be
structural: `dir:` values and layout order are asserted exactly, display titles are read from the
spec at assert time, never written out. The same rule applies to the template fence - anchor on a
section name from `sections`, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test.
Otherwise it is a tripwire that fires on an edit nobody connected to the test.