chemenu/tools/README.md

# tools/

Developer documentation for `wikitool` - how the CLI is built, how to change it,
and how to run its tests.

**This is not the command reference.** That is [CONTRACT.md](CONTRACT.md), which
`wikitool docs verify` checks against the registered commands. Copying the
command table here would create a second copy that drifts, so this file
deliberately has none - and `docs verify` now enforces that.

| Document | Audience |
|---|---|
| `README.md` (this file) | Humans working *on* wikitool |
| [`CONTRACT.md`](CONTRACT.md) | Agents working *with* wikitool - commands, error contracts, maintenance schedule |
| [`../AGENTS.md`](../AGENTS.md) | The invariants that say when using a command is mandatory |
| [`../CHANGES.md`](../CHANGES.md) | What changed in the stack, and when |

## Setup

```bash
cd tools
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

`jsonschema` and `PyYAML` are hard dependencies, not optional extras: schema
validation is the tool's whole safety net, so `cli.py` fails loudly with the
install command rather than degrading silently.

## Layout

```
tools/
  wikitool                 entry point
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate
    config.py              repo layout: root resolution and every path under it
    api.py                 the in-process entry point - point Chemenu at a corpus and read it
    errors.py              ChemenuError / ValidationError / BackendError
    corpus_cache.py        one parsed corpus per commit, never cached while the tree is dirty
    kb_scan.py             page iteration/loading over kb/
    blocks.py              generated regions in a page body, found by marker rather than by heading
    links.py               labelled edges in `related:` - the graph's semantics as data, not prose
    kb_collections.py      collection discovery (a directory with COLLECTION.md), and what one declares about itself
    conventions.py         kb/CONVENTIONS.md: what this instance decided about authoring, as opposed to what the stack enforces
    ownership.py           the stack-vs-instance boundary under a content stage - one predicate, read by `dist_cmd.py` and `commands/upstream_cmd.py` so the two cannot answer it differently
    type_resolver.py       type-spec loading and schema resolution
    catalog.py             how the corpus groups into collections and areas, and the shard threshold - with no CLI attached
    lint_core.py           the lint checks and the report, with no CLI attached
    types_core.py          type-spec listing/description, with no CLI attached
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    search/                pluggable search backends, plus service.py - the search core
    commands/              one module per command or command group: the terminal adapters
    tests/                 pytest suite
```

**Two consumers, one core.** The CLI is not the only caller any more. The cores
(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`) hold what
decides an answer and import no `typer` and no `rich`; the modules under `commands/` turn
those values into terminal output and those exceptions into exit codes.
`api.Corpus` is the in-process entry point over the same functions - it takes a
corpus root, returns exactly the structures the `--json` forms print, and
raises instead of exiting. A second consumer is therefore a second adapter
rather than a second implementation, and the write commands are unreachable
from `api` because nothing under `commands/` is imported there.

**Which corpus.** The root resolves by precedence: an explicit argument, then
`$CHEMENU_ROOT`, then a walk up from the package's own location. The walk-up is
the default, so the CLI is unaffected by any of this. Nothing under the root is
bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.

## Adding a command

1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
   app; a single command is a plain function.
2. Register it in `cli.py` (`app.add_typer(...)` or `app.command(...)`).
3. Add a row to [CONTRACT.md](CONTRACT.md)'s command table **and** to its error
   contract table. `docs verify` fails in both directions - an undocumented
   command and a documented non-command are equally reported. Write the rows
   without citing an issue number: `docs verify` also refuses any `.md` or
   `.template` `dist export` ships that carries one, because the tracker exists
   only in this repo (`instructions/dev/issue-tracking.md` § Citing an issue in
   the repo).
4. Add tests. Pure logic belongs in a function separate from the Typer callback
   (see `mass_update_gate_message`, `derive_run_key`, `run_export`), so a test
   does not need a CLI runner - and a Typer callback called directly from a test
   receives `OptionInfo` objects, not values, for any argument the test omits.
5. Raise the version: `wikitool version bump --minor --title "..."` for a new
   command (`--patch` for a fix, `--major` when existing content has to be
   migrated, which then also needs a document under `instructions/migrations/`).
   A new command reaches every future instance, and CI's version gate refuses a
   stack change that moved no version.

Every command is counted against the iteration budget unless it is listed in
`run_budget.SKIP_COMMANDS` / `SKIP_COMMAND_PATHS`. Only read-only retrieval
belongs there.

## Design notes

**Deterministic by default.** Anything an LLM would otherwise re-derive -
frontmatter, index statistics, cross-reference bookkeeping, log formatting,
confidence arithmetic - is computed here so it comes out the same every time.

**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the
Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
prompt-level limit is one an agent can talk itself past. Exemption lists are
constants, never flags.

**Clearance is an exit code, not an instruction.** `EXIT_NEEDS_CLEARANCE`
(42, in `commands/_util.py`) is a third outcome beside success and validation
error, meaning "a human has to see this output first". The command's own
message carries the reason, the evidence and the exact `--confirm <token>`
re-run line; the instruction layer deliberately holds none of it, because a
procedure written down in advance is one an agent can complete alone. Whether
a human *actually* saw it is not enforced here - that question is answered in
the eval layer (`evals/trajectory.py`, `clearance-ended-the-turn`).

**Nothing locates a region by its prose.** `xref` owns the links region and `cite` the footnotes
region, and each is delimited by a `<!-- wikitool:<name> -->` marker pair (`blocks.py`). The
heading inside is rendered from `kb/CONVENTIONS.md` and is replaced along with the rest of the
region on every write - so no heading text exists in Python, and changing the declaration cannot
split a page.

Both halves of that mattered. Matching on the heading made the KB language a compiler constant;
*guessing* where the region ended - at the next heading, and before that at the end of the file -
silently deleted content sitting after it on eight pages. `migrate verify` compares marker-pair
counts for the same reason it compares wikilink counts: a dropped marker is invisible otherwise.

**A relationship label is data, not prose.** `related:` carries `- <label>: <target>`
(`links.py`), the label drawn from `instructions/link-taxonomy.md` and authorised per
destination by the *source* collection's `outbound:` block. The body bullet is a rendering of
that, which is what removed the need to parse a German phrase back into a relationship - and why
the vocabulary can be checked at all, after drifting to 152 distinct labels while it could not
be. Both readers accept a bare title as an unlabelled edge: that is the shape a page is in
between the machinery landing and the migration reaching it, and `lint` is what reports it.

**An edge is authored in one direction.** `xref add` writes one, on the asserting page. The
inbound view is rendered from the graph rather than stored, so navigation does not depend on
anyone writing a mirror - and per-collection authorisation stays coherent, which it cannot be if
the tool writes edges into a collection whose rules the author never read.

**Generated output is never committed.** `reports/`, `.agents/skills/` and
`.claude/skills/` are build output; `docs verify` carries canaries in both
directions so an ignore rule can neither swallow tracked content nor stop
ignoring generated copies.

## Tests

```bash
cd tools && .venv/bin/python -m pytest -q
```

`pytest.ini` sets the import path. Tests use `tmp_path` fixtures and monkeypatch
`config` paths rather than touching the real `kb/`.

Coverage is optional locally and measured on every CI run:

```bash
.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov
```

`pytest-cov` is deliberately not in `requirements.txt`: that file is what an *instance*
needs at runtime and ships with `dist export`, and an instance does not measure this
suite. Configuration is `.coveragerc` (coverage.py does not read `pytest.ini`), which
measures `chemenu/` without `chemenu/tests/` and sets no threshold - see EVALS.md
for the measured number and how to read it.

**Two exceptions, and they bite.** `test_types_cmd.py` and `test_index_build.py` resolve the real
`types/` through the shared `resolver`, because what they assert *is* that behaviour comes from the
type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look
unrelated to it - translating the wiki broke six that way. Anything those tests pin must be
structural: `dir:` values and layout order are asserted exactly, display titles are read from the
spec at assert time, never written out. The same rule applies to the template fence - anchor on a
section name from `sections`, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test.
Otherwise it is a tripwire that fires on an edit nobody connected to the test.