chemenu/tools/README.md

# tools/

Developer documentation for `wikitool` - how the CLI is built, how to change it,
and how to run its tests.

**This is not the command reference.** That is [CONTRACT.md](CONTRACT.md), which
`wikitool docs verify` checks against the registered commands. Copying the
command table here would create a second copy that drifts, so this file
deliberately has none - and `docs verify` now enforces that.

| Document | Audience |
|---|---|
| `README.md` (this file) | Humans working *on* wikitool |
| [`CONTRACT.md`](CONTRACT.md) | Agents working *with* wikitool - commands, error contracts, maintenance schedule |
| [`../AGENTS.md`](../AGENTS.md) | The invariants that say when using a command is mandatory |
| [`../CHANGES.md`](../CHANGES.md) | What changed in the stack, and when |

## Setup

```bash
cd tools
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

`jsonschema` and `PyYAML` are hard dependencies, not optional extras: schema
validation is the tool's whole safety net, so `cli.py` fails loudly with the
install command rather than degrading silently.

## Layout

```
tools/
  wikitool                 entry point
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate
    config.py              repo layout: root resolution and every path under it
    api.py                 the in-process entry point - point Chemenu at a corpus and read it
    errors.py              ChemenuError / ValidationError / BackendError
    corpus_cache.py        one parsed corpus per commit, never cached while the tree is dirty
    kb_scan.py             page iteration/loading over kb/
    kb_collections.py      collection discovery (a directory with COLLECTION.md), and what one declares about itself
    conventions.py         kb/CONVENTIONS.md: what this instance decided about authoring, as opposed to what the stack enforces
    type_resolver.py       type-spec loading and schema resolution
    lint_core.py           the lint checks and the report, with no CLI attached
    types_core.py          type-spec listing/description, with no CLI attached
    sections.py            the section headings the tool reads and writes in a page body
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    search/                pluggable search backends, plus service.py - the search core
    commands/              one module per command or command group: the terminal adapters
    tests/                 pytest suite
```

**Two consumers, one core.** The CLI is not the only caller any more. The cores
(`search/service.py`, `lint_core.py`, `types_core.py`) hold what decides an
answer and import no `typer` and no `rich`; the modules under `commands/` turn
those values into terminal output and those exceptions into exit codes.
`api.Corpus` is the in-process entry point over the same functions - it takes a
corpus root, returns exactly the structures the `--json` forms print, and
raises instead of exiting. A second consumer is therefore a second adapter
rather than a second implementation, and the write commands are unreachable
from `api` because nothing under `commands/` is imported there.

**Which corpus.** The root resolves by precedence: an explicit argument, then
`$CHEMENU_ROOT`, then a walk up from the package's own location. The walk-up is
the default, so the CLI is unaffected by any of this. Nothing under the root is
bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.

## Adding a command

1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
   app; a single command is a plain function.
2. Register it in `cli.py` (`app.add_typer(...)` or `app.command(...)`).
3. Add a row to [CONTRACT.md](CONTRACT.md)'s command table **and** to its error
   contract table. `docs verify` fails in both directions - an undocumented
   command and a documented non-command are equally reported.
4. Add tests. Pure logic belongs in a function separate from the Typer callback
   (see `mass_update_gate_message`, `derive_run_key`, `run_export`), so a test
   does not need a CLI runner - and a Typer callback called directly from a test
   receives `OptionInfo` objects, not values, for any argument the test omits.
5. Raise the version: `wikitool version bump --minor --title "..."` for a new
   command (`--patch` for a fix, `--major` when existing content has to be
   migrated, which then also needs a document under `instructions/migrations/`).
   A new command reaches every future instance, and CI's version gate refuses a
   stack change that moved no version.

Every command is counted against the iteration budget unless it is listed in
`run_budget.SKIP_COMMANDS` / `SKIP_COMMAND_PATHS`. Only read-only retrieval
belongs there.

## Design notes

**Deterministic by default.** Anything an LLM would otherwise re-derive -
frontmatter, index statistics, cross-reference bookkeeping, log formatting,
confidence arithmetic - is computed here so it comes out the same every time.

**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the
Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
prompt-level limit is one an agent can talk itself past. Exemption lists are
constants, never flags.

**Clearance is an exit code, not an instruction.** `EXIT_NEEDS_CLEARANCE`
(42, in `commands/_util.py`) is a third outcome beside success and validation
error, meaning "a human has to see this output first". The command's own
message carries the reason, the evidence and the exact `--confirm <token>`
re-run line; the instruction layer deliberately holds none of it, because a
procedure written down in advance is one an agent can complete alone. Whether
a human *actually* saw it is not enforced here - that question is answered in
the eval layer (`evals/trajectory.py`, `clearance-ended-the-turn`).

**Section names are a vocabulary, not literals - and not the stack's.** `xref add` writes into
Relationships and See Also, and `cite add` owns the trailing Footnotes block, so those three
headings are structure the tool matches on. *Which words they are* is the corpus's own answer:
`conventions.py` reads them from `kb/CONVENTIONS.md`, `sections.py` resolves them on access
(PEP 562, the way `config` resolves its paths), and no heading text is written down in Python
except the pre-conventions fallback for an instance that has not declared one yet.

Each slot has one canonical spelling - what the tool writes - plus aliases it still recognizes.
That asymmetry is what let the wiki be translated page by page instead of atomically: an
untranslated `## Relationships` is still found and appended to. Dropping an alias is therefore a
breaking change for any page not yet converted, not a cleanup. Renaming a heading is a
migration's job; no other command may do it as a side effect (see `cite_block_heading` in
`provenance.py`, which exists solely so `cite sync` stays a no-op on an untranslated page).

Because the value is resolved rather than bound, nothing may capture it at import time - not a
module constant, not an evaluated default argument. That is why `provenance.CITE_BLOCK_HEADING`
is a module `__getattr__` and `render_cite_block(heading=None)` resolves inside the call.

**Generated output is never committed.** `reports/`, `.agents/skills/` and
`.claude/skills/` are build output; `docs verify` carries canaries in both
directions so an ignore rule can neither swallow tracked content nor stop
ignoring generated copies.

## Tests

```bash
cd tools && .venv/bin/python -m pytest -q
```

`pytest.ini` sets the import path. Tests use `tmp_path` fixtures and monkeypatch
`config` paths rather than touching the real `kb/`.

Coverage is optional locally and measured on every CI run:

```bash
.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov
```

`pytest-cov` is deliberately not in `requirements.txt`: that file is what an *instance*
needs at runtime and ships with `dist export`, and an instance does not measure this
suite. Configuration is `.coveragerc` (coverage.py does not read `pytest.ini`), which
measures `chemenu/` without `chemenu/tests/` and sets no threshold - see EVALS.md
for the measured number and how to read it.

**Two exceptions, and they bite.** `test_types_cmd.py` and `test_index_build.py` resolve the real
`types/` through the shared `resolver`, because what they assert *is* that behaviour comes from the
type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look
unrelated to it - translating the wiki broke six that way. Anything those tests pin must be
structural: `dir:` values and layout order are asserted exactly, display titles are read from the
spec at assert time, never written out. The same rule applies to the template fence - anchor on a
section name from `sections`, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test.
Otherwise it is a tripwire that fires on an edit nobody connected to the test.