chemenu/tools/README.md

# tools/

Developer documentation for `wikitool` - how the CLI is built, how to change it,
and how to run its tests.

**This is not the command reference.** That is [CONTRACT.md](CONTRACT.md), which
`wikitool docs verify` checks against the registered commands. Copying the
command table here would create a second copy that drifts, so this file
deliberately has none - and `docs verify` now enforces that.

| Document | Audience |
|---|---|
| `README.md` (this file) | Humans working *on* wikitool |
| [`CONTRACT.md`](CONTRACT.md) | Agents working *with* wikitool - commands, error contracts, maintenance schedule |
| [`../AGENTS.md`](../AGENTS.md) | The invariants that say when using a command is mandatory |
| [`../CHANGES.md`](../CHANGES.md) | What changed in the stack, and when |

## Setup

```bash
cd tools
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

`jsonschema` and `PyYAML` are hard dependencies, not optional extras: schema
validation is the tool's whole safety net, so `cli.py` fails loudly with the
install command rather than degrading silently.

## Layout

```
tools/
  wikitool                 entry point
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate
    config.py              repo layout constants (ROOT, RAW_DIR, KB_DIR, WORK_DIR, ...)
    kb_scan.py             page iteration/loading over kb/
    kb_collections.py      collection discovery (a directory with COLLECTION.md)
    type_resolver.py       type-spec loading and schema resolution
    sections.py            the section headings the tool reads and writes in a page body
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    search/                pluggable search backends (base protocol, ripgrep, filters, fuse)
    commands/              one module per command or command group
    tests/                 pytest suite
```

## Adding a command

1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
   app; a single command is a plain function.
2. Register it in `cli.py` (`app.add_typer(...)` or `app.command(...)`).
3. Add a row to [CONTRACT.md](CONTRACT.md)'s command table **and** to its error
   contract table. `docs verify` fails in both directions - an undocumented
   command and a documented non-command are equally reported.
4. Add tests. Pure logic belongs in a function separate from the Typer callback
   (see `mass_update_gate_message`, `derive_run_key`, `run_export`), so a test
   does not need a CLI runner - and a Typer callback called directly from a test
   receives `OptionInfo` objects, not values, for any argument the test omits.
5. Raise the version: `wikitool version bump --minor --title "..."` for a new
   command (`--patch` for a fix, `--major` when existing content has to be
   migrated, which then also needs a document under `instructions/migrations/`).
   A new command reaches every future instance, and CI's version gate refuses a
   stack change that moved no version.

Every command is counted against the iteration budget unless it is listed in
`run_budget.SKIP_COMMANDS` / `SKIP_COMMAND_PATHS`. Only read-only retrieval
belongs there.

## Design notes

**Deterministic by default.** Anything an LLM would otherwise re-derive -
frontmatter, index statistics, cross-reference bookkeeping, log formatting,
confidence arithmetic - is computed here so it comes out the same every time.

**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the
Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
prompt-level limit is one an agent can talk itself past. Exemption lists are
constants, never flags.

**Clearance is an exit code, not an instruction.** `EXIT_NEEDS_CLEARANCE`
(42, in `commands/_util.py`) is a third outcome beside success and validation
error, meaning "a human has to see this output first". The command's own
message carries the reason, the evidence and the exact `--confirm <token>`
re-run line; the instruction layer deliberately holds none of it, because a
procedure written down in advance is one an agent can complete alone. Whether
a human *actually* saw it is not enforced here - that question is answered in
the eval layer (`evals/trajectory.py`, `clearance-ended-the-turn`).

**Section names are a vocabulary, not literals.** `xref add` writes into Relationships and See
Also, and `cite add` owns the trailing Footnotes block, so those three headings are structure the
tool matches on. They are named once in `sections.py`, and each has one canonical spelling - what
the tool writes - plus aliases it still recognizes. That asymmetry is what let the wiki be
translated page by page instead of atomically: an untranslated `## Relationships` is still found
and appended to. Dropping an alias is therefore a breaking change for any page not yet converted,
not a cleanup. Renaming a heading is a migration's job; no other command may do it as a side
effect (see `cite_block_heading` in `provenance.py`, which exists solely so `cite sync` stays a
no-op on an untranslated page).

**Generated output is never committed.** `reports/`, `.agents/skills/` and
`.claude/skills/` are build output; `docs verify` carries canaries in both
directions so an ignore rule can neither swallow tracked content nor stop
ignoring generated copies.

## Tests

```bash
cd tools && .venv/bin/python -m pytest -q
```

`pytest.ini` sets the import path. Tests use `tmp_path` fixtures and monkeypatch
`config` paths rather than touching the real `kb/`.

Coverage is optional locally and measured on every CI run:

```bash
.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov
```

`pytest-cov` is deliberately not in `requirements.txt`: that file is what an *instance*
needs at runtime and ships with `dist export`, and an instance does not measure this
suite. Configuration is `.coveragerc` (coverage.py does not read `pytest.ini`), which
measures `chemenu/` without `chemenu/tests/` and sets no threshold - see EVALS.md
for the measured number and how to read it.

**Two exceptions, and they bite.** `test_types_cmd.py` and `test_index_build.py` resolve the real
`types/` through the shared `resolver`, because what they assert *is* that behaviour comes from the
type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look
unrelated to it - translating the wiki broke six that way. Anything those tests pin must be
structural: `dir:` values and layout order are asserted exactly, display titles are read from the
spec at assert time, never written out. The same rule applies to the template fence - anchor on a
section name from `sections`, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test.
Otherwise it is a tripwire that fires on an edit nobody connected to the test.