chemenu/tools/README.md

tools/

Developer documentation for wikitool - how the CLI is built, how to change it, and how to run its tests.

This is not the command reference. That is CONTRACT.md, which wikitool docs verify checks against the registered commands. Copying the command table here would create a second copy that drifts, so this file deliberately has none - and docs verify now enforces that.

Document	Audience
README.md (this file)	Humans working on wikitool
CONTRACT.md	Agents working with wikitool - commands, error contracts, maintenance schedule
../AGENTS.md	The invariants that say when using a command is mandatory
../CHANGES.md	What changed in the stack, and when

Setup

cd tools
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

jsonschema and PyYAML are hard dependencies, not optional extras: schema validation is the tool's whole safety net, so cli.py fails loudly with the install command rather than degrading silently.

Layout

tools/
  wikitool                 entry point
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate
    config.py              repo layout: root resolution and every path under it
    api.py                 the in-process entry point - point Chemenu at a corpus and read it
    errors.py              ChemenuError / ValidationError / BackendError
    corpus_cache.py        one parsed corpus per commit, never cached while the tree is dirty
    kb_scan.py             page iteration/loading over kb/
    kb_collections.py      collection discovery (a directory with COLLECTION.md), and what one declares about itself
    conventions.py         kb/CONVENTIONS.md: what this instance decided about authoring, as opposed to what the stack enforces
    type_resolver.py       type-spec loading and schema resolution
    lint_core.py           the lint checks and the report, with no CLI attached
    types_core.py          type-spec listing/description, with no CLI attached
    sections.py            the section headings the tool reads and writes in a page body
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    search/                pluggable search backends, plus service.py - the search core
    commands/              one module per command or command group: the terminal adapters
    tests/                 pytest suite

Two consumers, one core. The CLI is not the only caller any more. The cores (search/service.py, lint_core.py, types_core.py) hold what decides an answer and import no typer and no rich; the modules under commands/ turn those values into terminal output and those exceptions into exit codes. api.Corpus is the in-process entry point over the same functions - it takes a corpus root, returns exactly the structures the --json forms print, and raises instead of exiting. A second consumer is therefore a second adapter rather than a second implementation, and the write commands are unreachable from api because nothing under commands/ is imported there.

Which corpus. The root resolves by precedence: an explicit argument, then $CHEMENU_ROOT, then a walk up from the package's own location. The walk-up is the default, so the CLI is unaffected by any of this. Nothing under the root is bound at import time - KB_DIR and friends follow whatever ROOT currently is.

Adding a command

1. Write the module under chemenu/commands/. A group is a typer.Typer() app; a single command is a plain function.
2. Register it in cli.py (app.add_typer(...) or app.command(...)).
3. Add a row to CONTRACT.md's command table and to its error contract table. docs verify fails in both directions - an undocumented command and a documented non-command are equally reported.
4. Add tests. Pure logic belongs in a function separate from the Typer callback (see mass_update_gate_message, derive_run_key, run_export), so a test does not need a CLI runner - and a Typer callback called directly from a test receives OptionInfo objects, not values, for any argument the test omits.
5. Raise the version: wikitool version bump --minor --title "..." for a new command (--patch for a fix, --major when existing content has to be migrated, which then also needs a document under instructions/migrations/). A new command reaches every future instance, and CI's version gate refuses a stack change that moved no version.

Every command is counted against the iteration budget unless it is listed in run_budget.SKIP_COMMANDS / SKIP_COMMAND_PATHS. Only read-only retrieval belongs there.

Design notes

Deterministic by default. Anything an LLM would otherwise re-derive - frontmatter, index statistics, cross-reference bookkeeping, log formatting, confidence arithmetic - is computed here so it comes out the same every time.

Gates are code, not prompts. The Mass-Update Gate (git_publish.py) and the Iteration Budget Gate (run_budget.py) refuse in-process, because a prompt-level limit is one an agent can talk itself past. Exemption lists are constants, never flags.

Clearance is an exit code, not an instruction. EXIT_NEEDS_CLEARANCE (42, in commands/_util.py) is a third outcome beside success and validation error, meaning "a human has to see this output first". The command's own message carries the reason, the evidence and the exact --confirm <token> re-run line; the instruction layer deliberately holds none of it, because a procedure written down in advance is one an agent can complete alone. Whether a human actually saw it is not enforced here - that question is answered in the eval layer (evals/trajectory.py, clearance-ended-the-turn).

Section names are a vocabulary, not literals - and not the stack's. xref add writes into Relationships and See Also, and cite add owns the trailing Footnotes block, so those three headings are structure the tool matches on. Which words they are is the corpus's own answer: conventions.py reads them from kb/CONVENTIONS.md, sections.py resolves them on access (PEP 562, the way config resolves its paths), and no heading text is written down in Python except the pre-conventions fallback for an instance that has not declared one yet.

Each slot has one canonical spelling - what the tool writes - plus aliases it still recognizes. That asymmetry is what let the wiki be translated page by page instead of atomically: an untranslated ## Relationships is still found and appended to. Dropping an alias is therefore a breaking change for any page not yet converted, not a cleanup. Renaming a heading is a migration's job; no other command may do it as a side effect (see cite_block_heading in provenance.py, which exists solely so cite sync stays a no-op on an untranslated page).

Because the value is resolved rather than bound, nothing may capture it at import time - not a module constant, not an evaluated default argument. That is why provenance.CITE_BLOCK_HEADING is a module __getattr__ and render_cite_block(heading=None) resolves inside the call.

Generated output is never committed. reports/, .agents/skills/ and .claude/skills/ are build output; docs verify carries canaries in both directions so an ignore rule can neither swallow tracked content nor stop ignoring generated copies.

Tests

cd tools && .venv/bin/python -m pytest -q

pytest.ini sets the import path. Tests use tmp_path fixtures and monkeypatch config paths rather than touching the real kb/.

Coverage is optional locally and measured on every CI run:

.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov

pytest-cov is deliberately not in requirements.txt: that file is what an instance needs at runtime and ships with dist export, and an instance does not measure this suite. Configuration is .coveragerc (coverage.py does not read pytest.ini), which measures chemenu/ without chemenu/tests/ and sets no threshold - see EVALS.md for the measured number and how to read it.

Two exceptions, and they bite. test_types_cmd.py and test_index_build.py resolve the real types/ through the shared resolver, because what they assert is that behaviour comes from the type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look unrelated to it - translating the wiki broke six that way. Anything those tests pin must be structural: dir: values and layout order are asserted exactly, display titles are read from the spec at assert time, never written out. The same rule applies to the template fence - anchor on a section name from sections, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test. Otherwise it is a tripwire that fires on an edit nobody connected to the test.