chemenu/tools

tools/

Developer documentation for wikitool - how the CLI is built, how to change it, and how to run its tests.

This is not the command reference. That is CONTRACT.md, which wikitool docs verify checks against the registered commands. Copying its generated command records here would create a second copy that drifts, so this file deliberately has none - and docs verify now enforces that.

Document	Audience
README.md (this file)	Humans working on wikitool
CONTRACT.md	Agents working with wikitool - commands, error contracts, maintenance schedule
../AGENTS.md	The invariants that say when using a command is mandatory
../CHANGES.md	What changed in the stack, and when

Setup

tools/preflight.sh      # from the repo root

pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1   # PowerShell 7 on Windows

The preflight is the one way a checkout gets its environment: it checks the tools listed in prerequisites.txt, records their absolute paths in ../.wikitool-tools.json, creates .venv and installs requirements.txt into it with -m pip. It is POSIX sh because it has to run before Python is known to exist; exit 42 means the user has to act, and its output says how. The procedure an agent follows around it is instructions/preflight.md.

Each release also attaches preflight.sh and preflight.ps1 as assets, for the first install before any tree exists. release.yml writes the download address of that same release into two placeholder lines of the copy (RELEASE_ARCHIVE_URL/RELEASE_CHECKSUM_URL in the shell script, $ReleaseArchiveUrl/$ReleaseChecksumUrl in the PowerShell one; the tree copy leaves them empty). With no prerequisites.txt beside it, the script is in asset mode: it downloads the tarball and its .sha256, refuses on a mismatch, reads the folder limit out of the archive, unpacks into the folder it lies in (--into <path> to choose; a target holding anything but the script and a .git is refused), and runs the tree copy of itself, passing --set and its exit code through. It unpacks into a .chemenu-unpack.<pid> folder inside the target and then moves each top-level entry up; a move that fails is retried for about 30 seconds, each time from a fresh copy of the entry out of the archive, and one that still fails makes the script take back everything it moved before it exits 1. --archive <tarball> takes a local tarball instead of a download (its .sha256 has to lie next to it); that is also how the tests drive it, and CHEMENU_PREFLIGHT_MOVE_FAIL stands in for a move that fails.

tools/wikitool refuses to start (exit 42) until the preflight has written a complete .wikitool-tools.json and the venv exists. Past that, every git and rg the package starts goes through chemenu/toolpaths.py, which reads the recorded path - or, with no file at all (the test suite, a bare python -m chemenu.cli), falls back to the bare name. A file that is present but names a path that has gone raises ToolPathError, which the CLI turns into an ERROR line pointing at the preflight rather than a traceback. An optional tool - the 1Password CLI op, recorded only when the preflight found it - is asked for through toolpaths.optional(), which answers None when the file does not name it; the caller says what that costs.

The package runs natively on Windows as well, which CI never does. So the rules that keep it portable are held by reading the source, in the origin repository's tests/test_portability.py: a path that becomes a string goes through .as_posix(), a text file is opened with encoding= and written with newline="\n", a subprocess call with text=True names encoding="utf-8", and only filelock.py imports fcntl or msvcrt. An rg path comes back with \ on Windows even with --path-separator /, so search/ripgrep.py converts it where it parses the JSON. .gitattributes keeps every text file LF in a checkout, raw/ and incoming/ excepted.

jsonschema and PyYAML are hard dependencies, not optional extras: schema validation is the tool's whole safety net, so cli.py fails loudly with the fix rather than degrading silently. pypdf and olefile are in requirements.txt too but are imported only when a PDF or an Outlook .msg is derived, so a missing one stops that format alone; doctor reports each.

Layout

tools/
  wikitool                 entry point (POSIX sh): stops with exit 42 until the preflight has passed
  wikitool.ps1             the same entry point for PowerShell 7, which resolves `tools/wikitool` to this file first
  run_wikitool.py          what the launcher runs with the venv's Python - puts chemenu on sys.path without PYTHONPATH, sets stdout/stderr to UTF-8
  preflight.sh             checks prerequisites.txt, records .wikitool-tools.json, creates .venv (POSIX sh); as the release asset, downloads the stack and unpacks it into its own (empty) folder first
  preflight.ps1            the same for PowerShell 7; also checks the execution policy and the Mark of the Web
  prerequisites.txt        what the machine needs, one `|`-separated line per tool - read by the preflight and `doctor`
  trace-hook               what the harness hooks call: trace_ingest.py under the venv's Python
  trace-hook.ps1           the same for PowerShell, which resolves `./tools/trace-hook` to this file first - without it Windows asks which app opens the sh script
  bugreport                entry point for the bug-report collector (POSIX sh): finds a Python 3.8+ on PATH itself, skipping the Microsoft Store aliases, and runs bugreport.py - no preflight needed
  bugreport.ps1            the same for PowerShell, which resolves `tools/bugreport` to this file first; keeps to what Windows PowerShell 5.1 understands
  bugreport.py             the bug-report collector itself - see below
  chemenu/
    cli.py                 Typer app: registers every command, runs the budget gate, renders `-h`/`--help` from cli_contract
    cli_contract.py         one data record per command (name, synopsis, properties, exit status) - the source `-h`, the index, CONTRACT.md's generated region and the human reference under docs/reference/ render from
    config.py              repo layout: root resolution and every path under it
    config_shapes.py       one record per hand-written .wikitool-*.json (telemetry, remotes, upload, tasks): what its loader accepts, its defaults and `Expected:` text, and what docs/reference/configuration.md renders; also the sections of the tiered settings, each field with its tier; stdlib-only, since the telemetry policy reads it
    settings.py            the tiered settings: wikitool.json (instance), .wikitool-local.json (checkout), secrets by reference - placement checked per field, references resolved on access through WIKITOOL_SECRET_<NAME>, the secrets file or `op read`
    api.py                 the in-process entry point - point Chemenu at a corpus and read it
    errors.py              ChemenuError / ValidationError / BackendError
    toolpaths.py           where git and rg are started from: .wikitool-tools.json, bare name only without the file; an optional tool such as `op` is None when not recorded
    filelock.py            an exclusive lock on an open file, flock on POSIX and msvcrt on Windows - the only module that imports either
    prerequisites.py       prerequisites.txt read from Python, plus the platform and long-path questions `doctor` asks
    install_doc.py         INSTALL.md held to the instructions: its prerequisites lists generated from prerequisites.txt, its setup questions matched to setup-instance.md's markers
    reference_doc.py       the human reference under docs/reference/: one page per command group, an overview and the exit-42 causes, rendered from the records without what addresses an agent; and, in the origin repo only, the skill catalogue and the page types, rendered from the shipped SKILL.md files and type-specs; and the configuration-file shapes, from config_shapes.py
    corpus_cache.py        one parsed corpus per commit, never cached while the tree is dirty
    kb_scan.py             page iteration/loading over kb/
    blocks.py              generated regions in a page body, found by marker rather than by heading
    links.py               labelled edges in `related:` - the graph's semantics as data, not prose
    kb_collections.py      collection discovery (a directory with COLLECTION.md), and what one declares about itself
    conventions.py         kb/CONVENTIONS.md: what this instance decided about authoring, as opposed to what the stack enforces
    ownership.py           the stack-vs-instance boundary under a content stage - one predicate, so no caller keeps a list of its own
    type_resolver.py       type-spec loading and schema resolution
    catalog.py             how the corpus groups into collections and areas, and the shard threshold - with no CLI attached
    lint_core.py           the lint checks and the report, with no CLI attached
    types_core.py          type-spec listing/description, with no CLI attached
    review.py              the weekly review's five checks - the read-time join of kb/gtd/ against the tracker, with no CLI attached
    markdown_code.py       masks code spans/fences so a page may show wiki notation, not only use it
    version.py             the stack version: VERSION, the release stamp, the compatibility rule
    kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
    corpus_diff.py         invariant comparison of kb/ between two revisions
    web_capture.py         `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`) and one module per website, each registering the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text), standard library only
    filters/               the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
    repo_capture.py        `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
    guideline_export.py    `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
    search/                pluggable search backends, plus service.py - the search core
    tasks/                 the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (the tracker sections of the settings), one module per adapter - no instruction ever learns which provider it is
    commands/              one module per command or command group: the terminal adapters - except `raw/`, a package with one module per subcommand and the helpers several of them share in `_common.py`
    tests/                 pytest suite - origin repository only, `dist export` leaves it out

A script beside the package. bugreport.py is not part of chemenu and imports nothing from it: it is the bug-report collector (../instructions/bug-report.md), and the case it exists for is a checkout where wikitool does not start. It therefore uses the standard library only, keeps to Python 3.8 syntax so that an old interpreter can still run it, and exits 0 or 1 - never 42, since it opens no gate. It has a second mode: --pseudonymise replaces the identities it can read from the machine (stage 1), and --bundle/--candidates applies names a model found in the result (stage 2), both word by word so that length, separators and depth survive. The mapping, the review list and the candidate file stay beside the bundle directory. dist export ships it with the rest of tools/; its tests (tests/test_bugreport.py, in the origin repository) run it on a bare interpreter (-I -S) to keep that promise.

It is started through tools/bugreport (bugreport.ps1 under PowerShell), which assumes no more than the collector does: no preflight, no .wikitool-tools.json, no venv. It looks for the interpreter the way the preflight does - python3, python on PATH; on Windows python, py -3, python3 - then tries the venv's, probes each for 3.8 or newer, and never starts one under WindowsApps, where python3 in Git Bash is the Microsoft Store's alias. Finding none, it exits 1 and says why.

Two consumers, one core. The CLI is not the only caller any more. The cores (search/service.py, lint_core.py, types_core.py, catalog.py, guideline_export.py) hold what decides an answer and import no typer and no rich; the modules under commands/ turn those values into terminal output and those exceptions into exit codes. api.Corpus is the in-process entry point over the same functions - it takes a corpus root, returns exactly the structures the --json forms print, and raises instead of exiting. A second consumer is therefore a second adapter rather than a second implementation, and the write commands are unreachable from api because nothing under commands/ is imported there.

Which corpus. The root resolves by precedence: an explicit argument, then $CHEMENU_ROOT, then a walk up from the package's own location. The walk-up is the default, so the CLI is unaffected by any of this. Nothing under the root is bound at import time - KB_DIR and friends follow whatever ROOT currently is.

Adding a command

1. Write the module under chemenu/commands/. A group is a typer.Typer() app; a single command is a plain function.
2. Register it in cli.py (app.add_typer(...) or app.command(...)).
3. Attach a @cli_contract.record(cli_contract.CommandRecord(...)) decorator to the command function, in its own module, and add its path to the matching group in cli_contract.GROUPS. That one record is the source wikitool <cmd> -h, the index (wikitool -h, and the top of CONTRACT.md), and CONTRACT.md's generated #### <path> section all render from - see cli_contract.py's own module docstring for the record's shape, and the CommandRecord docstring for how its prose is written (NOTES as present-tense bullets, one Failure per cause with its reaction, copyable EXAMPLES, NEVER, SEE ALSO, and no "why" - that goes into a comment next to the code). docs verify fails in both directions - a command with no record and a GROUPS entry naming no real command are equally reported - and also checks that every non-hidden flag appears in the record's SYNOPSIS. Then regenerate both copies: wikitool docs contract --apply for CONTRACT.md and wikitool docs reference --apply for the human reference under docs/reference/commands/. A new group needs its page there first - docs/reference/commands/<group>.md with a title, a one-sentence head and the <!-- wikitool:reference-commands --> marker pair; the command never creates one. Write the record's prose without citing an issue number: docs verify also refuses one in a command's rendered --help text (a docstring above its \f marker, or an option's help=) and in any .md/.template dist export ships, because the tracker exists only in this repo (instructions/dev/issue-tracking.md § Citing an issue in the repo).
4. Add tests. Pure logic belongs in a function separate from the Typer callback (see mass_update_gate_message, derive_run_key, run_export), so a test does not need a CLI runner - and a Typer callback called directly from a test receives OptionInfo objects, not values, for any argument the test omits.
5. Raise the version: wikitool version bump --minor --title "..." for a new command (--patch for a fix, --major when the new version is not a drop-in replacement - a renamed flag or artefact, a stricter check that newly fails on shipped content an instance already had (a kb/ page, an instructions/*.md file), anything needing hand-work after the copy). A new command that is merely pickier about its own fresh input - a flag it did not previously accept, a write it now refuses without more from the caller - is the ordinary MINOR case: nothing an instance already has stops validating, there is simply more to say when the command is next invoked. Content migration is one way to land in the MAJOR row, not the definition of it: a --major may well ship --no-migration, and one that does migrate also needs a document under instructions/migrations/. The full test is instructions/dev/version-parts.md - read it before choosing --major. A new command reaches every future instance, and CI's version gate refuses a stack change that moved no version.

Every command is counted against the iteration budget unless its cli_contract record's budget: property says exempt (or, for version regrade's own shape, exempt_without_args) - run_budget.is_exempt reads it from there, not from a list of its own. Only read-only retrieval earns it.

Design notes

Deterministic by default. Anything an LLM would otherwise re-derive - frontmatter, index statistics, cross-reference bookkeeping, log formatting, version arithmetic - is computed here so it comes out the same every time.

Gates are code, not prompts. The Mass-Update Gate (git_publish.py), the Guideline Push Gate (export_cmd.py) and the Iteration Budget Gate (run_budget.py) refuse in-process, because a prompt-level limit is one an agent can talk itself past. Exemption lists are constants, never flags.

Clearance is an exit code, not an instruction. EXIT_NEEDS_CLEARANCE (42, in commands/_util.py) is a third outcome beside success and validation error, meaning "a human has to see this output first". The command's own message carries the reason, the evidence and the exact --confirm <token> re-run line; the instruction layer deliberately holds none of it, because a procedure written down in advance is one an agent can complete alone. Whether a human actually saw it is not enforced here - that question is answered in the eval layer (evals/trajectory.py, clearance-ended-the-turn).

Nothing locates a region by its prose. xref owns the links region and cite the footnotes region, and each is delimited by a <!-- wikitool:<name> --> marker pair (blocks.py). The heading inside is rendered from kb/CONVENTIONS.md and is replaced along with the rest of the region on every write - so no heading text exists in Python, and changing the declaration cannot split a page.

Both halves of that mattered. Matching on the heading made the KB language a compiler constant; guessing where the region ended - at the next heading, and before that at the end of the file - silently deleted content sitting after it on eight pages. migrate verify compares marker-pair counts for the same reason it compares wikilink counts: a dropped marker is invisible otherwise.

A relationship label is data, not prose. related: carries - <label>: <target> (links.py), the label drawn from instructions/link-taxonomy.md and authorised per destination by the source collection's outbound: block. The body bullet is a rendering of that, which is what removed the need to parse a German phrase back into a relationship - and why the vocabulary can be checked at all, after drifting to 152 distinct labels while it could not be. Both readers accept a bare title as an unlabelled edge: that is the shape a page is in between the machinery landing and the migration reaching it, and lint is what reports it.

An edge is authored in one direction. xref add writes one, on the asserting page. The inbound view is rendered from the graph rather than stored, so navigation does not depend on anyone writing a mirror - and per-collection authorisation stays coherent, which it cannot be if the tool writes edges into a collection whose rules the author never read.

Generated output is never committed. reports/, .agents/skills/ and .claude/skills/ are build output; docs verify carries canaries in both directions so an ignore rule can neither swallow tracked content nor stop ignoring generated copies.

Tests

The suite exists in the origin repository only: dist export leaves out chemenu/tests/, pytest.ini and .coveragerc, because the suite tests that repository's own type-specs and conventions, not an instance's.

cd tools && .venv/bin/python -m pytest -q

pytest.ini sets the import path. Tests use tmp_path fixtures and monkeypatch config paths rather than touching the real kb/.

Coverage is optional locally and measured on every CI run:

.venv/bin/pip install pytest-cov   # one time, CI-only dependency
.venv/bin/python -m pytest -q --cov

pytest-cov is deliberately not in requirements.txt: that file is what an instance needs at runtime and ships with dist export, and an instance does not measure this suite. Configuration is .coveragerc (coverage.py does not read pytest.ini), which measures chemenu/ without chemenu/tests/ and sets no threshold - the origin repository's developer guide, docs/dev/development.md § "How much of the stack the suite reaches", has the measured number and how to read it.

Two exceptions, and they bite. test_types_cmd.py and test_index_build.py resolve the real types/ through the shared resolver, because what they assert is that behaviour comes from the type-specs rather than from a hardcoded map. Editing a type-spec can therefore fail tests that look unrelated to it - translating the wiki broke six that way. Anything those tests pin must be structural: dir: values and layout order are asserted exactly, display titles are read from the spec at assert time, never written out. The same rule applies to the template fence - anchor on a section name from sections, not on a literal.

Assert on prose in a type-spec or a contract only when the prose is the subject of the test. Otherwise it is a tripwire that fires on an edit nobody connected to the test.