chemenu/docs/dev/development.md

# Developing this stack

This document is for people working on `tools/wikitool`, the type schema or the
instruction/skill layer itself - not for the operators of an instance. For the other side
(installing, upgrading and running an instance) see [INSTALL.md](../../INSTALL.md).

**This file is not shipped.** It lives under `docs/dev/`, the human counterpart of
`instructions/dev/`, and `tools/wikitool dist export` excludes both directories wholesale: a
distributed instance has no release workflow, no CI and no issue board, so it needs no guide to
them. The reason is a comment at `dist_cmd.DOCS_DEV_ONLY`, so that a later session does not
"repair" the apparent gap. And because it is not shipped, this file may link into
`instructions/dev/` - unlike a shipped document such as `README.md`, which
`instructions verify` checks on exactly that point.

<!-- wikitool:toc -->
## Contents

- [Development environment](#development-environment)
  - [`dist export` as a build and test tool](#dist-export-as-a-build-and-test-tool)
  - [A release feed in a non-public repository](#a-release-feed-in-a-non-public-repository)
- [The release workflow](#the-release-workflow)
- [The CI half](#the-ci-half)
  - [The Gitea wiki](#the-gitea-wiki)
  - [How much of the stack the suite reaches](#how-much-of-the-stack-the-suite-reaches)
- [Stack development as a session type of its own](#stack-development-as-a-session-type-of-its-own)
<!-- /wikitool:toc -->

## Development environment

Work on the stack happens in a clone of this repository. Such a clone is **not an instance** and
never becomes one. Instances come only from releases - see [INSTALL.md](../../INSTALL.md).

```bash
git clone https://gitea.nehmer.net/torben/chemenu.git
cd chemenu
```

Then have the agent run `instructions/bootstrap.md`: the preflight, then
`tools/wikitool instructions sync`. The agent's side of it - what differs here from an instance,
and how `dist export` runs as a test tool - is
[instructions/dev/dev-setup.md](../../instructions/dev/dev-setup.md).

Real documents for `raw derive` do not live in this repository. The corpus and its test harness
are the private repository `torben/chemenu-testfiles`, cloned by hand into `testfiles/` (gitignored
here, deliberately not a submodule, so a clone without access works unchanged):

```bash
git clone ssh://git@gitea.nehmer.net:10101/torben/chemenu-testfiles.git testfiles
cd tools && .venv/bin/python -m pytest -c pytest.ini ../testfiles/harness
```

It is private because the corpus is large (some 76 MB; one book takes 48 s per derivation) and
holds material that may not sit in a public repository, such as publishers' books and vendors'
documents. Without the clone, `pytest` runs unchanged and the real-document tests are simply not
there; the README of the private repository says what is in it, what each file's origin is and
whether it could be public. The tests run in their own workflow, see [The CI half](#the-ci-half).

What a clone has and an instance does not:

- **The demo corpus.** Some 170 pages documenting the stack itself: gates, lint, versioning,
  search, the wiki pattern. It is a testbed and a walkable example, not a productive knowledge
  base. What may be changed in it is governed by
  [instructions/dev/corpus-policy.md](../../instructions/dev/corpus-policy.md).
- **A demo persona** in `USER.md`/`SOUL.md`. `doctor` reports both as filled in. They describe
  the demo setup, not you.
- **Telemetry on.** Without `.wikitool-release.json` the clone is the measuring station the stack
  evaluates itself with
  ([telemetry-and-evaluation.md](../explanation/telemetry-and-evaluation.md#whether-it-runs-at-all)).
- **`instructions/dev/`, `docs/dev/`, `commonplace/` and `.gitea/`.** `dist export` ships none
  of them.

`ENVIRONMENT.md` is missing after every clone, because the file is gitignored: it describes a
checkout, not the repository. Creating it (template `ENVIRONMENT.md.template`, step 5 in
`instructions/bootstrap.md`) spares every stack session the questions about harness, `gitea-mcp`
and remote.

### `dist export` as a build and test tool

`tools/wikitool dist export <empty directory>` writes exactly the tree a release ships: the
machinery without wiki content, without git history, without `instructions/dev/` or `docs/dev/`.
That is how you check before a release what would ship (`--dry-run` only lists it). And it is
how you replay the installation path without waiting for a release: build a tarball from it the
way `.gitea/workflows/release.yml` does, then start the preflight script in an empty directory
with `--archive <tarball>`. The CI step "The distribution works as a fresh instance" in
`.gitea/workflows/ci.yml` does exactly that.

For a real instance such an export is not a way in. It lacks the release origin in its stamp,
and `dist upgrade --latest` later compares against a state that never existed as a release.

### A release feed in a non-public repository

Whoever runs the stack from a repository of their own that is not public, and upgrades instances
from there, points `WIKITOOL_UPDATE_URL` at its feed. There is one quirk: Gitea answers an
anonymous caller with the same `404` for an invisible repository as for one that does not exist
at all. A missing release and missing access then look identical - and "no update found" would
simply be wrong. A Gitea token with read access in `WIKITOOL_UPDATE_TOKEN` fixes that. It is sent
only to downloads on the same host as the feed.

## The release workflow

Between two releases the stack carries **one** running version candidate instead of a new
number per bump. The full model - where the state lives, how escalation works, why a number is
only consumed by a release - is in
[instructions/dev/version-parts.md](../../instructions/dev/version-parts.md) and
[docs/explanation/version-model.md](../explanation/version-model.md). Here only the sequence, in
the order a session actually goes through it:

1. **A bump opens or escalates the candidate, graded with `--impact`.**

   ```bash
   tools/wikitool version bump --minor --title "What changed" --impact medium
   ```

   Writes `VERSION` as `X.Y.Z-beta.N` and opens (or updates) the matching `CHANGES.md` entry.
   Several bumps for the same change are normal - each one updates the same entry instead of
   opening a new one. `--impact high|medium|low` (default `medium`) groups the entry;
   `tools/wikitool version regrade` corrects a grade later, when the candidate's overall shape
   changes the read on an earlier bump.

2. **The entry gets its prose - only where it is needed.** `bump` writes the skeleton (heading,
   date, author, the machine-managed, grouped list of bump titles, and the breaking and migration
   lines where they apply). A smaller change needs nothing beyond its bump title; a larger one
   gets a paragraph for its topic, and a short summary is added before the release. The form, and
   the size budget `docs verify` and `version release` check, are in
   [instructions/dev/version-parts.md](../../instructions/dev/version-parts.md) § The candidate
   model.

3. **Run the checks before anything is published:**

   ```bash
   cd tools && .venv/bin/python -m pytest -q
   tools/wikitool docs verify
   tools/wikitool instructions verify
   ```

4. **`version release` fixes the candidate** once it is meant to ship:

   ```bash
   tools/wikitool version release --title "Summarising title"
   ```

   Strips the `-beta.N` suffix from `VERSION` and closes the changelog entry. `--title` is
   optional - without it the last bump's title stays; with it, a candidate that collected several
   bump titles gets a summarising heading. Refused when the candidate collected two or more bumps
   and the summary from step 2 is still missing - a candidate with exactly one bump is exempt -
   and likewise when the entry is over its size budget. Commits and pushes nothing (invariant 5
   in [AGENTS.md](../../AGENTS.md)).

5. **Publish moves `VERSION` on `main`.**

   ```bash
   tools/wikitool publish --message "..."
   ```

   The Mass-Update Gate and the Publish-Remote Gate apply as on every other publish - see
   [instructions/gates.md](../../instructions/gates.md).

6. **CI does the rest.** `.gitea/workflows/release.yml` reacts to every push that moves
   `VERSION`: a `VERSION` with a suffix (a candidate) makes the job skip cleanly before it even
   queries the releases API - betas are never published. A suffix-free `VERSION` builds the
   distribution (`dist export`), creates the tag and the release, and uploads the tarball and its
   checksum, along with the two preflight scripts and the instructions `setup-instance.md` and
   `preflight.md` that the installation sentence in `INSTALL.md` points at. **CI sets the tag,
   never a session** - that keeps invariant 5 intact.

   The release note is the `CHANGES.md` entry; Gitea on MySQL stores at most 65535 bytes, and
   the job refuses from 60000 on, before it creates a tag. That is only the last net: `docs
   verify` already holds the topmost entry to at most 32000 bytes while it is a candidate (step
   2). If the job fails after `VERSION` is already on `main`, neither a push (the version does
   not rise again) nor a re-run (it takes the workflow file of the failed commit) helps: fix the
   cause, publish, and start `release.yml` on `main` via `workflow_dispatch`. The check for an
   existing release prevents a second one.

   A second job, `wiki-mirror`, runs after a suffix-free release and mirrors the docs of the
   release tree into the Gitea wiki ([The Gitea wiki](#the-gitea-wiki) below). It is a job of its
   own, so a failing mirror leaves the release and its tag standing and marks only itself red.

The three check commands are in step 3 above; what each of them checks is in
[tools/CONTRACT.md](../../tools/CONTRACT.md), where `docs verify` holds it against the actual
CLI. It is deliberately **not** repeated here: a second description of the same commands is
exactly the copy that drifts (AGENTS.md invariant 8), and this document lies outside the files
the command-record check of `docs verify` covers - so a drift here would go unnoticed. What
`pytest` expects of the developer at this point is in
[instructions/dev/testing-conventions.md](../../instructions/dev/testing-conventions.md).

## The CI half

`.gitea/workflows/ci.yml` runs on every push/PR against `main` (content paths excepted) and
runs the test suite, `docs verify`, `instructions verify` and a complete `setup-instance.md`
replay: a locally built release tarball, the preflight script with `--archive` in an empty
directory, then the instruction's steps - the same path a new user actually takes, just without
the download. `.gitea/workflows/nightly.yml` is the drift check against time rather than against
a commit. `.gitea/workflows/release.yml` is step 6 above, and `.gitea/workflows/wiki-mirror.yml`
refills the Gitea wiki by hand (see below).

`.gitea/workflows/converters.yml` runs the converters against the real documents of the private
repository (the clone above, run with the checkout's `tools/.venv`). It starts by itself on a push
to `main` that touches the converter surface - `tools/chemenu/filters/`, `raw/derive.py`,
`raw/accept.py`, `tests/test_raw_derive*.py`, `tests/binary_formats.py`, `tools/requirements.txt`
and the workflow itself - and by hand (`workflow_dispatch`), which is how to run it after a change
in the private repository alone: a push there starts nothing here. It always tests `main` of the
private repository, so a converter change that alters an expected output is two commits, the
private one first. Before a release without suffix, `release.yml` runs the same steps as a job its
release job waits for; a red harness ends the run before a release or a tag exists, and a
`-beta.N` candidate skips it. The job reads the private repository with the per-job token, which
works only because the private repository's own Actions settings grant `torben/chemenu` access;
without that grant the checkout step fails with "Determining the default branch … not found",
which reads like a missing repository rather than a missing permission.

That workflow shows that the corpus still derives, not that the derivation is good. The second
question is a session's: `harness/compare/` in the same private repository derives every corpus
file with the checkout's filters and with reference tools (pymupdf4llm, markitdown, pandoc, each in
a throwaway environment of its own, never in `tools/.venv`), measures wording fidelity, markup
overhead, missing semantics, structure, CHECK precision and runtime, and reports the difference to
a committed baseline. When a session runs it and what it does with the result is
[instructions/dev/converter-comparison.md](../../instructions/dev/converter-comparison.md).

Three more workflows carry the live tests of the tracker adapters (Super Productivity, CalDAV):
`ci.yml` runs the CalDAV half in a step of its own against Radicale as a process,
`tracker-live.yml` runs nightly and covers both providers, and `sp-live-image.yml` builds the
`chemenu-sp-live` image daily (when there is a new version) and monthly (always) with the current
Super Productivity version. The image follows the desktop clients' update channel, not a fixed
version. Once, after the very first push, the package has to be linked to the `torben/chemenu`
repository by hand. What the suite writes, how to run it against a tracker of your own, and what
a red run means is in
[instructions/dev/tracker-testing.md](../../instructions/dev/tracker-testing.md).

### The Gitea wiki

The wiki of `torben/chemenu` shows the documentation of the **latest release**. It is generated
and never edited: every run replaces its whole content, so a change made in the wiki is gone at
the next mirror, and nothing but CI writes to it (a push to the wiki repository by a session
would sit next to `publish`, which AGENTS.md invariant 5 forbids).

`.gitea/scripts/wiki_mirror.py` converts a tree into the wiki's page set: every `*.md` under
`docs/` (the `dev/` pages included), plus `README.md` as `Home` and `INSTALL.md` as `Install`.
Wiki pages in sub-directories are not listed reliably, so the names are flat: the path below
`docs/` with `.md` dropped, each segment capitalised and joined by `-`
(`docs/how-to/ingest-a-source.md` becomes `How-to-Ingest-a-source`; a `README.md` stands for its
directory). Two sources that would get the same name, compared without case, abort the run.
Relative links between mirrored files become wiki page names, relative links to anything else
become links to that file at the release tag, and a link whose target does not exist aborts the
run. A `_Sidebar` is generated, and every page opens with a line saying which file at which
version it came from. `.gitea/scripts/wiki-push.sh` then makes the wiki repository hold exactly
that output - a page the converter no longer produces is removed - and does not commit when
nothing changed.

Two things start it: the `wiki-mirror` job of `release.yml`, after each release (never for a
candidate), and `.gitea/workflows/wiki-mirror.yml` by hand with `workflow_dispatch`. The manual
workflow takes a `tag` (default: the latest release) and runs the converter and push script of
`main` against that tag's tree. Use it to fill the wiki before the first release that carries the
job, or to catch up a mirror that failed after its release was published. `ci.yml` runs
`wiki_mirror.py --check` against the working tree on every push, so a link the mirror could not
resolve fails there rather than at the release. The scripts live under `.gitea/`, which
`dist export` does not copy, so none of this reaches an instance.

Both jobs write with the job token, which needs `permissions: wiki: write` - Gitea gives the wiki
a token scope of its own, and `contents: write` does not cover it (the API answers 403). If the
repository's Actions settings ever cap the token below that, the mirror fails until the cap is
lifted. A wiki that has never had a page has no repository to clone yet; `wiki-push.sh` then
creates the first page through the API and clones afterwards.

### How much of the stack the suite reaches

Coverage is measured in CI - `pytest --cov`, config in `tools/.coveragerc`, HTML and XML
uploaded as the `coverage-<run id>` artifact of every run.
**Fetch that artifact from the run's own page, not from the API**: `upload-artifact@v3` writes
through the older artifact API, and the Actions artifact REST endpoints answer `total_count: 0`
for a run whose artifact the run page offers for download. The upload works; only the listing
does not see it. Do not re-derive this, and do not read the empty list as a failed upload.

It is enforced at a floor of **85%** (`fail_under` in `tools/.coveragerc`), which is what a red
suite from this axis means: coverage actually fell, not that a wrapper was added. The floor was
set only after the number had been watched - it was deliberately held back for exactly that, and
the two points between 85 and the measured 87.0% are the room the taxonomy below asks for. A
threshold at the measured number goes red on the next thin Typer wrapper, and a threshold that
goes red for a non-reason gets lowered rather than earned.

**Measured 2026-09-04, stack 4.7.1: 87.0% of 6498 statements across `chemenu/`, 975 tests** -
CI run 163. The first measurement, at stack 1.8.1 on 2026-08-31, was 86.9% of 5105 statements
over 730 tests (CI run 87). Both are what CI reported, never a local run: the local number
preceding a release measures a tree that is one commit short of the published one.

The pair says more than either number does. Between them the measured code grew by a quarter
and the suite by a third, and the quota moved by a tenth of a point - which is the observation a
threshold was waiting for, rather than the total itself. Reproduce either with
`cd tools && .venv/bin/python -m pytest -q --cov` (needs `pytest-cov`, which is CI-only and
deliberately absent from `tools/requirements.txt` - an instance runs the wiki, it does not
measure this suite).

The total stays the least interesting number here. What the report is for is *which* modules sit
low, and three kinds have to be told apart before any of it turns into work:

- **Thin Typer wrappers**, where the logic lives beside them and is tested there:
  `eval_cmd.py` (36%), `types_cmd.py` (40%), `search.py` (49%), `cli.py` (54%),
  `links_cmd.py` (61%). Low coverage on a wrapper is evidence of a good cut, not of a missing
  test - `search.py`'s uncovered block is its command body alone, while the backends under
  `chemenu/search/` that do the work sit between 91% and 98%.
- **Code that reaches the network or the filesystem's outside**, where the interesting half is
  already injectable and tested through the seam: `version.py`'s `fetch_latest()` takes a
  `fetcher` parameter for exactly that. The real `urllib` lines are exercised too, but only
  against a local `http.server` in `test_dist_upgrade.py` (`dist upgrade --latest`) - never
  against a real feed, which stays uncovered on purpose.
- **Genuine gaps**, where uncovered lines are logic nobody exercises: `provenance_cmd.py`
  (44%), `migrate_cmd.py` (65%), `type_resolver.py` (79%). This is the list worth reading, and
  the only one of the three that has not moved while everything around it did:
  `provenance_cmd.py` sits where it sat, and `migrate_cmd.py` fell from 71% because it grew and
  its new lines arrived untested. The floor freezes this; it does not close it. Closing it is
  Gitea #51.

## Stack development as a session type of its own

Three skills (`instructions/dev/`, present only in this origin repository) carry a session that
works on the stack itself rather than on wiki content through three phases: `stack-dev` works
the issue out until its body is "ready", `stack-build` builds, publishes and waits for a green CI
run, `stack-close` checks the body's final state and stale `docs/` and contract prose, closes
the issue and, where the body carries a `## Kette` section, names the next link of that chain
with the slash command that starts it. The handover goes through the state in the tracker, not through a session's context:
every phase change works in the same session or after `/clear`. `stack-dev` triggers
automatically; `stack-build` and `stack-close` are started only by the operator, by slash command
(`/stack-build #N`, `/stack-close`) - exactly where the choice falls whether to continue in the
same session or a new one, and on which model. None of the three offers a model switch in the
middle of a session. Rules and the phase table are in
[instructions/dev/stack-mode.md](../../instructions/dev/stack-mode.md), the issue tracker itself
in [instructions/dev/issue-tracking.md](../../instructions/dev/issue-tracking.md).