chemenu/docs/dev/development.md

# Developing this stack

This document is for people working on `tools/wikitool`, the type schema or the
instruction/skill layer itself - not for the operators of an instance. For the other side
(installing, upgrading and running an instance) see [INSTALL.md](../../INSTALL.md).

**This file is not shipped.** It lives under `docs/dev/`, the human counterpart of
`instructions/dev/`, and `tools/wikitool dist export` excludes both directories wholesale: a
distributed instance has no release workflow, no CI and no issue board, so it needs no guide to
them. The reason is a comment at `dist_cmd.DOCS_DEV_ONLY`, so that a later session does not
"repair" the apparent gap. And because it is not shipped, this file may link into
`instructions/dev/` - unlike a shipped document such as `README.md`, which
`instructions verify` checks on exactly that point.

<!-- wikitool:toc -->
## Contents

- [Development environment](#development-environment)
  - [`dist export` as a build and test tool](#dist-export-as-a-build-and-test-tool)
  - [A release feed in a non-public repository](#a-release-feed-in-a-non-public-repository)
- [The release workflow](#the-release-workflow)
- [The CI half](#the-ci-half)
  - [How much of the stack the suite reaches](#how-much-of-the-stack-the-suite-reaches)
- [Stack development as a session type of its own](#stack-development-as-a-session-type-of-its-own)
<!-- /wikitool:toc -->

## Development environment

Work on the stack happens in a clone of this repository. Such a clone is **not an instance** and
never becomes one. Instances come only from releases - see [INSTALL.md](../../INSTALL.md).

```bash
git clone https://gitea.nehmer.net/torben/chemenu.git
cd chemenu
```

Then have the agent run `instructions/bootstrap.md`: the preflight, then
`tools/wikitool instructions sync`. The agent's side of it - what differs here from an instance,
and how `dist export` runs as a test tool - is
[instructions/dev/dev-setup.md](../../instructions/dev/dev-setup.md).

What a clone has and an instance does not:

- **The demo corpus.** Some 170 pages documenting the stack itself: gates, lint, versioning,
  search, the wiki pattern. It is a testbed and a walkable example, not a productive knowledge
  base. What may be changed in it is governed by
  [instructions/dev/corpus-policy.md](../../instructions/dev/corpus-policy.md).
- **A demo persona** in `USER.md`/`SOUL.md`. `doctor` reports both as filled in. They describe
  the demo setup, not you.
- **Telemetry on.** Without `.wikitool-release.json` the clone is the measuring station the stack
  evaluates itself with
  ([telemetry-and-evaluation.md](../explanation/telemetry-and-evaluation.md#whether-it-runs-at-all)).
- **`instructions/dev/`, `docs/dev/`, `commonplace/` and `.gitea/`.** `dist export` ships none
  of them.

`ENVIRONMENT.md` is missing after every clone, because the file is gitignored: it describes a
checkout, not the repository. Creating it (template `ENVIRONMENT.md.template`, step 5 in
`instructions/bootstrap.md`) spares every stack session the questions about harness, `gitea-mcp`
and remote.

### `dist export` as a build and test tool

`tools/wikitool dist export <empty directory>` writes exactly the tree a release ships: the
machinery without wiki content, without git history, without `instructions/dev/` or `docs/dev/`.
That is how you check before a release what would ship (`--dry-run` only lists it). And it is
how you replay the installation path without waiting for a release: build a tarball from it the
way `.gitea/workflows/release.yml` does, then start the preflight script in an empty directory
with `--archive <tarball>`. The CI step "The distribution works as a fresh instance" in
`.gitea/workflows/ci.yml` does exactly that.

For a real instance such an export is not a way in. It lacks the release origin in its stamp,
and `dist upgrade --latest` later compares against a state that never existed as a release.

### A release feed in a non-public repository

Whoever runs the stack from a repository of their own that is not public, and upgrades instances
from there, points `WIKITOOL_UPDATE_URL` at its feed. There is one quirk: Gitea answers an
anonymous caller with the same `404` for an invisible repository as for one that does not exist
at all. A missing release and missing access then look identical - and "no update found" would
simply be wrong. A Gitea token with read access in `WIKITOOL_UPDATE_TOKEN` fixes that. It is sent
only to downloads on the same host as the feed.

## The release workflow

Between two releases the stack carries **one** running version candidate instead of a new
number per bump. The full model - where the state lives, how escalation works, why a number is
only consumed by a release - is in
[instructions/dev/version-parts.md](../../instructions/dev/version-parts.md) and
[docs/explanation/version-model.md](../explanation/version-model.md). Here only the sequence, in
the order a session actually goes through it:

1. **A bump opens or escalates the candidate, graded with `--impact`.**

   ```bash
   tools/wikitool version bump --minor --title "What changed" --impact medium
   ```

   Writes `VERSION` as `X.Y.Z-beta.N` and opens (or updates) the matching `CHANGES.md` entry.
   Several bumps for the same change are normal - each one updates the same entry instead of
   opening a new one. `--impact high|medium|low` (default `medium`) groups the entry;
   `tools/wikitool version regrade` corrects a grade later, when the candidate's overall shape
   changes the read on an earlier bump.

2. **The entry gets its prose - only where it is needed.** `bump` writes the skeleton (heading,
   date, author, the machine-managed, grouped list of bump titles, and the breaking and migration
   lines where they apply). A smaller change needs nothing beyond its bump title; a larger one
   gets a paragraph for its topic, and a short summary is added before the release. The form, and
   the size budget `docs verify` and `version release` check, are in
   [instructions/dev/version-parts.md](../../instructions/dev/version-parts.md) § The candidate
   model.

3. **Run the checks before anything is published:**

   ```bash
   cd tools && .venv/bin/python -m pytest -q
   tools/wikitool docs verify
   tools/wikitool instructions verify
   ```

4. **`version release` fixes the candidate** once it is meant to ship:

   ```bash
   tools/wikitool version release --title "Summarising title"
   ```

   Strips the `-beta.N` suffix from `VERSION` and closes the changelog entry. `--title` is
   optional - without it the last bump's title stays; with it, a candidate that collected several
   bump titles gets a summarising heading. Refused when the candidate collected two or more bumps
   and the summary from step 2 is still missing - a candidate with exactly one bump is exempt -
   and likewise when the entry is over its size budget. Commits and pushes nothing (invariant 5
   in [AGENTS.md](../../AGENTS.md)).

5. **Publish moves `VERSION` on `main`.**

   ```bash
   tools/wikitool publish --message "..."
   ```

   The Mass-Update Gate and the Publish-Remote Gate apply as on every other publish - see
   [instructions/gates.md](../../instructions/gates.md).

6. **CI does the rest.** `.gitea/workflows/release.yml` reacts to every push that moves
   `VERSION`: a `VERSION` with a suffix (a candidate) makes the job skip cleanly before it even
   queries the releases API - betas are never published. A suffix-free `VERSION` builds the
   distribution (`dist export`), creates the tag and the release, and uploads the tarball and its
   checksum, along with the two preflight scripts and the instructions `setup-instance.md` and
   `preflight.md` that the installation sentence in `INSTALL.md` points at. **CI sets the tag,
   never a session** - that keeps invariant 5 intact.

   The release note is the `CHANGES.md` entry; Gitea on MySQL stores at most 65535 bytes, and
   the job refuses from 60000 on, before it creates a tag. That is only the last net: `docs
   verify` already holds the topmost entry to at most 32000 bytes while it is a candidate (step
   2). If the job fails after `VERSION` is already on `main`, neither a push (the version does
   not rise again) nor a re-run (it takes the workflow file of the failed commit) helps: fix the
   cause, publish, and start `release.yml` on `main` via `workflow_dispatch`. The check for an
   existing release prevents a second one.

The three check commands are in step 3 above; what each of them checks is in
[tools/CONTRACT.md](../../tools/CONTRACT.md), where `docs verify` holds it against the actual
CLI. It is deliberately **not** repeated here: a second description of the same commands is
exactly the copy that drifts (AGENTS.md invariant 8), and this document lies outside the files
the command-record check of `docs verify` covers - so a drift here would go unnoticed. What
`pytest` expects of the developer at this point is in
[instructions/dev/testing-conventions.md](../../instructions/dev/testing-conventions.md).

## The CI half

`.gitea/workflows/ci.yml` runs on every push/PR against `main` (content paths excepted) and
runs the test suite, `docs verify`, `instructions verify` and a complete `setup-instance.md`
replay: a locally built release tarball, the preflight script with `--archive` in an empty
directory, then the instruction's steps - the same path a new user actually takes, just without
the download. `.gitea/workflows/nightly.yml` is the drift check against time rather than against
a commit. `.gitea/workflows/release.yml` is step 6 above.

Three more workflows carry the live tests of the tracker adapters (Super Productivity, CalDAV):
`ci.yml` runs the CalDAV half in a step of its own against Radicale as a process,
`tracker-live.yml` runs nightly and covers both providers, and `sp-live-image.yml` builds the
`chemenu-sp-live` image daily (when there is a new version) and monthly (always) with the current
Super Productivity version. The image follows the desktop clients' update channel, not a fixed
version. Once, after the very first push, the package has to be linked to the `torben/chemenu`
repository by hand. What the suite writes, how to run it against a tracker of your own, and what
a red run means is in
[instructions/dev/tracker-testing.md](../../instructions/dev/tracker-testing.md).

### How much of the stack the suite reaches

Coverage is measured in CI - `pytest --cov`, config in `tools/.coveragerc`, HTML and XML
uploaded as the `coverage-<run id>` artifact of every run.
**Fetch that artifact from the run's own page, not from the API**: `upload-artifact@v3` writes
through the older artifact API, and the Actions artifact REST endpoints answer `total_count: 0`
for a run whose artifact the run page offers for download. The upload works; only the listing
does not see it. Do not re-derive this, and do not read the empty list as a failed upload.

It is enforced at a floor of **85%** (`fail_under` in `tools/.coveragerc`), which is what a red
suite from this axis means: coverage actually fell, not that a wrapper was added. The floor was
set only after the number had been watched - it was deliberately held back for exactly that, and
the two points between 85 and the measured 87.0% are the room the taxonomy below asks for. A
threshold at the measured number goes red on the next thin Typer wrapper, and a threshold that
goes red for a non-reason gets lowered rather than earned.

**Measured 2026-09-04, stack 4.7.1: 87.0% of 6498 statements across `chemenu/`, 975 tests** -
CI run 163. The first measurement, at stack 1.8.1 on 2026-08-31, was 86.9% of 5105 statements
over 730 tests (CI run 87). Both are what CI reported, never a local run: the local number
preceding a release measures a tree that is one commit short of the published one.

The pair says more than either number does. Between them the measured code grew by a quarter
and the suite by a third, and the quota moved by a tenth of a point - which is the observation a
threshold was waiting for, rather than the total itself. Reproduce either with
`cd tools && .venv/bin/python -m pytest -q --cov` (needs `pytest-cov`, which is CI-only and
deliberately absent from `tools/requirements.txt` - an instance runs the wiki, it does not
measure this suite).

The total stays the least interesting number here. What the report is for is *which* modules sit
low, and three kinds have to be told apart before any of it turns into work:

- **Thin Typer wrappers**, where the logic lives beside them and is tested there:
  `eval_cmd.py` (36%), `types_cmd.py` (40%), `search.py` (49%), `cli.py` (54%),
  `links_cmd.py` (61%). Low coverage on a wrapper is evidence of a good cut, not of a missing
  test - `search.py`'s uncovered block is its command body alone, while the backends under
  `chemenu/search/` that do the work sit between 91% and 98%.
- **Code that reaches the network or the filesystem's outside**, where the interesting half is
  already injectable and tested through the seam: `version.py`'s `fetch_latest()` takes a
  `fetcher` parameter for exactly that. The real `urllib` lines are exercised too, but only
  against a local `http.server` in `test_dist_upgrade.py` (`dist upgrade --latest`) - never
  against a real feed, which stays uncovered on purpose.
- **Genuine gaps**, where uncovered lines are logic nobody exercises: `provenance_cmd.py`
  (44%), `migrate_cmd.py` (65%), `type_resolver.py` (79%). This is the list worth reading, and
  the only one of the three that has not moved while everything around it did:
  `provenance_cmd.py` sits where it sat, and `migrate_cmd.py` fell from 71% because it grew and
  its new lines arrived untested. The floor freezes this; it does not close it. Closing it is
  Gitea #51.

## Stack development as a session type of its own

Three skills (`instructions/dev/`, present only in this origin repository) carry a session that
works on the stack itself rather than on wiki content through three phases: `stack-dev` works
the issue out until its body is "ready", `stack-build` builds, publishes and waits for a green CI
run, `stack-close` checks the body's final state and stale `docs/` and contract prose, and closes
the issue. The handover goes through the state in the tracker, not through a session's context:
every phase change works in the same session or after `/clear`. `stack-dev` triggers
automatically; `stack-build` and `stack-close` are started only by the operator, by slash command
(`/stack-build #N`, `/stack-close`) - exactly where the choice falls whether to continue in the
same session or a new one, and on which model. None of the three offers a model switch in the
middle of a session. Rules and the phase table are in
[instructions/dev/stack-mode.md](../../instructions/dev/stack-mode.md), the issue tracker itself
in [instructions/dev/issue-tracking.md](../../instructions/dev/issue-tracking.md).