chemenu/docs/dev/development.md

Developing this stack

This document is for people working on tools/wikitool, the type schema or the instruction/skill layer itself - not for the operators of an instance. For the other side (installing, upgrading and running an instance) see INSTALL.md.

This file is not shipped. It lives under docs/dev/, the human counterpart of instructions/dev/, and tools/wikitool dist export excludes both directories wholesale: a distributed instance has no release workflow, no CI and no issue board, so it needs no guide to them. The reason is a comment at dist_cmd.DOCS_DEV_ONLY, so that a later session does not "repair" the apparent gap. And because it is not shipped, this file may link into instructions/dev/ - unlike a shipped document such as README.md, which instructions verify checks on exactly that point.

Contents

• Development environment
  • dist export as a build and test tool
  • A release feed in a non-public repository
• The release workflow
• The CI half
  • The Gitea wiki
  • How much of the stack the suite reaches
• Stack development as a session type of its own

Development environment

Work on the stack happens in a clone of this repository. Such a clone is not an instance and never becomes one. Instances come only from releases - see INSTALL.md.

git clone https://gitea.nehmer.net/torben/chemenu.git
cd chemenu

Then have the agent run instructions/bootstrap.md: the preflight, then tools/wikitool instructions sync. The agent's side of it - what differs here from an instance, and how dist export runs as a test tool - is instructions/dev/dev-setup.md.

Real documents for raw derive do not live in this repository. The corpus and its test harness are the private repository torben/chemenu-testfiles, cloned by hand into testfiles/ (gitignored here, deliberately not a submodule, so a clone without access works unchanged):

git clone ssh://git@gitea.nehmer.net:10101/torben/chemenu-testfiles.git testfiles
cd tools && .venv/bin/python -m pytest -c pytest.ini ../testfiles/harness

It is private because the corpus is large (some 76 MB; one book takes 48 s per derivation) and holds material that may not sit in a public repository, such as publishers' books and vendors' documents. Without the clone, pytest runs unchanged and the real-document tests are simply not there; the README of the private repository says what is in it, what each file's origin is and whether it could be public. The tests run in their own workflow, see The CI half.

What a clone has and an instance does not:

• The demo corpus. Some 170 pages documenting the stack itself: gates, lint, versioning, search, the wiki pattern. It is a testbed and a walkable example, not a productive knowledge base. What may be changed in it is governed by instructions/dev/corpus-policy.md.
• A demo persona in USER.md/SOUL.md. doctor reports both as filled in. They describe the demo setup, not you.
• Telemetry on. Without .wikitool-release.json the clone is the measuring station the stack evaluates itself with (telemetry-and-evaluation.md).
• instructions/dev/, docs/dev/, commonplace/ and .gitea/. dist export ships none of them.

ENVIRONMENT.md is missing after every clone, because the file is gitignored: it describes a checkout, not the repository. Creating it (template ENVIRONMENT.md.template, step 5 in instructions/bootstrap.md) spares every stack session the questions about harness, gitea-mcp and remote.

dist export as a build and test tool

tools/wikitool dist export <empty directory> writes exactly the tree a release ships: the machinery without wiki content, without git history, without instructions/dev/ or docs/dev/. That is how you check before a release what would ship (--dry-run only lists it). And it is how you replay the installation path without waiting for a release: build a tarball from it the way .gitea/workflows/release.yml does, then start the preflight script in an empty directory with --archive <tarball>. The CI step "The distribution works as a fresh instance" in .gitea/workflows/ci.yml does exactly that.

For a real instance such an export is not a way in. It lacks the release origin in its stamp, and dist upgrade --latest later compares against a state that never existed as a release.

A release feed in a non-public repository

Whoever runs the stack from a repository of their own that is not public, and upgrades instances from there, points WIKITOOL_UPDATE_URL at its feed. There is one quirk: Gitea answers an anonymous caller with the same 404 for an invisible repository as for one that does not exist at all. A missing release and missing access then look identical - and "no update found" would simply be wrong. A Gitea token with read access in WIKITOOL_UPDATE_TOKEN fixes that. It is sent only to downloads on the same host as the feed.

The release workflow

Between two releases the stack carries one running version candidate instead of a new number per bump. The full model - where the state lives, how escalation works, why a number is only consumed by a release - is in instructions/dev/version-parts.md and docs/explanation/version-model.md. Here only the sequence, in the order a session actually goes through it:

1. A bump opens or escalates the candidate, graded with --impact.

   tools/wikitool version bump --minor --title "What changed" --impact medium
   

   Writes VERSION as X.Y.Z-beta.N and opens (or updates) the matching CHANGES.md entry. Several bumps for the same change are normal - each one updates the same entry instead of opening a new one. --impact high|medium|low (default medium) groups the entry; tools/wikitool version regrade corrects a grade later, when the candidate's overall shape changes the read on an earlier bump.

2. The entry gets its prose - only where it is needed. bump writes the skeleton (heading, date, author, the machine-managed, grouped list of bump titles, and the breaking and migration lines where they apply). A smaller change needs nothing beyond its bump title; a larger one gets a paragraph for its topic, and a short summary is added before the release. The form, and the size budget docs verify and version release check, are in instructions/dev/version-parts.md § The candidate model.

3. Run the checks before anything is published:

   cd tools && .venv/bin/python -m pytest -q
   tools/wikitool docs verify
   tools/wikitool instructions verify
   

4. version release fixes the candidate once it is meant to ship:

   tools/wikitool version release --title "Summarising title"
   

   Strips the -beta.N suffix from VERSION and closes the changelog entry. --title is optional - without it the last bump's title stays; with it, a candidate that collected several bump titles gets a summarising heading. Refused when the candidate collected two or more bumps and the summary from step 2 is still missing - a candidate with exactly one bump is exempt - and likewise when the entry is over its size budget. Commits and pushes nothing (invariant 5 in AGENTS.md).

5. Publish moves VERSION on main.

   tools/wikitool publish --message "..."
   

   The Mass-Update Gate and the Publish-Remote Gate apply as on every other publish - see instructions/gates.md.

6. CI does the rest. .gitea/workflows/release.yml reacts to every push that moves VERSION: a VERSION with a suffix (a candidate) makes the job skip cleanly before it even queries the releases API - betas are never published. A suffix-free VERSION builds the distribution (dist export), creates the tag and the release, and uploads the tarball and its checksum, along with the two preflight scripts and the instructions setup-instance.md and preflight.md that the installation sentence in INSTALL.md points at. CI sets the tag, never a session - that keeps invariant 5 intact.

   The release note is the CHANGES.md entry; Gitea on MySQL stores at most 65535 bytes, and the job refuses from 60000 on, before it creates a tag. That is only the last net: docs verify already holds the topmost entry to at most 32000 bytes while it is a candidate (step 2). If the job fails after VERSION is already on main, neither a push (the version does not rise again) nor a re-run (it takes the workflow file of the failed commit) helps: fix the cause, publish, and start release.yml on main via workflow_dispatch. The check for an existing release prevents a second one.

   A second job, wiki-mirror, runs after a suffix-free release and mirrors the docs of the release tree into the Gitea wiki (The Gitea wiki below). It is a job of its own, so a failing mirror leaves the release and its tag standing and marks only itself red.

The three check commands are in step 3 above; what each of them checks is in tools/CONTRACT.md, where docs verify holds it against the actual CLI. It is deliberately not repeated here: a second description of the same commands is exactly the copy that drifts (AGENTS.md invariant 8), and this document lies outside the files the command-record check of docs verify covers - so a drift here would go unnoticed. What pytest expects of the developer at this point is in instructions/dev/testing-conventions.md.

The CI half

.gitea/workflows/ci.yml runs on every push/PR against main (content paths excepted) and runs the test suite, docs verify, instructions verify and a complete setup-instance.md replay: a locally built release tarball, the preflight script with --archive in an empty directory, then the instruction's steps - the same path a new user actually takes, just without the download. .gitea/workflows/nightly.yml is the drift check against time rather than against a commit. .gitea/workflows/release.yml is step 6 above, and .gitea/workflows/wiki-mirror.yml refills the Gitea wiki by hand (see below).

.gitea/workflows/converters.yml runs the converters against the real documents of the private repository (the clone above, run with the checkout's tools/.venv). It starts by itself on a push to main that touches the converter surface - tools/chemenu/filters/, raw/derive.py, raw/accept.py, tests/test_raw_derive*.py, tests/binary_formats.py, tools/requirements.txt and the workflow itself - and by hand (workflow_dispatch), which is how to run it after a change in the private repository alone: a push there starts nothing here. It always tests main of the private repository, so a converter change that alters an expected output is two commits, the private one first. Before a release without suffix, release.yml runs the same steps as a job its release job waits for; a red harness ends the run before a release or a tag exists, and a -beta.N candidate skips it. The job reads the private repository with the per-job token; if that instance ever stops allowing this, the workflow's header names the 1Password fallback.

Three more workflows carry the live tests of the tracker adapters (Super Productivity, CalDAV): ci.yml runs the CalDAV half in a step of its own against Radicale as a process, tracker-live.yml runs nightly and covers both providers, and sp-live-image.yml builds the chemenu-sp-live image daily (when there is a new version) and monthly (always) with the current Super Productivity version. The image follows the desktop clients' update channel, not a fixed version. Once, after the very first push, the package has to be linked to the torben/chemenu repository by hand. What the suite writes, how to run it against a tracker of your own, and what a red run means is in instructions/dev/tracker-testing.md.

The Gitea wiki

The wiki of torben/chemenu shows the documentation of the latest release. It is generated and never edited: every run replaces its whole content, so a change made in the wiki is gone at the next mirror, and nothing but CI writes to it (a push to the wiki repository by a session would sit next to publish, which AGENTS.md invariant 5 forbids).

.gitea/scripts/wiki_mirror.py converts a tree into the wiki's page set: every *.md under docs/ (the dev/ pages included), plus README.md as Home and INSTALL.md as Install. Wiki pages in sub-directories are not listed reliably, so the names are flat: the path below docs/ with .md dropped, each segment capitalised and joined by - (docs/how-to/ingest-a-source.md becomes How-to-Ingest-a-source; a README.md stands for its directory). Two sources that would get the same name, compared without case, abort the run. Relative links between mirrored files become wiki page names, relative links to anything else become links to that file at the release tag, and a link whose target does not exist aborts the run. A _Sidebar is generated, and every page opens with a line saying which file at which version it came from. .gitea/scripts/wiki-push.sh then makes the wiki repository hold exactly that output - a page the converter no longer produces is removed - and does not commit when nothing changed.

Two things start it: the wiki-mirror job of release.yml, after each release (never for a candidate), and .gitea/workflows/wiki-mirror.yml by hand with workflow_dispatch. The manual workflow takes a tag (default: the latest release) and runs the converter and push script of main against that tag's tree. Use it to fill the wiki before the first release that carries the job, or to catch up a mirror that failed after its release was published. ci.yml runs wiki_mirror.py --check against the working tree on every push, so a link the mirror could not resolve fails there rather than at the release. The scripts live under .gitea/, which dist export does not copy, so none of this reaches an instance.

Both jobs write with the job token, which needs permissions: wiki: write - Gitea gives the wiki a token scope of its own, and contents: write does not cover it (the API answers 403). If the repository's Actions settings ever cap the token below that, the mirror fails until the cap is lifted. A wiki that has never had a page has no repository to clone yet; wiki-push.sh then creates the first page through the API and clones afterwards.

How much of the stack the suite reaches

Coverage is measured in CI - pytest --cov, config in tools/.coveragerc, HTML and XML uploaded as the coverage-<run id> artifact of every run. Fetch that artifact from the run's own page, not from the API: upload-artifact@v3 writes through the older artifact API, and the Actions artifact REST endpoints answer total_count: 0 for a run whose artifact the run page offers for download. The upload works; only the listing does not see it. Do not re-derive this, and do not read the empty list as a failed upload.

It is enforced at a floor of 85% (fail_under in tools/.coveragerc), which is what a red suite from this axis means: coverage actually fell, not that a wrapper was added. The floor was set only after the number had been watched - it was deliberately held back for exactly that, and the two points between 85 and the measured 87.0% are the room the taxonomy below asks for. A threshold at the measured number goes red on the next thin Typer wrapper, and a threshold that goes red for a non-reason gets lowered rather than earned.

Measured 2026-09-04, stack 4.7.1: 87.0% of 6498 statements across chemenu/, 975 tests - CI run 163. The first measurement, at stack 1.8.1 on 2026-08-31, was 86.9% of 5105 statements over 730 tests (CI run 87). Both are what CI reported, never a local run: the local number preceding a release measures a tree that is one commit short of the published one.

The pair says more than either number does. Between them the measured code grew by a quarter and the suite by a third, and the quota moved by a tenth of a point - which is the observation a threshold was waiting for, rather than the total itself. Reproduce either with cd tools && .venv/bin/python -m pytest -q --cov (needs pytest-cov, which is CI-only and deliberately absent from tools/requirements.txt - an instance runs the wiki, it does not measure this suite).

The total stays the least interesting number here. What the report is for is which modules sit low, and three kinds have to be told apart before any of it turns into work:

• Thin Typer wrappers, where the logic lives beside them and is tested there: eval_cmd.py (36%), types_cmd.py (40%), search.py (49%), cli.py (54%), links_cmd.py (61%). Low coverage on a wrapper is evidence of a good cut, not of a missing test - search.py's uncovered block is its command body alone, while the backends under chemenu/search/ that do the work sit between 91% and 98%.
• Code that reaches the network or the filesystem's outside, where the interesting half is already injectable and tested through the seam: version.py's fetch_latest() takes a fetcher parameter for exactly that. The real urllib lines are exercised too, but only against a local http.server in test_dist_upgrade.py (dist upgrade --latest) - never against a real feed, which stays uncovered on purpose.
• Genuine gaps, where uncovered lines are logic nobody exercises: provenance_cmd.py (44%), migrate_cmd.py (65%), type_resolver.py (79%). This is the list worth reading, and the only one of the three that has not moved while everything around it did: provenance_cmd.py sits where it sat, and migrate_cmd.py fell from 71% because it grew and its new lines arrived untested. The floor freezes this; it does not close it. Closing it is Gitea #51.

Stack development as a session type of its own

Three skills (instructions/dev/, present only in this origin repository) carry a session that works on the stack itself rather than on wiki content through three phases: stack-dev works the issue out until its body is "ready", stack-build builds, publishes and waits for a green CI run, stack-close checks the body's final state and stale docs/ and contract prose, closes the issue and, where the body carries a ## Kette section, names the next link of that chain with the slash command that starts it. The handover goes through the state in the tracker, not through a session's context: every phase change works in the same session or after /clear. stack-dev triggers automatically; stack-build and stack-close are started only by the operator, by slash command (/stack-build #N, /stack-close) - exactly where the choice falls whether to continue in the same session or a new one, and on which model. None of the three offers a model switch in the middle of a session. Rules and the phase table are in instructions/dev/stack-mode.md, the issue tracker itself in instructions/dev/issue-tracking.md.