torben/chemenu
1
1 Fork 0
Code Issues 35 Pull Requests Actions Packages Releases 37 Activity

feat: live tracker suite - WIKITOOL_TASKS_CONFIG override, real-tracker tests for Super Productivity and CalDAV, nightly workflow and test image (#156)
CI / verify (push) Failing after 2m1s
Details
Release / release (push) Successful in 38s
Details

Browse Source 
Files changed:
- .gitea/scripts/start-radicale.sh
- .gitea/sp-live/Dockerfile
- .gitea/sp-live/resolve-version.sh
- .gitea/workflows/ci.yml
- .gitea/workflows/sp-live-image.yml
- .gitea/workflows/tracker-live.yml
- .gitignore
- CHANGES.md
- DEVELOPMENT.md
- INSTALL.md
- VERSION
- instructions/dev/doc-pull-through.md
- instructions/dev/stack-dev/SKILL.md
- instructions/dev/testing-conventions.md
- instructions/dev/tracker-testing.md
- kb/gtd/INDEX.md
- kb/gtd/technik/Chemenu 8.0.0 freigeben.md
- kb/gtd/technik/Windows nativ unterstützen.md
- kb/index.md
- tools/CONTRACT.md
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/review_cmd.py
- tools/chemenu/commands/task_cmd.py
- tools/chemenu/config.py
- tools/chemenu/tasks/config.py
- tools/chemenu/tests/conftest.py
- tools/chemenu/tests/fixtures/sp/MANIFEST.json
- tools/chemenu/tests/fixtures/sp/api/health.json
- tools/chemenu/tests/fixtures/sp/api/projects.json
- tools/chemenu/tests/fixtures/sp/api/tags.json
- tools/chemenu/tests/fixtures/sp/api/tasks.json
- tools/chemenu/tests/fixtures/sp/seed-backup.json
- tools/chemenu/tests/record_sp_fixtures.py
- tools/chemenu/tests/sp_headless.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_review.py
- tools/chemenu/tests/test_sp_recorded.py
- tools/chemenu/tests/test_task_cmd.py
- tools/chemenu/tests/test_tasks_config.py
- tools/chemenu/tests/test_tracker_live.py
- tools/chemenu/tests/tracker_live.py
- tools/pytest.ini
This commit is contained in:
torben committed 2026-09-30 13:23:24 +02:00
1 parent 529793b255
commit b0c64772cc
43 files changed
+2279 -32
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

instructions/dev/tracker-testing.md
+208
View File
@@ -0,0 +1,208 @@
---
type: types/instruction.md
name: tracker-testing
description: How the task-tracker adapters (Super Productivity, CalDAV) are tested against a real tracker - the live suite, the profile procedure for a tracker of your own, the nightly workflow and when an agent triggers it, what a red night means, and how to refresh the recorded fixtures.
---
# Test a tracker adapter against a real tracker
The default `pytest` run never talks to a tracker: it reads recorded answers
(`tools/chemenu/tests/fixtures/sp/`) and fakes. That is enough to keep the parsing honest and
not enough to know that the adapter still works against the tracker as it ships today - a
tracker is software somebody else releases (Gitea #156, and the adapter defects of #162 that
only a real Super Productivity could show). So a second suite exists, marked `live_tracker`,
that runs the documented `task new` / `task list` / `task close` / `review` workflow end to end.
It skips when no tracker is configured and fails when CI says one must be there.
<!-- wikitool:toc -->
## Contents
- [The three ways to name a tracker](#the-three-ways-to-name-a-tracker)
- [What the suite writes, and what it never does](#what-the-suite-writes-and-what-it-never-does)
- [Run it against a tracker of your own](#run-it-against-a-tracker-of-your-own)
- [The two workflows and the image](#the-two-workflows-and-the-image)
- [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run)
- [A red night](#a-red-night)
- [Refreshing the fixtures](#refreshing-the-fixtures)
- [Steps](#steps)
- [Scope](#scope)
<!-- /wikitool:toc -->
## The three ways to name a tracker
All through the environment; none of them is a fixed address, and none of them is read by a
plain `pytest` run unless you set it.
| Variable | Tracker | Who owns it |
|---|---|---|
| `CHEMENU_LIVE_SP_BINARY` | The packaged Super Productivity (`.deb` layout). The suite starts its own headless copy on a fresh profile, seeded from `fixtures/sp/seed-backup.json`, and stops it again. | The suite |
| `CHEMENU_LIVE_CALDAV_URL`, `_USER`, `_PASSWORD` (optional `_VERSION` for the report line) | A CalDAV collection. In CI it is a throwaway Radicale started by `.gitea/scripts/start-radicale.sh`. | The suite - it may create the marker calendar |
| `CHEMENU_LIVE_PROFILE=<name>` | A tracker of your own, described by `.wikitool-tasks.d/<name>.json` in the checkout, used exactly as configured. | You - the suite never creates anything but items |
`CHEMENU_LIVE_REQUIRE=sp,caldav,profile` (any subset) turns "no such tracker configured" into
a failure instead of a skip. CI sets it; without it a workflow that lost its server would go
green by skipping, which is the outcome the suite exists to rule out.
The commands under test read their configuration from `.wikitool-tasks.json`, or from the file
`WIKITOOL_TASKS_CONFIG` names when that variable is set. The live suite sets the variable per
test, so it never touches the checkout's own `.wikitool-tasks.json`; you can use the same
variable to run `task` and `review` by hand against a second tracker.
## What the suite writes, and what it never does
The safety rules are code (`tools/chemenu/tests/tracker_live.py`), and
`test_tracker_live.py` proves them against fakes on every run - they are not a promise in this
file.
- It writes only into the project named **`Chemenu Live-Test`** (`MARKER_PROJECT`). The project
must already exist. The suite creates it only in a tracker it owns end to end (the CalDAV
server it started); in a profile target, or in Super Productivity, a missing marker is an
abort before the first write - Super Productivity's API has no way to create a project anyway.
- Every item it creates starts with `[live-test <run id>]`. On the way out it closes what is
left of that prefix, and only that prefix.
- It never deletes anything. The only closing write is `task close`, the one this stack has.
- The headless Super Productivity refuses to start when anything already answers on the fixed
API port 3876: an answering app is somebody's real one.
- A profile path that starts with `~` is refused. Profiles carry absolute paths, because a `~`
resolves against whichever `HOME` the run has.
## Run it against a tracker of your own
Do this before you change an adapter for a tracker you actually use, or when a user reports one
that fails against theirs. The steps differ per tracker only in the profile and the marker.
1. **Create the marker project in that tracker**, by hand, named exactly `Chemenu Live-Test`.
Nothing else in the tracker is touched, but this project's items are created and closed.
2. **Write the profile** to `.wikitool-tasks.d/<name>.json` (gitignored - it holds
credentials). It is a normal `.wikitool-tasks.json` plus one optional key,
`live_test_version`, which is stripped before use and only names the tracker's version in
the report line. Paths are absolute.
Super Productivity, API access (the app must be running with the Local REST API on):
```json
{
"schema": 1,
"provider": "superproductivity",
"thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
"superproductivity": {"access": "api", "api_base_url": "http://127.0.0.1:3876", "api_token": "<token>"},
"live_test_version": "19.1.0"
}
```
CalDAV:
```json
{
"schema": 1,
"provider": "caldav",
"thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
"caldav": {"url": "https://<server>/<user>/", "username": "<user>", "app_password": "<password>",
"inbox_list": "Inbox", "someday_list": "Someday"},
"live_test_version": "<server and version>"
}
```
A Super Productivity profile with `access: "snapshot"` is accepted too, but the workflow
needs a write path, so the scenario fails on it by design: that access is read-only.
3. **Run it**, from `tools/`:
```bash
CHEMENU_LIVE_PROFILE=<name> CHEMENU_LIVE_REQUIRE=profile \
.venv/bin/python -m pytest -m live_tracker -k profile -s
```
`-s` shows the report line with the tracker version, which is what you paste into an issue.
4. **Read a failure as a finding about that tracker**, not as a flaky test: the scenario is
deterministic. Reproduce it with `task` and `review` under `WIKITOOL_TASKS_CONFIG`, then fix
the adapter and add a recorded fixture for the answer that broke it.
## The two workflows and the image
| Workflow | When | What |
|---|---|---|
| `ci.yml`, step "Live tracker suite (CalDAV)" | Every push and PR | Radicale as a process on loopback (`pip install radicale`), `CHEMENU_LIVE_REQUIRE=caldav`. The one half that needs no app and no display. |
| `tracker-live.yml` | Nightly 05:00 UTC, and by hand | Both kinds, `CHEMENU_LIVE_REQUIRE=sp,caldav`, inside the `chemenu-sp-live` image. It logs the installed Super Productivity version and warns when it differs from the update channel. |
| `sp-live-image.yml` | Daily 04:10 UTC, and by hand | Builds `gitea.nehmer.net/torben/chemenu-sp-live` when the registry lacks the channel's current version, and once a month regardless. |
The image follows the update channel (`latest-linux.yml`, resolved by
`.gitea/sp-live/resolve-version.sh`) and not a pin: installed desktop clients update
themselves, and a pinned old version would be tested while users run the new one. Tags are
`:<version>` and `:latest`; a manual build with an explicit `sp_version` never moves `:latest`,
which is how you reproduce a red night against the version it went red on.
**The first push of the package needs one manual step**, which no workflow can do: link the
package `chemenu-sp-live` to the repository `torben/chemenu` in the Gitea UI, once, and
dispatch `sp-live-image.yml` first - `tracker-live.yml` has nothing to pull before that. The
package is public, so the runner pulls it anonymously.
## When an agent triggers the nightly run
A session that changed the Super Productivity surface does not wait for the clock: after the
push has landed and CI is green, dispatch `tracker-live.yml` and poll it (Gitea MCP,
`actions_run_write` then `actions_run_read`; never an anonymous `curl`). The surface is
`tools/chemenu/tasks/superproductivity.py`, `tools/chemenu/tasks/config.py`,
`tools/chemenu/tests/sp_headless.py`, `tools/chemenu/tests/tracker_live.py`, the fixtures under
`tools/chemenu/tests/fixtures/sp/`, and `.gitea/sp-live/`. A change to the CalDAV adapter needs
no dispatch - `ci.yml` already ran it.
Say in the session summary which run answered and against which Super Productivity version.
## A red night
The nightly run is red for one of three reasons, and they are told apart by the log:
1. **The app changed.** The report line names a Super Productivity version newer than the last
green night. This is the finding the workflow exists for: the adapter must follow. Re-record
the fixtures (below), fix the adapter, and open an issue for it with the two version
numbers.
2. **The image is stale.** The step "Which Super Productivity is this" warned that the
installed version differs from the channel. Dispatch `sp-live-image.yml` with `force`, then
the run again.
3. **The environment.** The app did not start (the abort message carries the last lines of
the app's own log), or the runner could not pull the image. Nothing about the adapter is known yet.
Never skip the suite, never remove `CHEMENU_LIVE_REQUIRE` from a workflow and never mark the
job `continue-on-error` to get a green night: a red night with a cause is the product.
## Refreshing the fixtures
`tools/chemenu/tests/fixtures/sp/api/*.json` are the raw answers of a real Super Productivity;
`seed-backup.json` is a backup the app wrote itself, with the three projects the live suite and
the demo pages use. `MANIFEST.json` says which version and when. Refresh them when a red night
shows a new answer shape, and when the manifest's version is more than a few releases behind.
1. Get a Super Productivity to record from: the current `chemenu-sp-live` image with the
checkout mounted (`docker run --rm -v "$PWD:/work" -w /work/tools ...`, then a venv from
`requirements.txt` inside it), or any machine with the `.deb` installed and
`CHEMENU_LIVE_SP_BINARY` set.
2. From `tools/`, in an environment that has `requirements.txt`:
```bash
.venv/bin/python -m chemenu.tests.record_sp_fixtures chemenu/tests/fixtures/sp/api
```
3. Update `MANIFEST.json` (`sp_version`, `recorded`) by hand - it is a fixture, not a generated
page - and run `test_sp_recorded.py`. If the seed backup is stale because the app's backup
format moved, rebuild it from a backup the new version wrote and keep the three projects.
4. Do not edit the recorded answers to make a test pass. A test that fails against a fresh
recording is the adapter's problem.
## Steps
1. **Decide which kind of change this is.** A change to an adapter's parsing: extend the recorded
fixtures and their replay test first. A change that alters what the adapter sends: the live
suite has to run, so run it against a tracker (above) before publishing.
2. **Run the offline half** with the rest of the suite; it includes the safety tests for the
guard, the profile rules and the port refusal.
3. **Run the live half** for the tracker you changed - CalDAV locally with a Radicale of your
own, Super Productivity in the image or with a profile.
4. **After the push, dispatch `tracker-live.yml`** when the change is on the Super Productivity
surface (see [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run)).
## Scope
Applies to the task-tracker adapters under `tools/chemenu/tasks/` and to the live suite
itself. It is the one place the test conventions are relaxed on purpose:
[testing-conventions.md](testing-conventions.md) describes the hermetic default run, and the
live suite is opt-in, marked `live_tracker`, and may start a real application and talk to a
real server. The default run stays free of both.