--- type: types/instruction.md name: tracker-testing description: How the task-tracker adapters (Super Productivity, CalDAV) are tested against a real tracker - the live suite, the profile procedure for a tracker of your own, the nightly workflow and when an agent triggers it, what a red night means, and how to refresh the recorded fixtures. --- # Test a tracker adapter against a real tracker The default `pytest` run never talks to a tracker: it reads recorded answers (`tools/chemenu/tests/fixtures/sp/`) and fakes. That is enough to keep the parsing honest and not enough to know that the adapter still works against the tracker as it ships today - a tracker is software somebody else releases (Gitea #156, and the adapter defects of #162 that only a real Super Productivity could show). So a second suite exists, marked `live_tracker`, that runs the documented `task new` / `task list` / `task close` / `review` workflow end to end. It skips when no tracker is configured and fails when CI says one must be there. ## Contents - [The three ways to name a tracker](#the-three-ways-to-name-a-tracker) - [What the suite writes, and what it never does](#what-the-suite-writes-and-what-it-never-does) - [Run it against a tracker of your own](#run-it-against-a-tracker-of-your-own) - [The two workflows and the image](#the-two-workflows-and-the-image) - [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run) - [A red night](#a-red-night) - [Refreshing the fixtures](#refreshing-the-fixtures) - [Steps](#steps) - [Scope](#scope) ## The three ways to name a tracker All through the environment; none of them is a fixed address, and none of them is read by a plain `pytest` run unless you set it. | Variable | Tracker | Who owns it | |---|---|---| | `CHEMENU_LIVE_SP_BINARY` | The packaged Super Productivity (`.deb` layout). The suite starts its own headless copy on a fresh profile, seeded from `fixtures/sp/seed-backup.json`, and stops it again. | The suite | | `CHEMENU_LIVE_CALDAV_URL`, `_USER`, `_PASSWORD` (optional `_VERSION` for the report line) | A CalDAV collection. In CI it is a throwaway Radicale started by `.gitea/scripts/start-radicale.sh`. | The suite - it may create the marker calendar | | `CHEMENU_LIVE_PROFILE=` | A tracker of your own, described by `.wikitool-tasks.d/.json` in the checkout, used exactly as configured. | You - the suite never creates anything but items | `CHEMENU_LIVE_REQUIRE=sp,caldav,profile` (any subset) turns "no such tracker configured" into a failure instead of a skip. CI sets it; without it a workflow that lost its server would go green by skipping, which is the outcome the suite exists to rule out. The commands under test read their configuration from `.wikitool-tasks.json`, or from the file `WIKITOOL_TASKS_CONFIG` names when that variable is set. The live suite sets the variable per test, so it never touches the checkout's own `.wikitool-tasks.json`; you can use the same variable to run `task` and `review` by hand against a second tracker. ## What the suite writes, and what it never does The safety rules are code (`tools/chemenu/tests/tracker_live.py`), and `test_tracker_live.py` proves them against fakes on every run - they are not a promise in this file. - It writes only into the project named **`Chemenu Live-Test`** (`MARKER_PROJECT`). The project must already exist. The suite creates it only in a tracker it owns end to end (the CalDAV server it started); in a profile target, or in Super Productivity, a missing marker is an abort before the first write - Super Productivity's API has no way to create a project anyway. - Every item it creates starts with `[live-test ]`. On the way out it closes what is left of that prefix, and only that prefix. - It never deletes anything. The only closing write is `task close`, the one this stack has. - The headless Super Productivity refuses to start when anything already answers on the fixed API port 3876: an answering app is somebody's real one. - A profile path that starts with `~` is refused. Profiles carry absolute paths, because a `~` resolves against whichever `HOME` the run has. ## Run it against a tracker of your own Do this before you change an adapter for a tracker you actually use, or when a user reports one that fails against theirs. The steps differ per tracker only in the profile and the marker. 1. **Create the marker project in that tracker**, by hand, named exactly `Chemenu Live-Test`. Nothing else in the tracker is touched, but this project's items are created and closed. 2. **Write the profile** to `.wikitool-tasks.d/.json` (gitignored - it holds credentials). It is a normal `.wikitool-tasks.json` plus one optional key, `live_test_version`, which is stripped before use and only names the tracker's version in the report line. Paths are absolute. Super Productivity, API access (the app must be running with the Local REST API on): ```json { "schema": 1, "provider": "superproductivity", "thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5}, "superproductivity": {"access": "api", "api_base_url": "http://127.0.0.1:3876", "api_token": ""}, "live_test_version": "19.1.0" } ``` CalDAV: ```json { "schema": 1, "provider": "caldav", "thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5}, "caldav": {"url": "https:////", "username": "", "app_password": "", "inbox_list": "Inbox", "someday_list": "Someday"}, "live_test_version": "" } ``` A Super Productivity profile with `access: "snapshot"` is accepted too, but the workflow needs a write path, so the scenario fails on it by design: that access is read-only. 3. **Run it**, from `tools/`: ```bash CHEMENU_LIVE_PROFILE= CHEMENU_LIVE_REQUIRE=profile \ .venv/bin/python -m pytest -m live_tracker -k profile -s ``` `-s` shows the report line with the tracker version, which is what you paste into an issue. 4. **Read a failure as a finding about that tracker**, not as a flaky test: the scenario is deterministic. Reproduce it with `task` and `review` under `WIKITOOL_TASKS_CONFIG`, then fix the adapter and add a recorded fixture for the answer that broke it. ## The two workflows and the image | Workflow | When | What | |---|---|---| | `ci.yml`, step "Live tracker suite (CalDAV)" | Every push and PR | Radicale as a process on loopback (`pip install radicale`), `CHEMENU_LIVE_REQUIRE=caldav`. The one half that needs no app and no display. | | `tracker-live.yml` | Nightly 05:00 UTC, and by hand | Both kinds, `CHEMENU_LIVE_REQUIRE=sp,caldav`, inside the `chemenu-sp-live` image. It logs the installed Super Productivity version and warns when it differs from the update channel. | | `sp-live-image.yml` | Daily 04:10 UTC, and by hand | Builds `gitea.nehmer.net/torben/chemenu-sp-live` when the registry lacks the channel's current version, and once a month regardless. | The image follows the update channel (`latest-linux.yml`, resolved by `.gitea/sp-live/resolve-version.sh`) and not a pin: installed desktop clients update themselves, and a pinned old version would be tested while users run the new one. Tags are `:` and `:latest`; a manual build with an explicit `sp_version` never moves `:latest`, which is how you reproduce a red night against the version it went red on. **The first push of the package needs one manual step**, which no workflow can do: link the package `chemenu-sp-live` to the repository `torben/chemenu` in the Gitea UI, once, and dispatch `sp-live-image.yml` first - `tracker-live.yml` has nothing to pull before that. The package is public, so the runner pulls it anonymously. ## When an agent triggers the nightly run A session that changed the Super Productivity surface does not wait for the clock: after the push has landed and CI is green, dispatch `tracker-live.yml` and poll it (Gitea MCP, `actions_run_write` then `actions_run_read`; never an anonymous `curl`). The surface is `tools/chemenu/tasks/superproductivity.py`, `tools/chemenu/tasks/config.py`, `tools/chemenu/tests/sp_headless.py`, `tools/chemenu/tests/tracker_live.py`, the fixtures under `tools/chemenu/tests/fixtures/sp/`, and `.gitea/sp-live/`. A change to the CalDAV adapter needs no dispatch - `ci.yml` already ran it. Say in the session summary which run answered and against which Super Productivity version. ## A red night The nightly run is red for one of three reasons, and they are told apart by the log: 1. **The app changed.** The report line names a Super Productivity version newer than the last green night. This is the finding the workflow exists for: the adapter must follow. Re-record the fixtures (below), fix the adapter, and open an issue for it with the two version numbers. 2. **The image is stale.** The step "Which Super Productivity is this" warned that the installed version differs from the channel. Dispatch `sp-live-image.yml` with `force`, then the run again. 3. **The environment.** The app did not start (the abort message carries the last lines of the app's own log), or the runner could not pull the image. Nothing about the adapter is known yet. Never skip the suite, never remove `CHEMENU_LIVE_REQUIRE` from a workflow and never mark the job `continue-on-error` to get a green night: a red night with a cause is the product. ## Refreshing the fixtures `tools/chemenu/tests/fixtures/sp/api/*.json` are the raw answers of a real Super Productivity; `seed-backup.json` is a backup the app wrote itself, holding the marker project the live suite writes into and the demo projects - with their items - that the `kb/gtd/` pages of the same names join against. `MANIFEST.json` says which version and when, and names the one edit made to the backup after the app wrote it. Refresh them when a red night shows a new answer shape, and when the manifest's version is more than a few releases behind. 1. Get a Super Productivity to record from: the current `chemenu-sp-live` image with the checkout mounted (`docker run --rm -v "$PWD:/work" -w /work/tools ...`, then a venv from `requirements.txt` inside it), or any machine with the `.deb` installed and `CHEMENU_LIVE_SP_BINARY` set. 2. From `tools/`, in an environment that has `requirements.txt`: ```bash .venv/bin/python -m chemenu.tests.record_sp_fixtures chemenu/tests/fixtures/sp/api ``` 3. Update `MANIFEST.json` (`sp_version`, `recorded`) by hand - it is a fixture, not a generated page - and run `test_sp_recorded.py`. If the seed backup is stale because the app's backup format moved, rebuild it the same way: start the new version on the old seed, let it write its own backup, and keep the projects and items `test_sp_recorded.py` expects. A `kb/gtd/` page's name and the demo project's name must stay equal - the join is by name. 4. Do not edit the recorded answers to make a test pass. A test that fails against a fresh recording is the adapter's problem. ## Steps 1. **Decide which kind of change this is.** A change to an adapter's parsing: extend the recorded fixtures and their replay test first. A change that alters what the adapter sends: the live suite has to run, so run it against a tracker (above) before publishing. 2. **Run the offline half** with the rest of the suite; it includes the safety tests for the guard, the profile rules and the port refusal. 3. **Run the live half** for the tracker you changed - CalDAV locally with a Radicale of your own, Super Productivity in the image or with a profile. 4. **After the push, dispatch `tracker-live.yml`** when the change is on the Super Productivity surface (see [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run)). ## Scope Applies to the task-tracker adapters under `tools/chemenu/tasks/` and to the live suite itself. It is the one place the test conventions are relaxed on purpose: [testing-conventions.md](testing-conventions.md) describes the hermetic default run, and the live suite is opt-in, marked `live_tracker`, and may start a real application and talk to a real server. The default run stays free of both.