chemenu/instructions/dev/tracker-testing.md

---
type: types/instruction.md
name: tracker-testing
description: How the task-tracker adapters (Super Productivity, CalDAV) are tested against a real tracker - the live suite, the profile procedure for a tracker of your own, the nightly workflow and when an agent triggers it, what a red night means, and how to refresh the recorded fixtures.
---
# Test a tracker adapter against a real tracker

The default `pytest` run never talks to a tracker: it reads recorded answers
(`tools/chemenu/tests/fixtures/sp/`) and fakes. That is enough to keep the parsing honest and
not enough to know that the adapter still works against the tracker as it ships today - a
tracker is software somebody else releases (Gitea #156, and the adapter defects of #162 that
only a real Super Productivity could show). So a second suite exists, marked `live_tracker`,
that runs the documented `task new` / `task list` / `task close` / `review` workflow end to end.
It skips when no tracker is configured and fails when CI says one must be there.

<!-- wikitool:toc -->
## Contents

- [The three ways to name a tracker](#the-three-ways-to-name-a-tracker)
- [What the suite writes, and what it never does](#what-the-suite-writes-and-what-it-never-does)
- [Run it against a tracker of your own](#run-it-against-a-tracker-of-your-own)
- [The two workflows and the image](#the-two-workflows-and-the-image)
- [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run)
- [A red night](#a-red-night)
- [Refreshing the fixtures](#refreshing-the-fixtures)
- [Steps](#steps)
- [Scope](#scope)
<!-- /wikitool:toc -->

## The three ways to name a tracker

All through the environment; none of them is a fixed address, and none of them is read by a
plain `pytest` run unless you set it.

| Variable | Tracker | Who owns it |
|---|---|---|
| `CHEMENU_LIVE_SP_BINARY` | The packaged Super Productivity (`.deb` layout). The suite starts its own headless copy on a fresh profile, seeded from `fixtures/sp/seed-backup.json`, and stops it again. | The suite |
| `CHEMENU_LIVE_CALDAV_URL`, `_USER`, `_PASSWORD` (optional `_VERSION` for the report line) | A CalDAV collection. In CI it is a throwaway Radicale started by `.gitea/scripts/start-radicale.sh`. | The suite - it may create the marker calendar |
| `CHEMENU_LIVE_PROFILE=<name>` | A tracker of your own, described by a profile `.wikitool-tasks.d/<name>.json` in the checkout, used exactly as configured. | You - the suite never creates anything but items |

`CHEMENU_LIVE_REQUIRE=sp,caldav,profile` (any subset) turns "no such tracker configured" into
a failure instead of a skip. CI sets it; without it a workflow that lost its server would go
green by skipping, which is the outcome the suite exists to rule out.

The commands under test read the `tasks`, `superproductivity` and `caldav` sections of the
settings, or those of the profile `WIKITOOL_TASKS_CONFIG` names when that variable is set. The
live suite sets the variable per test, so it never touches the checkout's own settings; you can use the same
variable to run `task` and `review` by hand against a second tracker.

## What the suite writes, and what it never does

The safety rules are code (`tools/chemenu/tests/tracker_live.py`), and
`test_tracker_live.py` proves them against fakes on every run - they are not a promise in this
file.

- It writes only into the project named **`Chemenu Live-Test`** (`MARKER_PROJECT`). The project
  must already exist. The suite creates it only in a tracker it owns end to end (the CalDAV
  server it started); in a profile target, or in Super Productivity, a missing marker is an
  abort before the first write - Super Productivity's API has no way to create a project anyway.
- Every item it creates starts with `[live-test <run id>]`. On the way out it closes what is
  left of that prefix, and only that prefix.
- It never deletes anything. The only closing write is `task close`, the one this stack has.
- The headless Super Productivity refuses to start when anything already answers on the fixed
  API port 3876: an answering app is somebody's real one.
- A profile path that starts with `~` is refused. Profiles carry absolute paths, because a `~`
  resolves against whichever `HOME` the run has.

## Run it against a tracker of your own

Do this before you change an adapter for a tracker you actually use, or when a user reports one
that fails against theirs. The steps differ per tracker only in the profile and the marker.

1. **Create the marker project in that tracker**, by hand, named exactly `Chemenu Live-Test`.
   Nothing else in the tracker is touched, but this project's items are created and closed.
2. **Write the profile** to `.wikitool-tasks.d/<name>.json` (gitignored). It is in the format of
   `.wikitool-local.json`, holds only the sections `tasks`, `superproductivity` and `caldav`, and
   takes one optional key, `live_test_version`, which is stripped before use and only names the
   tracker's version in the report line. Paths are absolute. **A secret in a profile is a
   reference, never a value** - `{"ref": "<name>"}` - so the file holds no credential. Put the
   value where the reference resolves: the environment variable `WIKITOOL_SECRET_<NAME>` (name in
   capitals) for a run, or 1Password / the secrets file for a standing profile
   ([configuration reference](../../docs/reference/configuration.md#profiles)). The user does
   that; a value never goes through the chat or onto a command line.

   Super Productivity, API access (the app must be running with the Local REST API on):

   ```json
   {
     "schema": 1,
     "tasks": {"provider": "superproductivity", "stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
     "superproductivity": {"access": "api", "api_base_url": "http://127.0.0.1:3876", "api_token": {"ref": "live_sp_token"}},
     "live_test_version": "19.1.0"
   }
   ```

   CalDAV:

   ```json
   {
     "schema": 1,
     "tasks": {"provider": "caldav", "stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
     "caldav": {"url": "https://<server>/<user>/", "username": "<user>", "app_password": {"ref": "live_caldav_password"},
                "inbox_list": "Inbox", "someday_list": "Someday"},
     "live_test_version": "<server and version>"
   }
   ```

   A Super Productivity profile with `access: "snapshot"` is accepted too, but the workflow
   needs a write path, so the scenario fails on it by design: that access is read-only.
3. **Run it**, from `tools/`:

   ```bash
   CHEMENU_LIVE_PROFILE=<name> CHEMENU_LIVE_REQUIRE=profile \
     .venv/bin/python -m pytest -m live_tracker -k profile -s
   ```

   `-s` shows the report line with the tracker version, which is what you paste into an issue.
4. **Read a failure as a finding about that tracker**, not as a flaky test: the scenario is
   deterministic. Reproduce it with `task` and `review` under `WIKITOOL_TASKS_CONFIG`, then fix
   the adapter and add a recorded fixture for the answer that broke it.

## The two workflows and the image

| Workflow | When | What |
|---|---|---|
| `ci.yml`, step "Live tracker suite (CalDAV)" | Every push and PR | Radicale as a process on loopback (`pip install radicale`), `CHEMENU_LIVE_REQUIRE=caldav`. The one half that needs no app and no display. |
| `tracker-live.yml` | Nightly 05:00 UTC, and by hand | Both kinds, `CHEMENU_LIVE_REQUIRE=sp,caldav`, inside the `chemenu-sp-live` image. It logs the installed Super Productivity version and warns when it differs from the update channel. |
| `sp-live-image.yml` | Daily 04:10 UTC, and by hand | Builds `gitea.nehmer.net/torben/chemenu-sp-live` when the registry lacks the channel's current version, and once a month regardless. |

The image follows the update channel (`latest-linux.yml`, resolved by
`.gitea/sp-live/resolve-version.sh`) and not a pin: installed desktop clients update
themselves, and a pinned old version would be tested while users run the new one. Tags are
`:<version>` and `:latest`; a manual build with an explicit `sp_version` never moves `:latest`,
which is how you reproduce a red night against the version it went red on.

**The first push of the package needs one manual step**, which no workflow can do: link the
package `chemenu-sp-live` to the repository `torben/chemenu` in the Gitea UI, once, and
dispatch `sp-live-image.yml` first - `tracker-live.yml` has nothing to pull before that. The
package is public, so the runner pulls it anonymously.

## When an agent triggers the nightly run

A session that changed the Super Productivity surface does not wait for the clock: after the
push has landed and CI is green, dispatch `tracker-live.yml` and poll it (Gitea MCP,
`actions_run_write` then `actions_run_read`; never an anonymous `curl`). The surface is
`tools/chemenu/tasks/superproductivity.py`, `tools/chemenu/tasks/config.py`,
`tools/chemenu/tests/sp_headless.py`, `tools/chemenu/tests/tracker_live.py`, the fixtures under
`tools/chemenu/tests/fixtures/sp/`, and `.gitea/sp-live/`. A change to the CalDAV adapter needs
no dispatch - `ci.yml` already ran it.

Say in the session summary which run answered and against which Super Productivity version.

## A red night

The nightly run is red for one of three reasons, and they are told apart by the log:

1. **The app changed.** The report line names a Super Productivity version newer than the last
   green night. This is the finding the workflow exists for: the adapter must follow. Re-record
   the fixtures (below), fix the adapter, and open an issue for it with the two version
   numbers.
2. **The image is stale.** The step "Which Super Productivity is this" warned that the
   installed version differs from the channel. Dispatch `sp-live-image.yml` with `force`, then
   the run again.
3. **The environment.** The app did not start (the abort message carries the last lines of
   the app's own log), or the runner could not pull the image. Nothing about the adapter is known yet.

Never skip the suite, never remove `CHEMENU_LIVE_REQUIRE` from a workflow and never mark the
job `continue-on-error` to get a green night: a red night with a cause is the product.

## Refreshing the fixtures

`tools/chemenu/tests/fixtures/sp/api/*.json` are the raw answers of a real Super Productivity;
`seed-backup.json` is a backup the app wrote itself, holding the marker project the live suite
writes into and the demo projects - with their items - that the `kb/gtd/` pages of the same
names join against. `MANIFEST.json` says which version and when, and names the one edit made to
the backup after the app wrote it. Refresh them when a red night
shows a new answer shape, and when the manifest's version is more than a few releases behind.

1. Get a Super Productivity to record from: the current `chemenu-sp-live` image with the
   checkout mounted (`docker run --rm -v "$PWD:/work" -w /work/tools ...`, then a venv from
   `requirements.txt` inside it), or any machine with the `.deb` installed and
   `CHEMENU_LIVE_SP_BINARY` set.
2. From `tools/`, in an environment that has `requirements.txt`:

   ```bash
   .venv/bin/python -m chemenu.tests.record_sp_fixtures chemenu/tests/fixtures/sp/api
   ```

3. Update `MANIFEST.json` (`sp_version`, `recorded`) by hand - it is a fixture, not a generated
   page - and run `test_sp_recorded.py`. If the seed backup is stale because the app's backup
   format moved, rebuild it the same way: start the new version on the old seed, let it write
   its own backup, and keep the projects and items `test_sp_recorded.py` expects. A
   `kb/gtd/` page's name and the demo project's name must stay equal - the join is by name.
4. Do not edit the recorded answers to make a test pass. A test that fails against a fresh
   recording is the adapter's problem.

## Steps

1. **Decide which kind of change this is.** A change to an adapter's parsing: extend the recorded
   fixtures and their replay test first. A change that alters what the adapter sends: the live
   suite has to run, so run it against a tracker (above) before publishing.
2. **Run the offline half** with the rest of the suite; it includes the safety tests for the
   guard, the profile rules and the port refusal.
3. **Run the live half** for the tracker you changed - CalDAV locally with a Radicale of your
   own, Super Productivity in the image or with a profile.
4. **After the push, dispatch `tracker-live.yml`** when the change is on the Super Productivity
   surface (see [When an agent triggers the nightly run](#when-an-agent-triggers-the-nightly-run)).

## Scope

Applies to the task-tracker adapters under `tools/chemenu/tasks/` and to the live suite
itself. It is the one place the test conventions are relaxed on purpose:
[testing-conventions.md](testing-conventions.md) describes the hermetic default run, and the
live suite is opt-in, marked `live_tracker`, and may start a real application and talk to a
real server. The default run stays free of both.