chemenu/instructions/dev/tracker-testing.md

type, name, description

type	name	description
types/instruction.md	tracker-testing	How the task-tracker adapters (Super Productivity, CalDAV) are tested against a real tracker - the live suite, the profile procedure for a tracker of your own, the nightly workflow and when an agent triggers it, what a red night means, and how to refresh the recorded fixtures.

Test a tracker adapter against a real tracker

The default pytest run never talks to a tracker: it reads recorded answers (tools/chemenu/tests/fixtures/sp/) and fakes. That is enough to keep the parsing honest and not enough to know that the adapter still works against the tracker as it ships today - a tracker is software somebody else releases (Gitea #156, and the adapter defects of #162 that only a real Super Productivity could show). So a second suite exists, marked live_tracker, that runs the documented task new / task list / task close / review workflow end to end. It skips when no tracker is configured and fails when CI says one must be there.

Contents

• The three ways to name a tracker
• What the suite writes, and what it never does
• Run it against a tracker of your own
• The two workflows and the image
• When an agent triggers the nightly run
• A red night
• Refreshing the fixtures
• Steps
• Scope

The three ways to name a tracker

All through the environment; none of them is a fixed address, and none of them is read by a plain pytest run unless you set it.

Variable	Tracker	Who owns it
CHEMENU_LIVE_SP_BINARY	The packaged Super Productivity (.deb layout). The suite starts its own headless copy on a fresh profile, seeded from fixtures/sp/seed-backup.json, and stops it again.	The suite
CHEMENU_LIVE_CALDAV_URL, _USER, _PASSWORD (optional _VERSION for the report line)	A CalDAV collection. In CI it is a throwaway Radicale started by .gitea/scripts/start-radicale.sh.	The suite - it may create the marker calendar
CHEMENU_LIVE_PROFILE=<name>	A tracker of your own, described by .wikitool-tasks.d/<name>.json in the checkout, used exactly as configured.	You - the suite never creates anything but items

CHEMENU_LIVE_REQUIRE=sp,caldav,profile (any subset) turns "no such tracker configured" into a failure instead of a skip. CI sets it; without it a workflow that lost its server would go green by skipping, which is the outcome the suite exists to rule out.

The commands under test read their configuration from .wikitool-tasks.json, or from the file WIKITOOL_TASKS_CONFIG names when that variable is set. The live suite sets the variable per test, so it never touches the checkout's own .wikitool-tasks.json; you can use the same variable to run task and review by hand against a second tracker.

What the suite writes, and what it never does

The safety rules are code (tools/chemenu/tests/tracker_live.py), and test_tracker_live.py proves them against fakes on every run - they are not a promise in this file.

• It writes only into the project named Chemenu Live-Test (MARKER_PROJECT). The project must already exist. The suite creates it only in a tracker it owns end to end (the CalDAV server it started); in a profile target, or in Super Productivity, a missing marker is an abort before the first write - Super Productivity's API has no way to create a project anyway.
• Every item it creates starts with [live-test <run id>]. On the way out it closes what is left of that prefix, and only that prefix.
• It never deletes anything. The only closing write is task close, the one this stack has.
• The headless Super Productivity refuses to start when anything already answers on the fixed API port 3876: an answering app is somebody's real one.
• A profile path that starts with ~ is refused. Profiles carry absolute paths, because a ~ resolves against whichever HOME the run has.

Run it against a tracker of your own

Do this before you change an adapter for a tracker you actually use, or when a user reports one that fails against theirs. The steps differ per tracker only in the profile and the marker.

1. Create the marker project in that tracker, by hand, named exactly Chemenu Live-Test. Nothing else in the tracker is touched, but this project's items are created and closed.

2. Write the profile to .wikitool-tasks.d/<name>.json (gitignored - it holds credentials). It is a normal .wikitool-tasks.json plus one optional key, live_test_version, which is stripped before use and only names the tracker's version in the report line. Paths are absolute.

   Super Productivity, API access (the app must be running with the Local REST API on):

   {
     "schema": 1,
     "provider": "superproductivity",
     "thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
     "superproductivity": {"access": "api", "api_base_url": "http://127.0.0.1:3876", "api_token": "<token>"},
     "live_test_version": "19.1.0"
   }
   

   CalDAV:

   {
     "schema": 1,
     "provider": "caldav",
     "thresholds": {"stalled_waiting_days": 14, "unpaged_project_weeks": 3, "someday_stale_months": 5},
     "caldav": {"url": "https://<server>/<user>/", "username": "<user>", "app_password": "<password>",
                "inbox_list": "Inbox", "someday_list": "Someday"},
     "live_test_version": "<server and version>"
   }
   

   A Super Productivity profile with access: "snapshot" is accepted too, but the workflow needs a write path, so the scenario fails on it by design: that access is read-only.

3. Run it, from tools/:

   CHEMENU_LIVE_PROFILE=<name> CHEMENU_LIVE_REQUIRE=profile \
     .venv/bin/python -m pytest -m live_tracker -k profile -s
   

   -s shows the report line with the tracker version, which is what you paste into an issue.

4. Read a failure as a finding about that tracker, not as a flaky test: the scenario is deterministic. Reproduce it with task and review under WIKITOOL_TASKS_CONFIG, then fix the adapter and add a recorded fixture for the answer that broke it.

The two workflows and the image

Workflow	When	What
ci.yml, step "Live tracker suite (CalDAV)"	Every push and PR	Radicale as a process on loopback (pip install radicale), CHEMENU_LIVE_REQUIRE=caldav. The one half that needs no app and no display.
tracker-live.yml	Nightly 05:00 UTC, and by hand	Both kinds, CHEMENU_LIVE_REQUIRE=sp,caldav, inside the chemenu-sp-live image. It logs the installed Super Productivity version and warns when it differs from the update channel.
sp-live-image.yml	Daily 04:10 UTC, and by hand	Builds gitea.nehmer.net/torben/chemenu-sp-live when the registry lacks the channel's current version, and once a month regardless.

The image follows the update channel (latest-linux.yml, resolved by .gitea/sp-live/resolve-version.sh) and not a pin: installed desktop clients update themselves, and a pinned old version would be tested while users run the new one. Tags are :<version> and :latest; a manual build with an explicit sp_version never moves :latest, which is how you reproduce a red night against the version it went red on.

The first push of the package needs one manual step, which no workflow can do: link the package chemenu-sp-live to the repository torben/chemenu in the Gitea UI, once, and dispatch sp-live-image.yml first - tracker-live.yml has nothing to pull before that. The package is public, so the runner pulls it anonymously.

When an agent triggers the nightly run

A session that changed the Super Productivity surface does not wait for the clock: after the push has landed and CI is green, dispatch tracker-live.yml and poll it (Gitea MCP, actions_run_write then actions_run_read; never an anonymous curl). The surface is tools/chemenu/tasks/superproductivity.py, tools/chemenu/tasks/config.py, tools/chemenu/tests/sp_headless.py, tools/chemenu/tests/tracker_live.py, the fixtures under tools/chemenu/tests/fixtures/sp/, and .gitea/sp-live/. A change to the CalDAV adapter needs no dispatch - ci.yml already ran it.

Say in the session summary which run answered and against which Super Productivity version.

A red night

The nightly run is red for one of three reasons, and they are told apart by the log:

1. The app changed. The report line names a Super Productivity version newer than the last green night. This is the finding the workflow exists for: the adapter must follow. Re-record the fixtures (below), fix the adapter, and open an issue for it with the two version numbers.
2. The image is stale. The step "Which Super Productivity is this" warned that the installed version differs from the channel. Dispatch sp-live-image.yml with force, then the run again.
3. The environment. The app did not start (the abort message carries the last lines of the app's own log), or the runner could not pull the image. Nothing about the adapter is known yet.

Never skip the suite, never remove CHEMENU_LIVE_REQUIRE from a workflow and never mark the job continue-on-error to get a green night: a red night with a cause is the product.

Refreshing the fixtures

tools/chemenu/tests/fixtures/sp/api/*.json are the raw answers of a real Super Productivity; seed-backup.json is a backup the app wrote itself, with the three projects the live suite and the demo pages use. MANIFEST.json says which version and when. Refresh them when a red night shows a new answer shape, and when the manifest's version is more than a few releases behind.

1. Get a Super Productivity to record from: the current chemenu-sp-live image with the checkout mounted (docker run --rm -v "$PWD:/work" -w /work/tools ..., then a venv from requirements.txt inside it), or any machine with the .deb installed and CHEMENU_LIVE_SP_BINARY set.

2. From tools/, in an environment that has requirements.txt:

   .venv/bin/python -m chemenu.tests.record_sp_fixtures chemenu/tests/fixtures/sp/api
   

3. Update MANIFEST.json (sp_version, recorded) by hand - it is a fixture, not a generated page - and run test_sp_recorded.py. If the seed backup is stale because the app's backup format moved, rebuild it from a backup the new version wrote and keep the three projects.

4. Do not edit the recorded answers to make a test pass. A test that fails against a fresh recording is the adapter's problem.

Steps

1. Decide which kind of change this is. A change to an adapter's parsing: extend the recorded fixtures and their replay test first. A change that alters what the adapter sends: the live suite has to run, so run it against a tracker (above) before publishing.
2. Run the offline half with the rest of the suite; it includes the safety tests for the guard, the profile rules and the port refusal.
3. Run the live half for the tracker you changed - CalDAV locally with a Radicale of your own, Super Productivity in the image or with a profile.
4. After the push, dispatch tracker-live.yml when the change is on the Super Productivity surface (see When an agent triggers the nightly run).

Scope

Applies to the task-tracker adapters under tools/chemenu/tasks/ and to the live suite itself. It is the one place the test conventions are relaxed on purpose: testing-conventions.md describes the hermetic default run, and the live suite is opt-in, marked live_tracker, and may start a real application and talk to a real server. The default run stays free of both.