fix: Super Productivity API path unwraps the {ok, data} envelope, drops the inbox project, health waits for the renderer (#162)
CI / verify (push) Successful in 1m40s
Release / release (push) Successful in 37s

Files changed:
- CHANGES.md
- VERSION
- tools/CONTRACT.md
- tools/chemenu/commands/doctor.py
- tools/chemenu/tasks/superproductivity.py
- tools/chemenu/tests/test_new_page.py
- tools/chemenu/tests/test_superproductivity.py
- tools/chemenu/tests/test_task_cmd.py
This commit is contained in:
torben committed 2026-09-30 12:17:39 +02:00
1 parent 8be5e6e5f3
commit 529793b255
8 files changed
+267 -58

No files matched your search

+1 -1
View File
@@ -645,7 +645,7 @@ def run_doctor() -> list[Check]:
"(no tracker configured), malformed is a `FAIL`.",
"For a configured `superproductivity` provider, the configured `access` path's own "
"state: `access: \"api\"` reports whether its local REST API answers `GET /health` "
"right now, `access: \"snapshot\"` whether a backup file is ready. The other access "
"with a ready renderer right now, `access: \"snapshot\"` whether a backup file is ready. The other access "
"path is never attempted, and neither state is ever a `FAIL`.",
"For a configured `caldav` provider, whether the server is reachable and Basic auth "
"succeeds - never a `FAIL`; only a broken config block is.",
+90 -40
View File
@@ -38,11 +38,17 @@ CRUD do not exist. Every endpoint but `GET /health` requires
request are not the same failure - `GET /health` answers `503 APP_NOT_READY`
when the backend is up but the renderer is not yet, which this adapter
surfaces as its own message rather than folding into "unreachable".
`GET /projects` runs through `selectUnarchivedProjects` and excludes
`isArchived` projects server-side; the snapshot path below does the same
filtering itself, so the two access paths agree on that without either one
needing to know how the other got there (verified against `master`,
2026-09-20).
Every response is wrapped in an envelope: `{"ok": true, "data": ...}` on
success, `{"ok": false, "error": {"code": ..., "message": ...}}` on failure
(verified live against v19.1.0, Gitea #162). `_ApiClient` unwraps it and fails
loudly on a body that does not carry one. `GET /health` answers
`{"ok": true, "data": {"server": "up", "rendererReady": <bool>}}`; the app only
counts as healthy once `rendererReady` is true.
`GET /projects` excludes `isArchived` projects server-side, but **does** list
the inbox project (`id: "INBOX_PROJECT"`, v19.1.0, Gitea #162) - and the
snapshot path's `project` entity state carries it too. Both paths therefore
drop archived projects and the inbox themselves, so they agree without either
one needing to know how the other got there.
## The two GTD conventions this adapter encodes (#119 D9/D30, corrected #135)
@@ -94,11 +100,11 @@ path `TaskService.setDone(id)` itself takes
either the service or the task reducer. Concretely:
- `isDone` is in `ALLOWED_TASK_FIELDS`, so the route accepts it.
- Marking a task done through this API is **bit-identical** to the UI's own
checkbox: neither sets `doneOn` or any other field - `TaskCopy.doneOn`
exists on the model but nothing in `setDone`'s own call path writes it, so
a task closed here looks exactly like one a human clicked done on, not a
half-written state with a missing timestamp the UI would have set.
- The request carries `isDone: true` and nothing else. The app answers with
`isDone`, `doneOn` and `modified` set (observed live against v19.1.0, Gitea
#162): the store fills the timestamps itself, so a task closed here carries
the same fields as one a human ticked in the UI, not a half-written state.
This adapter never sends `doneOn` or `modified`.
- An unknown task id makes the same handler return `404 TASK_NOT_FOUND`
before any write happens, which this module's `_ApiClient` already turns
into an ordinary `ValidationError` - no separate existence preflight is
@@ -135,17 +141,13 @@ from chemenu.tasks.protocol import (
# - this instance's own convention, not something Super Productivity defines.
WAITING_TAG_TITLE = "waiting"
# Super Productivity's own inbox project id, verified against
# `project.const.ts`/`project.selectors.ts` on `master` (Gitea #132, 2026-09-20):
# a real project entity the store adds to itself if missing
# (`_addInboxProjectIfNecessary`), but `selectUnarchivedProjects` filters it out
# unconditionally by this exact id - so it never appears in `GET /projects`
# (nor in the snapshot path's own `project` entity state, which mirrors that
# filtering, module docstring). `create_item`'s `--inbox` route is the only
# place this module ever writes it; because of the same filter, an item filed
# there is invisible to every `chemenu.review` check that walks
# `TaskReader.projects()` - "Inbox" never appears as a project name to join
# against, not merely one this instance chooses to exclude.
# Super Productivity's own inbox project id. It is a real project entity: both
# `GET /projects` and the backup snapshot list it (verified live against
# v19.1.0, Gitea #162 - an earlier reading of `selectUnarchivedProjects` said
# otherwise). Both readers therefore drop it explicitly (`_is_tracker_project`),
# so "Inbox" never appears as a project name to join against in any
# `chemenu.review` check. `create_item`'s `--inbox` route is the only place
# this module ever writes it.
INBOX_PROJECT_ID = "INBOX_PROJECT"
DEFAULT_API_BASE_URL = "http://127.0.0.1:3876"
@@ -321,6 +323,15 @@ def _follow_up_at(task: dict) -> Optional[date]:
return _iso_day_to_date(task.get("dueDay"))
def _is_tracker_project(record: dict, record_id: Any = None) -> bool:
"""Whether a project record is one the tracker join sees: not archived
(Gitea #133) and not the inbox (Gitea #162). Both access paths apply this
themselves rather than trusting the other end to have filtered."""
if record.get("isArchived"):
return False
return (record.get("id") or record_id) != INBOX_PROJECT_ID
def _is_waiting(task: dict, tags_by_id: dict[str, dict]) -> bool:
for tag_id in task.get("tagIds") or []:
tag = tags_by_id.get(tag_id)
@@ -343,11 +354,7 @@ class SuperProductivityReader:
projects = _entity_state(snapshot, "project", path)
tasks = _entity_state(snapshot, "task", path)
tags = _entity_state(snapshot, "tag", path)
# Both access paths exclude archived projects (Gitea #133) - the API
# does it server-side (`selectUnarchivedProjects`), this path mirrors
# it explicitly so the two agree without either knowing about the
# other.
projects = {pid: p for pid, p in projects.items() if not p.get("isArchived")}
projects = {pid: p for pid, p in projects.items() if _is_tracker_project(p, pid)}
return path, projects, tasks, tags
def projects(self) -> list[ProjectSummary]:
@@ -417,16 +424,20 @@ class SuperProductivityReader:
def health(cfg: SuperProductivityConfig, *, timeout: float = 2.0) -> bool:
"""Whether the local REST API answers `GET /health` right now - the one
unauthenticated endpoint (module docstring). Never raises: an unreachable
app is an ordinary, expected state (`doctor` reports it, it does not
FAIL), not a defect in this adapter."""
"""Whether the local REST API answers `GET /health` with a ready renderer
right now - the one unauthenticated endpoint (module docstring). Never
raises: an unreachable or still-loading app is an ordinary, expected state
(`doctor` reports it, it does not FAIL), not a defect in this adapter."""
url = cfg.api_base_url.rstrip("/") + "/health"
try:
with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - localhost only
return 200 <= response.status < 300
if not 200 <= response.status < 300:
return False
payload = json.loads(response.read())
except (urllib.error.URLError, OSError, ValueError):
return False
data = payload.get("data") if isinstance(payload, dict) else None
return isinstance(data, dict) and data.get("rendererReady") is True
def _expect_list(value: Any, what: str) -> list[dict]:
@@ -438,6 +449,45 @@ def _expect_list(value: Any, what: str) -> list[dict]:
return value
def _envelope_error(payload: Any) -> Optional[dict]:
if isinstance(payload, dict) and payload.get("ok") is False:
error = payload.get("error")
return error if isinstance(error, dict) else {}
return None
def _error_detail(exc: urllib.error.HTTPError) -> str:
"""` (CODE: message)` from an error envelope, else nothing - an error
answer that is not an envelope must not hide the HTTP status."""
try:
error = _envelope_error(json.loads(exc.read()))
except (OSError, ValueError):
return ""
if not error:
return ""
return f" ({error.get('code', '?')}: {error.get('message', '')})"
def _unwrap_envelope(payload: Any, method: str, path: str) -> Any:
"""`data` of an `{"ok": true, "data": ...}` answer. An `ok: false` answer
raises with the API's own code and message; a body without the envelope
raises too - guessing would turn a drifted API into silent wrong data
(Gitea #162)."""
error = _envelope_error(payload)
if error is not None:
raise ValidationError(
f"superproductivity: API refused {method} {path}: "
f"{error.get('code', '?')} - {error.get('message', '')}"
)
if not isinstance(payload, dict) or payload.get("ok") is not True or "data" not in payload:
raise ValidationError(
f"superproductivity: API answer for {method} {path} is not a "
'{"ok": true, "data": ...} envelope - the response shape does not match what '
"this adapter expects."
)
return payload["data"]
class _ApiClient:
"""The one HTTP transport `SuperProductivityApiReader`/`SuperProductivityWriter`
use - a thin, loudly-failing wrapper, not a general REST client. `get` and
@@ -477,7 +527,8 @@ class _ApiClient:
"yet. Wait a moment and retry."
) from exc
raise ValidationError(
f"superproductivity: API returned HTTP {exc.code} for {method} {path}."
f"superproductivity: API returned HTTP {exc.code} for {method} {path}"
f"{_error_detail(exc)}."
) from exc
except (urllib.error.URLError, OSError) as exc:
raise ValidationError(
@@ -485,17 +536,18 @@ class _ApiClient:
"Is Super Productivity running?"
) from exc
try:
return json.loads(response_body)
payload = json.loads(response_body)
except json.JSONDecodeError as exc:
raise ValidationError(
f"superproductivity: API returned unparseable JSON for {method} {path}."
) from exc
return _unwrap_envelope(payload, method, path)
class SuperProductivityApiReader:
"""`TaskReader` over the local REST API (`access: "api"`) - the current
live state, re-fetched on every call. `GET /projects` already excludes
archived projects server-side; open items are counted against
live state, re-fetched on every call. Archived projects and the inbox are
dropped here (`_is_tracker_project`); open items are counted against
`project.taskIds` rather than `GET /tasks?projectId=`, because a subtask
inherits its parent's `projectId` and would otherwise be double-counted
against that filter (Gitea #133) - the same source of truth
@@ -509,11 +561,7 @@ class SuperProductivityApiReader:
projects = _expect_list(self._client.get("/projects"), "/projects")
tasks = _expect_list(self._client.get("/tasks"), "/tasks")
tags = _expect_list(self._client.get("/tags"), "/tags")
# `GET /projects` already runs through `selectUnarchivedProjects`
# server-side - filtered again here so both access paths hold the
# same guarantee (Gitea #133) rather than one of them trusting the
# other end to have done it.
projects = [p for p in projects if not p.get("isArchived")]
projects = [p for p in projects if _is_tracker_project(p)]
tasks_by_id = {t["id"]: t for t in tasks if isinstance(t.get("id"), str)}
tags_by_id = {t["id"]: t for t in tags if isinstance(t.get("id"), str)}
return projects, tasks_by_id, tags_by_id
@@ -671,6 +719,8 @@ class SuperProductivityWriter:
read path growing an SP-specific field for this one caller."""
target = normalize_project_name(project_name)
for record in _expect_list(self._client.get("/projects"), "/projects"):
if not _is_tracker_project(record):
continue
if normalize_project_name(str(record.get("title", ""))) == target:
project_id = record.get("id")
if isinstance(project_id, str) and project_id:
+7 -2
View File
@@ -80,7 +80,7 @@ def _make_api_handler(state: dict):
def do_GET(self): # noqa: N802 - stdlib method name
if self.path == "/health":
self._reply(200, {"ok": True})
self._reply(200, {"server": "up", "rendererReady": True})
return
if self.headers.get("Authorization") != f"Bearer {_API_TOKEN}":
self._reply(401, {"error": "unauthorized"})
@@ -100,7 +100,12 @@ def _make_api_handler(state: dict):
self._reply(404, {"error": "no such endpoint (fixture)"})
def _reply(self, code: int, payload) -> None:
body = json.dumps(payload).encode("utf-8")
# The app's real envelope (v19.1.0, Gitea #162).
if code < 400:
envelope = {"ok": True, "data": payload}
else:
envelope = {"ok": False, "error": {"code": "ERROR", "message": str(payload)}}
body = json.dumps(envelope).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
+130 -10
View File
@@ -4,8 +4,10 @@ The fixture snapshot below mirrors the real shape verified against
`super-productivity/super-productivity`'s `master` branch: a flat top-level
object with `project`/`task`/`tag` as `@ngrx/entity` `{"ids": [...],
"entities": {...}}` maps for the snapshot path, and the same records as a
flat list (the local REST API's own shape) for the API path (see the module
docstring for the exact source files). No test here starts a real Super
flat list for the API path (see the module docstring for the exact source
files). The API fakes answer in the app's real envelope - `{"ok": true,
"data": ...}` / `{"ok": false, "error": {...}}` (v19.1.0, Gitea #162); a fake
that answers without it is how that defect went unnoticed. No test here starts a real Super
Productivity instance or touches anything beyond a loopback socket and
`tmp_path` (`instructions/dev/testing-conventions.md`).
"""
@@ -25,6 +27,17 @@ from chemenu.errors import HumanInterventionRequired, ValidationError
from chemenu.tasks import superproductivity as sp
def _envelope(code: int, payload: Any) -> dict:
"""The local REST API's real answer shape for a fake's `payload`."""
if code < 400:
return {"ok": True, "data": payload}
detail = payload if isinstance(payload, dict) else {}
return {"ok": False, "error": {
"code": detail.get("code", "ERROR"),
"message": detail.get("message", str(detail.get("error", ""))),
}}
def _ms(year: int, month: int, day: int) -> int:
return int(datetime(year, month, day, tzinfo=timezone.utc).timestamp() * 1000)
@@ -408,12 +421,49 @@ class _HealthHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802 - stdlib method name
self.send_response(200)
self.end_headers()
self.wfile.write(b'{"ok": true}')
self.wfile.write(json.dumps(_envelope(200, {"server": "up", "rendererReady": True})).encode())
def log_message(self, *args): # silence stderr noise during the test run
pass
class _RendererLoadingHandler(_HealthHandler):
def do_GET(self): # noqa: N802 - stdlib method name
self.send_response(200)
self.end_headers()
self.wfile.write(json.dumps(_envelope(200, {"server": "up", "rendererReady": False})).encode())
class _BareOkHandler(_HealthHandler):
def do_GET(self): # noqa: N802 - stdlib method name
self.send_response(200)
self.end_headers()
self.wfile.write(b'{"ok": true}')
def _health_with(handler_cls) -> bool:
server = http.server.HTTPServer(("127.0.0.1", 0), handler_cls)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
cfg_ = sp.SuperProductivityConfig(
access=sp.ACCESS_API, backups_dir=None,
api_base_url=f"http://127.0.0.1:{server.server_address[1]}", api_token="t",
)
return sp.health(cfg_, timeout=2.0)
finally:
server.shutdown()
thread.join(timeout=2)
def test_health_is_false_while_the_renderer_is_still_loading():
assert _health_with(_RendererLoadingHandler) is False
def test_health_is_false_without_the_renderer_ready_flag():
assert _health_with(_BareOkHandler) is False
def test_health_is_true_when_the_endpoint_answers(tmp_path):
server = http.server.HTTPServer(("127.0.0.1", 0), _HealthHandler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
@@ -432,13 +482,18 @@ def test_health_is_true_when_the_endpoint_answers(tmp_path):
# --- the API read path (Gitea #133) -------------------------------------------------
def _make_api_handler(routes: dict[str, Any], token: str, *, status_override: dict[str, int] | None = None):
def _make_api_handler(
routes: dict[str, Any], token: str, *, status_override: dict[str, int] | None = None,
raw_body: Any = None,
):
"""`raw_body`, when given, is sent verbatim instead of the envelope - the
one way a test makes the fake answer like an API that has no envelope."""
status_override = status_override or {}
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802
if self.path == "/health":
self._reply(200, {"ok": True})
self._reply(200, {"server": "up", "rendererReady": True})
return
override = status_override.get(self.path)
if override is not None:
@@ -453,7 +508,7 @@ def _make_api_handler(routes: dict[str, Any], token: str, *, status_override: di
self._reply(404, {"error": "not found"})
def _reply(self, code: int, payload: Any) -> None:
body = json.dumps(payload).encode("utf-8")
body = json.dumps(raw_body if raw_body is not None else _envelope(code, payload)).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
@@ -466,8 +521,8 @@ def _make_api_handler(routes: dict[str, Any], token: str, *, status_override: di
@contextlib.contextmanager
def _api_server(routes: dict[str, Any], *, token: str = "test-token", status_override=None):
handler_cls = _make_api_handler(routes, token, status_override=status_override)
def _api_server(routes: dict[str, Any], *, token: str = "test-token", status_override=None, raw_body=None):
handler_cls = _make_api_handler(routes, token, status_override=status_override, raw_body=raw_body)
server = http.server.HTTPServer(("127.0.0.1", 0), handler_cls)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
@@ -569,6 +624,71 @@ def test_api_reader_non_list_response_fails_loud():
reader.projects()
def test_api_reader_rejects_a_bare_list_without_the_envelope():
projects, tasks, tags = _fixture_records()
with _api_server({"/projects": projects}, raw_body=projects) as server:
reader = sp.SuperProductivityApiReader(_api_cfg(server))
with pytest.raises(ValidationError, match="envelope"):
reader.projects()
def test_api_reader_ok_false_surfaces_the_error_code_and_message():
body = {"ok": False, "error": {"code": "APP_NOT_READY", "message": "renderer not ready"}}
with _api_server({"/projects": []}, raw_body=body) as server:
reader = sp.SuperProductivityApiReader(_api_cfg(server))
with pytest.raises(ValidationError, match="APP_NOT_READY - renderer not ready"):
reader.projects()
def test_api_http_error_names_the_error_envelope_code():
with _api_server({}, status_override={"/projects": 400}) as server:
reader = sp.SuperProductivityApiReader(_api_cfg(server))
with pytest.raises(ValidationError, match=r"HTTP 400 .*ERROR"):
reader.projects()
def test_api_reader_excludes_the_inbox_project():
projects, tasks, tags = _fixture_records()
inbox = {"id": sp.INBOX_PROJECT_ID, "title": "Inbox", "created": _ms(2026, 1, 1),
"taskIds": ["t2"], "backlogTaskIds": ["t3"]}
with _api_server({"/projects": [inbox, *projects], "/tasks": tasks, "/tags": tags}) as server:
reader = sp.SuperProductivityApiReader(_api_cfg(server))
assert "Inbox" not in {p.name for p in reader.projects()}
assert reader.open_items("Inbox").count == 0
assert [i.id for i in reader.someday_items()] == ["t3"] # only p1's, not the inbox's copy
def test_snapshot_reader_excludes_the_inbox_project(tmp_path):
snapshot = _snapshot()
snapshot["project"]["entities"][sp.INBOX_PROJECT_ID] = {
"id": sp.INBOX_PROJECT_ID, "title": "Inbox", "created": _ms(2026, 1, 1),
"taskIds": ["t2"], "backlogTaskIds": ["t3"],
}
snapshot["project"]["ids"].append(sp.INBOX_PROJECT_ID)
backups_dir = tmp_path / "backups"
backups_dir.mkdir()
_write_snapshot(backups_dir / "2026-03-01_120000.json", snapshot)
reader = sp.SuperProductivityReader(sp.SuperProductivityConfig(
access=sp.ACCESS_SNAPSHOT, backups_dir=backups_dir,
api_base_url=sp.DEFAULT_API_BASE_URL, api_token=None,
))
assert "Inbox" not in {p.name for p in reader.projects()}
assert reader.open_items("Inbox").count == 0
assert [i.id for i in reader.someday_items()] == ["t3"]
def test_project_id_never_resolves_to_the_inbox_even_under_a_user_project_name():
"""A user project titled like the inbox must not be answered with the
inbox's own id when the writer resolves a name."""
state = {"projects": [
{"id": sp.INBOX_PROJECT_ID, "title": "Inbox", "taskIds": [], "backlogTaskIds": []},
{"id": "p9", "title": "Inbox", "taskIds": [], "backlogTaskIds": []},
]}
with _write_api_server(state) as server:
_writer_for(server).create_item("x", project_name="Inbox")
assert state["posted"][0]["projectId"] == "p9"
def test_api_reader_source_is_live_and_needs_no_network():
"""`source()` on the API path must not itself perform a request - a
static description is correct regardless of reachability (Gitea #133)."""
@@ -591,7 +711,7 @@ def _make_write_handler(state: dict, *, token: str = "test-token"):
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802
if self.path == "/health":
self._reply(200, {"ok": True})
self._reply(200, {"server": "up", "rendererReady": True})
return
if self.headers.get("Authorization") != f"Bearer {token}":
self._reply(401, {"error": "unauthorized"})
@@ -641,7 +761,7 @@ def _make_write_handler(state: dict, *, token: str = "test-token"):
self._reply(404, {"error": "not found"})
def _reply(self, code: int, payload) -> None:
body = json.dumps(payload).encode("utf-8")
body = json.dumps(_envelope(code, payload)).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
+10 -2
View File
@@ -44,7 +44,7 @@ def _make_handler(state: dict, *, token: str = _API_TOKEN):
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802
if self.path == "/health":
self._reply(200, {"ok": True})
self._reply(200, {"server": "up", "rendererReady": True})
return
if self.headers.get("Authorization") != f"Bearer {token}":
self._reply(401, {"error": "unauthorized"})
@@ -91,7 +91,15 @@ def _make_handler(state: dict, *, token: str = _API_TOKEN):
self._reply(404, {"error": "not found"})
def _reply(self, code: int, payload: Any) -> None:
body = json.dumps(payload).encode("utf-8")
# The app's real envelope (v19.1.0, Gitea #162).
if code < 400:
envelope = {"ok": True, "data": payload}
else:
envelope = {"ok": False, "error": {
"code": payload.get("code", "ERROR"),
"message": payload.get("message", str(payload.get("error", ""))),
}}
body = json.dumps(envelope).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()