torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

feat: the shapes of the four hand-written .wikitool-*.json files live in one record each (config_shapes.py); loaders, doctor and docs/reference/configuration.md read them (#193)
CI / verify (push) Successful in 6m17s
Details
CI / pwsh (push) Successful in 2m0s
Details
Release / release (push) Successful in 34s
Details

Browse Source 
The telemetry, remotes, upload and tasks loaders take field names, defaults, coercions and their Expected: text from the record and accept exactly what they did before. doctor WARNs on a telemetry file the policy silently reads differently. configuration.md carries one generated region per file, also checked in an instance. docs reference --apply now refreshes several regions in one page in sequence instead of keeping only the last.

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/set-up-the-mcp-server.md
- docs/reference/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/types-instructions-and-docs.md
- docs/reference/configuration.md
- instructions/dev/doc-pull-through.md
- instructions/setup-instance.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/config.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/tasks/caldav.py
- tools/chemenu/tasks/config.py
- tools/chemenu/tasks/superproductivity.py
- tools/chemenu/telemetry/policy.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_reference_doc.py
- tools/chemenu/upload.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-07 08:44:50 +02:00
1 parent 61e18dc4aa
commit 893db6e640
25 files changed
+1646 -311
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/how-to/set-up-the-mcp-server.md
+3 -15
View File
@@ -161,21 +161,9 @@ reviewed yet.
## Optional: the submit path
Without this, `submit` does not exist as a tool - not unused, but not registered. A file
`.wikitool-upload.json` in the served corpus turns it on:
```json
{
"schema": 1,
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [".md", ".txt", ".pdf", ".html", ".csv", ".json", ".png", ".jpg"],
"quota": { "submissions_per_day": 20, "bytes_per_day": 52428800 }
}
```
Every field is required, and none has a built-in default except `identity_header`. A malformed file
stops the server from starting. It never means "no limits": the failure deliberately goes the safe
way. Size and quota are your deployment's decision, not defaults worth copying blindly.
`.wikitool-upload.json` in the served corpus turns it on. Its fields, defaults and what a malformed
file does are in the [configuration reference](../reference/configuration.md#mcp-submissions); a
broken file never means "no limits", it stops the server from starting.
**With `submit` on, the middleware has a second duty.** It has to set the identity header itself, on
every authenticated request, and discard any copy the client sent. The server trusts that header as
docs/reference/README.md
+6 -5
View File
@@ -14,9 +14,10 @@ written by hand.
- [Page types](page-types.md) - every page type the stack ships for `kb/`, its subtypes, where
its pages are written and its required fields, generated from the type-specs.
- [Configuration](configuration.md) - the environment variables `wikitool` reads and the
per-checkout files an operator may create. Written by hand: it has no source to be generated
from yet.
per-checkout files an operator may create. The shape of each file is generated from the record
its loader reads it with; the environment variables and what each setting does are written by
hand.
The last two describe the stack's shipped defaults, which an instance may change or extend;
`tools/wikitool instructions list` and `tools/wikitool types describe <type>` show what one
instance has.
The skill catalogue and the page types describe the stack's shipped defaults, which an instance
may change or extend; `tools/wikitool instructions list` and `tools/wikitool types describe <type>`
show what one instance has.
docs/reference/commands/instance-health.md
+1 -1
View File
@@ -39,7 +39,7 @@ tools/wikitool doctor --json
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
- For a configured `caldav` provider, whether the server is reachable and Basic auth succeeds - never a `FAIL`; only a broken config block is.
- Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
- Telemetry: on or off and why - installation-form default, `.wikitool-telemetry.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`.
- Telemetry: on or off and why - installation-form default, `.wikitool-telemetry.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when `.wikitool-telemetry.json` is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
- Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
- Read-only and exempt from the Iteration Budget Gate.
docs/reference/commands/types-instructions-and-docs.md
+2 -1
View File
@@ -288,7 +288,7 @@ tools/wikitool docs verify
- Every reference file `docs toc` covers carries the current table-of-contents region for its own headings - missing and stale are one check.
- Every relative markdown link in one of those reference files resolves to an existing file. A target's `#anchor` suffix is stripped first, and code fences and inline code spans are masked before scanning, so link syntax shown as an example is not mistaken for a real reference.
- Every file under `docs/` sits in one of its areas - `tutorials/`, `how-to/`, `reference/`, `explanation/` or `dev/` - never directly in `docs/` or in another directory.
- If `docs/` exists: every generated region of the human reference under `docs/reference/` matches what `docs reference` would write, every `cli_contract.GROUPS` entry has its page under `docs/reference/commands/`, and no page there belongs to no group. The skill catalogue and the page types are checked only in the origin repo (no `.wikitool-release.json`): they describe the stack's shipped defaults, which an instance's own type-specs and instructions may differ from.
- If `docs/` exists: every generated region of the human reference under `docs/reference/` matches what `docs reference` would write, every `cli_contract.GROUPS` entry has its page under `docs/reference/commands/`, and no page there belongs to no group. That includes the four configuration regions in `docs/reference/configuration.md`. The skill catalogue and the page types are checked only in the origin repo (no `.wikitool-release.json`): they describe the stack's shipped defaults, which an instance's own type-specs and instructions may differ from.
- `INSTALL.md` carries one generated `<!-- wikitool:prerequisites -->` region per platform value of `tools/prerequisites.txt` (`prerequisites-<platform>` for a platform-specific one), each current; and the `<!-- setup-question: <key> -->` markers in `instructions/setup-instance.md` and `INSTALL.md` name the same set of keys, so a question the agent asks is never one the human guide leaves out, nor the reverse.
- `VERSION` parses and the newest versioned `CHANGES.md` entry names it. An entry that crosses the compatibility boundary from the last release carries `**Breaking Change:**`, has a migration document or `**Migration:** none required`, and its `**Migration:**` line names every required migration document targeting it.
- The newest versioned `CHANGES.md` entry is at most 32 000 bytes (UTF-8) as `version notes` prints it, and each of its `###` sections is exactly one paragraph of at most 1 000 bytes, heading line excluded. Older entries and a changelog with no versioned entry are not checked.
@@ -495,6 +495,7 @@ tools/wikitool docs reference --apply
- Rewrites the generated region of every page of the human command reference from the `cli_contract` records: one `docs/reference/commands/<group>.md` per `cli_contract.GROUPS` entry (`<!-- wikitool:reference-commands -->`), the overview in `docs/reference/commands/README.md` (`<!-- wikitool:reference-commands-index -->`), and the gates and exit-42 causes in `docs/reference/exit-codes.md` (`<!-- wikitool:reference-exit-codes -->`).
- Also, in the origin repo only, the skill catalogue `docs/reference/skills.md` (`<!-- wikitool:reference-skills -->`: one row per shipped `instructions/<name>/SKILL.md`, its description split at the sentence starting "Use when") and the page types `docs/reference/page-types.md` (`<!-- wikitool:reference-page-types -->`: one section per `root: kb` type-spec, from what `types describe` reads). Both describe the stack's shipped defaults. In an instance - a tree with `.wikitool-release.json` - they are left alone and the output says so: an instance's adopted type-specs and own instructions may differ, and rewriting a shipped page there would block the next `dist upgrade`.
- Also the shapes of the four hand-written configuration files in `docs/reference/configuration.md`, one region each (`<!-- wikitool:reference-config-telemetry -->`, `-remotes`, `-tasks`, `-upload`) inside the hand-written section on that file, rendered from the records in `chemenu.config_shapes` the loaders read the files with. Those records ship with `tools/`, so these regions are written and checked in an instance as well. Several regions in one page are refreshed one after the other into the same text.
- Renders each command's summary, synopsis, options, examples, notes, properties in words, exit codes with their causes and see-also - but not the ON FAILURE reactions or NEVER, which address an agent; `tools/CONTRACT.md` keeps those.
- Refreshes the table of contents of each page it writes in the same step.
- Never creates a page or places a region: a page's title and one-sentence head are written by hand. A group's page is named after the group - lower case, every run of other characters one hyphen.
docs/reference/configuration.md
+185 -56
View File
@@ -4,9 +4,14 @@ The environment variables `wikitool` reads, and the per-checkout files an operat
root of an instance. None of them is needed for an instance to work: an instance installed from a
release runs against the origin's public release feed with no configuration at all.
This page is written by hand. Unlike the command reference, it has no single source in the code it
could be generated from yet, so a change to one of these files' shape has to be carried here
by whoever makes it.
The shape of each file - its fields, what an absent or a malformed file means, an example - is
generated from the record `wikitool` itself reads the file with, so it cannot describe a field the
tool does not take. What a setting does around it is written by hand.
Three more files at the root are written by `wikitool` and never by hand: `.wikitool-release.json`,
`.wikitool-kb.json` and `.wikitool-tools.json` ([AGENTS.md invariant 1](../../AGENTS.md#invariants)).
The last one has its own section [below](#tool-paths), because the preflight that writes it is
something you run.
<!-- wikitool:toc -->
## Contents
@@ -47,15 +52,33 @@ publish. Only a clone of the origin repository, used to develop the stack, has n
traces are the instrument the stack measures itself with, and they are **on**.
To turn the default around, create `.wikitool-telemetry.json` at the root. It is per checkout and
gitignored, with no `.template`, like `.wikitool-remotes.json`:
gitignored, with no `.template`, like `.wikitool-remotes.json`.
<!-- wikitool:reference-config-telemetry -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `enabled` | boolean | no | the installation-form default | Telemetry on or off. `WIKI_TRACE` beats it in both directions. Any other value is read by its truthiness, so `"no"` means on. |
| `max_session_bytes` | positive integer | no | `5242880` | Byte cap per session trace. `WIKI_TRACE_MAX_SESSION_BYTES` beats it. A number written as a string is accepted. |
| `keep_sessions` | positive integer | no | `250` | How many session directories retention keeps. `WIKI_TRACE_KEEP_SESSIONS` beats it. A number written as a string is accepted. |
- **Absent:** The installation-form default applies: on in a clone of the origin repository, off in an instance.
- **Malformed:** Never an error. Unreadable JSON or a non-object counts as an absent file, and a cap that is not a positive integer as an absent cap. `doctor` reports each such deviation as a `WARN` under `telemetry`, naming the field.
- **Unknown keys:** Ignored; `doctor` reports them as a `WARN`.
Example:
```json
{ "enabled": true, "max_session_bytes": 5242880, "keep_sessions": 250 }
{
"enabled": true,
"max_session_bytes": 5242880,
"keep_sessions": 250
}
```
<!-- /wikitool:reference-config-telemetry -->
All three keys are optional. `WIKI_TRACE` still overrides `enabled` in both directions and beats
this file. `wikitool doctor` reports the current state: on or off, why, and how much has been
recorded against both caps. What the caps do and what never reaches a trace:
`wikitool doctor` reports the current state: on or off, why, and how much has been recorded against
both caps. What the caps do and what never reaches a trace:
[telemetry-and-evaluation.md](../explanation/telemetry-and-evaluation.md).
## Tool paths
@@ -74,25 +97,124 @@ file by hand. `doctor` reports under `tool-paths` whether every path still works
## Allowed push URLs
`.wikitool-remotes.json` at the root restricts where `publish` may push, per checkout and
gitignored:
gitignored.
<!-- wikitool:reference-config-remotes -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `allowed_push_urls` | array of strings | yes | - | The push URLs `publish` may write to, compared with the URL the remote resolves to for a push - not with its name. An empty list allows no push at all. |
- **Absent:** Any remote is allowed; the Publish-Remote Gate does not apply.
- **Malformed:** `publish` stops with an error before committing anything. A broken file is never read as "no restriction".
- **Unknown keys:** Ignored.
Example:
```json
{ "allowed_push_urls": ["git@example.org:me/my-wiki.git"] }
{
"schema": 1,
"allowed_push_urls": [
"<url>"
]
}
```
<!-- /wikitool:reference-config-remotes -->
Without the file, any remote is allowed. With it, `publish` refuses with exit 42 (the Publish-Remote
Gate) before pushing to a URL not on the list. A malformed file is an error, not permission. Only you
edit this file. The gate has no token and no flag, and an agent that edits the list to get past a
refusal is opening a gate on its own initiative.
With the file, `publish` refuses with exit 42 (the Publish-Remote Gate) before pushing to a URL not
on the list. Only you edit this file. The gate has no token and no flag, and an agent that edits the
list to get past a refusal is opening a gate on its own initiative.
## Task tracker
**Optional.** The weekly review (`tools/wikitool review`, skill `gtd-weekly-review`) checks the
project pages under `kb/gtd/` against a task tracker. Which tracker that is goes in
`.wikitool-tasks.json` at the root. Like the telemetry and remotes files, it is per checkout and has
no `.template`, and it is **gitignored as soon as a token is in it**. Without the file, no tracker is
configured, which is a valid state, not an error. A broken file, on the other hand, is a `FAIL`: an
unreadable configuration must never pass as "no tracker".
no `.template`, and it is **gitignored as soon as a token is in it**.
<!-- wikitool:reference-config-tasks -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `provider` | string | yes | - | Which adapter the review goes through. The provider's own settings sit in a section of the same name. |
| `thresholds` | object | yes | - | The weekly review's three thresholds. |
| `thresholds.stalled_waiting_days` | positive integer | yes | - | A waiting-for item older than this many days counts as overdue. A number written as a string is accepted. |
| `thresholds.unpaged_project_weeks` | positive integer | yes | - | A tracker project with no `kb/` page is reported once it is older than this many weeks. A number written as a string is accepted. |
| `thresholds.someday_stale_months` | positive integer | yes | - | A someday/maybe item untouched for this many months counts as stale. A number written as a string is accepted. |
- **Absent:** No tracker is configured - a valid state. The weekly review has nothing to check against.
- **Malformed:** An error, and `doctor` reports `FAIL`. A broken file is never read as "no tracker configured".
- **Unknown keys:** Ignored, so the section of a provider not chosen right now can stay in the file.
**The `superproductivity` section.** Connection settings for the Super Productivity adapter.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `access` | string | yes | - | Which way in: `api`, the local REST API (reads the current state and carries the write path), or `snapshot`, the newest backup snapshot (read-only). |
With `access: "api"`:
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `api_base_url` | string | no | `"http://127.0.0.1:3876"` | Where the app's REST API listens. An empty value means the default. |
| `api_token` | string | yes, not empty | - | The token from the app's settings. |
With `access: "snapshot"`:
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `backups_dir` | string | yes, not empty | - | The directory the app writes its backup snapshots to; `~` is expanded. |
- **Unknown keys:** Rejected: a section names only one access path's own fields, so a `backups_dir` beside `access: "api"` is an error, not ignored.
```json
{
"superproductivity": {
"access": "api",
"api_base_url": "http://127.0.0.1:3876",
"api_token": "..."
}
}
```
```json
{
"superproductivity": {
"access": "snapshot",
"backups_dir": "~/.config/superProductivity/backups"
}
}
```
**The `caldav` section.** Connection settings for the CalDAV adapter.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `url` | string | yes, not empty | - | The account's CalDAV calendar home set. A trailing `/` is added when missing. |
| `username` | string | yes, not empty | - | The login name, which can differ from the user id in the URL. |
| `app_password` | string | yes, not empty | - | An app password is recommended over the account password. |
| `inbox_list` | string | yes, not empty | - | The name of the one inbox list. |
| `someday_list` | string | yes, not empty | - | The name of the one someday/maybe list. |
| `exclude_lists` | array of strings | no | `[]` | Existing plain task lists that are not projects. |
- **Unknown keys:** Rejected.
```json
{
"caldav": {
"url": "https://<host>/remote.php/dav/calendars/<user>/",
"username": "<login>",
"app_password": "...",
"inbox_list": "Inbox",
"someday_list": "Someday",
"exclude_lists": [
"..."
]
}
}
```
A complete example:
```json
{
@@ -106,21 +228,16 @@ unreadable configuration must never pass as "no tracker".
"superproductivity": {
"access": "api",
"api_base_url": "http://127.0.0.1:3876",
"api_token": "<token from the SP settings>"
"api_token": "..."
}
}
```
<!-- /wikitool:reference-config-tasks -->
`provider` picks the adapter. The stack ships `superproductivity` and `caldav`.
The `thresholds` block holds the review's three thresholds (configuration, not schema): when a
waiting-for counts as overdue, how old a tracker project without a `kb/` page has to be before it is
reported, and when a someday item counts as stale. A value the provider cannot supply at all - a
The `thresholds` block is configuration, not schema. A value the provider cannot supply at all - a
waiting item with no follow-up date, a tracker project with no creation date - is not skipped silently
but reported as its own finding (`waiting_no_follow_up`/`project_age_unknown`).
The provider block of the same name holds the connection details.
### Super Productivity
`access` is **mandatory and has no fallback**. It chooses between two mutually exclusive ways in:
@@ -129,20 +246,10 @@ The provider block of the same name holds the connection details.
under `backups_dir`. That works without the app running, but only reads. The tracker cannot be
written from there.
```json
"superproductivity": {
"access": "snapshot",
"backups_dir": "~/.config/superProductivity/backups"
}
```
- **`access: "api"`** for a desktop instance: it talks only to the local REST API. That answers only
while the app runs, but it gives the current state and carries the write path. `api_token` is
required, because every endpoint except `GET /health` demands `Authorization: Bearer <token>`.
The block names only the fields of its own way in. A `backups_dir` beside `access: "api"`, or an
`api_token` beside `access: "snapshot"`, is refused when the configuration is read, not ignored.
What can write:
- `tools/wikitool new project` creates a tracker project of the same name only with `access: "api"`,
@@ -159,27 +266,13 @@ from there. `tools/wikitool task list --project` only reads, so it answers with
`caldav` is the standards-based second adapter (RFC 4791/5545). It is verified against Nextcloud
Tasks, with iOS *Reminders* as the mobile client, and built for that split: tick items off on the
phone, maintain them at the desk. Unlike Super Productivity, there is only one way in. CalDAV is
always a network access, so no `access` field is needed:
always a network access, so no `access` field is needed.
```json
"caldav": {
"url": "https://<host>/remote.php/dav/calendars/<user>/",
"username": "<login>",
"app_password": "<Nextcloud app password>",
"inbox_list": "Inbox",
"someday_list": "Someday",
"exclude_lists": ["<an existing list that is not a project>"]
}
```
- `url` points at the account's CalDAV calendar home set. It may be an alias - Nextcloud accepts a
short name there - because every later address comes only from the `href`s the server returns,
never from this URL joined with a name.
- `username` is the login name, which can differ from the user id in the URL. A Nextcloud app
password is recommended, not the account password.
- `url` may be an alias - Nextcloud accepts a short name there - because every later address comes
only from the `href`s the server returns, never from this URL joined with a name.
- `inbox_list` and `someday_list` name the two fixed lists, exactly one of each per instance.
- `exclude_lists` leaves out existing plain task lists that are not projects. A list with a
`VEVENT` component never counts as a project anyway.
- A list with a `VEVENT` component never counts as a project, so `exclude_lists` is needed only for
plain task lists.
A project is a list whose only supported component is `VTODO`. `tools/wikitool new project` creates
it automatically with `MKCALENDAR` - without asking, unlike Super Productivity, because CalDAV has a
@@ -199,5 +292,41 @@ a list. Remove them by hand in Nextcloud when you want to.
## MCP submissions
`.wikitool-upload.json` in the served corpus turns on the MCP read server's optional `submit` tool.
Its shape, and what the authentication in front of the server has to guarantee before you create
it, are in [Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path).
What the authentication in front of the server has to guarantee before you create it is in
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path).
<!-- wikitool:reference-config-upload -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `identity_header` | string | no | `"X-Forwarded-User"` | The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default. |
| `max_bytes` | positive integer | yes | - | Largest single submission, in bytes. A number written as a string is accepted. |
| `allowed_extensions` | array of strings | yes | - | The file extensions `submit` accepts, each starting with `.`. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse. |
| `quota` | object | yes | - | Per-submitter limits over any rolling 24 hours. |
| `quota.submissions_per_day` | positive integer | yes | - | Submissions one submitter may make. A number written as a string is accepted. |
| `quota.bytes_per_day` | positive integer | yes | - | Bytes one submitter may submit. A number written as a string is accepted. |
- **Absent:** The `submit` tool is not registered: the write path does not exist.
- **Malformed:** An error. The MCP server does not start and `doctor` reports `FAIL`. A broken file is never read as "no limits".
- **Unknown keys:** Ignored.
Example:
```json
{
"schema": 1,
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [
".md",
".pdf"
],
"quota": {
"submissions_per_day": 20,
"bytes_per_day": 52428800
}
}
```
<!-- /wikitool:reference-config-upload -->
Size and quota are your deployment's decision, not defaults worth copying blindly.