Files
chemenu/docs/reference/configuration.md
T
torbenandClaude Code 21cdd787b7
CI / verify (push) Successful in 8m33s
CI / pwsh (push) Successful in 2m35s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped
sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6
Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
2026-10-11 18:36:10 +02:00

28 KiB

Configuration

The environment variables wikitool reads and the two tiered settings files an operator may create at the root of an instance. None of them is needed for an instance to work: an instance installed from a release runs against the origin's public release feed with no configuration at all.

Every setting of a feature - telemetry, the publish allowlist, the task tracker, the MCP submit limits - is a section of those two files, and the shape of each section - its fields, their tier, what an absent or a malformed section means, an example - is generated from the record wikitool itself reads it with, so it cannot describe a field the tool does not take. What a setting does around it is written by hand.

Three more files at the root are written by wikitool and never by hand: .wikitool-release.json, .wikitool-kb.json and .wikitool-tools.json (AGENTS.md invariant 1). The last one has its own section below, because the preflight that writes it is something you run.

Contents

Environment variables

Variable Purpose Fallback
WIKI_AUTHOR Overrides the author name of new source pages git config user.name. Without either, new stops with an ERROR
WIKITOOL_SESSION_ID Scopes the Iteration Budget Gate to a task rather than a terminal, and names the directory a trace lands in A session variable the harness sets itself, where one is known (for example CLAUDE_CODE_SESSION_ID), else the parent process id (instructions/session-setup.md)
WIKITOOL_UPDATE_URL The release feed version check, version notes and dist upgrade --latest ask The value in .wikitool-release.json, else the origin's feed
WIKITOOL_UPDATE_TOKEN A Gitea token for the release feed, sent only to downloads on the feed's own host None. Not needed against the origin, only against a feed in a private repository
WIKITOOL_TASKS_CONFIG Path to a tracker profile that task, review and doctor read instead of the tasks, superproductivity and caldav sections of the settings, for running one checkout against several trackers in turn (below) The sections in wikitool.json and .wikitool-local.json. A path to a file that does not exist is an error, never "no tracker configured"
WIKITOOL_SECRET_<NAME> The value behind a settings reference {"ref": "<name>"}, for CI and containers - <NAME> is the name upper-cased The secrets file in the user's configuration directory (below)
CHEMENU_ROOT Which corpus the package works on, for a caller outside the checkout itself The checkout the package sits in. tools/wikitool behaves the same without it
WIKI_TRACE 0/1 turns telemetry off or on, beating both the default and the telemetry section The installation-form default (below)
WIKI_TRACE_DIR Writes traces somewhere other than reports/telemetry/ reports/telemetry/
WIKI_TRACE_CONTENT 0 keeps only lengths and digests of prompts and replies instead of their text Text is kept
WIKI_TRACE_MAX_CONTENT Per-attribute content cap in characters 60 KiB
WIKI_TRACE_MAX_SESSION_BYTES Byte cap per session trace; beats the section's max_session_bytes 5 MiB
WIKI_TRACE_KEEP_SESSIONS How many session directories retention keeps; beats the section's keep_sessions 250

Tool paths

The preflight writes this file, not you. .wikitool-tools.json at the root holds the absolute paths of Python, git and ripgrep, as the preflight found them on this machine - and of the 1Password CLI op when it is installed. op is optional: the preflight records it when it finds it, reports ABSENT when it does not, and never asks for it. Only an op:// reference in the tiered settings needs it. wikitool starts git and rg from there instead of from PATH. It is per checkout and gitignored, because a path on one machine says nothing about the next.

If the file is missing or incomplete, tools/wikitool does not start (exit 42) and names the preflight. If a tool sits somewhere the preflight does not look, name it: tools/preflight.sh --set rg=<path>, or in PowerShell pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --set rg=<path>. Never edit the file by hand. doctor reports under tool-paths whether every path still works.

Tiered settings

Two files at the root hold the settings of the features that read them this way, and every field declares the tier it belongs to:

Tier File Holds
instance wikitool.json, committed What belongs to the corpus and is the same in every clone
checkout .wikitool-local.json, gitignored What is true only for this clone on this machine
secret neither Tokens and passwords - both files hold only a reference

Each file is a JSON object of sections, one per feature, beside two keys of its own:

{
  "schema": 1,
  "instance_id": "6f1c0e52-3d1b-4b8e-9a51-2f0e7d1c9b44",
  "<section>": {"<field>": "<value>"}
}

instance_id sits in wikitool.json only. The first wikitool config set creates the file and the id; the id names the directory of the secrets file below. The sections are telemetry, remotes, tasks, superproductivity and caldav and upload and sites. config set names the sections it knows, and doctor reports a section it does not read as a WARN.

One place per value. A value in the file its tier does not name is an error naming the file, the key and the right tier - never an override. The same holds for a key set in both files.

References. A field of tier secret takes only a reference; a field marked private - a host name, an internal URL - takes a plain value in .wikitool-local.json or a reference in either file. No other field takes one.

Reference Resolves through
{"ref": "<name>"} - lower-case letters, digits and single _ WIKITOOL_SECRET_<NAME>, else the key <name> in the secrets file
{"ref": "op://<vault>/<item>/<field>"} op read, the 1Password CLI the preflight recorded

The secrets file is ~/.config/chemenu/<instance_id>/secrets.json ($XDG_CONFIG_HOME instead of ~/.config when it is set; %APPDATA%\chemenu\<instance_id>\secrets.json on Windows), a JSON object of names to strings. Outside Windows it must be readable by its owner only (chmod 600); a file others can read is refused.

A reference is resolved when a command asks for that value, never when the files are read. A name nobody provides, an op that is not installed, or an item 1Password does not have ends the command with exit 1; a 1Password that is locked, signed out or waiting for an approval ends it with exit 42, for a human to unlock. Setting 1Password up: use-1password-for-settings.md.

Never in the committed file: a field that names a program to run or affects a gate - not even as a reference. config set does not write such a field either (exit 42); the user edits .wikitool-local.json by hand, and doctor checks it afterwards. The fields marked gate in the tables below are these: the publish allowlist and the upload limits.

wikitool doctor reports one line per tier - settings-instance, settings-checkout, settings-secrets - and for the references only whether they resolve. docs verify holds a committed wikitool.json to the placement rules above, a plain secret or private value above all. Neither file ships with dist export. Why the tiers exist, and what keeping secrets out of the tree does not protect against: configuration-tiers.md.

Telemetry

Telemetry is off in an instance. Every instance carries the .wikitool-release.json of its release, and that is how chemenu.telemetry.policy recognises that nobody asked for telemetry. The same holds for every further clone of the instance, because the file is committed with the first publish. Only a clone of the origin repository, used to develop the stack, has no stamp. There the traces are the instrument the stack measures itself with, and they are on.

To turn the default around, add a telemetry section to .wikitool-local.json, by hand or with wikitool config set telemetry.enabled true. The consent to write prompts to this disk belongs to the checkout, so the section has no instance-tier field and no gate.

Field Tier Type Required Default Description
enabled checkout boolean no the installation-form default Telemetry on or off. WIKI_TRACE beats it in both directions. Any other value is read by its truthiness, so "no" means on.
max_session_bytes checkout positive integer no 5242880 Byte cap per session trace. WIKI_TRACE_MAX_SESSION_BYTES beats it. A number written as a string is accepted.
keep_sessions checkout positive integer no 250 How many session directories retention keeps. WIKI_TRACE_KEEP_SESSIONS beats it. A number written as a string is accepted.
  • Absent: The installation-form default applies: on in a clone of the origin repository, off in an instance.
  • Malformed: The hook that reads it never fails on it: a value it cannot use counts as absent, and an unreadable .wikitool-local.json as an absent section. doctor reports each such deviation as a WARN under telemetry, naming the field.

Example:

{
  "telemetry": {
    "enabled": true,
    "max_session_bytes": 5242880,
    "keep_sessions": 250
  }
}

wikitool doctor reports the current state: on or off, why, and how much has been recorded against both caps. What the caps do and what never reaches a trace: telemetry-and-evaluation.md.

The telemetry hook runs on every tool call and stays silent, so it reads only this section, leniently, and never exits 42: a .wikitool-telemetry.json left over from an earlier version is ignored there (below).

Allowed push URLs

The remotes section of .wikitool-local.json restricts where publish may push.

Field Tier Type Required Default Description
allowed_push_urls checkout, gate array of strings yes - The push URLs publish may write to, compared with the URL the remote resolves to for a push - not with its name. An empty list allows no push at all.
  • Absent: Any remote is allowed; the Publish-Remote Gate does not apply.
  • Malformed: publish stops with an error before committing anything. A broken section is never read as "no restriction".

Example:

{
  "remotes": {
    "allowed_push_urls": [
      "<url>"
    ]
  }
}

With the section, publish refuses with exit 42 (the Publish-Remote Gate) before pushing to a URL not on the list. Only you edit it: the field is marked gate, so config set exits 42 for it. The gate has no token and no flag, and an agent that edits the list to get past a refusal is opening a gate on its own initiative.

Task tracker

Optional. The weekly review (tools/wikitool review, skill gtd-weekly-review) checks the project pages under kb/gtd/ against a task tracker. Which tracker that is, and its connection details, are three sections: tasks (which provider, the review's thresholds), and the section of the provider chosen, superproductivity or caldav. What is the same in every clone - the provider, the thresholds, the list names - sits in wikitool.json; the host, the user name and the folder this machine uses in .wikitool-local.json; a token or password is a reference to a secret, never a value.

Field Tier Type Required Default Description
provider instance string yes - Which adapter the review goes through.
stalled_waiting_days instance positive integer yes - A waiting-for item older than this many days counts as overdue. A number written as a string is accepted.
unpaged_project_weeks instance positive integer yes - A tracker project with no kb/ page is reported once it is older than this many weeks. A number written as a string is accepted.
someday_stale_months instance positive integer yes - A someday/maybe item untouched for this many months counts as stale. A number written as a string is accepted.
  • Absent: No tracker is configured - a valid state. The weekly review has nothing to check against.
  • Malformed: An error, and doctor reports FAIL. A broken section is never read as "no tracker configured".

Example:

{
  "tasks": {
    "provider": "superproductivity",
    "stalled_waiting_days": 14,
    "unpaged_project_weeks": 3,
    "someday_stale_months": 5
  }
}

The thresholds are configuration, not schema. A value the provider cannot supply at all - a waiting item with no follow-up date, a tracker project with no creation date - is not skipped silently but reported as its own finding (waiting_no_follow_up/project_age_unknown).

Super Productivity

Field Tier Type Required Default Description
access instance string yes - Which way in: api, the local REST API (reads the current state and carries the write path), or snapshot, the newest backup snapshot (read-only).
api_base_url checkout, private string no "http://127.0.0.1:3876" With access: "api": where the app's REST API listens. An empty value means the default.
api_token secret string no - With access: "api", required: the token from the app's settings. Only ever a reference.
backups_dir checkout string no - With access: "snapshot", required: the directory the app writes its backup snapshots to; ~ is expanded.
  • Absent: Read only when tasks.provider is superproductivity.
  • Malformed: An error when it is read: a field of the other access path beside access, or a missing api_token or backups_dir, stops the command with exit 1.

Examples:

{
  "superproductivity": {
    "access": "api",
    "api_base_url": "http://127.0.0.1:3876",
    "api_token": {
      "ref": "sp_token"
    }
  }
}
{
  "superproductivity": {
    "access": "snapshot",
    "backups_dir": "~/.config/superProductivity/backups"
  }
}

access is mandatory and has no fallback. The fields of the other access path are an error, not ignored: a backups_dir beside access: "api" stops the command with exit 1, as does an api_token missing from it. It chooses between two mutually exclusive ways in:

  • access: "snapshot" for an instance run headless: it reads only the newest backup snapshot under backups_dir. That works without the app running, but only reads. The tracker cannot be written from there.

  • access: "api" for a desktop instance: it talks only to the local REST API. That answers only while the app runs, but it gives the current state and carries the write path. api_token is required (as a reference), because every endpoint except GET /health demands Authorization: Bearer <token>.

What can write:

  • tools/wikitool new project creates a tracker project of the same name only with access: "api", and even there not automatically. The command stops (exit 42) and asks a human to create it, then continues with --resume.
  • tools/wikitool task new creates a single item without a kb/ page.
  • tools/wikitool task close --id marks an item done. It never deletes one.

With access: "snapshot", all three refuse completely with exit 1, because the tracker is read-only from there. tools/wikitool task list --project only reads, so it answers with either access.

CalDAV

Field Tier Type Required Default Description
url checkout, private string yes, not empty - The account's CalDAV calendar home set. A trailing / is added when missing.
username checkout, private string yes, not empty - The login name, which can differ from the user id in the URL.
app_password secret string yes, not empty - An app password is recommended over the account password. Only ever a reference.
inbox_list instance string yes, not empty - The name of the one inbox list.
someday_list instance string yes, not empty - The name of the one someday/maybe list.
exclude_lists instance array of strings no [] Existing plain task lists that are not projects.
  • Absent: Read only when tasks.provider is caldav.

Example:

{
  "caldav": {
    "url": {
      "ref": "op://Chemenu/caldav/url"
    },
    "username": "<login>",
    "app_password": {
      "ref": "caldav_app_password"
    },
    "inbox_list": "Inbox",
    "someday_list": "Someday",
    "exclude_lists": [
      "..."
    ]
  }
}

caldav is the standards-based second adapter (RFC 4791/5545). It is verified against Nextcloud Tasks, with iOS Reminders as the mobile client, and built for that split: tick items off on the phone, maintain them at the desk. Unlike Super Productivity, there is only one way in. CalDAV is always a network access, so no access field is needed.

  • url may be an alias - Nextcloud accepts a short name there - because every later address comes only from the hrefs the server returns, never from this URL joined with a name.
  • inbox_list and someday_list name the two fixed lists, exactly one of each per instance.
  • A list with a VEVENT component never counts as a project, so exclude_lists is needed only for plain task lists.

A project is a list whose only supported component is VTODO. tools/wikitool new project creates it automatically with MKCALENDAR - without asking, unlike Super Productivity, because CalDAV has a real create command for lists. The uniqueness check runs against every list in the account, including excluded lists, the inbox, someday and mixed calendars. If a new name collides with any of them, nothing is created and the collision is named. Renaming the existing list in Nextcloud stays a manual step.

task new and task close work without restriction on caldav, which has no read-only mode like access: "snapshot". When ticking an item off, task close changes only STATUS, COMPLETED, PERCENT-COMPLETE, LAST-MODIFIED and DTSTAMP on the existing .ics resource. Every other property, including an unknown X- property or an alarm, is kept unchanged. A resource that changed since it was read (an ETag conflict) is not written, and the command stops with exit 1. Nothing is ever deleted. The lists of finished projects stay after archiving, because the adapter never deletes a list. Remove them by hand in Nextcloud when you want to.

Profiles

WIKITOOL_TASKS_CONFIG points at a profile: a file in the format of .wikitool-local.json that holds only the sections tasks, superproductivity and caldav (and schema, and live_test_version for the live suite). While it is set, task, review and doctor read those three sections from the profile instead of from wikitool.json and .wikitool-local.json, so one checkout can be run against several trackers in turn. A profile is checked like the local file, except that a plain instance-tier value is allowed in it; a secret is a reference there too, never a value. The live suite takes its throwaway tokens from WIKITOOL_SECRET_<NAME>.

MCP submissions

The upload section of .wikitool-local.json in the served corpus turns on the MCP read server's optional submit tool. What the authentication in front of the server has to guarantee before you create it is in Set up the MCP read server. Every field is marked gate: the limits are the user's to set by hand, and config set exits 42 for them.

Field Tier Type Required Default Description
identity_header checkout, gate string no "X-Forwarded-User" The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default.
max_bytes checkout, gate positive integer yes - Largest single submission, in bytes. A number written as a string is accepted.
allowed_extensions checkout, gate array of strings yes - The file extensions submit accepts, each starting with .. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse.
quota_submissions_per_day checkout, gate positive integer yes - Submissions one submitter may make over any rolling 24 hours. A number written as a string is accepted.
quota_bytes_per_day checkout, gate positive integer yes - Bytes one submitter may submit over any rolling 24 hours. A number written as a string is accepted.
  • Absent: The submit tool is not registered: the write path does not exist.
  • Malformed: An error. The MCP server does not start and doctor reports FAIL. A broken section is never read as "no limits".

Example:

{
  "upload": {
    "identity_header": "X-Forwarded-User",
    "max_bytes": 10485760,
    "allowed_extensions": [
      ".md",
      ".pdf"
    ],
    "quota_submissions_per_day": 20,
    "quota_bytes_per_day": 52428800
  }
}

Size and quota are your deployment's decision, not defaults worth copying blindly.

Site handlers

tools/wikitool raw fetch knows how some websites serve their pages. For a URL of such a website a site handler fetches the page in the website's standard form, leaves the page furniture out of the derived text and prints hints about the page as CHECK lines; today there are two, both for learn.microsoft.com: learn-microsoft for every page of the host and learn-microsoft-powershell-module for a PowerShell module's overview page, which offers the cmdlet pages it lists as subpages for raw fetch --scope. Handlers are JSON files: the stack's, and the instance's own under sites/ (see How to write a site handler). The sites section is what the instance decides about the handlers, and it is optional: without it every handler applies, in the locale en-us.

  • disabled switches handlers off by name. A page of that website is then handled as any other: fetched under the URL as typed, with its page furniture in the derived text. A name no handler has is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the general one of the same website in force.
  • locale is the language segment a handler for a multi-language website puts into the URL it fetches. The header of the derived text keeps the URL as typed under requested_url.
  • notes adds hint lines of the instance's own, printed after a fetch or an --html derivation of a page of that host, whether or not a handler exists for it. An entry names a host (exact, lower case), optionally a path (a regular expression matched against the start of the URL path, after the standard form was applied) and one to three single-line lines. A host and path pattern appears once; an entry whose path is left out applies to the whole host. A line is plain text and is output as it is, so it is for a reminder to the reader of the capture, never for an instruction the tool should carry out.
  • max_pages and pause_seconds bound raw fetch --scope: the most subpages one page may offer before the fetch is refused, and the pause before each subpage request, which keeps the load on the website low.
Field Tier Type Required Default Description
disabled instance array of strings no [] The names of the site handlers that do not apply. raw fetch then treats a page of that website as it does any other. A name no handler has is an error when raw fetch reads the section, so a typo cannot leave a handler on.
locale instance string no "en-us" The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default.
notes instance array of objects no [] Hint lines of the instance's own, printed as CHECK lines after a fetch or an --html derivation of a page of that host. Each entry has host (exact, lower case), an optional path (a regular expression matched against the start of the URL path) and lines (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once.
max_pages instance positive integer no 200 The most subpages raw fetch --scope fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted.
pause_seconds instance number no 1 The pause, in seconds, before each subpage request of raw fetch --scope. 0 means none. A number written as a string is accepted.
  • Absent: Every handler applies, with the locale en-us and no further hint lines.
  • Malformed: An error when raw fetch reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".

Example:

{
  "sites": {
    "disabled": [
      "learn-microsoft-powershell-module"
    ],
    "locale": "en-us",
    "max_pages": 200,
    "pause_seconds": 1,
    "notes": [
      {
        "host": "learn.microsoft.com",
        "path": "^/[a-z-]+/azure/",
        "lines": [
          "Check the Azure retirement notice at the top of the page."
        ]
      }
    ]
  }
}

All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none of them names a program or opens a gate. A section that does not read correctly stops raw fetch with exit 1 before it fetches anything.

Files of earlier versions

Until version 10 every feature above had a file of its own at the root: .wikitool-telemetry.json, .wikitool-remotes.json, .wikitool-upload.json and .wikitool-tasks.json. Nothing reads them any more. A command that needs the setting such a file held stops with exit 42 and names tools/wikitool config migrate, doctor reports each one as a FAIL, and only the telemetry hook ignores it. config migrate moves the content into the sections above, each value into the file its tier names and each secret into the secrets file, and then removes the old files; it is run once per checkout. .wikitool-tools.json is not among them and stays. The migration step for an instance is 10.0.0-settings-tiers.md.