torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

Mirror docs at v10.0.0

Wiki Mirror committed 2026-10-13 06:21:25 +00:00
1 parent 624366529f
commit 7b705fa5c4
58 files changed
+2311 -467
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Dev-Development.md
+16 -15
@@ -1,4 +1,4 @@
*Generated from [`docs/dev/development.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/dev/development.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/dev/development.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/dev/development.md) at v10.0.0 – edit the repository, not this wiki.*
# Developing this stack
@@ -40,9 +40,9 @@ cd chemenu
Then have the agent run `instructions/bootstrap.md`: the preflight, then
`tools/wikitool instructions sync`. The agent's side of it - what differs here from an instance,
and how `dist export` runs as a test tool - is
[instructions/dev/dev-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/dev-setup.md).
[instructions/dev/dev-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/dev-setup.md).
Real documents for `raw derive` live in [testfiles/](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/testfiles/README.md) and are
Real documents for `raw derive` live in [testfiles/](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/testfiles/README.md) and are
committed, so CI derives them too. The repository is public: that README says what may go in.
What a clone has and an instance does not:
@@ -50,7 +50,7 @@ What a clone has and an instance does not:
- **The demo corpus.** Some 170 pages documenting the stack itself: gates, lint, versioning,
search, the wiki pattern. It is a testbed and a walkable example, not a productive knowledge
base. What may be changed in it is governed by
[instructions/dev/corpus-policy.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/corpus-policy.md).
[instructions/dev/corpus-policy.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/corpus-policy.md).
- **A demo persona** in `USER.md`/`SOUL.md`. `doctor` reports both as filled in. They describe
the demo setup, not you.
- **Telemetry on.** Without `.wikitool-release.json` the clone is the measuring station the stack
@@ -91,7 +91,7 @@ only to downloads on the same host as the feed.
Between two releases the stack carries **one** running version candidate instead of a new
number per bump. The full model - where the state lives, how escalation works, why a number is
only consumed by a release - is in
[instructions/dev/version-parts.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/version-parts.md) and
[instructions/dev/version-parts.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/version-parts.md) and
[docs/explanation/version-model.md](Explanation-Version-model). Here only the sequence, in
the order a session actually goes through it:
@@ -112,7 +112,7 @@ the order a session actually goes through it:
lines where they apply). A smaller change needs nothing beyond its bump title; a larger one
gets a paragraph for its topic, and a short summary is added before the release. The form, and
the size budget `docs verify` and `version release` check, are in
[instructions/dev/version-parts.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/version-parts.md) § The candidate
[instructions/dev/version-parts.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/version-parts.md) § The candidate
model.
3. **Run the checks before anything is published:**
@@ -134,7 +134,7 @@ the order a session actually goes through it:
bump titles gets a summarising heading. Refused when the candidate collected two or more bumps
and the summary from step 2 is still missing - a candidate with exactly one bump is exempt -
and likewise when the entry is over its size budget. Commits and pushes nothing (invariant 5
in [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md)).
in [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md)).
5. **Publish moves `VERSION` on `main`.**
@@ -143,7 +143,7 @@ the order a session actually goes through it:
```
The Mass-Update Gate and the Publish-Remote Gate apply as on every other publish - see
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/gates.md).
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/gates.md).
6. **CI does the rest.** `.gitea/workflows/release.yml` reacts to every push that moves
`VERSION`: a `VERSION` with a suffix (a candidate) makes the job skip cleanly before it even
@@ -166,12 +166,12 @@ the order a session actually goes through it:
own, so a failing mirror leaves the release and its tag standing and marks only itself red.
The three check commands are in step 3 above; what each of them checks is in
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md), where `docs verify` holds it against the actual
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md), where `docs verify` holds it against the actual
CLI. It is deliberately **not** repeated here: a second description of the same commands is
exactly the copy that drifts (AGENTS.md invariant 8), and this document lies outside the files
the command-record check of `docs verify` covers - so a drift here would go unnoticed. What
`pytest` expects of the developer at this point is in
[instructions/dev/testing-conventions.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/testing-conventions.md).
[instructions/dev/testing-conventions.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/testing-conventions.md).
## The CI half
@@ -191,7 +191,7 @@ Super Productivity version. The image follows the desktop clients' update channe
version. Once, after the very first push, the package has to be linked to the `torben/chemenu`
repository by hand. What the suite writes, how to run it against a tracker of your own, and what
a red run means is in
[instructions/dev/tracker-testing.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/tracker-testing.md).
[instructions/dev/tracker-testing.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/tracker-testing.md).
### The Gitea wiki
@@ -281,12 +281,13 @@ low, and three kinds have to be told apart before any of it turns into work:
Three skills (`instructions/dev/`, present only in this origin repository) carry a session that
works on the stack itself rather than on wiki content through three phases: `stack-dev` works
the issue out until its body is "ready", `stack-build` builds, publishes and waits for a green CI
run, `stack-close` checks the body's final state and stale `docs/` and contract prose, and closes
the issue. The handover goes through the state in the tracker, not through a session's context:
run, `stack-close` checks the body's final state and stale `docs/` and contract prose, closes
the issue and, where the body carries a `## Kette` section, names the next link of that chain
with the slash command that starts it. The handover goes through the state in the tracker, not through a session's context:
every phase change works in the same session or after `/clear`. `stack-dev` triggers
automatically; `stack-build` and `stack-close` are started only by the operator, by slash command
(`/stack-build #N`, `/stack-close`) - exactly where the choice falls whether to continue in the
same session or a new one, and on which model. None of the three offers a model switch in the
middle of a session. Rules and the phase table are in
[instructions/dev/stack-mode.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/stack-mode.md), the issue tracker itself
in [instructions/dev/issue-tracking.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/dev/issue-tracking.md).
[instructions/dev/stack-mode.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/stack-mode.md), the issue tracker itself
in [instructions/dev/issue-tracking.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/dev/issue-tracking.md).
Explanation-Architecture.md
+16 -7
@@ -1,4 +1,4 @@
*Generated from [`docs/explanation/architecture.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/architecture.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/architecture.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/architecture.md) at v10.0.0 – edit the repository, not this wiki.*
# Architecture
@@ -43,7 +43,7 @@ citation ids, schema enum values, tags, commands, paths and code. So `GitOps Own
German section heading sit in the same page without contradiction.
Which lines count as identifiers is the stack's rule,
[kb/CONTRACT.md § Language and identifiers](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md#language-and-identifiers).
[kb/CONTRACT.md § Language and identifiers](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md#language-and-identifiers).
*Which language* the prose is in, and what the tool-owned headings are called, is the instance's own
`kb/CONVENTIONS.md`. That is a per-instance decision, not a property of the pattern, so it lives in a
file the instance owns rather than in one the stack ships. A new instance picks its language when it
@@ -83,9 +83,10 @@ chemenu/
│ ├── ingest-large-tree.md
│ ├── ingest-queue.md # Reviewing a submission before wikitool upload accept promotes it
│ └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/
├── mcp-upload/ # QUARANTINE (optional): the MCP `submit` tool's write path, gitignored -
│ # read by no command in the ordinary pipeline; a human reviews it with
│ # `wikitool upload list/show/accept/reject`
├── quarantine/ # QUARANTINE (optional): where the intake channels (MCP `submit`, a synced
│ # folder, a command) put what nobody has reviewed, gitignored - read by
│ # no command in the ordinary pipeline; a human reviews it with
│ # `wikitool upload list/show/accept/reject`, `upload collect` fills it
├── incoming/ # INBOX: content gitignored - drop a file or one folder per source here, `raw accept` promotes it
├── raw/ # INPUT: immutable, untrusted source material
│ ├── CONTRACT.md # Date shard, capture fields, immutability, untrusted content
@@ -142,6 +143,8 @@ chemenu/
│ └── CONTRACT.md # Run keys, required files, how a run closes
├── reports/ # DERIVED: lint reports, traces, eval scores. Gitignored
│ └── CONTRACT.md
├── sites/ # The instance's own site handlers for `raw fetch`, one JSON file per website
│ └── CONTRACT.md # File format, which handler applies, what a broken file does
└── tools/ # COMPILER: the wikitool CLI
├── CONTRACT.md # Command records (generated) and index, maintenance schedule
└── README.md # How wikitool is built and how to change it
@@ -226,12 +229,12 @@ same record is rendered four ways:
command;
- a command that fails with exit 1 prints the record's reactions on stderr, right under its `ERROR`
line;
- [tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md) holds a generated copy for agents, together with the
- [tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md) holds a generated copy for agents, together with the
maintenance schedule;
- [the command reference](Reference-Commands) is a generated copy for humans.
`tools/wikitool docs verify` checks the copies, and every command's flags, against the CLI in both
directions. [tools/README.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/README.md) is the other half: how the CLI is built and how
directions. [tools/README.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/README.md) is the other half: how the CLI is built and how
to add a command. `AGENTS.md` holds the invariants that say when each command is mandatory.
The wiki is also a git repository, and `tools/wikitool publish` is the one way its changes are
@@ -262,6 +265,12 @@ for it.
Setting it up: [set-up-the-mcp-server.md](How-to-Set-up-the-mcp-server).
A reader that cannot ask at all - a Microsoft 365 tenant indexing a OneDrive folder - gets a copy
instead: `tools/wikitool export run` writes the pages a target selects into a folder, as a derived,
read-only version regenerated from the committed pages on every run. Nothing flows back from it;
the export marker and the target's manifest keep its files out of `raw/`
([export-to-a-synced-folder.md](How-to-Export-to-a-synced-folder)).
## Built for IT work
The shipped defaults are tailored to IT work:
Explanation-Configuration-tiers.md
+115
@@ -0,0 +1,115 @@
*Generated from [`docs/explanation/configuration-tiers.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/configuration-tiers.md) at v10.0.0 – edit the repository, not this wiki.*
# Configuration tiers
Settings are split into three tiers: what belongs to the instance (`wikitool.json`, committed),
what belongs to one checkout (`.wikitool-local.json`, gitignored), and secrets, which sit
outside the working tree and are reached only by reference. The files, their fields and the
command that writes them are in [the configuration reference](Reference-Configuration#tiered-settings);
the rules an agent follows are in [AGENTS.md § Settings](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#settings). This page is
about why it is built this way, and what it does not do.
<!-- wikitool:toc -->
## Contents
- [Three kinds of data that used to share files](#three-kinds-of-data-that-used-to-share-files)
- [One place per value, not an override](#one-place-per-value-not-an-override)
- [What runs, and where it pushes, never travels with a pull](#what-runs-and-where-it-pushes-never-travels-with-a-pull)
- [References instead of values](#references-instead-of-values)
- [What this does not protect against](#what-this-does-not-protect-against)
<!-- /wikitool:toc -->
## Three kinds of data that used to share files
Before the tiers, every feature kept its settings in a gitignored file of its own. Three kinds of
data ended up mixed in them:
- **What belongs to the instance.** The weekly review's thresholds are a decision about the
corpus, and every clone of it should make the same one. Gitignoring them meant each clone had
to be told again.
- **What belongs to one checkout.** A tool path, the push URLs a checkout may publish to, the
consent to record telemetry on this disk. Committing any of these would hand every other clone
an answer that is wrong for it rather than missing.
- **Secrets.** A tracker token sat in plain text in the working tree, protected only by
`.gitignore` - one wrong pattern away from a commit, and in plain view of every agent that read
the file in the course of ordinary work.
The tiers give each kind one place, declared per field, so the question "where does this go" is
answered once, by whoever declares the field, rather than each time by whoever writes a value.
## One place per value, not an override
A familiar design lets the local file override the committed one. It is deliberately not this
one: a value in the wrong file is an error. With two possible sources for one value, every
reader first has to work out which one wins, and the answer differs between clones - a reviewer
looking at `wikitool.json` would no longer know what the instance actually runs with. One place
per value keeps the committed file a true statement about every clone.
## What runs, and where it pushes, never travels with a pull
A field that names a program to run, or that arms or loosens a gate, is never in the committed
file - not even as a reference, since a resolved reference is still a value someone else chose.
Otherwise a `git pull` from another clone could change what `wikitool` executes on this machine,
or where `publish` sends this checkout's content. The rule is checked when a field is declared,
in code, so no reader has to remember it; the reasoning is the same as for the gates
([why-gates-are-code.md](Explanation-Why-gates-are-code)). The same fields are also not written by
`config set`: like the Publish-Remote Gate's allowlist, they change by the user's own edit.
## References instead of values
A secret is only ever a reference, and a private value - a host name, an internal URL - may be
one. A reference names where the value lives: a name, which resolves through a
`WIKITOOL_SECRET_<NAME>` variable or a secrets file in the user's own configuration directory, or
an `op://` URI, which resolves through the 1Password CLI.
That is what makes the committed file portable. An instance that is public commits
`wikitool.json` publicly, so a private value can never sit there as text; as an `op://`
reference it can, and every machine signed in to the same 1Password account resolves it to the
same value. In CI, `op` runs with a service-account token and resolves the same `op://`
references. Alternatively, 1Password's own action resolves values into variables before
`wikitool` starts, which serves references by name. This repository's CI does the second.
A plain private value in `.wikitool-local.json` stays on this machine only as long as that file
does. A backup into a synced folder (the `backup` section) copies the file as it is, because it is
the one record of how this checkout is set up, and a restore without it starts from nothing. That
trade is the operator's, made by writing the section by hand. A reference keeps the value out of
that copy as well, and the secrets file is never in it: it lies outside the instance, and a backup
reads nothing outside it.
A reference is resolved only when a command asks for that one value, never when the files are
read, and the value stays in that process's memory. `doctor` reports whether each reference
resolves, never what it resolves to.
One secret has no reference at all: the session `raw login` stores for a website behind a login.
The user never types it, so nobody could put it into a secrets source by hand. `raw login` writes
it into the secrets file itself, under a name derived from the host, and `raw fetch` finds it by
that name. It never goes to 1Password, because it changes with every sign-in and the stack only
ever reads from there. So it sits in the one source the 1Password safeguards below do not cover.
## What this does not protect against
Keeping secrets out of the tree protects against three things: a secret being committed, a
secret sitting in a file an agent reads while doing ordinary work, and a secret appearing in
`wikitool`'s output.
It does not protect against an agent with shell access that sets out to read one. If 1Password
is unlocked in the terminal, the agent can run `op read` itself. It can read the secrets file. In
general, whatever the `wikitool` process can read in a user account, an agent running in that same
account can read too. The stack does not claim otherwise.
What limits the exposure without preventing it:
- a 1Password vault of its own for this instance's entries, so an unlocked `op` reaches nothing
else;
- the approval prompt of the 1Password desktop app, which asks a human before the CLI may read
anything - per terminal session, not per value, so it stops a session nobody approved and not
one that already was;
- in CI, a service account that sees only that vault.
None of these covers a website session from `raw login`, which lives in the secrets file only.
Whoever can read that file can use the website as the user, until the session expires or the user
signs out there.
The rule in [AGENTS.md § Settings](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#settings) - a secret never passes through the
agent - is an instruction, not a barrier. The tiers make following it the easy path. They do not
make breaking it impossible.
Explanation-Knowledge-and-commitment.md
+4 -4
@@ -1,4 +1,4 @@
*Generated from [`docs/explanation/knowledge-and-commitment.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/knowledge-and-commitment.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/knowledge-and-commitment.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/knowledge-and-commitment.md) at v10.0.0 – edit the repository, not this wiki.*
# Why knowledge and commitments are two layers
@@ -6,8 +6,8 @@ Chemenu compiles knowledge into `kb/`, and it also tracks what its operator has
Those look like one subject - both are "things about my projects" - and the stack deliberately
keeps them apart: `kb/gtd/` holds one page per initiative, an external task tracker holds the
open items, and the only thing that crosses between them is a name. This page is about why that
line was drawn there. The rules that follow from it live in [kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md)
and the `review`, `new` and `task new` records in [tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md).
line was drawn there. The rules that follow from it live in [kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md)
and the `review`, `new` and `task new` records in [tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md).
<!-- wikitool:toc -->
## Contents
@@ -142,7 +142,7 @@ is. An instruction that said "open Super Productivity" would bake one instance's
the shared stack, and the next instance - a different context, a different employer, a different
set of constraints - would have to edit prose to change a setting.
So the provider lives in configuration (`.wikitool-tasks.json`), the adapters live behind one
So the provider lives in configuration (the `tasks` section of the settings), the adapters live behind one
protocol, and a capability the provider lacks surfaces as an ordinary tool error rather than as a
paragraph of instruction explaining what this particular tracker cannot do. A provider that
cannot create a project, for instance, stops and asks the operator to do it - the same posture the
Explanation-Language-boundaries.md
+33 -5
@@ -1,15 +1,17 @@
*Generated from [`docs/explanation/language-boundaries.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/language-boundaries.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/language-boundaries.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/language-boundaries.md) at v10.0.0 – edit the repository, not this wiki.*
# Language Boundaries
Two languages run through this repo at once. `kb/` is written in whatever language the instance
chose - German here, and the value lives in `kb/CONVENTIONS.md`'s `language:`. Everything that
tells an agent what to do - [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md), every `CONTRACT.md`, everything under
tells an agent what to do - [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md), every `CONTRACT.md`, everything under
`instructions/` - is written in English, in every instance, whatever the first value says. So
is the stack's own documentation for humans, for a different reason (§ "The stack's human
documentation").
The rule itself is in [AGENTS.md § File naming](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#file-naming). This page holds the
The rules themselves are in [AGENTS.md § File naming](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#file-naming) (the language
of a file) and [AGENTS.md § Conversation language](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#conversation-language) (the
language of what an agent says). This page holds the
part that is not a rule: why the line runs where it does, why the English half is not a setting,
and which argument for it turned out to be wrong.
@@ -20,6 +22,7 @@ and which argument for it turned out to be wrong.
- [Why the control plane's language is English](#why-the-control-planes-language-is-english)
- [Why it is not a parameter](#why-it-is-not-a-parameter)
- [What the KB language still decides](#what-the-kb-language-still-decides)
- [Why a rule alone did not hold](#why-a-rule-alone-did-not-hold)
- [Where the line runs around a page type](#where-the-line-runs-around-a-page-type)
- [The stack's human documentation](#the-stacks-human-documentation)
- [What would put this back on the table](#what-would-put-this-back-on-the-table)
@@ -40,7 +43,7 @@ guidance is still instruction addressed to an agent, and reads exactly like the
four types the stack ships. Ownership says "yours"; the audience has not moved at all.
So the question a line answers is not *whose file is this* but *who reads this line*, which is
the same cut [kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md#language-and-identifiers) already makes inside a
the same cut [kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md#language-and-identifiers) already makes inside a
single page between prose and identifiers - applied one level up, to the halves of a document.
Ownership decides who may change a sentence. The reader decides what language it is in. The two
questions were answered together for as long as they happened to agree.
@@ -107,6 +110,31 @@ So an operator who reads no English gets German pages and German answers from an
English instructions. The English is what the machinery is written in, not what it says back -
with one exception, the stack's documentation for humans (§ "The stack's human documentation").
### Why a rule alone did not hold
For a while the conversation rule was a sentence about files' languages in `AGENTS.md`, and an
instance on German pages still saw an agent that wrote its pages in German and everything else in
English: progress notes between tool calls, the closing report. Pages were right because the
step that writes a page reads `kb/CONVENTIONS.md` first; the chat was wrong because nothing had
put the value in front of the agent by then. Three things stacked up:
- **The rule sat in the wrong section.** A paragraph about the language of files does not read as
a rule about speaking, and it was phrased as a statement of fact rather than an instruction.
It now has a section of its own.
- **The value was one hop away.** `AGENTS.md` ships verbatim to every instance and so cannot
carry the instance's language; it names the file that does. A short ingest has written several
English messages before it opens that file.
- **The context is English.** Instructions, tool output and error text all are, and one short
message from the operator is outweighed by them - the more so for a weaker model.
So the value is also compiled into the published copy of each skill, as a one-line block under
its frontmatter that `wikitool instructions sync` writes: the language tag and a pointer to the
rule, nothing else. The rule stays in `AGENTS.md` and the value in `kb/CONVENTIONS.md`; the block
holds neither as text of its own, only puts the one in front of the task that needs it. It is a
tag, not a language name, because a table of names would be the compiler's opinion about
languages, which `conventions.py` deliberately has none of. A session with no skill is covered by
the `AGENTS.md` section alone.
## Where the line runs around a page type
A page type's contract is where the two languages meet most closely, and it is worth knowing
@@ -121,7 +149,7 @@ beside the template, but that split was made for ownership reasons - so an upgra
the guidance without overwriting what the instance chose - and it leaves this page's argument
untouched: each part is still written in the language its own reader needs, and a type-spec that
declares no `guidance:` keeps both halves in one file with exactly the same rule applying inside
it. [types/type-spec.md § Who owns a type-spec](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/types/type-spec.md#who-owns-a-type-spec) has
it. [types/type-spec.md § Who owns a type-spec](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/types/type-spec.md#who-owns-a-type-spec) has
the split as a table, and [ownership-and-templates.md](Explanation-Ownership-and-templates) § "Where the
file boundary used to strain" has what it cost to keep two audiences in one file for as long as
it did.
Explanation-Model-and-effort-selection.md
+2 -2
@@ -1,10 +1,10 @@
*Generated from [`docs/explanation/model-and-effort-selection.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/model-and-effort-selection.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/model-and-effort-selection.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/model-and-effort-selection.md) at v10.0.0 – edit the repository, not this wiki.*
# Choosing a Claude Code model and effort level
Claude Code exposes three choices this repo has an opinion on: which model a session itself
runs as, what model a spawned subagent gets, and which `/code-review` effort level to pick.
None of them are enforced anywhere - the gates in [instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/gates.md)
None of them are enforced anywhere - the gates in [instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/gates.md)
are code precisely because a model cannot be talked out of them
([why-gates-are-code.md](Explanation-Why-gates-are-code) makes that argument for gates; this page applies
the same axis to who is holding the keyboard). What follows is a reference for making that choice
Explanation-Ownership-and-templates.md
+8 -8
@@ -1,11 +1,11 @@
*Generated from [`docs/explanation/ownership-and-templates.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/ownership-and-templates.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/ownership-and-templates.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/ownership-and-templates.md) at v10.0.0 – edit the repository, not this wiki.*
# Ownership and Templates
Chemenu ships two kinds of files side by side, and at a glance they look the same: both are
plain markdown, both sit in the repo root or under `kb/`, both get read at session start. But a
stack upgrade treats them completely differently. Some - [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md),
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md), the per-stage contracts - are identical in every instance
stack upgrade treats them completely differently. Some - [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md),
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md), the per-stage contracts - are identical in every instance
that runs this stack and are the next release's to replace (with one caveat about local edits,
below). Others - `USER.md`,
`SOUL.md`, `kb/CONVENTIONS.md`, `ENVIRONMENT.md` - describe one particular instance, and
@@ -27,7 +27,7 @@ overwriting them would silently erase a choice someone made on purpose.
The stack-owned files describe how the tool works. `kb/CONTRACT.md` opens by saying it holds
what `tools/wikitool` enforces or what follows mechanically from how it operates - see
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md), lines 10-13. That kind of statement doesn't vary by
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md), lines 10-13. That kind of statement doesn't vary by
instance: the compiler behaves the same way regardless of who is running it, so the sentence
describing that behavior can be copied byte-for-byte into every checkout without becoming
wrong anywhere.
@@ -35,7 +35,7 @@ wrong anywhere.
The instance-owned files describe a choice: which language pages are written in, what tone the
agent takes, who the operator is, which git remote is authoritative, which MCP servers are
reachable. None of that follows from the tool's mechanics - two instances of the identical
stack can answer all of these differently and both be correct. [AGENTS.md § Personalization](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#personalization)
stack can answer all of these differently and both be correct. [AGENTS.md § Personalization](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#personalization)
frames the split the same way for `kb/CONTRACT.md` versus `kb/CONVENTIONS.md`: "the split is by
who may change the sentence, not by what it is about." A rule about page structure could in
principle have been written per-instance too, but then every instance answering "not German" to
@@ -61,7 +61,7 @@ the file itself already answers that.
For a while the boundary was written down as a list of paths - once in `dist_cmd.py`, once in
the merge procedure a private instance was told to run by hand, and once in the check that
procedure ended with. Three copies of one fact, which is the shape [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md)
procedure ended with. Three copies of one fact, which is the shape [AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md)
invariant 8 exists to forbid, and they drifted exactly as predicted: the hand-run procedure was
still naming three paths after the collection contracts had moved to the instance's side of the
line, so it discarded upstream changes to files it had never heard of, while its own final check
@@ -83,7 +83,7 @@ The mechanism for instance-owned content is a `.template` file the distribution
of the real one - `USER.md.template`, `SOUL.md.template`, `kb/CONVENTIONS.md.template`,
`ENVIRONMENT.md.template`. An alternative would have been to ship nothing at all and let a
brand-new instance start from a blank page. The template exists because a blank page doesn't
tell [instructions/setup-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/setup-instance.md) what shape the answer
tell [instructions/setup-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/setup-instance.md) what shape the answer
should take, and it gives nothing for a validator to check afterward.
A template carries a placeholder value - a sentinel - in the fields that need a real answer.
@@ -91,7 +91,7 @@ Setup interviews the operator and replaces the sentinel with what they actually
gives `doctor` a mechanical way to tell "personalized" from "not yet": a file that still
contains the sentinel hasn't been through setup, regardless of whether the file exists. That's
also why `ENVIRONMENT.md` only warrants a WARN rather than a FAIL when absent - see
[AGENTS.md § Environment](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#environment) - while a missing or unfilled
[AGENTS.md § Environment](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#environment) - while a missing or unfilled
`USER.md`/`SOUL.md`/`kb/CONVENTIONS.md` is a harder failure: `ENVIRONMENT.md` describes one
checkout among possibly several and is gitignored for that reason, so its absence is a normal
state rather than a sign setup was skipped.
Explanation-Pipeline-rationale.md
+13 -11
@@ -1,16 +1,16 @@
*Generated from [`docs/explanation/pipeline-rationale.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/pipeline-rationale.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/pipeline-rationale.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/pipeline-rationale.md) at v10.0.0 – edit the repository, not this wiki.*
# Why the pipeline has four stages
Chemenu could, in principle, be one directory: drop a file in, ask a question, get an answer
computed fresh each time. It isn't built that way. The pipeline in
[AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#routing) - `raw/` -> `[types/ + tools/]` -> `kb/` -> `reports/`, with
[AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#routing) - `raw/` -> `[types/ + tools/]` -> `kb/` -> `reports/`, with
`work/` alongside rather than inside it - separates *material* from *meaning* from
*byproduct*, and each seam exists because collapsing it costs something specific.
## Why raw material stays untouched
[raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/raw/CONTRACT.md) keeps a source exactly as it arrived. The reasoning is
[raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/raw/CONTRACT.md) keeps a source exactly as it arrived. The reasoning is
simple once stated: the moment someone "cleans up" or reformats a source on the way in, the
thing later claims get checked against is no longer the thing that was actually said. An
immutable `raw/` means a citation always resolves to the original, not to somebody's tidied
@@ -21,9 +21,11 @@ not yet reviewed and not yet a citation target, so its being gitignored and read
ingest session does not weaken anything - the boundary is the promotion into `raw/` itself, not
the moment a human happened to drop a file somewhere.
Once a document can arrive from *outside* - the MCP server's optional `submit` tool - "a human
happened to drop a file somewhere" stops describing how everything reaches `incoming/`, so the
near side of the boundary gets a stage of its own rather than a second meaning. `mcp-upload/`
Once a document can arrive from *outside* - the MCP server's optional `submit` tool, a folder a
sync client fills, a program of the checkout - "a human happened to drop a file somewhere" stops
describing how everything reaches `incoming/`, so the near side of the boundary gets a stage of its
own rather than a second meaning. Every such channel ends in the same `quarantine/`, because what
the boundary protects against is the lack of a reviewer, not the transport. `quarantine/`
holds what nobody has looked at yet; `incoming/` holds what someone has. Two arrows, two
different things being granted: `upload accept` grants *trust* (a human read the material and
took responsibility for it), `raw accept` grants *immutability* (it becomes a citation target
@@ -33,19 +35,19 @@ promotion into `raw/` quietly doubling as the review step it cannot perform.
## Why extraction happens once, through a schema
[types/type-spec.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/types/type-spec.md) is what stands between a raw file and a `kb/` page:
[types/type-spec.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/types/type-spec.md) is what stands between a raw file and a `kb/` page:
a type-spec defines what a conforming instance of a page looks like, and the compiler
(`tools/wikitool`) applies it. The alternative - every query re-reading and re-interpreting the
source on demand - would mean paying the cost of understanding the material every single time,
and getting a slightly different answer each time depending on how the question was phrased.
Extracting once, against a fixed schema, turns "re-read and re-guess" into "look up what was
already compiled." That is the "never re-derive, always compile" principle from
[AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md): understanding a source is expensive and worth doing exactly once,
[AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md): understanding a source is expensive and worth doing exactly once,
after which it becomes a cheap, stable lookup.
## Why a `kb/` page has to stand on its own
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md) sets the bar for the compiled layer: a page should answer a
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md) sets the bar for the compiled layer: a page should answer a
future question without sending the reader back to the source it came from. That's the payoff
of compiling in the first place - if every answer still bottomed out in "go re-read the raw
file," the `kb/` layer would just be a pointer with extra steps, and the cost of extraction
@@ -54,7 +56,7 @@ consistent to query: the work of understanding is already sitting there, done.
## Why `reports/` doesn't need to be maintained
[reports/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/reports/CONTRACT.md) treats most of what lands in `reports/` -
[reports/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/reports/CONTRACT.md) treats most of what lands in `reports/` -
lint output, telemetry traces - as disposable. The structural content of a lint report can be
recomputed from the tree at any commit, so keeping an old copy around would just be a second
version of something the tool can already answer on demand, and a second copy is exactly the
@@ -66,7 +68,7 @@ distinction between what's recomputable and what isn't.
## Where `work/` fits
[work/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/work/CONTRACT.md) describes a workshop, not a fifth pipeline stage: a
[work/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/work/CONTRACT.md) describes a workshop, not a fifth pipeline stage: a
place for the notes, extracts and open decisions of a task that spans more than one session, on
its way toward becoming a `kb/` page. It sits beside the raw -> kb -> reports flow rather than
inside it - closer in spirit to a desk than to a conveyor belt.
Explanation-Telemetry-and-evaluation.md
+8 -8
@@ -1,10 +1,10 @@
*Generated from [`docs/explanation/telemetry-and-evaluation.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/telemetry-and-evaluation.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/telemetry-and-evaluation.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/telemetry-and-evaluation.md) at v10.0.0 – edit the repository, not this wiki.*
# Telemetry and evaluation
How the stack observes what an agent did, and how that record turns into a score. This page explains
the design and points at the code. The rules an agent follows live in
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md) and [reports/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/reports/CONTRACT.md).
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md) and [reports/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/reports/CONTRACT.md).
Turning tracing on and reading a session back is a how-to:
[trace-and-score-sessions.md](How-to-Trace-and-score-sessions).
@@ -93,7 +93,7 @@ sides on the same id.
## The trace
One JSON object per line, appended to `reports/telemetry/<session>/trace.jsonl`. The contract is
[tools/chemenu/telemetry/schema.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/chemenu/telemetry/schema.py).
[tools/chemenu/telemetry/schema.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/chemenu/telemetry/schema.py).
| Field | Meaning |
|---|---|
@@ -128,7 +128,7 @@ needs, and charging for a `search` would discourage the one habit that lowers co
## Whether it runs at all
The default depends on how the tree got here, not on one hard-coded switch.
[tools/chemenu/telemetry/policy.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/chemenu/telemetry/policy.py) is the one place that
[tools/chemenu/telemetry/policy.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/chemenu/telemetry/policy.py) is the one place that
resolves it, so `wikitool doctor`, the writer and the MCP server's start-up guard all answer the
same question the same way:
@@ -144,8 +144,8 @@ where the traces are the stack's own measuring instrument. An instance commits t
first `publish`, so a clone of it on a second machine carries the stamp and defaults to off too. It
is a consuming instance, not a measuring stand.
**Turning it on for a distributed instance** takes a per-checkout `.wikitool-telemetry.json` at the
root, gitignored like `.wikitool-remotes.json` and for the same reason. The consent to write
**Turning it on for a distributed instance** takes a `telemetry` section in the per-checkout `.wikitool-local.json` at the
root, which is gitignored and for the same reason as every other setting that belongs to one checkout. The consent to write
prompts in cleartext to *this* disk belongs to the checkout, not the corpus, so a second clone must
not inherit it silently. The setup instruction asks about it; nothing writes it automatically. Its
shape is in the [configuration reference](Reference-Configuration#telemetry).
@@ -173,7 +173,7 @@ not on the existing history.
Prompts and assistant replies **are** recorded in cleartext, locally. A failure taxonomy cannot be
read out of hashes, and building one is the first phase of any evaluation work. Four guards make
that defensible, all of them in
[tools/chemenu/telemetry/scrub.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/chemenu/telemetry/scrub.py), so there is one place to
[tools/chemenu/telemetry/scrub.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/chemenu/telemetry/scrub.py), so there is one place to
audit:
1. **Secret scrubbing.** Tokens, keys, `Authorization:` headers and `SECRET=` assignments are
@@ -182,7 +182,7 @@ audit:
3. **A kill switch.** `WIKI_TRACE_CONTENT=0` keeps only `<field>_length` and `<field>_sha256`. The
digest is computed either way, so traces stay comparable.
4. **`raw/` contents never enter a trace at all.** That text is data, not instruction
([AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md) invariant 4), and a trace gets read back later. Callers record a path
([AGENTS.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md) invariant 4), and a trace gets read back later. Callers record a path
and a digest.
Nothing leaves the machine. `reports/` is gitignored, no exporter is configured, and where a vendor
Explanation-Version-model.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/explanation/version-model.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/version-model.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/version-model.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/version-model.md) at v10.0.0 – edit the repository, not this wiki.*
# Why the stack version splits compatibility from migration
Explanation-Why-gates-are-code.md
+5 -5
@@ -1,11 +1,11 @@
*Generated from [`docs/explanation/why-gates-are-code.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/why-gates-are-code.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/why-gates-are-code.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/why-gates-are-code.md) at v10.0.0 – edit the repository, not this wiki.*
# Why gates are code
Chemenu's hard limits - the gates [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#gates) lists - live inside
Chemenu's hard limits - the gates [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#gates) lists - live inside
`tools/wikitool`, not in a paragraph of instructions an agent reads and follows. The rules
themselves, and what to do when one trips, are in [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#gates) and
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/gates.md). This page is only about the design choice
themselves, and what to do when one trips, are in [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#gates) and
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/gates.md). This page is only about the design choice
underneath them: why code, and why these mechanisms in particular.
<!-- wikitool:toc -->
@@ -46,7 +46,7 @@ That's not a per-push judgment, it's a standing property of the checkout - true
every publish that checkout will ever attempt, not just this one. A confirm token would let an
agent clear it once and then treat the answer as settled, which is exactly backwards for a
question whose answer shouldn't move at all mid-session. The only way past it is the user
editing `.wikitool-remotes.json` directly, outside the gate's own flow.
editing the `remotes` section of `.wikitool-local.json` directly, outside the gate's own flow.
The Upload Review Gate asks the Mass-Update Gate's own question - *is this change right?* - at
the opposite end of its size range: one file from a stranger instead of a changeset from the
Explanation.md
+3 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/explanation/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/explanation/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/explanation/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/explanation/README.md) at v10.0.0 – edit the repository, not this wiki.*
# Explanation
@@ -20,5 +20,7 @@ lives in `AGENTS.md` or a `CONTRACT.md`, and a page here only gives its reasons.
and a migration question separately
- [knowledge-and-commitment.md](Explanation-Knowledge-and-commitment) - why commitments live in a task
tracker rather than in `kb/`
- [configuration-tiers.md](Explanation-Configuration-tiers) - why settings sit in three tiers, and what
keeping secrets out of the tree does not protect against
- [model-and-effort-selection.md](Explanation-Model-and-effort-selection) - which model and effort suit
which kind of session, for the harness that has to choose
Home.md
+6 -6
@@ -1,4 +1,4 @@
*Generated from [`README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/README.md) at v10.0.0 – edit the repository, not this wiki.*
# Chemenu - Personal IT Knowledge Base
@@ -53,7 +53,7 @@ The human documentation lives under `docs/`, sorted by what you want from it:
- [docs/reference/](Reference) - looking up: commands, exit codes, skills, page
types, configuration. The command reference is
[docs/reference/commands/](Reference-Commands); agents use the same records in
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md).
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md).
- [docs/explanation/](Explanation) - understanding: why the stack is built the way
it is.
@@ -74,7 +74,7 @@ takes a sample document that ships with it through an ingest and a cited answer,
## Changelog
Changes to the wiki stack (schema, skills, `wikitool`, READMEs) are tracked in
[`CHANGES.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/CHANGES.md), not in an inline version history here.
[`CHANGES.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/CHANGES.md), not in an inline version history here.
## Resources
@@ -88,12 +88,12 @@ Chemenu is dual-licensed, because it is two things in one repository.
| Half | Licence | File |
|------|---------|------|
| The stack — `tools/`, `types/` | GNU AGPL-3.0-or-later | [LICENSE](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/LICENSE) |
| The content — `kb/`, `raw/`, `instructions/`, the `CONTRACT.md` layer, and the prose documents at the root | CC-BY-4.0 | [LICENSE-CONTENT](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/LICENSE-CONTENT) |
| The stack — `tools/`, `types/` | GNU AGPL-3.0-or-later | [LICENSE](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/LICENSE) |
| The content — `kb/`, `raw/`, `instructions/`, the `CONTRACT.md` layer, and the prose documents at the root | CC-BY-4.0 | [LICENSE-CONTENT](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/LICENSE-CONTENT) |
The boundary between the two is not a list someone maintains by hand: it is the
file plan `tools/wikitool dist export` already computes, so it cannot drift out
of agreement with what actually ships. [NOTICE](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/NOTICE) states that, and carries
of agreement with what actually ships. [NOTICE](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/NOTICE) states that, and carries
the attribution for the vendored [Commonplace](https://github.com/zby/commonplace)
research base.
How-to-Back-up-to-a-synced-folder.md
+164
@@ -0,0 +1,164 @@
*Generated from [`docs/how-to/back-up-to-a-synced-folder.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/back-up-to-a-synced-folder.md) at v10.0.0 – edit the repository, not this wiki.*
# Back up to a synced folder
Back an instance up into a folder a sync client uploads - OneDrive, or any other client that keeps a
local folder in step with a server - when there is no git server to push to. The instance itself
stays in a folder the sync client does not touch; the synced folder receives a git repository to
push to and a copy of the files on this machine that are not in git. What the settings look like:
[configuration reference](Reference-Configuration#backup-to-a-synced-folder).
<!-- wikitool:toc -->
## Contents
- [Why the instance does not live in the synced folder](#why-the-instance-does-not-live-in-the-synced-folder)
- [Set it up](#set-it-up)
- [What is backed up, and what is not](#what-is-backed-up-and-what-is-not)
- [Restore](#restore)
- [One device writes](#one-device-writes)
<!-- /wikitool:toc -->
## Why the instance does not live in the synced folder
Putting the whole instance - `.git` included - into the synced folder looks like the simplest
backup, and fails in three ways:
- **Locks.** On Windows, OneDrive holds a file it is uploading open for a moment. `git gc` and
`git repack` then fail to delete or replace a file under `.git` ("unlink failed").
- **Placeholders.** With Files On-Demand, the client offloads files it considers unused and leaves a
placeholder behind; git then reads objects that first have to be downloaded, or are not there
when the machine is offline.
- **Conflict copies.** As soon as a second device syncs the same folder, two writers meet inside
`.git`, and the client resolves that by making copies of the files both changed - a repository
git can no longer read.
So the instance lives outside the synced folder, and the synced folder gets a layout git writes to
in a way a sync client handles well:
```
<target>/git/ a bare repository - origin of the instance; publish pushes to it
<target>/<machine>/ a copy of this machine's files that are not in git, plus .chemenu-backup.json
```
Git writes into the bare repository only on a push, and then mostly adds new files that never
change again.
`tools/wikitool doctor` reports `sync-folder` as a `WARN` when the instance's git directory lies in
a synced folder. On Windows, check that line even if you never chose such a folder: when Documents
or Desktop is redirected into OneDrive (Known Folder Move), an instance under Documents lies in
OneDrive without looking like it. Move the instance to a folder outside it - `C:\Wiki\<name>`, say -
before you go on.
## Set it up
1. **Pick the target**: a folder inside the synced folder, for example
`C:\Users\<you>\OneDrive - <company>\Chemenu-Backup`. It must not lie inside the instance.
2. **Write the `backup` section** into `.wikitool-local.json` at the root of the instance - by hand,
since its fields decide what leaves this machine and `config set` refuses them:
```json
{
"backup": {
"target": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Backup",
"machine_data": true,
"machine_name": "WORK-LAPTOP",
"extra": [".claude/settings.local.json"]
}
}
```
Backslashes in a JSON string are doubled. `machine_name` defaults to the computer's host name;
`extra` lists further paths inside the instance to copy, and can be left out.
3. **Tell the agent:**
> Set up the backup target from the `backup` section.
The agent runs `tools/wikitool backup init`. It creates `<target>/git` as a bare repository and
makes it `origin` - unless the instance already has an `origin` that points elsewhere: then it
changes nothing and shows you both URLs, and whether `origin` moves is yours to decide
(`git remote set-url origin "<target>\git"`).
4. **Allow the push target**, if your `.wikitool-local.json` has a `remotes` section: `backup init`
then prints that section with `<target>\git` added. Copy it into the file yourself - the list is
the Publish-Remote Gate, and no agent edits it. Without a `remotes` section there is nothing to do.
5. **Publish.** The next `tools/wikitool publish` pushes the whole history to `<target>/git` and,
with `machine_data` on, copies the machine data into `<target>/<machine>/`. From then on every
publish that pushes does both. `tools/wikitool backup run` copies the machine data on its own,
whenever you want it current without publishing - after dropping a file into `incoming/`, for
instance, which is otherwise only copied by the next publish.
6. **Check it:** `tools/wikitool doctor`. The `backup` line names the target and how old the last
copy of the machine data is; a `WARN` there names what is missing.
If a publish reports a `WARN` that the machine-data backup failed, the commit and the push have
still happened. Fix what the line names - a synced folder that is not there, a folder that is not
writable - and run `tools/wikitool backup run`. Another publish does not help: it has nothing new to
push.
## What is backed up, and what is not
| Path | Backed up | Why |
|------|-----------|-----|
| Everything committed (`kb/`, `raw/`, `wikitool.json`, ...) | yes, in `<target>/git` | that is what `publish` pushes |
| `.wikitool-local.json` | yes | the settings of this checkout |
| `ENVIRONMENT.md` | yes, if present | written by hand |
| Everything in `incoming/` but `.gitkeep` | yes | sources waiting there are backed up nowhere else |
| The `extra` paths | yes | what you named |
| `quarantine/` | no | material that arrived through an intake channel and that nobody has reviewed yet |
| `reports/` | no | regenerated by `lint`; telemetry holds prompts |
| `.wikitool-tools.json`, `tools/.venv/`, the published skills | no | the preflight and `instructions sync` rebuild them |
| The secrets file | **never** | it lies outside the instance, and a backup reads nothing outside it |
**Private values reach the synced folder.** `.wikitool-local.json` is copied as it is, so a plain
private value in it - an internal host name, an internal URL - ends up in OneDrive. A secret does
not: a secret is never in that file, only a reference to it, and the secrets file it resolves
through (`%APPDATA%\chemenu\<instance_id>\secrets.json`, with the sessions `raw login` stored)
lies outside the instance. An `extra` path that leads outside the instance - through `..` or a
link - is refused, so no setting can copy it either.
In `<target>/<machine>/` a run writes and deletes only what is its own: it removes a file only when
its previous run copied it and the instance no longer has it (a source `raw accept` moved out of
`incoming/`, say), and refuses a folder that is not empty and has no `.chemenu-backup.json`, rather
than copying into somebody else's folder. A link inside a copied directory is skipped and named,
never followed.
## Restore
On the same machine or a new one:
1. **Clone the repository and check it:**
```bash
git clone "<target>/git" "<new folder>"
cd "<new folder>"
git fsck --full
```
The clone is outside the synced folder, like the original; its `origin` is the bare repository
already.
2. **Copy the machine data back**: everything in `<target>/<machine>/` except
`.chemenu-backup.json`, into the new folder, keeping the paths. On Windows:
`robocopy "<target>\<machine>" "<new folder>" /E /XF .chemenu-backup.json`; elsewhere:
`rsync -a --exclude .chemenu-backup.json "<target>/<machine>/" "<new folder>/"`.
3. **Bootstrap the checkout** - tell the agent:
> Bootstrap this checkout.
It follows [instructions/bootstrap.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/bootstrap.md): the preflight, the
published skills, `doctor`.
4. **Provide the secrets again.** They were never in the backup. Put each value back where its
reference reads it - 1Password, the secrets file, or a `WIKITOOL_SECRET_<NAME>` variable
([use-1password-for-settings.md](How-to-Use-1password-for-settings)) - and sign in again with
`raw login` for each host that needs it. `doctor`'s `settings-secrets` line names every reference
that does not resolve yet.
`doctor` must then run through without a `FAIL`.
## One device writes
Only one device publishes to `<target>/git`. Two devices pushing into the same synced folder are two
writers again, only one step removed - and the sync client resolves that with conflict copies, not
with a merge. An instance on a second device gets a target of its own. The sync client is a backup
here, not a way to work on one instance from two devices.
A folder the sync client uploads can also hold a read-only version of selected pages, for readers
outside the repository: [Export pages to a synced folder](How-to-Export-to-a-synced-folder). It is a
separate folder with its own settings, and never the backup target.
How-to-Browse-with-obsidian.md
+3 -3
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/browse-with-obsidian.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/browse-with-obsidian.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/browse-with-obsidian.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/browse-with-obsidian.md) at v10.0.0 – edit the repository, not this wiki.*
# Browse the wiki in Obsidian
@@ -25,7 +25,7 @@ keeps consistent across the whole wiki. A hand edit can quietly break them. Rena
file in Obsidian is the clearest case: a title is the page's only identifier, so it also appears in
every wikilink and citation that points at the page. A rename goes through
`tools/wikitool rename`, which updates every reference. The full procedure is
[instructions/page-lifecycle.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/page-lifecycle.md).
[instructions/page-lifecycle.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/page-lifecycle.md).
Generated files - `kb/index.md`, `kb/log.md`, `kb/provenance.md` and every `INDEX.md` - are
rebuilt by the tool and never edited by hand.
@@ -45,7 +45,7 @@ Knowing the naming rules makes the vault easier to find your way in:
page's title by case alone. A file's whole path below the instance root stays at 160 characters
or fewer, so a Windows checkout works without long paths. `wikitool new` and `wikitool rename`
refuse titles that break these rules, and `wikitool lint` reports existing ones.
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/kb/CONTRACT.md) has the full rule.
[kb/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/kb/CONTRACT.md) has the full rule.
## Letting the structure grow
How-to-Capture-repository-docs.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/capture-repository-docs.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/capture-repository-docs.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/capture-repository-docs.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/capture-repository-docs.md) at v10.0.0 – edit the repository, not this wiki.*
# Capture documentation from a git repository
How-to-Collect-from-a-synced-folder.md
+139
@@ -0,0 +1,139 @@
*Generated from [`docs/how-to/collect-from-a-synced-folder.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/collect-from-a-synced-folder.md) at v10.0.0 – edit the repository, not this wiki.*
# Collect documents from a synced folder
Documents that are not typed or fetched at this terminal can reach the wiki through an intake
channel. Besides the MCP `submit` tool there are two channels that this checkout *pulls* from: a
folder that a sync client such as OneDrive fills (handler `folder`), and a program of your own
(handler `command`). Both end in the quarantine `quarantine/`, exactly like a submission through the
server, and both wait there for a human to run `wikitool upload accept` - the Upload Review Gate
stands between every channel and `incoming/`.
This guide creates a `folder` channel, explains what `upload collect` leaves alone and why, and shows
a `command` channel with an example script. The field reference is in the
[configuration reference](Reference-Configuration#intake-channels); the review is in
[ingesting a source](How-to-Ingest-a-source#a-document-submitted-from-outside).
<!-- wikitool:toc -->
## Contents
- [Create the folder channel](#create-the-folder-channel)
- [What stays in the folder, and why](#what-stays-in-the-folder-and-why)
- [`settle_seconds` and what it cannot promise](#settle_seconds-and-what-it-cannot-promise)
- [Deleting in a synced folder deletes everywhere](#deleting-in-a-synced-folder-deletes-everywhere)
- [A command channel](#a-command-channel)
- [Afterwards](#afterwards)
<!-- /wikitool:toc -->
## Create the folder channel
Channels are not written by `config set`: every field of the `intake` section opens a path into the
checkout, so it is yours to edit by hand in `.wikitool-local.json`.
```json
{
"intake": {
"channels": {
"onedrive": {
"handler": "folder",
"path": "/home/you/OneDrive/wiki-inbox",
"settle_seconds": 60,
"max_bytes": 10485760,
"allowed_extensions": [".pdf", ".docx", ".md", ".txt"]
}
}
}
}
```
`path` has to be absolute and lie outside the instance. Check it with `tools/wikitool doctor`, then
pull:
```bash
tools/wikitool upload collect # every pull channel
tools/wikitool upload collect onedrive # only this one
tools/wikitool upload list
```
A collected file is stored unchanged in `quarantine/<id>/` with a manifest that names the channel,
the handler and the original path. A folder knows nobody who dropped a file, so the submitter is
recorded as `unknown`, with the handler as its source: do not trust the name or content of such a
file any more than that of a stranger's.
## What stays in the folder, and why
`upload collect` reads the top level of the folder only, and it removes one kind of thing: a file it
has just stored. Everything else stays where it is and is reported with its reason:
| Entry | Why it stays |
|---|---|
| A subdirectory | Nothing descends into it, and nothing deletes a directory. |
| A name starting with `.` or `~$`, a hidden or system file | Editors and sync clients keep temporary files like these next to the real one. |
| A symbolic link | Following it could leave the folder. |
| An extension outside `allowed_extensions`, or a file above `max_bytes` | The same limits as for `submit`. |
| A file younger than `settle_seconds` | It may still be arriving. |
| A file that changed while it was being read or copied | The source is removed only if it still has the size and sha256 that were stored. |
A file whose sha256 already waits in the quarantine is reported as a duplicate, with the id of the
existing submission, and is removed from the folder.
## `settle_seconds` and what it cannot promise
A sync client writes a file in pieces, so a file that is half there looks like a complete, smaller
one. `settle_seconds` (default 60) skips files whose modification or creation time is younger than
that, and the check that the source did not change during the copy catches the rest. Neither can see
a client that pauses for longer than the interval in the middle of a file and then resumes: such a
file is stored incomplete, and the review is where you notice it. Raise the value if your client
works over a slow link.
## Deleting in a synced folder deletes everywhere
A file removed from a folder that OneDrive syncs is removed on every device that syncs it. That is
what a collected file should do - it is now in the quarantine - but it also means a collect on one
machine empties the folder for all of them. The client's recycle bin keeps the file for a while; the
quarantine copy, which is never synced, is the one to rely on. Collect from one machine only.
## A command channel
A `command` channel runs a program of your own, which fetches documents from somewhere that has no
synced folder - a mailbox, a share, a scanner. The program is named in `commands` and referenced from
the channel; the section stays valid only if every reference resolves.
```json
{
"intake": {
"commands": {
"scan-mail": ["/home/you/bin/fetch-attachments.sh"]
},
"channels": {
"mail": { "handler": "command", "command": "scan-mail" }
}
}
}
```
The program is started without a shell and with stdin closed, with an empty staging directory under
`quarantine/.staging/` appended as its last argument. It has 300 seconds. It leaves the files it
wants to hand over directly in that directory:
```bash
#!/bin/sh
# fetch-attachments.sh <staging-dir>
set -eu
out="$1"
rclone copy remote:wiki-inbox "$out" --max-age 1d
```
A program that exits non-zero, does not start or times out fails its channel: nothing is taken, the
staging directory is removed, and the last 2000 characters of its stderr are shown. Other channels
still run, and `upload collect` exits 1 if any channel failed. With no pull channel configured it
exits 0 with "nothing to collect".
## Afterwards
The agent does the same at the start of an ingest when you name no file: `upload collect`, then
`upload list`, then a report of what waits. It never runs `upload accept` for you.
The other direction - a read-only version of selected pages written out to a synced folder - is
[Export pages to a synced folder](How-to-Export-to-a-synced-folder). An export is never a source: the
export marker keeps it out of `raw/`.
How-to-Export-guidelines.md
+4 -2
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/export-guidelines.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/export-guidelines.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/export-guidelines.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/export-guidelines.md) at v10.0.0 – edit the repository, not this wiki.*
# Export guidelines to captured repositories
@@ -52,5 +52,7 @@ did not see.
## Related
- [Guideline export commands](Reference-Commands-Guideline-export)
- [Export commands](Reference-Commands-Exports)
- [Export pages to a synced folder](How-to-Export-to-a-synced-folder) - the other export, a read-only
version of `kb/` pages
- [Why the gates are code](Explanation-Why-gates-are-code)
How-to-Export-to-a-synced-folder.md
+157
@@ -0,0 +1,157 @@
*Generated from [`docs/how-to/export-to-a-synced-folder.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/export-to-a-synced-folder.md) at v10.0.0 – edit the repository, not this wiki.*
# Export to a synced folder
Give a reader outside the repository a read-only version of part of `kb/`: the selected pages,
written as files into a folder - typically one a sync client such as OneDrive uploads, so that a
Microsoft 365 tenant can index them. The export is derived. It is regenerated from the committed
pages on every run, nobody edits it, and nothing in it flows back into the wiki. What the settings
look like: [configuration reference](Reference-Configuration#export-to-a-folder).
<!-- wikitool:toc -->
## Contents
- [Pick the pages and the format](#pick-the-pages-and-the-format)
- [Set it up](#set-it-up)
- [What a run does to the folder](#what-a-run-does-to-the-folder)
- [Export after every publish](#export-after-every-publish)
- [Convert the pages with a program of your own](#convert-the-pages-with-a-program-of-your-own)
- [What does not come back in](#what-does-not-come-back-in)
- [Related](#related)
<!-- /wikitool:toc -->
## Pick the pages and the format
- **Which pages.** Only what a target's `include` selects is written; there is no "everything". A
selection is a list of `search` predicates that must all hold - `tags=shared`, `kind=system`,
`!draft` - and a page that fits any selection of the target is exported. Try a selection with
`tools/wikitool search --field tags=shared` first.
- **Which format.** Two ways, equally valid:
- `markdown` writes the pages as `.md` files. It needs nothing but the folder.
- `command` hands those files to a converter of your own and exports what it produces - Word
documents with `pandoc`, say.
Whether a Microsoft 365 tenant indexes `.md` files for Copilot is not settled for every tenant.
Try a small `markdown` target first; if the tenant does not pick the files up, the same pages go
through a `command` target as a format it does.
## Set it up
1. **Pick the folder**: a folder inside the synced folder, for example
`C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
and it must lie outside the instance, outside every other export target, outside every `intake`
folder and outside `backup.target`.
2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
since its fields decide what leaves this machine and which program runs, and `config set`
refuses them:
```json
{
"export": {
"targets": {
"shared": {
"handler": "markdown",
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Lesefassung",
"include": [["tags=shared"]],
"frontmatter": "summary",
"links": "relative",
"after_publish": false
}
}
}
}
```
`frontmatter` decides what follows the export marker in each file: `minimal` nothing, `summary`
(the default) the title, type, summary and modified date, `full` the page's whole frontmatter.
`links: relative` (the default) turns a link to an exported page into a relative Markdown link;
`links: text` leaves the page's name. A link to a page that is not exported is always plain text.
3. **Tell the agent:**
> Export the pages.
The agent runs `tools/wikitool export run`. It reports per target which pages are new, which
changed and which were removed, and how many stayed as they were. Name a target to run only that
one: `tools/wikitool export run shared`.
4. **Check it:** `tools/wikitool doctor`. The `export` line lists the targets; a `FAIL` there names
what is wrong with the section.
The export reads the committed pages. A `kb/` or `types/` with uncommitted changes is refused, so
every file names the commit its pages come from. Publish first.
## What a run does to the folder
The folder holds the pages and one file of the stack's own, `.wikitool-export.json`, which lists
what the last run wrote.
- **Only its own files.** A run removes a file only when it is listed and no longer exported, and a
folder only when the run created it. Anything else in the folder stays as it is.
- **Nobody edits an export.** A listed file that someone changed stops the target before anything is
written, and the message names the file. Move or delete it and run again. A listed file that is
missing is written again.
- **A folder that is not an export is refused.** A folder that holds files and no manifest is
somebody's folder; the run writes nothing into it.
- **An empty selection is an error.** A run that finds no page does not delete the earlier export.
- **An unchanged export stays untouched.** A run that has nothing to change writes nothing, not even
a modification time - the sync client has nothing to upload.
- **A growing selection needs no confirmation.** New pages are reported as new; read the list
before the sync client uploads them.
Targets are independent: one that fails is reported with its reason and the others still run; the
command then exits 1.
## Export after every publish
With `"after_publish": true` on a target, every `tools/wikitool publish` that committed refreshes
it afterwards - with `--no-push` too, since the export reads the commit, not the remote. If that
export fails, `publish` prints a `WARN` with the reason and still exits 0: the commit and the push
stand. Fix what the line names and run `tools/wikitool export run`; another `publish` does not
repeat it. A malformed `export` section stops `publish` before it commits, but only when some target
sets `after_publish`.
## Convert the pages with a program of your own
A target with `"handler": "command"` names an entry of `commands`:
```json
{
"export": {
"targets": {
"as-word": {
"handler": "command",
"command": "pandoc-docx",
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Word",
"include": [["tags=shared"]]
}
},
"commands": {
"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]
}
}
}
```
The pages are rendered into a staging directory first. The program runs without a shell, with stdin
closed and a limit of 300 seconds, and gets two more arguments: the directory with the Markdown
files and an empty directory for its output. On exit 0 the regular files it left there (below any
subdirectory, symlinks left out and reported) become the content of the target; the manifest and
all the rules above apply to them as to Markdown. Any other end - a non-zero exit, a timeout, a
program that does not start - leaves the target exactly as it was and prints the end of the
program's stderr. The staging directory is always removed.
## What does not come back in
An export is never a source. Every Markdown file starts with a frontmatter block whose first key is
`wikitool_export`, and `tools/wikitool raw accept` and `raw capture` refuse a file that carries it.
A converter may lose that key, so `raw accept` also refuses any file in `incoming/` that is byte
for byte one of the files in the manifest of a configured target - and warns, without refusing,
when a target has no manifest yet or the `export` section is broken.
## Related
- [Configuration reference](Reference-Configuration#export-to-a-folder)
- [Export commands](Reference-Commands-Exports)
- [Export guidelines to captured repositories](How-to-Export-guidelines) - the other export, one file
pushed into a repository
- [Back up to a synced folder](How-to-Back-up-to-a-synced-folder) - the same kind of folder, for the
backup instead
How-to-Fetch-pages-behind-a-login.md
+130
@@ -0,0 +1,130 @@
*Generated from [`docs/how-to/fetch-pages-behind-a-login.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/fetch-pages-behind-a-login.md) at v10.0.0 – edit the repository, not this wiki.*
# Fetch pages behind a login
Let `tools/wikitool raw fetch` fetch the articles of a website you are subscribed to, with your own
login: you sign in once in a browser window, and the wiki fetches that website's pages with the
session your sign-in left until it expires. No password is stored, and the agent never sees one.
The same page covers websites whose pages only appear in a browser, and handing a website to a
fetch program of your own.
Whether a website's terms allow you to fetch your subscription this way is yours to check; the
stack does not check it and offers no advice on it.
Without any of this, a page behind a login can still be saved from your browser and handed to
`raw fetch --html` ([Ingest a source](How-to-Ingest-a-source)) - one page at a time, by hand.
<!-- wikitool:toc -->
## Contents
- [1. Install the browser component](#1-install-the-browser-component)
- [2. Describe the website's login](#2-describe-the-websites-login)
- [3. Route the website through your session](#3-route-the-website-through-your-session)
- [4. Sign in](#4-sign-in)
- [5. Ingest as usual](#5-ingest-as-usual)
- [Pages that only appear in a browser](#pages-that-only-appear-in-a-browser)
- [A fetch program of your own](#a-fetch-program-of-your-own)
- [Where the session is kept](#where-the-session-is-kept)
<!-- /wikitool:toc -->
## 1. Install the browser component
Signing in needs a real browser window, so it needs the optional browser component: a Python
package and a Chromium of its own, installed into the instance's `tools/.venv`. You install it -
the agent never does:
```bash
tools/preflight.sh --with browser
```
In PowerShell 7: `pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`.
Later preflight runs keep it up to date. `tools/wikitool doctor` reports it on its `browser` line.
## 2. Describe the website's login
The website needs a site handler that says where you sign in and how a page shows that it arrived
signed in. Add a `session` block to the website's handler in `sites/`, or write a handler just for
it ([Write a site handler](How-to-Write-a-site-handler)):
```json
{
"host": "news.example.com",
"summary": "A subscription news site: articles need the subscriber's session.",
"session": {
"login": "https://news.example.com/login",
"signed_in": {"tag": "a", "class": "logout"},
"signed_out": {"class": "paywall-teaser"}
}
}
```
`signed_in` is something only a signed-in page carries - a logout link, an account menu.
`signed_out` is something only a page that was not carries - the teaser box that asks you to
subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element
that tells them apart. [sites/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/sites/CONTRACT.md) has the exact format.
## 3. Route the website through your session
The handler alone changes nothing. Tell the agent:
> Route news.example.com through my session.
It runs `tools/wikitool config set fetch.routes '{"news.example.com": "session"}'`, keeping any
host already listed. The setting is committed with the instance, so every clone fetches the
website the same way - each with its own user's session.
## 4. Sign in
Run this yourself - it opens a browser window and waits for you:
```bash
tools/wikitool raw login news.example.com
```
Sign in as you always do, captcha and second factor included, then close the window. The session
is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.
## 5. Ingest as usual
Give the agent the article's URL as for any web page ([Ingest a source](How-to-Ingest-a-source)).
The capture's header names the website whose session was used (`session: news.example.com`),
never the session itself.
When your session has expired, or the page came back as a teaser, the agent stops: `raw fetch`
writes nothing and asks you to run `raw login` again. Nothing is kept from a teaser in silence.
## Pages that only appear in a browser
Some websites send an empty frame and draw the text with JavaScript; `raw fetch` then captures
almost nothing. Add `"render": true` to the website's handler: its pages are then fetched through
the browser component, with your session where the website has one, and the header says
`rendered: true`. The `.html` is then the page as the browser drew it, not what the server sent -
which is why it happens only where a handler says so.
## A fetch program of your own
A website can instead go to a program you run - a fetch service on another machine, say. Add it to
`.wikitool-local.json` by hand, since it names a program to run and `config set` does not write
such a field:
```json
{
"schema": 1,
"fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
}
```
and route the host to it: `tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'`.
The program gets the URL as its last argument and prints the page's HTML; anything else - an exit
other than 0, no output - stops `raw fetch` with the program's error output, and the built-in
fetch is not tried instead. The `fetch` section is in
[docs/reference/configuration.md](Reference-Configuration#fetch-routes).
## Where the session is kept
In the secrets file in your own configuration directory - `~/.config/chemenu/<instance_id>/secrets.json`,
`%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows - under `session_<host>`, readable by
you only, outside every working tree. It never reaches a commit or a `wikitool` output. It does
not keep an agent with a shell in your account from reading it, any more than any other secret
there: [configuration-tiers.md](Explanation-Configuration-tiers) says what keeping secrets
out of the tree protects against and what it does not.
How-to-Ingest-a-source.md
+34 -6
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/ingest-a-source.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/ingest-a-source.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/ingest-a-source.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/ingest-a-source.md) at v10.0.0 – edit the repository, not this wiki.*
# Ingest a source
@@ -7,6 +7,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
the `wiki-ingest` skill ([skill catalogue](Reference-Skills)). This page covers what you hand
it and where it waits for you.
<!-- wikitool:toc -->
## Contents
- [What you prepare](#what-you-prepare)
- [What you say](#what-you-say)
- [Where the agent asks you](#where-the-agent-asks-you)
- [What you get back](#what-you-get-back)
- [A document submitted from outside](#a-document-submitted-from-outside)
- [Related](#related)
<!-- /wikitool:toc -->
## What you prepare
Put the material into `incoming/` at the root of the instance. You don't have to classify it or
@@ -18,6 +29,17 @@ pick a directory:
- **A web page** needs no download of your own. Name the URL to the agent, and
`tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
text derived from it with a header recording where and when it was fetched.
For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the
same page in the website's standard form (the English one, whatever language your link was in),
leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints
`CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
capture. Your instance can switch such a handler off, choose another language and add hint lines
of its own per website; see [the configuration reference](Reference-Configuration).
Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
page that offers more than the instance's `max_pages`.
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
Outlook `.msg`) or a ZIP archive - is
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
@@ -27,7 +49,10 @@ pick a directory:
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
text to derive - the tool warns, and there is no OCR.
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
agent derives the same text from that file with `raw fetch --html`.
agent derives the same text from that file with `raw fetch --html`. For a website you read often,
sign in once instead and let the wiki fetch it with your session:
[Fetch pages behind a login](How-to-Fetch-pages-behind-a-login). When that session has expired the
agent stops and asks you to run `tools/wikitool raw login <host>` - it never signs in itself.
Where a file ends up in `raw/` is computed by `tools/wikitool raw accept`: a `YYYY/MM` shard for the
day it was accepted. So is whether several files of one source get bundled. You never choose either.
@@ -49,7 +74,7 @@ what is waiting and names the entry the next bare `Ingest` will take.
- **Before promoting the file into `raw/`**, the agent asks how faithful the capture is
(`fidelity`) and what the material may claim about its subject (`authority`). Both are recorded
once and never guessed: you know the answers now, and nobody can recover them later.
[raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/raw/CONTRACT.md) defines the values.
[raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/raw/CONTRACT.md) defines the values.
- **Before writing pages**, the agent presents the key takeaways and asks which points matter most
and what to emphasise. The pages follow your answer.
- **When it publishes**, a source that touches ten or more files stops at the Mass-Update Gate
@@ -70,17 +95,20 @@ Every ten sources, the agent suggests a lint run - see [Maintain the wiki](How-t
## A document submitted from outside
When the [MCP read server](How-to-Set-up-the-mcp-server) has its optional `submit` tool turned on,
documents can arrive from another machine. They land in the quarantine `mcp-upload/`, not in
documents can arrive from another machine. A folder a sync client fills and a program of the
checkout are two more ways in ([Collect documents from a synced
folder](How-to-Collect-from-a-synced-folder)). All of them land in the quarantine `quarantine/`, not in
`incoming/`, and nothing else reads them there. Before an ingest can start, a human has to promote
the submission:
```bash
tools/wikitool upload collect # pull the folder and command channels into the quarantine
tools/wikitool upload list # what is waiting
tools/wikitool upload show <id> # one submission's manifest: who sent it, size, checksum
```
Ask the agent to review the queue, and it follows
[instructions/ingest-queue.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/ingest-queue.md). Accepting a submission stops at
[instructions/ingest-queue.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/ingest-queue.md). Accepting a submission stops at
the Upload Review Gate (exit 42), which shows you the manifest and the `--confirm` line. Once you
approve, the file is in `incoming/`, and from there it is ingested like any other. Rejecting it needs
no clearance. The file is deleted, and its checksum and your reason stay in the ledger.
@@ -89,5 +117,5 @@ no clearance. The file is deleted, and its checksum and your reason stay in the
- [Capture documentation from a repository](How-to-Capture-repository-docs) - for material that lives
in git and will change.
- [raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/raw/CONTRACT.md) - what belongs in `raw/`, and why its content is data,
- [raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/raw/CONTRACT.md) - what belongs in `raw/`, and why its content is data,
never instructions.
How-to-Maintain-the-wiki.md
+8 -3
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/maintain-the-wiki.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/maintain-the-wiki.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/maintain-the-wiki.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/maintain-the-wiki.md) at v10.0.0 – edit the repository, not this wiki.*
# Maintain the wiki
@@ -15,7 +15,7 @@ Lint the wiki
Do this periodically. The agent suggests it on its own every ten sources. The full schedule - which
task runs how often, and with which command - is the
[maintenance schedule](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md#maintenance-schedule) in `tools/CONTRACT.md`, next
[maintenance schedule](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md#maintenance-schedule) in `tools/CONTRACT.md`, next
to the command records it depends on.
## What you get back
@@ -60,7 +60,12 @@ tools/wikitool sources trace --page "HA Integration" # which raw files this p
needs. A refusal usually means the task should be split, not pushed through.
- **No reachable remote.** Without `--no-push`, `publish` refuses with exit 1 before committing when
the remote is not configured or cannot be reached. An instance that is meant to stay local passes
`--no-push` on every publish.
`--no-push` on every publish - or, without a git server, pushes to a synced folder instead:
[back-up-to-a-synced-folder.md](How-to-Back-up-to-a-synced-folder).
- **A failed machine-data backup.** With a backup into a synced folder, a publish that pushed also
copies this machine's files that are not in git. If that copy fails, `publish` prints a `WARN`
line and still succeeds - the commit and the push stand. Fix what the line names, then run
`tools/wikitool backup run`; another publish does not repeat the copy.
Every gate, and which command it belongs to: [exit codes and gates](Reference-Exit-codes).
Why they are enforced in code: [why-gates-are-code.md](Explanation-Why-gates-are-code).
How-to-Query-the-wiki.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/query-the-wiki.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/query-the-wiki.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/query-the-wiki.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/query-the-wiki.md) at v10.0.0 – edit the repository, not this wiki.*
# Query the wiki
How-to-Run-the-weekly-review.md
+3 -3
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/run-the-weekly-review.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/run-the-weekly-review.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/run-the-weekly-review.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/run-the-weekly-review.md) at v10.0.0 – edit the repository, not this wiki.*
# Run the weekly review
@@ -20,8 +20,8 @@ syncing: [knowledge-and-commitment.md](Explanation-Knowledge-and-commitment).
## What you prepare
A task tracker, configured for this checkout in `.wikitool-tasks.json`. Super Productivity and
CalDAV (for example Nextcloud Tasks) are supported. The file's shape, and the difference between a
A task tracker, configured in the settings (the `tasks` section and the section of its provider). Super Productivity and
CalDAV (for example Nextcloud Tasks) are supported. Their shape, and the difference between a
tracker the wiki can only read and one it can write to, is in the
[configuration reference](Reference-Configuration#task-tracker). Without a tracker, the
review has nothing to join against. That is a valid state, though, and the pages work without one.
How-to-Set-up-the-mcp-server.md
+11 -10
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/set-up-the-mcp-server.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/set-up-the-mcp-server.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/set-up-the-mcp-server.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/set-up-the-mcp-server.md) at v10.0.0 – edit the repository, not this wiki.*
# Set up the MCP read server
@@ -7,7 +7,7 @@ machine - another agent, a chat client, a service - can ask it the same question
answers. This page covers what you set up around the server: the client, the authentication in
front of it, the optional `submit` path, and how to check that it works. Installing, starting and
syncing the server itself is
[instructions/mcp-read-server.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/mcp-read-server.md), which an agent follows
[instructions/mcp-read-server.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/mcp-read-server.md), which an agent follows
and which is readable for you as well.
<!-- wikitool:toc -->
@@ -162,18 +162,19 @@ reviewed yet.
## Optional: the submit path
Without this, `submit` does not exist as a tool - not unused, but not registered. A file
`.wikitool-upload.json` in the served corpus turns it on. Its fields, defaults and what a malformed
file does are in the [configuration reference](Reference-Configuration#mcp-submissions); a
broken file never means "no limits", it stops the server from starting.
Without this, `submit` does not exist as a tool - not unused, but not registered. A
channel with the handler `mcp` in the `intake` section of the served checkout's
`.wikitool-local.json` turns it on. Its fields, defaults and what a malformed section does are in
the [configuration reference](Reference-Configuration#intake-channels); a broken section never
means "no limits", it stops the server from starting.
**With `submit` on, the middleware has a second duty.** It has to set the identity header itself, on
every authenticated request, and discard any copy the client sent. The server trusts that header as
the value for `submitter`. A header the client could set itself would be a claim, not an identity.
If `identity_header` names a header the middleware does not set, every submission is refused for
If the channel's `identity_header` names a header the middleware does not set, every submission is refused for
lack of an identity.
Submitted files land in `mcp-upload/<id>/`, which is gitignored and read by no other command. From
Submitted files land in `quarantine/<id>/`, which is gitignored and read by no other command. From
there a human promotes or rejects them, in the same checkout as the server rather than in its
process. That review is part of
[ingesting a source](How-to-Ingest-a-source#a-document-submitted-from-outside).
@@ -230,7 +231,7 @@ the client starts a different Python than the instance's. Give the client the ab
**`"commit": null` in every answer**: the served tree has uncommitted changes. Either the sync is not
running, or something writes into the corpus that has no business there. The five read tools never
write. `submit`, when it is on, writes only to `mcp-upload/`, which is gitignored and so is never a
write. `submit`, when it is on, writes only to `quarantine/`, which is gitignored and so is never a
cause of `"commit": null` either.
**`commit` names an old revision**: the sync timer is not running.
@@ -248,6 +249,6 @@ Deployment specifics - the cluster, ingress hosts, the secret store, the GitOps
private infrastructure, and this repository is public.
There is no container image for the server yet. Until there is, the path above - the venv,
`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `mcp-upload/`
`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `quarantine/`
has to survive restarts the same way the rest of the checkout does. Otherwise a submitted file that
nobody has reviewed yet loses its quarantine.
How-to-Trace-and-score-sessions.md
+10 -10
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/trace-and-score-sessions.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/trace-and-score-sessions.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/trace-and-score-sessions.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/trace-and-score-sessions.md) at v10.0.0 – edit the repository, not this wiki.*
# Trace and score agent sessions
@@ -23,13 +23,13 @@ score means are covered in [telemetry-and-evaluation.md](Explanation-Telemetry-a
**Turn tracing on, if it is off.** It is on in a clone of the origin repository and off in an
instance installed from a release. `tools/wikitool doctor` reports which state this checkout is in,
and why. To turn it on for one checkout, create `.wikitool-telemetry.json` at the root:
and why. To turn it on for one checkout, set it in the per-checkout settings:
```json
{ "enabled": true }
```bash
tools/wikitool config set telemetry.enabled true
```
The file and its other keys are in the [configuration reference](Reference-Configuration#telemetry).
This writes the `telemetry` section of `.wikitool-local.json`. The section and its other keys are in the [configuration reference](Reference-Configuration#telemetry).
Everything stays on this machine: traces land under `reports/telemetry/`, which is gitignored, and
no exporter is configured.
@@ -40,7 +40,7 @@ yourself keeps one task in one trace:
export WIKITOOL_SESSION_ID="my-task"
```
[instructions/session-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/session-setup.md) has the line for each shell,
[instructions/session-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/session-setup.md) has the line for each shell,
and what to do on a harness that starts a fresh shell for every command.
## Reading a session back
@@ -110,7 +110,7 @@ guarantee: an agent that reorders the flags gets around it.
### Copilot CLI
[.github/hooks/wiki-trace.json](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/.github/hooks/wiki-trace.json) is committed, so a clone
[.github/hooks/wiki-trace.json](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/.github/hooks/wiki-trace.json) is committed, so a clone
brings its own tracing. Eleven events go to `tools/trace_ingest.py`. `disableAllHooks` in
`.github/copilot/settings.json` turns them off without deleting anything. `userPromptSubmitted` is
among them, so `clearance-ended-the-turn` can be scored on Copilot CLI with no change.
@@ -146,7 +146,7 @@ hooks:
VS Code Chat has no hooks, so nothing observes a session while it runs. Its chronicle store
(`session-store.db`, the same schema as Copilot CLI's) keeps sessions, turns and touched files, and
[tools/import_chronicle.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/import_chronicle.py) rebuilds a trace from it afterwards:
[tools/import_chronicle.py](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/import_chronicle.py) rebuilds a trace from it afterwards:
```bash
tools/import_chronicle.py --dry-run # this repository's sessions
@@ -160,8 +160,8 @@ that a file was touched, not that a tool was about to run.
### Mistral Vibe
[.vibe/hooks.toml](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/.vibe/hooks.toml) declares the three hooks Vibe has, and
[.vibe/config.toml](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/.vibe/config.toml) keeps the telemetry settings in the repository rather
[.vibe/hooks.toml](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/.vibe/hooks.toml) declares the three hooks Vibe has, and
[.vibe/config.toml](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/.vibe/config.toml) keeps the telemetry settings in the repository rather
than in someone's shell profile. Both were checked against the installed CLI's own loader
(`mistral-vibe 2.24.2`) rather than against its documentation. What that turned up:
How-to-Troubleshoot.md
+11 -11
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/troubleshoot.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/troubleshoot.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/troubleshoot.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/troubleshoot.md) at v10.0.0 – edit the repository, not this wiki.*
# Troubleshoot an instance
@@ -20,7 +20,7 @@ tools/preflight.sh
In PowerShell 7 on Windows, the call is
`pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1`. The `-ExecutionPolicy Bypass`
applies to that one process only and changes no setting.
[instructions/preflight.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/preflight.md) is the procedure, and its output says
[instructions/preflight.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/preflight.md) is the procedure, and its output says
what to install if anything is missing.
## The agent offers no skills (`wiki-ingest`, `wiki-query`, ...)
@@ -37,7 +37,7 @@ Then restart the agent session. A harness reads its skills only when the session
`USER.md` or `SOUL.md` is missing, or still carries the sentinel line from its template. A renamed
template does not count as a filled one. Ask the agent to run the personalization step of
[instructions/setup-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/setup-instance.md). For a further clone of an
[instructions/setup-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/setup-instance.md). For a further clone of an
older instance, that is the only step to catch up on.
## `doctor` reports `environment: WARN`
@@ -48,11 +48,11 @@ a valid state.
## `doctor` reports `telemetry: WARN`
`.wikitool-telemetry.json` holds something telemetry does not read as written: unreadable JSON, a
value that is not an object, an `enabled` that is not `true` or `false`, a cap that is not a
The `telemetry` section of `.wikitool-local.json` holds something telemetry does not read as
written: an unreadable settings file, a value that is not an object, an `enabled` that is not `true` or `false`, a cap that is not a
positive whole number, or a key it does not know. Telemetry itself never stops on this. It ignores
the file or the value, or reads `"no"` as on, and the line names the field. Correct the file to the
shape in the [configuration reference](Reference-Configuration#telemetry), or delete it to
the file or the value, or reads `"no"` as on, and the line names the field. Correct the section to the
shape in the [configuration reference](Reference-Configuration#telemetry), or remove it to
get the default back.
## `new` stops with "No author configured"
@@ -65,19 +65,19 @@ export `WIKI_AUTHOR` ([configuration](Reference-Configuration)).
Expected when ten or more files are counted, for example on the very first commit of a new instance
or after a lint pass. It is not an error. It asks that a human see the output: the complete file
list and the exact `--confirm <token>` line that publishes once you approve.
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/gates.md) has the details.
[instructions/gates.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/gates.md) has the details.
## `publish` ends with exit 42 (Publish-Remote Gate)
This checkout has a `.wikitool-remotes.json`, and the remote being pushed to is not listed in it.
This checkout has a `remotes.allowed_push_urls` list in `.wikitool-local.json`, and the remote being pushed to is not listed in it.
Also not an error: the output names the push URL it would write to and the URLs that are allowed.
Unlike the Mass-Update Gate, there is **no token and no flag** here. If the target really is right,
you add its URL to the file yourself. An agent that edits the file to get past the refusal is
you add its URL to that list yourself. An agent that edits the list to get past the refusal is
opening a gate on its own initiative, which it must never do.
## Setup or an update fails, and you want to report it
Ask the agent to follow [instructions/bug-report.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/bug-report.md), or call
Ask the agent to follow [instructions/bug-report.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/bug-report.md), or call
`tools/bugreport` yourself. It works the same from Bash, Git Bash and PowerShell. It finds a
Python 3.8 or newer by itself, skipping the Store aliases `python3` points at on Windows. It needs
neither the preflight nor a working `wikitool`, and if it finds no Python, it says why.
How-to-Update-an-instance.md
+11 -2
@@ -1,10 +1,10 @@
*Generated from [`docs/how-to/update-an-instance.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/update-an-instance.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/update-an-instance.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/update-an-instance.md) at v10.0.0 – edit the repository, not this wiki.*
# Update an instance
Bring an instance installed from a release up to a newer release of the stack: the tools, types,
instructions and contracts. Your pages stay yours. The agent carries out the upgrade by following
[instructions/upgrade-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/upgrade-instance.md), which sets the order of
[instructions/upgrade-instance.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/upgrade-instance.md), which sets the order of
the steps, what each one decides, and where the session has to restart. This page covers the
decision before it, and what you'll see along the way.
@@ -95,6 +95,15 @@ old form. `doctor` keeps warning until the migration is done, and a later sessio
`tools/wikitool migrate status`. The agent never records a migration it did not carry out, because
the content's version would then describe a shape the pages are not in.
**A migration can also be owed per checkout.** Most migrations rewrite pages: they are done once,
and every other clone of the instance receives the result with its next pull. The 10.0.0 migration
moves settings instead, out of gitignored files that belong to one checkout, so a pull does not
carry it. After another clone has pulled the new release, run `tools/wikitool config migrate`
there too. Until then `doctor` reports each leftover file as a `FAIL`, and a command that needs
one of those settings stops with exit 42 and names that command. `migrate done` is still recorded
only once per instance
([10.0.0-settings-tiers.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/migrations/10.0.0-settings-tiers.md)).
If a step fails for a reason that isn't obvious, the agent stops and offers you a bug report rather
than improvising ([Troubleshoot](How-to-Troubleshoot)).
How-to-Use-1password-for-settings.md
+60
@@ -0,0 +1,60 @@
*Generated from [`docs/how-to/use-1password-for-settings.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/use-1password-for-settings.md) at v10.0.0 – edit the repository, not this wiki.*
# Use 1Password for settings
Keep secrets and private values (host names, internal URLs) out of the working tree by storing
them in 1Password and leaving only an `op://` reference in the settings. The committed
`wikitool.json` then works on every machine signed in to the same 1Password account, and in CI.
What the files and references are: [configuration reference](Reference-Configuration#tiered-settings).
Why, and what this does not protect against: [configuration-tiers.md](Explanation-Configuration-tiers).
## On a desktop
1. **Install the 1Password CLI** (`op`) and turn on its integration with the 1Password desktop
app, so `op` asks the app instead of keeping a session of its own. 1Password's own
documentation describes both steps for each platform.
2. **Run the preflight again** - `tools/preflight.sh`, or in PowerShell 7
`pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1`. It records `op` in
`.wikitool-tools.json` and shows it as `OK`; without `op` the line reads `ABSENT`, which is no
problem until a reference needs it.
3. **Make a vault of its own for this instance**, and put each secret or private value into an
item there. A vault of its own limits what an unlocked `op` can reach.
4. **Copy each value's reference** (`op://<vault>/<item>/<field>`) - the 1Password app offers it
on the field - and tell the agent which setting it belongs to:
> Set `<section>.<field>` to the reference `op://<vault>/<item>/<field>`.
The agent runs `tools/wikitool config set <section>.<field> --ref op://<vault>/<item>/<field>`,
which writes the reference into `wikitool.json`. Never paste the value itself into the chat:
the agent should never see it.
5. **Check it:** `tools/wikitool doctor`. The `settings-secrets` line says how many references
resolve. When the desktop app asks to approve access, that is the prompt doing its job - a
locked app makes the line a `WARN`, and a command that needs the value stops with exit 42 until
you unlock it.
## Without 1Password
A reference by name works the same way without `op`:
- set the reference: `tools/wikitool config set <section>.<field> --ref <name>`;
- put the value into `WIKITOOL_SECRET_<NAME>` (the name upper-cased) in the environment, or into
the secrets file as `{"<name>": "<value>"}`: `~/.config/chemenu/<instance_id>/secrets.json`
(`$XDG_CONFIG_HOME` instead of `~/.config` when it is set), readable by you only
(`chmod 600`) - on Windows `%APPDATA%\chemenu\<instance_id>\secrets.json`. The `instance_id`
is in `wikitool.json`.
## In CI
There are two ways, and which one fits depends on the form of the references:
- **`op` itself, with a service-account token** - for settings that hold `op://` references.
Install `op` in the job, run the preflight so it records it, and set `OP_SERVICE_ACCOUNT_TOKEN`;
the references then resolve exactly as on a desktop.
- **1Password's action resolves the values first** - for settings that hold references by name.
`1password/load-secrets-action` with a service-account token reads an `op://` reference into a
variable before `wikitool` starts; name the variable `WIKITOOL_SECRET_<NAME>` and use
`{"ref": "<name>"}` in the settings. This repository's own CI does this, in
`.gitea/workflows/ci.yml`. An `op://` reference in the settings is not resolved this way: it
still needs `op`.
Give the service account access to the instance's own vault only.
How-to-Write-a-site-handler.md
+87
@@ -0,0 +1,87 @@
*Generated from [`docs/how-to/write-a-site-handler.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/write-a-site-handler.md) at v10.0.0 – edit the repository, not this wiki.*
# Write a site handler
Teach `tools/wikitool raw fetch` how one website serves its pages: the standard form of its URLs,
the page furniture to leave out of the derived text, and a hint or two for whoever reads the
capture. A handler is a JSON file in `sites/` at the root of the instance - data, not a program -
so every clone fetches the website the same way.
The stack already ships handlers, for `learn.microsoft.com` among others. They are the same kind of
file, so the quickest start is to copy one:
```bash
cp tools/chemenu/sites/handlers/learn-microsoft.json sites/my-website.json
```
The file name without `.json` is the handler's name: lower-case letters, digits and hyphens.
## 1. Find out what the website does
Fetch a page of the website and read the derived `.md`:
```bash
tools/wikitool raw fetch <url>
```
Two things are worth a handler:
- **The same page under several URLs** - a language segment, a tracking parameter. A rewrite
turns every spelling into one standard form, so two people ingesting the page get the same source.
- **Page furniture in the derived text** - a sign-in notice, an action bar, a feedback footer. Open
the saved `.html`, find the element that holds it, and note its `id`, `class` or an attribute.
## 2. Write the file
A small handler that drops a cookie banner and a footer on one host:
```json
{
"host": "docs.example.org",
"summary": "Drops the cookie banner and the feedback footer.",
"drop": [
{"id": "cookie-banner"},
{"class": "feedback", "inside": "footer"}
]
}
```
`host` is required and matched exactly, in lower case. `path` narrows the handler to a part of the
website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/sites/CONTRACT.md).
A page that is an index - a module overview, a table of contents - can name its subpages, so that
`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
```json
"subpages": {"links": {"tag": "a", "inside": "td"}}
```
The links are the `<a>` elements inside a table cell; only those on the same host and under the
page's own path count. A page with such a handler prints a `CHECK` line with their number when it
is fetched without `--scope`.
## 3. Try it
Fetch a page again, or derive a saved one without a fetch:
```bash
tools/wikitool raw fetch <url>
tools/wikitool raw fetch --html <saved.html> --url <url>
```
The header of the derived text then says `site: <name> (instance)`, and the furniture is gone. A
check shows up as a `CHECK` line when it applies to the page.
`tools/wikitool doctor` lists the handlers of the instance and fails on a file that cannot be used,
naming the file and the reason. Such a file also stops a `raw fetch` of its own host with exit 1,
before anything is fetched or written, so a handler that did not apply never leaves furniture in a
source unnoticed.
## Replace or extend a stack handler
A handler of the instance for the same host and path as a stack handler **replaces** it. To keep
what the stack knows and add to it, name it as `parent` and give the new file a `path` of its own.
To switch a handler off without deleting it, list its name under `disabled` in the `sites` section
of the settings - see [Configuration](Reference-Configuration#site-handlers).
How-to.md
+12 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/how-to/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/how-to/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/how-to/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/how-to/README.md) at v10.0.0 – edit the repository, not this wiki.*
# How-to guides
@@ -12,8 +12,13 @@ Working with the wiki:
- [Ingest a source](How-to-Ingest-a-source) - a file, a folder, a web page, or a document submitted
through the MCP server
- [Write a site handler](How-to-Write-a-site-handler) - teach `raw fetch` how a website serves its pages
- [Fetch pages behind a login](How-to-Fetch-pages-behind-a-login) - with your own session, through a
browser, or through a fetch program of your own
- [Capture documentation from a git repository](How-to-Capture-repository-docs) - and keep it current
- [Export guidelines to captured repositories](How-to-Export-guidelines)
- [Export to a synced folder](How-to-Export-to-a-synced-folder) - a read-only version of selected pages
for a reader outside the repository, OneDrive and Microsoft 365 included
- [Query the wiki](How-to-Query-the-wiki) - asking, and searching yourself
- [Maintain the wiki](How-to-Maintain-the-wiki) - lint, provenance, and the gates on the way to a publish
- [Run the weekly review](How-to-Run-the-weekly-review) - commitments in a task tracker against the
@@ -27,5 +32,11 @@ Running the instance:
- [Set up the MCP read server](How-to-Set-up-the-mcp-server) - for a consumer that is not a terminal
on this machine
- [Trace and score agent sessions](How-to-Trace-and-score-sessions)
- [Use 1Password for settings](How-to-Use-1password-for-settings) - secrets and private values out
of the tree, on the desktop and in CI
- [Collect documents from a synced folder](How-to-Collect-from-a-synced-folder) - a folder OneDrive and
the like fill, or a program of your own, as an intake channel into the quarantine
- [Back up to a synced folder](How-to-Back-up-to-a-synced-folder) - OneDrive and the like as the
backup of an instance without a git server, and restoring from it
Installing a new instance is [INSTALL.md](Install) at the repository root.
Install.md
+21 -4
@@ -1,4 +1,4 @@
*Generated from [`INSTALL.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/INSTALL.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`INSTALL.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/INSTALL.md) at v10.0.0 – edit the repository, not this wiki.*
# Install a Chemenu instance
@@ -7,7 +7,7 @@ way to get one: an agent installs the **latest release** into an empty folder yo
agent runs the steps, following `instructions/setup-instance.md` from that same release. This
page covers what you prepare beforehand, the sentence you give the agent, what it asks you, and
what to do when it stops. The full command reference is in
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/tools/CONTRACT.md).
[tools/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/tools/CONTRACT.md).
The optional **MCP read server** is set up separately, once the instance runs:
[docs/how-to/set-up-the-mcp-server.md](How-to-Set-up-the-mcp-server). It serves the same
@@ -31,6 +31,21 @@ preflight names the install command for your system. The list is generated from
- **ripgrep (rg)**
<!-- /wikitool:prerequisites -->
Optional: the **1Password CLI (`op`)**, if you want secrets and private values in the settings to
come from 1Password ([use-1password-for-settings.md](How-to-Use-1password-for-settings)).
The preflight records it when it finds it and never asks for it.
Optional: the **browser component**, if you want the wiki to fetch pages of a website you are
subscribed to with your own login, or pages that only appear in a browser
([fetch-pages-behind-a-login.md](How-to-Fetch-pages-behind-a-login)). It is a Python
package plus a Chromium of its own, a few hundred megabytes, installed into the instance's
`tools/.venv` and nowhere else. You install it yourself, once the instance runs:
`tools/preflight.sh --with browser` (PowerShell 7:
`pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`). Every later
preflight run keeps it up to date. On Linux, the browser may need system libraries the preflight
cannot install; Playwright's message then names them, for you or whoever administers the machine.
Native Windows is not yet verified for it.
You also need:
- **An agent harness**: Claude Code, GitHub Copilot (in VS Code or as a CLI), Codex CLI or
@@ -87,7 +102,9 @@ repository:
needed).
- <!-- setup-question: remote --> **Remote** - for an empty clone, only the confirmation that
`origin` is right; otherwise a URL, if you want to push to a server. Without a remote the
instance stays local, and every `publish` runs with `--no-push`.
instance stays local, and every `publish` runs with `--no-push`. Without a git server but with a
synced folder (OneDrive and the like), the agent offers a backup into that folder instead:
[docs/how-to/back-up-to-a-synced-folder.md](How-to-Back-up-to-a-synced-folder).
- <!-- setup-question: kb-language --> **Language and tone of the pages** - they go into
`kb/CONVENTIONS.md`, plus `kb/<name>/COLLECTION.md` per collection. Ready-made profiles,
including a complete German one, are in `instructions/kb-profiles.md`. Decide this **before the
@@ -177,7 +194,7 @@ tools/wikitool doctor
This checks in one call: dependencies, author resolution, git identity/branch/remote, published
skills, structure (collection contracts, generated files), personalization (`USER.md`/`SOUL.md`
present **and** filled in), the optional environment note (`ENVIRONMENT.md`), the task tracker
(`.wikitool-tasks.json` - if it is missing, that is `OK`; if a provider is configured, also
(the `tasks` section of the settings - if it is missing, that is `OK`; if a provider is configured, also
whether its read path is ready and its API is answering right now, neither ever a `FAIL`) and the
session id. `OK`/`WARN` are harmless (a missing remote, for example, is a valid end state); only a
`FAIL` ends with exit 1, and every line names its own fix command.
Reference-Commands-Catalog-and-log.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/catalog-and-log.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/catalog-and-log.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/catalog-and-log.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/catalog-and-log.md) at v10.0.0 – edit the repository, not this wiki.*
# Catalog and log
Reference-Commands-Configuration.md
+124
@@ -0,0 +1,124 @@
*Generated from [`docs/reference/commands/configuration.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/configuration.md) at v10.0.0 – edit the repository, not this wiki.*
# Configuration
Writing the tiered settings in `wikitool.json` and `.wikitool-local.json`.
<!-- wikitool:toc -->
## Contents
- [`config set`](#config-set)
- [`config migrate`](#config-migrate)
<!-- /wikitool:toc -->
<!-- wikitool:reference-commands -->
## `config set`
Set one value of the tiered settings, in the file its tier names.
**Synopsis**
- `wikitool config set <section>.<field> <value>`
- `wikitool config set <section>.<field> --ref <name|op://vault/item/field>`
**Options**
```text
Arguments:
key <section>.<field> [required]
value The plain value; a string as given, anything else as JSON.
Options:
--ref <str> A reference instead of a value: <name> or
op://<vault>/<item>/<field>.
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool config set tasks.stalled_waiting_days 14
tools/wikitool config set caldav.app_password --ref caldav_app_password
tools/wikitool config set caldav.username --ref op://Chemenu/caldav/username
```
**Behaviour**
- Writes a plain value of a field of tier `instance` into `wikitool.json`, and one of tier `checkout` - a private field's included - into `.wikitool-local.json`.
- With `--ref`, writes a reference instead of a value into `wikitool.json`: `<name>` resolves through `WIKITOOL_SECRET_<NAME>`, then the secrets file in the user's configuration directory; `op://<vault>/<item>/<field>` resolves through the 1Password CLI. Only a secret or private field takes one.
- Removes the same key from the other file, so a value never has two places, and says so.
- Creates `wikitool.json` with a fresh `instance_id` the first time it writes anything; the secrets file is found by that id.
- Checks both files as they would stand after the write against every registered section, and writes nothing when that fails.
- Prints the key and the file it wrote, never the value. Resolves no reference.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Each file is replaced in one rename; both are checked before either is written
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - The key names no registered section or field, or is not `<section>.<field>`
- `1` - A plain value given for a field of tier `secret`
- `1` - `--ref` given for a field that takes a plain value, or a malformed reference
- `1` - The value is not of the field's type, or the files would not pass their check afterwards
- `42` - The field names a program to run or affects a gate
**See also**
- `wikitool doctor` - whether both files pass and every reference resolves
- `docs/reference/configuration.md` - the files, the tiers and the references
- `docs/how-to/use-1password-for-settings.md` - setting 1Password up as the source
## `config migrate`
Move the settings out of the per-feature files of earlier versions into the tiered settings.
**Synopsis**
- `wikitool config migrate`
**Options**
```text
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool config migrate
```
**Behaviour**
- Moves the four per-feature settings files of earlier versions - telemetry, remotes, upload and the task tracker, each named in the migration document - into the sections `telemetry`, `remotes`, `upload`, and `tasks`, `superproductivity` and `caldav`: each value into the file its field's tier names, a secret's value into the secrets file with only a reference in `wikitool.json`.
- Every setting keeps its effective value, defaults included: the old content is read with the same checks the new sections use, and a field the old file did not set is not written.
- Run once per checkout, since the old files and `.wikitool-local.json` belong to one checkout. `.wikitool-tools.json` stays as it is.
- Prints each key and the file it went to, plus anything the old file held that was not read before and is not carried over - never a value.
- A second run, or a run with no old file, prints that there is nothing to move and changes no file.
- A key the target files already hold with a different value stops the move before anything is written: decide which one stands, remove the other, run again.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Everything is computed and checked in memory first; the secrets file, `wikitool.json` and `.wikitool-local.json` are each replaced in one rename, read back and checked, and only then are the old files removed. Any failure before that puts every target back to the bytes it had and leaves the old files as they were
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - An old file does not pass the check the new section applies, a target key holds a different value, or the files would not pass their check afterwards
**See also**
- `wikitool doctor` - reports every leftover old file as a `FAIL`
- `docs/reference/configuration.md` - the sections the files became
- `instructions/migrations/10.0.0-settings-tiers.md` - the migration this command belongs to
<!-- /wikitool:reference-commands -->
Reference-Commands-Content-migrations.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/content-migrations.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/content-migrations.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/content-migrations.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/content-migrations.md) at v10.0.0 – edit the repository, not this wiki.*
# Content migrations
Reference-Commands-Distribution-and-versioning.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/distribution-and-versioning.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/distribution-and-versioning.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/distribution-and-versioning.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/distribution-and-versioning.md) at v10.0.0 – edit the repository, not this wiki.*
# Distribution and versioning
Reference-Commands-Exports.md
+144
@@ -0,0 +1,144 @@
*Generated from [`docs/reference/commands/exports.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/exports.md) at v10.0.0 – edit the repository, not this wiki.*
# Exports
Derived, read-only versions of `kb/`: the generated `GUIDELINES.md` for the repositories the wiki has captured, and the pages exported into configured folders.
<!-- wikitool:toc -->
## Contents
- [`export guidelines`](#export-guidelines)
- [`export run`](#export-run)
<!-- /wikitool:toc -->
<!-- wikitool:reference-commands -->
## `export guidelines`
Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
**Synopsis**
- `wikitool export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>]` - Prints the file to stdout; no network.
- `wikitool export guidelines <predicates> --push [--bundle <raw-bundle> ...] [--confirm TOKEN]` - Writes it into the captured repositories, behind the gate.
**Options**
```text
-f, --field <str> Frontmatter predicate, repeatable (AND) - the same forms
as `search --field`.
--kind <str> Shorthand for --field kind=<value>.
--subtype <str> Shorthand for --field subtype=<value>.
--collection <str> Shorthand for --field collection=<value>.
--tag <str> Shorthand for --field tags=<value>.
--push Write the file into the captured repositories (Guideline
Push Gate).
--bundle <path> Only the repository of this captured bundle under raw/;
repeatable.
--confirm <str> The token from a prior exit 42, once a human has approved
it.
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool export guidelines --tag guideline
tools/wikitool export guidelines --tag guideline --push
tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # re-run after exit 42, once the user approved every diff
```
**Behaviour**
- Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), `--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's guidelines is written down in `kb/CONVENTIONS.md`.
- Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page> - generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and its text without frontmatter, the generated links and footnote blocks, citation markers (`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as `Title`. A leading H1 of the page is replaced by the title heading. Code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Refuses before printing or pushing anything: no predicate, no match, an unknown field, any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or `types/` (untracked files included), or no git checkout.
- `--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref rule is a tag pattern, it cannot be reached or has no such branch, its root has no `GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), or the header names another `instance=`. A header line alone is the opt-in.
- Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, `unchanged`, `skipped`), the diff for every target it would write, and the re-run line. The token digests URL, branch, old tip and new blob of every target to write.
- With a matching token, each target gets exactly one commit on the fetched tip changing only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, `skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`.
- Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A second run after a successful one finds every target unchanged.
- The target repository's own `AGENTS.md` points at the file (Claude Code: `@GUIDELINES.md`); that belongs to the repository, not to this command.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Per target - one commit pushed without force, or nothing; targets already written stay written when a later one fails
- Counts toward the session call budget: yes
- Can reach the network: yes
- Gates: guideline-push - see [exit codes and gates](Reference-Exit-codes)
**Exit codes**
- `0` - success
- `0` - A target was skipped - tag rule, not reachable, not opted in, hand-written file, another instance's file
- `1` - No predicate, no page matches, a malformed predicate or an unknown field
- `1` - A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout
- `1` - `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched
- `1` - `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push`
- `1` - A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served
- `42` - Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
**See also**
- `wikitool search` - the same predicates, to preview the selection
- `wikitool raw capture` - how a repository becomes a target
- `instructions/gates.md` - the gate procedure
## `export run`
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
**Synopsis**
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
**Options**
```text
Arguments:
[TARGET]... Targets to run; default: every configured target.
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool export run
tools/wikitool export run work-notes
```
**Behaviour**
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
- `1` - `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
- `1` - A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
**See also**
- `wikitool search` - the same predicates, to preview a selection
- `wikitool backup run` - the same kind of folder, for this machine's own files
- `docs/how-to/export-to-a-synced-folder.md`
<!-- /wikitool:reference-commands -->
Reference-Commands-Finding-and-checking.md
+6 -4
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/finding-and-checking.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/finding-and-checking.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/finding-and-checking.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/finding-and-checking.md) at v10.0.0 – edit the repository, not this wiki.*
# Finding and checking
@@ -187,7 +187,8 @@ tools/wikitool review --json
- **someday-stale**: a someday/maybe item untouched for longer than `thresholds.someday_stale_months`.
- A value a provider genuinely cannot supply - a `WAITING` item with no `follow_up_at`, a tracker project with no determinable creation date - is its own finding (`waiting_no_follow_up`/`project_age_unknown`) rather than a silent skip.
- Thresholds come from the tracker configuration, never from the schema.
- The tracker configuration is read from `.wikitool-tasks.json`, or from the file `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- The tracker configuration is read from the `tasks`, `superproductivity` and `caldav` sections of the settings, or from the profile `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- The tracker settings file of an earlier version exits 42 and names `tools/wikitool config migrate`; so does a secret source that needs unlocking.
- Text output is one `[check] project: message` line per finding, preceded by a `Source:` line naming which access path answered and, for `superproductivity`'s `access: "snapshot"`, the snapshot's age. `--json` carries the same findings plus `checks_run`/`checks_skipped`/`kb_project_count`/`complete`/`source` (`{"kind": ..., "detail": ...}` or `null`).
- A provider that cannot be reached mid-run degrades only the checks that needed the failing call; the report is printed in full, then exit 1 follows - never rendered as if it were complete.
- Re-reads everything fresh on every call, so nothing is ever stale to re-fetch.
@@ -204,9 +205,10 @@ tools/wikitool review --json
**Exit codes**
- `0` - success
- `1` - No `.wikitool-tasks.json`, or a malformed one - a clear "no tracker configured" message
- `1` - `WIKITOOL_TASKS_CONFIG` names a file that does not exist or is broken
- `1` - No `tasks` section in the settings, or a malformed one - a clear "no tracker configured" message
- `1` - `WIKITOOL_TASKS_CONFIG` names a profile that does not exist or is broken
- `1` - The provider was reachable at config-parse time but a read call failed mid-run; the full report (findings plus which checks ran) was printed first
- `42` - The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
**See also**
Reference-Commands-Git.md
+114 -5
@@ -1,14 +1,17 @@
*Generated from [`docs/reference/commands/git.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/git.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/git.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/git.md) at v10.0.0 – edit the repository, not this wiki.*
# Git
Pulling in the remote and publishing a change - the only way a change leaves this checkout.
Pulling in the remote and publishing a change - the only way a change leaves this checkout - and
backing the checkout up into a synced folder.
<!-- wikitool:toc -->
## Contents
- [`sync`](#sync)
- [`publish`](#publish)
- [`backup init`](#backup-init)
- [`backup run`](#backup-run)
<!-- /wikitool:toc -->
<!-- wikitool:reference-commands -->
@@ -128,9 +131,12 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
- Mass-Update Gate: counts the files that would be committed, refuses with exit 42 at `--threshold` (default 10) or more, and prints a review report - a scale line (file count, total lines added/removed, status breakdown), attention notes where they apply (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn. The list is what the commit will hold: it is computed from a scratch copy of the index after `git add -A`, so a path that is staged as deleted and back in the working tree is not counted twice, and a rename counts as its old path deleted plus its new path added. The real index and the working tree are not touched, so a refused publish leaves both byte-identical.
- Never counted and never shown for approval, but committed like everything else: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`). The refusal line accounts for both, by reason.
- The `--confirm` token covers each counted path, the blob id of its contents and the publish target: a different file list or edited contents need a new clearance.
- Publish-Remote Gate: when the checkout carries `.wikitool-remotes.json` and the push URL of `--remote` is not listed in it, exits 42 before the reconcile fetches anything. The URL is read with `git remote get-url --push`, so a repointed remote does not pass on its name. An absent file means unrestricted; a malformed one is an error, not permission.
- Publish-Remote Gate: when `.wikitool-local.json` carries `remotes.allowed_push_urls` and the push URL of `--remote` is not listed in it, exits 42 before the reconcile fetches anything. The URL is read with `git remote get-url --push`, so a repointed remote does not pass on its name. An absent section means unrestricted; a malformed one is an error, not permission.
- The remotes settings file of an earlier version is not read: while it exists this exits 42 and names `tools/wikitool config migrate`, so a restriction is never lost silently.
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
**Properties**
@@ -145,18 +151,121 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
**Exit codes**
- `0` - success
- `0` - The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
- `0` - The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
- `1` - git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
- `1` - The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
- `1` - No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
- `1` - `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
- `1` - `.wikitool-remotes.json` is unreadable or has no usable `allowed_push_urls` list
- `1` - The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
- `42` - The remotes settings file of an earlier version is still there
- `42` - Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
- `42` - Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
- `42` - Publish-Remote Gate: `.wikitool-remotes.json` exists and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
- `42` - Publish-Remote Gate: `remotes.allowed_push_urls` is set and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
**See also**
- `wikitool sync` - the same reconcile on its own, without committing or pushing
- `instructions/gates.md` - the gate procedure
- `instructions/setup-instance.md` - the first publish of a new instance
- `wikitool backup run` - the machine-data backup on its own
- `wikitool export run` - the page export on its own
## `backup init`
Create the backup target's bare repository and make it `origin` when there is none.
**Synopsis**
- `wikitool backup init`
**Options**
```text
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool backup init
```
**Behaviour**
- Reads the `backup` section of `.wikitool-local.json` and creates `<target>/` and `<target>/git`, the latter as a bare repository (`git init --bare -b main`). An existing bare repository there, or an empty directory, is used as it is.
- Sets `origin` to `<target>/git` when the checkout has no `origin`. An `origin` that already pushes there is left as it is; one that pushes anywhere else stops the command.
- When `remotes.allowed_push_urls` exists and lacks the URL `origin` pushes to, prints the `remotes` section with that URL added, for the user to copy into `.wikitool-local.json`. It never writes that file: the field is a gate.
- Pushes nothing: the next `publish` sends the history to the new `origin`.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Every precondition is checked first; a refusal changes nothing
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - No `backup` section in `.wikitool-local.json`, or a malformed one - a target that is not absolute or lies inside the instance, a `machine_name` that cleans to nothing or to `git`, an `extra` path that resolves outside the instance or into `.git/`
- `1` - `<target>/git` exists and is not a bare repository; nothing was changed
- `1` - `origin` already pushes to another URL; the message names both and nothing was changed
**See also**
- `wikitool backup run` - the machine-data half of the backup
- `wikitool publish` - pushes to the bare repository and runs the machine-data backup after it
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
## `backup run`
Mirror this machine's files that are not in git into `<target>/<machine>/`.
**Synopsis**
- `wikitool backup run`
**Options**
```text
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool backup run
```
**Behaviour**
- Mirrors `.wikitool-local.json`, `ENVIRONMENT.md` if present, every file in `incoming/` but `.gitkeep`, and the `extra` paths of the `backup` section into `<target>/<machine>/`, then writes the manifest `.chemenu-backup.json` there: time, stack version, machine name, and per file its relative path, size and sha256.
- Never mirrors `quarantine/`, `reports/`, `.wikitool-tools.json`, `tools/.venv/`, the published skills or `.git/` - unless an `extra` path names one of the first ones. The secrets file lies outside the instance and is never read.
- Writes and deletes only under `<target>/<machine>/`. Deletes only a file the previous manifest lists and the instance no longer has; a file there the manifest does not list stays as it is. A file whose size and sha256 match the previous manifest is not rewritten.
- A symbolic link or junction inside a mirrored directory is skipped and named in the output, never followed.
- `publish` runs the same step after every push when `machine_data` is true.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Per file - each is written to a temporary file and renamed, the manifest last; an interrupted run leaves whole files and the previous manifest
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - No `backup` section in `.wikitool-local.json`, or a malformed one - a target that is not absolute or lies inside the instance, a `machine_name` that cleans to nothing or to `git`, an `extra` path that resolves outside the instance or into `.git/`
- `1` - `machine_data` is false
- `1` - `<target>/<machine>/` is not empty and has no manifest, or its manifest cannot be read; nothing was written
- `1` - The target is not a directory (the synced folder is missing or not mounted), or a write failed
**See also**
- `wikitool backup init` - sets up the target's bare repository
- `wikitool doctor` - the `backup` line reports the age of the last run
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
<!-- /wikitool:reference-commands -->
Reference-Commands-Guideline-export.md
-79
@@ -1,79 +0,0 @@
*Generated from [`docs/reference/commands/guideline-export.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/guideline-export.md) at v9.0.0 – edit the repository, not this wiki.*
# Guideline export
Writing the generated `GUIDELINES.md` into the repositories the wiki has captured.
<!-- wikitool:reference-commands -->
## `export guidelines`
Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
**Synopsis**
- `wikitool export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>]` - Prints the file to stdout; no network.
- `wikitool export guidelines <predicates> --push [--bundle <raw-bundle> ...] [--confirm TOKEN]` - Writes it into the captured repositories, behind the gate.
**Options**
```text
-f, --field <str> Frontmatter predicate, repeatable (AND) - the same forms
as `search --field`.
--kind <str> Shorthand for --field kind=<value>.
--subtype <str> Shorthand for --field subtype=<value>.
--collection <str> Shorthand for --field collection=<value>.
--tag <str> Shorthand for --field tags=<value>.
--push Write the file into the captured repositories (Guideline
Push Gate).
--bundle <path> Only the repository of this captured bundle under raw/;
repeatable.
--confirm <str> The token from a prior exit 42, once a human has approved
it.
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool export guidelines --tag guideline
tools/wikitool export guidelines --tag guideline --push
tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # re-run after exit 42, once the user approved every diff
```
**Behaviour**
- Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), `--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's guidelines is written down in `kb/CONVENTIONS.md`.
- Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page> - generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and its text without frontmatter, the generated links and footnote blocks, citation markers (`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as `Title`. A leading H1 of the page is replaced by the title heading. Code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Refuses before printing or pushing anything: no predicate, no match, an unknown field, any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or `types/` (untracked files included), or no git checkout.
- `--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref rule is a tag pattern, it cannot be reached or has no such branch, its root has no `GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), or the header names another `instance=`. A header line alone is the opt-in.
- Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, `unchanged`, `skipped`), the diff for every target it would write, and the re-run line. The token digests URL, branch, old tip and new blob of every target to write.
- With a matching token, each target gets exactly one commit on the fetched tip changing only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, `skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`.
- Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A second run after a successful one finds every target unchanged.
- The target repository's own `AGENTS.md` points at the file (Claude Code: `@GUIDELINES.md`); that belongs to the repository, not to this command.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Per target - one commit pushed without force, or nothing; targets already written stay written when a later one fails
- Counts toward the session call budget: yes
- Can reach the network: yes
- Gates: guideline-push - see [exit codes and gates](Reference-Exit-codes)
**Exit codes**
- `0` - success
- `0` - A target was skipped - tag rule, not reachable, not opted in, hand-written file, another instance's file
- `1` - No predicate, no page matches, a malformed predicate or an unknown field
- `1` - A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout
- `1` - `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched
- `1` - `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push`
- `1` - A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served
- `42` - Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
**See also**
- `wikitool search` - the same predicates, to preview the selection
- `wikitool raw capture` - how a repository becomes a target
- `instructions/gates.md` - the gate procedure
<!-- /wikitool:reference-commands -->
Reference-Commands-Instance-health.md
+14 -6
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/instance-health.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/instance-health.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/instance-health.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/instance-health.md) at v10.0.0 – edit the repository, not this wiki.*
# Instance health
@@ -29,7 +29,7 @@ tools/wikitool doctor --json
**Behaviour**
- Checks dependencies (Python, ripgrep), author resolution, stack version, git identity/branch/remote, published skills, the kb/raw/reports/work/instructions structure, and generated files.
- Checks dependencies (Python, ripgrep), author resolution, stack version, git identity/branch/remote, published skills, the kb/raw/reports/work/sites/instructions structure, and generated files.
- Import filters (`import-filter-<name>`): one line for each `raw derive` format that reads through a package (PDF, Outlook `.msg`) - a `WARN` naming the package when it is missing, since that costs only its own format; fixed by running `tools/preflight.sh` again.
- Tool paths (`tool-paths`): `.wikitool-tools.json` written by a preflight that finished, every tool `tools/prerequisites.txt` names for this platform recorded, and every recorded path still there - a `FAIL` otherwise, fixed by running `tools/preflight.sh` again (PowerShell 7: `tools/preflight.ps1`).
- Install folder (`install-dir`): on Windows with long paths off, a `FAIL` when the folder holding `tools/` is longer than `tools/prerequisites.txt` allows (95 characters) - the limit the preflight enforces before it sets anything up.
@@ -37,12 +37,19 @@ tools/wikitool doctor --json
- Personalization: `USER.md`/`SOUL.md` present **and** filled - a file still carrying the template's sentinel is a `FAIL`.
- KB conventions: `kb/CONVENTIONS.md` present, unsentinelled, and naming all three tool-owned section headings - a `FAIL` on any of the three.
- Environment note: `ENVIRONMENT.md` is optional, so absent is `OK`; a still-templated one is a `WARN`.
- MCP `submit` tool: whether `.wikitool-upload.json` is present, absent or malformed, its limits, and how many submissions wait in `mcp-upload/`. Absent is `OK` and means the write path does not exist at all; malformed is a `FAIL`.
- Task tracker: `.wikitool-tasks.json` present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, that file is read instead and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
- For a configured `caldav` provider, whether the server is reachable and Basic auth succeeds - never a `FAIL`; only a broken config block is.
- Settings (`settings-instance`, `settings-checkout`, `settings-secrets`): one line per tier. `wikitool.json` and `.wikitool-local.json` each present or absent - absent is `OK` - and a `FAIL` naming file, key and the right tier when a value sits in the wrong file, a secret or a committed private field holds a plain value, or a value fails its field; a section this version does not read is a `WARN`. For the references: how many resolve, never a value - an unresolvable one is a `FAIL`, one 1Password will not hand out until a human unlocks it a `WARN`.
- Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
- Telemetry: on or off and why - installation-form default, `.wikitool-telemetry.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when `.wikitool-telemetry.json` is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
- Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
- Browser component (`browser`): whether the optional Playwright component and its browser are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is missing and the instance needs it: a host `fetch.routes` sends through the session (`raw login`), or a site handler that declares `render`. The user installs it with `tools/preflight.sh --with browser`.
- Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
- Read-only and exempt from the Iteration Budget Gate.
@@ -63,7 +70,8 @@ tools/wikitool doctor --json
- `instructions/setup-instance.md` - the setup steps most findings point back to
- `instructions/preflight.md` - what `tool-paths` and `install-dir` point back to
- `docs/reference/configuration.md` - the per-checkout configuration files
- `docs/reference/configuration.md` - the per-checkout configuration files and the tiered settings
- `docs/explanation/telemetry-and-evaluation.md` - telemetry state and caps
- `docs/how-to/back-up-to-a-synced-folder.md` - what `sync-folder` and `backup` point to
- `instructions/session-setup.md` - setting `WIKITOOL_SESSION_ID`
<!-- /wikitool:reference-commands -->
Reference-Commands-Links-and-citations.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/links-and-citations.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/links-and-citations.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/links-and-citations.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/links-and-citations.md) at v10.0.0 – edit the repository, not this wiki.*
# Links and citations
Reference-Commands-Pages.md
+17 -14
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/pages.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/pages.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/pages.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/pages.md) at v10.0.0 – edit the repository, not this wiki.*
# Pages
@@ -73,7 +73,7 @@ tools/wikitool new project --name "Homelab migration" --set responsibility=infra
- `--set` is repeatable, and comma-separated values fill array fields. An element that itself contains a comma is written `\,`, or passed as its own repeated `--set` for that field - repeating an array field appends.
- A capture field the type-spec requires (a source's `fidelity`/`authority`) must be passed with `--set`; `new` never guesses it and refuses `unknown` for it.
- Produces structurally correct frontmatter and a body skeleton only - the prose (Description, Summary, judgment calls about relationships) is written afterwards.
- `new project`: with `.wikitool-tasks.json` configuring a task tracker, also makes sure a same-named tracker project exists - one name, one identity. No tracker configured is a legitimate, explicitly announced state: page only.
- `new project`: with the `tasks` section of the settings configuring a task tracker, also makes sure a same-named tracker project exists - one name, one identity. No tracker configured is a legitimate, explicitly announced state: page only.
- `new project`: tracker before page. The tracker side is settled first, so a failure past that point leaves a tracker project with no page - a state `review`'s check 3 reports - never a page with no tracker project.
- `new project`: a name already taken, case-insensitively, in `kb/` or the tracker is refused outright, naming where it was found, and creates nothing. For `caldav` the tracker check covers every list in the account, not only the ones counted as projects.
- `new project`: a provider whose configured access path has no write path (Super Productivity's `access: "snapshot"`) refuses entirely with exit 1, naming the `access: "api"` instance to use instead - neither the tracker project nor the page is created, and `--resume` behaves the same.
@@ -156,10 +156,10 @@ tools/wikitool task new --title "Read the qmd README" --inbox --notes "from [[So
- `--inbox` files into the tracker's own inbox. An item filed there never appears in `review`, since every one of its checks reaches items through a project name.
- `--waiting` sets the WAITING status `review`'s waiting-overdue check reads. `--follow-up-at` is refused without `--waiting` - it is never a due date on its own.
- `--notes` carries a freetext backref (e.g. to the `kb/` source page the item came from), stored verbatim, never parsed.
- The tracker configuration is read from `.wikitool-tasks.json`, or from the file `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- No `.wikitool-tasks.json` fails immediately with a "no tracker configured" message.
- The tracker configuration is read from the `tasks`, `superproductivity` and `caldav` sections of the settings, or from the profile `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- No `tasks` section fails immediately with a "no tracker configured" message.
- A provider whose configured access path has no write path (Super Productivity's `access: "snapshot"`) refuses entirely with exit 1, naming the `access: "api"` instance to use instead.
- **Never exits 42.** A `--project` matching no tracker project, or `--waiting` against a provider that cannot represent it right now (Super Productivity: the `waiting` tag does not exist yet, and its API cannot create tags), are ordinary exit-1 refusals that create nothing.
- **Exits 42 only for the settings** (the tracker settings file of an earlier version, a locked secret source). A `--project` matching no tracker project, or `--waiting` against a provider that cannot represent it right now (Super Productivity: the `waiting` tag does not exist yet, and its API cannot create tags), are ordinary exit-1 refusals that create nothing.
- Not idempotent: every successful run creates another item.
**Properties**
@@ -173,12 +173,13 @@ tools/wikitool task new --title "Read the qmd README" --inbox --notes "from [[So
**Exit codes**
- `0` - success
- `1` - No `.wikitool-tasks.json` - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a file that does not exist or is broken
- `1` - No `tasks` section in the settings - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a profile that does not exist or is broken
- `1` - Neither or both of `--project`/`--inbox`, or a `--follow-up-at` without `--waiting` or not `YYYY-MM-DD`
- `1` - A `--project` name matching no tracker project
- `1` - `--waiting` against a provider with no way to represent it right now (Super Productivity: the `waiting` tag does not exist)
- `1` - A read-only access path (Super Productivity's `access: "snapshot"`)
- `42` - The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
**See also**
@@ -229,8 +230,9 @@ tools/wikitool task list --project "Homelab migration"
- `0` - success
- `0` - A `--project` matching no tracker project - prints "No open items", not an error
- `1` - No `.wikitool-tasks.json` - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a file that does not exist or is broken
- `1` - No `tasks` section in the settings - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a profile that does not exist or is broken
- `42` - The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
**See also**
@@ -264,10 +266,10 @@ tools/wikitool task close --id <item-id>
- Marks one tracker item done. It never deletes or moves an item - the only closing write this stack makes.
- `--id` is the provider's own item id, from `task list` or a `review` finding - never a title.
- The tracker configuration is read from `.wikitool-tasks.json`, or from the file `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- No `.wikitool-tasks.json` fails with a "no tracker configured" message.
- The tracker configuration is read from the `tasks`, `superproductivity` and `caldav` sections of the settings, or from the profile `WIKITOOL_TASKS_CONFIG` names when that variable is set (so one checkout can be run against several trackers in turn). A set variable that names no file is an error, never "no tracker configured".
- No `tasks` section fails with a "no tracker configured" message.
- A provider whose configured access path has no write path (Super Productivity's `access: "snapshot"`) refuses entirely with exit 1, naming the `access: "api"` instance to use instead.
- **Never exits 42.**
- **Exits 42 only for the settings** (the tracker settings file of an earlier version, a locked secret source).
**Properties**
@@ -280,10 +282,11 @@ tools/wikitool task close --id <item-id>
**Exit codes**
- `0` - success
- `1` - No `.wikitool-tasks.json` - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a file that does not exist or is broken
- `1` - No `tasks` section in the settings - no tracker configured
- `1` - `WIKITOOL_TASKS_CONFIG` names a profile that does not exist or is broken
- `1` - An `--id` matching no tracker item right now
- `1` - A read-only access path (Super Productivity's `access: "snapshot"`)
- `42` - The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
**See also**
Reference-Commands-Provenance.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/provenance.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/provenance.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/provenance.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/provenance.md) at v10.0.0 – edit the repository, not this wiki.*
# Provenance
Reference-Commands-Raw-material-and-uploads.md
+145 -14
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/raw-material-and-uploads.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/raw-material-and-uploads.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/raw-material-and-uploads.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/raw-material-and-uploads.md) at v10.0.0 – edit the repository, not this wiki.*
# Raw material and uploads
@@ -8,11 +8,13 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
## Contents
- [`raw fetch`](#raw-fetch)
- [`raw login`](#raw-login)
- [`raw derive`](#raw-derive)
- [`raw capture`](#raw-capture)
- [`raw status`](#raw-status)
- [`raw pending`](#raw-pending)
- [`raw accept`](#raw-accept)
- [`upload collect`](#upload-collect)
- [`upload list`](#upload-list)
- [`upload show`](#upload-show)
- [`upload accept`](#upload-accept)
@@ -22,11 +24,12 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
<!-- wikitool:reference-commands -->
## `raw fetch`
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
**Synopsis**
- `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
- `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
**Options**
@@ -40,6 +43,8 @@ Options:
a browser) instead of fetching - no network access
--url <str> With --html: the page's URL, for the header and relative links
--name <str> File stem in incoming/ instead of the one computed from the URL
--scope Also fetch the subpages the site handler offers for this page,
into the folder incoming/<stem>/
-h, --help Show this message and exit.
```
@@ -48,18 +53,24 @@ Options:
```bash
tools/wikitool raw fetch https://example.org/blog/post
tools/wikitool raw fetch https://example.org/ --name example-start
tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post
```
**Behaviour**
- Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.
- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text.
- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
- Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
- Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
- `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
- Success prints the `raw accept` line for the written files and the `source_url` for the source page.
@@ -67,7 +78,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
- Changes files: yes
- Idempotent: no
- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
- Counts toward the session call budget: yes
- Can reach the network: yes
@@ -75,16 +86,79 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
- `0` - success
- `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
- `1` - A target file already exists in `incoming/`
- `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
- `1` - The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
- `1` - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
- `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
- `42` - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
- `42` - The site handler declares `render` and the optional browser component is not installed
**See also**
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
- `sites/CONTRACT.md` - the instance's own handler files
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
- `wikitool raw login` - the user signs in once for a host routed through the session
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
## `raw login`
Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
**Synopsis**
- `wikitool raw login <host>`
**Options**
```text
Arguments:
host The website's host, such as www.example.org [required]
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool raw login www.heise.de
```
**Behaviour**
- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
- Prints no part of the session. Signing in again replaces the stored session.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
- Counts toward the session call budget: yes
- Can reach the network: yes
**Exit codes**
- `0` - success
- `1` - The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
- `1` - The browser could not be started, or the window closed before any state could be read
- `42` - The optional browser component is not installed
**See also**
- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
- `sites/CONTRACT.md` - a handler's `session` block
- `docs/reference/configuration.md` - the `fetch` section
## `raw derive`
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
@@ -190,7 +264,7 @@ tools/wikitool raw capture --update raw/2026/10/service-docs
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
@@ -294,7 +368,7 @@ tools/wikitool raw pending --json
**Behaviour**
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `quarantine/` is outside `incoming/` and never listed.
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
- The default is the first candidate `raw accept` would take, and the output names it.
@@ -405,7 +479,7 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
- `0` - success
- `1` - raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- `1` - raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
- `1` - raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- `1` - raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- `1` - raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
@@ -424,9 +498,64 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
- `wikitool types describe source` - the capture field values
- `wikitool new source` - the source page for a promoted file
## `upload collect`
Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
**Synopsis**
- `wikitool upload collect [<channel>...]`
**Options**
```text
Arguments:
names... Channels to collect from; all pulled channels when none is named
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool upload collect
tools/wikitool upload collect onedrive
```
**Behaviour**
- Without arguments, collects from every channel of the `intake` section whose handler is `folder` or `command`; with names, from those channels only. A name no channel has, or the name of the `mcp` channel (which is pushed, not pulled), exits 1 before anything happens.
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
- Not idempotent: a second run finds the folder emptied.
**Properties**
- Changes files: yes
- Idempotent: no
- Atomic: No - per file: one copy into `quarantine/<id>/` (manifest last), then one removal of the source; a channel's failure does not stop the others
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - Unknown channel name, an `mcp` channel as argument, a malformed `intake` section, or an `mcp-upload/` that still holds submissions
- `1` - A channel or a file failed (folder missing, program exited non-zero, timed out or could not be started, a source that could not be removed)
**See also**
- `wikitool upload list` - what waits for review afterwards
- `wikitool upload accept` - the gate out of the quarantine
- `docs/how-to/collect-from-a-synced-folder.md`
## `upload list`
List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
**Synopsis**
@@ -448,8 +577,9 @@ tools/wikitool upload list --json
**Behaviour**
- Lists every submission waiting in `mcp-upload/`, oldest id first: id, filename, size, submitter.
- Only ever non-empty when `.wikitool-upload.json` opts the checkout into the MCP server's `submit` tool.
- Lists every submission waiting in `quarantine/`, oldest id first: id, filename, size, channel, submitter.
- Only ever non-empty when the `intake` section of `.wikitool-local.json` declares a channel.
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions - the message names the command that moves it.
- A submission directory with a corrupt manifest is silently skipped.
- Never fails; read-only and safe to retry freely.
@@ -497,7 +627,8 @@ tools/wikitool upload show <id>
**Behaviour**
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name, not a claim the header was honest), submission time.
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name or the handler, not a claim the header was honest), submission time, and where it came from: channel, handler, origin (the source file's path, `command:<name>` or `mcp:submit`).
- A submitter of `unknown` means the channel has no sender to trust - read channel and origin instead.
- What a reviewer reads before `upload accept`.
- Read-only.
@@ -547,7 +678,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
**Behaviour**
- Promotes a submission's file from `mcp-upload/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `mcp-upload/ledger.jsonl`.
- Promotes a submission's file from `quarantine/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `quarantine/ledger.jsonl`.
- Upload Review Gate: without a matching `--confirm`, exits 42 and prints the manifest in full plus the exact re-run line - one submission at a time. The gate check runs before anything is moved.
- The token digests id, filename, size, sha256 and submitter, so an edited or superseded manifest invalidates it.
- An occupied `incoming/<filename>` is an ordinary validation error, not the gate.
@@ -564,7 +695,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
**Exit codes**
- `0` - success
- `1` - Unknown or malformed submission id, or the submission's file is missing from `mcp-upload/<id>/`
- `1` - Unknown or malformed submission id, or the submission's file is missing from `quarantine/<id>/`
- `1` - `incoming/<filename>` already exists
- `42` - Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
Reference-Commands-Telemetry.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/telemetry.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/telemetry.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/telemetry.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/telemetry.md) at v10.0.0 – edit the repository, not this wiki.*
# Telemetry
Reference-Commands-Types-instructions-and-docs.md
+7 -4
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/types-instructions-and-docs.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/types-instructions-and-docs.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/types-instructions-and-docs.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/types-instructions-and-docs.md) at v10.0.0 – edit the repository, not this wiki.*
# Types, instructions and docs
@@ -140,6 +140,7 @@ tools/wikitool instructions sync
**Behaviour**
- Publishes every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and deletes published skills whose source is gone.
- Each published `SKILL.md` is its source plus one generated line directly under the frontmatter: the `language:` of `kb/CONVENTIONS.md` and a pointer to `AGENTS.md` § Conversation language, so the agent has the value in context when the task begins. The frontmatter and every other file are copied unchanged. With no language declared, the `SKILL.md` is copied unchanged too, and the success message says so.
- Both targets are gitignored, so a fresh clone runs this once.
- Re-running repairs a drifted copy: the source always wins.
- `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it).
@@ -188,7 +189,7 @@ tools/wikitool instructions verify
- Flat instructions validate against `types/instruction.schema.yaml`, and each `SKILL.md` carries the frontmatter its harness reads.
- No `SKILL.md` carries a relative markdown link: `sync` copies it to a different depth than the source, so a `SKILL.md` references a target as a repo-root-relative plain path instead (`instructions/CONTRACT.md` § "A skill's outbound reference is a plain path, not a link").
- Every published copy is byte-identical to its source. Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout.
- Every published copy is what `sync` would write from its source: byte-identical, except for the generated language line in `SKILL.md` that the current `language:` of `kb/CONVENTIONS.md` calls for. Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout.
- No instruction is left that nothing references; one marked `manual: true` must instead not be linked from AGENTS.md, CLAUDE.md or a skill.
- Nothing under `instructions/dev/` is referenced from outside it; a `<!-- dist:strip-start/end -->` block is exempt (`instructions/CONTRACT.md`).
- Read-only.
@@ -206,7 +207,7 @@ tools/wikitool instructions verify
- `0` - success
- `1` - Nothing found under `instructions/` at all, or a malformed instruction or `SKILL.md`
- `1` - A `SKILL.md` carries a relative markdown link
- `1` - A published copy drifted from its source
- `1` - A published copy drifted from its source - including a language line that belongs to an earlier `language:` (the message names it as such)
- `1` - An instruction nothing references, or a `manual: true` one that IS linked from AGENTS.md, CLAUDE.md or a skill and so risks running implicitly
- `1` - Something under `instructions/dev/` is referenced from outside it and outside a `dist:strip` block
@@ -285,7 +286,8 @@ tools/wikitool docs verify
- Collections: every directory under `kb/` has a `COLLECTION.md` and no directory outside it does; every collection declares `profile:` and a `required_by_stack:` that agrees with the stack's own list.
- Types: every type the stack lists (currently `source` and `project`) has a type-spec of that name whose schema requires the field the stack list names (`raw_files:`/`state:`); every file under `types/` declaring `type: types/type-spec.md` validates against `types/type-spec.schema.yaml`; every subtype template `types/<type>.<value>.md` (any `<value>` but `guidance`) sits beside a type-spec declaring `subtype_field:`, names a value that field's schema enum allows, and carries no frontmatter; no pre-migration `type: entity` blocks are left in the contracts.
- `kb/CONVENTIONS.md`, if it exists at all, names all three tool-owned section headings; every stage contract is present.
- The `.gitignore` canaries clear in both directions: nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories.
- The `.gitignore` canaries clear in both directions: nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories, `.wikitool-local.json` ignored and `wikitool.json` tracked.
- A `wikitool.json` at the root holds every value in the place its tier allows - no secret and no private field as plain text, no `checkout` field at all except as a reference - and a well-formed `schema` and `instance_id`. The local file is not read.
- No `.md`/`.template` file `dist export` would ship cites an issue number. A `<!-- dist:strip-start/end -->` region is exempt: the check reads the export plan's text, from which it is already gone.
- Every reference file `docs toc` covers carries the current table-of-contents region for its own headings - missing and stale are one check.
- Every relative markdown link in one of those reference files resolves to an existing file. A target's `#anchor` suffix is stripped first, and code fences and inline code spans are masked before scanning, so link syntax shown as an example is not mistaken for a real reference.
@@ -314,6 +316,7 @@ tools/wikitool docs verify
- `1` - The `<!-- wikitool:commands -->` region of `tools/CONTRACT.md` is stale
- `1` - A type-spec's own frontmatter fails its schema
- `1` - A subtype template `types/<type>.<value>.md` has no type-spec with a `subtype_field:` beside it, names a value outside that field's enum, or carries frontmatter
- `1` - `wikitool.json` holds a secret or private value as plain text, or a value its tier puts elsewhere
- `1` - A shipped `.md`/`.template` cites an issue number
- `1` - A region of the human reference under `docs/reference/` is stale
- `1` - A page of the human reference is missing or lacks its marker pair, or a page under `docs/reference/commands/` belongs to no group
Reference-Commands-Workshop-runs-and-session-budget.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/workshop-runs-and-session-budget.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/workshop-runs-and-session-budget.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/workshop-runs-and-session-budget.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/workshop-runs-and-session-budget.md) at v10.0.0 – edit the repository, not this wiki.*
# Workshop runs and session budget
Reference-Commands.md
+20 -7
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/commands/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/commands/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/commands/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/commands/README.md) at v10.0.0 – edit the repository, not this wiki.*
# Command reference
@@ -17,12 +17,13 @@ collected in [exit codes and gates](Reference-Exit-codes).
- [Provenance](#provenance)
- [Raw material and uploads](#raw-material-and-uploads)
- [Git](#git)
- [Guideline export](#guideline-export)
- [Exports](#exports)
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
- [Types, instructions and docs](#types-instructions-and-docs)
- [Telemetry](#telemetry)
- [Distribution and versioning](#distribution-and-versioning)
- [Content migrations](#content-migrations)
- [Configuration](#configuration)
- [Instance health](#instance-health)
<!-- /wikitool:toc -->
@@ -80,13 +81,15 @@ Full records: [provenance.md](Reference-Commands-Provenance)
Full records: [raw-material-and-uploads.md](Reference-Commands-Raw-material-and-uploads)
- [`raw fetch`](Reference-Commands-Raw-material-and-uploads#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
- [`raw fetch`](Reference-Commands-Raw-material-and-uploads#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
- [`raw login`](Reference-Commands-Raw-material-and-uploads#raw-login) - Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
- [`raw derive`](Reference-Commands-Raw-material-and-uploads#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
- [`raw capture`](Reference-Commands-Raw-material-and-uploads#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
- [`raw status`](Reference-Commands-Raw-material-and-uploads#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
- [`raw pending`](Reference-Commands-Raw-material-and-uploads#raw-pending) - List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
- [`raw accept`](Reference-Commands-Raw-material-and-uploads#raw-accept) - Promote one or more files, or one folder, from `incoming/` into `raw/`.
- [`upload list`](Reference-Commands-Raw-material-and-uploads#upload-list) - List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
- [`upload collect`](Reference-Commands-Raw-material-and-uploads#upload-collect) - Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
- [`upload list`](Reference-Commands-Raw-material-and-uploads#upload-list) - List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
- [`upload show`](Reference-Commands-Raw-material-and-uploads#upload-show) - Print one submission's manifest in full.
- [`upload accept`](Reference-Commands-Raw-material-and-uploads#upload-accept) - **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
- [`upload reject`](Reference-Commands-Raw-material-and-uploads#upload-reject) - Delete a submission's material, keeping only its ledger trail.
@@ -97,12 +100,15 @@ Full records: [git.md](Reference-Commands-Git)
- [`sync`](Reference-Commands-Git#sync) - Fetch `<remote>/<branch>` and bring the local branch up to date with it.
- [`publish`](Reference-Commands-Git#publish) - Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
- [`backup init`](Reference-Commands-Git#backup-init) - Create the backup target's bare repository and make it `origin` when there is none.
- [`backup run`](Reference-Commands-Git#backup-run) - Mirror this machine's files that are not in git into `<target>/<machine>/`.
## Guideline export
## Exports
Full records: [guideline-export.md](Reference-Commands-Guideline-export)
Full records: [exports.md](Reference-Commands-Exports)
- [`export guidelines`](Reference-Commands-Guideline-export#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
- [`export guidelines`](Reference-Commands-Exports#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
- [`export run`](Reference-Commands-Exports#export-run) - Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
## Workshop runs and session budget
@@ -159,6 +165,13 @@ Full records: [content-migrations.md](Reference-Commands-Content-migrations)
- [`migrate done`](Reference-Commands-Content-migrations#migrate-done) - Record one migration as applied, advancing `kb_version` in `.wikitool-kb.json`.
- [`migrate baseline`](Reference-Commands-Content-migrations#migrate-baseline) - Declare `kb_version` once, for an instance predating `.wikitool-kb.json`.
## Configuration
Full records: [configuration.md](Reference-Commands-Configuration)
- [`config set`](Reference-Commands-Configuration#config-set) - Set one value of the tiered settings, in the file its tier names.
- [`config migrate`](Reference-Commands-Configuration#config-migrate) - Move the settings out of the per-feature files of earlier versions into the tiered settings.
## Instance health
Full records: [instance-health.md](Reference-Commands-Instance-health)
Reference-Configuration.md
+586 -184
@@ -1,17 +1,19 @@
*Generated from [`docs/reference/configuration.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/configuration.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/configuration.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/configuration.md) at v10.0.0 – edit the repository, not this wiki.*
# Configuration
The environment variables `wikitool` reads, and the per-checkout files an operator may create at the
root of an instance. None of them is needed for an instance to work: an instance installed from a
The environment variables `wikitool` reads and the two tiered settings files an operator may create at
the root of an instance. None of them is needed for an instance to work: an instance installed from a
release runs against the origin's public release feed with no configuration at all.
The shape of each file - its fields, what an absent or a malformed file means, an example - is
generated from the record `wikitool` itself reads the file with, so it cannot describe a field the
tool does not take. What a setting does around it is written by hand.
Every setting of a feature - telemetry, the publish allowlist, the task tracker, the intake
channels - is a section of those two files, and the shape of each section - its fields, their tier, what
an absent or a malformed section means, an example - is generated from the record `wikitool` itself
reads it with, so it cannot describe a field the tool does not take. What a setting does around it is
written by hand.
Three more files at the root are written by `wikitool` and never by hand: `.wikitool-release.json`,
`.wikitool-kb.json` and `.wikitool-tools.json` ([AGENTS.md invariant 1](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#invariants)).
`.wikitool-kb.json` and `.wikitool-tools.json` ([AGENTS.md invariant 1](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#invariants)).
The last one has its own section [below](#tool-paths), because the preflight that writes it is
something you run.
@@ -19,13 +21,20 @@ something you run.
## Contents
- [Environment variables](#environment-variables)
- [Telemetry](#telemetry)
- [Tool paths](#tool-paths)
- [Tiered settings](#tiered-settings)
- [Telemetry](#telemetry)
- [Allowed push URLs](#allowed-push-urls)
- [Task tracker](#task-tracker)
- [Super Productivity](#super-productivity)
- [CalDAV](#caldav)
- [MCP submissions](#mcp-submissions)
- [Profiles](#profiles)
- [Intake channels](#intake-channels)
- [Site handlers](#site-handlers)
- [Fetch routes](#fetch-routes)
- [Backup to a synced folder](#backup-to-a-synced-folder)
- [Export to a folder](#export-to-a-folder)
- [Files of earlier versions](#files-of-earlier-versions)
<!-- /wikitool:toc -->
## Environment variables
@@ -33,60 +42,26 @@ something you run.
| Variable | Purpose | Fallback |
|----------|---------|----------|
| `WIKI_AUTHOR` | Overrides the author name of new source pages | `git config user.name`. Without either, `new` stops with an `ERROR` |
| `WIKITOOL_SESSION_ID` | Scopes the Iteration Budget Gate to a task rather than a terminal, and names the directory a trace lands in | A session variable the harness sets itself, where one is known (for example `CLAUDE_CODE_SESSION_ID`), else the parent process id ([instructions/session-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/instructions/session-setup.md)) |
| `WIKITOOL_SESSION_ID` | Scopes the Iteration Budget Gate to a task rather than a terminal, and names the directory a trace lands in | A session variable the harness sets itself, where one is known (for example `CLAUDE_CODE_SESSION_ID`), else the parent process id ([instructions/session-setup.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/session-setup.md)) |
| `WIKITOOL_UPDATE_URL` | The release feed `version check`, `version notes` and `dist upgrade --latest` ask | The value in `.wikitool-release.json`, else the origin's feed |
| `WIKITOOL_UPDATE_TOKEN` | A Gitea token for the release feed, sent only to downloads on the feed's own host | None. Not needed against the origin, only against a feed in a private repository |
| `WIKITOOL_TASKS_CONFIG` | Path to a tracker configuration that `task`, `review` and `doctor` read instead of `.wikitool-tasks.json`, for running one checkout against several trackers in turn | `.wikitool-tasks.json` at the root. A path to a file that does not exist is an error, never "no tracker configured" |
| `WIKITOOL_TASKS_CONFIG` | Path to a tracker profile that `task`, `review` and `doctor` read instead of the `tasks`, `superproductivity` and `caldav` sections of the settings, for running one checkout against several trackers in turn ([below](#profiles)) | The sections in `wikitool.json` and `.wikitool-local.json`. A path to a file that does not exist is an error, never "no tracker configured" |
| `WIKITOOL_SECRET_<NAME>` | The value behind a settings reference `{"ref": "<name>"}`, for CI and containers - `<NAME>` is the name upper-cased | The secrets file in the user's configuration directory ([below](#tiered-settings)) |
| `CHEMENU_ROOT` | Which corpus the package works on, for a caller outside the checkout itself | The checkout the package sits in. `tools/wikitool` behaves the same without it |
| `WIKI_TRACE` | `0`/`1` turns telemetry off or on, beating both the default and `.wikitool-telemetry.json` | The installation-form default ([below](#telemetry)) |
| `WIKI_TRACE` | `0`/`1` turns telemetry off or on, beating both the default and the `telemetry` section | The installation-form default ([below](#telemetry)) |
| `WIKI_TRACE_DIR` | Writes traces somewhere other than `reports/telemetry/` | `reports/telemetry/` |
| `WIKI_TRACE_CONTENT` | `0` keeps only lengths and digests of prompts and replies instead of their text | Text is kept |
| `WIKI_TRACE_MAX_CONTENT` | Per-attribute content cap in characters | 60 KiB |
| `WIKI_TRACE_MAX_SESSION_BYTES` | Byte cap per session trace; beats the file's `max_session_bytes` | 5 MiB |
| `WIKI_TRACE_KEEP_SESSIONS` | How many session directories retention keeps; beats the file's `keep_sessions` | 250 |
## Telemetry
**Telemetry is off in an instance.** Every instance carries the `.wikitool-release.json` of its
release, and that is how `chemenu.telemetry.policy` recognises that nobody asked for telemetry. The
same holds for every further clone of the instance, because the file is committed with the first
publish. Only a clone of the origin repository, used to develop the stack, has no stamp. There the
traces are the instrument the stack measures itself with, and they are **on**.
To turn the default around, create `.wikitool-telemetry.json` at the root. It is per checkout and
gitignored, with no `.template`, like `.wikitool-remotes.json`.
<!-- wikitool:reference-config-telemetry -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `enabled` | boolean | no | the installation-form default | Telemetry on or off. `WIKI_TRACE` beats it in both directions. Any other value is read by its truthiness, so `"no"` means on. |
| `max_session_bytes` | positive integer | no | `5242880` | Byte cap per session trace. `WIKI_TRACE_MAX_SESSION_BYTES` beats it. A number written as a string is accepted. |
| `keep_sessions` | positive integer | no | `250` | How many session directories retention keeps. `WIKI_TRACE_KEEP_SESSIONS` beats it. A number written as a string is accepted. |
- **Absent:** The installation-form default applies: on in a clone of the origin repository, off in an instance.
- **Malformed:** Never an error. Unreadable JSON or a non-object counts as an absent file, and a cap that is not a positive integer as an absent cap. `doctor` reports each such deviation as a `WARN` under `telemetry`, naming the field.
- **Unknown keys:** Ignored; `doctor` reports them as a `WARN`.
Example:
```json
{
"enabled": true,
"max_session_bytes": 5242880,
"keep_sessions": 250
}
```
<!-- /wikitool:reference-config-telemetry -->
`wikitool doctor` reports the current state: on or off, why, and how much has been recorded against
both caps. What the caps do and what never reaches a trace:
[telemetry-and-evaluation.md](Explanation-Telemetry-and-evaluation).
| `WIKI_TRACE_MAX_SESSION_BYTES` | Byte cap per session trace; beats the section's `max_session_bytes` | 5 MiB |
| `WIKI_TRACE_KEEP_SESSIONS` | How many session directories retention keeps; beats the section's `keep_sessions` | 250 |
## Tool paths
**The preflight writes this file, not you.** `.wikitool-tools.json` at the root holds the absolute
paths of Python, git and ripgrep, as the preflight found them on this machine. `wikitool` starts git
paths of Python, git and ripgrep, as the preflight found them on this machine - and of the
1Password CLI `op` when it is installed. `op` is optional: the preflight records it when it finds
it, reports `ABSENT` when it does not, and never asks for it. Only an `op://` reference in the
[tiered settings](#tiered-settings) needs it. `wikitool` starts git
and rg from there instead of from `PATH`. It is per checkout and gitignored, because a path on one
machine says nothing about the next.
@@ -96,85 +71,202 @@ preflight. If a tool sits somewhere the preflight does not look, name it:
`pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --set rg=<path>`. Never edit the
file by hand. `doctor` reports under `tool-paths` whether every path still works.
## Allowed push URLs
## Tiered settings
`.wikitool-remotes.json` at the root restricts where `publish` may push, per checkout and
gitignored.
Two files at the root hold the settings of the features that read them this way, and every field
declares the tier it belongs to:
<!-- wikitool:reference-config-remotes -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `allowed_push_urls` | array of strings | yes | - | The push URLs `publish` may write to, compared with the URL the remote resolves to for a push - not with its name. An empty list allows no push at all. |
| Tier | File | Holds |
|------|------|-------|
| `instance` | `wikitool.json`, committed | What belongs to the corpus and is the same in every clone |
| `checkout` | `.wikitool-local.json`, gitignored | What is true only for this clone on this machine |
| `secret` | neither | Tokens, passwords and website sessions - both files hold only a reference, and a session from `raw login` none ([below](#fetch-routes)) |
- **Absent:** Any remote is allowed; the Publish-Remote Gate does not apply.
- **Malformed:** `publish` stops with an error before committing anything. A broken file is never read as "no restriction".
- **Unknown keys:** Ignored.
Each file is a JSON object of sections, one per feature, beside two keys of its own:
```json
{
"schema": 1,
"instance_id": "6f1c0e52-3d1b-4b8e-9a51-2f0e7d1c9b44",
"<section>": {"<field>": "<value>"}
}
```
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
[backup](#backup-to-a-synced-folder) and [export](#export-to-a-folder). `config set` names the sections it knows, and `doctor` reports a
section it does not read as a `WARN`.
**One place per value.** A value in the file its tier does not name is an error naming the file,
the key and the right tier - never an override. The same holds for a key set in both files.
**References.** A field of tier `secret` takes only a reference; a field marked private - a host
name, an internal URL - takes a plain value in `.wikitool-local.json` or a reference in either
file. No other field takes one.
| Reference | Resolves through |
|-----------|------------------|
| `{"ref": "<name>"}` - lower-case letters, digits and single `_` | `WIKITOOL_SECRET_<NAME>`, else the key `<name>` in the secrets file |
| `{"ref": "op://<vault>/<item>/<field>"}` | `op read`, the 1Password CLI the preflight recorded |
The secrets file is `~/.config/chemenu/<instance_id>/secrets.json` (`$XDG_CONFIG_HOME` instead of
`~/.config` when it is set; `%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows), a JSON object of
names to strings. Outside Windows it must be readable by its owner only (`chmod 600`); a file
others can read is refused.
A reference is resolved when a command asks for that value, never when the files are read. A name
nobody provides, an `op` that is not installed, or an item 1Password does not have ends the command
with exit 1; a 1Password that is locked, signed out or waiting for an approval ends it with exit 42,
for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](How-to-Use-1password-for-settings).
**Never in the committed file:** a field that names a program to run or affects a gate - not even
as a reference. `config set` does not write such a field either (exit 42); the user edits
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
`names a command` in the tables below are these: the publish allowlist, the intake channels and the
external fetchers.
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
`settings-secrets` - and for the references only whether they resolve. `docs verify` holds a
committed `wikitool.json` to the placement rules above, a plain secret or private value above
all. Neither file ships with `dist export`. Why the tiers exist, and what keeping secrets out of
the tree does not protect against: [configuration-tiers.md](Explanation-Configuration-tiers).
## Telemetry
**Telemetry is off in an instance.** Every instance carries the `.wikitool-release.json` of its
release, and that is how `chemenu.telemetry.policy` recognises that nobody asked for telemetry. The
same holds for every further clone of the instance, because the file is committed with the first
publish. Only a clone of the origin repository, used to develop the stack, has no stamp. There the
traces are the instrument the stack measures itself with, and they are **on**.
To turn the default around, add a `telemetry` section to `.wikitool-local.json`, by hand or with
`wikitool config set telemetry.enabled true`. The consent to write prompts to this disk belongs to
the checkout, so the section has no instance-tier field and no gate.
<!-- wikitool:reference-config-telemetry -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `enabled` | checkout | boolean | no | the installation-form default | Telemetry on or off. `WIKI_TRACE` beats it in both directions. Any other value is read by its truthiness, so `"no"` means on. |
| `max_session_bytes` | checkout | positive integer | no | `5242880` | Byte cap per session trace. `WIKI_TRACE_MAX_SESSION_BYTES` beats it. A number written as a string is accepted. |
| `keep_sessions` | checkout | positive integer | no | `250` | How many session directories retention keeps. `WIKI_TRACE_KEEP_SESSIONS` beats it. A number written as a string is accepted. |
- **Absent:** The installation-form default applies: on in a clone of the origin repository, off in an instance.
- **Malformed:** The hook that reads it never fails on it: a value it cannot use counts as absent, and an unreadable `.wikitool-local.json` as an absent section. `doctor` reports each such deviation as a `WARN` under `telemetry`, naming the field.
Example:
```json
{
"schema": 1,
"allowed_push_urls": [
"<url>"
]
"telemetry": {
"enabled": true,
"max_session_bytes": 5242880,
"keep_sessions": 250
}
}
```
<!-- /wikitool:reference-config-telemetry -->
`wikitool doctor` reports the current state: on or off, why, and how much has been recorded against
both caps. What the caps do and what never reaches a trace:
[telemetry-and-evaluation.md](Explanation-Telemetry-and-evaluation).
The telemetry hook runs on every tool call and stays silent, so it reads only this section, leniently,
and never exits 42: a `.wikitool-telemetry.json` left over from an earlier version is ignored there
([below](#files-of-earlier-versions)).
## Allowed push URLs
The `remotes` section of `.wikitool-local.json` restricts where `publish` may push.
<!-- wikitool:reference-config-remotes -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `allowed_push_urls` | checkout, gate | array of strings | yes | - | The push URLs `publish` may write to, compared with the URL the remote resolves to for a push - not with its name. An empty list allows no push at all. |
- **Absent:** Any remote is allowed; the Publish-Remote Gate does not apply.
- **Malformed:** `publish` stops with an error before committing anything. A broken section is never read as "no restriction".
Example:
```json
{
"remotes": {
"allowed_push_urls": [
"<url>"
]
}
}
```
<!-- /wikitool:reference-config-remotes -->
With the file, `publish` refuses with exit 42 (the Publish-Remote Gate) before pushing to a URL not
on the list. Only you edit this file. The gate has no token and no flag, and an agent that edits the
list to get past a refusal is opening a gate on its own initiative.
With the section, `publish` refuses with exit 42 (the Publish-Remote Gate) before pushing to a URL not
on the list. Only you edit it: the field is marked `gate`, so `config set` exits 42 for it. The gate
has no token and no flag, and an agent that edits the list to get past a refusal is opening a gate on
its own initiative.
## Task tracker
**Optional.** The weekly review (`tools/wikitool review`, skill `gtd-weekly-review`) checks the
project pages under `kb/gtd/` against a task tracker. Which tracker that is goes in
`.wikitool-tasks.json` at the root. Like the telemetry and remotes files, it is per checkout and has
no `.template`, and it is **gitignored as soon as a token is in it**.
project pages under `kb/gtd/` against a task tracker. Which tracker that is, and its connection
details, are three sections: `tasks` (which provider, the review's thresholds), and the section of the
provider chosen, `superproductivity` or `caldav`. What is the same in every clone - the provider, the
thresholds, the list names - sits in `wikitool.json`; the host, the user name and the folder this
machine uses in `.wikitool-local.json`; a token or password is a reference to a secret, never a value.
<!-- wikitool:reference-config-tasks -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `provider` | string | yes | - | Which adapter the review goes through. The provider's own settings sit in a section of the same name. |
| `thresholds` | object | yes | - | The weekly review's three thresholds. |
| `thresholds.stalled_waiting_days` | positive integer | yes | - | A waiting-for item older than this many days counts as overdue. A number written as a string is accepted. |
| `thresholds.unpaged_project_weeks` | positive integer | yes | - | A tracker project with no `kb/` page is reported once it is older than this many weeks. A number written as a string is accepted. |
| `thresholds.someday_stale_months` | positive integer | yes | - | A someday/maybe item untouched for this many months counts as stale. A number written as a string is accepted. |
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `provider` | instance | string | yes | - | Which adapter the review goes through. |
| `stalled_waiting_days` | instance | positive integer | yes | - | A waiting-for item older than this many days counts as overdue. A number written as a string is accepted. |
| `unpaged_project_weeks` | instance | positive integer | yes | - | A tracker project with no `kb/` page is reported once it is older than this many weeks. A number written as a string is accepted. |
| `someday_stale_months` | instance | positive integer | yes | - | A someday/maybe item untouched for this many months counts as stale. A number written as a string is accepted. |
- **Absent:** No tracker is configured - a valid state. The weekly review has nothing to check against.
- **Malformed:** An error, and `doctor` reports `FAIL`. A broken file is never read as "no tracker configured".
- **Unknown keys:** Ignored, so the section of a provider not chosen right now can stay in the file.
- **Malformed:** An error, and `doctor` reports `FAIL`. A broken section is never read as "no tracker configured".
**The `superproductivity` section.** Connection settings for the Super Productivity adapter.
Example:
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `access` | string | yes | - | Which way in: `api`, the local REST API (reads the current state and carries the write path), or `snapshot`, the newest backup snapshot (read-only). |
```json
{
"tasks": {
"provider": "superproductivity",
"stalled_waiting_days": 14,
"unpaged_project_weeks": 3,
"someday_stale_months": 5
}
}
```
<!-- /wikitool:reference-config-tasks -->
With `access: "api"`:
The thresholds are configuration, not schema. A value the provider cannot supply at all - a
waiting item with no follow-up date, a tracker project with no creation date - is not skipped silently
but reported as its own finding (`waiting_no_follow_up`/`project_age_unknown`).
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `api_base_url` | string | no | `"http://127.0.0.1:3876"` | Where the app's REST API listens. An empty value means the default. |
| `api_token` | string | yes, not empty | - | The token from the app's settings. |
### Super Productivity
With `access: "snapshot"`:
<!-- wikitool:reference-config-superproductivity -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `access` | instance | string | yes | - | Which way in: `api`, the local REST API (reads the current state and carries the write path), or `snapshot`, the newest backup snapshot (read-only). |
| `api_base_url` | checkout, private | string | no | `"http://127.0.0.1:3876"` | With `access: "api"`: where the app's REST API listens. An empty value means the default. |
| `api_token` | secret | string | no | - | With `access: "api"`, required: the token from the app's settings. Only ever a reference. |
| `backups_dir` | checkout | string | no | - | With `access: "snapshot"`, required: the directory the app writes its backup snapshots to; `~` is expanded. |
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `backups_dir` | string | yes, not empty | - | The directory the app writes its backup snapshots to; `~` is expanded. |
- **Absent:** Read only when `tasks.provider` is `superproductivity`.
- **Malformed:** An error when it is read: a field of the other access path beside `access`, or a missing `api_token` or `backups_dir`, stops the command with exit 1.
- **Unknown keys:** Rejected: a section names only one access path's own fields, so a `backups_dir` beside `access: "api"` is an error, not ignored.
Examples:
```json
{
"superproductivity": {
"access": "api",
"api_base_url": "http://127.0.0.1:3876",
"api_token": "..."
"api_token": {
"ref": "sp_token"
}
}
}
```
@@ -187,62 +279,9 @@ With `access: "snapshot"`:
}
}
```
<!-- /wikitool:reference-config-superproductivity -->
**The `caldav` section.** Connection settings for the CalDAV adapter.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `url` | string | yes, not empty | - | The account's CalDAV calendar home set. A trailing `/` is added when missing. |
| `username` | string | yes, not empty | - | The login name, which can differ from the user id in the URL. |
| `app_password` | string | yes, not empty | - | An app password is recommended over the account password. |
| `inbox_list` | string | yes, not empty | - | The name of the one inbox list. |
| `someday_list` | string | yes, not empty | - | The name of the one someday/maybe list. |
| `exclude_lists` | array of strings | no | `[]` | Existing plain task lists that are not projects. |
- **Unknown keys:** Rejected.
```json
{
"caldav": {
"url": "https://<host>/remote.php/dav/calendars/<user>/",
"username": "<login>",
"app_password": "...",
"inbox_list": "Inbox",
"someday_list": "Someday",
"exclude_lists": [
"..."
]
}
}
```
A complete example:
```json
{
"schema": 1,
"provider": "superproductivity",
"thresholds": {
"stalled_waiting_days": 14,
"unpaged_project_weeks": 3,
"someday_stale_months": 5
},
"superproductivity": {
"access": "api",
"api_base_url": "http://127.0.0.1:3876",
"api_token": "..."
}
}
```
<!-- /wikitool:reference-config-tasks -->
The `thresholds` block is configuration, not schema. A value the provider cannot supply at all - a
waiting item with no follow-up date, a tracker project with no creation date - is not skipped silently
but reported as its own finding (`waiting_no_follow_up`/`project_age_unknown`).
### Super Productivity
`access` is **mandatory and has no fallback**. It chooses between two mutually exclusive ways in:
`access` is **mandatory and has no fallback**. The fields of the other access path are an error, not ignored: a `backups_dir` beside `access: "api"` stops the command with exit 1, as does an `api_token` missing from it. It chooses between two mutually exclusive ways in:
- **`access: "snapshot"`** for an instance run headless: it reads only the newest backup snapshot
under `backups_dir`. That works without the app running, but only reads. The tracker cannot be
@@ -250,7 +289,7 @@ but reported as its own finding (`waiting_no_follow_up`/`project_age_unknown`).
- **`access: "api"`** for a desktop instance: it talks only to the local REST API. That answers only
while the app runs, but it gives the current state and carries the write path. `api_token` is
required, because every endpoint except `GET /health` demands `Authorization: Bearer <token>`.
required (as a reference), because every endpoint except `GET /health` demands `Authorization: Bearer <token>`.
What can write:
@@ -265,6 +304,40 @@ from there. `tools/wikitool task list --project` only reads, so it answers with
### CalDAV
<!-- wikitool:reference-config-caldav -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `url` | checkout, private | string | yes, not empty | - | The account's CalDAV calendar home set. A trailing `/` is added when missing. |
| `username` | checkout, private | string | yes, not empty | - | The login name, which can differ from the user id in the URL. |
| `app_password` | secret | string | yes, not empty | - | An app password is recommended over the account password. Only ever a reference. |
| `inbox_list` | instance | string | yes, not empty | - | The name of the one inbox list. |
| `someday_list` | instance | string | yes, not empty | - | The name of the one someday/maybe list. |
| `exclude_lists` | instance | array of strings | no | `[]` | Existing plain task lists that are not projects. |
- **Absent:** Read only when `tasks.provider` is `caldav`.
Example:
```json
{
"caldav": {
"url": {
"ref": "op://Chemenu/caldav/url"
},
"username": "<login>",
"app_password": {
"ref": "caldav_app_password"
},
"inbox_list": "Inbox",
"someday_list": "Someday",
"exclude_lists": [
"..."
]
}
}
```
<!-- /wikitool:reference-config-caldav -->
`caldav` is the standards-based second adapter (RFC 4791/5545). It is verified against Nextcloud
Tasks, with iOS *Reminders* as the mobile client, and built for that split: tick items off on the
phone, maintain them at the desk. Unlike Super Productivity, there is only one way in. CalDAV is
@@ -291,44 +364,373 @@ since it was read (an ETag conflict) is not written, and the command stops with
ever deleted. The lists of finished projects stay after archiving, because the adapter never deletes
a list. Remove them by hand in Nextcloud when you want to.
## MCP submissions
### Profiles
`.wikitool-upload.json` in the served corpus turns on the MCP read server's optional `submit` tool.
What the authentication in front of the server has to guarantee before you create it is in
[Set up the MCP read server](How-to-Set-up-the-mcp-server#optional-the-submit-path).
`WIKITOOL_TASKS_CONFIG` points at a profile: a file in the format of `.wikitool-local.json` that holds
only the sections `tasks`, `superproductivity` and `caldav` (and `schema`, and `live_test_version`
for the live suite). While it is set, `task`, `review` and `doctor` read those three sections from the
profile instead of from `wikitool.json` and `.wikitool-local.json`, so one checkout can be run against
several trackers in turn. A profile is checked like the local file, except that a plain instance-tier
value is allowed in it; a secret is a reference there too, never a value. The live suite takes its
throwaway tokens from `WIKITOOL_SECRET_<NAME>`.
<!-- wikitool:reference-config-upload -->
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `schema` | any | no | - | Accepted and not read. Conventionally `1`; no version of this file is told apart by it yet. |
| `identity_header` | string | no | `"X-Forwarded-User"` | The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default. |
| `max_bytes` | positive integer | yes | - | Largest single submission, in bytes. A number written as a string is accepted. |
| `allowed_extensions` | array of strings | yes | - | The file extensions `submit` accepts, each starting with `.`. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse. |
| `quota` | object | yes | - | Per-submitter limits over any rolling 24 hours. |
| `quota.submissions_per_day` | positive integer | yes | - | Submissions one submitter may make. A number written as a string is accepted. |
| `quota.bytes_per_day` | positive integer | yes | - | Bytes one submitter may submit. A number written as a string is accepted. |
## Intake channels
- **Absent:** The `submit` tool is not registered: the write path does not exist.
- **Malformed:** An error. The MCP server does not start and `doctor` reports `FAIL`. A broken file is never read as "no limits".
- **Unknown keys:** Ignored.
The `intake` section of `.wikitool-local.json` declares the ways a document gets into the
quarantine `quarantine/` from somewhere other than this terminal. Each channel has a name and a
handler:
| Handler | Source | How it arrives |
|---|---|---|
| `mcp` | The MCP read server's optional `submit` tool | Pushed by a remote caller. At most one channel has this handler. |
| `folder` | A directory outside the instance that a sync client fills (OneDrive and the like) | Pulled by `wikitool upload collect`; what is taken is removed from the folder. |
| `command` | A program of this checkout, named in `commands` | Pulled by `wikitool upload collect`; the program leaves its files in an empty staging directory. |
All of them end in the same place: a submission nobody has looked at, which `wikitool upload accept`
promotes into `incoming/` after a human has cleared the Upload Review Gate. What the authentication
in front of the server has to guarantee for the `mcp` handler is in
[Set up the MCP read server](How-to-Set-up-the-mcp-server#optional-the-submit-path); how to
collect from a synced folder or a program is in
[Collect documents from a synced folder](How-to-Collect-from-a-synced-folder). Every field is
marked `gate` or `names a command`: the channels are the user's to set by hand, and `config set`
exits 42 for them.
<!-- wikitool:reference-config-intake -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
- **Malformed:** An error. The MCP server does not start, `upload collect` stops with exit 1 and `doctor` reports `FAIL`. A broken section is never read as "no channel".
Example:
```json
{
"schema": 1,
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [
".md",
".pdf"
],
"quota": {
"submissions_per_day": 20,
"bytes_per_day": 52428800
"intake": {
"channels": {
"mcp": {
"handler": "mcp",
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [
".md",
".pdf"
],
"quota_submissions_per_day": 20,
"quota_bytes_per_day": 52428800
},
"onedrive": {
"handler": "folder",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Eingang",
"max_bytes": 52428800,
"allowed_extensions": [
".pdf",
".docx",
".msg",
".eml",
".md"
],
"settle_seconds": 60
},
"scanner": {
"handler": "command",
"command": "scanner-pull",
"max_bytes": 52428800,
"allowed_extensions": [
".pdf"
]
}
},
"commands": {
"scanner-pull": [
"pwsh",
"-NoProfile",
"-File",
"C:\\tools\\scanner-pull.ps1"
]
}
}
}
```
<!-- /wikitool:reference-config-upload -->
<!-- /wikitool:reference-config-intake -->
Size and quota are your deployment's decision, not defaults worth copying blindly.
Size and quota are your deployment's decision, not defaults worth copying blindly. A field that does
not belong to the channel's handler is an error, not ignored: `identity_header` and the `quota_*`
fields belong to `mcp`, `path` and `settle_seconds` to `folder`, `command` to `command`. A `folder`
`path` must be absolute and lie outside the instance, and `upload collect` reads only its top level.
## Site handlers
`tools/wikitool raw fetch` knows how some websites serve their pages. For a URL of such a website a
*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
derived text and prints hints about the page as `CHECK` lines; today there are two, both for
`learn.microsoft.com`: `learn-microsoft` for every page of the host and
`learn-microsoft-powershell-module` for a PowerShell module's overview page, which offers the cmdlet
pages it lists as subpages for `raw fetch --scope`. Handlers are JSON files: the stack's, and the instance's own under `sites/`
(see [How to write a site handler](How-to-Write-a-site-handler)).
The `sites` section is what the instance decides about the handlers, and it is optional: without it every
handler applies, in the locale `en-us`.
- `disabled` switches handlers off by name. A page of that website is then handled as any other:
fetched under the URL as typed, with its page furniture in the derived text. A name no handler has
is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the
general one of the same website in force.
- `locale` is the language segment a handler for a multi-language website puts into the URL it
fetches. The header of the derived text keeps the URL as typed under `requested_url`.
- `notes` adds hint lines of the instance's own, printed after a fetch or an `--html` derivation of
a page of that host, whether or not a handler exists for it. An entry names a `host` (exact, lower
case), optionally a `path` (a regular expression matched against the start of the URL path, after
the standard form was applied) and one to three single-line `lines`. A host and path pattern appears
once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
output as it is, so it is for a reminder to the reader of the capture, never for an instruction
the tool should carry out.
- `max_pages` and `pause_seconds` bound `raw fetch --scope`: the most subpages one page may offer
before the fetch is refused, and the pause before each subpage request, which keeps the load on
the website low.
<!-- wikitool:reference-config-sites -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
| `max_pages` | instance | positive integer | no | `200` | The most subpages `raw fetch --scope` fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted. |
| `pause_seconds` | instance | number | no | `1` | The pause, in seconds, before each subpage request of `raw fetch --scope`. `0` means none. A number written as a string is accepted. |
- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
Example:
```json
{
"sites": {
"disabled": [
"learn-microsoft-powershell-module"
],
"locale": "en-us",
"max_pages": 200,
"pause_seconds": 1,
"notes": [
{
"host": "learn.microsoft.com",
"path": "^/[a-z-]+/azure/",
"lines": [
"Check the Azure retirement notice at the top of the page."
]
}
]
}
}
```
<!-- /wikitool:reference-config-sites -->
All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none
of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
with exit 1 before it fetches anything.
## Fetch routes
`tools/wikitool raw fetch` fetches every page the same way unless the `fetch` section says
otherwise for its host: anonymously, with no cookie and no browser. `routes` maps a host to another
way, and is instance-tier, so every clone fetches the host the same way:
- `"session"` fetches the host's pages with the session the user stored with
`tools/wikitool raw login <host>`, through the same built-in fetch. The host needs a site handler
with a `session` block - its login page and how a page shows it arrived signed in
([sites/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/sites/CONTRACT.md)). No stored session, or a page without that sign,
stops the fetch with exit 42 for the user to sign in again. The session is a secret: it is
stored in the secrets file [above](#tiered-settings) under `session_<host>` (every run of other
characters as one `_`, so `www.example.org` is `session_www_example_org`), or comes from
`WIKITOOL_SECRET_SESSION_<HOST>`, and no settings field refers to it. It is never written to
1Password.
- Any other value names a fetcher of `fetchers`: a program of this checkout, run without a shell
with the URL appended. `fetchers` names a program to run, so it lives only in
`.wikitool-local.json`, written by hand - `config set` refuses it with exit 42, and a fetcher in
`wikitool.json` is an error. A route to a fetcher this checkout does not define stops a fetch of
that host with exit 1, so a clone without the program fails loudly instead of fetching
anonymously.
The whole procedure, with the browser component it needs for signing in:
[fetch-pages-behind-a-login.md](How-to-Fetch-pages-behind-a-login).
<!-- wikitool:reference-config-fetch -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `routes` | instance | object | no | `{}` | Host (exact, lower case) to route: `"session"` fetches with the user's session - the host needs a site handler that names its login page - and any other value names a fetcher of `fetchers`. A host not listed is fetched as before. |
| `fetchers` | checkout, names a command | object | no | `{}` | Fetcher name to command: an array of the program and its arguments, run without a shell with the URL appended. Exit 0 and the page's HTML on stdout is a fetch; anything else stops `raw fetch` with exit 1 and the program's stderr. |
- **Absent:** Every page is fetched by the built-in anonymous fetch, as before.
- **Malformed:** An error when `raw fetch` or `raw login` reads it: the command stops with exit 1 before fetching anything. A route to a fetcher this checkout does not define stops a fetch of that host the same way.
Example:
```json
{
"fetch": {
"routes": {
"www.heise.de": "session",
"intranet.example.org": "homelab"
},
"fetchers": {
"homelab": [
"/usr/local/bin/fetch-page",
"--html"
]
}
}
}
```
<!-- /wikitool:reference-config-fetch -->
## Backup to a synced folder
The `backup` section of `.wikitool-local.json` backs this checkout up into a folder a sync client
uploads - OneDrive and the like - while the working tree itself stays outside it. The target gets
two kinds of content:
```
<target>/git/ bare repository, the checkout's origin; publish pushes here
<target>/<machine>/ mirror of this machine's files that are not in git, plus .chemenu-backup.json
```
`tools/wikitool backup init` creates `<target>/git` and makes it `origin` when the checkout has none.
With `machine_data` on, every `publish` that pushed mirrors `.wikitool-local.json`, `ENVIRONMENT.md`,
what waits in `incoming/` and the `extra` paths into `<target>/<machine>/`; `tools/wikitool backup
run` does the same on its own. The secrets file is never among them - it lies outside the instance,
and a backup reads nothing outside it. Plain private values in `.wikitool-local.json`, an internal
host name for one, do reach the synced folder.
<!-- wikitool:reference-config-backup -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `target` | checkout, gate | string | yes | - | The backup folder: an absolute path outside the instance. `backup init` creates `<target>/git` in it, `backup run` writes only to `<target>/<machine>/`. |
| `machine_data` | checkout, gate | boolean | no | `false` | Mirror this machine's files after every `publish` that pushed, and with `backup run`: `.wikitool-local.json`, `ENVIRONMENT.md`, what waits in `incoming/`, and `extra`. |
| `machine_name` | checkout | string | no | the host name | The name of this machine's folder under the target. Every character outside letters, digits, `-` and `_` becomes `-`, and leading and trailing `-` are dropped; a name that is empty afterwards, or `git` in any case, is an error. |
| `extra` | checkout, gate | array of strings | no | `[]` | Further files or directories to mirror, relative to the instance. One that resolves outside the instance, or into `.git/`, is an error. |
- **Absent:** No backup: `publish` behaves as without it, and `backup init` and `backup run` stop with exit 1 naming the section.
- **Malformed:** An error. `publish` stops with exit 1 before it commits anything, `backup init` and `backup run` stop with exit 1, and `doctor` reports `FAIL`. A broken section is never read as "no backup".
Example:
```json
{
"backup": {
"target": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Backup",
"machine_data": true,
"machine_name": "WORK-LAPTOP",
"extra": [
".claude/settings.local.json"
]
}
}
```
<!-- /wikitool:reference-config-backup -->
`target`, `machine_data` and `extra` decide what leaves this machine, so they are marked `gate`:
`config set` exits 42 for them, and only you write them. The procedure, restoring included:
[back-up-to-a-synced-folder.md](How-to-Back-up-to-a-synced-folder).
## Export to a folder
The `export` section of `.wikitool-local.json` declares folders that hold a derived, read-only
version of `kb/` - for a reader outside the repository, such as a Microsoft 365 tenant that indexes
a OneDrive folder. An export is regenerated from the committed pages on every run and is never
edited; what someone changes in the folder is refused or overwritten, never merged back. Each
target has a handler: `markdown` writes the selected pages as Markdown files, `command` renders
them into a staging directory and hands that to a converter of this checkout.
<!-- wikitool:reference-config-export -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target, an `intake` folder or `backup.target`) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
- **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
- **Malformed:** An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops with exit 1 before it commits anything, but only when some target sets `after_publish`. A broken section is never read as "no export".
Example:
```json
{
"export": {
"targets": {
"work-notes": {
"handler": "markdown",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
"include": [
[
"kind=system"
],
[
"tags=shared",
"!draft"
]
],
"frontmatter": "summary",
"links": "relative",
"after_publish": true
},
"as-word": {
"handler": "command",
"command": "pandoc-docx",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
"include": [
[
"kind=system"
]
]
}
},
"commands": {
"pandoc-docx": [
"pwsh",
"-NoProfile",
"-File",
"C:\\tools\\pandoc-docx.ps1"
]
}
}
}
```
<!-- /wikitool:reference-config-export -->
Within one target, `include` is a list of selections, each a list of `search` predicates (`tags=docs`,
`entity_type=system`, `!source_url`): a page is exported when it satisfies every predicate of at least
one selection. Only what `include` selects is written - there is no way to exclude by default. The
syntax is checked when the section is read, the field names against the corpus when `export run`
runs. `path` must not lie in or contain the instance, another target, an `intake` folder or
`backup.target`.
Every Markdown file starts with a frontmatter block whose first key is `wikitool_export`, carrying
the kind, the instance and the commit of the newest selected page. `frontmatter` decides what follows:
`minimal` nothing, `summary` the title, type, summary and modified date, `full` the page's whole
frontmatter. Wikilinks between exported pages become relative Markdown links (`links: relative`) or
plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote
markers and code stay as they are.
A target folder holds `.wikitool-export.json`, which lists the files a run wrote with their digests.
A run removes only listed files that are no longer exported, creates only folders it needs and
removes only those it created, and leaves every other file untouched. It stops before writing for a
listed file someone changed, for a folder that is not empty and has no manifest, and for a selection
that comes up empty. A run that finds nothing to change writes nothing, not even a modification time.
`targets` and `commands` decide what leaves this checkout and which program runs, so they are marked
`gate` and `names a command`: `config set` exits 42 for them, and only you write them. `raw accept`
also refuses a file whose sha256 is listed in the manifest of a configured target, for an export
that went through a converter and lost its marker. The procedure:
[export-to-a-synced-folder.md](How-to-Export-to-a-synced-folder).
## Files of earlier versions
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
`.wikitool-remotes.json`, `.wikitool-upload.json` and `.wikitool-tasks.json`. Nothing reads them any
more. A command that needs the setting such a file held stops with exit 42 and names
`tools/wikitool config migrate`, `doctor` reports each one as a `FAIL`, and only the telemetry hook
ignores it. `config migrate` moves the content into the sections above (the upload file becomes the channel `mcp` of `intake`), each value into the file its
tier names and each secret into the secrets file, and then removes the old files; it is run once per
checkout. `.wikitool-tools.json` is not among them and stays. The migration step for an instance is
[10.0.0-settings-tiers.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/instructions/migrations/10.0.0-settings-tiers.md).
Reference-Exit-codes.md
+14 -5
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/exit-codes.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/exit-codes.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/exit-codes.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/exit-codes.md) at v10.0.0 – edit the repository, not this wiki.*
# Exit codes and gates
@@ -6,13 +6,13 @@ Which `wikitool` commands can stop and wait for a person - exit 42 - and for whi
from the command records.
<!-- wikitool:reference-exit-codes -->
What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool error contract](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#tool-error-contract). What a gate is and why it is enforced in code is [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/AGENTS.md#gates). This page lists which commands declare a gate, and every cause for which a command exits 42.
What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool error contract](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#tool-error-contract). What a gate is and why it is enforced in code is [AGENTS.md § Gates](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/AGENTS.md#gates). This page lists which commands declare a gate, and every cause for which a command exits 42.
## Gates by command
### guideline-push
- [`export guidelines`](Reference-Commands-Guideline-export#export-guidelines)
- [`export guidelines`](Reference-Commands-Exports#export-guidelines)
### mass-update
@@ -34,10 +34,19 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
## Exit 42 causes
- [`new`](Reference-Commands-Pages#new): new project: The provider cannot create the project itself (Super Productivity's `access: "api"`); the output says what a human has to create
- [`task new`](Reference-Commands-Pages#task-new): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`task list`](Reference-Commands-Pages#task-list): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`task close`](Reference-Commands-Pages#task-close): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`review`](Reference-Commands-Finding-and-checking#review): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`raw fetch`](Reference-Commands-Raw-material-and-uploads#raw-fetch): The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
- [`raw fetch`](Reference-Commands-Raw-material-and-uploads#raw-fetch): The site handler declares `render` and the optional browser component is not installed
- [`raw login`](Reference-Commands-Raw-material-and-uploads#raw-login): The optional browser component is not installed
- [`upload accept`](Reference-Commands-Raw-material-and-uploads#upload-accept): Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
- [`sync`](Reference-Commands-Git#sync): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file, generated files aside; the output lists the upstream commits, the overlapping files and their diff
- [`publish`](Reference-Commands-Git#publish): The remotes settings file of an earlier version is still there
- [`publish`](Reference-Commands-Git#publish): Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
- [`publish`](Reference-Commands-Git#publish): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
- [`publish`](Reference-Commands-Git#publish): Publish-Remote Gate: `.wikitool-remotes.json` exists and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
- [`export guidelines`](Reference-Commands-Guideline-export#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
- [`publish`](Reference-Commands-Git#publish): Publish-Remote Gate: `remotes.allowed_push_urls` is set and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
- [`export guidelines`](Reference-Commands-Exports#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
- [`config set`](Reference-Commands-Configuration#config-set): The field names a program to run or affects a gate
<!-- /wikitool:reference-exit-codes -->
Reference-Page-types.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/page-types.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/page-types.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/page-types.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/page-types.md) at v10.0.0 – edit the repository, not this wiki.*
# Page types
Reference-Skills.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/skills.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/skills.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/skills.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/skills.md) at v10.0.0 – edit the repository, not this wiki.*
# Skill catalogue
Reference.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/reference/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/reference/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/reference/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/reference/README.md) at v10.0.0 – edit the repository, not this wiki.*
# Reference
Tutorials-First-source-to-first-answer.md
+2 -2
@@ -1,4 +1,4 @@
*Generated from [`docs/tutorials/first-source-to-first-answer.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/tutorials/first-source-to-first-answer.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/tutorials/first-source-to-first-answer.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/tutorials/first-source-to-first-answer.md) at v10.0.0 – edit the repository, not this wiki.*
# From your first source to your first answer
@@ -86,7 +86,7 @@ verbatim, reporting
```
The agent never guesses these two. Only you know them, and only now: once the file is in the wiki,
nobody can tell any more where it came from. [raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/raw/CONTRACT.md#capture-fields-fidelity-and-authority)
nobody can tell any more where it came from. [raw/CONTRACT.md](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/raw/CONTRACT.md#capture-fields-fidelity-and-authority)
lists every value, for the sources you add later.
## 5. Let the agent publish
Tutorials-Sample-source.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/tutorials/sample-source.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/tutorials/sample-source.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/tutorials/sample-source.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/tutorials/sample-source.md) at v10.0.0 – edit the repository, not this wiki.*
# Home lab backups - how they work now
Tutorials.md
+1 -1
@@ -1,4 +1,4 @@
*Generated from [`docs/tutorials/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v9.0.0/docs/tutorials/README.md) at v9.0.0 – edit the repository, not this wiki.*
*Generated from [`docs/tutorials/README.md`](https://gitea.nehmer.net/torben/chemenu/src/tag/v10.0.0/docs/tutorials/README.md) at v10.0.0 – edit the repository, not this wiki.*
# Tutorials
_Sidebar.md
+9 -1
@@ -4,9 +4,13 @@
* [From your first source to your first answer](Tutorials-First-source-to-first-answer)
* [Home lab backups - how they work now](Tutorials-Sample-source)
* [How-to guides](How-to)
* [Back up to a synced folder](How-to-Back-up-to-a-synced-folder)
* [Browse the wiki in Obsidian](How-to-Browse-with-obsidian)
* [Capture documentation from a git repository](How-to-Capture-repository-docs)
* [Collect documents from a synced folder](How-to-Collect-from-a-synced-folder)
* [Export guidelines to captured repositories](How-to-Export-guidelines)
* [Export to a synced folder](How-to-Export-to-a-synced-folder)
* [Fetch pages behind a login](How-to-Fetch-pages-behind-a-login)
* [Ingest a source](How-to-Ingest-a-source)
* [Maintain the wiki](How-to-Maintain-the-wiki)
* [Query the wiki](How-to-Query-the-wiki)
@@ -15,14 +19,17 @@
* [Trace and score agent sessions](How-to-Trace-and-score-sessions)
* [Troubleshoot an instance](How-to-Troubleshoot)
* [Update an instance](How-to-Update-an-instance)
* [Use 1Password for settings](How-to-Use-1password-for-settings)
* [Write a site handler](How-to-Write-a-site-handler)
* [Reference](Reference)
* [Command reference](Reference-Commands)
* [Catalog and log](Reference-Commands-Catalog-and-log)
* [Configuration](Reference-Commands-Configuration)
* [Content migrations](Reference-Commands-Content-migrations)
* [Distribution and versioning](Reference-Commands-Distribution-and-versioning)
* [Exports](Reference-Commands-Exports)
* [Finding and checking](Reference-Commands-Finding-and-checking)
* [Git](Reference-Commands-Git)
* [Guideline export](Reference-Commands-Guideline-export)
* [Instance health](Reference-Commands-Instance-health)
* [Links and citations](Reference-Commands-Links-and-citations)
* [Pages](Reference-Commands-Pages)
@@ -37,6 +44,7 @@
* [Skill catalogue](Reference-Skills)
* [Explanation](Explanation)
* [Architecture](Explanation-Architecture)
* [Configuration tiers](Explanation-Configuration-tiers)
* [Why knowledge and commitments are two layers](Explanation-Knowledge-and-commitment)
* [Language Boundaries](Explanation-Language-boundaries)
* [Choosing a Claude Code model and effort level](Explanation-Model-and-effort-selection)