intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11
CI / verify (push) Successful in 8m41s
CI / pwsh (push) Successful in 2m44s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

Files changed:
- .gitignore
- CHANGES.md
- VERSION
- docs/explanation/architecture.md
- docs/explanation/pipeline-rationale.md
- docs/how-to/README.md
- docs/how-to/back-up-to-a-synced-folder.md
- docs/how-to/collect-from-a-synced-folder.md
- docs/how-to/ingest-a-source.md
- docs/how-to/set-up-the-mcp-server.md
- docs/reference/commands/README.md
- docs/reference/commands/git.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- instructions/gates.md
- instructions/ingest-queue.md
- instructions/mcp-read-server.md
- instructions/migrations/10.0.0-settings-tiers.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli_contract.py
- tools/chemenu/collect.py
- tools/chemenu/commands/backup_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/pending.py
- tools/chemenu/commands/upload_cmd.py
- tools/chemenu/config.py
- tools/chemenu/config_shapes.py
- tools/chemenu/legacy_settings.py
- tools/chemenu/mcp/server.py
- tools/chemenu/reference_doc.py
- tools/chemenu/tests/test_backup.py
- tools/chemenu/tests/test_collect.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_legacy_settings.py
- tools/chemenu/tests/test_mcp_server.py
- tools/chemenu/tests/test_upload.py
- tools/chemenu/tests/test_upload_cmd.py
- tools/chemenu/upload.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-12 22:04:18 +02:00
1 parent 3627168403
commit cceb9a7bfe
44 files changed
+1910 -346

No files matched your search

+2 -1
View File
@@ -86,7 +86,8 @@ Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
- [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
- [`raw pending`](raw-material-and-uploads.md#raw-pending) - List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
- [`raw accept`](raw-material-and-uploads.md#raw-accept) - Promote one or more files, or one folder, from `incoming/` into `raw/`.
- [`upload list`](raw-material-and-uploads.md#upload-list) - List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
- [`upload collect`](raw-material-and-uploads.md#upload-collect) - Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
- [`upload list`](raw-material-and-uploads.md#upload-list) - List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
- [`upload show`](raw-material-and-uploads.md#upload-show) - Print one submission's manifest in full.
- [`upload accept`](raw-material-and-uploads.md#upload-accept) - **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
- [`upload reject`](raw-material-and-uploads.md#upload-reject) - Delete a submission's material, keeping only its ledger trail.
+1 -1
View File
@@ -237,7 +237,7 @@ tools/wikitool backup run
**Behaviour**
- Mirrors `.wikitool-local.json`, `ENVIRONMENT.md` if present, every file in `incoming/` but `.gitkeep`, and the `extra` paths of the `backup` section into `<target>/<machine>/`, then writes the manifest `.chemenu-backup.json` there: time, stack version, machine name, and per file its relative path, size and sha256.
- Never mirrors `mcp-upload/`, `reports/`, `.wikitool-tools.json`, `tools/.venv/`, the published skills or `.git/` - unless an `extra` path names one of the first ones. The secrets file lies outside the instance and is never read.
- Never mirrors `quarantine/`, `reports/`, `.wikitool-tools.json`, `tools/.venv/`, the published skills or `.git/` - unless an `extra` path names one of the first ones. The secrets file lies outside the instance and is never read.
- Writes and deletes only under `<target>/<machine>/`. Deletes only a file the previous manifest lists and the instance no longer has; a file there the manifest does not list stays as it is. A file whose size and sha256 match the previous manifest is not rewritten.
- A symbolic link or junction inside a mirrored directory is skipped and named in the output, never followed.
- `publish` runs the same step after every push when `machine_data` is true.
+1 -1
View File
@@ -38,7 +38,7 @@ tools/wikitool doctor --json
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
- MCP `submit` tool: whether the `upload` section of the settings is present, absent or malformed, its limits, and how many submissions wait in `mcp-upload/`. Absent is `OK` and means the write path does not exist at all; malformed is a `FAIL`.
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
- For a configured `caldav` provider, whether the server is reachable and Basic auth succeeds - never a `FAIL`; only a broken config block is.
@@ -12,6 +12,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
- [`raw status`](#raw-status)
- [`raw pending`](#raw-pending)
- [`raw accept`](#raw-accept)
- [`upload collect`](#upload-collect)
- [`upload list`](#upload-list)
- [`upload show`](#upload-show)
- [`upload accept`](#upload-accept)
@@ -365,7 +366,7 @@ tools/wikitool raw pending --json
**Behaviour**
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `quarantine/` is outside `incoming/` and never listed.
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
- The default is the first candidate `raw accept` would take, and the output names it.
@@ -495,9 +496,64 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
- `wikitool types describe source` - the capture field values
- `wikitool new source` - the source page for a promoted file
## `upload collect`
Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
**Synopsis**
- `wikitool upload collect [<channel>...]`
**Options**
```text
Arguments:
names... Channels to collect from; all pulled channels when none is named
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool upload collect
tools/wikitool upload collect onedrive
```
**Behaviour**
- Without arguments, collects from every channel of the `intake` section whose handler is `folder` or `command`; with names, from those channels only. A name no channel has, or the name of the `mcp` channel (which is pushed, not pulled), exits 1 before anything happens.
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
- Not idempotent: a second run finds the folder emptied.
**Properties**
- Changes files: yes
- Idempotent: no
- Atomic: No - per file: one copy into `quarantine/<id>/` (manifest last), then one removal of the source; a channel's failure does not stop the others
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - Unknown channel name, an `mcp` channel as argument, a malformed `intake` section, or an `mcp-upload/` that still holds submissions
- `1` - A channel or a file failed (folder missing, program exited non-zero, timed out or could not be started, a source that could not be removed)
**See also**
- `wikitool upload list` - what waits for review afterwards
- `wikitool upload accept` - the gate out of the quarantine
- `docs/how-to/collect-from-a-synced-folder.md`
## `upload list`
List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
**Synopsis**
@@ -519,8 +575,9 @@ tools/wikitool upload list --json
**Behaviour**
- Lists every submission waiting in `mcp-upload/`, oldest id first: id, filename, size, submitter.
- Only ever non-empty when the `upload` section of `.wikitool-local.json` opts the checkout into the MCP server's `submit` tool.
- Lists every submission waiting in `quarantine/`, oldest id first: id, filename, size, channel, submitter.
- Only ever non-empty when the `intake` section of `.wikitool-local.json` declares a channel.
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions - the message names the command that moves it.
- A submission directory with a corrupt manifest is silently skipped.
- Never fails; read-only and safe to retry freely.
@@ -568,7 +625,8 @@ tools/wikitool upload show <id>
**Behaviour**
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name, not a claim the header was honest), submission time.
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name or the handler, not a claim the header was honest), submission time, and where it came from: channel, handler, origin (the source file's path, `command:<name>` or `mcp:submit`).
- A submitter of `unknown` means the channel has no sender to trust - read channel and origin instead.
- What a reviewer reads before `upload accept`.
- Read-only.
@@ -618,7 +676,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
**Behaviour**
- Promotes a submission's file from `mcp-upload/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `mcp-upload/ledger.jsonl`.
- Promotes a submission's file from `quarantine/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `quarantine/ledger.jsonl`.
- Upload Review Gate: without a matching `--confirm`, exits 42 and prints the manifest in full plus the exact re-run line - one submission at a time. The gate check runs before anything is moved.
- The token digests id, filename, size, sha256 and submitter, so an edited or superseded manifest invalidates it.
- An occupied `incoming/<filename>` is an ordinary validation error, not the gate.
@@ -635,7 +693,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
**Exit codes**
- `0` - success
- `1` - Unknown or malformed submission id, or the submission's file is missing from `mcp-upload/<id>/`
- `1` - Unknown or malformed submission id, or the submission's file is missing from `quarantine/<id>/`
- `1` - `incoming/<filename>` already exists
- `42` - Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
+76 -29
View File
@@ -27,7 +27,7 @@ something you run.
- [Super Productivity](#super-productivity)
- [CalDAV](#caldav)
- [Profiles](#profiles)
- [MCP submissions](#mcp-submissions)
- [Intake channels](#intake-channels)
- [Site handlers](#site-handlers)
- [Fetch routes](#fetch-routes)
- [Backup to a synced folder](#backup-to-a-synced-folder)
@@ -92,7 +92,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
[upload](#mcp-submissions), [sites](#site-handlers), [fetch](#fetch-routes) and
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
[backup](#backup-to-a-synced-folder). `config set` names the sections it knows, and `doctor` reports a
section it does not read as a `WARN`.
@@ -121,7 +121,7 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
**Never in the committed file:** a field that names a program to run or affects a gate - not even
as a reference. `config set` does not write such a field either (exit 42); the user edits
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
`names a command` in the tables below are these: the publish allowlist, the upload limits and the
`names a command` in the tables below are these: the publish allowlist, the intake channels and the
external fetchers.
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
@@ -371,45 +371,92 @@ several trackers in turn. A profile is checked like the local file, except that
value is allowed in it; a secret is a reference there too, never a value. The live suite takes its
throwaway tokens from `WIKITOOL_SECRET_<NAME>`.
## MCP submissions
## Intake channels
The `upload` section of `.wikitool-local.json` in the served corpus turns on the MCP read server's
optional `submit` tool. What the authentication in front of the server has to guarantee before you
create it is in
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path). Every field
is marked `gate`: the limits are the user's to set by hand, and `config set` exits 42 for them.
The `intake` section of `.wikitool-local.json` declares the ways a document gets into the
quarantine `quarantine/` from somewhere other than this terminal. Each channel has a name and a
handler:
<!-- wikitool:reference-config-upload -->
| Handler | Source | How it arrives |
|---|---|---|
| `mcp` | The MCP read server's optional `submit` tool | Pushed by a remote caller. At most one channel has this handler. |
| `folder` | A directory outside the instance that a sync client fills (OneDrive and the like) | Pulled by `wikitool upload collect`; what is taken is removed from the folder. |
| `command` | A program of this checkout, named in `commands` | Pulled by `wikitool upload collect`; the program leaves its files in an empty staging directory. |
All of them end in the same place: a submission nobody has looked at, which `wikitool upload accept`
promotes into `incoming/` after a human has cleared the Upload Review Gate. What the authentication
in front of the server has to guarantee for the `mcp` handler is in
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path); how to
collect from a synced folder or a program is in
[Collect documents from a synced folder](../how-to/collect-from-a-synced-folder.md). Every field is
marked `gate` or `names a command`: the channels are the user's to set by hand, and `config set`
exits 42 for them.
<!-- wikitool:reference-config-intake -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `identity_header` | checkout, gate | string | no | `"X-Forwarded-User"` | The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default. |
| `max_bytes` | checkout, gate | positive integer | yes | - | Largest single submission, in bytes. A number written as a string is accepted. |
| `allowed_extensions` | checkout, gate | array of strings | yes | - | The file extensions `submit` accepts, each starting with `.`. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse. |
| `quota_submissions_per_day` | checkout, gate | positive integer | yes | - | Submissions one submitter may make over any rolling 24 hours. A number written as a string is accepted. |
| `quota_bytes_per_day` | checkout, gate | positive integer | yes | - | Bytes one submitter may submit over any rolling 24 hours. A number written as a string is accepted. |
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
- **Absent:** The `submit` tool is not registered: the write path does not exist.
- **Malformed:** An error. The MCP server does not start and `doctor` reports `FAIL`. A broken section is never read as "no limits".
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
- **Malformed:** An error. The MCP server does not start, `upload collect` stops with exit 1 and `doctor` reports `FAIL`. A broken section is never read as "no channel".
Example:
```json
{
"upload": {
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [
".md",
".pdf"
],
"quota_submissions_per_day": 20,
"quota_bytes_per_day": 52428800
"intake": {
"channels": {
"mcp": {
"handler": "mcp",
"identity_header": "X-Forwarded-User",
"max_bytes": 10485760,
"allowed_extensions": [
".md",
".pdf"
],
"quota_submissions_per_day": 20,
"quota_bytes_per_day": 52428800
},
"onedrive": {
"handler": "folder",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Eingang",
"max_bytes": 52428800,
"allowed_extensions": [
".pdf",
".docx",
".msg",
".eml",
".md"
],
"settle_seconds": 60
},
"scanner": {
"handler": "command",
"command": "scanner-pull",
"max_bytes": 52428800,
"allowed_extensions": [
".pdf"
]
}
},
"commands": {
"scanner-pull": [
"pwsh",
"-NoProfile",
"-File",
"C:\\tools\\scanner-pull.ps1"
]
}
}
}
```
<!-- /wikitool:reference-config-upload -->
<!-- /wikitool:reference-config-intake -->
Size and quota are your deployment's decision, not defaults worth copying blindly.
Size and quota are your deployment's decision, not defaults worth copying blindly. A field that does
not belong to the channel's handler is an error, not ignored: `identity_header` and the `quota_*`
fields belong to `mcp`, `path` and `settle_seconds` to `folder`, `command` to `command`. A `folder`
`path` must be absolute and lie outside the instance, and `upload collect` reads only its top level.
## Site handlers
@@ -590,7 +637,7 @@ Until version 10 every feature above had a file of its own at the root: `.wikito
`.wikitool-remotes.json`, `.wikitool-upload.json` and `.wikitool-tasks.json`. Nothing reads them any
more. A command that needs the setting such a file held stops with exit 42 and names
`tools/wikitool config migrate`, `doctor` reports each one as a `FAIL`, and only the telemetry hook
ignores it. `config migrate` moves the content into the sections above, each value into the file its
ignores it. `config migrate` moves the content into the sections above (the upload file becomes the channel `mcp` of `intake`), each value into the file its
tier names and each secret into the secrets file, and then removes the old files; it is run once per
checkout. `.wikitool-tools.json` is not among them and stays. The migration step for an instance is
[10.0.0-settings-tiers.md](../../instructions/migrations/10.0.0-settings-tiers.md).