intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11
Files changed: - .gitignore - CHANGES.md - VERSION - docs/explanation/architecture.md - docs/explanation/pipeline-rationale.md - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/ingest-a-source.md - docs/how-to/set-up-the-mcp-server.md - docs/reference/commands/README.md - docs/reference/commands/git.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - instructions/gates.md - instructions/ingest-queue.md - instructions/mcp-read-server.md - instructions/migrations/10.0.0-settings-tiers.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/collect.py - tools/chemenu/commands/backup_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config.py - tools/chemenu/config_shapes.py - tools/chemenu/legacy_settings.py - tools/chemenu/mcp/server.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_backup.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_legacy_settings.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_upload.py - tools/chemenu/tests/test_upload_cmd.py - tools/chemenu/upload.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed
1 parent
3627168403
commit
cceb9a7bfe
44 files changed
+1910
-346
No files matched your search
@@ -81,9 +81,10 @@ chemenu/
|
||||
│ ├── ingest-large-tree.md
|
||||
│ ├── ingest-queue.md # Reviewing a submission before wikitool upload accept promotes it
|
||||
│ └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/
|
||||
├── mcp-upload/ # QUARANTINE (optional): the MCP `submit` tool's write path, gitignored -
|
||||
│ # read by no command in the ordinary pipeline; a human reviews it with
|
||||
│ # `wikitool upload list/show/accept/reject`
|
||||
├── quarantine/ # QUARANTINE (optional): where the intake channels (MCP `submit`, a synced
|
||||
│ # folder, a command) put what nobody has reviewed, gitignored - read by
|
||||
│ # no command in the ordinary pipeline; a human reviews it with
|
||||
│ # `wikitool upload list/show/accept/reject`, `upload collect` fills it
|
||||
├── incoming/ # INBOX: content gitignored - drop a file or one folder per source here, `raw accept` promotes it
|
||||
├── raw/ # INPUT: immutable, untrusted source material
|
||||
│ ├── CONTRACT.md # Date shard, capture fields, immutability, untrusted content
|
||||
|
||||
@@ -19,9 +19,11 @@ not yet reviewed and not yet a citation target, so its being gitignored and read
|
||||
ingest session does not weaken anything - the boundary is the promotion into `raw/` itself, not
|
||||
the moment a human happened to drop a file somewhere.
|
||||
|
||||
Once a document can arrive from *outside* - the MCP server's optional `submit` tool - "a human
|
||||
happened to drop a file somewhere" stops describing how everything reaches `incoming/`, so the
|
||||
near side of the boundary gets a stage of its own rather than a second meaning. `mcp-upload/`
|
||||
Once a document can arrive from *outside* - the MCP server's optional `submit` tool, a folder a
|
||||
sync client fills, a program of the checkout - "a human happened to drop a file somewhere" stops
|
||||
describing how everything reaches `incoming/`, so the near side of the boundary gets a stage of its
|
||||
own rather than a second meaning. Every such channel ends in the same `quarantine/`, because what
|
||||
the boundary protects against is the lack of a reviewer, not the transport. `quarantine/`
|
||||
holds what nobody has looked at yet; `incoming/` holds what someone has. Two arrows, two
|
||||
different things being granted: `upload accept` grants *trust* (a human read the material and
|
||||
took responsibility for it), `raw accept` grants *immutability* (it becomes a citation target
|
||||
|
||||
@@ -30,6 +30,8 @@ Running the instance:
|
||||
- [Trace and score agent sessions](trace-and-score-sessions.md)
|
||||
- [Use 1Password for settings](use-1password-for-settings.md) - secrets and private values out
|
||||
of the tree, on the desktop and in CI
|
||||
- [Collect documents from a synced folder](collect-from-a-synced-folder.md) - a folder OneDrive and
|
||||
the like fill, or a program of your own, as an intake channel into the quarantine
|
||||
- [Back up to a synced folder](back-up-to-a-synced-folder.md) - OneDrive and the like as the
|
||||
backup of an instance without a git server, and restoring from it
|
||||
|
||||
|
||||
@@ -100,7 +100,7 @@ push.
|
||||
| `ENVIRONMENT.md` | yes, if present | written by hand |
|
||||
| Everything in `incoming/` but `.gitkeep` | yes | sources waiting there are backed up nowhere else |
|
||||
| The `extra` paths | yes | what you named |
|
||||
| `mcp-upload/` | no | material submitted from outside that nobody has reviewed yet |
|
||||
| `quarantine/` | no | material that arrived through an intake channel and that nobody has reviewed yet |
|
||||
| `reports/` | no | regenerated by `lint`; telemetry holds prompts |
|
||||
| `.wikitool-tools.json`, `tools/.venv/`, the published skills | no | the preflight and `instructions sync` rebuild them |
|
||||
| The secrets file | **never** | it lies outside the instance, and a backup reads nothing outside it |
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Collect documents from a synced folder
|
||||
|
||||
Documents that are not typed or fetched at this terminal can reach the wiki through an intake
|
||||
channel. Besides the MCP `submit` tool there are two channels that this checkout *pulls* from: a
|
||||
folder that a sync client such as OneDrive fills (handler `folder`), and a program of your own
|
||||
(handler `command`). Both end in the quarantine `quarantine/`, exactly like a submission through the
|
||||
server, and both wait there for a human to run `wikitool upload accept` - the Upload Review Gate
|
||||
stands between every channel and `incoming/`.
|
||||
|
||||
This guide creates a `folder` channel, explains what `upload collect` leaves alone and why, and shows
|
||||
a `command` channel with an example script. The field reference is in the
|
||||
[configuration reference](../reference/configuration.md#intake-channels); the review is in
|
||||
[ingesting a source](ingest-a-source.md#a-document-submitted-from-outside).
|
||||
|
||||
<!-- wikitool:toc -->
|
||||
## Contents
|
||||
|
||||
- [Create the folder channel](#create-the-folder-channel)
|
||||
- [What stays in the folder, and why](#what-stays-in-the-folder-and-why)
|
||||
- [`settle_seconds` and what it cannot promise](#settle_seconds-and-what-it-cannot-promise)
|
||||
- [Deleting in a synced folder deletes everywhere](#deleting-in-a-synced-folder-deletes-everywhere)
|
||||
- [A command channel](#a-command-channel)
|
||||
- [Afterwards](#afterwards)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
## Create the folder channel
|
||||
|
||||
Channels are not written by `config set`: every field of the `intake` section opens a path into the
|
||||
checkout, so it is yours to edit by hand in `.wikitool-local.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"intake": {
|
||||
"channels": {
|
||||
"onedrive": {
|
||||
"handler": "folder",
|
||||
"path": "/home/you/OneDrive/wiki-inbox",
|
||||
"settle_seconds": 60,
|
||||
"max_bytes": 10485760,
|
||||
"allowed_extensions": [".pdf", ".docx", ".md", ".txt"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`path` has to be absolute and lie outside the instance. Check it with `tools/wikitool doctor`, then
|
||||
pull:
|
||||
|
||||
```bash
|
||||
tools/wikitool upload collect # every pull channel
|
||||
tools/wikitool upload collect onedrive # only this one
|
||||
tools/wikitool upload list
|
||||
```
|
||||
|
||||
A collected file is stored unchanged in `quarantine/<id>/` with a manifest that names the channel,
|
||||
the handler and the original path. A folder knows nobody who dropped a file, so the submitter is
|
||||
recorded as `unknown`, with the handler as its source: do not trust the name or content of such a
|
||||
file any more than that of a stranger's.
|
||||
|
||||
## What stays in the folder, and why
|
||||
|
||||
`upload collect` reads the top level of the folder only, and it removes one kind of thing: a file it
|
||||
has just stored. Everything else stays where it is and is reported with its reason:
|
||||
|
||||
| Entry | Why it stays |
|
||||
|---|---|
|
||||
| A subdirectory | Nothing descends into it, and nothing deletes a directory. |
|
||||
| A name starting with `.` or `~$`, a hidden or system file | Editors and sync clients keep temporary files like these next to the real one. |
|
||||
| A symbolic link | Following it could leave the folder. |
|
||||
| An extension outside `allowed_extensions`, or a file above `max_bytes` | The same limits as for `submit`. |
|
||||
| A file younger than `settle_seconds` | It may still be arriving. |
|
||||
| A file that changed while it was being read or copied | The source is removed only if it still has the size and sha256 that were stored. |
|
||||
|
||||
A file whose sha256 already waits in the quarantine is reported as a duplicate, with the id of the
|
||||
existing submission, and is removed from the folder.
|
||||
|
||||
## `settle_seconds` and what it cannot promise
|
||||
|
||||
A sync client writes a file in pieces, so a file that is half there looks like a complete, smaller
|
||||
one. `settle_seconds` (default 60) skips files whose modification or creation time is younger than
|
||||
that, and the check that the source did not change during the copy catches the rest. Neither can see
|
||||
a client that pauses for longer than the interval in the middle of a file and then resumes: such a
|
||||
file is stored incomplete, and the review is where you notice it. Raise the value if your client
|
||||
works over a slow link.
|
||||
|
||||
## Deleting in a synced folder deletes everywhere
|
||||
|
||||
A file removed from a folder that OneDrive syncs is removed on every device that syncs it. That is
|
||||
what a collected file should do - it is now in the quarantine - but it also means a collect on one
|
||||
machine empties the folder for all of them. The client's recycle bin keeps the file for a while; the
|
||||
quarantine copy, which is never synced, is the one to rely on. Collect from one machine only.
|
||||
|
||||
## A command channel
|
||||
|
||||
A `command` channel runs a program of your own, which fetches documents from somewhere that has no
|
||||
synced folder - a mailbox, a share, a scanner. The program is named in `commands` and referenced from
|
||||
the channel; the section stays valid only if every reference resolves.
|
||||
|
||||
```json
|
||||
{
|
||||
"intake": {
|
||||
"commands": {
|
||||
"scan-mail": ["/home/you/bin/fetch-attachments.sh"]
|
||||
},
|
||||
"channels": {
|
||||
"mail": { "handler": "command", "command": "scan-mail" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The program is started without a shell and with stdin closed, with an empty staging directory under
|
||||
`quarantine/.staging/` appended as its last argument. It has 300 seconds. It leaves the files it
|
||||
wants to hand over directly in that directory:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
# fetch-attachments.sh <staging-dir>
|
||||
set -eu
|
||||
out="$1"
|
||||
rclone copy remote:wiki-inbox "$out" --max-age 1d
|
||||
```
|
||||
|
||||
A program that exits non-zero, does not start or times out fails its channel: nothing is taken, the
|
||||
staging directory is removed, and the last 2000 characters of its stderr are shown. Other channels
|
||||
still run, and `upload collect` exits 1 if any channel failed. With no pull channel configured it
|
||||
exits 0 with "nothing to collect".
|
||||
|
||||
## Afterwards
|
||||
|
||||
The agent does the same at the start of an ingest when you name no file: `upload collect`, then
|
||||
`upload list`, then a report of what waits. It never runs `upload accept` for you.
|
||||
@@ -93,11 +93,14 @@ Every ten sources, the agent suggests a lint run - see [Maintain the wiki](maint
|
||||
## A document submitted from outside
|
||||
|
||||
When the [MCP read server](set-up-the-mcp-server.md) has its optional `submit` tool turned on,
|
||||
documents can arrive from another machine. They land in the quarantine `mcp-upload/`, not in
|
||||
documents can arrive from another machine. A folder a sync client fills and a program of the
|
||||
checkout are two more ways in ([Collect documents from a synced
|
||||
folder](collect-from-a-synced-folder.md)). All of them land in the quarantine `quarantine/`, not in
|
||||
`incoming/`, and nothing else reads them there. Before an ingest can start, a human has to promote
|
||||
the submission:
|
||||
|
||||
```bash
|
||||
tools/wikitool upload collect # pull the folder and command channels into the quarantine
|
||||
tools/wikitool upload list # what is waiting
|
||||
tools/wikitool upload show <id> # one submission's manifest: who sent it, size, checksum
|
||||
```
|
||||
|
||||
@@ -160,18 +160,19 @@ reviewed yet.
|
||||
|
||||
## Optional: the submit path
|
||||
|
||||
Without this, `submit` does not exist as a tool - not unused, but not registered. An
|
||||
`upload` section in the served checkout's `.wikitool-local.json` turns it on. Its fields, defaults and what a malformed
|
||||
section does are in the [configuration reference](../reference/configuration.md#mcp-submissions); a
|
||||
broken section never means "no limits", it stops the server from starting.
|
||||
Without this, `submit` does not exist as a tool - not unused, but not registered. A
|
||||
channel with the handler `mcp` in the `intake` section of the served checkout's
|
||||
`.wikitool-local.json` turns it on. Its fields, defaults and what a malformed section does are in
|
||||
the [configuration reference](../reference/configuration.md#intake-channels); a broken section never
|
||||
means "no limits", it stops the server from starting.
|
||||
|
||||
**With `submit` on, the middleware has a second duty.** It has to set the identity header itself, on
|
||||
every authenticated request, and discard any copy the client sent. The server trusts that header as
|
||||
the value for `submitter`. A header the client could set itself would be a claim, not an identity.
|
||||
If `identity_header` names a header the middleware does not set, every submission is refused for
|
||||
If the channel's `identity_header` names a header the middleware does not set, every submission is refused for
|
||||
lack of an identity.
|
||||
|
||||
Submitted files land in `mcp-upload/<id>/`, which is gitignored and read by no other command. From
|
||||
Submitted files land in `quarantine/<id>/`, which is gitignored and read by no other command. From
|
||||
there a human promotes or rejects them, in the same checkout as the server rather than in its
|
||||
process. That review is part of
|
||||
[ingesting a source](ingest-a-source.md#a-document-submitted-from-outside).
|
||||
@@ -228,7 +229,7 @@ the client starts a different Python than the instance's. Give the client the ab
|
||||
|
||||
**`"commit": null` in every answer**: the served tree has uncommitted changes. Either the sync is not
|
||||
running, or something writes into the corpus that has no business there. The five read tools never
|
||||
write. `submit`, when it is on, writes only to `mcp-upload/`, which is gitignored and so is never a
|
||||
write. `submit`, when it is on, writes only to `quarantine/`, which is gitignored and so is never a
|
||||
cause of `"commit": null` either.
|
||||
|
||||
**`commit` names an old revision**: the sync timer is not running.
|
||||
@@ -246,6 +247,6 @@ Deployment specifics - the cluster, ingress hosts, the secret store, the GitOps
|
||||
private infrastructure, and this repository is public.
|
||||
|
||||
There is no container image for the server yet. Until there is, the path above - the venv,
|
||||
`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `mcp-upload/`
|
||||
`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `quarantine/`
|
||||
has to survive restarts the same way the rest of the checkout does. Otherwise a submitted file that
|
||||
nobody has reviewed yet loses its quarantine.
|
||||
@@ -86,7 +86,8 @@ Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
|
||||
- [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
|
||||
- [`raw pending`](raw-material-and-uploads.md#raw-pending) - List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
- [`raw accept`](raw-material-and-uploads.md#raw-accept) - Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- [`upload list`](raw-material-and-uploads.md#upload-list) - List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
- [`upload collect`](raw-material-and-uploads.md#upload-collect) - Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
|
||||
- [`upload list`](raw-material-and-uploads.md#upload-list) - List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
|
||||
- [`upload show`](raw-material-and-uploads.md#upload-show) - Print one submission's manifest in full.
|
||||
- [`upload accept`](raw-material-and-uploads.md#upload-accept) - **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
|
||||
- [`upload reject`](raw-material-and-uploads.md#upload-reject) - Delete a submission's material, keeping only its ledger trail.
|
||||
|
||||
@@ -237,7 +237,7 @@ tools/wikitool backup run
|
||||
**Behaviour**
|
||||
|
||||
- Mirrors `.wikitool-local.json`, `ENVIRONMENT.md` if present, every file in `incoming/` but `.gitkeep`, and the `extra` paths of the `backup` section into `<target>/<machine>/`, then writes the manifest `.chemenu-backup.json` there: time, stack version, machine name, and per file its relative path, size and sha256.
|
||||
- Never mirrors `mcp-upload/`, `reports/`, `.wikitool-tools.json`, `tools/.venv/`, the published skills or `.git/` - unless an `extra` path names one of the first ones. The secrets file lies outside the instance and is never read.
|
||||
- Never mirrors `quarantine/`, `reports/`, `.wikitool-tools.json`, `tools/.venv/`, the published skills or `.git/` - unless an `extra` path names one of the first ones. The secrets file lies outside the instance and is never read.
|
||||
- Writes and deletes only under `<target>/<machine>/`. Deletes only a file the previous manifest lists and the instance no longer has; a file there the manifest does not list stays as it is. A file whose size and sha256 match the previous manifest is not rewritten.
|
||||
- A symbolic link or junction inside a mirrored directory is skipped and named in the output, never followed.
|
||||
- `publish` runs the same step after every push when `machine_data` is true.
|
||||
|
||||
@@ -38,7 +38,7 @@ tools/wikitool doctor --json
|
||||
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
||||
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
||||
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
||||
- MCP `submit` tool: whether the `upload` section of the settings is present, absent or malformed, its limits, and how many submissions wait in `mcp-upload/`. Absent is `OK` and means the write path does not exist at all; malformed is a `FAIL`.
|
||||
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
||||
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
||||
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
||||
- For a configured `caldav` provider, whether the server is reachable and Basic auth succeeds - never a `FAIL`; only a broken config block is.
|
||||
|
||||
@@ -12,6 +12,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
|
||||
- [`raw status`](#raw-status)
|
||||
- [`raw pending`](#raw-pending)
|
||||
- [`raw accept`](#raw-accept)
|
||||
- [`upload collect`](#upload-collect)
|
||||
- [`upload list`](#upload-list)
|
||||
- [`upload show`](#upload-show)
|
||||
- [`upload accept`](#upload-accept)
|
||||
@@ -365,7 +366,7 @@ tools/wikitool raw pending --json
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `quarantine/` is outside `incoming/` and never listed.
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
@@ -495,9 +496,64 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
|
||||
- `wikitool types describe source` - the capture field values
|
||||
- `wikitool new source` - the source page for a promoted file
|
||||
|
||||
## `upload collect`
|
||||
|
||||
Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
- `wikitool upload collect [<channel>...]`
|
||||
|
||||
**Options**
|
||||
|
||||
```text
|
||||
Arguments:
|
||||
names... Channels to collect from; all pulled channels when none is named
|
||||
|
||||
Options:
|
||||
-h, --help Show this message and exit.
|
||||
```
|
||||
|
||||
**Examples**
|
||||
|
||||
```bash
|
||||
tools/wikitool upload collect
|
||||
tools/wikitool upload collect onedrive
|
||||
```
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Without arguments, collects from every channel of the `intake` section whose handler is `folder` or `command`; with names, from those channels only. A name no channel has, or the name of the `mcp` channel (which is pushed, not pulled), exits 1 before anything happens.
|
||||
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
||||
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
||||
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
||||
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
||||
- Not idempotent: a second run finds the folder emptied.
|
||||
|
||||
**Properties**
|
||||
|
||||
- Changes files: yes
|
||||
- Idempotent: no
|
||||
- Atomic: No - per file: one copy into `quarantine/<id>/` (manifest last), then one removal of the source; a channel's failure does not stop the others
|
||||
- Counts toward the session call budget: yes
|
||||
- Can reach the network: no
|
||||
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - Unknown channel name, an `mcp` channel as argument, a malformed `intake` section, or an `mcp-upload/` that still holds submissions
|
||||
- `1` - A channel or a file failed (folder missing, program exited non-zero, timed out or could not be started, a source that could not be removed)
|
||||
|
||||
**See also**
|
||||
|
||||
- `wikitool upload list` - what waits for review afterwards
|
||||
- `wikitool upload accept` - the gate out of the quarantine
|
||||
- `docs/how-to/collect-from-a-synced-folder.md`
|
||||
|
||||
## `upload list`
|
||||
|
||||
List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
@@ -519,8 +575,9 @@ tools/wikitool upload list --json
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Lists every submission waiting in `mcp-upload/`, oldest id first: id, filename, size, submitter.
|
||||
- Only ever non-empty when the `upload` section of `.wikitool-local.json` opts the checkout into the MCP server's `submit` tool.
|
||||
- Lists every submission waiting in `quarantine/`, oldest id first: id, filename, size, channel, submitter.
|
||||
- Only ever non-empty when the `intake` section of `.wikitool-local.json` declares a channel.
|
||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions - the message names the command that moves it.
|
||||
- A submission directory with a corrupt manifest is silently skipped.
|
||||
- Never fails; read-only and safe to retry freely.
|
||||
|
||||
@@ -568,7 +625,8 @@ tools/wikitool upload show <id>
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name, not a claim the header was honest), submission time.
|
||||
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name or the handler, not a claim the header was honest), submission time, and where it came from: channel, handler, origin (the source file's path, `command:<name>` or `mcp:submit`).
|
||||
- A submitter of `unknown` means the channel has no sender to trust - read channel and origin instead.
|
||||
- What a reviewer reads before `upload accept`.
|
||||
- Read-only.
|
||||
|
||||
@@ -618,7 +676,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Promotes a submission's file from `mcp-upload/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `mcp-upload/ledger.jsonl`.
|
||||
- Promotes a submission's file from `quarantine/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `quarantine/ledger.jsonl`.
|
||||
- Upload Review Gate: without a matching `--confirm`, exits 42 and prints the manifest in full plus the exact re-run line - one submission at a time. The gate check runs before anything is moved.
|
||||
- The token digests id, filename, size, sha256 and submitter, so an edited or superseded manifest invalidates it.
|
||||
- An occupied `incoming/<filename>` is an ordinary validation error, not the gate.
|
||||
@@ -635,7 +693,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - Unknown or malformed submission id, or the submission's file is missing from `mcp-upload/<id>/`
|
||||
- `1` - Unknown or malformed submission id, or the submission's file is missing from `quarantine/<id>/`
|
||||
- `1` - `incoming/<filename>` already exists
|
||||
- `42` - Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ something you run.
|
||||
- [Super Productivity](#super-productivity)
|
||||
- [CalDAV](#caldav)
|
||||
- [Profiles](#profiles)
|
||||
- [MCP submissions](#mcp-submissions)
|
||||
- [Intake channels](#intake-channels)
|
||||
- [Site handlers](#site-handlers)
|
||||
- [Fetch routes](#fetch-routes)
|
||||
- [Backup to a synced folder](#backup-to-a-synced-folder)
|
||||
@@ -92,7 +92,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
|
||||
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
|
||||
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
||||
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
||||
[upload](#mcp-submissions), [sites](#site-handlers), [fetch](#fetch-routes) and
|
||||
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
|
||||
[backup](#backup-to-a-synced-folder). `config set` names the sections it knows, and `doctor` reports a
|
||||
section it does not read as a `WARN`.
|
||||
|
||||
@@ -121,7 +121,7 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
|
||||
**Never in the committed file:** a field that names a program to run or affects a gate - not even
|
||||
as a reference. `config set` does not write such a field either (exit 42); the user edits
|
||||
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
|
||||
`names a command` in the tables below are these: the publish allowlist, the upload limits and the
|
||||
`names a command` in the tables below are these: the publish allowlist, the intake channels and the
|
||||
external fetchers.
|
||||
|
||||
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
|
||||
@@ -371,45 +371,92 @@ several trackers in turn. A profile is checked like the local file, except that
|
||||
value is allowed in it; a secret is a reference there too, never a value. The live suite takes its
|
||||
throwaway tokens from `WIKITOOL_SECRET_<NAME>`.
|
||||
|
||||
## MCP submissions
|
||||
## Intake channels
|
||||
|
||||
The `upload` section of `.wikitool-local.json` in the served corpus turns on the MCP read server's
|
||||
optional `submit` tool. What the authentication in front of the server has to guarantee before you
|
||||
create it is in
|
||||
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path). Every field
|
||||
is marked `gate`: the limits are the user's to set by hand, and `config set` exits 42 for them.
|
||||
The `intake` section of `.wikitool-local.json` declares the ways a document gets into the
|
||||
quarantine `quarantine/` from somewhere other than this terminal. Each channel has a name and a
|
||||
handler:
|
||||
|
||||
<!-- wikitool:reference-config-upload -->
|
||||
| Handler | Source | How it arrives |
|
||||
|---|---|---|
|
||||
| `mcp` | The MCP read server's optional `submit` tool | Pushed by a remote caller. At most one channel has this handler. |
|
||||
| `folder` | A directory outside the instance that a sync client fills (OneDrive and the like) | Pulled by `wikitool upload collect`; what is taken is removed from the folder. |
|
||||
| `command` | A program of this checkout, named in `commands` | Pulled by `wikitool upload collect`; the program leaves its files in an empty staging directory. |
|
||||
|
||||
All of them end in the same place: a submission nobody has looked at, which `wikitool upload accept`
|
||||
promotes into `incoming/` after a human has cleared the Upload Review Gate. What the authentication
|
||||
in front of the server has to guarantee for the `mcp` handler is in
|
||||
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path); how to
|
||||
collect from a synced folder or a program is in
|
||||
[Collect documents from a synced folder](../how-to/collect-from-a-synced-folder.md). Every field is
|
||||
marked `gate` or `names a command`: the channels are the user's to set by hand, and `config set`
|
||||
exits 42 for them.
|
||||
|
||||
<!-- wikitool:reference-config-intake -->
|
||||
| Field | Tier | Type | Required | Default | Description |
|
||||
|-------|------|------|----------|---------|-------------|
|
||||
| `identity_header` | checkout, gate | string | no | `"X-Forwarded-User"` | The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default. |
|
||||
| `max_bytes` | checkout, gate | positive integer | yes | - | Largest single submission, in bytes. A number written as a string is accepted. |
|
||||
| `allowed_extensions` | checkout, gate | array of strings | yes | - | The file extensions `submit` accepts, each starting with `.`. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse. |
|
||||
| `quota_submissions_per_day` | checkout, gate | positive integer | yes | - | Submissions one submitter may make over any rolling 24 hours. A number written as a string is accepted. |
|
||||
| `quota_bytes_per_day` | checkout, gate | positive integer | yes | - | Bytes one submitter may submit over any rolling 24 hours. A number written as a string is accepted. |
|
||||
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
|
||||
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
|
||||
|
||||
- **Absent:** The `submit` tool is not registered: the write path does not exist.
|
||||
- **Malformed:** An error. The MCP server does not start and `doctor` reports `FAIL`. A broken section is never read as "no limits".
|
||||
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
|
||||
- **Malformed:** An error. The MCP server does not start, `upload collect` stops with exit 1 and `doctor` reports `FAIL`. A broken section is never read as "no channel".
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"upload": {
|
||||
"identity_header": "X-Forwarded-User",
|
||||
"max_bytes": 10485760,
|
||||
"allowed_extensions": [
|
||||
".md",
|
||||
".pdf"
|
||||
],
|
||||
"quota_submissions_per_day": 20,
|
||||
"quota_bytes_per_day": 52428800
|
||||
"intake": {
|
||||
"channels": {
|
||||
"mcp": {
|
||||
"handler": "mcp",
|
||||
"identity_header": "X-Forwarded-User",
|
||||
"max_bytes": 10485760,
|
||||
"allowed_extensions": [
|
||||
".md",
|
||||
".pdf"
|
||||
],
|
||||
"quota_submissions_per_day": 20,
|
||||
"quota_bytes_per_day": 52428800
|
||||
},
|
||||
"onedrive": {
|
||||
"handler": "folder",
|
||||
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Eingang",
|
||||
"max_bytes": 52428800,
|
||||
"allowed_extensions": [
|
||||
".pdf",
|
||||
".docx",
|
||||
".msg",
|
||||
".eml",
|
||||
".md"
|
||||
],
|
||||
"settle_seconds": 60
|
||||
},
|
||||
"scanner": {
|
||||
"handler": "command",
|
||||
"command": "scanner-pull",
|
||||
"max_bytes": 52428800,
|
||||
"allowed_extensions": [
|
||||
".pdf"
|
||||
]
|
||||
}
|
||||
},
|
||||
"commands": {
|
||||
"scanner-pull": [
|
||||
"pwsh",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
"C:\\tools\\scanner-pull.ps1"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
<!-- /wikitool:reference-config-upload -->
|
||||
<!-- /wikitool:reference-config-intake -->
|
||||
|
||||
Size and quota are your deployment's decision, not defaults worth copying blindly.
|
||||
Size and quota are your deployment's decision, not defaults worth copying blindly. A field that does
|
||||
not belong to the channel's handler is an error, not ignored: `identity_header` and the `quota_*`
|
||||
fields belong to `mcp`, `path` and `settle_seconds` to `folder`, `command` to `command`. A `folder`
|
||||
`path` must be absolute and lie outside the instance, and `upload collect` reads only its top level.
|
||||
|
||||
## Site handlers
|
||||
|
||||
@@ -590,7 +637,7 @@ Until version 10 every feature above had a file of its own at the root: `.wikito
|
||||
`.wikitool-remotes.json`, `.wikitool-upload.json` and `.wikitool-tasks.json`. Nothing reads them any
|
||||
more. A command that needs the setting such a file held stops with exit 42 and names
|
||||
`tools/wikitool config migrate`, `doctor` reports each one as a `FAIL`, and only the telemetry hook
|
||||
ignores it. `config migrate` moves the content into the sections above, each value into the file its
|
||||
ignores it. `config migrate` moves the content into the sections above (the upload file becomes the channel `mcp` of `intake`), each value into the file its
|
||||
tier names and each secret into the secrets file, and then removes the old files; it is run once per
|
||||
checkout. `.wikitool-tools.json` is not among them and stays. The migration step for an instance is
|
||||
[10.0.0-settings-tiers.md](../../instructions/migrations/10.0.0-settings-tiers.md).
|
||||
Reference in new issue
Block a user