intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11

Files changed: - .gitignore - CHANGES.md - VERSION - docs/explanation/architecture.md - docs/explanation/pipeline-rationale.md - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/ingest-a-source.md - docs/how-to/set-up-the-mcp-server.md - docs/reference/commands/README.md - docs/reference/commands/git.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - instructions/gates.md - instructions/ingest-queue.md - instructions/mcp-read-server.md - instructions/migrations/10.0.0-settings-tiers.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/collect.py - tools/chemenu/commands/backup_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config.py - tools/chemenu/config_shapes.py - tools/chemenu/legacy_settings.py - tools/chemenu/mcp/server.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_backup.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_legacy_settings.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_upload.py - tools/chemenu/tests/test_upload_cmd.py - tools/chemenu/upload.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
2026-10-12 22:04:18 +02:00
1 parent 3627168403
commit cceb9a7bfe
@@ -30,6 +30,8 @@ Running the instance:
 - [Trace and score agent sessions](trace-and-score-sessions.md)
 - [Use 1Password for settings](use-1password-for-settings.md) - secrets and private values out
  of the tree, on the desktop and in CI
+- [Collect documents from a synced folder](collect-from-a-synced-folder.md) - a folder OneDrive and
+  the like fill, or a program of your own, as an intake channel into the quarantine
 - [Back up to a synced folder](back-up-to-a-synced-folder.md) - OneDrive and the like as the
  backup of an instance without a git server, and restoring from it

@@ -100,7 +100,7 @@ push.
 | `ENVIRONMENT.md` | yes, if present | written by hand |
 | Everything in `incoming/` but `.gitkeep` | yes | sources waiting there are backed up nowhere else |
 | The `extra` paths | yes | what you named |
-| `mcp-upload/` | no | material submitted from outside that nobody has reviewed yet |
+| `quarantine/` | no | material that arrived through an intake channel and that nobody has reviewed yet |
 | `reports/` | no | regenerated by `lint`; telemetry holds prompts |
 | `.wikitool-tools.json`, `tools/.venv/`, the published skills | no | the preflight and `instructions sync` rebuild them |
 | The secrets file | **never** | it lies outside the instance, and a backup reads nothing outside it |
@@ -0,0 +1,133 @@
+# Collect documents from a synced folder
+
+Documents that are not typed or fetched at this terminal can reach the wiki through an intake
+channel. Besides the MCP `submit` tool there are two channels that this checkout *pulls* from: a
+folder that a sync client such as OneDrive fills (handler `folder`), and a program of your own
+(handler `command`). Both end in the quarantine `quarantine/`, exactly like a submission through the
+server, and both wait there for a human to run `wikitool upload accept` - the Upload Review Gate
+stands between every channel and `incoming/`.
+
+This guide creates a `folder` channel, explains what `upload collect` leaves alone and why, and shows
+a `command` channel with an example script. The field reference is in the
+[configuration reference](../reference/configuration.md#intake-channels); the review is in
+[ingesting a source](ingest-a-source.md#a-document-submitted-from-outside).
+
+<!-- wikitool:toc -->
+## Contents
+
+- [Create the folder channel](#create-the-folder-channel)
+- [What stays in the folder, and why](#what-stays-in-the-folder-and-why)
+- [`settle_seconds` and what it cannot promise](#settle_seconds-and-what-it-cannot-promise)
+- [Deleting in a synced folder deletes everywhere](#deleting-in-a-synced-folder-deletes-everywhere)
+- [A command channel](#a-command-channel)
+- [Afterwards](#afterwards)
+<!-- /wikitool:toc -->
+
+## Create the folder channel
+
+Channels are not written by `config set`: every field of the `intake` section opens a path into the
+checkout, so it is yours to edit by hand in `.wikitool-local.json`.
+
+```json
+{
+  "intake": {
+    "channels": {
+      "onedrive": {
+        "handler": "folder",
+        "path": "/home/you/OneDrive/wiki-inbox",
+        "settle_seconds": 60,
+        "max_bytes": 10485760,
+        "allowed_extensions": [".pdf", ".docx", ".md", ".txt"]
+      }
+    }
+  }
+}
+```
+
+`path` has to be absolute and lie outside the instance. Check it with `tools/wikitool doctor`, then
+pull:
+
+```bash
+tools/wikitool upload collect            # every pull channel
+tools/wikitool upload collect onedrive   # only this one
+tools/wikitool upload list
+```
+
+A collected file is stored unchanged in `quarantine/<id>/` with a manifest that names the channel,
+the handler and the original path. A folder knows nobody who dropped a file, so the submitter is
+recorded as `unknown`, with the handler as its source: do not trust the name or content of such a
+file any more than that of a stranger's.
+
+## What stays in the folder, and why
+
+`upload collect` reads the top level of the folder only, and it removes one kind of thing: a file it
+has just stored. Everything else stays where it is and is reported with its reason:
+
+| Entry | Why it stays |
+|---|---|
+| A subdirectory | Nothing descends into it, and nothing deletes a directory. |
+| A name starting with `.` or `~$`, a hidden or system file | Editors and sync clients keep temporary files like these next to the real one. |
+| A symbolic link | Following it could leave the folder. |
+| An extension outside `allowed_extensions`, or a file above `max_bytes` | The same limits as for `submit`. |
+| A file younger than `settle_seconds` | It may still be arriving. |
+| A file that changed while it was being read or copied | The source is removed only if it still has the size and sha256 that were stored. |
+
+A file whose sha256 already waits in the quarantine is reported as a duplicate, with the id of the
+existing submission, and is removed from the folder.
+
+## `settle_seconds` and what it cannot promise
+
+A sync client writes a file in pieces, so a file that is half there looks like a complete, smaller
+one. `settle_seconds` (default 60) skips files whose modification or creation time is younger than
+that, and the check that the source did not change during the copy catches the rest. Neither can see
+a client that pauses for longer than the interval in the middle of a file and then resumes: such a
+file is stored incomplete, and the review is where you notice it. Raise the value if your client
+works over a slow link.
+
+## Deleting in a synced folder deletes everywhere
+
+A file removed from a folder that OneDrive syncs is removed on every device that syncs it. That is
+what a collected file should do - it is now in the quarantine - but it also means a collect on one
+machine empties the folder for all of them. The client's recycle bin keeps the file for a while; the
+quarantine copy, which is never synced, is the one to rely on. Collect from one machine only.
+
+## A command channel
+
+A `command` channel runs a program of your own, which fetches documents from somewhere that has no
+synced folder - a mailbox, a share, a scanner. The program is named in `commands` and referenced from
+the channel; the section stays valid only if every reference resolves.
+
+```json
+{
+  "intake": {
+    "commands": {
+      "scan-mail": ["/home/you/bin/fetch-attachments.sh"]
+    },
+    "channels": {
+      "mail": { "handler": "command", "command": "scan-mail" }
+    }
+  }
+}
+```
+
+The program is started without a shell and with stdin closed, with an empty staging directory under
+`quarantine/.staging/` appended as its last argument. It has 300 seconds. It leaves the files it
+wants to hand over directly in that directory:
+
+```bash
+#!/bin/sh
+# fetch-attachments.sh <staging-dir>
+set -eu
+out="$1"
+rclone copy remote:wiki-inbox "$out" --max-age 1d
+```
+
+A program that exits non-zero, does not start or times out fails its channel: nothing is taken, the
+staging directory is removed, and the last 2000 characters of its stderr are shown. Other channels
+still run, and `upload collect` exits 1 if any channel failed. With no pull channel configured it
+exits 0 with "nothing to collect".
+
+## Afterwards
+
+The agent does the same at the start of an ingest when you name no file: `upload collect`, then
+`upload list`, then a report of what waits. It never runs `upload accept` for you.
@@ -93,11 +93,14 @@ Every ten sources, the agent suggests a lint run - see [Maintain the wiki](maint
 ## A document submitted from outside

 When the [MCP read server](set-up-the-mcp-server.md) has its optional `submit` tool turned on,
-documents can arrive from another machine. They land in the quarantine `mcp-upload/`, not in
+documents can arrive from another machine. A folder a sync client fills and a program of the
+checkout are two more ways in ([Collect documents from a synced
+folder](collect-from-a-synced-folder.md)). All of them land in the quarantine `quarantine/`, not in
 `incoming/`, and nothing else reads them there. Before an ingest can start, a human has to promote
 the submission:

 ```bash
+tools/wikitool upload collect         # pull the folder and command channels into the quarantine
 tools/wikitool upload list            # what is waiting
 tools/wikitool upload show <id>       # one submission's manifest: who sent it, size, checksum
 ```
@@ -160,18 +160,19 @@ reviewed yet.

 ## Optional: the submit path

-Without this, `submit` does not exist as a tool - not unused, but not registered. An
-`upload` section in the served checkout's `.wikitool-local.json` turns it on. Its fields, defaults and what a malformed
-section does are in the [configuration reference](../reference/configuration.md#mcp-submissions); a
-broken section never means "no limits", it stops the server from starting.
+Without this, `submit` does not exist as a tool - not unused, but not registered. A
+channel with the handler `mcp` in the `intake` section of the served checkout's
+`.wikitool-local.json` turns it on. Its fields, defaults and what a malformed section does are in
+the [configuration reference](../reference/configuration.md#intake-channels); a broken section never
+means "no limits", it stops the server from starting.

 **With `submit` on, the middleware has a second duty.** It has to set the identity header itself, on
 every authenticated request, and discard any copy the client sent. The server trusts that header as
 the value for `submitter`. A header the client could set itself would be a claim, not an identity.
-If `identity_header` names a header the middleware does not set, every submission is refused for
+If the channel's `identity_header` names a header the middleware does not set, every submission is refused for
 lack of an identity.

-Submitted files land in `mcp-upload/<id>/`, which is gitignored and read by no other command. From
+Submitted files land in `quarantine/<id>/`, which is gitignored and read by no other command. From
 there a human promotes or rejects them, in the same checkout as the server rather than in its
 process. That review is part of
 [ingesting a source](ingest-a-source.md#a-document-submitted-from-outside).
@@ -228,7 +229,7 @@ the client starts a different Python than the instance's. Give the client the ab

 **`"commit": null` in every answer**: the served tree has uncommitted changes. Either the sync is not
 running, or something writes into the corpus that has no business there. The five read tools never
-write. `submit`, when it is on, writes only to `mcp-upload/`, which is gitignored and so is never a
+write. `submit`, when it is on, writes only to `quarantine/`, which is gitignored and so is never a
 cause of `"commit": null` either.

 **`commit` names an old revision**: the sync timer is not running.
@@ -246,6 +247,6 @@ Deployment specifics - the cluster, ingress hosts, the secret store, the GitOps
 private infrastructure, and this repository is public.

 There is no container image for the server yet. Until there is, the path above - the venv,
-`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `mcp-upload/`
+`python -m chemenu.mcp`, a proxy in front - is the complete one. With `submit` on, `quarantine/`
 has to survive restarts the same way the rest of the checkout does. Otherwise a submitted file that
 nobody has reviewed yet loses its quarantine.