intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11

Files changed: - .gitignore - CHANGES.md - VERSION - docs/explanation/architecture.md - docs/explanation/pipeline-rationale.md - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/ingest-a-source.md - docs/how-to/set-up-the-mcp-server.md - docs/reference/commands/README.md - docs/reference/commands/git.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - instructions/gates.md - instructions/ingest-queue.md - instructions/mcp-read-server.md - instructions/migrations/10.0.0-settings-tiers.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/collect.py - tools/chemenu/commands/backup_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config.py - tools/chemenu/config_shapes.py - tools/chemenu/legacy_settings.py - tools/chemenu/mcp/server.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_backup.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_legacy_settings.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_upload.py - tools/chemenu/tests/test_upload_cmd.py - tools/chemenu/upload.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
2026-10-12 22:04:18 +02:00
1 parent 3627168403
commit cceb9a7bfe
@@ -117,11 +117,13 @@ it leaves open exactly the window it closes.

 ### Upload Review Gate

-The MCP server's opt-in `submit` tool (the `upload` section of `.wikitool-local.json`) writes into a
-quarantine, `mcp-upload/`, that no other command reads. This gate is the only door out of it:
+The intake channels (the `intake` section of `.wikitool-local.json`: the MCP server's opt-in
+`submit` tool, a synced `folder`, a `command`) write into a quarantine, `quarantine/`, that no other
+command reads. This gate is the only door out of it:
 `wikitool upload accept <id>` refuses without a matching `--confirm <token>`, printing the
-submission's manifest in full - filename, size, sha256, submitter, and `submitter_source` (the
-*header* the submitter's name came from, not a verified fact) - plus the exact re-run line.
+submission's manifest in full - filename, size, sha256, submitter, `submitter_source` (the
+*header* the submitter's name came from, not a verified fact; the handler for a channel with no
+sender, whose `submitter` is `unknown`), and `channel`, `handler` and `origin` - plus the exact re-run line.

 Same shape as the Mass-Update Gate, scoped to one submission instead of a changeset: the token
 digests id/filename/size/sha256/submitter, so an edited or superseded manifest invalidates it the
@@ -132,7 +134,7 @@ Mass-Update Gate's review report versus this file's exit-42 procedure.

 `wikitool upload reject <id> --reason "<why>"` is the other way out, and it has **no gate at
 all** - rejecting needs no clearance, only accepting a stranger's file into the pipeline does. It
-deletes the material and keeps only the reason and a sha256 in `mcp-upload/ledger.jsonl`.
+deletes the material and keeps only the reason and a sha256 in `quarantine/ledger.jsonl`.

 ### Guideline Push Gate

@@ -1,18 +1,19 @@
 ---
 type: types/instruction.md
 name: ingest-queue
-description: Review a document submitted from outside through the MCP submit tool before it is promoted into incoming/ - what to check, how the quarantine and its ledger work, and the Upload Review Gate that stands between a submission and raw/
+description: Review a document that reached the quarantine from outside - through the MCP submit tool, a synced folder or a command of this checkout - before it is promoted into incoming/ - what to check, how the quarantine and its ledger work, and the Upload Review Gate that stands between a submission and raw/
 ---

 # Reviewing an external submission

-An `upload` section in `.wikitool-local.json` opts a checkout into a sixth MCP tool, `submit` -
-documents pushed by a caller that is not this terminal, into a quarantine no
-ordinary command reads. This is the human half of that path: what a reviewer
-checks before letting one through, and how `upload accept`/`upload reject`
-work. What the tool itself enforces (identity, size, extension, quota,
-duplicate-hash) is `chemenu/upload.py`'s job and is not repeated here - read
-this when a submission is already waiting and a decision is due.
+The `intake` section in `.wikitool-local.json` opts a checkout into one or more *intake channels*:
+a sixth MCP tool, `submit` (handler `mcp`), a folder a sync client fills (`folder`) or a program of
+this checkout (`command`) - documents that come from somewhere other than this terminal, into a
+quarantine, `quarantine/`, that no ordinary command reads. This is the human half of that path:
+what a reviewer checks before letting one through, and how `upload accept`/`upload reject` work.
+What the channels themselves enforce (identity, size, extension, quota, duplicate-hash, settle
+time) is `chemenu/upload.py`'s and `chemenu/collect.py`'s job and is not repeated here - read this
+when a submission is already waiting and a decision is due.

 <!-- wikitool:toc -->
 ## Contents
@@ -26,7 +27,8 @@ this when a submission is already waiting and a decision is due.

 ## When to run

- `wikitool upload list` shows one or more submissions waiting.
+- `wikitool upload list` shows one or more submissions waiting - after
+  `wikitool upload collect` has pulled what the `folder` and `command` channels hold.
 - A `submit` call reported success and named an id worth looking at now
  rather than later.
 - Standing up the `submit` tool for the first time - see § Arming the intake
@@ -37,12 +39,18 @@ this when a submission is already waiting and a decision is due.
 1. **Read the manifest, not just the file.** `wikitool upload show <id>`
   prints filename, size, sha256, submitter, and - as important as the
   submitter's name - `submitter_source`: the *header* the value came from,
-   naming where the claim rests rather than asserting it as fact. Read
+   naming where the claim rests rather than asserting it as fact. The
+   manifest also names `channel`, `handler` and `origin` (the source file's
+   path for a `folder` channel, `command:<name>` for a `command` channel,
+   `mcp:submit` for the tool). **`submitter: unknown` means there is no sender
+   to trust at all** - a synced folder or a script knows who dropped the file
+   no better than the file system does - so the file is judged on its
+   content alone. Read
   [docs/explanation/why-gates-are-code.md](../docs/explanation/why-gates-are-code.md) once for why
   this gate exists as code rather than as this paragraph alone.

 2. **Read the file itself before promoting anything.** The quarantine holds
-   it at `mcp-upload/<id>/<filename>` for exactly this purpose. Check it
+   it at `quarantine/<id>/<filename>` for exactly this purpose. Check it
   against [raw/CONTRACT.md](../raw/CONTRACT.md) "What does not belong here":
   secrets or credentials, content that is not worth a source page, anything
   the LLM itself wrote presented as a source.
@@ -80,28 +88,29 @@ this when a submission is already waiting and a decision is due.
   - **Reject:** `wikitool upload reject <id> --reason "<why>"` deletes the
     material immediately - no gate, because deleting needs no clearance,
     only accepting a stranger's file into the pipeline does. The reason and
-     the file's sha256 survive in `mcp-upload/ledger.jsonl`; the bytes do
+     the file's sha256 survive in `quarantine/ledger.jsonl`; the bytes do
     not. Write a reason a later reader can act on ("license unclear",
     "looks like a prompt injection attempt", "duplicate of an existing
     source under a different name") rather than a bare "no".

-6. **Never promote by hand.** Moving the file out of `mcp-upload/` with `mv`
+6. **Never promote by hand.** Moving the file out of `quarantine/` with `mv`
   or by editing `incoming/` directly skips the ledger entry and the gate
   both - the same "never hand-craft what the tool would have produced"
   principle as everywhere else in this stack (AGENTS.md invariant 7).

 ## Arming the intake

-`submit` does not exist as a tool until the `upload` section exists in the
-served root's `.wikitool-local.json` - absence means the write path is not registered at all, not
-that it is unrestricted (see the section's own shape in
+`submit` does not exist as a tool until the `intake` section of the served root's
+`.wikitool-local.json` declares a channel with the handler `mcp` - absence means the write path is
+not registered at all, not that it is unrestricted. A `folder` or `command` channel opens no remote
+path: it is pulled by `wikitool upload collect`, run by someone at this checkout (see the section's own shape in
 [raw/CONTRACT.md](../raw/CONTRACT.md) and `tools/chemenu/upload.py`). Two
 things belong to the *deployment*, not to this repository, and are named here
 because a reviewer needs to know they hold, not because this file configures
 them:

 - **The identity header is set by the middleware, never by the client.**
-  `identity_header` (default `X-Forwarded-User`) names an HTTP header the
+  The `mcp` channel's `identity_header` (default `X-Forwarded-User`) names an HTTP header the
  Traefik authentication layer in front of the process must set on every
  authenticated request and strip from any copy the client itself sent - the
  same posture [instructions/mcp-read-server.md](mcp-read-server.md) already
@@ -110,7 +119,7 @@ them:
  be trustworthy enough to write into `submitter` and stay there.
 - **Quota and size limits are a deployment decision, not a default worth
  copying blindly.** `max_bytes`, `allowed_extensions`,
-  `submissions_per_day`, `bytes_per_day` all live in the same file - see
+  `quota_submissions_per_day`, `quota_bytes_per_day` all live on the channel in the same file - see
  [tools/CONTRACT.md](../tools/CONTRACT.md) for the exact shape.

 ## Decision points
@@ -96,7 +96,7 @@ everything an operator needs that is *true of the software* rather than of one i
   stamped `"commit": null`, because a dirty tree corresponds to no revision.

   **Never add `git clean` to this sync.** `reset --hard` leaves every gitignored path alone by
-   design, which is exactly what keeps `mcp-upload/` (the `submit` tool's own quarantine) and
+   design, which is exactly what keeps `quarantine/` (the intake channels' quarantine) and
   `reports/telemetry/` intact across a sync - a `git clean -xd` bolted on "to tidy up" would
   delete a submission nobody has reviewed yet, silently, on the next poll.

@@ -112,8 +112,8 @@ everything an operator needs that is *true of the software* rather than of one i
 - **Asked to expose a write tool?** Five of the six tools have none, structurally: the server
  imports nothing under `chemenu.commands`, so `new`, `touch`, `xref`, `cite`, `publish` and
  `migrate` are unreachable from it rather than filtered out of a list. The one exception is
-  `submit` (opt-in via the `upload` settings section): it may write, but only into
-  `mcp-upload/`, a quarantine no other command reads - a **positive list** enforced in code
+  `submit` (opt-in via a channel with the handler `mcp` in the `intake` settings section): it may write, but only into
+  `quarantine/`, a quarantine no other command reads - a **positive list** enforced in code
  (`chemenu.upload._write_atomic_within`), not an absence. The commands that move a submission
  *out* of that quarantine (`upload accept`/`upload reject`) still have the absence property:
  they live under `chemenu.commands` and stay unreachable from the server. Reviewing what
@@ -1,7 +1,7 @@
 ---
 type: types/instruction.md
 name: 10.0.0-settings-tiers
-description: Move the settings of the four per-feature files .wikitool-telemetry.json, .wikitool-remotes.json, .wikitool-upload.json and .wikitool-tasks.json into the tiered settings with `wikitool config migrate` - run once per checkout; without it the Publish-Remote Gate and the upload limits are not in effect and the commands that need them exit 42.
+description: Move the settings of the four per-feature files .wikitool-telemetry.json, .wikitool-remotes.json, .wikitool-upload.json and .wikitool-tasks.json into the tiered settings with `wikitool config migrate`, and move the quarantine mcp-upload/ to quarantine/ - run once per checkout; without it the Publish-Remote Gate and the upload limits are not in effect and the commands that need them exit 42.
 manual: true
 migrates_to: 10.0.0
 migration_kind: mechanical
@@ -13,7 +13,8 @@ Until 10.0.0 four features kept their settings in a file of their own at the roo
 checkout: `.wikitool-telemetry.json`, `.wikitool-remotes.json`, `.wikitool-upload.json` and
 `.wikitool-tasks.json`. 10.0.0 reads none of them. The same settings are now sections of the
 [tiered settings](../../docs/reference/configuration.md#tiered-settings) - `telemetry`,
-`remotes`, `upload`, and `tasks` with `superproductivity` and `caldav` - and each field sits in
+`remotes`, `intake` (the old `upload` file becomes its channel `mcp`), and `tasks` with
+`superproductivity` and `caldav` - and each field sits in
 the file its tier names: `wikitool.json` (committed) for what is the same in every clone,
 `.wikitool-local.json` for what is true of one checkout, and for a token or a password the
 secrets file, with only a reference in `wikitool.json`.
@@ -23,6 +24,13 @@ Publish-Remote Gate, no upload limits, no tracker. `wikitool config migrate` mov
 removes the old files. It is `mechanical` because nothing is decided per file: every setting keeps
 its effective value, defaults included.

+10.0.0 also renames the quarantine of the intake channels from `mcp-upload/` to `quarantine/`,
+because it no longer holds only what the MCP `submit` tool received. A checkout whose `mcp-upload/`
+still holds submissions that wait for review moves that directory once, by hand - nothing moves it
+automatically, since material that waits for a decision is not the tool's to relocate. Until it
+is moved, `doctor` reports a `FAIL` on `upload-intake` with the command, the `upload` commands
+exit 1 naming it, and the MCP server does not start.
+
 <!-- wikitool:toc -->
 ## Contents

@@ -52,8 +60,9 @@ command below.
   tools/wikitool doctor
   ```

-   Each old file that is still present is one `legacy-settings` line. None, and the checkout has
-   nothing to move: go to step 4.
+   Each old file that is still present is one `legacy-settings` line, and an `mcp-upload/` that
+   still holds submissions is a `FAIL` on the `upload-intake` line. Neither, and the checkout has
+   nothing to move: go to step 5.

 2. **Move it:**

@@ -67,18 +76,33 @@ command below.
   removes the old ones; if anything fails, nothing is changed and the old files are as they were.
   A second run says there is nothing to move.

-3. **Check it:**
+3. **Move the quarantine, if `mcp-upload/` still holds submissions.** In the instance root:
+
+   ```bash
+   mv mcp-upload quarantine              # POSIX shells
+   Move-Item mcp-upload quarantine       # PowerShell
+   ```
+
+   If `quarantine/` exists already (the new version created it), move the entries of
+   `mcp-upload/` into it one by one - each is a directory named by its id, plus `ledger.jsonl`,
+   which is appended to the existing ledger rather than replacing it - and remove the empty
+   `mcp-upload/` afterwards. Open submissions keep their ids and are listed by `upload list` again.
+   The root `.gitignore` of an instance carries `/quarantine/` from this version on; `dist upgrade`
+   brings the entry, and the quarantine writes its own `.gitignore` as well.
+
+4. **Check it:**

   ```bash
   tools/wikitool doctor
   git status --short
   ```

-   `doctor` shows no `legacy-settings` line, and the settings lines are `OK`. `git status` shows
+   `doctor` shows no `legacy-settings` line, the `upload-intake` line is not a `FAIL`, and the
+   settings lines are `OK`. `git status` shows
   `wikitool.json` as new or changed: it holds the instance-tier values and the references, and it
   is committed with the next `publish`. `.wikitool-local.json` does not appear, being ignored.

-4. **Record it, once per instance** - not once per checkout:
+5. **Record it, once per instance** - not once per checkout:

   ```bash
   tools/wikitool migrate done 10.0.0 --pages 0
@@ -89,8 +113,9 @@ command below.

 ## How to tell a migrated checkout from an unmigrated one

-An unmigrated checkout has at least one of the four files at its root, and `doctor` names each.
-A migrated one has none; `.wikitool-tools.json` is not among them and stays.
+An unmigrated checkout has at least one of the four files at its root, or an `mcp-upload/` that
+still holds submissions; `doctor` names each. A migrated one has none; `.wikitool-tools.json` is
+not among them and stays.

 ## Decision points

@@ -103,7 +128,8 @@ A migrated one has none; `.wikitool-tools.json` is not among them and stays.
  carried over.
 - **The old Publish-Remote allowlist or upload limits go where?** Into `.wikitool-local.json`:
  both affect a gate, so `config set` does not write them, and the migration is the one command
-  that does. Later changes to them are the user's own edit.
+  that does. The upload limits become the channel `mcp` of the `intake` section. Later changes to
+  them are the user's own edit.
 - **A profile file `WIKITOOL_TASKS_CONFIG` points at still holds a plain token?** It is not
  touched by `config migrate`, and it is no longer read as the old format: a profile has the layout of
  `.wikitool-local.json` and holds references only
@@ -112,6 +138,6 @@ A migrated one has none; `.wikitool-tools.json` is not among them and stays.

 ## Scope

-The four files named above, in the checkout the command runs in. `.wikitool-tools.json` and the
+The four files named above and the `mcp-upload/` directory, in the checkout the command runs in. `.wikitool-tools.json` and the
 two tiered settings files are not old files. The machinery - the readers, `config migrate`,
 `doctor`'s check - arrived with `dist upgrade` and needs nothing from this document.
@@ -48,12 +48,27 @@ validator complains - and the ticked list is the only record that they happened.
   `tools/wikitool raw derive <path>` first and read the `.md` - its output names what to check.
   Any other binary or image: note its presence and what it shows.

-   **The user named nothing** ("ingest", "process the inbox")? Pick the entry from the queue:
+   **The user named nothing** ("ingest", "process the inbox")? First pull what the configured
+   intake channels hold, then look at what waits for a reviewer, then pick the entry from the queue:

   ```bash
+   tools/wikitool upload collect
+   tools/wikitool upload list
   tools/wikitool raw pending
   ```

+   `upload collect` takes what the `folder` and `command` channels of the `intake` settings
+   section hold into the quarantine (`quarantine/`) and prints, per channel, what it took and what
+   it left with the reason. With no pull channel it says so ("nothing to collect") and the run
+   goes on. A channel that fails is reported to the user and the run continues with `incoming/` -
+   it does not stop the run. `upload list` shows every submission waiting in the quarantine,
+   whichever channel it came through; if any, report "N submissions are waiting" with their ids
+   and say that `upload show <id>` reads a manifest and `upload accept <id>` is the reviewer's
+   call (`instructions/ingest-queue.md`). **Never call `upload accept`
+   here** - it is the Upload Review Gate, a human decision. Nothing in the quarantine is a
+   candidate for this run until it has been accepted into `incoming/`. If `incoming/` is empty
+   and submissions are waiting, the run ends with that list.
+
   It lists what waits in `incoming/`, oldest first, and marks the default - the oldest entry
   `raw accept` would take as it stands. **Announce it and carry on with it:** which entry, why
   this one (the oldest that can be accepted), and how many wait after it. Ask nothing here -
@@ -290,8 +305,9 @@ validator complains - and the ticked list is the only record that they happened.
   `tools/wikitool raw accept incoming/<bundle>`, which refuses them. `_capture.json` goes in no
   `raw_files:`; the success message already leaves it out.

-   **A file that arrived through the MCP `submit` tool is not yet in `incoming/`** - it sits in
-   `mcp-upload/<id>/`, a quarantine no command in this step reads. A reviewer promotes it first
+   **A file that arrived through an intake channel (the MCP `submit` tool, a synced folder, a
+   command) is not yet in `incoming/`** - it sits in `quarantine/<id>/`, a quarantine no command in
+   this step reads. A reviewer promotes it first
   with `wikitool upload accept <id> --confirm <token>`, per
   `instructions/ingest-queue.md`; once accepted it is an ordinary file in
   `incoming/` and this step applies to it exactly as to anything dropped there by hand.