intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11
Files changed: - .gitignore - CHANGES.md - VERSION - docs/explanation/architecture.md - docs/explanation/pipeline-rationale.md - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/ingest-a-source.md - docs/how-to/set-up-the-mcp-server.md - docs/reference/commands/README.md - docs/reference/commands/git.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - instructions/gates.md - instructions/ingest-queue.md - instructions/mcp-read-server.md - instructions/migrations/10.0.0-settings-tiers.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/collect.py - tools/chemenu/commands/backup_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config.py - tools/chemenu/config_shapes.py - tools/chemenu/legacy_settings.py - tools/chemenu/mcp/server.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_backup.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_legacy_settings.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_upload.py - tools/chemenu/tests/test_upload_cmd.py - tools/chemenu/upload.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
3627168403
commit
cceb9a7bfe
44 files changed
+1910
-346
No files matched your search
@@ -12,6 +12,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
|
||||
- [`raw status`](#raw-status)
|
||||
- [`raw pending`](#raw-pending)
|
||||
- [`raw accept`](#raw-accept)
|
||||
- [`upload collect`](#upload-collect)
|
||||
- [`upload list`](#upload-list)
|
||||
- [`upload show`](#upload-show)
|
||||
- [`upload accept`](#upload-accept)
|
||||
@@ -365,7 +366,7 @@ tools/wikitool raw pending --json
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `quarantine/` is outside `incoming/` and never listed.
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
@@ -495,9 +496,64 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
|
||||
- `wikitool types describe source` - the capture field values
|
||||
- `wikitool new source` - the source page for a promoted file
|
||||
|
||||
## `upload collect`
|
||||
|
||||
Pull material from the configured `folder` and `command` intake channels into the quarantine - never past it.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
- `wikitool upload collect [<channel>...]`
|
||||
|
||||
**Options**
|
||||
|
||||
```text
|
||||
Arguments:
|
||||
names... Channels to collect from; all pulled channels when none is named
|
||||
|
||||
Options:
|
||||
-h, --help Show this message and exit.
|
||||
```
|
||||
|
||||
**Examples**
|
||||
|
||||
```bash
|
||||
tools/wikitool upload collect
|
||||
tools/wikitool upload collect onedrive
|
||||
```
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Without arguments, collects from every channel of the `intake` section whose handler is `folder` or `command`; with names, from those channels only. A name no channel has, or the name of the `mcp` channel (which is pushed, not pulled), exits 1 before anything happens.
|
||||
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
||||
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
||||
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
||||
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
||||
- Not idempotent: a second run finds the folder emptied.
|
||||
|
||||
**Properties**
|
||||
|
||||
- Changes files: yes
|
||||
- Idempotent: no
|
||||
- Atomic: No - per file: one copy into `quarantine/<id>/` (manifest last), then one removal of the source; a channel's failure does not stop the others
|
||||
- Counts toward the session call budget: yes
|
||||
- Can reach the network: no
|
||||
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - Unknown channel name, an `mcp` channel as argument, a malformed `intake` section, or an `mcp-upload/` that still holds submissions
|
||||
- `1` - A channel or a file failed (folder missing, program exited non-zero, timed out or could not be started, a source that could not be removed)
|
||||
|
||||
**See also**
|
||||
|
||||
- `wikitool upload list` - what waits for review afterwards
|
||||
- `wikitool upload accept` - the gate out of the quarantine
|
||||
- `docs/how-to/collect-from-a-synced-folder.md`
|
||||
|
||||
## `upload list`
|
||||
|
||||
List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
List every submission currently waiting in the quarantine (`quarantine/`), whatever channel it came through.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
@@ -519,8 +575,9 @@ tools/wikitool upload list --json
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Lists every submission waiting in `mcp-upload/`, oldest id first: id, filename, size, submitter.
|
||||
- Only ever non-empty when the `upload` section of `.wikitool-local.json` opts the checkout into the MCP server's `submit` tool.
|
||||
- Lists every submission waiting in `quarantine/`, oldest id first: id, filename, size, channel, submitter.
|
||||
- Only ever non-empty when the `intake` section of `.wikitool-local.json` declares a channel.
|
||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions - the message names the command that moves it.
|
||||
- A submission directory with a corrupt manifest is silently skipped.
|
||||
- Never fails; read-only and safe to retry freely.
|
||||
|
||||
@@ -568,7 +625,8 @@ tools/wikitool upload show <id>
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name, not a claim the header was honest), submission time.
|
||||
- Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name or the handler, not a claim the header was honest), submission time, and where it came from: channel, handler, origin (the source file's path, `command:<name>` or `mcp:submit`).
|
||||
- A submitter of `unknown` means the channel has no sender to trust - read channel and origin instead.
|
||||
- What a reviewer reads before `upload accept`.
|
||||
- Read-only.
|
||||
|
||||
@@ -618,7 +676,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Promotes a submission's file from `mcp-upload/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `mcp-upload/ledger.jsonl`.
|
||||
- Promotes a submission's file from `quarantine/<id>/` into `incoming/`, deletes the quarantine directory, and appends an `accepted` event to `quarantine/ledger.jsonl`.
|
||||
- Upload Review Gate: without a matching `--confirm`, exits 42 and prints the manifest in full plus the exact re-run line - one submission at a time. The gate check runs before anything is moved.
|
||||
- The token digests id, filename, size, sha256 and submitter, so an edited or superseded manifest invalidates it.
|
||||
- An occupied `incoming/<filename>` is an ordinary validation error, not the gate.
|
||||
@@ -635,7 +693,7 @@ tools/wikitool upload accept <id> --confirm <token> # re-run after exit 42, o
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - Unknown or malformed submission id, or the submission's file is missing from `mcp-upload/<id>/`
|
||||
- `1` - Unknown or malformed submission id, or the submission's file is missing from `quarantine/<id>/`
|
||||
- `1` - `incoming/<filename>` already exists
|
||||
- `42` - Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
|
||||
|
||||
|
||||
Reference in new issue
Block a user