intake: several configured intake channels (mcp, folder, command) all end in quarantine/ (was mcp-upload/), and upload collect pulls the synced-folder and command channels into it (#213); 10.0.0-beta.11
Files changed: - .gitignore - CHANGES.md - VERSION - docs/explanation/architecture.md - docs/explanation/pipeline-rationale.md - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/ingest-a-source.md - docs/how-to/set-up-the-mcp-server.md - docs/reference/commands/README.md - docs/reference/commands/git.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - instructions/gates.md - instructions/ingest-queue.md - instructions/mcp-read-server.md - instructions/migrations/10.0.0-settings-tiers.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/collect.py - tools/chemenu/commands/backup_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config.py - tools/chemenu/config_shapes.py - tools/chemenu/legacy_settings.py - tools/chemenu/mcp/server.py - tools/chemenu/reference_doc.py - tools/chemenu/tests/test_backup.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_legacy_settings.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_upload.py - tools/chemenu/tests/test_upload_cmd.py - tools/chemenu/upload.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed
1 parent
3627168403
commit
cceb9a7bfe
44 files changed
+1910
-346
No files matched your search
@@ -27,7 +27,7 @@ something you run.
|
||||
- [Super Productivity](#super-productivity)
|
||||
- [CalDAV](#caldav)
|
||||
- [Profiles](#profiles)
|
||||
- [MCP submissions](#mcp-submissions)
|
||||
- [Intake channels](#intake-channels)
|
||||
- [Site handlers](#site-handlers)
|
||||
- [Fetch routes](#fetch-routes)
|
||||
- [Backup to a synced folder](#backup-to-a-synced-folder)
|
||||
@@ -92,7 +92,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
|
||||
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
|
||||
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
||||
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
||||
[upload](#mcp-submissions), [sites](#site-handlers), [fetch](#fetch-routes) and
|
||||
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
|
||||
[backup](#backup-to-a-synced-folder). `config set` names the sections it knows, and `doctor` reports a
|
||||
section it does not read as a `WARN`.
|
||||
|
||||
@@ -121,7 +121,7 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
|
||||
**Never in the committed file:** a field that names a program to run or affects a gate - not even
|
||||
as a reference. `config set` does not write such a field either (exit 42); the user edits
|
||||
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
|
||||
`names a command` in the tables below are these: the publish allowlist, the upload limits and the
|
||||
`names a command` in the tables below are these: the publish allowlist, the intake channels and the
|
||||
external fetchers.
|
||||
|
||||
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
|
||||
@@ -371,45 +371,92 @@ several trackers in turn. A profile is checked like the local file, except that
|
||||
value is allowed in it; a secret is a reference there too, never a value. The live suite takes its
|
||||
throwaway tokens from `WIKITOOL_SECRET_<NAME>`.
|
||||
|
||||
## MCP submissions
|
||||
## Intake channels
|
||||
|
||||
The `upload` section of `.wikitool-local.json` in the served corpus turns on the MCP read server's
|
||||
optional `submit` tool. What the authentication in front of the server has to guarantee before you
|
||||
create it is in
|
||||
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path). Every field
|
||||
is marked `gate`: the limits are the user's to set by hand, and `config set` exits 42 for them.
|
||||
The `intake` section of `.wikitool-local.json` declares the ways a document gets into the
|
||||
quarantine `quarantine/` from somewhere other than this terminal. Each channel has a name and a
|
||||
handler:
|
||||
|
||||
<!-- wikitool:reference-config-upload -->
|
||||
| Handler | Source | How it arrives |
|
||||
|---|---|---|
|
||||
| `mcp` | The MCP read server's optional `submit` tool | Pushed by a remote caller. At most one channel has this handler. |
|
||||
| `folder` | A directory outside the instance that a sync client fills (OneDrive and the like) | Pulled by `wikitool upload collect`; what is taken is removed from the folder. |
|
||||
| `command` | A program of this checkout, named in `commands` | Pulled by `wikitool upload collect`; the program leaves its files in an empty staging directory. |
|
||||
|
||||
All of them end in the same place: a submission nobody has looked at, which `wikitool upload accept`
|
||||
promotes into `incoming/` after a human has cleared the Upload Review Gate. What the authentication
|
||||
in front of the server has to guarantee for the `mcp` handler is in
|
||||
[Set up the MCP read server](../how-to/set-up-the-mcp-server.md#optional-the-submit-path); how to
|
||||
collect from a synced folder or a program is in
|
||||
[Collect documents from a synced folder](../how-to/collect-from-a-synced-folder.md). Every field is
|
||||
marked `gate` or `names a command`: the channels are the user's to set by hand, and `config set`
|
||||
exits 42 for them.
|
||||
|
||||
<!-- wikitool:reference-config-intake -->
|
||||
| Field | Tier | Type | Required | Default | Description |
|
||||
|-------|------|------|----------|---------|-------------|
|
||||
| `identity_header` | checkout, gate | string | no | `"X-Forwarded-User"` | The request header the authenticating middleware sets to the submitter's identity on every request. An empty value means the default. |
|
||||
| `max_bytes` | checkout, gate | positive integer | yes | - | Largest single submission, in bytes. A number written as a string is accepted. |
|
||||
| `allowed_extensions` | checkout, gate | array of strings | yes | - | The file extensions `submit` accepts, each starting with `.`. Compared case-insensitively. Entries are read as strings and lower-cased; duplicates collapse. |
|
||||
| `quota_submissions_per_day` | checkout, gate | positive integer | yes | - | Submissions one submitter may make over any rolling 24 hours. A number written as a string is accepted. |
|
||||
| `quota_bytes_per_day` | checkout, gate | positive integer | yes | - | Bytes one submitter may submit over any rolling 24 hours. A number written as a string is accepted. |
|
||||
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
|
||||
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
|
||||
|
||||
- **Absent:** The `submit` tool is not registered: the write path does not exist.
|
||||
- **Malformed:** An error. The MCP server does not start and `doctor` reports `FAIL`. A broken section is never read as "no limits".
|
||||
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
|
||||
- **Malformed:** An error. The MCP server does not start, `upload collect` stops with exit 1 and `doctor` reports `FAIL`. A broken section is never read as "no channel".
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"upload": {
|
||||
"identity_header": "X-Forwarded-User",
|
||||
"max_bytes": 10485760,
|
||||
"allowed_extensions": [
|
||||
".md",
|
||||
".pdf"
|
||||
],
|
||||
"quota_submissions_per_day": 20,
|
||||
"quota_bytes_per_day": 52428800
|
||||
"intake": {
|
||||
"channels": {
|
||||
"mcp": {
|
||||
"handler": "mcp",
|
||||
"identity_header": "X-Forwarded-User",
|
||||
"max_bytes": 10485760,
|
||||
"allowed_extensions": [
|
||||
".md",
|
||||
".pdf"
|
||||
],
|
||||
"quota_submissions_per_day": 20,
|
||||
"quota_bytes_per_day": 52428800
|
||||
},
|
||||
"onedrive": {
|
||||
"handler": "folder",
|
||||
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Eingang",
|
||||
"max_bytes": 52428800,
|
||||
"allowed_extensions": [
|
||||
".pdf",
|
||||
".docx",
|
||||
".msg",
|
||||
".eml",
|
||||
".md"
|
||||
],
|
||||
"settle_seconds": 60
|
||||
},
|
||||
"scanner": {
|
||||
"handler": "command",
|
||||
"command": "scanner-pull",
|
||||
"max_bytes": 52428800,
|
||||
"allowed_extensions": [
|
||||
".pdf"
|
||||
]
|
||||
}
|
||||
},
|
||||
"commands": {
|
||||
"scanner-pull": [
|
||||
"pwsh",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
"C:\\tools\\scanner-pull.ps1"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
<!-- /wikitool:reference-config-upload -->
|
||||
<!-- /wikitool:reference-config-intake -->
|
||||
|
||||
Size and quota are your deployment's decision, not defaults worth copying blindly.
|
||||
Size and quota are your deployment's decision, not defaults worth copying blindly. A field that does
|
||||
not belong to the channel's handler is an error, not ignored: `identity_header` and the `quota_*`
|
||||
fields belong to `mcp`, `path` and `settle_seconds` to `folder`, `command` to `command`. A `folder`
|
||||
`path` must be absolute and lie outside the instance, and `upload collect` reads only its top level.
|
||||
|
||||
## Site handlers
|
||||
|
||||
@@ -590,7 +637,7 @@ Until version 10 every feature above had a file of its own at the root: `.wikito
|
||||
`.wikitool-remotes.json`, `.wikitool-upload.json` and `.wikitool-tasks.json`. Nothing reads them any
|
||||
more. A command that needs the setting such a file held stops with exit 42 and names
|
||||
`tools/wikitool config migrate`, `doctor` reports each one as a `FAIL`, and only the telemetry hook
|
||||
ignores it. `config migrate` moves the content into the sections above, each value into the file its
|
||||
ignores it. `config migrate` moves the content into the sections above (the upload file becomes the channel `mcp` of `intake`), each value into the file its
|
||||
tier names and each secret into the secrets file, and then removes the old files; it is run once per
|
||||
checkout. `.wikitool-tools.json` is not among them and stays. The migration step for an instance is
|
||||
[10.0.0-settings-tiers.md](../../instructions/migrations/10.0.0-settings-tiers.md).
|
||||
Reference in new issue
Block a user