tools: Export- und Eingangsordner im Backup-Ziel - Überschneidungsregel nur noch auf <ziel>/git und den eigenen Maschinenordner, Laufzeitprüfung auf Vorfahren in upload collect und export run (#216); 10.1.0-beta.16

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/back-up-to-a-synced-folder.md
- docs/how-to/collect-from-a-synced-folder.md
- docs/how-to/export-to-a-synced-folder.md
- docs/reference/commands/exports.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- tools/CONTRACT.md
- tools/chemenu/backup.py
- tools/chemenu/collect.py
- tools/chemenu/commands/export_cmd.py
- tools/chemenu/commands/upload_cmd.py
- tools/chemenu/config_shapes.py
- tools/chemenu/page_export.py
- tools/chemenu/tests/test_collect.py
- tools/chemenu/tests/test_export_run.py
- tools/chemenu/upload.py

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/back-up-to-a-synced-folder.md

@@ -11,6 +11,7 @@ push to and a copy of the files on this machine that are not in git. What the se
 
 - [Why the instance does not live in the synced folder](#why-the-instance-does-not-live-in-the-synced-folder)
 - [Set it up](#set-it-up)
+- [One folder for the backup, the intake and the export](#one-folder-for-the-backup-the-intake-and-the-export)
 - [What is backed up, and what is not](#what-is-backed-up-and-what-is-not)
 - [Restore](#restore)
 - [One device writes](#one-device-writes)
@@ -91,6 +92,29 @@ still happened. Fix what the line names - a synced folder that is not there, a f
 writable - and run `tools/wikitool backup run`. Another publish does not help: it has nothing new to
 push.
 
+## One folder for the backup, the intake and the export
+
+The target may be the one folder your whole setup shares. The backup writes only into `git` and into
+the folder of its own machine; everything else in the target belongs to the sync folder, so an
+[intake folder](collect-from-a-synced-folder.md) and an
+[export folder](export-to-a-synced-folder.md) can sit beside them:
+
+```
+<OneDrive - company>\Chemenu\          backup.target
+    git\                               the bare repository (backup init, publish)
+    <MACHINE>\                         this machine's data (backup run, publish)
+    Inbox\                             intake channel, handler folder
+    Export\                            export target
+```
+
+The rule behind it: no export and no intake folder may be, or lie in, `<target>\git` or a machine
+folder. For the export that holds from above as well (its path must not contain them either); an
+intake folder may sit above them, since `upload collect` reads only the top level of its folder.
+`doctor` checks the part that the settings can show. What they cannot show - the folder of another
+machine, or one that a machine starts to fill later - is checked when the command runs: `upload
+collect` and `export run` refuse a folder that holds or lies in a `.chemenu-backup.json`, a bare
+repository or (for `upload collect`) a `.wikitool-export.json`, and touch nothing in it.
+
 ## What is backed up, and what is not
 
 | Path | Backed up | Why |

docs/how-to/collect-from-a-synced-folder.md

@@ -44,8 +44,13 @@ checkout, so it is yours to edit by hand in `.wikitool-local.json`.
 }
 ```
 
-`path` has to be absolute and lie outside the instance. Check it with `tools/wikitool doctor`, then
-pull:
+`path` has to be absolute and lie outside the instance. It may sit in the folder that also holds
+the backup and an export, as `<OneDrive - company>\Chemenu\Inbox` beside `git`, the machine folder
+and `Export` ([the layout](back-up-to-a-synced-folder.md#one-folder-for-the-backup-the-intake-and-the-export)),
+but it must not be, or lie in, `<backup.target>/git` or a machine folder of the backup. `upload
+collect` also refuses, for that channel alone and before it reads anything, a folder that holds or
+lies in a machine folder of any machine, a bare repository or an export: it empties its folder, and
+these are not submissions. Check it with `tools/wikitool doctor`, then pull:
 
 ```bash
 tools/wikitool upload collect            # every pull channel

docs/how-to/export-to-a-synced-folder.md

@@ -38,8 +38,12 @@ look like: [configuration reference](../reference/configuration.md#export-to-a-f
 
 1. **Pick the folder**: a folder inside the synced folder, for example
    `C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
-   and it must lie outside the instance, outside every other export target, outside every `intake`
-   folder and outside `backup.target`.
+   and it must lie outside the instance, outside every other export target and every `intake`
+   folder, and apart from the folders the backup writes into: it may not be, lie in or contain
+   `<backup.target>\git` or the backup's machine folder, nor lie in another machine's folder or a
+   bare repository. A folder beside them is fine, so one synced folder can hold the backup, an
+   intake folder and the export, with the export in `<OneDrive - company>\Chemenu\Export`
+   ([the layout](back-up-to-a-synced-folder.md#one-folder-for-the-backup-the-intake-and-the-export)).
 2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
    since its fields decide what leaves this machine and which program runs, and `config set`
    refuses them:

docs/reference/commands/exports.md

@@ -113,7 +113,7 @@ tools/wikitool export run work-notes
 - Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
 - Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
 - The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
-- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
+- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a folder that holds or lies in backup data (a `.chemenu-backup.json` - any machine's, also one this checkout does not know - or a bare git repository), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
 - Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
 - Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
 - A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.

docs/reference/commands/raw-material-and-uploads.md

@@ -531,6 +531,7 @@ tools/wikitool upload collect onedrive
 - Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
 - `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
 - `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
+- `folder` never empties backup or export data: a `path` that holds a `.chemenu-backup.json` (a machine's folder of the backup, any machine's) or a `.wikitool-export.json` (an export), is a bare git repository, or lies in one of these, is that channel's error - nothing is read and nothing removed. A `path` above such folders is fine, since only its top level is read.
 - Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
 - Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
 - Not idempotent: a second run finds the folder emptied.

docs/reference/configuration.md

@@ -396,7 +396,7 @@ exits 42 for them.
 <!-- wikitool:reference-config-intake -->
 | Field | Tier | Type | Required | Default | Description |
 |-------|------|------|----------|---------|-------------|
-| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
+| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories; it must not be or lie in `<backup.target>/git` or `<backup.target>/<machine>`, and `upload collect` refuses a folder that holds or lies in any machine folder, bare repository or export) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
 | `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
 
 - **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
@@ -604,7 +604,7 @@ host name for one, do reach the synced folder.
 <!-- wikitool:reference-config-backup -->
 | Field | Tier | Type | Required | Default | Description |
 |-------|------|------|----------|---------|-------------|
-| `target` | checkout, gate | string | yes | - | The backup folder: an absolute path outside the instance. `backup init` creates `<target>/git` in it, `backup run` writes only to `<target>/<machine>/`. |
+| `target` | checkout, gate | string | yes | - | The backup folder: an absolute path outside the instance. `backup init` creates `<target>/git` in it, `backup run` writes only to `<target>/<machine>/`. Everything else in the folder is the sync folder's: an `export` target or an `intake` folder may sit beside `git` and the machine folders (`<target>/Export`, `<target>/Inbox`), but none may be, or lie in, one of them. |
 | `machine_data` | checkout, gate | boolean | no | `false` | Mirror this machine's files after every `publish` that pushed, and with `backup run`: `.wikitool-local.json`, `ENVIRONMENT.md`, what waits in `incoming/`, and `extra`. |
 | `machine_name` | checkout | string | no | the host name | The name of this machine's folder under the target. Every character outside letters, digits, `-` and `_` becomes `-`, and leading and trailing `-` are dropped; a name that is empty afterwards, or `git` in any case, is an error. |
 | `extra` | checkout, gate | array of strings | no | `[]` | Further files or directories to mirror, relative to the instance. One that resolves outside the instance, or into `.git/`, is an error. |
@@ -644,7 +644,7 @@ them into a staging directory and hands that to a converter of this checkout.
 <!-- wikitool:reference-config-export -->
 | Field | Tier | Type | Required | Default | Description |
 |-------|------|------|----------|---------|-------------|
-| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target, an `intake` folder or `backup.target`) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
+| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target or an `intake` folder, and not be, lie in or contain `<backup.target>/git` or `<backup.target>/<machine>`, nor lie in any other machine's folder or a bare repository) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
 | `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
 
 - **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.