tools: Export- und Eingangsordner im Backup-Ziel - Überschneidungsregel nur noch auf <ziel>/git und den eigenen Maschinenordner, Laufzeitprüfung auf Vorfahren in upload collect und export run (#216); 10.1.0-beta.16
Files changed: - CHANGES.md - VERSION - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/export-to-a-synced-folder.md - docs/reference/commands/exports.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - tools/CONTRACT.md - tools/chemenu/backup.py - tools/chemenu/collect.py - tools/chemenu/commands/export_cmd.py - tools/chemenu/commands/upload_cmd.py - tools/chemenu/config_shapes.py - tools/chemenu/page_export.py - tools/chemenu/tests/test_collect.py - tools/chemenu/tests/test_export_run.py - tools/chemenu/upload.py Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
b48a35b224
commit
02bf3b0403
18 files changed
+312
-41
No files matched your search
+2
-1
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10.1.0-beta.15 - 2026-10-16 - tools/bugreport: eine nach dem Sammeln korrigierte Chronologie wird mit --bundle/--chronology in das vorhandene Bündel getauscht (gleiche Verarbeitung, kein zweiter Lauf, kein zweites Bündel); bug-report.md prüft die Chronologie vor dem Sammeln (#220)
|
## 10.1.0-beta.16 - 2026-10-16 - Export- und Eingangsordner dürfen im Backup-Ziel neben git/ und den Maschinenordnern liegen: die Überschneidungsregel gilt nur noch für <ziel>/git und den eigenen Maschinenordner, upload collect und export run weisen Ordner in oder unter Backup-/Export-Daten fremder Maschinen zur Laufzeit ab
|
||||||
|
|
||||||
**Author:** Torben Nehmer
|
**Author:** Torben Nehmer
|
||||||
|
|
||||||
@@ -29,6 +29,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
- export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221)
|
- export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221)
|
||||||
- wiki-ingest: ein Lauf endet nach einer Quelle mit Stopp und /clear-Empfehlung - keine Schleife über mehrere Einträge im selben Kontext; capture-session delegiert wiki-ingest nicht mehr an Subagents (#219)
|
- wiki-ingest: ein Lauf endet nach einer Quelle mit Stopp und /clear-Empfehlung - keine Schleife über mehrere Einträge im selben Kontext; capture-session delegiert wiki-ingest nicht mehr an Subagents (#219)
|
||||||
- tools/bugreport: eine nach dem Sammeln korrigierte Chronologie wird mit --bundle/--chronology in das vorhandene Bündel getauscht (gleiche Verarbeitung, kein zweiter Lauf, kein zweites Bündel); bug-report.md prüft die Chronologie vor dem Sammeln (#220)
|
- tools/bugreport: eine nach dem Sammeln korrigierte Chronologie wird mit --bundle/--chronology in das vorhandene Bündel getauscht (gleiche Verarbeitung, kein zweiter Lauf, kein zweites Bündel); bug-report.md prüft die Chronologie vor dem Sammeln (#220)
|
||||||
|
- Export- und Eingangsordner dürfen im Backup-Ziel neben git/ und den Maschinenordnern liegen: die Überschneidungsregel gilt nur noch für <ziel>/git und den eigenen Maschinenordner, upload collect und export run weisen Ordner in oder unter Backup-/Export-Daten fremder Maschinen zur Laufzeit ab
|
||||||
|
|
||||||
**Low impact**
|
**Low impact**
|
||||||
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ push to and a copy of the files on this machine that are not in git. What the se
|
|||||||
|
|
||||||
- [Why the instance does not live in the synced folder](#why-the-instance-does-not-live-in-the-synced-folder)
|
- [Why the instance does not live in the synced folder](#why-the-instance-does-not-live-in-the-synced-folder)
|
||||||
- [Set it up](#set-it-up)
|
- [Set it up](#set-it-up)
|
||||||
|
- [One folder for the backup, the intake and the export](#one-folder-for-the-backup-the-intake-and-the-export)
|
||||||
- [What is backed up, and what is not](#what-is-backed-up-and-what-is-not)
|
- [What is backed up, and what is not](#what-is-backed-up-and-what-is-not)
|
||||||
- [Restore](#restore)
|
- [Restore](#restore)
|
||||||
- [One device writes](#one-device-writes)
|
- [One device writes](#one-device-writes)
|
||||||
@@ -91,6 +92,29 @@ still happened. Fix what the line names - a synced folder that is not there, a f
|
|||||||
writable - and run `tools/wikitool backup run`. Another publish does not help: it has nothing new to
|
writable - and run `tools/wikitool backup run`. Another publish does not help: it has nothing new to
|
||||||
push.
|
push.
|
||||||
|
|
||||||
|
## One folder for the backup, the intake and the export
|
||||||
|
|
||||||
|
The target may be the one folder your whole setup shares. The backup writes only into `git` and into
|
||||||
|
the folder of its own machine; everything else in the target belongs to the sync folder, so an
|
||||||
|
[intake folder](collect-from-a-synced-folder.md) and an
|
||||||
|
[export folder](export-to-a-synced-folder.md) can sit beside them:
|
||||||
|
|
||||||
|
```
|
||||||
|
<OneDrive - company>\Chemenu\ backup.target
|
||||||
|
git\ the bare repository (backup init, publish)
|
||||||
|
<MACHINE>\ this machine's data (backup run, publish)
|
||||||
|
Inbox\ intake channel, handler folder
|
||||||
|
Export\ export target
|
||||||
|
```
|
||||||
|
|
||||||
|
The rule behind it: no export and no intake folder may be, or lie in, `<target>\git` or a machine
|
||||||
|
folder. For the export that holds from above as well (its path must not contain them either); an
|
||||||
|
intake folder may sit above them, since `upload collect` reads only the top level of its folder.
|
||||||
|
`doctor` checks the part that the settings can show. What they cannot show - the folder of another
|
||||||
|
machine, or one that a machine starts to fill later - is checked when the command runs: `upload
|
||||||
|
collect` and `export run` refuse a folder that holds or lies in a `.chemenu-backup.json`, a bare
|
||||||
|
repository or (for `upload collect`) a `.wikitool-export.json`, and touch nothing in it.
|
||||||
|
|
||||||
## What is backed up, and what is not
|
## What is backed up, and what is not
|
||||||
|
|
||||||
| Path | Backed up | Why |
|
| Path | Backed up | Why |
|
||||||
|
|||||||
@@ -44,8 +44,13 @@ checkout, so it is yours to edit by hand in `.wikitool-local.json`.
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`path` has to be absolute and lie outside the instance. Check it with `tools/wikitool doctor`, then
|
`path` has to be absolute and lie outside the instance. It may sit in the folder that also holds
|
||||||
pull:
|
the backup and an export, as `<OneDrive - company>\Chemenu\Inbox` beside `git`, the machine folder
|
||||||
|
and `Export` ([the layout](back-up-to-a-synced-folder.md#one-folder-for-the-backup-the-intake-and-the-export)),
|
||||||
|
but it must not be, or lie in, `<backup.target>/git` or a machine folder of the backup. `upload
|
||||||
|
collect` also refuses, for that channel alone and before it reads anything, a folder that holds or
|
||||||
|
lies in a machine folder of any machine, a bare repository or an export: it empties its folder, and
|
||||||
|
these are not submissions. Check it with `tools/wikitool doctor`, then pull:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tools/wikitool upload collect # every pull channel
|
tools/wikitool upload collect # every pull channel
|
||||||
|
|||||||
@@ -38,8 +38,12 @@ look like: [configuration reference](../reference/configuration.md#export-to-a-f
|
|||||||
|
|
||||||
1. **Pick the folder**: a folder inside the synced folder, for example
|
1. **Pick the folder**: a folder inside the synced folder, for example
|
||||||
`C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
|
`C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
|
||||||
and it must lie outside the instance, outside every other export target, outside every `intake`
|
and it must lie outside the instance, outside every other export target and every `intake`
|
||||||
folder and outside `backup.target`.
|
folder, and apart from the folders the backup writes into: it may not be, lie in or contain
|
||||||
|
`<backup.target>\git` or the backup's machine folder, nor lie in another machine's folder or a
|
||||||
|
bare repository. A folder beside them is fine, so one synced folder can hold the backup, an
|
||||||
|
intake folder and the export, with the export in `<OneDrive - company>\Chemenu\Export`
|
||||||
|
([the layout](back-up-to-a-synced-folder.md#one-folder-for-the-backup-the-intake-and-the-export)).
|
||||||
2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
|
2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
|
||||||
since its fields decide what leaves this machine and which program runs, and `config set`
|
since its fields decide what leaves this machine and which program runs, and `config set`
|
||||||
refuses them:
|
refuses them:
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ tools/wikitool export run work-notes
|
|||||||
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
|
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
|
||||||
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
||||||
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
||||||
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a folder that holds or lies in backup data (a `.chemenu-backup.json` - any machine's, also one this checkout does not know - or a bare git repository), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
||||||
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
||||||
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
||||||
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
||||||
|
|||||||
@@ -531,6 +531,7 @@ tools/wikitool upload collect onedrive
|
|||||||
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
||||||
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
||||||
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
||||||
|
- `folder` never empties backup or export data: a `path` that holds a `.chemenu-backup.json` (a machine's folder of the backup, any machine's) or a `.wikitool-export.json` (an export), is a bare git repository, or lies in one of these, is that channel's error - nothing is read and nothing removed. A `path` above such folders is fine, since only its top level is read.
|
||||||
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
||||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
||||||
- Not idempotent: a second run finds the folder emptied.
|
- Not idempotent: a second run finds the folder emptied.
|
||||||
|
|||||||
@@ -396,7 +396,7 @@ exits 42 for them.
|
|||||||
<!-- wikitool:reference-config-intake -->
|
<!-- wikitool:reference-config-intake -->
|
||||||
| Field | Tier | Type | Required | Default | Description |
|
| Field | Tier | Type | Required | Default | Description |
|
||||||
|-------|------|------|----------|---------|-------------|
|
|-------|------|------|----------|---------|-------------|
|
||||||
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
|
| `channels` | checkout, gate | object | yes | - | Channel name (lower-case letters, digits and hyphens) to channel. The name goes into the manifest of what the channel delivers and is the argument of `upload collect`. Every channel has `handler` (`mcp`, `folder` or `command`), `max_bytes` (the largest single file) and `allowed_extensions` (each starting with `.`, compared case-insensitively). Then per handler: `mcp` takes `identity_header` (the request header the authenticating middleware sets to the submitter's identity, default `X-Forwarded-User`) and, required, `quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any rolling 24 hours); `folder` takes `path` (required: an absolute path outside the instance, read without descending into subdirectories; it must not be or lie in `<backup.target>/git` or `<backup.target>/<machine>`, and `upload collect` refuses a folder that holds or lies in any machine folder, bare repository or export) and `settle_seconds` (default 60: a file whose modification or creation time is younger is left alone); `command` takes `command` (required: the name of an entry of `commands`). At most one channel has the handler `mcp`. A field the handler does not take is an error. |
|
||||||
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
|
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s, and a staging directory appended as the last argument. Exit 0 turns each regular file at the top level of that directory into a submission; anything else takes nothing over and prints the end of the program's stderr. |
|
||||||
|
|
||||||
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
|
- **Absent:** No channel: the `submit` tool is not registered and `upload collect` has nothing to collect. A valid state; `doctor` reports `OK`.
|
||||||
@@ -604,7 +604,7 @@ host name for one, do reach the synced folder.
|
|||||||
<!-- wikitool:reference-config-backup -->
|
<!-- wikitool:reference-config-backup -->
|
||||||
| Field | Tier | Type | Required | Default | Description |
|
| Field | Tier | Type | Required | Default | Description |
|
||||||
|-------|------|------|----------|---------|-------------|
|
|-------|------|------|----------|---------|-------------|
|
||||||
| `target` | checkout, gate | string | yes | - | The backup folder: an absolute path outside the instance. `backup init` creates `<target>/git` in it, `backup run` writes only to `<target>/<machine>/`. |
|
| `target` | checkout, gate | string | yes | - | The backup folder: an absolute path outside the instance. `backup init` creates `<target>/git` in it, `backup run` writes only to `<target>/<machine>/`. Everything else in the folder is the sync folder's: an `export` target or an `intake` folder may sit beside `git` and the machine folders (`<target>/Export`, `<target>/Inbox`), but none may be, or lie in, one of them. |
|
||||||
| `machine_data` | checkout, gate | boolean | no | `false` | Mirror this machine's files after every `publish` that pushed, and with `backup run`: `.wikitool-local.json`, `ENVIRONMENT.md`, what waits in `incoming/`, and `extra`. |
|
| `machine_data` | checkout, gate | boolean | no | `false` | Mirror this machine's files after every `publish` that pushed, and with `backup run`: `.wikitool-local.json`, `ENVIRONMENT.md`, what waits in `incoming/`, and `extra`. |
|
||||||
| `machine_name` | checkout | string | no | the host name | The name of this machine's folder under the target. Every character outside letters, digits, `-` and `_` becomes `-`, and leading and trailing `-` are dropped; a name that is empty afterwards, or `git` in any case, is an error. |
|
| `machine_name` | checkout | string | no | the host name | The name of this machine's folder under the target. Every character outside letters, digits, `-` and `_` becomes `-`, and leading and trailing `-` are dropped; a name that is empty afterwards, or `git` in any case, is an error. |
|
||||||
| `extra` | checkout, gate | array of strings | no | `[]` | Further files or directories to mirror, relative to the instance. One that resolves outside the instance, or into `.git/`, is an error. |
|
| `extra` | checkout, gate | array of strings | no | `[]` | Further files or directories to mirror, relative to the instance. One that resolves outside the instance, or into `.git/`, is an error. |
|
||||||
@@ -644,7 +644,7 @@ them into a staging directory and hands that to a converter of this checkout.
|
|||||||
<!-- wikitool:reference-config-export -->
|
<!-- wikitool:reference-config-export -->
|
||||||
| Field | Tier | Type | Required | Default | Description |
|
| Field | Tier | Type | Required | Default | Description |
|
||||||
|-------|------|------|----------|---------|-------------|
|
|-------|------|------|----------|---------|-------------|
|
||||||
| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target, an `intake` folder or `backup.target`) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
|
| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target or an `intake` folder, and not be, lie in or contain `<backup.target>/git` or `<backup.target>/<machine>`, nor lie in any other machine's folder or a bare repository) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
|
||||||
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
|
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
|
||||||
|
|
||||||
- **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
|
- **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
|
||||||
|
|||||||
+2
-1
@@ -1913,6 +1913,7 @@ Pull material from the configured `folder` and `command` intake channels into th
|
|||||||
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
- Everything a channel delivers lands in `quarantine/<id>/` unchanged, with a manifest that names `channel`, `handler` and `origin`; the submitter is `unknown`. Nothing reaches `incoming/` or `raw/` - that is `upload accept`, behind its gate, and an agent may run `collect` because collecting gives no trust.
|
||||||
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
- `folder`: only the top level of `path`. A file is taken once its modification time and, where the platform has it, its creation time are older than `settle_seconds`, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with `.` or `~$`, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
|
||||||
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
- `command`: the program of `commands.<name>` runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under `quarantine/.staging/` as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
|
||||||
|
- `folder` never empties backup or export data: a `path` that holds a `.chemenu-backup.json` (a machine's folder of the backup, any machine's) or a `.wikitool-export.json` (an export), is a bare git repository, or lies in one of these, is that channel's error - nothing is read and nothing removed. A `path` above such folders is fine, since only its top level is read.
|
||||||
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
- Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints `nothing to collect` and exits 0.
|
||||||
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
- Exits 1, before it does anything, while an `mcp-upload/` of an earlier version still holds submissions, and when the `intake` section is malformed.
|
||||||
- Not idempotent: a second run finds the folder emptied.
|
- Not idempotent: a second run finds the folder emptied.
|
||||||
@@ -2459,7 +2460,7 @@ Write the derived, read-only reading copy of `kb/` into the folders the `export`
|
|||||||
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
|
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
|
||||||
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
||||||
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
||||||
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a folder that holds or lies in backup data (a `.chemenu-backup.json` - any machine's, also one this checkout does not know - or a bare git repository), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
||||||
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
||||||
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
||||||
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
||||||
|
|||||||
@@ -209,6 +209,47 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
|
|||||||
machine_name=machine_name, extra=tuple(section["extra"]))
|
machine_name=machine_name, extra=tuple(section["extra"]))
|
||||||
|
|
||||||
|
|
||||||
|
def written_folders(root: "Path | str | None" = None) -> list[tuple[str, Path]]:
|
||||||
|
"""The folders this instance's backup writes into - `<target>/git` and
|
||||||
|
`<target>/<own machine>` - each with a label for a message. Whatever else
|
||||||
|
lies in `<target>` is the sync folder's, not the backup's, so an export or
|
||||||
|
an intake folder may sit beside these two. Read as leniently as the other
|
||||||
|
features read each other's folders: a broken section yields none, it is
|
||||||
|
reported where it is read (`doctor`'s `backup` check)."""
|
||||||
|
try:
|
||||||
|
cfg = read_config(root)
|
||||||
|
except ValidationError:
|
||||||
|
return []
|
||||||
|
if cfg is None:
|
||||||
|
return []
|
||||||
|
return [(f"the backup's git folder ({cfg.git_dir})", cfg.git_dir),
|
||||||
|
(f"the backup's machine folder ({cfg.machine_dir})", cfg.machine_dir)]
|
||||||
|
|
||||||
|
|
||||||
|
def _is_bare_repo_dir(folder: Path) -> bool:
|
||||||
|
return (folder / "HEAD").is_file() and (folder / "objects").is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def data_marker(folder: Path, extra_manifests: tuple[str, ...] = ()) -> Optional[str]:
|
||||||
|
"""What marks `folder`, or one of its ancestors, as data that belongs to the
|
||||||
|
backup - a machine folder (the backup manifest), a bare repository - or to
|
||||||
|
a manifest named in `extra_manifests`; None when nothing does. The
|
||||||
|
configuration cannot see another machine's folder, or one a machine starts
|
||||||
|
to fill later, so a run asks the folder itself; the ancestors cover a
|
||||||
|
folder inside one. Looked at through links, and without a `git` call."""
|
||||||
|
resolved = real_path(folder)
|
||||||
|
for candidate in (resolved, *resolved.parents):
|
||||||
|
try:
|
||||||
|
for name in (MANIFEST_NAME, *extra_manifests):
|
||||||
|
if (candidate / name).is_file():
|
||||||
|
return f"{candidate} holds {name}"
|
||||||
|
if _is_bare_repo_dir(candidate):
|
||||||
|
return f"{candidate} is a git repository"
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def require_config(root: "Path | str | None" = None) -> BackupConfig:
|
def require_config(root: "Path | str | None" = None) -> BackupConfig:
|
||||||
found = read_config(root)
|
found = read_config(root)
|
||||||
if found is None:
|
if found is None:
|
||||||
|
|||||||
@@ -19,6 +19,9 @@ What a pull guarantees:
|
|||||||
in the channel's `path`. It writes nothing there, deletes no directory and
|
in the channel's `path`. It writes nothing there, deletes no directory and
|
||||||
follows no symbolic link.
|
follows no symbolic link.
|
||||||
- Whatever it does not take stays where it is and is reported with its reason.
|
- Whatever it does not take stays where it is and is reported with its reason.
|
||||||
|
- It never empties data of the backup or of an export: a folder that holds a
|
||||||
|
backup or export manifest or is a bare repository, or lies in one, is the
|
||||||
|
channel's error - nothing read, nothing removed.
|
||||||
|
|
||||||
No `typer`, no `rich`: the CLI renders a `ChannelResult`.
|
No `typer`, no `rich`: the CLI renders a `ChannelResult`.
|
||||||
"""
|
"""
|
||||||
@@ -33,7 +36,7 @@ from dataclasses import dataclass, field
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Callable, Optional
|
from typing import Callable, Optional
|
||||||
|
|
||||||
from chemenu import upload
|
from chemenu import backup, page_export, upload
|
||||||
from chemenu.errors import ValidationError
|
from chemenu.errors import ValidationError
|
||||||
|
|
||||||
COMMAND_TIMEOUT_SECONDS = 300
|
COMMAND_TIMEOUT_SECONDS = 300
|
||||||
@@ -170,6 +173,11 @@ def collect_folder(root: "Path | str", channel: upload.Channel, now: Optional[fl
|
|||||||
if folder is None or not folder.is_dir():
|
if folder is None or not folder.is_dir():
|
||||||
result.error = f"the folder {folder} does not exist or is not a directory"
|
result.error = f"the folder {folder} does not exist or is not a directory"
|
||||||
return result
|
return result
|
||||||
|
marker = backup.data_marker(folder, (page_export.MANIFEST_NAME,))
|
||||||
|
if marker:
|
||||||
|
result.error = (f"the folder {folder} lies in backup or export data ({marker}) - a channel empties "
|
||||||
|
"its folder, so nothing was read or removed")
|
||||||
|
return result
|
||||||
clock = time.time() if now is None else now
|
clock = time.time() if now is None else now
|
||||||
try:
|
try:
|
||||||
entries = sorted(os.scandir(folder), key=lambda entry: entry.name)
|
entries = sorted(os.scandir(folder), key=lambda entry: entry.name)
|
||||||
|
|||||||
@@ -483,7 +483,9 @@ def export_guidelines_command(
|
|||||||
"nothing - not even a modification time - for a file that did not change. A run that "
|
"nothing - not even a modification time - for a file that did not change. A run that "
|
||||||
"changes nothing writes nothing at all.",
|
"changes nothing writes nothing at all.",
|
||||||
"Refuses a target, before writing anything into it: a non-empty folder with no manifest "
|
"Refuses a target, before writing anything into it: a non-empty folder with no manifest "
|
||||||
"(it is somebody's, not an export), a manifest file that was changed in the folder since "
|
"(it is somebody's, not an export), a folder that holds or lies in backup data (a "
|
||||||
|
"`.chemenu-backup.json` - any machine's, also one this checkout does not know - or a "
|
||||||
|
"bare git repository), a manifest file that was changed in the folder since "
|
||||||
"the last run (naming it), a foreign file where an exported one belongs, an empty "
|
"the last run (naming it), a foreign file where an exported one belongs, an empty "
|
||||||
"selection (it would delete the earlier export), an unknown field in a predicate. A "
|
"selection (it would delete the earlier export), an unknown field in a predicate. A "
|
||||||
"missing folder is created.",
|
"missing folder is created.",
|
||||||
|
|||||||
@@ -322,6 +322,11 @@ def _result_lines(result: collect_module.ChannelResult) -> list[str]:
|
|||||||
"argument. On exit 0 each regular file at the top of that directory with an allowed "
|
"argument. On exit 0 each regular file at the top of that directory with an allowed "
|
||||||
"extension and size becomes a submission; on anything else nothing is taken and the end of "
|
"extension and size becomes a submission; on anything else nothing is taken and the end of "
|
||||||
"stderr is printed. The staging directory is always removed.",
|
"stderr is printed. The staging directory is always removed.",
|
||||||
|
"`folder` never empties backup or export data: a `path` that holds a `.chemenu-backup.json` "
|
||||||
|
"(a machine's folder of the backup, any machine's) or a `.wikitool-export.json` (an "
|
||||||
|
"export), is a bare git repository, or lies in one of these, is that channel's error - "
|
||||||
|
"nothing is read and nothing removed. A `path` above such folders is fine, since only "
|
||||||
|
"its top level is read.",
|
||||||
"Channels are independent: when one fails the others still run, and the exit status is 1 "
|
"Channels are independent: when one fails the others still run, and the exit status is 1 "
|
||||||
"if any channel or file failed. With no pulled channel it prints `nothing to collect` "
|
"if any channel or file failed. With no pulled channel it prints `nothing to collect` "
|
||||||
"and exits 0.",
|
"and exits 0.",
|
||||||
|
|||||||
@@ -494,7 +494,10 @@ INTAKE = Shape(
|
|||||||
f"submitter's identity, default `{INTAKE_DEFAULT_IDENTITY_HEADER}`) and, required, "
|
f"submitter's identity, default `{INTAKE_DEFAULT_IDENTITY_HEADER}`) and, required, "
|
||||||
"`quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any "
|
"`quota_submissions_per_day` and `quota_bytes_per_day` (per submitter, over any "
|
||||||
"rolling 24 hours); `folder` takes `path` (required: an absolute path outside the "
|
"rolling 24 hours); `folder` takes `path` (required: an absolute path outside the "
|
||||||
"instance, read without descending into subdirectories) and `settle_seconds` (default "
|
"instance, read without descending into subdirectories; it must not be or lie in "
|
||||||
|
"`<backup.target>/git` or `<backup.target>/<machine>`, and `upload collect` refuses "
|
||||||
|
"a folder that holds or lies in any machine folder, bare repository or export) "
|
||||||
|
"and `settle_seconds` (default "
|
||||||
f"{INTAKE_SETTLE_SECONDS}: a file whose modification or creation time is younger is "
|
f"{INTAKE_SETTLE_SECONDS}: a file whose modification or creation time is younger is "
|
||||||
"left alone); `command` takes `command` (required: the name of an entry of "
|
"left alone); `command` takes `command` (required: the name of an entry of "
|
||||||
"`commands`). At most one channel has the handler `mcp`. A field the handler does not "
|
"`commands`). At most one channel has the handler `mcp`. A field the handler does not "
|
||||||
@@ -775,7 +778,9 @@ BACKUP = Shape(
|
|||||||
Field(
|
Field(
|
||||||
"target", "string", "The backup folder: an absolute path outside the instance. "
|
"target", "string", "The backup folder: an absolute path outside the instance. "
|
||||||
"`backup init` creates `<target>/git` in it, `backup run` writes only to "
|
"`backup init` creates `<target>/git` in it, `backup run` writes only to "
|
||||||
"`<target>/<machine>/`.",
|
"`<target>/<machine>/`. Everything else in the folder is the sync folder's: an "
|
||||||
|
"`export` target or an `intake` folder may sit beside `git` and the machine folders "
|
||||||
|
"(`<target>/Export`, `<target>/Inbox`), but none may be, or lie in, one of them.",
|
||||||
required=True, json_type="string", coerce=exact_str, check=absolute_path_problem,
|
required=True, json_type="string", coerce=exact_str, check=absolute_path_problem,
|
||||||
tier=CHECKOUT, gate=True,
|
tier=CHECKOUT, gate=True,
|
||||||
),
|
),
|
||||||
@@ -913,7 +918,9 @@ EXPORT = Shape(
|
|||||||
"the name is the argument of `export run`. Every target has `handler` (`markdown`: the "
|
"the name is the argument of `export run`. Every target has `handler` (`markdown`: the "
|
||||||
"pages as Markdown files; `command`: the same Markdown run through a converter of this "
|
"pages as Markdown files; `command`: the same Markdown run through a converter of this "
|
||||||
"checkout), `path` (an absolute path outside the instance; it must not lie in or "
|
"checkout), `path` (an absolute path outside the instance; it must not lie in or "
|
||||||
"contain another target, an `intake` folder or `backup.target`) and `include` (an "
|
"contain another target or an `intake` folder, and not be, lie in or contain "
|
||||||
|
"`<backup.target>/git` or `<backup.target>/<machine>`, nor lie in any other "
|
||||||
|
"machine's folder or a bare repository) and `include` (an "
|
||||||
"array of selections, each an array of `search --field` predicates: all predicates of "
|
"array of selections, each an array of `search --field` predicates: all predicates of "
|
||||||
"a selection must hold, and a page that fits any selection is exported; there is no "
|
"a selection must hold, and a page that fits any selection is exported; there is no "
|
||||||
"\"everything\"). Optional: `frontmatter` (`minimal`: only the export marker; "
|
"\"everything\"). Optional: `frontmatter` (`minimal`: only the export marker; "
|
||||||
|
|||||||
@@ -81,8 +81,8 @@ class Target:
|
|||||||
|
|
||||||
def _others_paths(root: Path) -> list[tuple[str, Path]]:
|
def _others_paths(root: Path) -> list[tuple[str, Path]]:
|
||||||
"""The folders other features of this checkout write into or read from -
|
"""The folders other features of this checkout write into or read from -
|
||||||
the `intake` folders and the backup target - for the conflict check. A
|
the `intake` folders and the two folders the backup writes into - for the
|
||||||
section that is itself broken is that feature's own problem and is skipped
|
conflict check. A section that is itself broken is that feature's own problem and is skipped
|
||||||
here: it is reported where it is read."""
|
here: it is reported where it is read."""
|
||||||
from chemenu import upload
|
from chemenu import upload
|
||||||
|
|
||||||
@@ -93,12 +93,7 @@ def _others_paths(root: Path) -> list[tuple[str, Path]]:
|
|||||||
found.append((f"the intake channel {name!r} ({channel.path})", channel.path))
|
found.append((f"the intake channel {name!r} ({channel.path})", channel.path))
|
||||||
except ValidationError:
|
except ValidationError:
|
||||||
pass
|
pass
|
||||||
try:
|
found.extend(backup.written_folders(root))
|
||||||
backup_cfg = backup.read_config(root)
|
|
||||||
except ValidationError:
|
|
||||||
backup_cfg = None
|
|
||||||
if backup_cfg is not None:
|
|
||||||
found.append((f"backup.target ({backup_cfg.target})", backup_cfg.target))
|
|
||||||
return found
|
return found
|
||||||
|
|
||||||
|
|
||||||
@@ -121,7 +116,7 @@ def read_config(root: "Path | str | None" = None) -> Optional[tuple[Target, ...]
|
|||||||
that is broken - including what the record cannot judge without the
|
that is broken - including what the record cannot judge without the
|
||||||
instance and the rest of the settings: a predicate that does not parse, a
|
instance and the rest of the settings: a predicate that does not parse, a
|
||||||
path on this machine that is not absolute, inside the instance, or in the
|
path on this machine that is not absolute, inside the instance, or in the
|
||||||
way of another target, an `intake` folder or the backup target."""
|
way of another target, an `intake` folder or a folder the backup writes into."""
|
||||||
root = Path(config.ROOT if root is None else root)
|
root = Path(config.ROOT if root is None else root)
|
||||||
key = config_shapes.EXPORT.key
|
key = config_shapes.EXPORT.key
|
||||||
section = settings.load(root, only=(key,)).section(key)
|
section = settings.load(root, only=(key,)).section(key)
|
||||||
@@ -406,6 +401,10 @@ def mirror(folder: Path, files: dict[str, bytes], result: TargetResult, *, name:
|
|||||||
"so nothing was changed")
|
"so nothing was changed")
|
||||||
if folder.exists() and not folder.is_dir():
|
if folder.exists() and not folder.is_dir():
|
||||||
raise ExportError(f"{folder} exists and is not a folder")
|
raise ExportError(f"{folder} exists and is not a folder")
|
||||||
|
marker = backup.data_marker(folder)
|
||||||
|
if marker:
|
||||||
|
raise ExportError(f"{folder} lies in backup data ({marker}) - an export belongs in a folder of its "
|
||||||
|
"own, beside the backup's. Nothing was written.")
|
||||||
manifest = read_manifest(folder) if folder.is_dir() else None
|
manifest = read_manifest(folder) if folder.is_dir() else None
|
||||||
if folder.is_dir() and manifest is None and any(folder.iterdir()):
|
if folder.is_dir() and manifest is None and any(folder.iterdir()):
|
||||||
raise ExportError(
|
raise ExportError(
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import pytest
|
|||||||
import typer
|
import typer
|
||||||
|
|
||||||
from chemenu import collect, config, upload
|
from chemenu import collect, config, upload
|
||||||
|
from chemenu.errors import ValidationError
|
||||||
from chemenu.commands.upload_cmd import upload_collect_command, upload_list_command
|
from chemenu.commands.upload_cmd import upload_collect_command, upload_list_command
|
||||||
|
|
||||||
OLD = time.time() - 3600
|
OLD = time.time() - 3600
|
||||||
@@ -295,3 +296,90 @@ def test_collected_material_shows_in_no_git_status_even_without_a_root_entry(roo
|
|||||||
collect.collect(root, _command_channel(_script("(out / 'm.md').write_bytes(b'mail')")))
|
collect.collect(root, _command_channel(_script("(out / 'm.md').write_bytes(b'mail')")))
|
||||||
assert (root / "quarantine" / ".gitignore").read_text().strip() == "*"
|
assert (root / "quarantine" / ".gitignore").read_text().strip() == "*"
|
||||||
assert _git(root, "status", "--porcelain", "--untracked-files=all") == ""
|
assert _git(root, "status", "--porcelain", "--untracked-files=all") == ""
|
||||||
|
|
||||||
|
|
||||||
|
# --- backup and export data -------------------------------------------------
|
||||||
|
|
||||||
|
def _backup_target(root: Path, tmp_path: Path) -> Path:
|
||||||
|
shared = tmp_path / "shared"
|
||||||
|
(shared / "git").mkdir(parents=True)
|
||||||
|
body = {"backup": {"target": str(shared), "machine_data": False, "machine_name": "WORK-LAPTOP"}}
|
||||||
|
(root / config.LOCAL_SETTINGS_FILENAME).write_text(json.dumps({"schema": 1, **body}), encoding="utf-8")
|
||||||
|
return shared
|
||||||
|
|
||||||
|
|
||||||
|
def _channel_at(root: Path, folder: Path) -> None:
|
||||||
|
path = root / config.LOCAL_SETTINGS_FILENAME
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8")) if path.exists() else {"schema": 1}
|
||||||
|
data["intake"] = {"channels": {"inbox": {
|
||||||
|
"handler": "folder", "path": str(folder), "max_bytes": 1024, "allowed_extensions": [".md"]}}}
|
||||||
|
path.write_text(json.dumps(data), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("where", ["git", "git/inner", "WORK-LAPTOP", "WORK-LAPTOP/inner"])
|
||||||
|
def test_an_intake_folder_equal_to_or_in_a_folder_the_backup_writes_is_refused(root, tmp_path, where):
|
||||||
|
shared = _backup_target(root, tmp_path)
|
||||||
|
_channel_at(root, shared / where)
|
||||||
|
with pytest.raises(ValidationError, match="the backup's"):
|
||||||
|
upload.read_channels(root)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_intake_folder_beside_or_over_the_backup_folders_is_valid(root, tmp_path):
|
||||||
|
shared = _backup_target(root, tmp_path)
|
||||||
|
for folder in (shared / "Inbox", shared, shared.parent):
|
||||||
|
_channel_at(root, folder)
|
||||||
|
assert upload.read_channels(root)["inbox"].path == folder
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_broken_backup_section_does_not_break_the_intake_section(root, tmp_path):
|
||||||
|
_backup_target(root, tmp_path)
|
||||||
|
path = root / config.LOCAL_SETTINGS_FILENAME
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
data["backup"] = {"target": "relative/path"}
|
||||||
|
path.write_text(json.dumps(data), encoding="utf-8")
|
||||||
|
_channel_at(root, tmp_path / "shared" / "git")
|
||||||
|
assert upload.read_channels(root)["inbox"].path == tmp_path / "shared" / "git"
|
||||||
|
|
||||||
|
|
||||||
|
def _backup_manifest(folder: Path) -> None:
|
||||||
|
(folder / ".chemenu-backup.json").write_text("{}", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _bare_repository(folder: Path) -> None:
|
||||||
|
(folder / "objects").mkdir()
|
||||||
|
(folder / "HEAD").write_text("ref: refs/heads/main\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _export_manifest(folder: Path) -> None:
|
||||||
|
(folder / ".wikitool-export.json").write_text("{}", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mark, nested", [
|
||||||
|
(_backup_manifest, False), (_backup_manifest, True),
|
||||||
|
(_bare_repository, False), (_bare_repository, True),
|
||||||
|
(_export_manifest, False), (_export_manifest, True),
|
||||||
|
], ids=["machine", "in-machine", "bare", "in-bare", "export", "in-export"])
|
||||||
|
def test_a_pull_from_backup_or_export_data_fails_its_channel_and_leaves_the_folder_as_it_was(
|
||||||
|
root, drop, mark, nested):
|
||||||
|
base = drop / "inner" if nested else drop
|
||||||
|
base.mkdir(exist_ok=True)
|
||||||
|
mark(drop)
|
||||||
|
_put(base, "ENVIRONMENT.md", b"machine data\n")
|
||||||
|
before = _tree(drop)
|
||||||
|
result = collect.collect(root, _folder_channel(base))
|
||||||
|
assert result.error and "lies in" in result.error
|
||||||
|
assert not (result.taken or result.failed or result.skipped)
|
||||||
|
assert _tree(drop) == before
|
||||||
|
assert _manifests(root) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_other_channels_still_run_when_one_lies_in_backup_data(root, drop, tmp_path, capsys):
|
||||||
|
_two_channels(root, drop)
|
||||||
|
_backup_manifest(drop)
|
||||||
|
_put(drop, "a.md", b"alpha\n")
|
||||||
|
before = _tree(drop)
|
||||||
|
with pytest.raises(typer.Exit) as excinfo:
|
||||||
|
upload_collect_command(names=None)
|
||||||
|
assert excinfo.value.exit_code == 1
|
||||||
|
assert _tree(drop) == before
|
||||||
|
assert {m["channel"] for m in _manifests(root)} == {"mail"}
|
||||||
@@ -191,21 +191,97 @@ def test_targets_lying_in_one_another_are_refused(instance):
|
|||||||
assert "lie in one another" in _flat(result.output)
|
assert "lie in one another" in _flat(result.output)
|
||||||
|
|
||||||
|
|
||||||
def test_a_target_in_the_backup_target_or_an_intake_folder_is_refused(instance):
|
def _local(root: Path) -> dict:
|
||||||
backup_target = _out(instance, "backup")
|
return json.loads((root / config.LOCAL_SETTINGS_FILENAME).read_text(encoding="utf-8"))
|
||||||
path = instance / config.LOCAL_SETTINGS_FILENAME
|
|
||||||
_configure(instance, pages={"path": str(backup_target / "pages")})
|
|
||||||
data = json.loads(path.read_text(encoding="utf-8"))
|
def _save(root: Path, data: dict) -> None:
|
||||||
data["backup"] = {"target": str(backup_target)}
|
(root / config.LOCAL_SETTINGS_FILENAME).write_text(json.dumps(data), encoding="utf-8")
|
||||||
path.write_text(json.dumps(data), encoding="utf-8")
|
|
||||||
assert _run().exit_code == 1
|
|
||||||
data.pop("backup")
|
def _backup_section(root: Path) -> Path:
|
||||||
|
"""A backup whose target is the shared folder itself; returns that folder."""
|
||||||
|
shared = _out(root, "Chemenu")
|
||||||
|
data = _local(root)
|
||||||
|
data["backup"] = {"target": str(shared), "machine_data": False, "machine_name": "WORK-LAPTOP"}
|
||||||
|
_save(root, data)
|
||||||
|
return shared
|
||||||
|
|
||||||
|
|
||||||
|
def _channel(root: Path, path: Path) -> None:
|
||||||
|
data = _local(root)
|
||||||
data["intake"] = {"channels": {"inbox": {
|
data["intake"] = {"channels": {"inbox": {
|
||||||
"handler": "folder", "path": str(_out(instance, "pages") / "in"), "max_bytes": 1024,
|
"handler": "folder", "path": str(path), "max_bytes": 1024, "allowed_extensions": [".md"]}}}
|
||||||
"allowed_extensions": [".md"]}}}
|
_save(root, data)
|
||||||
data["export"]["targets"]["pages"]["path"] = str(_out(instance, "pages"))
|
|
||||||
path.write_text(json.dumps(data), encoding="utf-8")
|
|
||||||
|
def test_export_intake_and_backup_may_share_one_synced_folder(instance):
|
||||||
|
shared = _out(instance, "Chemenu")
|
||||||
|
_configure(instance, pages={"path": str(shared / "Export")})
|
||||||
|
(shared / "Inbox").mkdir(parents=True)
|
||||||
|
_backup_section(instance)
|
||||||
|
_channel(instance, shared / "Inbox")
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert (shared / "Export" / ".wikitool-export.json").is_file()
|
||||||
|
for check in (doctor.check_export(), doctor.check_upload_intake(), doctor.check_backup()):
|
||||||
|
assert check.status != "FAIL", (check.name, check.detail)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("where", ["git", "git/inner", "WORK-LAPTOP", "WORK-LAPTOP/inner", ""])
|
||||||
|
def test_an_export_equal_to_in_or_over_a_folder_the_backup_writes_is_refused(instance, where):
|
||||||
|
shared = _out(instance, "Chemenu")
|
||||||
|
_configure(instance, pages={"path": str(shared / where) if where else str(shared)})
|
||||||
|
_backup_section(instance)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "lie in one another" in _flat(result.output)
|
||||||
|
assert "the backup's" in _flat(result.output)
|
||||||
|
assert not (shared / "git").exists()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("where", ["git", "git/inner", "WORK-LAPTOP", "WORK-LAPTOP/inner"])
|
||||||
|
def test_doctor_fails_an_intake_folder_in_a_folder_the_backup_writes(instance, where):
|
||||||
|
shared = _out(instance, "Chemenu")
|
||||||
|
_configure(instance, pages={"path": str(shared / "Export")})
|
||||||
|
_backup_section(instance)
|
||||||
|
_channel(instance, shared / where)
|
||||||
|
check = doctor.check_upload_intake()
|
||||||
|
assert check.status == "FAIL" and "the backup's" in check.detail
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_in_an_intake_folder_is_refused(instance):
|
||||||
|
_configure(instance, pages={"path": str(_out(instance, "pages"))})
|
||||||
|
_channel(instance, _out(instance, "pages") / "in")
|
||||||
assert _run().exit_code == 1
|
assert _run().exit_code == 1
|
||||||
|
_channel(instance, _out(instance, "pages").parent)
|
||||||
|
assert _run().exit_code == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_below_another_machines_folder_is_refused_before_anything_is_written(instance):
|
||||||
|
shared = _out(instance, "Chemenu")
|
||||||
|
foreign = shared / "OTHER-PC"
|
||||||
|
foreign.mkdir(parents=True)
|
||||||
|
(foreign / ".chemenu-backup.json").write_text("{}", encoding="utf-8")
|
||||||
|
_configure(instance, pages={"path": str(foreign / "deeper" / "Export")})
|
||||||
|
_backup_section(instance)
|
||||||
|
before = {p: p.read_bytes() for p in shared.rglob("*") if p.is_file()}
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "backup data" in _flat(result.output)
|
||||||
|
assert {p: p.read_bytes() for p in shared.rglob("*") if p.is_file()} == before
|
||||||
|
assert not (foreign / "deeper").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_in_a_bare_repository_is_refused(instance):
|
||||||
|
bare = _out(instance, "repo.git")
|
||||||
|
(bare / "objects").mkdir(parents=True)
|
||||||
|
(bare / "HEAD").write_text("ref: refs/heads/main\n", encoding="utf-8")
|
||||||
|
_configure(instance, pages={"path": str(bare / "Export")})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "is a git repository" in _flat(result.output)
|
||||||
|
assert not (bare / "Export").exists()
|
||||||
|
|
||||||
|
|
||||||
def test_the_operator_only_fields_cannot_be_set_by_the_agent():
|
def test_the_operator_only_fields_cannot_be_set_by_the_agent():
|
||||||
|
|||||||
+12
-3
@@ -102,9 +102,18 @@ def _folder_problem(name: str, raw: str, root: "Path | str") -> Optional[str]:
|
|||||||
try:
|
try:
|
||||||
resolved.relative_to(real_root)
|
resolved.relative_to(real_root)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return None
|
pass
|
||||||
return (f"intake.channels.{name}.path {raw!r} lies inside the instance - a folder the channel "
|
else:
|
||||||
"empties belongs outside it")
|
return (f"intake.channels.{name}.path {raw!r} lies inside the instance - a folder the channel "
|
||||||
|
"empties belongs outside it")
|
||||||
|
# Imported here: the MCP server process imports this module, and only a folder channel needs it.
|
||||||
|
from chemenu import backup
|
||||||
|
|
||||||
|
for label, written in backup.written_folders(root):
|
||||||
|
if backup.is_under(resolved, backup.real_path(written)):
|
||||||
|
return (f"intake.channels.{name}.path {raw!r} is, or lies in, {label} - a folder the channel "
|
||||||
|
"empties belongs beside the backup's folders, not in them")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def read_channels(root: "Path | str") -> dict[str, Channel]:
|
def read_channels(root: "Path | str") -> dict[str, Channel]:
|
||||||
|
|||||||
Reference in new issue
Block a user