torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

export: export run writes selected kb/ pages as a derived, read-only version into configured target folders (markdown and command handlers, manifest, after_publish), raw accept also refuses manifest-listed files, doctor checks the section (#214); 10.0.0-beta.13
CI / verify (push) Successful in 9m2s
Details
CI / pwsh (push) Successful in 2m48s
Details
Release / release (push) Successful in 34s
Details
Release / wiki-mirror (push) Skipped
Details

Browse Source 
Files changed:
- CHANGES.md
- VERSION
- docs/how-to/README.md
- docs/how-to/back-up-to-a-synced-folder.md
- docs/how-to/collect-from-a-synced-folder.md
- docs/how-to/export-guidelines.md
- docs/how-to/export-to-a-synced-folder.md
- docs/reference/commands/README.md
- docs/reference/commands/exports.md
- docs/reference/commands/git.md
- docs/reference/commands/guideline-export.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/setup-instance.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/backup.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/export_cmd.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/commands/raw/_common.py
- tools/chemenu/commands/raw/accept.py
- tools/chemenu/commands/raw/capture.py
- tools/chemenu/config_shapes.py
- tools/chemenu/guideline_export.py
- tools/chemenu/page_export.py
- tools/chemenu/reference_doc.py
- tools/chemenu/repo_capture.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_export_run.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-13 05:53:48 +02:00
1 parent ecefe0a033
commit 99945c4841
33 files changed
+2316 -96
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGES.md
+2 -1
View File
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
---
## 10.0.0-beta.12 - 2026-10-12 - AGENTS.md § Gates and mcp-read-server name every intake channel behind the Upload Review Gate, not only submit (#213 close)
## 10.0.0-beta.13 - 2026-10-12 - Exporte als konfigurierte Ziele: export run schreibt eine abgeleitete, schreibgeschützte Lesefassung von kb/ je Zielordner (Markdown oder über ein Konvertierprogramm), auf Wunsch nach jedem publish; raw accept lehnt auch Dateien ab, die in einem Export-Manifest stehen
**Author:** Torben Nehmer
@@ -37,6 +37,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
- raw fetch hinter einem Login: raw login speichert die Session des Nutzers als Secret, raw fetch holt damit; gerenderte Seiten über die optionale Browser-Komponente, externe Abrufer pro Host
- Gesprächssprache: AGENTS.md § Conversation language legt fest, dass der Agent im Chat die KB-Sprache spricht; instructions sync schreibt den Sprachwert als generierte Zeile in jedes veröffentlichte SKILL.md, instructions verify und doctor melden eine geänderte Sprache als Drift
- Backup in einen Sync-Ordner (OneDrive): backup init legt ein Bare-Repo unter <ziel>/git als origin an, backup run und publish spiegeln die Maschinendaten nach <ziel>/<maschine>/, doctor warnt vor .git im Sync-Ordner
- Exporte als konfigurierte Ziele: export run schreibt eine abgeleitete, schreibgeschützte Lesefassung von kb/ je Zielordner (Markdown oder über ein Konvertierprogramm), auf Wunsch nach jedem publish; raw accept lehnt auch Dateien ab, die in einem Export-Manifest stehen
**Low impact**
- Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
VERSION
+1 -1
View File
@@ -1 +1 @@
10.0.0-beta.12
10.0.0-beta.13
docs/how-to/README.md
+2
View File
@@ -15,6 +15,8 @@ Working with the wiki:
browser, or through a fetch program of your own
- [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
- [Export guidelines to captured repositories](export-guidelines.md)
- [Export to a synced folder](export-to-a-synced-folder.md) - a read-only version of selected pages
for a reader outside the repository, OneDrive and Microsoft 365 included
- [Query the wiki](query-the-wiki.md) - asking, and searching yourself
- [Maintain the wiki](maintain-the-wiki.md) - lint, provenance, and the gates on the way to a publish
- [Run the weekly review](run-the-weekly-review.md) - commitments in a task tracker against the
docs/how-to/back-up-to-a-synced-folder.md
+4
View File
@@ -156,3 +156,7 @@ Only one device publishes to `<target>/git`. Two devices pushing into the same s
writers again, only one step removed - and the sync client resolves that with conflict copies, not
with a merge. An instance on a second device gets a target of its own. The sync client is a backup
here, not a way to work on one instance from two devices.
A folder the sync client uploads can also hold a read-only version of selected pages, for readers
outside the repository: [Export pages to a synced folder](export-to-a-synced-folder.md). It is a
separate folder with its own settings, and never the backup target.
docs/how-to/collect-from-a-synced-folder.md
+4
View File
@@ -131,3 +131,7 @@ exits 0 with "nothing to collect".
The agent does the same at the start of an ingest when you name no file: `upload collect`, then
`upload list`, then a report of what waits. It never runs `upload accept` for you.
The other direction - a read-only version of selected pages written out to a synced folder - is
[Export pages to a synced folder](export-to-a-synced-folder.md). An export is never a source: the
export marker keeps it out of `raw/`.
docs/how-to/export-guidelines.md
+3 -1
View File
@@ -50,5 +50,7 @@ did not see.
## Related
- [Guideline export commands](../reference/commands/guideline-export.md)
- [Export commands](../reference/commands/exports.md)
- [Export pages to a synced folder](export-to-a-synced-folder.md) - the other export, a read-only
version of `kb/` pages
- [Why the gates are code](../explanation/why-gates-are-code.md)
docs/how-to/export-to-a-synced-folder.md
+155
View File
@@ -0,0 +1,155 @@
# Export to a synced folder
Give a reader outside the repository a read-only version of part of `kb/`: the selected pages,
written as files into a folder - typically one a sync client such as OneDrive uploads, so that a
Microsoft 365 tenant can index them. The export is derived. It is regenerated from the committed
pages on every run, nobody edits it, and nothing in it flows back into the wiki. What the settings
look like: [configuration reference](../reference/configuration.md#export-to-a-folder).
<!-- wikitool:toc -->
## Contents
- [Pick the pages and the format](#pick-the-pages-and-the-format)
- [Set it up](#set-it-up)
- [What a run does to the folder](#what-a-run-does-to-the-folder)
- [Export after every publish](#export-after-every-publish)
- [Convert the pages with a program of your own](#convert-the-pages-with-a-program-of-your-own)
- [What does not come back in](#what-does-not-come-back-in)
- [Related](#related)
<!-- /wikitool:toc -->
## Pick the pages and the format
- **Which pages.** Only what a target's `include` selects is written; there is no "everything". A
selection is a list of `search` predicates that must all hold - `tags=shared`, `kind=system`,
`!draft` - and a page that fits any selection of the target is exported. Try a selection with
`tools/wikitool search --field tags=shared` first.
- **Which format.** Two ways, equally valid:
- `markdown` writes the pages as `.md` files. It needs nothing but the folder.
- `command` hands those files to a converter of your own and exports what it produces - Word
documents with `pandoc`, say.
Whether a Microsoft 365 tenant indexes `.md` files for Copilot is not settled for every tenant.
Try a small `markdown` target first; if the tenant does not pick the files up, the same pages go
through a `command` target as a format it does.
## Set it up
1. **Pick the folder**: a folder inside the synced folder, for example
`C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
and it must lie outside the instance, outside every other export target, outside every `intake`
folder and outside `backup.target`.
2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
since its fields decide what leaves this machine and which program runs, and `config set`
refuses them:
```json
{
"export": {
"targets": {
"shared": {
"handler": "markdown",
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Lesefassung",
"include": [["tags=shared"]],
"frontmatter": "summary",
"links": "relative",
"after_publish": false
}
}
}
}
```
`frontmatter` decides what follows the export marker in each file: `minimal` nothing, `summary`
(the default) the title, type, summary and modified date, `full` the page's whole frontmatter.
`links: relative` (the default) turns a link to an exported page into a relative Markdown link;
`links: text` leaves the page's name. A link to a page that is not exported is always plain text.
3. **Tell the agent:**
> Export the pages.
The agent runs `tools/wikitool export run`. It reports per target which pages are new, which
changed and which were removed, and how many stayed as they were. Name a target to run only that
one: `tools/wikitool export run shared`.
4. **Check it:** `tools/wikitool doctor`. The `export` line lists the targets; a `FAIL` there names
what is wrong with the section.
The export reads the committed pages. A `kb/` or `types/` with uncommitted changes is refused, so
every file names the commit its pages come from. Publish first.
## What a run does to the folder
The folder holds the pages and one file of the stack's own, `.wikitool-export.json`, which lists
what the last run wrote.
- **Only its own files.** A run removes a file only when it is listed and no longer exported, and a
folder only when the run created it. Anything else in the folder stays as it is.
- **Nobody edits an export.** A listed file that someone changed stops the target before anything is
written, and the message names the file. Move or delete it and run again. A listed file that is
missing is written again.
- **A folder that is not an export is refused.** A folder that holds files and no manifest is
somebody's folder; the run writes nothing into it.
- **An empty selection is an error.** A run that finds no page does not delete the earlier export.
- **An unchanged export stays untouched.** A run that has nothing to change writes nothing, not even
a modification time - the sync client has nothing to upload.
- **A growing selection needs no confirmation.** New pages are reported as new; read the list
before the sync client uploads them.
Targets are independent: one that fails is reported with its reason and the others still run; the
command then exits 1.
## Export after every publish
With `"after_publish": true` on a target, every `tools/wikitool publish` that committed refreshes
it afterwards - with `--no-push` too, since the export reads the commit, not the remote. If that
export fails, `publish` prints a `WARN` with the reason and still exits 0: the commit and the push
stand. Fix what the line names and run `tools/wikitool export run`; another `publish` does not
repeat it. A malformed `export` section stops `publish` before it commits, but only when some target
sets `after_publish`.
## Convert the pages with a program of your own
A target with `"handler": "command"` names an entry of `commands`:
```json
{
"export": {
"targets": {
"as-word": {
"handler": "command",
"command": "pandoc-docx",
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Word",
"include": [["tags=shared"]]
}
},
"commands": {
"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]
}
}
}
```
The pages are rendered into a staging directory first. The program runs without a shell, with stdin
closed and a limit of 300 seconds, and gets two more arguments: the directory with the Markdown
files and an empty directory for its output. On exit 0 the regular files it left there (below any
subdirectory, symlinks left out and reported) become the content of the target; the manifest and
all the rules above apply to them as to Markdown. Any other end - a non-zero exit, a timeout, a
program that does not start - leaves the target exactly as it was and prints the end of the
program's stderr. The staging directory is always removed.
## What does not come back in
An export is never a source. Every Markdown file starts with a frontmatter block whose first key is
`wikitool_export`, and `tools/wikitool raw accept` and `raw capture` refuse a file that carries it.
A converter may lose that key, so `raw accept` also refuses any file in `incoming/` that is byte
for byte one of the files in the manifest of a configured target - and warns, without refusing,
when a target has no manifest yet or the `export` section is broken.
## Related
- [Configuration reference](../reference/configuration.md#export-to-a-folder)
- [Export commands](../reference/commands/exports.md)
- [Export guidelines to captured repositories](export-guidelines.md) - the other export, one file
pushed into a repository
- [Back up to a synced folder](back-up-to-a-synced-folder.md) - the same kind of folder, for the
backup instead
docs/reference/commands/README.md
+5 -4
View File
@@ -15,7 +15,7 @@ collected in [exit codes and gates](../exit-codes.md).
- [Provenance](#provenance)
- [Raw material and uploads](#raw-material-and-uploads)
- [Git](#git)
- [Guideline export](#guideline-export)
- [Exports](#exports)
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
- [Types, instructions and docs](#types-instructions-and-docs)
- [Telemetry](#telemetry)
@@ -101,11 +101,12 @@ Full records: [git.md](git.md)
- [`backup init`](git.md#backup-init) - Create the backup target's bare repository and make it `origin` when there is none.
- [`backup run`](git.md#backup-run) - Mirror this machine's files that are not in git into `<target>/<machine>/`.
## Guideline export
## Exports
Full records: [guideline-export.md](guideline-export.md)
Full records: [exports.md](exports.md)
- [`export guidelines`](guideline-export.md#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
- [`export guidelines`](exports.md#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
- [`export run`](exports.md#export-run) - Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
## Workshop runs and session budget
docs/reference/commands/guideline-export.md → docs/reference/commands/exports.md
+67 -2
View File
@@ -1,6 +1,13 @@
# Guideline export
# Exports
Writing the generated `GUIDELINES.md` into the repositories the wiki has captured.
Derived, read-only versions of `kb/`: the generated `GUIDELINES.md` for the repositories the wiki has captured, and the pages exported into configured folders.
<!-- wikitool:toc -->
## Contents
- [`export guidelines`](#export-guidelines)
- [`export run`](#export-run)
<!-- /wikitool:toc -->
<!-- wikitool:reference-commands -->
## `export guidelines`
@@ -74,4 +81,62 @@ tools/wikitool export guidelines --confirm <token> --push --field tags=guideline
- `wikitool search` - the same predicates, to preview the selection
- `wikitool raw capture` - how a repository becomes a target
- `instructions/gates.md` - the gate procedure
## `export run`
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
**Synopsis**
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
**Options**
```text
Arguments:
[TARGET]... Targets to run; default: every configured target.
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool export run
tools/wikitool export run work-notes
```
**Behaviour**
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
- Counts toward the session call budget: yes
- Can reach the network: no
**Exit codes**
- `0` - success
- `1` - No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
- `1` - `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
- `1` - A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
**See also**
- `wikitool search` - the same predicates, to preview a selection
- `wikitool backup run` - the same kind of folder, for this machine's own files
- `docs/how-to/export-to-a-synced-folder.md`
<!-- /wikitool:reference-commands -->
docs/reference/commands/git.md
+4 -1
View File
@@ -134,6 +134,7 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
**Properties**
@@ -149,11 +150,12 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
- `0` - success
- `0` - The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
- `0` - The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
- `1` - git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
- `1` - The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
- `1` - No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
- `1` - `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
- `1` - The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed
- `1` - The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
- `42` - The remotes settings file of an earlier version is still there
- `42` - Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
- `42` - Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
@@ -165,6 +167,7 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
- `instructions/gates.md` - the gate procedure
- `instructions/setup-instance.md` - the first publish of a new instance
- `wikitool backup run` - the machine-data backup on its own
- `wikitool export run` - the page export on its own
## `backup init`
docs/reference/commands/instance-health.md
+1
View File
@@ -38,6 +38,7 @@ tools/wikitool doctor --json
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
docs/reference/commands/raw-material-and-uploads.md
+2 -2
View File
@@ -262,7 +262,7 @@ tools/wikitool raw capture --update raw/2026/10/service-docs
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
@@ -477,7 +477,7 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
- `0` - success
- `1` - raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- `1` - raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
- `1` - raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- `1` - raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- `1` - raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
docs/reference/configuration.md
+92 -1
View File
@@ -31,6 +31,7 @@ something you run.
- [Site handlers](#site-handlers)
- [Fetch routes](#fetch-routes)
- [Backup to a synced folder](#backup-to-a-synced-folder)
- [Export to a folder](#export-to-a-folder)
- [Files of earlier versions](#files-of-earlier-versions)
<!-- /wikitool:toc -->
@@ -93,7 +94,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
[backup](#backup-to-a-synced-folder). `config set` names the sections it knows, and `doctor` reports a
[backup](#backup-to-a-synced-folder) and [export](#export-to-a-folder). `config set` names the sections it knows, and `doctor` reports a
section it does not read as a `WARN`.
**One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -631,6 +632,96 @@ Example:
`config set` exits 42 for them, and only you write them. The procedure, restoring included:
[back-up-to-a-synced-folder.md](../how-to/back-up-to-a-synced-folder.md).
## Export to a folder
The `export` section of `.wikitool-local.json` declares folders that hold a derived, read-only
version of `kb/` - for a reader outside the repository, such as a Microsoft 365 tenant that indexes
a OneDrive folder. An export is regenerated from the committed pages on every run and is never
edited; what someone changes in the folder is refused or overwritten, never merged back. Each
target has a handler: `markdown` writes the selected pages as Markdown files, `command` renders
them into a staging directory and hands that to a converter of this checkout.
<!-- wikitool:reference-config-export -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target, an `intake` folder or `backup.target`) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
- **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
- **Malformed:** An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops with exit 1 before it commits anything, but only when some target sets `after_publish`. A broken section is never read as "no export".
Example:
```json
{
"export": {
"targets": {
"work-notes": {
"handler": "markdown",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
"include": [
[
"kind=system"
],
[
"tags=shared",
"!draft"
]
],
"frontmatter": "summary",
"links": "relative",
"after_publish": true
},
"as-word": {
"handler": "command",
"command": "pandoc-docx",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
"include": [
[
"kind=system"
]
]
}
},
"commands": {
"pandoc-docx": [
"pwsh",
"-NoProfile",
"-File",
"C:\\tools\\pandoc-docx.ps1"
]
}
}
}
```
<!-- /wikitool:reference-config-export -->
Within one target, `include` is a list of selections, each a list of `search` predicates (`tags=docs`,
`entity_type=system`, `!source_url`): a page is exported when it satisfies every predicate of at least
one selection. Only what `include` selects is written - there is no way to exclude by default. The
syntax is checked when the section is read, the field names against the corpus when `export run`
runs. `path` must not lie in or contain the instance, another target, an `intake` folder or
`backup.target`.
Every Markdown file starts with a frontmatter block whose first key is `wikitool_export`, carrying
the kind, the instance and the commit of the newest selected page. `frontmatter` decides what follows:
`minimal` nothing, `summary` the title, type, summary and modified date, `full` the page's whole
frontmatter. Wikilinks between exported pages become relative Markdown links (`links: relative`) or
plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote
markers and code stay as they are.
A target folder holds `.wikitool-export.json`, which lists the files a run wrote with their digests.
A run removes only listed files that are no longer exported, creates only folders it needs and
removes only those it created, and leaves every other file untouched. It stops before writing for a
listed file someone changed, for a folder that is not empty and has no manifest, and for a selection
that comes up empty. A run that finds nothing to change writes nothing, not even a modification time.
`targets` and `commands` decide what leaves this checkout and which program runs, so they are marked
`gate` and `names a command`: `config set` exits 42 for them, and only you write them. `raw accept`
also refuses a file whose sha256 is listed in the manifest of a configured target, for an export
that went through a converter and lost its marker. The procedure:
[export-to-a-synced-folder.md](../how-to/export-to-a-synced-folder.md).
## Files of earlier versions
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
docs/reference/exit-codes.md
+2 -2
View File
@@ -10,7 +10,7 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
### guideline-push
- [`export guidelines`](commands/guideline-export.md#export-guidelines)
- [`export guidelines`](commands/exports.md#export-guidelines)
### mass-update
@@ -45,6 +45,6 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
- [`publish`](commands/git.md#publish): Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
- [`publish`](commands/git.md#publish): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
- [`publish`](commands/git.md#publish): Publish-Remote Gate: `remotes.allowed_push_urls` is set and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
- [`export guidelines`](commands/guideline-export.md#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
- [`export guidelines`](commands/exports.md#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
- [`config set`](commands/configuration.md#config-set): The field names a program to run or affects a gate
<!-- /wikitool:reference-exit-codes -->
instructions/setup-instance.md
+5
View File
@@ -130,6 +130,11 @@ one it needs before that.
show the user any `remotes.allowed_push_urls` entry it prints, for them to add. The whole
procedure is [docs/how-to/back-up-to-a-synced-folder.md](../docs/how-to/back-up-to-a-synced-folder.md).
Until it is done, treat the instance as local.
- A reader outside the repository - a Microsoft 365 tenant that indexes a OneDrive folder, say
- who should get a read-only version of part of `kb/`: offer an export. The user writes the
`export` section of `.wikitool-local.json` by hand (its fields are gates, so `config set`
refuses them); then run `tools/wikitool export run`. The whole procedure is
[docs/how-to/export-to-a-synced-folder.md](../docs/how-to/export-to-a-synced-folder.md).
4. **Decision point - authoring conventions.** The release ships no filled-in conventions, only
`kb/CONVENTIONS.md.template` and one `kb/<name>/COLLECTION.md.template` per collection. Both
raw/CONTRACT.md
+13 -8
View File
@@ -95,12 +95,16 @@ That is what keeps the clean-up safe: only directories the moves emptied are rem
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
both ways out - the whole folder, or the file moved up into `incoming/`.
**A guideline export never comes back in.** A file whose first line (after an optional BOM)
starts with `<!-- wikitool:export` is this stack's own output - `export guidelines` renders the
wiki's guideline pages into it for other repositories - and `raw accept` refuses it, given on its
own, inside a folder, or as `--replaces`/`--replaces-bundle` material, naming each one. Taken in,
it would turn the wiki's knowledge on its way out into a source the wiki then cites for the same
claims. `raw capture` leaves such a file out of a repository for the same reason
**An export never comes back in.** A file that carries the stack's export marker is this stack's
own output and `raw accept` refuses it, given on its own, inside a folder, or as
`--replaces`/`--replaces-bundle` material, naming each one. The marker is either a first line
(after an optional BOM) that starts with `<!-- wikitool:export` - the generated `GUIDELINES.md` of
`export guidelines` - or, in a leading frontmatter block, a `wikitool_export:` key - a page
`export run` wrote. A converter of an `export` target may drop the marker, so `raw accept` also
refuses a file whose sha256 is listed in the manifest of a configured target; a missing manifest
or a broken `export` section is a `WARN` and the check is skipped. Taken in, an export would turn
the wiki's knowledge on its way out into a source the wiki then cites for the same claims.
`raw capture` leaves a file with the marker out of a repository for the same reason
([below](#getting-a-repository-in-raw-capture)).
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
@@ -447,8 +451,9 @@ is captured from a repository or accepted from `incoming/` anywhere but at the t
folder.
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its
way out, which must not come back in as a source), a symlink, a submodule, a file over 25 MiB, a
starts with `<!-- wikitool:export` or opens a frontmatter block that holds a `wikitool_export:` key
(this stack's own export - the wiki's knowledge on its way out, which must not come back in as a
source), a symlink, a submodule, a file over 25 MiB, a
path with a segment starting with `.` (`raw accept` refuses hidden entries anyway), a file named
`_capture.json`, and a Git LFS pointer, which holds a reference to the content rather than the
content.
tools/CONTRACT.md
+70 -7
View File
@@ -42,7 +42,7 @@ ON FAILURE and NEVER, which address an agent.
- [Provenance](#provenance)
- [Raw material and uploads](#raw-material-and-uploads)
- [Git](#git)
- [Guideline export](#guideline-export)
- [Exports](#exports)
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
- [Types, instructions and docs](#types-instructions-and-docs)
- [Telemetry](#telemetry)
@@ -140,6 +140,7 @@ publish write non-idempotent budget:counted exit:0,1,
backup init write idempotent budget:counted exit:0,1 Create the backup target's bare repository and make it `origin` when there is none.
backup run write idempotent budget:counted exit:0,1 Mirror this machine's files that are not in git into `<target>/<machine>/`.
export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
export run write idempotent budget:counted exit:0,1 Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run.
work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop.
budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history.
@@ -1671,7 +1672,7 @@ Capture documentation from a git repository into `incoming/<bundle>/`, with a ma
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
@@ -1806,7 +1807,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- 0 success
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
@@ -1821,7 +1822,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
@@ -2178,11 +2179,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
- 0 success
- 0 The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
- 0 The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
- 1 The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
- 1 No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
- 1 `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
- 42 The remotes settings file of an earlier version is still there
- 42 Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
- 42 Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
@@ -2191,11 +2193,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
**ON FAILURE**
- The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool backup run` repeats the backup
- The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool export run` repeats the export
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change) -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
- The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case -> Check out the branch you mean to publish, or pass `--branch <checked-out branch>`, then retry once
- No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed -> Show the message to the user and ask whether to commit locally with `--no-push`. Never push by hand - the next `publish` that reaches the remote sends that commit
- `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error -> Drop it. The Mass-Update Gate is cleared only with `--confirm <token>` from the gate's own refusal output
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
- The remotes settings file of an earlier version is still there -> Show the output to the user verbatim and stop; `tools/wikitool config migrate` moves it, and the user decides when
- Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
- Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
@@ -2225,6 +2228,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
**SEE ALSO**
@@ -2233,6 +2237,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
- `instructions/gates.md` - the gate procedure
- `instructions/setup-instance.md` - the first publish of a new instance
- `wikitool backup run` - the machine-data backup on its own
- `wikitool export run` - the page export on its own
#### `backup init`
@@ -2337,7 +2342,7 @@ Mirror this machine's files that are not in git into `<target>/<machine>/`.
- `wikitool doctor` - the `backup` line reports the age of the last run
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
### Guideline export
### Exports
#### `export guidelines`
@@ -2405,6 +2410,63 @@ Render the guideline pages into one generated `GUIDELINES.md`, or push it into t
- `wikitool raw capture` - how a repository becomes a target
- `instructions/gates.md` - the gate procedure
#### `export run`
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
**SYNOPSIS**
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
**PROPERTIES**
- effect: write
- idempotent: yes
- atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
- budget: counted
- network: no
**EXAMPLES**
- `tools/wikitool export run`
- `tools/wikitool export run work-notes`
**EXIT STATUS**
- 0 success
- 1 No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
- 1 `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
- 1 A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
**ON FAILURE**
- No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target) -> Show the message to the user; the section is theirs to write by hand, never yours
- `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter -> Fix the named pages, or publish the changes first, then retry once
- A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0 -> Show the target's reason to the user; the other targets were still served. Never delete or edit files in a target folder to make a run pass
**NEVER**
- Never edit a file in a target folder to change an export - the next run refuses or overwrites it; change the page, or the target's `include`, instead.
- Never put an export folder into `incoming/` or `raw/`.
**NOTES**
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
**SEE ALSO**
- `wikitool search` - the same predicates, to preview a selection
- `wikitool backup run` - the same kind of folder, for this machine's own files
- `docs/how-to/export-to-a-synced-folder.md`
### Workshop runs and session budget
#### `work new`
@@ -4073,6 +4135,7 @@ Check that this instance is correctly configured.
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
tools/README.md
+2 -1
View File
@@ -132,7 +132,8 @@ tools/
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
backup.py `backup init`/`backup run`'s core and the step `publish` runs after a push: the bare repository under `<target>/git`, the machine-data mirror under `<target>/<machine>/` with its manifest, and which synced folders `doctor` knows
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`is_export`, for both the guideline comment line and the page frontmatter key) - with no CLI attached
page_export.py `export run`'s core and the step `publish` runs after a commit: the `export` section, rendering selected pages as Markdown, the target folder with its manifest, and the converter handler - with no CLI attached
search/ pluggable search backends, plus service.py - the search core
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (the tracker sections of the settings), one module per adapter - no instruction ever learns which provider it is
commands/ one module per command or command group: the terminal adapters - except `raw/`, a package with one module per subcommand and the helpers several of them share in `_common.py`
tools/chemenu/backup.py
+29 -29
View File
@@ -99,12 +99,12 @@ class BackupConfig:
# ---------------------------------------------------------------------------
def _is_link(path: Path) -> bool:
def is_link(path: Path) -> bool:
is_junction = getattr(path, "is_junction", None) # Python 3.12+
return path.is_symlink() or bool(is_junction and is_junction())
def _real(path: Path) -> Path:
def real_path(path: Path) -> Path:
return Path(os.path.realpath(path))
@@ -121,7 +121,7 @@ def _names_git(parts) -> bool:
return any(part.casefold() == GIT_NAME for part in parts)
def _safe_relative(rel: str) -> bool:
def safe_relative(rel: str) -> bool:
"""Whether `rel`, a `/`-separated path, stays below the folder it is joined
to on every platform: no empty, `.` or `..` part, no backslash, no drive."""
parts = rel.split("/")
@@ -132,13 +132,13 @@ def _safe_relative(rel: str) -> bool:
def _within(machine_dir: Path, rel: str) -> Path:
"""`rel` below `machine_dir`, refusing a path that would leave it - by its
spelling, or through a link somewhere in the folder."""
if not _safe_relative(rel):
if not safe_relative(rel):
raise BackupError(f"{rel!r} is not a path below {machine_dir} - nothing further was written")
dest = machine_dir.joinpath(*rel.split("/"))
current = machine_dir
for part in rel.split("/")[:-1]:
current = current / part
if _is_link(current):
if is_link(current):
raise BackupError(f"{current} is a link - a backup writes nothing through one")
return dest
@@ -174,13 +174,13 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
if not section.configured:
return None
problems = []
real_root = _real(root)
real_root = real_path(root)
target = Path(section["target"])
if not os.path.isabs(section["target"]):
problems.append(f"backup.target {section['target']!r} is not an absolute path on this machine")
else:
real_target = _real(target)
real_target = real_path(target)
if is_under(real_target, real_root):
problems.append(f"backup.target {target} lies inside the instance - it belongs outside it")
@@ -195,7 +195,7 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
if not problems:
for inner in (target / config_shapes.BACKUP_GIT_DIR, target / machine_name):
if is_under(real_root, _real(inner)):
if is_under(real_root, real_path(inner)):
problems.append(f"the instance lies inside {inner}, which the backup writes to")
for entry in section["extra"]:
@@ -249,14 +249,14 @@ def same_location(url: str, path: Path, root: Path) -> bool:
local = url[len("file://"):] if url.startswith("file://") else url
if "://" in local or (":" in local and not os.path.isabs(local)):
return False
a, b = os.path.normcase(_real(root / local)), os.path.normcase(_real(path))
a, b = os.path.normcase(real_path(root / local)), os.path.normcase(real_path(path))
return a.casefold() == b.casefold() if IGNORE_CASE else a == b
def _directory_state(path: Path) -> str:
if not path.exists() and not _is_link(path):
if not path.exists() and not is_link(path):
return "absent"
if path.is_dir() and not _is_link(path) and not any(path.iterdir()):
if path.is_dir() and not is_link(path) and not any(path.iterdir()):
return "empty"
return "bare" if is_bare_repo(path) else "other"
@@ -329,7 +329,7 @@ class Collected:
def _walk(path: Path, rel: str, out: Collected) -> None:
if _is_link(path):
if is_link(path):
out.skipped.append(f"{rel} (a link - not followed)")
return
if path.name.casefold() == GIT_NAME:
@@ -337,7 +337,7 @@ def _walk(path: Path, rel: str, out: Collected) -> None:
if path.is_file():
if rel == MANIFEST_NAME:
out.skipped.append(f"{rel} (the manifest's own name)")
elif not _safe_relative(rel):
elif not safe_relative(rel):
out.skipped.append(f"{rel} (a name that cannot be mirrored on every platform)")
else:
out.files.setdefault(rel, path)
@@ -351,13 +351,13 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
"""What a machine-data run mirrors: the root files that exist, everything in
`incoming/` but its `.gitkeep`, and `extra`."""
root = Path(config.ROOT if root is None else root)
real_root = _real(root)
real_root = real_path(root)
out = Collected()
for name in ROOT_FILES:
if (root / name).exists() or _is_link(root / name):
if (root / name).exists() or is_link(root / name):
_walk(root / name, name, out)
incoming = root / INCOMING_DIR
if _is_link(incoming):
if is_link(incoming):
out.skipped.append(f"{INCOMING_DIR} (a link - not followed)")
elif incoming.is_dir():
for child in sorted(incoming.iterdir()):
@@ -375,7 +375,7 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
return out
def _digest(path: Path) -> tuple[int, str]:
def file_digest(path: Path) -> tuple[int, str]:
sha, size = hashlib.sha256(), 0
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1 << 20), b""):
@@ -395,7 +395,7 @@ def _replace(tmp: str, dest: Path) -> None:
time.sleep(_REPLACE_PAUSE_SECONDS)
def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
def write_atomic(dest: Path, chunks) -> tuple[int, str]:
dest.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(prefix=".chemenu-", suffix=".tmp", dir=str(dest.parent))
sha, size = hashlib.sha256(), 0
@@ -416,7 +416,7 @@ def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
def _copy_atomic(src: Path, dest: Path) -> tuple[int, str]:
with open(src, "rb") as handle:
return _write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
return write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
def read_manifest(machine_dir: Path) -> Optional[dict]:
@@ -432,7 +432,7 @@ def read_manifest(machine_dir: Path) -> Optional[dict]:
files = data.get("files") if isinstance(data, dict) else None
if not isinstance(files, list) or not all(
isinstance(entry, dict) and isinstance(entry.get("path"), str)
and _safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
and safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
and isinstance(entry.get("sha256"), str)
for entry in files
):
@@ -456,7 +456,7 @@ class RunResult:
return line
def _prune_empty(directory: Path, stop: Path) -> None:
def prune_empty(directory: Path, stop: Path) -> None:
while directory != stop and is_under(directory, stop):
try:
directory.rmdir()
@@ -484,7 +484,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
"synced folder there? `tools/wikitool backup init` creates the target"
)
machine_dir = cfg.machine_dir
if _is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
if is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
raise BackupError(f"{machine_dir} is not a plain directory - nothing was written")
previous = read_manifest(machine_dir)
if previous is None and machine_dir.is_dir() and any(machine_dir.iterdir()):
@@ -500,10 +500,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
machine_dir.mkdir(exist_ok=True)
for rel, source in sorted(collected.files.items()):
dest = _within(machine_dir, rel)
size, digest = _digest(source)
size, digest = file_digest(source)
old = before.get(rel)
if (old and old["size"] == size and old["sha256"] == digest and dest.is_file()
and not _is_link(dest) and dest.stat().st_size == size):
and not is_link(dest) and dest.stat().st_size == size):
result.unchanged.append(rel)
else:
size, digest = _copy_atomic(source, dest)
@@ -512,10 +512,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
for rel in sorted(set(before) - set(collected.files)):
dest = _within(machine_dir, rel)
if dest.is_file() and not _is_link(dest):
if dest.is_file() and not is_link(dest):
dest.unlink()
result.removed.append(rel)
_prune_empty(dest.parent, machine_dir)
prune_empty(dest.parent, machine_dir)
manifest = {
"schema": MANIFEST_SCHEMA,
@@ -525,7 +525,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
"files": entries,
}
text = json.dumps(manifest, indent=2, ensure_ascii=False) + "\n"
_write_atomic(cfg.manifest, [text.encode("utf-8")])
write_atomic(cfg.manifest, [text.encode("utf-8")])
return result
@@ -544,9 +544,9 @@ def sync_roots() -> list[Path]:
def synced_folder_of(path: Path) -> Optional[Path]:
"""The synced folder `path` lies in, links and junctions resolved on both
sides, or None."""
real = _real(path)
real = real_path(path)
for candidate in sync_roots():
if is_under(real, _real(candidate)):
if is_under(real, real_path(candidate)):
return candidate
return None
tools/chemenu/cli_contract.py
+2 -2
View File
@@ -293,8 +293,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
("Git", (
"sync", "publish", "backup init", "backup run",
)),
("Guideline export", (
"export guidelines",
("Exports", (
"export guidelines", "export run",
)),
("Workshop runs and session budget", (
"work new", "work close", "budget status", "budget reset",
tools/chemenu/commands/doctor.py
+26
View File
@@ -53,6 +53,7 @@ def _git(args: list[str]) -> Optional[subprocess.CompletedProcess]:
PREFLIGHT_FIX = f"Run tools/preflight.sh (PowerShell 7: {toolpaths.PREFLIGHT_PWSH})"
BACKUP_HOW_TO = "docs/how-to/back-up-to-a-synced-folder.md"
EXPORT_HOW_TO = "docs/how-to/export-to-a-synced-folder.md"
def check_python() -> Check:
@@ -640,6 +641,27 @@ def _telemetry_file_deviations() -> list[str]:
return config_shapes.deviations(config_shapes.TELEMETRY, data[config_shapes.TELEMETRY.key])
def check_export() -> Check:
"""The `export` section: absent is `OK` (no page export configured), a
broken one is a `FAIL`, and a configured one lists its targets."""
from chemenu import page_export
try:
targets = page_export.read_config(config.ROOT)
except ChemenuError as exc:
return Check(
"export", "FAIL", str(exc),
f"Fix the export section as the message says - see {EXPORT_HOW_TO}",
)
if targets is None:
return Check("export", "OK", "absent - no export section in the settings")
parts = [
f"{t.name} ({t.handler}, {t.path}{', after publish' if t.after_publish else ''})"
for t in targets
]
return Check("export", "OK", f"{len(targets)} target(s): " + "; ".join(parts))
def check_upload_intake() -> Check:
"""Which intake channels this checkout has, how many submissions wait in
the quarantine, and whether the quarantine of earlier versions
@@ -1036,6 +1058,7 @@ def run_doctor() -> list[Check]:
check_sync_folder(),
check_backup(),
check_upload_intake(),
check_export(),
check_tasks_provider(),
*check_settings(),
check_site_handlers(),
@@ -1095,6 +1118,9 @@ def run_doctor() -> list[Check]:
"Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not "
"push there, or - with `machine_data` true - no machine-data backup exists yet; "
"otherwise `OK` with the age of the last machine-data backup.",
"Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; "
"otherwise `OK` listing each target with its handler, folder and whether it runs after "
"`publish`.",
"Intake channels (`upload-intake`): the channels of the `intake` section with their "
"handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no "
"channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an "
tools/chemenu/commands/export_cmd.py
+129 -3
View File
@@ -1,6 +1,8 @@
"""`wikitool export guidelines` - the instance's guideline pages as one
generated `GUIDELINES.md`, printed, or written into the captured repositories
behind the Guideline Push Gate.
behind the Guideline Push Gate - and `wikitool export run`, the derived reading
copy of `kb/` in the folders the `export` section configures
(`chemenu/page_export.py`; no gate, since a target is a folder the user named).
The core - selection, rendering, the header - is `chemenu/guideline_export.py`;
the git write is `repo_capture.commit_file`/`push_commit`. This module decides
@@ -42,12 +44,12 @@ from typing import Optional
import typer
from rich.markup import escape
from chemenu import cli_contract, config, guideline_export, repo_capture
from chemenu import cli_contract, config, guideline_export, page_export, repo_capture
from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path
from chemenu.errors import BackendError, ValidationError
from chemenu.search import filters
app = typer.Typer(help="Generate files from the wiki for other repositories.")
app = typer.Typer(help="Generate files from the wiki for other repositories and folders.")
GATE = "guideline-push"
@@ -427,3 +429,127 @@ def export_guidelines_command(
"written; run the command again for the rest."
)
raise typer.Exit(code=1)
@app.command("run")
@cli_contract.record(cli_contract.CommandRecord(
path="export run",
summary="Write the derived, read-only reading copy of `kb/` into the folders the `export` "
"section configures - the selected pages as Markdown, or converted by a program of this "
"checkout.",
synopsis=(
cli_contract.Variant(
usage="export run [<target> ...]",
notes="Every configured target, or the named ones. Needs an `export` section in "
f"`{config.LOCAL_SETTINGS_FILENAME}`; the user writes it by hand.",
),
),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.YES,
atomic="Per file within a target (a temporary file in the target, then a move over its "
"place), the manifest last; targets are independent, so a failed target does not undo "
"another",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"An export is derived: regenerated from `kb/` on every run, never edited, never read back "
"into the wiki. Each target names its pages with an allowlist of `search --field` "
"predicates (`include`: a list of selections, AND within one, OR across them), a handler "
"(`markdown`, or `command` for the operator's converter) and the folder (`path`, outside "
"the instance). Nothing is exported that no selection names.",
"Every Markdown file starts with a frontmatter block whose `wikitool_export` line says "
"`kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit "
"touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the "
"page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, "
"`full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A "
"wikilink to an exported page becomes a relative Markdown link (`links: relative`, the "
"default) or plain text (`links: text`); a link to a page that is not exported is always "
"plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: "
"the same pages give the same bytes.",
"Handler `command`: the Markdown is rendered into a staging directory, the program of "
"`export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives "
"the Markdown directory and an empty output directory as its last two arguments. On exit "
"0 the regular files in the output directory (recursively) become the target's content; "
"any other end leaves the target unchanged and prints the end of its stderr. The staging "
"directory is always removed.",
"The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: "
"the files an earlier run wrote, each with its sha256. A run removes only manifest files "
"that are no longer exported (and folders it created, once empty), rewrites a manifest "
"file that is missing, leaves every other file byte for byte as it was, and writes "
"nothing - not even a modification time - for a file that did not change. A run that "
"changes nothing writes nothing at all.",
"Refuses a target, before writing anything into it: a non-empty folder with no manifest "
"(it is somebody's, not an export), a manifest file that was changed in the folder since "
"the last run (naming it), a foreign file where an exported one belongs, an empty "
"selection (it would delete the earlier export), an unknown field in a predicate. A "
"missing folder is created.",
"Refuses everything, before any target: uncommitted changes in `kb/` or `types/` "
"(untracked files included), no git checkout, a page under `kb/` with unreadable "
"frontmatter, an unknown target name, a broken `export` section.",
"Output per target: `new` (with the page titles), `changed`, `removed`, the count "
"`unchanged`, or `failed` with the reason. A page that newly enters a selection is "
"reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the "
"others were still served.",
"A target with `after_publish: true` is also refreshed by every `publish` that "
"committed, `--no-push` included; a failure there is a warning, the commit stands, and "
"this command is the repeat.",
"`raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries "
"the marker: an export never goes back into `raw/`.",
),
failures=(
cli_contract.Failure(
cause="No `export` section, an unknown target name, or a broken section (a predicate "
"that does not parse, a path inside the instance or in the way of another target, an "
"`intake` folder or the backup target)",
reaction="Show the message to the user; the section is theirs to write by hand, never "
"yours",
),
cli_contract.Failure(
cause="`kb/` or `types/` has uncommitted changes, the instance is not a git checkout, "
"or a page under `kb/` has unreadable frontmatter",
reaction="Fix the named pages, or publish the changes first, then retry once",
),
cli_contract.Failure(
cause="A target failed: a foreign folder, a file changed in the target, an empty "
"selection, an unknown field, a converter that did not exit 0",
reaction="Show the target's reason to the user; the other targets were still served. "
"Never delete or edit files in a target folder to make a run pass",
),
),
examples=(
"tools/wikitool export run",
"tools/wikitool export run work-notes",
),
never=(
"Never edit a file in a target folder to change an export - the next run refuses or "
"overwrites it; change the page, or the target's `include`, instead.",
"Never put an export folder into `incoming/` or `raw/`.",
),
see_also=(
"`wikitool search` - the same predicates, to preview a selection",
"`wikitool backup run` - the same kind of folder, for this machine's own files",
"`docs/how-to/export-to-a-synced-folder.md`",
),
))
def export_run_command(
names: Optional[list[str]] = typer.Argument(
None, metavar="[TARGET]...", help="Targets to run; default: every configured target."
),
):
"""Write the reading copy of kb/ into the configured export folders."""
try:
targets = page_export.choose(page_export.require_config(), names or [])
results = page_export.run(targets)
except ValidationError as exc:
fail(escape(str(exc)))
return
failed = 0
for result in results:
for line in page_export.report_lines(result):
typer.echo(line)
failed += not result.ok
if failed:
console.print(f"[bold red]ERROR[/bold red] {failed} of {len(results)} target(s) failed - the "
"others were served; fix the reason above and run again.")
raise typer.Exit(code=1)
tools/chemenu/commands/git_publish.py
+65 -1
View File
@@ -180,6 +180,52 @@ def read_backup_config():
fail(str(exc))
# --- Export after publish -------------------------------------------------------
#
# A target of the `export` section with `after_publish` is refreshed once a
# commit exists - `--no-push` included, since the export reads the committed
# state of `kb/`, not what reached the remote. The commit has happened by then,
# so a failing export is a WARN and exit 0, for the same reason as the backup
# above: `export run` repeats it, another `publish` cannot.
def read_export_targets():
"""The targets to refresh after this publish; exit 1 for a broken `export`
section when some target asks for the refresh."""
from chemenu import page_export
from chemenu.errors import ChemenuError
try:
return page_export.after_publish_targets()
except ChemenuError as exc:
fail(str(exc))
def export_after_publish(targets) -> None:
from rich.markup import escape
from chemenu import page_export
from chemenu.errors import ChemenuError
try:
results = page_export.run(targets)
except (ChemenuError, OSError) as exc:
console.print(
f"[yellow]WARN[/yellow] The export failed: {escape(str(exc))}\n"
" The commit stands. Show this to the user; once the cause is fixed, "
"`tools/wikitool export run` repeats it - another `publish` does not."
)
return
for result in results:
for line in page_export.report_lines(result):
if result.ok:
typer.echo(line)
else:
console.print(f"[yellow]WARN[/yellow] {escape(line)}")
if not result.ok:
console.print(" The commit stands; `tools/wikitool export run` repeats this target.")
def back_up_machine_data(cfg) -> None:
from rich.markup import escape
@@ -1614,6 +1660,12 @@ def commit_regenerated(remote: str, branch: str) -> None:
"stand. No backup with `--no-push`, after a failed push, or when there was nothing to "
"push. A malformed `backup` section stops the call with exit 1 before anything is "
"committed, `--no-push` included.",
"Export after publish: every target of the `export` section with `after_publish: true` "
"is refreshed once a commit was made - the same step as `export run`, `--no-push` "
"included, since the export reads the committed `kb/`. A failure there prints a `WARN` "
"line with the reason and still exits 0, since the commit stands; `export run` repeats "
"it. A malformed `export` section stops the call with exit 1 before anything is "
"committed, but only when some target sets `after_publish`.",
"After a successful commit or push whose changed files include `tools/`, `types/`, "
"`instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder "
"line: the phase past this point (an issue-body rewrite, `docs/` staleness, a "
@@ -1651,7 +1703,8 @@ def commit_regenerated(remote: str, branch: str) -> None:
),
cli_contract.Failure(
cause="The settings files do not pass their check, `remotes.allowed_push_urls` is "
"not a list of strings, or the `backup` section is malformed; nothing was committed",
"not a list of strings, the `backup` section is malformed, or the `export` section is "
"malformed while a target sets `after_publish`; nothing was committed",
reaction="Show the error to the user and stop - a malformed section is not "
"permission, and fixing it is theirs to do",
),
@@ -1662,6 +1715,13 @@ def commit_regenerated(remote: str, branch: str) -> None:
"fixed, `tools/wikitool backup run` repeats the backup",
code=0,
),
cli_contract.Failure(
cause="The export after the commit failed: a `WARN` line names the target and the "
"reason, and the commit stands",
reaction="Show the `WARN` line to the user. Do not publish again - once the cause is "
"fixed, `tools/wikitool export run` repeats the export",
code=0,
),
cli_contract.Failure(
cause="The remotes settings file of an earlier version is still there",
reaction="Show the output to the user verbatim and stop; `tools/wikitool config "
@@ -1710,6 +1770,7 @@ def commit_regenerated(remote: str, branch: str) -> None:
"`instructions/gates.md` - the gate procedure",
"`instructions/setup-instance.md` - the first publish of a new instance",
"`wikitool backup run` - the machine-data backup on its own",
"`wikitool export run` - the page export on its own",
),
))
def publish_command(
@@ -1760,6 +1821,7 @@ def publish_command(
# Read before anything is committed, `--no-push` included: a broken section
# stops here rather than turning into a skipped backup after the push.
backup_cfg = read_backup_config()
export_targets = read_export_targets()
paths = list(path or [])
# What is staged and committed: `paths` plus any uncommitted generated file outside them.
@@ -1932,5 +1994,7 @@ def publish_command(
success(f"Published changes to {remote}/{branch}." if push else "Committed changes (not pushed).")
if push and backup_cfg is not None and backup_cfg.machine_data:
back_up_machine_data(backup_cfg)
if export_targets:
export_after_publish(export_targets)
if touches_stack_machinery(changed_files):
typer.echo(STACK_MACHINERY_NOTE)
tools/chemenu/commands/raw/_common.py
+29 -7
View File
@@ -11,9 +11,10 @@ import re
from pathlib import Path
from typing import Optional
import typer
from rich.markup import escape
from chemenu import config, filters, guideline_export, repo_capture
from chemenu import backup, config, filters, guideline_export, page_export, repo_capture
from chemenu.commands._util import fail, path_budget_problem_for, rel_path
from chemenu.errors import ValidationError
from chemenu.provenance import citing_pages, source_pages_by_raw_file
@@ -155,16 +156,37 @@ def _check_files(resolved: list[Path]) -> None:
def _check_no_export(paths: list[Path]) -> None:
"""Refuse a file a wiki exported (`export guidelines`): it is generated
from `kb/`, so taking it into `raw/` would feed the wiki its own output
back as a source. `raw capture` already leaves such a file out; this
closes the way in through `incoming/`."""
"""Refuse a file a wiki exported (`export guidelines`, `export run`): it is
generated from `kb/`, so taking it into `raw/` would feed the wiki its own
output back as a source. `raw capture` already leaves such a file out; this
closes the way in through `incoming/`. The marker catches a file as written;
the digest of a configured target's manifest catches one whose marker a
converter dropped."""
exports = [p for p in paths if guideline_export.file_is_export(p)]
if exports:
listed = "\n".join(f" - {rel_path(p)}" for p in exports)
raise _Refused(
"These files are a wiki's guideline export - their first line starts with "
"<!-- wikitool:export. They are generated from kb/ and never go back into raw/:\n"
"These files are a wiki's export (a guideline export or a page export) - they carry "
"the wikitool export marker. They are generated from kb/ and never go back into "
f"raw/:\n{listed}\n Remove them from incoming/ and accept again; nothing was moved."
)
digests, warnings = page_export.known_digests()
for warning in warnings:
typer.echo(f"WARN {warning}")
if not digests:
return
exported = []
for p in paths:
try:
if backup.file_digest(p)[1] in digests:
exported.append(p)
except OSError:
continue
if exported:
listed = "\n".join(f" - {rel_path(p)}" for p in exported)
raise _Refused(
"These files are byte for byte a file of a configured export target (see "
"`export run`). They are generated from kb/ and never go back into raw/:\n"
f"{listed}\n Remove them from incoming/ and accept again; nothing was moved."
)
tools/chemenu/commands/raw/accept.py
+4 -2
View File
@@ -668,8 +668,10 @@ def _warn_underived(resolved: list[Path]) -> None:
),
cli_contract.Failure(
cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` "
"material - is a guideline export: its first line, after an optional BOM, starts with "
"`<!-- wikitool:export`",
"material - is an export of this wiki: it carries the export marker (a first line "
"starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading "
"frontmatter block), or it is byte for byte a file in the manifest of a configured "
"`export` target",
reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes "
"back into `raw/`. Remove it from `incoming/`; nothing was moved",
),
tools/chemenu/commands/raw/capture.py
+1 -1
View File
@@ -121,7 +121,7 @@ def _write_bundle(target: Path, files: dict[str, bytes], manifest: repo_capture.
"spans any number of directories; a glob without wildcards also takes everything below it "
"as a directory. Files keep their repository paths inside the bundle.",
"Never captured, each named in the output with its reason: a file whose first line starts "
"with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a "
"with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a "
"segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.",
"`incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, "
"`captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that "
tools/chemenu/config_shapes.py
+150 -1
View File
@@ -813,11 +813,160 @@ BACKUP = Shape(
"\"no backup\".",
)
EXPORT_HANDLER_MARKDOWN = "markdown"
EXPORT_HANDLER_COMMAND = "command"
EXPORT_HANDLERS = (EXPORT_HANDLER_MARKDOWN, EXPORT_HANDLER_COMMAND)
EXPORT_FRONTMATTER_MODES = ("minimal", "summary", "full")
EXPORT_DEFAULT_FRONTMATTER = "summary"
EXPORT_LINK_MODES = ("relative", "text")
EXPORT_DEFAULT_LINKS = "relative"
# What each handler takes besides the keys every target has. A key outside
# both sets is an error, so a typo or a field of the other handler is not
# silently ignored.
_EXPORT_COMMON_KEYS = ("handler", "path", "include", "frontmatter", "links", "after_publish")
_EXPORT_HANDLER_KEYS = {
EXPORT_HANDLER_MARKDOWN: (),
EXPORT_HANDLER_COMMAND: ("command",),
}
def _export_include_problem(where: str, include: Any) -> Optional[str]:
if not isinstance(include, list) or not include:
return f"{where}: `include` must be a non-empty array of selections"
for number, selection in enumerate(include, start=1):
if not isinstance(selection, list) or not selection:
return f"{where}: `include` selection {number} must be a non-empty array of predicates"
if not all(isinstance(predicate, str) and predicate.strip() for predicate in selection):
return f"{where}: `include` selection {number} holds an entry that is not a non-empty string"
return None
def _export_target_problem(name: str, target: Any) -> Optional[str]:
where = f"targets.{name}"
if not isinstance(target, dict):
return f"{where} must be an object"
handler = target.get("handler")
if handler not in EXPORT_HANDLERS:
return f"{where}: `handler` must be one of {', '.join(EXPORT_HANDLERS)}"
allowed = _EXPORT_COMMON_KEYS + _EXPORT_HANDLER_KEYS[handler]
stray = sorted(set(target) - set(allowed))
if stray:
return (f"{where}: {', '.join(f'`{key}`' for key in stray)} not taken by the handler "
f"{handler} (it takes {', '.join(f'`{key}`' for key in allowed)})")
for key in ("path", "include"):
if key not in target:
return f"{where}: `{key}` is required"
path = target["path"]
if not isinstance(path, str) or not path.strip():
return f"{where}: `path` must be a non-empty string"
problem = absolute_path_problem(path)
if problem:
return f"{where}: `path` {problem}"
problem = _export_include_problem(where, target["include"])
if problem:
return problem
for key, choices in (("frontmatter", EXPORT_FRONTMATTER_MODES), ("links", EXPORT_LINK_MODES)):
if key in target and target[key] not in choices:
return f"{where}: `{key}` must be one of {', '.join(choices)}"
if "after_publish" in target and not isinstance(target["after_publish"], bool):
return f"{where}: `after_publish` must be true or false"
if handler == EXPORT_HANDLER_COMMAND:
ref = target.get("command")
if not isinstance(ref, str) or not _FETCHER_NAME.fullmatch(ref):
return (f"{where}: `command` is required and names an entry of `commands` - lower-case "
"letters, digits and hyphens")
return None
def export_targets_problem(value: dict) -> Optional[str]:
if not value:
return "must hold at least one target"
for name, target in value.items():
if not _FETCHER_NAME.fullmatch(name):
return (f"has the target name {name!r} - a target's name is lower-case letters, digits "
"and hyphens")
problem = _export_target_problem(name, target)
if problem:
return problem
return None
def export_section_problem(values: dict) -> Optional[str]:
commands = values.get("commands") or {}
for name, target in values["targets"].items():
if target["handler"] == EXPORT_HANDLER_COMMAND and target["command"] not in commands:
return (f"targets.{name}.command names {target['command']!r}, which `commands` does not "
"define")
return None
EXPORT = Shape(
key="export",
title="Exports to folders",
purpose="Declares the folders `export run` writes a derived, read-only reading copy of `kb/` "
"into: per target the pages it takes (an allowlist of `search` predicates), how they are "
"written, and whether `publish` refreshes it. An export is regenerated from `kb/` and never "
"edited; what a person changes in the folder is overwritten or refused, never merged back.",
fields=(
Field(
"targets", "object", "Target name (lower-case letters, digits and hyphens) to target; "
"the name is the argument of `export run`. Every target has `handler` (`markdown`: the "
"pages as Markdown files; `command`: the same Markdown run through a converter of this "
"checkout), `path` (an absolute path outside the instance; it must not lie in or "
"contain another target, an `intake` folder or `backup.target`) and `include` (an "
"array of selections, each an array of `search --field` predicates: all predicates of "
"a selection must hold, and a page that fits any selection is exported; there is no "
"\"everything\"). Optional: `frontmatter` (`minimal`: only the export marker; "
f"`summary`, the default: also `title`, `type`, `summary` and `modified` where the page "
"has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a "
"wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a "
"link to a page that is not exported is always plain text) and `after_publish` "
"(default false: refresh this target after every `publish` that committed). The "
"handler `command` takes `command` (required: the name of an entry of `commands`). A "
"field the handler does not take is an error.",
required=True, json_type="object", coerce=object_value, check=export_targets_problem,
tier=CHECKOUT, gate=True,
),
Field(
"commands", "object", "Command name to command: an array of the program and its "
"arguments, run by the `command` handler without a shell, with stdin closed and a "
"timeout of 300 s. The last two arguments appended are the directory holding the "
"Markdown and an empty output directory. Exit 0 makes the regular files in the output "
"directory the target's content; anything else leaves the target unchanged and prints "
"the end of the program's stderr.",
default={}, json_type="object", coerce=object_value, check=intake_commands_problem,
tier=CHECKOUT, names_command=True,
),
),
examples=({
"targets": {
"work-notes": {
"handler": "markdown",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
"include": [["kind=system"], ["tags=shared", "!draft"]],
"frontmatter": "summary", "links": "relative", "after_publish": True,
},
"as-word": {
"handler": "command", "command": "pandoc-docx",
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
"include": [["kind=system"]],
},
},
"commands": {"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]},
},),
absent="No export: `export run` stops with exit 1 naming the section, `publish` exports nothing "
"and `doctor` reports `OK`.",
malformed="An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops "
"with exit 1 before it commits anything, but only when some target sets `after_publish`. A "
"broken section is never read as \"no export\".",
section_check=export_section_problem,
)
# The sections of the tiered settings files, `wikitool.json` and
# `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
# stand in a section of its own.
SETTINGS_SECTIONS: tuple[Shape, ...] = (
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP,
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP, EXPORT,
)
# The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
tools/chemenu/guideline_export.py
+47 -13
View File
@@ -6,7 +6,7 @@ again when it comes back.
side - writing the file into a captured repository's branch - lives beside
`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under
`chemenu.commands`, and nothing here imports `repo_capture`, which imports
`EXPORT_MARKER` from here: that is the direction the dependency runs.
`is_export` from here: that is the direction the dependency runs.
Three properties carry the design, and each has a test:
@@ -38,11 +38,19 @@ from chemenu.search import filters
from chemenu.search.service import load_pages_by_path, unreadable_pages
from chemenu.search.types import Predicate
# The first bytes of every file this stack generates for another repository.
# Defined once, here: `repo_capture` excludes a file carrying it from a
# capture, `raw accept` refuses one, and the export writes it.
EXPORT_MARKER = b"<!-- wikitool:export"
# What marks a file this stack generated for somewhere else - the one place
# the definition of an export lives. `repo_capture` excludes a file that
# `is_export` recognises from a capture, `raw accept` refuses one, and the
# exports write it: `export guidelines` as the first-line comment, `export run`
# as a frontmatter key. Either form is an export; nothing else is.
COMMENT_MARKER = b"<!-- wikitool:export"
FRONTMATTER_KEY = b"wikitool_export:"
FRONTMATTER_FENCE = b"---"
# How much of a file `is_export` looks at to find the frontmatter's closing
# fence; a frontmatter that does not close within it is not an export.
SCAN_BYTES = 64 * 1024
GUIDELINES_KIND = "guidelines"
PAGES_KIND = "pages"
GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}"
TARGET_PATH = "GUIDELINES.md"
LOCAL_INSTANCE = "local"
@@ -57,16 +65,42 @@ _SCP_USERINFO = re.compile(r"^[^/@:]+@(?=[^/:]+:)")
# --- recognising an export ------------------------------------------------------
def _without_bom(data: bytes) -> bytes:
return data[len(_BOM):] if data.startswith(_BOM) else data
def _first_line(data: bytes) -> bytes:
data = data[len(_BOM):] if data.startswith(_BOM) else data
return data.split(b"\n", 1)[0]
return _without_bom(data).split(b"\n", 1)[0]
def is_export(data: bytes) -> bool:
"""Whether `data` is a file this stack exported: its first line, after an
optional BOM, starts with `EXPORT_MARKER`. Reads no more than the first
line, so a caller may pass only a file's first bytes."""
return _first_line(data).startswith(EXPORT_MARKER)
"""Whether `data` is a file this stack exported, in either form:
1. its first line, after an optional BOM, starts with `COMMENT_MARKER`
(`export guidelines`);
2. its first line is `---` and a line starting with `FRONTMATTER_KEY`
stands before the closing `---` line (`export run`).
Form 2 is a line scan, not a YAML parse, so a frontmatter that is no longer
valid YAML is still recognised - the marker, not the YAML, is what says
where a file came from. Looks at no more than `SCAN_BYTES`, so a caller may
pass only a file's first bytes: a frontmatter with no closing line in that
range is not an export, and neither is a key that only follows it."""
data = _without_bom(data)
head = data[:SCAN_BYTES]
lines = head.split(b"\n")
if len(head) == SCAN_BYTES and len(data) > SCAN_BYTES:
lines.pop() # a line the cut may have shortened
if lines[0].startswith(COMMENT_MARKER):
return True
if lines[0].rstrip() != FRONTMATTER_FENCE:
return False
marked = False
for line in lines[1:]:
if line.rstrip() == FRONTMATTER_FENCE:
return marked
marked = marked or line.startswith(FRONTMATTER_KEY)
return False
def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
@@ -80,9 +114,9 @@ def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
def file_is_export(path: Path) -> bool:
"""`is_export` for a file on disk, reading only its first line's worth."""
"""`is_export` for a file on disk, reading no more than it looks at."""
with open(path, "rb") as handle:
return is_export(handle.read(len(_BOM) + len(EXPORT_MARKER)))
return is_export(handle.read(len(_BOM) + SCAN_BYTES + 1))
# --- the instance it comes from -----------------------------------------------
tools/chemenu/page_export.py
+631
View File
@@ -0,0 +1,631 @@
"""The page export, with no CLI attached: a derived, read-only reading copy of
`kb/` per configured target folder.
`wikitool export run` is the terminal adapter over this module, `publish`
calls `run` for the targets that ask for it, and `raw accept` asks
`known_digests` whether a file is one this module wrote. Nothing here imports
`typer` or anything under `chemenu.commands`.
An export is never a working copy: it is regenerated from `kb/` and what a
person changes in the folder is overwritten or refused, never merged back. The
properties that carry that, each with a test:
- **Deterministic.** The output depends on the selected pages and the commit
that last touched one of them - no timestamp, no `HEAD` - so a run on an
unchanged corpus writes nothing, not even a file's modification time.
- **Only its own files are touched.** What an earlier run wrote is recorded in
a manifest in the target (`.wikitool-export.json`, with a digest per file).
A run removes only manifest files that are no longer exported, leaves every
other file byte for byte as it found it, and stops on a manifest file a
person has changed rather than overwrite the change. A non-empty folder
without a manifest is refused: it is somebody's folder, not an export.
- **Nothing is deleted on a doubtful selection.** Zero selected pages is an
error before anything is touched.
- **Atomic per file.** Each file is written to a temporary file in the target
and moved over its place; the manifest comes last.
- **One definition of an export.** The marker every file carries, and the test
for it, live in `chemenu.guideline_export`.
"""
from __future__ import annotations
import hashlib
import json
import os
import posixpath
import subprocess
import tempfile
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Optional, Sequence
from urllib.parse import quote
from chemenu import backup, config, config_shapes, guideline_export, settings
from chemenu.errors import ValidationError
from chemenu.frontmatter_io import dump_frontmatter
from chemenu.kb_scan import LINK_RE, normalize_link_target
from chemenu.markdown_code import strip_code_spans
from chemenu.page import Page
from chemenu.search import filters
from chemenu.search.service import load_pages_by_path, unreadable_pages
from chemenu.search.types import Predicate
MANIFEST_NAME = ".wikitool-export.json"
MANIFEST_SCHEMA = 1
COMMAND_TIMEOUT_SECONDS = 300
_STDERR_TAIL_CHARS = 2000
SUMMARY_KEYS = ("type", "summary", "modified")
class ExportError(ValidationError):
"""An export step refused or failed; exit 1 with the message."""
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class Target:
name: str
handler: str
path: Path
include: tuple[tuple[Predicate, ...], ...]
frontmatter: str
links: str
after_publish: bool
command_name: str = ""
command: tuple[str, ...] = ()
def _others_paths(root: Path) -> list[tuple[str, Path]]:
"""The folders other features of this checkout write into or read from -
the `intake` folders and the backup target - for the conflict check. A
section that is itself broken is that feature's own problem and is skipped
here: it is reported where it is read."""
from chemenu import upload
found: list[tuple[str, Path]] = []
try:
for name, channel in upload.read_channels(root).items():
if channel.path is not None:
found.append((f"the intake channel {name!r} ({channel.path})", channel.path))
except ValidationError:
pass
try:
backup_cfg = backup.read_config(root)
except ValidationError:
backup_cfg = None
if backup_cfg is not None:
found.append((f"backup.target ({backup_cfg.target})", backup_cfg.target))
return found
def _parse_include(name: str, include: list[list[str]], problems: list[str]) -> tuple[tuple[Predicate, ...], ...]:
selections = []
for number, selection in enumerate(include, start=1):
parsed = []
for raw in selection:
try:
parsed.append(filters.parse_predicate(raw))
except ValidationError as exc:
problems.append(f"export.targets.{name}.include selection {number}: {exc}")
selections.append(tuple(parsed))
return tuple(selections)
def read_config(root: "Path | str | None" = None) -> Optional[tuple[Target, ...]]:
"""The targets of the `export` section in the order the file lists them, or
None when the settings carry none. Raises `ValidationError` for a section
that is broken - including what the record cannot judge without the
instance and the rest of the settings: a predicate that does not parse, a
path on this machine that is not absolute, inside the instance, or in the
way of another target, an `intake` folder or the backup target."""
root = Path(config.ROOT if root is None else root)
key = config_shapes.EXPORT.key
section = settings.load(root, only=(key,)).section(key)
if not section.configured:
return None
problems: list[str] = []
targets: list[Target] = []
commands = section["commands"]
real_root = backup.real_path(root)
for name, raw in section["targets"].items():
path = Path(raw["path"])
if not os.path.isabs(raw["path"]):
problems.append(f"export.targets.{name}.path {raw['path']!r} is not an absolute path on this machine")
continue
real = backup.real_path(path)
if backup.is_under(real, real_root) or backup.is_under(real_root, real):
problems.append(f"export.targets.{name}.path {path} and the instance lie in one another - "
"an export belongs outside it")
continue
include = _parse_include(name, raw["include"], problems)
command_name = raw.get("command", "")
targets.append(Target(
name=name, handler=raw["handler"], path=path, include=include,
frontmatter=raw.get("frontmatter", config_shapes.EXPORT_DEFAULT_FRONTMATTER),
links=raw.get("links", config_shapes.EXPORT_DEFAULT_LINKS),
after_publish=raw.get("after_publish", False),
command_name=command_name, command=tuple(commands.get(command_name, ())),
))
if not problems:
placed = [(f"export.targets.{t.name}", backup.real_path(t.path)) for t in targets]
others = [(label, backup.real_path(path)) for label, path in _others_paths(root)]
for index, (label, real) in enumerate(placed):
for other_label, other in placed[index + 1:] + others:
if backup.is_under(real, other) or backup.is_under(other, real):
problems.append(f"{label} and {other_label} lie in one another - "
"every export folder is its own")
if problems:
raise ExportError("; ".join(problems))
return tuple(targets)
def require_config(root: "Path | str | None" = None) -> tuple[Target, ...]:
found = read_config(root)
if found is None:
raise ExportError(
f"no export section in {config.LOCAL_SETTINGS_FILENAME} - the user adds it by hand "
"(docs/how-to/export-to-a-synced-folder.md)"
)
return found
def _asks_for_after_publish(root: Path) -> bool:
"""Whether the raw `export` section has a target that sets `after_publish`,
read without judging the rest of it - so a section too broken to read as a
whole can still tell `publish` that it matters."""
try:
data = settings.read_file(settings.local_path(root)) or {}
except ValidationError:
return False
section = data.get(config_shapes.EXPORT.key)
targets = section.get("targets") if isinstance(section, dict) else None
if not isinstance(targets, dict):
return False
return any(isinstance(t, dict) and t.get("after_publish") for t in targets.values())
def after_publish_targets(root: "Path | str | None" = None) -> tuple[Target, ...]:
"""The targets `publish` refreshes after a commit. A broken `export` section
raises only when some target asks for that - a checkout that never set
`after_publish` is not held up by a section it does not use there."""
root = Path(config.ROOT if root is None else root)
try:
targets = read_config(root)
except ValidationError:
if _asks_for_after_publish(root):
raise
return ()
return tuple(t for t in targets or () if t.after_publish)
def choose(targets: Sequence[Target], names: Sequence[str]) -> list[Target]:
"""The targets `names` select, in the order named; all of them for no name.
An unknown name is an error before anything runs."""
if not names:
return list(targets)
by_name = {t.name: t for t in targets}
unknown = [n for n in names if n not in by_name]
if unknown:
raise ExportError(
f"unknown export target(s): {', '.join(unknown)}. Configured: {', '.join(by_name)}"
)
chosen: list[Target] = []
for name in names:
if by_name[name] not in chosen:
chosen.append(by_name[name])
return chosen
# ---------------------------------------------------------------------------
# Selection and rendering
# ---------------------------------------------------------------------------
def select(pages: dict[str, Page], target: Target, kb_dir: Path) -> list[Page]:
"""The pages of `target`: the union over its selections, each the pages
that satisfy every predicate of it, sorted by `(title.casefold(), title)`."""
predicates = tuple(p for selection in target.include for p in selection)
filters.validate_fields(predicates, pages)
chosen: dict[str, Page] = {}
for selection in target.include:
chosen.update(filters.apply_predicates(pages, selection, kb_dir))
return sorted(chosen.values(), key=lambda p: (p.title.casefold(), p.title))
def file_path(page: Page, kb_dir: Path) -> str:
return page.path.relative_to(kb_dir).as_posix()
def _rewrite_links(text: str, here: str, destinations: dict[str, Optional[str]], links: str) -> str:
"""Wikilinks outside code to Markdown: a link to an exported page relative
to `here` (`links: relative`), everything else to its display text. Matches
are found on the masked copy and applied to the original, which share
offsets, so code reaches the output byte for byte."""
masked = strip_code_spans(text)
out: list[str] = []
pos = 0
for m in LINK_RE.finditer(masked):
title = normalize_link_target(text[m.start(1):m.end(1)])
rest = text[m.start(2):m.end(2)]
display = rest.split("|", 1)[1] if "|" in rest else title
destination = destinations.get(title)
if links == "relative" and destination:
relative = posixpath.relpath(destination, posixpath.dirname(here) or ".")
replacement = f"[{display}]({quote(relative, safe='/')})"
else:
replacement = display
out.append(text[pos:m.start()])
out.append(replacement)
pos = m.end()
out.append(text[pos:])
return "".join(out)
def marker_value(instance: str, commit: str) -> str:
return f"kind={guideline_export.PAGES_KIND} instance={instance} commit={commit}"
def render_page(page: Page, here: str, destinations: dict[str, Optional[str]], target: Target,
instance: str, commit: str) -> bytes:
"""One page as the file a target holds: the frontmatter fence with the
export marker first, then the frontmatter `target.frontmatter` asks for,
then the body - its generated links block kept, wikilinks rewritten, code
untouched. LF endings, exactly one trailing newline."""
own: dict[str, Any] = {}
if target.frontmatter == "summary":
own["title"] = page.title
own.update({k: page.frontmatter[k] for k in SUMMARY_KEYS if k in page.frontmatter})
elif target.frontmatter == "full":
own.update({k: v for k, v in page.frontmatter.items() if k != "wikitool_export"})
marker = json.dumps(marker_value(instance, commit), ensure_ascii=False)
head = [f"wikitool_export: {marker}"]
if own:
head.append(dump_frontmatter(own))
body = page.body.replace("\r\n", "\n").strip("\n")
body = _rewrite_links(body, here, destinations, target.links)
text = "---\n" + "\n".join(head) + "\n---\n" + ("\n" + body + "\n" if body else "")
return text.encode("utf-8")
def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, commit: str) -> dict[str, bytes]:
"""The files of a target for `pages`, by path below the target. A page
that cannot be written on every platform, or two that would share a file
name in a case-insensitive folder, is an `ExportError`."""
paths = {id(page): file_path(page, kb_dir) for page in pages}
seen: dict[str, str] = {}
for page in pages:
rel = paths[id(page)]
if not backup.safe_relative(rel):
raise ExportError(
f"{rel} cannot be written to a folder on every platform (a name with `:`, a backslash "
"or an empty part) - rename the page or leave it out of the selection"
)
if seen.setdefault(rel.casefold(), rel) != rel:
raise ExportError(f"{rel} and {seen[rel.casefold()]} differ only in case - "
"they would be one file in a synced folder")
by_title: dict[str, Optional[str]] = {}
for page in pages:
by_title[page.title] = None if page.title in by_title else paths[id(page)]
files: dict[str, bytes] = {}
for page in pages:
rel = paths[id(page)]
files[rel] = render_page(page, rel, by_title, target, instance, commit)
return files
# ---------------------------------------------------------------------------
# The target folder
# ---------------------------------------------------------------------------
def _sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def read_manifest(folder: Path) -> Optional[dict]:
"""The manifest of the last run into `folder`, or None when there is none.
Raises `ExportError` for one that cannot be read as one."""
path = folder / MANIFEST_NAME
if not path.exists():
return None
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
raise ExportError(f"{path} cannot be read ({exc}) - nothing was written") from None
files = data.get("files") if isinstance(data, dict) else None
dirs = data.get("dirs", []) if isinstance(data, dict) else None
if (not isinstance(files, list) or not isinstance(dirs, list)
or not all(isinstance(d, str) and backup.safe_relative(d) for d in dirs)
or not all(
isinstance(entry, dict) and isinstance(entry.get("path"), str)
and backup.safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
and isinstance(entry.get("sha256"), str)
for entry in files)):
raise ExportError(f"{path} is not an export manifest - nothing was written")
return data
@dataclass
class TargetResult:
name: str
path: Path
commit: str = ""
new: list = field(default_factory=list)
changed: list = field(default_factory=list)
removed: list = field(default_factory=list)
unchanged: int = 0
skipped: list = field(default_factory=list)
error: Optional[str] = None
titles: dict = field(default_factory=dict)
@property
def ok(self) -> bool:
return self.error is None
@property
def touched(self) -> bool:
return bool(self.new or self.changed or self.removed)
def _check_parents(folder: Path, rel: str) -> None:
current = folder
for part in rel.split("/")[:-1]:
current = current / part
if backup.is_link(current):
raise ExportError(f"{current} is a link - an export writes nothing through one")
if current.exists() and not current.is_dir():
raise ExportError(f"{current} is a file where the export needs a folder - move it away; "
"nothing was written")
def _on_disk(folder: Path, rel: str) -> Optional[str]:
"""The sha256 of `rel` in `folder`, None when it is missing. A directory or
link there is an `ExportError`."""
path = folder.joinpath(*rel.split("/"))
if backup.is_link(path):
raise ExportError(f"{path} is a link - an export touches no link; nothing was written")
if not path.exists():
return None
if not path.is_file():
raise ExportError(f"{path} is not a file - nothing was written")
return backup.file_digest(path)[1]
def mirror(folder: Path, files: dict[str, bytes], result: TargetResult, *, name: str, instance: str,
commit: str) -> None:
"""Bring `folder` to exactly `files` plus its manifest, filling `result`.
Everything that can refuse refuses before the first write."""
if not files:
raise ExportError("nothing to export - an empty result would delete the earlier export, "
"so nothing was changed")
if folder.exists() and not folder.is_dir():
raise ExportError(f"{folder} exists and is not a folder")
manifest = read_manifest(folder) if folder.is_dir() else None
if folder.is_dir() and manifest is None and any(folder.iterdir()):
raise ExportError(
f"{folder} is not empty and holds no {MANIFEST_NAME} - it is not an export of this stack, "
"and an export writes only into an empty or its own folder. Nothing was written."
)
old = {entry["path"]: entry["sha256"] for entry in (manifest or {}).get("files", [])}
new_digests = {rel: _sha256(data) for rel, data in files.items()}
changed_by_hand = []
for rel, digest in old.items():
present = _on_disk(folder, rel)
if present is not None and present not in (digest, new_digests.get(rel)):
changed_by_hand.append(rel)
if changed_by_hand:
raise ExportError(
"changed in the target since the last export: " + ", ".join(changed_by_hand)
+ ". An export is never edited; move or delete the file and run again. Nothing was written."
)
for rel, data in files.items():
_check_parents(folder, rel)
present = _on_disk(folder, rel)
if rel in old:
if present == new_digests[rel]:
result.unchanged += 1
else:
result.changed.append(rel)
elif present is None:
result.new.append(rel)
elif present == new_digests[rel]:
result.new.append(rel)
else:
raise ExportError(f"{folder.joinpath(*rel.split('/'))} exists and is not one of the export's "
"files - move it away. Nothing was written.")
result.removed = sorted(rel for rel in old if rel not in files and _on_disk(folder, rel) is not None)
dirs = set((manifest or {}).get("dirs", []))
if manifest is not None and old == new_digests and not (result.new or result.changed or result.removed):
return
if not folder.exists():
folder.mkdir(parents=True)
for rel in result.new + result.changed:
current, parts = folder, rel.split("/")[:-1]
for index, part in enumerate(parts):
current = current / part
if not current.exists():
current.mkdir()
dirs.add("/".join(parts[:index + 1]))
backup.write_atomic(folder.joinpath(*rel.split("/")), [files[rel]])
for rel in result.removed:
victim = folder.joinpath(*rel.split("/"))
victim.unlink()
parent = posixpath.dirname(rel)
while parent and parent in dirs:
try:
folder.joinpath(*parent.split("/")).rmdir()
except OSError:
break
dirs.discard(parent)
parent = posixpath.dirname(parent)
listing = [{"path": rel, "size": len(files[rel]), "sha256": new_digests[rel]} for rel in sorted(files)]
body = {
"schema": MANIFEST_SCHEMA, "kind": guideline_export.PAGES_KIND, "target": name,
"instance": instance, "commit": commit, "dirs": sorted(dirs), "files": listing,
}
data = (json.dumps(body, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
backup.write_atomic(folder / MANIFEST_NAME, [data])
# ---------------------------------------------------------------------------
# The command handler
# ---------------------------------------------------------------------------
def _tail(stderr: bytes) -> str:
text = stderr.decode("utf-8", errors="replace").strip()
if len(text) > _STDERR_TAIL_CHARS:
text = "..." + text[-_STDERR_TAIL_CHARS:]
return text or "(nothing on stderr)"
def convert(target: Target, markdown: dict[str, bytes], result: TargetResult) -> dict[str, bytes]:
"""Run the target's converter over `markdown` and return what it left in
its output directory. The staging directory is always removed; any end but
exit 0 is an `ExportError` with the end of the program's stderr."""
where = f"the command {target.command_name!r}"
with tempfile.TemporaryDirectory(prefix="wikitool-export-") as tmp:
source, output = Path(tmp) / "markdown", Path(tmp) / "output"
output.mkdir()
for rel, data in markdown.items():
dest = source.joinpath(*rel.split("/"))
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_bytes(data)
try:
ran = subprocess.run(
[*target.command, str(source), str(output)], capture_output=True,
stdin=subprocess.DEVNULL, timeout=COMMAND_TIMEOUT_SECONDS, check=False, cwd=tmp,
)
except FileNotFoundError:
raise ExportError(f"{where} could not be started: {target.command[0]} does not exist") from None
except subprocess.TimeoutExpired:
raise ExportError(f"{where} did not finish within {COMMAND_TIMEOUT_SECONDS} s") from None
except OSError as exc:
raise ExportError(f"{where} could not be started ({exc})") from None
if ran.returncode != 0:
raise ExportError(f"{where} ended with exit {ran.returncode}. Its stderr:\n{_tail(ran.stderr)}")
produced: dict[str, bytes] = {}
for directory, subdirs, names in os.walk(output, followlinks=False):
base = Path(directory)
subdirs[:] = [d for d in subdirs if not backup.is_link(base / d)]
for entry in sorted(names):
path = base / entry
rel = path.relative_to(output).as_posix()
if backup.is_link(path) or not path.is_file():
result.skipped.append(rel)
continue
if not backup.safe_relative(rel):
raise ExportError(f"{where} produced {rel!r}, which is not a name that can be "
"written on every platform")
produced[rel] = path.read_bytes()
casefolded: dict[str, str] = {}
for rel in produced:
if casefolded.setdefault(rel.casefold(), rel) != rel:
raise ExportError(f"{where} produced {rel} and {casefolded[rel.casefold()]}, which differ "
"only in case")
return produced
# ---------------------------------------------------------------------------
# A run
# ---------------------------------------------------------------------------
def run(targets: Sequence[Target], root: "Path | str | None" = None) -> list[TargetResult]:
"""Export each target in `targets`, independently: one that fails is
reported in its result and the others still run. What is not per target -
a checkout that is not clean, a page whose frontmatter does not parse - is
an `ExportError` before any target is touched."""
root = Path(config.ROOT if root is None else root)
kb_dir = config.KB_DIR
guideline_export.check_clean(root)
pages = load_pages_by_path(kb_dir, root)
unreadable = unreadable_pages(pages)
if unreadable:
listed = "\n".join(f" - {u['path']} ({u['reason']})" for u in unreadable)
raise ExportError(
"These pages have frontmatter that does not parse, so the export cannot tell whether "
f"they are selected:\n{listed}\n Fix them first; nothing was exported."
)
instance = guideline_export.instance_id(root)
results = []
for target in targets:
result = TargetResult(target.name, target.path)
results.append(result)
try:
selected = select(pages, target, kb_dir)
if not selected:
raise ExportError("no page matches the selection - an empty export would delete the "
"earlier one, so nothing was changed")
result.commit = guideline_export.last_commit(selected, root)
files = render(selected, target, kb_dir, instance, result.commit)
result.titles = {file_path(p, kb_dir): p.title for p in selected}
if target.handler == config_shapes.EXPORT_HANDLER_COMMAND:
files = convert(target, files, result)
result.titles = {}
mirror(target.path, files, result, name=target.name, instance=instance, commit=result.commit)
except (ValidationError, OSError) as exc:
result.error = str(exc)
return results
# ---------------------------------------------------------------------------
# What `raw accept` asks
# ---------------------------------------------------------------------------
def known_digests(root: "Path | str | None" = None) -> tuple[set[str], list[str]]:
"""The sha256 of every file the manifests of the configured targets list,
and a warning for each reason the answer is incomplete: a section that is
broken, a target whose manifest is missing or unreadable. The check this
feeds is a second line behind the marker, so it skips what it cannot read
rather than stop a `raw accept`."""
root = Path(config.ROOT if root is None else root)
try:
targets = read_config(root)
except ValidationError as exc:
return set(), [f"the export section is broken, so files in incoming/ were not compared with "
f"the exported ones ({exc})"]
digests: set[str] = set()
warnings: list[str] = []
for target in targets or ():
try:
manifest = read_manifest(target.path)
except (ValidationError, OSError) as exc:
warnings.append(f"export target {target.name}: {exc}")
continue
if manifest is None:
warnings.append(f"export target {target.name} has no manifest at {target.path}, so its files "
"were not compared with incoming/")
continue
digests.update(entry["sha256"] for entry in manifest["files"])
return digests, warnings
def report_lines(result: TargetResult) -> list[str]:
"""What a run did to one target, for the terminal: its new files with the
page titles, then changed, removed, the count left untouched, and the
reason it failed."""
head = f"{result.name} -> {result.path}"
if result.error is not None:
return [f"{head}: failed", *(f" {line}" for line in result.error.splitlines())]
lines = [f"{head} (commit {result.commit[:12]})"]
def named(rel: str) -> str:
title = result.titles.get(rel)
return f"{title} ({rel})" if title and title != rel else rel
for label, rels in (("new", result.new), ("changed", result.changed), ("removed", result.removed)):
if rels:
lines.append(f" {label} ({len(rels)}): " + ", ".join(named(rel) for rel in rels))
lines.append(f" unchanged: {result.unchanged}")
if result.skipped:
lines.append(f" not regular files, skipped: {', '.join(result.skipped)}")
return lines
tools/chemenu/reference_doc.py
+1
View File
@@ -71,6 +71,7 @@ CONFIG_REGIONS = (
("reference-config-sites", "SITES"),
("reference-config-fetch", "FETCH"),
("reference-config-backup", "BACKUP"),
("reference-config-export", "EXPORT"),
)
_SLUG_RE = re.compile(r"[^a-z0-9]+")
tools/chemenu/repo_capture.py
+3 -5
View File
@@ -41,7 +41,7 @@ from typing import Iterator, Optional
from chemenu import config, filelock, toolpaths, web_capture
from chemenu.errors import BackendError, ValidationError
from chemenu.guideline_export import EXPORT_MARKER
from chemenu.guideline_export import is_export
MANIFEST_NAME = "_capture.json"
SCHEMA = 1
@@ -56,7 +56,6 @@ ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
GIT_TIMEOUT_SECONDS = 120.0
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
_BOM = b"\xef\xbb\xbf"
_MODE_SYMLINK = "120000"
_MODE_SUBMODULE = "160000"
@@ -446,9 +445,8 @@ def _structural_exclusion(entry: TreeEntry) -> Optional[str]:
def _content_exclusion(data: bytes) -> Optional[str]:
if data.startswith(LFS_MARKER):
return "Git LFS pointer - the content is not in the repository"
first = data[len(_BOM):] if data.startswith(_BOM) else data
if first.startswith(EXPORT_MARKER):
return "guideline export (first line starts with <!-- wikitool:export)"
if is_export(data):
return "guideline export (carries the wikitool export marker)"
return None
tools/chemenu/tests/test_config_shapes.py
+1 -1
View File
@@ -79,7 +79,7 @@ def test_the_telemetry_policy_reads_the_record():
def test_one_record_per_section():
assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
"telemetry", "remotes", "intake", "tasks", "superproductivity", "caldav", "sites", "fetch",
"backup",
"backup", "export",
]
tools/chemenu/tests/test_export_run.py
+764
View File
@@ -0,0 +1,764 @@
"""`export run`, the `export` settings section, the `after_publish` step of
`publish`, the `doctor` line, and the digest check of `raw accept` - against a
real checkout and target folders in `tmp_path`.
The target's path carries a space on purpose: the folder a sync client keeps on
Windows is `OneDrive - <company>`."""
from __future__ import annotations
import hashlib
import json
import os
import shutil
import subprocess
import sys
import textwrap
from pathlib import Path
import pytest
import typer
from typer.testing import CliRunner
from chemenu import config, config_shapes, guideline_export, page_export
from chemenu.cli import app
from chemenu.commands import doctor
from chemenu.commands.git_publish import publish_command
from chemenu.frontmatter_io import write_page
from chemenu.type_resolver import resolver
runner = CliRunner()
ALPHA = """
# Alpha
See [[Beta]], [[Beta|the beta page]] and [[Gamma]] for more, also [[Beta#Scope]].
Cited[^s-one]. Inline `[[Beta]]` stays.
```
[[Beta]] and [[Gamma]]
```
<!-- wikitool:footnotes -->
## Footnotes
[^s-one]: [[Source - One]]
<!-- /wikitool:footnotes -->
"""
BETA = """
# Beta
Back to [[Alpha]].
"""
def _git(root: Path, *args: str) -> str:
result = subprocess.run(["git", *args], cwd=root, capture_output=True, text=True)
assert result.returncode == 0, result.stderr
return result.stdout.strip()
def _commit(root: Path, message: str = "change") -> str:
_git(root, "add", "-A")
_git(root, "commit", "-q", "-m", message)
return _git(root, "rev-parse", "HEAD")
def _page(root: Path, rel: str, body: str, **frontmatter) -> Path:
path = root / "kb" / rel
meta = {"type": "types/concept.md", "tags": ["docs"], "summary": f"About {path.stem}.",
"modified": "2026-10-01", "related": ["Beta"], **frontmatter}
write_page(path, meta, body)
return path
@pytest.fixture
def instance(tmp_path, monkeypatch):
root = tmp_path / "instance"
root.mkdir()
_git(root, "init", "-q", "-b", "main")
_git(root, "config", "user.name", "Test")
_git(root, "config", "user.email", "test@example.com")
(root / "incoming").mkdir()
(root / "raw").mkdir()
(root / "incoming" / ".gitkeep").write_text("", encoding="utf-8")
(root / ".gitignore").write_text(
"/incoming/*\n!/incoming/.gitkeep\n/.wikitool-local.json\n/quarantine/\n/reports/\n",
encoding="utf-8")
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
monkeypatch.setattr(config, "TYPES_DIR", root / "types")
monkeypatch.setattr(resolver, "_repo_root", root)
_page(root, "concepts/Alpha.md", ALPHA)
_page(root, "concepts/protocols/Beta.md", BETA)
_page(root, "concepts/Gamma.md", "\n# Gamma\n\nPrivate.\n", tags=["private"])
_commit(root, "init")
(tmp_path / "OneDrive - Example").mkdir()
monkeypatch.setattr(config, "ROOT", root)
monkeypatch.setenv("WIKITOOL_SESSION_ID", "test-session")
return root
def _out(root: Path, name: str = "Pages") -> Path:
return root.parent / "OneDrive - Example" / name
def _configure(root: Path, **targets) -> None:
"""One target `pages` by default, with `docs` pages selected."""
if not targets:
targets = {"pages": {}}
section = {}
for name, extra in targets.items():
body = {"handler": "markdown", "path": str(_out(root, name)), "include": [["tags=docs"]]}
body.update(extra)
section[name] = body
commands = {body["command"]: ["unset"] for body in section.values() if body.get("command")}
export = {"targets": section, **({"commands": commands} if commands else {})}
(root / config.LOCAL_SETTINGS_FILENAME).write_text(
json.dumps({"schema": 1, "export": export}), encoding="utf-8")
def _set_commands(root: Path, commands: dict) -> None:
path = root / config.LOCAL_SETTINGS_FILENAME
data = json.loads(path.read_text(encoding="utf-8"))
data["export"]["commands"] = commands
path.write_text(json.dumps(data), encoding="utf-8")
def _run(*names: str):
return runner.invoke(app, ["export", "run", *names])
def _flat(text: str) -> str:
return " ".join(text.split())
def _snapshot(directory: Path) -> dict:
return {p.relative_to(directory).as_posix(): p.read_bytes()
for p in sorted(directory.rglob("*")) if p.is_file()}
def _mtimes(directory: Path) -> dict:
return {p.relative_to(directory).as_posix(): p.stat().st_mtime_ns
for p in sorted(directory.rglob("*")) if p.is_file()}
def _manifest(folder: Path) -> dict:
return json.loads((folder / page_export.MANIFEST_NAME).read_text(encoding="utf-8"))
# --- the section -------------------------------------------------------------
def test_no_section_is_an_error_naming_it_and_doctor_says_ok(instance):
result = _run()
assert result.exit_code == 1
assert "no export section" in _flat(result.output)
assert _check("export").status == "OK"
@pytest.mark.parametrize("target, needle", [
({"path": "relative/folder"}, "absolute"),
({"include": []}, "include"),
({"include": [[]]}, "include"),
({"include": [["not a predicate"]]}, "predicate"),
({"handler": "pdf"}, "handler"),
({"frontmatter": "everything"}, "frontmatter"),
({"links": "hyper"}, "links"),
({"after_publish": "yes"}, "after_publish"),
({"command": "convert"}, "command"),
({"handler": "command"}, "command"),
])
def test_a_broken_section_stops_run_and_doctor_fails(instance, target, needle):
_configure(instance, pages=target)
result = _run()
assert result.exit_code == 1, result.output
assert needle in _flat(result.output)
assert _check("export").status == "FAIL"
assert not _out(instance, "pages").exists()
def test_a_target_inside_the_instance_is_refused(instance):
_configure(instance, pages={"path": str(instance / "site")})
assert _run().exit_code == 1
assert _check("export").status == "FAIL"
def test_targets_lying_in_one_another_are_refused(instance):
_configure(instance, one={"path": str(_out(instance, "one"))},
two={"path": str(_out(instance, "one") / "inner")})
result = _run()
assert result.exit_code == 1
assert "lie in one another" in _flat(result.output)
def test_a_target_in_the_backup_target_or_an_intake_folder_is_refused(instance):
backup_target = _out(instance, "backup")
path = instance / config.LOCAL_SETTINGS_FILENAME
_configure(instance, pages={"path": str(backup_target / "pages")})
data = json.loads(path.read_text(encoding="utf-8"))
data["backup"] = {"target": str(backup_target)}
path.write_text(json.dumps(data), encoding="utf-8")
assert _run().exit_code == 1
data.pop("backup")
data["intake"] = {"channels": {"inbox": {
"handler": "folder", "path": str(_out(instance, "pages") / "in"), "max_bytes": 1024,
"allowed_extensions": [".md"]}}}
data["export"]["targets"]["pages"]["path"] = str(_out(instance, "pages"))
path.write_text(json.dumps(data), encoding="utf-8")
assert _run().exit_code == 1
def test_the_operator_only_fields_cannot_be_set_by_the_agent():
export = config_shapes.EXPORT
assert export.field("targets").user_only and export.field("commands").user_only
assert export.field("targets").tier == config_shapes.CHECKOUT
assert export.field("commands").tier == config_shapes.CHECKOUT
def test_an_unknown_target_name_is_an_error_and_changes_nothing(instance):
_configure(instance)
result = _run("pages", "nothing")
assert result.exit_code == 1
assert "unknown export target" in _flat(result.output)
assert not _out(instance, "pages").exists()
# --- what a run writes -------------------------------------------------------
def test_a_run_writes_the_selected_pages_with_marker_and_links(instance):
_configure(instance)
result = _run()
assert result.exit_code == 0, result.output
folder = _out(instance, "pages")
assert set(_snapshot(folder)) == {
"concepts/Alpha.md", "concepts/protocols/Beta.md", page_export.MANIFEST_NAME}
commit = _git(instance, "rev-parse", "HEAD")
alpha = (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
lines = alpha.split("\n")
assert lines[0] == "---"
assert lines[1] == f'wikitool_export: "kind=pages instance=local commit={commit}"'
assert "title: Alpha" in alpha and "summary: About Alpha." in alpha
assert "tags" not in alpha and "related" not in alpha
assert "[Beta](protocols/Beta.md)" in alpha and "[the beta page](protocols/Beta.md)" in alpha
assert "also [Beta](protocols/Beta.md)" in alpha
assert "and Gamma for more" in alpha
assert "Cited[^s-one]." in alpha and "[^s-one]: " in alpha
assert "Inline `[[Beta]]` stays." in alpha
assert "```\n[[Beta]] and [[Gamma]]\n```" in alpha
assert alpha.endswith("\n") and not alpha.endswith("\n\n") and "\r" not in alpha
beta = (folder / "concepts" / "protocols" / "Beta.md").read_text(encoding="utf-8")
assert "[Alpha](../Alpha.md)" in beta
assert guideline_export.file_is_export(folder / "concepts" / "Alpha.md")
out = _flat(result.output)
assert "new (2)" in out and "Alpha" in out and "unchanged: 0" in out
def test_frontmatter_modes_and_links_text(instance):
_configure(instance, minimal={"frontmatter": "minimal", "links": "text"},
full={"frontmatter": "full"})
assert _run().exit_code == 0
minimal = (_out(instance, "minimal") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
assert minimal.startswith('---\nwikitool_export: "kind=pages')
assert minimal.split("---\n")[1].count("\n") == 1
assert "[Beta](" not in minimal and "See Beta, the beta page and Gamma" in minimal
full = (_out(instance, "full") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
assert "tags:" in full and "type: types/concept.md" in full and "related:" in full
def test_selections_are_or_and_predicates_are_and(instance):
_configure(instance, pages={"include": [["tags=docs", "summary~Alpha"], ["tags=private"]]})
assert _run().exit_code == 0
assert set(_snapshot(_out(instance, "pages"))) == {
"concepts/Alpha.md", "concepts/Gamma.md", page_export.MANIFEST_NAME}
def test_an_unknown_field_is_an_error_at_run(instance):
_configure(instance, pages={"include": [["no_such_field=x"]]})
result = _run()
assert result.exit_code == 1
assert "unknown field" in _flat(result.output)
assert not _out(instance, "pages").exists()
def test_a_rerun_on_an_unchanged_corpus_touches_nothing(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
before, stamps = _snapshot(folder), _mtimes(folder)
result = _run()
assert result.exit_code == 0
assert _snapshot(folder) == before and _mtimes(folder) == stamps
assert "unchanged: 2" in _flat(result.output)
assert "new (" not in result.output and "changed (" not in result.output
def test_an_instance_commit_that_touches_no_selected_page_changes_nothing(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
stamps = _mtimes(folder)
(instance / "README.md").write_text("other\n", encoding="utf-8")
_commit(instance, "unrelated")
assert _run().exit_code == 0
assert _mtimes(folder) == stamps
def test_a_changed_page_a_new_page_and_a_removed_page(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_page(instance, "concepts/Delta.md", "\n# Delta\n\nNew.\n")
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
commit = _commit(instance, "edit")
result = _run()
assert result.exit_code == 0, result.output
out = _flat(result.output)
assert "new (1): Delta" in out and "changed (1):" in out and "removed (1):" in out
assert (folder / "concepts" / "Delta.md").is_file()
assert not (folder / "concepts" / "protocols" / "Beta.md").exists()
assert not (folder / "concepts" / "protocols").exists()
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
assert _manifest(folder)["commit"] == commit
assert [e["path"] for e in _manifest(folder)["files"]] == ["concepts/Alpha.md", "concepts/Delta.md"]
def test_the_selection_growing_needs_no_gate(instance):
_configure(instance)
assert _run().exit_code == 0
for index in range(30):
_page(instance, f"concepts/Many{index}.md", f"\n# Many{index}\n\nText.\n")
_commit(instance, "many")
result = _run()
assert result.exit_code == 0, result.output
assert "new (30)" in _flat(result.output)
def test_foreign_files_stay_and_only_created_directories_are_pruned(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
(folder / "notes.txt").write_bytes(b"mine\r\n")
(folder / "concepts" / "mine.txt").write_bytes(b"also mine")
(folder / "concepts" / "protocols" / "mine.txt").write_bytes(b"in a created dir")
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
_commit(instance, "drop beta")
assert _run().exit_code == 0
assert (folder / "notes.txt").read_bytes() == b"mine\r\n"
assert (folder / "concepts" / "mine.txt").read_bytes() == b"also mine"
assert (folder / "concepts" / "protocols" / "mine.txt").read_bytes() == b"in a created dir"
assert (folder / "concepts" / "Alpha.md").is_file()
def test_a_directory_that_existed_before_the_export_is_kept(instance):
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
_commit(instance, "no beta yet")
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
(folder / "concepts" / "protocols").mkdir()
_page(instance, "concepts/protocols/Beta.md", BETA)
_commit(instance, "beta")
assert _run().exit_code == 0
assert "concepts/protocols" not in _manifest(folder)["dirs"]
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
_commit(instance, "drop beta")
assert _run().exit_code == 0
assert (folder / "concepts" / "protocols").is_dir()
def test_a_non_empty_folder_without_a_manifest_is_refused(instance):
folder = _out(instance, "pages")
folder.mkdir()
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
_configure(instance)
result = _run()
assert result.exit_code == 1
assert "not empty" in _flat(result.output) and "mine.md" not in _snapshot(folder).keys() - {"mine.md"}
assert _snapshot(folder) == {"mine.md": b"mine\n"}
def test_an_empty_existing_folder_is_taken(instance):
_out(instance, "pages").mkdir()
_configure(instance)
assert _run().exit_code == 0
def test_a_file_changed_by_hand_aborts_the_target_before_any_write(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
edited = folder / "concepts" / "protocols" / "Beta.md"
edited.write_text("my own words\n", encoding="utf-8")
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_commit(instance, "edit")
before = _snapshot(folder)
result = _run()
assert result.exit_code == 1
assert "concepts/protocols/Beta.md" in _flat(result.output)
assert _snapshot(folder) == before
def test_a_missing_manifest_file_is_rewritten(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
expected = _snapshot(folder)
(folder / "concepts" / "Alpha.md").unlink()
result = _run()
assert result.exit_code == 0, result.output
assert _snapshot(folder) == expected
def test_a_crash_between_the_files_and_the_manifest_is_recovered(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
old_manifest = (folder / page_export.MANIFEST_NAME).read_bytes()
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_commit(instance, "edit")
assert _run().exit_code == 0
new_state = _snapshot(folder)
(folder / page_export.MANIFEST_NAME).write_bytes(old_manifest)
assert _run().exit_code == 0
assert _snapshot(folder) == new_state
def test_zero_selected_pages_is_an_error_and_deletes_nothing(instance):
_configure(instance)
assert _run().exit_code == 0
folder = _out(instance, "pages")
before = _snapshot(folder)
_configure(instance, pages={"include": [["tags=nothing-has-this"]]})
result = _run()
assert result.exit_code == 1
assert "no page matches" in _flat(result.output)
assert _snapshot(folder) == before
def test_a_dirty_kb_is_refused(instance):
_configure(instance)
_page(instance, "concepts/Alpha.md", ALPHA + "\nunsaved\n")
result = _run()
assert result.exit_code == 1
assert "uncommitted" in _flat(result.output)
assert not _out(instance, "pages").exists()
def test_a_page_with_unreadable_frontmatter_stops_the_run(instance):
_configure(instance)
(instance / "kb" / "concepts" / "Broken.md").write_text("---\ntype: [unclosed\n---\nBody\n", encoding="utf-8")
_commit(instance, "broken")
result = _run()
assert result.exit_code == 1
assert "Broken.md" in result.output
assert not _out(instance, "pages").exists()
def test_targets_are_independent_and_any_failure_is_exit_1(instance):
bad = _out(instance, "bad")
bad.mkdir()
(bad / "mine.md").write_text("mine\n", encoding="utf-8")
_configure(instance, bad={}, good={})
result = _run()
assert result.exit_code == 1
assert (_out(instance, "good") / "concepts" / "Alpha.md").is_file()
assert _snapshot(bad) == {"mine.md": b"mine\n"}
assert "bad" in result.output and "failed" in result.output
def test_named_targets_run_alone(instance):
_configure(instance, one={}, two={})
assert _run("two").exit_code == 0
assert _out(instance, "two").is_dir() and not _out(instance, "one").exists()
def test_a_case_only_collision_is_refused(instance):
_page(instance, "concepts/alpha.md", "\n# alpha\n\nLower.\n")
_commit(instance, "case")
_configure(instance)
result = _run()
assert result.exit_code == 1
assert "differ only in case" in _flat(result.output)
assert not _out(instance, "pages").exists()
# --- the command handler -----------------------------------------------------
def _converter(root: Path, body: str) -> list[str]:
script = root.parent / "convert.py"
script.write_text(textwrap.dedent(body), encoding="utf-8")
return [sys.executable, str(script)]
def _command_target(root: Path, program: list[str]) -> None:
_configure(root, pages={"handler": "command", "command": "convert"})
_set_commands(root, {"convert": program})
COPY_AS_TXT = """
import pathlib, sys
src, out = map(pathlib.Path, sys.argv[1:3])
for f in src.rglob("*.md"):
dest = out / f.relative_to(src).with_suffix(".txt")
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(f.read_text(encoding="utf-8").upper(), encoding="utf-8")
"""
def test_the_command_handler_exports_what_the_program_leaves(instance):
_command_target(instance, _converter(instance, COPY_AS_TXT))
result = _run()
assert result.exit_code == 0, result.output
folder = _out(instance, "pages")
assert set(_snapshot(folder)) == {
"concepts/Alpha.txt", "concepts/protocols/Beta.txt", page_export.MANIFEST_NAME}
assert "WIKITOOL_EXPORT" in (folder / "concepts" / "Alpha.txt").read_text(encoding="utf-8")
stamps = _mtimes(folder)
assert _run().exit_code == 0 and _mtimes(folder) == stamps
def test_the_command_gets_two_directories_no_stdin_and_the_staging_is_removed(instance, tmp_path):
record = tmp_path / "record.json"
program = _converter(instance, f"""
import json, os, pathlib, sys
src, out = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
stdin = sys.stdin.read()
pathlib.Path({str(record)!r}).write_text(json.dumps({{
"args": sys.argv[1:], "stdin": stdin, "cwd": os.getcwd(),
"src_files": sorted(p.name for p in src.rglob("*.md")), "out_files": os.listdir(out)}}))
(out / "all.txt").write_text("done")
""")
_command_target(instance, program)
assert _run().exit_code == 0
seen = json.loads(record.read_text(encoding="utf-8"))
assert seen["stdin"] == "" and seen["out_files"] == [] and seen["src_files"] == ["Alpha.md", "Beta.md"]
assert Path(seen["args"][0]).name == "markdown" and Path(seen["args"][1]).name == "output"
assert not Path(seen["args"][0]).parent.exists()
def test_a_failing_command_leaves_the_target_as_it_was(instance):
_command_target(instance, _converter(instance, COPY_AS_TXT))
assert _run().exit_code == 0
folder = _out(instance, "pages")
before = _snapshot(folder)
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_commit(instance, "edit")
_set_commands(instance, {"convert": _converter(instance, """
import sys
sys.stderr.write("x" * 5000 + "boom at the end")
sys.exit(3)
""")})
result = _run()
assert result.exit_code == 1
text = _flat(result.output)
assert "exit 3" in text and "boom at the end" in text and "x" * 3000 not in text
assert _snapshot(folder) == before
def test_a_missing_program_and_a_timeout_are_target_errors(instance, monkeypatch):
_command_target(instance, ["/nonexistent/converter"])
result = _run()
assert result.exit_code == 1 and "could not be started" in _flat(result.output)
_set_commands(instance, {"convert": _converter(instance, "import time; time.sleep(30)")})
monkeypatch.setattr(page_export, "COMMAND_TIMEOUT_SECONDS", 1)
result = _run()
assert result.exit_code == 1 and "did not finish" in _flat(result.output)
assert not _out(instance, "pages").exists()
@pytest.mark.skipif(os.name == "nt", reason="symlinks")
def test_symlinks_in_the_output_are_skipped_and_reported(instance):
_command_target(instance, _converter(instance, """
import os, pathlib, sys
out = pathlib.Path(sys.argv[2])
(out / "real.txt").write_text("real")
os.symlink("/etc/hostname", out / "link.txt")
"""))
result = _run()
assert result.exit_code == 0, result.output
assert set(_snapshot(_out(instance, "pages"))) == {"real.txt", page_export.MANIFEST_NAME}
assert "link.txt" in result.output
def test_a_converter_that_leaves_nothing_is_an_error(instance):
_command_target(instance, _converter(instance, "pass"))
assert _run().exit_code == 1
assert not _out(instance, "pages").exists()
# --- publish -----------------------------------------------------------------
def _publish(**overrides):
kwargs = dict(message="change", push=False, confirm=None, confirm_rebase=None, yes=False,
threshold=10, remote="origin", branch="main", path=None)
kwargs.update(overrides)
publish_command(**kwargs)
def test_publish_exports_after_the_commit_with_no_push_too(instance):
_configure(instance, pages={"after_publish": True}, manual={})
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_publish()
folder = _out(instance, "pages")
head = _git(instance, "rev-parse", "HEAD")
assert _manifest(folder)["commit"] == head
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
assert not _out(instance, "manual").exists()
def test_a_failed_export_after_publish_is_a_warning_and_the_commit_stands(instance, capsys):
folder = _out(instance, "pages")
folder.mkdir()
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
_configure(instance, pages={"after_publish": True})
head = _git(instance, "rev-parse", "HEAD")
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_publish()
out = capsys.readouterr().out
assert "WARN" in out and "export run" in out
assert _git(instance, "rev-parse", "HEAD") != head
def test_publish_without_after_publish_exports_nothing(instance):
_configure(instance)
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
_publish()
assert not _out(instance, "pages").exists()
def test_a_malformed_section_stops_publish_only_when_a_target_asks_for_it(instance):
_configure(instance, pages={"after_publish": True, "include": []})
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
head = _git(instance, "rev-parse", "HEAD")
with pytest.raises(typer.Exit) as exc:
_publish()
assert exc.value.exit_code == 1 and _git(instance, "rev-parse", "HEAD") == head
_configure(instance, pages={"include": []})
_publish()
assert _git(instance, "rev-parse", "HEAD") != head
# --- doctor ------------------------------------------------------------------
def _check(name: str):
assert name == "export"
return doctor.check_export()
def test_doctor_lists_the_targets(instance):
_configure(instance, pages={"after_publish": True})
check = _check("export")
assert check.status == "OK"
assert "pages" in check.detail and "after publish" in check.detail
# --- raw accept --------------------------------------------------------------
def _accept(path: Path):
return runner.invoke(app, ["raw", "accept", str(path), "--fidelity", "verbatim",
"--authority", "normative"])
def test_raw_accept_refuses_a_file_carrying_the_page_export_marker(instance):
_configure(instance)
assert _run().exit_code == 0
exported = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
(instance / "incoming" / "Alpha.md").write_bytes(exported)
result = _accept(instance / "incoming" / "Alpha.md")
assert result.exit_code == 1
assert "export" in _flat(result.output) and "nothing was moved" in _flat(result.output)
assert (instance / "incoming" / "Alpha.md").is_file()
assert not any((instance / "raw").rglob("*.md"))
def test_raw_accept_refuses_an_exported_file_whose_marker_a_converter_dropped(instance):
_command_target(instance, _converter(instance, COPY_AS_TXT))
assert _run().exit_code == 0
exported = (_out(instance, "pages") / "concepts" / "Alpha.txt").read_bytes()
folder = instance / "incoming" / "pack"
folder.mkdir()
(folder / "note.txt").write_text("ordinary\n", encoding="utf-8")
(folder / "Alpha.txt").write_bytes(exported)
(instance / "incoming" / "Alpha.txt").write_bytes(exported)
for target in (folder, instance / "incoming" / "Alpha.txt"):
before = _snapshot(instance / "incoming")
result = _accept(target)
assert result.exit_code == 1, result.output
assert "configured export target" in _flat(result.output)
assert "Alpha.txt" in result.output
assert _snapshot(instance / "incoming") == before
assert not any(p.is_file() for p in (instance / "raw").rglob("*"))
def test_raw_accept_warns_and_goes_on_when_a_manifest_is_missing(instance):
_configure(instance)
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
result = _accept(instance / "incoming" / "note.md")
assert "WARN" in result.output and "no manifest" in _flat(result.output)
assert "configured export target" not in result.output
def test_raw_accept_warns_and_goes_on_when_the_section_is_broken(instance):
_configure(instance, pages={"include": []})
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
result = _accept(instance / "incoming" / "note.md")
assert "WARN" in result.output and "export section is broken" in _flat(result.output)
assert "configured export target" not in result.output
def test_known_digests_lists_the_manifest_files(instance):
_configure(instance)
assert _run().exit_code == 0
digests, warnings = page_export.known_digests()
alpha = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
assert hashlib.sha256(alpha).hexdigest() in digests and warnings == []
# --- the marker --------------------------------------------------------------
FRONT = b'---\nwikitool_export: "kind=pages instance=local commit=abc"\ntitle: A\n---\n\nBody\n'
@pytest.mark.parametrize("data, expected", [
(b"<!-- wikitool:export kind=guidelines -->\nbody", True),
(FRONT, True),
(b"\xef\xbb\xbf" + FRONT, True),
(FRONT.replace(b"\n", b"\r\n"), True),
(b"---\ntitle: A\n---\nwikitool_export: x\n", False),
(b"---\ntitle: A\nwikitool_export: x\n", False),
(b"# Title\n---\nwikitool_export: x\n---\n", False),
(b"---\ntitle: A\n---\n\nBody\n", False),
(b"", False),
])
def test_is_export_recognises_both_forms(data, expected):
assert guideline_export.is_export(data) is expected
def test_the_key_beyond_the_scan_window_is_not_seen():
padding = b"x: " + b"y" * 100 + b"\n"
data = b"---\n" + padding * (guideline_export.SCAN_BYTES // len(padding) + 2) + b"wikitool_export: z\n---\n"
assert guideline_export.is_export(data) is False
def test_file_is_export_reads_files(tmp_path):
(tmp_path / "a.md").write_bytes(FRONT)
(tmp_path / "b.md").write_bytes(b"plain\n")
assert guideline_export.file_is_export(tmp_path / "a.md")
assert not guideline_export.file_is_export(tmp_path / "b.md")
def test_the_marker_key_is_defined_once():
package = Path(guideline_export.__file__).parent
definitions = [
path for path in package.rglob("*.py")
if "tests" not in path.parts and "wikitool_export:" in path.read_text("utf-8")
and 'b"wikitool_export:"' in path.read_text("utf-8")
]
assert definitions == [Path(guideline_export.__file__)]