export: export run writes selected kb/ pages as a derived, read-only version into configured target folders (markdown and command handlers, manifest, after_publish), raw accept also refuses manifest-listed files, doctor checks the section (#214); 10.0.0-beta.13
Files changed: - CHANGES.md - VERSION - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/export-guidelines.md - docs/how-to/export-to-a-synced-folder.md - docs/reference/commands/README.md - docs/reference/commands/exports.md - docs/reference/commands/git.md - docs/reference/commands/guideline-export.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/setup-instance.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/backup.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/export_cmd.py - tools/chemenu/commands/git_publish.py - tools/chemenu/commands/raw/_common.py - tools/chemenu/commands/raw/accept.py - tools/chemenu/commands/raw/capture.py - tools/chemenu/config_shapes.py - tools/chemenu/guideline_export.py - tools/chemenu/page_export.py - tools/chemenu/reference_doc.py - tools/chemenu/repo_capture.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_export_run.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
ecefe0a033
commit
99945c4841
33 files changed
+2316
-96
No files matched your search
+70
-7
@@ -42,7 +42,7 @@ ON FAILURE and NEVER, which address an agent.
|
||||
- [Provenance](#provenance)
|
||||
- [Raw material and uploads](#raw-material-and-uploads)
|
||||
- [Git](#git)
|
||||
- [Guideline export](#guideline-export)
|
||||
- [Exports](#exports)
|
||||
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
|
||||
- [Types, instructions and docs](#types-instructions-and-docs)
|
||||
- [Telemetry](#telemetry)
|
||||
@@ -140,6 +140,7 @@ publish write non-idempotent budget:counted exit:0,1,
|
||||
backup init write idempotent budget:counted exit:0,1 Create the backup target's bare repository and make it `origin` when there is none.
|
||||
backup run write idempotent budget:counted exit:0,1 Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
||||
export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
|
||||
export run write idempotent budget:counted exit:0,1 Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||
work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run.
|
||||
work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop.
|
||||
budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history.
|
||||
@@ -1671,7 +1672,7 @@ Capture documentation from a git repository into `incoming/<bundle>/`, with a ma
|
||||
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
||||
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
||||
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
||||
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
||||
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
||||
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
||||
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
||||
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
||||
@@ -1806,7 +1807,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- 0 success
|
||||
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
||||
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
|
||||
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
|
||||
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
||||
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
||||
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
||||
@@ -1821,7 +1822,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
|
||||
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form
|
||||
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
|
||||
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
|
||||
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
||||
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
|
||||
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
|
||||
@@ -2178,11 +2179,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
|
||||
- 0 success
|
||||
- 0 The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
|
||||
- 0 The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
|
||||
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
||||
- 1 The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
||||
- 1 No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
||||
- 1 `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
||||
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed
|
||||
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
|
||||
- 42 The remotes settings file of an earlier version is still there
|
||||
- 42 Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
||||
- 42 Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
||||
@@ -2191,11 +2193,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
**ON FAILURE**
|
||||
|
||||
- The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool backup run` repeats the backup
|
||||
- The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool export run` repeats the export
|
||||
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change) -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
|
||||
- The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case -> Check out the branch you mean to publish, or pass `--branch <checked-out branch>`, then retry once
|
||||
- No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed -> Show the message to the user and ask whether to commit locally with `--no-push`. Never push by hand - the next `publish` that reaches the remote sends that commit
|
||||
- `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error -> Drop it. The Mass-Update Gate is cleared only with `--confirm <token>` from the gate's own refusal output
|
||||
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
|
||||
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
|
||||
- The remotes settings file of an earlier version is still there -> Show the output to the user verbatim and stop; `tools/wikitool config migrate` moves it, and the user decides when
|
||||
- Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||
- Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||
@@ -2225,6 +2228,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
|
||||
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
|
||||
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
|
||||
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
|
||||
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
|
||||
|
||||
**SEE ALSO**
|
||||
@@ -2233,6 +2237,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
- `instructions/gates.md` - the gate procedure
|
||||
- `instructions/setup-instance.md` - the first publish of a new instance
|
||||
- `wikitool backup run` - the machine-data backup on its own
|
||||
- `wikitool export run` - the page export on its own
|
||||
|
||||
#### `backup init`
|
||||
|
||||
@@ -2337,7 +2342,7 @@ Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
||||
- `wikitool doctor` - the `backup` line reports the age of the last run
|
||||
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
|
||||
|
||||
### Guideline export
|
||||
### Exports
|
||||
|
||||
#### `export guidelines`
|
||||
|
||||
@@ -2405,6 +2410,63 @@ Render the guideline pages into one generated `GUIDELINES.md`, or push it into t
|
||||
- `wikitool raw capture` - how a repository becomes a target
|
||||
- `instructions/gates.md` - the gate procedure
|
||||
|
||||
#### `export run`
|
||||
|
||||
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: yes
|
||||
- atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool export run`
|
||||
- `tools/wikitool export run work-notes`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
|
||||
- 1 `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
|
||||
- 1 A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target) -> Show the message to the user; the section is theirs to write by hand, never yours
|
||||
- `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter -> Fix the named pages, or publish the changes first, then retry once
|
||||
- A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0 -> Show the target's reason to the user; the other targets were still served. Never delete or edit files in a target folder to make a run pass
|
||||
|
||||
**NEVER**
|
||||
|
||||
- Never edit a file in a target folder to change an export - the next run refuses or overwrites it; change the page, or the target's `include`, instead.
|
||||
- Never put an export folder into `incoming/` or `raw/`.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
|
||||
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
|
||||
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
||||
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
||||
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
||||
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
||||
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
||||
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
||||
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool search` - the same predicates, to preview a selection
|
||||
- `wikitool backup run` - the same kind of folder, for this machine's own files
|
||||
- `docs/how-to/export-to-a-synced-folder.md`
|
||||
|
||||
### Workshop runs and session budget
|
||||
|
||||
#### `work new`
|
||||
@@ -4073,6 +4135,7 @@ Check that this instance is correctly configured.
|
||||
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
||||
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
||||
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
||||
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
|
||||
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
||||
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
||||
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
||||
|
||||
+2
-1
@@ -132,7 +132,8 @@ tools/
|
||||
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
|
||||
backup.py `backup init`/`backup run`'s core and the step `publish` runs after a push: the bare repository under `<target>/git`, the machine-data mirror under `<target>/<machine>/` with its manifest, and which synced folders `doctor` knows
|
||||
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
|
||||
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
|
||||
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`is_export`, for both the guideline comment line and the page frontmatter key) - with no CLI attached
|
||||
page_export.py `export run`'s core and the step `publish` runs after a commit: the `export` section, rendering selected pages as Markdown, the target folder with its manifest, and the converter handler - with no CLI attached
|
||||
search/ pluggable search backends, plus service.py - the search core
|
||||
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (the tracker sections of the settings), one module per adapter - no instruction ever learns which provider it is
|
||||
commands/ one module per command or command group: the terminal adapters - except `raw/`, a package with one module per subcommand and the helpers several of them share in `_common.py`
|
||||
|
||||
+29
-29
@@ -99,12 +99,12 @@ class BackupConfig:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _is_link(path: Path) -> bool:
|
||||
def is_link(path: Path) -> bool:
|
||||
is_junction = getattr(path, "is_junction", None) # Python 3.12+
|
||||
return path.is_symlink() or bool(is_junction and is_junction())
|
||||
|
||||
|
||||
def _real(path: Path) -> Path:
|
||||
def real_path(path: Path) -> Path:
|
||||
return Path(os.path.realpath(path))
|
||||
|
||||
|
||||
@@ -121,7 +121,7 @@ def _names_git(parts) -> bool:
|
||||
return any(part.casefold() == GIT_NAME for part in parts)
|
||||
|
||||
|
||||
def _safe_relative(rel: str) -> bool:
|
||||
def safe_relative(rel: str) -> bool:
|
||||
"""Whether `rel`, a `/`-separated path, stays below the folder it is joined
|
||||
to on every platform: no empty, `.` or `..` part, no backslash, no drive."""
|
||||
parts = rel.split("/")
|
||||
@@ -132,13 +132,13 @@ def _safe_relative(rel: str) -> bool:
|
||||
def _within(machine_dir: Path, rel: str) -> Path:
|
||||
"""`rel` below `machine_dir`, refusing a path that would leave it - by its
|
||||
spelling, or through a link somewhere in the folder."""
|
||||
if not _safe_relative(rel):
|
||||
if not safe_relative(rel):
|
||||
raise BackupError(f"{rel!r} is not a path below {machine_dir} - nothing further was written")
|
||||
dest = machine_dir.joinpath(*rel.split("/"))
|
||||
current = machine_dir
|
||||
for part in rel.split("/")[:-1]:
|
||||
current = current / part
|
||||
if _is_link(current):
|
||||
if is_link(current):
|
||||
raise BackupError(f"{current} is a link - a backup writes nothing through one")
|
||||
return dest
|
||||
|
||||
@@ -174,13 +174,13 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
|
||||
if not section.configured:
|
||||
return None
|
||||
problems = []
|
||||
real_root = _real(root)
|
||||
real_root = real_path(root)
|
||||
|
||||
target = Path(section["target"])
|
||||
if not os.path.isabs(section["target"]):
|
||||
problems.append(f"backup.target {section['target']!r} is not an absolute path on this machine")
|
||||
else:
|
||||
real_target = _real(target)
|
||||
real_target = real_path(target)
|
||||
if is_under(real_target, real_root):
|
||||
problems.append(f"backup.target {target} lies inside the instance - it belongs outside it")
|
||||
|
||||
@@ -195,7 +195,7 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
|
||||
|
||||
if not problems:
|
||||
for inner in (target / config_shapes.BACKUP_GIT_DIR, target / machine_name):
|
||||
if is_under(real_root, _real(inner)):
|
||||
if is_under(real_root, real_path(inner)):
|
||||
problems.append(f"the instance lies inside {inner}, which the backup writes to")
|
||||
|
||||
for entry in section["extra"]:
|
||||
@@ -249,14 +249,14 @@ def same_location(url: str, path: Path, root: Path) -> bool:
|
||||
local = url[len("file://"):] if url.startswith("file://") else url
|
||||
if "://" in local or (":" in local and not os.path.isabs(local)):
|
||||
return False
|
||||
a, b = os.path.normcase(_real(root / local)), os.path.normcase(_real(path))
|
||||
a, b = os.path.normcase(real_path(root / local)), os.path.normcase(real_path(path))
|
||||
return a.casefold() == b.casefold() if IGNORE_CASE else a == b
|
||||
|
||||
|
||||
def _directory_state(path: Path) -> str:
|
||||
if not path.exists() and not _is_link(path):
|
||||
if not path.exists() and not is_link(path):
|
||||
return "absent"
|
||||
if path.is_dir() and not _is_link(path) and not any(path.iterdir()):
|
||||
if path.is_dir() and not is_link(path) and not any(path.iterdir()):
|
||||
return "empty"
|
||||
return "bare" if is_bare_repo(path) else "other"
|
||||
|
||||
@@ -329,7 +329,7 @@ class Collected:
|
||||
|
||||
|
||||
def _walk(path: Path, rel: str, out: Collected) -> None:
|
||||
if _is_link(path):
|
||||
if is_link(path):
|
||||
out.skipped.append(f"{rel} (a link - not followed)")
|
||||
return
|
||||
if path.name.casefold() == GIT_NAME:
|
||||
@@ -337,7 +337,7 @@ def _walk(path: Path, rel: str, out: Collected) -> None:
|
||||
if path.is_file():
|
||||
if rel == MANIFEST_NAME:
|
||||
out.skipped.append(f"{rel} (the manifest's own name)")
|
||||
elif not _safe_relative(rel):
|
||||
elif not safe_relative(rel):
|
||||
out.skipped.append(f"{rel} (a name that cannot be mirrored on every platform)")
|
||||
else:
|
||||
out.files.setdefault(rel, path)
|
||||
@@ -351,13 +351,13 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
|
||||
"""What a machine-data run mirrors: the root files that exist, everything in
|
||||
`incoming/` but its `.gitkeep`, and `extra`."""
|
||||
root = Path(config.ROOT if root is None else root)
|
||||
real_root = _real(root)
|
||||
real_root = real_path(root)
|
||||
out = Collected()
|
||||
for name in ROOT_FILES:
|
||||
if (root / name).exists() or _is_link(root / name):
|
||||
if (root / name).exists() or is_link(root / name):
|
||||
_walk(root / name, name, out)
|
||||
incoming = root / INCOMING_DIR
|
||||
if _is_link(incoming):
|
||||
if is_link(incoming):
|
||||
out.skipped.append(f"{INCOMING_DIR} (a link - not followed)")
|
||||
elif incoming.is_dir():
|
||||
for child in sorted(incoming.iterdir()):
|
||||
@@ -375,7 +375,7 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
|
||||
return out
|
||||
|
||||
|
||||
def _digest(path: Path) -> tuple[int, str]:
|
||||
def file_digest(path: Path) -> tuple[int, str]:
|
||||
sha, size = hashlib.sha256(), 0
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||
@@ -395,7 +395,7 @@ def _replace(tmp: str, dest: Path) -> None:
|
||||
time.sleep(_REPLACE_PAUSE_SECONDS)
|
||||
|
||||
|
||||
def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
||||
def write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(prefix=".chemenu-", suffix=".tmp", dir=str(dest.parent))
|
||||
sha, size = hashlib.sha256(), 0
|
||||
@@ -416,7 +416,7 @@ def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
||||
|
||||
def _copy_atomic(src: Path, dest: Path) -> tuple[int, str]:
|
||||
with open(src, "rb") as handle:
|
||||
return _write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
|
||||
return write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
|
||||
|
||||
|
||||
def read_manifest(machine_dir: Path) -> Optional[dict]:
|
||||
@@ -432,7 +432,7 @@ def read_manifest(machine_dir: Path) -> Optional[dict]:
|
||||
files = data.get("files") if isinstance(data, dict) else None
|
||||
if not isinstance(files, list) or not all(
|
||||
isinstance(entry, dict) and isinstance(entry.get("path"), str)
|
||||
and _safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
||||
and safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
||||
and isinstance(entry.get("sha256"), str)
|
||||
for entry in files
|
||||
):
|
||||
@@ -456,7 +456,7 @@ class RunResult:
|
||||
return line
|
||||
|
||||
|
||||
def _prune_empty(directory: Path, stop: Path) -> None:
|
||||
def prune_empty(directory: Path, stop: Path) -> None:
|
||||
while directory != stop and is_under(directory, stop):
|
||||
try:
|
||||
directory.rmdir()
|
||||
@@ -484,7 +484,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
||||
"synced folder there? `tools/wikitool backup init` creates the target"
|
||||
)
|
||||
machine_dir = cfg.machine_dir
|
||||
if _is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
|
||||
if is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
|
||||
raise BackupError(f"{machine_dir} is not a plain directory - nothing was written")
|
||||
previous = read_manifest(machine_dir)
|
||||
if previous is None and machine_dir.is_dir() and any(machine_dir.iterdir()):
|
||||
@@ -500,10 +500,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
||||
machine_dir.mkdir(exist_ok=True)
|
||||
for rel, source in sorted(collected.files.items()):
|
||||
dest = _within(machine_dir, rel)
|
||||
size, digest = _digest(source)
|
||||
size, digest = file_digest(source)
|
||||
old = before.get(rel)
|
||||
if (old and old["size"] == size and old["sha256"] == digest and dest.is_file()
|
||||
and not _is_link(dest) and dest.stat().st_size == size):
|
||||
and not is_link(dest) and dest.stat().st_size == size):
|
||||
result.unchanged.append(rel)
|
||||
else:
|
||||
size, digest = _copy_atomic(source, dest)
|
||||
@@ -512,10 +512,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
||||
|
||||
for rel in sorted(set(before) - set(collected.files)):
|
||||
dest = _within(machine_dir, rel)
|
||||
if dest.is_file() and not _is_link(dest):
|
||||
if dest.is_file() and not is_link(dest):
|
||||
dest.unlink()
|
||||
result.removed.append(rel)
|
||||
_prune_empty(dest.parent, machine_dir)
|
||||
prune_empty(dest.parent, machine_dir)
|
||||
|
||||
manifest = {
|
||||
"schema": MANIFEST_SCHEMA,
|
||||
@@ -525,7 +525,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
||||
"files": entries,
|
||||
}
|
||||
text = json.dumps(manifest, indent=2, ensure_ascii=False) + "\n"
|
||||
_write_atomic(cfg.manifest, [text.encode("utf-8")])
|
||||
write_atomic(cfg.manifest, [text.encode("utf-8")])
|
||||
return result
|
||||
|
||||
|
||||
@@ -544,9 +544,9 @@ def sync_roots() -> list[Path]:
|
||||
def synced_folder_of(path: Path) -> Optional[Path]:
|
||||
"""The synced folder `path` lies in, links and junctions resolved on both
|
||||
sides, or None."""
|
||||
real = _real(path)
|
||||
real = real_path(path)
|
||||
for candidate in sync_roots():
|
||||
if is_under(real, _real(candidate)):
|
||||
if is_under(real, real_path(candidate)):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
@@ -293,8 +293,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("Git", (
|
||||
"sync", "publish", "backup init", "backup run",
|
||||
)),
|
||||
("Guideline export", (
|
||||
"export guidelines",
|
||||
("Exports", (
|
||||
"export guidelines", "export run",
|
||||
)),
|
||||
("Workshop runs and session budget", (
|
||||
"work new", "work close", "budget status", "budget reset",
|
||||
|
||||
@@ -53,6 +53,7 @@ def _git(args: list[str]) -> Optional[subprocess.CompletedProcess]:
|
||||
|
||||
PREFLIGHT_FIX = f"Run tools/preflight.sh (PowerShell 7: {toolpaths.PREFLIGHT_PWSH})"
|
||||
BACKUP_HOW_TO = "docs/how-to/back-up-to-a-synced-folder.md"
|
||||
EXPORT_HOW_TO = "docs/how-to/export-to-a-synced-folder.md"
|
||||
|
||||
|
||||
def check_python() -> Check:
|
||||
@@ -640,6 +641,27 @@ def _telemetry_file_deviations() -> list[str]:
|
||||
return config_shapes.deviations(config_shapes.TELEMETRY, data[config_shapes.TELEMETRY.key])
|
||||
|
||||
|
||||
def check_export() -> Check:
|
||||
"""The `export` section: absent is `OK` (no page export configured), a
|
||||
broken one is a `FAIL`, and a configured one lists its targets."""
|
||||
from chemenu import page_export
|
||||
|
||||
try:
|
||||
targets = page_export.read_config(config.ROOT)
|
||||
except ChemenuError as exc:
|
||||
return Check(
|
||||
"export", "FAIL", str(exc),
|
||||
f"Fix the export section as the message says - see {EXPORT_HOW_TO}",
|
||||
)
|
||||
if targets is None:
|
||||
return Check("export", "OK", "absent - no export section in the settings")
|
||||
parts = [
|
||||
f"{t.name} ({t.handler}, {t.path}{', after publish' if t.after_publish else ''})"
|
||||
for t in targets
|
||||
]
|
||||
return Check("export", "OK", f"{len(targets)} target(s): " + "; ".join(parts))
|
||||
|
||||
|
||||
def check_upload_intake() -> Check:
|
||||
"""Which intake channels this checkout has, how many submissions wait in
|
||||
the quarantine, and whether the quarantine of earlier versions
|
||||
@@ -1036,6 +1058,7 @@ def run_doctor() -> list[Check]:
|
||||
check_sync_folder(),
|
||||
check_backup(),
|
||||
check_upload_intake(),
|
||||
check_export(),
|
||||
check_tasks_provider(),
|
||||
*check_settings(),
|
||||
check_site_handlers(),
|
||||
@@ -1095,6 +1118,9 @@ def run_doctor() -> list[Check]:
|
||||
"Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not "
|
||||
"push there, or - with `machine_data` true - no machine-data backup exists yet; "
|
||||
"otherwise `OK` with the age of the last machine-data backup.",
|
||||
"Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; "
|
||||
"otherwise `OK` listing each target with its handler, folder and whether it runs after "
|
||||
"`publish`.",
|
||||
"Intake channels (`upload-intake`): the channels of the `intake` section with their "
|
||||
"handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no "
|
||||
"channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an "
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""`wikitool export guidelines` - the instance's guideline pages as one
|
||||
generated `GUIDELINES.md`, printed, or written into the captured repositories
|
||||
behind the Guideline Push Gate.
|
||||
behind the Guideline Push Gate - and `wikitool export run`, the derived reading
|
||||
copy of `kb/` in the folders the `export` section configures
|
||||
(`chemenu/page_export.py`; no gate, since a target is a folder the user named).
|
||||
|
||||
The core - selection, rendering, the header - is `chemenu/guideline_export.py`;
|
||||
the git write is `repo_capture.commit_file`/`push_commit`. This module decides
|
||||
@@ -42,12 +44,12 @@ from typing import Optional
|
||||
import typer
|
||||
from rich.markup import escape
|
||||
|
||||
from chemenu import cli_contract, config, guideline_export, repo_capture
|
||||
from chemenu import cli_contract, config, guideline_export, page_export, repo_capture
|
||||
from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path
|
||||
from chemenu.errors import BackendError, ValidationError
|
||||
from chemenu.search import filters
|
||||
|
||||
app = typer.Typer(help="Generate files from the wiki for other repositories.")
|
||||
app = typer.Typer(help="Generate files from the wiki for other repositories and folders.")
|
||||
|
||||
GATE = "guideline-push"
|
||||
|
||||
@@ -427,3 +429,127 @@ def export_guidelines_command(
|
||||
"written; run the command again for the rest."
|
||||
)
|
||||
raise typer.Exit(code=1)
|
||||
|
||||
|
||||
@app.command("run")
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="export run",
|
||||
summary="Write the derived, read-only reading copy of `kb/` into the folders the `export` "
|
||||
"section configures - the selected pages as Markdown, or converted by a program of this "
|
||||
"checkout.",
|
||||
synopsis=(
|
||||
cli_contract.Variant(
|
||||
usage="export run [<target> ...]",
|
||||
notes="Every configured target, or the named ones. Needs an `export` section in "
|
||||
f"`{config.LOCAL_SETTINGS_FILENAME}`; the user writes it by hand.",
|
||||
),
|
||||
),
|
||||
properties=cli_contract.Properties(
|
||||
effect=cli_contract.Effect.WRITE,
|
||||
idempotent=cli_contract.Idempotent.YES,
|
||||
atomic="Per file within a target (a temporary file in the target, then a move over its "
|
||||
"place), the manifest last; targets are independent, so a failed target does not undo "
|
||||
"another",
|
||||
budget=cli_contract.Budget.COUNTED,
|
||||
),
|
||||
notes=(
|
||||
"An export is derived: regenerated from `kb/` on every run, never edited, never read back "
|
||||
"into the wiki. Each target names its pages with an allowlist of `search --field` "
|
||||
"predicates (`include`: a list of selections, AND within one, OR across them), a handler "
|
||||
"(`markdown`, or `command` for the operator's converter) and the folder (`path`, outside "
|
||||
"the instance). Nothing is exported that no selection names.",
|
||||
"Every Markdown file starts with a frontmatter block whose `wikitool_export` line says "
|
||||
"`kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit "
|
||||
"touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the "
|
||||
"page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, "
|
||||
"`full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A "
|
||||
"wikilink to an exported page becomes a relative Markdown link (`links: relative`, the "
|
||||
"default) or plain text (`links: text`); a link to a page that is not exported is always "
|
||||
"plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: "
|
||||
"the same pages give the same bytes.",
|
||||
"Handler `command`: the Markdown is rendered into a staging directory, the program of "
|
||||
"`export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives "
|
||||
"the Markdown directory and an empty output directory as its last two arguments. On exit "
|
||||
"0 the regular files in the output directory (recursively) become the target's content; "
|
||||
"any other end leaves the target unchanged and prints the end of its stderr. The staging "
|
||||
"directory is always removed.",
|
||||
"The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: "
|
||||
"the files an earlier run wrote, each with its sha256. A run removes only manifest files "
|
||||
"that are no longer exported (and folders it created, once empty), rewrites a manifest "
|
||||
"file that is missing, leaves every other file byte for byte as it was, and writes "
|
||||
"nothing - not even a modification time - for a file that did not change. A run that "
|
||||
"changes nothing writes nothing at all.",
|
||||
"Refuses a target, before writing anything into it: a non-empty folder with no manifest "
|
||||
"(it is somebody's, not an export), a manifest file that was changed in the folder since "
|
||||
"the last run (naming it), a foreign file where an exported one belongs, an empty "
|
||||
"selection (it would delete the earlier export), an unknown field in a predicate. A "
|
||||
"missing folder is created.",
|
||||
"Refuses everything, before any target: uncommitted changes in `kb/` or `types/` "
|
||||
"(untracked files included), no git checkout, a page under `kb/` with unreadable "
|
||||
"frontmatter, an unknown target name, a broken `export` section.",
|
||||
"Output per target: `new` (with the page titles), `changed`, `removed`, the count "
|
||||
"`unchanged`, or `failed` with the reason. A page that newly enters a selection is "
|
||||
"reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the "
|
||||
"others were still served.",
|
||||
"A target with `after_publish: true` is also refreshed by every `publish` that "
|
||||
"committed, `--no-push` included; a failure there is a warning, the commit stands, and "
|
||||
"this command is the repeat.",
|
||||
"`raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries "
|
||||
"the marker: an export never goes back into `raw/`.",
|
||||
),
|
||||
failures=(
|
||||
cli_contract.Failure(
|
||||
cause="No `export` section, an unknown target name, or a broken section (a predicate "
|
||||
"that does not parse, a path inside the instance or in the way of another target, an "
|
||||
"`intake` folder or the backup target)",
|
||||
reaction="Show the message to the user; the section is theirs to write by hand, never "
|
||||
"yours",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="`kb/` or `types/` has uncommitted changes, the instance is not a git checkout, "
|
||||
"or a page under `kb/` has unreadable frontmatter",
|
||||
reaction="Fix the named pages, or publish the changes first, then retry once",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="A target failed: a foreign folder, a file changed in the target, an empty "
|
||||
"selection, an unknown field, a converter that did not exit 0",
|
||||
reaction="Show the target's reason to the user; the other targets were still served. "
|
||||
"Never delete or edit files in a target folder to make a run pass",
|
||||
),
|
||||
),
|
||||
examples=(
|
||||
"tools/wikitool export run",
|
||||
"tools/wikitool export run work-notes",
|
||||
),
|
||||
never=(
|
||||
"Never edit a file in a target folder to change an export - the next run refuses or "
|
||||
"overwrites it; change the page, or the target's `include`, instead.",
|
||||
"Never put an export folder into `incoming/` or `raw/`.",
|
||||
),
|
||||
see_also=(
|
||||
"`wikitool search` - the same predicates, to preview a selection",
|
||||
"`wikitool backup run` - the same kind of folder, for this machine's own files",
|
||||
"`docs/how-to/export-to-a-synced-folder.md`",
|
||||
),
|
||||
))
|
||||
def export_run_command(
|
||||
names: Optional[list[str]] = typer.Argument(
|
||||
None, metavar="[TARGET]...", help="Targets to run; default: every configured target."
|
||||
),
|
||||
):
|
||||
"""Write the reading copy of kb/ into the configured export folders."""
|
||||
try:
|
||||
targets = page_export.choose(page_export.require_config(), names or [])
|
||||
results = page_export.run(targets)
|
||||
except ValidationError as exc:
|
||||
fail(escape(str(exc)))
|
||||
return
|
||||
failed = 0
|
||||
for result in results:
|
||||
for line in page_export.report_lines(result):
|
||||
typer.echo(line)
|
||||
failed += not result.ok
|
||||
if failed:
|
||||
console.print(f"[bold red]ERROR[/bold red] {failed} of {len(results)} target(s) failed - the "
|
||||
"others were served; fix the reason above and run again.")
|
||||
raise typer.Exit(code=1)
|
||||
@@ -180,6 +180,52 @@ def read_backup_config():
|
||||
fail(str(exc))
|
||||
|
||||
|
||||
# --- Export after publish -------------------------------------------------------
|
||||
#
|
||||
# A target of the `export` section with `after_publish` is refreshed once a
|
||||
# commit exists - `--no-push` included, since the export reads the committed
|
||||
# state of `kb/`, not what reached the remote. The commit has happened by then,
|
||||
# so a failing export is a WARN and exit 0, for the same reason as the backup
|
||||
# above: `export run` repeats it, another `publish` cannot.
|
||||
|
||||
|
||||
def read_export_targets():
|
||||
"""The targets to refresh after this publish; exit 1 for a broken `export`
|
||||
section when some target asks for the refresh."""
|
||||
from chemenu import page_export
|
||||
from chemenu.errors import ChemenuError
|
||||
|
||||
try:
|
||||
return page_export.after_publish_targets()
|
||||
except ChemenuError as exc:
|
||||
fail(str(exc))
|
||||
|
||||
|
||||
def export_after_publish(targets) -> None:
|
||||
from rich.markup import escape
|
||||
|
||||
from chemenu import page_export
|
||||
from chemenu.errors import ChemenuError
|
||||
|
||||
try:
|
||||
results = page_export.run(targets)
|
||||
except (ChemenuError, OSError) as exc:
|
||||
console.print(
|
||||
f"[yellow]WARN[/yellow] The export failed: {escape(str(exc))}\n"
|
||||
" The commit stands. Show this to the user; once the cause is fixed, "
|
||||
"`tools/wikitool export run` repeats it - another `publish` does not."
|
||||
)
|
||||
return
|
||||
for result in results:
|
||||
for line in page_export.report_lines(result):
|
||||
if result.ok:
|
||||
typer.echo(line)
|
||||
else:
|
||||
console.print(f"[yellow]WARN[/yellow] {escape(line)}")
|
||||
if not result.ok:
|
||||
console.print(" The commit stands; `tools/wikitool export run` repeats this target.")
|
||||
|
||||
|
||||
def back_up_machine_data(cfg) -> None:
|
||||
from rich.markup import escape
|
||||
|
||||
@@ -1614,6 +1660,12 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
||||
"stand. No backup with `--no-push`, after a failed push, or when there was nothing to "
|
||||
"push. A malformed `backup` section stops the call with exit 1 before anything is "
|
||||
"committed, `--no-push` included.",
|
||||
"Export after publish: every target of the `export` section with `after_publish: true` "
|
||||
"is refreshed once a commit was made - the same step as `export run`, `--no-push` "
|
||||
"included, since the export reads the committed `kb/`. A failure there prints a `WARN` "
|
||||
"line with the reason and still exits 0, since the commit stands; `export run` repeats "
|
||||
"it. A malformed `export` section stops the call with exit 1 before anything is "
|
||||
"committed, but only when some target sets `after_publish`.",
|
||||
"After a successful commit or push whose changed files include `tools/`, `types/`, "
|
||||
"`instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder "
|
||||
"line: the phase past this point (an issue-body rewrite, `docs/` staleness, a "
|
||||
@@ -1651,7 +1703,8 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="The settings files do not pass their check, `remotes.allowed_push_urls` is "
|
||||
"not a list of strings, or the `backup` section is malformed; nothing was committed",
|
||||
"not a list of strings, the `backup` section is malformed, or the `export` section is "
|
||||
"malformed while a target sets `after_publish`; nothing was committed",
|
||||
reaction="Show the error to the user and stop - a malformed section is not "
|
||||
"permission, and fixing it is theirs to do",
|
||||
),
|
||||
@@ -1662,6 +1715,13 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
||||
"fixed, `tools/wikitool backup run` repeats the backup",
|
||||
code=0,
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="The export after the commit failed: a `WARN` line names the target and the "
|
||||
"reason, and the commit stands",
|
||||
reaction="Show the `WARN` line to the user. Do not publish again - once the cause is "
|
||||
"fixed, `tools/wikitool export run` repeats the export",
|
||||
code=0,
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="The remotes settings file of an earlier version is still there",
|
||||
reaction="Show the output to the user verbatim and stop; `tools/wikitool config "
|
||||
@@ -1710,6 +1770,7 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
||||
"`instructions/gates.md` - the gate procedure",
|
||||
"`instructions/setup-instance.md` - the first publish of a new instance",
|
||||
"`wikitool backup run` - the machine-data backup on its own",
|
||||
"`wikitool export run` - the page export on its own",
|
||||
),
|
||||
))
|
||||
def publish_command(
|
||||
@@ -1760,6 +1821,7 @@ def publish_command(
|
||||
# Read before anything is committed, `--no-push` included: a broken section
|
||||
# stops here rather than turning into a skipped backup after the push.
|
||||
backup_cfg = read_backup_config()
|
||||
export_targets = read_export_targets()
|
||||
|
||||
paths = list(path or [])
|
||||
# What is staged and committed: `paths` plus any uncommitted generated file outside them.
|
||||
@@ -1932,5 +1994,7 @@ def publish_command(
|
||||
success(f"Published changes to {remote}/{branch}." if push else "Committed changes (not pushed).")
|
||||
if push and backup_cfg is not None and backup_cfg.machine_data:
|
||||
back_up_machine_data(backup_cfg)
|
||||
if export_targets:
|
||||
export_after_publish(export_targets)
|
||||
if touches_stack_machinery(changed_files):
|
||||
typer.echo(STACK_MACHINERY_NOTE)
|
||||
@@ -11,9 +11,10 @@ import re
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich.markup import escape
|
||||
|
||||
from chemenu import config, filters, guideline_export, repo_capture
|
||||
from chemenu import backup, config, filters, guideline_export, page_export, repo_capture
|
||||
from chemenu.commands._util import fail, path_budget_problem_for, rel_path
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.provenance import citing_pages, source_pages_by_raw_file
|
||||
@@ -155,16 +156,37 @@ def _check_files(resolved: list[Path]) -> None:
|
||||
|
||||
|
||||
def _check_no_export(paths: list[Path]) -> None:
|
||||
"""Refuse a file a wiki exported (`export guidelines`): it is generated
|
||||
from `kb/`, so taking it into `raw/` would feed the wiki its own output
|
||||
back as a source. `raw capture` already leaves such a file out; this
|
||||
closes the way in through `incoming/`."""
|
||||
"""Refuse a file a wiki exported (`export guidelines`, `export run`): it is
|
||||
generated from `kb/`, so taking it into `raw/` would feed the wiki its own
|
||||
output back as a source. `raw capture` already leaves such a file out; this
|
||||
closes the way in through `incoming/`. The marker catches a file as written;
|
||||
the digest of a configured target's manifest catches one whose marker a
|
||||
converter dropped."""
|
||||
exports = [p for p in paths if guideline_export.file_is_export(p)]
|
||||
if exports:
|
||||
listed = "\n".join(f" - {rel_path(p)}" for p in exports)
|
||||
raise _Refused(
|
||||
"These files are a wiki's guideline export - their first line starts with "
|
||||
"<!-- wikitool:export. They are generated from kb/ and never go back into raw/:\n"
|
||||
"These files are a wiki's export (a guideline export or a page export) - they carry "
|
||||
"the wikitool export marker. They are generated from kb/ and never go back into "
|
||||
f"raw/:\n{listed}\n Remove them from incoming/ and accept again; nothing was moved."
|
||||
)
|
||||
digests, warnings = page_export.known_digests()
|
||||
for warning in warnings:
|
||||
typer.echo(f"WARN {warning}")
|
||||
if not digests:
|
||||
return
|
||||
exported = []
|
||||
for p in paths:
|
||||
try:
|
||||
if backup.file_digest(p)[1] in digests:
|
||||
exported.append(p)
|
||||
except OSError:
|
||||
continue
|
||||
if exported:
|
||||
listed = "\n".join(f" - {rel_path(p)}" for p in exported)
|
||||
raise _Refused(
|
||||
"These files are byte for byte a file of a configured export target (see "
|
||||
"`export run`). They are generated from kb/ and never go back into raw/:\n"
|
||||
f"{listed}\n Remove them from incoming/ and accept again; nothing was moved."
|
||||
)
|
||||
|
||||
|
||||
@@ -668,8 +668,10 @@ def _warn_underived(resolved: list[Path]) -> None:
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` "
|
||||
"material - is a guideline export: its first line, after an optional BOM, starts with "
|
||||
"`<!-- wikitool:export`",
|
||||
"material - is an export of this wiki: it carries the export marker (a first line "
|
||||
"starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading "
|
||||
"frontmatter block), or it is byte for byte a file in the manifest of a configured "
|
||||
"`export` target",
|
||||
reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes "
|
||||
"back into `raw/`. Remove it from `incoming/`; nothing was moved",
|
||||
),
|
||||
|
||||
@@ -121,7 +121,7 @@ def _write_bundle(target: Path, files: dict[str, bytes], manifest: repo_capture.
|
||||
"spans any number of directories; a glob without wildcards also takes everything below it "
|
||||
"as a directory. Files keep their repository paths inside the bundle.",
|
||||
"Never captured, each named in the output with its reason: a file whose first line starts "
|
||||
"with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a "
|
||||
"with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a "
|
||||
"segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.",
|
||||
"`incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, "
|
||||
"`captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that "
|
||||
|
||||
@@ -813,11 +813,160 @@ BACKUP = Shape(
|
||||
"\"no backup\".",
|
||||
)
|
||||
|
||||
EXPORT_HANDLER_MARKDOWN = "markdown"
|
||||
EXPORT_HANDLER_COMMAND = "command"
|
||||
EXPORT_HANDLERS = (EXPORT_HANDLER_MARKDOWN, EXPORT_HANDLER_COMMAND)
|
||||
EXPORT_FRONTMATTER_MODES = ("minimal", "summary", "full")
|
||||
EXPORT_DEFAULT_FRONTMATTER = "summary"
|
||||
EXPORT_LINK_MODES = ("relative", "text")
|
||||
EXPORT_DEFAULT_LINKS = "relative"
|
||||
# What each handler takes besides the keys every target has. A key outside
|
||||
# both sets is an error, so a typo or a field of the other handler is not
|
||||
# silently ignored.
|
||||
_EXPORT_COMMON_KEYS = ("handler", "path", "include", "frontmatter", "links", "after_publish")
|
||||
_EXPORT_HANDLER_KEYS = {
|
||||
EXPORT_HANDLER_MARKDOWN: (),
|
||||
EXPORT_HANDLER_COMMAND: ("command",),
|
||||
}
|
||||
|
||||
|
||||
def _export_include_problem(where: str, include: Any) -> Optional[str]:
|
||||
if not isinstance(include, list) or not include:
|
||||
return f"{where}: `include` must be a non-empty array of selections"
|
||||
for number, selection in enumerate(include, start=1):
|
||||
if not isinstance(selection, list) or not selection:
|
||||
return f"{where}: `include` selection {number} must be a non-empty array of predicates"
|
||||
if not all(isinstance(predicate, str) and predicate.strip() for predicate in selection):
|
||||
return f"{where}: `include` selection {number} holds an entry that is not a non-empty string"
|
||||
return None
|
||||
|
||||
|
||||
def _export_target_problem(name: str, target: Any) -> Optional[str]:
|
||||
where = f"targets.{name}"
|
||||
if not isinstance(target, dict):
|
||||
return f"{where} must be an object"
|
||||
handler = target.get("handler")
|
||||
if handler not in EXPORT_HANDLERS:
|
||||
return f"{where}: `handler` must be one of {', '.join(EXPORT_HANDLERS)}"
|
||||
allowed = _EXPORT_COMMON_KEYS + _EXPORT_HANDLER_KEYS[handler]
|
||||
stray = sorted(set(target) - set(allowed))
|
||||
if stray:
|
||||
return (f"{where}: {', '.join(f'`{key}`' for key in stray)} not taken by the handler "
|
||||
f"{handler} (it takes {', '.join(f'`{key}`' for key in allowed)})")
|
||||
for key in ("path", "include"):
|
||||
if key not in target:
|
||||
return f"{where}: `{key}` is required"
|
||||
path = target["path"]
|
||||
if not isinstance(path, str) or not path.strip():
|
||||
return f"{where}: `path` must be a non-empty string"
|
||||
problem = absolute_path_problem(path)
|
||||
if problem:
|
||||
return f"{where}: `path` {problem}"
|
||||
problem = _export_include_problem(where, target["include"])
|
||||
if problem:
|
||||
return problem
|
||||
for key, choices in (("frontmatter", EXPORT_FRONTMATTER_MODES), ("links", EXPORT_LINK_MODES)):
|
||||
if key in target and target[key] not in choices:
|
||||
return f"{where}: `{key}` must be one of {', '.join(choices)}"
|
||||
if "after_publish" in target and not isinstance(target["after_publish"], bool):
|
||||
return f"{where}: `after_publish` must be true or false"
|
||||
if handler == EXPORT_HANDLER_COMMAND:
|
||||
ref = target.get("command")
|
||||
if not isinstance(ref, str) or not _FETCHER_NAME.fullmatch(ref):
|
||||
return (f"{where}: `command` is required and names an entry of `commands` - lower-case "
|
||||
"letters, digits and hyphens")
|
||||
return None
|
||||
|
||||
|
||||
def export_targets_problem(value: dict) -> Optional[str]:
|
||||
if not value:
|
||||
return "must hold at least one target"
|
||||
for name, target in value.items():
|
||||
if not _FETCHER_NAME.fullmatch(name):
|
||||
return (f"has the target name {name!r} - a target's name is lower-case letters, digits "
|
||||
"and hyphens")
|
||||
problem = _export_target_problem(name, target)
|
||||
if problem:
|
||||
return problem
|
||||
return None
|
||||
|
||||
|
||||
def export_section_problem(values: dict) -> Optional[str]:
|
||||
commands = values.get("commands") or {}
|
||||
for name, target in values["targets"].items():
|
||||
if target["handler"] == EXPORT_HANDLER_COMMAND and target["command"] not in commands:
|
||||
return (f"targets.{name}.command names {target['command']!r}, which `commands` does not "
|
||||
"define")
|
||||
return None
|
||||
|
||||
|
||||
EXPORT = Shape(
|
||||
key="export",
|
||||
title="Exports to folders",
|
||||
purpose="Declares the folders `export run` writes a derived, read-only reading copy of `kb/` "
|
||||
"into: per target the pages it takes (an allowlist of `search` predicates), how they are "
|
||||
"written, and whether `publish` refreshes it. An export is regenerated from `kb/` and never "
|
||||
"edited; what a person changes in the folder is overwritten or refused, never merged back.",
|
||||
fields=(
|
||||
Field(
|
||||
"targets", "object", "Target name (lower-case letters, digits and hyphens) to target; "
|
||||
"the name is the argument of `export run`. Every target has `handler` (`markdown`: the "
|
||||
"pages as Markdown files; `command`: the same Markdown run through a converter of this "
|
||||
"checkout), `path` (an absolute path outside the instance; it must not lie in or "
|
||||
"contain another target, an `intake` folder or `backup.target`) and `include` (an "
|
||||
"array of selections, each an array of `search --field` predicates: all predicates of "
|
||||
"a selection must hold, and a page that fits any selection is exported; there is no "
|
||||
"\"everything\"). Optional: `frontmatter` (`minimal`: only the export marker; "
|
||||
f"`summary`, the default: also `title`, `type`, `summary` and `modified` where the page "
|
||||
"has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a "
|
||||
"wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a "
|
||||
"link to a page that is not exported is always plain text) and `after_publish` "
|
||||
"(default false: refresh this target after every `publish` that committed). The "
|
||||
"handler `command` takes `command` (required: the name of an entry of `commands`). A "
|
||||
"field the handler does not take is an error.",
|
||||
required=True, json_type="object", coerce=object_value, check=export_targets_problem,
|
||||
tier=CHECKOUT, gate=True,
|
||||
),
|
||||
Field(
|
||||
"commands", "object", "Command name to command: an array of the program and its "
|
||||
"arguments, run by the `command` handler without a shell, with stdin closed and a "
|
||||
"timeout of 300 s. The last two arguments appended are the directory holding the "
|
||||
"Markdown and an empty output directory. Exit 0 makes the regular files in the output "
|
||||
"directory the target's content; anything else leaves the target unchanged and prints "
|
||||
"the end of the program's stderr.",
|
||||
default={}, json_type="object", coerce=object_value, check=intake_commands_problem,
|
||||
tier=CHECKOUT, names_command=True,
|
||||
),
|
||||
),
|
||||
examples=({
|
||||
"targets": {
|
||||
"work-notes": {
|
||||
"handler": "markdown",
|
||||
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
|
||||
"include": [["kind=system"], ["tags=shared", "!draft"]],
|
||||
"frontmatter": "summary", "links": "relative", "after_publish": True,
|
||||
},
|
||||
"as-word": {
|
||||
"handler": "command", "command": "pandoc-docx",
|
||||
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
|
||||
"include": [["kind=system"]],
|
||||
},
|
||||
},
|
||||
"commands": {"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]},
|
||||
},),
|
||||
absent="No export: `export run` stops with exit 1 naming the section, `publish` exports nothing "
|
||||
"and `doctor` reports `OK`.",
|
||||
malformed="An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops "
|
||||
"with exit 1 before it commits anything, but only when some target sets `after_publish`. A "
|
||||
"broken section is never read as \"no export\".",
|
||||
section_check=export_section_problem,
|
||||
)
|
||||
|
||||
# The sections of the tiered settings files, `wikitool.json` and
|
||||
# `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
|
||||
# stand in a section of its own.
|
||||
SETTINGS_SECTIONS: tuple[Shape, ...] = (
|
||||
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP,
|
||||
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP, EXPORT,
|
||||
)
|
||||
|
||||
# The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
|
||||
|
||||
@@ -6,7 +6,7 @@ again when it comes back.
|
||||
side - writing the file into a captured repository's branch - lives beside
|
||||
`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under
|
||||
`chemenu.commands`, and nothing here imports `repo_capture`, which imports
|
||||
`EXPORT_MARKER` from here: that is the direction the dependency runs.
|
||||
`is_export` from here: that is the direction the dependency runs.
|
||||
|
||||
Three properties carry the design, and each has a test:
|
||||
|
||||
@@ -38,11 +38,19 @@ from chemenu.search import filters
|
||||
from chemenu.search.service import load_pages_by_path, unreadable_pages
|
||||
from chemenu.search.types import Predicate
|
||||
|
||||
# The first bytes of every file this stack generates for another repository.
|
||||
# Defined once, here: `repo_capture` excludes a file carrying it from a
|
||||
# capture, `raw accept` refuses one, and the export writes it.
|
||||
EXPORT_MARKER = b"<!-- wikitool:export"
|
||||
# What marks a file this stack generated for somewhere else - the one place
|
||||
# the definition of an export lives. `repo_capture` excludes a file that
|
||||
# `is_export` recognises from a capture, `raw accept` refuses one, and the
|
||||
# exports write it: `export guidelines` as the first-line comment, `export run`
|
||||
# as a frontmatter key. Either form is an export; nothing else is.
|
||||
COMMENT_MARKER = b"<!-- wikitool:export"
|
||||
FRONTMATTER_KEY = b"wikitool_export:"
|
||||
FRONTMATTER_FENCE = b"---"
|
||||
# How much of a file `is_export` looks at to find the frontmatter's closing
|
||||
# fence; a frontmatter that does not close within it is not an export.
|
||||
SCAN_BYTES = 64 * 1024
|
||||
GUIDELINES_KIND = "guidelines"
|
||||
PAGES_KIND = "pages"
|
||||
GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}"
|
||||
TARGET_PATH = "GUIDELINES.md"
|
||||
LOCAL_INSTANCE = "local"
|
||||
@@ -57,16 +65,42 @@ _SCP_USERINFO = re.compile(r"^[^/@:]+@(?=[^/:]+:)")
|
||||
# --- recognising an export ------------------------------------------------------
|
||||
|
||||
|
||||
def _without_bom(data: bytes) -> bytes:
|
||||
return data[len(_BOM):] if data.startswith(_BOM) else data
|
||||
|
||||
|
||||
def _first_line(data: bytes) -> bytes:
|
||||
data = data[len(_BOM):] if data.startswith(_BOM) else data
|
||||
return data.split(b"\n", 1)[0]
|
||||
return _without_bom(data).split(b"\n", 1)[0]
|
||||
|
||||
|
||||
def is_export(data: bytes) -> bool:
|
||||
"""Whether `data` is a file this stack exported: its first line, after an
|
||||
optional BOM, starts with `EXPORT_MARKER`. Reads no more than the first
|
||||
line, so a caller may pass only a file's first bytes."""
|
||||
return _first_line(data).startswith(EXPORT_MARKER)
|
||||
"""Whether `data` is a file this stack exported, in either form:
|
||||
|
||||
1. its first line, after an optional BOM, starts with `COMMENT_MARKER`
|
||||
(`export guidelines`);
|
||||
2. its first line is `---` and a line starting with `FRONTMATTER_KEY`
|
||||
stands before the closing `---` line (`export run`).
|
||||
|
||||
Form 2 is a line scan, not a YAML parse, so a frontmatter that is no longer
|
||||
valid YAML is still recognised - the marker, not the YAML, is what says
|
||||
where a file came from. Looks at no more than `SCAN_BYTES`, so a caller may
|
||||
pass only a file's first bytes: a frontmatter with no closing line in that
|
||||
range is not an export, and neither is a key that only follows it."""
|
||||
data = _without_bom(data)
|
||||
head = data[:SCAN_BYTES]
|
||||
lines = head.split(b"\n")
|
||||
if len(head) == SCAN_BYTES and len(data) > SCAN_BYTES:
|
||||
lines.pop() # a line the cut may have shortened
|
||||
if lines[0].startswith(COMMENT_MARKER):
|
||||
return True
|
||||
if lines[0].rstrip() != FRONTMATTER_FENCE:
|
||||
return False
|
||||
marked = False
|
||||
for line in lines[1:]:
|
||||
if line.rstrip() == FRONTMATTER_FENCE:
|
||||
return marked
|
||||
marked = marked or line.startswith(FRONTMATTER_KEY)
|
||||
return False
|
||||
|
||||
|
||||
def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
|
||||
@@ -80,9 +114,9 @@ def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
|
||||
|
||||
|
||||
def file_is_export(path: Path) -> bool:
|
||||
"""`is_export` for a file on disk, reading only its first line's worth."""
|
||||
"""`is_export` for a file on disk, reading no more than it looks at."""
|
||||
with open(path, "rb") as handle:
|
||||
return is_export(handle.read(len(_BOM) + len(EXPORT_MARKER)))
|
||||
return is_export(handle.read(len(_BOM) + SCAN_BYTES + 1))
|
||||
|
||||
|
||||
# --- the instance it comes from -----------------------------------------------
|
||||
|
||||
@@ -0,0 +1,631 @@
|
||||
"""The page export, with no CLI attached: a derived, read-only reading copy of
|
||||
`kb/` per configured target folder.
|
||||
|
||||
`wikitool export run` is the terminal adapter over this module, `publish`
|
||||
calls `run` for the targets that ask for it, and `raw accept` asks
|
||||
`known_digests` whether a file is one this module wrote. Nothing here imports
|
||||
`typer` or anything under `chemenu.commands`.
|
||||
|
||||
An export is never a working copy: it is regenerated from `kb/` and what a
|
||||
person changes in the folder is overwritten or refused, never merged back. The
|
||||
properties that carry that, each with a test:
|
||||
|
||||
- **Deterministic.** The output depends on the selected pages and the commit
|
||||
that last touched one of them - no timestamp, no `HEAD` - so a run on an
|
||||
unchanged corpus writes nothing, not even a file's modification time.
|
||||
- **Only its own files are touched.** What an earlier run wrote is recorded in
|
||||
a manifest in the target (`.wikitool-export.json`, with a digest per file).
|
||||
A run removes only manifest files that are no longer exported, leaves every
|
||||
other file byte for byte as it found it, and stops on a manifest file a
|
||||
person has changed rather than overwrite the change. A non-empty folder
|
||||
without a manifest is refused: it is somebody's folder, not an export.
|
||||
- **Nothing is deleted on a doubtful selection.** Zero selected pages is an
|
||||
error before anything is touched.
|
||||
- **Atomic per file.** Each file is written to a temporary file in the target
|
||||
and moved over its place; the manifest comes last.
|
||||
- **One definition of an export.** The marker every file carries, and the test
|
||||
for it, live in `chemenu.guideline_export`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import posixpath
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, Sequence
|
||||
from urllib.parse import quote
|
||||
|
||||
from chemenu import backup, config, config_shapes, guideline_export, settings
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.frontmatter_io import dump_frontmatter
|
||||
from chemenu.kb_scan import LINK_RE, normalize_link_target
|
||||
from chemenu.markdown_code import strip_code_spans
|
||||
from chemenu.page import Page
|
||||
from chemenu.search import filters
|
||||
from chemenu.search.service import load_pages_by_path, unreadable_pages
|
||||
from chemenu.search.types import Predicate
|
||||
|
||||
MANIFEST_NAME = ".wikitool-export.json"
|
||||
MANIFEST_SCHEMA = 1
|
||||
COMMAND_TIMEOUT_SECONDS = 300
|
||||
_STDERR_TAIL_CHARS = 2000
|
||||
SUMMARY_KEYS = ("type", "summary", "modified")
|
||||
|
||||
|
||||
class ExportError(ValidationError):
|
||||
"""An export step refused or failed; exit 1 with the message."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Target:
|
||||
name: str
|
||||
handler: str
|
||||
path: Path
|
||||
include: tuple[tuple[Predicate, ...], ...]
|
||||
frontmatter: str
|
||||
links: str
|
||||
after_publish: bool
|
||||
command_name: str = ""
|
||||
command: tuple[str, ...] = ()
|
||||
|
||||
|
||||
def _others_paths(root: Path) -> list[tuple[str, Path]]:
|
||||
"""The folders other features of this checkout write into or read from -
|
||||
the `intake` folders and the backup target - for the conflict check. A
|
||||
section that is itself broken is that feature's own problem and is skipped
|
||||
here: it is reported where it is read."""
|
||||
from chemenu import upload
|
||||
|
||||
found: list[tuple[str, Path]] = []
|
||||
try:
|
||||
for name, channel in upload.read_channels(root).items():
|
||||
if channel.path is not None:
|
||||
found.append((f"the intake channel {name!r} ({channel.path})", channel.path))
|
||||
except ValidationError:
|
||||
pass
|
||||
try:
|
||||
backup_cfg = backup.read_config(root)
|
||||
except ValidationError:
|
||||
backup_cfg = None
|
||||
if backup_cfg is not None:
|
||||
found.append((f"backup.target ({backup_cfg.target})", backup_cfg.target))
|
||||
return found
|
||||
|
||||
|
||||
def _parse_include(name: str, include: list[list[str]], problems: list[str]) -> tuple[tuple[Predicate, ...], ...]:
|
||||
selections = []
|
||||
for number, selection in enumerate(include, start=1):
|
||||
parsed = []
|
||||
for raw in selection:
|
||||
try:
|
||||
parsed.append(filters.parse_predicate(raw))
|
||||
except ValidationError as exc:
|
||||
problems.append(f"export.targets.{name}.include selection {number}: {exc}")
|
||||
selections.append(tuple(parsed))
|
||||
return tuple(selections)
|
||||
|
||||
|
||||
def read_config(root: "Path | str | None" = None) -> Optional[tuple[Target, ...]]:
|
||||
"""The targets of the `export` section in the order the file lists them, or
|
||||
None when the settings carry none. Raises `ValidationError` for a section
|
||||
that is broken - including what the record cannot judge without the
|
||||
instance and the rest of the settings: a predicate that does not parse, a
|
||||
path on this machine that is not absolute, inside the instance, or in the
|
||||
way of another target, an `intake` folder or the backup target."""
|
||||
root = Path(config.ROOT if root is None else root)
|
||||
key = config_shapes.EXPORT.key
|
||||
section = settings.load(root, only=(key,)).section(key)
|
||||
if not section.configured:
|
||||
return None
|
||||
problems: list[str] = []
|
||||
targets: list[Target] = []
|
||||
commands = section["commands"]
|
||||
real_root = backup.real_path(root)
|
||||
for name, raw in section["targets"].items():
|
||||
path = Path(raw["path"])
|
||||
if not os.path.isabs(raw["path"]):
|
||||
problems.append(f"export.targets.{name}.path {raw['path']!r} is not an absolute path on this machine")
|
||||
continue
|
||||
real = backup.real_path(path)
|
||||
if backup.is_under(real, real_root) or backup.is_under(real_root, real):
|
||||
problems.append(f"export.targets.{name}.path {path} and the instance lie in one another - "
|
||||
"an export belongs outside it")
|
||||
continue
|
||||
include = _parse_include(name, raw["include"], problems)
|
||||
command_name = raw.get("command", "")
|
||||
targets.append(Target(
|
||||
name=name, handler=raw["handler"], path=path, include=include,
|
||||
frontmatter=raw.get("frontmatter", config_shapes.EXPORT_DEFAULT_FRONTMATTER),
|
||||
links=raw.get("links", config_shapes.EXPORT_DEFAULT_LINKS),
|
||||
after_publish=raw.get("after_publish", False),
|
||||
command_name=command_name, command=tuple(commands.get(command_name, ())),
|
||||
))
|
||||
if not problems:
|
||||
placed = [(f"export.targets.{t.name}", backup.real_path(t.path)) for t in targets]
|
||||
others = [(label, backup.real_path(path)) for label, path in _others_paths(root)]
|
||||
for index, (label, real) in enumerate(placed):
|
||||
for other_label, other in placed[index + 1:] + others:
|
||||
if backup.is_under(real, other) or backup.is_under(other, real):
|
||||
problems.append(f"{label} and {other_label} lie in one another - "
|
||||
"every export folder is its own")
|
||||
if problems:
|
||||
raise ExportError("; ".join(problems))
|
||||
return tuple(targets)
|
||||
|
||||
|
||||
def require_config(root: "Path | str | None" = None) -> tuple[Target, ...]:
|
||||
found = read_config(root)
|
||||
if found is None:
|
||||
raise ExportError(
|
||||
f"no export section in {config.LOCAL_SETTINGS_FILENAME} - the user adds it by hand "
|
||||
"(docs/how-to/export-to-a-synced-folder.md)"
|
||||
)
|
||||
return found
|
||||
|
||||
|
||||
def _asks_for_after_publish(root: Path) -> bool:
|
||||
"""Whether the raw `export` section has a target that sets `after_publish`,
|
||||
read without judging the rest of it - so a section too broken to read as a
|
||||
whole can still tell `publish` that it matters."""
|
||||
try:
|
||||
data = settings.read_file(settings.local_path(root)) or {}
|
||||
except ValidationError:
|
||||
return False
|
||||
section = data.get(config_shapes.EXPORT.key)
|
||||
targets = section.get("targets") if isinstance(section, dict) else None
|
||||
if not isinstance(targets, dict):
|
||||
return False
|
||||
return any(isinstance(t, dict) and t.get("after_publish") for t in targets.values())
|
||||
|
||||
|
||||
def after_publish_targets(root: "Path | str | None" = None) -> tuple[Target, ...]:
|
||||
"""The targets `publish` refreshes after a commit. A broken `export` section
|
||||
raises only when some target asks for that - a checkout that never set
|
||||
`after_publish` is not held up by a section it does not use there."""
|
||||
root = Path(config.ROOT if root is None else root)
|
||||
try:
|
||||
targets = read_config(root)
|
||||
except ValidationError:
|
||||
if _asks_for_after_publish(root):
|
||||
raise
|
||||
return ()
|
||||
return tuple(t for t in targets or () if t.after_publish)
|
||||
|
||||
|
||||
def choose(targets: Sequence[Target], names: Sequence[str]) -> list[Target]:
|
||||
"""The targets `names` select, in the order named; all of them for no name.
|
||||
An unknown name is an error before anything runs."""
|
||||
if not names:
|
||||
return list(targets)
|
||||
by_name = {t.name: t for t in targets}
|
||||
unknown = [n for n in names if n not in by_name]
|
||||
if unknown:
|
||||
raise ExportError(
|
||||
f"unknown export target(s): {', '.join(unknown)}. Configured: {', '.join(by_name)}"
|
||||
)
|
||||
chosen: list[Target] = []
|
||||
for name in names:
|
||||
if by_name[name] not in chosen:
|
||||
chosen.append(by_name[name])
|
||||
return chosen
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Selection and rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def select(pages: dict[str, Page], target: Target, kb_dir: Path) -> list[Page]:
|
||||
"""The pages of `target`: the union over its selections, each the pages
|
||||
that satisfy every predicate of it, sorted by `(title.casefold(), title)`."""
|
||||
predicates = tuple(p for selection in target.include for p in selection)
|
||||
filters.validate_fields(predicates, pages)
|
||||
chosen: dict[str, Page] = {}
|
||||
for selection in target.include:
|
||||
chosen.update(filters.apply_predicates(pages, selection, kb_dir))
|
||||
return sorted(chosen.values(), key=lambda p: (p.title.casefold(), p.title))
|
||||
|
||||
|
||||
def file_path(page: Page, kb_dir: Path) -> str:
|
||||
return page.path.relative_to(kb_dir).as_posix()
|
||||
|
||||
|
||||
def _rewrite_links(text: str, here: str, destinations: dict[str, Optional[str]], links: str) -> str:
|
||||
"""Wikilinks outside code to Markdown: a link to an exported page relative
|
||||
to `here` (`links: relative`), everything else to its display text. Matches
|
||||
are found on the masked copy and applied to the original, which share
|
||||
offsets, so code reaches the output byte for byte."""
|
||||
masked = strip_code_spans(text)
|
||||
out: list[str] = []
|
||||
pos = 0
|
||||
for m in LINK_RE.finditer(masked):
|
||||
title = normalize_link_target(text[m.start(1):m.end(1)])
|
||||
rest = text[m.start(2):m.end(2)]
|
||||
display = rest.split("|", 1)[1] if "|" in rest else title
|
||||
destination = destinations.get(title)
|
||||
if links == "relative" and destination:
|
||||
relative = posixpath.relpath(destination, posixpath.dirname(here) or ".")
|
||||
replacement = f"[{display}]({quote(relative, safe='/')})"
|
||||
else:
|
||||
replacement = display
|
||||
out.append(text[pos:m.start()])
|
||||
out.append(replacement)
|
||||
pos = m.end()
|
||||
out.append(text[pos:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def marker_value(instance: str, commit: str) -> str:
|
||||
return f"kind={guideline_export.PAGES_KIND} instance={instance} commit={commit}"
|
||||
|
||||
|
||||
def render_page(page: Page, here: str, destinations: dict[str, Optional[str]], target: Target,
|
||||
instance: str, commit: str) -> bytes:
|
||||
"""One page as the file a target holds: the frontmatter fence with the
|
||||
export marker first, then the frontmatter `target.frontmatter` asks for,
|
||||
then the body - its generated links block kept, wikilinks rewritten, code
|
||||
untouched. LF endings, exactly one trailing newline."""
|
||||
own: dict[str, Any] = {}
|
||||
if target.frontmatter == "summary":
|
||||
own["title"] = page.title
|
||||
own.update({k: page.frontmatter[k] for k in SUMMARY_KEYS if k in page.frontmatter})
|
||||
elif target.frontmatter == "full":
|
||||
own.update({k: v for k, v in page.frontmatter.items() if k != "wikitool_export"})
|
||||
marker = json.dumps(marker_value(instance, commit), ensure_ascii=False)
|
||||
head = [f"wikitool_export: {marker}"]
|
||||
if own:
|
||||
head.append(dump_frontmatter(own))
|
||||
body = page.body.replace("\r\n", "\n").strip("\n")
|
||||
body = _rewrite_links(body, here, destinations, target.links)
|
||||
text = "---\n" + "\n".join(head) + "\n---\n" + ("\n" + body + "\n" if body else "")
|
||||
return text.encode("utf-8")
|
||||
|
||||
|
||||
def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, commit: str) -> dict[str, bytes]:
|
||||
"""The files of a target for `pages`, by path below the target. A page
|
||||
that cannot be written on every platform, or two that would share a file
|
||||
name in a case-insensitive folder, is an `ExportError`."""
|
||||
paths = {id(page): file_path(page, kb_dir) for page in pages}
|
||||
seen: dict[str, str] = {}
|
||||
for page in pages:
|
||||
rel = paths[id(page)]
|
||||
if not backup.safe_relative(rel):
|
||||
raise ExportError(
|
||||
f"{rel} cannot be written to a folder on every platform (a name with `:`, a backslash "
|
||||
"or an empty part) - rename the page or leave it out of the selection"
|
||||
)
|
||||
if seen.setdefault(rel.casefold(), rel) != rel:
|
||||
raise ExportError(f"{rel} and {seen[rel.casefold()]} differ only in case - "
|
||||
"they would be one file in a synced folder")
|
||||
by_title: dict[str, Optional[str]] = {}
|
||||
for page in pages:
|
||||
by_title[page.title] = None if page.title in by_title else paths[id(page)]
|
||||
files: dict[str, bytes] = {}
|
||||
for page in pages:
|
||||
rel = paths[id(page)]
|
||||
files[rel] = render_page(page, rel, by_title, target, instance, commit)
|
||||
return files
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The target folder
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sha256(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def read_manifest(folder: Path) -> Optional[dict]:
|
||||
"""The manifest of the last run into `folder`, or None when there is none.
|
||||
Raises `ExportError` for one that cannot be read as one."""
|
||||
path = folder / MANIFEST_NAME
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
raise ExportError(f"{path} cannot be read ({exc}) - nothing was written") from None
|
||||
files = data.get("files") if isinstance(data, dict) else None
|
||||
dirs = data.get("dirs", []) if isinstance(data, dict) else None
|
||||
if (not isinstance(files, list) or not isinstance(dirs, list)
|
||||
or not all(isinstance(d, str) and backup.safe_relative(d) for d in dirs)
|
||||
or not all(
|
||||
isinstance(entry, dict) and isinstance(entry.get("path"), str)
|
||||
and backup.safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
||||
and isinstance(entry.get("sha256"), str)
|
||||
for entry in files)):
|
||||
raise ExportError(f"{path} is not an export manifest - nothing was written")
|
||||
return data
|
||||
|
||||
|
||||
@dataclass
|
||||
class TargetResult:
|
||||
name: str
|
||||
path: Path
|
||||
commit: str = ""
|
||||
new: list = field(default_factory=list)
|
||||
changed: list = field(default_factory=list)
|
||||
removed: list = field(default_factory=list)
|
||||
unchanged: int = 0
|
||||
skipped: list = field(default_factory=list)
|
||||
error: Optional[str] = None
|
||||
titles: dict = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.error is None
|
||||
|
||||
@property
|
||||
def touched(self) -> bool:
|
||||
return bool(self.new or self.changed or self.removed)
|
||||
|
||||
|
||||
def _check_parents(folder: Path, rel: str) -> None:
|
||||
current = folder
|
||||
for part in rel.split("/")[:-1]:
|
||||
current = current / part
|
||||
if backup.is_link(current):
|
||||
raise ExportError(f"{current} is a link - an export writes nothing through one")
|
||||
if current.exists() and not current.is_dir():
|
||||
raise ExportError(f"{current} is a file where the export needs a folder - move it away; "
|
||||
"nothing was written")
|
||||
|
||||
|
||||
def _on_disk(folder: Path, rel: str) -> Optional[str]:
|
||||
"""The sha256 of `rel` in `folder`, None when it is missing. A directory or
|
||||
link there is an `ExportError`."""
|
||||
path = folder.joinpath(*rel.split("/"))
|
||||
if backup.is_link(path):
|
||||
raise ExportError(f"{path} is a link - an export touches no link; nothing was written")
|
||||
if not path.exists():
|
||||
return None
|
||||
if not path.is_file():
|
||||
raise ExportError(f"{path} is not a file - nothing was written")
|
||||
return backup.file_digest(path)[1]
|
||||
|
||||
|
||||
def mirror(folder: Path, files: dict[str, bytes], result: TargetResult, *, name: str, instance: str,
|
||||
commit: str) -> None:
|
||||
"""Bring `folder` to exactly `files` plus its manifest, filling `result`.
|
||||
Everything that can refuse refuses before the first write."""
|
||||
if not files:
|
||||
raise ExportError("nothing to export - an empty result would delete the earlier export, "
|
||||
"so nothing was changed")
|
||||
if folder.exists() and not folder.is_dir():
|
||||
raise ExportError(f"{folder} exists and is not a folder")
|
||||
manifest = read_manifest(folder) if folder.is_dir() else None
|
||||
if folder.is_dir() and manifest is None and any(folder.iterdir()):
|
||||
raise ExportError(
|
||||
f"{folder} is not empty and holds no {MANIFEST_NAME} - it is not an export of this stack, "
|
||||
"and an export writes only into an empty or its own folder. Nothing was written."
|
||||
)
|
||||
old = {entry["path"]: entry["sha256"] for entry in (manifest or {}).get("files", [])}
|
||||
new_digests = {rel: _sha256(data) for rel, data in files.items()}
|
||||
|
||||
changed_by_hand = []
|
||||
for rel, digest in old.items():
|
||||
present = _on_disk(folder, rel)
|
||||
if present is not None and present not in (digest, new_digests.get(rel)):
|
||||
changed_by_hand.append(rel)
|
||||
if changed_by_hand:
|
||||
raise ExportError(
|
||||
"changed in the target since the last export: " + ", ".join(changed_by_hand)
|
||||
+ ". An export is never edited; move or delete the file and run again. Nothing was written."
|
||||
)
|
||||
|
||||
for rel, data in files.items():
|
||||
_check_parents(folder, rel)
|
||||
present = _on_disk(folder, rel)
|
||||
if rel in old:
|
||||
if present == new_digests[rel]:
|
||||
result.unchanged += 1
|
||||
else:
|
||||
result.changed.append(rel)
|
||||
elif present is None:
|
||||
result.new.append(rel)
|
||||
elif present == new_digests[rel]:
|
||||
result.new.append(rel)
|
||||
else:
|
||||
raise ExportError(f"{folder.joinpath(*rel.split('/'))} exists and is not one of the export's "
|
||||
"files - move it away. Nothing was written.")
|
||||
result.removed = sorted(rel for rel in old if rel not in files and _on_disk(folder, rel) is not None)
|
||||
|
||||
dirs = set((manifest or {}).get("dirs", []))
|
||||
if manifest is not None and old == new_digests and not (result.new or result.changed or result.removed):
|
||||
return
|
||||
if not folder.exists():
|
||||
folder.mkdir(parents=True)
|
||||
for rel in result.new + result.changed:
|
||||
current, parts = folder, rel.split("/")[:-1]
|
||||
for index, part in enumerate(parts):
|
||||
current = current / part
|
||||
if not current.exists():
|
||||
current.mkdir()
|
||||
dirs.add("/".join(parts[:index + 1]))
|
||||
backup.write_atomic(folder.joinpath(*rel.split("/")), [files[rel]])
|
||||
for rel in result.removed:
|
||||
victim = folder.joinpath(*rel.split("/"))
|
||||
victim.unlink()
|
||||
parent = posixpath.dirname(rel)
|
||||
while parent and parent in dirs:
|
||||
try:
|
||||
folder.joinpath(*parent.split("/")).rmdir()
|
||||
except OSError:
|
||||
break
|
||||
dirs.discard(parent)
|
||||
parent = posixpath.dirname(parent)
|
||||
listing = [{"path": rel, "size": len(files[rel]), "sha256": new_digests[rel]} for rel in sorted(files)]
|
||||
body = {
|
||||
"schema": MANIFEST_SCHEMA, "kind": guideline_export.PAGES_KIND, "target": name,
|
||||
"instance": instance, "commit": commit, "dirs": sorted(dirs), "files": listing,
|
||||
}
|
||||
data = (json.dumps(body, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
|
||||
backup.write_atomic(folder / MANIFEST_NAME, [data])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The command handler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tail(stderr: bytes) -> str:
|
||||
text = stderr.decode("utf-8", errors="replace").strip()
|
||||
if len(text) > _STDERR_TAIL_CHARS:
|
||||
text = "..." + text[-_STDERR_TAIL_CHARS:]
|
||||
return text or "(nothing on stderr)"
|
||||
|
||||
|
||||
def convert(target: Target, markdown: dict[str, bytes], result: TargetResult) -> dict[str, bytes]:
|
||||
"""Run the target's converter over `markdown` and return what it left in
|
||||
its output directory. The staging directory is always removed; any end but
|
||||
exit 0 is an `ExportError` with the end of the program's stderr."""
|
||||
where = f"the command {target.command_name!r}"
|
||||
with tempfile.TemporaryDirectory(prefix="wikitool-export-") as tmp:
|
||||
source, output = Path(tmp) / "markdown", Path(tmp) / "output"
|
||||
output.mkdir()
|
||||
for rel, data in markdown.items():
|
||||
dest = source.joinpath(*rel.split("/"))
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_bytes(data)
|
||||
try:
|
||||
ran = subprocess.run(
|
||||
[*target.command, str(source), str(output)], capture_output=True,
|
||||
stdin=subprocess.DEVNULL, timeout=COMMAND_TIMEOUT_SECONDS, check=False, cwd=tmp,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
raise ExportError(f"{where} could not be started: {target.command[0]} does not exist") from None
|
||||
except subprocess.TimeoutExpired:
|
||||
raise ExportError(f"{where} did not finish within {COMMAND_TIMEOUT_SECONDS} s") from None
|
||||
except OSError as exc:
|
||||
raise ExportError(f"{where} could not be started ({exc})") from None
|
||||
if ran.returncode != 0:
|
||||
raise ExportError(f"{where} ended with exit {ran.returncode}. Its stderr:\n{_tail(ran.stderr)}")
|
||||
produced: dict[str, bytes] = {}
|
||||
for directory, subdirs, names in os.walk(output, followlinks=False):
|
||||
base = Path(directory)
|
||||
subdirs[:] = [d for d in subdirs if not backup.is_link(base / d)]
|
||||
for entry in sorted(names):
|
||||
path = base / entry
|
||||
rel = path.relative_to(output).as_posix()
|
||||
if backup.is_link(path) or not path.is_file():
|
||||
result.skipped.append(rel)
|
||||
continue
|
||||
if not backup.safe_relative(rel):
|
||||
raise ExportError(f"{where} produced {rel!r}, which is not a name that can be "
|
||||
"written on every platform")
|
||||
produced[rel] = path.read_bytes()
|
||||
casefolded: dict[str, str] = {}
|
||||
for rel in produced:
|
||||
if casefolded.setdefault(rel.casefold(), rel) != rel:
|
||||
raise ExportError(f"{where} produced {rel} and {casefolded[rel.casefold()]}, which differ "
|
||||
"only in case")
|
||||
return produced
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# A run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run(targets: Sequence[Target], root: "Path | str | None" = None) -> list[TargetResult]:
|
||||
"""Export each target in `targets`, independently: one that fails is
|
||||
reported in its result and the others still run. What is not per target -
|
||||
a checkout that is not clean, a page whose frontmatter does not parse - is
|
||||
an `ExportError` before any target is touched."""
|
||||
root = Path(config.ROOT if root is None else root)
|
||||
kb_dir = config.KB_DIR
|
||||
guideline_export.check_clean(root)
|
||||
pages = load_pages_by_path(kb_dir, root)
|
||||
unreadable = unreadable_pages(pages)
|
||||
if unreadable:
|
||||
listed = "\n".join(f" - {u['path']} ({u['reason']})" for u in unreadable)
|
||||
raise ExportError(
|
||||
"These pages have frontmatter that does not parse, so the export cannot tell whether "
|
||||
f"they are selected:\n{listed}\n Fix them first; nothing was exported."
|
||||
)
|
||||
instance = guideline_export.instance_id(root)
|
||||
results = []
|
||||
for target in targets:
|
||||
result = TargetResult(target.name, target.path)
|
||||
results.append(result)
|
||||
try:
|
||||
selected = select(pages, target, kb_dir)
|
||||
if not selected:
|
||||
raise ExportError("no page matches the selection - an empty export would delete the "
|
||||
"earlier one, so nothing was changed")
|
||||
result.commit = guideline_export.last_commit(selected, root)
|
||||
files = render(selected, target, kb_dir, instance, result.commit)
|
||||
result.titles = {file_path(p, kb_dir): p.title for p in selected}
|
||||
if target.handler == config_shapes.EXPORT_HANDLER_COMMAND:
|
||||
files = convert(target, files, result)
|
||||
result.titles = {}
|
||||
mirror(target.path, files, result, name=target.name, instance=instance, commit=result.commit)
|
||||
except (ValidationError, OSError) as exc:
|
||||
result.error = str(exc)
|
||||
return results
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# What `raw accept` asks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def known_digests(root: "Path | str | None" = None) -> tuple[set[str], list[str]]:
|
||||
"""The sha256 of every file the manifests of the configured targets list,
|
||||
and a warning for each reason the answer is incomplete: a section that is
|
||||
broken, a target whose manifest is missing or unreadable. The check this
|
||||
feeds is a second line behind the marker, so it skips what it cannot read
|
||||
rather than stop a `raw accept`."""
|
||||
root = Path(config.ROOT if root is None else root)
|
||||
try:
|
||||
targets = read_config(root)
|
||||
except ValidationError as exc:
|
||||
return set(), [f"the export section is broken, so files in incoming/ were not compared with "
|
||||
f"the exported ones ({exc})"]
|
||||
digests: set[str] = set()
|
||||
warnings: list[str] = []
|
||||
for target in targets or ():
|
||||
try:
|
||||
manifest = read_manifest(target.path)
|
||||
except (ValidationError, OSError) as exc:
|
||||
warnings.append(f"export target {target.name}: {exc}")
|
||||
continue
|
||||
if manifest is None:
|
||||
warnings.append(f"export target {target.name} has no manifest at {target.path}, so its files "
|
||||
"were not compared with incoming/")
|
||||
continue
|
||||
digests.update(entry["sha256"] for entry in manifest["files"])
|
||||
return digests, warnings
|
||||
|
||||
|
||||
|
||||
def report_lines(result: TargetResult) -> list[str]:
|
||||
"""What a run did to one target, for the terminal: its new files with the
|
||||
page titles, then changed, removed, the count left untouched, and the
|
||||
reason it failed."""
|
||||
head = f"{result.name} -> {result.path}"
|
||||
if result.error is not None:
|
||||
return [f"{head}: failed", *(f" {line}" for line in result.error.splitlines())]
|
||||
lines = [f"{head} (commit {result.commit[:12]})"]
|
||||
|
||||
def named(rel: str) -> str:
|
||||
title = result.titles.get(rel)
|
||||
return f"{title} ({rel})" if title and title != rel else rel
|
||||
|
||||
for label, rels in (("new", result.new), ("changed", result.changed), ("removed", result.removed)):
|
||||
if rels:
|
||||
lines.append(f" {label} ({len(rels)}): " + ", ".join(named(rel) for rel in rels))
|
||||
lines.append(f" unchanged: {result.unchanged}")
|
||||
if result.skipped:
|
||||
lines.append(f" not regular files, skipped: {', '.join(result.skipped)}")
|
||||
return lines
|
||||
@@ -71,6 +71,7 @@ CONFIG_REGIONS = (
|
||||
("reference-config-sites", "SITES"),
|
||||
("reference-config-fetch", "FETCH"),
|
||||
("reference-config-backup", "BACKUP"),
|
||||
("reference-config-export", "EXPORT"),
|
||||
)
|
||||
|
||||
_SLUG_RE = re.compile(r"[^a-z0-9]+")
|
||||
|
||||
@@ -41,7 +41,7 @@ from typing import Iterator, Optional
|
||||
|
||||
from chemenu import config, filelock, toolpaths, web_capture
|
||||
from chemenu.errors import BackendError, ValidationError
|
||||
from chemenu.guideline_export import EXPORT_MARKER
|
||||
from chemenu.guideline_export import is_export
|
||||
|
||||
MANIFEST_NAME = "_capture.json"
|
||||
SCHEMA = 1
|
||||
@@ -56,7 +56,6 @@ ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
|
||||
GIT_TIMEOUT_SECONDS = 120.0
|
||||
|
||||
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
|
||||
_BOM = b"\xef\xbb\xbf"
|
||||
|
||||
_MODE_SYMLINK = "120000"
|
||||
_MODE_SUBMODULE = "160000"
|
||||
@@ -446,9 +445,8 @@ def _structural_exclusion(entry: TreeEntry) -> Optional[str]:
|
||||
def _content_exclusion(data: bytes) -> Optional[str]:
|
||||
if data.startswith(LFS_MARKER):
|
||||
return "Git LFS pointer - the content is not in the repository"
|
||||
first = data[len(_BOM):] if data.startswith(_BOM) else data
|
||||
if first.startswith(EXPORT_MARKER):
|
||||
return "guideline export (first line starts with <!-- wikitool:export)"
|
||||
if is_export(data):
|
||||
return "guideline export (carries the wikitool export marker)"
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -79,7 +79,7 @@ def test_the_telemetry_policy_reads_the_record():
|
||||
def test_one_record_per_section():
|
||||
assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
|
||||
"telemetry", "remotes", "intake", "tasks", "superproductivity", "caldav", "sites", "fetch",
|
||||
"backup",
|
||||
"backup", "export",
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,764 @@
|
||||
"""`export run`, the `export` settings section, the `after_publish` step of
|
||||
`publish`, the `doctor` line, and the digest check of `raw accept` - against a
|
||||
real checkout and target folders in `tmp_path`.
|
||||
|
||||
The target's path carries a space on purpose: the folder a sync client keeps on
|
||||
Windows is `OneDrive - <company>`."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import typer
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from chemenu import config, config_shapes, guideline_export, page_export
|
||||
from chemenu.cli import app
|
||||
from chemenu.commands import doctor
|
||||
from chemenu.commands.git_publish import publish_command
|
||||
from chemenu.frontmatter_io import write_page
|
||||
from chemenu.type_resolver import resolver
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
ALPHA = """
|
||||
# Alpha
|
||||
|
||||
See [[Beta]], [[Beta|the beta page]] and [[Gamma]] for more, also [[Beta#Scope]].
|
||||
Cited[^s-one]. Inline `[[Beta]]` stays.
|
||||
|
||||
```
|
||||
[[Beta]] and [[Gamma]]
|
||||
```
|
||||
|
||||
<!-- wikitool:footnotes -->
|
||||
## Footnotes
|
||||
|
||||
[^s-one]: [[Source - One]]
|
||||
<!-- /wikitool:footnotes -->
|
||||
"""
|
||||
|
||||
BETA = """
|
||||
# Beta
|
||||
|
||||
Back to [[Alpha]].
|
||||
"""
|
||||
|
||||
|
||||
def _git(root: Path, *args: str) -> str:
|
||||
result = subprocess.run(["git", *args], cwd=root, capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def _commit(root: Path, message: str = "change") -> str:
|
||||
_git(root, "add", "-A")
|
||||
_git(root, "commit", "-q", "-m", message)
|
||||
return _git(root, "rev-parse", "HEAD")
|
||||
|
||||
|
||||
def _page(root: Path, rel: str, body: str, **frontmatter) -> Path:
|
||||
path = root / "kb" / rel
|
||||
meta = {"type": "types/concept.md", "tags": ["docs"], "summary": f"About {path.stem}.",
|
||||
"modified": "2026-10-01", "related": ["Beta"], **frontmatter}
|
||||
write_page(path, meta, body)
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def instance(tmp_path, monkeypatch):
|
||||
root = tmp_path / "instance"
|
||||
root.mkdir()
|
||||
_git(root, "init", "-q", "-b", "main")
|
||||
_git(root, "config", "user.name", "Test")
|
||||
_git(root, "config", "user.email", "test@example.com")
|
||||
(root / "incoming").mkdir()
|
||||
(root / "raw").mkdir()
|
||||
(root / "incoming" / ".gitkeep").write_text("", encoding="utf-8")
|
||||
(root / ".gitignore").write_text(
|
||||
"/incoming/*\n!/incoming/.gitkeep\n/.wikitool-local.json\n/quarantine/\n/reports/\n",
|
||||
encoding="utf-8")
|
||||
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
|
||||
monkeypatch.setattr(config, "TYPES_DIR", root / "types")
|
||||
monkeypatch.setattr(resolver, "_repo_root", root)
|
||||
_page(root, "concepts/Alpha.md", ALPHA)
|
||||
_page(root, "concepts/protocols/Beta.md", BETA)
|
||||
_page(root, "concepts/Gamma.md", "\n# Gamma\n\nPrivate.\n", tags=["private"])
|
||||
_commit(root, "init")
|
||||
(tmp_path / "OneDrive - Example").mkdir()
|
||||
monkeypatch.setattr(config, "ROOT", root)
|
||||
monkeypatch.setenv("WIKITOOL_SESSION_ID", "test-session")
|
||||
return root
|
||||
|
||||
|
||||
def _out(root: Path, name: str = "Pages") -> Path:
|
||||
return root.parent / "OneDrive - Example" / name
|
||||
|
||||
|
||||
def _configure(root: Path, **targets) -> None:
|
||||
"""One target `pages` by default, with `docs` pages selected."""
|
||||
if not targets:
|
||||
targets = {"pages": {}}
|
||||
section = {}
|
||||
for name, extra in targets.items():
|
||||
body = {"handler": "markdown", "path": str(_out(root, name)), "include": [["tags=docs"]]}
|
||||
body.update(extra)
|
||||
section[name] = body
|
||||
commands = {body["command"]: ["unset"] for body in section.values() if body.get("command")}
|
||||
export = {"targets": section, **({"commands": commands} if commands else {})}
|
||||
(root / config.LOCAL_SETTINGS_FILENAME).write_text(
|
||||
json.dumps({"schema": 1, "export": export}), encoding="utf-8")
|
||||
|
||||
|
||||
def _set_commands(root: Path, commands: dict) -> None:
|
||||
path = root / config.LOCAL_SETTINGS_FILENAME
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
data["export"]["commands"] = commands
|
||||
path.write_text(json.dumps(data), encoding="utf-8")
|
||||
|
||||
|
||||
def _run(*names: str):
|
||||
return runner.invoke(app, ["export", "run", *names])
|
||||
|
||||
|
||||
def _flat(text: str) -> str:
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _snapshot(directory: Path) -> dict:
|
||||
return {p.relative_to(directory).as_posix(): p.read_bytes()
|
||||
for p in sorted(directory.rglob("*")) if p.is_file()}
|
||||
|
||||
|
||||
def _mtimes(directory: Path) -> dict:
|
||||
return {p.relative_to(directory).as_posix(): p.stat().st_mtime_ns
|
||||
for p in sorted(directory.rglob("*")) if p.is_file()}
|
||||
|
||||
|
||||
def _manifest(folder: Path) -> dict:
|
||||
return json.loads((folder / page_export.MANIFEST_NAME).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
# --- the section -------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_section_is_an_error_naming_it_and_doctor_says_ok(instance):
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "no export section" in _flat(result.output)
|
||||
assert _check("export").status == "OK"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("target, needle", [
|
||||
({"path": "relative/folder"}, "absolute"),
|
||||
({"include": []}, "include"),
|
||||
({"include": [[]]}, "include"),
|
||||
({"include": [["not a predicate"]]}, "predicate"),
|
||||
({"handler": "pdf"}, "handler"),
|
||||
({"frontmatter": "everything"}, "frontmatter"),
|
||||
({"links": "hyper"}, "links"),
|
||||
({"after_publish": "yes"}, "after_publish"),
|
||||
({"command": "convert"}, "command"),
|
||||
({"handler": "command"}, "command"),
|
||||
])
|
||||
def test_a_broken_section_stops_run_and_doctor_fails(instance, target, needle):
|
||||
_configure(instance, pages=target)
|
||||
result = _run()
|
||||
assert result.exit_code == 1, result.output
|
||||
assert needle in _flat(result.output)
|
||||
assert _check("export").status == "FAIL"
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
def test_a_target_inside_the_instance_is_refused(instance):
|
||||
_configure(instance, pages={"path": str(instance / "site")})
|
||||
assert _run().exit_code == 1
|
||||
assert _check("export").status == "FAIL"
|
||||
|
||||
|
||||
def test_targets_lying_in_one_another_are_refused(instance):
|
||||
_configure(instance, one={"path": str(_out(instance, "one"))},
|
||||
two={"path": str(_out(instance, "one") / "inner")})
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "lie in one another" in _flat(result.output)
|
||||
|
||||
|
||||
def test_a_target_in_the_backup_target_or_an_intake_folder_is_refused(instance):
|
||||
backup_target = _out(instance, "backup")
|
||||
path = instance / config.LOCAL_SETTINGS_FILENAME
|
||||
_configure(instance, pages={"path": str(backup_target / "pages")})
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
data["backup"] = {"target": str(backup_target)}
|
||||
path.write_text(json.dumps(data), encoding="utf-8")
|
||||
assert _run().exit_code == 1
|
||||
data.pop("backup")
|
||||
data["intake"] = {"channels": {"inbox": {
|
||||
"handler": "folder", "path": str(_out(instance, "pages") / "in"), "max_bytes": 1024,
|
||||
"allowed_extensions": [".md"]}}}
|
||||
data["export"]["targets"]["pages"]["path"] = str(_out(instance, "pages"))
|
||||
path.write_text(json.dumps(data), encoding="utf-8")
|
||||
assert _run().exit_code == 1
|
||||
|
||||
|
||||
def test_the_operator_only_fields_cannot_be_set_by_the_agent():
|
||||
export = config_shapes.EXPORT
|
||||
assert export.field("targets").user_only and export.field("commands").user_only
|
||||
assert export.field("targets").tier == config_shapes.CHECKOUT
|
||||
assert export.field("commands").tier == config_shapes.CHECKOUT
|
||||
|
||||
|
||||
def test_an_unknown_target_name_is_an_error_and_changes_nothing(instance):
|
||||
_configure(instance)
|
||||
result = _run("pages", "nothing")
|
||||
assert result.exit_code == 1
|
||||
assert "unknown export target" in _flat(result.output)
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
# --- what a run writes -------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_run_writes_the_selected_pages_with_marker_and_links(instance):
|
||||
_configure(instance)
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
folder = _out(instance, "pages")
|
||||
assert set(_snapshot(folder)) == {
|
||||
"concepts/Alpha.md", "concepts/protocols/Beta.md", page_export.MANIFEST_NAME}
|
||||
commit = _git(instance, "rev-parse", "HEAD")
|
||||
alpha = (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||
lines = alpha.split("\n")
|
||||
assert lines[0] == "---"
|
||||
assert lines[1] == f'wikitool_export: "kind=pages instance=local commit={commit}"'
|
||||
assert "title: Alpha" in alpha and "summary: About Alpha." in alpha
|
||||
assert "tags" not in alpha and "related" not in alpha
|
||||
assert "[Beta](protocols/Beta.md)" in alpha and "[the beta page](protocols/Beta.md)" in alpha
|
||||
assert "also [Beta](protocols/Beta.md)" in alpha
|
||||
assert "and Gamma for more" in alpha
|
||||
assert "Cited[^s-one]." in alpha and "[^s-one]: " in alpha
|
||||
assert "Inline `[[Beta]]` stays." in alpha
|
||||
assert "```\n[[Beta]] and [[Gamma]]\n```" in alpha
|
||||
assert alpha.endswith("\n") and not alpha.endswith("\n\n") and "\r" not in alpha
|
||||
beta = (folder / "concepts" / "protocols" / "Beta.md").read_text(encoding="utf-8")
|
||||
assert "[Alpha](../Alpha.md)" in beta
|
||||
assert guideline_export.file_is_export(folder / "concepts" / "Alpha.md")
|
||||
out = _flat(result.output)
|
||||
assert "new (2)" in out and "Alpha" in out and "unchanged: 0" in out
|
||||
|
||||
|
||||
def test_frontmatter_modes_and_links_text(instance):
|
||||
_configure(instance, minimal={"frontmatter": "minimal", "links": "text"},
|
||||
full={"frontmatter": "full"})
|
||||
assert _run().exit_code == 0
|
||||
minimal = (_out(instance, "minimal") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||
assert minimal.startswith('---\nwikitool_export: "kind=pages')
|
||||
assert minimal.split("---\n")[1].count("\n") == 1
|
||||
assert "[Beta](" not in minimal and "See Beta, the beta page and Gamma" in minimal
|
||||
full = (_out(instance, "full") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||
assert "tags:" in full and "type: types/concept.md" in full and "related:" in full
|
||||
|
||||
|
||||
def test_selections_are_or_and_predicates_are_and(instance):
|
||||
_configure(instance, pages={"include": [["tags=docs", "summary~Alpha"], ["tags=private"]]})
|
||||
assert _run().exit_code == 0
|
||||
assert set(_snapshot(_out(instance, "pages"))) == {
|
||||
"concepts/Alpha.md", "concepts/Gamma.md", page_export.MANIFEST_NAME}
|
||||
|
||||
|
||||
def test_an_unknown_field_is_an_error_at_run(instance):
|
||||
_configure(instance, pages={"include": [["no_such_field=x"]]})
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "unknown field" in _flat(result.output)
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
def test_a_rerun_on_an_unchanged_corpus_touches_nothing(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
before, stamps = _snapshot(folder), _mtimes(folder)
|
||||
result = _run()
|
||||
assert result.exit_code == 0
|
||||
assert _snapshot(folder) == before and _mtimes(folder) == stamps
|
||||
assert "unchanged: 2" in _flat(result.output)
|
||||
assert "new (" not in result.output and "changed (" not in result.output
|
||||
|
||||
|
||||
def test_an_instance_commit_that_touches_no_selected_page_changes_nothing(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
stamps = _mtimes(folder)
|
||||
(instance / "README.md").write_text("other\n", encoding="utf-8")
|
||||
_commit(instance, "unrelated")
|
||||
assert _run().exit_code == 0
|
||||
assert _mtimes(folder) == stamps
|
||||
|
||||
|
||||
def test_a_changed_page_a_new_page_and_a_removed_page(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_page(instance, "concepts/Delta.md", "\n# Delta\n\nNew.\n")
|
||||
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||
commit = _commit(instance, "edit")
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = _flat(result.output)
|
||||
assert "new (1): Delta" in out and "changed (1):" in out and "removed (1):" in out
|
||||
assert (folder / "concepts" / "Delta.md").is_file()
|
||||
assert not (folder / "concepts" / "protocols" / "Beta.md").exists()
|
||||
assert not (folder / "concepts" / "protocols").exists()
|
||||
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||
assert _manifest(folder)["commit"] == commit
|
||||
assert [e["path"] for e in _manifest(folder)["files"]] == ["concepts/Alpha.md", "concepts/Delta.md"]
|
||||
|
||||
|
||||
def test_the_selection_growing_needs_no_gate(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
for index in range(30):
|
||||
_page(instance, f"concepts/Many{index}.md", f"\n# Many{index}\n\nText.\n")
|
||||
_commit(instance, "many")
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "new (30)" in _flat(result.output)
|
||||
|
||||
|
||||
def test_foreign_files_stay_and_only_created_directories_are_pruned(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
(folder / "notes.txt").write_bytes(b"mine\r\n")
|
||||
(folder / "concepts" / "mine.txt").write_bytes(b"also mine")
|
||||
(folder / "concepts" / "protocols" / "mine.txt").write_bytes(b"in a created dir")
|
||||
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||
_commit(instance, "drop beta")
|
||||
assert _run().exit_code == 0
|
||||
assert (folder / "notes.txt").read_bytes() == b"mine\r\n"
|
||||
assert (folder / "concepts" / "mine.txt").read_bytes() == b"also mine"
|
||||
assert (folder / "concepts" / "protocols" / "mine.txt").read_bytes() == b"in a created dir"
|
||||
assert (folder / "concepts" / "Alpha.md").is_file()
|
||||
|
||||
|
||||
def test_a_directory_that_existed_before_the_export_is_kept(instance):
|
||||
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||
_commit(instance, "no beta yet")
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
(folder / "concepts" / "protocols").mkdir()
|
||||
_page(instance, "concepts/protocols/Beta.md", BETA)
|
||||
_commit(instance, "beta")
|
||||
assert _run().exit_code == 0
|
||||
assert "concepts/protocols" not in _manifest(folder)["dirs"]
|
||||
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||
_commit(instance, "drop beta")
|
||||
assert _run().exit_code == 0
|
||||
assert (folder / "concepts" / "protocols").is_dir()
|
||||
|
||||
|
||||
def test_a_non_empty_folder_without_a_manifest_is_refused(instance):
|
||||
folder = _out(instance, "pages")
|
||||
folder.mkdir()
|
||||
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||
_configure(instance)
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "not empty" in _flat(result.output) and "mine.md" not in _snapshot(folder).keys() - {"mine.md"}
|
||||
assert _snapshot(folder) == {"mine.md": b"mine\n"}
|
||||
|
||||
|
||||
def test_an_empty_existing_folder_is_taken(instance):
|
||||
_out(instance, "pages").mkdir()
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
|
||||
|
||||
def test_a_file_changed_by_hand_aborts_the_target_before_any_write(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
edited = folder / "concepts" / "protocols" / "Beta.md"
|
||||
edited.write_text("my own words\n", encoding="utf-8")
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_commit(instance, "edit")
|
||||
before = _snapshot(folder)
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "concepts/protocols/Beta.md" in _flat(result.output)
|
||||
assert _snapshot(folder) == before
|
||||
|
||||
|
||||
def test_a_missing_manifest_file_is_rewritten(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
expected = _snapshot(folder)
|
||||
(folder / "concepts" / "Alpha.md").unlink()
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
assert _snapshot(folder) == expected
|
||||
|
||||
|
||||
def test_a_crash_between_the_files_and_the_manifest_is_recovered(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
old_manifest = (folder / page_export.MANIFEST_NAME).read_bytes()
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_commit(instance, "edit")
|
||||
assert _run().exit_code == 0
|
||||
new_state = _snapshot(folder)
|
||||
(folder / page_export.MANIFEST_NAME).write_bytes(old_manifest)
|
||||
assert _run().exit_code == 0
|
||||
assert _snapshot(folder) == new_state
|
||||
|
||||
|
||||
def test_zero_selected_pages_is_an_error_and_deletes_nothing(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
before = _snapshot(folder)
|
||||
_configure(instance, pages={"include": [["tags=nothing-has-this"]]})
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "no page matches" in _flat(result.output)
|
||||
assert _snapshot(folder) == before
|
||||
|
||||
|
||||
def test_a_dirty_kb_is_refused(instance):
|
||||
_configure(instance)
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nunsaved\n")
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "uncommitted" in _flat(result.output)
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
def test_a_page_with_unreadable_frontmatter_stops_the_run(instance):
|
||||
_configure(instance)
|
||||
(instance / "kb" / "concepts" / "Broken.md").write_text("---\ntype: [unclosed\n---\nBody\n", encoding="utf-8")
|
||||
_commit(instance, "broken")
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "Broken.md" in result.output
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
def test_targets_are_independent_and_any_failure_is_exit_1(instance):
|
||||
bad = _out(instance, "bad")
|
||||
bad.mkdir()
|
||||
(bad / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||
_configure(instance, bad={}, good={})
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert (_out(instance, "good") / "concepts" / "Alpha.md").is_file()
|
||||
assert _snapshot(bad) == {"mine.md": b"mine\n"}
|
||||
assert "bad" in result.output and "failed" in result.output
|
||||
|
||||
|
||||
def test_named_targets_run_alone(instance):
|
||||
_configure(instance, one={}, two={})
|
||||
assert _run("two").exit_code == 0
|
||||
assert _out(instance, "two").is_dir() and not _out(instance, "one").exists()
|
||||
|
||||
|
||||
def test_a_case_only_collision_is_refused(instance):
|
||||
_page(instance, "concepts/alpha.md", "\n# alpha\n\nLower.\n")
|
||||
_commit(instance, "case")
|
||||
_configure(instance)
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
assert "differ only in case" in _flat(result.output)
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
# --- the command handler -----------------------------------------------------
|
||||
|
||||
|
||||
def _converter(root: Path, body: str) -> list[str]:
|
||||
script = root.parent / "convert.py"
|
||||
script.write_text(textwrap.dedent(body), encoding="utf-8")
|
||||
return [sys.executable, str(script)]
|
||||
|
||||
|
||||
def _command_target(root: Path, program: list[str]) -> None:
|
||||
_configure(root, pages={"handler": "command", "command": "convert"})
|
||||
_set_commands(root, {"convert": program})
|
||||
|
||||
|
||||
COPY_AS_TXT = """
|
||||
import pathlib, sys
|
||||
src, out = map(pathlib.Path, sys.argv[1:3])
|
||||
for f in src.rglob("*.md"):
|
||||
dest = out / f.relative_to(src).with_suffix(".txt")
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(f.read_text(encoding="utf-8").upper(), encoding="utf-8")
|
||||
"""
|
||||
|
||||
|
||||
def test_the_command_handler_exports_what_the_program_leaves(instance):
|
||||
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
folder = _out(instance, "pages")
|
||||
assert set(_snapshot(folder)) == {
|
||||
"concepts/Alpha.txt", "concepts/protocols/Beta.txt", page_export.MANIFEST_NAME}
|
||||
assert "WIKITOOL_EXPORT" in (folder / "concepts" / "Alpha.txt").read_text(encoding="utf-8")
|
||||
stamps = _mtimes(folder)
|
||||
assert _run().exit_code == 0 and _mtimes(folder) == stamps
|
||||
|
||||
|
||||
def test_the_command_gets_two_directories_no_stdin_and_the_staging_is_removed(instance, tmp_path):
|
||||
record = tmp_path / "record.json"
|
||||
program = _converter(instance, f"""
|
||||
import json, os, pathlib, sys
|
||||
src, out = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
|
||||
stdin = sys.stdin.read()
|
||||
pathlib.Path({str(record)!r}).write_text(json.dumps({{
|
||||
"args": sys.argv[1:], "stdin": stdin, "cwd": os.getcwd(),
|
||||
"src_files": sorted(p.name for p in src.rglob("*.md")), "out_files": os.listdir(out)}}))
|
||||
(out / "all.txt").write_text("done")
|
||||
""")
|
||||
_command_target(instance, program)
|
||||
assert _run().exit_code == 0
|
||||
seen = json.loads(record.read_text(encoding="utf-8"))
|
||||
assert seen["stdin"] == "" and seen["out_files"] == [] and seen["src_files"] == ["Alpha.md", "Beta.md"]
|
||||
assert Path(seen["args"][0]).name == "markdown" and Path(seen["args"][1]).name == "output"
|
||||
assert not Path(seen["args"][0]).parent.exists()
|
||||
|
||||
|
||||
def test_a_failing_command_leaves_the_target_as_it_was(instance):
|
||||
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||
assert _run().exit_code == 0
|
||||
folder = _out(instance, "pages")
|
||||
before = _snapshot(folder)
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_commit(instance, "edit")
|
||||
_set_commands(instance, {"convert": _converter(instance, """
|
||||
import sys
|
||||
sys.stderr.write("x" * 5000 + "boom at the end")
|
||||
sys.exit(3)
|
||||
""")})
|
||||
result = _run()
|
||||
assert result.exit_code == 1
|
||||
text = _flat(result.output)
|
||||
assert "exit 3" in text and "boom at the end" in text and "x" * 3000 not in text
|
||||
assert _snapshot(folder) == before
|
||||
|
||||
|
||||
def test_a_missing_program_and_a_timeout_are_target_errors(instance, monkeypatch):
|
||||
_command_target(instance, ["/nonexistent/converter"])
|
||||
result = _run()
|
||||
assert result.exit_code == 1 and "could not be started" in _flat(result.output)
|
||||
_set_commands(instance, {"convert": _converter(instance, "import time; time.sleep(30)")})
|
||||
monkeypatch.setattr(page_export, "COMMAND_TIMEOUT_SECONDS", 1)
|
||||
result = _run()
|
||||
assert result.exit_code == 1 and "did not finish" in _flat(result.output)
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="symlinks")
|
||||
def test_symlinks_in_the_output_are_skipped_and_reported(instance):
|
||||
_command_target(instance, _converter(instance, """
|
||||
import os, pathlib, sys
|
||||
out = pathlib.Path(sys.argv[2])
|
||||
(out / "real.txt").write_text("real")
|
||||
os.symlink("/etc/hostname", out / "link.txt")
|
||||
"""))
|
||||
result = _run()
|
||||
assert result.exit_code == 0, result.output
|
||||
assert set(_snapshot(_out(instance, "pages"))) == {"real.txt", page_export.MANIFEST_NAME}
|
||||
assert "link.txt" in result.output
|
||||
|
||||
|
||||
def test_a_converter_that_leaves_nothing_is_an_error(instance):
|
||||
_command_target(instance, _converter(instance, "pass"))
|
||||
assert _run().exit_code == 1
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
# --- publish -----------------------------------------------------------------
|
||||
|
||||
|
||||
def _publish(**overrides):
|
||||
kwargs = dict(message="change", push=False, confirm=None, confirm_rebase=None, yes=False,
|
||||
threshold=10, remote="origin", branch="main", path=None)
|
||||
kwargs.update(overrides)
|
||||
publish_command(**kwargs)
|
||||
|
||||
|
||||
def test_publish_exports_after_the_commit_with_no_push_too(instance):
|
||||
_configure(instance, pages={"after_publish": True}, manual={})
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_publish()
|
||||
folder = _out(instance, "pages")
|
||||
head = _git(instance, "rev-parse", "HEAD")
|
||||
assert _manifest(folder)["commit"] == head
|
||||
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||
assert not _out(instance, "manual").exists()
|
||||
|
||||
|
||||
def test_a_failed_export_after_publish_is_a_warning_and_the_commit_stands(instance, capsys):
|
||||
folder = _out(instance, "pages")
|
||||
folder.mkdir()
|
||||
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||
_configure(instance, pages={"after_publish": True})
|
||||
head = _git(instance, "rev-parse", "HEAD")
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_publish()
|
||||
out = capsys.readouterr().out
|
||||
assert "WARN" in out and "export run" in out
|
||||
assert _git(instance, "rev-parse", "HEAD") != head
|
||||
|
||||
|
||||
def test_publish_without_after_publish_exports_nothing(instance):
|
||||
_configure(instance)
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
_publish()
|
||||
assert not _out(instance, "pages").exists()
|
||||
|
||||
|
||||
def test_a_malformed_section_stops_publish_only_when_a_target_asks_for_it(instance):
|
||||
_configure(instance, pages={"after_publish": True, "include": []})
|
||||
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||
head = _git(instance, "rev-parse", "HEAD")
|
||||
with pytest.raises(typer.Exit) as exc:
|
||||
_publish()
|
||||
assert exc.value.exit_code == 1 and _git(instance, "rev-parse", "HEAD") == head
|
||||
_configure(instance, pages={"include": []})
|
||||
_publish()
|
||||
assert _git(instance, "rev-parse", "HEAD") != head
|
||||
|
||||
|
||||
# --- doctor ------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check(name: str):
|
||||
assert name == "export"
|
||||
return doctor.check_export()
|
||||
|
||||
|
||||
def test_doctor_lists_the_targets(instance):
|
||||
_configure(instance, pages={"after_publish": True})
|
||||
check = _check("export")
|
||||
assert check.status == "OK"
|
||||
assert "pages" in check.detail and "after publish" in check.detail
|
||||
|
||||
|
||||
# --- raw accept --------------------------------------------------------------
|
||||
|
||||
|
||||
def _accept(path: Path):
|
||||
return runner.invoke(app, ["raw", "accept", str(path), "--fidelity", "verbatim",
|
||||
"--authority", "normative"])
|
||||
|
||||
|
||||
def test_raw_accept_refuses_a_file_carrying_the_page_export_marker(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
exported = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
|
||||
(instance / "incoming" / "Alpha.md").write_bytes(exported)
|
||||
result = _accept(instance / "incoming" / "Alpha.md")
|
||||
assert result.exit_code == 1
|
||||
assert "export" in _flat(result.output) and "nothing was moved" in _flat(result.output)
|
||||
assert (instance / "incoming" / "Alpha.md").is_file()
|
||||
assert not any((instance / "raw").rglob("*.md"))
|
||||
|
||||
|
||||
def test_raw_accept_refuses_an_exported_file_whose_marker_a_converter_dropped(instance):
|
||||
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||
assert _run().exit_code == 0
|
||||
exported = (_out(instance, "pages") / "concepts" / "Alpha.txt").read_bytes()
|
||||
folder = instance / "incoming" / "pack"
|
||||
folder.mkdir()
|
||||
(folder / "note.txt").write_text("ordinary\n", encoding="utf-8")
|
||||
(folder / "Alpha.txt").write_bytes(exported)
|
||||
(instance / "incoming" / "Alpha.txt").write_bytes(exported)
|
||||
for target in (folder, instance / "incoming" / "Alpha.txt"):
|
||||
before = _snapshot(instance / "incoming")
|
||||
result = _accept(target)
|
||||
assert result.exit_code == 1, result.output
|
||||
assert "configured export target" in _flat(result.output)
|
||||
assert "Alpha.txt" in result.output
|
||||
assert _snapshot(instance / "incoming") == before
|
||||
assert not any(p.is_file() for p in (instance / "raw").rglob("*"))
|
||||
|
||||
|
||||
def test_raw_accept_warns_and_goes_on_when_a_manifest_is_missing(instance):
|
||||
_configure(instance)
|
||||
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
|
||||
result = _accept(instance / "incoming" / "note.md")
|
||||
assert "WARN" in result.output and "no manifest" in _flat(result.output)
|
||||
assert "configured export target" not in result.output
|
||||
|
||||
|
||||
def test_raw_accept_warns_and_goes_on_when_the_section_is_broken(instance):
|
||||
_configure(instance, pages={"include": []})
|
||||
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
|
||||
result = _accept(instance / "incoming" / "note.md")
|
||||
assert "WARN" in result.output and "export section is broken" in _flat(result.output)
|
||||
assert "configured export target" not in result.output
|
||||
|
||||
|
||||
def test_known_digests_lists_the_manifest_files(instance):
|
||||
_configure(instance)
|
||||
assert _run().exit_code == 0
|
||||
digests, warnings = page_export.known_digests()
|
||||
alpha = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
|
||||
assert hashlib.sha256(alpha).hexdigest() in digests and warnings == []
|
||||
|
||||
|
||||
# --- the marker --------------------------------------------------------------
|
||||
|
||||
|
||||
FRONT = b'---\nwikitool_export: "kind=pages instance=local commit=abc"\ntitle: A\n---\n\nBody\n'
|
||||
|
||||
|
||||
@pytest.mark.parametrize("data, expected", [
|
||||
(b"<!-- wikitool:export kind=guidelines -->\nbody", True),
|
||||
(FRONT, True),
|
||||
(b"\xef\xbb\xbf" + FRONT, True),
|
||||
(FRONT.replace(b"\n", b"\r\n"), True),
|
||||
(b"---\ntitle: A\n---\nwikitool_export: x\n", False),
|
||||
(b"---\ntitle: A\nwikitool_export: x\n", False),
|
||||
(b"# Title\n---\nwikitool_export: x\n---\n", False),
|
||||
(b"---\ntitle: A\n---\n\nBody\n", False),
|
||||
(b"", False),
|
||||
])
|
||||
def test_is_export_recognises_both_forms(data, expected):
|
||||
assert guideline_export.is_export(data) is expected
|
||||
|
||||
|
||||
def test_the_key_beyond_the_scan_window_is_not_seen():
|
||||
padding = b"x: " + b"y" * 100 + b"\n"
|
||||
data = b"---\n" + padding * (guideline_export.SCAN_BYTES // len(padding) + 2) + b"wikitool_export: z\n---\n"
|
||||
assert guideline_export.is_export(data) is False
|
||||
|
||||
|
||||
def test_file_is_export_reads_files(tmp_path):
|
||||
(tmp_path / "a.md").write_bytes(FRONT)
|
||||
(tmp_path / "b.md").write_bytes(b"plain\n")
|
||||
assert guideline_export.file_is_export(tmp_path / "a.md")
|
||||
assert not guideline_export.file_is_export(tmp_path / "b.md")
|
||||
|
||||
|
||||
def test_the_marker_key_is_defined_once():
|
||||
package = Path(guideline_export.__file__).parent
|
||||
definitions = [
|
||||
path for path in package.rglob("*.py")
|
||||
if "tests" not in path.parts and "wikitool_export:" in path.read_text("utf-8")
|
||||
and 'b"wikitool_export:"' in path.read_text("utf-8")
|
||||
]
|
||||
assert definitions == [Path(guideline_export.__file__)]
|
||||
Reference in new issue
Block a user