export: export run writes selected kb/ pages as a derived, read-only version into configured target folders (markdown and command handlers, manifest, after_publish), raw accept also refuses manifest-listed files, doctor checks the section (#214); 10.0.0-beta.13
Files changed: - CHANGES.md - VERSION - docs/how-to/README.md - docs/how-to/back-up-to-a-synced-folder.md - docs/how-to/collect-from-a-synced-folder.md - docs/how-to/export-guidelines.md - docs/how-to/export-to-a-synced-folder.md - docs/reference/commands/README.md - docs/reference/commands/exports.md - docs/reference/commands/git.md - docs/reference/commands/guideline-export.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/setup-instance.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/backup.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/export_cmd.py - tools/chemenu/commands/git_publish.py - tools/chemenu/commands/raw/_common.py - tools/chemenu/commands/raw/accept.py - tools/chemenu/commands/raw/capture.py - tools/chemenu/config_shapes.py - tools/chemenu/guideline_export.py - tools/chemenu/page_export.py - tools/chemenu/reference_doc.py - tools/chemenu/repo_capture.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_export_run.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
ecefe0a033
commit
99945c4841
33 files changed
+2316
-96
No files matched your search
+2
-1
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10.0.0-beta.12 - 2026-10-12 - AGENTS.md § Gates and mcp-read-server name every intake channel behind the Upload Review Gate, not only submit (#213 close)
|
## 10.0.0-beta.13 - 2026-10-12 - Exporte als konfigurierte Ziele: export run schreibt eine abgeleitete, schreibgeschützte Lesefassung von kb/ je Zielordner (Markdown oder über ein Konvertierprogramm), auf Wunsch nach jedem publish; raw accept lehnt auch Dateien ab, die in einem Export-Manifest stehen
|
||||||
|
|
||||||
**Author:** Torben Nehmer
|
**Author:** Torben Nehmer
|
||||||
|
|
||||||
@@ -37,6 +37,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
- raw fetch hinter einem Login: raw login speichert die Session des Nutzers als Secret, raw fetch holt damit; gerenderte Seiten über die optionale Browser-Komponente, externe Abrufer pro Host
|
- raw fetch hinter einem Login: raw login speichert die Session des Nutzers als Secret, raw fetch holt damit; gerenderte Seiten über die optionale Browser-Komponente, externe Abrufer pro Host
|
||||||
- Gesprächssprache: AGENTS.md § Conversation language legt fest, dass der Agent im Chat die KB-Sprache spricht; instructions sync schreibt den Sprachwert als generierte Zeile in jedes veröffentlichte SKILL.md, instructions verify und doctor melden eine geänderte Sprache als Drift
|
- Gesprächssprache: AGENTS.md § Conversation language legt fest, dass der Agent im Chat die KB-Sprache spricht; instructions sync schreibt den Sprachwert als generierte Zeile in jedes veröffentlichte SKILL.md, instructions verify und doctor melden eine geänderte Sprache als Drift
|
||||||
- Backup in einen Sync-Ordner (OneDrive): backup init legt ein Bare-Repo unter <ziel>/git als origin an, backup run und publish spiegeln die Maschinendaten nach <ziel>/<maschine>/, doctor warnt vor .git im Sync-Ordner
|
- Backup in einen Sync-Ordner (OneDrive): backup init legt ein Bare-Repo unter <ziel>/git als origin an, backup run und publish spiegeln die Maschinendaten nach <ziel>/<maschine>/, doctor warnt vor .git im Sync-Ordner
|
||||||
|
- Exporte als konfigurierte Ziele: export run schreibt eine abgeleitete, schreibgeschützte Lesefassung von kb/ je Zielordner (Markdown oder über ein Konvertierprogramm), auf Wunsch nach jedem publish; raw accept lehnt auch Dateien ab, die in einem Export-Manifest stehen
|
||||||
|
|
||||||
**Low impact**
|
**Low impact**
|
||||||
- Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
|
- Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ Working with the wiki:
|
|||||||
browser, or through a fetch program of your own
|
browser, or through a fetch program of your own
|
||||||
- [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
|
- [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
|
||||||
- [Export guidelines to captured repositories](export-guidelines.md)
|
- [Export guidelines to captured repositories](export-guidelines.md)
|
||||||
|
- [Export to a synced folder](export-to-a-synced-folder.md) - a read-only version of selected pages
|
||||||
|
for a reader outside the repository, OneDrive and Microsoft 365 included
|
||||||
- [Query the wiki](query-the-wiki.md) - asking, and searching yourself
|
- [Query the wiki](query-the-wiki.md) - asking, and searching yourself
|
||||||
- [Maintain the wiki](maintain-the-wiki.md) - lint, provenance, and the gates on the way to a publish
|
- [Maintain the wiki](maintain-the-wiki.md) - lint, provenance, and the gates on the way to a publish
|
||||||
- [Run the weekly review](run-the-weekly-review.md) - commitments in a task tracker against the
|
- [Run the weekly review](run-the-weekly-review.md) - commitments in a task tracker against the
|
||||||
|
|||||||
@@ -156,3 +156,7 @@ Only one device publishes to `<target>/git`. Two devices pushing into the same s
|
|||||||
writers again, only one step removed - and the sync client resolves that with conflict copies, not
|
writers again, only one step removed - and the sync client resolves that with conflict copies, not
|
||||||
with a merge. An instance on a second device gets a target of its own. The sync client is a backup
|
with a merge. An instance on a second device gets a target of its own. The sync client is a backup
|
||||||
here, not a way to work on one instance from two devices.
|
here, not a way to work on one instance from two devices.
|
||||||
|
|
||||||
|
A folder the sync client uploads can also hold a read-only version of selected pages, for readers
|
||||||
|
outside the repository: [Export pages to a synced folder](export-to-a-synced-folder.md). It is a
|
||||||
|
separate folder with its own settings, and never the backup target.
|
||||||
@@ -131,3 +131,7 @@ exits 0 with "nothing to collect".
|
|||||||
|
|
||||||
The agent does the same at the start of an ingest when you name no file: `upload collect`, then
|
The agent does the same at the start of an ingest when you name no file: `upload collect`, then
|
||||||
`upload list`, then a report of what waits. It never runs `upload accept` for you.
|
`upload list`, then a report of what waits. It never runs `upload accept` for you.
|
||||||
|
|
||||||
|
The other direction - a read-only version of selected pages written out to a synced folder - is
|
||||||
|
[Export pages to a synced folder](export-to-a-synced-folder.md). An export is never a source: the
|
||||||
|
export marker keeps it out of `raw/`.
|
||||||
@@ -50,5 +50,7 @@ did not see.
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Guideline export commands](../reference/commands/guideline-export.md)
|
- [Export commands](../reference/commands/exports.md)
|
||||||
|
- [Export pages to a synced folder](export-to-a-synced-folder.md) - the other export, a read-only
|
||||||
|
version of `kb/` pages
|
||||||
- [Why the gates are code](../explanation/why-gates-are-code.md)
|
- [Why the gates are code](../explanation/why-gates-are-code.md)
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# Export to a synced folder
|
||||||
|
|
||||||
|
Give a reader outside the repository a read-only version of part of `kb/`: the selected pages,
|
||||||
|
written as files into a folder - typically one a sync client such as OneDrive uploads, so that a
|
||||||
|
Microsoft 365 tenant can index them. The export is derived. It is regenerated from the committed
|
||||||
|
pages on every run, nobody edits it, and nothing in it flows back into the wiki. What the settings
|
||||||
|
look like: [configuration reference](../reference/configuration.md#export-to-a-folder).
|
||||||
|
|
||||||
|
<!-- wikitool:toc -->
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- [Pick the pages and the format](#pick-the-pages-and-the-format)
|
||||||
|
- [Set it up](#set-it-up)
|
||||||
|
- [What a run does to the folder](#what-a-run-does-to-the-folder)
|
||||||
|
- [Export after every publish](#export-after-every-publish)
|
||||||
|
- [Convert the pages with a program of your own](#convert-the-pages-with-a-program-of-your-own)
|
||||||
|
- [What does not come back in](#what-does-not-come-back-in)
|
||||||
|
- [Related](#related)
|
||||||
|
<!-- /wikitool:toc -->
|
||||||
|
|
||||||
|
## Pick the pages and the format
|
||||||
|
|
||||||
|
- **Which pages.** Only what a target's `include` selects is written; there is no "everything". A
|
||||||
|
selection is a list of `search` predicates that must all hold - `tags=shared`, `kind=system`,
|
||||||
|
`!draft` - and a page that fits any selection of the target is exported. Try a selection with
|
||||||
|
`tools/wikitool search --field tags=shared` first.
|
||||||
|
- **Which format.** Two ways, equally valid:
|
||||||
|
- `markdown` writes the pages as `.md` files. It needs nothing but the folder.
|
||||||
|
- `command` hands those files to a converter of your own and exports what it produces - Word
|
||||||
|
documents with `pandoc`, say.
|
||||||
|
|
||||||
|
Whether a Microsoft 365 tenant indexes `.md` files for Copilot is not settled for every tenant.
|
||||||
|
Try a small `markdown` target first; if the tenant does not pick the files up, the same pages go
|
||||||
|
through a `command` target as a format it does.
|
||||||
|
|
||||||
|
## Set it up
|
||||||
|
|
||||||
|
1. **Pick the folder**: a folder inside the synced folder, for example
|
||||||
|
`C:\Users\<you>\OneDrive - <company>\Chemenu-Lesefassung`. It must be empty or not exist yet,
|
||||||
|
and it must lie outside the instance, outside every other export target, outside every `intake`
|
||||||
|
folder and outside `backup.target`.
|
||||||
|
2. **Write the `export` section** into `.wikitool-local.json` at the root of the instance - by hand,
|
||||||
|
since its fields decide what leaves this machine and which program runs, and `config set`
|
||||||
|
refuses them:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"export": {
|
||||||
|
"targets": {
|
||||||
|
"shared": {
|
||||||
|
"handler": "markdown",
|
||||||
|
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Lesefassung",
|
||||||
|
"include": [["tags=shared"]],
|
||||||
|
"frontmatter": "summary",
|
||||||
|
"links": "relative",
|
||||||
|
"after_publish": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`frontmatter` decides what follows the export marker in each file: `minimal` nothing, `summary`
|
||||||
|
(the default) the title, type, summary and modified date, `full` the page's whole frontmatter.
|
||||||
|
`links: relative` (the default) turns a link to an exported page into a relative Markdown link;
|
||||||
|
`links: text` leaves the page's name. A link to a page that is not exported is always plain text.
|
||||||
|
3. **Tell the agent:**
|
||||||
|
|
||||||
|
> Export the pages.
|
||||||
|
|
||||||
|
The agent runs `tools/wikitool export run`. It reports per target which pages are new, which
|
||||||
|
changed and which were removed, and how many stayed as they were. Name a target to run only that
|
||||||
|
one: `tools/wikitool export run shared`.
|
||||||
|
4. **Check it:** `tools/wikitool doctor`. The `export` line lists the targets; a `FAIL` there names
|
||||||
|
what is wrong with the section.
|
||||||
|
|
||||||
|
The export reads the committed pages. A `kb/` or `types/` with uncommitted changes is refused, so
|
||||||
|
every file names the commit its pages come from. Publish first.
|
||||||
|
|
||||||
|
## What a run does to the folder
|
||||||
|
|
||||||
|
The folder holds the pages and one file of the stack's own, `.wikitool-export.json`, which lists
|
||||||
|
what the last run wrote.
|
||||||
|
|
||||||
|
- **Only its own files.** A run removes a file only when it is listed and no longer exported, and a
|
||||||
|
folder only when the run created it. Anything else in the folder stays as it is.
|
||||||
|
- **Nobody edits an export.** A listed file that someone changed stops the target before anything is
|
||||||
|
written, and the message names the file. Move or delete it and run again. A listed file that is
|
||||||
|
missing is written again.
|
||||||
|
- **A folder that is not an export is refused.** A folder that holds files and no manifest is
|
||||||
|
somebody's folder; the run writes nothing into it.
|
||||||
|
- **An empty selection is an error.** A run that finds no page does not delete the earlier export.
|
||||||
|
- **An unchanged export stays untouched.** A run that has nothing to change writes nothing, not even
|
||||||
|
a modification time - the sync client has nothing to upload.
|
||||||
|
- **A growing selection needs no confirmation.** New pages are reported as new; read the list
|
||||||
|
before the sync client uploads them.
|
||||||
|
|
||||||
|
Targets are independent: one that fails is reported with its reason and the others still run; the
|
||||||
|
command then exits 1.
|
||||||
|
|
||||||
|
## Export after every publish
|
||||||
|
|
||||||
|
With `"after_publish": true` on a target, every `tools/wikitool publish` that committed refreshes
|
||||||
|
it afterwards - with `--no-push` too, since the export reads the commit, not the remote. If that
|
||||||
|
export fails, `publish` prints a `WARN` with the reason and still exits 0: the commit and the push
|
||||||
|
stand. Fix what the line names and run `tools/wikitool export run`; another `publish` does not
|
||||||
|
repeat it. A malformed `export` section stops `publish` before it commits, but only when some target
|
||||||
|
sets `after_publish`.
|
||||||
|
|
||||||
|
## Convert the pages with a program of your own
|
||||||
|
|
||||||
|
A target with `"handler": "command"` names an entry of `commands`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"export": {
|
||||||
|
"targets": {
|
||||||
|
"as-word": {
|
||||||
|
"handler": "command",
|
||||||
|
"command": "pandoc-docx",
|
||||||
|
"path": "C:\\Users\\<you>\\OneDrive - <company>\\Chemenu-Word",
|
||||||
|
"include": [["tags=shared"]]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"commands": {
|
||||||
|
"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The pages are rendered into a staging directory first. The program runs without a shell, with stdin
|
||||||
|
closed and a limit of 300 seconds, and gets two more arguments: the directory with the Markdown
|
||||||
|
files and an empty directory for its output. On exit 0 the regular files it left there (below any
|
||||||
|
subdirectory, symlinks left out and reported) become the content of the target; the manifest and
|
||||||
|
all the rules above apply to them as to Markdown. Any other end - a non-zero exit, a timeout, a
|
||||||
|
program that does not start - leaves the target exactly as it was and prints the end of the
|
||||||
|
program's stderr. The staging directory is always removed.
|
||||||
|
|
||||||
|
## What does not come back in
|
||||||
|
|
||||||
|
An export is never a source. Every Markdown file starts with a frontmatter block whose first key is
|
||||||
|
`wikitool_export`, and `tools/wikitool raw accept` and `raw capture` refuse a file that carries it.
|
||||||
|
A converter may lose that key, so `raw accept` also refuses any file in `incoming/` that is byte
|
||||||
|
for byte one of the files in the manifest of a configured target - and warns, without refusing,
|
||||||
|
when a target has no manifest yet or the `export` section is broken.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Configuration reference](../reference/configuration.md#export-to-a-folder)
|
||||||
|
- [Export commands](../reference/commands/exports.md)
|
||||||
|
- [Export guidelines to captured repositories](export-guidelines.md) - the other export, one file
|
||||||
|
pushed into a repository
|
||||||
|
- [Back up to a synced folder](back-up-to-a-synced-folder.md) - the same kind of folder, for the
|
||||||
|
backup instead
|
||||||
@@ -15,7 +15,7 @@ collected in [exit codes and gates](../exit-codes.md).
|
|||||||
- [Provenance](#provenance)
|
- [Provenance](#provenance)
|
||||||
- [Raw material and uploads](#raw-material-and-uploads)
|
- [Raw material and uploads](#raw-material-and-uploads)
|
||||||
- [Git](#git)
|
- [Git](#git)
|
||||||
- [Guideline export](#guideline-export)
|
- [Exports](#exports)
|
||||||
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
|
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
|
||||||
- [Types, instructions and docs](#types-instructions-and-docs)
|
- [Types, instructions and docs](#types-instructions-and-docs)
|
||||||
- [Telemetry](#telemetry)
|
- [Telemetry](#telemetry)
|
||||||
@@ -101,11 +101,12 @@ Full records: [git.md](git.md)
|
|||||||
- [`backup init`](git.md#backup-init) - Create the backup target's bare repository and make it `origin` when there is none.
|
- [`backup init`](git.md#backup-init) - Create the backup target's bare repository and make it `origin` when there is none.
|
||||||
- [`backup run`](git.md#backup-run) - Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
- [`backup run`](git.md#backup-run) - Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
||||||
|
|
||||||
## Guideline export
|
## Exports
|
||||||
|
|
||||||
Full records: [guideline-export.md](guideline-export.md)
|
Full records: [exports.md](exports.md)
|
||||||
|
|
||||||
- [`export guidelines`](guideline-export.md#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
|
- [`export guidelines`](exports.md#export-guidelines) - Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
|
||||||
|
- [`export run`](exports.md#export-run) - Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||||
|
|
||||||
## Workshop runs and session budget
|
## Workshop runs and session budget
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,13 @@
|
|||||||
# Guideline export
|
# Exports
|
||||||
|
|
||||||
Writing the generated `GUIDELINES.md` into the repositories the wiki has captured.
|
Derived, read-only versions of `kb/`: the generated `GUIDELINES.md` for the repositories the wiki has captured, and the pages exported into configured folders.
|
||||||
|
|
||||||
|
<!-- wikitool:toc -->
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- [`export guidelines`](#export-guidelines)
|
||||||
|
- [`export run`](#export-run)
|
||||||
|
<!-- /wikitool:toc -->
|
||||||
|
|
||||||
<!-- wikitool:reference-commands -->
|
<!-- wikitool:reference-commands -->
|
||||||
## `export guidelines`
|
## `export guidelines`
|
||||||
@@ -74,4 +81,62 @@ tools/wikitool export guidelines --confirm <token> --push --field tags=guideline
|
|||||||
- `wikitool search` - the same predicates, to preview the selection
|
- `wikitool search` - the same predicates, to preview the selection
|
||||||
- `wikitool raw capture` - how a repository becomes a target
|
- `wikitool raw capture` - how a repository becomes a target
|
||||||
- `instructions/gates.md` - the gate procedure
|
- `instructions/gates.md` - the gate procedure
|
||||||
|
|
||||||
|
## `export run`
|
||||||
|
|
||||||
|
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||||
|
|
||||||
|
**Synopsis**
|
||||||
|
|
||||||
|
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
|
||||||
|
|
||||||
|
**Options**
|
||||||
|
|
||||||
|
```text
|
||||||
|
Arguments:
|
||||||
|
[TARGET]... Targets to run; default: every configured target.
|
||||||
|
|
||||||
|
Options:
|
||||||
|
-h, --help Show this message and exit.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Examples**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tools/wikitool export run
|
||||||
|
tools/wikitool export run work-notes
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behaviour**
|
||||||
|
|
||||||
|
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
|
||||||
|
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
|
||||||
|
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
||||||
|
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
||||||
|
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
||||||
|
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
||||||
|
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
||||||
|
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
||||||
|
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
|
||||||
|
|
||||||
|
**Properties**
|
||||||
|
|
||||||
|
- Changes files: yes
|
||||||
|
- Idempotent: yes
|
||||||
|
- Atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
|
||||||
|
- Counts toward the session call budget: yes
|
||||||
|
- Can reach the network: no
|
||||||
|
|
||||||
|
**Exit codes**
|
||||||
|
|
||||||
|
- `0` - success
|
||||||
|
- `1` - No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
|
||||||
|
- `1` - `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
|
||||||
|
- `1` - A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
|
||||||
|
|
||||||
|
**See also**
|
||||||
|
|
||||||
|
- `wikitool search` - the same predicates, to preview a selection
|
||||||
|
- `wikitool backup run` - the same kind of folder, for this machine's own files
|
||||||
|
- `docs/how-to/export-to-a-synced-folder.md`
|
||||||
<!-- /wikitool:reference-commands -->
|
<!-- /wikitool:reference-commands -->
|
||||||
@@ -134,6 +134,7 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
|
|||||||
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
|
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
|
||||||
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
|
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
|
||||||
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
|
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
|
||||||
|
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
|
||||||
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
|
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
|
||||||
|
|
||||||
**Properties**
|
**Properties**
|
||||||
@@ -149,11 +150,12 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
|
|||||||
|
|
||||||
- `0` - success
|
- `0` - success
|
||||||
- `0` - The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
|
- `0` - The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
|
||||||
|
- `0` - The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
|
||||||
- `1` - git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
- `1` - git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
||||||
- `1` - The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
- `1` - The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
||||||
- `1` - No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
- `1` - No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
||||||
- `1` - `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
- `1` - `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
||||||
- `1` - The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed
|
- `1` - The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
|
||||||
- `42` - The remotes settings file of an earlier version is still there
|
- `42` - The remotes settings file of an earlier version is still there
|
||||||
- `42` - Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
- `42` - Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
||||||
- `42` - Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
- `42` - Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
||||||
@@ -165,6 +167,7 @@ tools/wikitool publish --confirm-rebase <token> --message "ingest: docker-cheats
|
|||||||
- `instructions/gates.md` - the gate procedure
|
- `instructions/gates.md` - the gate procedure
|
||||||
- `instructions/setup-instance.md` - the first publish of a new instance
|
- `instructions/setup-instance.md` - the first publish of a new instance
|
||||||
- `wikitool backup run` - the machine-data backup on its own
|
- `wikitool backup run` - the machine-data backup on its own
|
||||||
|
- `wikitool export run` - the page export on its own
|
||||||
|
|
||||||
## `backup init`
|
## `backup init`
|
||||||
|
|
||||||
|
|||||||
@@ -38,6 +38,7 @@ tools/wikitool doctor --json
|
|||||||
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
||||||
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
||||||
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
||||||
|
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
|
||||||
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
||||||
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
||||||
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
||||||
|
|||||||
@@ -262,7 +262,7 @@ tools/wikitool raw capture --update raw/2026/10/service-docs
|
|||||||
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
||||||
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
||||||
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
||||||
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
||||||
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
||||||
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
||||||
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
||||||
@@ -477,7 +477,7 @@ tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu
|
|||||||
- `0` - success
|
- `0` - success
|
||||||
- `1` - raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
- `1` - raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||||
- `1` - raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
- `1` - raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
||||||
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
|
- `1` - A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
|
||||||
- `1` - raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
- `1` - raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
||||||
- `1` - raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
- `1` - raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
||||||
- `1` - raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
- `1` - raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ something you run.
|
|||||||
- [Site handlers](#site-handlers)
|
- [Site handlers](#site-handlers)
|
||||||
- [Fetch routes](#fetch-routes)
|
- [Fetch routes](#fetch-routes)
|
||||||
- [Backup to a synced folder](#backup-to-a-synced-folder)
|
- [Backup to a synced folder](#backup-to-a-synced-folder)
|
||||||
|
- [Export to a folder](#export-to-a-folder)
|
||||||
- [Files of earlier versions](#files-of-earlier-versions)
|
- [Files of earlier versions](#files-of-earlier-versions)
|
||||||
<!-- /wikitool:toc -->
|
<!-- /wikitool:toc -->
|
||||||
|
|
||||||
@@ -93,7 +94,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
|
|||||||
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
||||||
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
||||||
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
|
[intake](#intake-channels), [sites](#site-handlers), [fetch](#fetch-routes) and
|
||||||
[backup](#backup-to-a-synced-folder). `config set` names the sections it knows, and `doctor` reports a
|
[backup](#backup-to-a-synced-folder) and [export](#export-to-a-folder). `config set` names the sections it knows, and `doctor` reports a
|
||||||
section it does not read as a `WARN`.
|
section it does not read as a `WARN`.
|
||||||
|
|
||||||
**One place per value.** A value in the file its tier does not name is an error naming the file,
|
**One place per value.** A value in the file its tier does not name is an error naming the file,
|
||||||
@@ -631,6 +632,96 @@ Example:
|
|||||||
`config set` exits 42 for them, and only you write them. The procedure, restoring included:
|
`config set` exits 42 for them, and only you write them. The procedure, restoring included:
|
||||||
[back-up-to-a-synced-folder.md](../how-to/back-up-to-a-synced-folder.md).
|
[back-up-to-a-synced-folder.md](../how-to/back-up-to-a-synced-folder.md).
|
||||||
|
|
||||||
|
## Export to a folder
|
||||||
|
|
||||||
|
The `export` section of `.wikitool-local.json` declares folders that hold a derived, read-only
|
||||||
|
version of `kb/` - for a reader outside the repository, such as a Microsoft 365 tenant that indexes
|
||||||
|
a OneDrive folder. An export is regenerated from the committed pages on every run and is never
|
||||||
|
edited; what someone changes in the folder is refused or overwritten, never merged back. Each
|
||||||
|
target has a handler: `markdown` writes the selected pages as Markdown files, `command` renders
|
||||||
|
them into a staging directory and hands that to a converter of this checkout.
|
||||||
|
|
||||||
|
<!-- wikitool:reference-config-export -->
|
||||||
|
| Field | Tier | Type | Required | Default | Description |
|
||||||
|
|-------|------|------|----------|---------|-------------|
|
||||||
|
| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target, an `intake` folder or `backup.target`) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
|
||||||
|
| `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
|
||||||
|
|
||||||
|
- **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
|
||||||
|
- **Malformed:** An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops with exit 1 before it commits anything, but only when some target sets `after_publish`. A broken section is never read as "no export".
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"export": {
|
||||||
|
"targets": {
|
||||||
|
"work-notes": {
|
||||||
|
"handler": "markdown",
|
||||||
|
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
|
||||||
|
"include": [
|
||||||
|
[
|
||||||
|
"kind=system"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"tags=shared",
|
||||||
|
"!draft"
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"frontmatter": "summary",
|
||||||
|
"links": "relative",
|
||||||
|
"after_publish": true
|
||||||
|
},
|
||||||
|
"as-word": {
|
||||||
|
"handler": "command",
|
||||||
|
"command": "pandoc-docx",
|
||||||
|
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
|
||||||
|
"include": [
|
||||||
|
[
|
||||||
|
"kind=system"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"commands": {
|
||||||
|
"pandoc-docx": [
|
||||||
|
"pwsh",
|
||||||
|
"-NoProfile",
|
||||||
|
"-File",
|
||||||
|
"C:\\tools\\pandoc-docx.ps1"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
<!-- /wikitool:reference-config-export -->
|
||||||
|
|
||||||
|
Within one target, `include` is a list of selections, each a list of `search` predicates (`tags=docs`,
|
||||||
|
`entity_type=system`, `!source_url`): a page is exported when it satisfies every predicate of at least
|
||||||
|
one selection. Only what `include` selects is written - there is no way to exclude by default. The
|
||||||
|
syntax is checked when the section is read, the field names against the corpus when `export run`
|
||||||
|
runs. `path` must not lie in or contain the instance, another target, an `intake` folder or
|
||||||
|
`backup.target`.
|
||||||
|
|
||||||
|
Every Markdown file starts with a frontmatter block whose first key is `wikitool_export`, carrying
|
||||||
|
the kind, the instance and the commit of the newest selected page. `frontmatter` decides what follows:
|
||||||
|
`minimal` nothing, `summary` the title, type, summary and modified date, `full` the page's whole
|
||||||
|
frontmatter. Wikilinks between exported pages become relative Markdown links (`links: relative`) or
|
||||||
|
plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote
|
||||||
|
markers and code stay as they are.
|
||||||
|
|
||||||
|
A target folder holds `.wikitool-export.json`, which lists the files a run wrote with their digests.
|
||||||
|
A run removes only listed files that are no longer exported, creates only folders it needs and
|
||||||
|
removes only those it created, and leaves every other file untouched. It stops before writing for a
|
||||||
|
listed file someone changed, for a folder that is not empty and has no manifest, and for a selection
|
||||||
|
that comes up empty. A run that finds nothing to change writes nothing, not even a modification time.
|
||||||
|
|
||||||
|
`targets` and `commands` decide what leaves this checkout and which program runs, so they are marked
|
||||||
|
`gate` and `names a command`: `config set` exits 42 for them, and only you write them. `raw accept`
|
||||||
|
also refuses a file whose sha256 is listed in the manifest of a configured target, for an export
|
||||||
|
that went through a converter and lost its marker. The procedure:
|
||||||
|
[export-to-a-synced-folder.md](../how-to/export-to-a-synced-folder.md).
|
||||||
|
|
||||||
## Files of earlier versions
|
## Files of earlier versions
|
||||||
|
|
||||||
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
|
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
|
|||||||
|
|
||||||
### guideline-push
|
### guideline-push
|
||||||
|
|
||||||
- [`export guidelines`](commands/guideline-export.md#export-guidelines)
|
- [`export guidelines`](commands/exports.md#export-guidelines)
|
||||||
|
|
||||||
### mass-update
|
### mass-update
|
||||||
|
|
||||||
@@ -45,6 +45,6 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
|
|||||||
- [`publish`](commands/git.md#publish): Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
- [`publish`](commands/git.md#publish): Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
||||||
- [`publish`](commands/git.md#publish): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
- [`publish`](commands/git.md#publish): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
||||||
- [`publish`](commands/git.md#publish): Publish-Remote Gate: `remotes.allowed_push_urls` is set and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
|
- [`publish`](commands/git.md#publish): Publish-Remote Gate: `remotes.allowed_push_urls` is set and the push URL of `--remote` is not listed in it, or `--remote` resolves to no push URL
|
||||||
- [`export guidelines`](commands/guideline-export.md#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
|
- [`export guidelines`](commands/exports.md#export-guidelines): Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
|
||||||
- [`config set`](commands/configuration.md#config-set): The field names a program to run or affects a gate
|
- [`config set`](commands/configuration.md#config-set): The field names a program to run or affects a gate
|
||||||
<!-- /wikitool:reference-exit-codes -->
|
<!-- /wikitool:reference-exit-codes -->
|
||||||
@@ -130,6 +130,11 @@ one it needs before that.
|
|||||||
show the user any `remotes.allowed_push_urls` entry it prints, for them to add. The whole
|
show the user any `remotes.allowed_push_urls` entry it prints, for them to add. The whole
|
||||||
procedure is [docs/how-to/back-up-to-a-synced-folder.md](../docs/how-to/back-up-to-a-synced-folder.md).
|
procedure is [docs/how-to/back-up-to-a-synced-folder.md](../docs/how-to/back-up-to-a-synced-folder.md).
|
||||||
Until it is done, treat the instance as local.
|
Until it is done, treat the instance as local.
|
||||||
|
- A reader outside the repository - a Microsoft 365 tenant that indexes a OneDrive folder, say
|
||||||
|
- who should get a read-only version of part of `kb/`: offer an export. The user writes the
|
||||||
|
`export` section of `.wikitool-local.json` by hand (its fields are gates, so `config set`
|
||||||
|
refuses them); then run `tools/wikitool export run`. The whole procedure is
|
||||||
|
[docs/how-to/export-to-a-synced-folder.md](../docs/how-to/export-to-a-synced-folder.md).
|
||||||
|
|
||||||
4. **Decision point - authoring conventions.** The release ships no filled-in conventions, only
|
4. **Decision point - authoring conventions.** The release ships no filled-in conventions, only
|
||||||
`kb/CONVENTIONS.md.template` and one `kb/<name>/COLLECTION.md.template` per collection. Both
|
`kb/CONVENTIONS.md.template` and one `kb/<name>/COLLECTION.md.template` per collection. Both
|
||||||
|
|||||||
+13
-8
@@ -95,12 +95,16 @@ That is what keeps the clean-up safe: only directories the moves emptied are rem
|
|||||||
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
|
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
|
||||||
both ways out - the whole folder, or the file moved up into `incoming/`.
|
both ways out - the whole folder, or the file moved up into `incoming/`.
|
||||||
|
|
||||||
**A guideline export never comes back in.** A file whose first line (after an optional BOM)
|
**An export never comes back in.** A file that carries the stack's export marker is this stack's
|
||||||
starts with `<!-- wikitool:export` is this stack's own output - `export guidelines` renders the
|
own output and `raw accept` refuses it, given on its own, inside a folder, or as
|
||||||
wiki's guideline pages into it for other repositories - and `raw accept` refuses it, given on its
|
`--replaces`/`--replaces-bundle` material, naming each one. The marker is either a first line
|
||||||
own, inside a folder, or as `--replaces`/`--replaces-bundle` material, naming each one. Taken in,
|
(after an optional BOM) that starts with `<!-- wikitool:export` - the generated `GUIDELINES.md` of
|
||||||
it would turn the wiki's knowledge on its way out into a source the wiki then cites for the same
|
`export guidelines` - or, in a leading frontmatter block, a `wikitool_export:` key - a page
|
||||||
claims. `raw capture` leaves such a file out of a repository for the same reason
|
`export run` wrote. A converter of an `export` target may drop the marker, so `raw accept` also
|
||||||
|
refuses a file whose sha256 is listed in the manifest of a configured target; a missing manifest
|
||||||
|
or a broken `export` section is a `WARN` and the check is skipped. Taken in, an export would turn
|
||||||
|
the wiki's knowledge on its way out into a source the wiki then cites for the same claims.
|
||||||
|
`raw capture` leaves a file with the marker out of a repository for the same reason
|
||||||
([below](#getting-a-repository-in-raw-capture)).
|
([below](#getting-a-repository-in-raw-capture)).
|
||||||
|
|
||||||
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
|
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
|
||||||
@@ -447,8 +451,9 @@ is captured from a repository or accepted from `incoming/` anywhere but at the t
|
|||||||
folder.
|
folder.
|
||||||
|
|
||||||
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
|
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
|
||||||
starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its
|
starts with `<!-- wikitool:export` or opens a frontmatter block that holds a `wikitool_export:` key
|
||||||
way out, which must not come back in as a source), a symlink, a submodule, a file over 25 MiB, a
|
(this stack's own export - the wiki's knowledge on its way out, which must not come back in as a
|
||||||
|
source), a symlink, a submodule, a file over 25 MiB, a
|
||||||
path with a segment starting with `.` (`raw accept` refuses hidden entries anyway), a file named
|
path with a segment starting with `.` (`raw accept` refuses hidden entries anyway), a file named
|
||||||
`_capture.json`, and a Git LFS pointer, which holds a reference to the content rather than the
|
`_capture.json`, and a Git LFS pointer, which holds a reference to the content rather than the
|
||||||
content.
|
content.
|
||||||
|
|||||||
+70
-7
@@ -42,7 +42,7 @@ ON FAILURE and NEVER, which address an agent.
|
|||||||
- [Provenance](#provenance)
|
- [Provenance](#provenance)
|
||||||
- [Raw material and uploads](#raw-material-and-uploads)
|
- [Raw material and uploads](#raw-material-and-uploads)
|
||||||
- [Git](#git)
|
- [Git](#git)
|
||||||
- [Guideline export](#guideline-export)
|
- [Exports](#exports)
|
||||||
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
|
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
|
||||||
- [Types, instructions and docs](#types-instructions-and-docs)
|
- [Types, instructions and docs](#types-instructions-and-docs)
|
||||||
- [Telemetry](#telemetry)
|
- [Telemetry](#telemetry)
|
||||||
@@ -140,6 +140,7 @@ publish write non-idempotent budget:counted exit:0,1,
|
|||||||
backup init write idempotent budget:counted exit:0,1 Create the backup target's bare repository and make it `origin` when there is none.
|
backup init write idempotent budget:counted exit:0,1 Create the backup target's bare repository and make it `origin` when there is none.
|
||||||
backup run write idempotent budget:counted exit:0,1 Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
backup run write idempotent budget:counted exit:0,1 Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
||||||
export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
|
export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
|
||||||
|
export run write idempotent budget:counted exit:0,1 Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||||
work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run.
|
work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run.
|
||||||
work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop.
|
work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop.
|
||||||
budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history.
|
budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history.
|
||||||
@@ -1671,7 +1672,7 @@ Capture documentation from a git repository into `incoming/<bundle>/`, with a ma
|
|||||||
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
||||||
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
||||||
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
||||||
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
||||||
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
||||||
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
||||||
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
||||||
@@ -1806,7 +1807,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
|||||||
- 0 success
|
- 0 success
|
||||||
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||||
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
||||||
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
|
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target
|
||||||
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
||||||
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
||||||
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
||||||
@@ -1821,7 +1822,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
|||||||
|
|
||||||
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||||
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form
|
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form
|
||||||
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
|
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is an export of this wiki: it carries the export marker (a first line starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured `export` target -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
|
||||||
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
||||||
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
|
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
|
||||||
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
|
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
|
||||||
@@ -2178,11 +2179,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
|||||||
|
|
||||||
- 0 success
|
- 0 success
|
||||||
- 0 The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
|
- 0 The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand
|
||||||
|
- 0 The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands
|
||||||
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
||||||
- 1 The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
- 1 The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
||||||
- 1 No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
- 1 No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
||||||
- 1 `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
- 1 `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
||||||
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed
|
- 1 The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed
|
||||||
- 42 The remotes settings file of an earlier version is still there
|
- 42 The remotes settings file of an earlier version is still there
|
||||||
- 42 Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
- 42 Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset
|
||||||
- 42 Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
- 42 Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
||||||
@@ -2191,11 +2193,12 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
|||||||
**ON FAILURE**
|
**ON FAILURE**
|
||||||
|
|
||||||
- The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool backup run` repeats the backup
|
- The machine-data backup after the push failed: a `WARN` line names the reason, and the commit and the push stand -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool backup run` repeats the backup
|
||||||
|
- The export after the commit failed: a `WARN` line names the target and the reason, and the commit stands -> Show the `WARN` line to the user. Do not publish again - once the cause is fixed, `tools/wikitool export run` repeats the export
|
||||||
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change) -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
|
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change) -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
|
||||||
- The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case -> Check out the branch you mean to publish, or pass `--branch <checked-out branch>`, then retry once
|
- The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case -> Check out the branch you mean to publish, or pass `--branch <checked-out branch>`, then retry once
|
||||||
- No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed -> Show the message to the user and ask whether to commit locally with `--no-push`. Never push by hand - the next `publish` that reaches the remote sends that commit
|
- No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed -> Show the message to the user and ask whether to commit locally with `--no-push`. Never push by hand - the next `publish` that reaches the remote sends that commit
|
||||||
- `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error -> Drop it. The Mass-Update Gate is cleared only with `--confirm <token>` from the gate's own refusal output
|
- `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error -> Drop it. The Mass-Update Gate is cleared only with `--confirm <token>` from the gate's own refusal output
|
||||||
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, or the `backup` section is malformed; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
|
- The settings files do not pass their check, `remotes.allowed_push_urls` is not a list of strings, the `backup` section is malformed, or the `export` section is malformed while a target sets `after_publish`; nothing was committed -> Show the error to the user and stop - a malformed section is not permission, and fixing it is theirs to do
|
||||||
- The remotes settings file of an earlier version is still there -> Show the output to the user verbatim and stop; `tools/wikitool config migrate` moves it, and the user decides when
|
- The remotes settings file of an earlier version is still there -> Show the output to the user verbatim and stop; `tools/wikitool config migrate` moves it, and the user decides when
|
||||||
- Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
- Mass-Update Gate: `--threshold` (default 10) or more counted files would be committed, or the `--confirm` token does not match this changeset -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||||
- Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
- Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||||
@@ -2225,6 +2228,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
|||||||
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
|
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree, with one exception: uncommitted generated files outside `--path` (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`) are staged and committed too, and a line names them - whether a reconcile or the change itself wrote them. They stay out of the gate count, and the `--confirm` token and the re-run line cover the `--path` given. Every other file outside `--path` is left as it is.
|
||||||
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
|
- Commit message: `--message`, then a `Files changed:` paragraph listing every committed path. When `--message` ends in a paragraph git reads as a trailer block (`Co-Authored-By:` and the like), that block stays the last paragraph and the list goes in front of it, since git reads trailers from the last paragraph only. `git interpret-trailers` decides whether there is such a block, and the list moves only when git reads the same trailers from the result; otherwise it is appended at the end. `--message` is not part of any gate token.
|
||||||
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
|
- Machine-data backup: with a `backup` section in `.wikitool-local.json` whose `machine_data` is true, a call that pushed then mirrors this machine's files that are not in git into `<target>/<machine>/` - the same step as `backup run`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit and the push stand. No backup with `--no-push`, after a failed push, or when there was nothing to push. A malformed `backup` section stops the call with exit 1 before anything is committed, `--no-push` included.
|
||||||
|
- Export after publish: every target of the `export` section with `after_publish: true` is refreshed once a commit was made - the same step as `export run`, `--no-push` included, since the export reads the committed `kb/`. A failure there prints a `WARN` line with the reason and still exits 0, since the commit stands; `export run` repeats it. A malformed `export` section stops the call with exit 1 before anything is committed, but only when some target sets `after_publish`.
|
||||||
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
|
- After a successful commit or push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder line: the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. It is not a gate: no exit code change, nothing to clear, and silent for an ordinary content publish.
|
||||||
|
|
||||||
**SEE ALSO**
|
**SEE ALSO**
|
||||||
@@ -2233,6 +2237,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
|||||||
- `instructions/gates.md` - the gate procedure
|
- `instructions/gates.md` - the gate procedure
|
||||||
- `instructions/setup-instance.md` - the first publish of a new instance
|
- `instructions/setup-instance.md` - the first publish of a new instance
|
||||||
- `wikitool backup run` - the machine-data backup on its own
|
- `wikitool backup run` - the machine-data backup on its own
|
||||||
|
- `wikitool export run` - the page export on its own
|
||||||
|
|
||||||
#### `backup init`
|
#### `backup init`
|
||||||
|
|
||||||
@@ -2337,7 +2342,7 @@ Mirror this machine's files that are not in git into `<target>/<machine>/`.
|
|||||||
- `wikitool doctor` - the `backup` line reports the age of the last run
|
- `wikitool doctor` - the `backup` line reports the age of the last run
|
||||||
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
|
- `docs/how-to/back-up-to-a-synced-folder.md` - the whole procedure, restoring included
|
||||||
|
|
||||||
### Guideline export
|
### Exports
|
||||||
|
|
||||||
#### `export guidelines`
|
#### `export guidelines`
|
||||||
|
|
||||||
@@ -2405,6 +2410,63 @@ Render the guideline pages into one generated `GUIDELINES.md`, or push it into t
|
|||||||
- `wikitool raw capture` - how a repository becomes a target
|
- `wikitool raw capture` - how a repository becomes a target
|
||||||
- `instructions/gates.md` - the gate procedure
|
- `instructions/gates.md` - the gate procedure
|
||||||
|
|
||||||
|
#### `export run`
|
||||||
|
|
||||||
|
Write the derived, read-only reading copy of `kb/` into the folders the `export` section configures - the selected pages as Markdown, or converted by a program of this checkout.
|
||||||
|
|
||||||
|
**SYNOPSIS**
|
||||||
|
|
||||||
|
- `wikitool export run [<target> ...]` - Every configured target, or the named ones. Needs an `export` section in `.wikitool-local.json`; the user writes it by hand.
|
||||||
|
|
||||||
|
**PROPERTIES**
|
||||||
|
|
||||||
|
- effect: write
|
||||||
|
- idempotent: yes
|
||||||
|
- atomic: Per file within a target (a temporary file in the target, then a move over its place), the manifest last; targets are independent, so a failed target does not undo another
|
||||||
|
- budget: counted
|
||||||
|
- network: no
|
||||||
|
|
||||||
|
**EXAMPLES**
|
||||||
|
|
||||||
|
- `tools/wikitool export run`
|
||||||
|
- `tools/wikitool export run work-notes`
|
||||||
|
|
||||||
|
**EXIT STATUS**
|
||||||
|
|
||||||
|
- 0 success
|
||||||
|
- 1 No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target)
|
||||||
|
- 1 `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
|
||||||
|
- 1 A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
|
||||||
|
|
||||||
|
**ON FAILURE**
|
||||||
|
|
||||||
|
- No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or the backup target) -> Show the message to the user; the section is theirs to write by hand, never yours
|
||||||
|
- `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter -> Fix the named pages, or publish the changes first, then retry once
|
||||||
|
- A target failed: a foreign folder, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0 -> Show the target's reason to the user; the other targets were still served. Never delete or edit files in a target folder to make a run pass
|
||||||
|
|
||||||
|
**NEVER**
|
||||||
|
|
||||||
|
- Never edit a file in a target folder to change an export - the next run refuses or overwrites it; change the page, or the target's `include`, instead.
|
||||||
|
- Never put an export folder into `incoming/` or `raw/`.
|
||||||
|
|
||||||
|
**NOTES**
|
||||||
|
|
||||||
|
- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
|
||||||
|
- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
|
||||||
|
- Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
|
||||||
|
- The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
|
||||||
|
- Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created.
|
||||||
|
- Refuses everything, before any target: uncommitted changes in `kb/` or `types/` (untracked files included), no git checkout, a page under `kb/` with unreadable frontmatter, an unknown target name, a broken `export` section.
|
||||||
|
- Output per target: `new` (with the page titles), `changed`, `removed`, the count `unchanged`, or `failed` with the reason. A page that newly enters a selection is reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the others were still served.
|
||||||
|
- A target with `after_publish: true` is also refreshed by every `publish` that committed, `--no-push` included; a failure there is a warning, the commit stands, and this command is the repeat.
|
||||||
|
- `raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries the marker: an export never goes back into `raw/`.
|
||||||
|
|
||||||
|
**SEE ALSO**
|
||||||
|
|
||||||
|
- `wikitool search` - the same predicates, to preview a selection
|
||||||
|
- `wikitool backup run` - the same kind of folder, for this machine's own files
|
||||||
|
- `docs/how-to/export-to-a-synced-folder.md`
|
||||||
|
|
||||||
### Workshop runs and session budget
|
### Workshop runs and session budget
|
||||||
|
|
||||||
#### `work new`
|
#### `work new`
|
||||||
@@ -4073,6 +4135,7 @@ Check that this instance is correctly configured.
|
|||||||
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
- Legacy settings (`legacy-settings`): one `FAIL` for each per-feature settings file of an earlier version (telemetry, remotes, upload, task tracker) that is still there. Nothing reads them any more, so the settings they hold are not in effect; the fix is `tools/wikitool config migrate`.
|
||||||
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
- Synced folder (`sync-folder`): a `WARN` when the git directory, links and junctions resolved, lies under a folder named by `OneDrive`, `OneDriveCommercial` or `OneDriveConsumer` (compared without regard to case on Windows) or under `~/Library/CloudStorage/` (macOS); `OK` otherwise, never a `FAIL`.
|
||||||
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
- Backup (`backup`): the `backup` section absent is `OK`, malformed is a `FAIL`. Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not push there, or - with `machine_data` true - no machine-data backup exists yet; otherwise `OK` with the age of the last machine-data backup.
|
||||||
|
- Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; otherwise `OK` listing each target with its handler, folder and whether it runs after `publish`.
|
||||||
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
- Intake channels (`upload-intake`): the channels of the `intake` section with their handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an `mcp-upload/` of an earlier version that still holds submissions (the message names the command that moves it). A `folder` channel whose `path` does not exist is a `WARN`.
|
||||||
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
- Task tracker: the `tasks` section present, absent or malformed - absent is `OK` (no tracker configured), malformed is a `FAIL`. When `WIKITOOL_TASKS_CONFIG` is set, the profile it names is read instead of the `tasks`, `superproductivity` and `caldav` sections and the finding names it; a set variable that names no file is a `FAIL`, never `OK`.
|
||||||
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
- For a configured `superproductivity` provider, the configured `access` path's own state: `access: "api"` reports whether its local REST API answers `GET /health` with a ready renderer right now, `access: "snapshot"` whether a backup file is ready. The other access path is never attempted, and neither state is ever a `FAIL`.
|
||||||
|
|||||||
+2
-1
@@ -132,7 +132,8 @@ tools/
|
|||||||
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
|
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
|
||||||
backup.py `backup init`/`backup run`'s core and the step `publish` runs after a push: the bare repository under `<target>/git`, the machine-data mirror under `<target>/<machine>/` with its manifest, and which synced folders `doctor` knows
|
backup.py `backup init`/`backup run`'s core and the step `publish` runs after a push: the bare repository under `<target>/git`, the machine-data mirror under `<target>/<machine>/` with its manifest, and which synced folders `doctor` knows
|
||||||
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
|
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
|
||||||
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
|
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`is_export`, for both the guideline comment line and the page frontmatter key) - with no CLI attached
|
||||||
|
page_export.py `export run`'s core and the step `publish` runs after a commit: the `export` section, rendering selected pages as Markdown, the target folder with its manifest, and the converter handler - with no CLI attached
|
||||||
search/ pluggable search backends, plus service.py - the search core
|
search/ pluggable search backends, plus service.py - the search core
|
||||||
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (the tracker sections of the settings), one module per adapter - no instruction ever learns which provider it is
|
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (the tracker sections of the settings), one module per adapter - no instruction ever learns which provider it is
|
||||||
commands/ one module per command or command group: the terminal adapters - except `raw/`, a package with one module per subcommand and the helpers several of them share in `_common.py`
|
commands/ one module per command or command group: the terminal adapters - except `raw/`, a package with one module per subcommand and the helpers several of them share in `_common.py`
|
||||||
|
|||||||
+29
-29
@@ -99,12 +99,12 @@ class BackupConfig:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _is_link(path: Path) -> bool:
|
def is_link(path: Path) -> bool:
|
||||||
is_junction = getattr(path, "is_junction", None) # Python 3.12+
|
is_junction = getattr(path, "is_junction", None) # Python 3.12+
|
||||||
return path.is_symlink() or bool(is_junction and is_junction())
|
return path.is_symlink() or bool(is_junction and is_junction())
|
||||||
|
|
||||||
|
|
||||||
def _real(path: Path) -> Path:
|
def real_path(path: Path) -> Path:
|
||||||
return Path(os.path.realpath(path))
|
return Path(os.path.realpath(path))
|
||||||
|
|
||||||
|
|
||||||
@@ -121,7 +121,7 @@ def _names_git(parts) -> bool:
|
|||||||
return any(part.casefold() == GIT_NAME for part in parts)
|
return any(part.casefold() == GIT_NAME for part in parts)
|
||||||
|
|
||||||
|
|
||||||
def _safe_relative(rel: str) -> bool:
|
def safe_relative(rel: str) -> bool:
|
||||||
"""Whether `rel`, a `/`-separated path, stays below the folder it is joined
|
"""Whether `rel`, a `/`-separated path, stays below the folder it is joined
|
||||||
to on every platform: no empty, `.` or `..` part, no backslash, no drive."""
|
to on every platform: no empty, `.` or `..` part, no backslash, no drive."""
|
||||||
parts = rel.split("/")
|
parts = rel.split("/")
|
||||||
@@ -132,13 +132,13 @@ def _safe_relative(rel: str) -> bool:
|
|||||||
def _within(machine_dir: Path, rel: str) -> Path:
|
def _within(machine_dir: Path, rel: str) -> Path:
|
||||||
"""`rel` below `machine_dir`, refusing a path that would leave it - by its
|
"""`rel` below `machine_dir`, refusing a path that would leave it - by its
|
||||||
spelling, or through a link somewhere in the folder."""
|
spelling, or through a link somewhere in the folder."""
|
||||||
if not _safe_relative(rel):
|
if not safe_relative(rel):
|
||||||
raise BackupError(f"{rel!r} is not a path below {machine_dir} - nothing further was written")
|
raise BackupError(f"{rel!r} is not a path below {machine_dir} - nothing further was written")
|
||||||
dest = machine_dir.joinpath(*rel.split("/"))
|
dest = machine_dir.joinpath(*rel.split("/"))
|
||||||
current = machine_dir
|
current = machine_dir
|
||||||
for part in rel.split("/")[:-1]:
|
for part in rel.split("/")[:-1]:
|
||||||
current = current / part
|
current = current / part
|
||||||
if _is_link(current):
|
if is_link(current):
|
||||||
raise BackupError(f"{current} is a link - a backup writes nothing through one")
|
raise BackupError(f"{current} is a link - a backup writes nothing through one")
|
||||||
return dest
|
return dest
|
||||||
|
|
||||||
@@ -174,13 +174,13 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
|
|||||||
if not section.configured:
|
if not section.configured:
|
||||||
return None
|
return None
|
||||||
problems = []
|
problems = []
|
||||||
real_root = _real(root)
|
real_root = real_path(root)
|
||||||
|
|
||||||
target = Path(section["target"])
|
target = Path(section["target"])
|
||||||
if not os.path.isabs(section["target"]):
|
if not os.path.isabs(section["target"]):
|
||||||
problems.append(f"backup.target {section['target']!r} is not an absolute path on this machine")
|
problems.append(f"backup.target {section['target']!r} is not an absolute path on this machine")
|
||||||
else:
|
else:
|
||||||
real_target = _real(target)
|
real_target = real_path(target)
|
||||||
if is_under(real_target, real_root):
|
if is_under(real_target, real_root):
|
||||||
problems.append(f"backup.target {target} lies inside the instance - it belongs outside it")
|
problems.append(f"backup.target {target} lies inside the instance - it belongs outside it")
|
||||||
|
|
||||||
@@ -195,7 +195,7 @@ def read_config(root: "Path | str | None" = None) -> Optional[BackupConfig]:
|
|||||||
|
|
||||||
if not problems:
|
if not problems:
|
||||||
for inner in (target / config_shapes.BACKUP_GIT_DIR, target / machine_name):
|
for inner in (target / config_shapes.BACKUP_GIT_DIR, target / machine_name):
|
||||||
if is_under(real_root, _real(inner)):
|
if is_under(real_root, real_path(inner)):
|
||||||
problems.append(f"the instance lies inside {inner}, which the backup writes to")
|
problems.append(f"the instance lies inside {inner}, which the backup writes to")
|
||||||
|
|
||||||
for entry in section["extra"]:
|
for entry in section["extra"]:
|
||||||
@@ -249,14 +249,14 @@ def same_location(url: str, path: Path, root: Path) -> bool:
|
|||||||
local = url[len("file://"):] if url.startswith("file://") else url
|
local = url[len("file://"):] if url.startswith("file://") else url
|
||||||
if "://" in local or (":" in local and not os.path.isabs(local)):
|
if "://" in local or (":" in local and not os.path.isabs(local)):
|
||||||
return False
|
return False
|
||||||
a, b = os.path.normcase(_real(root / local)), os.path.normcase(_real(path))
|
a, b = os.path.normcase(real_path(root / local)), os.path.normcase(real_path(path))
|
||||||
return a.casefold() == b.casefold() if IGNORE_CASE else a == b
|
return a.casefold() == b.casefold() if IGNORE_CASE else a == b
|
||||||
|
|
||||||
|
|
||||||
def _directory_state(path: Path) -> str:
|
def _directory_state(path: Path) -> str:
|
||||||
if not path.exists() and not _is_link(path):
|
if not path.exists() and not is_link(path):
|
||||||
return "absent"
|
return "absent"
|
||||||
if path.is_dir() and not _is_link(path) and not any(path.iterdir()):
|
if path.is_dir() and not is_link(path) and not any(path.iterdir()):
|
||||||
return "empty"
|
return "empty"
|
||||||
return "bare" if is_bare_repo(path) else "other"
|
return "bare" if is_bare_repo(path) else "other"
|
||||||
|
|
||||||
@@ -329,7 +329,7 @@ class Collected:
|
|||||||
|
|
||||||
|
|
||||||
def _walk(path: Path, rel: str, out: Collected) -> None:
|
def _walk(path: Path, rel: str, out: Collected) -> None:
|
||||||
if _is_link(path):
|
if is_link(path):
|
||||||
out.skipped.append(f"{rel} (a link - not followed)")
|
out.skipped.append(f"{rel} (a link - not followed)")
|
||||||
return
|
return
|
||||||
if path.name.casefold() == GIT_NAME:
|
if path.name.casefold() == GIT_NAME:
|
||||||
@@ -337,7 +337,7 @@ def _walk(path: Path, rel: str, out: Collected) -> None:
|
|||||||
if path.is_file():
|
if path.is_file():
|
||||||
if rel == MANIFEST_NAME:
|
if rel == MANIFEST_NAME:
|
||||||
out.skipped.append(f"{rel} (the manifest's own name)")
|
out.skipped.append(f"{rel} (the manifest's own name)")
|
||||||
elif not _safe_relative(rel):
|
elif not safe_relative(rel):
|
||||||
out.skipped.append(f"{rel} (a name that cannot be mirrored on every platform)")
|
out.skipped.append(f"{rel} (a name that cannot be mirrored on every platform)")
|
||||||
else:
|
else:
|
||||||
out.files.setdefault(rel, path)
|
out.files.setdefault(rel, path)
|
||||||
@@ -351,13 +351,13 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
|
|||||||
"""What a machine-data run mirrors: the root files that exist, everything in
|
"""What a machine-data run mirrors: the root files that exist, everything in
|
||||||
`incoming/` but its `.gitkeep`, and `extra`."""
|
`incoming/` but its `.gitkeep`, and `extra`."""
|
||||||
root = Path(config.ROOT if root is None else root)
|
root = Path(config.ROOT if root is None else root)
|
||||||
real_root = _real(root)
|
real_root = real_path(root)
|
||||||
out = Collected()
|
out = Collected()
|
||||||
for name in ROOT_FILES:
|
for name in ROOT_FILES:
|
||||||
if (root / name).exists() or _is_link(root / name):
|
if (root / name).exists() or is_link(root / name):
|
||||||
_walk(root / name, name, out)
|
_walk(root / name, name, out)
|
||||||
incoming = root / INCOMING_DIR
|
incoming = root / INCOMING_DIR
|
||||||
if _is_link(incoming):
|
if is_link(incoming):
|
||||||
out.skipped.append(f"{INCOMING_DIR} (a link - not followed)")
|
out.skipped.append(f"{INCOMING_DIR} (a link - not followed)")
|
||||||
elif incoming.is_dir():
|
elif incoming.is_dir():
|
||||||
for child in sorted(incoming.iterdir()):
|
for child in sorted(incoming.iterdir()):
|
||||||
@@ -375,7 +375,7 @@ def collect(cfg: BackupConfig, root: "Path | str | None" = None) -> Collected:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _digest(path: Path) -> tuple[int, str]:
|
def file_digest(path: Path) -> tuple[int, str]:
|
||||||
sha, size = hashlib.sha256(), 0
|
sha, size = hashlib.sha256(), 0
|
||||||
with open(path, "rb") as handle:
|
with open(path, "rb") as handle:
|
||||||
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||||
@@ -395,7 +395,7 @@ def _replace(tmp: str, dest: Path) -> None:
|
|||||||
time.sleep(_REPLACE_PAUSE_SECONDS)
|
time.sleep(_REPLACE_PAUSE_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
def write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
||||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
fd, tmp = tempfile.mkstemp(prefix=".chemenu-", suffix=".tmp", dir=str(dest.parent))
|
fd, tmp = tempfile.mkstemp(prefix=".chemenu-", suffix=".tmp", dir=str(dest.parent))
|
||||||
sha, size = hashlib.sha256(), 0
|
sha, size = hashlib.sha256(), 0
|
||||||
@@ -416,7 +416,7 @@ def _write_atomic(dest: Path, chunks) -> tuple[int, str]:
|
|||||||
|
|
||||||
def _copy_atomic(src: Path, dest: Path) -> tuple[int, str]:
|
def _copy_atomic(src: Path, dest: Path) -> tuple[int, str]:
|
||||||
with open(src, "rb") as handle:
|
with open(src, "rb") as handle:
|
||||||
return _write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
|
return write_atomic(dest, iter(lambda: handle.read(1 << 20), b""))
|
||||||
|
|
||||||
|
|
||||||
def read_manifest(machine_dir: Path) -> Optional[dict]:
|
def read_manifest(machine_dir: Path) -> Optional[dict]:
|
||||||
@@ -432,7 +432,7 @@ def read_manifest(machine_dir: Path) -> Optional[dict]:
|
|||||||
files = data.get("files") if isinstance(data, dict) else None
|
files = data.get("files") if isinstance(data, dict) else None
|
||||||
if not isinstance(files, list) or not all(
|
if not isinstance(files, list) or not all(
|
||||||
isinstance(entry, dict) and isinstance(entry.get("path"), str)
|
isinstance(entry, dict) and isinstance(entry.get("path"), str)
|
||||||
and _safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
and safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
||||||
and isinstance(entry.get("sha256"), str)
|
and isinstance(entry.get("sha256"), str)
|
||||||
for entry in files
|
for entry in files
|
||||||
):
|
):
|
||||||
@@ -456,7 +456,7 @@ class RunResult:
|
|||||||
return line
|
return line
|
||||||
|
|
||||||
|
|
||||||
def _prune_empty(directory: Path, stop: Path) -> None:
|
def prune_empty(directory: Path, stop: Path) -> None:
|
||||||
while directory != stop and is_under(directory, stop):
|
while directory != stop and is_under(directory, stop):
|
||||||
try:
|
try:
|
||||||
directory.rmdir()
|
directory.rmdir()
|
||||||
@@ -484,7 +484,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
|||||||
"synced folder there? `tools/wikitool backup init` creates the target"
|
"synced folder there? `tools/wikitool backup init` creates the target"
|
||||||
)
|
)
|
||||||
machine_dir = cfg.machine_dir
|
machine_dir = cfg.machine_dir
|
||||||
if _is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
|
if is_link(machine_dir) or (machine_dir.exists() and not machine_dir.is_dir()):
|
||||||
raise BackupError(f"{machine_dir} is not a plain directory - nothing was written")
|
raise BackupError(f"{machine_dir} is not a plain directory - nothing was written")
|
||||||
previous = read_manifest(machine_dir)
|
previous = read_manifest(machine_dir)
|
||||||
if previous is None and machine_dir.is_dir() and any(machine_dir.iterdir()):
|
if previous is None and machine_dir.is_dir() and any(machine_dir.iterdir()):
|
||||||
@@ -500,10 +500,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
|||||||
machine_dir.mkdir(exist_ok=True)
|
machine_dir.mkdir(exist_ok=True)
|
||||||
for rel, source in sorted(collected.files.items()):
|
for rel, source in sorted(collected.files.items()):
|
||||||
dest = _within(machine_dir, rel)
|
dest = _within(machine_dir, rel)
|
||||||
size, digest = _digest(source)
|
size, digest = file_digest(source)
|
||||||
old = before.get(rel)
|
old = before.get(rel)
|
||||||
if (old and old["size"] == size and old["sha256"] == digest and dest.is_file()
|
if (old and old["size"] == size and old["sha256"] == digest and dest.is_file()
|
||||||
and not _is_link(dest) and dest.stat().st_size == size):
|
and not is_link(dest) and dest.stat().st_size == size):
|
||||||
result.unchanged.append(rel)
|
result.unchanged.append(rel)
|
||||||
else:
|
else:
|
||||||
size, digest = _copy_atomic(source, dest)
|
size, digest = _copy_atomic(source, dest)
|
||||||
@@ -512,10 +512,10 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
|||||||
|
|
||||||
for rel in sorted(set(before) - set(collected.files)):
|
for rel in sorted(set(before) - set(collected.files)):
|
||||||
dest = _within(machine_dir, rel)
|
dest = _within(machine_dir, rel)
|
||||||
if dest.is_file() and not _is_link(dest):
|
if dest.is_file() and not is_link(dest):
|
||||||
dest.unlink()
|
dest.unlink()
|
||||||
result.removed.append(rel)
|
result.removed.append(rel)
|
||||||
_prune_empty(dest.parent, machine_dir)
|
prune_empty(dest.parent, machine_dir)
|
||||||
|
|
||||||
manifest = {
|
manifest = {
|
||||||
"schema": MANIFEST_SCHEMA,
|
"schema": MANIFEST_SCHEMA,
|
||||||
@@ -525,7 +525,7 @@ def run(cfg: BackupConfig, root: "Path | str | None" = None) -> RunResult:
|
|||||||
"files": entries,
|
"files": entries,
|
||||||
}
|
}
|
||||||
text = json.dumps(manifest, indent=2, ensure_ascii=False) + "\n"
|
text = json.dumps(manifest, indent=2, ensure_ascii=False) + "\n"
|
||||||
_write_atomic(cfg.manifest, [text.encode("utf-8")])
|
write_atomic(cfg.manifest, [text.encode("utf-8")])
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
@@ -544,9 +544,9 @@ def sync_roots() -> list[Path]:
|
|||||||
def synced_folder_of(path: Path) -> Optional[Path]:
|
def synced_folder_of(path: Path) -> Optional[Path]:
|
||||||
"""The synced folder `path` lies in, links and junctions resolved on both
|
"""The synced folder `path` lies in, links and junctions resolved on both
|
||||||
sides, or None."""
|
sides, or None."""
|
||||||
real = _real(path)
|
real = real_path(path)
|
||||||
for candidate in sync_roots():
|
for candidate in sync_roots():
|
||||||
if is_under(real, _real(candidate)):
|
if is_under(real, real_path(candidate)):
|
||||||
return candidate
|
return candidate
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|||||||
@@ -293,8 +293,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
|||||||
("Git", (
|
("Git", (
|
||||||
"sync", "publish", "backup init", "backup run",
|
"sync", "publish", "backup init", "backup run",
|
||||||
)),
|
)),
|
||||||
("Guideline export", (
|
("Exports", (
|
||||||
"export guidelines",
|
"export guidelines", "export run",
|
||||||
)),
|
)),
|
||||||
("Workshop runs and session budget", (
|
("Workshop runs and session budget", (
|
||||||
"work new", "work close", "budget status", "budget reset",
|
"work new", "work close", "budget status", "budget reset",
|
||||||
|
|||||||
@@ -53,6 +53,7 @@ def _git(args: list[str]) -> Optional[subprocess.CompletedProcess]:
|
|||||||
|
|
||||||
PREFLIGHT_FIX = f"Run tools/preflight.sh (PowerShell 7: {toolpaths.PREFLIGHT_PWSH})"
|
PREFLIGHT_FIX = f"Run tools/preflight.sh (PowerShell 7: {toolpaths.PREFLIGHT_PWSH})"
|
||||||
BACKUP_HOW_TO = "docs/how-to/back-up-to-a-synced-folder.md"
|
BACKUP_HOW_TO = "docs/how-to/back-up-to-a-synced-folder.md"
|
||||||
|
EXPORT_HOW_TO = "docs/how-to/export-to-a-synced-folder.md"
|
||||||
|
|
||||||
|
|
||||||
def check_python() -> Check:
|
def check_python() -> Check:
|
||||||
@@ -640,6 +641,27 @@ def _telemetry_file_deviations() -> list[str]:
|
|||||||
return config_shapes.deviations(config_shapes.TELEMETRY, data[config_shapes.TELEMETRY.key])
|
return config_shapes.deviations(config_shapes.TELEMETRY, data[config_shapes.TELEMETRY.key])
|
||||||
|
|
||||||
|
|
||||||
|
def check_export() -> Check:
|
||||||
|
"""The `export` section: absent is `OK` (no page export configured), a
|
||||||
|
broken one is a `FAIL`, and a configured one lists its targets."""
|
||||||
|
from chemenu import page_export
|
||||||
|
|
||||||
|
try:
|
||||||
|
targets = page_export.read_config(config.ROOT)
|
||||||
|
except ChemenuError as exc:
|
||||||
|
return Check(
|
||||||
|
"export", "FAIL", str(exc),
|
||||||
|
f"Fix the export section as the message says - see {EXPORT_HOW_TO}",
|
||||||
|
)
|
||||||
|
if targets is None:
|
||||||
|
return Check("export", "OK", "absent - no export section in the settings")
|
||||||
|
parts = [
|
||||||
|
f"{t.name} ({t.handler}, {t.path}{', after publish' if t.after_publish else ''})"
|
||||||
|
for t in targets
|
||||||
|
]
|
||||||
|
return Check("export", "OK", f"{len(targets)} target(s): " + "; ".join(parts))
|
||||||
|
|
||||||
|
|
||||||
def check_upload_intake() -> Check:
|
def check_upload_intake() -> Check:
|
||||||
"""Which intake channels this checkout has, how many submissions wait in
|
"""Which intake channels this checkout has, how many submissions wait in
|
||||||
the quarantine, and whether the quarantine of earlier versions
|
the quarantine, and whether the quarantine of earlier versions
|
||||||
@@ -1036,6 +1058,7 @@ def run_doctor() -> list[Check]:
|
|||||||
check_sync_folder(),
|
check_sync_folder(),
|
||||||
check_backup(),
|
check_backup(),
|
||||||
check_upload_intake(),
|
check_upload_intake(),
|
||||||
|
check_export(),
|
||||||
check_tasks_provider(),
|
check_tasks_provider(),
|
||||||
*check_settings(),
|
*check_settings(),
|
||||||
check_site_handlers(),
|
check_site_handlers(),
|
||||||
@@ -1095,6 +1118,9 @@ def run_doctor() -> list[Check]:
|
|||||||
"Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not "
|
"Configured, a `WARN` when `<target>/git` is not a bare repository, `origin` does not "
|
||||||
"push there, or - with `machine_data` true - no machine-data backup exists yet; "
|
"push there, or - with `machine_data` true - no machine-data backup exists yet; "
|
||||||
"otherwise `OK` with the age of the last machine-data backup.",
|
"otherwise `OK` with the age of the last machine-data backup.",
|
||||||
|
"Page export (`export`): the `export` section absent is `OK`, malformed is a `FAIL`; "
|
||||||
|
"otherwise `OK` listing each target with its handler, folder and whether it runs after "
|
||||||
|
"`publish`.",
|
||||||
"Intake channels (`upload-intake`): the channels of the `intake` section with their "
|
"Intake channels (`upload-intake`): the channels of the `intake` section with their "
|
||||||
"handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no "
|
"handler, and how many submissions wait in `quarantine/`. Absent is `OK` and means no "
|
||||||
"channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an "
|
"channel - no `submit` tool, nothing to collect; malformed is a `FAIL`, and so is an "
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
"""`wikitool export guidelines` - the instance's guideline pages as one
|
"""`wikitool export guidelines` - the instance's guideline pages as one
|
||||||
generated `GUIDELINES.md`, printed, or written into the captured repositories
|
generated `GUIDELINES.md`, printed, or written into the captured repositories
|
||||||
behind the Guideline Push Gate.
|
behind the Guideline Push Gate - and `wikitool export run`, the derived reading
|
||||||
|
copy of `kb/` in the folders the `export` section configures
|
||||||
|
(`chemenu/page_export.py`; no gate, since a target is a folder the user named).
|
||||||
|
|
||||||
The core - selection, rendering, the header - is `chemenu/guideline_export.py`;
|
The core - selection, rendering, the header - is `chemenu/guideline_export.py`;
|
||||||
the git write is `repo_capture.commit_file`/`push_commit`. This module decides
|
the git write is `repo_capture.commit_file`/`push_commit`. This module decides
|
||||||
@@ -42,12 +44,12 @@ from typing import Optional
|
|||||||
import typer
|
import typer
|
||||||
from rich.markup import escape
|
from rich.markup import escape
|
||||||
|
|
||||||
from chemenu import cli_contract, config, guideline_export, repo_capture
|
from chemenu import cli_contract, config, guideline_export, page_export, repo_capture
|
||||||
from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path
|
from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path
|
||||||
from chemenu.errors import BackendError, ValidationError
|
from chemenu.errors import BackendError, ValidationError
|
||||||
from chemenu.search import filters
|
from chemenu.search import filters
|
||||||
|
|
||||||
app = typer.Typer(help="Generate files from the wiki for other repositories.")
|
app = typer.Typer(help="Generate files from the wiki for other repositories and folders.")
|
||||||
|
|
||||||
GATE = "guideline-push"
|
GATE = "guideline-push"
|
||||||
|
|
||||||
@@ -427,3 +429,127 @@ def export_guidelines_command(
|
|||||||
"written; run the command again for the rest."
|
"written; run the command again for the rest."
|
||||||
)
|
)
|
||||||
raise typer.Exit(code=1)
|
raise typer.Exit(code=1)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command("run")
|
||||||
|
@cli_contract.record(cli_contract.CommandRecord(
|
||||||
|
path="export run",
|
||||||
|
summary="Write the derived, read-only reading copy of `kb/` into the folders the `export` "
|
||||||
|
"section configures - the selected pages as Markdown, or converted by a program of this "
|
||||||
|
"checkout.",
|
||||||
|
synopsis=(
|
||||||
|
cli_contract.Variant(
|
||||||
|
usage="export run [<target> ...]",
|
||||||
|
notes="Every configured target, or the named ones. Needs an `export` section in "
|
||||||
|
f"`{config.LOCAL_SETTINGS_FILENAME}`; the user writes it by hand.",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
properties=cli_contract.Properties(
|
||||||
|
effect=cli_contract.Effect.WRITE,
|
||||||
|
idempotent=cli_contract.Idempotent.YES,
|
||||||
|
atomic="Per file within a target (a temporary file in the target, then a move over its "
|
||||||
|
"place), the manifest last; targets are independent, so a failed target does not undo "
|
||||||
|
"another",
|
||||||
|
budget=cli_contract.Budget.COUNTED,
|
||||||
|
),
|
||||||
|
notes=(
|
||||||
|
"An export is derived: regenerated from `kb/` on every run, never edited, never read back "
|
||||||
|
"into the wiki. Each target names its pages with an allowlist of `search --field` "
|
||||||
|
"predicates (`include`: a list of selections, AND within one, OR across them), a handler "
|
||||||
|
"(`markdown`, or `command` for the operator's converter) and the folder (`path`, outside "
|
||||||
|
"the instance). Nothing is exported that no selection names.",
|
||||||
|
"Every Markdown file starts with a frontmatter block whose `wikitool_export` line says "
|
||||||
|
"`kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit "
|
||||||
|
"touching a selected page>`, followed by `title`, `type`, `summary`, `modified` where the "
|
||||||
|
"page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, "
|
||||||
|
"`full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A "
|
||||||
|
"wikilink to an exported page becomes a relative Markdown link (`links: relative`, the "
|
||||||
|
"default) or plain text (`links: text`); a link to a page that is not exported is always "
|
||||||
|
"plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: "
|
||||||
|
"the same pages give the same bytes.",
|
||||||
|
"Handler `command`: the Markdown is rendered into a staging directory, the program of "
|
||||||
|
"`export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives "
|
||||||
|
"the Markdown directory and an empty output directory as its last two arguments. On exit "
|
||||||
|
"0 the regular files in the output directory (recursively) become the target's content; "
|
||||||
|
"any other end leaves the target unchanged and prints the end of its stderr. The staging "
|
||||||
|
"directory is always removed.",
|
||||||
|
"The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: "
|
||||||
|
"the files an earlier run wrote, each with its sha256. A run removes only manifest files "
|
||||||
|
"that are no longer exported (and folders it created, once empty), rewrites a manifest "
|
||||||
|
"file that is missing, leaves every other file byte for byte as it was, and writes "
|
||||||
|
"nothing - not even a modification time - for a file that did not change. A run that "
|
||||||
|
"changes nothing writes nothing at all.",
|
||||||
|
"Refuses a target, before writing anything into it: a non-empty folder with no manifest "
|
||||||
|
"(it is somebody's, not an export), a manifest file that was changed in the folder since "
|
||||||
|
"the last run (naming it), a foreign file where an exported one belongs, an empty "
|
||||||
|
"selection (it would delete the earlier export), an unknown field in a predicate. A "
|
||||||
|
"missing folder is created.",
|
||||||
|
"Refuses everything, before any target: uncommitted changes in `kb/` or `types/` "
|
||||||
|
"(untracked files included), no git checkout, a page under `kb/` with unreadable "
|
||||||
|
"frontmatter, an unknown target name, a broken `export` section.",
|
||||||
|
"Output per target: `new` (with the page titles), `changed`, `removed`, the count "
|
||||||
|
"`unchanged`, or `failed` with the reason. A page that newly enters a selection is "
|
||||||
|
"reported as new, not gated. Exit 0 when every target ran; exit 1 when any failed - the "
|
||||||
|
"others were still served.",
|
||||||
|
"A target with `after_publish: true` is also refreshed by every `publish` that "
|
||||||
|
"committed, `--no-push` included; a failure there is a warning, the commit stands, and "
|
||||||
|
"this command is the repeat.",
|
||||||
|
"`raw accept` refuses a file whose sha256 a target's manifest lists, and one that carries "
|
||||||
|
"the marker: an export never goes back into `raw/`.",
|
||||||
|
),
|
||||||
|
failures=(
|
||||||
|
cli_contract.Failure(
|
||||||
|
cause="No `export` section, an unknown target name, or a broken section (a predicate "
|
||||||
|
"that does not parse, a path inside the instance or in the way of another target, an "
|
||||||
|
"`intake` folder or the backup target)",
|
||||||
|
reaction="Show the message to the user; the section is theirs to write by hand, never "
|
||||||
|
"yours",
|
||||||
|
),
|
||||||
|
cli_contract.Failure(
|
||||||
|
cause="`kb/` or `types/` has uncommitted changes, the instance is not a git checkout, "
|
||||||
|
"or a page under `kb/` has unreadable frontmatter",
|
||||||
|
reaction="Fix the named pages, or publish the changes first, then retry once",
|
||||||
|
),
|
||||||
|
cli_contract.Failure(
|
||||||
|
cause="A target failed: a foreign folder, a file changed in the target, an empty "
|
||||||
|
"selection, an unknown field, a converter that did not exit 0",
|
||||||
|
reaction="Show the target's reason to the user; the other targets were still served. "
|
||||||
|
"Never delete or edit files in a target folder to make a run pass",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
examples=(
|
||||||
|
"tools/wikitool export run",
|
||||||
|
"tools/wikitool export run work-notes",
|
||||||
|
),
|
||||||
|
never=(
|
||||||
|
"Never edit a file in a target folder to change an export - the next run refuses or "
|
||||||
|
"overwrites it; change the page, or the target's `include`, instead.",
|
||||||
|
"Never put an export folder into `incoming/` or `raw/`.",
|
||||||
|
),
|
||||||
|
see_also=(
|
||||||
|
"`wikitool search` - the same predicates, to preview a selection",
|
||||||
|
"`wikitool backup run` - the same kind of folder, for this machine's own files",
|
||||||
|
"`docs/how-to/export-to-a-synced-folder.md`",
|
||||||
|
),
|
||||||
|
))
|
||||||
|
def export_run_command(
|
||||||
|
names: Optional[list[str]] = typer.Argument(
|
||||||
|
None, metavar="[TARGET]...", help="Targets to run; default: every configured target."
|
||||||
|
),
|
||||||
|
):
|
||||||
|
"""Write the reading copy of kb/ into the configured export folders."""
|
||||||
|
try:
|
||||||
|
targets = page_export.choose(page_export.require_config(), names or [])
|
||||||
|
results = page_export.run(targets)
|
||||||
|
except ValidationError as exc:
|
||||||
|
fail(escape(str(exc)))
|
||||||
|
return
|
||||||
|
failed = 0
|
||||||
|
for result in results:
|
||||||
|
for line in page_export.report_lines(result):
|
||||||
|
typer.echo(line)
|
||||||
|
failed += not result.ok
|
||||||
|
if failed:
|
||||||
|
console.print(f"[bold red]ERROR[/bold red] {failed} of {len(results)} target(s) failed - the "
|
||||||
|
"others were served; fix the reason above and run again.")
|
||||||
|
raise typer.Exit(code=1)
|
||||||
@@ -180,6 +180,52 @@ def read_backup_config():
|
|||||||
fail(str(exc))
|
fail(str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
# --- Export after publish -------------------------------------------------------
|
||||||
|
#
|
||||||
|
# A target of the `export` section with `after_publish` is refreshed once a
|
||||||
|
# commit exists - `--no-push` included, since the export reads the committed
|
||||||
|
# state of `kb/`, not what reached the remote. The commit has happened by then,
|
||||||
|
# so a failing export is a WARN and exit 0, for the same reason as the backup
|
||||||
|
# above: `export run` repeats it, another `publish` cannot.
|
||||||
|
|
||||||
|
|
||||||
|
def read_export_targets():
|
||||||
|
"""The targets to refresh after this publish; exit 1 for a broken `export`
|
||||||
|
section when some target asks for the refresh."""
|
||||||
|
from chemenu import page_export
|
||||||
|
from chemenu.errors import ChemenuError
|
||||||
|
|
||||||
|
try:
|
||||||
|
return page_export.after_publish_targets()
|
||||||
|
except ChemenuError as exc:
|
||||||
|
fail(str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
def export_after_publish(targets) -> None:
|
||||||
|
from rich.markup import escape
|
||||||
|
|
||||||
|
from chemenu import page_export
|
||||||
|
from chemenu.errors import ChemenuError
|
||||||
|
|
||||||
|
try:
|
||||||
|
results = page_export.run(targets)
|
||||||
|
except (ChemenuError, OSError) as exc:
|
||||||
|
console.print(
|
||||||
|
f"[yellow]WARN[/yellow] The export failed: {escape(str(exc))}\n"
|
||||||
|
" The commit stands. Show this to the user; once the cause is fixed, "
|
||||||
|
"`tools/wikitool export run` repeats it - another `publish` does not."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
for result in results:
|
||||||
|
for line in page_export.report_lines(result):
|
||||||
|
if result.ok:
|
||||||
|
typer.echo(line)
|
||||||
|
else:
|
||||||
|
console.print(f"[yellow]WARN[/yellow] {escape(line)}")
|
||||||
|
if not result.ok:
|
||||||
|
console.print(" The commit stands; `tools/wikitool export run` repeats this target.")
|
||||||
|
|
||||||
|
|
||||||
def back_up_machine_data(cfg) -> None:
|
def back_up_machine_data(cfg) -> None:
|
||||||
from rich.markup import escape
|
from rich.markup import escape
|
||||||
|
|
||||||
@@ -1614,6 +1660,12 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
|||||||
"stand. No backup with `--no-push`, after a failed push, or when there was nothing to "
|
"stand. No backup with `--no-push`, after a failed push, or when there was nothing to "
|
||||||
"push. A malformed `backup` section stops the call with exit 1 before anything is "
|
"push. A malformed `backup` section stops the call with exit 1 before anything is "
|
||||||
"committed, `--no-push` included.",
|
"committed, `--no-push` included.",
|
||||||
|
"Export after publish: every target of the `export` section with `after_publish: true` "
|
||||||
|
"is refreshed once a commit was made - the same step as `export run`, `--no-push` "
|
||||||
|
"included, since the export reads the committed `kb/`. A failure there prints a `WARN` "
|
||||||
|
"line with the reason and still exits 0, since the commit stands; `export run` repeats "
|
||||||
|
"it. A malformed `export` section stops the call with exit 1 before anything is "
|
||||||
|
"committed, but only when some target sets `after_publish`.",
|
||||||
"After a successful commit or push whose changed files include `tools/`, `types/`, "
|
"After a successful commit or push whose changed files include `tools/`, `types/`, "
|
||||||
"`instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder "
|
"`instructions/`, `AGENTS.md` or a path ending in `CONTRACT.md`, prints one reminder "
|
||||||
"line: the phase past this point (an issue-body rewrite, `docs/` staleness, a "
|
"line: the phase past this point (an issue-body rewrite, `docs/` staleness, a "
|
||||||
@@ -1651,7 +1703,8 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
|||||||
),
|
),
|
||||||
cli_contract.Failure(
|
cli_contract.Failure(
|
||||||
cause="The settings files do not pass their check, `remotes.allowed_push_urls` is "
|
cause="The settings files do not pass their check, `remotes.allowed_push_urls` is "
|
||||||
"not a list of strings, or the `backup` section is malformed; nothing was committed",
|
"not a list of strings, the `backup` section is malformed, or the `export` section is "
|
||||||
|
"malformed while a target sets `after_publish`; nothing was committed",
|
||||||
reaction="Show the error to the user and stop - a malformed section is not "
|
reaction="Show the error to the user and stop - a malformed section is not "
|
||||||
"permission, and fixing it is theirs to do",
|
"permission, and fixing it is theirs to do",
|
||||||
),
|
),
|
||||||
@@ -1662,6 +1715,13 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
|||||||
"fixed, `tools/wikitool backup run` repeats the backup",
|
"fixed, `tools/wikitool backup run` repeats the backup",
|
||||||
code=0,
|
code=0,
|
||||||
),
|
),
|
||||||
|
cli_contract.Failure(
|
||||||
|
cause="The export after the commit failed: a `WARN` line names the target and the "
|
||||||
|
"reason, and the commit stands",
|
||||||
|
reaction="Show the `WARN` line to the user. Do not publish again - once the cause is "
|
||||||
|
"fixed, `tools/wikitool export run` repeats the export",
|
||||||
|
code=0,
|
||||||
|
),
|
||||||
cli_contract.Failure(
|
cli_contract.Failure(
|
||||||
cause="The remotes settings file of an earlier version is still there",
|
cause="The remotes settings file of an earlier version is still there",
|
||||||
reaction="Show the output to the user verbatim and stop; `tools/wikitool config "
|
reaction="Show the output to the user verbatim and stop; `tools/wikitool config "
|
||||||
@@ -1710,6 +1770,7 @@ def commit_regenerated(remote: str, branch: str) -> None:
|
|||||||
"`instructions/gates.md` - the gate procedure",
|
"`instructions/gates.md` - the gate procedure",
|
||||||
"`instructions/setup-instance.md` - the first publish of a new instance",
|
"`instructions/setup-instance.md` - the first publish of a new instance",
|
||||||
"`wikitool backup run` - the machine-data backup on its own",
|
"`wikitool backup run` - the machine-data backup on its own",
|
||||||
|
"`wikitool export run` - the page export on its own",
|
||||||
),
|
),
|
||||||
))
|
))
|
||||||
def publish_command(
|
def publish_command(
|
||||||
@@ -1760,6 +1821,7 @@ def publish_command(
|
|||||||
# Read before anything is committed, `--no-push` included: a broken section
|
# Read before anything is committed, `--no-push` included: a broken section
|
||||||
# stops here rather than turning into a skipped backup after the push.
|
# stops here rather than turning into a skipped backup after the push.
|
||||||
backup_cfg = read_backup_config()
|
backup_cfg = read_backup_config()
|
||||||
|
export_targets = read_export_targets()
|
||||||
|
|
||||||
paths = list(path or [])
|
paths = list(path or [])
|
||||||
# What is staged and committed: `paths` plus any uncommitted generated file outside them.
|
# What is staged and committed: `paths` plus any uncommitted generated file outside them.
|
||||||
@@ -1932,5 +1994,7 @@ def publish_command(
|
|||||||
success(f"Published changes to {remote}/{branch}." if push else "Committed changes (not pushed).")
|
success(f"Published changes to {remote}/{branch}." if push else "Committed changes (not pushed).")
|
||||||
if push and backup_cfg is not None and backup_cfg.machine_data:
|
if push and backup_cfg is not None and backup_cfg.machine_data:
|
||||||
back_up_machine_data(backup_cfg)
|
back_up_machine_data(backup_cfg)
|
||||||
|
if export_targets:
|
||||||
|
export_after_publish(export_targets)
|
||||||
if touches_stack_machinery(changed_files):
|
if touches_stack_machinery(changed_files):
|
||||||
typer.echo(STACK_MACHINERY_NOTE)
|
typer.echo(STACK_MACHINERY_NOTE)
|
||||||
@@ -11,9 +11,10 @@ import re
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
|
import typer
|
||||||
from rich.markup import escape
|
from rich.markup import escape
|
||||||
|
|
||||||
from chemenu import config, filters, guideline_export, repo_capture
|
from chemenu import backup, config, filters, guideline_export, page_export, repo_capture
|
||||||
from chemenu.commands._util import fail, path_budget_problem_for, rel_path
|
from chemenu.commands._util import fail, path_budget_problem_for, rel_path
|
||||||
from chemenu.errors import ValidationError
|
from chemenu.errors import ValidationError
|
||||||
from chemenu.provenance import citing_pages, source_pages_by_raw_file
|
from chemenu.provenance import citing_pages, source_pages_by_raw_file
|
||||||
@@ -155,16 +156,37 @@ def _check_files(resolved: list[Path]) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def _check_no_export(paths: list[Path]) -> None:
|
def _check_no_export(paths: list[Path]) -> None:
|
||||||
"""Refuse a file a wiki exported (`export guidelines`): it is generated
|
"""Refuse a file a wiki exported (`export guidelines`, `export run`): it is
|
||||||
from `kb/`, so taking it into `raw/` would feed the wiki its own output
|
generated from `kb/`, so taking it into `raw/` would feed the wiki its own
|
||||||
back as a source. `raw capture` already leaves such a file out; this
|
output back as a source. `raw capture` already leaves such a file out; this
|
||||||
closes the way in through `incoming/`."""
|
closes the way in through `incoming/`. The marker catches a file as written;
|
||||||
|
the digest of a configured target's manifest catches one whose marker a
|
||||||
|
converter dropped."""
|
||||||
exports = [p for p in paths if guideline_export.file_is_export(p)]
|
exports = [p for p in paths if guideline_export.file_is_export(p)]
|
||||||
if exports:
|
if exports:
|
||||||
listed = "\n".join(f" - {rel_path(p)}" for p in exports)
|
listed = "\n".join(f" - {rel_path(p)}" for p in exports)
|
||||||
raise _Refused(
|
raise _Refused(
|
||||||
"These files are a wiki's guideline export - their first line starts with "
|
"These files are a wiki's export (a guideline export or a page export) - they carry "
|
||||||
"<!-- wikitool:export. They are generated from kb/ and never go back into raw/:\n"
|
"the wikitool export marker. They are generated from kb/ and never go back into "
|
||||||
|
f"raw/:\n{listed}\n Remove them from incoming/ and accept again; nothing was moved."
|
||||||
|
)
|
||||||
|
digests, warnings = page_export.known_digests()
|
||||||
|
for warning in warnings:
|
||||||
|
typer.echo(f"WARN {warning}")
|
||||||
|
if not digests:
|
||||||
|
return
|
||||||
|
exported = []
|
||||||
|
for p in paths:
|
||||||
|
try:
|
||||||
|
if backup.file_digest(p)[1] in digests:
|
||||||
|
exported.append(p)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
if exported:
|
||||||
|
listed = "\n".join(f" - {rel_path(p)}" for p in exported)
|
||||||
|
raise _Refused(
|
||||||
|
"These files are byte for byte a file of a configured export target (see "
|
||||||
|
"`export run`). They are generated from kb/ and never go back into raw/:\n"
|
||||||
f"{listed}\n Remove them from incoming/ and accept again; nothing was moved."
|
f"{listed}\n Remove them from incoming/ and accept again; nothing was moved."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -668,8 +668,10 @@ def _warn_underived(resolved: list[Path]) -> None:
|
|||||||
),
|
),
|
||||||
cli_contract.Failure(
|
cli_contract.Failure(
|
||||||
cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` "
|
cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` "
|
||||||
"material - is a guideline export: its first line, after an optional BOM, starts with "
|
"material - is an export of this wiki: it carries the export marker (a first line "
|
||||||
"`<!-- wikitool:export`",
|
"starting with `<!-- wikitool:export`, or a `wikitool_export:` key in a leading "
|
||||||
|
"frontmatter block), or it is byte for byte a file in the manifest of a configured "
|
||||||
|
"`export` target",
|
||||||
reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes "
|
reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes "
|
||||||
"back into `raw/`. Remove it from `incoming/`; nothing was moved",
|
"back into `raw/`. Remove it from `incoming/`; nothing was moved",
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -121,7 +121,7 @@ def _write_bundle(target: Path, files: dict[str, bytes], manifest: repo_capture.
|
|||||||
"spans any number of directories; a glob without wildcards also takes everything below it "
|
"spans any number of directories; a glob without wildcards also takes everything below it "
|
||||||
"as a directory. Files keep their repository paths inside the bundle.",
|
"as a directory. Files keep their repository paths inside the bundle.",
|
||||||
"Never captured, each named in the output with its reason: a file whose first line starts "
|
"Never captured, each named in the output with its reason: a file whose first line starts "
|
||||||
"with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a "
|
"with `<!-- wikitool:export` or a leading frontmatter block with a `wikitool_export:` key, a symlink, a submodule, a file over 25 MiB, a path with a "
|
||||||
"segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.",
|
"segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.",
|
||||||
"`incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, "
|
"`incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, "
|
||||||
"`captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that "
|
"`captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that "
|
||||||
|
|||||||
@@ -813,11 +813,160 @@ BACKUP = Shape(
|
|||||||
"\"no backup\".",
|
"\"no backup\".",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
EXPORT_HANDLER_MARKDOWN = "markdown"
|
||||||
|
EXPORT_HANDLER_COMMAND = "command"
|
||||||
|
EXPORT_HANDLERS = (EXPORT_HANDLER_MARKDOWN, EXPORT_HANDLER_COMMAND)
|
||||||
|
EXPORT_FRONTMATTER_MODES = ("minimal", "summary", "full")
|
||||||
|
EXPORT_DEFAULT_FRONTMATTER = "summary"
|
||||||
|
EXPORT_LINK_MODES = ("relative", "text")
|
||||||
|
EXPORT_DEFAULT_LINKS = "relative"
|
||||||
|
# What each handler takes besides the keys every target has. A key outside
|
||||||
|
# both sets is an error, so a typo or a field of the other handler is not
|
||||||
|
# silently ignored.
|
||||||
|
_EXPORT_COMMON_KEYS = ("handler", "path", "include", "frontmatter", "links", "after_publish")
|
||||||
|
_EXPORT_HANDLER_KEYS = {
|
||||||
|
EXPORT_HANDLER_MARKDOWN: (),
|
||||||
|
EXPORT_HANDLER_COMMAND: ("command",),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _export_include_problem(where: str, include: Any) -> Optional[str]:
|
||||||
|
if not isinstance(include, list) or not include:
|
||||||
|
return f"{where}: `include` must be a non-empty array of selections"
|
||||||
|
for number, selection in enumerate(include, start=1):
|
||||||
|
if not isinstance(selection, list) or not selection:
|
||||||
|
return f"{where}: `include` selection {number} must be a non-empty array of predicates"
|
||||||
|
if not all(isinstance(predicate, str) and predicate.strip() for predicate in selection):
|
||||||
|
return f"{where}: `include` selection {number} holds an entry that is not a non-empty string"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _export_target_problem(name: str, target: Any) -> Optional[str]:
|
||||||
|
where = f"targets.{name}"
|
||||||
|
if not isinstance(target, dict):
|
||||||
|
return f"{where} must be an object"
|
||||||
|
handler = target.get("handler")
|
||||||
|
if handler not in EXPORT_HANDLERS:
|
||||||
|
return f"{where}: `handler` must be one of {', '.join(EXPORT_HANDLERS)}"
|
||||||
|
allowed = _EXPORT_COMMON_KEYS + _EXPORT_HANDLER_KEYS[handler]
|
||||||
|
stray = sorted(set(target) - set(allowed))
|
||||||
|
if stray:
|
||||||
|
return (f"{where}: {', '.join(f'`{key}`' for key in stray)} not taken by the handler "
|
||||||
|
f"{handler} (it takes {', '.join(f'`{key}`' for key in allowed)})")
|
||||||
|
for key in ("path", "include"):
|
||||||
|
if key not in target:
|
||||||
|
return f"{where}: `{key}` is required"
|
||||||
|
path = target["path"]
|
||||||
|
if not isinstance(path, str) or not path.strip():
|
||||||
|
return f"{where}: `path` must be a non-empty string"
|
||||||
|
problem = absolute_path_problem(path)
|
||||||
|
if problem:
|
||||||
|
return f"{where}: `path` {problem}"
|
||||||
|
problem = _export_include_problem(where, target["include"])
|
||||||
|
if problem:
|
||||||
|
return problem
|
||||||
|
for key, choices in (("frontmatter", EXPORT_FRONTMATTER_MODES), ("links", EXPORT_LINK_MODES)):
|
||||||
|
if key in target and target[key] not in choices:
|
||||||
|
return f"{where}: `{key}` must be one of {', '.join(choices)}"
|
||||||
|
if "after_publish" in target and not isinstance(target["after_publish"], bool):
|
||||||
|
return f"{where}: `after_publish` must be true or false"
|
||||||
|
if handler == EXPORT_HANDLER_COMMAND:
|
||||||
|
ref = target.get("command")
|
||||||
|
if not isinstance(ref, str) or not _FETCHER_NAME.fullmatch(ref):
|
||||||
|
return (f"{where}: `command` is required and names an entry of `commands` - lower-case "
|
||||||
|
"letters, digits and hyphens")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def export_targets_problem(value: dict) -> Optional[str]:
|
||||||
|
if not value:
|
||||||
|
return "must hold at least one target"
|
||||||
|
for name, target in value.items():
|
||||||
|
if not _FETCHER_NAME.fullmatch(name):
|
||||||
|
return (f"has the target name {name!r} - a target's name is lower-case letters, digits "
|
||||||
|
"and hyphens")
|
||||||
|
problem = _export_target_problem(name, target)
|
||||||
|
if problem:
|
||||||
|
return problem
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def export_section_problem(values: dict) -> Optional[str]:
|
||||||
|
commands = values.get("commands") or {}
|
||||||
|
for name, target in values["targets"].items():
|
||||||
|
if target["handler"] == EXPORT_HANDLER_COMMAND and target["command"] not in commands:
|
||||||
|
return (f"targets.{name}.command names {target['command']!r}, which `commands` does not "
|
||||||
|
"define")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
EXPORT = Shape(
|
||||||
|
key="export",
|
||||||
|
title="Exports to folders",
|
||||||
|
purpose="Declares the folders `export run` writes a derived, read-only reading copy of `kb/` "
|
||||||
|
"into: per target the pages it takes (an allowlist of `search` predicates), how they are "
|
||||||
|
"written, and whether `publish` refreshes it. An export is regenerated from `kb/` and never "
|
||||||
|
"edited; what a person changes in the folder is overwritten or refused, never merged back.",
|
||||||
|
fields=(
|
||||||
|
Field(
|
||||||
|
"targets", "object", "Target name (lower-case letters, digits and hyphens) to target; "
|
||||||
|
"the name is the argument of `export run`. Every target has `handler` (`markdown`: the "
|
||||||
|
"pages as Markdown files; `command`: the same Markdown run through a converter of this "
|
||||||
|
"checkout), `path` (an absolute path outside the instance; it must not lie in or "
|
||||||
|
"contain another target, an `intake` folder or `backup.target`) and `include` (an "
|
||||||
|
"array of selections, each an array of `search --field` predicates: all predicates of "
|
||||||
|
"a selection must hold, and a page that fits any selection is exported; there is no "
|
||||||
|
"\"everything\"). Optional: `frontmatter` (`minimal`: only the export marker; "
|
||||||
|
f"`summary`, the default: also `title`, `type`, `summary` and `modified` where the page "
|
||||||
|
"has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a "
|
||||||
|
"wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a "
|
||||||
|
"link to a page that is not exported is always plain text) and `after_publish` "
|
||||||
|
"(default false: refresh this target after every `publish` that committed). The "
|
||||||
|
"handler `command` takes `command` (required: the name of an entry of `commands`). A "
|
||||||
|
"field the handler does not take is an error.",
|
||||||
|
required=True, json_type="object", coerce=object_value, check=export_targets_problem,
|
||||||
|
tier=CHECKOUT, gate=True,
|
||||||
|
),
|
||||||
|
Field(
|
||||||
|
"commands", "object", "Command name to command: an array of the program and its "
|
||||||
|
"arguments, run by the `command` handler without a shell, with stdin closed and a "
|
||||||
|
"timeout of 300 s. The last two arguments appended are the directory holding the "
|
||||||
|
"Markdown and an empty output directory. Exit 0 makes the regular files in the output "
|
||||||
|
"directory the target's content; anything else leaves the target unchanged and prints "
|
||||||
|
"the end of the program's stderr.",
|
||||||
|
default={}, json_type="object", coerce=object_value, check=intake_commands_problem,
|
||||||
|
tier=CHECKOUT, names_command=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
examples=({
|
||||||
|
"targets": {
|
||||||
|
"work-notes": {
|
||||||
|
"handler": "markdown",
|
||||||
|
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Lesefassung",
|
||||||
|
"include": [["kind=system"], ["tags=shared", "!draft"]],
|
||||||
|
"frontmatter": "summary", "links": "relative", "after_publish": True,
|
||||||
|
},
|
||||||
|
"as-word": {
|
||||||
|
"handler": "command", "command": "pandoc-docx",
|
||||||
|
"path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-Word",
|
||||||
|
"include": [["kind=system"]],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"commands": {"pandoc-docx": ["pwsh", "-NoProfile", "-File", "C:\\tools\\pandoc-docx.ps1"]},
|
||||||
|
},),
|
||||||
|
absent="No export: `export run` stops with exit 1 naming the section, `publish` exports nothing "
|
||||||
|
"and `doctor` reports `OK`.",
|
||||||
|
malformed="An error. `export run` stops with exit 1 and `doctor` reports `FAIL`; `publish` stops "
|
||||||
|
"with exit 1 before it commits anything, but only when some target sets `after_publish`. A "
|
||||||
|
"broken section is never read as \"no export\".",
|
||||||
|
section_check=export_section_problem,
|
||||||
|
)
|
||||||
|
|
||||||
# The sections of the tiered settings files, `wikitool.json` and
|
# The sections of the tiered settings files, `wikitool.json` and
|
||||||
# `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
|
# `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
|
||||||
# stand in a section of its own.
|
# stand in a section of its own.
|
||||||
SETTINGS_SECTIONS: tuple[Shape, ...] = (
|
SETTINGS_SECTIONS: tuple[Shape, ...] = (
|
||||||
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP,
|
TELEMETRY, REMOTES, INTAKE, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH, BACKUP, EXPORT,
|
||||||
)
|
)
|
||||||
|
|
||||||
# The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
|
# The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ again when it comes back.
|
|||||||
side - writing the file into a captured repository's branch - lives beside
|
side - writing the file into a captured repository's branch - lives beside
|
||||||
`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under
|
`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under
|
||||||
`chemenu.commands`, and nothing here imports `repo_capture`, which imports
|
`chemenu.commands`, and nothing here imports `repo_capture`, which imports
|
||||||
`EXPORT_MARKER` from here: that is the direction the dependency runs.
|
`is_export` from here: that is the direction the dependency runs.
|
||||||
|
|
||||||
Three properties carry the design, and each has a test:
|
Three properties carry the design, and each has a test:
|
||||||
|
|
||||||
@@ -38,11 +38,19 @@ from chemenu.search import filters
|
|||||||
from chemenu.search.service import load_pages_by_path, unreadable_pages
|
from chemenu.search.service import load_pages_by_path, unreadable_pages
|
||||||
from chemenu.search.types import Predicate
|
from chemenu.search.types import Predicate
|
||||||
|
|
||||||
# The first bytes of every file this stack generates for another repository.
|
# What marks a file this stack generated for somewhere else - the one place
|
||||||
# Defined once, here: `repo_capture` excludes a file carrying it from a
|
# the definition of an export lives. `repo_capture` excludes a file that
|
||||||
# capture, `raw accept` refuses one, and the export writes it.
|
# `is_export` recognises from a capture, `raw accept` refuses one, and the
|
||||||
EXPORT_MARKER = b"<!-- wikitool:export"
|
# exports write it: `export guidelines` as the first-line comment, `export run`
|
||||||
|
# as a frontmatter key. Either form is an export; nothing else is.
|
||||||
|
COMMENT_MARKER = b"<!-- wikitool:export"
|
||||||
|
FRONTMATTER_KEY = b"wikitool_export:"
|
||||||
|
FRONTMATTER_FENCE = b"---"
|
||||||
|
# How much of a file `is_export` looks at to find the frontmatter's closing
|
||||||
|
# fence; a frontmatter that does not close within it is not an export.
|
||||||
|
SCAN_BYTES = 64 * 1024
|
||||||
GUIDELINES_KIND = "guidelines"
|
GUIDELINES_KIND = "guidelines"
|
||||||
|
PAGES_KIND = "pages"
|
||||||
GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}"
|
GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}"
|
||||||
TARGET_PATH = "GUIDELINES.md"
|
TARGET_PATH = "GUIDELINES.md"
|
||||||
LOCAL_INSTANCE = "local"
|
LOCAL_INSTANCE = "local"
|
||||||
@@ -57,16 +65,42 @@ _SCP_USERINFO = re.compile(r"^[^/@:]+@(?=[^/:]+:)")
|
|||||||
# --- recognising an export ------------------------------------------------------
|
# --- recognising an export ------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _without_bom(data: bytes) -> bytes:
|
||||||
|
return data[len(_BOM):] if data.startswith(_BOM) else data
|
||||||
|
|
||||||
|
|
||||||
def _first_line(data: bytes) -> bytes:
|
def _first_line(data: bytes) -> bytes:
|
||||||
data = data[len(_BOM):] if data.startswith(_BOM) else data
|
return _without_bom(data).split(b"\n", 1)[0]
|
||||||
return data.split(b"\n", 1)[0]
|
|
||||||
|
|
||||||
|
|
||||||
def is_export(data: bytes) -> bool:
|
def is_export(data: bytes) -> bool:
|
||||||
"""Whether `data` is a file this stack exported: its first line, after an
|
"""Whether `data` is a file this stack exported, in either form:
|
||||||
optional BOM, starts with `EXPORT_MARKER`. Reads no more than the first
|
|
||||||
line, so a caller may pass only a file's first bytes."""
|
1. its first line, after an optional BOM, starts with `COMMENT_MARKER`
|
||||||
return _first_line(data).startswith(EXPORT_MARKER)
|
(`export guidelines`);
|
||||||
|
2. its first line is `---` and a line starting with `FRONTMATTER_KEY`
|
||||||
|
stands before the closing `---` line (`export run`).
|
||||||
|
|
||||||
|
Form 2 is a line scan, not a YAML parse, so a frontmatter that is no longer
|
||||||
|
valid YAML is still recognised - the marker, not the YAML, is what says
|
||||||
|
where a file came from. Looks at no more than `SCAN_BYTES`, so a caller may
|
||||||
|
pass only a file's first bytes: a frontmatter with no closing line in that
|
||||||
|
range is not an export, and neither is a key that only follows it."""
|
||||||
|
data = _without_bom(data)
|
||||||
|
head = data[:SCAN_BYTES]
|
||||||
|
lines = head.split(b"\n")
|
||||||
|
if len(head) == SCAN_BYTES and len(data) > SCAN_BYTES:
|
||||||
|
lines.pop() # a line the cut may have shortened
|
||||||
|
if lines[0].startswith(COMMENT_MARKER):
|
||||||
|
return True
|
||||||
|
if lines[0].rstrip() != FRONTMATTER_FENCE:
|
||||||
|
return False
|
||||||
|
marked = False
|
||||||
|
for line in lines[1:]:
|
||||||
|
if line.rstrip() == FRONTMATTER_FENCE:
|
||||||
|
return marked
|
||||||
|
marked = marked or line.startswith(FRONTMATTER_KEY)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
|
def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
|
||||||
@@ -80,9 +114,9 @@ def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
|
|||||||
|
|
||||||
|
|
||||||
def file_is_export(path: Path) -> bool:
|
def file_is_export(path: Path) -> bool:
|
||||||
"""`is_export` for a file on disk, reading only its first line's worth."""
|
"""`is_export` for a file on disk, reading no more than it looks at."""
|
||||||
with open(path, "rb") as handle:
|
with open(path, "rb") as handle:
|
||||||
return is_export(handle.read(len(_BOM) + len(EXPORT_MARKER)))
|
return is_export(handle.read(len(_BOM) + SCAN_BYTES + 1))
|
||||||
|
|
||||||
|
|
||||||
# --- the instance it comes from -----------------------------------------------
|
# --- the instance it comes from -----------------------------------------------
|
||||||
|
|||||||
@@ -0,0 +1,631 @@
|
|||||||
|
"""The page export, with no CLI attached: a derived, read-only reading copy of
|
||||||
|
`kb/` per configured target folder.
|
||||||
|
|
||||||
|
`wikitool export run` is the terminal adapter over this module, `publish`
|
||||||
|
calls `run` for the targets that ask for it, and `raw accept` asks
|
||||||
|
`known_digests` whether a file is one this module wrote. Nothing here imports
|
||||||
|
`typer` or anything under `chemenu.commands`.
|
||||||
|
|
||||||
|
An export is never a working copy: it is regenerated from `kb/` and what a
|
||||||
|
person changes in the folder is overwritten or refused, never merged back. The
|
||||||
|
properties that carry that, each with a test:
|
||||||
|
|
||||||
|
- **Deterministic.** The output depends on the selected pages and the commit
|
||||||
|
that last touched one of them - no timestamp, no `HEAD` - so a run on an
|
||||||
|
unchanged corpus writes nothing, not even a file's modification time.
|
||||||
|
- **Only its own files are touched.** What an earlier run wrote is recorded in
|
||||||
|
a manifest in the target (`.wikitool-export.json`, with a digest per file).
|
||||||
|
A run removes only manifest files that are no longer exported, leaves every
|
||||||
|
other file byte for byte as it found it, and stops on a manifest file a
|
||||||
|
person has changed rather than overwrite the change. A non-empty folder
|
||||||
|
without a manifest is refused: it is somebody's folder, not an export.
|
||||||
|
- **Nothing is deleted on a doubtful selection.** Zero selected pages is an
|
||||||
|
error before anything is touched.
|
||||||
|
- **Atomic per file.** Each file is written to a temporary file in the target
|
||||||
|
and moved over its place; the manifest comes last.
|
||||||
|
- **One definition of an export.** The marker every file carries, and the test
|
||||||
|
for it, live in `chemenu.guideline_export`.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import posixpath
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Optional, Sequence
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from chemenu import backup, config, config_shapes, guideline_export, settings
|
||||||
|
from chemenu.errors import ValidationError
|
||||||
|
from chemenu.frontmatter_io import dump_frontmatter
|
||||||
|
from chemenu.kb_scan import LINK_RE, normalize_link_target
|
||||||
|
from chemenu.markdown_code import strip_code_spans
|
||||||
|
from chemenu.page import Page
|
||||||
|
from chemenu.search import filters
|
||||||
|
from chemenu.search.service import load_pages_by_path, unreadable_pages
|
||||||
|
from chemenu.search.types import Predicate
|
||||||
|
|
||||||
|
MANIFEST_NAME = ".wikitool-export.json"
|
||||||
|
MANIFEST_SCHEMA = 1
|
||||||
|
COMMAND_TIMEOUT_SECONDS = 300
|
||||||
|
_STDERR_TAIL_CHARS = 2000
|
||||||
|
SUMMARY_KEYS = ("type", "summary", "modified")
|
||||||
|
|
||||||
|
|
||||||
|
class ExportError(ValidationError):
|
||||||
|
"""An export step refused or failed; exit 1 with the message."""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Configuration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Target:
|
||||||
|
name: str
|
||||||
|
handler: str
|
||||||
|
path: Path
|
||||||
|
include: tuple[tuple[Predicate, ...], ...]
|
||||||
|
frontmatter: str
|
||||||
|
links: str
|
||||||
|
after_publish: bool
|
||||||
|
command_name: str = ""
|
||||||
|
command: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
def _others_paths(root: Path) -> list[tuple[str, Path]]:
|
||||||
|
"""The folders other features of this checkout write into or read from -
|
||||||
|
the `intake` folders and the backup target - for the conflict check. A
|
||||||
|
section that is itself broken is that feature's own problem and is skipped
|
||||||
|
here: it is reported where it is read."""
|
||||||
|
from chemenu import upload
|
||||||
|
|
||||||
|
found: list[tuple[str, Path]] = []
|
||||||
|
try:
|
||||||
|
for name, channel in upload.read_channels(root).items():
|
||||||
|
if channel.path is not None:
|
||||||
|
found.append((f"the intake channel {name!r} ({channel.path})", channel.path))
|
||||||
|
except ValidationError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
backup_cfg = backup.read_config(root)
|
||||||
|
except ValidationError:
|
||||||
|
backup_cfg = None
|
||||||
|
if backup_cfg is not None:
|
||||||
|
found.append((f"backup.target ({backup_cfg.target})", backup_cfg.target))
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_include(name: str, include: list[list[str]], problems: list[str]) -> tuple[tuple[Predicate, ...], ...]:
|
||||||
|
selections = []
|
||||||
|
for number, selection in enumerate(include, start=1):
|
||||||
|
parsed = []
|
||||||
|
for raw in selection:
|
||||||
|
try:
|
||||||
|
parsed.append(filters.parse_predicate(raw))
|
||||||
|
except ValidationError as exc:
|
||||||
|
problems.append(f"export.targets.{name}.include selection {number}: {exc}")
|
||||||
|
selections.append(tuple(parsed))
|
||||||
|
return tuple(selections)
|
||||||
|
|
||||||
|
|
||||||
|
def read_config(root: "Path | str | None" = None) -> Optional[tuple[Target, ...]]:
|
||||||
|
"""The targets of the `export` section in the order the file lists them, or
|
||||||
|
None when the settings carry none. Raises `ValidationError` for a section
|
||||||
|
that is broken - including what the record cannot judge without the
|
||||||
|
instance and the rest of the settings: a predicate that does not parse, a
|
||||||
|
path on this machine that is not absolute, inside the instance, or in the
|
||||||
|
way of another target, an `intake` folder or the backup target."""
|
||||||
|
root = Path(config.ROOT if root is None else root)
|
||||||
|
key = config_shapes.EXPORT.key
|
||||||
|
section = settings.load(root, only=(key,)).section(key)
|
||||||
|
if not section.configured:
|
||||||
|
return None
|
||||||
|
problems: list[str] = []
|
||||||
|
targets: list[Target] = []
|
||||||
|
commands = section["commands"]
|
||||||
|
real_root = backup.real_path(root)
|
||||||
|
for name, raw in section["targets"].items():
|
||||||
|
path = Path(raw["path"])
|
||||||
|
if not os.path.isabs(raw["path"]):
|
||||||
|
problems.append(f"export.targets.{name}.path {raw['path']!r} is not an absolute path on this machine")
|
||||||
|
continue
|
||||||
|
real = backup.real_path(path)
|
||||||
|
if backup.is_under(real, real_root) or backup.is_under(real_root, real):
|
||||||
|
problems.append(f"export.targets.{name}.path {path} and the instance lie in one another - "
|
||||||
|
"an export belongs outside it")
|
||||||
|
continue
|
||||||
|
include = _parse_include(name, raw["include"], problems)
|
||||||
|
command_name = raw.get("command", "")
|
||||||
|
targets.append(Target(
|
||||||
|
name=name, handler=raw["handler"], path=path, include=include,
|
||||||
|
frontmatter=raw.get("frontmatter", config_shapes.EXPORT_DEFAULT_FRONTMATTER),
|
||||||
|
links=raw.get("links", config_shapes.EXPORT_DEFAULT_LINKS),
|
||||||
|
after_publish=raw.get("after_publish", False),
|
||||||
|
command_name=command_name, command=tuple(commands.get(command_name, ())),
|
||||||
|
))
|
||||||
|
if not problems:
|
||||||
|
placed = [(f"export.targets.{t.name}", backup.real_path(t.path)) for t in targets]
|
||||||
|
others = [(label, backup.real_path(path)) for label, path in _others_paths(root)]
|
||||||
|
for index, (label, real) in enumerate(placed):
|
||||||
|
for other_label, other in placed[index + 1:] + others:
|
||||||
|
if backup.is_under(real, other) or backup.is_under(other, real):
|
||||||
|
problems.append(f"{label} and {other_label} lie in one another - "
|
||||||
|
"every export folder is its own")
|
||||||
|
if problems:
|
||||||
|
raise ExportError("; ".join(problems))
|
||||||
|
return tuple(targets)
|
||||||
|
|
||||||
|
|
||||||
|
def require_config(root: "Path | str | None" = None) -> tuple[Target, ...]:
|
||||||
|
found = read_config(root)
|
||||||
|
if found is None:
|
||||||
|
raise ExportError(
|
||||||
|
f"no export section in {config.LOCAL_SETTINGS_FILENAME} - the user adds it by hand "
|
||||||
|
"(docs/how-to/export-to-a-synced-folder.md)"
|
||||||
|
)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_for_after_publish(root: Path) -> bool:
|
||||||
|
"""Whether the raw `export` section has a target that sets `after_publish`,
|
||||||
|
read without judging the rest of it - so a section too broken to read as a
|
||||||
|
whole can still tell `publish` that it matters."""
|
||||||
|
try:
|
||||||
|
data = settings.read_file(settings.local_path(root)) or {}
|
||||||
|
except ValidationError:
|
||||||
|
return False
|
||||||
|
section = data.get(config_shapes.EXPORT.key)
|
||||||
|
targets = section.get("targets") if isinstance(section, dict) else None
|
||||||
|
if not isinstance(targets, dict):
|
||||||
|
return False
|
||||||
|
return any(isinstance(t, dict) and t.get("after_publish") for t in targets.values())
|
||||||
|
|
||||||
|
|
||||||
|
def after_publish_targets(root: "Path | str | None" = None) -> tuple[Target, ...]:
|
||||||
|
"""The targets `publish` refreshes after a commit. A broken `export` section
|
||||||
|
raises only when some target asks for that - a checkout that never set
|
||||||
|
`after_publish` is not held up by a section it does not use there."""
|
||||||
|
root = Path(config.ROOT if root is None else root)
|
||||||
|
try:
|
||||||
|
targets = read_config(root)
|
||||||
|
except ValidationError:
|
||||||
|
if _asks_for_after_publish(root):
|
||||||
|
raise
|
||||||
|
return ()
|
||||||
|
return tuple(t for t in targets or () if t.after_publish)
|
||||||
|
|
||||||
|
|
||||||
|
def choose(targets: Sequence[Target], names: Sequence[str]) -> list[Target]:
|
||||||
|
"""The targets `names` select, in the order named; all of them for no name.
|
||||||
|
An unknown name is an error before anything runs."""
|
||||||
|
if not names:
|
||||||
|
return list(targets)
|
||||||
|
by_name = {t.name: t for t in targets}
|
||||||
|
unknown = [n for n in names if n not in by_name]
|
||||||
|
if unknown:
|
||||||
|
raise ExportError(
|
||||||
|
f"unknown export target(s): {', '.join(unknown)}. Configured: {', '.join(by_name)}"
|
||||||
|
)
|
||||||
|
chosen: list[Target] = []
|
||||||
|
for name in names:
|
||||||
|
if by_name[name] not in chosen:
|
||||||
|
chosen.append(by_name[name])
|
||||||
|
return chosen
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Selection and rendering
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def select(pages: dict[str, Page], target: Target, kb_dir: Path) -> list[Page]:
|
||||||
|
"""The pages of `target`: the union over its selections, each the pages
|
||||||
|
that satisfy every predicate of it, sorted by `(title.casefold(), title)`."""
|
||||||
|
predicates = tuple(p for selection in target.include for p in selection)
|
||||||
|
filters.validate_fields(predicates, pages)
|
||||||
|
chosen: dict[str, Page] = {}
|
||||||
|
for selection in target.include:
|
||||||
|
chosen.update(filters.apply_predicates(pages, selection, kb_dir))
|
||||||
|
return sorted(chosen.values(), key=lambda p: (p.title.casefold(), p.title))
|
||||||
|
|
||||||
|
|
||||||
|
def file_path(page: Page, kb_dir: Path) -> str:
|
||||||
|
return page.path.relative_to(kb_dir).as_posix()
|
||||||
|
|
||||||
|
|
||||||
|
def _rewrite_links(text: str, here: str, destinations: dict[str, Optional[str]], links: str) -> str:
|
||||||
|
"""Wikilinks outside code to Markdown: a link to an exported page relative
|
||||||
|
to `here` (`links: relative`), everything else to its display text. Matches
|
||||||
|
are found on the masked copy and applied to the original, which share
|
||||||
|
offsets, so code reaches the output byte for byte."""
|
||||||
|
masked = strip_code_spans(text)
|
||||||
|
out: list[str] = []
|
||||||
|
pos = 0
|
||||||
|
for m in LINK_RE.finditer(masked):
|
||||||
|
title = normalize_link_target(text[m.start(1):m.end(1)])
|
||||||
|
rest = text[m.start(2):m.end(2)]
|
||||||
|
display = rest.split("|", 1)[1] if "|" in rest else title
|
||||||
|
destination = destinations.get(title)
|
||||||
|
if links == "relative" and destination:
|
||||||
|
relative = posixpath.relpath(destination, posixpath.dirname(here) or ".")
|
||||||
|
replacement = f"[{display}]({quote(relative, safe='/')})"
|
||||||
|
else:
|
||||||
|
replacement = display
|
||||||
|
out.append(text[pos:m.start()])
|
||||||
|
out.append(replacement)
|
||||||
|
pos = m.end()
|
||||||
|
out.append(text[pos:])
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def marker_value(instance: str, commit: str) -> str:
|
||||||
|
return f"kind={guideline_export.PAGES_KIND} instance={instance} commit={commit}"
|
||||||
|
|
||||||
|
|
||||||
|
def render_page(page: Page, here: str, destinations: dict[str, Optional[str]], target: Target,
|
||||||
|
instance: str, commit: str) -> bytes:
|
||||||
|
"""One page as the file a target holds: the frontmatter fence with the
|
||||||
|
export marker first, then the frontmatter `target.frontmatter` asks for,
|
||||||
|
then the body - its generated links block kept, wikilinks rewritten, code
|
||||||
|
untouched. LF endings, exactly one trailing newline."""
|
||||||
|
own: dict[str, Any] = {}
|
||||||
|
if target.frontmatter == "summary":
|
||||||
|
own["title"] = page.title
|
||||||
|
own.update({k: page.frontmatter[k] for k in SUMMARY_KEYS if k in page.frontmatter})
|
||||||
|
elif target.frontmatter == "full":
|
||||||
|
own.update({k: v for k, v in page.frontmatter.items() if k != "wikitool_export"})
|
||||||
|
marker = json.dumps(marker_value(instance, commit), ensure_ascii=False)
|
||||||
|
head = [f"wikitool_export: {marker}"]
|
||||||
|
if own:
|
||||||
|
head.append(dump_frontmatter(own))
|
||||||
|
body = page.body.replace("\r\n", "\n").strip("\n")
|
||||||
|
body = _rewrite_links(body, here, destinations, target.links)
|
||||||
|
text = "---\n" + "\n".join(head) + "\n---\n" + ("\n" + body + "\n" if body else "")
|
||||||
|
return text.encode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, commit: str) -> dict[str, bytes]:
|
||||||
|
"""The files of a target for `pages`, by path below the target. A page
|
||||||
|
that cannot be written on every platform, or two that would share a file
|
||||||
|
name in a case-insensitive folder, is an `ExportError`."""
|
||||||
|
paths = {id(page): file_path(page, kb_dir) for page in pages}
|
||||||
|
seen: dict[str, str] = {}
|
||||||
|
for page in pages:
|
||||||
|
rel = paths[id(page)]
|
||||||
|
if not backup.safe_relative(rel):
|
||||||
|
raise ExportError(
|
||||||
|
f"{rel} cannot be written to a folder on every platform (a name with `:`, a backslash "
|
||||||
|
"or an empty part) - rename the page or leave it out of the selection"
|
||||||
|
)
|
||||||
|
if seen.setdefault(rel.casefold(), rel) != rel:
|
||||||
|
raise ExportError(f"{rel} and {seen[rel.casefold()]} differ only in case - "
|
||||||
|
"they would be one file in a synced folder")
|
||||||
|
by_title: dict[str, Optional[str]] = {}
|
||||||
|
for page in pages:
|
||||||
|
by_title[page.title] = None if page.title in by_title else paths[id(page)]
|
||||||
|
files: dict[str, bytes] = {}
|
||||||
|
for page in pages:
|
||||||
|
rel = paths[id(page)]
|
||||||
|
files[rel] = render_page(page, rel, by_title, target, instance, commit)
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The target folder
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(data: bytes) -> str:
|
||||||
|
return hashlib.sha256(data).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def read_manifest(folder: Path) -> Optional[dict]:
|
||||||
|
"""The manifest of the last run into `folder`, or None when there is none.
|
||||||
|
Raises `ExportError` for one that cannot be read as one."""
|
||||||
|
path = folder / MANIFEST_NAME
|
||||||
|
if not path.exists():
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError) as exc:
|
||||||
|
raise ExportError(f"{path} cannot be read ({exc}) - nothing was written") from None
|
||||||
|
files = data.get("files") if isinstance(data, dict) else None
|
||||||
|
dirs = data.get("dirs", []) if isinstance(data, dict) else None
|
||||||
|
if (not isinstance(files, list) or not isinstance(dirs, list)
|
||||||
|
or not all(isinstance(d, str) and backup.safe_relative(d) for d in dirs)
|
||||||
|
or not all(
|
||||||
|
isinstance(entry, dict) and isinstance(entry.get("path"), str)
|
||||||
|
and backup.safe_relative(entry["path"]) and isinstance(entry.get("size"), int)
|
||||||
|
and isinstance(entry.get("sha256"), str)
|
||||||
|
for entry in files)):
|
||||||
|
raise ExportError(f"{path} is not an export manifest - nothing was written")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TargetResult:
|
||||||
|
name: str
|
||||||
|
path: Path
|
||||||
|
commit: str = ""
|
||||||
|
new: list = field(default_factory=list)
|
||||||
|
changed: list = field(default_factory=list)
|
||||||
|
removed: list = field(default_factory=list)
|
||||||
|
unchanged: int = 0
|
||||||
|
skipped: list = field(default_factory=list)
|
||||||
|
error: Optional[str] = None
|
||||||
|
titles: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def ok(self) -> bool:
|
||||||
|
return self.error is None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def touched(self) -> bool:
|
||||||
|
return bool(self.new or self.changed or self.removed)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_parents(folder: Path, rel: str) -> None:
|
||||||
|
current = folder
|
||||||
|
for part in rel.split("/")[:-1]:
|
||||||
|
current = current / part
|
||||||
|
if backup.is_link(current):
|
||||||
|
raise ExportError(f"{current} is a link - an export writes nothing through one")
|
||||||
|
if current.exists() and not current.is_dir():
|
||||||
|
raise ExportError(f"{current} is a file where the export needs a folder - move it away; "
|
||||||
|
"nothing was written")
|
||||||
|
|
||||||
|
|
||||||
|
def _on_disk(folder: Path, rel: str) -> Optional[str]:
|
||||||
|
"""The sha256 of `rel` in `folder`, None when it is missing. A directory or
|
||||||
|
link there is an `ExportError`."""
|
||||||
|
path = folder.joinpath(*rel.split("/"))
|
||||||
|
if backup.is_link(path):
|
||||||
|
raise ExportError(f"{path} is a link - an export touches no link; nothing was written")
|
||||||
|
if not path.exists():
|
||||||
|
return None
|
||||||
|
if not path.is_file():
|
||||||
|
raise ExportError(f"{path} is not a file - nothing was written")
|
||||||
|
return backup.file_digest(path)[1]
|
||||||
|
|
||||||
|
|
||||||
|
def mirror(folder: Path, files: dict[str, bytes], result: TargetResult, *, name: str, instance: str,
|
||||||
|
commit: str) -> None:
|
||||||
|
"""Bring `folder` to exactly `files` plus its manifest, filling `result`.
|
||||||
|
Everything that can refuse refuses before the first write."""
|
||||||
|
if not files:
|
||||||
|
raise ExportError("nothing to export - an empty result would delete the earlier export, "
|
||||||
|
"so nothing was changed")
|
||||||
|
if folder.exists() and not folder.is_dir():
|
||||||
|
raise ExportError(f"{folder} exists and is not a folder")
|
||||||
|
manifest = read_manifest(folder) if folder.is_dir() else None
|
||||||
|
if folder.is_dir() and manifest is None and any(folder.iterdir()):
|
||||||
|
raise ExportError(
|
||||||
|
f"{folder} is not empty and holds no {MANIFEST_NAME} - it is not an export of this stack, "
|
||||||
|
"and an export writes only into an empty or its own folder. Nothing was written."
|
||||||
|
)
|
||||||
|
old = {entry["path"]: entry["sha256"] for entry in (manifest or {}).get("files", [])}
|
||||||
|
new_digests = {rel: _sha256(data) for rel, data in files.items()}
|
||||||
|
|
||||||
|
changed_by_hand = []
|
||||||
|
for rel, digest in old.items():
|
||||||
|
present = _on_disk(folder, rel)
|
||||||
|
if present is not None and present not in (digest, new_digests.get(rel)):
|
||||||
|
changed_by_hand.append(rel)
|
||||||
|
if changed_by_hand:
|
||||||
|
raise ExportError(
|
||||||
|
"changed in the target since the last export: " + ", ".join(changed_by_hand)
|
||||||
|
+ ". An export is never edited; move or delete the file and run again. Nothing was written."
|
||||||
|
)
|
||||||
|
|
||||||
|
for rel, data in files.items():
|
||||||
|
_check_parents(folder, rel)
|
||||||
|
present = _on_disk(folder, rel)
|
||||||
|
if rel in old:
|
||||||
|
if present == new_digests[rel]:
|
||||||
|
result.unchanged += 1
|
||||||
|
else:
|
||||||
|
result.changed.append(rel)
|
||||||
|
elif present is None:
|
||||||
|
result.new.append(rel)
|
||||||
|
elif present == new_digests[rel]:
|
||||||
|
result.new.append(rel)
|
||||||
|
else:
|
||||||
|
raise ExportError(f"{folder.joinpath(*rel.split('/'))} exists and is not one of the export's "
|
||||||
|
"files - move it away. Nothing was written.")
|
||||||
|
result.removed = sorted(rel for rel in old if rel not in files and _on_disk(folder, rel) is not None)
|
||||||
|
|
||||||
|
dirs = set((manifest or {}).get("dirs", []))
|
||||||
|
if manifest is not None and old == new_digests and not (result.new or result.changed or result.removed):
|
||||||
|
return
|
||||||
|
if not folder.exists():
|
||||||
|
folder.mkdir(parents=True)
|
||||||
|
for rel in result.new + result.changed:
|
||||||
|
current, parts = folder, rel.split("/")[:-1]
|
||||||
|
for index, part in enumerate(parts):
|
||||||
|
current = current / part
|
||||||
|
if not current.exists():
|
||||||
|
current.mkdir()
|
||||||
|
dirs.add("/".join(parts[:index + 1]))
|
||||||
|
backup.write_atomic(folder.joinpath(*rel.split("/")), [files[rel]])
|
||||||
|
for rel in result.removed:
|
||||||
|
victim = folder.joinpath(*rel.split("/"))
|
||||||
|
victim.unlink()
|
||||||
|
parent = posixpath.dirname(rel)
|
||||||
|
while parent and parent in dirs:
|
||||||
|
try:
|
||||||
|
folder.joinpath(*parent.split("/")).rmdir()
|
||||||
|
except OSError:
|
||||||
|
break
|
||||||
|
dirs.discard(parent)
|
||||||
|
parent = posixpath.dirname(parent)
|
||||||
|
listing = [{"path": rel, "size": len(files[rel]), "sha256": new_digests[rel]} for rel in sorted(files)]
|
||||||
|
body = {
|
||||||
|
"schema": MANIFEST_SCHEMA, "kind": guideline_export.PAGES_KIND, "target": name,
|
||||||
|
"instance": instance, "commit": commit, "dirs": sorted(dirs), "files": listing,
|
||||||
|
}
|
||||||
|
data = (json.dumps(body, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
|
||||||
|
backup.write_atomic(folder / MANIFEST_NAME, [data])
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The command handler
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _tail(stderr: bytes) -> str:
|
||||||
|
text = stderr.decode("utf-8", errors="replace").strip()
|
||||||
|
if len(text) > _STDERR_TAIL_CHARS:
|
||||||
|
text = "..." + text[-_STDERR_TAIL_CHARS:]
|
||||||
|
return text or "(nothing on stderr)"
|
||||||
|
|
||||||
|
|
||||||
|
def convert(target: Target, markdown: dict[str, bytes], result: TargetResult) -> dict[str, bytes]:
|
||||||
|
"""Run the target's converter over `markdown` and return what it left in
|
||||||
|
its output directory. The staging directory is always removed; any end but
|
||||||
|
exit 0 is an `ExportError` with the end of the program's stderr."""
|
||||||
|
where = f"the command {target.command_name!r}"
|
||||||
|
with tempfile.TemporaryDirectory(prefix="wikitool-export-") as tmp:
|
||||||
|
source, output = Path(tmp) / "markdown", Path(tmp) / "output"
|
||||||
|
output.mkdir()
|
||||||
|
for rel, data in markdown.items():
|
||||||
|
dest = source.joinpath(*rel.split("/"))
|
||||||
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
dest.write_bytes(data)
|
||||||
|
try:
|
||||||
|
ran = subprocess.run(
|
||||||
|
[*target.command, str(source), str(output)], capture_output=True,
|
||||||
|
stdin=subprocess.DEVNULL, timeout=COMMAND_TIMEOUT_SECONDS, check=False, cwd=tmp,
|
||||||
|
)
|
||||||
|
except FileNotFoundError:
|
||||||
|
raise ExportError(f"{where} could not be started: {target.command[0]} does not exist") from None
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
raise ExportError(f"{where} did not finish within {COMMAND_TIMEOUT_SECONDS} s") from None
|
||||||
|
except OSError as exc:
|
||||||
|
raise ExportError(f"{where} could not be started ({exc})") from None
|
||||||
|
if ran.returncode != 0:
|
||||||
|
raise ExportError(f"{where} ended with exit {ran.returncode}. Its stderr:\n{_tail(ran.stderr)}")
|
||||||
|
produced: dict[str, bytes] = {}
|
||||||
|
for directory, subdirs, names in os.walk(output, followlinks=False):
|
||||||
|
base = Path(directory)
|
||||||
|
subdirs[:] = [d for d in subdirs if not backup.is_link(base / d)]
|
||||||
|
for entry in sorted(names):
|
||||||
|
path = base / entry
|
||||||
|
rel = path.relative_to(output).as_posix()
|
||||||
|
if backup.is_link(path) or not path.is_file():
|
||||||
|
result.skipped.append(rel)
|
||||||
|
continue
|
||||||
|
if not backup.safe_relative(rel):
|
||||||
|
raise ExportError(f"{where} produced {rel!r}, which is not a name that can be "
|
||||||
|
"written on every platform")
|
||||||
|
produced[rel] = path.read_bytes()
|
||||||
|
casefolded: dict[str, str] = {}
|
||||||
|
for rel in produced:
|
||||||
|
if casefolded.setdefault(rel.casefold(), rel) != rel:
|
||||||
|
raise ExportError(f"{where} produced {rel} and {casefolded[rel.casefold()]}, which differ "
|
||||||
|
"only in case")
|
||||||
|
return produced
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# A run
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def run(targets: Sequence[Target], root: "Path | str | None" = None) -> list[TargetResult]:
|
||||||
|
"""Export each target in `targets`, independently: one that fails is
|
||||||
|
reported in its result and the others still run. What is not per target -
|
||||||
|
a checkout that is not clean, a page whose frontmatter does not parse - is
|
||||||
|
an `ExportError` before any target is touched."""
|
||||||
|
root = Path(config.ROOT if root is None else root)
|
||||||
|
kb_dir = config.KB_DIR
|
||||||
|
guideline_export.check_clean(root)
|
||||||
|
pages = load_pages_by_path(kb_dir, root)
|
||||||
|
unreadable = unreadable_pages(pages)
|
||||||
|
if unreadable:
|
||||||
|
listed = "\n".join(f" - {u['path']} ({u['reason']})" for u in unreadable)
|
||||||
|
raise ExportError(
|
||||||
|
"These pages have frontmatter that does not parse, so the export cannot tell whether "
|
||||||
|
f"they are selected:\n{listed}\n Fix them first; nothing was exported."
|
||||||
|
)
|
||||||
|
instance = guideline_export.instance_id(root)
|
||||||
|
results = []
|
||||||
|
for target in targets:
|
||||||
|
result = TargetResult(target.name, target.path)
|
||||||
|
results.append(result)
|
||||||
|
try:
|
||||||
|
selected = select(pages, target, kb_dir)
|
||||||
|
if not selected:
|
||||||
|
raise ExportError("no page matches the selection - an empty export would delete the "
|
||||||
|
"earlier one, so nothing was changed")
|
||||||
|
result.commit = guideline_export.last_commit(selected, root)
|
||||||
|
files = render(selected, target, kb_dir, instance, result.commit)
|
||||||
|
result.titles = {file_path(p, kb_dir): p.title for p in selected}
|
||||||
|
if target.handler == config_shapes.EXPORT_HANDLER_COMMAND:
|
||||||
|
files = convert(target, files, result)
|
||||||
|
result.titles = {}
|
||||||
|
mirror(target.path, files, result, name=target.name, instance=instance, commit=result.commit)
|
||||||
|
except (ValidationError, OSError) as exc:
|
||||||
|
result.error = str(exc)
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# What `raw accept` asks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def known_digests(root: "Path | str | None" = None) -> tuple[set[str], list[str]]:
|
||||||
|
"""The sha256 of every file the manifests of the configured targets list,
|
||||||
|
and a warning for each reason the answer is incomplete: a section that is
|
||||||
|
broken, a target whose manifest is missing or unreadable. The check this
|
||||||
|
feeds is a second line behind the marker, so it skips what it cannot read
|
||||||
|
rather than stop a `raw accept`."""
|
||||||
|
root = Path(config.ROOT if root is None else root)
|
||||||
|
try:
|
||||||
|
targets = read_config(root)
|
||||||
|
except ValidationError as exc:
|
||||||
|
return set(), [f"the export section is broken, so files in incoming/ were not compared with "
|
||||||
|
f"the exported ones ({exc})"]
|
||||||
|
digests: set[str] = set()
|
||||||
|
warnings: list[str] = []
|
||||||
|
for target in targets or ():
|
||||||
|
try:
|
||||||
|
manifest = read_manifest(target.path)
|
||||||
|
except (ValidationError, OSError) as exc:
|
||||||
|
warnings.append(f"export target {target.name}: {exc}")
|
||||||
|
continue
|
||||||
|
if manifest is None:
|
||||||
|
warnings.append(f"export target {target.name} has no manifest at {target.path}, so its files "
|
||||||
|
"were not compared with incoming/")
|
||||||
|
continue
|
||||||
|
digests.update(entry["sha256"] for entry in manifest["files"])
|
||||||
|
return digests, warnings
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def report_lines(result: TargetResult) -> list[str]:
|
||||||
|
"""What a run did to one target, for the terminal: its new files with the
|
||||||
|
page titles, then changed, removed, the count left untouched, and the
|
||||||
|
reason it failed."""
|
||||||
|
head = f"{result.name} -> {result.path}"
|
||||||
|
if result.error is not None:
|
||||||
|
return [f"{head}: failed", *(f" {line}" for line in result.error.splitlines())]
|
||||||
|
lines = [f"{head} (commit {result.commit[:12]})"]
|
||||||
|
|
||||||
|
def named(rel: str) -> str:
|
||||||
|
title = result.titles.get(rel)
|
||||||
|
return f"{title} ({rel})" if title and title != rel else rel
|
||||||
|
|
||||||
|
for label, rels in (("new", result.new), ("changed", result.changed), ("removed", result.removed)):
|
||||||
|
if rels:
|
||||||
|
lines.append(f" {label} ({len(rels)}): " + ", ".join(named(rel) for rel in rels))
|
||||||
|
lines.append(f" unchanged: {result.unchanged}")
|
||||||
|
if result.skipped:
|
||||||
|
lines.append(f" not regular files, skipped: {', '.join(result.skipped)}")
|
||||||
|
return lines
|
||||||
@@ -71,6 +71,7 @@ CONFIG_REGIONS = (
|
|||||||
("reference-config-sites", "SITES"),
|
("reference-config-sites", "SITES"),
|
||||||
("reference-config-fetch", "FETCH"),
|
("reference-config-fetch", "FETCH"),
|
||||||
("reference-config-backup", "BACKUP"),
|
("reference-config-backup", "BACKUP"),
|
||||||
|
("reference-config-export", "EXPORT"),
|
||||||
)
|
)
|
||||||
|
|
||||||
_SLUG_RE = re.compile(r"[^a-z0-9]+")
|
_SLUG_RE = re.compile(r"[^a-z0-9]+")
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ from typing import Iterator, Optional
|
|||||||
|
|
||||||
from chemenu import config, filelock, toolpaths, web_capture
|
from chemenu import config, filelock, toolpaths, web_capture
|
||||||
from chemenu.errors import BackendError, ValidationError
|
from chemenu.errors import BackendError, ValidationError
|
||||||
from chemenu.guideline_export import EXPORT_MARKER
|
from chemenu.guideline_export import is_export
|
||||||
|
|
||||||
MANIFEST_NAME = "_capture.json"
|
MANIFEST_NAME = "_capture.json"
|
||||||
SCHEMA = 1
|
SCHEMA = 1
|
||||||
@@ -56,7 +56,6 @@ ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
|
|||||||
GIT_TIMEOUT_SECONDS = 120.0
|
GIT_TIMEOUT_SECONDS = 120.0
|
||||||
|
|
||||||
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
|
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
|
||||||
_BOM = b"\xef\xbb\xbf"
|
|
||||||
|
|
||||||
_MODE_SYMLINK = "120000"
|
_MODE_SYMLINK = "120000"
|
||||||
_MODE_SUBMODULE = "160000"
|
_MODE_SUBMODULE = "160000"
|
||||||
@@ -446,9 +445,8 @@ def _structural_exclusion(entry: TreeEntry) -> Optional[str]:
|
|||||||
def _content_exclusion(data: bytes) -> Optional[str]:
|
def _content_exclusion(data: bytes) -> Optional[str]:
|
||||||
if data.startswith(LFS_MARKER):
|
if data.startswith(LFS_MARKER):
|
||||||
return "Git LFS pointer - the content is not in the repository"
|
return "Git LFS pointer - the content is not in the repository"
|
||||||
first = data[len(_BOM):] if data.startswith(_BOM) else data
|
if is_export(data):
|
||||||
if first.startswith(EXPORT_MARKER):
|
return "guideline export (carries the wikitool export marker)"
|
||||||
return "guideline export (first line starts with <!-- wikitool:export)"
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ def test_the_telemetry_policy_reads_the_record():
|
|||||||
def test_one_record_per_section():
|
def test_one_record_per_section():
|
||||||
assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
|
assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
|
||||||
"telemetry", "remotes", "intake", "tasks", "superproductivity", "caldav", "sites", "fetch",
|
"telemetry", "remotes", "intake", "tasks", "superproductivity", "caldav", "sites", "fetch",
|
||||||
"backup",
|
"backup", "export",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,764 @@
|
|||||||
|
"""`export run`, the `export` settings section, the `after_publish` step of
|
||||||
|
`publish`, the `doctor` line, and the digest check of `raw accept` - against a
|
||||||
|
real checkout and target folders in `tmp_path`.
|
||||||
|
|
||||||
|
The target's path carries a space on purpose: the folder a sync client keeps on
|
||||||
|
Windows is `OneDrive - <company>`."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import textwrap
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import typer
|
||||||
|
from typer.testing import CliRunner
|
||||||
|
|
||||||
|
from chemenu import config, config_shapes, guideline_export, page_export
|
||||||
|
from chemenu.cli import app
|
||||||
|
from chemenu.commands import doctor
|
||||||
|
from chemenu.commands.git_publish import publish_command
|
||||||
|
from chemenu.frontmatter_io import write_page
|
||||||
|
from chemenu.type_resolver import resolver
|
||||||
|
|
||||||
|
runner = CliRunner()
|
||||||
|
|
||||||
|
ALPHA = """
|
||||||
|
# Alpha
|
||||||
|
|
||||||
|
See [[Beta]], [[Beta|the beta page]] and [[Gamma]] for more, also [[Beta#Scope]].
|
||||||
|
Cited[^s-one]. Inline `[[Beta]]` stays.
|
||||||
|
|
||||||
|
```
|
||||||
|
[[Beta]] and [[Gamma]]
|
||||||
|
```
|
||||||
|
|
||||||
|
<!-- wikitool:footnotes -->
|
||||||
|
## Footnotes
|
||||||
|
|
||||||
|
[^s-one]: [[Source - One]]
|
||||||
|
<!-- /wikitool:footnotes -->
|
||||||
|
"""
|
||||||
|
|
||||||
|
BETA = """
|
||||||
|
# Beta
|
||||||
|
|
||||||
|
Back to [[Alpha]].
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _git(root: Path, *args: str) -> str:
|
||||||
|
result = subprocess.run(["git", *args], cwd=root, capture_output=True, text=True)
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _commit(root: Path, message: str = "change") -> str:
|
||||||
|
_git(root, "add", "-A")
|
||||||
|
_git(root, "commit", "-q", "-m", message)
|
||||||
|
return _git(root, "rev-parse", "HEAD")
|
||||||
|
|
||||||
|
|
||||||
|
def _page(root: Path, rel: str, body: str, **frontmatter) -> Path:
|
||||||
|
path = root / "kb" / rel
|
||||||
|
meta = {"type": "types/concept.md", "tags": ["docs"], "summary": f"About {path.stem}.",
|
||||||
|
"modified": "2026-10-01", "related": ["Beta"], **frontmatter}
|
||||||
|
write_page(path, meta, body)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def instance(tmp_path, monkeypatch):
|
||||||
|
root = tmp_path / "instance"
|
||||||
|
root.mkdir()
|
||||||
|
_git(root, "init", "-q", "-b", "main")
|
||||||
|
_git(root, "config", "user.name", "Test")
|
||||||
|
_git(root, "config", "user.email", "test@example.com")
|
||||||
|
(root / "incoming").mkdir()
|
||||||
|
(root / "raw").mkdir()
|
||||||
|
(root / "incoming" / ".gitkeep").write_text("", encoding="utf-8")
|
||||||
|
(root / ".gitignore").write_text(
|
||||||
|
"/incoming/*\n!/incoming/.gitkeep\n/.wikitool-local.json\n/quarantine/\n/reports/\n",
|
||||||
|
encoding="utf-8")
|
||||||
|
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
|
||||||
|
monkeypatch.setattr(config, "TYPES_DIR", root / "types")
|
||||||
|
monkeypatch.setattr(resolver, "_repo_root", root)
|
||||||
|
_page(root, "concepts/Alpha.md", ALPHA)
|
||||||
|
_page(root, "concepts/protocols/Beta.md", BETA)
|
||||||
|
_page(root, "concepts/Gamma.md", "\n# Gamma\n\nPrivate.\n", tags=["private"])
|
||||||
|
_commit(root, "init")
|
||||||
|
(tmp_path / "OneDrive - Example").mkdir()
|
||||||
|
monkeypatch.setattr(config, "ROOT", root)
|
||||||
|
monkeypatch.setenv("WIKITOOL_SESSION_ID", "test-session")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _out(root: Path, name: str = "Pages") -> Path:
|
||||||
|
return root.parent / "OneDrive - Example" / name
|
||||||
|
|
||||||
|
|
||||||
|
def _configure(root: Path, **targets) -> None:
|
||||||
|
"""One target `pages` by default, with `docs` pages selected."""
|
||||||
|
if not targets:
|
||||||
|
targets = {"pages": {}}
|
||||||
|
section = {}
|
||||||
|
for name, extra in targets.items():
|
||||||
|
body = {"handler": "markdown", "path": str(_out(root, name)), "include": [["tags=docs"]]}
|
||||||
|
body.update(extra)
|
||||||
|
section[name] = body
|
||||||
|
commands = {body["command"]: ["unset"] for body in section.values() if body.get("command")}
|
||||||
|
export = {"targets": section, **({"commands": commands} if commands else {})}
|
||||||
|
(root / config.LOCAL_SETTINGS_FILENAME).write_text(
|
||||||
|
json.dumps({"schema": 1, "export": export}), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _set_commands(root: Path, commands: dict) -> None:
|
||||||
|
path = root / config.LOCAL_SETTINGS_FILENAME
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
data["export"]["commands"] = commands
|
||||||
|
path.write_text(json.dumps(data), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _run(*names: str):
|
||||||
|
return runner.invoke(app, ["export", "run", *names])
|
||||||
|
|
||||||
|
|
||||||
|
def _flat(text: str) -> str:
|
||||||
|
return " ".join(text.split())
|
||||||
|
|
||||||
|
|
||||||
|
def _snapshot(directory: Path) -> dict:
|
||||||
|
return {p.relative_to(directory).as_posix(): p.read_bytes()
|
||||||
|
for p in sorted(directory.rglob("*")) if p.is_file()}
|
||||||
|
|
||||||
|
|
||||||
|
def _mtimes(directory: Path) -> dict:
|
||||||
|
return {p.relative_to(directory).as_posix(): p.stat().st_mtime_ns
|
||||||
|
for p in sorted(directory.rglob("*")) if p.is_file()}
|
||||||
|
|
||||||
|
|
||||||
|
def _manifest(folder: Path) -> dict:
|
||||||
|
return json.loads((folder / page_export.MANIFEST_NAME).read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
# --- the section -------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_section_is_an_error_naming_it_and_doctor_says_ok(instance):
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "no export section" in _flat(result.output)
|
||||||
|
assert _check("export").status == "OK"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("target, needle", [
|
||||||
|
({"path": "relative/folder"}, "absolute"),
|
||||||
|
({"include": []}, "include"),
|
||||||
|
({"include": [[]]}, "include"),
|
||||||
|
({"include": [["not a predicate"]]}, "predicate"),
|
||||||
|
({"handler": "pdf"}, "handler"),
|
||||||
|
({"frontmatter": "everything"}, "frontmatter"),
|
||||||
|
({"links": "hyper"}, "links"),
|
||||||
|
({"after_publish": "yes"}, "after_publish"),
|
||||||
|
({"command": "convert"}, "command"),
|
||||||
|
({"handler": "command"}, "command"),
|
||||||
|
])
|
||||||
|
def test_a_broken_section_stops_run_and_doctor_fails(instance, target, needle):
|
||||||
|
_configure(instance, pages=target)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1, result.output
|
||||||
|
assert needle in _flat(result.output)
|
||||||
|
assert _check("export").status == "FAIL"
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_inside_the_instance_is_refused(instance):
|
||||||
|
_configure(instance, pages={"path": str(instance / "site")})
|
||||||
|
assert _run().exit_code == 1
|
||||||
|
assert _check("export").status == "FAIL"
|
||||||
|
|
||||||
|
|
||||||
|
def test_targets_lying_in_one_another_are_refused(instance):
|
||||||
|
_configure(instance, one={"path": str(_out(instance, "one"))},
|
||||||
|
two={"path": str(_out(instance, "one") / "inner")})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "lie in one another" in _flat(result.output)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_in_the_backup_target_or_an_intake_folder_is_refused(instance):
|
||||||
|
backup_target = _out(instance, "backup")
|
||||||
|
path = instance / config.LOCAL_SETTINGS_FILENAME
|
||||||
|
_configure(instance, pages={"path": str(backup_target / "pages")})
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
data["backup"] = {"target": str(backup_target)}
|
||||||
|
path.write_text(json.dumps(data), encoding="utf-8")
|
||||||
|
assert _run().exit_code == 1
|
||||||
|
data.pop("backup")
|
||||||
|
data["intake"] = {"channels": {"inbox": {
|
||||||
|
"handler": "folder", "path": str(_out(instance, "pages") / "in"), "max_bytes": 1024,
|
||||||
|
"allowed_extensions": [".md"]}}}
|
||||||
|
data["export"]["targets"]["pages"]["path"] = str(_out(instance, "pages"))
|
||||||
|
path.write_text(json.dumps(data), encoding="utf-8")
|
||||||
|
assert _run().exit_code == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_operator_only_fields_cannot_be_set_by_the_agent():
|
||||||
|
export = config_shapes.EXPORT
|
||||||
|
assert export.field("targets").user_only and export.field("commands").user_only
|
||||||
|
assert export.field("targets").tier == config_shapes.CHECKOUT
|
||||||
|
assert export.field("commands").tier == config_shapes.CHECKOUT
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unknown_target_name_is_an_error_and_changes_nothing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
result = _run("pages", "nothing")
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "unknown export target" in _flat(result.output)
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
# --- what a run writes -------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_run_writes_the_selected_pages_with_marker_and_links(instance):
|
||||||
|
_configure(instance)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
assert set(_snapshot(folder)) == {
|
||||||
|
"concepts/Alpha.md", "concepts/protocols/Beta.md", page_export.MANIFEST_NAME}
|
||||||
|
commit = _git(instance, "rev-parse", "HEAD")
|
||||||
|
alpha = (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||||
|
lines = alpha.split("\n")
|
||||||
|
assert lines[0] == "---"
|
||||||
|
assert lines[1] == f'wikitool_export: "kind=pages instance=local commit={commit}"'
|
||||||
|
assert "title: Alpha" in alpha and "summary: About Alpha." in alpha
|
||||||
|
assert "tags" not in alpha and "related" not in alpha
|
||||||
|
assert "[Beta](protocols/Beta.md)" in alpha and "[the beta page](protocols/Beta.md)" in alpha
|
||||||
|
assert "also [Beta](protocols/Beta.md)" in alpha
|
||||||
|
assert "and Gamma for more" in alpha
|
||||||
|
assert "Cited[^s-one]." in alpha and "[^s-one]: " in alpha
|
||||||
|
assert "Inline `[[Beta]]` stays." in alpha
|
||||||
|
assert "```\n[[Beta]] and [[Gamma]]\n```" in alpha
|
||||||
|
assert alpha.endswith("\n") and not alpha.endswith("\n\n") and "\r" not in alpha
|
||||||
|
beta = (folder / "concepts" / "protocols" / "Beta.md").read_text(encoding="utf-8")
|
||||||
|
assert "[Alpha](../Alpha.md)" in beta
|
||||||
|
assert guideline_export.file_is_export(folder / "concepts" / "Alpha.md")
|
||||||
|
out = _flat(result.output)
|
||||||
|
assert "new (2)" in out and "Alpha" in out and "unchanged: 0" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_frontmatter_modes_and_links_text(instance):
|
||||||
|
_configure(instance, minimal={"frontmatter": "minimal", "links": "text"},
|
||||||
|
full={"frontmatter": "full"})
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
minimal = (_out(instance, "minimal") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||||
|
assert minimal.startswith('---\nwikitool_export: "kind=pages')
|
||||||
|
assert minimal.split("---\n")[1].count("\n") == 1
|
||||||
|
assert "[Beta](" not in minimal and "See Beta, the beta page and Gamma" in minimal
|
||||||
|
full = (_out(instance, "full") / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||||
|
assert "tags:" in full and "type: types/concept.md" in full and "related:" in full
|
||||||
|
|
||||||
|
|
||||||
|
def test_selections_are_or_and_predicates_are_and(instance):
|
||||||
|
_configure(instance, pages={"include": [["tags=docs", "summary~Alpha"], ["tags=private"]]})
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert set(_snapshot(_out(instance, "pages"))) == {
|
||||||
|
"concepts/Alpha.md", "concepts/Gamma.md", page_export.MANIFEST_NAME}
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unknown_field_is_an_error_at_run(instance):
|
||||||
|
_configure(instance, pages={"include": [["no_such_field=x"]]})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "unknown field" in _flat(result.output)
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rerun_on_an_unchanged_corpus_touches_nothing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
before, stamps = _snapshot(folder), _mtimes(folder)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0
|
||||||
|
assert _snapshot(folder) == before and _mtimes(folder) == stamps
|
||||||
|
assert "unchanged: 2" in _flat(result.output)
|
||||||
|
assert "new (" not in result.output and "changed (" not in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_instance_commit_that_touches_no_selected_page_changes_nothing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
stamps = _mtimes(folder)
|
||||||
|
(instance / "README.md").write_text("other\n", encoding="utf-8")
|
||||||
|
_commit(instance, "unrelated")
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert _mtimes(folder) == stamps
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_changed_page_a_new_page_and_a_removed_page(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_page(instance, "concepts/Delta.md", "\n# Delta\n\nNew.\n")
|
||||||
|
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||||
|
commit = _commit(instance, "edit")
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
out = _flat(result.output)
|
||||||
|
assert "new (1): Delta" in out and "changed (1):" in out and "removed (1):" in out
|
||||||
|
assert (folder / "concepts" / "Delta.md").is_file()
|
||||||
|
assert not (folder / "concepts" / "protocols" / "Beta.md").exists()
|
||||||
|
assert not (folder / "concepts" / "protocols").exists()
|
||||||
|
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||||
|
assert _manifest(folder)["commit"] == commit
|
||||||
|
assert [e["path"] for e in _manifest(folder)["files"]] == ["concepts/Alpha.md", "concepts/Delta.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_selection_growing_needs_no_gate(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
for index in range(30):
|
||||||
|
_page(instance, f"concepts/Many{index}.md", f"\n# Many{index}\n\nText.\n")
|
||||||
|
_commit(instance, "many")
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert "new (30)" in _flat(result.output)
|
||||||
|
|
||||||
|
|
||||||
|
def test_foreign_files_stay_and_only_created_directories_are_pruned(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
(folder / "notes.txt").write_bytes(b"mine\r\n")
|
||||||
|
(folder / "concepts" / "mine.txt").write_bytes(b"also mine")
|
||||||
|
(folder / "concepts" / "protocols" / "mine.txt").write_bytes(b"in a created dir")
|
||||||
|
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||||
|
_commit(instance, "drop beta")
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert (folder / "notes.txt").read_bytes() == b"mine\r\n"
|
||||||
|
assert (folder / "concepts" / "mine.txt").read_bytes() == b"also mine"
|
||||||
|
assert (folder / "concepts" / "protocols" / "mine.txt").read_bytes() == b"in a created dir"
|
||||||
|
assert (folder / "concepts" / "Alpha.md").is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_directory_that_existed_before_the_export_is_kept(instance):
|
||||||
|
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||||
|
_commit(instance, "no beta yet")
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
(folder / "concepts" / "protocols").mkdir()
|
||||||
|
_page(instance, "concepts/protocols/Beta.md", BETA)
|
||||||
|
_commit(instance, "beta")
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert "concepts/protocols" not in _manifest(folder)["dirs"]
|
||||||
|
_page(instance, "concepts/protocols/Beta.md", BETA, tags=["private"])
|
||||||
|
_commit(instance, "drop beta")
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert (folder / "concepts" / "protocols").is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_non_empty_folder_without_a_manifest_is_refused(instance):
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
folder.mkdir()
|
||||||
|
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||||
|
_configure(instance)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "not empty" in _flat(result.output) and "mine.md" not in _snapshot(folder).keys() - {"mine.md"}
|
||||||
|
assert _snapshot(folder) == {"mine.md": b"mine\n"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_existing_folder_is_taken(instance):
|
||||||
|
_out(instance, "pages").mkdir()
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_file_changed_by_hand_aborts_the_target_before_any_write(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
edited = folder / "concepts" / "protocols" / "Beta.md"
|
||||||
|
edited.write_text("my own words\n", encoding="utf-8")
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_commit(instance, "edit")
|
||||||
|
before = _snapshot(folder)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "concepts/protocols/Beta.md" in _flat(result.output)
|
||||||
|
assert _snapshot(folder) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_manifest_file_is_rewritten(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
expected = _snapshot(folder)
|
||||||
|
(folder / "concepts" / "Alpha.md").unlink()
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert _snapshot(folder) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_crash_between_the_files_and_the_manifest_is_recovered(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
old_manifest = (folder / page_export.MANIFEST_NAME).read_bytes()
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_commit(instance, "edit")
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
new_state = _snapshot(folder)
|
||||||
|
(folder / page_export.MANIFEST_NAME).write_bytes(old_manifest)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
assert _snapshot(folder) == new_state
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_selected_pages_is_an_error_and_deletes_nothing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
before = _snapshot(folder)
|
||||||
|
_configure(instance, pages={"include": [["tags=nothing-has-this"]]})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "no page matches" in _flat(result.output)
|
||||||
|
assert _snapshot(folder) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_dirty_kb_is_refused(instance):
|
||||||
|
_configure(instance)
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nunsaved\n")
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "uncommitted" in _flat(result.output)
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_page_with_unreadable_frontmatter_stops_the_run(instance):
|
||||||
|
_configure(instance)
|
||||||
|
(instance / "kb" / "concepts" / "Broken.md").write_text("---\ntype: [unclosed\n---\nBody\n", encoding="utf-8")
|
||||||
|
_commit(instance, "broken")
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "Broken.md" in result.output
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_targets_are_independent_and_any_failure_is_exit_1(instance):
|
||||||
|
bad = _out(instance, "bad")
|
||||||
|
bad.mkdir()
|
||||||
|
(bad / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||||
|
_configure(instance, bad={}, good={})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert (_out(instance, "good") / "concepts" / "Alpha.md").is_file()
|
||||||
|
assert _snapshot(bad) == {"mine.md": b"mine\n"}
|
||||||
|
assert "bad" in result.output and "failed" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_named_targets_run_alone(instance):
|
||||||
|
_configure(instance, one={}, two={})
|
||||||
|
assert _run("two").exit_code == 0
|
||||||
|
assert _out(instance, "two").is_dir() and not _out(instance, "one").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_case_only_collision_is_refused(instance):
|
||||||
|
_page(instance, "concepts/alpha.md", "\n# alpha\n\nLower.\n")
|
||||||
|
_commit(instance, "case")
|
||||||
|
_configure(instance)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "differ only in case" in _flat(result.output)
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
# --- the command handler -----------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _converter(root: Path, body: str) -> list[str]:
|
||||||
|
script = root.parent / "convert.py"
|
||||||
|
script.write_text(textwrap.dedent(body), encoding="utf-8")
|
||||||
|
return [sys.executable, str(script)]
|
||||||
|
|
||||||
|
|
||||||
|
def _command_target(root: Path, program: list[str]) -> None:
|
||||||
|
_configure(root, pages={"handler": "command", "command": "convert"})
|
||||||
|
_set_commands(root, {"convert": program})
|
||||||
|
|
||||||
|
|
||||||
|
COPY_AS_TXT = """
|
||||||
|
import pathlib, sys
|
||||||
|
src, out = map(pathlib.Path, sys.argv[1:3])
|
||||||
|
for f in src.rglob("*.md"):
|
||||||
|
dest = out / f.relative_to(src).with_suffix(".txt")
|
||||||
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
dest.write_text(f.read_text(encoding="utf-8").upper(), encoding="utf-8")
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_command_handler_exports_what_the_program_leaves(instance):
|
||||||
|
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
assert set(_snapshot(folder)) == {
|
||||||
|
"concepts/Alpha.txt", "concepts/protocols/Beta.txt", page_export.MANIFEST_NAME}
|
||||||
|
assert "WIKITOOL_EXPORT" in (folder / "concepts" / "Alpha.txt").read_text(encoding="utf-8")
|
||||||
|
stamps = _mtimes(folder)
|
||||||
|
assert _run().exit_code == 0 and _mtimes(folder) == stamps
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_command_gets_two_directories_no_stdin_and_the_staging_is_removed(instance, tmp_path):
|
||||||
|
record = tmp_path / "record.json"
|
||||||
|
program = _converter(instance, f"""
|
||||||
|
import json, os, pathlib, sys
|
||||||
|
src, out = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
|
||||||
|
stdin = sys.stdin.read()
|
||||||
|
pathlib.Path({str(record)!r}).write_text(json.dumps({{
|
||||||
|
"args": sys.argv[1:], "stdin": stdin, "cwd": os.getcwd(),
|
||||||
|
"src_files": sorted(p.name for p in src.rglob("*.md")), "out_files": os.listdir(out)}}))
|
||||||
|
(out / "all.txt").write_text("done")
|
||||||
|
""")
|
||||||
|
_command_target(instance, program)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
seen = json.loads(record.read_text(encoding="utf-8"))
|
||||||
|
assert seen["stdin"] == "" and seen["out_files"] == [] and seen["src_files"] == ["Alpha.md", "Beta.md"]
|
||||||
|
assert Path(seen["args"][0]).name == "markdown" and Path(seen["args"][1]).name == "output"
|
||||||
|
assert not Path(seen["args"][0]).parent.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failing_command_leaves_the_target_as_it_was(instance):
|
||||||
|
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
before = _snapshot(folder)
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_commit(instance, "edit")
|
||||||
|
_set_commands(instance, {"convert": _converter(instance, """
|
||||||
|
import sys
|
||||||
|
sys.stderr.write("x" * 5000 + "boom at the end")
|
||||||
|
sys.exit(3)
|
||||||
|
""")})
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1
|
||||||
|
text = _flat(result.output)
|
||||||
|
assert "exit 3" in text and "boom at the end" in text and "x" * 3000 not in text
|
||||||
|
assert _snapshot(folder) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_program_and_a_timeout_are_target_errors(instance, monkeypatch):
|
||||||
|
_command_target(instance, ["/nonexistent/converter"])
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1 and "could not be started" in _flat(result.output)
|
||||||
|
_set_commands(instance, {"convert": _converter(instance, "import time; time.sleep(30)")})
|
||||||
|
monkeypatch.setattr(page_export, "COMMAND_TIMEOUT_SECONDS", 1)
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 1 and "did not finish" in _flat(result.output)
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(os.name == "nt", reason="symlinks")
|
||||||
|
def test_symlinks_in_the_output_are_skipped_and_reported(instance):
|
||||||
|
_command_target(instance, _converter(instance, """
|
||||||
|
import os, pathlib, sys
|
||||||
|
out = pathlib.Path(sys.argv[2])
|
||||||
|
(out / "real.txt").write_text("real")
|
||||||
|
os.symlink("/etc/hostname", out / "link.txt")
|
||||||
|
"""))
|
||||||
|
result = _run()
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert set(_snapshot(_out(instance, "pages"))) == {"real.txt", page_export.MANIFEST_NAME}
|
||||||
|
assert "link.txt" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_converter_that_leaves_nothing_is_an_error(instance):
|
||||||
|
_command_target(instance, _converter(instance, "pass"))
|
||||||
|
assert _run().exit_code == 1
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
# --- publish -----------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _publish(**overrides):
|
||||||
|
kwargs = dict(message="change", push=False, confirm=None, confirm_rebase=None, yes=False,
|
||||||
|
threshold=10, remote="origin", branch="main", path=None)
|
||||||
|
kwargs.update(overrides)
|
||||||
|
publish_command(**kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_publish_exports_after_the_commit_with_no_push_too(instance):
|
||||||
|
_configure(instance, pages={"after_publish": True}, manual={})
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_publish()
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
head = _git(instance, "rev-parse", "HEAD")
|
||||||
|
assert _manifest(folder)["commit"] == head
|
||||||
|
assert "More." in (folder / "concepts" / "Alpha.md").read_text(encoding="utf-8")
|
||||||
|
assert not _out(instance, "manual").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failed_export_after_publish_is_a_warning_and_the_commit_stands(instance, capsys):
|
||||||
|
folder = _out(instance, "pages")
|
||||||
|
folder.mkdir()
|
||||||
|
(folder / "mine.md").write_text("mine\n", encoding="utf-8")
|
||||||
|
_configure(instance, pages={"after_publish": True})
|
||||||
|
head = _git(instance, "rev-parse", "HEAD")
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_publish()
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "WARN" in out and "export run" in out
|
||||||
|
assert _git(instance, "rev-parse", "HEAD") != head
|
||||||
|
|
||||||
|
|
||||||
|
def test_publish_without_after_publish_exports_nothing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
_publish()
|
||||||
|
assert not _out(instance, "pages").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_malformed_section_stops_publish_only_when_a_target_asks_for_it(instance):
|
||||||
|
_configure(instance, pages={"after_publish": True, "include": []})
|
||||||
|
_page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n")
|
||||||
|
head = _git(instance, "rev-parse", "HEAD")
|
||||||
|
with pytest.raises(typer.Exit) as exc:
|
||||||
|
_publish()
|
||||||
|
assert exc.value.exit_code == 1 and _git(instance, "rev-parse", "HEAD") == head
|
||||||
|
_configure(instance, pages={"include": []})
|
||||||
|
_publish()
|
||||||
|
assert _git(instance, "rev-parse", "HEAD") != head
|
||||||
|
|
||||||
|
|
||||||
|
# --- doctor ------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _check(name: str):
|
||||||
|
assert name == "export"
|
||||||
|
return doctor.check_export()
|
||||||
|
|
||||||
|
|
||||||
|
def test_doctor_lists_the_targets(instance):
|
||||||
|
_configure(instance, pages={"after_publish": True})
|
||||||
|
check = _check("export")
|
||||||
|
assert check.status == "OK"
|
||||||
|
assert "pages" in check.detail and "after publish" in check.detail
|
||||||
|
|
||||||
|
|
||||||
|
# --- raw accept --------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _accept(path: Path):
|
||||||
|
return runner.invoke(app, ["raw", "accept", str(path), "--fidelity", "verbatim",
|
||||||
|
"--authority", "normative"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_raw_accept_refuses_a_file_carrying_the_page_export_marker(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
exported = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
|
||||||
|
(instance / "incoming" / "Alpha.md").write_bytes(exported)
|
||||||
|
result = _accept(instance / "incoming" / "Alpha.md")
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "export" in _flat(result.output) and "nothing was moved" in _flat(result.output)
|
||||||
|
assert (instance / "incoming" / "Alpha.md").is_file()
|
||||||
|
assert not any((instance / "raw").rglob("*.md"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_raw_accept_refuses_an_exported_file_whose_marker_a_converter_dropped(instance):
|
||||||
|
_command_target(instance, _converter(instance, COPY_AS_TXT))
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
exported = (_out(instance, "pages") / "concepts" / "Alpha.txt").read_bytes()
|
||||||
|
folder = instance / "incoming" / "pack"
|
||||||
|
folder.mkdir()
|
||||||
|
(folder / "note.txt").write_text("ordinary\n", encoding="utf-8")
|
||||||
|
(folder / "Alpha.txt").write_bytes(exported)
|
||||||
|
(instance / "incoming" / "Alpha.txt").write_bytes(exported)
|
||||||
|
for target in (folder, instance / "incoming" / "Alpha.txt"):
|
||||||
|
before = _snapshot(instance / "incoming")
|
||||||
|
result = _accept(target)
|
||||||
|
assert result.exit_code == 1, result.output
|
||||||
|
assert "configured export target" in _flat(result.output)
|
||||||
|
assert "Alpha.txt" in result.output
|
||||||
|
assert _snapshot(instance / "incoming") == before
|
||||||
|
assert not any(p.is_file() for p in (instance / "raw").rglob("*"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_raw_accept_warns_and_goes_on_when_a_manifest_is_missing(instance):
|
||||||
|
_configure(instance)
|
||||||
|
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
|
||||||
|
result = _accept(instance / "incoming" / "note.md")
|
||||||
|
assert "WARN" in result.output and "no manifest" in _flat(result.output)
|
||||||
|
assert "configured export target" not in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_raw_accept_warns_and_goes_on_when_the_section_is_broken(instance):
|
||||||
|
_configure(instance, pages={"include": []})
|
||||||
|
(instance / "incoming" / "note.md").write_text("# Note\n\nOrdinary.\n", encoding="utf-8")
|
||||||
|
result = _accept(instance / "incoming" / "note.md")
|
||||||
|
assert "WARN" in result.output and "export section is broken" in _flat(result.output)
|
||||||
|
assert "configured export target" not in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_known_digests_lists_the_manifest_files(instance):
|
||||||
|
_configure(instance)
|
||||||
|
assert _run().exit_code == 0
|
||||||
|
digests, warnings = page_export.known_digests()
|
||||||
|
alpha = (_out(instance, "pages") / "concepts" / "Alpha.md").read_bytes()
|
||||||
|
assert hashlib.sha256(alpha).hexdigest() in digests and warnings == []
|
||||||
|
|
||||||
|
|
||||||
|
# --- the marker --------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
FRONT = b'---\nwikitool_export: "kind=pages instance=local commit=abc"\ntitle: A\n---\n\nBody\n'
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("data, expected", [
|
||||||
|
(b"<!-- wikitool:export kind=guidelines -->\nbody", True),
|
||||||
|
(FRONT, True),
|
||||||
|
(b"\xef\xbb\xbf" + FRONT, True),
|
||||||
|
(FRONT.replace(b"\n", b"\r\n"), True),
|
||||||
|
(b"---\ntitle: A\n---\nwikitool_export: x\n", False),
|
||||||
|
(b"---\ntitle: A\nwikitool_export: x\n", False),
|
||||||
|
(b"# Title\n---\nwikitool_export: x\n---\n", False),
|
||||||
|
(b"---\ntitle: A\n---\n\nBody\n", False),
|
||||||
|
(b"", False),
|
||||||
|
])
|
||||||
|
def test_is_export_recognises_both_forms(data, expected):
|
||||||
|
assert guideline_export.is_export(data) is expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_key_beyond_the_scan_window_is_not_seen():
|
||||||
|
padding = b"x: " + b"y" * 100 + b"\n"
|
||||||
|
data = b"---\n" + padding * (guideline_export.SCAN_BYTES // len(padding) + 2) + b"wikitool_export: z\n---\n"
|
||||||
|
assert guideline_export.is_export(data) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_file_is_export_reads_files(tmp_path):
|
||||||
|
(tmp_path / "a.md").write_bytes(FRONT)
|
||||||
|
(tmp_path / "b.md").write_bytes(b"plain\n")
|
||||||
|
assert guideline_export.file_is_export(tmp_path / "a.md")
|
||||||
|
assert not guideline_export.file_is_export(tmp_path / "b.md")
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_marker_key_is_defined_once():
|
||||||
|
package = Path(guideline_export.__file__).parent
|
||||||
|
definitions = [
|
||||||
|
path for path in package.rglob("*.py")
|
||||||
|
if "tests" not in path.parts and "wikitool_export:" in path.read_text("utf-8")
|
||||||
|
and 'b"wikitool_export:"' in path.read_text("utf-8")
|
||||||
|
]
|
||||||
|
assert definitions == [Path(guideline_export.__file__)]
|
||||||
Reference in new issue
Block a user