diff --git a/AGENTS.md b/AGENTS.md index db82736..239f5ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,7 +152,7 @@ exists, why silent overwrite is the failure it guards against, and why an instan from a release), [docs/language-boundaries.md](docs/language-boundaries.md) (why the control plane is English everywhere and the KB language is a value, and why the axis is the reader rather than the -owner), [docs/why-gates-are-code.md](docs/why-gates-are-code.md) (why the four gates in +owner), [docs/why-gates-are-code.md](docs/why-gates-are-code.md) (why the five gates in [Gates](#gates) are code rather than instruction), [docs/version-model.md](docs/version-model.md) (why a version number answers a compatibility question and a migration question separately), and @@ -267,7 +267,7 @@ that take a title. A result cut short by `--limit` says so and names the total. ## Gates -Four limits are enforced in code rather than by instruction, because a prompt-level limit is +Five limits are enforced in code rather than by instruction, because a prompt-level limit is one an agent can talk itself past. - **Mass-Update Gate.** `publish` exits **42** on a change touching too many files, printing @@ -279,6 +279,10 @@ one an agent can talk itself past. - **Upload Review Gate.** `upload accept` exits **42** on an MCP `submit` tool submission nobody has cleared yet, printing its manifest and the `--confirm ` line that promotes it once the user approves - same shape as the Mass-Update Gate, one submission at a time. +- **Guideline Push Gate.** `export guidelines --push` exits **42** before writing the generated + `GUIDELINES.md` into any captured repository, printing every target's status, the diff for + each one it would write, and the `--confirm ` line that pushes exactly that set once the + user approves. - **Iteration Budget Gate / Loop-Breaker.** Past 60 `wikitool` calls in a session, or after 3 identical calls in a row, further calls are refused. diff --git a/CHANGES.md b/CHANGES.md index e0f461e..1ba4bda 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse. --- -## 8.0.0-beta.43 - 2026-10-05 - wiki-ingest: updating the captured repositories is a step-1 branch over raw status +## 8.0.0-beta.44 - 2026-10-06 - export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate **Author:** Torben Nehmer @@ -110,6 +110,7 @@ concern - readable here, never shipped as something to parse. - Comparison and source pages accept the sources: that cite add writes; sources may cite sources - raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting - wiki-ingest: updating the captured repositories is a step-1 branch over raw status +- export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate **Low impact** - version bump no longer points at version release in its output @@ -156,6 +157,51 @@ concern - readable here, never shipped as something to parse. - raw accept: an occupied folder name held by a captured bundle points at --replaces-bundle +### export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate + +The other direction of `raw capture`. Project repositories each carried their own copy of the +same agent rules, drifting apart; the instance now holds them once and delivers them as a +generated, committed file, so each repository works on its own - in CI, without MCP. + +- **`export guidelines `** selects pages with `search`'s own predicates and no text + (`--field`, `--kind`, `--subtype`, `--collection`, `--tag`), and prints one `GUIDELINES.md`. + Line 1 is ``; + then each page by title as `# ` and its text without frontmatter, the generated links and + footnotes regions and citation markers, with wikilinks turned into their text - code left + byte for byte. No timestamp and not `HEAD`: the same pages give the same bytes, so an instance + commit that touches no guideline changes no target repository. It refuses with no predicate, no + match, an unknown field, any unreadable frontmatter under `kb/`, or uncommitted changes in `kb/` + or `types/`. +- **`--push`** writes that file into every captured repository - the `(repo, ref)` of each + `_capture.json`, `--bundle` narrowing it - that opted in by carrying a `GUIDELINES.md` whose + first line is the export header; a header line alone is the opt-in. A hand-written file, one + from another `instance=`, a tag rule, and an unreachable repository are skipped with the reason. + Each written target gets one commit on the fetched tip changing only `GUIDELINES.md`, built with + plumbing in the bare capture cache (no working tree), authored as the instance checkout's + `user.name`/`user.email`, and pushed without force; a branch that moved since the fetch is + rejected, read from `push --porcelain`'s status flag rather than git's translated messages. +- **The Guideline Push Gate** - a fifth named gate. `--push` without a matching `--confirm` + pushes nothing and exits 42 with each target's status, the diff of every file it would write, + and the re-run line; the token digests URL, branch, old tip and new blob of every target to + write. Nothing to write means no gate. The Publish-Remote Gate does not apply - the targets are + declared by the committed manifests already. +- **No way back in:** `raw accept` refuses a file whose first line starts with + `<!-- wikitool:export`, given on its own, inside a folder, or as `--replaces`/`--replaces-bundle` + material - `raw capture` already left such files out. `EXPORT_MARKER` now lives once, in the new + core module `guideline_export.py`. + +Which pages are guidelines is an instance decision: `kb/CONVENTIONS.md` (and its template) has a +new section § Guidelines for other repositories; this instance's filter is `--tag guideline`. +`kb/CONTRACT.md` says what leaving the wiki does to a page; `raw/CONTRACT.md`, `AGENTS.md` +§ Gates, `instructions/gates.md`, `docs/why-gates-are-code.md`, `README.md` and +`tools/README.md` follow. Internals: `kb_scan.LINK_RE` (moved from `commands/page_ops`, which +re-exports it), `filters.raw_predicates` shared with `search`, `repo_capture.run_git_result` and +`captured_manifests` shared with `raw status`. + +A new command and a new refusal for files no tool produced before - drop-in in both directions, +no page or manifest changes (Gitea #179). + ### wiki-ingest: updating the captured repositories is a step-1 branch over raw status `raw status`, `raw capture --update` and `raw accept --replaces-bundle` existed, but nothing told diff --git a/README.md b/README.md index 7c58667..37cc35f 100644 --- a/README.md +++ b/README.md @@ -227,6 +227,18 @@ tell the LLM `Update the captured repositories`, and it checks them, takes one c run into the wiki and says how many are still behind. Git uses your own keys and credential helpers - nothing is stored in the repository. +The same repositories can get something back: the wiki's guidelines, as one generated +`GUIDELINES.md` in their root. Which pages count as guidelines is your decision, written down in +`kb/CONVENTIONS.md` (§ Guidelines for other repositories); `tools/wikitool export guidelines +--tag guideline` prints the file as it would be written. A repository takes part by carrying a +`GUIDELINES.md` whose first line is `<!-- wikitool:export kind=guidelines -->` - commit that one +line there, and point the repository's own `AGENTS.md` at the file (Claude Code: +`@GUIDELINES.md`). A hand-written `GUIDELINES.md` is never touched. Then tell the LLM +`Roll out the guidelines to the captured repositories`: it runs `export guidelines --push`, which +stops before pushing anything (the **Guideline Push Gate**) and shows you, per repository, what +would change - the LLM puts every diff in front of you, and only your approval pushes one commit +per repository, changing only that file, straight onto its branch. + ### Querying Knowledge Ask questions naturally: @@ -407,6 +419,11 @@ this" code, not an error - printing the full file list and the `--confirm <token>` line that publishes it. The token digests that file list, so a clearance never carries to a changeset the user did not see. +**Guideline push.** `tools/wikitool export guidelines --push` exits **42** the same way (the +**Guideline Push Gate**) before it writes `GUIDELINES.md` into any captured repository, printing +every target's status and diff and the `--confirm <token>` line. The token digests each target's +branch tip and the file, so a moved branch or an edited guideline asks again. + **Iteration/cost limits.** Every `tools/wikitool` call is checked against a hard, code-enforced per-session budget before it runs (default: 60 calls, or 3 identical calls in a row) - not just a prompt instruction to stop. Past the diff --git a/VERSION b/VERSION index 3385b23..40a5ab9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -8.0.0-beta.43 +8.0.0-beta.44 diff --git a/docs/why-gates-are-code.md b/docs/why-gates-are-code.md index 2e1442b..ee4e61f 100644 --- a/docs/why-gates-are-code.md +++ b/docs/why-gates-are-code.md @@ -1,17 +1,18 @@ # Why gates are code -Chemenu has four hard limits - the Mass-Update Gate, the Publish-Remote Gate, the Upload Review -Gate, and the Iteration Budget Gate - and all four live inside `tools/wikitool`, not in a +Chemenu has five hard limits - the Mass-Update Gate, the Publish-Remote Gate, the Upload Review +Gate, the Guideline Push Gate, and the Iteration Budget Gate - and all five live inside +`tools/wikitool`, not in a paragraph of instructions an agent reads and follows. The rules themselves, and what to do when one trips, are in [AGENTS.md § Gates](../AGENTS.md#gates) and [instructions/gates.md](../instructions/gates.md). This page is only about the design choice -underneath them: why code, and why these four mechanisms in particular. +underneath them: why code, and why these mechanisms in particular. <!-- wikitool:toc --> ## Contents - [A suggestion an agent can talk itself past](#a-suggestion-an-agent-can-talk-itself-past) -- [Why four different mechanisms, not one](#why-four-different-mechanisms-not-one) +- [Why different mechanisms, not one](#why-different-mechanisms-not-one) - [Exit 42 is a posture, and it outgrew the gates](#exit-42-is-a-posture-and-it-outgrew-the-gates) - [A gate in code still has to be reachable](#a-gate-in-code-still-has-to-be-reachable) - [Numbers that come from measurement, not intuition](#numbers-that-come-from-measurement-not-intuition) @@ -31,10 +32,10 @@ conversation at all. It runs before the command dispatches, regardless of how co case for skipping it seemed a moment earlier. The difference isn't that code is smarter than a well-written instruction - it's that code doesn't get talked into anything. -## Why four different mechanisms, not one +## Why different mechanisms, not one -The four gates ask four different questions, and each one's shape follows from what kind of -question it is. +The five gates ask four different kinds of question, and each one's shape follows from what kind +of question it is. The Mass-Update Gate asks *is this change too large to publish unreviewed* - a judgment that varies changeset by changeset, so it clears with a `--confirm` token tied to the specific @@ -56,6 +57,17 @@ is answering, so nothing about the mechanism needed to change - only the boundar did, since the submission lives in a quarantine the ordinary pipeline never reads at all rather than in the working tree `publish` is about to commit. +The Guideline Push Gate asks the same question once more, at the boundary facing outwards: *is +this content right for these repositories, now?* `export guidelines --push` writes a generated +file straight onto another repository's branch, with no review on the receiving side, and a +public target publishes it the moment it lands. That is a per-run judgment - the guidelines +change, the set of opted-in repositories changes, a branch moves - so it takes the token shape, +digesting exactly what would be written where. It deliberately does not take the Publish-Remote +Gate's shape, although it too pushes to a remote: which repositories are targets is already a +standing, committed declaration - the capture manifests under `raw/` - and asking for the same +URLs in an allowlist as well would be a second declaration of one fact. What is left to ask is +the per-run question, and that is the one a token answers. + The Iteration Budget Gate asks a fourth kind of question - not "is this instance correct" but "has this session stopped making progress." That's read from the shape of the call history itself (call count, repeated identical calls), not from anything about the content of any one @@ -63,13 +75,13 @@ call. ## Exit 42 is a posture, and it outgrew the gates -Those four are the named gates, and they are not the only thing that exits 42 any more. When the +Those five are the named gates, and they are not the only thing that exits 42 any more. When the task-tracker provider layer arrived, it brought a case that looks like a gate from the outside and is not one: a provider whose API cannot create a project (Super Productivity's local REST API reads projects but does not write them) raises `HumanInterventionRequired`, and the command prints what a human has to do and exits 42. -Reusing the code was deliberate, and so was not calling it a fifth gate. What the four gates share +Reusing the code was deliberate, and so was not calling it another gate. What the named gates share is a *refusal*: the operation was possible and the tool declined to perform it unreviewed. This is the opposite situation - the operation is not possible at all, and no token could make it possible. What the two have in common is only what the exit code actually communicates: **stop, @@ -106,7 +118,7 @@ That failure has no symptom of its own. A gate that fires announces that it exis *cannot* fire looks identical to a gate nobody happened to need - the same clean runs, the same silence - and what finally told the two apart was reading a trace for an unrelated reason. So there is a third property to keep alongside living in code and carrying measured numbers: each -gate has to leave evidence that it can still fire. The three that clear by token or by a +gate has to leave evidence that it can still fire. The ones that clear by token or by a deliberate edit have it by construction, because clearing one is a visible event in somebody's terminal. The budget gate, whose ordinary outcome is silence, is the one that had to be given it - which is why its session id now carries where it came from, into both the trace and diff --git a/instructions/gates.md b/instructions/gates.md index f77a7fa..90e2578 100644 --- a/instructions/gates.md +++ b/instructions/gates.md @@ -1,7 +1,7 @@ --- type: types/instruction.md name: gates -description: What to do when wikitool refuses a call - exit 42 (user clearance required) on publish, and the Iteration Budget Gate and loop-breaker on every command. +description: What to do when wikitool refuses a call - exit 42 (user clearance required) on publish, upload accept and export guidelines --push, and the Iteration Budget Gate and loop-breaker on every command. --- # When a gate refuses a call @@ -25,6 +25,7 @@ Read the exit code first - it says which of these applies: - [Exit 42: user clearance required](#exit-42-user-clearance-required) - [Publish-Remote Gate](#publish-remote-gate) - [Upload Review Gate](#upload-review-gate) + - [Guideline Push Gate](#guideline-push-gate) - [Iteration Budget Gate and loop-breaker](#iteration-budget-gate-and-loop-breaker) - [Taking a new session id](#taking-a-new-session-id) - [Scope](#scope) @@ -33,12 +34,13 @@ Read the exit code first - it says which of these applies: ## Exit 42: user clearance required A `wikitool` command that exits **42** is not reporting an error. It is refusing to act until a -human has *read its output*. Four gates use it today - the Mass-Update Gate (`publish`, on a +human has *read its output*. Five gates use it today - the Mass-Update Gate (`publish`, on a change touching 10 or more counted files), the rebase-review gate (`sync` and `publish`, on a rebase whose incoming commits touch a file this session is also changing), the -Publish-Remote Gate (`publish`, on a push to a target this checkout has not declared), and the -Upload Review Gate (`upload accept`, on a submission nobody has cleared yet) - but the -rule is about the exit code, not the command: +Publish-Remote Gate (`publish`, on a push to a target this checkout has not declared), the +Upload Review Gate (`upload accept`, on a submission nobody has cleared yet), and the Guideline +Push Gate (`export guidelines --push`, before a generated `GUIDELINES.md` goes into any captured +repository) - but the rule is about the exit code, not the command: > **Copy the command's output into your reply - the substance of it, not a description of it - > and stop.** Run no further commands in that turn. @@ -104,7 +106,7 @@ checkout is in, WARNing only when there is more than one remote and no allowlist file is an error rather than "no restriction": a corrupted safeguard must not read as a disabled one. -**This gate has no `--confirm` token, on purpose.** The other three clear with a token because the +**This gate has no `--confirm` token, on purpose.** The others clear with a token because the question they ask ("is this change right?") is one the agent can put to the user and the user can answer for that one changeset. This one asks "does this content belong to that repository?", which is a standing property of the checkout, not a per-push judgment. The way past it is for the user @@ -133,6 +135,39 @@ Mass-Update Gate's review report versus this file's exit-42 procedure. all** - rejecting needs no clearance, only accepting a stranger's file into the pipeline does. It deletes the material and keeps only the reason and a sha256 in `mcp-upload/ledger.jsonl`. +### Guideline Push Gate + +`wikitool export guidelines --push` writes this instance's guideline pages, rendered into one +generated `GUIDELINES.md`, straight onto the branch of every captured repository that opted in - +no pull request, no review on the other side. That is the one write this stack makes into a +repository it does not own, and on a public target the content is public the moment it lands. So +the first run fetches and builds everything, pushes nothing, and exits 42. + +Its substance, which your reply reproduces in full: + +- **Every target and its status** - `new`, `changed`, `unchanged`, or `skipped` with the reason + (a tag rule, not reachable, not opted in, a hand-written file, another instance's file). A + skipped line is part of the answer: the user may have expected that repository to be written. +- **The diff of `GUIDELINES.md` for every target it would write**, against what that repository + carries now. This is what the user approves - the text that will appear there, not a count of + repositories. +- **The re-run line** with `--confirm <token>`, which you run only after the user approved this + exact set. + +The token digests each target's URL, branch, the tip the commit builds on, and the file's +content, so a branch that moved, a guideline edited since, or a newly captured repository makes +it stale: the next run is gated again with the current state. A run with nothing to write ends +with exit 0 and no gate. + +**Not the Publish-Remote Gate.** That one stays on `publish`. The targets here are declared by +the committed `_capture.json` manifests under `raw/`; asking for the same URLs in +`.wikitool-remotes.json` as well would be a second declaration of the same thing. What this gate +asks instead is the per-run question - whether this content belongs in these repositories now - +which is the kind a token answers. + +A push the branch moved under is rejected, never forced, and reported as `rejected`; running the +whole command again (and clearing it again) serves that repository. + ## Iteration Budget Gate and loop-breaker Every `wikitool` call is counted per session. Calls are refused past **60 in a session**, or diff --git a/instructions/kb-profiles.md b/instructions/kb-profiles.md index 6390822..660e056 100644 --- a/instructions/kb-profiles.md +++ b/instructions/kb-profiles.md @@ -55,7 +55,7 @@ is not a profile. | File | Holds | Profiles below | |---|---|---| - | `kb/CONVENTIONS.md` | Language, section headings, naming forms, tone, relationship labels, the hedging rule - once per instance | [Language profiles](#language-profiles) | + | `kb/CONVENTIONS.md` | Language, section headings, naming forms, tone, relationship labels, the hedging rule, which pages leave as guidelines - once per instance | [Language profiles](#language-profiles) | | `kb/<name>/COLLECTION.md` | What one collection holds, its quality goal, its local linking and naming rules | [Collection profiles](#collection-profiles) | 2. **Copy the entry's text into the file**, then edit it until it is true of this instance. diff --git a/kb/CONTRACT.md b/kb/CONTRACT.md index 3028003..1456594 100644 --- a/kb/CONTRACT.md +++ b/kb/CONTRACT.md @@ -13,7 +13,7 @@ how it works, so it is identical everywhere and `dist export` ships it verbatim. **What an instance decides for itself is next door, in [kb/CONVENTIONS.md](CONVENTIONS.md)** - the language pages are written in, the headings its two -generated regions render under, the naming forms, the tone, the hedging rule. That file binds exactly as this one does; it is simply owned by the instance +generated regions render under, the naming forms, the tone, the hedging rule, which pages leave the wiki as guidelines. That file binds exactly as this one does; it is simply owned by the instance rather than by the stack, so the distribution ships only its `.template` and the instance writes the real one. Read both, plus the target collection's `kb/<name>/COLLECTION.md` (also instance-owned), before writing or editing a page. @@ -40,6 +40,7 @@ looks like) are in neither - they belong to the type-specs and are printed by - [Generated regions](#generated-regions) - [Linking](#linking) - [Provenance and citation](#provenance-and-citation) +- [Pages that leave the wiki: guidelines](#pages-that-leave-the-wiki-guidelines) - [What does not belong here](#what-does-not-belong-here) <!-- /wikitool:toc --> @@ -329,6 +330,23 @@ Every claim is either traceable to a raw file or explicitly marked as not. If no raw file or existing page backs an answer, say so explicitly rather than synthesizing one - and never file the synthesized version back into the wiki. +## Pages that leave the wiki: guidelines + +`tools/wikitool export guidelines` renders a selection of pages into one generated +`GUIDELINES.md` and, behind the Guideline Push Gate, writes it into the captured repositories that +opted in ([raw/CONTRACT.md](../raw/CONTRACT.md#getting-a-repository-in-raw-capture)). Which pages +that is, is not decided here: the stack defines no type and no field for a guideline, and the +selection - a set of `search` predicates - is written down in +[kb/CONVENTIONS.md](CONVENTIONS.md) by the instance. + +What the stack does decide is how a page reads once it has left. The export is mechanical: +frontmatter, the generated links and footnotes regions and every citation marker are dropped, +`[[Title|Text]]` becomes `Text` and `[[Title]]` becomes `Title`, and code is left untouched. So a +guideline has to stand on its own in another repository - without its links to follow and without +the sources behind it - and an edit to one reaches every target repository on the next export. +The file there is never edited by hand: the next export overwrites it, so a correction goes into +the page. + ## What does not belong here - Raw source material - it stays immutable under `raw/`. diff --git a/kb/CONVENTIONS.md b/kb/CONVENTIONS.md index e287328..1e0b0f3 100644 --- a/kb/CONVENTIONS.md +++ b/kb/CONVENTIONS.md @@ -35,6 +35,7 @@ those regions and nothing else. Nothing matches on this text. - [Tone](#tone) - [Relationship labels](#relationship-labels) - [Hedging](#hedging) +- [Guidelines for other repositories](#guidelines-for-other-repositories) - [Keeping this file honest](#keeping-this-file-honest) <!-- /wikitool:toc --> @@ -140,6 +141,18 @@ carry, not against a threshold. This is `SOUL.md`'s existing standard ("Was nicht belegt ist, ist nicht gewusst, nur vermutet - und wird auch so benannt"), applied to `kb/` without a number competing next to it. +## Guidelines for other repositories + +`tools/wikitool export guidelines` renders the pages this filter selects into one generated +`GUIDELINES.md` for the repositories this instance captured (`raw/CONTRACT.md`). The stack defines +no type or field for a guideline - which pages are guidelines is this instance's decision, and it +is written down here and nowhere else. + +**The filter is `--tag guideline`.** A page carries the tag when its content is a rule an agent +working in another repository should follow there as it stands - written so that it reads without +its links and citations, because the export turns `[[links]]` into plain text and drops every +footnote. No page in this corpus carries it yet. + ## Keeping this file honest Change it when a convention actually changes. `sections:` is safe to change at any time - the diff --git a/kb/CONVENTIONS.md.template b/kb/CONVENTIONS.md.template index 387b1fb..21a63f9 100644 --- a/kb/CONVENTIONS.md.template +++ b/kb/CONVENTIONS.md.template @@ -34,6 +34,7 @@ marker pair, so a rename re-renders words and nothing else. - [Tone](#tone) - [Relationship labels](#relationship-labels) - [Hedging](#hedging) +- [Guidelines for other repositories](#guidelines-for-other-repositories) - [Keeping this file honest](#keeping-this-file-honest) <!-- /wikitool:toc --> @@ -112,6 +113,16 @@ sourced claim, in the KB language.} (`raw/CONTRACT.md`'s `authority` axis) from one resting on `opinion`, and how it signals disagreement between sources.} +## Guidelines for other repositories + +`tools/wikitool export guidelines` renders the pages this filter selects into one generated +`GUIDELINES.md` for the repositories this instance captured (`raw/CONTRACT.md`). The stack defines +no type or field for a guideline - which pages are guidelines is this instance's decision, and it +is written down here and nowhere else. + +{The selection, as `search` predicates - for example `--tag guideline` - or "none" if this +instance exports no guidelines.} + ## Keeping this file honest Change it when a convention actually changes. `wikitool doctor` FAILs on a missing or unfilled diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index 67fdec2..511e491 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -94,6 +94,14 @@ That is what keeps the clean-up safe: only directories the moves emptied are rem can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names both ways out - the whole folder, or the file moved up into `incoming/`. +**A guideline export never comes back in.** A file whose first line (after an optional BOM) +starts with `<!-- wikitool:export` is this stack's own output - `export guidelines` renders the +wiki's guideline pages into it for other repositories - and `raw accept` refuses it, given on its +own, inside a folder, or as `--replaces`/`--replaces-bundle` material, naming each one. Taken in, +it would turn the wiki's knowledge on its way out into a source the wiki then cites for the same +claims. `raw capture` leaves such a file out of a repository for the same reason +([below](#getting-a-repository-in-raw-capture)). + A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries no type any more - the kind of source comes from its content, as `source_type:` on the source page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs @@ -276,9 +284,13 @@ names it by its path inside the bundle (`docs/runbook.md`), not by its base name **`_capture.json` is the bundle's manifest** - `repo`, the `ref` rule, the `commit`, the `paths` globs, when it was `captured`, `fidelity`, `authority` and the `files` list. It is the only declaration that and how this instance follows the repository; there is no second configuration -file. The name is reserved: it is metadata of the bundle, not a source, so `sources coverage` and -`lint` never report it, no `raw_files:` lists it, and no file of that name is captured from a -repository or accepted from `incoming/` anywhere but at the top of a captured folder. +file. It is read in the other direction too: the captured repositories are exactly the ones +`export guidelines --push` writes this instance's `GUIDELINES.md` into - each one that opted in +by carrying the file, on the branch its ref rule names (a tag rule takes no commit, so such a +repository is skipped). The name is reserved: it is metadata of the bundle, not a source, so +`sources coverage` and `lint` never report it, no `raw_files:` lists it, and no file of that name +is captured from a repository or accepted from `incoming/` anywhere but at the top of a captured +folder. **Some files are never captured, by mechanism, each named in the output:** a file whose first line starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index bfbac5c..e4a81fe 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -37,6 +37,7 @@ file end to end is for changing the CLI itself. - [Provenance](#provenance) - [Raw material and uploads](#raw-material-and-uploads) - [Git](#git) + - [Guideline export](#guideline-export) - [Workshop runs and session budget](#workshop-runs-and-session-budget) - [Types, instructions and docs](#types-instructions-and-docs) - [Telemetry](#telemetry) @@ -125,6 +126,7 @@ upload accept write non-idempotent budget:counted exit:0,1, upload reject write non-idempotent budget:counted exit:0,1 Delete a submission's material, keeping only its ledger trail. sync write idempotent budget:counted exit:0,1,42 Fetch `<remote>/<branch>` and bring the local branch up to date with it. publish write non-idempotent budget:counted exit:0,1,42 Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push. +export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**). work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run. work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop. budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history. @@ -1643,6 +1645,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`. - 0 success - 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) - 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` +- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` - 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum - 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own - 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value @@ -1657,6 +1660,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`. - raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged - raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form +- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved - raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once - raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait - raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces` @@ -2002,6 +2006,74 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push. - `instructions/gates.md` - the gate procedure - `instructions/setup-instance.md` - the first publish of a new instance +### Guideline export + +#### `export guidelines` + +Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**). + +**SYNOPSIS** + +- `wikitool export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>]` - Prints the file to stdout; no network. +- `wikitool export guidelines <predicates> --push [--bundle <raw-bundle> ...] [--confirm TOKEN]` - Writes it into the captured repositories, behind the gate. + +**PROPERTIES** + +- effect: write +- idempotent: yes +- atomic: Per target - one commit pushed without force, or nothing; targets already written stay written when a later one fails +- budget: counted +- network: yes +- gates: guideline-push + +**EXAMPLES** + +- `tools/wikitool export guidelines --tag guideline` +- `tools/wikitool export guidelines --tag guideline --push` +- `tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # re-run after exit 42, once the user approved every diff` + +**EXIT STATUS** + +- 0 success +- 0 A target was skipped - tag rule, not reachable, not opted in, hand-written file, another instance's file +- 1 No predicate, no page matches, a malformed predicate or an unknown field +- 1 A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout +- 1 `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched +- 1 `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push` +- 1 A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served +- 42 Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets + +**ON FAILURE** + +- No predicate, no page matches, a malformed predicate or an unknown field -> Use the filter `kb/CONVENTIONS.md` names for this instance's guidelines, and retry once +- A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout -> Fix the named pages, or publish the changes first, then retry once +- `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched -> Show the message to the user; the identity is theirs to set, never yours +- `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push` -> Fix the argument and retry once +- A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served -> Show the lines to the user. A rejected target is served by running the whole command again, which asks for clearance again +- Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state + +**NEVER** + +- Never pass a `--confirm` token the user has not seen and approved. +- Never edit a `GUIDELINES.md` in a target repository by hand to make it match - the next export overwrites it; change the guideline page instead. + +**NOTES** + +- Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), `--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's guidelines is written down in `kb/CONVENTIONS.md`. +- Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page> - generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and its text without frontmatter, the generated links and footnote blocks, citation markers (`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as `Title`. A leading H1 of the page is replaced by the title heading. Code is never rewritten. LF endings, no timestamp: the same pages give the same bytes. +- Refuses before printing or pushing anything: no predicate, no match, an unknown field, any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or `types/` (untracked files included), or no git checkout. +- `--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref rule is a tag pattern, it cannot be reached or has no such branch, its root has no `GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), or the header names another `instance=`. A header line alone is the opt-in. +- Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, `unchanged`, `skipped`), the diff for every target it would write, and the re-run line. The token digests URL, branch, old tip and new blob of every target to write. +- With a matching token, each target gets exactly one commit on the fetched tip changing only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, `skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`. +- Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A second run after a successful one finds every target unchanged. +- The target repository's own `AGENTS.md` points at the file (Claude Code: `@GUIDELINES.md`); that belongs to the repository, not to this command. + +**SEE ALSO** + +- `wikitool search` - the same predicates, to preview the selection +- `wikitool raw capture` - how a repository becomes a target +- `instructions/gates.md` - the gate procedure + ### Workshop runs and session budget #### `work new` diff --git a/tools/README.md b/tools/README.md index 025b2d7..125a805 100644 --- a/tools/README.md +++ b/tools/README.md @@ -106,7 +106,8 @@ tools/ kb_state.py the KB version (.wikitool-kb.json) and the migration chain corpus_diff.py invariant comparison of kb/ between two revisions web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic - repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt + repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines` + guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached search/ pluggable search backends, plus service.py - the search core tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (.wikitool-tasks.json), one module per adapter - no instruction ever learns which provider it is commands/ one module per command or command group: the terminal adapters @@ -132,7 +133,7 @@ interpreter the way the preflight does - `python3`, `python` on `PATH`; on Windo and says why. **Two consumers, one core.** The CLI is not the only caller any more. The cores -(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`) hold what +(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`, `guideline_export.py`) hold what decides an answer and import no `typer` and no `rich`; the modules under `commands/` turn those values into terminal output and those exceptions into exit codes. `api.Corpus` is the in-process entry point over the same functions - it takes a @@ -201,8 +202,8 @@ not from a list of its own. Only read-only retrieval earns it. frontmatter, index statistics, cross-reference bookkeeping, log formatting, version arithmetic - is computed here so it comes out the same every time. -**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the -Iteration Budget Gate (`run_budget.py`) refuse in-process, because a +**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`), the Guideline Push +Gate (`export_cmd.py`) and the Iteration Budget Gate (`run_budget.py`) refuse in-process, because a prompt-level limit is one an agent can talk itself past. Exemption lists are constants, never flags. diff --git a/tools/chemenu/cli.py b/tools/chemenu/cli.py index 684edce..28e580b 100644 --- a/tools/chemenu/cli.py +++ b/tools/chemenu/cli.py @@ -31,6 +31,7 @@ try: doctor, docs_verify, eval_cmd, + export_cmd, git_publish, index_build, instructions_cmd, @@ -178,6 +179,7 @@ app.add_typer(dist_cmd.app, name="dist") app.add_typer(version_cmd.app, name="version") app.add_typer(migrate_cmd.app, name="migrate") app.add_typer(task_cmd.app, name="task") +app.add_typer(export_cmd.app, name="export") app.command("new")(new_page.new_page_command) app.command("touch")(touch_module.touch_command) app.command("rename")(page_ops.rename_command) diff --git a/tools/chemenu/cli_contract.py b/tools/chemenu/cli_contract.py index e4fb50b..5aaccf9 100644 --- a/tools/chemenu/cli_contract.py +++ b/tools/chemenu/cli_contract.py @@ -265,6 +265,9 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = ( ("Git", ( "sync", "publish", )), + ("Guideline export", ( + "export guidelines", + )), ("Workshop runs and session budget", ( "work new", "work close", "budget status", "budget reset", )), diff --git a/tools/chemenu/commands/export_cmd.py b/tools/chemenu/commands/export_cmd.py new file mode 100644 index 0000000..e8f8949 --- /dev/null +++ b/tools/chemenu/commands/export_cmd.py @@ -0,0 +1,429 @@ +"""`wikitool export guidelines` - the instance's guideline pages as one +generated `GUIDELINES.md`, printed, or written into the captured repositories +behind the Guideline Push Gate. + +The core - selection, rendering, the header - is `chemenu/guideline_export.py`; +the git write is `repo_capture.commit_file`/`push_commit`. This module decides +which captured repositories are targets, which of them take part, and when the +gate opens. + +**Targets are the captured repositories, and nothing else.** The `repo` and +`ref` of every `_capture.json` under `raw/` - the one declaration that this +instance follows a repository, now read in both directions. A second list of +targets would be a second declaration of the same thing to drift. + +**A repository takes part by carrying the file.** Only a branch whose root +`GUIDELINES.md` already starts with the guideline export header is written to; +a hand-written `GUIDELINES.md` is never replaced, and one whose header names a +different `instance=` belongs to that instance. A bare header line is the +opt-in. + +**Why a gate, and why this shape.** A push into another repository is the most +irreversible thing the stack does, and a public target publishes the content - +the `instance=` URL included - the moment it lands. So it is held to the same +token form as the Mass-Update and Upload Review gates: the first run fetches, +checks and builds everything, pushes nothing and exits 42 with every target's +status and diff; the token digests exactly that set (URL, branch, old tip, new +blob), so a moved branch, a changed guideline or a new target asks again. The +Publish-Remote Gate is deliberately not applied: the targets are declared by +committed manifests already, and listing the same URLs in +`.wikitool-remotes.json` would be the second declaration again. +""" +from __future__ import annotations + +import difflib +import hashlib +import json +import shlex +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional + +import typer +from rich.markup import escape + +from chemenu import cli_contract, config, guideline_export, repo_capture +from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path +from chemenu.errors import BackendError, ValidationError +from chemenu.search import filters + +app = typer.Typer(help="Generate files from the wiki for other repositories.") + +GATE = "guideline-push" + + +@dataclass +class Target: + """One `(repo, ref)` pair - several bundles captured from one branch are + one target. `error` is set when a manifest could not be read.""" + repo: str + ref: str + bundles: list[str] = field(default_factory=list) + error: str = "" + + @property + def label(self) -> str: + return f"{self.repo} {self.ref}" + + +@dataclass +class Plan: + target: Target + status: str # new | changed | unchanged | skipped + reason: str = "" + tip: str = "" + blob: str = "" + old: bytes = b"" + + @property + def writes(self) -> bool: + return self.status in ("new", "changed") + + +def _targets(only: list[Path]) -> list[Target]: + manifests = repo_capture.captured_manifests(config.RAW_DIR) + if only: + wanted = {p.resolve() for p in only} + known = {m.parent.resolve() for m in manifests} + unknown = sorted(rel_path(p) for p in only if p.resolve() not in known) + if unknown: + raise ValidationError( + f"--bundle {', '.join(unknown)}: not a captured bundle under raw/ (a directory " + f"holding {repo_capture.MANIFEST_NAME})." + ) + manifests = [m for m in manifests if m.parent.resolve() in wanted] + targets: dict[tuple[str, str], Target] = {} + broken: list[Target] = [] + for path in manifests: + bundle = rel_path(path.parent) + try: + manifest = repo_capture.read_manifest(path) + except ValidationError as exc: + broken.append(Target(bundle, "?", [bundle], f"manifest unreadable: {exc}")) + continue + target = targets.setdefault((manifest.repo, manifest.ref), Target(manifest.repo, manifest.ref)) + target.bundles.append(bundle) + return [*targets.values(), *broken] + + +def _plan(target: Target, export: guideline_export.Export) -> Plan: + """Where this target stands - fetched, never written to.""" + if target.error: + return Plan(target, "skipped", target.error) + if repo_capture.is_tag_pattern(target.ref): + return Plan(target, "skipped", "captured by a tag pattern - a tag takes no commit") + try: + with repo_capture.cache_repo(target.repo) as repo: + resolved = repo_capture.resolve_ref(target.repo, target.ref, repo) + tip = repo_capture.fetch(target.repo, resolved.refname, repo) + existing = repo_capture.read_path(repo, tip, guideline_export.TARGET_PATH) + blob = repo_capture.blob_id(repo, export.data) + except BackendError as exc: + return Plan(target, "skipped", f"not reachable: {exc}") + except ValidationError as exc: # the branch does not exist + return Plan(target, "skipped", str(exc)) + if existing is None: + return Plan(target, "skipped", f"no {guideline_export.TARGET_PATH} - not opted in", tip) + fields = guideline_export.guidelines_header(existing) + if fields is None: + return Plan( + target, "skipped", + f"{guideline_export.TARGET_PATH} has no export header - hand-written, never replaced", tip, + ) + owner = fields.get("instance") + if owner is not None and owner != export.instance: + return Plan(target, "skipped", f"delivered by another instance ({owner})", tip) + if existing == export.data: + return Plan(target, "unchanged", tip=tip) + return Plan(target, "changed" if owner else "new", tip=tip, blob=blob, old=existing) + + +def confirm_token(plans: list[Plan]) -> str: + """sha256 over every target the run would write - URL, branch, the tip it + builds on and the blob it writes - cut to 12 hex chars, the shape of + `git_publish.changeset_token`. The new commit's own id is not part of it: + `commit-tree` stamps the time, so it differs on every run.""" + payload = json.dumps( + sorted([p.target.repo, p.target.ref, p.tip, p.blob] for p in plans if p.writes), + sort_keys=True, + ) + return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12] + + +def _rerun(token: str, predicates: list[str], bundles: list[Path]) -> str: + parts = ["tools/wikitool", "export", "guidelines", "--confirm", token, "--push"] + for raw in predicates: + parts += ["--field", raw] + for bundle in bundles: + parts += ["--bundle", rel_path(bundle)] + return " ".join(shlex.quote(part) for part in parts) + + +def _diff(plan: Plan, export: guideline_export.Export) -> str: + old = plan.old.decode("utf-8", "replace").splitlines(keepends=True) + new = export.text.splitlines(keepends=True) + name = guideline_export.TARGET_PATH + return "".join(difflib.unified_diff(old, new, f"a/{name}", f"b/{name}")) + + +def _status_line(plan: Plan) -> str: + detail = f" ({plan.reason})" if plan.reason else "" + return f" {plan.target.label}: {plan.status}{detail}" + + +def _clearance(plans: list[Plan], export, token: str, rerun: str, stale: Optional[str]) -> str: + writes = [p for p in plans if p.writes] + lines = [ + f"This run would push one commit changing only {guideline_export.TARGET_PATH} into " + f"{len(writes)} repositor{'y' if len(writes) == 1 else 'ies'}. A push there is visible at " + "once - publicly, for a public repository - and not cheaply reversible. Nothing was pushed.", + "", + ] + if stale: + lines += [ + f"The token you passed ({stale}) does not match this set - a branch moved, a guideline " + "changed, a target came or went, or the token was invented. Here is the current state.", + "", + ] + lines += [ + "THE USER CANNOT SEE THIS OUTPUT. It went to your context, not to their screen.", + "Reproduce every target line and every diff below in your reply, and stop there. Run no " + "further commands in this turn.", + "", + f"TARGETS ({len(plans)}) - reproduce this in your reply:", + "", + *(_status_line(p) for p in plans), + ] + for plan in writes: + lines += ["", f"DIFF {plan.target.label}:", "", _diff(plan, export).rstrip("\n")] + lines += [ + "", + "Once they have replied approving exactly this, the line that pushes it is:", + "", + f" {rerun}", + "", + f"The token {token} covers each target's branch tip and the exact file; if either moves, " + "clearance is asked again.", + ] + return "\n".join(lines) + + +def _write(plan: Plan, export: guideline_export.Export, identity: tuple[str, str]) -> tuple[str, bool]: + """Commit and push one target. Returns its result line and whether it + counts as a failure (rejected, or a git error).""" + message = f"Update {guideline_export.TARGET_PATH} from {export.instance} at {export.commit[:12]}" + label = plan.target.label + try: + with repo_capture.cache_repo(plan.target.repo) as repo: + commit = repo_capture.commit_file( + repo, plan.tip, guideline_export.TARGET_PATH, export.data, message, identity + ) + result = repo_capture.push_commit(plan.target.repo, repo, commit, plan.target.ref) + except BackendError as exc: + return f" {label}: failed ({exc})", True + if not result.accepted: + return f" {label}: rejected ({result.detail}) - the branch moved; run again", True + return f" {label}: written {plan.tip[:12]} -> {commit[:12]}", False + + +@app.command("guidelines") +@cli_contract.record(cli_contract.CommandRecord( + path="export guidelines", + summary="Render the guideline pages into one generated `GUIDELINES.md`, or push it into the " + "captured repositories that opted in (**Guideline Push Gate**).", + synopsis=( + cli_contract.Variant( + usage="export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/" + "--tag <v>]", + notes="Prints the file to stdout; no network.", + ), + cli_contract.Variant( + usage="export guidelines <predicates> --push [--bundle <raw-bundle> ...] " + "[--confirm TOKEN]", + notes="Writes it into the captured repositories, behind the gate.", + ), + ), + properties=cli_contract.Properties( + effect=cli_contract.Effect.WRITE, + idempotent=cli_contract.Idempotent.YES, + atomic="Per target - one commit pushed without force, or nothing; targets already " + "written stay written when a later one fails", + budget=cli_contract.Budget.COUNTED, + network=cli_contract.Network.YES, + gates=(GATE,), + ), + notes=( + "Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), " + "`--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's " + "guidelines is written down in `kb/CONVENTIONS.md`.", + "Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin " + "URL without userinfo, or local> commit=<newest commit touching a selected page> - " + "generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and " + "its text without frontmatter, the generated links and footnote blocks, citation markers " + "(`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as " + "`Title`. A leading H1 of the page is replaced by the title heading. Code is never " + "rewritten. LF endings, no timestamp: the same pages give the same bytes.", + "Refuses before printing or pushing anything: no predicate, no match, an unknown field, " + "any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or " + "`types/` (untracked files included), or no git checkout.", + "`--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` " + "under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref " + "rule is a tag pattern, it cannot be reached or has no such branch, its root has no " + "`GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), " + "or the header names another `instance=`. A header line alone is the opt-in.", + "Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds " + "everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, " + "`unchanged`, `skipped`), the diff for every target it would write, and the re-run line. " + "The token digests URL, branch, old tip and new blob of every target to write.", + "With a matching token, each target gets exactly one commit on the fetched tip changing " + "only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, " + "pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, " + "`skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`.", + "Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A " + "second run after a successful one finds every target unchanged.", + "The target repository's own `AGENTS.md` points at the file (Claude Code: " + "`@GUIDELINES.md`); that belongs to the repository, not to this command.", + ), + failures=( + cli_contract.Failure( + cause="No predicate, no page matches, a malformed predicate or an unknown field", + reaction="Use the filter `kb/CONVENTIONS.md` names for this instance's guidelines, and " + "retry once", + ), + cli_contract.Failure( + cause="A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has " + "uncommitted changes, or the instance is not a git checkout", + reaction="Fix the named pages, or publish the changes first, then retry once", + ), + cli_contract.Failure( + cause="`--push` with no `user.name` or `user.email` in the instance checkout's git " + "configuration - checked before any target is fetched", + reaction="Show the message to the user; the identity is theirs to set, never yours", + ), + cli_contract.Failure( + cause="`--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` " + "was given without `--push`", + reaction="Fix the argument and retry once", + ), + cli_contract.Failure( + cause="A target rejected the push (its branch moved since the fetch) or a git call " + "failed - the other targets were still served", + reaction="Show the lines to the user. A rejected target is served by running the " + "whole command again, which asks for clearance again", + ), + cli_contract.Failure( + code=0, + cause="A target was skipped - tag rule, not reachable, not opted in, hand-written " + "file, another instance's file", + reaction="", + ), + cli_contract.Failure( + cause="Guideline Push Gate: `--push` without `--confirm`, or with a token that does " + "not match the current set of targets", + reaction=cli_contract.token_gate_reaction("--confirm"), + code=42, + ), + ), + examples=( + "tools/wikitool export guidelines --tag guideline", + "tools/wikitool export guidelines --tag guideline --push", + "tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # " + "re-run after exit 42, once the user approved every diff", + ), + never=( + "Never pass a `--confirm` token the user has not seen and approved.", + "Never edit a `GUIDELINES.md` in a target repository by hand to make it match - the " + "next export overwrites it; change the guideline page instead.", + ), + see_also=( + "`wikitool search` - the same predicates, to preview the selection", + "`wikitool raw capture` - how a repository becomes a target", + "`instructions/gates.md` - the gate procedure", + ), +)) +def export_guidelines_command( + field_: Optional[list[str]] = typer.Option( + None, "--field", "-f", + help="Frontmatter predicate, repeatable (AND) - the same forms as `search --field`.", + ), + kind: Optional[str] = typer.Option(None, "--kind", help="Shorthand for --field kind=<value>."), + subtype: Optional[str] = typer.Option(None, "--subtype", help="Shorthand for --field subtype=<value>."), + collection: Optional[str] = typer.Option( + None, "--collection", help="Shorthand for --field collection=<value>." + ), + tag: Optional[str] = typer.Option(None, "--tag", help="Shorthand for --field tags=<value>."), + push: bool = typer.Option( + False, "--push", help="Write the file into the captured repositories (Guideline Push Gate)." + ), + bundle: Optional[list[Path]] = typer.Option( + None, "--bundle", help="Only the repository of this captured bundle under raw/; repeatable." + ), + confirm: Optional[str] = typer.Option( + None, "--confirm", help="The token from a prior exit 42, once a human has approved it." + ), +): + """Render the guideline pages into GUIDELINES.md - printed, or pushed into + the captured repositories that carry one.""" + bundles = [b if b.is_absolute() else config.ROOT / b for b in (bundle or [])] + if not push and (bundles or confirm): + fail("--bundle and --confirm only apply with --push.") + raw = filters.raw_predicates(field_, kind, subtype, collection, tag) + try: + predicates = tuple(filters.parse_predicate(r) for r in raw) + export = guideline_export.build(predicates) + except ValidationError as exc: + fail(escape(str(exc))) + return + + if not push: + typer.echo(export.text, nl=False) + return + + identity = guideline_export.git_identity() + if identity is None: + fail( + "No git identity in this instance checkout - `git config user.name` and `git config " + "user.email` must both be set, since the commit in each target repository carries them. " + "Nothing was fetched or pushed." + ) + return + try: + targets = _targets(bundles) + except ValidationError as exc: + fail(escape(str(exc))) + return + if not targets: + typer.echo("No captured repository under raw/ - nothing to write.") + return + + plans = [_plan(t, export) for t in targets] + if not any(p.writes for p in plans): + typer.echo(f"{len(plans)} target(s), nothing to write:") + for plan in plans: + typer.echo(_status_line(plan)) + return + + token = confirm_token(plans) + if confirm != token: + console.print("[bold yellow]NEEDS USER CLEARANCE[/bold yellow] Guideline Push Gate") + typer.echo(_clearance(plans, export, token, _rerun(token, raw, bundles), confirm)) + raise typer.Exit(code=EXIT_NEEDS_CLEARANCE) + + failed = 0 + typer.echo(f"{len(plans)} target(s):") + for plan in plans: + if not plan.writes: + typer.echo(_status_line(plan)) + continue + line, bad = _write(plan, export, identity) + failed += bad + typer.echo(line) + written = sum(1 for p in plans if p.writes) - failed + typer.echo(f"{written} written, {failed} rejected or failed, {len(plans) - written - failed} " + "unchanged or skipped.") + if failed: + console.print( + f"[bold red]ERROR[/bold red] {failed} target(s) not written - what was written stays " + "written; run the command again for the rest." + ) + raise typer.Exit(code=1) diff --git a/tools/chemenu/commands/page_ops.py b/tools/chemenu/commands/page_ops.py index cd3a435..bb6df5e 100644 --- a/tools/chemenu/commands/page_ops.py +++ b/tools/chemenu/commands/page_ops.py @@ -25,7 +25,7 @@ from typing import Optional import typer -from chemenu import cli_contract, config, links +from chemenu import cli_contract, config, kb_scan, links from chemenu.commands._util import ( check_collision, check_path_budget, @@ -50,14 +50,10 @@ from chemenu.provenance import ( ) from chemenu.type_resolver import resolver -# `[[Target]]`, `[[Target|alias]]`, `[[Target#anchor]]` - including the -# `[[Target]]` inside a `[^cite-id]: [[Target]]` Footnotes definition, which -# is exactly what lets retarget_body() repoint a citation's link target on a -# rename. Group 1 is the target title; group 2 keeps any alias/anchor suffix -# untouched. The id itself is a separate concern - see retarget_cite_ids(). -# Group 1 may span a line break; it is compared through -# kb_scan.normalize_link_target(), the same reading `lint` gives it. -LINK_RE = re.compile(r"\[\[([^\[\]|#]+)((?:[|#][^\[\]]*)?)\]\]") +# `kb_scan.LINK_RE` - the whole wikilink, defined there so the guideline export +# reads links the same way without importing a command module. The cite id is a +# separate concern from the link target - see retarget_cite_ids(). +LINK_RE = kb_scan.LINK_RE def page_ref_fields(page: Page) -> list[str]: diff --git a/tools/chemenu/commands/raw_cmd.py b/tools/chemenu/commands/raw_cmd.py index 9c32ebd..ea4be25 100644 --- a/tools/chemenu/commands/raw_cmd.py +++ b/tools/chemenu/commands/raw_cmd.py @@ -87,7 +87,7 @@ from typing import Annotated, Optional import typer from rich.markup import escape -from chemenu import cli_contract, config, repo_capture, web_capture +from chemenu import cli_contract, config, guideline_export, repo_capture, web_capture from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success from chemenu.errors import BackendError, ChemenuError, ValidationError from chemenu.frontmatter_io import write_page @@ -226,11 +226,27 @@ def _check_files(resolved: list[Path]) -> None: _check_directly_in_incoming(path, incoming) if path.name == repo_capture.MANIFEST_NAME: raise _Refused(_reserved_manifest_message(path)) + _check_no_export(resolved) names = [path.name for path in resolved] if len(names) != len(set(names)): raise _Refused("Two files share a filename; rename one before promoting.") +def _check_no_export(paths: list[Path]) -> None: + """Refuse a file a wiki exported (`export guidelines`): it is generated + from `kb/`, so taking it into `raw/` would feed the wiki its own output + back as a source. `raw capture` already leaves such a file out; this + closes the way in through `incoming/`.""" + exports = [p for p in paths if guideline_export.file_is_export(p)] + if exports: + listed = "\n".join(f" - {rel_path(p)}" for p in exports) + raise _Refused( + "These files are a wiki's guideline export - their first line starts with " + "<!-- wikitool:export. They are generated from kb/ and never go back into raw/:\n" + f"{listed}\n Remove them from incoming/ and accept again; nothing was moved." + ) + + def _check_target(dst: Path, remedy: str) -> None: problem = path_budget_problem_for(dst) if problem: @@ -409,6 +425,7 @@ def _plan_folder(folder: Path) -> tuple[Path, list[tuple[Path, Path]], list[Path stray = [f for f in files if f.name == repo_capture.MANIFEST_NAME and f.parent != folder] if stray: raise _Refused(_reserved_manifest_message(stray[0])) + _check_no_export(files) holder = _occupied_stems(config.RAW_DIR).get(folder.name) if holder is not None: @@ -490,6 +507,7 @@ def _replace( if not incoming_path.is_file(): fail(f"{rel_path(incoming_path)} does not exist or is not a file.") _validate_under_incoming(incoming_path, _incoming_dir()) + _refusals_fail(_check_no_export, [incoming_path]) target = _resolve(replaces) try: @@ -784,6 +802,7 @@ def _plan_bundle_replacement(folder: Path, bundle: Path, pages) -> _BundlePlan: _check_matches_manifest(folder, manifest) new_files = repo_capture.bundle_files(folder) + _check_no_export(list(new_files.values())) old_files = repo_capture.bundle_files(bundle) changes = repo_capture.diff(old_files, {r: p.read_bytes() for r, p in new_files.items()}) changed = {r for _s, r in changes} @@ -1061,6 +1080,13 @@ def _replace_bundle( "incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename " "the folder in `incoming/` - such a folder has no replacement form", ), + cli_contract.Failure( + cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` " + "material - is a guideline export: its first line, after an optional BOM, starts with " + "`<!-- wikitool:export`", + reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes " + "back into `raw/`. Remove it from `incoming/`; nothing was moved", + ), cli_contract.Failure( label="raw accept", cause="`--fidelity`/`--authority` is missing, or names `unknown` or a value outside " @@ -2155,13 +2181,7 @@ def raw_status_command( ): """Report which captured bundles have fallen behind their repository. See raw/CONTRACT.md "Getting a repository in: `raw capture`".""" - manifests = [] - if config.RAW_DIR.is_dir(): - manifests = [ - p for p in sorted(config.RAW_DIR.rglob(repo_capture.MANIFEST_NAME)) - if p.is_file() and not any(part.startswith(".") for part in p.relative_to(config.RAW_DIR).parts) - ] - rows = [_bundle_status(p) for p in manifests] + rows = [_bundle_status(p) for p in repo_capture.captured_manifests(config.RAW_DIR)] if json_out: typer.echo(json.dumps(rows, indent=2, ensure_ascii=False)) diff --git a/tools/chemenu/commands/search.py b/tools/chemenu/commands/search.py index a1f889a..64f1633 100644 --- a/tools/chemenu/commands/search.py +++ b/tools/chemenu/commands/search.py @@ -240,12 +240,7 @@ def search_command( json_out: bool = typer.Option(False, "--json", help="Print the results as JSON."), ): """Search kb/ by text, by frontmatter, or by both.""" - raw_predicates = list(field or []) - for value, name in ((kind, "kind"), (subtype, "subtype"), (collection, "collection")): - if value: - raw_predicates.append(f"{name}={value}") - if tag: - raw_predicates.append(f"tags={tag}") + raw_predicates = filters.raw_predicates(field, kind, subtype, collection, tag) if not text and not raw_predicates: fail("Nothing to search for: give a query, or at least one --field predicate.") diff --git a/tools/chemenu/guideline_export.py b/tools/chemenu/guideline_export.py new file mode 100644 index 0000000..e3746bb --- /dev/null +++ b/tools/chemenu/guideline_export.py @@ -0,0 +1,296 @@ +"""The guideline export, with no CLI attached: select pages by frontmatter, +render them mechanically into one `GUIDELINES.md`, and recognise such a file +again when it comes back. + +`wikitool export guidelines` is the terminal adapter over this module; the git +side - writing the file into a captured repository's branch - lives beside +`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under +`chemenu.commands`, and nothing here imports `repo_capture`, which imports +`EXPORT_MARKER` from here: that is the direction the dependency runs. + +Three properties carry the design, and each has a test: + +- **Deterministic.** The output depends only on the selected pages and the + commit that last touched one of them - no timestamp, no `HEAD`. A run on an + unchanged corpus, or after an instance commit that touched no guideline, is + byte-identical, and so writes nothing into any target repository. +- **One filter semantics.** Selection is `search`'s own predicate machinery + (`search.filters`), with no text argument: a guideline is a declared choice in + frontmatter, not a full-text hit. +- **Code is never rewritten.** Every transformation of the page text runs on a + copy with code masked out (`markdown_code.strip_code_spans`, which keeps + offsets), so inline code and fenced blocks reach the output byte for byte. +""" +from __future__ import annotations + +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +from chemenu import blocks, config, corpus_cache, provenance, toolpaths +from chemenu.errors import ValidationError +from chemenu.kb_scan import LINK_RE, normalize_link_target +from chemenu.markdown_code import strip_code_spans +from chemenu.page import Page +from chemenu.search import filters +from chemenu.search.service import load_pages_by_path, unreadable_pages +from chemenu.search.types import Predicate + +# The first bytes of every file this stack generates for another repository. +# Defined once, here: `repo_capture` excludes a file carrying it from a +# capture, `raw accept` refuses one, and the export writes it. +EXPORT_MARKER = b"<!-- wikitool:export" +GUIDELINES_KIND = "guidelines" +GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}" +TARGET_PATH = "GUIDELINES.md" +LOCAL_INSTANCE = "local" + +_BOM = b"\xef\xbb\xbf" +_HEADER_FIELD = re.compile(r"\b(instance|commit)=(\S+)") +_H1 = re.compile(r"#(?:[ \t]|$)") +_SCHEME_USERINFO = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*://)[^/]*@") +_SCP_USERINFO = re.compile(r"^[^/@:]+@(?=[^/:]+:)") + + +# --- recognising an export ------------------------------------------------------ + + +def _first_line(data: bytes) -> bytes: + data = data[len(_BOM):] if data.startswith(_BOM) else data + return data.split(b"\n", 1)[0] + + +def is_export(data: bytes) -> bool: + """Whether `data` is a file this stack exported: its first line, after an + optional BOM, starts with `EXPORT_MARKER`. Reads no more than the first + line, so a caller may pass only a file's first bytes.""" + return _first_line(data).startswith(EXPORT_MARKER) + + +def guidelines_header(data: bytes) -> Optional[dict[str, str]]: + """The fields of a guideline export's first line - `instance`, `commit`, + whichever it carries - or None when `data` is not a guideline export at + all. A header with no fields (`{}`) is the bare opt-in stub.""" + line = _first_line(data).decode("utf-8", "replace") + if not line.startswith(GUIDELINES_PREFIX): + return None + return dict(_HEADER_FIELD.findall(line[len(GUIDELINES_PREFIX):])) + + +def file_is_export(path: Path) -> bool: + """`is_export` for a file on disk, reading only its first line's worth.""" + with open(path, "rb") as handle: + return is_export(handle.read(len(_BOM) + len(EXPORT_MARKER))) + + +# --- the instance it comes from ----------------------------------------------- + + +def _git(args: list[str], root: Path) -> Optional[str]: + """`git <args>` in the instance checkout: stdout on success, None on any + failure. Only local reads - nothing here reaches a remote.""" + try: + result = subprocess.run( + [toolpaths.git(), *args], cwd=root, capture_output=True, text=True, + encoding="utf-8", timeout=30, check=False, + ) + except (OSError, subprocess.SubprocessError): + return None + return result.stdout if result.returncode == 0 else None + + +def strip_userinfo(url: str) -> str: + """`url` without a user or token in front of the host - + `https://user:token@host/x` becomes `https://host/x`, `git@host:x` becomes + `host:x`. The header lands in other repositories, some of them public.""" + stripped = _SCHEME_USERINFO.sub(r"\1", url, count=1) + if stripped != url: + return stripped + return _SCP_USERINFO.sub("", url, count=1) + + +def instance_id(root: Optional[Path] = None) -> str: + """The `instance=` value: `origin`'s fetch URL without userinfo, or + `local` for a checkout without an `origin`.""" + url = (_git(["remote", "get-url", "origin"], root or config.ROOT) or "").strip() + if not url: + return LOCAL_INSTANCE + return "".join(strip_userinfo(url).split()) or LOCAL_INSTANCE + + +def git_identity(root: Optional[Path] = None) -> Optional[tuple[str, str]]: + """`(user.name, user.email)` as the instance checkout resolves them - its + local configuration before the global one - or None if either is unset. + The commit in a target repository is made in a bare cache repository, which + would see only the global configuration; this is the identity the + instance's own commits carry.""" + root = root or config.ROOT + name = (_git(["config", "user.name"], root) or "").strip() + email = (_git(["config", "user.email"], root) or "").strip() + return (name, email) if name and email else None + + +# --- selection ---------------------------------------------------------------- + + +def select(predicates: tuple[Predicate, ...], kb_dir: Optional[Path] = None, + root: Optional[Path] = None) -> list[Page]: + """The pages the predicates select, sorted by `(title.casefold(), title)`. + + Raises `ValidationError` instead of returning something incomplete: with + no predicate (a forgotten filter would export the whole wiki), on an + unknown field (as `search` does), on any page under `kb/` whose + frontmatter does not parse (it could be a guideline that silently drops + out), and on zero hits (an empty file would delete every repository's + guidelines).""" + if not predicates: + raise ValidationError( + "No predicate - the export takes the guidelines by frontmatter, and without a filter " + "it would take the whole wiki. kb/CONVENTIONS.md names this instance's filter." + ) + kb_dir = kb_dir or config.KB_DIR + pages = load_pages_by_path(kb_dir, root or config.ROOT) + unreadable = unreadable_pages(pages) + if unreadable: + listed = "\n".join(f" - {u['path']} ({u['reason']})" for u in unreadable) + raise ValidationError( + "These pages have frontmatter that does not parse, so the export cannot tell whether " + f"they are guidelines:\n{listed}\n Fix them first; nothing was exported." + ) + filters.validate_fields(predicates, pages) + selected = filters.apply_predicates(pages, predicates, kb_dir) + if not selected: + rendered = " ".join(p.render() for p in predicates) + raise ValidationError( + f"No page matches {rendered} - an empty export would delete every repository's " + "guidelines, so nothing was exported." + ) + return sorted(selected.values(), key=lambda p: (p.title.casefold(), p.title)) + + +def check_clean(root: Optional[Path] = None) -> None: + """Refuse a checkout that is not a git repository, or whose `kb/` or + `types/` has uncommitted changes, untracked files included - otherwise + `commit=` would name a state the export was not made from. `types/` counts + because `kind` and `subtype` are resolved through the type-specs.""" + root = root or config.ROOT + if corpus_cache.head_commit(root) is None: + raise ValidationError( + f"{root} is not a git checkout with a commit - the export names the commit its pages " + "come from." + ) + dirty = [ + Path(path).relative_to(root).as_posix() if Path(path).is_relative_to(root) else str(path) + for path in (config.KB_DIR, config.TYPES_DIR) + if corpus_cache.is_dirty(root, path) + ] + if dirty: + raise ValidationError( + f"{' and '.join(dirty)} have uncommitted changes (untracked files count). Publish them " + "first - the export names the commit its pages come from." + ) + + +def last_commit(pages: list[Page], root: Optional[Path] = None) -> str: + """The newest commit touching one of `pages`' files - not `HEAD`, so an + instance commit that touched no guideline leaves the output unchanged.""" + root = root or config.ROOT + paths = [Path(p.path).resolve().relative_to(Path(root).resolve()).as_posix() for p in pages] + sha = (_git(["log", "-1", "--format=%H", "--", *paths], root) or "").strip() + if not sha: + raise ValidationError("None of the selected pages is in a commit yet - publish them first.") + return sha + + +# --- rendering ---------------------------------------------------------------- + + +def _inline(text: str) -> str: + """Wikilinks to their display text; `[^cite-id]` and legacy `^[[...]]` + citations removed - all of it outside code only. Matches are found on the + masked copy and applied to the original, which share offsets.""" + masked = strip_code_spans(text) + edits: list[tuple[int, int, str]] = [] + for m in provenance.LEGACY_CITE_RE.finditer(masked): + edits.append((m.start(), m.end(), "")) + for m in provenance.CITE_REF_RE.finditer(masked): + edits.append((m.start(), m.end(), "")) + for m in LINK_RE.finditer(masked): + rest = text[m.start(2):m.end(2)] + display = ( + rest.split("|", 1)[1] if "|" in rest else normalize_link_target(text[m.start(1):m.end(1)]) + ) + edits.append((m.start(), m.end(), display)) + # A legacy citation contains a wikilink; the one starting first wins. + edits.sort(key=lambda e: (e[0], -e[1])) + out: list[str] = [] + pos = 0 + for start, end, replacement in edits: + if start < pos: + continue + out.append(text[pos:start]) + out.append(replacement) + pos = end + out.append(text[pos:]) + return "".join(out) + + +def _trim_blank(lines: list[str]) -> list[str]: + start, end = 0, len(lines) + while start < end and not lines[start].strip(): + start += 1 + while end > start and not lines[end - 1].strip(): + end -= 1 + return lines[start:end] + + +def render_page(page: Page) -> str: + """One page: `# <title>`, then its text without the generated links and + footnote blocks, citations removed and wikilinks resolved to text. A + leading H1 of the page's own is replaced by the title heading; every other + heading keeps its level.""" + body = page.body.replace("\r\n", "\n") + body = blocks.strip(body, blocks.LINKS) + body, _definitions = provenance.split_cite_block(body) + lines = _trim_blank(_inline(body).split("\n")) + if lines and _H1.match(lines[0]): + lines = _trim_blank(lines[1:]) + return "\n".join([f"# {page.title}", "", *lines]) if lines else f"# {page.title}" + + +def header(instance: str, commit: str) -> str: + return ( + f"{GUIDELINES_PREFIX} instance={instance} commit={commit} " + "- generated, do not edit by hand -->" + ) + + +def render(pages: list[Page], instance: str, commit: str) -> str: + """The whole file: the header line, then each page, one blank line + between. LF line endings, exactly one trailing newline.""" + parts = [header(instance, commit), *(render_page(p) for p in pages)] + return "\n\n".join(parts) + "\n" + + +@dataclass(frozen=True) +class Export: + text: str + pages: tuple[Page, ...] + instance: str + commit: str + + @property + def data(self) -> bytes: + return self.text.encode("utf-8") + + +def build(predicates: tuple[Predicate, ...], root: Optional[Path] = None) -> Export: + """Select, check and render in one go. Raises `ValidationError`.""" + root = root or config.ROOT + pages = select(predicates, root=root) + check_clean(root) + commit = last_commit(pages, root) + instance = instance_id(root) + return Export(render(pages, instance, commit), tuple(pages), instance, commit) diff --git a/tools/chemenu/kb_scan.py b/tools/chemenu/kb_scan.py index 0bfb649..9426e35 100644 --- a/tools/chemenu/kb_scan.py +++ b/tools/chemenu/kb_scan.py @@ -12,6 +12,14 @@ from chemenu.page import Page WIKILINK_RE = re.compile(r"\[\[([^\]|#]+)") +# The whole link: `[[Target]]`, `[[Target|alias]]`, `[[Target#anchor]]` - +# including the `[[Target]]` inside a `[^cite-id]: [[Target]]` Footnotes +# definition, which is exactly what lets `page_ops.retarget_body()` repoint a +# citation's link target on a rename. Group 1 is the target title; group 2 keeps +# any alias/anchor suffix untouched. Group 1 may span a line break; it is +# compared through `normalize_link_target()`, the same reading `lint` gives it. +LINK_RE = re.compile(r"\[\[([^\[\]|#]+)((?:[|#][^\[\]]*)?)\]\]") + # A line break inside `[[...]]`, with the indentation around it. The target # class above admits a newline, so a link someone wrapped at a fixed column - # `[[Foo Bar\n Target]]` - captures the break as part of the title, matches no diff --git a/tools/chemenu/repo_capture.py b/tools/chemenu/repo_capture.py index 414dc86..af3e365 100644 --- a/tools/chemenu/repo_capture.py +++ b/tools/chemenu/repo_capture.py @@ -1,10 +1,12 @@ """Capturing documentation from a git repository for `raw/`: resolve a ref rule to one commit, read the files the globs select straight out of that -commit, and describe the result in a manifest - with no CLI attached. +commit, and describe the result in a manifest - with no CLI attached. And the +one write in the other direction: a single commit that changes one file on a +captured repository's branch, for `export guidelines --push`. -`wikitool raw capture`, `raw status` and `raw accept --replaces-bundle` are the -terminal adapters over this module; it decides nothing about `incoming/` or -`raw/` paths beyond the manifest's own name. +`wikitool raw capture`, `raw status`, `raw accept --replaces-bundle` and +`export guidelines` are the terminal adapters over this module; it decides +nothing about `incoming/` or `raw/` paths beyond the manifest's own name. Three properties carry the design, and each has a test: @@ -39,6 +41,7 @@ from typing import Iterator, Optional from chemenu import config, filelock, toolpaths, web_capture from chemenu.errors import BackendError, ValidationError +from chemenu.guideline_export import EXPORT_MARKER MANIFEST_NAME = "_capture.json" SCHEMA = 1 @@ -52,7 +55,6 @@ ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https") GIT_TIMEOUT_SECONDS = 120.0 -EXPORT_MARKER = b"<!-- wikitool:export" LFS_MARKER = b"version https://git-lfs.github.com/spec/v1" _BOM = b"\xef\xbb\xbf" @@ -199,21 +201,36 @@ def _git_env() -> dict[str, str]: return env -def run_git( +@dataclass(frozen=True) +class GitResult: + returncode: int + stdout: bytes + stderr: bytes + + def last_error_line(self) -> str: + message = self.stderr.decode("utf-8", "replace").strip().splitlines() + return message[-1] if message else f"exit {self.returncode}" + + +def run_git_result( args: list[str], git_dir: Path, *, stdin: Optional[bytes] = None, + env: Optional[dict[str, str]] = None, timeout: float = GIT_TIMEOUT_SECONDS, -) -> bytes: - """Run `git --git-dir=<git_dir> <args>` and return its stdout. +) -> GitResult: + """Run `git --git-dir=<git_dir> <args>` and return its exit code and both + streams, whatever the exit code - for a caller that reads stdout on a + failure too (`push --porcelain` reports a rejected ref there, with exit 1). No call can prompt: no terminal prompt, no askpass program (an empty `core.askPass` stops git from falling back to `SSH_ASKPASS`), and on POSIX a session of its own, so `ssh` has no controlling terminal to ask on either. A timeout kills the whole process group, `ssh` included - killing `git` - alone would leave the pipe open and the read below hanging. Raises - `BackendError` on a non-zero exit, a timeout or a git that cannot start.""" + alone would leave the pipe open and the read below hanging. `env` is laid + over that environment. Raises `BackendError` on a timeout or a git that + cannot start.""" argv = [toolpaths.git(), "-c", "core.askPass=", f"--git-dir={git_dir}", *args] posix = os.name == "posix" try: @@ -222,7 +239,7 @@ def run_git( stdin=subprocess.PIPE if stdin is not None else subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - env=_git_env(), + env={**_git_env(), **(env or {})}, start_new_session=posix, ) except OSError as exc: @@ -239,10 +256,23 @@ def run_git( proc.kill() proc.communicate() raise BackendError(f"git {args[0]} did not finish within {timeout:.0f} s") from None - if proc.returncode != 0: - message = err.decode("utf-8", "replace").strip().splitlines() - raise BackendError(f"git {args[0]} failed: {message[-1] if message else f'exit {proc.returncode}'}") - return out + return GitResult(proc.returncode, out, err) + + +def run_git( + args: list[str], + git_dir: Path, + *, + stdin: Optional[bytes] = None, + env: Optional[dict[str, str]] = None, + timeout: float = GIT_TIMEOUT_SECONDS, +) -> bytes: + """`run_git_result`, returning stdout and raising `BackendError` on a + non-zero exit as well.""" + result = run_git_result(args, git_dir, stdin=stdin, env=env, timeout=timeout) + if result.returncode != 0: + raise BackendError(f"git {args[0]} failed: {result.last_error_line()}") + return result.stdout def cache_root() -> Path: @@ -470,6 +500,84 @@ def remote_commit(url: str, rule: str) -> ResolvedRef: return resolve_ref(url, rule, repo) +# --- writing one file back ---------------------------------------------------- +# +# `export guidelines --push` - the only write into another repository. It needs +# no working tree and touches none: blob, tree and commit are built with +# plumbing in the bare cache repository, on top of the tip `fetch` brought in, +# and pushed by commit id. The cache's shallow history is enough, because the +# server already has the parent. + + +def read_path(repo: Path, commit: str, path: str) -> Optional[bytes]: + """The bytes of the file at repository path `path` in `commit`, or None + if there is no regular file there.""" + out = run_git(["ls-tree", "-z", commit, "--", path], repo) + entry = out.split(b"\0", 1)[0] + if not entry: + return None + meta, _, _name = entry.partition(b"\t") + mode, kind, sha = meta.decode("ascii").split() + if kind != "blob" or mode == _MODE_SYMLINK: + return None + return read_blobs(repo, [sha])[sha] + + +def blob_id(repo: Path, data: bytes) -> str: + """The object id `data` has as a blob in `repo`, without writing it.""" + return run_git(["hash-object", "--no-filters", "--stdin"], repo, stdin=data).decode().strip() + + +def commit_file( + repo: Path, parent: str, path: str, data: bytes, message: str, identity: tuple[str, str] +) -> str: + """A new commit on `parent` whose tree differs from the parent's only in + `path`, now holding `data` - built in a temporary index, so nothing else of + the tree can change. The file keeps an executable bit it already had. + `identity` is `(name, email)` for author and committer alike. Returns the + new commit's id; nothing is pushed.""" + blob = run_git(["hash-object", "-w", "--no-filters", "--stdin"], repo, stdin=data).decode().strip() + existing = run_git(["ls-tree", "-z", parent, "--", path], repo).split(b"\0", 1)[0] + mode = "100755" if existing.startswith(b"100755 ") else "100644" + name, email = identity + index = repo / f"wikitool-export-{os.getpid()}.index" + env = { + "GIT_INDEX_FILE": str(index), + "GIT_AUTHOR_NAME": name, "GIT_AUTHOR_EMAIL": email, + "GIT_COMMITTER_NAME": name, "GIT_COMMITTER_EMAIL": email, + } + try: + run_git(["read-tree", parent], repo, env=env) + run_git(["update-index", "--add", "--cacheinfo", f"{mode},{blob},{path}"], repo, env=env) + tree = run_git(["write-tree"], repo, env=env).decode().strip() + return run_git(["commit-tree", tree, "-p", parent, "-m", message], repo, env=env).decode().strip() + finally: + index.unlink(missing_ok=True) + + +@dataclass(frozen=True) +class PushResult: + accepted: bool + detail: str # git's own summary for the ref, e.g. "[rejected] (fetch first)" + + +def push_commit(url: str, repo: Path, commit: str, branch: str) -> PushResult: + """Push `commit` to `refs/heads/<branch>` at `url` - never forced, so a + branch that moved since the fetch rejects it and stays where it is. + + Accepted or rejected is read from `--porcelain`'s status flag for the ref + (`!` is rejected), never from the wording on stderr, which git translates + into the host's locale. Raises `BackendError` when there is no status for + the ref at all - the remote was not reached.""" + ref = f"refs/heads/{branch}" + result = run_git_result(["push", "--porcelain", "--", url, f"{commit}:{ref}"], repo) + for line in result.stdout.decode("utf-8", "replace").splitlines(): + fields = line.split("\t") + if len(fields) >= 3 and fields[1].endswith(f":{ref}") and fields[0] in (" ", "*", "=", "!", "+", "-"): + return PushResult(fields[0] != "!", fields[2].strip()) + raise BackendError(f"git push failed: {result.last_error_line()}") + + # --- manifest ----------------------------------------------------------------- @@ -537,6 +645,18 @@ def read_manifest(path: Path) -> Manifest: ) +def captured_manifests(raw_dir: Path) -> list[Path]: + """Every captured bundle's `_capture.json` under `raw_dir`, sorted, none + below a hidden directory - the one list both `raw status` and + `export guidelines` work from.""" + if not raw_dir.is_dir(): + return [] + return [ + p for p in sorted(raw_dir.rglob(MANIFEST_NAME)) + if p.is_file() and not any(part.startswith(".") for part in p.relative_to(raw_dir).parts) + ] + + def bundle_files(bundle: Path) -> dict[str, Path]: """Every file of a captured bundle on disk, keyed by its repository path - the manifest itself left out.""" diff --git a/tools/chemenu/search/filters.py b/tools/chemenu/search/filters.py index 4e571e0..699d7dc 100644 --- a/tools/chemenu/search/filters.py +++ b/tools/chemenu/search/filters.py @@ -70,6 +70,26 @@ def parse_predicate(raw: str) -> Predicate: ) +def raw_predicates( + fields: Optional[list[str]] = None, + kind: Optional[str] = None, + subtype: Optional[str] = None, + collection: Optional[str] = None, + tag: Optional[str] = None, +) -> list[str]: + """The `--field` arguments a call's shorthand options stand for, after its + own `--field` values: `--kind/--subtype/--collection X` is `<name>=X`, + `--tag X` is `tags=X`. Every command taking these options goes through + here, so the shorthands mean the same thing everywhere.""" + out = list(fields or []) + for value, name in ((kind, "kind"), (subtype, "subtype"), (collection, "collection")): + if value: + out.append(f"{name}={value}") + if tag: + out.append(f"tags={tag}") + return out + + def known_fields(pages: dict[str, Page]) -> set[str]: """Every field name a predicate may legitimately name: the union of all frontmatter keys actually present in the corpus, plus the virtual ones.""" diff --git a/tools/chemenu/tests/test_cli.py b/tools/chemenu/tests/test_cli.py index e9cd2f6..1adf031 100644 --- a/tools/chemenu/tests/test_cli.py +++ b/tools/chemenu/tests/test_cli.py @@ -340,13 +340,15 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list since `--latest`, which asks the release feed and downloads from it, and `raw fetch` since it exists (Gitea #120) - its `--html` form stays offline, which does not turn the - command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177). Pinned as an explicit + command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177), and so + does `export guidelines --push` (Gitea #179). Pinned as an explicit set so a command gaining or losing that reach is a deliberate edit here, not a silent drift between the property and what the command actually does.""" expected = { "raw fetch", "raw capture", "raw status", + "export guidelines", "sync", "publish", "version check", diff --git a/tools/chemenu/tests/test_export_guidelines.py b/tools/chemenu/tests/test_export_guidelines.py new file mode 100644 index 0000000..c7f5e25 --- /dev/null +++ b/tools/chemenu/tests/test_export_guidelines.py @@ -0,0 +1,553 @@ +"""`export guidelines` and the Guideline Push Gate, against local repositories +over `file://` - never the network. + +The instance is the `kb_dir` fixture tree made into a git checkout with its own +identity and a copy of the shipped `types/`, so the cleanliness check sees the +same two directories it sees in a real instance. Target repositories are bare +repositories, each fed from a work clone, because a push into a checked-out +branch is refused by git itself. +""" +from __future__ import annotations + +import json +import re +import shutil +import subprocess +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from chemenu import config, guideline_export, repo_capture +from chemenu.cli import app +from chemenu.commands import export_cmd +from chemenu.commands.raw_cmd import raw_accept_command, raw_capture_command, raw_status_command +from chemenu.frontmatter_io import write_page +from chemenu.markdown_code import strip_code_spans +from chemenu.search import filters +from chemenu.search.registry import resolve +from chemenu.search.service import load_pages_by_path, run_search +from chemenu.search.types import SearchQuery +from chemenu.type_resolver import resolver + +runner = CliRunner() + +TAG = ["--tag", "guideline"] + + +def _git(cwd: Path, *args: str, check: bool = True) -> str: + return subprocess.run( + ["git", "-c", "user.name=Fixture", "-c", "user.email=f@example.org", *args], + cwd=cwd, check=check, capture_output=True, text=True, + ).stdout.strip() + + +def _bare(bare: Path, *args: str) -> str: + return subprocess.run( + ["git", f"--git-dir={bare}", *args], check=True, capture_output=True, text=True + ).stdout.strip() + + +class Target: + """A bare repository - what the export pushes into - fed from a work + clone that plays the project's own developers.""" + + def __init__(self, base: Path, guidelines: bytes | None = None) -> None: + self.work = base / "work" + self.bare = base / "remote.git" + self.work.mkdir(parents=True) + _git(self.work, "init", "-q", "-b", "main") + (self.work / "README.md").write_text("# Project\n", encoding="utf-8") + (self.work / "docs").mkdir() + (self.work / "docs" / "a.md").write_text("# A\n", encoding="utf-8") + if guidelines is not None: + (self.work / "GUIDELINES.md").write_bytes(guidelines) + self.commit("initial") + _git(base, "clone", "-q", "--bare", str(self.work), str(self.bare)) + _git(self.work, "remote", "add", "origin", self.url) + + @property + def url(self) -> str: + return self.bare.as_uri() + + def commit(self, message: str = "change") -> None: + _git(self.work, "add", "-A") + _git(self.work, "commit", "-q", "--allow-empty", "-m", message) + + def advance(self, message: str = "upstream change") -> str: + """A commit by someone else, pushed to the remote branch.""" + _git(self.work, "pull", "-q", "--ff-only", "origin", "main", check=False) + (self.work / "docs" / "a.md").write_text(f"# A\n\n{message}\n", encoding="utf-8") + self.commit(message) + _git(self.work, "push", "-q", "origin", "HEAD:main") + return self.tip() + + def tip(self) -> str: + return _bare(self.bare, "rev-parse", "refs/heads/main") + + def refs(self) -> str: + return _bare(self.bare, "for-each-ref", "--format=%(refname) %(objectname)") + + def file(self, rev: str = "refs/heads/main") -> bytes: + return subprocess.run( + ["git", f"--git-dir={self.bare}", "show", f"{rev}:GUIDELINES.md"], + check=True, capture_output=True, + ).stdout + + +STUB = b"<!-- wikitool:export kind=guidelines -->\n" + +ALPHA_BODY = """ +# Alpha Rule + +Keep secrets in `.env` only[^s-handbook]. See [[Beta Rule]] and [[Beta Rule|the beta]], +also [[Beta Rule#Scope]] and [[Beta Rule#Scope|its scope]]. Legacy^[[Source - Handbook]] too. + +## Details + +Inline `[[Not A Link]]` and `[^not-a-cite]` stay as written. + +``` +[[Fenced|kept]] and [^fenced] and ^[[Fenced]] +``` + +<!-- wikitool:links --> +## Beziehungen + +- **related:** [[Beta Rule]] +<!-- /wikitool:links --> + +<!-- wikitool:footnotes --> +## Fußnoten + +[^s-handbook]: [[Source - Handbook]] +<!-- /wikitool:footnotes --> +""" + +BETA_BODY = """ +Beta has no heading of its own. + +### Scope + +Everything. + +## Footnotes + +[^s-handbook]: [[Source - Handbook]] +""" + + +def _guideline(kb: Path, title: str, body: str, tags=("guideline",)) -> Path: + path = kb / "concepts" / f"{title}.md" + write_page(path, {"type": "types/concept.md", "tags": list(tags), "summary": f"{title}."}, body) + return path + + +def _commit_instance(root: Path, message: str = "instance change") -> None: + _git(root, "add", "-A") + _git(root, "commit", "-q", "-m", message) + + +@pytest.fixture +def instance(kb_dir, monkeypatch): + root = kb_dir.parent + shutil.copytree(config._PACKAGE_ROOT / "types", root / "types") + monkeypatch.setattr(config, "TYPES_DIR", root / "types") + monkeypatch.setattr(resolver, "_repo_root", root) + monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file")) + (root / "raw").mkdir(exist_ok=True) + (root / "incoming").mkdir(exist_ok=True) + (root / ".gitignore").write_text("tools/\n", encoding="utf-8") + _guideline(kb_dir, "Alpha Rule", ALPHA_BODY) + _guideline(kb_dir, "beta rule", BETA_BODY) + _git(root, "init", "-q", "-b", "main") + _git(root, "config", "user.name", "Instance Author") + _git(root, "config", "user.email", "author@example.org") + _commit_instance(root, "instance") + return root + + +def _manifest(root: Path, name: str, repo: str, ref: str = "main") -> Path: + bundle = root / "raw" / "2026" / "10" / name + bundle.mkdir(parents=True) + data = { + "schema": 1, "repo": repo, "ref": ref, "commit": "0" * 40, "paths": ["docs/**"], + "captured": "2026-10-05T00:00:00Z", "fidelity": "verbatim", "authority": "normative", + "files": [], + } + (bundle / repo_capture.MANIFEST_NAME).write_text(json.dumps(data), encoding="utf-8") + return bundle + + +@pytest.fixture +def target(instance, tmp_path): + t = Target(tmp_path / "targets" / "one", STUB) + _manifest(instance, "one", t.url) + _commit_instance(instance, "capture one") + return t + + +def _run(*args: str): + return runner.invoke(app, ["export", "guidelines", *args]) + + +def _token(output: str) -> str: + match = re.search(r"--confirm (\w+) --push", output) + assert match, output + return match.group(1) + + +def _push(*extra: str): + """Gate run, then the confirmed run - what an approved push looks like.""" + gated = _run(*TAG, "--push", *extra) + assert gated.exit_code == 42, gated.output + return _run(*TAG, "--push", "--confirm", _token(gated.output), *extra) + + +# --- selection ---------------------------------------------------------------- + + +def test_selection_is_searchs_own(instance, kb_dir): + _guideline(kb_dir, "Gamma Elsewhere", "\nNot a guideline.\n", tags=("other",)) + _commit_instance(instance) + result = _run(*TAG) + assert result.exit_code == 0, result.output + exported = re.findall(r"^# (.+)$", result.output, re.MULTILINE) + predicates = tuple(filters.parse_predicate(r) for r in filters.raw_predicates(tag="guideline")) + search = run_search(SearchQuery(text=None, predicates=predicates, limit=0), + load_pages_by_path(), resolve(None)) + expected = sorted((h.title for h in search.hits), key=lambda t: (t.casefold(), t)) + assert exported == expected == ["Alpha Rule", "beta rule"] + + +@pytest.mark.parametrize("args, needle", [ + ((), "No predicate"), + (("--tag", "nothing-has-this"), "No page matches"), + (("--field", "no_such_field=x"), "unknown field"), +]) +def test_refusals_print_no_export(instance, target, args, needle): + before = target.refs() + for extra in ((), ("--push",)): + result = _run(*args, *extra) + assert result.exit_code == 1, result.output + assert needle in result.output + assert "<!-- wikitool:export" not in result.output + assert target.refs() == before + + +def test_unreadable_frontmatter_anywhere_refuses(instance, kb_dir, target): + (kb_dir / "concepts" / "Broken.md").write_text("---\ntype: [unclosed\n---\nBody\n", encoding="utf-8") + _commit_instance(instance) + before = target.refs() + result = _run(*TAG, "--push") + assert result.exit_code == 1 + assert "Broken.md" in result.output and "<!-- wikitool:export" not in result.output + assert target.refs() == before + + +@pytest.mark.parametrize("change", ["modified", "untracked-kb", "untracked-types"]) +def test_dirty_kb_or_types_refuses(instance, kb_dir, target, change): + if change == "modified": + page = kb_dir / "concepts" / "Alpha Rule.md" + page.write_text(page.read_text(encoding="utf-8") + "\nmore\n", encoding="utf-8") + elif change == "untracked-kb": + (kb_dir / "concepts" / "draft.txt").write_text("x\n", encoding="utf-8") + else: + (instance / "types" / "draft.md").write_text("x\n", encoding="utf-8") + before = target.refs() + for extra in ((), ("--push",)): + result = _run(*TAG, *extra) + assert result.exit_code == 1, result.output + assert "uncommitted changes" in result.output + assert "<!-- wikitool:export" not in result.output + assert target.refs() == before + + +# --- rendering ---------------------------------------------------------------- + + +def _export(*args: str) -> str: + result = _run(*(args or TAG)) + assert result.exit_code == 0, result.output + return result.output + + +def test_rendering_resolves_links_and_drops_citations_outside_code(instance): + out = _export() + assert out.endswith("\n") and not out.endswith("\n\n") and "\r" not in out + assert "See Beta Rule and the beta,\nalso Beta Rule and its scope. Legacy too." in out + assert "Keep secrets in `.env` only." in out + assert "Inline `[[Not A Link]]` and `[^not-a-cite]` stay as written." in out + assert "```\n[[Fenced|kept]] and [^fenced] and ^[[Fenced]]\n```" in out + prose = strip_code_spans(out) + for forbidden in ("[[", "[^", "^[[", "wikitool:links", "wikitool:footnotes", "Beziehungen", + "Fußnoten", "Footnotes", "type: types/", "---\n"): + assert forbidden not in prose, forbidden + + +def test_headings(instance): + out = _export() + lines = out.split("\n") + assert lines[1] == "" and lines[2] == "# Alpha Rule" and lines[3] == "" + assert out.count("# Alpha Rule") == 1 + assert "\n## Details\n" in out and "\n### Scope\n" in out + assert "\n\n# beta rule\n\nBeta has no heading of its own." in out + + +def test_header_line(instance): + first = _export().split("\n", 1)[0] + commit = _git(instance, "log", "-1", "--format=%H", "--", "kb/concepts/Alpha Rule.md", + "kb/concepts/beta rule.md") + assert first == ( + f"<!-- wikitool:export kind=guidelines instance=local commit={commit} " + "- generated, do not edit by hand -->" + ) + + +def test_instance_is_origin_without_userinfo(instance): + _git(instance, "remote", "add", "origin", "https://user:s3cret@git.example.org/team/wiki.git") + first = _export().split("\n", 1)[0] + assert "instance=https://git.example.org/team/wiki.git " in first + assert "s3cret" not in first and "user" not in first + + +@pytest.mark.parametrize("url, expected", [ + ("https://user:tok@host/x.git", "https://host/x.git"), + ("ssh://git@host:2222/x.git", "ssh://host:2222/x.git"), + ("git@host:team/x.git", "host:team/x.git"), + ("https://host/x.git", "https://host/x.git"), +]) +def test_strip_userinfo(url, expected): + assert guideline_export.strip_userinfo(url) == expected + + +def test_output_is_deterministic_and_ignores_unrelated_commits(instance, kb_dir): + first = _export() + assert _export() == first + (kb_dir / "concepts" / "Unrelated.md").write_text( + "---\ntype: types/concept.md\ntags: [other]\n---\n\nx\n", encoding="utf-8" + ) + _commit_instance(instance, "unrelated") + assert _export() == first + + +# --- the gate and the push ---------------------------------------------------- + + +def test_push_without_identity_refuses_before_any_target(instance, target): + _git(instance, "config", "--unset", "user.email") + before = target.refs() + result = _run(*TAG, "--push") + assert result.exit_code == 1, result.output + assert "user.email" in result.output + assert not re.search(r"--confirm \w{12}", result.output) + assert "NEEDS USER CLEARANCE" not in result.output + assert target.refs() == before + + +def test_push_without_token_is_gated(instance, target, tmp_path): + hand = Target(tmp_path / "targets" / "hand", b"# Our own rules\n") + _manifest(instance, "hand", hand.url) + _commit_instance(instance) + before = (target.refs(), hand.refs()) + result = _run(*TAG, "--push") + assert result.exit_code == 42, result.output + assert (target.refs(), hand.refs()) == before + assert f"{target.url} main: new" in result.output + assert f"{hand.url} main: skipped" in result.output + assert "+# Alpha Rule" in result.output and "-<!-- wikitool:export kind=guidelines -->" in result.output + assert re.search(r"tools/wikitool export guidelines --confirm \w{12} --push --field tags=guideline", + result.output) + + +def test_confirmed_push_writes_one_commit_with_only_guidelines(instance, target): + old = target.tip() + stdout = _export() + result = _push() + assert result.exit_code == 0, result.output + new = target.tip() + assert f"written {old[:12]} -> {new[:12]}" in result.output + assert _bare(target.bare, "rev-parse", f"{new}^") == old + assert _bare(target.bare, "diff", "--name-only", old, new) == "GUIDELINES.md" + assert target.file() == stdout.encode("utf-8") + assert _bare(target.bare, "log", "-1", "--format=%an <%ae>|%cn <%ce>", new) == ( + "Instance Author <author@example.org>|Instance Author <author@example.org>" + ) + commit = stdout.split("commit=", 1)[1][:12] + assert _bare(target.bare, "log", "-1", "--format=%s", new) == ( + f"Update GUIDELINES.md from local at {commit}" + ) + + again = _run(*TAG, "--push") + assert again.exit_code == 0, again.output + assert f"{target.url} main: unchanged" in again.output + assert "--confirm" not in again.output + assert target.tip() == new + + +def test_stale_tokens_are_gated_again(instance, kb_dir, target, tmp_path): + token = _token(_run(*TAG, "--push").output) + + target.advance() + moved = _run(*TAG, "--push", "--confirm", token) + assert moved.exit_code == 42 and "does not match" in moved.output + token = _token(moved.output) + + page = kb_dir / "concepts" / "beta rule.md" + page.write_text(page.read_text(encoding="utf-8") + "\nAmended.\n", encoding="utf-8") + _commit_instance(instance, "amend beta") + changed = _run(*TAG, "--push", "--confirm", token) + assert changed.exit_code == 42 and "+Amended." in changed.output + token = _token(changed.output) + + second = Target(tmp_path / "targets" / "two", STUB) + _manifest(instance, "two", second.url) + _commit_instance(instance, "capture two") + added = _run(*TAG, "--push", "--confirm", token) + assert added.exit_code == 42 and f"{second.url} main: new" in added.output + + before = (target.refs(), second.refs()) + invented = _run(*TAG, "--push", "--confirm", "000000000000") + assert invented.exit_code == 42 + assert (target.refs(), second.refs()) == before + + +def test_targets_that_do_not_take_part_are_left_alone(instance, target, tmp_path): + hand = Target(tmp_path / "targets" / "hand", b"# Our own rules\n") + none = Target(tmp_path / "targets" / "none") + foreign = Target( + tmp_path / "targets" / "foreign", + b"<!-- wikitool:export kind=guidelines instance=https://other.example/wiki commit=abc -->\n", + ) + tagged = Target(tmp_path / "targets" / "tagged", STUB) + for name, t, ref in (("hand", hand, "main"), ("none", none, "main"), + ("foreign", foreign, "main"), ("tagged", tagged, "v*")): + _manifest(instance, name, t.url, ref) + _commit_instance(instance) + before = {t.url: t.refs() for t in (hand, none, foreign, tagged)} + result = _push() + assert result.exit_code == 0, result.output + assert "hand-written" in result.output + assert "not opted in" in result.output + assert "another instance (https://other.example/wiki)" in result.output + assert "tag pattern" in result.output + assert {t.url: t.refs() for t in (hand, none, foreign, tagged)} == before + assert "written" in result.output # the participating target was still served + + +def test_a_branch_that_moved_is_rejected_not_forced(instance, target, monkeypatch): + # Translated git messages must not change the verdict: it is read from the + # porcelain status flag, not from the text. + monkeypatch.setenv("LANGUAGE", "de") + monkeypatch.setenv("LANG", "de_DE.UTF-8") + token = _token(_run(*TAG, "--push").output) + real_commit_file = repo_capture.commit_file + foreign: list[str] = [] + + def commit_then_race(*args, **kwargs): + foreign.append(target.advance("raced in")) + return real_commit_file(*args, **kwargs) + + monkeypatch.setattr(repo_capture, "commit_file", commit_then_race) + result = _run(*TAG, "--push", "--confirm", token) + assert result.exit_code == 1, result.output + assert f"{target.url} main: rejected" in result.output + assert target.tip() == foreign[0] + + +def test_an_unreachable_target_does_not_stop_the_others(instance, target, tmp_path): + _manifest(instance, "gone", (tmp_path / "does-not-exist.git").as_uri()) + _commit_instance(instance) + result = _push() + assert result.exit_code == 0, result.output + assert "does-not-exist.git main: skipped (not reachable" in result.output + assert f"{target.url} main: written" in result.output + + +def test_bundle_narrows_the_targets(instance, target, tmp_path): + other = Target(tmp_path / "targets" / "other", STUB) + _manifest(instance, "other", other.url) + _commit_instance(instance) + before = other.refs() + result = _push("--bundle", "raw/2026/10/one") + assert result.exit_code == 0, result.output + assert other.url not in result.output + assert other.refs() == before + unknown = _run(*TAG, "--push", "--bundle", "raw/2026/10/nope") + assert unknown.exit_code == 1 and "not a captured bundle" in unknown.output + + +def test_bundle_and_confirm_need_push(instance): + assert _run(*TAG, "--confirm", "abc").exit_code == 1 + assert _run(*TAG, "--bundle", "raw/x").exit_code == 1 + + +def test_raw_status_does_not_report_a_pushed_export(instance, tmp_path, capsys): + t = Target(tmp_path / "targets" / "captured", STUB) + raw_capture_command(repo_url=t.url, ref="main", paths=["**/*.md"], name="captured", + fidelity="verbatim", authority="normative", update=None) + raw_accept_command(files=[instance / "incoming" / "captured"], fidelity=None, authority=None, + page=None, replaces=None, dry_run=False) + _commit_instance(instance, "capture") + capsys.readouterr() + result = _push() + assert result.exit_code == 0 and "written" in result.output, result.output + raw_status_command(json_out=True) + rows = json.loads(capsys.readouterr().out) + assert len(rows) == 1 and rows[0]["error"] is None and rows[0]["changed"] is False + + +# --- no way back in ----------------------------------------------------------- + + +def test_export_marker_is_defined_once(): + package = Path(guideline_export.__file__).parent + definitions = [ + path for path in package.rglob("*.py") + if "tests" not in path.parts and re.search(r'= b"<!-- wikitool:export"', path.read_text("utf-8")) + ] + assert definitions == [Path(guideline_export.__file__)] + + +def test_a_real_export_is_excluded_from_capture(instance): + data = _export().encode("utf-8") + assert repo_capture._content_exclusion(data) is not None + assert repo_capture._content_exclusion(b"\xef\xbb\xbf" + data) is not None + + +def _snapshot(*dirs: Path) -> dict[str, bytes]: + return {p.as_posix(): p.read_bytes() for d in dirs for p in sorted(d.rglob("*")) if p.is_file()} + + +def test_raw_accept_refuses_an_export_file_and_a_folder_holding_one(instance): + incoming, raw = instance / "incoming", instance / "raw" + export = _export().encode("utf-8") + (incoming / "GUIDELINES.md").write_bytes(export) + (incoming / "pack").mkdir() + (incoming / "pack" / "notes.md").write_text("# Notes\n", encoding="utf-8") + (incoming / "pack" / "sub").mkdir() + (incoming / "pack" / "sub" / "rules.md").write_bytes(b"\xef\xbb\xbf" + export) + before = _snapshot(incoming, raw) + for path in (incoming / "GUIDELINES.md", incoming / "pack"): + result = runner.invoke(app, ["raw", "accept", str(path), "--fidelity", "verbatim", + "--authority", "normative"]) + assert result.exit_code == 1, result.output + assert "guideline export" in result.output + assert _snapshot(incoming, raw) == before + + +def test_is_export_reads_the_first_line_only(): + assert guideline_export.is_export(b"<!-- wikitool:export kind=x -->\nbody") + assert not guideline_export.is_export(b"# Title\n<!-- wikitool:export kind=x -->\n") + assert guideline_export.guidelines_header(STUB) == {} + assert guideline_export.guidelines_header(b"<!-- wikitool:export kind=other -->") is None + + +def test_confirm_token_ignores_targets_it_does_not_write(): + t = export_cmd.Target("u", "main") + writes = export_cmd.Plan(t, "new", tip="a", blob="b") + skipped = export_cmd.Plan(export_cmd.Target("v", "main"), "skipped", "x") + assert export_cmd.confirm_token([writes, skipped]) == export_cmd.confirm_token([writes]) + assert export_cmd.confirm_token([writes]) != export_cmd.confirm_token( + [export_cmd.Plan(t, "new", tip="a2", blob="b")] + )