feat: export guidelines - the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate (#179)
CI / verify (push) Successful in 5m40s
CI / pwsh (push) Successful in 1m59s
Release / release (push) Successful in 35s

Files changed:
- AGENTS.md
- CHANGES.md
- README.md
- VERSION
- docs/why-gates-are-code.md
- instructions/gates.md
- instructions/kb-profiles.md
- kb/CONTRACT.md
- kb/CONVENTIONS.md
- kb/CONVENTIONS.md.template
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/export_cmd.py
- tools/chemenu/commands/page_ops.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/commands/search.py
- tools/chemenu/guideline_export.py
- tools/chemenu/kb_scan.py
- tools/chemenu/repo_capture.py
- tools/chemenu/search/filters.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_export_guidelines.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-06 06:48:07 +02:00
1 parent ccede04b97
commit 283cdae8be
25 files changed
+1753 -68

No files matched your search

+6 -2
View File
@@ -152,7 +152,7 @@ exists, why silent overwrite is the failure it guards against, and why an instan
from a release),
[docs/language-boundaries.md](docs/language-boundaries.md) (why the control plane is English
everywhere and the KB language is a value, and why the axis is the reader rather than the
owner), [docs/why-gates-are-code.md](docs/why-gates-are-code.md) (why the four gates in
owner), [docs/why-gates-are-code.md](docs/why-gates-are-code.md) (why the five gates in
[Gates](#gates) are code rather than instruction),
[docs/version-model.md](docs/version-model.md) (why a version number answers a compatibility
question and a migration question separately), and
@@ -267,7 +267,7 @@ that take a title. A result cut short by `--limit` says so and names the total.
## Gates
Four limits are enforced in code rather than by instruction, because a prompt-level limit is
Five limits are enforced in code rather than by instruction, because a prompt-level limit is
one an agent can talk itself past.
- **Mass-Update Gate.** `publish` exits **42** on a change touching too many files, printing
@@ -279,6 +279,10 @@ one an agent can talk itself past.
- **Upload Review Gate.** `upload accept` exits **42** on an MCP `submit` tool submission
nobody has cleared yet, printing its manifest and the `--confirm <token>` line that promotes
it once the user approves - same shape as the Mass-Update Gate, one submission at a time.
- **Guideline Push Gate.** `export guidelines --push` exits **42** before writing the generated
`GUIDELINES.md` into any captured repository, printing every target's status, the diff for
each one it would write, and the `--confirm <token>` line that pushes exactly that set once the
user approves.
- **Iteration Budget Gate / Loop-Breaker.** Past 60 `wikitool` calls in a session, or after 3
identical calls in a row, further calls are refused.
+47 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.43 - 2026-10-05 - wiki-ingest: updating the captured repositories is a step-1 branch over raw status
## 8.0.0-beta.44 - 2026-10-06 - export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate
**Author:** Torben Nehmer
@@ -110,6 +110,7 @@ concern - readable here, never shipped as something to parse.
- Comparison and source pages accept the sources: that cite add writes; sources may cite sources
- raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
- wiki-ingest: updating the captured repositories is a step-1 branch over raw status
- export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate
**Low impact**
- version bump no longer points at version release in its output
@@ -156,6 +157,51 @@ concern - readable here, never shipped as something to parse.
- raw accept: an occupied folder name held by a captured bundle points at --replaces-bundle
<!-- /wikitool:bumps -->
### export guidelines: the guideline pages as a generated GUIDELINES.md, pushed into the captured repositories behind the Guideline Push Gate
The other direction of `raw capture`. Project repositories each carried their own copy of the
same agent rules, drifting apart; the instance now holds them once and delivers them as a
generated, committed file, so each repository works on its own - in CI, without MCP.
- **`export guidelines <predicates>`** selects pages with `search`'s own predicates and no text
(`--field`, `--kind`, `--subtype`, `--collection`, `--tag`), and prints one `GUIDELINES.md`.
Line 1 is `<!-- wikitool:export kind=guidelines instance=<origin URL without userinfo, or
local> commit=<newest commit touching a selected page> - generated, do not edit by hand -->`;
then each page by title as `# <title>` and its text without frontmatter, the generated links and
footnotes regions and citation markers, with wikilinks turned into their text - code left
byte for byte. No timestamp and not `HEAD`: the same pages give the same bytes, so an instance
commit that touches no guideline changes no target repository. It refuses with no predicate, no
match, an unknown field, any unreadable frontmatter under `kb/`, or uncommitted changes in `kb/`
or `types/`.
- **`--push`** writes that file into every captured repository - the `(repo, ref)` of each
`_capture.json`, `--bundle` narrowing it - that opted in by carrying a `GUIDELINES.md` whose
first line is the export header; a header line alone is the opt-in. A hand-written file, one
from another `instance=`, a tag rule, and an unreachable repository are skipped with the reason.
Each written target gets one commit on the fetched tip changing only `GUIDELINES.md`, built with
plumbing in the bare capture cache (no working tree), authored as the instance checkout's
`user.name`/`user.email`, and pushed without force; a branch that moved since the fetch is
rejected, read from `push --porcelain`'s status flag rather than git's translated messages.
- **The Guideline Push Gate** - a fifth named gate. `--push` without a matching `--confirm`
pushes nothing and exits 42 with each target's status, the diff of every file it would write,
and the re-run line; the token digests URL, branch, old tip and new blob of every target to
write. Nothing to write means no gate. The Publish-Remote Gate does not apply - the targets are
declared by the committed manifests already.
- **No way back in:** `raw accept` refuses a file whose first line starts with
`<!-- wikitool:export`, given on its own, inside a folder, or as `--replaces`/`--replaces-bundle`
material - `raw capture` already left such files out. `EXPORT_MARKER` now lives once, in the new
core module `guideline_export.py`.
Which pages are guidelines is an instance decision: `kb/CONVENTIONS.md` (and its template) has a
new section § Guidelines for other repositories; this instance's filter is `--tag guideline`.
`kb/CONTRACT.md` says what leaving the wiki does to a page; `raw/CONTRACT.md`, `AGENTS.md`
§ Gates, `instructions/gates.md`, `docs/why-gates-are-code.md`, `README.md` and
`tools/README.md` follow. Internals: `kb_scan.LINK_RE` (moved from `commands/page_ops`, which
re-exports it), `filters.raw_predicates` shared with `search`, `repo_capture.run_git_result` and
`captured_manifests` shared with `raw status`.
A new command and a new refusal for files no tool produced before - drop-in in both directions,
no page or manifest changes (Gitea #179).
### wiki-ingest: updating the captured repositories is a step-1 branch over raw status
`raw status`, `raw capture --update` and `raw accept --replaces-bundle` existed, but nothing told
+17
View File
@@ -227,6 +227,18 @@ tell the LLM `Update the captured repositories`, and it checks them, takes one c
run into the wiki and says how many are still behind. Git uses your own keys and credential
helpers - nothing is stored in the repository.
The same repositories can get something back: the wiki's guidelines, as one generated
`GUIDELINES.md` in their root. Which pages count as guidelines is your decision, written down in
`kb/CONVENTIONS.md` (§ Guidelines for other repositories); `tools/wikitool export guidelines
--tag guideline` prints the file as it would be written. A repository takes part by carrying a
`GUIDELINES.md` whose first line is `<!-- wikitool:export kind=guidelines -->` - commit that one
line there, and point the repository's own `AGENTS.md` at the file (Claude Code:
`@GUIDELINES.md`). A hand-written `GUIDELINES.md` is never touched. Then tell the LLM
`Roll out the guidelines to the captured repositories`: it runs `export guidelines --push`, which
stops before pushing anything (the **Guideline Push Gate**) and shows you, per repository, what
would change - the LLM puts every diff in front of you, and only your approval pushes one commit
per repository, changing only that file, straight onto its branch.
### Querying Knowledge
Ask questions naturally:
@@ -407,6 +419,11 @@ this" code, not an error - printing the full file list and the
`--confirm <token>` line that publishes it. The token digests that file list,
so a clearance never carries to a changeset the user did not see.
**Guideline push.** `tools/wikitool export guidelines --push` exits **42** the same way (the
**Guideline Push Gate**) before it writes `GUIDELINES.md` into any captured repository, printing
every target's status and diff and the `--confirm <token>` line. The token digests each target's
branch tip and the file, so a moved branch or an edited guideline asks again.
**Iteration/cost limits.** Every `tools/wikitool` call is checked against a
hard, code-enforced per-session budget before it runs (default: 60 calls, or
3 identical calls in a row) - not just a prompt instruction to stop. Past the
+1 -1
View File
@@ -1 +1 @@
8.0.0-beta.43
8.0.0-beta.44
+22 -10
View File
@@ -1,17 +1,18 @@
# Why gates are code
Chemenu has four hard limits - the Mass-Update Gate, the Publish-Remote Gate, the Upload Review
Gate, and the Iteration Budget Gate - and all four live inside `tools/wikitool`, not in a
Chemenu has five hard limits - the Mass-Update Gate, the Publish-Remote Gate, the Upload Review
Gate, the Guideline Push Gate, and the Iteration Budget Gate - and all five live inside
`tools/wikitool`, not in a
paragraph of instructions an agent reads and follows. The rules themselves, and what to do when
one trips, are in [AGENTS.md § Gates](../AGENTS.md#gates) and
[instructions/gates.md](../instructions/gates.md). This page is only about the design choice
underneath them: why code, and why these four mechanisms in particular.
underneath them: why code, and why these mechanisms in particular.
<!-- wikitool:toc -->
## Contents
- [A suggestion an agent can talk itself past](#a-suggestion-an-agent-can-talk-itself-past)
- [Why four different mechanisms, not one](#why-four-different-mechanisms-not-one)
- [Why different mechanisms, not one](#why-different-mechanisms-not-one)
- [Exit 42 is a posture, and it outgrew the gates](#exit-42-is-a-posture-and-it-outgrew-the-gates)
- [A gate in code still has to be reachable](#a-gate-in-code-still-has-to-be-reachable)
- [Numbers that come from measurement, not intuition](#numbers-that-come-from-measurement-not-intuition)
@@ -31,10 +32,10 @@ conversation at all. It runs before the command dispatches, regardless of how co
case for skipping it seemed a moment earlier. The difference isn't that code is smarter than a
well-written instruction - it's that code doesn't get talked into anything.
## Why four different mechanisms, not one
## Why different mechanisms, not one
The four gates ask four different questions, and each one's shape follows from what kind of
question it is.
The five gates ask four different kinds of question, and each one's shape follows from what kind
of question it is.
The Mass-Update Gate asks *is this change too large to publish unreviewed* - a judgment that
varies changeset by changeset, so it clears with a `--confirm` token tied to the specific
@@ -56,6 +57,17 @@ is answering, so nothing about the mechanism needed to change - only the boundar
did, since the submission lives in a quarantine the ordinary pipeline never reads at all rather
than in the working tree `publish` is about to commit.
The Guideline Push Gate asks the same question once more, at the boundary facing outwards: *is
this content right for these repositories, now?* `export guidelines --push` writes a generated
file straight onto another repository's branch, with no review on the receiving side, and a
public target publishes it the moment it lands. That is a per-run judgment - the guidelines
change, the set of opted-in repositories changes, a branch moves - so it takes the token shape,
digesting exactly what would be written where. It deliberately does not take the Publish-Remote
Gate's shape, although it too pushes to a remote: which repositories are targets is already a
standing, committed declaration - the capture manifests under `raw/` - and asking for the same
URLs in an allowlist as well would be a second declaration of one fact. What is left to ask is
the per-run question, and that is the one a token answers.
The Iteration Budget Gate asks a fourth kind of question - not "is this instance correct" but
"has this session stopped making progress." That's read from the shape of the call history
itself (call count, repeated identical calls), not from anything about the content of any one
@@ -63,13 +75,13 @@ call.
## Exit 42 is a posture, and it outgrew the gates
Those four are the named gates, and they are not the only thing that exits 42 any more. When the
Those five are the named gates, and they are not the only thing that exits 42 any more. When the
task-tracker provider layer arrived, it brought a case that looks like a gate from the outside and
is not one: a provider whose API cannot create a project (Super Productivity's local REST API
reads projects but does not write them) raises `HumanInterventionRequired`, and the command prints
what a human has to do and exits 42.
Reusing the code was deliberate, and so was not calling it a fifth gate. What the four gates share
Reusing the code was deliberate, and so was not calling it another gate. What the named gates share
is a *refusal*: the operation was possible and the tool declined to perform it unreviewed. This is
the opposite situation - the operation is not possible at all, and no token could make it
possible. What the two have in common is only what the exit code actually communicates: **stop,
@@ -106,7 +118,7 @@ That failure has no symptom of its own. A gate that fires announces that it exis
*cannot* fire looks identical to a gate nobody happened to need - the same clean runs, the same
silence - and what finally told the two apart was reading a trace for an unrelated reason. So
there is a third property to keep alongside living in code and carrying measured numbers: each
gate has to leave evidence that it can still fire. The three that clear by token or by a
gate has to leave evidence that it can still fire. The ones that clear by token or by a
deliberate edit have it by construction, because clearing one is a visible event in somebody's
terminal. The budget gate, whose ordinary outcome is silence, is the one that had to be given
it - which is why its session id now carries where it came from, into both the trace and
+41 -6
View File
@@ -1,7 +1,7 @@
---
type: types/instruction.md
name: gates
description: What to do when wikitool refuses a call - exit 42 (user clearance required) on publish, and the Iteration Budget Gate and loop-breaker on every command.
description: What to do when wikitool refuses a call - exit 42 (user clearance required) on publish, upload accept and export guidelines --push, and the Iteration Budget Gate and loop-breaker on every command.
---
# When a gate refuses a call
@@ -25,6 +25,7 @@ Read the exit code first - it says which of these applies:
- [Exit 42: user clearance required](#exit-42-user-clearance-required)
- [Publish-Remote Gate](#publish-remote-gate)
- [Upload Review Gate](#upload-review-gate)
- [Guideline Push Gate](#guideline-push-gate)
- [Iteration Budget Gate and loop-breaker](#iteration-budget-gate-and-loop-breaker)
- [Taking a new session id](#taking-a-new-session-id)
- [Scope](#scope)
@@ -33,12 +34,13 @@ Read the exit code first - it says which of these applies:
## Exit 42: user clearance required
A `wikitool` command that exits **42** is not reporting an error. It is refusing to act until a
human has *read its output*. Four gates use it today - the Mass-Update Gate (`publish`, on a
human has *read its output*. Five gates use it today - the Mass-Update Gate (`publish`, on a
change touching 10 or more counted files), the rebase-review gate (`sync` and `publish`, on
a rebase whose incoming commits touch a file this session is also changing), the
Publish-Remote Gate (`publish`, on a push to a target this checkout has not declared), and the
Upload Review Gate (`upload accept`, on a submission nobody has cleared yet) - but the
rule is about the exit code, not the command:
Publish-Remote Gate (`publish`, on a push to a target this checkout has not declared), the
Upload Review Gate (`upload accept`, on a submission nobody has cleared yet), and the Guideline
Push Gate (`export guidelines --push`, before a generated `GUIDELINES.md` goes into any captured
repository) - but the rule is about the exit code, not the command:
> **Copy the command's output into your reply - the substance of it, not a description of it -
> and stop.** Run no further commands in that turn.
@@ -104,7 +106,7 @@ checkout is in, WARNing only when there is more than one remote and no allowlist
file is an error rather than "no restriction": a corrupted safeguard must not read as a disabled
one.
**This gate has no `--confirm` token, on purpose.** The other three clear with a token because the
**This gate has no `--confirm` token, on purpose.** The others clear with a token because the
question they ask ("is this change right?") is one the agent can put to the user and the user can
answer for that one changeset. This one asks "does this content belong to that repository?", which
is a standing property of the checkout, not a per-push judgment. The way past it is for the user
@@ -133,6 +135,39 @@ Mass-Update Gate's review report versus this file's exit-42 procedure.
all** - rejecting needs no clearance, only accepting a stranger's file into the pipeline does. It
deletes the material and keeps only the reason and a sha256 in `mcp-upload/ledger.jsonl`.
### Guideline Push Gate
`wikitool export guidelines --push` writes this instance's guideline pages, rendered into one
generated `GUIDELINES.md`, straight onto the branch of every captured repository that opted in -
no pull request, no review on the other side. That is the one write this stack makes into a
repository it does not own, and on a public target the content is public the moment it lands. So
the first run fetches and builds everything, pushes nothing, and exits 42.
Its substance, which your reply reproduces in full:
- **Every target and its status** - `new`, `changed`, `unchanged`, or `skipped` with the reason
(a tag rule, not reachable, not opted in, a hand-written file, another instance's file). A
skipped line is part of the answer: the user may have expected that repository to be written.
- **The diff of `GUIDELINES.md` for every target it would write**, against what that repository
carries now. This is what the user approves - the text that will appear there, not a count of
repositories.
- **The re-run line** with `--confirm <token>`, which you run only after the user approved this
exact set.
The token digests each target's URL, branch, the tip the commit builds on, and the file's
content, so a branch that moved, a guideline edited since, or a newly captured repository makes
it stale: the next run is gated again with the current state. A run with nothing to write ends
with exit 0 and no gate.
**Not the Publish-Remote Gate.** That one stays on `publish`. The targets here are declared by
the committed `_capture.json` manifests under `raw/`; asking for the same URLs in
`.wikitool-remotes.json` as well would be a second declaration of the same thing. What this gate
asks instead is the per-run question - whether this content belongs in these repositories now -
which is the kind a token answers.
A push the branch moved under is rejected, never forced, and reported as `rejected`; running the
whole command again (and clearing it again) serves that repository.
## Iteration Budget Gate and loop-breaker
Every `wikitool` call is counted per session. Calls are refused past **60 in a session**, or
+1 -1
View File
@@ -55,7 +55,7 @@ is not a profile.
| File | Holds | Profiles below |
|---|---|---|
| `kb/CONVENTIONS.md` | Language, section headings, naming forms, tone, relationship labels, the hedging rule - once per instance | [Language profiles](#language-profiles) |
| `kb/CONVENTIONS.md` | Language, section headings, naming forms, tone, relationship labels, the hedging rule, which pages leave as guidelines - once per instance | [Language profiles](#language-profiles) |
| `kb/<name>/COLLECTION.md` | What one collection holds, its quality goal, its local linking and naming rules | [Collection profiles](#collection-profiles) |
2. **Copy the entry's text into the file**, then edit it until it is true of this instance.
+19 -1
View File
@@ -13,7 +13,7 @@ how it works, so it is identical everywhere and `dist export` ships it verbatim.
**What an instance decides for itself is next door, in
[kb/CONVENTIONS.md](CONVENTIONS.md)** - the language pages are written in, the headings its two
generated regions render under, the naming forms, the tone, the hedging rule. That file binds exactly as this one does; it is simply owned by the instance
generated regions render under, the naming forms, the tone, the hedging rule, which pages leave the wiki as guidelines. That file binds exactly as this one does; it is simply owned by the instance
rather than by the stack, so the distribution ships only its `.template` and the instance writes
the real one. Read both, plus the target collection's `kb/<name>/COLLECTION.md` (also
instance-owned), before writing or editing a page.
@@ -40,6 +40,7 @@ looks like) are in neither - they belong to the type-specs and are printed by
- [Generated regions](#generated-regions)
- [Linking](#linking)
- [Provenance and citation](#provenance-and-citation)
- [Pages that leave the wiki: guidelines](#pages-that-leave-the-wiki-guidelines)
- [What does not belong here](#what-does-not-belong-here)
<!-- /wikitool:toc -->
@@ -329,6 +330,23 @@ Every claim is either traceable to a raw file or explicitly marked as not.
If no raw file or existing page backs an answer, say so explicitly rather than synthesizing
one - and never file the synthesized version back into the wiki.
## Pages that leave the wiki: guidelines
`tools/wikitool export guidelines` renders a selection of pages into one generated
`GUIDELINES.md` and, behind the Guideline Push Gate, writes it into the captured repositories that
opted in ([raw/CONTRACT.md](../raw/CONTRACT.md#getting-a-repository-in-raw-capture)). Which pages
that is, is not decided here: the stack defines no type and no field for a guideline, and the
selection - a set of `search` predicates - is written down in
[kb/CONVENTIONS.md](CONVENTIONS.md) by the instance.
What the stack does decide is how a page reads once it has left. The export is mechanical:
frontmatter, the generated links and footnotes regions and every citation marker are dropped,
`[[Title|Text]]` becomes `Text` and `[[Title]]` becomes `Title`, and code is left untouched. So a
guideline has to stand on its own in another repository - without its links to follow and without
the sources behind it - and an edit to one reaches every target repository on the next export.
The file there is never edited by hand: the next export overwrites it, so a correction goes into
the page.
## What does not belong here
- Raw source material - it stays immutable under `raw/`.
+13
View File
@@ -35,6 +35,7 @@ those regions and nothing else. Nothing matches on this text.
- [Tone](#tone)
- [Relationship labels](#relationship-labels)
- [Hedging](#hedging)
- [Guidelines for other repositories](#guidelines-for-other-repositories)
- [Keeping this file honest](#keeping-this-file-honest)
<!-- /wikitool:toc -->
@@ -140,6 +141,18 @@ carry, not against a threshold.
This is `SOUL.md`'s existing standard ("Was nicht belegt ist, ist nicht gewusst, nur vermutet -
und wird auch so benannt"), applied to `kb/` without a number competing next to it.
## Guidelines for other repositories
`tools/wikitool export guidelines` renders the pages this filter selects into one generated
`GUIDELINES.md` for the repositories this instance captured (`raw/CONTRACT.md`). The stack defines
no type or field for a guideline - which pages are guidelines is this instance's decision, and it
is written down here and nowhere else.
**The filter is `--tag guideline`.** A page carries the tag when its content is a rule an agent
working in another repository should follow there as it stands - written so that it reads without
its links and citations, because the export turns `[[links]]` into plain text and drops every
footnote. No page in this corpus carries it yet.
## Keeping this file honest
Change it when a convention actually changes. `sections:` is safe to change at any time - the
+11
View File
@@ -34,6 +34,7 @@ marker pair, so a rename re-renders words and nothing else.
- [Tone](#tone)
- [Relationship labels](#relationship-labels)
- [Hedging](#hedging)
- [Guidelines for other repositories](#guidelines-for-other-repositories)
- [Keeping this file honest](#keeping-this-file-honest)
<!-- /wikitool:toc -->
@@ -112,6 +113,16 @@ sourced claim, in the KB language.}
(`raw/CONTRACT.md`'s `authority` axis) from one resting on `opinion`, and how it signals
disagreement between sources.}
## Guidelines for other repositories
`tools/wikitool export guidelines` renders the pages this filter selects into one generated
`GUIDELINES.md` for the repositories this instance captured (`raw/CONTRACT.md`). The stack defines
no type or field for a guideline - which pages are guidelines is this instance's decision, and it
is written down here and nowhere else.
{The selection, as `search` predicates - for example `--tag guideline` - or "none" if this
instance exports no guidelines.}
## Keeping this file honest
Change it when a convention actually changes. `wikitool doctor` FAILs on a missing or unfilled
+15 -3
View File
@@ -94,6 +94,14 @@ That is what keeps the clean-up safe: only directories the moves emptied are rem
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
both ways out - the whole folder, or the file moved up into `incoming/`.
**A guideline export never comes back in.** A file whose first line (after an optional BOM)
starts with `<!-- wikitool:export` is this stack's own output - `export guidelines` renders the
wiki's guideline pages into it for other repositories - and `raw accept` refuses it, given on its
own, inside a folder, or as `--replaces`/`--replaces-bundle` material, naming each one. Taken in,
it would turn the wiki's knowledge on its way out into a source the wiki then cites for the same
claims. `raw capture` leaves such a file out of a repository for the same reason
([below](#getting-a-repository-in-raw-capture)).
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
no type any more - the kind of source comes from its content, as `source_type:` on the source
page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs
@@ -276,9 +284,13 @@ names it by its path inside the bundle (`docs/runbook.md`), not by its base name
**`_capture.json` is the bundle's manifest** - `repo`, the `ref` rule, the `commit`, the `paths`
globs, when it was `captured`, `fidelity`, `authority` and the `files` list. It is the only
declaration that and how this instance follows the repository; there is no second configuration
file. The name is reserved: it is metadata of the bundle, not a source, so `sources coverage` and
`lint` never report it, no `raw_files:` lists it, and no file of that name is captured from a
repository or accepted from `incoming/` anywhere but at the top of a captured folder.
file. It is read in the other direction too: the captured repositories are exactly the ones
`export guidelines --push` writes this instance's `GUIDELINES.md` into - each one that opted in
by carrying the file, on the branch its ref rule names (a tag rule takes no commit, so such a
repository is skipped). The name is reserved: it is metadata of the bundle, not a source, so
`sources coverage` and `lint` never report it, no `raw_files:` lists it, and no file of that name
is captured from a repository or accepted from `incoming/` anywhere but at the top of a captured
folder.
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its
+72
View File
@@ -37,6 +37,7 @@ file end to end is for changing the CLI itself.
- [Provenance](#provenance)
- [Raw material and uploads](#raw-material-and-uploads)
- [Git](#git)
- [Guideline export](#guideline-export)
- [Workshop runs and session budget](#workshop-runs-and-session-budget)
- [Types, instructions and docs](#types-instructions-and-docs)
- [Telemetry](#telemetry)
@@ -125,6 +126,7 @@ upload accept write non-idempotent budget:counted exit:0,1,
upload reject write non-idempotent budget:counted exit:0,1 Delete a submission's material, keeping only its ledger trail.
sync write idempotent budget:counted exit:0,1,42 Fetch `<remote>/<branch>` and bring the local branch up to date with it.
publish write non-idempotent budget:counted exit:0,1,42 Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
export guidelines write idempotent budget:counted exit:0,1,42 Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
work new write non-idempotent budget:counted exit:0,1 Scaffold `work/<runkey>/` for one workshop run.
work close write non-idempotent budget:counted exit:0,1 Delete a finished workshop.
budget status read idempotent budget:exempt exit:0 Show the current session's `wikitool` call count and recent command history.
@@ -1643,6 +1645,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- 0 success
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- 1 A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export`
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
@@ -1657,6 +1660,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once. For an occupied name held by a captured bundle, the folder is its new edition: `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename the folder in `incoming/` - such a folder has no replacement form
- A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` material - is a guideline export: its first line, after an optional BOM, starts with `<!-- wikitool:export` -> Not fixed by retrying: it is generated from a wiki's `kb/` and never goes back into `raw/`. Remove it from `incoming/`; nothing was moved
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
@@ -2002,6 +2006,74 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
- `instructions/gates.md` - the gate procedure
- `instructions/setup-instance.md` - the first publish of a new instance
### Guideline export
#### `export guidelines`
Render the guideline pages into one generated `GUIDELINES.md`, or push it into the captured repositories that opted in (**Guideline Push Gate**).
**SYNOPSIS**
- `wikitool export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>]` - Prints the file to stdout; no network.
- `wikitool export guidelines <predicates> --push [--bundle <raw-bundle> ...] [--confirm TOKEN]` - Writes it into the captured repositories, behind the gate.
**PROPERTIES**
- effect: write
- idempotent: yes
- atomic: Per target - one commit pushed without force, or nothing; targets already written stay written when a later one fails
- budget: counted
- network: yes
- gates: guideline-push
**EXAMPLES**
- `tools/wikitool export guidelines --tag guideline`
- `tools/wikitool export guidelines --tag guideline --push`
- `tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # re-run after exit 42, once the user approved every diff`
**EXIT STATUS**
- 0 success
- 0 A target was skipped - tag rule, not reachable, not opted in, hand-written file, another instance's file
- 1 No predicate, no page matches, a malformed predicate or an unknown field
- 1 A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout
- 1 `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched
- 1 `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push`
- 1 A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served
- 42 Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets
**ON FAILURE**
- No predicate, no page matches, a malformed predicate or an unknown field -> Use the filter `kb/CONVENTIONS.md` names for this instance's guidelines, and retry once
- A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has uncommitted changes, or the instance is not a git checkout -> Fix the named pages, or publish the changes first, then retry once
- `--push` with no `user.name` or `user.email` in the instance checkout's git configuration - checked before any target is fetched -> Show the message to the user; the identity is theirs to set, never yours
- `--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` was given without `--push` -> Fix the argument and retry once
- A target rejected the push (its branch moved since the fetch) or a git call failed - the other targets were still served -> Show the lines to the user. A rejected target is served by running the whole command again, which asks for clearance again
- Guideline Push Gate: `--push` without `--confirm`, or with a token that does not match the current set of targets -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
**NEVER**
- Never pass a `--confirm` token the user has not seen and approved.
- Never edit a `GUIDELINES.md` in a target repository by hand to make it match - the next export overwrites it; change the guideline page instead.
**NOTES**
- Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), `--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's guidelines is written down in `kb/CONVENTIONS.md`.
- Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin URL without userinfo, or local> commit=<newest commit touching a selected page> - generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and its text without frontmatter, the generated links and footnote blocks, citation markers (`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as `Title`. A leading H1 of the page is replaced by the title heading. Code is never rewritten. LF endings, no timestamp: the same pages give the same bytes.
- Refuses before printing or pushing anything: no predicate, no match, an unknown field, any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or `types/` (untracked files included), or no git checkout.
- `--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref rule is a tag pattern, it cannot be reached or has no such branch, its root has no `GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), or the header names another `instance=`. A header line alone is the opt-in.
- Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, `unchanged`, `skipped`), the diff for every target it would write, and the re-run line. The token digests URL, branch, old tip and new blob of every target to write.
- With a matching token, each target gets exactly one commit on the fetched tip changing only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, `skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`.
- Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A second run after a successful one finds every target unchanged.
- The target repository's own `AGENTS.md` points at the file (Claude Code: `@GUIDELINES.md`); that belongs to the repository, not to this command.
**SEE ALSO**
- `wikitool search` - the same predicates, to preview the selection
- `wikitool raw capture` - how a repository becomes a target
- `instructions/gates.md` - the gate procedure
### Workshop runs and session budget
#### `work new`
+5 -4
View File
@@ -106,7 +106,8 @@ tools/
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
corpus_diff.py invariant comparison of kb/ between two revisions
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
search/ pluggable search backends, plus service.py - the search core
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (.wikitool-tasks.json), one module per adapter - no instruction ever learns which provider it is
commands/ one module per command or command group: the terminal adapters
@@ -132,7 +133,7 @@ interpreter the way the preflight does - `python3`, `python` on `PATH`; on Windo
and says why.
**Two consumers, one core.** The CLI is not the only caller any more. The cores
(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`) hold what
(`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`, `guideline_export.py`) hold what
decides an answer and import no `typer` and no `rich`; the modules under `commands/` turn
those values into terminal output and those exceptions into exit codes.
`api.Corpus` is the in-process entry point over the same functions - it takes a
@@ -201,8 +202,8 @@ not from a list of its own. Only read-only retrieval earns it.
frontmatter, index statistics, cross-reference bookkeeping, log formatting,
version arithmetic - is computed here so it comes out the same every time.
**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`) and the
Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
**Gates are code, not prompts.** The Mass-Update Gate (`git_publish.py`), the Guideline Push
Gate (`export_cmd.py`) and the Iteration Budget Gate (`run_budget.py`) refuse in-process, because a
prompt-level limit is one an agent can talk itself past. Exemption lists are
constants, never flags.
+2
View File
@@ -31,6 +31,7 @@ try:
doctor,
docs_verify,
eval_cmd,
export_cmd,
git_publish,
index_build,
instructions_cmd,
@@ -178,6 +179,7 @@ app.add_typer(dist_cmd.app, name="dist")
app.add_typer(version_cmd.app, name="version")
app.add_typer(migrate_cmd.app, name="migrate")
app.add_typer(task_cmd.app, name="task")
app.add_typer(export_cmd.app, name="export")
app.command("new")(new_page.new_page_command)
app.command("touch")(touch_module.touch_command)
app.command("rename")(page_ops.rename_command)
+3
View File
@@ -265,6 +265,9 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
("Git", (
"sync", "publish",
)),
("Guideline export", (
"export guidelines",
)),
("Workshop runs and session budget", (
"work new", "work close", "budget status", "budget reset",
)),
+429
View File
@@ -0,0 +1,429 @@
"""`wikitool export guidelines` - the instance's guideline pages as one
generated `GUIDELINES.md`, printed, or written into the captured repositories
behind the Guideline Push Gate.
The core - selection, rendering, the header - is `chemenu/guideline_export.py`;
the git write is `repo_capture.commit_file`/`push_commit`. This module decides
which captured repositories are targets, which of them take part, and when the
gate opens.
**Targets are the captured repositories, and nothing else.** The `repo` and
`ref` of every `_capture.json` under `raw/` - the one declaration that this
instance follows a repository, now read in both directions. A second list of
targets would be a second declaration of the same thing to drift.
**A repository takes part by carrying the file.** Only a branch whose root
`GUIDELINES.md` already starts with the guideline export header is written to;
a hand-written `GUIDELINES.md` is never replaced, and one whose header names a
different `instance=` belongs to that instance. A bare header line is the
opt-in.
**Why a gate, and why this shape.** A push into another repository is the most
irreversible thing the stack does, and a public target publishes the content -
the `instance=` URL included - the moment it lands. So it is held to the same
token form as the Mass-Update and Upload Review gates: the first run fetches,
checks and builds everything, pushes nothing and exits 42 with every target's
status and diff; the token digests exactly that set (URL, branch, old tip, new
blob), so a moved branch, a changed guideline or a new target asks again. The
Publish-Remote Gate is deliberately not applied: the targets are declared by
committed manifests already, and listing the same URLs in
`.wikitool-remotes.json` would be the second declaration again.
"""
from __future__ import annotations
import difflib
import hashlib
import json
import shlex
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
import typer
from rich.markup import escape
from chemenu import cli_contract, config, guideline_export, repo_capture
from chemenu.commands._util import EXIT_NEEDS_CLEARANCE, console, fail, rel_path
from chemenu.errors import BackendError, ValidationError
from chemenu.search import filters
app = typer.Typer(help="Generate files from the wiki for other repositories.")
GATE = "guideline-push"
@dataclass
class Target:
"""One `(repo, ref)` pair - several bundles captured from one branch are
one target. `error` is set when a manifest could not be read."""
repo: str
ref: str
bundles: list[str] = field(default_factory=list)
error: str = ""
@property
def label(self) -> str:
return f"{self.repo} {self.ref}"
@dataclass
class Plan:
target: Target
status: str # new | changed | unchanged | skipped
reason: str = ""
tip: str = ""
blob: str = ""
old: bytes = b""
@property
def writes(self) -> bool:
return self.status in ("new", "changed")
def _targets(only: list[Path]) -> list[Target]:
manifests = repo_capture.captured_manifests(config.RAW_DIR)
if only:
wanted = {p.resolve() for p in only}
known = {m.parent.resolve() for m in manifests}
unknown = sorted(rel_path(p) for p in only if p.resolve() not in known)
if unknown:
raise ValidationError(
f"--bundle {', '.join(unknown)}: not a captured bundle under raw/ (a directory "
f"holding {repo_capture.MANIFEST_NAME})."
)
manifests = [m for m in manifests if m.parent.resolve() in wanted]
targets: dict[tuple[str, str], Target] = {}
broken: list[Target] = []
for path in manifests:
bundle = rel_path(path.parent)
try:
manifest = repo_capture.read_manifest(path)
except ValidationError as exc:
broken.append(Target(bundle, "?", [bundle], f"manifest unreadable: {exc}"))
continue
target = targets.setdefault((manifest.repo, manifest.ref), Target(manifest.repo, manifest.ref))
target.bundles.append(bundle)
return [*targets.values(), *broken]
def _plan(target: Target, export: guideline_export.Export) -> Plan:
"""Where this target stands - fetched, never written to."""
if target.error:
return Plan(target, "skipped", target.error)
if repo_capture.is_tag_pattern(target.ref):
return Plan(target, "skipped", "captured by a tag pattern - a tag takes no commit")
try:
with repo_capture.cache_repo(target.repo) as repo:
resolved = repo_capture.resolve_ref(target.repo, target.ref, repo)
tip = repo_capture.fetch(target.repo, resolved.refname, repo)
existing = repo_capture.read_path(repo, tip, guideline_export.TARGET_PATH)
blob = repo_capture.blob_id(repo, export.data)
except BackendError as exc:
return Plan(target, "skipped", f"not reachable: {exc}")
except ValidationError as exc: # the branch does not exist
return Plan(target, "skipped", str(exc))
if existing is None:
return Plan(target, "skipped", f"no {guideline_export.TARGET_PATH} - not opted in", tip)
fields = guideline_export.guidelines_header(existing)
if fields is None:
return Plan(
target, "skipped",
f"{guideline_export.TARGET_PATH} has no export header - hand-written, never replaced", tip,
)
owner = fields.get("instance")
if owner is not None and owner != export.instance:
return Plan(target, "skipped", f"delivered by another instance ({owner})", tip)
if existing == export.data:
return Plan(target, "unchanged", tip=tip)
return Plan(target, "changed" if owner else "new", tip=tip, blob=blob, old=existing)
def confirm_token(plans: list[Plan]) -> str:
"""sha256 over every target the run would write - URL, branch, the tip it
builds on and the blob it writes - cut to 12 hex chars, the shape of
`git_publish.changeset_token`. The new commit's own id is not part of it:
`commit-tree` stamps the time, so it differs on every run."""
payload = json.dumps(
sorted([p.target.repo, p.target.ref, p.tip, p.blob] for p in plans if p.writes),
sort_keys=True,
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
def _rerun(token: str, predicates: list[str], bundles: list[Path]) -> str:
parts = ["tools/wikitool", "export", "guidelines", "--confirm", token, "--push"]
for raw in predicates:
parts += ["--field", raw]
for bundle in bundles:
parts += ["--bundle", rel_path(bundle)]
return " ".join(shlex.quote(part) for part in parts)
def _diff(plan: Plan, export: guideline_export.Export) -> str:
old = plan.old.decode("utf-8", "replace").splitlines(keepends=True)
new = export.text.splitlines(keepends=True)
name = guideline_export.TARGET_PATH
return "".join(difflib.unified_diff(old, new, f"a/{name}", f"b/{name}"))
def _status_line(plan: Plan) -> str:
detail = f" ({plan.reason})" if plan.reason else ""
return f" {plan.target.label}: {plan.status}{detail}"
def _clearance(plans: list[Plan], export, token: str, rerun: str, stale: Optional[str]) -> str:
writes = [p for p in plans if p.writes]
lines = [
f"This run would push one commit changing only {guideline_export.TARGET_PATH} into "
f"{len(writes)} repositor{'y' if len(writes) == 1 else 'ies'}. A push there is visible at "
"once - publicly, for a public repository - and not cheaply reversible. Nothing was pushed.",
"",
]
if stale:
lines += [
f"The token you passed ({stale}) does not match this set - a branch moved, a guideline "
"changed, a target came or went, or the token was invented. Here is the current state.",
"",
]
lines += [
"THE USER CANNOT SEE THIS OUTPUT. It went to your context, not to their screen.",
"Reproduce every target line and every diff below in your reply, and stop there. Run no "
"further commands in this turn.",
"",
f"TARGETS ({len(plans)}) - reproduce this in your reply:",
"",
*(_status_line(p) for p in plans),
]
for plan in writes:
lines += ["", f"DIFF {plan.target.label}:", "", _diff(plan, export).rstrip("\n")]
lines += [
"",
"Once they have replied approving exactly this, the line that pushes it is:",
"",
f" {rerun}",
"",
f"The token {token} covers each target's branch tip and the exact file; if either moves, "
"clearance is asked again.",
]
return "\n".join(lines)
def _write(plan: Plan, export: guideline_export.Export, identity: tuple[str, str]) -> tuple[str, bool]:
"""Commit and push one target. Returns its result line and whether it
counts as a failure (rejected, or a git error)."""
message = f"Update {guideline_export.TARGET_PATH} from {export.instance} at {export.commit[:12]}"
label = plan.target.label
try:
with repo_capture.cache_repo(plan.target.repo) as repo:
commit = repo_capture.commit_file(
repo, plan.tip, guideline_export.TARGET_PATH, export.data, message, identity
)
result = repo_capture.push_commit(plan.target.repo, repo, commit, plan.target.ref)
except BackendError as exc:
return f" {label}: failed ({exc})", True
if not result.accepted:
return f" {label}: rejected ({result.detail}) - the branch moved; run again", True
return f" {label}: written {plan.tip[:12]} -> {commit[:12]}", False
@app.command("guidelines")
@cli_contract.record(cli_contract.CommandRecord(
path="export guidelines",
summary="Render the guideline pages into one generated `GUIDELINES.md`, or push it into the "
"captured repositories that opted in (**Guideline Push Gate**).",
synopsis=(
cli_contract.Variant(
usage="export guidelines [--field <predicate> ...] [--kind/--subtype/--collection/"
"--tag <v>]",
notes="Prints the file to stdout; no network.",
),
cli_contract.Variant(
usage="export guidelines <predicates> --push [--bundle <raw-bundle> ...] "
"[--confirm TOKEN]",
notes="Writes it into the captured repositories, behind the gate.",
),
),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.YES,
atomic="Per target - one commit pushed without force, or nothing; targets already "
"written stay written when a later one fails",
budget=cli_contract.Budget.COUNTED,
network=cli_contract.Network.YES,
gates=(GATE,),
),
notes=(
"Selects pages with `search`'s own predicates and no text: `--field` (repeatable, AND), "
"`--kind`, `--subtype`, `--collection`, `--tag`. Which filter selects this instance's "
"guidelines is written down in `kb/CONVENTIONS.md`.",
"Renders mechanically: line 1 is `<!-- wikitool:export kind=guidelines instance=<origin "
"URL without userinfo, or local> commit=<newest commit touching a selected page> - "
"generated, do not edit by hand -->`; then each page sorted by title, as `# <title>` and "
"its text without frontmatter, the generated links and footnote blocks, citation markers "
"(`[^id]`, legacy `^[[...]]`), with `[[Title|Text]]` as `Text` and `[[Title]]` as "
"`Title`. A leading H1 of the page is replaced by the title heading. Code is never "
"rewritten. LF endings, no timestamp: the same pages give the same bytes.",
"Refuses before printing or pushing anything: no predicate, no match, an unknown field, "
"any page under `kb/` with unreadable frontmatter, uncommitted changes in `kb/` or "
"`types/` (untracked files included), or no git checkout.",
"`--push` targets every captured repository - each `(repo, ref)` from a `_capture.json` "
"under `raw/`, `--bundle` narrowing it. A target is skipped with its reason when its ref "
"rule is a tag pattern, it cannot be reached or has no such branch, its root has no "
"`GUIDELINES.md` (not opted in), its `GUIDELINES.md` has no export header (hand-written), "
"or the header names another `instance=`. A header line alone is the opt-in.",
"Guideline Push Gate: `--push` without a matching `--confirm` fetches and builds "
"everything, pushes nothing, and exits 42 with each target's status (`new`, `changed`, "
"`unchanged`, `skipped`), the diff for every target it would write, and the re-run line. "
"The token digests URL, branch, old tip and new blob of every target to write.",
"With a matching token, each target gets exactly one commit on the fetched tip changing "
"only `GUIDELINES.md`, authored as the instance checkout's `user.name`/`user.email`, "
"pushed without force; one line per target: `unchanged`, `written <old> -> <new>`, "
"`skipped (<reason>)`, `rejected (<reason>)` or `failed (<reason>)`.",
"Nothing to write (every target unchanged or skipped) ends with exit 0 and no gate. A "
"second run after a successful one finds every target unchanged.",
"The target repository's own `AGENTS.md` points at the file (Claude Code: "
"`@GUIDELINES.md`); that belongs to the repository, not to this command.",
),
failures=(
cli_contract.Failure(
cause="No predicate, no page matches, a malformed predicate or an unknown field",
reaction="Use the filter `kb/CONVENTIONS.md` names for this instance's guidelines, and "
"retry once",
),
cli_contract.Failure(
cause="A page under `kb/` has unreadable frontmatter, `kb/` or `types/` has "
"uncommitted changes, or the instance is not a git checkout",
reaction="Fix the named pages, or publish the changes first, then retry once",
),
cli_contract.Failure(
cause="`--push` with no `user.name` or `user.email` in the instance checkout's git "
"configuration - checked before any target is fetched",
reaction="Show the message to the user; the identity is theirs to set, never yours",
),
cli_contract.Failure(
cause="`--bundle` names no captured bundle under `raw/`, or `--bundle`/`--confirm` "
"was given without `--push`",
reaction="Fix the argument and retry once",
),
cli_contract.Failure(
cause="A target rejected the push (its branch moved since the fetch) or a git call "
"failed - the other targets were still served",
reaction="Show the lines to the user. A rejected target is served by running the "
"whole command again, which asks for clearance again",
),
cli_contract.Failure(
code=0,
cause="A target was skipped - tag rule, not reachable, not opted in, hand-written "
"file, another instance's file",
reaction="",
),
cli_contract.Failure(
cause="Guideline Push Gate: `--push` without `--confirm`, or with a token that does "
"not match the current set of targets",
reaction=cli_contract.token_gate_reaction("--confirm"),
code=42,
),
),
examples=(
"tools/wikitool export guidelines --tag guideline",
"tools/wikitool export guidelines --tag guideline --push",
"tools/wikitool export guidelines --confirm <token> --push --field tags=guideline # "
"re-run after exit 42, once the user approved every diff",
),
never=(
"Never pass a `--confirm` token the user has not seen and approved.",
"Never edit a `GUIDELINES.md` in a target repository by hand to make it match - the "
"next export overwrites it; change the guideline page instead.",
),
see_also=(
"`wikitool search` - the same predicates, to preview the selection",
"`wikitool raw capture` - how a repository becomes a target",
"`instructions/gates.md` - the gate procedure",
),
))
def export_guidelines_command(
field_: Optional[list[str]] = typer.Option(
None, "--field", "-f",
help="Frontmatter predicate, repeatable (AND) - the same forms as `search --field`.",
),
kind: Optional[str] = typer.Option(None, "--kind", help="Shorthand for --field kind=<value>."),
subtype: Optional[str] = typer.Option(None, "--subtype", help="Shorthand for --field subtype=<value>."),
collection: Optional[str] = typer.Option(
None, "--collection", help="Shorthand for --field collection=<value>."
),
tag: Optional[str] = typer.Option(None, "--tag", help="Shorthand for --field tags=<value>."),
push: bool = typer.Option(
False, "--push", help="Write the file into the captured repositories (Guideline Push Gate)."
),
bundle: Optional[list[Path]] = typer.Option(
None, "--bundle", help="Only the repository of this captured bundle under raw/; repeatable."
),
confirm: Optional[str] = typer.Option(
None, "--confirm", help="The token from a prior exit 42, once a human has approved it."
),
):
"""Render the guideline pages into GUIDELINES.md - printed, or pushed into
the captured repositories that carry one."""
bundles = [b if b.is_absolute() else config.ROOT / b for b in (bundle or [])]
if not push and (bundles or confirm):
fail("--bundle and --confirm only apply with --push.")
raw = filters.raw_predicates(field_, kind, subtype, collection, tag)
try:
predicates = tuple(filters.parse_predicate(r) for r in raw)
export = guideline_export.build(predicates)
except ValidationError as exc:
fail(escape(str(exc)))
return
if not push:
typer.echo(export.text, nl=False)
return
identity = guideline_export.git_identity()
if identity is None:
fail(
"No git identity in this instance checkout - `git config user.name` and `git config "
"user.email` must both be set, since the commit in each target repository carries them. "
"Nothing was fetched or pushed."
)
return
try:
targets = _targets(bundles)
except ValidationError as exc:
fail(escape(str(exc)))
return
if not targets:
typer.echo("No captured repository under raw/ - nothing to write.")
return
plans = [_plan(t, export) for t in targets]
if not any(p.writes for p in plans):
typer.echo(f"{len(plans)} target(s), nothing to write:")
for plan in plans:
typer.echo(_status_line(plan))
return
token = confirm_token(plans)
if confirm != token:
console.print("[bold yellow]NEEDS USER CLEARANCE[/bold yellow] Guideline Push Gate")
typer.echo(_clearance(plans, export, token, _rerun(token, raw, bundles), confirm))
raise typer.Exit(code=EXIT_NEEDS_CLEARANCE)
failed = 0
typer.echo(f"{len(plans)} target(s):")
for plan in plans:
if not plan.writes:
typer.echo(_status_line(plan))
continue
line, bad = _write(plan, export, identity)
failed += bad
typer.echo(line)
written = sum(1 for p in plans if p.writes) - failed
typer.echo(f"{written} written, {failed} rejected or failed, {len(plans) - written - failed} "
"unchanged or skipped.")
if failed:
console.print(
f"[bold red]ERROR[/bold red] {failed} target(s) not written - what was written stays "
"written; run the command again for the rest."
)
raise typer.Exit(code=1)
+5 -9
View File
@@ -25,7 +25,7 @@ from typing import Optional
import typer
from chemenu import cli_contract, config, links
from chemenu import cli_contract, config, kb_scan, links
from chemenu.commands._util import (
check_collision,
check_path_budget,
@@ -50,14 +50,10 @@ from chemenu.provenance import (
)
from chemenu.type_resolver import resolver
# `[[Target]]`, `[[Target|alias]]`, `[[Target#anchor]]` - including the
# `[[Target]]` inside a `[^cite-id]: [[Target]]` Footnotes definition, which
# is exactly what lets retarget_body() repoint a citation's link target on a
# rename. Group 1 is the target title; group 2 keeps any alias/anchor suffix
# untouched. The id itself is a separate concern - see retarget_cite_ids().
# Group 1 may span a line break; it is compared through
# kb_scan.normalize_link_target(), the same reading `lint` gives it.
LINK_RE = re.compile(r"\[\[([^\[\]|#]+)((?:[|#][^\[\]]*)?)\]\]")
# `kb_scan.LINK_RE` - the whole wikilink, defined there so the guideline export
# reads links the same way without importing a command module. The cite id is a
# separate concern from the link target - see retarget_cite_ids().
LINK_RE = kb_scan.LINK_RE
def page_ref_fields(page: Page) -> list[str]:
+28 -8
View File
@@ -87,7 +87,7 @@ from typing import Annotated, Optional
import typer
from rich.markup import escape
from chemenu import cli_contract, config, repo_capture, web_capture
from chemenu import cli_contract, config, guideline_export, repo_capture, web_capture
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
from chemenu.errors import BackendError, ChemenuError, ValidationError
from chemenu.frontmatter_io import write_page
@@ -226,11 +226,27 @@ def _check_files(resolved: list[Path]) -> None:
_check_directly_in_incoming(path, incoming)
if path.name == repo_capture.MANIFEST_NAME:
raise _Refused(_reserved_manifest_message(path))
_check_no_export(resolved)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
raise _Refused("Two files share a filename; rename one before promoting.")
def _check_no_export(paths: list[Path]) -> None:
"""Refuse a file a wiki exported (`export guidelines`): it is generated
from `kb/`, so taking it into `raw/` would feed the wiki its own output
back as a source. `raw capture` already leaves such a file out; this
closes the way in through `incoming/`."""
exports = [p for p in paths if guideline_export.file_is_export(p)]
if exports:
listed = "\n".join(f" - {rel_path(p)}" for p in exports)
raise _Refused(
"These files are a wiki's guideline export - their first line starts with "
"<!-- wikitool:export. They are generated from kb/ and never go back into raw/:\n"
f"{listed}\n Remove them from incoming/ and accept again; nothing was moved."
)
def _check_target(dst: Path, remedy: str) -> None:
problem = path_budget_problem_for(dst)
if problem:
@@ -409,6 +425,7 @@ def _plan_folder(folder: Path) -> tuple[Path, list[tuple[Path, Path]], list[Path
stray = [f for f in files if f.name == repo_capture.MANIFEST_NAME and f.parent != folder]
if stray:
raise _Refused(_reserved_manifest_message(stray[0]))
_check_no_export(files)
holder = _occupied_stems(config.RAW_DIR).get(folder.name)
if holder is not None:
@@ -490,6 +507,7 @@ def _replace(
if not incoming_path.is_file():
fail(f"{rel_path(incoming_path)} does not exist or is not a file.")
_validate_under_incoming(incoming_path, _incoming_dir())
_refusals_fail(_check_no_export, [incoming_path])
target = _resolve(replaces)
try:
@@ -784,6 +802,7 @@ def _plan_bundle_replacement(folder: Path, bundle: Path, pages) -> _BundlePlan:
_check_matches_manifest(folder, manifest)
new_files = repo_capture.bundle_files(folder)
_check_no_export(list(new_files.values()))
old_files = repo_capture.bundle_files(bundle)
changes = repo_capture.diff(old_files, {r: p.read_bytes() for r, p in new_files.items()})
changed = {r for _s, r in changes}
@@ -1061,6 +1080,13 @@ def _replace_bundle(
"incoming/<bundle> --replaces-bundle <raw-bundle>`. Any other occupied name: rename "
"the folder in `incoming/` - such a folder has no replacement form",
),
cli_contract.Failure(
cause="A file - given directly, inside a folder, or as `--replaces`/`--replaces-bundle` "
"material - is a guideline export: its first line, after an optional BOM, starts with "
"`<!-- wikitool:export`",
reaction="Not fixed by retrying: it is generated from a wiki's `kb/` and never goes "
"back into `raw/`. Remove it from `incoming/`; nothing was moved",
),
cli_contract.Failure(
label="raw accept",
cause="`--fidelity`/`--authority` is missing, or names `unknown` or a value outside "
@@ -2155,13 +2181,7 @@ def raw_status_command(
):
"""Report which captured bundles have fallen behind their repository.
See raw/CONTRACT.md "Getting a repository in: `raw capture`"."""
manifests = []
if config.RAW_DIR.is_dir():
manifests = [
p for p in sorted(config.RAW_DIR.rglob(repo_capture.MANIFEST_NAME))
if p.is_file() and not any(part.startswith(".") for part in p.relative_to(config.RAW_DIR).parts)
]
rows = [_bundle_status(p) for p in manifests]
rows = [_bundle_status(p) for p in repo_capture.captured_manifests(config.RAW_DIR)]
if json_out:
typer.echo(json.dumps(rows, indent=2, ensure_ascii=False))
+1 -6
View File
@@ -240,12 +240,7 @@ def search_command(
json_out: bool = typer.Option(False, "--json", help="Print the results as JSON."),
):
"""Search kb/ by text, by frontmatter, or by both."""
raw_predicates = list(field or [])
for value, name in ((kind, "kind"), (subtype, "subtype"), (collection, "collection")):
if value:
raw_predicates.append(f"{name}={value}")
if tag:
raw_predicates.append(f"tags={tag}")
raw_predicates = filters.raw_predicates(field, kind, subtype, collection, tag)
if not text and not raw_predicates:
fail("Nothing to search for: give a query, or at least one --field predicate.")
+296
View File
@@ -0,0 +1,296 @@
"""The guideline export, with no CLI attached: select pages by frontmatter,
render them mechanically into one `GUIDELINES.md`, and recognise such a file
again when it comes back.
`wikitool export guidelines` is the terminal adapter over this module; the git
side - writing the file into a captured repository's branch - lives beside
`fetch` in `repo_capture.py`. Nothing here imports `typer` or anything under
`chemenu.commands`, and nothing here imports `repo_capture`, which imports
`EXPORT_MARKER` from here: that is the direction the dependency runs.
Three properties carry the design, and each has a test:
- **Deterministic.** The output depends only on the selected pages and the
commit that last touched one of them - no timestamp, no `HEAD`. A run on an
unchanged corpus, or after an instance commit that touched no guideline, is
byte-identical, and so writes nothing into any target repository.
- **One filter semantics.** Selection is `search`'s own predicate machinery
(`search.filters`), with no text argument: a guideline is a declared choice in
frontmatter, not a full-text hit.
- **Code is never rewritten.** Every transformation of the page text runs on a
copy with code masked out (`markdown_code.strip_code_spans`, which keeps
offsets), so inline code and fenced blocks reach the output byte for byte.
"""
from __future__ import annotations
import re
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
from chemenu import blocks, config, corpus_cache, provenance, toolpaths
from chemenu.errors import ValidationError
from chemenu.kb_scan import LINK_RE, normalize_link_target
from chemenu.markdown_code import strip_code_spans
from chemenu.page import Page
from chemenu.search import filters
from chemenu.search.service import load_pages_by_path, unreadable_pages
from chemenu.search.types import Predicate
# The first bytes of every file this stack generates for another repository.
# Defined once, here: `repo_capture` excludes a file carrying it from a
# capture, `raw accept` refuses one, and the export writes it.
EXPORT_MARKER = b"<!-- wikitool:export"
GUIDELINES_KIND = "guidelines"
GUIDELINES_PREFIX = f"<!-- wikitool:export kind={GUIDELINES_KIND}"
TARGET_PATH = "GUIDELINES.md"
LOCAL_INSTANCE = "local"
_BOM = b"\xef\xbb\xbf"
_HEADER_FIELD = re.compile(r"\b(instance|commit)=(\S+)")
_H1 = re.compile(r"#(?:[ \t]|$)")
_SCHEME_USERINFO = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*://)[^/]*@")
_SCP_USERINFO = re.compile(r"^[^/@:]+@(?=[^/:]+:)")
# --- recognising an export ------------------------------------------------------
def _first_line(data: bytes) -> bytes:
data = data[len(_BOM):] if data.startswith(_BOM) else data
return data.split(b"\n", 1)[0]
def is_export(data: bytes) -> bool:
"""Whether `data` is a file this stack exported: its first line, after an
optional BOM, starts with `EXPORT_MARKER`. Reads no more than the first
line, so a caller may pass only a file's first bytes."""
return _first_line(data).startswith(EXPORT_MARKER)
def guidelines_header(data: bytes) -> Optional[dict[str, str]]:
"""The fields of a guideline export's first line - `instance`, `commit`,
whichever it carries - or None when `data` is not a guideline export at
all. A header with no fields (`{}`) is the bare opt-in stub."""
line = _first_line(data).decode("utf-8", "replace")
if not line.startswith(GUIDELINES_PREFIX):
return None
return dict(_HEADER_FIELD.findall(line[len(GUIDELINES_PREFIX):]))
def file_is_export(path: Path) -> bool:
"""`is_export` for a file on disk, reading only its first line's worth."""
with open(path, "rb") as handle:
return is_export(handle.read(len(_BOM) + len(EXPORT_MARKER)))
# --- the instance it comes from -----------------------------------------------
def _git(args: list[str], root: Path) -> Optional[str]:
"""`git <args>` in the instance checkout: stdout on success, None on any
failure. Only local reads - nothing here reaches a remote."""
try:
result = subprocess.run(
[toolpaths.git(), *args], cwd=root, capture_output=True, text=True,
encoding="utf-8", timeout=30, check=False,
)
except (OSError, subprocess.SubprocessError):
return None
return result.stdout if result.returncode == 0 else None
def strip_userinfo(url: str) -> str:
"""`url` without a user or token in front of the host -
`https://user:token@host/x` becomes `https://host/x`, `git@host:x` becomes
`host:x`. The header lands in other repositories, some of them public."""
stripped = _SCHEME_USERINFO.sub(r"\1", url, count=1)
if stripped != url:
return stripped
return _SCP_USERINFO.sub("", url, count=1)
def instance_id(root: Optional[Path] = None) -> str:
"""The `instance=` value: `origin`'s fetch URL without userinfo, or
`local` for a checkout without an `origin`."""
url = (_git(["remote", "get-url", "origin"], root or config.ROOT) or "").strip()
if not url:
return LOCAL_INSTANCE
return "".join(strip_userinfo(url).split()) or LOCAL_INSTANCE
def git_identity(root: Optional[Path] = None) -> Optional[tuple[str, str]]:
"""`(user.name, user.email)` as the instance checkout resolves them - its
local configuration before the global one - or None if either is unset.
The commit in a target repository is made in a bare cache repository, which
would see only the global configuration; this is the identity the
instance's own commits carry."""
root = root or config.ROOT
name = (_git(["config", "user.name"], root) or "").strip()
email = (_git(["config", "user.email"], root) or "").strip()
return (name, email) if name and email else None
# --- selection ----------------------------------------------------------------
def select(predicates: tuple[Predicate, ...], kb_dir: Optional[Path] = None,
root: Optional[Path] = None) -> list[Page]:
"""The pages the predicates select, sorted by `(title.casefold(), title)`.
Raises `ValidationError` instead of returning something incomplete: with
no predicate (a forgotten filter would export the whole wiki), on an
unknown field (as `search` does), on any page under `kb/` whose
frontmatter does not parse (it could be a guideline that silently drops
out), and on zero hits (an empty file would delete every repository's
guidelines)."""
if not predicates:
raise ValidationError(
"No predicate - the export takes the guidelines by frontmatter, and without a filter "
"it would take the whole wiki. kb/CONVENTIONS.md names this instance's filter."
)
kb_dir = kb_dir or config.KB_DIR
pages = load_pages_by_path(kb_dir, root or config.ROOT)
unreadable = unreadable_pages(pages)
if unreadable:
listed = "\n".join(f" - {u['path']} ({u['reason']})" for u in unreadable)
raise ValidationError(
"These pages have frontmatter that does not parse, so the export cannot tell whether "
f"they are guidelines:\n{listed}\n Fix them first; nothing was exported."
)
filters.validate_fields(predicates, pages)
selected = filters.apply_predicates(pages, predicates, kb_dir)
if not selected:
rendered = " ".join(p.render() for p in predicates)
raise ValidationError(
f"No page matches {rendered} - an empty export would delete every repository's "
"guidelines, so nothing was exported."
)
return sorted(selected.values(), key=lambda p: (p.title.casefold(), p.title))
def check_clean(root: Optional[Path] = None) -> None:
"""Refuse a checkout that is not a git repository, or whose `kb/` or
`types/` has uncommitted changes, untracked files included - otherwise
`commit=` would name a state the export was not made from. `types/` counts
because `kind` and `subtype` are resolved through the type-specs."""
root = root or config.ROOT
if corpus_cache.head_commit(root) is None:
raise ValidationError(
f"{root} is not a git checkout with a commit - the export names the commit its pages "
"come from."
)
dirty = [
Path(path).relative_to(root).as_posix() if Path(path).is_relative_to(root) else str(path)
for path in (config.KB_DIR, config.TYPES_DIR)
if corpus_cache.is_dirty(root, path)
]
if dirty:
raise ValidationError(
f"{' and '.join(dirty)} have uncommitted changes (untracked files count). Publish them "
"first - the export names the commit its pages come from."
)
def last_commit(pages: list[Page], root: Optional[Path] = None) -> str:
"""The newest commit touching one of `pages`' files - not `HEAD`, so an
instance commit that touched no guideline leaves the output unchanged."""
root = root or config.ROOT
paths = [Path(p.path).resolve().relative_to(Path(root).resolve()).as_posix() for p in pages]
sha = (_git(["log", "-1", "--format=%H", "--", *paths], root) or "").strip()
if not sha:
raise ValidationError("None of the selected pages is in a commit yet - publish them first.")
return sha
# --- rendering ----------------------------------------------------------------
def _inline(text: str) -> str:
"""Wikilinks to their display text; `[^cite-id]` and legacy `^[[...]]`
citations removed - all of it outside code only. Matches are found on the
masked copy and applied to the original, which share offsets."""
masked = strip_code_spans(text)
edits: list[tuple[int, int, str]] = []
for m in provenance.LEGACY_CITE_RE.finditer(masked):
edits.append((m.start(), m.end(), ""))
for m in provenance.CITE_REF_RE.finditer(masked):
edits.append((m.start(), m.end(), ""))
for m in LINK_RE.finditer(masked):
rest = text[m.start(2):m.end(2)]
display = (
rest.split("|", 1)[1] if "|" in rest else normalize_link_target(text[m.start(1):m.end(1)])
)
edits.append((m.start(), m.end(), display))
# A legacy citation contains a wikilink; the one starting first wins.
edits.sort(key=lambda e: (e[0], -e[1]))
out: list[str] = []
pos = 0
for start, end, replacement in edits:
if start < pos:
continue
out.append(text[pos:start])
out.append(replacement)
pos = end
out.append(text[pos:])
return "".join(out)
def _trim_blank(lines: list[str]) -> list[str]:
start, end = 0, len(lines)
while start < end and not lines[start].strip():
start += 1
while end > start and not lines[end - 1].strip():
end -= 1
return lines[start:end]
def render_page(page: Page) -> str:
"""One page: `# <title>`, then its text without the generated links and
footnote blocks, citations removed and wikilinks resolved to text. A
leading H1 of the page's own is replaced by the title heading; every other
heading keeps its level."""
body = page.body.replace("\r\n", "\n")
body = blocks.strip(body, blocks.LINKS)
body, _definitions = provenance.split_cite_block(body)
lines = _trim_blank(_inline(body).split("\n"))
if lines and _H1.match(lines[0]):
lines = _trim_blank(lines[1:])
return "\n".join([f"# {page.title}", "", *lines]) if lines else f"# {page.title}"
def header(instance: str, commit: str) -> str:
return (
f"{GUIDELINES_PREFIX} instance={instance} commit={commit} "
"- generated, do not edit by hand -->"
)
def render(pages: list[Page], instance: str, commit: str) -> str:
"""The whole file: the header line, then each page, one blank line
between. LF line endings, exactly one trailing newline."""
parts = [header(instance, commit), *(render_page(p) for p in pages)]
return "\n\n".join(parts) + "\n"
@dataclass(frozen=True)
class Export:
text: str
pages: tuple[Page, ...]
instance: str
commit: str
@property
def data(self) -> bytes:
return self.text.encode("utf-8")
def build(predicates: tuple[Predicate, ...], root: Optional[Path] = None) -> Export:
"""Select, check and render in one go. Raises `ValidationError`."""
root = root or config.ROOT
pages = select(predicates, root=root)
check_clean(root)
commit = last_commit(pages, root)
instance = instance_id(root)
return Export(render(pages, instance, commit), tuple(pages), instance, commit)
+8
View File
@@ -12,6 +12,14 @@ from chemenu.page import Page
WIKILINK_RE = re.compile(r"\[\[([^\]|#]+)")
# The whole link: `[[Target]]`, `[[Target|alias]]`, `[[Target#anchor]]` -
# including the `[[Target]]` inside a `[^cite-id]: [[Target]]` Footnotes
# definition, which is exactly what lets `page_ops.retarget_body()` repoint a
# citation's link target on a rename. Group 1 is the target title; group 2 keeps
# any alias/anchor suffix untouched. Group 1 may span a line break; it is
# compared through `normalize_link_target()`, the same reading `lint` gives it.
LINK_RE = re.compile(r"\[\[([^\[\]|#]+)((?:[|#][^\[\]]*)?)\]\]")
# A line break inside `[[...]]`, with the indentation around it. The target
# class above admits a newline, so a link someone wrapped at a fixed column -
# `[[Foo Bar\n Target]]` - captures the break as part of the title, matches no
+135 -15
View File
@@ -1,10 +1,12 @@
"""Capturing documentation from a git repository for `raw/`: resolve a ref
rule to one commit, read the files the globs select straight out of that
commit, and describe the result in a manifest - with no CLI attached.
commit, and describe the result in a manifest - with no CLI attached. And the
one write in the other direction: a single commit that changes one file on a
captured repository's branch, for `export guidelines --push`.
`wikitool raw capture`, `raw status` and `raw accept --replaces-bundle` are the
terminal adapters over this module; it decides nothing about `incoming/` or
`raw/` paths beyond the manifest's own name.
`wikitool raw capture`, `raw status`, `raw accept --replaces-bundle` and
`export guidelines` are the terminal adapters over this module; it decides
nothing about `incoming/` or `raw/` paths beyond the manifest's own name.
Three properties carry the design, and each has a test:
@@ -39,6 +41,7 @@ from typing import Iterator, Optional
from chemenu import config, filelock, toolpaths, web_capture
from chemenu.errors import BackendError, ValidationError
from chemenu.guideline_export import EXPORT_MARKER
MANIFEST_NAME = "_capture.json"
SCHEMA = 1
@@ -52,7 +55,6 @@ ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
GIT_TIMEOUT_SECONDS = 120.0
EXPORT_MARKER = b"<!-- wikitool:export"
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
_BOM = b"\xef\xbb\xbf"
@@ -199,21 +201,36 @@ def _git_env() -> dict[str, str]:
return env
def run_git(
@dataclass(frozen=True)
class GitResult:
returncode: int
stdout: bytes
stderr: bytes
def last_error_line(self) -> str:
message = self.stderr.decode("utf-8", "replace").strip().splitlines()
return message[-1] if message else f"exit {self.returncode}"
def run_git_result(
args: list[str],
git_dir: Path,
*,
stdin: Optional[bytes] = None,
env: Optional[dict[str, str]] = None,
timeout: float = GIT_TIMEOUT_SECONDS,
) -> bytes:
"""Run `git --git-dir=<git_dir> <args>` and return its stdout.
) -> GitResult:
"""Run `git --git-dir=<git_dir> <args>` and return its exit code and both
streams, whatever the exit code - for a caller that reads stdout on a
failure too (`push --porcelain` reports a rejected ref there, with exit 1).
No call can prompt: no terminal prompt, no askpass program (an empty
`core.askPass` stops git from falling back to `SSH_ASKPASS`), and on POSIX a
session of its own, so `ssh` has no controlling terminal to ask on either.
A timeout kills the whole process group, `ssh` included - killing `git`
alone would leave the pipe open and the read below hanging. Raises
`BackendError` on a non-zero exit, a timeout or a git that cannot start."""
alone would leave the pipe open and the read below hanging. `env` is laid
over that environment. Raises `BackendError` on a timeout or a git that
cannot start."""
argv = [toolpaths.git(), "-c", "core.askPass=", f"--git-dir={git_dir}", *args]
posix = os.name == "posix"
try:
@@ -222,7 +239,7 @@ def run_git(
stdin=subprocess.PIPE if stdin is not None else subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=_git_env(),
env={**_git_env(), **(env or {})},
start_new_session=posix,
)
except OSError as exc:
@@ -239,10 +256,23 @@ def run_git(
proc.kill()
proc.communicate()
raise BackendError(f"git {args[0]} did not finish within {timeout:.0f} s") from None
if proc.returncode != 0:
message = err.decode("utf-8", "replace").strip().splitlines()
raise BackendError(f"git {args[0]} failed: {message[-1] if message else f'exit {proc.returncode}'}")
return out
return GitResult(proc.returncode, out, err)
def run_git(
args: list[str],
git_dir: Path,
*,
stdin: Optional[bytes] = None,
env: Optional[dict[str, str]] = None,
timeout: float = GIT_TIMEOUT_SECONDS,
) -> bytes:
"""`run_git_result`, returning stdout and raising `BackendError` on a
non-zero exit as well."""
result = run_git_result(args, git_dir, stdin=stdin, env=env, timeout=timeout)
if result.returncode != 0:
raise BackendError(f"git {args[0]} failed: {result.last_error_line()}")
return result.stdout
def cache_root() -> Path:
@@ -470,6 +500,84 @@ def remote_commit(url: str, rule: str) -> ResolvedRef:
return resolve_ref(url, rule, repo)
# --- writing one file back ----------------------------------------------------
#
# `export guidelines --push` - the only write into another repository. It needs
# no working tree and touches none: blob, tree and commit are built with
# plumbing in the bare cache repository, on top of the tip `fetch` brought in,
# and pushed by commit id. The cache's shallow history is enough, because the
# server already has the parent.
def read_path(repo: Path, commit: str, path: str) -> Optional[bytes]:
"""The bytes of the file at repository path `path` in `commit`, or None
if there is no regular file there."""
out = run_git(["ls-tree", "-z", commit, "--", path], repo)
entry = out.split(b"\0", 1)[0]
if not entry:
return None
meta, _, _name = entry.partition(b"\t")
mode, kind, sha = meta.decode("ascii").split()
if kind != "blob" or mode == _MODE_SYMLINK:
return None
return read_blobs(repo, [sha])[sha]
def blob_id(repo: Path, data: bytes) -> str:
"""The object id `data` has as a blob in `repo`, without writing it."""
return run_git(["hash-object", "--no-filters", "--stdin"], repo, stdin=data).decode().strip()
def commit_file(
repo: Path, parent: str, path: str, data: bytes, message: str, identity: tuple[str, str]
) -> str:
"""A new commit on `parent` whose tree differs from the parent's only in
`path`, now holding `data` - built in a temporary index, so nothing else of
the tree can change. The file keeps an executable bit it already had.
`identity` is `(name, email)` for author and committer alike. Returns the
new commit's id; nothing is pushed."""
blob = run_git(["hash-object", "-w", "--no-filters", "--stdin"], repo, stdin=data).decode().strip()
existing = run_git(["ls-tree", "-z", parent, "--", path], repo).split(b"\0", 1)[0]
mode = "100755" if existing.startswith(b"100755 ") else "100644"
name, email = identity
index = repo / f"wikitool-export-{os.getpid()}.index"
env = {
"GIT_INDEX_FILE": str(index),
"GIT_AUTHOR_NAME": name, "GIT_AUTHOR_EMAIL": email,
"GIT_COMMITTER_NAME": name, "GIT_COMMITTER_EMAIL": email,
}
try:
run_git(["read-tree", parent], repo, env=env)
run_git(["update-index", "--add", "--cacheinfo", f"{mode},{blob},{path}"], repo, env=env)
tree = run_git(["write-tree"], repo, env=env).decode().strip()
return run_git(["commit-tree", tree, "-p", parent, "-m", message], repo, env=env).decode().strip()
finally:
index.unlink(missing_ok=True)
@dataclass(frozen=True)
class PushResult:
accepted: bool
detail: str # git's own summary for the ref, e.g. "[rejected] (fetch first)"
def push_commit(url: str, repo: Path, commit: str, branch: str) -> PushResult:
"""Push `commit` to `refs/heads/<branch>` at `url` - never forced, so a
branch that moved since the fetch rejects it and stays where it is.
Accepted or rejected is read from `--porcelain`'s status flag for the ref
(`!` is rejected), never from the wording on stderr, which git translates
into the host's locale. Raises `BackendError` when there is no status for
the ref at all - the remote was not reached."""
ref = f"refs/heads/{branch}"
result = run_git_result(["push", "--porcelain", "--", url, f"{commit}:{ref}"], repo)
for line in result.stdout.decode("utf-8", "replace").splitlines():
fields = line.split("\t")
if len(fields) >= 3 and fields[1].endswith(f":{ref}") and fields[0] in (" ", "*", "=", "!", "+", "-"):
return PushResult(fields[0] != "!", fields[2].strip())
raise BackendError(f"git push failed: {result.last_error_line()}")
# --- manifest -----------------------------------------------------------------
@@ -537,6 +645,18 @@ def read_manifest(path: Path) -> Manifest:
)
def captured_manifests(raw_dir: Path) -> list[Path]:
"""Every captured bundle's `_capture.json` under `raw_dir`, sorted, none
below a hidden directory - the one list both `raw status` and
`export guidelines` work from."""
if not raw_dir.is_dir():
return []
return [
p for p in sorted(raw_dir.rglob(MANIFEST_NAME))
if p.is_file() and not any(part.startswith(".") for part in p.relative_to(raw_dir).parts)
]
def bundle_files(bundle: Path) -> dict[str, Path]:
"""Every file of a captured bundle on disk, keyed by its repository path -
the manifest itself left out."""
+20
View File
@@ -70,6 +70,26 @@ def parse_predicate(raw: str) -> Predicate:
)
def raw_predicates(
fields: Optional[list[str]] = None,
kind: Optional[str] = None,
subtype: Optional[str] = None,
collection: Optional[str] = None,
tag: Optional[str] = None,
) -> list[str]:
"""The `--field` arguments a call's shorthand options stand for, after its
own `--field` values: `--kind/--subtype/--collection X` is `<name>=X`,
`--tag X` is `tags=X`. Every command taking these options goes through
here, so the shorthands mean the same thing everywhere."""
out = list(fields or [])
for value, name in ((kind, "kind"), (subtype, "subtype"), (collection, "collection")):
if value:
out.append(f"{name}={value}")
if tag:
out.append(f"tags={tag}")
return out
def known_fields(pages: dict[str, Page]) -> set[str]:
"""Every field name a predicate may legitimately name: the union of all
frontmatter keys actually present in the corpus, plus the virtual ones."""
+3 -1
View File
@@ -340,13 +340,15 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list
since `--latest`, which asks the release feed and downloads from it, and `raw fetch`
since it exists (Gitea #120) - its `--html` form stays offline, which does not turn the
command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177). Pinned as an explicit
command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177), and so
does `export guidelines --push` (Gitea #179). Pinned as an explicit
set so a command gaining or losing that reach is a deliberate edit here, not a silent
drift between the property and what the command actually does."""
expected = {
"raw fetch",
"raw capture",
"raw status",
"export guidelines",
"sync",
"publish",
"version check",
@@ -0,0 +1,553 @@
"""`export guidelines` and the Guideline Push Gate, against local repositories
over `file://` - never the network.
The instance is the `kb_dir` fixture tree made into a git checkout with its own
identity and a copy of the shipped `types/`, so the cleanliness check sees the
same two directories it sees in a real instance. Target repositories are bare
repositories, each fed from a work clone, because a push into a checked-out
branch is refused by git itself.
"""
from __future__ import annotations
import json
import re
import shutil
import subprocess
from pathlib import Path
import pytest
from typer.testing import CliRunner
from chemenu import config, guideline_export, repo_capture
from chemenu.cli import app
from chemenu.commands import export_cmd
from chemenu.commands.raw_cmd import raw_accept_command, raw_capture_command, raw_status_command
from chemenu.frontmatter_io import write_page
from chemenu.markdown_code import strip_code_spans
from chemenu.search import filters
from chemenu.search.registry import resolve
from chemenu.search.service import load_pages_by_path, run_search
from chemenu.search.types import SearchQuery
from chemenu.type_resolver import resolver
runner = CliRunner()
TAG = ["--tag", "guideline"]
def _git(cwd: Path, *args: str, check: bool = True) -> str:
return subprocess.run(
["git", "-c", "user.name=Fixture", "-c", "user.email=f@example.org", *args],
cwd=cwd, check=check, capture_output=True, text=True,
).stdout.strip()
def _bare(bare: Path, *args: str) -> str:
return subprocess.run(
["git", f"--git-dir={bare}", *args], check=True, capture_output=True, text=True
).stdout.strip()
class Target:
"""A bare repository - what the export pushes into - fed from a work
clone that plays the project's own developers."""
def __init__(self, base: Path, guidelines: bytes | None = None) -> None:
self.work = base / "work"
self.bare = base / "remote.git"
self.work.mkdir(parents=True)
_git(self.work, "init", "-q", "-b", "main")
(self.work / "README.md").write_text("# Project\n", encoding="utf-8")
(self.work / "docs").mkdir()
(self.work / "docs" / "a.md").write_text("# A\n", encoding="utf-8")
if guidelines is not None:
(self.work / "GUIDELINES.md").write_bytes(guidelines)
self.commit("initial")
_git(base, "clone", "-q", "--bare", str(self.work), str(self.bare))
_git(self.work, "remote", "add", "origin", self.url)
@property
def url(self) -> str:
return self.bare.as_uri()
def commit(self, message: str = "change") -> None:
_git(self.work, "add", "-A")
_git(self.work, "commit", "-q", "--allow-empty", "-m", message)
def advance(self, message: str = "upstream change") -> str:
"""A commit by someone else, pushed to the remote branch."""
_git(self.work, "pull", "-q", "--ff-only", "origin", "main", check=False)
(self.work / "docs" / "a.md").write_text(f"# A\n\n{message}\n", encoding="utf-8")
self.commit(message)
_git(self.work, "push", "-q", "origin", "HEAD:main")
return self.tip()
def tip(self) -> str:
return _bare(self.bare, "rev-parse", "refs/heads/main")
def refs(self) -> str:
return _bare(self.bare, "for-each-ref", "--format=%(refname) %(objectname)")
def file(self, rev: str = "refs/heads/main") -> bytes:
return subprocess.run(
["git", f"--git-dir={self.bare}", "show", f"{rev}:GUIDELINES.md"],
check=True, capture_output=True,
).stdout
STUB = b"<!-- wikitool:export kind=guidelines -->\n"
ALPHA_BODY = """
# Alpha Rule
Keep secrets in `.env` only[^s-handbook]. See [[Beta Rule]] and [[Beta Rule|the beta]],
also [[Beta Rule#Scope]] and [[Beta Rule#Scope|its scope]]. Legacy^[[Source - Handbook]] too.
## Details
Inline `[[Not A Link]]` and `[^not-a-cite]` stay as written.
```
[[Fenced|kept]] and [^fenced] and ^[[Fenced]]
```
<!-- wikitool:links -->
## Beziehungen
- **related:** [[Beta Rule]]
<!-- /wikitool:links -->
<!-- wikitool:footnotes -->
## Fußnoten
[^s-handbook]: [[Source - Handbook]]
<!-- /wikitool:footnotes -->
"""
BETA_BODY = """
Beta has no heading of its own.
### Scope
Everything.
## Footnotes
[^s-handbook]: [[Source - Handbook]]
"""
def _guideline(kb: Path, title: str, body: str, tags=("guideline",)) -> Path:
path = kb / "concepts" / f"{title}.md"
write_page(path, {"type": "types/concept.md", "tags": list(tags), "summary": f"{title}."}, body)
return path
def _commit_instance(root: Path, message: str = "instance change") -> None:
_git(root, "add", "-A")
_git(root, "commit", "-q", "-m", message)
@pytest.fixture
def instance(kb_dir, monkeypatch):
root = kb_dir.parent
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
monkeypatch.setattr(config, "TYPES_DIR", root / "types")
monkeypatch.setattr(resolver, "_repo_root", root)
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file"))
(root / "raw").mkdir(exist_ok=True)
(root / "incoming").mkdir(exist_ok=True)
(root / ".gitignore").write_text("tools/\n", encoding="utf-8")
_guideline(kb_dir, "Alpha Rule", ALPHA_BODY)
_guideline(kb_dir, "beta rule", BETA_BODY)
_git(root, "init", "-q", "-b", "main")
_git(root, "config", "user.name", "Instance Author")
_git(root, "config", "user.email", "author@example.org")
_commit_instance(root, "instance")
return root
def _manifest(root: Path, name: str, repo: str, ref: str = "main") -> Path:
bundle = root / "raw" / "2026" / "10" / name
bundle.mkdir(parents=True)
data = {
"schema": 1, "repo": repo, "ref": ref, "commit": "0" * 40, "paths": ["docs/**"],
"captured": "2026-10-05T00:00:00Z", "fidelity": "verbatim", "authority": "normative",
"files": [],
}
(bundle / repo_capture.MANIFEST_NAME).write_text(json.dumps(data), encoding="utf-8")
return bundle
@pytest.fixture
def target(instance, tmp_path):
t = Target(tmp_path / "targets" / "one", STUB)
_manifest(instance, "one", t.url)
_commit_instance(instance, "capture one")
return t
def _run(*args: str):
return runner.invoke(app, ["export", "guidelines", *args])
def _token(output: str) -> str:
match = re.search(r"--confirm (\w+) --push", output)
assert match, output
return match.group(1)
def _push(*extra: str):
"""Gate run, then the confirmed run - what an approved push looks like."""
gated = _run(*TAG, "--push", *extra)
assert gated.exit_code == 42, gated.output
return _run(*TAG, "--push", "--confirm", _token(gated.output), *extra)
# --- selection ----------------------------------------------------------------
def test_selection_is_searchs_own(instance, kb_dir):
_guideline(kb_dir, "Gamma Elsewhere", "\nNot a guideline.\n", tags=("other",))
_commit_instance(instance)
result = _run(*TAG)
assert result.exit_code == 0, result.output
exported = re.findall(r"^# (.+)$", result.output, re.MULTILINE)
predicates = tuple(filters.parse_predicate(r) for r in filters.raw_predicates(tag="guideline"))
search = run_search(SearchQuery(text=None, predicates=predicates, limit=0),
load_pages_by_path(), resolve(None))
expected = sorted((h.title for h in search.hits), key=lambda t: (t.casefold(), t))
assert exported == expected == ["Alpha Rule", "beta rule"]
@pytest.mark.parametrize("args, needle", [
((), "No predicate"),
(("--tag", "nothing-has-this"), "No page matches"),
(("--field", "no_such_field=x"), "unknown field"),
])
def test_refusals_print_no_export(instance, target, args, needle):
before = target.refs()
for extra in ((), ("--push",)):
result = _run(*args, *extra)
assert result.exit_code == 1, result.output
assert needle in result.output
assert "<!-- wikitool:export" not in result.output
assert target.refs() == before
def test_unreadable_frontmatter_anywhere_refuses(instance, kb_dir, target):
(kb_dir / "concepts" / "Broken.md").write_text("---\ntype: [unclosed\n---\nBody\n", encoding="utf-8")
_commit_instance(instance)
before = target.refs()
result = _run(*TAG, "--push")
assert result.exit_code == 1
assert "Broken.md" in result.output and "<!-- wikitool:export" not in result.output
assert target.refs() == before
@pytest.mark.parametrize("change", ["modified", "untracked-kb", "untracked-types"])
def test_dirty_kb_or_types_refuses(instance, kb_dir, target, change):
if change == "modified":
page = kb_dir / "concepts" / "Alpha Rule.md"
page.write_text(page.read_text(encoding="utf-8") + "\nmore\n", encoding="utf-8")
elif change == "untracked-kb":
(kb_dir / "concepts" / "draft.txt").write_text("x\n", encoding="utf-8")
else:
(instance / "types" / "draft.md").write_text("x\n", encoding="utf-8")
before = target.refs()
for extra in ((), ("--push",)):
result = _run(*TAG, *extra)
assert result.exit_code == 1, result.output
assert "uncommitted changes" in result.output
assert "<!-- wikitool:export" not in result.output
assert target.refs() == before
# --- rendering ----------------------------------------------------------------
def _export(*args: str) -> str:
result = _run(*(args or TAG))
assert result.exit_code == 0, result.output
return result.output
def test_rendering_resolves_links_and_drops_citations_outside_code(instance):
out = _export()
assert out.endswith("\n") and not out.endswith("\n\n") and "\r" not in out
assert "See Beta Rule and the beta,\nalso Beta Rule and its scope. Legacy too." in out
assert "Keep secrets in `.env` only." in out
assert "Inline `[[Not A Link]]` and `[^not-a-cite]` stay as written." in out
assert "```\n[[Fenced|kept]] and [^fenced] and ^[[Fenced]]\n```" in out
prose = strip_code_spans(out)
for forbidden in ("[[", "[^", "^[[", "wikitool:links", "wikitool:footnotes", "Beziehungen",
"Fußnoten", "Footnotes", "type: types/", "---\n"):
assert forbidden not in prose, forbidden
def test_headings(instance):
out = _export()
lines = out.split("\n")
assert lines[1] == "" and lines[2] == "# Alpha Rule" and lines[3] == ""
assert out.count("# Alpha Rule") == 1
assert "\n## Details\n" in out and "\n### Scope\n" in out
assert "\n\n# beta rule\n\nBeta has no heading of its own." in out
def test_header_line(instance):
first = _export().split("\n", 1)[0]
commit = _git(instance, "log", "-1", "--format=%H", "--", "kb/concepts/Alpha Rule.md",
"kb/concepts/beta rule.md")
assert first == (
f"<!-- wikitool:export kind=guidelines instance=local commit={commit} "
"- generated, do not edit by hand -->"
)
def test_instance_is_origin_without_userinfo(instance):
_git(instance, "remote", "add", "origin", "https://user:s3cret@git.example.org/team/wiki.git")
first = _export().split("\n", 1)[0]
assert "instance=https://git.example.org/team/wiki.git " in first
assert "s3cret" not in first and "user" not in first
@pytest.mark.parametrize("url, expected", [
("https://user:tok@host/x.git", "https://host/x.git"),
("ssh://git@host:2222/x.git", "ssh://host:2222/x.git"),
("git@host:team/x.git", "host:team/x.git"),
("https://host/x.git", "https://host/x.git"),
])
def test_strip_userinfo(url, expected):
assert guideline_export.strip_userinfo(url) == expected
def test_output_is_deterministic_and_ignores_unrelated_commits(instance, kb_dir):
first = _export()
assert _export() == first
(kb_dir / "concepts" / "Unrelated.md").write_text(
"---\ntype: types/concept.md\ntags: [other]\n---\n\nx\n", encoding="utf-8"
)
_commit_instance(instance, "unrelated")
assert _export() == first
# --- the gate and the push ----------------------------------------------------
def test_push_without_identity_refuses_before_any_target(instance, target):
_git(instance, "config", "--unset", "user.email")
before = target.refs()
result = _run(*TAG, "--push")
assert result.exit_code == 1, result.output
assert "user.email" in result.output
assert not re.search(r"--confirm \w{12}", result.output)
assert "NEEDS USER CLEARANCE" not in result.output
assert target.refs() == before
def test_push_without_token_is_gated(instance, target, tmp_path):
hand = Target(tmp_path / "targets" / "hand", b"# Our own rules\n")
_manifest(instance, "hand", hand.url)
_commit_instance(instance)
before = (target.refs(), hand.refs())
result = _run(*TAG, "--push")
assert result.exit_code == 42, result.output
assert (target.refs(), hand.refs()) == before
assert f"{target.url} main: new" in result.output
assert f"{hand.url} main: skipped" in result.output
assert "+# Alpha Rule" in result.output and "-<!-- wikitool:export kind=guidelines -->" in result.output
assert re.search(r"tools/wikitool export guidelines --confirm \w{12} --push --field tags=guideline",
result.output)
def test_confirmed_push_writes_one_commit_with_only_guidelines(instance, target):
old = target.tip()
stdout = _export()
result = _push()
assert result.exit_code == 0, result.output
new = target.tip()
assert f"written {old[:12]} -> {new[:12]}" in result.output
assert _bare(target.bare, "rev-parse", f"{new}^") == old
assert _bare(target.bare, "diff", "--name-only", old, new) == "GUIDELINES.md"
assert target.file() == stdout.encode("utf-8")
assert _bare(target.bare, "log", "-1", "--format=%an <%ae>|%cn <%ce>", new) == (
"Instance Author <author@example.org>|Instance Author <author@example.org>"
)
commit = stdout.split("commit=", 1)[1][:12]
assert _bare(target.bare, "log", "-1", "--format=%s", new) == (
f"Update GUIDELINES.md from local at {commit}"
)
again = _run(*TAG, "--push")
assert again.exit_code == 0, again.output
assert f"{target.url} main: unchanged" in again.output
assert "--confirm" not in again.output
assert target.tip() == new
def test_stale_tokens_are_gated_again(instance, kb_dir, target, tmp_path):
token = _token(_run(*TAG, "--push").output)
target.advance()
moved = _run(*TAG, "--push", "--confirm", token)
assert moved.exit_code == 42 and "does not match" in moved.output
token = _token(moved.output)
page = kb_dir / "concepts" / "beta rule.md"
page.write_text(page.read_text(encoding="utf-8") + "\nAmended.\n", encoding="utf-8")
_commit_instance(instance, "amend beta")
changed = _run(*TAG, "--push", "--confirm", token)
assert changed.exit_code == 42 and "+Amended." in changed.output
token = _token(changed.output)
second = Target(tmp_path / "targets" / "two", STUB)
_manifest(instance, "two", second.url)
_commit_instance(instance, "capture two")
added = _run(*TAG, "--push", "--confirm", token)
assert added.exit_code == 42 and f"{second.url} main: new" in added.output
before = (target.refs(), second.refs())
invented = _run(*TAG, "--push", "--confirm", "000000000000")
assert invented.exit_code == 42
assert (target.refs(), second.refs()) == before
def test_targets_that_do_not_take_part_are_left_alone(instance, target, tmp_path):
hand = Target(tmp_path / "targets" / "hand", b"# Our own rules\n")
none = Target(tmp_path / "targets" / "none")
foreign = Target(
tmp_path / "targets" / "foreign",
b"<!-- wikitool:export kind=guidelines instance=https://other.example/wiki commit=abc -->\n",
)
tagged = Target(tmp_path / "targets" / "tagged", STUB)
for name, t, ref in (("hand", hand, "main"), ("none", none, "main"),
("foreign", foreign, "main"), ("tagged", tagged, "v*")):
_manifest(instance, name, t.url, ref)
_commit_instance(instance)
before = {t.url: t.refs() for t in (hand, none, foreign, tagged)}
result = _push()
assert result.exit_code == 0, result.output
assert "hand-written" in result.output
assert "not opted in" in result.output
assert "another instance (https://other.example/wiki)" in result.output
assert "tag pattern" in result.output
assert {t.url: t.refs() for t in (hand, none, foreign, tagged)} == before
assert "written" in result.output # the participating target was still served
def test_a_branch_that_moved_is_rejected_not_forced(instance, target, monkeypatch):
# Translated git messages must not change the verdict: it is read from the
# porcelain status flag, not from the text.
monkeypatch.setenv("LANGUAGE", "de")
monkeypatch.setenv("LANG", "de_DE.UTF-8")
token = _token(_run(*TAG, "--push").output)
real_commit_file = repo_capture.commit_file
foreign: list[str] = []
def commit_then_race(*args, **kwargs):
foreign.append(target.advance("raced in"))
return real_commit_file(*args, **kwargs)
monkeypatch.setattr(repo_capture, "commit_file", commit_then_race)
result = _run(*TAG, "--push", "--confirm", token)
assert result.exit_code == 1, result.output
assert f"{target.url} main: rejected" in result.output
assert target.tip() == foreign[0]
def test_an_unreachable_target_does_not_stop_the_others(instance, target, tmp_path):
_manifest(instance, "gone", (tmp_path / "does-not-exist.git").as_uri())
_commit_instance(instance)
result = _push()
assert result.exit_code == 0, result.output
assert "does-not-exist.git main: skipped (not reachable" in result.output
assert f"{target.url} main: written" in result.output
def test_bundle_narrows_the_targets(instance, target, tmp_path):
other = Target(tmp_path / "targets" / "other", STUB)
_manifest(instance, "other", other.url)
_commit_instance(instance)
before = other.refs()
result = _push("--bundle", "raw/2026/10/one")
assert result.exit_code == 0, result.output
assert other.url not in result.output
assert other.refs() == before
unknown = _run(*TAG, "--push", "--bundle", "raw/2026/10/nope")
assert unknown.exit_code == 1 and "not a captured bundle" in unknown.output
def test_bundle_and_confirm_need_push(instance):
assert _run(*TAG, "--confirm", "abc").exit_code == 1
assert _run(*TAG, "--bundle", "raw/x").exit_code == 1
def test_raw_status_does_not_report_a_pushed_export(instance, tmp_path, capsys):
t = Target(tmp_path / "targets" / "captured", STUB)
raw_capture_command(repo_url=t.url, ref="main", paths=["**/*.md"], name="captured",
fidelity="verbatim", authority="normative", update=None)
raw_accept_command(files=[instance / "incoming" / "captured"], fidelity=None, authority=None,
page=None, replaces=None, dry_run=False)
_commit_instance(instance, "capture")
capsys.readouterr()
result = _push()
assert result.exit_code == 0 and "written" in result.output, result.output
raw_status_command(json_out=True)
rows = json.loads(capsys.readouterr().out)
assert len(rows) == 1 and rows[0]["error"] is None and rows[0]["changed"] is False
# --- no way back in -----------------------------------------------------------
def test_export_marker_is_defined_once():
package = Path(guideline_export.__file__).parent
definitions = [
path for path in package.rglob("*.py")
if "tests" not in path.parts and re.search(r'= b"<!-- wikitool:export"', path.read_text("utf-8"))
]
assert definitions == [Path(guideline_export.__file__)]
def test_a_real_export_is_excluded_from_capture(instance):
data = _export().encode("utf-8")
assert repo_capture._content_exclusion(data) is not None
assert repo_capture._content_exclusion(b"\xef\xbb\xbf" + data) is not None
def _snapshot(*dirs: Path) -> dict[str, bytes]:
return {p.as_posix(): p.read_bytes() for d in dirs for p in sorted(d.rglob("*")) if p.is_file()}
def test_raw_accept_refuses_an_export_file_and_a_folder_holding_one(instance):
incoming, raw = instance / "incoming", instance / "raw"
export = _export().encode("utf-8")
(incoming / "GUIDELINES.md").write_bytes(export)
(incoming / "pack").mkdir()
(incoming / "pack" / "notes.md").write_text("# Notes\n", encoding="utf-8")
(incoming / "pack" / "sub").mkdir()
(incoming / "pack" / "sub" / "rules.md").write_bytes(b"\xef\xbb\xbf" + export)
before = _snapshot(incoming, raw)
for path in (incoming / "GUIDELINES.md", incoming / "pack"):
result = runner.invoke(app, ["raw", "accept", str(path), "--fidelity", "verbatim",
"--authority", "normative"])
assert result.exit_code == 1, result.output
assert "guideline export" in result.output
assert _snapshot(incoming, raw) == before
def test_is_export_reads_the_first_line_only():
assert guideline_export.is_export(b"<!-- wikitool:export kind=x -->\nbody")
assert not guideline_export.is_export(b"# Title\n<!-- wikitool:export kind=x -->\n")
assert guideline_export.guidelines_header(STUB) == {}
assert guideline_export.guidelines_header(b"<!-- wikitool:export kind=other -->") is None
def test_confirm_token_ignores_targets_it_does_not_write():
t = export_cmd.Target("u", "main")
writes = export_cmd.Plan(t, "new", tip="a", blob="b")
skipped = export_cmd.Plan(export_cmd.Target("v", "main"), "skipped", "x")
assert export_cmd.confirm_token([writes, skipped]) == export_cmd.confirm_token([writes])
assert export_cmd.confirm_token([writes]) != export_cmd.confirm_token(
[export_cmd.Plan(t, "new", tip="a2", blob="b")]
)