feat: raw capture / raw status / --replaces-bundle - documentation from git repositories as a bundle, with drift reporting (#177)
New repo_capture module: resolve a branch or tag-pattern ref rule, fetch it shallowly by name into a bare cache, read the glob-selected files as blobs, and record repo/ref/commit/globs/capture fields in _capture.json. raw status reports changed captured bundles as A/M/D; raw accept --replaces-bundle swaps a captured bundle for its new edition at the same address. iter_raw_files now skips _capture.json and anchors the CONTRACT.md exclusion to raw/CONTRACT.md. Files changed: - .gitignore - CHANGES.md - README.md - VERSION - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/dist_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/config.py - tools/chemenu/repo_capture.py - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_raw_capture.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed
1 parent
8ff22ad6b0
commit
311c8ee059
16 files changed
+2515
-23
No files matched your search
@@ -73,6 +73,11 @@ npm-debug.log*
|
||||
# "Gates") - local, per-session, never committed
|
||||
/tools/.wikitool_session/
|
||||
|
||||
# `wikitool raw capture`/`raw status` git cache (see raw/CONTRACT.md "Getting a
|
||||
# repository in") - one bare repository per captured URL, refetched on demand.
|
||||
# Derived and per-checkout; never committed, never shipped.
|
||||
/tools/.wikitool_capture/
|
||||
|
||||
# Go
|
||||
/go.mod
|
||||
/go.sum
|
||||
|
||||
+49
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
||||
|
||||
---
|
||||
|
||||
## 8.0.0-beta.40 - 2026-10-04 - Comparison and source pages accept the sources: that cite add writes; sources may cite sources
|
||||
## 8.0.0-beta.41 - 2026-10-05 - raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -108,6 +108,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- Organisationsseiten: Personen als Abschnitt mit Aufstieg, entity_type organization, member-of, Lint-Befund broken_anchors
|
||||
- lint: Unfilled Template Sections - a section still holding only its template's TODO placeholders (advisory)
|
||||
- Comparison and source pages accept the sources: that cite add writes; sources may cite sources
|
||||
- raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
|
||||
|
||||
**Low impact**
|
||||
- version bump no longer points at version release in its output
|
||||
@@ -153,6 +154,53 @@ concern - readable here, never shipped as something to parse.
|
||||
- wiki-ingest and wiki-manage call xref add with --rel, not the --rel-a/--rel-b removed in 4.0.0
|
||||
<!-- /wikitool:bumps -->
|
||||
|
||||
### raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
|
||||
|
||||
Documentation from a git repository used to reach an instance only by hand: copy the files into
|
||||
`incoming/`, rename them around the global name rule, and remember nowhere which repository and
|
||||
commit they came from. Three commands replace that:
|
||||
|
||||
- **`raw capture <repo-url> --ref <rule> --path <glob>... --name <bundle> --fidelity <v>
|
||||
--authority <v>`** resolves the ref rule - a branch, or a tag pattern such as `v*` that takes the
|
||||
newest matching tag by version order - to one commit. It fetches that commit by ref name,
|
||||
shallowly, into a bare cache per URL under `tools/.wikitool_capture/` (gitignored, never
|
||||
exported), and writes the files the globs select into `incoming/<bundle>/` at their repository
|
||||
paths. Files are read as blobs, never through a checkout, so they are byte-identical to the
|
||||
repository even under `core.autocrlf`. Globs follow git's `:(glob)` pathspec, checked against
|
||||
git itself in the tests. `_capture.json` beside the files records repository, ref rule, commit,
|
||||
globs, capture time, `fidelity`, `authority` and the file list. `--update <raw-bundle>` captures
|
||||
the current state from that manifest.
|
||||
- **`raw status`** reports every captured bundle whose files changed in the repository, as
|
||||
`A`/`M`/`D` grouped by owning source page, and stays quiet about a commit that moved without a
|
||||
change inside the globs. An unreachable repository is one line, never an abort; `--json` is
|
||||
for the coming intake run.
|
||||
- **`raw accept --replaces-bundle <raw-bundle> incoming/<bundle>`** replaces a captured bundle as
|
||||
a whole at its existing address. Afterwards it holds exactly the new manifest's files plus
|
||||
`_capture.json`; files the repository dropped are removed, and emptied directories `rmdir`ed.
|
||||
`raw_files:` is left alone, as with `--replaces`, and the command prints the `touch --add/--remove
|
||||
raw_files=` lines that follow.
|
||||
|
||||
Mechanical exclusions, each named in the output: a file whose first line starts with
|
||||
`<!-- wikitool:export`, symlinks, submodules, files over 25 MiB, hidden path segments, files
|
||||
named `_capture.json`, and Git LFS pointers. Git runs with the host's credentials, limited to
|
||||
`ssh`/`https` through `GIT_ALLOW_PROTOCOL`, with no terminal prompt, no askpass and a timeout. A
|
||||
repository asking for a password is reported as unreachable rather than hanging an unattended
|
||||
run. A URL carrying a password is refused, because the manifest is committed.
|
||||
|
||||
`raw accept` takes a captured folder's capture fields from its manifest only, and refuses them on
|
||||
the command line. `--replaces` and `--page` refuse a target inside a captured bundle.
|
||||
|
||||
**Fixed along the way:** `sources coverage` and `lint` ignored every `CONTRACT.md` under `raw/`
|
||||
at any depth, not just the stage's own `raw/CONTRACT.md`. A captured repository's contract
|
||||
file would have been invisible. The exclusion is now anchored to `raw/CONTRACT.md`, and
|
||||
`_capture.json` is excluded instead. An instance with a `CONTRACT.md` somewhere below `raw/` may
|
||||
see it reported as uncovered for the first time - a finding about a file that was always there,
|
||||
nothing to migrate.
|
||||
|
||||
Drop-in in both directions: without a `_capture.json` under `raw/` no existing command behaves
|
||||
differently. An older version would report a manifest as an uncovered raw file. `raw/CONTRACT.md`
|
||||
§ "Getting a repository in" and `wiki-ingest` describe the workflow (Gitea #177).
|
||||
|
||||
### Comparison and source pages accept the sources: that cite add writes; sources may cite sources
|
||||
|
||||
`cite add` writes the cited source into the page's `sources:`, but neither the `comparison` nor
|
||||
|
||||
@@ -217,6 +217,14 @@ text derived from it with a header recording where and when it was fetched. Behi
|
||||
login, save the page from your browser into `incoming/` (HTML only) instead; the LLM derives the
|
||||
same text from that file with `raw fetch --html`. See [raw/CONTRACT.md](raw/CONTRACT.md).
|
||||
|
||||
Documentation that lives in a git repository is captured, not copied: `tools/wikitool raw capture
|
||||
<repo-url> --ref main --path 'docs/**/*.md' --name <bundle> ...` writes the matching files, byte
|
||||
for byte, into `incoming/<bundle>/` together with a `_capture.json` manifest naming the repository,
|
||||
the ref rule and the commit. Later, `tools/wikitool raw status` tells you which captured bundles
|
||||
have fallen behind their repository, file by file; `raw capture --update` and `raw accept
|
||||
--replaces-bundle` take the new edition in as a whole. Git uses your own keys and credential
|
||||
helpers - nothing is stored in the repository.
|
||||
|
||||
### Querying Knowledge
|
||||
|
||||
Ask questions naturally:
|
||||
|
||||
@@ -107,7 +107,8 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
(`work new --input <path>`) refuses any path outside `raw/`, so the hand-off needs the
|
||||
material already promoted; there is no later point at which this skill still controls the
|
||||
file. Ask `--fidelity`/`--authority` immediately, with the same posture step 5 states below,
|
||||
and run `raw accept` before switching over. This does not weaken the property step 5 exists
|
||||
and run `raw accept` before switching over - a folder `raw capture` wrote takes neither flag
|
||||
(step 5). This does not weaken the property step 5 exists
|
||||
for: a large-tree run is not atomic - it publishes unit by unit over days, and asks its own
|
||||
commitment question per unit, in that procedure's step 5d, long after this promotion. The
|
||||
raw-file-without-page state that stands until then is the one `sources coverage` and `lint`
|
||||
@@ -230,6 +231,12 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
A file inside a subdirectory of `incoming/` is refused on its own - the subdirectory is the
|
||||
source.
|
||||
|
||||
**A folder with a `_capture.json` at its top was written by `raw capture`** - documentation
|
||||
from a git repository (`raw/CONTRACT.md` "Getting a repository in"). Its capture fields were
|
||||
asked when it was captured and sit in that manifest: accept it with neither flag,
|
||||
`tools/wikitool raw accept incoming/<bundle>`, which refuses them. `_capture.json` goes in no
|
||||
`raw_files:`; the success message already leaves it out.
|
||||
|
||||
**A file that arrived through the MCP `submit` tool is not yet in `incoming/`** - it sits in
|
||||
`mcp-upload/<id>/`, a quarantine no command in this step reads. A reviewer promotes it first
|
||||
with `wikitool upload accept <id> --confirm <token>`, per
|
||||
@@ -329,7 +336,9 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
While drafting, cite every hard fact - an IP, port, version, path, command or config value -
|
||||
with `tools/wikitool cite add --page "<Name>" --source "Source - <Title>"`, which mints the
|
||||
`[^cite-id]`, upserts its Footnotes definition, and adds the source to `sources:`; paste the
|
||||
marker it prints at the fact.
|
||||
marker it prints at the fact. Citing one file of a captured bundle, pass `--file` with its path
|
||||
inside the bundle (`--file docs/runbook.md`), never its base name - a repository has a
|
||||
`README.md` in many directories.
|
||||
|
||||
8. **Create or update concept pages** - only if the source produced any. Same pattern, including
|
||||
step 7's rule about which subjects earn a page at all, reading
|
||||
@@ -399,6 +408,14 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
- **One source names far more subjects than usual?** That is breadth, not volume. It is not
|
||||
split into several sources - it cannot be - and it does not get a page per name either:
|
||||
`instructions/ingest-large-tree.md` § A broad source is not cut.
|
||||
- **The entry is a new edition of a captured bundle?** `raw status` reported it, `raw capture
|
||||
--update` wrote it, and `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>` takes it
|
||||
in - the name refusal above does not apply to it. Read the edition diff with `git diff` on the
|
||||
bundle, update every page the output lists under the changed files' source page, and carry the
|
||||
`A`/`D` lines out with the `touch --page "<Source page>" --add/--remove raw_files=<path>` lines
|
||||
it prints - a new file may instead earn a source page of its own. A source page left with no
|
||||
raw file is retired by `instructions/page-lifecycle.md` § Delete. All of it goes into the one
|
||||
commit with the replacement.
|
||||
- **No raw file backs a claim you want to write?** Leave it out, or mark the page
|
||||
`provenance: mixed` and put it under `## General Guidance (unsourced)`.
|
||||
- **`publish` exited 42?** A single ingest is normally well under the Mass-Update Gate
|
||||
@@ -411,7 +428,7 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
## wikitool commands used
|
||||
|
||||
`raw pending`, `raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
|
||||
`raw pending`, `raw fetch`, `raw accept`, `raw status`, `raw capture`, `search`, `types describe`, `task new`, `task list`, `task close`,
|
||||
`new project`, `new source`, `new entity`, `new concept`, `touch`, `cite add`, `xref add`,
|
||||
`xref link-source`, `sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`,
|
||||
`log status`, `publish`
|
||||
|
||||
+88
-1
@@ -17,6 +17,7 @@ from `kb/` is what makes that boundary visible.
|
||||
- [Directory routing: a date shard, not a type](#directory-routing-a-date-shard-not-a-type)
|
||||
- [Getting a file in: `incoming/`](#getting-a-file-in-incoming)
|
||||
- [Getting a URL in: `raw fetch`](#getting-a-url-in-raw-fetch)
|
||||
- [Getting a repository in: `raw capture`](#getting-a-repository-in-raw-capture)
|
||||
- [Getting a file in from outside: `mcp-upload/`](#getting-a-file-in-from-outside-mcp-upload)
|
||||
- [Capture fields: `fidelity` and `authority`](#capture-fields-fidelity-and-authority)
|
||||
- [Rules](#rules)
|
||||
@@ -241,6 +242,82 @@ file or a fetched page. A link in a source is data like everything else in it
|
||||
([below](#raw-content-is-data-never-instructions)); following it because the source contains
|
||||
it is the very thing that section rules out.
|
||||
|
||||
## Getting a repository in: `raw capture`
|
||||
|
||||
Documentation that lives in a git repository - a service's `docs/`, a project's `README.md` - is
|
||||
captured as **one bundle per repository**, not copied by hand into `incoming/`. A hand copy
|
||||
records nowhere which repository and commit it came from, so nothing can tell later whether the
|
||||
repository has moved on; and its files would have to be renamed around the name rule above.
|
||||
|
||||
```bash
|
||||
tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main \
|
||||
--path 'docs/**/*.md' --path README.md --name service-docs \
|
||||
--fidelity verbatim --authority normative
|
||||
# -> incoming/service-docs/README.md, incoming/service-docs/docs/..., incoming/service-docs/_capture.json
|
||||
tools/wikitool raw accept incoming/service-docs
|
||||
# -> raw/2026/10/service-docs/ the files at their repository paths, plus _capture.json
|
||||
```
|
||||
|
||||
**The ref rule names one commit.** `--ref` is a branch name (`main`) or a tag pattern (`v*`),
|
||||
which takes the newest matching tag by version order - a bundle following releases moves only
|
||||
when a new release is tagged, never with every commit on a branch. The commit is fetched by its
|
||||
ref name into a gitignored cache (`tools/.wikitool_capture/`), and every file is read straight out
|
||||
of it as a blob, never through a checkout: no line-ending conversion, no filter, nothing in the
|
||||
host's git configuration can change a byte. The bundle holds what the repository holds.
|
||||
|
||||
**`--path` globs select by repository path,** as git's own `:(glob)` pathspec does: `*`, `?` and
|
||||
`[...]` stop at `/`, `**` as a whole segment spans any number of directories, and a glob without
|
||||
wildcards also takes everything below it. Files keep their repository paths inside the bundle, so
|
||||
two `README.md` in different directories are no conflict - which is also why a citation of one
|
||||
names it by its path inside the bundle (`docs/runbook.md`), not by its base name.
|
||||
|
||||
**`_capture.json` is the bundle's manifest** - `repo`, the `ref` rule, the `commit`, the `paths`
|
||||
globs, when it was `captured`, `fidelity`, `authority` and the `files` list. It is the only
|
||||
declaration that and how this instance follows the repository; there is no second configuration
|
||||
file. The name is reserved: it is metadata of the bundle, not a source, so `sources coverage` and
|
||||
`lint` never report it, no `raw_files:` lists it, and no file of that name is captured from a
|
||||
repository or accepted from `incoming/` anywhere but at the top of a captured folder.
|
||||
|
||||
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
|
||||
starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its
|
||||
way out, which must not come back in as a source), a symlink, a submodule, a file over 25 MiB, a
|
||||
path with a segment starting with `.` (`raw accept` refuses hidden entries anyway), a file named
|
||||
`_capture.json`, and a Git LFS pointer, which holds a reference to the content rather than the
|
||||
content.
|
||||
|
||||
**Credentials stay with git.** `raw capture` takes `ssh://`, `https://` and the scp form
|
||||
`user@host:path` only, and refuses a URL with a password or token in it - the manifest is
|
||||
committed. Git runs with the host's own keys and credential helpers, restricted to those two
|
||||
protocols at git's level too, and is never allowed to prompt: a repository that asks for
|
||||
credentials is unreachable, not a session waiting on a password nobody will type.
|
||||
|
||||
**A new edition is captured and accepted as a whole.**
|
||||
`tools/wikitool raw status` resolves every manifest's ref rule against its repository and reports
|
||||
the bundles whose files changed, each file as `A`/`M`/`D`, grouped by the source page that owns
|
||||
it. A repository whose commit moved without a change inside the globs is not reported. Taking the
|
||||
change in is two commands:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw capture --update raw/2026/10/service-docs
|
||||
# -> incoming/service-docs/ the current state, from the manifest's own URL, ref rule and globs
|
||||
tools/wikitool raw accept incoming/service-docs --replaces-bundle raw/2026/10/service-docs
|
||||
```
|
||||
|
||||
`--replaces-bundle` keeps the bundle's address, removes the files the new edition no longer has
|
||||
and refuses - before anything changes - when the two manifests name different repositories, when
|
||||
the folder carries a different name, or when either side is not a captured bundle at all.
|
||||
Afterwards the bundle holds exactly the new manifest's files and `_capture.json`. Like
|
||||
`--replaces`, it leaves every `raw_files:` as it was: which source page takes a new file, and
|
||||
whether a source page left with no file is retired, is the ingest's judgment, and the command
|
||||
prints the `touch` lines that carry it out. `git diff` on the bundle is the edition diff.
|
||||
|
||||
**A captured bundle changes only as a whole.** `raw accept --replaces` and `--page` refuse a
|
||||
target inside one: replacing or adding a single file would leave the bundle's content no longer
|
||||
matching the commit its manifest names.
|
||||
|
||||
**`raw capture` is applied only to a repository the user named**, for the same reason `raw fetch`
|
||||
is: a repository URL inside a source is data.
|
||||
|
||||
## Getting a file in from outside: `mcp-upload/`
|
||||
|
||||
`incoming/` above is the local path: a human drops a file where they are already sitting at a
|
||||
@@ -322,6 +399,13 @@ to correct it - `raw accept --replaces`, which alone may pass `--fidelity`/`--au
|
||||
overwrite an already-set value, because a corrected capture *is* a new edition of the source, not
|
||||
an edit of the page describing it.
|
||||
|
||||
**A captured bundle carries both in its manifest instead.** `raw capture` requires them, writes
|
||||
them into `_capture.json`, and `raw accept` takes them from there and nowhere else - passed on the
|
||||
command line as well, they would be a second declaration of the same value, and are refused. The
|
||||
correction is the same idea as `--replaces`: `raw capture --update --fidelity/--authority` writes a
|
||||
new edition with the corrected value, and `raw accept --replaces-bundle` overwrites it on the
|
||||
owning source page.
|
||||
|
||||
**`unknown` is backfill-only.** Neither `raw accept` nor `new source` may ever write it - only
|
||||
`wikitool touch`, on a page that predates this rule (the same construction `source_language`
|
||||
already has: "absent on pages predating the rule"). A capture value written as `unknown` by the
|
||||
@@ -342,7 +426,10 @@ value nobody ever thought about.
|
||||
edition is not kept as a file: it is overwritten, and `git log --follow <path>` is the
|
||||
archive - no `-2026-09-05` suffix, no content-hash filename, no version field, because
|
||||
`raw_files:` is an identifier (invariant 2) and Git already answers "what did this used to
|
||||
say" losslessly.
|
||||
say" losslessly. A captured bundle is the one source replaced as a *bundle*: `raw accept
|
||||
--replaces-bundle` swaps all of its files for a new edition of the same repository at the
|
||||
same address, and a file the repository dropped is dropped from the bundle
|
||||
([Getting a repository in](#getting-a-repository-in-raw-capture)).
|
||||
- **Binary and image files still get ingested**, noting their presence and what they show,
|
||||
even when their content cannot be read directly.
|
||||
- **Every file is expected to be covered** by some source page, and one source page may cover
|
||||
|
||||
+130
-1
@@ -115,6 +115,8 @@ sources coverage read idempotent budget:counted exit:0
|
||||
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
|
||||
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
|
||||
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
|
||||
raw capture write non-idempotent budget:counted exit:0,1 Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
raw status read idempotent budget:counted exit:0 Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
|
||||
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
@@ -1455,6 +1457,120 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
#### `raw capture`
|
||||
|
||||
Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw capture <repo-url> --ref <branch|tag-pattern> --path <glob> [--path <glob> ...] --name <bundle> --fidelity <v> --authority <v>` - First capture of a repository
|
||||
- `wikitool raw capture --update <raw-bundle> [--fidelity <v>] [--authority <v>]` - Capture the current state of an already-accepted bundle, for `raw accept --replaces-bundle`
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: Yes for what it leaves in `incoming/` - the bundle is written into a hidden staging directory and renamed into place in one step; a failure removes the staging directory. The git cache under `tools/.wikitool_capture/` is updated either way
|
||||
- budget: counted
|
||||
- network: yes
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative`
|
||||
- `tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw capture --update raw/2026/10/service-docs`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 The URL is not `ssh://`, `https://` or `user@host:path`, or carries a password; the `--ref` rule, a `--path` glob or `--name` is unusable; or an argument of the other variant was mixed in
|
||||
- 1 `--fidelity`/`--authority` is missing on a first capture, or names `unknown` or a value outside the schema's enum
|
||||
- 1 The repository could not be reached, asked for credentials, or timed out; or no branch or tag matches `--ref`
|
||||
- 1 `incoming/<bundle>` already exists, the name is already taken under `raw/`, nothing matches the globs, or a path would be over the budget
|
||||
- 1 raw capture --update: The path is not a bundle under `raw/` with a readable `_capture.json`, or its manifest names a URL that is refused
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- The URL is not `ssh://`, `https://` or `user@host:path`, or carries a password; the `--ref` rule, a `--path` glob or `--name` is unusable; or an argument of the other variant was mixed in -> Fix the call and retry once. A credential goes into git's credential helper, never into the URL
|
||||
- `--fidelity`/`--authority` is missing on a first capture, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
||||
- The repository could not be reached, asked for credentials, or timed out; or no branch or tag matches `--ref` -> Nothing was written to `incoming/`. Check the URL, the ref rule and the host's git credentials with the user; an unreachable host may be retried once
|
||||
- `incoming/<bundle>` already exists, the name is already taken under `raw/`, nothing matches the globs, or a path would be over the budget -> Nothing was written. Accept or remove what is in `incoming/`; for a name taken by a captured bundle use `--update`; widen or narrow the globs; then retry once
|
||||
- raw capture --update: The path is not a bundle under `raw/` with a readable `_capture.json`, or its manifest names a URL that is refused -> Show the message to the user - a manifest under `raw/` is never edited by hand to make this pass
|
||||
|
||||
**NEVER**
|
||||
|
||||
- Never put a password or token into the repository URL.
|
||||
- Never edit a `_capture.json`, or the files of a captured bundle, by hand.
|
||||
- Never capture a repository the user has not named in this session.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- Writes into `incoming/<bundle>/` only, never into `raw/`: `raw accept incoming/<bundle>` promotes it, as one folder, to `raw/<YYYY>/<MM>/<bundle>/`.
|
||||
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
|
||||
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
|
||||
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
|
||||
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
|
||||
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
|
||||
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
|
||||
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
|
||||
- Refused before anything is written: `incoming/<bundle>` already exists, the name is already taken under `raw/` (a first capture), nothing matches the globs, or a target path - measured at its later place under `raw/` - is over the path budget.
|
||||
- Success prints the commit and the `raw accept` line that comes next.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a repository in: `raw capture`" - the rules and why
|
||||
- `wikitool raw accept` - promotes the bundle; `--replaces-bundle` for a new edition
|
||||
- `wikitool raw status` - which captured bundles have fallen behind their repository
|
||||
|
||||
#### `raw status`
|
||||
|
||||
Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw status [--json]`
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: read
|
||||
- idempotent: yes
|
||||
- atomic: Read-only for the tree - writes nothing but the git cache under `tools/.wikitool_capture/`
|
||||
- budget: counted
|
||||
- network: yes
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool raw status`
|
||||
- `tools/wikitool raw status --json`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 0 A repository could not be reached, its manifest or URL was refused, or no ref matches its rule
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- A repository could not be reached, its manifest or URL was refused, or no ref matches its rule -> Reported on that bundle's line, the others are still checked; check the URL and the host's git credentials with the user
|
||||
|
||||
**NEVER**
|
||||
|
||||
- Never edit a `_capture.json` to silence a line - a new edition goes through `raw capture --update` and `raw accept --replaces-bundle`.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- Reads every `_capture.json` under `raw/` and resolves its ref rule with `git ls-remote`. Only where the commit moved is the new commit fetched, and compared - with the same globs and exclusions `raw capture` applies - against the bundle's files under `raw/`.
|
||||
- A bundle whose commit moved without a change inside its globs is reported as unchanged, not listed. A tag pattern follows new matching tags only, never new commits on a branch.
|
||||
- Each changed bundle prints `old -> new` and its files as `A`/`M`/`D`, grouped by owning source page with its citing pages, followed by the `raw capture --update` and `raw accept --replaces-bundle` lines that take the new edition in.
|
||||
- An unreachable repository, a refused URL or an unreadable manifest is one line beside the others, never an abort; the exit status is 0 as long as the command itself ran.
|
||||
- `--json` prints one object per bundle: `bundle`, `repo`, `ref`, `old`, `new`, `changed`, `files` (`[{path, status}]`), `error` (or null).
|
||||
- Git runs exactly as for `raw capture`: the host's credentials, only `ssh`/`https`, never a prompt, 120 s per call.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool raw capture` - `--update` captures the new edition
|
||||
- `wikitool raw accept` - `--replaces-bundle` takes it in
|
||||
- `raw/CONTRACT.md` "Getting a repository in: `raw capture`"
|
||||
|
||||
#### `raw pending`
|
||||
|
||||
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
@@ -1504,12 +1620,13 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
|
||||
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
|
||||
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
|
||||
- `wikitool raw accept incoming/<bundle> --replaces-bundle <raw-bundle> [--dry-run]` - Replace a captured bundle as a whole with the new edition `raw capture --update` wrote
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write. `raw accept --replaces-bundle`: No - one `unlink()` per removed file, one `rename()` per added or modified file, then the manifest, then one `rmdir` per emptied directory, plus one page write per owning page whose capture fields change; a half-replaced bundle is not resumed
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
@@ -1519,6 +1636,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
|
||||
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
|
||||
- `tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
@@ -1531,6 +1649,9 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
|
||||
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
|
||||
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
|
||||
- 1 raw accept --replaces / --page: The target, or a file the page already has, lies in a captured bundle
|
||||
- 1 raw accept incoming/<bundle>: A captured folder is given `--fidelity`/`--authority`, its `_capture.json` cannot be read or names an invalid value, or its files differ from the ones the manifest lists
|
||||
- 1 raw accept --replaces-bundle: Either side has no `_capture.json`, the two manifests name different repositories, the folder's name differs from the bundle's, `--page`, `--replaces`, `--fidelity` or `--authority` was also given, the folder does not match its manifest, a new path is over the budget, or a file of the bundle has more than one owning source page
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
@@ -1542,6 +1663,9 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
|
||||
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
|
||||
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
|
||||
- raw accept --replaces / --page: The target, or a file the page already has, lies in a captured bundle -> Not fixed by retrying: a captured bundle changes only as a whole - `raw capture --update <raw-bundle>`, then `raw accept --replaces-bundle`
|
||||
- raw accept incoming/<bundle>: A captured folder is given `--fidelity`/`--authority`, its `_capture.json` cannot be read or names an invalid value, or its files differ from the ones the manifest lists -> Nothing moved. Drop the two flags and retry once; a manifest that does not match its folder is fixed by removing the folder and capturing it again, never by editing either
|
||||
- raw accept --replaces-bundle: Either side has no `_capture.json`, the two manifests name different repositories, the folder's name differs from the bundle's, `--page`, `--replaces`, `--fidelity` or `--authority` was also given, the folder does not match its manifest, a new path is over the budget, or a file of the bundle has more than one owning source page -> Nothing under `raw/` or in `incoming/` changed. Fix what the message names and retry once; different repositories or a different name are a separate source, not a new edition - show the message to the user
|
||||
|
||||
**NEVER**
|
||||
|
||||
@@ -1563,12 +1687,17 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
- `--replaces` refuses a target with more than one owning source page; with none, it replaces anyway and says so. It cannot be combined with `--page` or with more than one incoming file.
|
||||
- `--replaces` prints the source page (if any) and its citing pages, so their update lands in the same commit as the replacement.
|
||||
- A file already at its computed destination is what "already exists" reports, not a partial prior run to resume - safe to retry as-is once a cause is fixed.
|
||||
- A folder carrying `_capture.json` at its top is a captured bundle (`raw capture`): `--fidelity`/`--authority` come from that manifest only and are refused on the command line; the folder must hold exactly the files the manifest lists. `_capture.json` goes in no `raw_files:`.
|
||||
- `--replaces-bundle <raw-bundle>` replaces a captured bundle under `raw/` with the new edition in `incoming/<bundle>` at the same address: files gone from the new edition are removed, emptied directories `rmdir`ed, and afterwards the bundle holds exactly the new manifest's files plus `_capture.json`. `raw_files:` is left untouched; an owning source page's `fidelity`/`authority` is overwritten where the new manifest differs.
|
||||
- `--replaces-bundle` prints each changed file as `A`/`M`/`D`, grouped by owning source page with its citing pages, and the `touch --page ... --add/--remove raw_files=` lines that follow; `git diff` on the bundle shows the edition's changes.
|
||||
- `--replaces` and `--page` refuse a target inside a captured bundle; the refusal names `raw capture --update` and `--replaces-bundle`. A file named `_capture.json` anywhere but at a captured folder's top is refused - the name is reserved.
|
||||
- `--dry-run` reports the moves without making them.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
|
||||
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
|
||||
- `wikitool raw capture` - writes a captured bundle, or its new edition, into `incoming/`
|
||||
- `wikitool types describe source` - the capture field values
|
||||
- `wikitool new source` - the source page for a promoted file
|
||||
|
||||
|
||||
@@ -106,6 +106,7 @@ tools/
|
||||
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
|
||||
corpus_diff.py invariant comparison of kb/ between two revisions
|
||||
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
|
||||
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt
|
||||
search/ pluggable search backends, plus service.py - the search core
|
||||
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (.wikitool-tasks.json), one module per adapter - no instruction ever learns which provider it is
|
||||
commands/ one module per command or command group: the terminal adapters
|
||||
|
||||
@@ -259,7 +259,7 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
"sources coverage", "sources trace", "sources rebuild-index",
|
||||
)),
|
||||
("Raw material and uploads", (
|
||||
"raw fetch", "raw pending", "raw accept",
|
||||
"raw fetch", "raw capture", "raw status", "raw pending", "raw accept",
|
||||
"upload list", "upload show", "upload accept", "upload reject",
|
||||
)),
|
||||
("Git", (
|
||||
|
||||
@@ -127,7 +127,7 @@ HOOK_DIRS = (".github/hooks", ".vibe")
|
||||
# it, measured against the source repo's own test run. Matched by directory
|
||||
# name at any depth, which is right for a cache and wrong for anything with an
|
||||
# ordinary name - that is what `TOOLS_DEV_ONLY` below is for.
|
||||
TOOLS_EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", ".wikitool_session", "htmlcov"}
|
||||
TOOLS_EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", ".wikitool_session", ".wikitool_capture", "htmlcov"}
|
||||
|
||||
# tools/-relative paths that are dev-only rather than derived: the test suite
|
||||
# and the two files that configure running and measuring it. The suite tests
|
||||
|
||||
@@ -160,6 +160,9 @@ REQUIRED_IGNORE_CANARIES = (
|
||||
".wikitool-tasks.json",
|
||||
# The live suite's tracker profiles (Gitea #156) - credentials again, one file per tracker.
|
||||
".wikitool-tasks.d/probe.json",
|
||||
# `raw capture`'s git cache (Gitea #177) - fetched repositories, recomputable,
|
||||
# and in the way of `publish`'s `git add -A` like the coverage output above.
|
||||
"tools/.wikitool_capture/0123abcd/HEAD",
|
||||
)
|
||||
REQUIRED_TRACKED_PATHS = (
|
||||
"reports/CONTRACT.md",
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+11
-3
@@ -167,9 +167,15 @@ def __getattr__(name: str):
|
||||
def __dir__() -> list[str]:
|
||||
return sorted([*globals(), "ROOT", *_DERIVED, *_KB_DERIVED])
|
||||
|
||||
# Files/patterns to ignore when scanning raw/ for ingest coverage.
|
||||
# CONTRACT.md is the layer's source contract, not source material.
|
||||
RAW_IGNORE_NAMES = {".gitkeep", ".DS_Store", "CONTRACT.md"}
|
||||
# Files to ignore when scanning raw/ for ingest coverage, by name at any depth.
|
||||
# `_capture.json` is a captured bundle's manifest (`chemenu.repo_capture`) -
|
||||
# metadata of the bundle, not source material; `raw capture` never takes a
|
||||
# repository file of that name, so every one under raw/ is a manifest.
|
||||
RAW_IGNORE_NAMES = {".gitkeep", ".DS_Store", "_capture.json"}
|
||||
|
||||
# Ignored only directly in raw/: the stage's own contract. Matched by name it
|
||||
# hid every `CONTRACT.md` a captured repository brought along (Gitea #177).
|
||||
RAW_IGNORE_TOP_LEVEL = {"CONTRACT.md"}
|
||||
|
||||
# Per-instance personalization: who operates this wiki (`USER.md`) and how this
|
||||
# instance sounds while doing it (`SOUL.md`). Both are read every session and
|
||||
@@ -325,5 +331,7 @@ def iter_raw_files(raw_dir: Path):
|
||||
continue
|
||||
if path.name in RAW_IGNORE_NAMES or path.name.startswith("."):
|
||||
continue
|
||||
if path.parent == raw_dir and path.name in RAW_IGNORE_TOP_LEVEL:
|
||||
continue
|
||||
yield path
|
||||
|
||||
@@ -0,0 +1,578 @@
|
||||
"""Capturing documentation from a git repository for `raw/`: resolve a ref
|
||||
rule to one commit, read the files the globs select straight out of that
|
||||
commit, and describe the result in a manifest - with no CLI attached.
|
||||
|
||||
`wikitool raw capture`, `raw status` and `raw accept --replaces-bundle` are the
|
||||
terminal adapters over this module; it decides nothing about `incoming/` or
|
||||
`raw/` paths beyond the manifest's own name.
|
||||
|
||||
Three properties carry the design, and each has a test:
|
||||
|
||||
- **Byte-identical to the blob.** Files are read with `git cat-file` out of a
|
||||
bare cache repository, never from a checked-out working tree, so
|
||||
`core.autocrlf`, a smudge filter or anything else in the host's git
|
||||
configuration cannot change a byte.
|
||||
- **One selection, two callers.** `select_files` is the only place the globs
|
||||
and the exclusions are applied; `raw capture` writes what it returns and
|
||||
`raw status` compares against it. Two copies of the exclusion list would let
|
||||
every excluded file show up as a permanent `A` in `raw status`.
|
||||
- **Git, not wikitool, holds the credentials.** Git runs with the host's own
|
||||
configuration and credential helpers. Nothing here reads, stores or passes a
|
||||
secret, a URL carrying a password is refused (the manifest is committed), and
|
||||
no call may prompt: an unattended intake run must find a repository that asks
|
||||
for credentials unreachable, not hang on it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import urllib.parse
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterator, Optional
|
||||
|
||||
from chemenu import config, filelock, toolpaths, web_capture
|
||||
from chemenu.errors import BackendError, ValidationError
|
||||
|
||||
MANIFEST_NAME = "_capture.json"
|
||||
SCHEMA = 1
|
||||
|
||||
# What `git` may talk to, as URL schemes and as `GIT_ALLOW_PROTOCOL` alike. The
|
||||
# test suite widens this tuple through a fixture to reach local test
|
||||
# repositories over `file://`; nothing the command itself reads - no option, no
|
||||
# environment variable - can, because `GIT_ALLOW_PROTOCOL` is set from this
|
||||
# tuple on every call and overrides whatever the caller's environment held.
|
||||
ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
|
||||
|
||||
GIT_TIMEOUT_SECONDS = 120.0
|
||||
|
||||
EXPORT_MARKER = b"<!-- wikitool:export"
|
||||
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
|
||||
_BOM = b"\xef\xbb\xbf"
|
||||
|
||||
_MODE_SYMLINK = "120000"
|
||||
_MODE_SUBMODULE = "160000"
|
||||
|
||||
# `user@host:path` - git's scp-like form. A colon before the first slash is
|
||||
# what makes git read it that way; `::` is excluded because that is the
|
||||
# `<transport>::<address>` form (`ext::`, `fd::`), never an scp address.
|
||||
_SCP_FORM = re.compile(r"[A-Za-z0-9._~-]+@[A-Za-z0-9.-]+:(?!:)[^\s]+")
|
||||
_SCHEME = re.compile(r"([A-Za-z][A-Za-z0-9+.-]*)://")
|
||||
|
||||
|
||||
# --- URLs ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def check_repo_url(url: str) -> None:
|
||||
"""Refuse anything but `ssh://`, `https://` and `user@host:path`, and any
|
||||
URL that carries a password. Raises `ValidationError`.
|
||||
|
||||
Applied to a URL from the command line and to one read back out of a
|
||||
manifest under `raw/` alike: a manifest is committed text, not a trusted
|
||||
command."""
|
||||
if not isinstance(url, str) or not url or url != url.strip() or url.startswith("-"):
|
||||
raise ValidationError(f"{url!r} is not a repository URL.")
|
||||
if any(ord(c) < 0x20 for c in url):
|
||||
raise ValidationError(f"{url!r} contains a control character.")
|
||||
allowed = ", ".join(f"{s}://" for s in ALLOWED_SCHEMES)
|
||||
scheme_match = _SCHEME.match(url)
|
||||
if scheme_match is None:
|
||||
if _SCP_FORM.fullmatch(url) and "ssh" in ALLOWED_SCHEMES:
|
||||
return
|
||||
raise ValidationError(
|
||||
f"{url!r} is not a repository URL raw capture takes - only {allowed} and the scp form "
|
||||
"user@host:path. A local path, file://, ext:: and fd:: are refused."
|
||||
)
|
||||
scheme = scheme_match.group(1).lower()
|
||||
if scheme not in ALLOWED_SCHEMES:
|
||||
raise ValidationError(
|
||||
f"{url!r} uses {scheme}://, which raw capture does not take - only {allowed} and the "
|
||||
"scp form user@host:path."
|
||||
)
|
||||
parsed = urllib.parse.urlsplit(url)
|
||||
if parsed.password is not None:
|
||||
raise ValidationError(
|
||||
f"The URL for {parsed.hostname or 'this repository'} carries a password or token in it. "
|
||||
"The manifest is committed, so a credential never goes into the URL - configure it in "
|
||||
"git's own credential helper instead, and pass the URL without it."
|
||||
)
|
||||
|
||||
|
||||
# --- globs --------------------------------------------------------------------
|
||||
|
||||
|
||||
_WILDCARDS = frozenset("*?[\\")
|
||||
|
||||
|
||||
def check_glob(pattern: str) -> None:
|
||||
if not pattern or pattern.startswith("/") or "\0" in pattern:
|
||||
raise ValidationError(
|
||||
f"--path {pattern!r} is not a usable glob: it is matched against the repository's "
|
||||
"paths, relative to its root, so it is not empty and does not start with '/'."
|
||||
)
|
||||
if any(part in ("", ".", "..") for part in pattern.rstrip("/").split("/")):
|
||||
raise ValidationError(f"--path {pattern!r} has an empty, '.' or '..' segment.")
|
||||
|
||||
|
||||
def _segment_regex(segment: str) -> str:
|
||||
out: list[str] = []
|
||||
i, n = 0, len(segment)
|
||||
while i < n:
|
||||
c = segment[i]
|
||||
if c == "*":
|
||||
out.append("[^/]*")
|
||||
elif c == "?":
|
||||
out.append("[^/]")
|
||||
elif c == "\\" and i + 1 < n:
|
||||
i += 1
|
||||
out.append(re.escape(segment[i]))
|
||||
elif c == "[":
|
||||
j = i + 1
|
||||
negate = j < n and segment[j] in "!^"
|
||||
if negate:
|
||||
j += 1
|
||||
if j < n and segment[j] == "]":
|
||||
j += 1
|
||||
while j < n and segment[j] != "]":
|
||||
j += 1
|
||||
if j >= n:
|
||||
out.append(re.escape(c)) # an unclosed '[' is a literal, as in git
|
||||
else:
|
||||
body = segment[i + 1 + (1 if negate else 0):j]
|
||||
body = body.replace("\\", "\\\\")
|
||||
out.append(f"[^/{body}]" if negate else f"(?!/)[{body}]")
|
||||
i = j
|
||||
else:
|
||||
out.append(re.escape(c))
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def glob_regex(pattern: str) -> "re.Pattern[str]":
|
||||
"""The regex for one glob, with git's `:(glob)` pathspec semantics: `*`,
|
||||
`?` and `[...]` never cross a `/`; `**` as a whole segment spans any
|
||||
number of directories, none included; anywhere else it is a plain `*`.
|
||||
|
||||
`fnmatch` would not do - its `*` crosses `/` - and `PurePath.full_match`
|
||||
only exists from Python 3.13."""
|
||||
segments = pattern.split("/")
|
||||
parts: list[str] = []
|
||||
for index, segment in enumerate(segments):
|
||||
last = index == len(segments) - 1
|
||||
if segment == "**":
|
||||
parts.append(".*" if last else "(?:[^/]*/)*")
|
||||
continue
|
||||
parts.append(_segment_regex(segment))
|
||||
if not last:
|
||||
parts.append("/")
|
||||
return re.compile("".join(parts), re.DOTALL)
|
||||
|
||||
|
||||
def glob_matches(pattern: str, path: str) -> bool:
|
||||
"""Whether repository path `path` falls under `pattern`. A pattern with no
|
||||
wildcard also matches everything below it as a directory - `docs` takes
|
||||
`docs/x/b.md` - the way git's own pathspec does; a pattern with one does
|
||||
not (`docs/x*` takes nothing below `docs/x/`)."""
|
||||
if not any(c in _WILDCARDS for c in pattern):
|
||||
literal = pattern.rstrip("/")
|
||||
return path == literal or path.startswith(literal + "/")
|
||||
return glob_regex(pattern).fullmatch(path) is not None
|
||||
|
||||
|
||||
# --- git ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _git_env() -> dict[str, str]:
|
||||
env = dict(os.environ)
|
||||
for name in ("GIT_DIR", "GIT_WORK_TREE", "GIT_ASKPASS", "SSH_ASKPASS"):
|
||||
env.pop(name, None)
|
||||
env["GIT_ALLOW_PROTOCOL"] = ":".join(ALLOWED_SCHEMES)
|
||||
env["GIT_TERMINAL_PROMPT"] = "0"
|
||||
env["SSH_ASKPASS_REQUIRE"] = "never"
|
||||
env["GCM_INTERACTIVE"] = "never"
|
||||
return env
|
||||
|
||||
|
||||
def run_git(
|
||||
args: list[str],
|
||||
git_dir: Path,
|
||||
*,
|
||||
stdin: Optional[bytes] = None,
|
||||
timeout: float = GIT_TIMEOUT_SECONDS,
|
||||
) -> bytes:
|
||||
"""Run `git --git-dir=<git_dir> <args>` and return its stdout.
|
||||
|
||||
No call can prompt: no terminal prompt, no askpass program (an empty
|
||||
`core.askPass` stops git from falling back to `SSH_ASKPASS`), and on POSIX a
|
||||
session of its own, so `ssh` has no controlling terminal to ask on either.
|
||||
A timeout kills the whole process group, `ssh` included - killing `git`
|
||||
alone would leave the pipe open and the read below hanging. Raises
|
||||
`BackendError` on a non-zero exit, a timeout or a git that cannot start."""
|
||||
argv = [toolpaths.git(), "-c", "core.askPass=", f"--git-dir={git_dir}", *args]
|
||||
posix = os.name == "posix"
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
argv,
|
||||
stdin=subprocess.PIPE if stdin is not None else subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
env=_git_env(),
|
||||
start_new_session=posix,
|
||||
)
|
||||
except OSError as exc:
|
||||
raise BackendError(f"git could not be started: {exc}") from exc
|
||||
try:
|
||||
out, err = proc.communicate(stdin, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
if posix:
|
||||
try:
|
||||
os.killpg(proc.pid, signal.SIGKILL)
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
proc.kill()
|
||||
proc.communicate()
|
||||
raise BackendError(f"git {args[0]} did not finish within {timeout:.0f} s") from None
|
||||
if proc.returncode != 0:
|
||||
message = err.decode("utf-8", "replace").strip().splitlines()
|
||||
raise BackendError(f"git {args[0]} failed: {message[-1] if message else f'exit {proc.returncode}'}")
|
||||
return out
|
||||
|
||||
|
||||
def cache_root() -> Path:
|
||||
return config.ROOT / "tools" / ".wikitool_capture"
|
||||
|
||||
|
||||
def _url_key(url: str) -> str:
|
||||
return hashlib.sha256(url.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def cache_repo(url: str) -> Iterator[Path]:
|
||||
"""The bare cache repository for `url`, created on first use, held under an
|
||||
exclusive lock for the duration of the block - two runs against one URL
|
||||
must not fetch into the same repository at once."""
|
||||
root = cache_root()
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
key = _url_key(url)
|
||||
repo = root / key
|
||||
with open(root / f"{key}.lock", "a+b") as handle, filelock.exclusive(handle):
|
||||
if not (repo / "HEAD").is_file():
|
||||
repo.mkdir(exist_ok=True)
|
||||
run_git(["init", "--bare", "-q"], repo)
|
||||
yield repo
|
||||
|
||||
|
||||
# --- refs ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def _version_key(name: str) -> tuple:
|
||||
# `git tag --sort=-v:refname` without a configured suffix order: runs of
|
||||
# digits compare as numbers, everything else as text. `re.split` with a
|
||||
# capture group always alternates text, digits, text, ... starting with
|
||||
# text, so the tuple positions never mix types.
|
||||
parts = re.split(r"(\d+)", name)
|
||||
return tuple(int(p) if i % 2 else p for i, p in enumerate(parts))
|
||||
|
||||
|
||||
def is_tag_pattern(rule: str) -> bool:
|
||||
return any(c in rule for c in "*?[")
|
||||
|
||||
|
||||
def check_ref_rule(rule: str) -> None:
|
||||
if not rule or rule.startswith("-") or any(c.isspace() or ord(c) < 0x20 for c in rule):
|
||||
raise ValidationError(
|
||||
f"--ref {rule!r} is neither a branch name (main) nor a tag pattern (v*)."
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedRef:
|
||||
refname: str # what is fetched: refs/heads/<branch> or refs/tags/<tag>
|
||||
commit: str # the commit it names - peeled, for an annotated tag
|
||||
|
||||
|
||||
def resolve_ref(url: str, rule: str, repo: Path) -> ResolvedRef:
|
||||
"""Resolve `rule` against the remote's advertised refs: a branch name names
|
||||
`refs/heads/<rule>`; a rule with a wildcard is a tag pattern and names the
|
||||
newest matching tag by version order. Raises `BackendError` when the
|
||||
remote is unreachable, `ValidationError` when nothing matches."""
|
||||
out = run_git(["ls-remote", "--heads", "--tags", "--", url], repo)
|
||||
direct: dict[str, str] = {}
|
||||
peeled: dict[str, str] = {}
|
||||
for line in out.decode("utf-8", "replace").splitlines():
|
||||
sha, _, ref = line.partition("\t")
|
||||
if not ref:
|
||||
continue
|
||||
if ref.endswith("^{}"):
|
||||
peeled[ref[:-3]] = sha
|
||||
else:
|
||||
direct[ref] = sha
|
||||
if is_tag_pattern(rule):
|
||||
regex = glob_regex(rule)
|
||||
tags = [
|
||||
ref[len("refs/tags/"):]
|
||||
for ref in direct
|
||||
if ref.startswith("refs/tags/") and regex.fullmatch(ref[len("refs/tags/"):])
|
||||
]
|
||||
if not tags:
|
||||
raise ValidationError(f"No tag in {url} matches --ref {rule!r}.")
|
||||
refname = f"refs/tags/{max(tags, key=_version_key)}"
|
||||
else:
|
||||
refname = f"refs/heads/{rule}"
|
||||
if refname not in direct:
|
||||
raise ValidationError(f"{url} has no branch {rule!r}.")
|
||||
return ResolvedRef(refname, peeled.get(refname, direct[refname]))
|
||||
|
||||
|
||||
def fetch(url: str, refname: str, repo: Path) -> str:
|
||||
"""Fetch `refname` shallowly into `repo`, by name - never by SHA, which a
|
||||
server only serves with `allowReachableSHA1InWant` - and return the commit
|
||||
it pointed at when fetched."""
|
||||
run_git(["fetch", "--depth", "1", "--no-tags", "-q", "--", url, f"+{refname}:refs/wikitool/fetched"], repo)
|
||||
return run_git(["rev-parse", "--verify", "refs/wikitool/fetched^{commit}"], repo).decode().strip()
|
||||
|
||||
|
||||
# --- selection ----------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TreeEntry:
|
||||
mode: str
|
||||
kind: str
|
||||
sha: str
|
||||
size: Optional[int]
|
||||
path: str
|
||||
|
||||
|
||||
def list_tree(repo: Path, commit: str) -> list[TreeEntry]:
|
||||
out = run_git(["ls-tree", "-r", "-z", "--long", commit], repo)
|
||||
entries: list[TreeEntry] = []
|
||||
for record in out.split(b"\0"):
|
||||
if not record:
|
||||
continue
|
||||
meta, _, raw_path = record.partition(b"\t")
|
||||
mode, kind, sha, size = meta.decode("ascii").split(None, 3)
|
||||
try:
|
||||
path = raw_path.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
path = raw_path.decode("utf-8", "backslashreplace")
|
||||
kind = "undecodable"
|
||||
entries.append(TreeEntry(mode, kind, sha, None if size.strip() == "-" else int(size), path))
|
||||
return entries
|
||||
|
||||
|
||||
def read_blobs(repo: Path, shas: list[str]) -> dict[str, bytes]:
|
||||
"""Every blob in `shas`, read raw with `cat-file --batch` - no filter, no
|
||||
line-ending conversion."""
|
||||
if not shas:
|
||||
return {}
|
||||
unique = list(dict.fromkeys(shas))
|
||||
out = run_git(["cat-file", "--batch"], repo, stdin="".join(f"{s}\n" for s in unique).encode())
|
||||
blobs: dict[str, bytes] = {}
|
||||
pos = 0
|
||||
for sha in unique:
|
||||
newline = out.index(b"\n", pos)
|
||||
header = out[pos:newline].decode("ascii").split()
|
||||
if len(header) != 3:
|
||||
raise BackendError(f"git cat-file could not read {sha}: {' '.join(header)}")
|
||||
size = int(header[2])
|
||||
start = newline + 1
|
||||
blobs[sha] = out[start:start + size]
|
||||
pos = start + size + 1
|
||||
return blobs
|
||||
|
||||
|
||||
@dataclass
|
||||
class Selection:
|
||||
files: dict[str, bytes] = field(default_factory=dict) # repo path -> blob bytes
|
||||
excluded: list[tuple[str, str]] = field(default_factory=list) # (repo path, reason)
|
||||
|
||||
|
||||
def _structural_exclusion(entry: TreeEntry) -> Optional[str]:
|
||||
if entry.mode == _MODE_SYMLINK:
|
||||
return "symlink"
|
||||
if entry.mode == _MODE_SUBMODULE or entry.kind == "commit":
|
||||
return "submodule"
|
||||
if entry.kind == "undecodable":
|
||||
return "path is not UTF-8"
|
||||
if entry.kind != "blob":
|
||||
return f"not a file ({entry.kind})"
|
||||
if any(part.startswith(".") for part in entry.path.split("/")):
|
||||
return "hidden path segment"
|
||||
if entry.path.rsplit("/", 1)[-1] == MANIFEST_NAME:
|
||||
return f"reserved name {MANIFEST_NAME}"
|
||||
if entry.size is not None and entry.size > web_capture.MAX_BYTES:
|
||||
return f"over {web_capture.MAX_BYTES // (1024 * 1024)} MiB"
|
||||
return None
|
||||
|
||||
|
||||
def _content_exclusion(data: bytes) -> Optional[str]:
|
||||
if data.startswith(LFS_MARKER):
|
||||
return "Git LFS pointer - the content is not in the repository"
|
||||
first = data[len(_BOM):] if data.startswith(_BOM) else data
|
||||
if first.startswith(EXPORT_MARKER):
|
||||
return "guideline export (first line starts with <!-- wikitool:export)"
|
||||
return None
|
||||
|
||||
|
||||
def select_files(repo: Path, commit: str, globs: list[str]) -> Selection:
|
||||
"""The files of `commit` that match at least one glob, minus the
|
||||
exclusions - each excluded path named with its reason. The single place
|
||||
both are applied (module docstring)."""
|
||||
selection = Selection()
|
||||
candidates: list[TreeEntry] = []
|
||||
for entry in list_tree(repo, commit):
|
||||
if not any(glob_matches(g, entry.path) for g in globs):
|
||||
continue
|
||||
reason = _structural_exclusion(entry)
|
||||
if reason:
|
||||
selection.excluded.append((entry.path, reason))
|
||||
else:
|
||||
candidates.append(entry)
|
||||
blobs = read_blobs(repo, [e.sha for e in candidates])
|
||||
for entry in candidates:
|
||||
data = blobs[entry.sha]
|
||||
reason = _content_exclusion(data)
|
||||
if reason:
|
||||
selection.excluded.append((entry.path, reason))
|
||||
else:
|
||||
selection.files[entry.path] = data
|
||||
selection.excluded.sort()
|
||||
return selection
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Snapshot:
|
||||
refname: str
|
||||
commit: str
|
||||
selection: Selection
|
||||
|
||||
|
||||
def snapshot(url: str, rule: str, globs: list[str]) -> Snapshot:
|
||||
"""Resolve, fetch and select in one go, under the cache lock."""
|
||||
with cache_repo(url) as repo:
|
||||
resolved = resolve_ref(url, rule, repo)
|
||||
commit = fetch(url, resolved.refname, repo)
|
||||
return Snapshot(resolved.refname, commit, select_files(repo, commit, globs))
|
||||
|
||||
|
||||
def remote_commit(url: str, rule: str) -> ResolvedRef:
|
||||
"""`resolve_ref` without a fetch - what `raw status` asks first, so an
|
||||
unchanged repository costs one `ls-remote` and nothing else."""
|
||||
with cache_repo(url) as repo:
|
||||
return resolve_ref(url, rule, repo)
|
||||
|
||||
|
||||
# --- manifest -----------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Manifest:
|
||||
repo: str
|
||||
ref: str
|
||||
commit: str
|
||||
paths: tuple[str, ...]
|
||||
captured: str
|
||||
fidelity: str
|
||||
authority: str
|
||||
files: tuple[str, ...]
|
||||
|
||||
def to_json(self) -> str:
|
||||
data = {
|
||||
"schema": SCHEMA,
|
||||
"repo": self.repo,
|
||||
"ref": self.ref,
|
||||
"commit": self.commit,
|
||||
"paths": list(self.paths),
|
||||
"captured": self.captured,
|
||||
"fidelity": self.fidelity,
|
||||
"authority": self.authority,
|
||||
"files": sorted(self.files),
|
||||
}
|
||||
return json.dumps(data, indent=2, ensure_ascii=False) + "\n"
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
)
|
||||
|
||||
|
||||
def read_manifest(path: Path) -> Manifest:
|
||||
"""Parse `_capture.json`. Raises `ValidationError` naming what is wrong
|
||||
with it - including a repository URL `check_repo_url` refuses."""
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
raise ValidationError(f"{MANIFEST_NAME} cannot be read: {exc}") from exc
|
||||
if not isinstance(data, dict) or data.get("schema") != SCHEMA:
|
||||
raise ValidationError(f"{MANIFEST_NAME} is not schema {SCHEMA}.")
|
||||
strings = ("repo", "ref", "commit", "captured", "fidelity", "authority")
|
||||
lists = ("paths", "files")
|
||||
bad = [k for k in strings if not isinstance(data.get(k), str) or not data.get(k)]
|
||||
bad += [
|
||||
k for k in lists
|
||||
if not isinstance(data.get(k), list) or not all(isinstance(v, str) and v for v in data[k])
|
||||
]
|
||||
if bad:
|
||||
raise ValidationError(f"{MANIFEST_NAME} lacks a valid {', '.join(bad)}.")
|
||||
check_repo_url(data["repo"])
|
||||
check_ref_rule(data["ref"])
|
||||
for pattern in data["paths"]:
|
||||
check_glob(pattern)
|
||||
for rel in data["files"]:
|
||||
parts = rel.split("/")
|
||||
if rel.startswith("/") or "\\" in rel or any(p in ("", ".", "..") or p.startswith(".") for p in parts):
|
||||
raise ValidationError(f"{MANIFEST_NAME} lists a file path it may not: {rel!r}.")
|
||||
return Manifest(
|
||||
data["repo"], data["ref"], data["commit"], tuple(data["paths"]), data["captured"],
|
||||
data["fidelity"], data["authority"], tuple(data["files"]),
|
||||
)
|
||||
|
||||
|
||||
def bundle_files(bundle: Path) -> dict[str, Path]:
|
||||
"""Every file of a captured bundle on disk, keyed by its repository path -
|
||||
the manifest itself left out."""
|
||||
found: dict[str, Path] = {}
|
||||
for path in sorted(bundle.rglob("*")):
|
||||
if path.is_file() and not path.is_symlink():
|
||||
rel = path.relative_to(bundle).as_posix()
|
||||
if rel != MANIFEST_NAME:
|
||||
found[rel] = path
|
||||
return found
|
||||
|
||||
|
||||
def captured_bundle_of(path: Path, raw_dir: Path) -> Optional[Path]:
|
||||
"""The captured bundle `path` lies in - the nearest directory above it,
|
||||
below `raw_dir`, holding a `_capture.json` - or None."""
|
||||
try:
|
||||
path.relative_to(raw_dir)
|
||||
except ValueError:
|
||||
return None
|
||||
current = path if path.is_dir() else path.parent
|
||||
while current != raw_dir and raw_dir in current.parents:
|
||||
if (current / MANIFEST_NAME).is_file():
|
||||
return current
|
||||
current = current.parent
|
||||
return None
|
||||
|
||||
|
||||
def diff(files_on_disk: dict[str, Path], new: dict[str, bytes]) -> list[tuple[str, str]]:
|
||||
"""`(status, repo path)` for every file that differs between a bundle on
|
||||
disk and a new selection: `A` new, `M` bytes differ, `D` gone."""
|
||||
changes: list[tuple[str, str]] = []
|
||||
for rel in sorted(set(files_on_disk) | set(new)):
|
||||
if rel not in files_on_disk:
|
||||
changes.append(("A", rel))
|
||||
elif rel not in new:
|
||||
changes.append(("D", rel))
|
||||
elif files_on_disk[rel].read_bytes() != new[rel]:
|
||||
changes.append(("M", rel))
|
||||
return changes
|
||||
@@ -340,11 +340,13 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
|
||||
not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list
|
||||
since `--latest`, which asks the release feed and downloads from it, and `raw fetch`
|
||||
since it exists (Gitea #120) - its `--html` form stays offline, which does not turn the
|
||||
command back to `no`. Pinned as an explicit
|
||||
command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177). Pinned as an explicit
|
||||
set so a command gaining or losing that reach is a deliberate edit here, not a silent
|
||||
drift between the property and what the command actually does."""
|
||||
expected = {
|
||||
"raw fetch",
|
||||
"raw capture",
|
||||
"raw status",
|
||||
"sync",
|
||||
"publish",
|
||||
"version check",
|
||||
|
||||
@@ -0,0 +1,733 @@
|
||||
"""`raw capture`, `raw status` and `raw accept --replaces-bundle` (Gitea #177),
|
||||
against local test repositories - never the network.
|
||||
|
||||
The `file` transport those repositories need is unlocked by the `capture`
|
||||
fixture widening `repo_capture.ALLOWED_SCHEMES`, the one switch the command
|
||||
itself cannot reach: it sets `GIT_ALLOW_PROTOCOL` from that tuple on every git
|
||||
call, so no option and no environment variable opens it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import typer
|
||||
|
||||
from chemenu import config, repo_capture
|
||||
from chemenu.commands.raw_cmd import (
|
||||
raw_accept_command,
|
||||
raw_capture_command,
|
||||
raw_pending_command,
|
||||
raw_status_command,
|
||||
)
|
||||
from chemenu.errors import BackendError, ValidationError
|
||||
from chemenu.frontmatter_io import read_page, write_page
|
||||
from chemenu.kb_scan import load_kb_pages
|
||||
from chemenu.provenance import uncovered_raw_files
|
||||
|
||||
posix_only = pytest.mark.skipif(os.name != "posix", reason="needs a POSIX shell and symlinks")
|
||||
|
||||
|
||||
def _git(cwd: Path, *args: str) -> str:
|
||||
return subprocess.run(
|
||||
["git", "-c", "user.name=Fixture", "-c", "user.email=f@example.org", *args],
|
||||
cwd=cwd, check=True, capture_output=True, text=True,
|
||||
).stdout.strip()
|
||||
|
||||
|
||||
class Repo:
|
||||
"""A work repository whose `file://` URL is what the commands are given."""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = path
|
||||
path.mkdir(parents=True)
|
||||
_git(path, "init", "-q", "-b", "main")
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return self.path.as_uri()
|
||||
|
||||
def write(self, rel: str, data) -> None:
|
||||
target = self.path / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if isinstance(data, str):
|
||||
data = data.encode("utf-8")
|
||||
target.write_bytes(data)
|
||||
|
||||
def remove(self, rel: str) -> None:
|
||||
_git(self.path, "rm", "-q", rel)
|
||||
|
||||
def commit(self, message: str = "change") -> str:
|
||||
_git(self.path, "add", "-A")
|
||||
_git(self.path, "commit", "-q", "--allow-empty", "-m", message)
|
||||
return _git(self.path, "rev-parse", "HEAD")
|
||||
|
||||
def tag(self, name: str, annotated: bool = False) -> None:
|
||||
if annotated:
|
||||
_git(self.path, "tag", "-a", name, "-m", name)
|
||||
else:
|
||||
_git(self.path, "tag", name)
|
||||
|
||||
|
||||
def _shard() -> str:
|
||||
today = datetime.date.today()
|
||||
return f"raw/{today.year:04d}/{today.month:02d}"
|
||||
|
||||
|
||||
def _out(capsys) -> str:
|
||||
return " ".join(capsys.readouterr().out.split())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def capture(kb_dir, monkeypatch):
|
||||
root = kb_dir.parent
|
||||
(root / "raw").mkdir(exist_ok=True)
|
||||
(root / "incoming").mkdir(exist_ok=True)
|
||||
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file"))
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(tmp_path):
|
||||
r = Repo(tmp_path / "upstream")
|
||||
r.write("README.md", "# Service\r\nCRLF line endings, kept as they are.\r\n")
|
||||
r.write("docs/a.md", "# A\n")
|
||||
r.write("docs/x/y/b.md", "# B\n")
|
||||
r.write("docs/config.yaml", "key: value\n")
|
||||
r.write("src/main.py", "print('hi')\n")
|
||||
r.commit("initial")
|
||||
return r
|
||||
|
||||
|
||||
def _capture(url=None, ref="main", paths=("docs/**/*.md", "README.md"), name="svc",
|
||||
fidelity="verbatim", authority="normative", update=None):
|
||||
return raw_capture_command(
|
||||
repo_url=url, ref=None if update else ref, paths=None if update else list(paths),
|
||||
name=None if update else name, fidelity=fidelity, authority=authority,
|
||||
update=Path(update) if update else None,
|
||||
)
|
||||
|
||||
|
||||
def _update(bundle, fidelity=None, authority=None):
|
||||
return raw_capture_command(
|
||||
repo_url=None, ref=None, paths=None, name=None, fidelity=fidelity, authority=authority,
|
||||
update=Path(bundle),
|
||||
)
|
||||
|
||||
|
||||
def _accept(path, **kwargs):
|
||||
defaults = dict(fidelity=None, authority=None, page=None, replaces=None, dry_run=False)
|
||||
defaults.update(kwargs)
|
||||
return raw_accept_command(files=[Path(path)], **defaults)
|
||||
|
||||
|
||||
def _tree(*dirs: Path) -> dict[str, bytes]:
|
||||
"""Every file below `dirs`, with its bytes - what "nothing changed" is
|
||||
asserted against."""
|
||||
snapshot = {}
|
||||
for top in dirs:
|
||||
for path in sorted(top.rglob("*")):
|
||||
if path.is_file():
|
||||
snapshot[path.as_posix()] = path.read_bytes()
|
||||
return snapshot
|
||||
|
||||
|
||||
def _write_source(kb_dir, title, raw_files, fidelity="verbatim", authority="normative"):
|
||||
write_page(
|
||||
kb_dir / "sources" / f"{title}.md",
|
||||
{
|
||||
"type": "types/source.md", "source_type": "document", "author": "Fixture",
|
||||
"raw_files": list(raw_files), "date": "2026-10-01", "tags": [], "entities": [],
|
||||
"concepts": [], "summary": "Test source.", "fidelity": fidelity, "authority": authority,
|
||||
},
|
||||
f"\n# {title}\n\n## Summary\n\nTest.\n",
|
||||
)
|
||||
|
||||
|
||||
def _captured_and_accepted(capture, repo, **kwargs) -> Path:
|
||||
_capture(repo.url, **kwargs)
|
||||
_accept(capture / "incoming" / kwargs.get("name", "svc"))
|
||||
return capture / _shard() / kwargs.get("name", "svc")
|
||||
|
||||
|
||||
# --- globs --------------------------------------------------------------------
|
||||
|
||||
|
||||
GLOB_FILES = (
|
||||
"README.md", "docs/a.md", "docs/x/b.md", "docs/x/y/c.md", "docs/x/y/c.yaml", "docs/.h.md",
|
||||
".github/w.md", "docx/n.md", "docs/x[1].md",
|
||||
)
|
||||
GLOB_PATTERNS = (
|
||||
"docs/*.md", "docs/**/*.md", "docs", "docs/", "docs/x", "**/*.md", "docs/**", "*.md", "do*",
|
||||
"docs/x*", "docs/**b.md", "*", "**", "docs/?.md", "docs/[ab].md", "docs/[!a].md", "**/c.*",
|
||||
"docs/x/**/c.md",
|
||||
)
|
||||
|
||||
|
||||
def test_glob_semantics_are_gits_own(tmp_path):
|
||||
"""Every pattern selects exactly what `git ls-files ':(glob)<p>'` selects -
|
||||
the reference the issue names, asked of git itself rather than restated."""
|
||||
work = tmp_path / "globs"
|
||||
work.mkdir()
|
||||
_git(work, "init", "-q", "-b", "main")
|
||||
for rel in GLOB_FILES:
|
||||
(work / rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
(work / rel).write_text("x\n", encoding="utf-8")
|
||||
_git(work, "add", "-A")
|
||||
for pattern in GLOB_PATTERNS:
|
||||
expected = set(_git(work, "ls-files", "--", f":(glob){pattern}").splitlines())
|
||||
actual = {rel for rel in GLOB_FILES if repo_capture.glob_matches(pattern, rel)}
|
||||
assert actual == expected, pattern
|
||||
|
||||
|
||||
def test_named_glob_cases():
|
||||
assert repo_capture.glob_matches("docs/**/*.md", "docs/a.md")
|
||||
assert repo_capture.glob_matches("docs/**/*.md", "docs/x/y/b.md")
|
||||
assert not repo_capture.glob_matches("docs/*.md", "docs/x/b.md")
|
||||
assert not repo_capture.glob_matches("docs/**/*.md", "docs/config.yaml")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pattern", ["", "/docs/**", "docs/../x", "docs//a.md", "./docs"])
|
||||
def test_unusable_globs_are_refused(pattern):
|
||||
with pytest.raises(ValidationError):
|
||||
repo_capture.check_glob(pattern)
|
||||
|
||||
|
||||
# --- capture + accept ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_capture_and_accept_hold_exactly_the_matching_files_byte_for_byte(capture, repo, monkeypatch, tmp_path):
|
||||
# The host's git would convert line endings on a checkout; a blob read must not.
|
||||
gitconfig = tmp_path / "gitconfig"
|
||||
gitconfig.write_text("[core]\n\tautocrlf = true\n", encoding="utf-8")
|
||||
monkeypatch.setenv("GIT_CONFIG_GLOBAL", str(gitconfig))
|
||||
|
||||
_capture(repo.url)
|
||||
incoming = capture / "incoming" / "svc"
|
||||
manifest = json.loads((incoming / "_capture.json").read_text(encoding="utf-8"))
|
||||
assert manifest["schema"] == 1
|
||||
assert manifest["repo"] == repo.url
|
||||
assert manifest["ref"] == "main"
|
||||
assert manifest["commit"] == _git(repo.path, "rev-parse", "HEAD")
|
||||
assert manifest["paths"] == ["docs/**/*.md", "README.md"]
|
||||
assert manifest["fidelity"] == "verbatim" and manifest["authority"] == "normative"
|
||||
assert manifest["files"] == ["README.md", "docs/a.md", "docs/x/y/b.md"]
|
||||
assert manifest["captured"].endswith("Z")
|
||||
|
||||
_accept(incoming)
|
||||
bundle = capture / _shard() / "svc"
|
||||
on_disk = sorted(p.relative_to(bundle).as_posix() for p in bundle.rglob("*") if p.is_file())
|
||||
assert on_disk == ["README.md", "_capture.json", "docs/a.md", "docs/x/y/b.md"]
|
||||
for rel in ("README.md", "docs/a.md", "docs/x/y/b.md"):
|
||||
assert (bundle / rel).read_bytes() == (repo.path / rel).read_bytes()
|
||||
assert (bundle / "README.md").read_bytes().count(b"\r\n") == 2
|
||||
assert not incoming.exists()
|
||||
|
||||
|
||||
def test_accepting_a_captured_folder_takes_the_capture_fields_from_its_manifest(capture, repo, capsys):
|
||||
_capture(repo.url, fidelity="published", authority="reporting")
|
||||
_accept(capture / "incoming" / "svc")
|
||||
out = _out(capsys)
|
||||
assert "--set fidelity=published --set authority=reporting" in out
|
||||
assert "_capture.json" not in out.split("--set raw_files=")[1].split()[0]
|
||||
|
||||
|
||||
def test_accepting_a_captured_folder_with_capture_flags_is_refused(capture, repo):
|
||||
_capture(repo.url)
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc", fidelity="verbatim", authority="normative")
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_a_captured_folder_edited_after_capture_is_refused(capture, repo, capsys):
|
||||
_capture(repo.url)
|
||||
(capture / "incoming" / "svc" / "docs" / "extra.md").write_text("added by hand\n", encoding="utf-8")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc")
|
||||
assert "no longer matches" in _out(capsys)
|
||||
|
||||
|
||||
def test_pending_judges_a_captured_folder_by_its_manifest(capture, repo, capsys):
|
||||
_capture(repo.url)
|
||||
capsys.readouterr()
|
||||
raw_pending_command(json_out=True)
|
||||
[candidate] = json.loads(capsys.readouterr().out)
|
||||
assert candidate["kind"] == "folder" and candidate["acceptable"] is True
|
||||
|
||||
|
||||
def test_capture_refuses_an_existing_incoming_folder(capture, repo):
|
||||
(capture / "incoming" / "svc").mkdir()
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(repo.url)
|
||||
assert list((capture / "incoming" / "svc").iterdir()) == []
|
||||
|
||||
|
||||
def test_capture_refuses_a_name_a_captured_bundle_holds_and_names_update(capture, repo, capsys):
|
||||
_captured_and_accepted(capture, repo)
|
||||
capsys.readouterr()
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(repo.url)
|
||||
assert "raw capture --update" in _out(capsys)
|
||||
assert not (capture / "incoming" / "svc").exists()
|
||||
|
||||
|
||||
def test_capture_refuses_when_nothing_matches(capture, repo):
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(repo.url, paths=("nothing/**",))
|
||||
assert list((capture / "incoming").iterdir()) == []
|
||||
|
||||
|
||||
def test_capture_refuses_a_path_over_the_budget_before_writing(capture, tmp_path, capsys):
|
||||
r = Repo(tmp_path / "deep")
|
||||
deep = "/".join(["d" * 30] * 6) + "/page.md"
|
||||
r.write(deep, "# deep\n")
|
||||
r.commit()
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(r.url, paths=("**",))
|
||||
assert "path budget" in _out(capsys)
|
||||
assert list((capture / "incoming").iterdir()) == []
|
||||
|
||||
|
||||
def test_capture_follows_the_newest_tag_by_version_order(capture, repo):
|
||||
repo.tag("v1.9")
|
||||
repo.write("docs/a.md", "# A at 1.10\n")
|
||||
v110 = repo.commit()
|
||||
repo.tag("v1.10", annotated=True)
|
||||
repo.write("docs/a.md", "# A on main, untagged\n")
|
||||
repo.commit()
|
||||
_capture(repo.url, ref="v*")
|
||||
incoming = capture / "incoming" / "svc"
|
||||
manifest = json.loads((incoming / "_capture.json").read_text(encoding="utf-8"))
|
||||
assert manifest["commit"] == v110 # the peeled commit, not the tag object
|
||||
assert (incoming / "docs/a.md").read_text(encoding="utf-8") == "# A at 1.10\n"
|
||||
|
||||
|
||||
def test_capture_refuses_a_branch_that_does_not_exist(capture, repo):
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(repo.url, ref="nope")
|
||||
assert list((capture / "incoming").iterdir()) == []
|
||||
|
||||
|
||||
# --- exclusions ---------------------------------------------------------------
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_excluded_paths_are_never_captured_and_each_is_named(capture, repo, capsys):
|
||||
repo.write("docs/guideline.md", "<!-- wikitool:export from=kb -->\n# Exported\n")
|
||||
repo.write(".github/workflow.md", "# CI\n")
|
||||
repo.write("docs/big.bin", "version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 9\n")
|
||||
repo.write("docs/sub/_capture.json", "{}\n")
|
||||
os.symlink("a.md", repo.path / "docs" / "link.md")
|
||||
sub = repo.commit()
|
||||
_git(repo.path, "update-index", "--add", "--cacheinfo", f"160000,{sub},vendor/lib")
|
||||
_git(repo.path, "commit", "-q", "-m", "submodule")
|
||||
|
||||
_capture(repo.url, paths=("**",))
|
||||
out = _out(capsys)
|
||||
for path, reason in (
|
||||
("docs/guideline.md", "guideline export"),
|
||||
(".github/workflow.md", "hidden path segment"),
|
||||
("docs/big.bin", "Git LFS pointer"),
|
||||
("docs/sub/_capture.json", "reserved name"),
|
||||
("docs/link.md", "symlink"),
|
||||
("vendor/lib", "submodule"),
|
||||
):
|
||||
assert f"excluded {path} ({reason}" in out
|
||||
files = json.loads((capture / "incoming" / "svc" / "_capture.json").read_text(encoding="utf-8"))["files"]
|
||||
assert files == ["README.md", "docs/a.md", "docs/config.yaml", "docs/x/y/b.md", "src/main.py"]
|
||||
|
||||
_accept(capture / "incoming" / "svc")
|
||||
repo.write("docs/a.md", "# A, changed\n")
|
||||
repo.commit()
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert row["files"] == [{"path": f"{_shard()}/svc/docs/a.md", "status": "M"}]
|
||||
|
||||
|
||||
def test_a_file_over_the_size_limit_is_excluded(capture, repo, monkeypatch, capsys):
|
||||
from chemenu import web_capture
|
||||
|
||||
monkeypatch.setattr(web_capture, "MAX_BYTES", 5)
|
||||
_capture(repo.url, paths=("docs/x/**",))
|
||||
assert "excluded docs/x/y/b.md (over" not in _out(capsys) # 4 bytes: under
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(repo.url, paths=("README.md",), name="svc2")
|
||||
assert "excluded README.md (over" in _out(capsys)
|
||||
|
||||
|
||||
# --- URLs and credentials -----------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", [
|
||||
"ext::sh -c touch% /tmp/x",
|
||||
"fd::17",
|
||||
"file:///srv/repo.git",
|
||||
"/srv/repo.git",
|
||||
"http://example.org/repo.git",
|
||||
"https://user:token@example.org/repo.git",
|
||||
"ssh://user:secret@example.org/repo.git",
|
||||
"-uhelp@example.org:x",
|
||||
])
|
||||
def test_refused_urls(url):
|
||||
with pytest.raises(ValidationError):
|
||||
repo_capture.check_repo_url(url)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", [
|
||||
"https://example.org/team/repo.git",
|
||||
"https://user@example.org/team/repo.git",
|
||||
"ssh://git@example.org:2222/team/repo.git",
|
||||
"git@example.org:team/repo.git",
|
||||
])
|
||||
def test_accepted_urls(url):
|
||||
repo_capture.check_repo_url(url)
|
||||
|
||||
|
||||
def _ext_url(marker: Path) -> str:
|
||||
return f"ext::sh -c touch% {marker}"
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_capture_refuses_an_ext_url_and_runs_nothing(capture, tmp_path):
|
||||
marker = tmp_path / "ext-ran"
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(_ext_url(marker))
|
||||
assert not marker.exists()
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_git_itself_refuses_the_ext_transport(capture, tmp_path):
|
||||
"""Below the URL check: the git layer alone, handed the URL directly."""
|
||||
marker = tmp_path / "ext-ran"
|
||||
with pytest.raises(BackendError):
|
||||
with repo_capture.cache_repo("ext-probe") as cache:
|
||||
repo_capture.resolve_ref(_ext_url(marker), "main", cache)
|
||||
assert not marker.exists()
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_the_ext_probe_would_run_if_git_allowed_it(capture, tmp_path, monkeypatch):
|
||||
"""The control for the two tests above: the same URL does run its command
|
||||
once `ext` is allowed, so their empty marker means something."""
|
||||
marker = tmp_path / "ext-ran"
|
||||
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ext",))
|
||||
with pytest.raises(BackendError):
|
||||
with repo_capture.cache_repo("ext-probe") as cache:
|
||||
repo_capture.resolve_ref(_ext_url(marker), "main", cache)
|
||||
assert marker.exists()
|
||||
|
||||
|
||||
def test_a_refused_url_leaves_no_credential_anywhere(capture, tmp_path):
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture("https://user:s3cr3t-token@example.org/repo.git")
|
||||
for path in tmp_path.rglob("*"):
|
||||
if path.is_file():
|
||||
assert b"s3cr3t-token" not in path.read_bytes(), path
|
||||
|
||||
|
||||
def _plant_bundle(capture, name, url, commit="0" * 40, files=("README.md",)):
|
||||
"""A captured bundle under raw/ as an earlier accept would have left it."""
|
||||
bundle = capture / _shard() / name
|
||||
bundle.mkdir(parents=True)
|
||||
for rel in files:
|
||||
(bundle / rel).write_text("planted\n", encoding="utf-8")
|
||||
manifest = {
|
||||
"schema": 1, "repo": url, "ref": "main", "commit": commit, "paths": ["README.md"],
|
||||
"captured": "2026-10-01T00:00:00Z", "fidelity": "verbatim", "authority": "normative",
|
||||
"files": list(files),
|
||||
}
|
||||
(bundle / "_capture.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
return bundle
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_update_and_status_refuse_an_ext_url_read_from_a_manifest(capture, tmp_path, capsys):
|
||||
marker = tmp_path / "ext-ran"
|
||||
bundle = _plant_bundle(capture, "evil", _ext_url(marker))
|
||||
with pytest.raises(typer.Exit):
|
||||
_update(bundle)
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert "refused" in row["error"]
|
||||
assert not marker.exists()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def asks_for_credentials(monkeypatch):
|
||||
"""An HTTP server that answers every request with a Basic-auth challenge -
|
||||
the repository that would make an unguarded git prompt for a password."""
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self): # noqa: N802 - http.server's name
|
||||
self.send_response(401)
|
||||
self.send_header("WWW-Authenticate", 'Basic realm="repo"')
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file", "http"))
|
||||
yield f"http://127.0.0.1:{server.server_address[1]}/repo.git"
|
||||
server.shutdown()
|
||||
|
||||
|
||||
def test_a_repository_asking_for_credentials_is_unreachable_not_a_hang(capture, asks_for_credentials, capsys):
|
||||
_plant_bundle(capture, "locked", asks_for_credentials)
|
||||
started = time.monotonic()
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert row["error"].startswith("not reachable")
|
||||
with pytest.raises(typer.Exit):
|
||||
_capture(asks_for_credentials, name="locked2")
|
||||
assert time.monotonic() - started < 30
|
||||
|
||||
|
||||
# --- raw status ---------------------------------------------------------------
|
||||
|
||||
|
||||
def test_status_reports_changed_docs_and_goes_quiet_after_the_replacement(capture, repo, capsys):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
repo.write("docs/a.md", "# A, second edition\n")
|
||||
repo.remove("docs/x/y/b.md")
|
||||
repo.write("docs/new.md", "# New\n")
|
||||
repo.commit()
|
||||
capsys.readouterr()
|
||||
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
prefix = f"{_shard()}/svc"
|
||||
assert row["changed"] is True and row["error"] is None
|
||||
assert row["new"] == _git(repo.path, "rev-parse", "HEAD")
|
||||
assert row["files"] == [
|
||||
{"path": f"{prefix}/docs/a.md", "status": "M"},
|
||||
{"path": f"{prefix}/docs/new.md", "status": "A"},
|
||||
{"path": f"{prefix}/docs/x/y/b.md", "status": "D"},
|
||||
]
|
||||
|
||||
raw_status_command(json_out=False)
|
||||
out = _out(capsys)
|
||||
assert f"raw capture --update {prefix}" in out
|
||||
assert f"--replaces-bundle {prefix}" in out
|
||||
|
||||
_update(bundle)
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert row["changed"] is False and row["files"] == []
|
||||
|
||||
|
||||
def test_status_ignores_a_change_outside_the_globs(capture, repo, capsys):
|
||||
_captured_and_accepted(capture, repo)
|
||||
repo.write("src/main.py", "print('changed')\n")
|
||||
repo.commit()
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert row["old"] != row["new"] and row["changed"] is False
|
||||
raw_status_command(json_out=False)
|
||||
out = _out(capsys)
|
||||
assert "1 unchanged" in out and "svc" not in out.split("raw/:")[1].replace("1 unchanged.", "")
|
||||
|
||||
|
||||
def test_status_follows_new_tags_only_not_new_commits_on_main(capture, repo, capsys):
|
||||
repo.tag("v1.0")
|
||||
_captured_and_accepted(capture, repo, ref="v*")
|
||||
repo.write("docs/a.md", "# A on main\n")
|
||||
repo.commit()
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
assert json.loads(capsys.readouterr().out)[0]["changed"] is False
|
||||
repo.tag("v1.1")
|
||||
raw_status_command(json_out=True)
|
||||
[row] = json.loads(capsys.readouterr().out)
|
||||
assert row["changed"] is True
|
||||
assert row["files"] == [{"path": f"{_shard()}/svc/docs/a.md", "status": "M"}]
|
||||
|
||||
|
||||
def test_an_unreachable_repository_is_one_line_beside_the_others(capture, repo, tmp_path, capsys):
|
||||
_captured_and_accepted(capture, repo)
|
||||
_plant_bundle(capture, "gone", (tmp_path / "does-not-exist").as_uri())
|
||||
repo.write("docs/a.md", "# A, changed\n")
|
||||
repo.commit()
|
||||
capsys.readouterr()
|
||||
raw_status_command(json_out=True)
|
||||
rows = {Path(r["bundle"]).name: r for r in json.loads(capsys.readouterr().out)}
|
||||
assert rows["gone"]["error"].startswith("not reachable")
|
||||
assert rows["svc"]["changed"] is True and rows["svc"]["error"] is None
|
||||
raw_status_command(json_out=False) # exits normally, both on their own line
|
||||
out = capsys.readouterr().out
|
||||
assert "not reachable" in out and "docs/a.md" in out
|
||||
|
||||
|
||||
def test_status_with_no_captured_bundle(capture, capsys):
|
||||
raw_status_command(json_out=False)
|
||||
assert "No captured bundle" in capsys.readouterr().out
|
||||
|
||||
|
||||
# --- raw accept --replaces-bundle ---------------------------------------------
|
||||
|
||||
|
||||
def test_replaces_bundle_leaves_exactly_the_new_edition_in_place(capture, repo, kb_dir, capsys):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
prefix = f"{_shard()}/svc"
|
||||
owned = [f"{prefix}/README.md", f"{prefix}/docs/a.md", f"{prefix}/docs/x/y/b.md"]
|
||||
_write_source(kb_dir, "Source - Svc", owned)
|
||||
|
||||
repo.write("docs/a.md", "# A, second edition\n")
|
||||
repo.remove("docs/x/y/b.md")
|
||||
repo.write("docs/new.md", "# New\n")
|
||||
new_commit = repo.commit()
|
||||
_update(bundle)
|
||||
capsys.readouterr()
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
|
||||
out = _out(capsys)
|
||||
|
||||
on_disk = sorted(p.relative_to(bundle).as_posix() for p in bundle.rglob("*") if p.is_file())
|
||||
assert on_disk == ["README.md", "_capture.json", "docs/a.md", "docs/new.md"]
|
||||
for rel in ("README.md", "docs/a.md", "docs/new.md"):
|
||||
assert (bundle / rel).read_bytes() == (repo.path / rel).read_bytes()
|
||||
assert not (bundle / "docs" / "x").exists()
|
||||
assert json.loads((bundle / "_capture.json").read_text(encoding="utf-8"))["commit"] == new_commit
|
||||
assert not (capture / "incoming" / "svc").exists()
|
||||
assert read_page(kb_dir / "sources" / "Source - Svc.md")[0]["raw_files"] == owned
|
||||
|
||||
assert f"M {prefix}/docs/a.md" in out and f"D {prefix}/docs/x/y/b.md" in out
|
||||
assert f"A {prefix}/docs/new.md" in out
|
||||
assert f'touch --page "Source - Svc" --remove raw_files={prefix}/docs/x/y/b.md' in out
|
||||
assert f'touch --page "Source - Svc" --add raw_files={prefix}/docs/new.md' in out
|
||||
|
||||
|
||||
def test_replaces_bundle_overwrites_changed_capture_fields_on_the_owning_page(capture, repo, kb_dir):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
_write_source(kb_dir, "Source - Svc", [f"{_shard()}/svc/README.md"])
|
||||
_update(bundle, authority="reporting")
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
|
||||
assert read_page(kb_dir / "sources" / "Source - Svc.md")[0]["authority"] == "reporting"
|
||||
|
||||
|
||||
def test_replaces_bundle_refuses_another_repository(capture, repo, tmp_path):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
other = Repo(tmp_path / "other")
|
||||
other.write("README.md", "# Other\n")
|
||||
other.commit()
|
||||
_capture(other.url, paths=("README.md",), name="other")
|
||||
(capture / "incoming" / "other").rename(capture / "incoming" / "svc")
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_replaces_bundle_refuses_a_different_folder_name(capture, repo):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
_update(bundle)
|
||||
(capture / "incoming" / "svc").rename(capture / "incoming" / "svc-renamed")
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc-renamed", replaces_bundle=bundle)
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_replaces_bundle_refuses_a_bundle_without_a_manifest(capture, repo):
|
||||
plain = capture / _shard() / "svc"
|
||||
plain.mkdir(parents=True)
|
||||
(plain / "README.md").write_text("hand-made folder bundle\n", encoding="utf-8")
|
||||
_capture(repo.url, name="svc-new")
|
||||
(capture / "incoming" / "svc-new").rename(capture / "incoming" / "svc")
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=plain)
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_replaces_bundle_refuses_page_replaces_and_capture_flags(capture, repo):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
_update(bundle)
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
for extra in ({"page": "Source - Svc"}, {"replaces": bundle / "README.md"}, {"fidelity": "verbatim"}):
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle, **extra)
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_replaces_bundle_refuses_a_file_with_two_owners(capture, repo, kb_dir):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
readme = f"{_shard()}/svc/README.md"
|
||||
_write_source(kb_dir, "Source - One", [readme])
|
||||
_write_source(kb_dir, "Source - Two", [readme])
|
||||
_update(bundle)
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
# --- --replaces / --page inside a captured bundle -----------------------------
|
||||
|
||||
|
||||
def test_replaces_refuses_a_target_inside_a_captured_bundle(capture, repo, capsys):
|
||||
bundle = _captured_and_accepted(capture, repo)
|
||||
(capture / "incoming" / "b.md").write_text("# B by hand\n", encoding="utf-8")
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "b.md", replaces=bundle / "docs" / "x" / "y" / "b.md")
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
assert "--replaces-bundle" in _out(capsys)
|
||||
|
||||
|
||||
def test_page_refuses_to_grow_a_captured_bundle(capture, tmp_path, kb_dir):
|
||||
flat = Repo(tmp_path / "flat")
|
||||
flat.write("README.md", "# R\n")
|
||||
flat.write("AGENTS.md", "# A\n")
|
||||
flat.commit()
|
||||
_capture(flat.url, paths=("*.md",), name="flat")
|
||||
_accept(capture / "incoming" / "flat")
|
||||
prefix = f"{_shard()}/flat"
|
||||
_write_source(kb_dir, "Source - Flat", [f"{prefix}/AGENTS.md", f"{prefix}/README.md"])
|
||||
(capture / "incoming" / "notes.md").write_text("# notes\n", encoding="utf-8")
|
||||
before = _tree(capture / "incoming", capture / "raw")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "notes.md", fidelity="verbatim", authority="normative",
|
||||
page="Source - Flat")
|
||||
assert _tree(capture / "incoming", capture / "raw") == before
|
||||
|
||||
|
||||
def test_a_loose_file_named_like_the_manifest_is_refused(capture):
|
||||
(capture / "incoming" / "_capture.json").write_text("{}\n", encoding="utf-8")
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(capture / "incoming" / "_capture.json", fidelity="verbatim", authority="normative")
|
||||
|
||||
|
||||
# --- coverage -----------------------------------------------------------------
|
||||
|
||||
|
||||
def test_coverage_skips_the_manifest_and_sees_a_bundled_contract(capture, tmp_path):
|
||||
r = Repo(tmp_path / "with-contract")
|
||||
r.write("README.md", "# R\n")
|
||||
r.write("raw/CONTRACT.md", "# a repository's own stage contract\n")
|
||||
r.commit()
|
||||
_capture(r.url, paths=("README.md", "raw/CONTRACT.md"), name="wc")
|
||||
_accept(capture / "incoming" / "wc")
|
||||
(capture / "raw" / "CONTRACT.md").write_text("# the stage contract\n", encoding="utf-8")
|
||||
|
||||
uncovered = uncovered_raw_files(config.RAW_DIR, load_kb_pages(config.KB_DIR))
|
||||
prefix = f"{_shard()}/wc"
|
||||
assert f"{prefix}/raw/CONTRACT.md" in uncovered
|
||||
assert f"{prefix}/README.md" in uncovered
|
||||
assert f"{prefix}/_capture.json" not in uncovered
|
||||
assert "raw/CONTRACT.md" not in uncovered
|
||||
Reference in new issue
Block a user