feat: raw capture / raw status / --replaces-bundle - documentation from git repositories as a bundle, with drift reporting (#177)
CI / verify (push) Successful in 5m39s
CI / pwsh (push) Successful in 2m15s
Release / release (push) Successful in 36s

New repo_capture module: resolve a branch or tag-pattern ref rule, fetch it
shallowly by name into a bare cache, read the glob-selected files as blobs,
and record repo/ref/commit/globs/capture fields in _capture.json. raw status
reports changed captured bundles as A/M/D; raw accept --replaces-bundle swaps
a captured bundle for its new edition at the same address. iter_raw_files now
skips _capture.json and anchors the CONTRACT.md exclusion to raw/CONTRACT.md.

Files changed:
- .gitignore
- CHANGES.md
- README.md
- VERSION
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/config.py
- tools/chemenu/repo_capture.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_raw_capture.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-05 11:59:59 +02:00
1 parent 8ff22ad6b0
commit 311c8ee059
16 files changed
+2515 -23

No files matched your search

+49 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.40 - 2026-10-04 - Comparison and source pages accept the sources: that cite add writes; sources may cite sources
## 8.0.0-beta.41 - 2026-10-05 - raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
**Author:** Torben Nehmer
@@ -108,6 +108,7 @@ concern - readable here, never shipped as something to parse.
- Organisationsseiten: Personen als Abschnitt mit Aufstieg, entity_type organization, member-of, Lint-Befund broken_anchors
- lint: Unfilled Template Sections - a section still holding only its template's TODO placeholders (advisory)
- Comparison and source pages accept the sources: that cite add writes; sources may cite sources
- raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
**Low impact**
- version bump no longer points at version release in its output
@@ -153,6 +154,53 @@ concern - readable here, never shipped as something to parse.
- wiki-ingest and wiki-manage call xref add with --rel, not the --rel-a/--rel-b removed in 4.0.0
<!-- /wikitool:bumps -->
### raw capture / raw status / --replaces-bundle: documentation from git repositories as a bundle, with drift reporting
Documentation from a git repository used to reach an instance only by hand: copy the files into
`incoming/`, rename them around the global name rule, and remember nowhere which repository and
commit they came from. Three commands replace that:
- **`raw capture <repo-url> --ref <rule> --path <glob>... --name <bundle> --fidelity <v>
--authority <v>`** resolves the ref rule - a branch, or a tag pattern such as `v*` that takes the
newest matching tag by version order - to one commit. It fetches that commit by ref name,
shallowly, into a bare cache per URL under `tools/.wikitool_capture/` (gitignored, never
exported), and writes the files the globs select into `incoming/<bundle>/` at their repository
paths. Files are read as blobs, never through a checkout, so they are byte-identical to the
repository even under `core.autocrlf`. Globs follow git's `:(glob)` pathspec, checked against
git itself in the tests. `_capture.json` beside the files records repository, ref rule, commit,
globs, capture time, `fidelity`, `authority` and the file list. `--update <raw-bundle>` captures
the current state from that manifest.
- **`raw status`** reports every captured bundle whose files changed in the repository, as
`A`/`M`/`D` grouped by owning source page, and stays quiet about a commit that moved without a
change inside the globs. An unreachable repository is one line, never an abort; `--json` is
for the coming intake run.
- **`raw accept --replaces-bundle <raw-bundle> incoming/<bundle>`** replaces a captured bundle as
a whole at its existing address. Afterwards it holds exactly the new manifest's files plus
`_capture.json`; files the repository dropped are removed, and emptied directories `rmdir`ed.
`raw_files:` is left alone, as with `--replaces`, and the command prints the `touch --add/--remove
raw_files=` lines that follow.
Mechanical exclusions, each named in the output: a file whose first line starts with
`<!-- wikitool:export`, symlinks, submodules, files over 25 MiB, hidden path segments, files
named `_capture.json`, and Git LFS pointers. Git runs with the host's credentials, limited to
`ssh`/`https` through `GIT_ALLOW_PROTOCOL`, with no terminal prompt, no askpass and a timeout. A
repository asking for a password is reported as unreachable rather than hanging an unattended
run. A URL carrying a password is refused, because the manifest is committed.
`raw accept` takes a captured folder's capture fields from its manifest only, and refuses them on
the command line. `--replaces` and `--page` refuse a target inside a captured bundle.
**Fixed along the way:** `sources coverage` and `lint` ignored every `CONTRACT.md` under `raw/`
at any depth, not just the stage's own `raw/CONTRACT.md`. A captured repository's contract
file would have been invisible. The exclusion is now anchored to `raw/CONTRACT.md`, and
`_capture.json` is excluded instead. An instance with a `CONTRACT.md` somewhere below `raw/` may
see it reported as uncovered for the first time - a finding about a file that was always there,
nothing to migrate.
Drop-in in both directions: without a `_capture.json` under `raw/` no existing command behaves
differently. An older version would report a manifest as an uncovered raw file. `raw/CONTRACT.md`
§ "Getting a repository in" and `wiki-ingest` describe the workflow (Gitea #177).
### Comparison and source pages accept the sources: that cite add writes; sources may cite sources
`cite add` writes the cited source into the page's `sources:`, but neither the `comparison` nor