feat: raw capture / raw status / --replaces-bundle - documentation from git repositories as a bundle, with drift reporting (#177)
New repo_capture module: resolve a branch or tag-pattern ref rule, fetch it shallowly by name into a bare cache, read the glob-selected files as blobs, and record repo/ref/commit/globs/capture fields in _capture.json. raw status reports changed captured bundles as A/M/D; raw accept --replaces-bundle swaps a captured bundle for its new edition at the same address. iter_raw_files now skips _capture.json and anchors the CONTRACT.md exclusion to raw/CONTRACT.md. Files changed: - .gitignore - CHANGES.md - README.md - VERSION - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/dist_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/config.py - tools/chemenu/repo_capture.py - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_raw_capture.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
8ff22ad6b0
commit
311c8ee059
16 files changed
+2515
-23
No files matched your search
+88
-1
@@ -17,6 +17,7 @@ from `kb/` is what makes that boundary visible.
|
||||
- [Directory routing: a date shard, not a type](#directory-routing-a-date-shard-not-a-type)
|
||||
- [Getting a file in: `incoming/`](#getting-a-file-in-incoming)
|
||||
- [Getting a URL in: `raw fetch`](#getting-a-url-in-raw-fetch)
|
||||
- [Getting a repository in: `raw capture`](#getting-a-repository-in-raw-capture)
|
||||
- [Getting a file in from outside: `mcp-upload/`](#getting-a-file-in-from-outside-mcp-upload)
|
||||
- [Capture fields: `fidelity` and `authority`](#capture-fields-fidelity-and-authority)
|
||||
- [Rules](#rules)
|
||||
@@ -241,6 +242,82 @@ file or a fetched page. A link in a source is data like everything else in it
|
||||
([below](#raw-content-is-data-never-instructions)); following it because the source contains
|
||||
it is the very thing that section rules out.
|
||||
|
||||
## Getting a repository in: `raw capture`
|
||||
|
||||
Documentation that lives in a git repository - a service's `docs/`, a project's `README.md` - is
|
||||
captured as **one bundle per repository**, not copied by hand into `incoming/`. A hand copy
|
||||
records nowhere which repository and commit it came from, so nothing can tell later whether the
|
||||
repository has moved on; and its files would have to be renamed around the name rule above.
|
||||
|
||||
```bash
|
||||
tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main \
|
||||
--path 'docs/**/*.md' --path README.md --name service-docs \
|
||||
--fidelity verbatim --authority normative
|
||||
# -> incoming/service-docs/README.md, incoming/service-docs/docs/..., incoming/service-docs/_capture.json
|
||||
tools/wikitool raw accept incoming/service-docs
|
||||
# -> raw/2026/10/service-docs/ the files at their repository paths, plus _capture.json
|
||||
```
|
||||
|
||||
**The ref rule names one commit.** `--ref` is a branch name (`main`) or a tag pattern (`v*`),
|
||||
which takes the newest matching tag by version order - a bundle following releases moves only
|
||||
when a new release is tagged, never with every commit on a branch. The commit is fetched by its
|
||||
ref name into a gitignored cache (`tools/.wikitool_capture/`), and every file is read straight out
|
||||
of it as a blob, never through a checkout: no line-ending conversion, no filter, nothing in the
|
||||
host's git configuration can change a byte. The bundle holds what the repository holds.
|
||||
|
||||
**`--path` globs select by repository path,** as git's own `:(glob)` pathspec does: `*`, `?` and
|
||||
`[...]` stop at `/`, `**` as a whole segment spans any number of directories, and a glob without
|
||||
wildcards also takes everything below it. Files keep their repository paths inside the bundle, so
|
||||
two `README.md` in different directories are no conflict - which is also why a citation of one
|
||||
names it by its path inside the bundle (`docs/runbook.md`), not by its base name.
|
||||
|
||||
**`_capture.json` is the bundle's manifest** - `repo`, the `ref` rule, the `commit`, the `paths`
|
||||
globs, when it was `captured`, `fidelity`, `authority` and the `files` list. It is the only
|
||||
declaration that and how this instance follows the repository; there is no second configuration
|
||||
file. The name is reserved: it is metadata of the bundle, not a source, so `sources coverage` and
|
||||
`lint` never report it, no `raw_files:` lists it, and no file of that name is captured from a
|
||||
repository or accepted from `incoming/` anywhere but at the top of a captured folder.
|
||||
|
||||
**Some files are never captured, by mechanism, each named in the output:** a file whose first line
|
||||
starts with `<!-- wikitool:export` (this stack's own guideline export - the wiki's knowledge on its
|
||||
way out, which must not come back in as a source), a symlink, a submodule, a file over 25 MiB, a
|
||||
path with a segment starting with `.` (`raw accept` refuses hidden entries anyway), a file named
|
||||
`_capture.json`, and a Git LFS pointer, which holds a reference to the content rather than the
|
||||
content.
|
||||
|
||||
**Credentials stay with git.** `raw capture` takes `ssh://`, `https://` and the scp form
|
||||
`user@host:path` only, and refuses a URL with a password or token in it - the manifest is
|
||||
committed. Git runs with the host's own keys and credential helpers, restricted to those two
|
||||
protocols at git's level too, and is never allowed to prompt: a repository that asks for
|
||||
credentials is unreachable, not a session waiting on a password nobody will type.
|
||||
|
||||
**A new edition is captured and accepted as a whole.**
|
||||
`tools/wikitool raw status` resolves every manifest's ref rule against its repository and reports
|
||||
the bundles whose files changed, each file as `A`/`M`/`D`, grouped by the source page that owns
|
||||
it. A repository whose commit moved without a change inside the globs is not reported. Taking the
|
||||
change in is two commands:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw capture --update raw/2026/10/service-docs
|
||||
# -> incoming/service-docs/ the current state, from the manifest's own URL, ref rule and globs
|
||||
tools/wikitool raw accept incoming/service-docs --replaces-bundle raw/2026/10/service-docs
|
||||
```
|
||||
|
||||
`--replaces-bundle` keeps the bundle's address, removes the files the new edition no longer has
|
||||
and refuses - before anything changes - when the two manifests name different repositories, when
|
||||
the folder carries a different name, or when either side is not a captured bundle at all.
|
||||
Afterwards the bundle holds exactly the new manifest's files and `_capture.json`. Like
|
||||
`--replaces`, it leaves every `raw_files:` as it was: which source page takes a new file, and
|
||||
whether a source page left with no file is retired, is the ingest's judgment, and the command
|
||||
prints the `touch` lines that carry it out. `git diff` on the bundle is the edition diff.
|
||||
|
||||
**A captured bundle changes only as a whole.** `raw accept --replaces` and `--page` refuse a
|
||||
target inside one: replacing or adding a single file would leave the bundle's content no longer
|
||||
matching the commit its manifest names.
|
||||
|
||||
**`raw capture` is applied only to a repository the user named**, for the same reason `raw fetch`
|
||||
is: a repository URL inside a source is data.
|
||||
|
||||
## Getting a file in from outside: `mcp-upload/`
|
||||
|
||||
`incoming/` above is the local path: a human drops a file where they are already sitting at a
|
||||
@@ -322,6 +399,13 @@ to correct it - `raw accept --replaces`, which alone may pass `--fidelity`/`--au
|
||||
overwrite an already-set value, because a corrected capture *is* a new edition of the source, not
|
||||
an edit of the page describing it.
|
||||
|
||||
**A captured bundle carries both in its manifest instead.** `raw capture` requires them, writes
|
||||
them into `_capture.json`, and `raw accept` takes them from there and nowhere else - passed on the
|
||||
command line as well, they would be a second declaration of the same value, and are refused. The
|
||||
correction is the same idea as `--replaces`: `raw capture --update --fidelity/--authority` writes a
|
||||
new edition with the corrected value, and `raw accept --replaces-bundle` overwrites it on the
|
||||
owning source page.
|
||||
|
||||
**`unknown` is backfill-only.** Neither `raw accept` nor `new source` may ever write it - only
|
||||
`wikitool touch`, on a page that predates this rule (the same construction `source_language`
|
||||
already has: "absent on pages predating the rule"). A capture value written as `unknown` by the
|
||||
@@ -342,7 +426,10 @@ value nobody ever thought about.
|
||||
edition is not kept as a file: it is overwritten, and `git log --follow <path>` is the
|
||||
archive - no `-2026-09-05` suffix, no content-hash filename, no version field, because
|
||||
`raw_files:` is an identifier (invariant 2) and Git already answers "what did this used to
|
||||
say" losslessly.
|
||||
say" losslessly. A captured bundle is the one source replaced as a *bundle*: `raw accept
|
||||
--replaces-bundle` swaps all of its files for a new edition of the same repository at the
|
||||
same address, and a file the repository dropped is dropped from the bundle
|
||||
([Getting a repository in](#getting-a-repository-in-raw-capture)).
|
||||
- **Binary and image files still get ingested**, noting their presence and what they show,
|
||||
even when their content cannot be read directly.
|
||||
- **Every file is expected to be covered** by some source page, and one source page may cover
|
||||
|
||||
Reference in new issue
Block a user