feat: raw capture / raw status / --replaces-bundle - documentation from git repositories as a bundle, with drift reporting (#177)
CI / verify (push) Successful in 5m39s
CI / pwsh (push) Successful in 2m15s
Release / release (push) Successful in 36s

New repo_capture module: resolve a branch or tag-pattern ref rule, fetch it
shallowly by name into a bare cache, read the glob-selected files as blobs,
and record repo/ref/commit/globs/capture fields in _capture.json. raw status
reports changed captured bundles as A/M/D; raw accept --replaces-bundle swaps
a captured bundle for its new edition at the same address. iter_raw_files now
skips _capture.json and anchors the CONTRACT.md exclusion to raw/CONTRACT.md.

Files changed:
- .gitignore
- CHANGES.md
- README.md
- VERSION
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/config.py
- tools/chemenu/repo_capture.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_raw_capture.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-05 11:59:59 +02:00
1 parent 8ff22ad6b0
commit 311c8ee059
16 files changed
+2515 -23

No files matched your search

+130 -1
View File
@@ -115,6 +115,8 @@ sources coverage read idempotent budget:counted exit:0
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
raw capture write non-idempotent budget:counted exit:0,1 Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
raw status read idempotent budget:counted exit:0 Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
@@ -1455,6 +1457,120 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
#### `raw capture`
Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
**SYNOPSIS**
- `wikitool raw capture <repo-url> --ref <branch|tag-pattern> --path <glob> [--path <glob> ...] --name <bundle> --fidelity <v> --authority <v>` - First capture of a repository
- `wikitool raw capture --update <raw-bundle> [--fidelity <v>] [--authority <v>]` - Capture the current state of an already-accepted bundle, for `raw accept --replaces-bundle`
**PROPERTIES**
- effect: write
- idempotent: no
- atomic: Yes for what it leaves in `incoming/` - the bundle is written into a hidden staging directory and renamed into place in one step; a failure removes the staging directory. The git cache under `tools/.wikitool_capture/` is updated either way
- budget: counted
- network: yes
**EXAMPLES**
- `tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative`
- `tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting`
- `tools/wikitool raw capture --update raw/2026/10/service-docs`
**EXIT STATUS**
- 0 success
- 1 The URL is not `ssh://`, `https://` or `user@host:path`, or carries a password; the `--ref` rule, a `--path` glob or `--name` is unusable; or an argument of the other variant was mixed in
- 1 `--fidelity`/`--authority` is missing on a first capture, or names `unknown` or a value outside the schema's enum
- 1 The repository could not be reached, asked for credentials, or timed out; or no branch or tag matches `--ref`
- 1 `incoming/<bundle>` already exists, the name is already taken under `raw/`, nothing matches the globs, or a path would be over the budget
- 1 raw capture --update: The path is not a bundle under `raw/` with a readable `_capture.json`, or its manifest names a URL that is refused
**ON FAILURE**
- The URL is not `ssh://`, `https://` or `user@host:path`, or carries a password; the `--ref` rule, a `--path` glob or `--name` is unusable; or an argument of the other variant was mixed in -> Fix the call and retry once. A credential goes into git's credential helper, never into the URL
- `--fidelity`/`--authority` is missing on a first capture, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
- The repository could not be reached, asked for credentials, or timed out; or no branch or tag matches `--ref` -> Nothing was written to `incoming/`. Check the URL, the ref rule and the host's git credentials with the user; an unreachable host may be retried once
- `incoming/<bundle>` already exists, the name is already taken under `raw/`, nothing matches the globs, or a path would be over the budget -> Nothing was written. Accept or remove what is in `incoming/`; for a name taken by a captured bundle use `--update`; widen or narrow the globs; then retry once
- raw capture --update: The path is not a bundle under `raw/` with a readable `_capture.json`, or its manifest names a URL that is refused -> Show the message to the user - a manifest under `raw/` is never edited by hand to make this pass
**NEVER**
- Never put a password or token into the repository URL.
- Never edit a `_capture.json`, or the files of a captured bundle, by hand.
- Never capture a repository the user has not named in this session.
**NOTES**
- Writes into `incoming/<bundle>/` only, never into `raw/`: `raw accept incoming/<bundle>` promotes it, as one folder, to `raw/<YYYY>/<MM>/<bundle>/`.
- `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
- The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
- `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
- Never captured, each named in the output with its reason: a file whose first line starts with `<!-- wikitool:export`, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with `.`, a file named `_capture.json`, and a Git LFS pointer.
- `incoming/<bundle>/_capture.json` records `schema`, `repo`, `ref`, `commit`, `paths`, `captured` (UTC), `fidelity`, `authority` and `files`. It is the only declaration that and how this instance follows the repository.
- Only `ssh://`, `https://` and the scp form `user@host:path`; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
- `--update <raw-bundle>` takes URL, ref rule and globs from that bundle's manifest and writes the current state to `incoming/<same bundle name>/`; `--fidelity`/`--authority` default to the manifest's values, and passing one is the way to correct it.
- Refused before anything is written: `incoming/<bundle>` already exists, the name is already taken under `raw/` (a first capture), nothing matches the globs, or a target path - measured at its later place under `raw/` - is over the path budget.
- Success prints the commit and the `raw accept` line that comes next.
**SEE ALSO**
- `raw/CONTRACT.md` "Getting a repository in: `raw capture`" - the rules and why
- `wikitool raw accept` - promotes the bundle; `--replaces-bundle` for a new edition
- `wikitool raw status` - which captured bundles have fallen behind their repository
#### `raw status`
Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
**SYNOPSIS**
- `wikitool raw status [--json]`
**PROPERTIES**
- effect: read
- idempotent: yes
- atomic: Read-only for the tree - writes nothing but the git cache under `tools/.wikitool_capture/`
- budget: counted
- network: yes
**EXAMPLES**
- `tools/wikitool raw status`
- `tools/wikitool raw status --json`
**EXIT STATUS**
- 0 success
- 0 A repository could not be reached, its manifest or URL was refused, or no ref matches its rule
**ON FAILURE**
- A repository could not be reached, its manifest or URL was refused, or no ref matches its rule -> Reported on that bundle's line, the others are still checked; check the URL and the host's git credentials with the user
**NEVER**
- Never edit a `_capture.json` to silence a line - a new edition goes through `raw capture --update` and `raw accept --replaces-bundle`.
**NOTES**
- Reads every `_capture.json` under `raw/` and resolves its ref rule with `git ls-remote`. Only where the commit moved is the new commit fetched, and compared - with the same globs and exclusions `raw capture` applies - against the bundle's files under `raw/`.
- A bundle whose commit moved without a change inside its globs is reported as unchanged, not listed. A tag pattern follows new matching tags only, never new commits on a branch.
- Each changed bundle prints `old -> new` and its files as `A`/`M`/`D`, grouped by owning source page with its citing pages, followed by the `raw capture --update` and `raw accept --replaces-bundle` lines that take the new edition in.
- An unreachable repository, a refused URL or an unreadable manifest is one line beside the others, never an abort; the exit status is 0 as long as the command itself ran.
- `--json` prints one object per bundle: `bundle`, `repo`, `ref`, `old`, `new`, `changed`, `files` (`[{path, status}]`), `error` (or null).
- Git runs exactly as for `raw capture`: the host's credentials, only `ssh`/`https`, never a prompt, 120 s per call.
**SEE ALSO**
- `wikitool raw capture` - `--update` captures the new edition
- `wikitool raw accept` - `--replaces-bundle` takes it in
- `raw/CONTRACT.md` "Getting a repository in: `raw capture`"
#### `raw pending`
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
@@ -1504,12 +1620,13 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
- `wikitool raw accept incoming/<bundle> --replaces-bundle <raw-bundle> [--dry-run]` - Replace a captured bundle as a whole with the new edition `raw capture --update` wrote
**PROPERTIES**
- effect: write
- idempotent: no
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write. `raw accept --replaces-bundle`: No - one `unlink()` per removed file, one `rename()` per added or modified file, then the manifest, then one `rmdir` per emptied directory, plus one page write per owning page whose capture fields change; a half-replaced bundle is not resumed
- budget: counted
- network: no
@@ -1519,6 +1636,7 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
- `tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu`
**EXIT STATUS**
@@ -1531,6 +1649,9 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
- 1 raw accept --replaces / --page: The target, or a file the page already has, lies in a captured bundle
- 1 raw accept incoming/<bundle>: A captured folder is given `--fidelity`/`--authority`, its `_capture.json` cannot be read or names an invalid value, or its files differ from the ones the manifest lists
- 1 raw accept --replaces-bundle: Either side has no `_capture.json`, the two manifests name different repositories, the folder's name differs from the bundle's, `--page`, `--replaces`, `--fidelity` or `--authority` was also given, the folder does not match its manifest, a new path is over the budget, or a file of the bundle has more than one owning source page
**ON FAILURE**
@@ -1542,6 +1663,9 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
- raw accept --replaces / --page: The target, or a file the page already has, lies in a captured bundle -> Not fixed by retrying: a captured bundle changes only as a whole - `raw capture --update <raw-bundle>`, then `raw accept --replaces-bundle`
- raw accept incoming/<bundle>: A captured folder is given `--fidelity`/`--authority`, its `_capture.json` cannot be read or names an invalid value, or its files differ from the ones the manifest lists -> Nothing moved. Drop the two flags and retry once; a manifest that does not match its folder is fixed by removing the folder and capturing it again, never by editing either
- raw accept --replaces-bundle: Either side has no `_capture.json`, the two manifests name different repositories, the folder's name differs from the bundle's, `--page`, `--replaces`, `--fidelity` or `--authority` was also given, the folder does not match its manifest, a new path is over the budget, or a file of the bundle has more than one owning source page -> Nothing under `raw/` or in `incoming/` changed. Fix what the message names and retry once; different repositories or a different name are a separate source, not a new edition - show the message to the user
**NEVER**
@@ -1563,12 +1687,17 @@ Promote one or more files, or one folder, from `incoming/` into `raw/`.
- `--replaces` refuses a target with more than one owning source page; with none, it replaces anyway and says so. It cannot be combined with `--page` or with more than one incoming file.
- `--replaces` prints the source page (if any) and its citing pages, so their update lands in the same commit as the replacement.
- A file already at its computed destination is what "already exists" reports, not a partial prior run to resume - safe to retry as-is once a cause is fixed.
- A folder carrying `_capture.json` at its top is a captured bundle (`raw capture`): `--fidelity`/`--authority` come from that manifest only and are refused on the command line; the folder must hold exactly the files the manifest lists. `_capture.json` goes in no `raw_files:`.
- `--replaces-bundle <raw-bundle>` replaces a captured bundle under `raw/` with the new edition in `incoming/<bundle>` at the same address: files gone from the new edition are removed, emptied directories `rmdir`ed, and afterwards the bundle holds exactly the new manifest's files plus `_capture.json`. `raw_files:` is left untouched; an owning source page's `fidelity`/`authority` is overwritten where the new manifest differs.
- `--replaces-bundle` prints each changed file as `A`/`M`/`D`, grouped by owning source page with its citing pages, and the `touch --page ... --add/--remove raw_files=` lines that follow; `git diff` on the bundle shows the edition's changes.
- `--replaces` and `--page` refuse a target inside a captured bundle; the refusal names `raw capture --update` and `--replaces-bundle`. A file named `_capture.json` anywhere but at a captured folder's top is refused - the name is reserved.
- `--dry-run` reports the moves without making them.
**SEE ALSO**
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
- `wikitool raw capture` - writes a captured bundle, or its new edition, into `incoming/`
- `wikitool types describe source` - the capture field values
- `wikitool new source` - the source page for a promoted file
+1
View File
@@ -106,6 +106,7 @@ tools/
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
corpus_diff.py invariant comparison of kb/ between two revisions
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt
search/ pluggable search backends, plus service.py - the search core
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (.wikitool-tasks.json), one module per adapter - no instruction ever learns which provider it is
commands/ one module per command or command group: the terminal adapters
+1 -1
View File
@@ -259,7 +259,7 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
"sources coverage", "sources trace", "sources rebuild-index",
)),
("Raw material and uploads", (
"raw fetch", "raw pending", "raw accept",
"raw fetch", "raw capture", "raw status", "raw pending", "raw accept",
"upload list", "upload show", "upload accept", "upload reject",
)),
("Git", (
+1 -1
View File
@@ -127,7 +127,7 @@ HOOK_DIRS = (".github/hooks", ".vibe")
# it, measured against the source repo's own test run. Matched by directory
# name at any depth, which is right for a cache and wrong for anything with an
# ordinary name - that is what `TOOLS_DEV_ONLY` below is for.
TOOLS_EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", ".wikitool_session", "htmlcov"}
TOOLS_EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", ".wikitool_session", ".wikitool_capture", "htmlcov"}
# tools/-relative paths that are dev-only rather than derived: the test suite
# and the two files that configure running and measuring it. The suite tests
+3
View File
@@ -160,6 +160,9 @@ REQUIRED_IGNORE_CANARIES = (
".wikitool-tasks.json",
# The live suite's tracker profiles (Gitea #156) - credentials again, one file per tracker.
".wikitool-tasks.d/probe.json",
# `raw capture`'s git cache (Gitea #177) - fetched repositories, recomputable,
# and in the way of `publish`'s `git add -A` like the coverage output above.
"tools/.wikitool_capture/0123abcd/HEAD",
)
REQUIRED_TRACKED_PATHS = (
"reports/CONTRACT.md",
File diff suppressed because it is too large. Load diff
+11 -3
View File
@@ -167,9 +167,15 @@ def __getattr__(name: str):
def __dir__() -> list[str]:
return sorted([*globals(), "ROOT", *_DERIVED, *_KB_DERIVED])
# Files/patterns to ignore when scanning raw/ for ingest coverage.
# CONTRACT.md is the layer's source contract, not source material.
RAW_IGNORE_NAMES = {".gitkeep", ".DS_Store", "CONTRACT.md"}
# Files to ignore when scanning raw/ for ingest coverage, by name at any depth.
# `_capture.json` is a captured bundle's manifest (`chemenu.repo_capture`) -
# metadata of the bundle, not source material; `raw capture` never takes a
# repository file of that name, so every one under raw/ is a manifest.
RAW_IGNORE_NAMES = {".gitkeep", ".DS_Store", "_capture.json"}
# Ignored only directly in raw/: the stage's own contract. Matched by name it
# hid every `CONTRACT.md` a captured repository brought along (Gitea #177).
RAW_IGNORE_TOP_LEVEL = {"CONTRACT.md"}
# Per-instance personalization: who operates this wiki (`USER.md`) and how this
# instance sounds while doing it (`SOUL.md`). Both are read every session and
@@ -325,5 +331,7 @@ def iter_raw_files(raw_dir: Path):
continue
if path.name in RAW_IGNORE_NAMES or path.name.startswith("."):
continue
if path.parent == raw_dir and path.name in RAW_IGNORE_TOP_LEVEL:
continue
yield path
+578
View File
@@ -0,0 +1,578 @@
"""Capturing documentation from a git repository for `raw/`: resolve a ref
rule to one commit, read the files the globs select straight out of that
commit, and describe the result in a manifest - with no CLI attached.
`wikitool raw capture`, `raw status` and `raw accept --replaces-bundle` are the
terminal adapters over this module; it decides nothing about `incoming/` or
`raw/` paths beyond the manifest's own name.
Three properties carry the design, and each has a test:
- **Byte-identical to the blob.** Files are read with `git cat-file` out of a
bare cache repository, never from a checked-out working tree, so
`core.autocrlf`, a smudge filter or anything else in the host's git
configuration cannot change a byte.
- **One selection, two callers.** `select_files` is the only place the globs
and the exclusions are applied; `raw capture` writes what it returns and
`raw status` compares against it. Two copies of the exclusion list would let
every excluded file show up as a permanent `A` in `raw status`.
- **Git, not wikitool, holds the credentials.** Git runs with the host's own
configuration and credential helpers. Nothing here reads, stores or passes a
secret, a URL carrying a password is refused (the manifest is committed), and
no call may prompt: an unattended intake run must find a repository that asks
for credentials unreachable, not hang on it.
"""
from __future__ import annotations
import datetime
import hashlib
import json
import os
import re
import signal
import subprocess
import urllib.parse
from contextlib import contextmanager
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterator, Optional
from chemenu import config, filelock, toolpaths, web_capture
from chemenu.errors import BackendError, ValidationError
MANIFEST_NAME = "_capture.json"
SCHEMA = 1
# What `git` may talk to, as URL schemes and as `GIT_ALLOW_PROTOCOL` alike. The
# test suite widens this tuple through a fixture to reach local test
# repositories over `file://`; nothing the command itself reads - no option, no
# environment variable - can, because `GIT_ALLOW_PROTOCOL` is set from this
# tuple on every call and overrides whatever the caller's environment held.
ALLOWED_SCHEMES: tuple[str, ...] = ("ssh", "https")
GIT_TIMEOUT_SECONDS = 120.0
EXPORT_MARKER = b"<!-- wikitool:export"
LFS_MARKER = b"version https://git-lfs.github.com/spec/v1"
_BOM = b"\xef\xbb\xbf"
_MODE_SYMLINK = "120000"
_MODE_SUBMODULE = "160000"
# `user@host:path` - git's scp-like form. A colon before the first slash is
# what makes git read it that way; `::` is excluded because that is the
# `<transport>::<address>` form (`ext::`, `fd::`), never an scp address.
_SCP_FORM = re.compile(r"[A-Za-z0-9._~-]+@[A-Za-z0-9.-]+:(?!:)[^\s]+")
_SCHEME = re.compile(r"([A-Za-z][A-Za-z0-9+.-]*)://")
# --- URLs ---------------------------------------------------------------------
def check_repo_url(url: str) -> None:
"""Refuse anything but `ssh://`, `https://` and `user@host:path`, and any
URL that carries a password. Raises `ValidationError`.
Applied to a URL from the command line and to one read back out of a
manifest under `raw/` alike: a manifest is committed text, not a trusted
command."""
if not isinstance(url, str) or not url or url != url.strip() or url.startswith("-"):
raise ValidationError(f"{url!r} is not a repository URL.")
if any(ord(c) < 0x20 for c in url):
raise ValidationError(f"{url!r} contains a control character.")
allowed = ", ".join(f"{s}://" for s in ALLOWED_SCHEMES)
scheme_match = _SCHEME.match(url)
if scheme_match is None:
if _SCP_FORM.fullmatch(url) and "ssh" in ALLOWED_SCHEMES:
return
raise ValidationError(
f"{url!r} is not a repository URL raw capture takes - only {allowed} and the scp form "
"user@host:path. A local path, file://, ext:: and fd:: are refused."
)
scheme = scheme_match.group(1).lower()
if scheme not in ALLOWED_SCHEMES:
raise ValidationError(
f"{url!r} uses {scheme}://, which raw capture does not take - only {allowed} and the "
"scp form user@host:path."
)
parsed = urllib.parse.urlsplit(url)
if parsed.password is not None:
raise ValidationError(
f"The URL for {parsed.hostname or 'this repository'} carries a password or token in it. "
"The manifest is committed, so a credential never goes into the URL - configure it in "
"git's own credential helper instead, and pass the URL without it."
)
# --- globs --------------------------------------------------------------------
_WILDCARDS = frozenset("*?[\\")
def check_glob(pattern: str) -> None:
if not pattern or pattern.startswith("/") or "\0" in pattern:
raise ValidationError(
f"--path {pattern!r} is not a usable glob: it is matched against the repository's "
"paths, relative to its root, so it is not empty and does not start with '/'."
)
if any(part in ("", ".", "..") for part in pattern.rstrip("/").split("/")):
raise ValidationError(f"--path {pattern!r} has an empty, '.' or '..' segment.")
def _segment_regex(segment: str) -> str:
out: list[str] = []
i, n = 0, len(segment)
while i < n:
c = segment[i]
if c == "*":
out.append("[^/]*")
elif c == "?":
out.append("[^/]")
elif c == "\\" and i + 1 < n:
i += 1
out.append(re.escape(segment[i]))
elif c == "[":
j = i + 1
negate = j < n and segment[j] in "!^"
if negate:
j += 1
if j < n and segment[j] == "]":
j += 1
while j < n and segment[j] != "]":
j += 1
if j >= n:
out.append(re.escape(c)) # an unclosed '[' is a literal, as in git
else:
body = segment[i + 1 + (1 if negate else 0):j]
body = body.replace("\\", "\\\\")
out.append(f"[^/{body}]" if negate else f"(?!/)[{body}]")
i = j
else:
out.append(re.escape(c))
i += 1
return "".join(out)
def glob_regex(pattern: str) -> "re.Pattern[str]":
"""The regex for one glob, with git's `:(glob)` pathspec semantics: `*`,
`?` and `[...]` never cross a `/`; `**` as a whole segment spans any
number of directories, none included; anywhere else it is a plain `*`.
`fnmatch` would not do - its `*` crosses `/` - and `PurePath.full_match`
only exists from Python 3.13."""
segments = pattern.split("/")
parts: list[str] = []
for index, segment in enumerate(segments):
last = index == len(segments) - 1
if segment == "**":
parts.append(".*" if last else "(?:[^/]*/)*")
continue
parts.append(_segment_regex(segment))
if not last:
parts.append("/")
return re.compile("".join(parts), re.DOTALL)
def glob_matches(pattern: str, path: str) -> bool:
"""Whether repository path `path` falls under `pattern`. A pattern with no
wildcard also matches everything below it as a directory - `docs` takes
`docs/x/b.md` - the way git's own pathspec does; a pattern with one does
not (`docs/x*` takes nothing below `docs/x/`)."""
if not any(c in _WILDCARDS for c in pattern):
literal = pattern.rstrip("/")
return path == literal or path.startswith(literal + "/")
return glob_regex(pattern).fullmatch(path) is not None
# --- git ----------------------------------------------------------------------
def _git_env() -> dict[str, str]:
env = dict(os.environ)
for name in ("GIT_DIR", "GIT_WORK_TREE", "GIT_ASKPASS", "SSH_ASKPASS"):
env.pop(name, None)
env["GIT_ALLOW_PROTOCOL"] = ":".join(ALLOWED_SCHEMES)
env["GIT_TERMINAL_PROMPT"] = "0"
env["SSH_ASKPASS_REQUIRE"] = "never"
env["GCM_INTERACTIVE"] = "never"
return env
def run_git(
args: list[str],
git_dir: Path,
*,
stdin: Optional[bytes] = None,
timeout: float = GIT_TIMEOUT_SECONDS,
) -> bytes:
"""Run `git --git-dir=<git_dir> <args>` and return its stdout.
No call can prompt: no terminal prompt, no askpass program (an empty
`core.askPass` stops git from falling back to `SSH_ASKPASS`), and on POSIX a
session of its own, so `ssh` has no controlling terminal to ask on either.
A timeout kills the whole process group, `ssh` included - killing `git`
alone would leave the pipe open and the read below hanging. Raises
`BackendError` on a non-zero exit, a timeout or a git that cannot start."""
argv = [toolpaths.git(), "-c", "core.askPass=", f"--git-dir={git_dir}", *args]
posix = os.name == "posix"
try:
proc = subprocess.Popen(
argv,
stdin=subprocess.PIPE if stdin is not None else subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=_git_env(),
start_new_session=posix,
)
except OSError as exc:
raise BackendError(f"git could not be started: {exc}") from exc
try:
out, err = proc.communicate(stdin, timeout=timeout)
except subprocess.TimeoutExpired:
if posix:
try:
os.killpg(proc.pid, signal.SIGKILL)
except OSError:
pass
else:
proc.kill()
proc.communicate()
raise BackendError(f"git {args[0]} did not finish within {timeout:.0f} s") from None
if proc.returncode != 0:
message = err.decode("utf-8", "replace").strip().splitlines()
raise BackendError(f"git {args[0]} failed: {message[-1] if message else f'exit {proc.returncode}'}")
return out
def cache_root() -> Path:
return config.ROOT / "tools" / ".wikitool_capture"
def _url_key(url: str) -> str:
return hashlib.sha256(url.encode("utf-8")).hexdigest()
@contextmanager
def cache_repo(url: str) -> Iterator[Path]:
"""The bare cache repository for `url`, created on first use, held under an
exclusive lock for the duration of the block - two runs against one URL
must not fetch into the same repository at once."""
root = cache_root()
root.mkdir(parents=True, exist_ok=True)
key = _url_key(url)
repo = root / key
with open(root / f"{key}.lock", "a+b") as handle, filelock.exclusive(handle):
if not (repo / "HEAD").is_file():
repo.mkdir(exist_ok=True)
run_git(["init", "--bare", "-q"], repo)
yield repo
# --- refs ---------------------------------------------------------------------
def _version_key(name: str) -> tuple:
# `git tag --sort=-v:refname` without a configured suffix order: runs of
# digits compare as numbers, everything else as text. `re.split` with a
# capture group always alternates text, digits, text, ... starting with
# text, so the tuple positions never mix types.
parts = re.split(r"(\d+)", name)
return tuple(int(p) if i % 2 else p for i, p in enumerate(parts))
def is_tag_pattern(rule: str) -> bool:
return any(c in rule for c in "*?[")
def check_ref_rule(rule: str) -> None:
if not rule or rule.startswith("-") or any(c.isspace() or ord(c) < 0x20 for c in rule):
raise ValidationError(
f"--ref {rule!r} is neither a branch name (main) nor a tag pattern (v*)."
)
@dataclass(frozen=True)
class ResolvedRef:
refname: str # what is fetched: refs/heads/<branch> or refs/tags/<tag>
commit: str # the commit it names - peeled, for an annotated tag
def resolve_ref(url: str, rule: str, repo: Path) -> ResolvedRef:
"""Resolve `rule` against the remote's advertised refs: a branch name names
`refs/heads/<rule>`; a rule with a wildcard is a tag pattern and names the
newest matching tag by version order. Raises `BackendError` when the
remote is unreachable, `ValidationError` when nothing matches."""
out = run_git(["ls-remote", "--heads", "--tags", "--", url], repo)
direct: dict[str, str] = {}
peeled: dict[str, str] = {}
for line in out.decode("utf-8", "replace").splitlines():
sha, _, ref = line.partition("\t")
if not ref:
continue
if ref.endswith("^{}"):
peeled[ref[:-3]] = sha
else:
direct[ref] = sha
if is_tag_pattern(rule):
regex = glob_regex(rule)
tags = [
ref[len("refs/tags/"):]
for ref in direct
if ref.startswith("refs/tags/") and regex.fullmatch(ref[len("refs/tags/"):])
]
if not tags:
raise ValidationError(f"No tag in {url} matches --ref {rule!r}.")
refname = f"refs/tags/{max(tags, key=_version_key)}"
else:
refname = f"refs/heads/{rule}"
if refname not in direct:
raise ValidationError(f"{url} has no branch {rule!r}.")
return ResolvedRef(refname, peeled.get(refname, direct[refname]))
def fetch(url: str, refname: str, repo: Path) -> str:
"""Fetch `refname` shallowly into `repo`, by name - never by SHA, which a
server only serves with `allowReachableSHA1InWant` - and return the commit
it pointed at when fetched."""
run_git(["fetch", "--depth", "1", "--no-tags", "-q", "--", url, f"+{refname}:refs/wikitool/fetched"], repo)
return run_git(["rev-parse", "--verify", "refs/wikitool/fetched^{commit}"], repo).decode().strip()
# --- selection ----------------------------------------------------------------
@dataclass(frozen=True)
class TreeEntry:
mode: str
kind: str
sha: str
size: Optional[int]
path: str
def list_tree(repo: Path, commit: str) -> list[TreeEntry]:
out = run_git(["ls-tree", "-r", "-z", "--long", commit], repo)
entries: list[TreeEntry] = []
for record in out.split(b"\0"):
if not record:
continue
meta, _, raw_path = record.partition(b"\t")
mode, kind, sha, size = meta.decode("ascii").split(None, 3)
try:
path = raw_path.decode("utf-8")
except UnicodeDecodeError:
path = raw_path.decode("utf-8", "backslashreplace")
kind = "undecodable"
entries.append(TreeEntry(mode, kind, sha, None if size.strip() == "-" else int(size), path))
return entries
def read_blobs(repo: Path, shas: list[str]) -> dict[str, bytes]:
"""Every blob in `shas`, read raw with `cat-file --batch` - no filter, no
line-ending conversion."""
if not shas:
return {}
unique = list(dict.fromkeys(shas))
out = run_git(["cat-file", "--batch"], repo, stdin="".join(f"{s}\n" for s in unique).encode())
blobs: dict[str, bytes] = {}
pos = 0
for sha in unique:
newline = out.index(b"\n", pos)
header = out[pos:newline].decode("ascii").split()
if len(header) != 3:
raise BackendError(f"git cat-file could not read {sha}: {' '.join(header)}")
size = int(header[2])
start = newline + 1
blobs[sha] = out[start:start + size]
pos = start + size + 1
return blobs
@dataclass
class Selection:
files: dict[str, bytes] = field(default_factory=dict) # repo path -> blob bytes
excluded: list[tuple[str, str]] = field(default_factory=list) # (repo path, reason)
def _structural_exclusion(entry: TreeEntry) -> Optional[str]:
if entry.mode == _MODE_SYMLINK:
return "symlink"
if entry.mode == _MODE_SUBMODULE or entry.kind == "commit":
return "submodule"
if entry.kind == "undecodable":
return "path is not UTF-8"
if entry.kind != "blob":
return f"not a file ({entry.kind})"
if any(part.startswith(".") for part in entry.path.split("/")):
return "hidden path segment"
if entry.path.rsplit("/", 1)[-1] == MANIFEST_NAME:
return f"reserved name {MANIFEST_NAME}"
if entry.size is not None and entry.size > web_capture.MAX_BYTES:
return f"over {web_capture.MAX_BYTES // (1024 * 1024)} MiB"
return None
def _content_exclusion(data: bytes) -> Optional[str]:
if data.startswith(LFS_MARKER):
return "Git LFS pointer - the content is not in the repository"
first = data[len(_BOM):] if data.startswith(_BOM) else data
if first.startswith(EXPORT_MARKER):
return "guideline export (first line starts with <!-- wikitool:export)"
return None
def select_files(repo: Path, commit: str, globs: list[str]) -> Selection:
"""The files of `commit` that match at least one glob, minus the
exclusions - each excluded path named with its reason. The single place
both are applied (module docstring)."""
selection = Selection()
candidates: list[TreeEntry] = []
for entry in list_tree(repo, commit):
if not any(glob_matches(g, entry.path) for g in globs):
continue
reason = _structural_exclusion(entry)
if reason:
selection.excluded.append((entry.path, reason))
else:
candidates.append(entry)
blobs = read_blobs(repo, [e.sha for e in candidates])
for entry in candidates:
data = blobs[entry.sha]
reason = _content_exclusion(data)
if reason:
selection.excluded.append((entry.path, reason))
else:
selection.files[entry.path] = data
selection.excluded.sort()
return selection
@dataclass(frozen=True)
class Snapshot:
refname: str
commit: str
selection: Selection
def snapshot(url: str, rule: str, globs: list[str]) -> Snapshot:
"""Resolve, fetch and select in one go, under the cache lock."""
with cache_repo(url) as repo:
resolved = resolve_ref(url, rule, repo)
commit = fetch(url, resolved.refname, repo)
return Snapshot(resolved.refname, commit, select_files(repo, commit, globs))
def remote_commit(url: str, rule: str) -> ResolvedRef:
"""`resolve_ref` without a fetch - what `raw status` asks first, so an
unchanged repository costs one `ls-remote` and nothing else."""
with cache_repo(url) as repo:
return resolve_ref(url, rule, repo)
# --- manifest -----------------------------------------------------------------
@dataclass(frozen=True)
class Manifest:
repo: str
ref: str
commit: str
paths: tuple[str, ...]
captured: str
fidelity: str
authority: str
files: tuple[str, ...]
def to_json(self) -> str:
data = {
"schema": SCHEMA,
"repo": self.repo,
"ref": self.ref,
"commit": self.commit,
"paths": list(self.paths),
"captured": self.captured,
"fidelity": self.fidelity,
"authority": self.authority,
"files": sorted(self.files),
}
return json.dumps(data, indent=2, ensure_ascii=False) + "\n"
def utc_now() -> str:
return datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat().replace(
"+00:00", "Z"
)
def read_manifest(path: Path) -> Manifest:
"""Parse `_capture.json`. Raises `ValidationError` naming what is wrong
with it - including a repository URL `check_repo_url` refuses."""
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
raise ValidationError(f"{MANIFEST_NAME} cannot be read: {exc}") from exc
if not isinstance(data, dict) or data.get("schema") != SCHEMA:
raise ValidationError(f"{MANIFEST_NAME} is not schema {SCHEMA}.")
strings = ("repo", "ref", "commit", "captured", "fidelity", "authority")
lists = ("paths", "files")
bad = [k for k in strings if not isinstance(data.get(k), str) or not data.get(k)]
bad += [
k for k in lists
if not isinstance(data.get(k), list) or not all(isinstance(v, str) and v for v in data[k])
]
if bad:
raise ValidationError(f"{MANIFEST_NAME} lacks a valid {', '.join(bad)}.")
check_repo_url(data["repo"])
check_ref_rule(data["ref"])
for pattern in data["paths"]:
check_glob(pattern)
for rel in data["files"]:
parts = rel.split("/")
if rel.startswith("/") or "\\" in rel or any(p in ("", ".", "..") or p.startswith(".") for p in parts):
raise ValidationError(f"{MANIFEST_NAME} lists a file path it may not: {rel!r}.")
return Manifest(
data["repo"], data["ref"], data["commit"], tuple(data["paths"]), data["captured"],
data["fidelity"], data["authority"], tuple(data["files"]),
)
def bundle_files(bundle: Path) -> dict[str, Path]:
"""Every file of a captured bundle on disk, keyed by its repository path -
the manifest itself left out."""
found: dict[str, Path] = {}
for path in sorted(bundle.rglob("*")):
if path.is_file() and not path.is_symlink():
rel = path.relative_to(bundle).as_posix()
if rel != MANIFEST_NAME:
found[rel] = path
return found
def captured_bundle_of(path: Path, raw_dir: Path) -> Optional[Path]:
"""The captured bundle `path` lies in - the nearest directory above it,
below `raw_dir`, holding a `_capture.json` - or None."""
try:
path.relative_to(raw_dir)
except ValueError:
return None
current = path if path.is_dir() else path.parent
while current != raw_dir and raw_dir in current.parents:
if (current / MANIFEST_NAME).is_file():
return current
current = current.parent
return None
def diff(files_on_disk: dict[str, Path], new: dict[str, bytes]) -> list[tuple[str, str]]:
"""`(status, repo path)` for every file that differs between a bundle on
disk and a new selection: `A` new, `M` bytes differ, `D` gone."""
changes: list[tuple[str, str]] = []
for rel in sorted(set(files_on_disk) | set(new)):
if rel not in files_on_disk:
changes.append(("A", rel))
elif rel not in new:
changes.append(("D", rel))
elif files_on_disk[rel].read_bytes() != new[rel]:
changes.append(("M", rel))
return changes
+3 -1
View File
@@ -340,11 +340,13 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list
since `--latest`, which asks the release feed and downloads from it, and `raw fetch`
since it exists (Gitea #120) - its `--html` form stays offline, which does not turn the
command back to `no`. Pinned as an explicit
command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177). Pinned as an explicit
set so a command gaining or losing that reach is a deliberate edit here, not a silent
drift between the property and what the command actually does."""
expected = {
"raw fetch",
"raw capture",
"raw status",
"sync",
"publish",
"version check",
+733
View File
@@ -0,0 +1,733 @@
"""`raw capture`, `raw status` and `raw accept --replaces-bundle` (Gitea #177),
against local test repositories - never the network.
The `file` transport those repositories need is unlocked by the `capture`
fixture widening `repo_capture.ALLOWED_SCHEMES`, the one switch the command
itself cannot reach: it sets `GIT_ALLOW_PROTOCOL` from that tuple on every git
call, so no option and no environment variable opens it.
"""
from __future__ import annotations
import datetime
import http.server
import json
import os
import subprocess
import threading
import time
from pathlib import Path
import pytest
import typer
from chemenu import config, repo_capture
from chemenu.commands.raw_cmd import (
raw_accept_command,
raw_capture_command,
raw_pending_command,
raw_status_command,
)
from chemenu.errors import BackendError, ValidationError
from chemenu.frontmatter_io import read_page, write_page
from chemenu.kb_scan import load_kb_pages
from chemenu.provenance import uncovered_raw_files
posix_only = pytest.mark.skipif(os.name != "posix", reason="needs a POSIX shell and symlinks")
def _git(cwd: Path, *args: str) -> str:
return subprocess.run(
["git", "-c", "user.name=Fixture", "-c", "user.email=f@example.org", *args],
cwd=cwd, check=True, capture_output=True, text=True,
).stdout.strip()
class Repo:
"""A work repository whose `file://` URL is what the commands are given."""
def __init__(self, path: Path) -> None:
self.path = path
path.mkdir(parents=True)
_git(path, "init", "-q", "-b", "main")
@property
def url(self) -> str:
return self.path.as_uri()
def write(self, rel: str, data) -> None:
target = self.path / rel
target.parent.mkdir(parents=True, exist_ok=True)
if isinstance(data, str):
data = data.encode("utf-8")
target.write_bytes(data)
def remove(self, rel: str) -> None:
_git(self.path, "rm", "-q", rel)
def commit(self, message: str = "change") -> str:
_git(self.path, "add", "-A")
_git(self.path, "commit", "-q", "--allow-empty", "-m", message)
return _git(self.path, "rev-parse", "HEAD")
def tag(self, name: str, annotated: bool = False) -> None:
if annotated:
_git(self.path, "tag", "-a", name, "-m", name)
else:
_git(self.path, "tag", name)
def _shard() -> str:
today = datetime.date.today()
return f"raw/{today.year:04d}/{today.month:02d}"
def _out(capsys) -> str:
return " ".join(capsys.readouterr().out.split())
@pytest.fixture
def capture(kb_dir, monkeypatch):
root = kb_dir.parent
(root / "raw").mkdir(exist_ok=True)
(root / "incoming").mkdir(exist_ok=True)
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file"))
return root
@pytest.fixture
def repo(tmp_path):
r = Repo(tmp_path / "upstream")
r.write("README.md", "# Service\r\nCRLF line endings, kept as they are.\r\n")
r.write("docs/a.md", "# A\n")
r.write("docs/x/y/b.md", "# B\n")
r.write("docs/config.yaml", "key: value\n")
r.write("src/main.py", "print('hi')\n")
r.commit("initial")
return r
def _capture(url=None, ref="main", paths=("docs/**/*.md", "README.md"), name="svc",
fidelity="verbatim", authority="normative", update=None):
return raw_capture_command(
repo_url=url, ref=None if update else ref, paths=None if update else list(paths),
name=None if update else name, fidelity=fidelity, authority=authority,
update=Path(update) if update else None,
)
def _update(bundle, fidelity=None, authority=None):
return raw_capture_command(
repo_url=None, ref=None, paths=None, name=None, fidelity=fidelity, authority=authority,
update=Path(bundle),
)
def _accept(path, **kwargs):
defaults = dict(fidelity=None, authority=None, page=None, replaces=None, dry_run=False)
defaults.update(kwargs)
return raw_accept_command(files=[Path(path)], **defaults)
def _tree(*dirs: Path) -> dict[str, bytes]:
"""Every file below `dirs`, with its bytes - what "nothing changed" is
asserted against."""
snapshot = {}
for top in dirs:
for path in sorted(top.rglob("*")):
if path.is_file():
snapshot[path.as_posix()] = path.read_bytes()
return snapshot
def _write_source(kb_dir, title, raw_files, fidelity="verbatim", authority="normative"):
write_page(
kb_dir / "sources" / f"{title}.md",
{
"type": "types/source.md", "source_type": "document", "author": "Fixture",
"raw_files": list(raw_files), "date": "2026-10-01", "tags": [], "entities": [],
"concepts": [], "summary": "Test source.", "fidelity": fidelity, "authority": authority,
},
f"\n# {title}\n\n## Summary\n\nTest.\n",
)
def _captured_and_accepted(capture, repo, **kwargs) -> Path:
_capture(repo.url, **kwargs)
_accept(capture / "incoming" / kwargs.get("name", "svc"))
return capture / _shard() / kwargs.get("name", "svc")
# --- globs --------------------------------------------------------------------
GLOB_FILES = (
"README.md", "docs/a.md", "docs/x/b.md", "docs/x/y/c.md", "docs/x/y/c.yaml", "docs/.h.md",
".github/w.md", "docx/n.md", "docs/x[1].md",
)
GLOB_PATTERNS = (
"docs/*.md", "docs/**/*.md", "docs", "docs/", "docs/x", "**/*.md", "docs/**", "*.md", "do*",
"docs/x*", "docs/**b.md", "*", "**", "docs/?.md", "docs/[ab].md", "docs/[!a].md", "**/c.*",
"docs/x/**/c.md",
)
def test_glob_semantics_are_gits_own(tmp_path):
"""Every pattern selects exactly what `git ls-files ':(glob)<p>'` selects -
the reference the issue names, asked of git itself rather than restated."""
work = tmp_path / "globs"
work.mkdir()
_git(work, "init", "-q", "-b", "main")
for rel in GLOB_FILES:
(work / rel).parent.mkdir(parents=True, exist_ok=True)
(work / rel).write_text("x\n", encoding="utf-8")
_git(work, "add", "-A")
for pattern in GLOB_PATTERNS:
expected = set(_git(work, "ls-files", "--", f":(glob){pattern}").splitlines())
actual = {rel for rel in GLOB_FILES if repo_capture.glob_matches(pattern, rel)}
assert actual == expected, pattern
def test_named_glob_cases():
assert repo_capture.glob_matches("docs/**/*.md", "docs/a.md")
assert repo_capture.glob_matches("docs/**/*.md", "docs/x/y/b.md")
assert not repo_capture.glob_matches("docs/*.md", "docs/x/b.md")
assert not repo_capture.glob_matches("docs/**/*.md", "docs/config.yaml")
@pytest.mark.parametrize("pattern", ["", "/docs/**", "docs/../x", "docs//a.md", "./docs"])
def test_unusable_globs_are_refused(pattern):
with pytest.raises(ValidationError):
repo_capture.check_glob(pattern)
# --- capture + accept ---------------------------------------------------------
def test_capture_and_accept_hold_exactly_the_matching_files_byte_for_byte(capture, repo, monkeypatch, tmp_path):
# The host's git would convert line endings on a checkout; a blob read must not.
gitconfig = tmp_path / "gitconfig"
gitconfig.write_text("[core]\n\tautocrlf = true\n", encoding="utf-8")
monkeypatch.setenv("GIT_CONFIG_GLOBAL", str(gitconfig))
_capture(repo.url)
incoming = capture / "incoming" / "svc"
manifest = json.loads((incoming / "_capture.json").read_text(encoding="utf-8"))
assert manifest["schema"] == 1
assert manifest["repo"] == repo.url
assert manifest["ref"] == "main"
assert manifest["commit"] == _git(repo.path, "rev-parse", "HEAD")
assert manifest["paths"] == ["docs/**/*.md", "README.md"]
assert manifest["fidelity"] == "verbatim" and manifest["authority"] == "normative"
assert manifest["files"] == ["README.md", "docs/a.md", "docs/x/y/b.md"]
assert manifest["captured"].endswith("Z")
_accept(incoming)
bundle = capture / _shard() / "svc"
on_disk = sorted(p.relative_to(bundle).as_posix() for p in bundle.rglob("*") if p.is_file())
assert on_disk == ["README.md", "_capture.json", "docs/a.md", "docs/x/y/b.md"]
for rel in ("README.md", "docs/a.md", "docs/x/y/b.md"):
assert (bundle / rel).read_bytes() == (repo.path / rel).read_bytes()
assert (bundle / "README.md").read_bytes().count(b"\r\n") == 2
assert not incoming.exists()
def test_accepting_a_captured_folder_takes_the_capture_fields_from_its_manifest(capture, repo, capsys):
_capture(repo.url, fidelity="published", authority="reporting")
_accept(capture / "incoming" / "svc")
out = _out(capsys)
assert "--set fidelity=published --set authority=reporting" in out
assert "_capture.json" not in out.split("--set raw_files=")[1].split()[0]
def test_accepting_a_captured_folder_with_capture_flags_is_refused(capture, repo):
_capture(repo.url)
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc", fidelity="verbatim", authority="normative")
assert _tree(capture / "incoming", capture / "raw") == before
def test_a_captured_folder_edited_after_capture_is_refused(capture, repo, capsys):
_capture(repo.url)
(capture / "incoming" / "svc" / "docs" / "extra.md").write_text("added by hand\n", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc")
assert "no longer matches" in _out(capsys)
def test_pending_judges_a_captured_folder_by_its_manifest(capture, repo, capsys):
_capture(repo.url)
capsys.readouterr()
raw_pending_command(json_out=True)
[candidate] = json.loads(capsys.readouterr().out)
assert candidate["kind"] == "folder" and candidate["acceptable"] is True
def test_capture_refuses_an_existing_incoming_folder(capture, repo):
(capture / "incoming" / "svc").mkdir()
with pytest.raises(typer.Exit):
_capture(repo.url)
assert list((capture / "incoming" / "svc").iterdir()) == []
def test_capture_refuses_a_name_a_captured_bundle_holds_and_names_update(capture, repo, capsys):
_captured_and_accepted(capture, repo)
capsys.readouterr()
with pytest.raises(typer.Exit):
_capture(repo.url)
assert "raw capture --update" in _out(capsys)
assert not (capture / "incoming" / "svc").exists()
def test_capture_refuses_when_nothing_matches(capture, repo):
with pytest.raises(typer.Exit):
_capture(repo.url, paths=("nothing/**",))
assert list((capture / "incoming").iterdir()) == []
def test_capture_refuses_a_path_over_the_budget_before_writing(capture, tmp_path, capsys):
r = Repo(tmp_path / "deep")
deep = "/".join(["d" * 30] * 6) + "/page.md"
r.write(deep, "# deep\n")
r.commit()
with pytest.raises(typer.Exit):
_capture(r.url, paths=("**",))
assert "path budget" in _out(capsys)
assert list((capture / "incoming").iterdir()) == []
def test_capture_follows_the_newest_tag_by_version_order(capture, repo):
repo.tag("v1.9")
repo.write("docs/a.md", "# A at 1.10\n")
v110 = repo.commit()
repo.tag("v1.10", annotated=True)
repo.write("docs/a.md", "# A on main, untagged\n")
repo.commit()
_capture(repo.url, ref="v*")
incoming = capture / "incoming" / "svc"
manifest = json.loads((incoming / "_capture.json").read_text(encoding="utf-8"))
assert manifest["commit"] == v110 # the peeled commit, not the tag object
assert (incoming / "docs/a.md").read_text(encoding="utf-8") == "# A at 1.10\n"
def test_capture_refuses_a_branch_that_does_not_exist(capture, repo):
with pytest.raises(typer.Exit):
_capture(repo.url, ref="nope")
assert list((capture / "incoming").iterdir()) == []
# --- exclusions ---------------------------------------------------------------
@posix_only
def test_excluded_paths_are_never_captured_and_each_is_named(capture, repo, capsys):
repo.write("docs/guideline.md", "<!-- wikitool:export from=kb -->\n# Exported\n")
repo.write(".github/workflow.md", "# CI\n")
repo.write("docs/big.bin", "version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 9\n")
repo.write("docs/sub/_capture.json", "{}\n")
os.symlink("a.md", repo.path / "docs" / "link.md")
sub = repo.commit()
_git(repo.path, "update-index", "--add", "--cacheinfo", f"160000,{sub},vendor/lib")
_git(repo.path, "commit", "-q", "-m", "submodule")
_capture(repo.url, paths=("**",))
out = _out(capsys)
for path, reason in (
("docs/guideline.md", "guideline export"),
(".github/workflow.md", "hidden path segment"),
("docs/big.bin", "Git LFS pointer"),
("docs/sub/_capture.json", "reserved name"),
("docs/link.md", "symlink"),
("vendor/lib", "submodule"),
):
assert f"excluded {path} ({reason}" in out
files = json.loads((capture / "incoming" / "svc" / "_capture.json").read_text(encoding="utf-8"))["files"]
assert files == ["README.md", "docs/a.md", "docs/config.yaml", "docs/x/y/b.md", "src/main.py"]
_accept(capture / "incoming" / "svc")
repo.write("docs/a.md", "# A, changed\n")
repo.commit()
capsys.readouterr()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert row["files"] == [{"path": f"{_shard()}/svc/docs/a.md", "status": "M"}]
def test_a_file_over_the_size_limit_is_excluded(capture, repo, monkeypatch, capsys):
from chemenu import web_capture
monkeypatch.setattr(web_capture, "MAX_BYTES", 5)
_capture(repo.url, paths=("docs/x/**",))
assert "excluded docs/x/y/b.md (over" not in _out(capsys) # 4 bytes: under
with pytest.raises(typer.Exit):
_capture(repo.url, paths=("README.md",), name="svc2")
assert "excluded README.md (over" in _out(capsys)
# --- URLs and credentials -----------------------------------------------------
@pytest.mark.parametrize("url", [
"ext::sh -c touch% /tmp/x",
"fd::17",
"file:///srv/repo.git",
"/srv/repo.git",
"http://example.org/repo.git",
"https://user:token@example.org/repo.git",
"ssh://user:secret@example.org/repo.git",
"-uhelp@example.org:x",
])
def test_refused_urls(url):
with pytest.raises(ValidationError):
repo_capture.check_repo_url(url)
@pytest.mark.parametrize("url", [
"https://example.org/team/repo.git",
"https://user@example.org/team/repo.git",
"ssh://git@example.org:2222/team/repo.git",
"git@example.org:team/repo.git",
])
def test_accepted_urls(url):
repo_capture.check_repo_url(url)
def _ext_url(marker: Path) -> str:
return f"ext::sh -c touch% {marker}"
@posix_only
def test_capture_refuses_an_ext_url_and_runs_nothing(capture, tmp_path):
marker = tmp_path / "ext-ran"
with pytest.raises(typer.Exit):
_capture(_ext_url(marker))
assert not marker.exists()
@posix_only
def test_git_itself_refuses_the_ext_transport(capture, tmp_path):
"""Below the URL check: the git layer alone, handed the URL directly."""
marker = tmp_path / "ext-ran"
with pytest.raises(BackendError):
with repo_capture.cache_repo("ext-probe") as cache:
repo_capture.resolve_ref(_ext_url(marker), "main", cache)
assert not marker.exists()
@posix_only
def test_the_ext_probe_would_run_if_git_allowed_it(capture, tmp_path, monkeypatch):
"""The control for the two tests above: the same URL does run its command
once `ext` is allowed, so their empty marker means something."""
marker = tmp_path / "ext-ran"
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ext",))
with pytest.raises(BackendError):
with repo_capture.cache_repo("ext-probe") as cache:
repo_capture.resolve_ref(_ext_url(marker), "main", cache)
assert marker.exists()
def test_a_refused_url_leaves_no_credential_anywhere(capture, tmp_path):
with pytest.raises(typer.Exit):
_capture("https://user:s3cr3t-token@example.org/repo.git")
for path in tmp_path.rglob("*"):
if path.is_file():
assert b"s3cr3t-token" not in path.read_bytes(), path
def _plant_bundle(capture, name, url, commit="0" * 40, files=("README.md",)):
"""A captured bundle under raw/ as an earlier accept would have left it."""
bundle = capture / _shard() / name
bundle.mkdir(parents=True)
for rel in files:
(bundle / rel).write_text("planted\n", encoding="utf-8")
manifest = {
"schema": 1, "repo": url, "ref": "main", "commit": commit, "paths": ["README.md"],
"captured": "2026-10-01T00:00:00Z", "fidelity": "verbatim", "authority": "normative",
"files": list(files),
}
(bundle / "_capture.json").write_text(json.dumps(manifest), encoding="utf-8")
return bundle
@posix_only
def test_update_and_status_refuse_an_ext_url_read_from_a_manifest(capture, tmp_path, capsys):
marker = tmp_path / "ext-ran"
bundle = _plant_bundle(capture, "evil", _ext_url(marker))
with pytest.raises(typer.Exit):
_update(bundle)
capsys.readouterr()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert "refused" in row["error"]
assert not marker.exists()
@pytest.fixture
def asks_for_credentials(monkeypatch):
"""An HTTP server that answers every request with a Basic-auth challenge -
the repository that would make an unguarded git prompt for a password."""
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802 - http.server's name
self.send_response(401)
self.send_header("WWW-Authenticate", 'Basic realm="repo"')
self.send_header("Content-Length", "0")
self.end_headers()
def log_message(self, *args):
pass
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
monkeypatch.setattr(repo_capture, "ALLOWED_SCHEMES", ("ssh", "https", "file", "http"))
yield f"http://127.0.0.1:{server.server_address[1]}/repo.git"
server.shutdown()
def test_a_repository_asking_for_credentials_is_unreachable_not_a_hang(capture, asks_for_credentials, capsys):
_plant_bundle(capture, "locked", asks_for_credentials)
started = time.monotonic()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert row["error"].startswith("not reachable")
with pytest.raises(typer.Exit):
_capture(asks_for_credentials, name="locked2")
assert time.monotonic() - started < 30
# --- raw status ---------------------------------------------------------------
def test_status_reports_changed_docs_and_goes_quiet_after_the_replacement(capture, repo, capsys):
bundle = _captured_and_accepted(capture, repo)
repo.write("docs/a.md", "# A, second edition\n")
repo.remove("docs/x/y/b.md")
repo.write("docs/new.md", "# New\n")
repo.commit()
capsys.readouterr()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
prefix = f"{_shard()}/svc"
assert row["changed"] is True and row["error"] is None
assert row["new"] == _git(repo.path, "rev-parse", "HEAD")
assert row["files"] == [
{"path": f"{prefix}/docs/a.md", "status": "M"},
{"path": f"{prefix}/docs/new.md", "status": "A"},
{"path": f"{prefix}/docs/x/y/b.md", "status": "D"},
]
raw_status_command(json_out=False)
out = _out(capsys)
assert f"raw capture --update {prefix}" in out
assert f"--replaces-bundle {prefix}" in out
_update(bundle)
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
capsys.readouterr()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert row["changed"] is False and row["files"] == []
def test_status_ignores_a_change_outside_the_globs(capture, repo, capsys):
_captured_and_accepted(capture, repo)
repo.write("src/main.py", "print('changed')\n")
repo.commit()
capsys.readouterr()
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert row["old"] != row["new"] and row["changed"] is False
raw_status_command(json_out=False)
out = _out(capsys)
assert "1 unchanged" in out and "svc" not in out.split("raw/:")[1].replace("1 unchanged.", "")
def test_status_follows_new_tags_only_not_new_commits_on_main(capture, repo, capsys):
repo.tag("v1.0")
_captured_and_accepted(capture, repo, ref="v*")
repo.write("docs/a.md", "# A on main\n")
repo.commit()
capsys.readouterr()
raw_status_command(json_out=True)
assert json.loads(capsys.readouterr().out)[0]["changed"] is False
repo.tag("v1.1")
raw_status_command(json_out=True)
[row] = json.loads(capsys.readouterr().out)
assert row["changed"] is True
assert row["files"] == [{"path": f"{_shard()}/svc/docs/a.md", "status": "M"}]
def test_an_unreachable_repository_is_one_line_beside_the_others(capture, repo, tmp_path, capsys):
_captured_and_accepted(capture, repo)
_plant_bundle(capture, "gone", (tmp_path / "does-not-exist").as_uri())
repo.write("docs/a.md", "# A, changed\n")
repo.commit()
capsys.readouterr()
raw_status_command(json_out=True)
rows = {Path(r["bundle"]).name: r for r in json.loads(capsys.readouterr().out)}
assert rows["gone"]["error"].startswith("not reachable")
assert rows["svc"]["changed"] is True and rows["svc"]["error"] is None
raw_status_command(json_out=False) # exits normally, both on their own line
out = capsys.readouterr().out
assert "not reachable" in out and "docs/a.md" in out
def test_status_with_no_captured_bundle(capture, capsys):
raw_status_command(json_out=False)
assert "No captured bundle" in capsys.readouterr().out
# --- raw accept --replaces-bundle ---------------------------------------------
def test_replaces_bundle_leaves_exactly_the_new_edition_in_place(capture, repo, kb_dir, capsys):
bundle = _captured_and_accepted(capture, repo)
prefix = f"{_shard()}/svc"
owned = [f"{prefix}/README.md", f"{prefix}/docs/a.md", f"{prefix}/docs/x/y/b.md"]
_write_source(kb_dir, "Source - Svc", owned)
repo.write("docs/a.md", "# A, second edition\n")
repo.remove("docs/x/y/b.md")
repo.write("docs/new.md", "# New\n")
new_commit = repo.commit()
_update(bundle)
capsys.readouterr()
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
out = _out(capsys)
on_disk = sorted(p.relative_to(bundle).as_posix() for p in bundle.rglob("*") if p.is_file())
assert on_disk == ["README.md", "_capture.json", "docs/a.md", "docs/new.md"]
for rel in ("README.md", "docs/a.md", "docs/new.md"):
assert (bundle / rel).read_bytes() == (repo.path / rel).read_bytes()
assert not (bundle / "docs" / "x").exists()
assert json.loads((bundle / "_capture.json").read_text(encoding="utf-8"))["commit"] == new_commit
assert not (capture / "incoming" / "svc").exists()
assert read_page(kb_dir / "sources" / "Source - Svc.md")[0]["raw_files"] == owned
assert f"M {prefix}/docs/a.md" in out and f"D {prefix}/docs/x/y/b.md" in out
assert f"A {prefix}/docs/new.md" in out
assert f'touch --page "Source - Svc" --remove raw_files={prefix}/docs/x/y/b.md' in out
assert f'touch --page "Source - Svc" --add raw_files={prefix}/docs/new.md' in out
def test_replaces_bundle_overwrites_changed_capture_fields_on_the_owning_page(capture, repo, kb_dir):
bundle = _captured_and_accepted(capture, repo)
_write_source(kb_dir, "Source - Svc", [f"{_shard()}/svc/README.md"])
_update(bundle, authority="reporting")
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
assert read_page(kb_dir / "sources" / "Source - Svc.md")[0]["authority"] == "reporting"
def test_replaces_bundle_refuses_another_repository(capture, repo, tmp_path):
bundle = _captured_and_accepted(capture, repo)
other = Repo(tmp_path / "other")
other.write("README.md", "# Other\n")
other.commit()
_capture(other.url, paths=("README.md",), name="other")
(capture / "incoming" / "other").rename(capture / "incoming" / "svc")
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
assert _tree(capture / "incoming", capture / "raw") == before
def test_replaces_bundle_refuses_a_different_folder_name(capture, repo):
bundle = _captured_and_accepted(capture, repo)
_update(bundle)
(capture / "incoming" / "svc").rename(capture / "incoming" / "svc-renamed")
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc-renamed", replaces_bundle=bundle)
assert _tree(capture / "incoming", capture / "raw") == before
def test_replaces_bundle_refuses_a_bundle_without_a_manifest(capture, repo):
plain = capture / _shard() / "svc"
plain.mkdir(parents=True)
(plain / "README.md").write_text("hand-made folder bundle\n", encoding="utf-8")
_capture(repo.url, name="svc-new")
(capture / "incoming" / "svc-new").rename(capture / "incoming" / "svc")
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc", replaces_bundle=plain)
assert _tree(capture / "incoming", capture / "raw") == before
def test_replaces_bundle_refuses_page_replaces_and_capture_flags(capture, repo):
bundle = _captured_and_accepted(capture, repo)
_update(bundle)
before = _tree(capture / "incoming", capture / "raw")
for extra in ({"page": "Source - Svc"}, {"replaces": bundle / "README.md"}, {"fidelity": "verbatim"}):
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc", replaces_bundle=bundle, **extra)
assert _tree(capture / "incoming", capture / "raw") == before
def test_replaces_bundle_refuses_a_file_with_two_owners(capture, repo, kb_dir):
bundle = _captured_and_accepted(capture, repo)
readme = f"{_shard()}/svc/README.md"
_write_source(kb_dir, "Source - One", [readme])
_write_source(kb_dir, "Source - Two", [readme])
_update(bundle)
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "svc", replaces_bundle=bundle)
assert _tree(capture / "incoming", capture / "raw") == before
# --- --replaces / --page inside a captured bundle -----------------------------
def test_replaces_refuses_a_target_inside_a_captured_bundle(capture, repo, capsys):
bundle = _captured_and_accepted(capture, repo)
(capture / "incoming" / "b.md").write_text("# B by hand\n", encoding="utf-8")
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "b.md", replaces=bundle / "docs" / "x" / "y" / "b.md")
assert _tree(capture / "incoming", capture / "raw") == before
assert "--replaces-bundle" in _out(capsys)
def test_page_refuses_to_grow_a_captured_bundle(capture, tmp_path, kb_dir):
flat = Repo(tmp_path / "flat")
flat.write("README.md", "# R\n")
flat.write("AGENTS.md", "# A\n")
flat.commit()
_capture(flat.url, paths=("*.md",), name="flat")
_accept(capture / "incoming" / "flat")
prefix = f"{_shard()}/flat"
_write_source(kb_dir, "Source - Flat", [f"{prefix}/AGENTS.md", f"{prefix}/README.md"])
(capture / "incoming" / "notes.md").write_text("# notes\n", encoding="utf-8")
before = _tree(capture / "incoming", capture / "raw")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "notes.md", fidelity="verbatim", authority="normative",
page="Source - Flat")
assert _tree(capture / "incoming", capture / "raw") == before
def test_a_loose_file_named_like_the_manifest_is_refused(capture):
(capture / "incoming" / "_capture.json").write_text("{}\n", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(capture / "incoming" / "_capture.json", fidelity="verbatim", authority="normative")
# --- coverage -----------------------------------------------------------------
def test_coverage_skips_the_manifest_and_sees_a_bundled_contract(capture, tmp_path):
r = Repo(tmp_path / "with-contract")
r.write("README.md", "# R\n")
r.write("raw/CONTRACT.md", "# a repository's own stage contract\n")
r.commit()
_capture(r.url, paths=("README.md", "raw/CONTRACT.md"), name="wc")
_accept(capture / "incoming" / "wc")
(capture / "raw" / "CONTRACT.md").write_text("# the stage contract\n", encoding="utf-8")
uncovered = uncovered_raw_files(config.RAW_DIR, load_kb_pages(config.KB_DIR))
prefix = f"{_shard()}/wc"
assert f"{prefix}/raw/CONTRACT.md" in uncovered
assert f"{prefix}/README.md" in uncovered
assert f"{prefix}/_capture.json" not in uncovered
assert "raw/CONTRACT.md" not in uncovered