fix: publish/sync merge generated files mechanically and carry non-overlapping uncommitted work through a rebase (#180)
Overlap only in kb/index.md, kb/log.md, kb/provenance.md and kb/**/INDEX.md no longer fails a reconcile or reaches the rebase-review gate: the log keeps both sides' entries, the catalog and provenance are regenerated. Uncommitted work no incoming commit touches rides through the rebase via --autostash; the working tree is backed up under refs/wikitool/reconcile-backup first. is_generated is narrowed to kb/. Files changed: - CHANGES.md - VERSION - instructions/gates.md - instructions/session-setup.md - tools/CONTRACT.md - tools/chemenu/commands/git_publish.py - tools/chemenu/commands/index_build.py - tools/chemenu/commands/provenance_cmd.py - tools/chemenu/tests/test_git_publish.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
40839d956d
commit
d4638bacde
9 files changed
+907
-58
No files matched your search
+15
-9
@@ -1888,7 +1888,7 @@ Fetch `<remote>/<branch>` and bring the local branch up to date with it.
|
||||
|
||||
- effect: write
|
||||
- idempotent: yes
|
||||
- atomic: No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure
|
||||
- atomic: No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure with the working tree restored; a success that merged generated files then rewrites the catalog
|
||||
- budget: counted
|
||||
- network: yes
|
||||
- gates: rebase-review
|
||||
@@ -1903,13 +1903,15 @@ Fetch `<remote>/<branch>` and bring the local branch up to date with it.
|
||||
- 0 success
|
||||
- 0 No remote configured - reported and skipped, not a failure
|
||||
- 0 The remote cannot be reached - reported and skipped, not a failure
|
||||
- 1 The automatic rebase hit a real conflict (git failed); it is aborted cleanly
|
||||
- 42 Rebase-review gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff
|
||||
- 1 The automatic rebase hit a real conflict outside the generated files (git failed), or `kb/log.md` was edited rather than appended to; it is aborted cleanly and the working tree restored
|
||||
- 1 An uncommitted change - or an untracked file - sits on a path the incoming commits also change; nothing was changed
|
||||
- 42 Rebase-review gate: `<remote>/<branch>` moved and both sides changed the same file, generated files aside; the output lists the upstream commits, the overlapping files and their diff
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- The automatic rebase hit a real conflict (git failed); it is aborted cleanly -> Do not retry and do not force - resolve the conflict manually, then re-run
|
||||
- Rebase-review gate: `<remote>/<branch>` moved and both sides changed the same file; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||
- The automatic rebase hit a real conflict outside the generated files (git failed), or `kb/log.md` was edited rather than appended to; it is aborted cleanly and the working tree restored -> Do not retry and do not force - resolve the conflict manually, then re-run
|
||||
- An uncommitted change - or an untracked file - sits on a path the incoming commits also change; nothing was changed -> Show the user the file it names. Once that change is committed or moved out of the way, re-run once
|
||||
- Rebase-review gate: `<remote>/<branch>` moved and both sides changed the same file, generated files aside; the output lists the upstream commits, the overlapping files and their diff -> Show the user the command's full output verbatim and stop. Once they have approved it, run the re-run line the output prints, which carries `--confirm-rebase <token>`. Without that token, or with a wrong, invented or superseded one, it exits 42 again with the current state
|
||||
|
||||
**NEVER**
|
||||
|
||||
@@ -1919,8 +1921,11 @@ Fetch `<remote>/<branch>` and bring the local branch up to date with it.
|
||||
**NOTES**
|
||||
|
||||
- Fetches `<remote>/<branch>`, then: fast-forwards when only the remote moved; rebases the local commits on top when both sides moved but touched disjoint files; exits 42 (rebase-review gate) when both sides touched the same file.
|
||||
- Files `wikitool` generates (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `kb/**/INDEX.md`) never count as an overlap and never reach the gate. Where they are what stops git - uncommitted ones in the way of a fast-forward or rebase, or a rebase conflict in them - they are merged mechanically: `kb/log.md` becomes the incoming log plus the entries only this side appended, and the catalog and `kb/provenance.md` are regenerated from the merged pages and left as an uncommitted change. A reconcile git can do on its own is unchanged and regenerates nothing.
|
||||
- Uncommitted changes that no incoming commit touches no longer stop a rebase: they are carried through it (`git rebase --autostash`) and come back unchanged.
|
||||
- Before anything is set aside, the working tree is stored as a commit under `refs/wikitool/reconcile-backup`. A successful reconcile drops it; a failed one restores the working tree and names the ref.
|
||||
- A refused call performs no rebase attempt and leaves the branch where it was.
|
||||
- The `--confirm-rebase` token covers the exact upstream state and the set of files touched on both sides; either one moving makes it stale.
|
||||
- The `--confirm-rebase` token covers the exact upstream state and the set of files touched on both sides, generated files excluded; either one moving makes it stale.
|
||||
- Makes no commit, no push, and no forced operation of any kind.
|
||||
- Three messages for a fetch that fails: no remote of that name, a remote that answers but has no such branch yet (a new, empty repository), and a remote that cannot be reached. All three exit 0 here; `publish` stops on the first and the last.
|
||||
- Run it once at the start of a writing session.
|
||||
@@ -1957,7 +1962,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase
|
||||
- 1 git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change)
|
||||
- 1 The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case
|
||||
- 1 No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed
|
||||
- 1 `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error
|
||||
@@ -1968,7 +1973,7 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
|
||||
- git failed - `git add`, `git commit`, `git push`, or the reconcile's automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, or an uncommitted change on a path the incoming commits change) -> Do not retry and do not force - report and ask the user. `publish` has already made its one retry of a rejected push itself, where a reconcile resolved the rejection
|
||||
- The push target (`--branch`) is not the checked-out branch, or HEAD is detached; the unborn branch of a fresh `git init` is not this case -> Check out the branch you mean to publish, or pass `--branch <checked-out branch>`, then retry once
|
||||
- No `--no-push`, and the remote is not configured or cannot be reached; nothing was committed -> Show the message to the user and ask whether to commit locally with `--no-push`. Never push by hand - the next `publish` that reaches the remote sends that commit
|
||||
- `--yes`/`-y` was passed - the flag does not exist and fails with an explicit error -> Drop it. The Mass-Update Gate is cleared only with `--confirm <token>` from the gate's own refusal output
|
||||
@@ -1988,12 +1993,13 @@ Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.
|
||||
- Order: branch check and Publish-Remote Gate, then the reconcile with `<remote>/<branch>`, then the Mass-Update Gate, then `git add -A`, commit and push. `--no-push` skips all but the Mass-Update Gate and the commit.
|
||||
- Without `--no-push`, a remote that is not configured or cannot be reached ends the call with exit 1 at the reconcile - before the gate, `git add` and the commit, and also on a clean tree. Nothing is committed, the index and the working tree are unchanged, and the message names `--no-push` as the way to a local commit. The next `publish` that reaches the remote pushes that commit along with whatever is new. A remote that answers but has no `<branch>` yet (a new, empty repository) is not this case: the first publish of an instance commits and pushes as before.
|
||||
- Reconcile: fetches `<remote>/<branch>`, fast-forwards when only the remote moved, rebases the local commits on top when both sides moved but touched disjoint files, and exits 42 (rebase-review gate) when both sides touched the same file. A refused reconcile performs no rebase attempt. The `--confirm-rebase` token covers the exact upstream state and the set of files touched on both sides.
|
||||
- Generated files are never an overlap: where they are all that stops the reconcile, it merges them as `sync` does (both sides' log entries kept, the catalog and `kb/provenance.md` regenerated) and carries uncommitted changes no incoming commit touches through the rebase. The proactive reconcile runs before staging, so this publish commits the regenerated files; on the retry after a rejected push they get a commit of their own before the second push.
|
||||
- The push target must be the checked-out branch; this is checked before anything is staged. The unborn branch of a fresh `git init -b main` counts as checked out, so the first publish of a new instance works; a real detached HEAD is refused.
|
||||
- With nothing new to stage, a local commit the remote lacks is still pushed: one left behind by an earlier publish whose push failed, or every commit when the remote answers but does not have the branch yet (a new, empty remote repository).
|
||||
- A rejected push that finds the remote unreachable on its one retry reports the original push error.
|
||||
- A rejected push gets exactly one more reconcile-and-push; never more than one.
|
||||
- Mass-Update Gate: counts the files that would be committed, refuses with exit 42 at `--threshold` (default 10) or more, and prints a review report - a scale line (file count, total lines added/removed, status breakdown), attention notes where they apply (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn. The list is what the commit will hold: it is computed from a scratch copy of the index after `git add -A`, so a path that is staged as deleted and back in the working tree is not counted twice, and a rename counts as its old path deleted plus its new path added. The real index and the working tree are not touched, so a refused publish leaves both byte-identical.
|
||||
- Never counted and never shown for approval, but committed like everything else: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md`). The refusal line accounts for both, by reason.
|
||||
- Never counted and never shown for approval, but committed like everything else: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`). The refusal line accounts for both, by reason.
|
||||
- The `--confirm` token covers each counted path, the blob id of its contents and the publish target: a different file list or edited contents need a new clearance.
|
||||
- Publish-Remote Gate: when the checkout carries `.wikitool-remotes.json` and the push URL of `--remote` is not listed in it, exits 42 before the reconcile fetches anything. The URL is read with `git remote get-url --push`, so a repointed remote does not pass on its name. An absent file means unrestricted; a malformed one is an error, not permission.
|
||||
- `--path` (repeatable) scopes the whole operation - gate count, staging and commit - to that subtree.
|
||||
|
||||
@@ -12,7 +12,9 @@ same exit-42 idiom as the gate below - when it might. This is what keeps a
|
||||
rejected push from stranding the commit that made it: a previous `publish`
|
||||
whose push failed leaves a real, unpushed commit sitting on the branch, and
|
||||
the next `publish` now pushes it instead of reporting "Nothing to commit"
|
||||
forever.
|
||||
forever. Files `wikitool` generates never count as a collision: where they
|
||||
are what stops git, the reconcile merges them mechanically and regenerates
|
||||
the catalog (see "Reconciling around generated files" below).
|
||||
|
||||
Also implements the Mass-Update Gate (kb/concepts/workflows/Mass-Update Gate.md):
|
||||
a push to origin/main is the one action in this system with a real,
|
||||
@@ -80,12 +82,14 @@ DEFAULT_MASS_UPDATE_THRESHOLD = 10
|
||||
GATE_EXEMPT_PREFIXES = ("work/",)
|
||||
|
||||
|
||||
def _run(args: list[str], input: Optional[str] = None) -> subprocess.CompletedProcess:
|
||||
"""Run a `git ...` argument list, starting git from its recorded path."""
|
||||
def _run(args: list[str], input: Optional[str] = None,
|
||||
env: Optional[dict[str, str]] = None) -> subprocess.CompletedProcess:
|
||||
"""Run a `git ...` argument list, starting git from its recorded path. `env` is laid over
|
||||
the inherited environment, not substituted for it."""
|
||||
if args and args[0] == "git":
|
||||
args = [toolpaths.git(), *args[1:]]
|
||||
return subprocess.run(args, cwd=config.ROOT, capture_output=True, text=True, encoding="utf-8",
|
||||
input=input)
|
||||
input=input, env={**os.environ, **env} if env else None)
|
||||
|
||||
|
||||
# --- Publish-Remote Gate -----------------------------------------------------
|
||||
@@ -352,12 +356,17 @@ def branch_mismatch_message(checked_out: Optional[str], branch: str) -> str:
|
||||
# them pushed ordinary single-source ingests over a threshold meant for mass
|
||||
# updates. Three consecutive ingests on 2026-08-31 each stopped at the gate with
|
||||
# 5-7 of their files generated; none of them was a mass update.
|
||||
#
|
||||
# Only under `kb/`, which is also what AGENTS.md invariant 1 names (`kb/**/INDEX.md`). An
|
||||
# `INDEX.md` in a captured repository under `raw/` is source material that happens to share the
|
||||
# name - and `reconcile` discards and regenerates what this predicate calls generated.
|
||||
GENERATED_SUFFIXES = ("/INDEX.md",)
|
||||
GENERATED_PATHS = ("kb/index.md", "kb/log.md", "kb/provenance.md")
|
||||
GENERATED_LOG = "kb/log.md"
|
||||
|
||||
|
||||
def is_generated(path: str) -> bool:
|
||||
return path in GENERATED_PATHS or path.endswith(GENERATED_SUFFIXES)
|
||||
return path in GENERATED_PATHS or (path.startswith("kb/") and path.endswith(GENERATED_SUFFIXES))
|
||||
|
||||
|
||||
def is_exempt(path: str, prefixes: tuple[str, ...] = GATE_EXEMPT_PREFIXES) -> bool:
|
||||
@@ -767,12 +776,276 @@ def rebase_review_token(remote: str, branch: str, local_before: str, remote_tip:
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
|
||||
|
||||
|
||||
# --- Reconciling around generated files ---------------------------------------
|
||||
#
|
||||
# Every ingest touches kb/index.md and kb/log.md, so two ingests on two checkouts always
|
||||
# "overlap" - in files that carry no decision and can be recomputed from the result. Left to
|
||||
# git, that overlap stopped every such reconcile: a fast-forward refused over the uncommitted
|
||||
# catalog, a rebase went to review and then failed on the two appends at the end of the log,
|
||||
# and `git rebase` refused outright over any uncommitted file at all.
|
||||
#
|
||||
# The path below takes the generated files out of git's way and puts the result back
|
||||
# together mechanically: the log is the incoming version plus what only this side appended,
|
||||
# and the catalog and kb/provenance.md are regenerated from the merged pages. It runs only
|
||||
# where git would otherwise have failed, so a reconcile that succeeds today is unchanged -
|
||||
# in particular it does not rebuild, because `index rebuild` stamps today's date into
|
||||
# kb/index.md and every `sync` would then end with a modified file.
|
||||
#
|
||||
# Uncommitted work that is not generated is never discarded: it rides through a rebase in
|
||||
# `--autostash` and comes back unchanged, which needs only that no incoming commit touches it.
|
||||
# Before the first write, the whole working tree is stored under BACKUP_REF.
|
||||
|
||||
BACKUP_REF = "refs/wikitool/reconcile-backup"
|
||||
|
||||
# The identity of the backup commit, so taking one never depends on - or fails for want of -
|
||||
# a configured user.
|
||||
_BACKUP_IDENTITY = {
|
||||
"GIT_AUTHOR_NAME": "wikitool", "GIT_AUTHOR_EMAIL": "wikitool@localhost",
|
||||
"GIT_COMMITTER_NAME": "wikitool", "GIT_COMMITTER_EMAIL": "wikitool@localhost",
|
||||
}
|
||||
|
||||
# A rebase that stops on a conflict this module resolves is continued without an editor.
|
||||
# GIT_EDITOR and not `-c core.editor`, which an exported GIT_EDITOR would override.
|
||||
_NO_EDITOR = {"GIT_EDITOR": "true"}
|
||||
|
||||
|
||||
def _git_bytes(*args: str) -> Optional[bytes]:
|
||||
"""stdout of `git <args>` as raw bytes, or None when git fails. Bytes, because a file's
|
||||
content goes back to disk exactly as git holds it - text mode would fold line endings."""
|
||||
result = subprocess.run([toolpaths.git(), *args], cwd=config.ROOT, capture_output=True)
|
||||
return result.stdout if result.returncode == 0 else None
|
||||
|
||||
|
||||
def _dirty_paths() -> tuple[list[str], list[str]]:
|
||||
"""(tracked paths whose index or working copy differs from HEAD, untracked paths). Ignored
|
||||
files are neither - nothing here ever touches one."""
|
||||
result = _run(["git", "status", "--porcelain=v1", "-z", "--untracked-files=all", "--no-renames"])
|
||||
tracked: list[str] = []
|
||||
untracked: list[str] = []
|
||||
for record in result.stdout.split("\0"):
|
||||
if len(record) < 4:
|
||||
continue
|
||||
(untracked if record[:2] == "??" else tracked).append(record[3:])
|
||||
return tracked, untracked
|
||||
|
||||
|
||||
def _read(path: str) -> Optional[bytes]:
|
||||
target = config.ROOT / path
|
||||
return target.read_bytes() if target.is_file() else None
|
||||
|
||||
|
||||
def _log_suffix() -> Optional[bytes]:
|
||||
"""The bytes the working copy of kb/log.md has beyond its HEAD version - b"" when it has
|
||||
none - or None when the working copy is not the HEAD version with something appended.
|
||||
`log append` only ever appends, so None means a hand edit, and the log is then left to
|
||||
git as before."""
|
||||
work = _read(GENERATED_LOG)
|
||||
head = _git_bytes("show", f"HEAD:{GENERATED_LOG}")
|
||||
if work is None:
|
||||
return b"" if head is None else None
|
||||
head = head or b""
|
||||
return work[len(head):] if work.startswith(head) else None
|
||||
|
||||
|
||||
def _in_head(path: str) -> bool:
|
||||
return _run(["git", "cat-file", "-e", f"HEAD:{path}"]).returncode == 0
|
||||
|
||||
|
||||
def _rebase_in_progress() -> bool:
|
||||
return any((config.ROOT / _git_path(name)).exists() for name in ("rebase-merge", "rebase-apply"))
|
||||
|
||||
|
||||
def _back_up_working_tree() -> Optional[str]:
|
||||
"""Store the working tree as it stands - tracked and untracked files, not ignored ones -
|
||||
as a commit on top of HEAD under BACKUP_REF, and return its id; None if that failed.
|
||||
Staged through a scratch copy of the index, the same way `collect_changes` stages, so the
|
||||
real index is not touched."""
|
||||
index_path = config.ROOT / _git_path("index")
|
||||
scratch = Path(tempfile.mkdtemp(prefix="wikitool-backup-"))
|
||||
try:
|
||||
temp_index = scratch / "index"
|
||||
if index_path.is_file():
|
||||
shutil.copyfile(index_path, temp_index)
|
||||
env = {"GIT_INDEX_FILE": str(temp_index)}
|
||||
if _run(["git", "add", "-A"], env=env).returncode != 0:
|
||||
return None
|
||||
tree = _run(["git", "write-tree"], env=env)
|
||||
finally:
|
||||
shutil.rmtree(scratch, ignore_errors=True)
|
||||
if tree.returncode != 0:
|
||||
return None
|
||||
commit = _run(
|
||||
["git", "commit-tree", tree.stdout.strip(), "-p", "HEAD",
|
||||
"-m", "wikitool reconcile: the working tree before generated files were set aside"],
|
||||
env=_BACKUP_IDENTITY,
|
||||
)
|
||||
if commit.returncode != 0:
|
||||
return None
|
||||
commit_id = commit.stdout.strip()
|
||||
if _run(["git", "update-ref", BACKUP_REF, commit_id]).returncode != 0:
|
||||
return None
|
||||
return commit_id
|
||||
|
||||
|
||||
@dataclass
|
||||
class _SetAside:
|
||||
"""The generated files taken out of git's way: each one's working-copy bytes before
|
||||
(None where it did not exist), and the bytes only this side appended to kb/log.md."""
|
||||
originals: dict[str, Optional[bytes]]
|
||||
log_suffix: bytes
|
||||
|
||||
|
||||
def _set_aside(paths: list[str]) -> _SetAside:
|
||||
"""Return each generated path in `paths` to its HEAD state - or remove it, where HEAD
|
||||
has none - after recording what was there."""
|
||||
aside = _SetAside(
|
||||
originals={path: _read(path) for path in paths},
|
||||
log_suffix=(_log_suffix() or b"") if GENERATED_LOG in paths else b"",
|
||||
)
|
||||
in_head = [path for path in paths if _in_head(path)]
|
||||
if in_head:
|
||||
_run(["git", "checkout", "HEAD", "--", *in_head])
|
||||
for path in paths:
|
||||
if path not in in_head:
|
||||
_run(["git", "rm", "-q", "--cached", "--ignore-unmatch", "--", path])
|
||||
(config.ROOT / path).unlink(missing_ok=True)
|
||||
return aside
|
||||
|
||||
|
||||
def _put_back(aside: _SetAside) -> None:
|
||||
"""Undo `_set_aside` byte for byte in the working tree."""
|
||||
for path, data in aside.originals.items():
|
||||
target = config.ROOT / path
|
||||
if data is None:
|
||||
target.unlink(missing_ok=True)
|
||||
else:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(data)
|
||||
|
||||
|
||||
def _generated_to_set_aside(incoming: set[str]) -> tuple[Optional[list[str]], str]:
|
||||
"""The dirty generated paths, or (None, why) when uncommitted work stands in the way of
|
||||
reconciling with `incoming` at all - a non-generated file both sides change, or a
|
||||
kb/log.md that was edited rather than appended to. Reads only."""
|
||||
tracked, untracked = _dirty_paths()
|
||||
for path in tracked:
|
||||
if not is_generated(path) and path in incoming:
|
||||
return None, f"{path} has uncommitted changes and the incoming commits change it too"
|
||||
for path in untracked:
|
||||
if not is_generated(path) and path in incoming:
|
||||
return None, f"the untracked {path} would be overwritten by an incoming commit"
|
||||
generated = sorted(path for path in tracked + untracked if is_generated(path))
|
||||
if GENERATED_LOG in generated and _log_suffix() is None:
|
||||
return None, f"{GENERATED_LOG} has uncommitted changes that are not only appended entries"
|
||||
return generated, ""
|
||||
|
||||
|
||||
def _resolve_generated_conflicts() -> Optional[str]:
|
||||
"""Resolve the conflicts of a stopped rebase step when every one of them is in a generated
|
||||
file; None on success, else why not. Stage 2 is the side being rebased onto, stage 3 the
|
||||
commit being replayed: the catalog and provenance take stage 2 (they are regenerated
|
||||
afterwards anyway), the log takes stage 2 plus what stage 3 appended to stage 1."""
|
||||
listed = _run(["git", "diff", "--name-only", "--diff-filter=U", "-z"]).stdout
|
||||
unmerged = [path for path in listed.split("\0") if path]
|
||||
if not unmerged:
|
||||
return "the rebase stopped without a file conflict"
|
||||
foreign = [path for path in unmerged if not is_generated(path)]
|
||||
if foreign:
|
||||
return f"conflict in {', '.join(foreign)}"
|
||||
for path in unmerged:
|
||||
ours = _git_bytes("show", f":2:{path}")
|
||||
target = config.ROOT / path
|
||||
if path == GENERATED_LOG:
|
||||
base = _git_bytes("show", f":1:{path}") or b""
|
||||
theirs = _git_bytes("show", f":3:{path}")
|
||||
if ours is None or theirs is None or not theirs.startswith(base):
|
||||
return f"{path} was changed on one side by more than appended entries"
|
||||
target.write_bytes(ours + theirs[len(base):])
|
||||
elif ours is None:
|
||||
_run(["git", "rm", "-q", "--cached", "--", path])
|
||||
target.unlink(missing_ok=True)
|
||||
continue
|
||||
else:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(ours)
|
||||
if _run(["git", "add", "--", path]).returncode != 0:
|
||||
return f"git add {path} failed"
|
||||
return None
|
||||
|
||||
|
||||
def _rebase_resolving_generated(remote_ref: str) -> tuple[bool, str, bool]:
|
||||
"""`git rebase --autostash <remote_ref>`, resolving each step whose conflicts are all in
|
||||
generated files. Returns (succeeded, detail, resolved anything). Any other stop is aborted,
|
||||
which also hands the autostash back."""
|
||||
result = _run(["git", "rebase", "--autostash", remote_ref], env=_NO_EDITOR)
|
||||
resolved = False
|
||||
while result.returncode != 0:
|
||||
if not _rebase_in_progress():
|
||||
return False, result.stderr, resolved
|
||||
reason = _resolve_generated_conflicts()
|
||||
if reason:
|
||||
_run(["git", "rebase", "--abort"])
|
||||
return False, f"{reason}\n{result.stderr}", resolved
|
||||
resolved = True
|
||||
result = _run(["git", "rebase", "--continue"], env=_NO_EDITOR)
|
||||
return True, "", resolved
|
||||
|
||||
|
||||
def regenerate_generated(log_suffix: bytes = b"") -> None:
|
||||
"""Append `log_suffix` to kb/log.md, then rewrite the catalog and kb/provenance.md from the
|
||||
pages in the working tree - what `index rebuild` and `sources rebuild-index` write. The log
|
||||
goes first: it is the one part that cannot be recomputed."""
|
||||
from chemenu.commands.index_build import write_index
|
||||
from chemenu.commands.provenance_cmd import write_provenance_index
|
||||
|
||||
if log_suffix:
|
||||
with (config.ROOT / GENERATED_LOG).open("ab") as log:
|
||||
log.write(log_suffix)
|
||||
write_index(config.KB_DIR)
|
||||
write_provenance_index(config.KB_DIR, config.RAW_DIR)
|
||||
|
||||
|
||||
def _around_generated(generated: list[str], operation, regenerate: bool) -> tuple[bool, str, bool]:
|
||||
"""Back up the working tree, set `generated` aside, run `operation` - which returns
|
||||
(succeeded, detail, resolved generated conflicts) - and then either regenerate or put
|
||||
everything back. Returns (succeeded, detail, regenerated).
|
||||
|
||||
Regenerates when something generated was set aside or resolved, or when `regenerate` says
|
||||
both sides' commits touched a generated file - a textual merge of two catalogs can succeed
|
||||
and still be wrong. The backup ref is dropped on success and kept, and named, on any
|
||||
failure."""
|
||||
backup = _back_up_working_tree()
|
||||
if backup is None:
|
||||
return False, "could not back up the working tree first, so nothing was changed", False
|
||||
aside = _set_aside(generated)
|
||||
try:
|
||||
succeeded, detail, resolved = operation()
|
||||
except BaseException:
|
||||
if _rebase_in_progress():
|
||||
_run(["git", "rebase", "--abort"])
|
||||
_put_back(aside)
|
||||
raise
|
||||
if not succeeded:
|
||||
_put_back(aside)
|
||||
return False, (
|
||||
f"{detail.rstrip()}\nThe working tree was restored; as it stood before, it is also "
|
||||
f"kept at {BACKUP_REF} ({backup[:12]})."
|
||||
), False
|
||||
regenerated = bool(generated) or resolved or regenerate
|
||||
if regenerated:
|
||||
regenerate_generated(aside.log_suffix)
|
||||
_run(["git", "update-ref", "-d", BACKUP_REF])
|
||||
return True, "", regenerated
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReconcileOutcome:
|
||||
"""What happened when the local branch was brought up to date with the remote before a
|
||||
publish/sync. `status` is one of: "no-remote", "remote-lacks-branch", "unreachable",
|
||||
"up-to-date", "fast-forwarded", "local-ahead", "rebased", "needs-review", "conflict".
|
||||
The first three are the ways the fetch can fail; `detail` then carries the remote's URL."""
|
||||
The first three are the ways the fetch can fail; `detail` then carries the remote's URL.
|
||||
`regenerated` says the catalog and kb/provenance.md were rewritten afterwards and are an
|
||||
uncommitted change now."""
|
||||
status: str
|
||||
pulled_commits: list[str] = field(default_factory=list)
|
||||
overlap_files: list[str] = field(default_factory=list)
|
||||
@@ -780,6 +1053,7 @@ class ReconcileOutcome:
|
||||
token: str = ""
|
||||
was_reviewed: bool = False
|
||||
detail: str = ""
|
||||
regenerated: bool = False
|
||||
|
||||
|
||||
def reconcile(remote: str, branch: str, confirm_rebase: Optional[str] = None) -> ReconcileOutcome:
|
||||
@@ -791,6 +1065,11 @@ def reconcile(remote: str, branch: str, confirm_rebase: Optional[str] = None) ->
|
||||
actually rewrites history: an overlap that has not been cleared performs no rebase attempt
|
||||
at all, so a refused call leaves the branch exactly where it was found.
|
||||
|
||||
Generated files (`is_generated`) are left out of the overlap: where they - or uncommitted
|
||||
work that no incoming commit touches - are all that stops git, the reconcile goes ahead
|
||||
around them (see "Reconciling around generated files" above) and leaves the regenerated
|
||||
catalog as an uncommitted change.
|
||||
|
||||
Never commits, never pushes, never force-anything. Callers are `sync` (this is its whole
|
||||
job) and `publish` (proactively before staging, and once more if the eventual push is
|
||||
rejected - the real, narrow race this whole module exists to close)."""
|
||||
@@ -812,12 +1091,24 @@ def reconcile(remote: str, branch: str, confirm_rebase: Optional[str] = None) ->
|
||||
if state == "ff-possible":
|
||||
pulled = oneline_log(branch, remote_ref)
|
||||
result = _run(["git", "merge", "--ff-only", remote_ref])
|
||||
if result.returncode != 0:
|
||||
# Only reachable if uncommitted local changes would be overwritten by the merge -
|
||||
# git itself refuses and touches nothing, so this is a safe abort, not a half-done
|
||||
# state.
|
||||
if result.returncode == 0:
|
||||
return ReconcileOutcome(status="fast-forwarded", pulled_commits=pulled)
|
||||
# Only reachable if uncommitted local changes would be overwritten by the merge -
|
||||
# git itself refuses and touches nothing, so this is a safe abort, not a half-done
|
||||
# state. When the only such changes are generated files, they are set aside instead.
|
||||
incoming = touched_files("HEAD", remote_ref)
|
||||
generated, _why = _generated_to_set_aside(incoming)
|
||||
if not generated or not incoming.intersection(generated):
|
||||
return ReconcileOutcome(status="conflict", detail=result.stderr)
|
||||
return ReconcileOutcome(status="fast-forwarded", pulled_commits=pulled)
|
||||
|
||||
def fast_forward() -> tuple[bool, str, bool]:
|
||||
retry = _run(["git", "merge", "--ff-only", remote_ref])
|
||||
return retry.returncode == 0, retry.stderr, False
|
||||
|
||||
succeeded, detail, regenerated = _around_generated(generated, fast_forward, False)
|
||||
if not succeeded:
|
||||
return ReconcileOutcome(status="conflict", detail=detail)
|
||||
return ReconcileOutcome(status="fast-forwarded", pulled_commits=pulled, regenerated=regenerated)
|
||||
|
||||
if state == "local-ahead":
|
||||
return ReconcileOutcome(status="local-ahead")
|
||||
@@ -830,7 +1121,10 @@ def reconcile(remote: str, branch: str, confirm_rebase: Optional[str] = None) ->
|
||||
return ReconcileOutcome(status="conflict", detail="no common history with remote")
|
||||
|
||||
upstream_commits = oneline_log(base, remote_tip)
|
||||
overlap = sorted(touched_files(base, local_before) & touched_files(base, remote_tip))
|
||||
incoming = touched_files(base, remote_tip)
|
||||
both_sides = touched_files(base, local_before) & incoming
|
||||
overlap = sorted(path for path in both_sides if not is_generated(path))
|
||||
generated_overlap = any(is_generated(path) for path in both_sides)
|
||||
was_reviewed = False
|
||||
token = ""
|
||||
|
||||
@@ -844,13 +1138,34 @@ def reconcile(remote: str, branch: str, confirm_rebase: Optional[str] = None) ->
|
||||
)
|
||||
was_reviewed = True
|
||||
|
||||
rebase_result = _run(["git", "rebase", remote_ref])
|
||||
if rebase_result.returncode != 0:
|
||||
_run(["git", "rebase", "--abort"])
|
||||
return ReconcileOutcome(status="conflict", detail=rebase_result.stderr)
|
||||
tracked, untracked = _dirty_paths()
|
||||
in_the_way = tracked + [path for path in untracked if is_generated(path)]
|
||||
if not in_the_way and not generated_overlap:
|
||||
# Nothing generated on both sides and nothing uncommitted that `git rebase` refuses
|
||||
# over: the plain rebase, exactly as it always ran.
|
||||
rebase_result = _run(["git", "rebase", remote_ref])
|
||||
if rebase_result.returncode != 0:
|
||||
_run(["git", "rebase", "--abort"])
|
||||
return ReconcileOutcome(status="conflict", detail=rebase_result.stderr)
|
||||
return ReconcileOutcome(
|
||||
status="rebased", pulled_commits=upstream_commits,
|
||||
overlap_files=overlap, was_reviewed=was_reviewed, token=token,
|
||||
)
|
||||
|
||||
generated, why = _generated_to_set_aside(incoming)
|
||||
if generated is None:
|
||||
return ReconcileOutcome(
|
||||
status="conflict",
|
||||
detail=f"Nothing was changed: {why}. Commit or set aside that change, then re-run.",
|
||||
)
|
||||
succeeded, detail, regenerated = _around_generated(
|
||||
generated, lambda: _rebase_resolving_generated(remote_ref), generated_overlap,
|
||||
)
|
||||
if not succeeded:
|
||||
return ReconcileOutcome(status="conflict", detail=detail)
|
||||
return ReconcileOutcome(
|
||||
status="rebased", pulled_commits=upstream_commits,
|
||||
overlap_files=overlap, was_reviewed=was_reviewed, token=token,
|
||||
overlap_files=overlap, was_reviewed=was_reviewed, token=token, regenerated=regenerated,
|
||||
)
|
||||
|
||||
|
||||
@@ -920,13 +1235,26 @@ def rebase_review_message(outcome: ReconcileOutcome, remote: str, branch: str, c
|
||||
def _reconcile_summary(outcome: ReconcileOutcome, remote: str, branch: str) -> str:
|
||||
"""One line for the outcomes that do not fail or gate - what to tell the caller, or "" for
|
||||
`conflict` and `needs-review`, which `apply_reconcile` raises on instead."""
|
||||
regenerated = (
|
||||
" Generated files were merged mechanically: kb/log.md keeps both sides' entries, and the "
|
||||
"catalog and kb/provenance.md were regenerated - an uncommitted change now."
|
||||
if outcome.regenerated else ""
|
||||
)
|
||||
if outcome.status == "fast-forwarded":
|
||||
return f"Pulled {len(outcome.pulled_commits)} commit(s) from {remote}/{branch}."
|
||||
return f"Pulled {len(outcome.pulled_commits)} commit(s) from {remote}/{branch}.{regenerated}"
|
||||
if outcome.status == "local-ahead":
|
||||
return f"Local branch is ahead of {remote}/{branch}; nothing to pull."
|
||||
if outcome.status == "rebased":
|
||||
reviewed = " after review" if outcome.was_reviewed else " (no file overlap, automatic)"
|
||||
return f"Rebased onto {len(outcome.pulled_commits)} new commit(s) from {remote}/{branch}{reviewed}."
|
||||
if outcome.was_reviewed:
|
||||
how = " after review"
|
||||
elif outcome.regenerated:
|
||||
how = " (no overlap outside generated files, automatic)"
|
||||
else:
|
||||
how = " (no file overlap, automatic)"
|
||||
return (
|
||||
f"Rebased onto {len(outcome.pulled_commits)} new commit(s) from "
|
||||
f"{remote}/{branch}{how}.{regenerated}"
|
||||
)
|
||||
if outcome.status == "up-to-date":
|
||||
return f"Already up to date with {remote}/{branch}."
|
||||
if outcome.status == "no-remote":
|
||||
@@ -1003,7 +1331,9 @@ def apply_reconcile(outcome: ReconcileOutcome, remote: str, branch: str, command
|
||||
properties=cli_contract.Properties(
|
||||
effect=cli_contract.Effect.WRITE,
|
||||
idempotent=cli_contract.Idempotent.YES,
|
||||
atomic="No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure",
|
||||
atomic="No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure "
|
||||
"with the working tree restored; a success that merged generated files then rewrites "
|
||||
"the catalog",
|
||||
budget=cli_contract.Budget.COUNTED,
|
||||
network=cli_contract.Network.YES,
|
||||
gates=("rebase-review",),
|
||||
@@ -1012,9 +1342,21 @@ def apply_reconcile(outcome: ReconcileOutcome, remote: str, branch: str, command
|
||||
"Fetches `<remote>/<branch>`, then: fast-forwards when only the remote moved; rebases "
|
||||
"the local commits on top when both sides moved but touched disjoint files; exits 42 "
|
||||
"(rebase-review gate) when both sides touched the same file.",
|
||||
"Files `wikitool` generates (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every "
|
||||
"`kb/**/INDEX.md`) never count as an overlap and never reach the gate. Where they are "
|
||||
"what stops git - uncommitted ones in the way of a fast-forward or rebase, or a rebase "
|
||||
"conflict in them - they are merged mechanically: `kb/log.md` becomes the incoming log "
|
||||
"plus the entries only this side appended, and the catalog and `kb/provenance.md` are "
|
||||
"regenerated from the merged pages and left as an uncommitted change. A reconcile git "
|
||||
"can do on its own is unchanged and regenerates nothing.",
|
||||
"Uncommitted changes that no incoming commit touches no longer stop a rebase: they are "
|
||||
"carried through it (`git rebase --autostash`) and come back unchanged.",
|
||||
"Before anything is set aside, the working tree is stored as a commit under "
|
||||
"`refs/wikitool/reconcile-backup`. A successful reconcile drops it; a failed one "
|
||||
"restores the working tree and names the ref.",
|
||||
"A refused call performs no rebase attempt and leaves the branch where it was.",
|
||||
"The `--confirm-rebase` token covers the exact upstream state and the set of files "
|
||||
"touched on both sides; either one moving makes it stale.",
|
||||
"touched on both sides, generated files excluded; either one moving makes it stale.",
|
||||
"Makes no commit, no push, and no forced operation of any kind.",
|
||||
"Three messages for a fetch that fails: no remote of that name, a remote that answers "
|
||||
"but has no such branch yet (a new, empty repository), and a remote that cannot be "
|
||||
@@ -1033,13 +1375,21 @@ def apply_reconcile(outcome: ReconcileOutcome, remote: str, branch: str, command
|
||||
code=0,
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="The automatic rebase hit a real conflict (git failed); it is aborted cleanly",
|
||||
cause="The automatic rebase hit a real conflict outside the generated files (git "
|
||||
"failed), or `kb/log.md` was edited rather than appended to; it is aborted cleanly "
|
||||
"and the working tree restored",
|
||||
reaction="Do not retry and do not force - resolve the conflict manually, then re-run",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="An uncommitted change - or an untracked file - sits on a path the incoming "
|
||||
"commits also change; nothing was changed",
|
||||
reaction="Show the user the file it names. Once that change is committed or moved "
|
||||
"out of the way, re-run once",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
cause="Rebase-review gate: `<remote>/<branch>` moved and both sides changed the "
|
||||
"same file; the output lists the upstream commits, the overlapping files and their "
|
||||
"diff",
|
||||
"same file, generated files aside; the output lists the upstream commits, the "
|
||||
"overlapping files and their diff",
|
||||
reaction=cli_contract.token_gate_reaction("--confirm-rebase"),
|
||||
code=42,
|
||||
),
|
||||
@@ -1116,6 +1466,24 @@ def commit_message(message: str, changed_files: list[str]) -> str:
|
||||
return candidate if _trailers(candidate) == trailers else plain
|
||||
|
||||
|
||||
def commit_regenerated(remote: str, branch: str) -> None:
|
||||
"""Commit the generated files a reconcile just rewrote - and only those, whatever else is
|
||||
staged - with a message of their own."""
|
||||
tracked, untracked = _dirty_paths()
|
||||
paths = sorted(path for path in tracked + untracked if is_generated(path))
|
||||
if not paths:
|
||||
return
|
||||
add_result = _run(["git", "add", "-A", "--", *paths])
|
||||
if add_result.returncode != 0:
|
||||
fail(f"git add failed:\n{add_result.stderr}")
|
||||
message = commit_message(
|
||||
f"chore: regenerate the catalog after reconciling with {remote}/{branch}", paths,
|
||||
)
|
||||
commit_result = _run(["git", "commit", "-m", message, "--", *paths])
|
||||
if commit_result.returncode != 0:
|
||||
fail(f"git commit failed:\n{commit_result.stderr}")
|
||||
|
||||
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="publish",
|
||||
summary="Reconcile with `<remote>/<branch>`, then stage all changes, commit, and push.",
|
||||
@@ -1150,6 +1518,12 @@ def commit_message(message: str, changed_files: list[str]) -> str:
|
||||
"and exits 42 (rebase-review gate) when both sides touched the same file. A refused "
|
||||
"reconcile performs no rebase attempt. The `--confirm-rebase` token covers the exact "
|
||||
"upstream state and the set of files touched on both sides.",
|
||||
"Generated files are never an overlap: where they are all that stops the reconcile, it "
|
||||
"merges them as `sync` does (both sides' log entries kept, the catalog and "
|
||||
"`kb/provenance.md` regenerated) and carries uncommitted changes no incoming commit "
|
||||
"touches through the rebase. The proactive reconcile runs before staging, so this "
|
||||
"publish commits the regenerated files; on the retry after a rejected push they get a "
|
||||
"commit of their own before the second push.",
|
||||
"The push target must be the checked-out branch; this is checked before anything is "
|
||||
"staged. The unborn branch of a fresh `git init -b main` counts as checked out, so the "
|
||||
"first publish of a new instance works; a real detached HEAD is refused.",
|
||||
@@ -1171,8 +1545,8 @@ def commit_message(message: str, changed_files: list[str]) -> str:
|
||||
"refused publish leaves both byte-identical.",
|
||||
"Never counted and never shown for approval, but committed like everything else: "
|
||||
"anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, "
|
||||
"`kb/log.md`, `kb/provenance.md`, every `INDEX.md`). The refusal line accounts for "
|
||||
"both, by reason.",
|
||||
"`kb/log.md`, `kb/provenance.md`, every `INDEX.md` under `kb/`). The refusal line "
|
||||
"accounts for both, by reason.",
|
||||
"The `--confirm` token covers each counted path, the blob id of its contents and the "
|
||||
"publish target: a different file list or edited contents need a new clearance.",
|
||||
"Publish-Remote Gate: when the checkout carries `.wikitool-remotes.json` and the push "
|
||||
@@ -1199,7 +1573,8 @@ def commit_message(message: str, changed_files: list[str]) -> str:
|
||||
failures=(
|
||||
cli_contract.Failure(
|
||||
cause="git failed - `git add`, `git commit`, `git push`, or the reconcile's "
|
||||
"automatic rebase",
|
||||
"automatic rebase (a conflict outside the generated files, an edited `kb/log.md`, "
|
||||
"or an uncommitted change on a path the incoming commits change)",
|
||||
reaction="Do not retry and do not force - report and ask the user. `publish` has "
|
||||
"already made its one retry of a rejected push itself, where a reconcile resolved "
|
||||
"the rejection",
|
||||
@@ -1443,6 +1818,10 @@ def publish_command(
|
||||
retry_summary = apply_reconcile(retry_outcome, remote, branch, "publish")
|
||||
if retry_summary:
|
||||
typer.echo(retry_summary)
|
||||
if retry_outcome.regenerated:
|
||||
# Staging is long past here, so the regenerated catalog would otherwise
|
||||
# stay behind and the push would carry the stale one it was merged from.
|
||||
commit_regenerated(remote, branch)
|
||||
push_result = _run(["git", "push", remote, branch])
|
||||
elif retry_outcome.status in ("conflict", "needs-review"):
|
||||
apply_reconcile(retry_outcome, remote, branch, "publish") # raises
|
||||
|
||||
@@ -210,6 +210,21 @@ def stale_shards(kb_dir: Path, plan: dict[Path, str]) -> list[Path]:
|
||||
return sorted(p for p in kb_dir.rglob(GENERATED_INDEX) if p not in plan)
|
||||
|
||||
|
||||
def write_index(kb_dir: Path) -> tuple[dict[Path, str], list[Path]]:
|
||||
"""Write every catalog file and remove every stale shard; returns (plan, removed).
|
||||
|
||||
The writing half of `index rebuild`, as a function so `reconcile` can regenerate the
|
||||
catalog in-process after it has merged two sides' generated files."""
|
||||
plan = plan_index(kb_dir)
|
||||
stale = stale_shards(kb_dir, plan)
|
||||
for path, content in plan.items():
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8", newline="\n")
|
||||
for path in stale:
|
||||
path.unlink()
|
||||
return plan, stale
|
||||
|
||||
|
||||
def build_index(kb_dir: Path) -> str:
|
||||
"""The root map. Kept as a named function because callers (and tests) ask
|
||||
for "the index" meaning the entry point, not the whole plan."""
|
||||
@@ -276,24 +291,17 @@ def index_rebuild(
|
||||
"See `wikitool lint`'s Nested Pages finding."
|
||||
)
|
||||
|
||||
plan = plan_index(config.KB_DIR)
|
||||
stale = stale_shards(config.KB_DIR, plan)
|
||||
|
||||
if dry_run:
|
||||
plan = plan_index(config.KB_DIR)
|
||||
for path in sorted(plan):
|
||||
typer.echo(f"--- {rel_path(path)}")
|
||||
# nl=False: the content already ends in a newline.
|
||||
typer.echo(plan[path], nl=False)
|
||||
for path in stale:
|
||||
for path in stale_shards(config.KB_DIR, plan):
|
||||
typer.echo(f"--- would remove stale shard: {rel_path(path)}")
|
||||
return
|
||||
|
||||
for path, content in plan.items():
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8", newline="\n")
|
||||
for path in stale:
|
||||
path.unlink()
|
||||
|
||||
plan, stale = write_index(config.KB_DIR)
|
||||
shards = len(plan) - 1
|
||||
removed = f", removed {len(stale)} stale" if stale else ""
|
||||
success(f"Rebuilt {rel_path(config.INDEX_FILE)} and {shards} shard(s){removed}")
|
||||
@@ -231,6 +231,14 @@ def build_provenance_index(kb_dir: Path, raw_dir: Path) -> str:
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_provenance_index(kb_dir: Path, raw_dir: Path) -> Path:
|
||||
"""Write `kb/provenance.md` from scratch and return its path - the writing half of
|
||||
`sources rebuild-index`, callable in-process by `reconcile`."""
|
||||
provenance_file = kb_dir / "provenance.md"
|
||||
provenance_file.write_text(build_provenance_index(kb_dir, raw_dir), encoding="utf-8", newline="\n")
|
||||
return provenance_file
|
||||
|
||||
|
||||
@app.command("rebuild-index")
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="sources rebuild-index",
|
||||
@@ -267,12 +275,10 @@ def rebuild_index(
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Print the result instead of writing kb/provenance.md"),
|
||||
):
|
||||
"""Regenerate the `kb/provenance.md` reverse index."""
|
||||
content = build_provenance_index(config.KB_DIR, config.RAW_DIR)
|
||||
provenance_file = config.KB_DIR / "provenance.md"
|
||||
if dry_run:
|
||||
# nl=False so the preview is byte-identical to the file that would be
|
||||
# written; see the same note in index_build.py.
|
||||
typer.echo(content, nl=False)
|
||||
typer.echo(build_provenance_index(config.KB_DIR, config.RAW_DIR), nl=False)
|
||||
return
|
||||
provenance_file.write_text(content, encoding="utf-8", newline="\n")
|
||||
provenance_file = write_provenance_index(config.KB_DIR, config.RAW_DIR)
|
||||
success(f"Rebuilt {rel_path(provenance_file)}")
|
||||
@@ -1,4 +1,5 @@
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
@@ -367,8 +368,8 @@ def _publish(**overrides):
|
||||
"""Call `publish_command` directly. Every parameter must be given a real
|
||||
value: bypassing Typer's CLI parsing means an omitted argument keeps its
|
||||
`typer.Option(...)` sentinel instead of the value it wraps."""
|
||||
kwargs = dict(message="change", push=True, confirm=None, yes=False, threshold=10,
|
||||
remote="origin", branch="main", path=None)
|
||||
kwargs = dict(message="change", push=True, confirm=None, confirm_rebase=None, yes=False,
|
||||
threshold=10, remote="origin", branch="main", path=None)
|
||||
kwargs.update(overrides)
|
||||
publish_command(**kwargs)
|
||||
|
||||
@@ -515,6 +516,14 @@ def test_generated_files_are_recognised_wherever_they_sit():
|
||||
assert not is_generated("kb/concepts/protocols/Modbus.md")
|
||||
|
||||
|
||||
def test_an_index_md_outside_kb_is_not_generated():
|
||||
"""A captured repository under raw/ may carry its own INDEX.md. It is source material, and
|
||||
`reconcile` discards and regenerates whatever this predicate calls generated."""
|
||||
assert not is_generated("raw/repos/project/INDEX.md")
|
||||
assert not is_generated("docs/INDEX.md")
|
||||
assert counted_files(["raw/repos/project/INDEX.md"]) == ["raw/repos/project/INDEX.md"]
|
||||
|
||||
|
||||
def test_paths_land_in_the_group_a_reviewer_expects():
|
||||
assert group_of("kb/concepts/X.md")[0] == "Published knowledge"
|
||||
assert group_of("AGENTS.md")[0] == "Agent control plane"
|
||||
@@ -1497,3 +1506,395 @@ def test_a_mass_update_token_does_not_depend_on_the_message(repo):
|
||||
assert _git(
|
||||
repo, "log", "-1", "--format=%(trailers:key=Co-Authored-By,valueonly)"
|
||||
).stdout.strip() == "A <a@x>"
|
||||
|
||||
|
||||
# --- Reconciling around generated files: two ingests on two checkouts -----------------------
|
||||
#
|
||||
# Every ingest touches kb/index.md and kb/log.md. These tests build a real catalog with the
|
||||
# shipped rebuild functions, so "the catalog is current" is checked against what `index
|
||||
# rebuild`/`sources rebuild-index` actually write, not against a hand-made stand-in.
|
||||
|
||||
from contextlib import contextmanager
|
||||
|
||||
from chemenu.commands.log_append import format_log_entry, parse_log_entries
|
||||
from chemenu.frontmatter_io import write_page
|
||||
from chemenu.tests.conftest import use_shipped_type_specs
|
||||
|
||||
GENERATED_FILES = ("kb/index.md", "kb/log.md", "kb/provenance.md", "kb/concepts/INDEX.md")
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _at(root):
|
||||
"""Point `config.ROOT` at another checkout for the duration - the writer clone builds its
|
||||
catalog with the same functions, against its own tree."""
|
||||
previous = config.ROOT
|
||||
config.ROOT = root
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
config.ROOT = previous
|
||||
|
||||
|
||||
def _write_concept(root, title, text="A concept.\n"):
|
||||
write_page(
|
||||
root / f"kb/concepts/protocols/{title}.md",
|
||||
{"type": "types/concept.md", "concept_type": "protocol", "tags": [],
|
||||
"created": "2026-10-06", "modified": "2026-10-06", "related": [], "sources": []},
|
||||
f"\n# {title}\n\n## Definition\n\n{text}",
|
||||
)
|
||||
|
||||
|
||||
def _ingest(root, title, text="A concept.\n"):
|
||||
"""What an ingest leaves in the working tree: a page, the regenerated catalog and
|
||||
provenance, and one appended log entry."""
|
||||
_write_concept(root, title, text)
|
||||
with _at(root):
|
||||
git_publish.regenerate_generated()
|
||||
with (root / "kb/log.md").open("a", encoding="utf-8", newline="\n") as log:
|
||||
log.write("\n" + format_log_entry("ingest", title, today="2026-10-06"))
|
||||
|
||||
|
||||
def _commit_all(root, message):
|
||||
_git(root, "add", "-A")
|
||||
_git(root, "commit", "-m", message)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wiki_repo(repo, monkeypatch):
|
||||
"""`repo` with a real, pushed kb/: one collection, one page, its catalog and provenance
|
||||
built by the shipped functions, and a log."""
|
||||
use_shipped_type_specs(monkeypatch)
|
||||
(repo / "kb/concepts").mkdir(parents=True)
|
||||
(repo / "kb/concepts/COLLECTION.md").write_text(
|
||||
"---\nprofile: concepts\nrequired_by_stack: false\noutbound:\n any: [see-also]\n---\n\n"
|
||||
"# kb/concepts/ - Collection Contract\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(repo / "raw/notes").mkdir(parents=True)
|
||||
(repo / "raw/notes/a.md").write_text("raw\n", encoding="utf-8")
|
||||
(repo / "kb/log.md").write_text("# Wiki Log\n", encoding="utf-8")
|
||||
_ingest(repo, "Base")
|
||||
_commit_all(repo, "wiki")
|
||||
_git(repo, "push", "origin", "main")
|
||||
return repo
|
||||
|
||||
|
||||
def _writer_ingest(writer, title):
|
||||
_ingest(writer, title)
|
||||
_commit_all(writer, f"writer ingest: {title}")
|
||||
_git(writer, "push", "origin", "main")
|
||||
|
||||
|
||||
def _tree(root):
|
||||
"""Every file of the working tree outside .git, as bytes."""
|
||||
return {
|
||||
path.relative_to(root).as_posix(): path.read_bytes()
|
||||
for path in root.rglob("*")
|
||||
if path.is_file() and ".git" not in path.relative_to(root).parts
|
||||
}
|
||||
|
||||
|
||||
def _non_generated(tree):
|
||||
return {path: data for path, data in tree.items() if not is_generated(path)}
|
||||
|
||||
|
||||
def _assert_kept(before, root):
|
||||
"""Every non-generated file `before` holds is still there, byte for byte. Files the
|
||||
incoming commits add are new, not changed."""
|
||||
after = _tree(root)
|
||||
assert {path: after.get(path) for path in _non_generated(before)} == _non_generated(before)
|
||||
|
||||
|
||||
def _check_tree(root):
|
||||
"""A fresh clone of the remote tip, as a place to compare its generated files against a
|
||||
rebuild of that same tip."""
|
||||
check = root.parent / "check"
|
||||
shutil.rmtree(check, ignore_errors=True)
|
||||
subprocess.run(["git", "clone", str(_remote_for(root)), str(check)], check=True,
|
||||
capture_output=True)
|
||||
return check
|
||||
|
||||
|
||||
def _assert_generated_current(root):
|
||||
"""The generated files at `root` are byte for byte what a rebuild of `root` writes today."""
|
||||
before = {path: (root / path).read_bytes() for path in GENERATED_FILES if path != "kb/log.md"}
|
||||
with _at(root):
|
||||
git_publish.regenerate_generated()
|
||||
after = {path: (root / path).read_bytes() for path in before}
|
||||
assert before == after
|
||||
|
||||
|
||||
def _logged_titles(root):
|
||||
return [title for _date, _op, title in parse_log_entries(
|
||||
(root / "kb/log.md").read_text(encoding="utf-8"))]
|
||||
|
||||
|
||||
def test_case_1_an_uncommitted_ingest_publishes_over_an_ingest_that_landed_meanwhile(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
|
||||
_publish(message="ingest: Local") # must not raise - no gate, no failure
|
||||
|
||||
check = _check_tree(wiki_repo)
|
||||
assert (check / "kb/concepts/protocols/Writer.md").is_file()
|
||||
assert (check / "kb/concepts/protocols/Local.md").is_file()
|
||||
assert _logged_titles(check) == ["Base", "Writer", "Local"]
|
||||
_assert_generated_current(check)
|
||||
assert _status(wiki_repo) == ""
|
||||
|
||||
|
||||
def test_case_1_on_sync_leaves_the_regenerated_catalog_uncommitted(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
page = (wiki_repo / "kb/concepts/protocols/Local.md").read_bytes()
|
||||
|
||||
outcome = reconcile("origin", "main", None)
|
||||
|
||||
assert (outcome.status, outcome.regenerated) == ("fast-forwarded", True)
|
||||
assert (wiki_repo / "kb/concepts/protocols/Local.md").read_bytes() == page
|
||||
assert _logged_titles(wiki_repo) == ["Base", "Writer", "Local"]
|
||||
_assert_generated_current(wiki_repo)
|
||||
assert "kb/index.md" in _status(wiki_repo)
|
||||
assert subprocess.run(["git", "show-ref", "--verify", "-q", git_publish.BACKUP_REF],
|
||||
cwd=wiki_repo).returncode != 0 # dropped on success
|
||||
|
||||
|
||||
def test_case_2_two_committed_ingests_rebase_without_the_review_gate(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
_commit_all(wiki_repo, "local ingest, push failed") # the stranded commit
|
||||
|
||||
_publish(message="retry") # must not raise
|
||||
|
||||
check = _check_tree(wiki_repo)
|
||||
assert _logged_titles(check) == ["Base", "Writer", "Local"]
|
||||
_assert_generated_current(check)
|
||||
assert _status(wiki_repo) == ""
|
||||
|
||||
|
||||
def test_case_2_on_sync_ends_with_exit_0_and_both_sides_in_the_log(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
_commit_all(wiki_repo, "local ingest")
|
||||
|
||||
_sync() # must not raise
|
||||
|
||||
assert _logged_titles(wiki_repo) == ["Base", "Writer", "Local"]
|
||||
_assert_generated_current(wiki_repo)
|
||||
assert "writer ingest: Writer" in _git(wiki_repo, "log", "--oneline").stdout
|
||||
|
||||
|
||||
def test_case_2_still_gates_on_an_overlapping_page_and_names_no_generated_file(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_ingest(writer, "Base", text="Writer's definition.\n")
|
||||
_commit_all(writer, "writer edits Base")
|
||||
_git(writer, "push", "origin", "main")
|
||||
_ingest(wiki_repo, "Base", text="Local definition.\n")
|
||||
_commit_all(wiki_repo, "local edits Base")
|
||||
|
||||
outcome = reconcile("origin", "main", None)
|
||||
|
||||
assert outcome.status == "needs-review"
|
||||
assert outcome.overlap_files == ["kb/concepts/protocols/Base.md"]
|
||||
assert "kb/log.md" not in outcome.overlap_diff and "kb/index.md" not in outcome.overlap_diff
|
||||
|
||||
|
||||
def test_case_2_in_the_retry_path_commits_the_regenerated_catalog_before_the_push(
|
||||
wiki_repo, monkeypatch
|
||||
):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
raced = {"done": False}
|
||||
real_run = git_publish._run
|
||||
|
||||
def racy_run(args, **kwargs):
|
||||
if args[:2] == ["git", "push"] and not raced["done"]:
|
||||
raced["done"] = True
|
||||
_writer_ingest(writer, "Writer")
|
||||
return real_run(args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(git_publish, "_run", racy_run)
|
||||
_ingest(wiki_repo, "Local")
|
||||
|
||||
_publish(message="ingest: Local") # must not raise
|
||||
|
||||
assert raced["done"]
|
||||
check = _check_tree(wiki_repo)
|
||||
assert _logged_titles(check) == ["Base", "Writer", "Local"]
|
||||
_assert_generated_current(check)
|
||||
assert "chore: regenerate the catalog" in _git(check, "log", "-1", "--format=%s").stdout
|
||||
assert _status(wiki_repo) == ""
|
||||
|
||||
|
||||
def _stranded_ingest_plus_uncommitted_ingest(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Stranded")
|
||||
_commit_all(wiki_repo, "stranded ingest")
|
||||
_ingest(wiki_repo, "Uncommitted")
|
||||
(wiki_repo / "README.md").write_text("init\nedited, not committed\n", encoding="utf-8")
|
||||
|
||||
|
||||
def test_case_3_publish_carries_uncommitted_work_over_a_stranded_commit(wiki_repo):
|
||||
_stranded_ingest_plus_uncommitted_ingest(wiki_repo)
|
||||
|
||||
_publish(message="ingest: Uncommitted") # must not raise
|
||||
|
||||
check = _check_tree(wiki_repo)
|
||||
assert (check / "kb/concepts/protocols/Uncommitted.md").is_file()
|
||||
assert (check / "README.md").read_text(encoding="utf-8") == "init\nedited, not committed\n"
|
||||
assert _logged_titles(check) == ["Base", "Writer", "Stranded", "Uncommitted"]
|
||||
_assert_generated_current(check)
|
||||
|
||||
|
||||
def test_case_3_sync_leaves_the_uncommitted_pages_exactly_as_they_were(wiki_repo):
|
||||
_stranded_ingest_plus_uncommitted_ingest(wiki_repo)
|
||||
before = _tree(wiki_repo)
|
||||
|
||||
_sync() # must not raise
|
||||
|
||||
_assert_kept(before, wiki_repo)
|
||||
status = _status(wiki_repo)
|
||||
assert "README.md" in status and "Uncommitted.md" in status # still uncommitted
|
||||
assert _logged_titles(wiki_repo) == ["Base", "Writer", "Stranded", "Uncommitted"]
|
||||
_assert_generated_current(wiki_repo)
|
||||
|
||||
|
||||
def test_case_3_without_generated_files_rebases_and_rebuilds_nothing(repo, monkeypatch):
|
||||
"""A stranded commit and an unrelated uncommitted edit: `git rebase` used to refuse over
|
||||
the edit. Nothing generated is involved, so nothing is regenerated."""
|
||||
monkeypatch.setattr(git_publish, "regenerate_generated",
|
||||
lambda *a, **k: pytest.fail("nothing generated was involved"))
|
||||
writer = _clone_writer(repo)
|
||||
_push_from_writer(writer, "from-writer.md", "writer\n")
|
||||
(repo / "local.md").write_text("local\n", encoding="utf-8")
|
||||
_commit_all(repo, "stranded")
|
||||
(repo / "README.md").write_text("uncommitted\n", encoding="utf-8")
|
||||
|
||||
outcome = reconcile("origin", "main", None)
|
||||
|
||||
assert (outcome.status, outcome.regenerated) == ("rebased", False)
|
||||
assert (repo / "README.md").read_text(encoding="utf-8") == "uncommitted\n"
|
||||
assert (repo / "from-writer.md").is_file()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("how", ["raises", "fails"])
|
||||
def test_uncommitted_work_survives_an_abort_in_the_middle_of_the_rebase(wiki_repo, monkeypatch, how):
|
||||
"""The invariant: whatever happens mid-rebase, every non-generated file in the working tree
|
||||
is byte-identical afterwards, and the branch has not moved. A failure also restores the
|
||||
generated files and names the backup ref."""
|
||||
_stranded_ingest_plus_uncommitted_ingest(wiki_repo)
|
||||
before_tree = _tree(wiki_repo)
|
||||
before_head = _git(wiki_repo, "rev-parse", "HEAD").stdout
|
||||
|
||||
def forced(*_args):
|
||||
if how == "raises":
|
||||
raise RuntimeError("forced abort mid-rebase")
|
||||
return "forced failure"
|
||||
|
||||
monkeypatch.setattr(git_publish, "_resolve_generated_conflicts", forced)
|
||||
|
||||
if how == "raises":
|
||||
with pytest.raises(RuntimeError):
|
||||
reconcile("origin", "main", None)
|
||||
else:
|
||||
outcome = reconcile("origin", "main", None)
|
||||
assert outcome.status == "conflict"
|
||||
assert git_publish.BACKUP_REF in outcome.detail
|
||||
assert _tree(wiki_repo) == before_tree
|
||||
|
||||
assert _non_generated(_tree(wiki_repo)) == _non_generated(before_tree)
|
||||
assert _git(wiki_repo, "rev-parse", "HEAD").stdout == before_head
|
||||
assert not git_publish._rebase_in_progress()
|
||||
assert _git(wiki_repo, "stash", "list").stdout == ""
|
||||
# The backup holds the working tree as it stood, uncommitted page included.
|
||||
assert "Uncommitted" in _git(
|
||||
wiki_repo, "show", f"{git_publish.BACKUP_REF}:kb/concepts/protocols/Uncommitted.md"
|
||||
).stdout
|
||||
|
||||
|
||||
def _assert_untouched(repo, run):
|
||||
tree, head = _tree(repo), _git(repo, "rev-parse", "HEAD").stdout
|
||||
with pytest.raises(typer.Exit) as excinfo:
|
||||
run()
|
||||
assert excinfo.value.exit_code == 1
|
||||
assert _tree(repo) == tree
|
||||
assert _git(repo, "rev-parse", "HEAD").stdout == head
|
||||
|
||||
|
||||
def test_a_hand_edited_log_is_left_to_git_as_before(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
log = wiki_repo / "kb/log.md"
|
||||
log.write_text(log.read_text(encoding="utf-8").replace("# Wiki Log", "# Edited Log"),
|
||||
encoding="utf-8")
|
||||
|
||||
_assert_untouched(wiki_repo, _sync)
|
||||
|
||||
|
||||
def test_a_log_rewritten_in_a_local_commit_aborts_the_rebase_as_before(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Local")
|
||||
log = wiki_repo / "kb/log.md"
|
||||
log.write_text(log.read_text(encoding="utf-8").replace("# Wiki Log", "# Edited Log"),
|
||||
encoding="utf-8")
|
||||
_commit_all(wiki_repo, "local ingest with a rewritten log")
|
||||
|
||||
_assert_untouched(wiki_repo, _sync)
|
||||
|
||||
|
||||
def test_an_uncommitted_change_to_a_file_the_incoming_commits_change_stops_as_before(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_ingest(writer, "Base", text="Writer's definition.\n")
|
||||
_commit_all(writer, "writer edits Base")
|
||||
_git(writer, "push", "origin", "main")
|
||||
_ingest(wiki_repo, "Stranded")
|
||||
_commit_all(wiki_repo, "stranded")
|
||||
_write_concept(wiki_repo, "Base", "Uncommitted local definition.\n")
|
||||
|
||||
_assert_untouched(wiki_repo, _sync)
|
||||
|
||||
|
||||
def test_an_untracked_file_on_an_incoming_path_stops_as_before(wiki_repo):
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
_ingest(wiki_repo, "Stranded")
|
||||
_commit_all(wiki_repo, "stranded")
|
||||
_write_concept(wiki_repo, "Writer", "Same title, written here and never committed.\n")
|
||||
|
||||
_assert_untouched(wiki_repo, _sync)
|
||||
|
||||
|
||||
def test_a_plain_fast_forward_regenerates_nothing(wiki_repo, monkeypatch):
|
||||
"""None of the three cases: the result is what it always was, and in particular no rebuild
|
||||
runs - it would stamp today's date into kb/index.md and leave every sync dirty."""
|
||||
writer = _clone_writer(wiki_repo)
|
||||
_writer_ingest(writer, "Writer")
|
||||
monkeypatch.setattr(git_publish, "regenerate_generated",
|
||||
lambda *a, **k: pytest.fail("no rebuild without a collision"))
|
||||
|
||||
outcome = reconcile("origin", "main", None)
|
||||
|
||||
assert (outcome.status, outcome.regenerated) == ("fast-forwarded", False)
|
||||
assert _status(wiki_repo) == ""
|
||||
|
||||
|
||||
def test_an_index_md_under_raw_still_goes_to_review_when_both_sides_change_it(repo):
|
||||
(repo / "raw/repos/project").mkdir(parents=True)
|
||||
(repo / "raw/repos/project/INDEX.md").write_text("original\n", encoding="utf-8")
|
||||
_commit_all(repo, "capture")
|
||||
_git(repo, "push", "origin", "main")
|
||||
writer = _clone_writer(repo)
|
||||
_push_from_writer(writer, "raw/repos/project/INDEX.md", "writer\n")
|
||||
(repo / "raw/repos/project/INDEX.md").write_text("local\n", encoding="utf-8")
|
||||
_commit_all(repo, "local")
|
||||
|
||||
outcome = reconcile("origin", "main", None)
|
||||
|
||||
assert outcome.status == "needs-review"
|
||||
assert outcome.overlap_files == ["raw/repos/project/INDEX.md"]
|
||||
Reference in new issue
Block a user