diff --git a/CHANGES.md b/CHANGES.md index 5e5cdb5..1d936fd 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.1.0-beta.11 - 2026-10-15 - wiki-query liest den Wortlaut geführt in der Rohdatei nach: über Seite, Source-Seite und Gliederung der Ableitung bis zum Zeilenbereich, nie freies Suchen in raw/; kb/CONTRACT.md präzisiert das Qualitätsziel auf die Substanz (#227) +## 10.1.0-beta.12 - 2026-10-15 - export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221) **Author:** Torben Nehmer @@ -26,6 +26,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. - raw derive PDF: Kopf nennt die Bildseiten (images) und die nicht zugeordneten Seiten (unmatched); neue Anweisung read-page-images für den gezielten Blick auf diese Seiten - Länge als dritte Größenachse beim Ingest: raw derive und raw fetch melden einen Text über der Längenschwelle mit einer CHECK-Zeile, raw pending markiert eine lange Textdatei; ingest-large-tree liest ihn in Teilen nach der Gliederung, mit einer Faktenliste je Teil (#226) - wiki-query liest den Wortlaut geführt in der Rohdatei nach: über Seite, Source-Seite und Gliederung der Ableitung bis zum Zeilenbereich, nie freies Suchen in raw/; kb/CONTRACT.md präzisiert das Qualitätsziel auf die Substanz (#227) +- export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221) **Low impact** - instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie diff --git a/VERSION b/VERSION index 42a0d36..ff7064d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.1.0-beta.11 +10.1.0-beta.12 diff --git a/docs/how-to/export-to-a-synced-folder.md b/docs/how-to/export-to-a-synced-folder.md index 92edf34..625612b 100644 --- a/docs/how-to/export-to-a-synced-folder.md +++ b/docs/how-to/export-to-a-synced-folder.md @@ -76,7 +76,7 @@ look like: [configuration reference](../reference/configuration.md#export-to-a-f what is wrong with the section. The export reads the committed pages. A `kb/` or `types/` with uncommitted changes is refused, so -every file names the commit its pages come from. Publish first. +every file names the commit its page comes from. Publish first. ## What a run does to the folder @@ -93,6 +93,11 @@ what the last run wrote. - **An empty selection is an error.** A run that finds no page does not delete the earlier export. - **An unchanged export stays untouched.** A run that has nothing to change writes nothing, not even a modification time - the sync client has nothing to upload. +- **A commit rewrites the files of the pages it touched.** Each file names the last commit of its own + page, so an ingest that changed three pages rewrites three files, however large the selection. + A file of a page the commit did not touch changes only when its text does: a link to a page that + has just entered the selection becomes a relative link, one to a page that left it plain text. + The first run after updating to a version that brought this rewrites every file once. - **A growing selection needs no confirmation.** New pages are reported as new; read the list before the sync client uploads them. diff --git a/docs/reference/commands/exports.md b/docs/reference/commands/exports.md index ffbe763..9f4326c 100644 --- a/docs/reference/commands/exports.md +++ b/docs/reference/commands/exports.md @@ -110,7 +110,7 @@ tools/wikitool export run work-notes **Behaviour** - An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names. -- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance= commit=`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes. +- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance= commit=`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected. - Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed. - The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all. - Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 70f9938..ed264ec 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -704,7 +704,7 @@ runs. `path` must not lie in or contain the instance, another target, an `intake `backup.target`. Every Markdown file starts with a frontmatter block whose first key is `wikitool_export`, carrying -the kind, the instance and the commit of the newest selected page. `frontmatter` decides what follows: +the kind, the instance and the newest commit that touched that page. `frontmatter` decides what follows: `minimal` nothing, `summary` the title, type, summary and modified date, `full` the page's whole frontmatter. Wikilinks between exported pages become relative Markdown links (`links: relative`) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 7797f4b..258c088 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -2456,7 +2456,7 @@ Write the derived, read-only reading copy of `kb/` into the folders the `export` **NOTES** - An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names. -- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance= commit=`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes. +- Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance= commit=`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected. - Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed. - The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all. - Refuses a target, before writing anything into it: a non-empty folder with no manifest (it is somebody's, not an export), a manifest file that was changed in the folder since the last run (naming it), a foreign file where an exported one belongs, an empty selection (it would delete the earlier export), an unknown field in a predicate. A missing folder is created. diff --git a/tools/chemenu/commands/export_cmd.py b/tools/chemenu/commands/export_cmd.py index e7bc7ac..0faab77 100644 --- a/tools/chemenu/commands/export_cmd.py +++ b/tools/chemenu/commands/export_cmd.py @@ -460,13 +460,16 @@ def export_guidelines_command( "the instance). Nothing is exported that no selection names.", "Every Markdown file starts with a frontmatter block whose `wikitool_export` line says " "`kind=pages instance= commit=`, followed by `title`, `type`, `summary`, `modified` where the " + "touching this page>`, followed by `title`, `type`, `summary`, `modified` where the " "page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, " "`full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A " "wikilink to an exported page becomes a relative Markdown link (`links: relative`, the " "default) or plain text (`links: text`); a link to a page that is not exported is always " "plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: " - "the same pages give the same bytes.", + "the same pages give the same bytes, and a commit that touches one page rewrites that " + "page's file alone - plus any file whose links change because a page entered or left the " + "selection. The manifest's `commit` is the newest commit touching any selected page; all " + "of them come from one `git log`, however many pages are selected.", "Handler `command`: the Markdown is rendered into a staging directory, the program of " "`export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives " "the Markdown directory and an empty output directory as its last two arguments. On exit " diff --git a/tools/chemenu/guideline_export.py b/tools/chemenu/guideline_export.py index e789d81..9cff338 100644 --- a/tools/chemenu/guideline_export.py +++ b/tools/chemenu/guideline_export.py @@ -122,12 +122,12 @@ def file_is_export(path: Path) -> bool: # --- the instance it comes from ----------------------------------------------- -def _git(args: list[str], root: Path) -> Optional[str]: +def _git(args: list[str], root: Path, stdin: Optional[str] = None) -> Optional[str]: """`git ` in the instance checkout: stdout on success, None on any failure. Only local reads - nothing here reaches a remote.""" try: result = subprocess.run( - [toolpaths.git(), *args], cwd=root, capture_output=True, text=True, + [toolpaths.git(), *args], cwd=root, input=stdin, capture_output=True, text=True, encoding="utf-8", timeout=30, check=False, ) except (OSError, subprocess.SubprocessError): @@ -227,17 +227,51 @@ def check_clean(root: Optional[Path] = None) -> None: ) +_NOT_COMMITTED = "None of the selected pages is in a commit yet - publish them first." + + +def _repo_paths(pages: list[Page], root: Path) -> list[str]: + return [Path(p.path).resolve().relative_to(Path(root).resolve()).as_posix() for p in pages] + + def last_commit(pages: list[Page], root: Optional[Path] = None) -> str: """The newest commit touching one of `pages`' files - not `HEAD`, so an instance commit that touched no guideline leaves the output unchanged.""" root = root or config.ROOT - paths = [Path(p.path).resolve().relative_to(Path(root).resolve()).as_posix() for p in pages] - sha = (_git(["log", "-1", "--format=%H", "--", *paths], root) or "").strip() + sha = (_git(["log", "-1", "--format=%H", "--", *_repo_paths(pages, root)], root) or "").strip() if not sha: - raise ValidationError("None of the selected pages is in a commit yet - publish them first.") + raise ValidationError(_NOT_COMMITTED) return sha +def page_commits(pages: list[Page], root: Optional[Path] = None) -> tuple[str, dict[Path, str]]: + """`last_commit(pages)`, and for each page's `path` the newest commit + touching that one file - from a single `git log`, however many pages there + are, since every process start costs on Windows. The paths go in on stdin, + so a whole corpus does not run into the command-line limit, and literally, + so a `[` in a title is not a glob. Without `--follow`, a page `rename` moved + gets the commit that moved it. A page no listed commit names - one that + only a merge's own resolution touched - falls back to the newest commit.""" + root = root or config.ROOT + paths = _repo_paths(pages, root) + out = _git(["--literal-pathspecs", "log", "--stdin", "-z", "--format=commit %H", "--name-only"], root, + stdin="--\n" + "".join(f"{path}\n" for path in paths)) or "" + newest = "" + current = "" + first: dict[str, str] = {} + # `-z` ends each header and each name with NUL; a commit's first name + # follows its header after a newline. + for token in out.split("\0"): + if token.startswith("commit "): + current = token[len("commit "):] + newest = newest or current + elif token.strip("\n"): + first.setdefault(token.lstrip("\n"), current) + if not newest: + raise ValidationError(_NOT_COMMITTED) + return newest, {page.path: first.get(path, newest) for page, path in zip(pages, paths)} + + # --- rendering ---------------------------------------------------------------- diff --git a/tools/chemenu/page_export.py b/tools/chemenu/page_export.py index 0e5ad4e..db58811 100644 --- a/tools/chemenu/page_export.py +++ b/tools/chemenu/page_export.py @@ -10,9 +10,10 @@ An export is never a working copy: it is regenerated from `kb/` and what a person changes in the folder is overwritten or refused, never merged back. The properties that carry that, each with a test: -- **Deterministic.** The output depends on the selected pages and the commit - that last touched one of them - no timestamp, no `HEAD` - so a run on an - unchanged corpus writes nothing, not even a file's modification time. +- **Deterministic.** A file depends on its page, the selection it links into + and the commit that last touched that page - no timestamp, no `HEAD` - so a + run on an unchanged corpus writes nothing, not even a file's modification + time, and a commit to one page rewrites that page's file alone. - **Only its own files are touched.** What an earlier run wrote is recorded in a manifest in the target (`.wikitool-export.json`, with a digest per file). A run removes only manifest files that are no longer exported, leaves every @@ -290,8 +291,10 @@ def render_page(page: Page, here: str, destinations: dict[str, Optional[str]], t return text.encode("utf-8") -def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, commit: str) -> dict[str, bytes]: - """The files of a target for `pages`, by path below the target. A page +def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, + commits: dict[Path, str]) -> dict[str, bytes]: + """The files of a target for `pages`, by path below the target, each + marked with its own page's commit from `commits`. A page that cannot be written on every platform, or two that would share a file name in a case-insensitive folder, is an `ExportError`.""" paths = {id(page): file_path(page, kb_dir) for page in pages} @@ -312,7 +315,7 @@ def render(pages: list[Page], target: Target, kb_dir: Path, instance: str, commi files: dict[str, bytes] = {} for page in pages: rel = paths[id(page)] - files[rel] = render_page(page, rel, by_title, target, instance, commit) + files[rel] = render_page(page, rel, by_title, target, instance, commits[page.path]) return files @@ -563,8 +566,8 @@ def run(targets: Sequence[Target], root: "Path | str | None" = None) -> list[Tar if not selected: raise ExportError("no page matches the selection - an empty export would delete the " "earlier one, so nothing was changed") - result.commit = guideline_export.last_commit(selected, root) - files = render(selected, target, kb_dir, instance, result.commit) + result.commit, commits = guideline_export.page_commits(selected, root) + files = render(selected, target, kb_dir, instance, commits) result.titles = {file_path(p, kb_dir): p.title for p in selected} if target.handler == config_shapes.EXPORT_HANDLER_COMMAND: files = convert(target, files, result) diff --git a/tools/chemenu/tests/test_export_run.py b/tools/chemenu/tests/test_export_run.py index 3aff488..d950483 100644 --- a/tools/chemenu/tests/test_export_run.py +++ b/tools/chemenu/tests/test_export_run.py @@ -304,6 +304,100 @@ def test_an_instance_commit_that_touches_no_selected_page_changes_nothing(instan assert _mtimes(folder) == stamps +def _marker_commit(path: Path) -> str: + line = path.read_text(encoding="utf-8").split("\n")[1] + return line.rsplit("commit=", 1)[1].rstrip('"') + + +def test_a_commit_to_one_page_rewrites_that_file_alone(instance): + _configure(instance, pages={"include": [["tags=docs"], ["tags=private"]]}) + assert _run().exit_code == 0 + folder = _out(instance, "pages") + before, stamps = _snapshot(folder), _mtimes(folder) + _page(instance, "concepts/Alpha.md", ALPHA + "\nMore.\n") + commit = _commit(instance, "edit alpha") + result = _run() + assert result.exit_code == 0, result.output + out = _flat(result.output) + assert "changed (1): Alpha" in out and "unchanged: 2" in out and "new (" not in out + after, after_stamps = _snapshot(folder), _mtimes(folder) + for rel in ("concepts/protocols/Beta.md", "concepts/Gamma.md"): + assert after[rel] == before[rel] and after_stamps[rel] == stamps[rel] + assert _marker_commit(folder / "concepts" / "Alpha.md") == commit + + +def test_each_file_carries_its_own_pages_commit_and_the_manifest_the_newest(instance): + _page(instance, "concepts/Löwe [1].md", "\n# Löwe\n\nText.\n") + first = _commit(instance, "lion") + _page(instance, "concepts/protocols/Beta.md", BETA + "\nMore.\n") + newest = _commit(instance, "edit beta") + (instance / "README.md").write_text("other\n", encoding="utf-8") + _commit(instance, "unrelated") + _configure(instance) + assert _run().exit_code == 0 + folder = _out(instance, "pages") + init = _git(instance, "rev-list", "--max-parents=0", "HEAD") + assert _marker_commit(folder / "concepts" / "Alpha.md") == init + assert _marker_commit(folder / "concepts" / "Löwe [1].md") == first + assert _marker_commit(folder / "concepts" / "protocols" / "Beta.md") == newest + assert _manifest(folder)["commit"] == newest + + +def test_a_page_entering_the_selection_rewrites_only_the_files_whose_text_changes(instance): + _configure(instance, pages={"include": [["tags=docs"], ["summary~Gamma"]]}) + _page(instance, "concepts/Gamma.md", "\n# Gamma\n\nPrivate.\n", tags=["private"], summary="Hidden.") + _commit(instance, "hide gamma") + assert _run().exit_code == 0 + folder = _out(instance, "pages") + before, stamps = _snapshot(folder), _mtimes(folder) + assert "and Gamma for more" in before["concepts/Alpha.md"].decode("utf-8") + _page(instance, "concepts/Gamma.md", "\n# Gamma\n\nPrivate.\n", tags=["private"]) + _commit(instance, "show gamma") + result = _run() + assert result.exit_code == 0, result.output + out = _flat(result.output) + assert "new (1): Gamma" in out and "changed (1): Alpha" in out and "unchanged: 1" in out + after, after_stamps = _snapshot(folder), _mtimes(folder) + assert "[Gamma](Gamma.md)" in after["concepts/Alpha.md"].decode("utf-8") + beta = "concepts/protocols/Beta.md" + assert after[beta] == before[beta] and after_stamps[beta] == stamps[beta] + + +def test_a_renamed_page_carries_the_commit_of_the_rename(instance): + (instance / "README.md").write_text("other\n", encoding="utf-8") + _commit(instance, "unrelated") + result = runner.invoke(app, ["rename", "--from", "Beta", "--to", "Epsilon"]) + assert result.exit_code == 0, result.output + renamed = _commit(instance, "rename") + _configure(instance) + assert _run().exit_code == 0 + folder = _out(instance, "pages") + assert _marker_commit(folder / "concepts" / "protocols" / "Epsilon.md") == renamed + + +def test_the_git_calls_do_not_grow_with_the_pages(instance, monkeypatch): + _configure(instance) + calls = [] + real = guideline_export._git + + def counted(args, root, stdin=None): + calls.append(args) + return real(args, root, stdin) + + monkeypatch.setattr(guideline_export, "_git", counted) + assert _run().exit_code == 0 + few = len(calls) + for index in range(30): + _page(instance, f"concepts/Many{index}.md", f"\n# Many{index}\n\nText.\n") + _commit(instance, "many") + calls.clear() + result = _run() + assert result.exit_code == 0, result.output + assert "new (30)" in _flat(result.output) + assert len(calls) == few + assert sum(1 for args in calls if "log" in args) == 1 + + def test_a_changed_page_a_new_page_and_a_removed_page(instance): _configure(instance) assert _run().exit_code == 0