feat: the test suite no longer ships, and dist upgrade deletes what a release stops shipping (#113)
CI / verify (push) Successful in 5m26s
CI / pwsh (push) Successful in 2m7s
Release / release (push) Successful in 35s

dist export leaves out tools/chemenu/tests/, tools/pytest.ini and tools/.coveragerc by exact
path - the suite tests the origin repository, and 257 of its tests failed in a fresh export.
dist upgrade now deletes a no-longer-shipped file that is unchanged since install, with any
directory that leaves empty, and blocks one changed since install like any local change
(--take-release deletes it, --keep-local keeps it). --prune is accepted and ignored.

Files changed:
- CHANGES.md
- EVALS.md
- VERSION
- instructions/mcp-read-server.md
- instructions/upgrade-instance.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_dist_upgrade.py
- tools/requirements-mcp.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-03 20:12:32 +02:00
1 parent dd565d249f
commit 0d3499ab04
11 files changed
+479 -87

No files matched your search

+42 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.32 - 2026-10-03 - Page-material passages in type-spec.md, type-guidance.md and language-boundaries.md name subtype templates
## 8.0.0-beta.33 - 2026-10-03 - The test suite no longer ships, and dist upgrade deletes what a release stops shipping
**Author:** Torben Nehmer
@@ -103,6 +103,7 @@ concern - readable here, never shipped as something to parse.
- Stack-Entwicklung in drei Phasen: stack-dev (Design), stack-build, stack-close - Übergabe über den Tracker, kein Modellwechsel in der Sitzung
- raw fetch: a sanctioned intake for a URL into incoming/
- new: a subtype gets its own page skeleton from types/<type>.<value>.md
- The test suite no longer ships, and dist upgrade deletes what a release stops shipping
**Low impact**
- version bump no longer points at version release in its output
@@ -145,6 +146,46 @@ concern - readable here, never shipped as something to parse.
- Page-material passages in type-spec.md, type-guidance.md and language-boundaries.md name subtype templates
<!-- /wikitool:bumps -->
### The test suite no longer ships, and dist upgrade deletes what a release stops shipping
`dist export` used to ship all of `tools/chemenu/tests/` with `tools/pytest.ini` and
`tools/.coveragerc` - about a quarter of every release's files. The suite tests the origin
repository: it reads that repository's own type-specs, conventions and collections, where an
instance carries templates or its own decisions, so in a freshly exported tree 257 of its tests
failed. All three now stay behind, matched by exact path, so a directory that only happens to be
called `tests` elsewhere under `tools/` still ships. `tools/README.md` drops its Tests section on
export; `tools/CONTRACT.md`, `EVALS.md`, `instructions/mcp-read-server.md` and
`tools/requirements-mcp.txt` say the suite lives in the origin repository.
`dist upgrade` used to keep a file the new release no longer ships unless `--prune` was passed.
After that run the new stamp no longer named the path, so no later upgrade could find the file
again. Now the upgrade settles it in the run that sees the file go. If the file is unchanged
since install, it is deleted, just as an unchanged file is overwritten without asking. Any
directory that this leaves holding nothing (at most a `__pycache__/` of `*.pyc`) goes with it.
If the file was changed since install, it is listed as locally changed ("no longer shipped") and
needs the usual answer: `--take-release` deletes it, and `--keep-local` keeps it as the
instance's own file. The upgrade never deletes a file that neither stamp names, and never an
instance-owned seed (`CHANGES.md`, `kb/log.md`, `.wikitool-kb.json`). `--prune` is still
accepted and does nothing (Gitea #113).
**For an existing instance:** the upgrade *into* this release still runs the installed
`dist upgrade`, which keeps no-longer-shipped files unless told otherwise. Pass `--prune` on that
one run. It then removes the test suite, `tools/pytest.ini` and `tools/.coveragerc`, and also
`instructions/private-instance.md` and `tools/chemenu/commands/upstream_cmd.py`, which this
release drops as well. If they are already left over, remove them by hand, together with any of
these three, which earlier releases dropped:
`tools/chemenu/sections.py`, `instructions/claude-code-model-selection.md`,
`tools/chemenu/commands/confidence_decay.py`.
```bash
git rm -r --ignore-unmatch tools/chemenu/tests tools/pytest.ini tools/.coveragerc \
instructions/private-instance.md tools/chemenu/commands/upstream_cmd.py \
tools/chemenu/sections.py instructions/claude-code-model-selection.md \
tools/chemenu/commands/confidence_decay.py
```
From the next upgrade on, this happens on its own.
### Page-material passages in type-spec.md, type-guidance.md and language-boundaries.md name subtype templates
Found in the closing check of the change below: three documents listed what in `types/` is page
+2 -2
View File
@@ -9,8 +9,8 @@ that [AGENTS.md](AGENTS.md) exists to prevent.
## Why, beyond the unit tests
The pytest suite under `tools/chemenu/tests/` checks the **compiler**: given this input,
does `wikitool` produce that output. It says nothing about the two things that actually go
The pytest suite under `tools/chemenu/tests/` - in the origin repository; `dist export` does
not ship it - checks the **compiler**: given this input, does `wikitool` produce that output. It says nothing about the two things that actually go
wrong in practice - whether the *agent* followed the contracts, and whether the pages it wrote
are any good.
+1 -1
View File
@@ -1 +1 @@
8.0.0-beta.32
8.0.0-beta.33
+2 -2
View File
@@ -106,8 +106,8 @@ everything an operator needs that is *true of the software* rather than of one i
sync is not running. If it is `null`, the served tree has uncommitted changes - something is
writing into the corpus that should not be.
- **The server disagrees with `wikitool` on the same query?** That is a defect, not a
configuration difference: the two go through the same functions and a golden test holds their
output together (`tools/chemenu/tests/test_mcp_server.py`). Check first that both are pointed
configuration difference: the two go through the same functions and a golden test in the origin
repository holds their output together. Check first that both are pointed
at the same root - `CHEMENU_ROOT` is easy to set for one and not the other.
- **Asked to expose a write tool?** Five of the six tools have none, structurally: the server
imports nothing under `chemenu.commands`, so `new`, `touch`, `xref`, `cite`, `publish` and
+6 -4
View File
@@ -115,8 +115,10 @@ a further checkout of this one ([bootstrap.md](bootstrap.md)).
(`tools/wikitool dist adopt types/<type>.<value>.md.template`) and `wikitool new` scaffolds
pages of that subtype from it; leave it lying and they keep the type's `## Template` block.
Both are valid - [subtype-templates.md](subtype-templates.md) is how to judge whether the
corpus wants it. `locally changed` is step 6. `removed` matters only if `--prune` is wanted, which is optional
and never required.
corpus wants it. `locally changed` is step 6. `removed` needs no decision: a file the release
no longer ships and that is unchanged since install is deleted, together with any directory
that leaves empty, and the report lists both. One that was changed since install appears
under `locally changed` as "no longer shipped" instead, and is step 6.
6. **Only if a file is reported as locally changed: decide whose file it is, then reconcile it.**
The classification is against the sha256 the *installed* release recorded, so "locally
@@ -126,8 +128,8 @@ a further checkout of this one ([bootstrap.md](bootstrap.md)).
| Whose file | What to do |
|---|---|
| The instance's own | Cannot appear here, which is worth knowing so a report that looks like it is read again rather than acted on: a file the instance owns either ships only as `<name>.template` (`kb/CONVENTIONS.md`, each `COLLECTION.md`, `USER.md`/`SOUL.md`/`ENVIRONMENT.md`) and is never classified at all, or is seeded once and then kept out of the write set (`.wikitool-kb.json`, `CHANGES.md`) |
| Machinery (a `CONTRACT.md`, anything under `tools/`, `types/`, `instructions/`, `AGENTS.md`, and every `<name>.template` beside an owned file) | It should not have local changes at all. Take the release's version: `--take-release <path>`, one per file |
| Machinery this instance changed **on purpose** | `--keep-local` keeps every listed file untouched - but the new stamp records the release digest anyway, so the same file is reported again at every future upgrade. That is the right answer only for a difference the instance intends to carry indefinitely |
| Machinery (a `CONTRACT.md`, anything under `tools/`, `types/`, `instructions/`, `AGENTS.md`, and every `<name>.template` beside an owned file) | It should not have local changes at all. Take the release's version: `--take-release <path>`, one per file. For a file marked "no longer shipped" the release's version is no file at all, so taking it deletes it |
| Machinery this instance changed **on purpose** | `--keep-local` keeps every listed file untouched - but the new stamp records the release digest anyway, so the same file is reported again at every future upgrade. That is the right answer only for a difference the instance intends to carry indefinitely. A file marked "no longer shipped" is the exception: no stamp names it after this run, so keeping it makes it the instance's own and it is never reported again |
The decision is per path, and the two flags compose - which is what a mixed report needs, one
file reset and another kept. Preview it before it writes:
+9 -4
View File
@@ -2599,7 +2599,7 @@ Write a contentless, distributable copy of this repo's machinery.
**NOTES**
- Writes a contentless, distributable copy of this repo's machinery into an empty or new `<target>` directory.
- Ships `AGENTS.md`/`README.md`/`EVALS.md` with any `dist:strip-start`...`dist:strip-end` marker region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs, their schemas and their subtype templates `types/<type>.<value>.md` re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas) and `VERSION`.
- Ships `AGENTS.md`/`README.md`/`EVALS.md` with any `dist:strip-start`...`dist:strip-end` marker region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs, their schemas and their subtype templates `types/<type>.<value>.md` re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches, and not the dev-only test suite `tools/chemenu/tests/` with its `pytest.ini`/`.coveragerc`), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas) and `VERSION`.
- Ships `raw/` and `incoming/` as flat roots, each with a `.gitkeep` and no subdirectories. `incoming/.gitkeep` is trackable and survives becoming a git repository, so a plain clone gets the directory without any bootstrap step.
- Ships templates, never the filled files: `USER.md.template`/`SOUL.md.template`, `kb/CONVENTIONS.md.template`, and each collection's contract re-keyed as `kb/<name>/COLLECTION.md.template`. The filled `USER.md`/`SOUL.md`/`kb/CONVENTIONS.md`/`kb/<name>/COLLECTION.md`/`types/<page-type>.md`/`types/<page-type>.<value>.md` bind their instance; `find_leaks` refuses a plan carrying one.
- Writes a generated `.wikitool-release.json` stamp: version, export date, origin, and a sha256 per exported file - the base a later upgrade compares against.
@@ -2738,9 +2738,10 @@ Apply a stack update `dist export` produced - the write half of `version check`.
- Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten.
- Locally changed files abort the run with the full list; the abort text names the three answers with the command line filled in, and none of them is the default.
- `--keep-local` proceeds and leaves every locally changed file untouched. The new stamp is still written whole, recording the release's digest for files that were not written, so a skipped file keeps diverging and is reported again on every later run.
- `--take-release <path>` (repeatable) writes the release's version over the named path, discarding the local change, and re-creates it if it was locally deleted. The path then matches the stamp and stops being reported.
- `--take-release <path>` (repeatable) writes the release's version over the named path, discarding the local change, re-creates it if it was locally deleted, and deletes it if the release no longer ships it. The path then matches the stamp and stops being reported.
- The two are decided per path and compose on one call: without `--keep-local`, a locally changed path that no `--take-release` names still aborts the run.
- A path in the old stamp but not the new one is reported as no longer part of the release and left alone, unless `--prune` is passed, which removes it only if it is still unchanged since installation.
- A path in the old stamp but not the new one is no longer part of the release. Still unchanged since installation, it is deleted, as silently as an unchanged file is overwritten; changed since, it is locally changed like any other and needs one of the same answers - `--take-release` deletes it, `--keep-local` keeps it as the instance's own file, which no later upgrade reports again. Either way it is decided in this run: the new stamp no longer names the path. What the instance owns (`kb/log.md`, `CHANGES.md`, `.wikitool-kb.json`, ...) is never a candidate, and neither is any file the old stamp does not name.
- A directory left holding nothing - at most a `__pycache__/` of `*.pyc` - once those files are deleted is removed with them, up to but never including the instance root. `--prune` is accepted and ignored; it used to switch the deletion on.
- Reports the migration chain the new machinery would owe, but never runs any of it - there is no `migrate run`.
- Reports, but does not block on, a crossed compatibility boundary.
- The local preconditions - `VERSION`, the release stamp, `.wikitool-kb.json`, no outstanding migration, a clean working tree - are checked before the source is read. Not being a git repository at all is a WARN, not a refusal.
@@ -3426,6 +3427,10 @@ Check that this instance is correctly configured.
## Tests
The suite and its configuration (`chemenu/tests/`, `pytest.ini`, `.coveragerc`) exist in the
origin repository only - `dist export` leaves them out, because they test that repository's own
type-specs and conventions rather than an instance's. There:
```bash
cd tools
.venv/bin/pip install pytest pytest-cov # one time; pytest-cov is optional
@@ -3433,7 +3438,7 @@ cd tools
```
Neither is in `requirements.txt`, and `pytest-cov` is CI-only by intent - see
[README.md](README.md#tests) and [EVALS.md](../EVALS.md).
[EVALS.md](../EVALS.md).
## Maintenance schedule
+9 -3
View File
@@ -52,7 +52,7 @@ but names a path that has gone raises `ToolPathError`, which the CLI turns
into an `ERROR` line pointing at the preflight rather than a traceback.
The package runs natively on Windows as well, which CI never does. So the rules that keep
it portable are held by reading the source, in `tests/test_portability.py`: a path that
it portable are held by reading the source, in the origin repository's `tests/test_portability.py`: a path that
becomes a string goes through `.as_posix()`, a text file is opened with `encoding=` and
written with `newline="\n"`, a `subprocess` call with `text=True` names `encoding="utf-8"`,
and only `filelock.py` imports `fcntl` or `msvcrt`. An `rg` path comes back with `\` on
@@ -109,7 +109,7 @@ tools/
search/ pluggable search backends, plus service.py - the search core
tasks/ the task-tracker provider layer: protocol.py (TaskReader/TaskWriter), config.py (.wikitool-tasks.json), one module per adapter - no instruction ever learns which provider it is
commands/ one module per command or command group: the terminal adapters
tests/ pytest suite
tests/ pytest suite - origin repository only, `dist export` leaves it out
```
**A script beside the package.** `bugreport.py` is not part of `chemenu` and imports nothing
@@ -121,7 +121,7 @@ identities it can read from the machine (stage 1), and `--bundle`/`--candidates`
model found in the result (stage 2), both word by word so that length, separators and depth
survive. The mapping, the review list and the candidate file stay beside the bundle directory.
`dist export` ships it with the rest of `tools/`;
its tests (`tests/test_bugreport.py`) run it on a bare interpreter (`-I -S`) to keep that promise.
its tests (`tests/test_bugreport.py`, in the origin repository) run it on a bare interpreter (`-I -S`) to keep that promise.
It is started through `tools/bugreport` (`bugreport.ps1` under PowerShell), which assumes no more
than the collector does: no preflight, no `.wikitool-tools.json`, no venv. It looks for the
@@ -243,8 +243,13 @@ the tool writes edges into a collection whose rules the author never read.
directions so an ignore rule can neither swallow tracked content nor stop
ignoring generated copies.
<!-- dist:strip-start -->
## Tests
The suite exists in the origin repository only: `dist export` leaves out `chemenu/tests/`,
`pytest.ini` and `.coveragerc`, because the suite tests that repository's own type-specs and
conventions, not an instance's.
```bash
cd tools && .venv/bin/python -m pytest -q
```
@@ -275,3 +280,4 @@ section name from `sections`, not on a literal.
Assert on prose in a type-spec or a contract only when the prose is the subject of the test.
Otherwise it is a tripwire that fires on an edit nobody connected to the test.
<!-- dist:strip-end -->
+201 -50
View File
@@ -25,7 +25,9 @@ shapes of "does not belong in someone else's instance":
- Build output under `tools/` is dropped, by directory (`TOOLS_EXCLUDE_DIRS`)
where it has one, and by filename (`_is_coverage_output`) where it does not.
Not dev-only but *derived*: recomputable, and measured against this repo's
own test run rather than the receiving instance's.
own test run rather than the receiving instance's. The test suite and its
configuration (`TOOLS_DEV_ONLY`) go the same way, by exact path: dev-only
whole files again, like `instructions/dev/`, only under `tools/`.
"""
from __future__ import annotations
@@ -122,10 +124,20 @@ HOOK_DIRS = (".github/hooks", ".vibe")
# tools/ subpaths never copied - build/venv/cache artifacts, not machinery.
# `htmlcov/` is coverage.py's HTML report: derived output, and a large tree of
# it, measured against the source repo's own test run. `.coveragerc` beside it
# *does* ship, the same way `pytest.ini` does - it is configuration, not output.
# it, measured against the source repo's own test run. Matched by directory
# name at any depth, which is right for a cache and wrong for anything with an
# ordinary name - that is what `TOOLS_DEV_ONLY` below is for.
TOOLS_EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", ".wikitool_session", "htmlcov"}
# tools/-relative paths that are dev-only rather than derived: the test suite
# and the two files that configure running and measuring it. The suite tests
# *this* repository - it reads the shipped `types/`, `kb/CONVENTIONS.md` and
# collections as this repo has them, where an instance carries templates or its
# own decisions - so in an exported tree a quarter of it fails, and `pytest` is
# not in requirements.txt to run it with anyway. Matched by exact path, not by
# name: a directory called `tests` elsewhere under tools/ would be machinery.
TOOLS_DEV_ONLY = frozenset({"chemenu/tests", "pytest.ini", ".coveragerc"})
# The rest of coverage's output lands beside the code rather than in a directory
# of its own - `.coverage`, `coverage.xml`, and `.coverage.<host>.<pid>` under a
# parallel run - so a directory exclusion cannot reach it. Same argument as
@@ -243,22 +255,31 @@ def _copy_tree(
dest_prefix: str,
exclude_dirs: frozenset[str],
exclude_file: Optional[Callable[[str], bool]] = None,
exclude_paths: frozenset[str] = frozenset(),
) -> dict[str, PlannedFile]:
"""Every file under source_root, marker-stripped, keyed by its
destination-relative path. Excluded directories are pruned during the
walk rather than filtered after, so a large `.venv/` is never read.
`exclude_file` drops individual files by name, for output that lands
beside the code instead of in a directory a prune could catch."""
beside the code instead of in a directory a prune could catch.
`exclude_paths` drops files or whole directories by their exact
source_root-relative path, for one whose name alone says nothing."""
files: dict[str, PlannedFile] = {}
if not source_root.is_dir():
return files
for dirpath, dirnames, filenames in os.walk(source_root):
dirnames[:] = sorted(d for d in dirnames if d not in exclude_dirs)
here = Path(dirpath).relative_to(source_root)
dirnames[:] = sorted(
d for d in dirnames
if d not in exclude_dirs and (here / d).as_posix() not in exclude_paths
)
for filename in sorted(filenames):
if exclude_file is not None and exclude_file(filename):
continue
path = Path(dirpath) / filename
relative = path.relative_to(source_root).as_posix()
if relative in exclude_paths:
continue
dest_rel = f"{dest_prefix}/{relative}"
files[dest_rel] = _read_planned_file(path, dest_rel)
return files
@@ -435,7 +456,8 @@ def build_plan(origin: Optional[Origin] = None) -> dict[str, PlannedFile]:
plan.update(_copy_tree(config.INSTRUCTIONS_DIR, "instructions", frozenset(INSTRUCTIONS_EXCLUDE_DIRS)))
plan.update(_plan_types())
plan.update(_copy_tree(
config.ROOT / "tools", "tools", frozenset(TOOLS_EXCLUDE_DIRS), _is_coverage_output
config.ROOT / "tools", "tools", frozenset(TOOLS_EXCLUDE_DIRS), _is_coverage_output,
TOOLS_DEV_ONLY,
))
for hook_dir in HOOK_DIRS:
plan.update(_copy_tree(config.ROOT / hook_dir, hook_dir, frozenset()))
@@ -607,7 +629,9 @@ def _write_plan(target: Path, plan: dict[str, PlannedFile]) -> None:
"Ships `AGENTS.md`/`README.md`/`EVALS.md` with any `dist:strip-start`...`dist:strip-end` "
"marker region removed, `instructions/` (minus `instructions/dev/`), `types/` (the "
"`root: kb` page type-specs, their schemas and their subtype templates "
"`types/<type>.<value>.md` re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the "
"`types/<type>.<value>.md` re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no "
"venv/caches, and not the dev-only test suite `tools/chemenu/tests/` with its "
"`pytest.ini`/`.coveragerc`), the "
"`.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, "
"`kb/CONTRACT.md` (no pages, no areas) and `VERSION`.",
"Ships `raw/` and `incoming/` as flat roots, each with a `.gitkeep` and no "
@@ -878,20 +902,27 @@ def run_adopt(templates: Sequence[Path], dry_run: bool = False) -> None:
@dataclass(frozen=True)
class FileClassification:
"""The four-way split of every path `dist upgrade` would touch, plus the
fifth direction (`removed`) that has no write set of its own."""
"""The four-way split of every path `dist upgrade` would write, plus the
paths the new release no longer ships (`retired`), split the same way.
A retired file still unchanged since install is deleted, as silently as an
unchanged file is overwritten. A retired file changed since install is
blocked like any other local change. Neither can be left for later: the new
stamp no longer names the path, so the next upgrade could not tell it from a
file the instance wrote itself."""
unchanged: list[str]
modified: list[str]
deleted: list[str]
new: list[str]
removed: list[str]
retired: list[str]
retired_modified: list[str]
@property
def blocked(self) -> list[str]:
"""Locally changed paths - modified or deleted - which are never
silently overwritten."""
return sorted(self.modified + self.deleted)
"""Locally changed paths - modified, deleted, or modified and no
longer shipped - which are never silently overwritten or removed."""
return sorted(self.modified + self.deleted + self.retired_modified)
def _write_candidates(new_files: dict) -> set[str]:
@@ -930,8 +961,84 @@ def _classify_files(old_files: dict, new_files: dict) -> FileClassification:
else:
deleted.append(relative)
removed = sorted(set(old_files) - set(new_files))
return FileClassification(unchanged, modified, deleted, new, removed)
# The same exemptions as the write set, for the same reason: a path an
# export seeds and the instance owns from then on is the instance's to keep,
# whether or not a later release still seeds it.
gone = {
relative: digest
for relative, digest in old_files.items()
if relative not in new_files
and not ownership.is_export_stub(Path(relative).name)
and not ownership.is_upgrade_preserved(relative)
}
retired: list[str] = []
retired_modified: list[str] = []
for relative, status in kb_state.compare_against_stamp(gone).items():
if status == kb_state.UNCHANGED:
retired.append(relative)
elif status == kb_state.MODIFIED:
retired_modified.append(relative)
# DELETED: already gone, nothing left to decide.
return FileClassification(
unchanged, modified, deleted, new, sorted(retired), sorted(retired_modified)
)
# Everything in a `__pycache__/` is derived from the sources beside it and
# gitignored, so it never keeps an otherwise emptied directory alive.
_DERIVED_DIR = "__pycache__"
def _only_bytecode(directory: Path) -> bool:
return all(entry.is_file() and entry.suffix == ".pyc" for entry in directory.iterdir())
def _emptied_directories(root: Path, deleted: Sequence[str]) -> list[Path]:
"""The directories that hold nothing once `deleted` (root-relative files)
are gone, deepest first - walking up from each deleted file, never as far
as `root` itself.
"Nothing" allows exactly one thing: a `__pycache__/` holding only `*.pyc`.
Any other entry - a file no stamp knows about, a subdirectory still in use -
keeps the directory and every parent above it. Computed before anything is
removed, so `--dry-run` reports the same list the writing run acts on."""
gone: set[Path] = {root / relative for relative in deleted}
emptied: list[Path] = []
candidates = sorted(
{parent for path in gone for parent in path.parents if root in parent.parents},
key=lambda p: len(p.parts),
reverse=True,
)
for directory in candidates:
if not directory.is_dir():
continue
removable = True
for entry in directory.iterdir():
if entry in gone:
continue
if entry.name == _DERIVED_DIR and entry.is_dir() and _only_bytecode(entry):
continue
removable = False
break
if removable:
gone.add(directory)
emptied.append(directory)
return emptied
def _remove_emptied(directory: Path) -> bool:
"""Remove one directory `_emptied_directories` named, re-checking rather
than trusting the plan: its bytecode cache goes only while it still holds
nothing but `*.pyc`, and `rmdir` refuses a directory that is not empty, so
nothing that appeared since the plan was made is ever taken with it."""
cache = directory / _DERIVED_DIR
if cache.is_dir() and _only_bytecode(cache):
shutil.rmtree(cache)
try:
directory.rmdir()
except OSError:
return False
return True
def _verify_sha256_sidecar(archive: Path) -> None:
@@ -1116,8 +1223,9 @@ def _resolve_take_release(
listed = "\n".join(f" - {path}" for path in classification.blocked) or " (none)"
fail(
f"--take-release names {len(unknown)} path(s) that are not locally changed: "
f"{', '.join(unknown)}. Only a path this run reports as locally modified or "
f"locally deleted can be taken from the release. Reported as locally changed:\n"
f"{', '.join(unknown)}. Only a path this run reports as locally modified, "
f"locally deleted, or locally modified and no longer shipped can be taken from the "
f"release. Reported as locally changed:\n"
f"{listed}"
)
return set() # unreachable: fail() raises typer.Exit
@@ -1141,15 +1249,24 @@ def _refusal_for_blocked(
if len(classification.blocked) > 1
else "it is reported again on every future upgrade"
)
retired = [path for path in undecided if path in classification.retired_modified]
retired_note = (
f" A file the release no longer ships ({len(retired)} of them) has no release "
"version: taking it deletes it, and keeping it makes it this instance's own file - no "
"later upgrade reports it again, because no stamp names it any more.\n"
if retired
else ""
)
return (
f"{len(undecided)} locally changed file(s) (listed above) would be silently "
f"overwritten. Nothing was written, and none of these three is the default:\n"
f"overwritten or removed. Nothing was written, and none of these three is the default:\n"
f" - take the release's version and discard the local change:\n"
f" dist upgrade {target} --take-release {paths}\n"
f" - keep every local change and upgrade around them ({kept_again}):\n"
f" dist upgrade {target} --keep-local\n"
f" - reconcile them by hand first, then re-run."
)
f" - reconcile them by hand first, then re-run.\n"
f"{retired_note}"
).rstrip("\n")
def _report_plan(
@@ -1159,6 +1276,7 @@ def _report_plan(
local_version: "version_mod.Version",
new_version: "version_mod.Version",
taken: set[str] = frozenset(),
emptied: Sequence[Path] = (),
) -> None:
console.print(f"{local_version} -> {new_version}")
if boundary_crossing:
@@ -1169,12 +1287,16 @@ def _report_plan(
)
console.print(
f"{len(classification.unchanged)} unchanged, {len(classification.new)} new, "
f"{len(classification.blocked)} locally changed, {len(classification.removed)} removed "
f"{len(classification.blocked)} locally changed, {len(classification.retired)} removed "
"from the release."
)
def _mark(relative: str) -> str:
return " [cyan](--take-release: overwritten from the release)[/cyan]" if relative in taken else ""
if relative not in taken:
return ""
if relative in classification.retired_modified:
return " [cyan](--take-release: deleted, as the release no longer ships it)[/cyan]"
return " [cyan](--take-release: overwritten from the release)[/cyan]"
if classification.modified:
console.print(f"[bold]Locally modified ({len(classification.modified)}):[/bold]")
@@ -1184,10 +1306,23 @@ def _report_plan(
console.print(f"[bold]Locally deleted ({len(classification.deleted)}):[/bold]")
for relative in classification.deleted:
console.print(f" - {relative}{_mark(relative)}")
if classification.removed:
console.print("[dim]No longer part of the release, not written or removed by default:[/dim]")
for relative in classification.removed:
console.print(f" [dim]- {relative}[/dim]")
if classification.retired_modified:
console.print(
f"[bold]Locally modified, no longer shipped ({len(classification.retired_modified)}):[/bold]"
)
for relative in classification.retired_modified:
console.print(f" - {relative}{_mark(relative)}")
if classification.retired:
console.print(
f"No longer part of the release, unchanged since install - deleted "
f"({len(classification.retired)}):"
)
for relative in classification.retired:
console.print(f" - {relative}")
if emptied:
console.print(f"Directories left empty by that, removed ({len(emptied)}):")
for directory in emptied:
console.print(f" - {rel_path(directory)}/")
if migration_chain:
console.print(
f"[cyan]{len(migration_chain)} migration(s) will be outstanding after this "
@@ -1248,13 +1383,22 @@ def _report_plan(
"not written, so a skipped file keeps diverging and is reported again on every later "
"run.",
"`--take-release <path>` (repeatable) writes the release's version over the named "
"path, discarding the local change, and re-creates it if it was locally deleted. The "
"path then matches the stamp and stops being reported.",
"path, discarding the local change, re-creates it if it was locally deleted, and "
"deletes it if the release no longer ships it. The path then matches the stamp and "
"stops being reported.",
"The two are decided per path and compose on one call: without `--keep-local`, a "
"locally changed path that no `--take-release` names still aborts the run.",
"A path in the old stamp but not the new one is reported as no longer part of the "
"release and left alone, unless `--prune` is passed, which removes it only if it is "
"still unchanged since installation.",
"A path in the old stamp but not the new one is no longer part of the release. Still "
"unchanged since installation, it is deleted, as silently as an unchanged file is "
"overwritten; changed since, it is locally changed like any other and needs one of the "
"same answers - `--take-release` deletes it, `--keep-local` keeps it as the instance's "
"own file, which no later upgrade reports again. Either way it is decided in this run: "
"the new stamp no longer names the path. What the instance owns (`kb/log.md`, "
"`CHANGES.md`, `.wikitool-kb.json`, ...) is never a candidate, and neither is any file "
"the old stamp does not name.",
"A directory left holding nothing - at most a `__pycache__/` of `*.pyc` - once those "
"files are deleted is removed with them, up to but never including the instance root. "
"`--prune` is accepted and ignored; it used to switch the deletion on.",
"Reports the migration chain the new machinery would owe, but never runs any of it - "
"there is no `migrate run`.",
"Reports, but does not block on, a crossed compatibility boundary.",
@@ -1403,7 +1547,8 @@ def upgrade_command(
),
prune: bool = typer.Option(
False, "--prune",
help="Also delete files the new release no longer ships, if they are unchanged since install",
help="Accepted and ignored: files the new release no longer ships are now deleted by "
"default when unchanged since install",
),
allow_pre: bool = typer.Option(
False, "--pre", help="Allow a pre-release (-beta.N) source tree - release.yml never publishes one",
@@ -1422,7 +1567,9 @@ def upgrade_command(
overwritten silently, new files are created, and a locally modified or
deleted file is never silently overwritten - `dist upgrade` aborts unless
`--keep-local` says to leave it alone or `--take-release <path>` names it
as one to overwrite from the release. Reports the migration chain the new
as one to overwrite from the release. A file the release no longer ships
is deleted if unchanged since install, and blocked like any local change
otherwise; `--prune` is accepted and ignored. Reports the migration chain the new
machinery would owe without running any of it (there is no `migrate run`).
Refuses on a missing local release stamp, a downgrade, a pre-release
source without `--pre`, a dirty working tree, or a `--take-release` path
@@ -1445,7 +1592,7 @@ def run_upgrade(
dry_run: bool = False,
keep_local: bool = False,
take_release: Optional[Sequence[str]] = None,
prune: bool = False,
prune: bool = False, # accepted and ignored - retired files are always classified now
allow_pre: bool = False,
latest: bool = False,
expect: Optional[str] = None,
@@ -1582,8 +1729,13 @@ def run_upgrade(
boundary_crossing = local_version.compat_key != new_version.compat_key
taken = _resolve_take_release(take_release, classification)
to_delete = sorted(
classification.retired + [p for p in classification.retired_modified if p in taken]
)
emptied = _emptied_directories(config.ROOT, to_delete)
_report_plan(
classification, migration_chain, boundary_crossing, local_version, new_version, taken
classification, migration_chain, boundary_crossing, local_version, new_version, taken,
emptied,
)
# Dry-run's whole purpose is to preview this classification - including
@@ -1603,7 +1755,10 @@ def run_upgrade(
fail(_refusal_for_blocked(target, undecided, classification))
return
to_write = sorted(classification.unchanged + classification.new + sorted(taken))
to_write = sorted(
classification.unchanged + classification.new
+ [p for p in taken if p not in classification.retired_modified]
)
for relative in to_write:
src = new_root / relative
dst = config.ROOT / relative
@@ -1617,26 +1772,22 @@ def run_upgrade(
# digest recorded here and stops being reported at all.
shutil.copy2(stamp_path, config.ROOT / version_mod.RELEASE_STAMP_FILENAME)
pruned: list[str] = []
if prune:
for relative in classification.removed:
digest = old_files.get(relative)
if digest is None:
continue
status = kb_state.compare_against_stamp({relative: digest}).get(relative)
if status != kb_state.UNCHANGED:
continue
target = config.ROOT / relative
if target.is_file():
target.unlink()
pruned.append(relative)
# Retired files go regardless of `--prune`, which is accepted and ignored: an
# unchanged retired file is deleted as silently as an unchanged file is
# overwritten, and only now - the new stamp just written no longer names it, so
# no later run could find it again. Only paths classified above are touched;
# a file neither stamp names is never a candidate.
for relative in to_delete:
(config.ROOT / relative).unlink(missing_ok=True)
removed_dirs = [directory for directory in emptied if _remove_emptied(directory)]
skipped = undecided if keep_local else []
summary = (
f"Upgraded {local_version} -> {new_version}: {len(to_write)} file(s) written"
+ (f", {len(taken)} taken from the release (--take-release)" if taken else "")
+ (f", {len(skipped)} left untouched (--keep-local)" if skipped else "")
+ (f", {len(pruned)} pruned" if pruned else "")
+ (f", {len(to_delete)} no longer shipped and deleted" if to_delete else "")
+ (f", {len(removed_dirs)} emptied director(ies) removed" if removed_dirs else "")
+ "."
)
if migration_chain:
+28 -2
View File
@@ -281,8 +281,34 @@ def test_plan_excludes_coverage_output(repo):
assert "tools/.coverage.host.4242" not in plan
assert "tools/coverage.xml" not in plan
assert not any(relative.startswith("tools/htmlcov/") for relative in plan)
# Configuration is machinery and ships, the way pytest.ini does.
assert "tools/.coveragerc" in plan
# Its configuration is dev-only with the suite it measures - see the next test.
assert "tools/.coveragerc" not in plan
def test_plan_excludes_the_test_suite_and_its_configuration_by_exact_path(repo):
"""The suite tests this repository, not an instance: in an exported tree it
reads templates where it expects this repo's own type-specs and conventions,
and a quarter of it fails. It goes by exact path - a directory that merely
shares the name `tests` elsewhere under tools/ is machinery and still ships."""
tools_dir = repo / "tools"
suite = tools_dir / "chemenu" / "tests"
(suite / "fixtures" / "sp").mkdir(parents=True)
(suite / "__init__.py").write_text("", encoding="utf-8")
(suite / "test_something.py").write_text("def test_x():\n pass\n", encoding="utf-8")
(suite / "fixtures" / "sp" / "tasks.json").write_text("[]\n", encoding="utf-8")
(tools_dir / "pytest.ini").write_text("[pytest]\n", encoding="utf-8")
(tools_dir / ".coveragerc").write_text("[run]\n", encoding="utf-8")
(tools_dir / "chemenu" / "evals" / "tests").mkdir(parents=True)
(tools_dir / "chemenu" / "evals" / "tests" / "keep.py").write_text("", encoding="utf-8")
(tools_dir / "chemenu" / "tests.py").write_text("", encoding="utf-8")
plan = dist_cmd.build_plan()
assert not any(relative.startswith("tools/chemenu/tests/") for relative in plan)
assert "tools/pytest.ini" not in plan
assert "tools/.coveragerc" not in plan
assert "tools/chemenu/evals/tests/keep.py" in plan
assert "tools/chemenu/tests.py" in plan
def test_plan_ships_the_environment_template_but_not_the_filled_file(repo):
+177 -16
View File
@@ -130,11 +130,17 @@ def test_dry_run_classifies_every_case_and_writes_nothing(instance, tmp_path):
assert json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["version"] == "1.0.0"
def test_removed_file_is_reported_and_left_alone_without_prune(instance, tmp_path):
def test_dry_run_reports_a_retired_file_and_its_emptied_directory_and_keeps_both(
instance, tmp_path, capsys
):
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"})
# tools/wikitool is in the old stamp but absent from the new one.
dist_cmd.run_upgrade(release, dry_run=True)
assert (instance / "tools" / "wikitool").is_file()
out = capsys.readouterr().out
assert "tools/wikitool" in out
assert "tools/" in out.split("Directories left empty")[1]
# --- local changes are never silently overwritten ---------------------------
@@ -520,27 +526,182 @@ def test_source_that_does_not_exist_is_refused(instance, tmp_path):
dist_cmd.run_upgrade(tmp_path / "nowhere")
# --- prune -------------------------------------------------------------------
# --- files the release no longer ships ---------------------------------------
#
# Decided in the run that sees them go: the new stamp no longer names such a
# path, so no later run could tell it from a file the instance wrote itself.
def test_prune_removes_only_removed_files_still_unchanged_since_install(instance, tmp_path):
(instance / "extra.txt").write_text("shipped once, edited since\n", encoding="utf-8")
old_stamp = json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())
old_stamp["files"]["extra.txt"] = _digest("shipped once, unedited\n") # deliberately stale
old_stamp["files"]["gone.txt"] = _digest("also shipped once\n")
(instance / version_mod.RELEASE_STAMP_FILENAME).write_text(json.dumps(old_stamp), encoding="utf-8")
(instance / "gone.txt").write_text("also shipped once\n", encoding="utf-8")
def _record_in_stamp(root: Path, files: dict[str, str]) -> None:
"""Add `files` (relative -> content as installed) to the local stamp, and
put each on disk with that content."""
stamp_path = root / version_mod.RELEASE_STAMP_FILENAME
stamp = json.loads(stamp_path.read_text())
for relative, content in files.items():
path = root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
stamp["files"][relative] = _digest(content)
stamp_path.write_text(json.dumps(stamp), encoding="utf-8")
_CURRENT = {"AGENTS.md": "core\n", "tools/wikitool": "#!/bin/sh\n"}
def test_retired_file_unchanged_since_install_is_deleted_without_any_flag(instance, tmp_path):
_record_in_stamp(instance, {"gone.txt": "shipped once\n"})
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release)
assert not (instance / "gone.txt").exists()
assert (instance / "tools" / "wikitool").is_file()
def test_prune_is_still_accepted_and_changes_nothing(instance, tmp_path):
_record_in_stamp(instance, {"gone.txt": "shipped once\n"})
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
release = _release(
tmp_path, "release", "1.1.0",
{"AGENTS.md": "core\n", "tools/wikitool": "#!/bin/sh\n"},
)
dist_cmd.run_upgrade(release, prune=True)
# gone.txt matched its recorded digest -> pruned.
assert not (instance / "gone.txt").exists()
# extra.txt was locally edited relative to its recorded digest -> kept.
assert (instance / "extra.txt").read_text(encoding="utf-8") == "shipped once, edited since\n"
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n"
def test_directory_emptied_by_a_retired_file_goes_with_its_bytecode_cache(instance, tmp_path):
_record_in_stamp(instance, {
"tools/chemenu/tests/__init__.py": "",
"tools/chemenu/tests/fixtures/sp/api/tasks.json": "[]\n",
})
cache = instance / "tools" / "chemenu" / "tests" / "__pycache__"
cache.mkdir()
(cache / "__init__.cpython-313.pyc").write_bytes(b"\x00bytecode")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release)
assert not (instance / "tools" / "chemenu" / "tests").exists()
# tools/chemenu held nothing else, so it went too; tools/ still holds wikitool.
assert not (instance / "tools" / "chemenu").exists()
assert (instance / "tools" / "wikitool").is_file()
def test_a_file_neither_stamp_names_keeps_itself_and_its_directory(instance, tmp_path):
_record_in_stamp(instance, {"instructions/retired.md": "old\n"})
own = instance / "instructions" / "my-own.md"
own.write_text("the instance wrote this\n", encoding="utf-8")
stray_cache = instance / "instructions" / "__pycache__"
stray_cache.mkdir()
(stray_cache / "notes.txt").write_text("not bytecode\n", encoding="utf-8")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release)
assert not (instance / "instructions" / "retired.md").exists()
assert own.read_text(encoding="utf-8") == "the instance wrote this\n"
assert (stray_cache / "notes.txt").is_file()
def test_a_directory_with_a_non_bytecode_cache_entry_is_kept(instance, tmp_path):
_record_in_stamp(instance, {"lib/old.py": "x = 1\n"})
cache = instance / "lib" / "__pycache__"
cache.mkdir()
(cache / "old.cpython-313.pyc").write_bytes(b"\x00")
(cache / "keep.txt").write_text("someone put this here\n", encoding="utf-8")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release)
assert not (instance / "lib" / "old.py").exists()
assert (cache / "keep.txt").is_file()
def test_instance_owned_paths_are_never_retired(instance, tmp_path):
"""`CHANGES.md` and `kb/log.md` sit in the old stamp because an export seeds
them - and belong to the instance from then on, shipped again or not."""
_record_in_stamp(instance, {"CHANGES.md": "seeded\n", "kb/log.md": "seeded\n"})
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release)
assert (instance / "CHANGES.md").read_text(encoding="utf-8") == "seeded\n"
assert (instance / "kb" / "log.md").read_text(encoding="utf-8") == "seeded\n"
def test_retired_file_changed_since_install_blocks_and_nothing_is_written(instance, tmp_path):
_record_in_stamp(instance, {"gone.txt": "shipped once\n", "also-gone.txt": "same\n"})
(instance / "gone.txt").write_text("edited since\n", encoding="utf-8")
release = _release(
tmp_path, "release", "1.1.0", {**_CURRENT, "types/new.md": "new\n"}
)
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(release)
assert (instance / "gone.txt").read_text(encoding="utf-8") == "edited since\n"
# Neither the unchanged retired file nor a new one was touched by the refusal.
assert (instance / "also-gone.txt").is_file()
assert not (instance / "types").exists()
assert json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["version"] == "1.0.0"
def test_keep_local_keeps_a_changed_retired_file(instance, tmp_path):
_record_in_stamp(instance, {"gone.txt": "shipped once\n"})
(instance / "gone.txt").write_text("edited since\n", encoding="utf-8")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release, keep_local=True)
assert (instance / "gone.txt").read_text(encoding="utf-8") == "edited since\n"
assert "gone.txt" not in json.loads(
(instance / version_mod.RELEASE_STAMP_FILENAME).read_text()
)["files"]
def test_take_release_deletes_a_changed_retired_file(instance, tmp_path):
_record_in_stamp(instance, {"sub/gone.txt": "shipped once\n"})
(instance / "sub" / "gone.txt").write_text("edited since\n", encoding="utf-8")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
dist_cmd.run_upgrade(release, take_release=["sub/gone.txt"])
assert not (instance / "sub").exists()
def test_refusal_says_a_kept_retired_file_becomes_the_instances_own(instance, tmp_path, capsys):
_record_in_stamp(instance, {"gone.txt": "shipped once\n"})
(instance / "gone.txt").write_text("edited since\n", encoding="utf-8")
release = _release(tmp_path, "release", "1.1.0", _CURRENT)
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(release)
out = " ".join(capsys.readouterr().out.split())
assert "no longer shipped" in out
assert "taking it deletes it" in out
def test_dry_run_leaves_the_tree_byte_identical(instance, tmp_path):
_record_in_stamp(instance, {
"gone.txt": "shipped once\n",
"deep/er/gone.txt": "also\n",
"edited.txt": "shipped\n",
})
(instance / "edited.txt").write_text("edited\n", encoding="utf-8")
before = {
p.relative_to(instance).as_posix(): p.read_bytes()
for p in sorted(instance.rglob("*")) if p.is_file()
}
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core v2\n"})
dist_cmd.run_upgrade(release, dry_run=True)
after = {
p.relative_to(instance).as_posix(): p.read_bytes()
for p in sorted(instance.rglob("*")) if p.is_file()
}
assert after == before
assert (instance / "deep" / "er").is_dir()
# --- tarball sources ----------------------------------------------------------
+2 -2
View File
@@ -8,6 +8,6 @@
# tools/.venv/bin/pip install -r tools/requirements-mcp.txt
# WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp --transport stdio
#
# The tests under `chemenu/tests/test_mcp_server.py` skip without it; CI
# installs it, so they do run.
# The origin repository's server tests skip without it; its CI installs it,
# so they do run.
mcp>=2.0