diff --git a/CHANGES.md b/CHANGES.md index 79165cc..9d2c859 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -59,13 +59,14 @@ concern - readable here, never shipped as something to parse. --- -## 7.1.0-beta.31 - 2026-09-26 - new_page/type_resolver comments no longer claim only entities declare a layout: +## 7.1.0-beta.32 - 2026-09-29 - dist upgrade --latest: one-command update from the release feed **Author:** Torben Nehmer **High impact** - wikitool: one data record per command - `-h`, index and CONTRACT.md render from cli_contract (Gitea #121 Phase 1) +- dist upgrade --latest: one-command update from the release feed **Medium impact** - CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them @@ -102,6 +103,50 @@ concern - readable here, never shipped as something to parse. - new_page/type_resolver comments no longer claim only entities declare a layout: +### dist upgrade --latest: one-command update from the release feed (Gitea #161) + +Updating a tarball instance took a manual detour: `version notes`, then fetching the `.tar.gz` and +its `.sha256` from the release page by hand, then `dist upgrade `. `dist upgrade --latest` +now does the middle part. It asks the release feed (`update_url` from the stamp, +`$WIKITOOL_UPDATE_URL` and `$WIKITOOL_UPDATE_TOKEN` as before) which release is latest, downloads +the release's archive and checksum into a scratch directory, verifies the archive against the +checksum, and hands over to the existing upgrade path unchanged - classification, +`--keep-local`/`--take-release`, migration report and every refusal are the same code. `` +becomes optional; exactly one of it and `--latest` must be given. + +The order is what matters. The local preconditions run first, then the feed is asked, and the +version it reports is judged **before any download**: already installed is a success no-op, an +older version is a downgrade refusal, a `-beta.N` version needs `--pre`. Only then are the two +assets looked up - by exact name, `chemenu-stack-.tar.gz` and its `.sha256`, in the +`assets` of the release object the version query already returned, using the feed's own +`browser_download_url` values; no URL is composed. A release missing either asset is refused +before the first download with the assets it does have and its release page named. The checksum +is mandatory on this path (a `` archive without a sibling `.sha256` is still only a +WARN), and the archive's own `VERSION` must equal the feed's version, otherwise nothing is +applied. `--dry-run` downloads and verifies too - classification needs the tree - and removes +everything again; the scratch directory goes away on every exit. + +`--expect ` (only with `--latest`) closes the gap between reading the notes and applying +the update: the feed only offers its *latest* release, so if a newer one appeared since +`version notes` was read, the run refuses before downloading anything and names both versions. +`instructions/upgrade-instance.md` passes the version from step 2 in the dry run and in the real +run. The comparison is on parsed versions, so a `v` prefix does not matter. + +`$WIKITOOL_UPDATE_TOKEN` is sent to an asset download only when the asset URL has the feed's +scheme, host and port, and as an unredirected header, so a redirect cannot carry it elsewhere. +There is no https enforcement: the checksum comes from the same host as the archive and protects +against transfer errors, not against a compromised feed - `INSTALL.md` says so. + +The asset names are a Python constant (`version.ARCHIVE_NAME`) that a test ties to +`.gitea/workflows/release.yml`, so renaming them in one place fails the suite instead of the next +upgrade. `dist upgrade` is now `network: yes` in its record and in the pinned set in `test_cli.py`, +the `version check` record's "never reached implicitly" note names the one explicit exception, the +`fetch_latest` docstring says "three commands", and `tools/CONTRACT.md` is regenerated. The tests +run against a local `http.server` that logs every request and its `Authorization` header, so the +token rule and the "no asset request before the checks pass" rule are asserted on the wire. The +`upstream merge` pointers in the `dist upgrade` record and failure message stay as they are; they +belong to the separate work on the clone path. + ### new_page/type_resolver comments no longer claim only entities declare a layout: Two code comments - `new_page.py`'s module docstring and `TypeResolver.get_layout`'s - still said diff --git a/INSTALL.md b/INSTALL.md index 9f3c2a5..9721d3e 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -152,12 +152,14 @@ tools/wikitool version # was läuft hier, und woher kommt es tools/wikitool version check # gibt es ein neueres Release? ``` -`version check` und `version notes` sind die einzigen Befehle, die ins Netz gehen, und beide -fragen denselben Release-Feed der Ursprungs-Instanz (`$WIKITOOL_UPDATE_URL` überschreibt; sonst -der Wert aus dem Stamp). `version check` ist dafür da; `version notes` greift nur dann darauf -zurück, wenn die lokale `CHANGES.md` den Eintrag nicht hat - auf einer Instanz also immer, siehe -unten - und sagt vorher auf stderr, welche URL es fragt. Ein nicht erreichbarer Feed wird als -Fehler gemeldet - **nie** als „aktuell" und nie als „keine Notes". +`version check`, `version notes` und `dist upgrade --latest` sind die einzigen Befehle, die +ins Netz gehen, und alle drei fragen denselben Release-Feed der Ursprungs-Instanz +(`$WIKITOOL_UPDATE_URL` überschreibt; sonst der Wert aus dem Stamp). `version check` ist dafür da; +`version notes` greift nur dann darauf zurück, wenn die lokale `CHANGES.md` den Eintrag nicht +hat - auf einer Instanz also immer, siehe unten - und sagt vorher auf stderr, welche URL es +fragt; `dist upgrade` fragt den Feed nur, wenn `--latest` dasteht, und lädt dann auch das Release +herunter (siehe „Eine Instanz aktualisieren"). Ein nicht erreichbarer Feed wird als Fehler +gemeldet - **nie** als „aktuell" und nie als „keine Notes". **Was die Versionsnummer aussagt:** kompatibel ist, was in der *linkesten von Null verschiedenen Stelle* übereinstimmt. `0.1.3 → 0.1.4` ist ein sicheres Update, `0.1.3 → 0.2.0` @@ -203,9 +205,32 @@ Agent-Sitzung ausgeführt wird; eine zweite Fassung derselben Schrittfolge an di genau die Kopie, die irgendwann auseinanderläuft. Wer den Lauf selbst fahren will, liest dieselbe Datei. -Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, woher -der Tarball kommt (§ „Version und Updates" und Weg A oben) - und der eine Sonderfall, den die -Instruktion nicht abdecken kann, weil es sie dort noch nicht gibt: +Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, und +wem die Instanz als Quelle vertraut - und zwei Sonderfälle, die die Instruktion nicht abdecken +kann, weil es sie dort noch nicht gibt. + +**Das Release kommt mit einem Befehl.** `tools/wikitool dist upgrade --latest --expect ` +fragt den Release-Feed, lädt Tarball und `.sha256` in ein Arbeitsverzeichnis, prüft den Tarball +gegen die Summe und wendet ihn an; das Arbeitsverzeichnis verschwindet bei jedem Ausgang wieder, +auch bei `--dry-run`. `` ist die, die `version notes` gedruckt hat: der Feed kennt nur +sein *neuestes* Release, und `--expect` verweigert den Lauf **vor** dem Download, wenn inzwischen +ein neueres erschienen ist, statt es ungelesen einzuspielen. Beides zusammen entscheidet vor dem +Download über „schon aktuell", Downgrade und Vor-Release (`-beta.N` braucht `--pre`); fehlt dem +Release der Tarball oder die Summe, bricht der Befehl vor dem ersten Download ab und nennt die +Release-Seite. Wer offline arbeitet oder einen Tarball vom Betreiber bekommen hat, gibt statt +`--latest` weiter die Datei an (`dist upgrade `). + +**Was die Prüfsumme leistet - und was nicht.** Die `.sha256` liegt beim selben Feed wie der +Tarball. Sie schützt vor einer beschädigten Übertragung, nicht vor einem Feed, der selbst +kompromittiert ist: die Echtheit eines Releases beruht auf dem Vertrauen in den Host, dessen +Feed die Instanz fragt (`update_url` im Stamp). Es gibt keinen https-Zwang; wer einen +`http://`-Feed konfiguriert, tut das bewusst. `$WIKITOOL_UPDATE_TOKEN` geht nur an Downloads +auf demselben Host wie der Feed. + +**Beim ersten Sprung auf ein Release, das `--latest` kennt, gibt es die Option in der Instanz +noch nicht** - die Instruktion, die dort steht, gehört zum Release, das die Instanz verlässt. +Dann den Tarball einmal von Hand holen (Weg A oben), prüfen und `dist upgrade ` geben; +ab dem Release danach trägt die Instanz `--latest` selbst. **Beim ersten Sprung auf `4.5.0` oder höher gibt es `dist upgrade` in der Instanz noch nicht** - es kam erst mit `4.5.0`. Dann das Werkzeug aus dem entpackten *neuen* Tarball verwenden, gegen die @@ -233,9 +258,9 @@ sind. Einer Instanz, die älter ist als `.wikitool-kb.json`, fehlt die Datei gan **Fallstricke.** Eine Instanz ohne lokale `.wikitool-release.json` (oder eine ohne `files`-Block, aus der Zeit vor `4.5.0`) hat für `dist upgrade` keine Basis, gegen die es eine lokale Änderung erkennen könnte, und verweigert den Tausch - dafür gibt es heute keine Reparatur. -Der Befehl lädt selbst nichts herunter: `` muss vorher aus Weg A -geholt werden, und ein Tarball muss genau ein Top-Level-Verzeichnis enthalten - die Form, in der -`.gitea/workflows/release.yml` es baut. +Mit `` lädt der Befehl selbst nichts herunter; die Datei muss vorher +aus Weg A geholt werden. Nur `--latest` lädt, und ein Tarball muss in beiden Fällen genau ein +Top-Level-Verzeichnis enthalten - die Form, in der `.gitea/workflows/release.yml` es baut. Vor `4.5.0` stand hier ein rein manueller Ablauf (Maschinerie von Hand kopieren, `kb/CONTRACT.md` eingeschlossen, sha256-Vergleich von Hand). `dist upgrade` ersetzt genau diesen Teil; wer ihn diff --git a/VERSION b/VERSION index d4cc798..f0a43e9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -7.1.0-beta.31 +7.1.0-beta.32 diff --git a/instructions/upgrade-instance.md b/instructions/upgrade-instance.md index 8dd7a01..65a0d15 100644 --- a/instructions/upgrade-instance.md +++ b/instructions/upgrade-instance.md @@ -81,14 +81,25 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream step 12's, run against the migration documents this instance already has. An `offered` upgrade listed separately blocks nothing and is decided later, in step 12. -4. **Fetch the tarball and verify it.** `dist upgrade` downloads nothing; the file has to be - there already. Take the `.tar.gz` and its `.sha256` from the release page found in step 2 and - check them before unpacking. A tarball must unpack to exactly one top-level directory. +4. **Nothing to fetch by hand.** `dist upgrade --latest` asks the release feed for the latest + release, downloads its `.tar.gz` and `.sha256` into a scratch directory, checks the archive + against the checksum and removes both again - all inside the calls of steps 5 to 7. Note the + version step 2's `version notes` printed: steps 5 to 7 pass it as `--expect`, so a release that + appeared in the meantime is refused before anything is downloaded, rather than applied unread. + + The offline alternative is the tarball path: with the feed unreachable, or an archive the + operator supplies, take the `.tar.gz` and its `.sha256` from the release page named in step 2, + check the archive against the checksum before unpacking, and pass the file as `` + where the steps below say `--latest --expect `. A tarball must unpack to exactly one + top-level directory. The checksum comes from the same host as the archive, so it catches a + damaged transfer, not a compromised host - who is trusted to publish releases is the + operator's decision, made before this file starts ([INSTALL.md](../INSTALL.md) § "Version und + Updates"). 5. **Dry-run the swap and read all four counts:** ```bash - tools/wikitool dist upgrade --dry-run + tools/wikitool dist upgrade --latest --expect --dry-run ``` `unchanged` / `new` / `locally changed` / `removed from the release`. `unchanged` needs no @@ -115,7 +126,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream file reset and another kept. Preview it before it writes: ```bash - tools/wikitool dist upgrade --dry-run --take-release [--take-release ] + tools/wikitool dist upgrade --latest --expect --dry-run --take-release [--take-release ] ``` The preview marks every named path as one it would overwrite from the release, and a path that @@ -134,7 +145,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream ```bash git rev-parse --short HEAD # the pre-swap commit; keep it - tools/wikitool dist upgrade [--take-release ] [--keep-local] + tools/wikitool dist upgrade --latest --expect [--take-release ] [--keep-local] ``` It writes, and commits nothing. diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 9b75816..6ed6e81 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -2418,7 +2418,7 @@ Apply a stack update `dist export` produced - the write half of `version check`. **SYNOPSIS** -- `wikitool dist upgrade [--dry-run] [--keep-local] [--take-release ]... [--prune] [--pre]` +- `wikitool dist upgrade ( | --latest [--expect ]) [--dry-run] [--keep-local] [--take-release ]... [--prune] [--pre]` **PROPERTIES** @@ -2426,23 +2426,30 @@ Apply a stack update `dist export` produced - the write half of `version check`. - idempotent: no - atomic: Yes for every refusal - nothing is written. Once writing starts it is a plain sequential file copy with no partial-state cleanup: an interruption mid-copy (killed process, disk full) can leave the tree part-old, part-new - budget: counted -- network: no +- network: yes **EXAMPLES** - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run` - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz` - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md` +- `tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run` **EXIT STATUS** - 0 success - 0 The source's version equals the installed one - a no-op success +- 1 Both `` and `--latest`, or neither; or `--expect` without `--latest` - 1 Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block - 1 `.wikitool-kb.json` is missing - 1 A migration is already outstanding against the *installed* machinery - 1 The working tree is dirty - 1 `` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory +- 1 `--latest`: the release feed cannot be reached, or answers with something that is not a release +- 1 `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded +- 1 `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded +- 1 `--latest`: an asset download fails, or the archive fails its `.sha256` +- 1 `--latest`: the downloaded archive's `VERSION` is not the version the feed announced - 1 The source carries no `VERSION`, `.wikitool-release.json` or `files` block - 1 The source's version is older than the installed one, or a pre-release without `--pre` - 1 A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises @@ -2450,11 +2457,17 @@ Apply a stack update `dist export` produced - the write half of `version check`. **ON FAILURE** +- Both `` and `--latest`, or neither; or `--expect` without `--latest` -> Not transient - name exactly one source, and pass `--expect` only with `--latest` - Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block -> Not transient - fix the named precondition and retry. A checkout with shared git history takes stack updates with `wikitool upstream merge` instead - `.wikitool-kb.json` is missing -> Run `wikitool migrate baseline `, then retry - A migration is already outstanding against the *installed* machinery -> Finish it first - `wikitool migrate status` names it - then retry - The working tree is dirty -> Commit or stash first, then retry - `` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory -> Fix the path or re-download the release archive, then retry +- `--latest`: the release feed cannot be reached, or answers with something that is not a release -> Transient - retry once; then report the URL from the message, or take the release page's archive by hand and pass it as `` +- `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded -> Not transient - read `wikitool version notes` for the version the message names, then either expect that one or stop +- `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded -> Not transient - the message lists the assets present and the release page; report it there rather than upgrading without the checksum +- `--latest`: an asset download fails, or the archive fails its `.sha256` -> Retry once; if it fails again, report the exact message - do not fall back to an unchecked archive +- `--latest`: the downloaded archive's `VERSION` is not the version the feed announced -> Not transient - an inconsistent release; report it against the release page - The source carries no `VERSION`, `.wikitool-release.json` or `files` block -> Point `` at a distribution export, then retry - The source's version is older than the installed one, or a pre-release without `--pre` -> Not transient - choose another source, or pass `--pre` for a pre-release - A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises -> Correct it against the locally-changed list the refusal prints, then retry @@ -2466,7 +2479,9 @@ Apply a stack update `dist export` produced - the write half of `version check`. **NOTES** -- Never downloads anything: `` is an already-fetched export directory or `.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs. +- Exactly one of `` and `--latest` names the release. `` is an already-fetched export directory or `.tar.gz` release archive and touches no network: the archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs. +- `--latest` asks the release feed (`update_url` of the local stamp, overridable with `$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, already installed - before any download, then downloads the archive and its `.sha256` into a scratch directory removed on every exit. The checksum is mandatory here (a release without one, or an archive that fails it, is an error, not a WARN) and the archive's own `VERSION` must equal the feed's version. The asset URLs are the feed's own `browser_download_url` values, and the token reaches an asset download only if it is on the feed's origin. The checksum protects against transfer errors, not against a feed that is itself compromised - authenticity is the trust in the feed's host. +- `--expect ` (only with `--latest`) pins the release the feed may announce: a different latest version is refused before anything is downloaded. Pass the version `wikitool version notes` was read for, in the dry run and in the real run alike. - The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md`), plus the stamp itself, always rewritten. - Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten. - Locally changed files abort the run with the full list; the abort text names the three answers with the command line filled in, and none of them is the default. @@ -2478,13 +2493,14 @@ Apply a stack update `dist export` produced - the write half of `version check`. - Reports, but does not block on, a crossed compatibility boundary. - The local preconditions - `VERSION`, the release stamp, `.wikitool-kb.json`, no outstanding migration, a clean working tree - are checked before the source is read. Not being a git repository at all is a WARN, not a refusal. - Never touches git - no commit, no push. -- `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`. +- `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`. With `--latest` it still downloads and verifies the archive - that is the only way to classify - and removes it again. - An interrupted write is not resumed automatically: compare the tree against the printed classification and finish or revert by hand. - The closing report names `instructions/upgrade-instance.md`, which carries the order for everything after the swap and resumes at `instructions sync`. **SEE ALSO** - `wikitool version check` - finds out whether an update exists +- `wikitool version notes` - the notes of the release `--expect` should name - `instructions/upgrade-instance.md` - the order after the swap - `INSTALL.md` § "Version und Updates" - which release, whether to take it, where the tarball comes from - `wikitool upstream merge` - the update path for a checkout with shared git history @@ -2571,8 +2587,8 @@ Ask the origin's release feed whether a newer stack exists. **NOTES** - Asks the release feed for its latest release and compares it with `VERSION`: `state` is `current`, `update`, `migration` (the step crosses a compatibility boundary) or `ahead`. -- The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance. -- Never reached implicitly from another command, needs no key, and times out after `--timeout` seconds (default 10). +- The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance, and `dist upgrade --latest` asks it which release to download. +- Never reached implicitly from another command (`dist upgrade` asks only when passed `--latest`), needs no key, and times out after `--timeout` seconds (default 10). - The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously. - For `update` or `migration` it prints that applying the release is a separate, manual step (`INSTALL.md` § "Eine Instanz aktualisieren"). - Read-only; exempt from the Iteration Budget Gate. diff --git a/tools/chemenu/commands/dist_cmd.py b/tools/chemenu/commands/dist_cmd.py index 866f23a..8d0cbe5 100644 --- a/tools/chemenu/commands/dist_cmd.py +++ b/tools/chemenu/commands/dist_cmd.py @@ -736,8 +736,8 @@ def run_export(target: Path, dry_run: bool = False, origin: Optional[Origin] = N # modified/deleted file is never silently overwritten: the run aborts unless # `--keep-local` keeps it or `--take-release ` names it, which is the # difference between a file the instance means to carry and one that drifted. -# This never calls a release feed; the caller supplies an already-downloaded -# tree or archive. +# The classification never touches the network: the tree or archive it works +# on is either supplied by the caller or fetched by `--latest` beforehand. @dataclass(frozen=True) @@ -867,6 +867,87 @@ def _resolved_source(source: Path): yield _extract_single_top_level_dir(source, Path(tmp)) +def _version_gate( + new_version: "version_mod.Version", + local_version: "version_mod.Version", + allow_pre: bool, + origin: str, +) -> bool: + """The three version refusals shared by a source tree and the feed: a + candidate without `--pre`, a downgrade, an equal version. `True` means go + on; `False` means the run ended successfully as a no-op. `origin` names + what reported the version, for the message.""" + if new_version.is_prerelease and not allow_pre: + fail( + f"{origin} reports {new_version}, a running candidate (-beta.N). " + "`.gitea/workflows/release.yml` never publishes one, so it can only come from a " + "dev checkout or a hand-made release - pass --pre if that is deliberate." + ) + if new_version < local_version: + fail(f"{origin} reports {new_version}, older than the installed {local_version} - refusing a downgrade.") + if new_version == local_version: + success(f"Already at {local_version}. Nothing to do.") + return False + return True + + +def _query_feed( + stamp: dict, + local_version: "version_mod.Version", + allow_pre: bool, + expect: Optional["version_mod.Version"], +) -> Optional[tuple["version_mod.LatestRelease", str, Optional[str]]]: + """Ask the release feed which release is latest and decide, before any + download, whether there is anything to fetch. Returns the release with the + feed URL and token, or `None` when the installed version is already it.""" + url = version_mod.update_url(stamp) + token = os.environ.get(version_mod.UPDATE_TOKEN_ENV, "").strip() or None + console.print(f"Asking the release feed {url}", highlight=False) + try: + release = version_mod.fetch_latest_assets(url, token) + except version_mod.VersionError as exc: + fail(str(exc)) + if expect is not None and release.version != expect: + fail( + f"The feed's latest release is {release.version}, not the expected {expect} - " + "nothing was downloaded. The feed moved on since the notes were read, or " + f"the wrong version was named: read `wikitool version notes` for {release.version} " + "before deciding to go there instead." + ) + if not _version_gate(release.version, local_version, allow_pre, "the release feed"): + return None + return release, url, token + + +@contextmanager +def _downloaded_source(release: "version_mod.LatestRelease", feed_url: str, token: Optional[str]): + """Yield the directory of the release the feed announced: download its + archive and checksum into a scratch directory, hold the archive to the + checksum (a missing checksum is an error, never a WARN), and unpack it. + A release without both assets is refused before the first download. The + scratch directory is removed on every exit, `--dry-run` included.""" + archive_url, sum_url = _asset_urls(release) + with tempfile.TemporaryDirectory(prefix="wikitool-upgrade-") as tmp: + archive = Path(tmp) / release.archive_name + checksum = archive.with_name(archive.name + version_mod.CHECKSUM_SUFFIX) + for url, dest in ((archive_url, archive), (sum_url, checksum)): + console.print(f"Downloading {url}", highlight=False) + try: + version_mod.download_asset(url, dest, version_mod.token_for_asset(feed_url, url, token)) + except version_mod.VersionError as exc: + fail(str(exc)) + _verify_sha256_sidecar(archive) + yield _extract_single_top_level_dir(archive, Path(tmp) / "unpacked") + + +def _asset_urls(release: "version_mod.LatestRelease") -> tuple[str, str]: + try: + return release.asset_urls() + except version_mod.VersionError as exc: + fail(str(exc)) + raise # unreachable: fail() raises typer.Exit + + def _git_working_tree_status() -> Optional[str]: """`git status --porcelain` for `config.ROOT`, or None if it is not a git repository at all - which is a valid, if unprotected, state for a tarball @@ -907,7 +988,7 @@ def _resolve_take_release( def _refusal_for_blocked( - source: Path, undecided: list[str], classification: FileClassification + target: str, undecided: list[str], classification: FileClassification ) -> str: """The abort text for blocked paths no flag has answered for. @@ -927,9 +1008,9 @@ def _refusal_for_blocked( f"{len(undecided)} locally changed file(s) (listed above) would be silently " f"overwritten. Nothing was written, and none of these three is the default:\n" f" - take the release's version and discard the local change:\n" - f" dist upgrade {rel_path(source)} --take-release {paths}\n" + f" dist upgrade {target} --take-release {paths}\n" f" - keep every local change and upgrade around them ({kept_again}):\n" - f" dist upgrade {rel_path(source)} --keep-local\n" + f" dist upgrade {target} --keep-local\n" f" - reconcile them by hand first, then re-run." ) @@ -984,8 +1065,8 @@ def _report_plan( path="dist upgrade", summary="Apply a stack update `dist export` produced - the write half of `version check`.", synopsis=(cli_contract.Variant( - usage="dist upgrade [--dry-run] [--keep-local] [--take-release ]... " - "[--prune] [--pre]", + usage="dist upgrade ( | --latest [--expect ]) [--dry-run] " + "[--keep-local] [--take-release ]... [--prune] [--pre]", ),), properties=cli_contract.Properties( effect=cli_contract.Effect.WRITE, @@ -994,12 +1075,28 @@ def _report_plan( "sequential file copy with no partial-state cleanup: an interruption mid-copy " "(killed process, disk full) can leave the tree part-old, part-new", budget=cli_contract.Budget.COUNTED, + network=cli_contract.Network.YES, ), notes=( - "Never downloads anything: `` is an already-fetched export directory or " - "`.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one " - "is present (a missing one is a WARN, not a block) and must unpack to exactly one " - "top-level directory - the shape `.gitea/workflows/release.yml` packs.", + "Exactly one of `` and `--latest` names the release. `` is an " + "already-fetched export directory or `.tar.gz` release archive and touches no network: " + "the archive is verified against a sibling `.sha256` if one is present (a missing one " + "is a WARN, not a block) and must unpack to exactly one top-level directory - the " + "shape `.gitea/workflows/release.yml` packs.", + "`--latest` asks the release feed (`update_url` of the local stamp, overridable with " + "`$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is " + "latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, " + "already installed - before any download, then downloads the archive and its " + "`.sha256` into a scratch directory removed on every exit. The checksum is mandatory " + "here (a release without one, or an archive that fails it, is an error, not a WARN) " + "and the archive's own `VERSION` must equal the feed's version. The asset URLs are the " + "feed's own `browser_download_url` values, and the token reaches an asset download only " + "if it is on the feed's origin. The checksum protects against transfer errors, not " + "against a feed that is itself compromised - authenticity is the trust in the feed's " + "host.", + "`--expect ` (only with `--latest`) pins the release the feed may announce: a " + "different latest version is refused before anything is downloaded. Pass the version " + "`wikitool version notes` was read for, in the dry run and in the real run alike.", "The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus " "what an export re-seeds from a blank template every time (`kb/log.md`, " "`raw/.gitkeep`) or seeds once and the instance owns from then on " @@ -1029,7 +1126,8 @@ def _report_plan( "Not being a git repository at all is a WARN, not a refusal.", "Never touches git - no commit, no push.", "`--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source " - "needs `--pre`.", + "needs `--pre`. With `--latest` it still downloads and verifies the archive - that is " + "the only way to classify - and removes it again.", "An interrupted write is not resumed automatically: compare the tree against the " "printed classification and finish or revert by hand.", "The closing report names `instructions/upgrade-instance.md`, which carries the order " @@ -1041,6 +1139,11 @@ def _report_plan( reaction="", code=0, ), + cli_contract.Failure( + cause="Both `` and `--latest`, or neither; or `--expect` without `--latest`", + reaction="Not transient - name exactly one source, and pass `--expect` only with " + "`--latest`", + ), cli_contract.Failure( cause="Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` " "block", @@ -1064,6 +1167,34 @@ def _report_plan( "one top-level directory", reaction="Fix the path or re-download the release archive, then retry", ), + cli_contract.Failure( + cause="`--latest`: the release feed cannot be reached, or answers with something " + "that is not a release", + reaction="Transient - retry once; then report the URL from the message, or take the " + "release page's archive by hand and pass it as ``", + ), + cli_contract.Failure( + cause="`--latest --expect`: the feed's latest release is another version than the " + "one expected; nothing was downloaded", + reaction="Not transient - read `wikitool version notes` for the version the message " + "names, then either expect that one or stop", + ), + cli_contract.Failure( + cause="`--latest`: the release publishes no archive or no `.sha256` under the " + "expected name; nothing was downloaded", + reaction="Not transient - the message lists the assets present and the release page; " + "report it there rather than upgrading without the checksum", + ), + cli_contract.Failure( + cause="`--latest`: an asset download fails, or the archive fails its `.sha256`", + reaction="Retry once; if it fails again, report the exact message - do not fall back " + "to an unchecked archive", + ), + cli_contract.Failure( + cause="`--latest`: the downloaded archive's `VERSION` is not the version the feed " + "announced", + reaction="Not transient - an inconsistent release; report it against the release page", + ), cli_contract.Failure( cause="The source carries no `VERSION`, `.wikitool-release.json` or `files` block", reaction="Point `` at a distribution export, then retry", @@ -1091,12 +1222,14 @@ def _report_plan( "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run", "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz", "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md", + "tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run", ), never=( "Never treat any of the three answers to locally changed files as the default.", ), see_also=( "`wikitool version check` - finds out whether an update exists", + "`wikitool version notes` - the notes of the release `--expect` should name", "`instructions/upgrade-instance.md` - the order after the swap", "`INSTALL.md` § \"Version und Updates\" - which release, whether to take it, where " "the tarball comes from", @@ -1106,8 +1239,17 @@ def _report_plan( )) @app.command("upgrade") def upgrade_command( - source: Path = typer.Argument( - ..., help="An extracted distribution directory, or a release .tar.gz archive" + source: Optional[Path] = typer.Argument( + None, help="An extracted distribution directory, or a release .tar.gz archive - or use --latest" + ), + latest: bool = typer.Option( + False, "--latest", + help="Download the release the feed announces as latest instead of naming a ", + ), + expect: Optional[str] = typer.Option( + None, "--expect", + help="With --latest: refuse, before downloading anything, unless the feed's latest release " + "is this version (the one `version notes` was read for)", ), dry_run: bool = typer.Option( False, "--dry-run", help="Classify and report, without writing anything" @@ -1133,8 +1275,10 @@ def upgrade_command( ): """Apply a stack update `dist export` produced - the write half of `version check`. \f - Never downloads anything: `source` is an already-fetched - export directory or `.tar.gz` archive. Writes exactly the new release + `source` is an already-fetched export directory or `.tar.gz` archive; + `--latest` instead downloads the archive and its checksum from the + release feed (`--expect ` pins which release that may be) and + holds it to the checksum before anything else. Writes exactly the new release stamp's `files` block, minus what an export re-seeds every time (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md`), classifying every candidate @@ -1155,19 +1299,41 @@ def upgrade_command( take_release=take_release, prune=prune, allow_pre=allow_pre, + latest=latest, + expect=expect, ) def run_upgrade( - source: Path, + source: Optional[Path] = None, dry_run: bool = False, keep_local: bool = False, take_release: Optional[Sequence[str]] = None, prune: bool = False, allow_pre: bool = False, + latest: bool = False, + expect: Optional[str] = None, ) -> None: """The upgrade itself, free of Typer's option objects - see `run_export` for why this split exists.""" + if latest == (source is not None): + fail( + "Name exactly one source: `` (an export directory or a release .tar.gz " + "already on disk) or `--latest` (the release the feed announces). " + + ("Both were given." if latest else "Neither was given.") + ) + return + expected_version = None + if expect is not None: + if not latest: + fail("--expect only makes sense with --latest: it pins the release the feed may announce.") + return + try: + expected_version = version_mod.Version.parse(expect) + except version_mod.VersionError as exc: + fail(f"--expect: {exc}") + return + try: local_version = version_mod.read_version() except version_mod.VersionError as exc: @@ -1223,7 +1389,19 @@ def run_upgrade( ) return - with _resolved_source(source) as new_root: + feed = None + if latest: + feed = _query_feed(old_stamp, local_version, allow_pre, expected_version) + if feed is None: + return + release, feed_url, feed_token = feed + source_cm = _downloaded_source(release, feed_url, feed_token) + target = f"--latest --expect {release.version}" + else: + source_cm = _resolved_source(source) + target = rel_path(source) + + with source_cm as new_root: version_path = new_root / version_mod.VERSION_FILENAME if not version_path.is_file(): fail(f"{rel_path(new_root)} has no VERSION - not a distribution export.") @@ -1234,18 +1412,16 @@ def run_upgrade( fail(str(exc)) return - if new_version.is_prerelease and not allow_pre: + if feed is not None and new_version != feed[0].version: fail( - f"{new_version} is a running candidate (-beta.N). `.gitea/workflows/release.yml` " - "never publishes one, so a candidate tree can only come from a dev checkout by " - "hand - pass --pre if that is deliberate." + f"The downloaded archive's VERSION says {new_version}, but the feed announced " + f"{feed[0].version} (tag {feed[0].tag}) - this release is inconsistent, so " + "nothing was applied. Report it against the release page" + + (f" {feed[0].html_url}" if feed[0].html_url else "") + + "." ) return - if new_version < local_version: - fail(f"{new_version} is older than the installed {local_version} - refusing a downgrade.") - return - if new_version == local_version: - success(f"Already at {local_version}. Nothing to do.") + if not _version_gate(new_version, local_version, allow_pre, "the source"): return stamp_path = new_root / version_mod.RELEASE_STAMP_FILENAME @@ -1289,7 +1465,7 @@ def run_upgrade( undecided = [path for path in classification.blocked if path not in taken] if undecided and not keep_local: - fail(_refusal_for_blocked(source, undecided, classification)) + fail(_refusal_for_blocked(target, undecided, classification)) return to_write = sorted(classification.unchanged + classification.new + sorted(taken)) diff --git a/tools/chemenu/commands/version_cmd.py b/tools/chemenu/commands/version_cmd.py index fcc7a3b..0ed1198 100644 --- a/tools/chemenu/commands/version_cmd.py +++ b/tools/chemenu/commands/version_cmd.py @@ -29,6 +29,9 @@ number means, and `instructions/dev/version-parts.md` for the candidate model): stderr before asking and takes `--offline`. Both need no key, both time out, and a feed that cannot be reached is reported as an error rather than silently answered as "up to date" or "no notes". +- `dist upgrade --latest` reaches the same feed as a third, equally explicit + caller: it asks which release is latest, then downloads that release's + assets. Only `version_mod` talks to the feed; this module is not involved. """ from __future__ import annotations @@ -158,8 +161,10 @@ def show_command( "is `current`, `update`, `migration` (the step crosses a compatibility boundary) or " "`ahead`.", "The only command whose whole job is the network call - `version notes` reaches the " - "same feed too, but only as a fallback on a distributed instance.", - "Never reached implicitly from another command, needs no key, and times out after " + "same feed too, but only as a fallback on a distributed instance, and `dist upgrade " + "--latest` asks it which release to download.", + "Never reached implicitly from another command (`dist upgrade` asks only when passed " + "`--latest`), needs no key, and times out after " "`--timeout` seconds (default 10).", "The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the " "built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable " diff --git a/tools/chemenu/tests/test_cli.py b/tools/chemenu/tests/test_cli.py index 5a670d9..f950585 100644 --- a/tools/chemenu/tests/test_cli.py +++ b/tools/chemenu/tests/test_cli.py @@ -337,7 +337,8 @@ def test_every_gated_record_shows_its_re_run_after_exit_42(): def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkout(): """Gitea #144: `network:` means *any* reach outside this checkout - an HTTP call `wikitool` makes itself, or a git operation against a remote (fetch/ls-remote/push) - - not only the two `version_cmd.py` used to claim exclusivity for. Pinned as an explicit + not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list + since `--latest`, which asks the release feed and downloads from it. Pinned as an explicit set so a command gaining or losing that reach is a deliberate edit here, not a silent drift between the property and what the command actually does.""" expected = { @@ -346,6 +347,7 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou "upstream merge", "version check", "version notes", + "dist upgrade", "review", "task new", "task list", diff --git a/tools/chemenu/tests/test_dist_upgrade.py b/tools/chemenu/tests/test_dist_upgrade.py index 05e23e6..e2e16f8 100644 --- a/tools/chemenu/tests/test_dist_upgrade.py +++ b/tools/chemenu/tests/test_dist_upgrade.py @@ -4,9 +4,13 @@ execution, and every refusal before anything is written. See Gitea #7.""" from __future__ import annotations import hashlib +import http.server import json +import shutil import subprocess import tarfile +import tempfile +import threading from pathlib import Path import pytest @@ -586,3 +590,426 @@ def test_tarball_sha256_sidecar_mismatch_is_refused(instance, tmp_path): with pytest.raises(typer.Exit): dist_cmd.run_upgrade(archive) + + +# --- --latest: the release feed --------------------------------------------- +# +# A real `http.server` on 127.0.0.1 rather than a fetcher stub: the download +# path is `urllib` streaming into a file, and the same-origin token rule is a +# statement about which requests carry which header - both only observable on +# the wire. Every server records `(path, Authorization)` for each request. + + +class FeedServer: + def __init__(self) -> None: + self.routes: dict[str, tuple[int, bytes]] = {} + self.requests: list[tuple[str, str | None]] = [] + outer = self + + class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): # noqa: N802 - http.server's name + outer.requests.append((self.path, self.headers.get("Authorization"))) + status, body = outer.routes.get(self.path, (404, b"not found")) + self.send_response(status) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): # silence the test output + pass + + self.httpd = http.server.HTTPServer(("127.0.0.1", 0), Handler) + self.thread = threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True) + self.thread.start() + + @property + def base(self) -> str: + return f"http://127.0.0.1:{self.httpd.server_address[1]}" + + def stop(self) -> None: + self.httpd.shutdown() + self.httpd.server_close() + + def paths(self) -> list[str]: + return [path for path, _ in self.requests] + + def asset_paths(self) -> list[str]: + return [path for path in self.paths() if path.startswith("/assets/")] + + +@pytest.fixture +def feed(monkeypatch: pytest.MonkeyPatch): + server = FeedServer() + monkeypatch.setenv("WIKITOOL_UPDATE_URL", f"{server.base}/feed") + yield server + server.stop() + + +@pytest.fixture +def other_origin(): + server = FeedServer() + yield server + server.stop() + + +@pytest.fixture +def scratch_tmp(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """Redirect `tempfile` into a directory the test can inspect afterwards.""" + scratch = tmp_path / "scratch-tmp" + scratch.mkdir() + monkeypatch.setattr(tempfile, "tempdir", str(scratch)) + return scratch + + +def _publish( + feed: FeedServer, + tmp_path: Path, + version: str, + files: dict[str, str] | None = None, + *, + tag: str | None = None, + inner_version: str | None = None, + checksum: str | None = "ok", + archive: bool = True, + asset_server: FeedServer | None = None, +) -> str: + """Serve `version` as the feed's latest release and return its archive name. + + `inner_version` is what the archive's own VERSION says (default: the same), + `checksum` is `"ok"`, `"wrong"` or `None` for no `.sha256` asset.""" + name = f"chemenu-stack-{version}" + tree = _release( + tmp_path, f"served-{name}", inner_version or version, + files or {"AGENTS.md": "core v2\n", "tools/wikitool": "#!/bin/sh\n"}, + ) + packed = tmp_path / f"{name}.tar.gz" + with tarfile.open(packed, "w:gz") as tf: + tf.add(tree, arcname=name) + data = packed.read_bytes() + digest = hashlib.sha256(data).hexdigest() if checksum == "ok" else "0" * 64 + holder = asset_server or feed + assets = [] + if archive: + holder.routes[f"/assets/{name}.tar.gz"] = (200, data) + assets.append({"name": f"{name}.tar.gz", "browser_download_url": f"{holder.base}/assets/{name}.tar.gz"}) + if checksum is not None: + holder.routes[f"/assets/{name}.tar.gz.sha256"] = (200, f"{digest} {name}.tar.gz\n".encode()) + assets.append({ + "name": f"{name}.tar.gz.sha256", + "browser_download_url": f"{holder.base}/assets/{name}.tar.gz.sha256", + }) + feed.routes["/feed"] = ( + 200, + json.dumps({ + "tag_name": tag or f"v{version}", + "html_url": f"{feed.base}/releases/tag/v{version}", + "assets": assets, + }).encode(), + ) + return name + + +def _stamp_version(instance: Path) -> str: + return json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["version"] + + +def test_latest_downloads_verifies_and_applies_the_feeds_release(instance, tmp_path, feed, scratch_tmp): + name = _publish(feed, tmp_path, "1.1.0") + + dist_cmd.run_upgrade(latest=True) + + assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core v2\n" + assert _stamp_version(instance) == "1.1.0" + assert feed.paths() == ["/feed", f"/assets/{name}.tar.gz", f"/assets/{name}.tar.gz.sha256"] + assert list(scratch_tmp.iterdir()) == [] + recorded = json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["files"] + assert recorded + for relative, digest in recorded.items(): + assert _digest((instance / relative).read_text(encoding="utf-8")) == digest + + +def test_latest_yields_the_same_tree_as_the_archive_it_downloaded(instance, tmp_path, feed, monkeypatch): + name = _publish(feed, tmp_path, "1.1.0") + served = tmp_path / f"{name}.tar.gz" + snapshot = tmp_path / "instance-copy" + shutil.copytree(instance, snapshot) + + dist_cmd.run_upgrade(latest=True) + via_feed = {p.relative_to(instance): p.read_bytes() for p in instance.rglob("*") if p.is_file()} + + monkeypatch.setattr(config, "ROOT", snapshot) + dist_cmd.run_upgrade(served) + via_archive = {p.relative_to(snapshot): p.read_bytes() for p in snapshot.rglob("*") if p.is_file()} + + assert via_feed == via_archive + + +def test_latest_dry_run_downloads_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp): + name = _publish(feed, tmp_path, "1.1.0") + + dist_cmd.run_upgrade(latest=True, dry_run=True) + + assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n" + assert _stamp_version(instance) == "1.0.0" + assert f"/assets/{name}.tar.gz" in feed.paths() + assert list(scratch_tmp.iterdir()) == [] + + +def test_latest_already_current_is_a_no_op_without_any_download(instance, tmp_path, feed): + _publish(feed, tmp_path, "1.0.0") + + dist_cmd.run_upgrade(latest=True) + + assert feed.paths() == ["/feed"] + assert _stamp_version(instance) == "1.0.0" + + +def test_latest_older_than_installed_is_a_downgrade_refusal_before_any_download(instance, tmp_path, feed, capsys): + _publish(feed, tmp_path, "0.9.0") + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + assert "downgrade" in capsys.readouterr().out + assert feed.asset_paths() == [] + + +def test_latest_prerelease_needs_pre_and_is_refused_before_any_download(instance, tmp_path, feed): + _publish(feed, tmp_path, "1.1.0-beta.1") + + with pytest.raises(typer.Exit): + dist_cmd.run_upgrade(latest=True) + assert feed.asset_paths() == [] + + dist_cmd.run_upgrade(latest=True, allow_pre=True) + assert _stamp_version(instance) == "1.1.0-beta.1" + + +@pytest.mark.parametrize("missing", ["checksum", "archive"]) +def test_latest_release_without_an_asset_is_refused_before_any_download( + instance, tmp_path, feed, capsys, missing +): + _publish( + feed, tmp_path, "1.1.0", + checksum=None if missing == "checksum" else "ok", + archive=missing != "archive", + ) + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + out = " ".join(capsys.readouterr().out.split()) + assert "chemenu-stack-1.1.0.tar.gz" in out + assert "assets present" in out + assert "/releases/tag/v1.1.0" in out + assert feed.asset_paths() == [] + assert _stamp_version(instance) == "1.0.0" + + +def test_latest_checksum_mismatch_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp, capsys): + _publish(feed, tmp_path, "1.1.0", checksum="wrong") + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + assert "does not match" in " ".join(capsys.readouterr().out.split()) + assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n" + assert _stamp_version(instance) == "1.0.0" + assert list(scratch_tmp.iterdir()) == [] + + +def test_latest_archive_version_that_differs_from_the_feed_is_refused(instance, tmp_path, feed, scratch_tmp, capsys): + _publish(feed, tmp_path, "1.1.0", inner_version="1.2.0") + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + out = " ".join(capsys.readouterr().out.split()) + assert "1.2.0" in out and "1.1.0" in out + assert _stamp_version(instance) == "1.0.0" + assert list(scratch_tmp.iterdir()) == [] + + +def test_latest_asset_download_error_is_reported_with_its_url(instance, tmp_path, feed, capsys): + name = _publish(feed, tmp_path, "1.1.0") + del feed.routes[f"/assets/{name}.tar.gz"] + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + assert f"/assets/{name}.tar.gz answered HTTP 404" in " ".join(capsys.readouterr().out.split()) + assert _stamp_version(instance) == "1.0.0" + + +def test_latest_unreachable_feed_is_an_error_naming_the_url(instance, monkeypatch, capsys): + monkeypatch.setenv("WIKITOOL_UPDATE_URL", "http://127.0.0.1:1/feed") + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True) + + assert excinfo.value.exit_code == 1 + assert "Could not reach http://127.0.0.1:1/feed" in " ".join(capsys.readouterr().out.split()) + + +def test_latest_respects_the_local_preconditions_before_asking_the_feed(instance, tmp_path, feed): + _publish(feed, tmp_path, "1.1.0") + (instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8") + subprocess.run(["git", "init", "-q"], cwd=instance, check=True) + + with pytest.raises(typer.Exit): + dist_cmd.run_upgrade(latest=True) + + assert feed.requests == [] + + +def test_latest_blocked_refusal_names_the_pinned_latest_invocation(instance, tmp_path, feed, capsys): + _publish(feed, tmp_path, "1.1.0") + (instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8") + + with pytest.raises(typer.Exit): + dist_cmd.run_upgrade(latest=True) + + out = " ".join(capsys.readouterr().out.split()) + assert "dist upgrade --latest --expect 1.1.0 --take-release AGENTS.md" in out + assert "dist upgrade --latest --expect 1.1.0 --keep-local" in out + + +def test_source_and_latest_together_or_neither_is_refused_before_any_check(instance, tmp_path, feed): + release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"}) + + with pytest.raises(typer.Exit) as both: + dist_cmd.run_upgrade(release, latest=True) + with pytest.raises(typer.Exit) as neither: + dist_cmd.run_upgrade() + + assert both.value.exit_code == 1 and neither.value.exit_code == 1 + assert feed.requests == [] + + +# --- --expect --------------------------------------------------------------- + + +@pytest.mark.parametrize("spelling", ["1.1.0", "v1.1.0"]) +def test_expect_matching_the_feed_proceeds(instance, tmp_path, feed, spelling): + _publish(feed, tmp_path, "1.1.0") + + dist_cmd.run_upgrade(latest=True, expect=spelling) + + assert _stamp_version(instance) == "1.1.0" + + +def test_expect_mismatch_is_refused_before_any_download_naming_both_versions(instance, tmp_path, feed, capsys): + _publish(feed, tmp_path, "1.2.0") + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(latest=True, expect="1.1.0") + + assert excinfo.value.exit_code == 1 + out = " ".join(capsys.readouterr().out.split()) + assert "1.2.0" in out and "1.1.0" in out and "version notes" in out + assert feed.asset_paths() == [] + assert _stamp_version(instance) == "1.0.0" + + +def test_expect_mismatch_wins_over_an_already_current_install(instance, tmp_path, feed): + _publish(feed, tmp_path, "1.0.0") + + with pytest.raises(typer.Exit): + dist_cmd.run_upgrade(latest=True, expect="1.1.0") + + +def test_expect_without_latest_is_refused(instance, tmp_path, feed): + release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"}) + + with pytest.raises(typer.Exit) as excinfo: + dist_cmd.run_upgrade(release, expect="1.1.0") + + assert excinfo.value.exit_code == 1 + assert feed.requests == [] + + +def test_expect_that_is_not_a_version_is_refused(instance, feed): + with pytest.raises(typer.Exit): + dist_cmd.run_upgrade(latest=True, expect="latest") + assert feed.requests == [] + + +# --- the token -------------------------------------------------------------- + + +def test_token_goes_to_the_feed_and_same_origin_assets_only(instance, tmp_path, feed, other_origin, monkeypatch): + monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret") + name = _publish(feed, tmp_path, "1.1.0") + + dist_cmd.run_upgrade(latest=True) + + assert {auth for _, auth in feed.requests} == {"token s3cret"} + assert f"/assets/{name}.tar.gz" in feed.paths() + assert other_origin.requests == [] + + +def test_token_is_withheld_from_an_asset_on_another_origin(instance, tmp_path, feed, other_origin, monkeypatch): + monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret") + _publish(feed, tmp_path, "1.1.0", asset_server=other_origin) + + dist_cmd.run_upgrade(latest=True) + + assert _stamp_version(instance) == "1.1.0" + assert feed.requests == [("/feed", "token s3cret")] + assert len(other_origin.requests) == 2 + assert {auth for _, auth in other_origin.requests} == {None} + + +@pytest.mark.parametrize( + "feed_url, asset_url, sent", + [ + ("https://git.example/api/v1/releases/latest", "https://git.example/dl/a.tar.gz", True), + ("https://git.example/api/v1/releases/latest", "https://git.example:443/dl/a.tar.gz", True), + ("https://git.example/api/v1/releases/latest", "http://git.example/dl/a.tar.gz", False), + ("https://git.example/api/v1/releases/latest", "https://cdn.example/dl/a.tar.gz", False), + ("http://git.example:3000/x", "http://git.example:3001/x", False), + ], +) +def test_token_for_asset_compares_scheme_host_and_port(feed_url, asset_url, sent): + assert version_mod.token_for_asset(feed_url, asset_url, "t") == ("t" if sent else None) + assert version_mod.token_for_asset(feed_url, asset_url, None) is None + + +# --- the asset names -------------------------------------------------------- + + +def test_release_asset_names_match_the_workflow(): + workflow = Path(__file__).resolve().parents[3] / ".gitea" / "workflows" / "release.yml" + if not workflow.is_file(): + pytest.skip("a distributed instance carries no release workflow") + text = workflow.read_text(encoding="utf-8") + + archive = version_mod.ARCHIVE_NAME.format(version="${VERSION}") + stem = archive.removesuffix(".tar.gz") + assert f'name="{stem}"' in text + assert archive.replace(stem, "${name}") in text + assert (archive + version_mod.CHECKSUM_SUFFIX).replace(stem, "${name}") in text + + +def test_fetch_latest_assets_reads_urls_off_the_release_and_skips_malformed_entries(): + payload = json.dumps({ + "tag_name": "v1.1.0", + "assets": [ + {"name": "a.tar.gz", "browser_download_url": "https://h/a"}, + {"name": "no-url"}, + "not an object", + ], + }).encode() + + release = version_mod.fetch_latest_assets("https://h/feed", fetcher=lambda u, t, to: payload) + + assert release.version == version_mod.Version.parse("1.1.0") + assert release.tag == "v1.1.0" + assert release.assets == {"a.tar.gz": "https://h/a"} + with pytest.raises(version_mod.VersionError, match="assets present: a.tar.gz"): + release.asset_urls() diff --git a/tools/chemenu/version.py b/tools/chemenu/version.py index d9ee03e..76e0266 100644 --- a/tools/chemenu/version.py +++ b/tools/chemenu/version.py @@ -37,7 +37,9 @@ import functools import json import os import re +import shutil import urllib.error +import urllib.parse import urllib.request from dataclasses import dataclass from pathlib import Path @@ -349,12 +351,15 @@ def fetch_latest( The network call sits behind `fetcher` so every caller above this line - and every test - can run without a network. This is the one place that - talks to the **release feed**, and only two commands reach it: `version - check`, whose whole job it is, and `version notes` on a *distributed* + talks to the **release feed**, and only three commands reach it: `version + check`, whose whole job it is, `version notes` on a *distributed* instance, whose local `CHANGES.md` is a stub with no entry to print (see - `fetch_latest_notes`). Neither is implicit - `check` exists for the call, - and `notes` announces the URL it is asking before it asks, on stderr, and - takes `--offline` for a caller that wants none of it. + `fetch_latest_notes`), and `dist upgrade --latest`, which asks the feed + which release to download (see `fetch_latest_assets`). None is implicit - + `check` exists for the call, `notes` announces the URL it is asking before + it asks, on stderr, and takes `--offline` for a caller that wants none of + it, and `upgrade` reaches the feed only when the operator passes + `--latest`. """ fetch = fetcher or _urlopen_fetch try: @@ -462,6 +467,120 @@ def fetch_latest_notes( return version, body, (str(html_url) if html_url else None) +# --- release assets -------------------------------------------------------- +# +# `dist upgrade --latest` downloads the two assets `.gitea/workflows/release.yml` +# attaches to every release. The names are that workflow's `name=` line plus +# `.tar.gz` / `.tar.gz.sha256`; `test_release_asset_names_match_the_workflow` +# ties this constant to the workflow so one cannot move without the other. + +ARCHIVE_NAME = "chemenu-stack-{version}.tar.gz" +CHECKSUM_SUFFIX = ".sha256" +DOWNLOAD_TIMEOUT = 60.0 + + +@dataclass(frozen=True) +class LatestRelease: + """What the feed's latest release offers: its version, its tag as the feed + spells it, its own page, and every asset as `name -> download URL`.""" + + version: Version + tag: str + html_url: Optional[str] + assets: dict[str, str] + + @property + def archive_name(self) -> str: + return ARCHIVE_NAME.format(version=self.version) + + def asset_urls(self) -> tuple[str, str]: + """`(archive URL, checksum URL)`, read off the release object and never + composed. Raises `VersionError` naming the release page when either is + missing - an upgrade without its checksum is refused, not downgraded to + an unchecked one.""" + archive = self.archive_name + wanted = (archive, archive + CHECKSUM_SUFFIX) + missing = [name for name in wanted if name not in self.assets] + if missing: + present = ", ".join(sorted(self.assets)) or "none" + page = f" - see the release page {self.html_url}" if self.html_url else "" + raise VersionError( + f"release {self.tag} does not publish {' and '.join(missing)} " + f"(assets present: {present}){page}" + ) + return self.assets[wanted[0]], self.assets[wanted[1]] + + +def fetch_latest_assets( + url: str, + token: Optional[str] = None, + timeout: float = 10.0, + fetcher: Optional[Fetcher] = None, +) -> LatestRelease: + """`fetch_latest`'s version plus the release's assets, from the one + response - the asset URLs are the feed's own `browser_download_url` + values, never a path this code composes (AGENTS.md invariant 7).""" + version, captured = _fetch_latest_object(url, token, timeout, fetcher) + assets: dict[str, str] = {} + raw_assets = captured.get("assets") + if isinstance(raw_assets, list): + for entry in raw_assets: + if not isinstance(entry, dict): + continue + name = str(entry.get("name") or "").strip() + download = str(entry.get("browser_download_url") or "").strip() + if name and download: + assets[name] = download + html_url = captured.get("html_url") or captured.get("url") + return LatestRelease( + version=version, + tag=str(captured.get("tag_name") or "").strip(), + html_url=str(html_url) if html_url else None, + assets=assets, + ) + + +def _origin(url: str) -> tuple[str, str, int]: + parts = urllib.parse.urlsplit(url) + scheme = parts.scheme.lower() + default = {"http": 80, "https": 443}.get(scheme, 0) + return scheme, (parts.hostname or "").lower(), parts.port or default + + +def token_for_asset(feed_url: str, asset_url: str, token: Optional[str]) -> Optional[str]: + """The token to send with an asset download: `token` when the asset is on + the feed's own origin (scheme, host, port), `None` otherwise. An asset URL + is data from a JSON answer, so the credential does not follow it to + another host.""" + return token if token and _origin(feed_url) == _origin(asset_url) else None + + +def download_asset( + url: str, + dest: Path, + token: Optional[str] = None, + timeout: float = DOWNLOAD_TIMEOUT, +) -> None: + """Stream `url` into `dest`. `timeout` is a socket timeout per read, not a + deadline for the whole transfer. The token goes out as an *unredirected* + header, so a redirect to another host does not carry it along. Every + failure is a `VersionError` naming the URL.""" + request = urllib.request.Request(url) + if token: + request.add_unredirected_header("Authorization", f"token {token}") + try: + with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310 - URL from the feed + with dest.open("wb") as out: + shutil.copyfileobj(response, out) + except urllib.error.HTTPError as exc: + hint = "" + if exc.code in (401, 403): + hint = f" - the download needs authentication; set ${UPDATE_TOKEN_ENV}" + raise VersionError(f"{url} answered HTTP {exc.code}{hint}") from exc + except (urllib.error.URLError, OSError, TimeoutError) as exc: + raise VersionError(f"Could not download {url}: {exc}") from exc + + # --- CHANGES.md ------------------------------------------------------------ # # The changelog is prose and stays the author's job. What is mechanical is the