feat: dist upgrade --latest downloads and verifies the release from the feed, --expect pins the version (#161)
CI / verify (push) Successful in 1m42s
Release / release (push) Successful in 38s

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- instructions/upgrade-instance.md
- tools/CONTRACT.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/version_cmd.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_dist_upgrade.py
- tools/chemenu/version.py
This commit is contained in:
torben committed 2026-09-29 22:08:00 +02:00
1 parent dd885db625
commit cf892315e6
10 files changed
+888 -62

No files matched your search

+46 -1
View File
@@ -59,13 +59,14 @@ concern - readable here, never shipped as something to parse.
---
## 7.1.0-beta.31 - 2026-09-26 - new_page/type_resolver comments no longer claim only entities declare a layout:
## 7.1.0-beta.32 - 2026-09-29 - dist upgrade --latest: one-command update from the release feed
**Author:** Torben Nehmer
<!-- wikitool:bumps -->
**High impact**
- wikitool: one data record per command - `-h`, index and CONTRACT.md render from cli_contract (Gitea #121 Phase 1)
- dist upgrade --latest: one-command update from the release feed
**Medium impact**
- CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
@@ -102,6 +103,50 @@ concern - readable here, never shipped as something to parse.
- new_page/type_resolver comments no longer claim only entities declare a layout:
<!-- /wikitool:bumps -->
### dist upgrade --latest: one-command update from the release feed (Gitea #161)
Updating a tarball instance took a manual detour: `version notes`, then fetching the `.tar.gz` and
its `.sha256` from the release page by hand, then `dist upgrade <tarball>`. `dist upgrade --latest`
now does the middle part. It asks the release feed (`update_url` from the stamp,
`$WIKITOOL_UPDATE_URL` and `$WIKITOOL_UPDATE_TOKEN` as before) which release is latest, downloads
the release's archive and checksum into a scratch directory, verifies the archive against the
checksum, and hands over to the existing upgrade path unchanged - classification,
`--keep-local`/`--take-release`, migration report and every refusal are the same code. `<source>`
becomes optional; exactly one of it and `--latest` must be given.
The order is what matters. The local preconditions run first, then the feed is asked, and the
version it reports is judged **before any download**: already installed is a success no-op, an
older version is a downgrade refusal, a `-beta.N` version needs `--pre`. Only then are the two
assets looked up - by exact name, `chemenu-stack-<version>.tar.gz` and its `.sha256`, in the
`assets` of the release object the version query already returned, using the feed's own
`browser_download_url` values; no URL is composed. A release missing either asset is refused
before the first download with the assets it does have and its release page named. The checksum
is mandatory on this path (a `<source>` archive without a sibling `.sha256` is still only a
WARN), and the archive's own `VERSION` must equal the feed's version, otherwise nothing is
applied. `--dry-run` downloads and verifies too - classification needs the tree - and removes
everything again; the scratch directory goes away on every exit.
`--expect <version>` (only with `--latest`) closes the gap between reading the notes and applying
the update: the feed only offers its *latest* release, so if a newer one appeared since
`version notes` was read, the run refuses before downloading anything and names both versions.
`instructions/upgrade-instance.md` passes the version from step 2 in the dry run and in the real
run. The comparison is on parsed versions, so a `v` prefix does not matter.
`$WIKITOOL_UPDATE_TOKEN` is sent to an asset download only when the asset URL has the feed's
scheme, host and port, and as an unredirected header, so a redirect cannot carry it elsewhere.
There is no https enforcement: the checksum comes from the same host as the archive and protects
against transfer errors, not against a compromised feed - `INSTALL.md` says so.
The asset names are a Python constant (`version.ARCHIVE_NAME`) that a test ties to
`.gitea/workflows/release.yml`, so renaming them in one place fails the suite instead of the next
upgrade. `dist upgrade` is now `network: yes` in its record and in the pinned set in `test_cli.py`,
the `version check` record's "never reached implicitly" note names the one explicit exception, the
`fetch_latest` docstring says "three commands", and `tools/CONTRACT.md` is regenerated. The tests
run against a local `http.server` that logs every request and its `Authorization` header, so the
token rule and the "no asset request before the checks pass" rule are asserted on the wire. The
`upstream merge` pointers in the `dist upgrade` record and failure message stay as they are; they
belong to the separate work on the clone path.
### new_page/type_resolver comments no longer claim only entities declare a layout:
Two code comments - `new_page.py`'s module docstring and `TypeResolver.get_layout`'s - still said
+37 -12
View File
@@ -152,12 +152,14 @@ tools/wikitool version # was läuft hier, und woher kommt es
tools/wikitool version check # gibt es ein neueres Release?
```
`version check` und `version notes` sind die einzigen Befehle, die ins Netz gehen, und beide
fragen denselben Release-Feed der Ursprungs-Instanz (`$WIKITOOL_UPDATE_URL` überschreibt; sonst
der Wert aus dem Stamp). `version check` ist dafür da; `version notes` greift nur dann darauf
zurück, wenn die lokale `CHANGES.md` den Eintrag nicht hat - auf einer Instanz also immer, siehe
unten - und sagt vorher auf stderr, welche URL es fragt. Ein nicht erreichbarer Feed wird als
Fehler gemeldet - **nie** als „aktuell" und nie als „keine Notes".
`version check`, `version notes` und `dist upgrade --latest` sind die einzigen Befehle, die
ins Netz gehen, und alle drei fragen denselben Release-Feed der Ursprungs-Instanz
(`$WIKITOOL_UPDATE_URL` überschreibt; sonst der Wert aus dem Stamp). `version check` ist dafür da;
`version notes` greift nur dann darauf zurück, wenn die lokale `CHANGES.md` den Eintrag nicht
hat - auf einer Instanz also immer, siehe unten - und sagt vorher auf stderr, welche URL es
fragt; `dist upgrade` fragt den Feed nur, wenn `--latest` dasteht, und lädt dann auch das Release
herunter (siehe „Eine Instanz aktualisieren"). Ein nicht erreichbarer Feed wird als Fehler
gemeldet - **nie** als „aktuell" und nie als „keine Notes".
**Was die Versionsnummer aussagt:** kompatibel ist, was in der *linkesten von Null
verschiedenen Stelle* übereinstimmt. `0.1.3 → 0.1.4` ist ein sicheres Update, `0.1.3 → 0.2.0`
@@ -203,9 +205,32 @@ Agent-Sitzung ausgeführt wird; eine zweite Fassung derselben Schrittfolge an di
genau die Kopie, die irgendwann auseinanderläuft. Wer den Lauf selbst fahren will, liest dieselbe
Datei.
Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, woher
der Tarball kommt (§ „Version und Updates" und Weg A oben) - und der eine Sonderfall, den die
Instruktion nicht abdecken kann, weil es sie dort noch nicht gibt:
Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, und
wem die Instanz als Quelle vertraut - und zwei Sonderfälle, die die Instruktion nicht abdecken
kann, weil es sie dort noch nicht gibt.
**Das Release kommt mit einem Befehl.** `tools/wikitool dist upgrade --latest --expect <version>`
fragt den Release-Feed, lädt Tarball und `.sha256` in ein Arbeitsverzeichnis, prüft den Tarball
gegen die Summe und wendet ihn an; das Arbeitsverzeichnis verschwindet bei jedem Ausgang wieder,
auch bei `--dry-run`. `<version>` ist die, die `version notes` gedruckt hat: der Feed kennt nur
sein *neuestes* Release, und `--expect` verweigert den Lauf **vor** dem Download, wenn inzwischen
ein neueres erschienen ist, statt es ungelesen einzuspielen. Beides zusammen entscheidet vor dem
Download über „schon aktuell", Downgrade und Vor-Release (`-beta.N` braucht `--pre`); fehlt dem
Release der Tarball oder die Summe, bricht der Befehl vor dem ersten Download ab und nennt die
Release-Seite. Wer offline arbeitet oder einen Tarball vom Betreiber bekommen hat, gibt statt
`--latest` weiter die Datei an (`dist upgrade <tarball>`).
**Was die Prüfsumme leistet - und was nicht.** Die `.sha256` liegt beim selben Feed wie der
Tarball. Sie schützt vor einer beschädigten Übertragung, nicht vor einem Feed, der selbst
kompromittiert ist: die Echtheit eines Releases beruht auf dem Vertrauen in den Host, dessen
Feed die Instanz fragt (`update_url` im Stamp). Es gibt keinen https-Zwang; wer einen
`http://`-Feed konfiguriert, tut das bewusst. `$WIKITOOL_UPDATE_TOKEN` geht nur an Downloads
auf demselben Host wie der Feed.
**Beim ersten Sprung auf ein Release, das `--latest` kennt, gibt es die Option in der Instanz
noch nicht** - die Instruktion, die dort steht, gehört zum Release, das die Instanz verlässt.
Dann den Tarball einmal von Hand holen (Weg A oben), prüfen und `dist upgrade <tarball>` geben;
ab dem Release danach trägt die Instanz `--latest` selbst.
**Beim ersten Sprung auf `4.5.0` oder höher gibt es `dist upgrade` in der Instanz noch nicht** -
es kam erst mit `4.5.0`. Dann das Werkzeug aus dem entpackten *neuen* Tarball verwenden, gegen die
@@ -233,9 +258,9 @@ sind. Einer Instanz, die älter ist als `.wikitool-kb.json`, fehlt die Datei gan
**Fallstricke.** Eine Instanz ohne lokale `.wikitool-release.json` (oder eine ohne `files`-Block,
aus der Zeit vor `4.5.0`) hat für `dist upgrade` keine Basis, gegen die es eine lokale Änderung
erkennen könnte, und verweigert den Tausch - dafür gibt es heute keine Reparatur.
Der Befehl lädt selbst nichts herunter: `<tarball-oder-verzeichnis>` muss vorher aus Weg A
geholt werden, und ein Tarball muss genau ein Top-Level-Verzeichnis enthalten - die Form, in der
`.gitea/workflows/release.yml` es baut.
Mit `<tarball-oder-verzeichnis>` lädt der Befehl selbst nichts herunter; die Datei muss vorher
aus Weg A geholt werden. Nur `--latest` lädt, und ein Tarball muss in beiden Fällen genau ein
Top-Level-Verzeichnis enthalten - die Form, in der `.gitea/workflows/release.yml` es baut.
Vor `4.5.0` stand hier ein rein manueller Ablauf (Maschinerie von Hand kopieren, `kb/CONTRACT.md`
eingeschlossen, sha256-Vergleich von Hand). `dist upgrade` ersetzt genau diesen Teil; wer ihn
+1 -1
View File
@@ -1 +1 @@
7.1.0-beta.31
7.1.0-beta.32
+17 -6
View File
@@ -81,14 +81,25 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
step 12's, run against the migration documents this instance already has. An `offered` upgrade
listed separately blocks nothing and is decided later, in step 12.
4. **Fetch the tarball and verify it.** `dist upgrade` downloads nothing; the file has to be
there already. Take the `.tar.gz` and its `.sha256` from the release page found in step 2 and
check them before unpacking. A tarball must unpack to exactly one top-level directory.
4. **Nothing to fetch by hand.** `dist upgrade --latest` asks the release feed for the latest
release, downloads its `.tar.gz` and `.sha256` into a scratch directory, checks the archive
against the checksum and removes both again - all inside the calls of steps 5 to 7. Note the
version step 2's `version notes` printed: steps 5 to 7 pass it as `--expect`, so a release that
appeared in the meantime is refused before anything is downloaded, rather than applied unread.
The offline alternative is the tarball path: with the feed unreachable, or an archive the
operator supplies, take the `.tar.gz` and its `.sha256` from the release page named in step 2,
check the archive against the checksum before unpacking, and pass the file as `<tarball>`
where the steps below say `--latest --expect <version>`. A tarball must unpack to exactly one
top-level directory. The checksum comes from the same host as the archive, so it catches a
damaged transfer, not a compromised host - who is trusted to publish releases is the
operator's decision, made before this file starts ([INSTALL.md](../INSTALL.md) § "Version und
Updates").
5. **Dry-run the swap and read all four counts:**
```bash
tools/wikitool dist upgrade <tarball> --dry-run
tools/wikitool dist upgrade --latest --expect <version from step 2> --dry-run
```
`unchanged` / `new` / `locally changed` / `removed from the release`. `unchanged` needs no
@@ -115,7 +126,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
file reset and another kept. Preview it before it writes:
```bash
tools/wikitool dist upgrade <tarball> --dry-run --take-release <path> [--take-release <path>]
tools/wikitool dist upgrade --latest --expect <version from step 2> --dry-run --take-release <path> [--take-release <path>]
```
The preview marks every named path as one it would overwrite from the release, and a path that
@@ -134,7 +145,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
```bash
git rev-parse --short HEAD # the pre-swap commit; keep it
tools/wikitool dist upgrade <tarball> [--take-release <path>] [--keep-local]
tools/wikitool dist upgrade --latest --expect <version from step 2> [--take-release <path>] [--keep-local]
```
It writes, and commits nothing.
+22 -6
View File
@@ -2418,7 +2418,7 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**SYNOPSIS**
- `wikitool dist upgrade <source> [--dry-run] [--keep-local] [--take-release <path>]... [--prune] [--pre]`
- `wikitool dist upgrade (<source> | --latest [--expect <version>]) [--dry-run] [--keep-local] [--take-release <path>]... [--prune] [--pre]`
**PROPERTIES**
@@ -2426,23 +2426,30 @@ Apply a stack update `dist export` produced - the write half of `version check`.
- idempotent: no
- atomic: Yes for every refusal - nothing is written. Once writing starts it is a plain sequential file copy with no partial-state cleanup: an interruption mid-copy (killed process, disk full) can leave the tree part-old, part-new
- budget: counted
- network: no
- network: yes
**EXAMPLES**
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run`
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz`
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md`
- `tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run`
**EXIT STATUS**
- 0 success
- 0 The source's version equals the installed one - a no-op success
- 1 Both `<source>` and `--latest`, or neither; or `--expect` without `--latest`
- 1 Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block
- 1 `.wikitool-kb.json` is missing
- 1 A migration is already outstanding against the *installed* machinery
- 1 The working tree is dirty
- 1 `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory
- 1 `--latest`: the release feed cannot be reached, or answers with something that is not a release
- 1 `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded
- 1 `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded
- 1 `--latest`: an asset download fails, or the archive fails its `.sha256`
- 1 `--latest`: the downloaded archive's `VERSION` is not the version the feed announced
- 1 The source carries no `VERSION`, `.wikitool-release.json` or `files` block
- 1 The source's version is older than the installed one, or a pre-release without `--pre`
- 1 A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises
@@ -2450,11 +2457,17 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**ON FAILURE**
- Both `<source>` and `--latest`, or neither; or `--expect` without `--latest` -> Not transient - name exactly one source, and pass `--expect` only with `--latest`
- Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block -> Not transient - fix the named precondition and retry. A checkout with shared git history takes stack updates with `wikitool upstream merge` instead
- `.wikitool-kb.json` is missing -> Run `wikitool migrate baseline <version>`, then retry
- A migration is already outstanding against the *installed* machinery -> Finish it first - `wikitool migrate status` names it - then retry
- The working tree is dirty -> Commit or stash first, then retry
- `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory -> Fix the path or re-download the release archive, then retry
- `--latest`: the release feed cannot be reached, or answers with something that is not a release -> Transient - retry once; then report the URL from the message, or take the release page's archive by hand and pass it as `<source>`
- `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded -> Not transient - read `wikitool version notes` for the version the message names, then either expect that one or stop
- `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded -> Not transient - the message lists the assets present and the release page; report it there rather than upgrading without the checksum
- `--latest`: an asset download fails, or the archive fails its `.sha256` -> Retry once; if it fails again, report the exact message - do not fall back to an unchecked archive
- `--latest`: the downloaded archive's `VERSION` is not the version the feed announced -> Not transient - an inconsistent release; report it against the release page
- The source carries no `VERSION`, `.wikitool-release.json` or `files` block -> Point `<source>` at a distribution export, then retry
- The source's version is older than the installed one, or a pre-release without `--pre` -> Not transient - choose another source, or pass `--pre` for a pre-release
- A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises -> Correct it against the locally-changed list the refusal prints, then retry
@@ -2466,7 +2479,9 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**NOTES**
- Never downloads anything: `<source>` is an already-fetched export directory or `.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs.
- Exactly one of `<source>` and `--latest` names the release. `<source>` is an already-fetched export directory or `.tar.gz` release archive and touches no network: the archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs.
- `--latest` asks the release feed (`update_url` of the local stamp, overridable with `$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, already installed - before any download, then downloads the archive and its `.sha256` into a scratch directory removed on every exit. The checksum is mandatory here (a release without one, or an archive that fails it, is an error, not a WARN) and the archive's own `VERSION` must equal the feed's version. The asset URLs are the feed's own `browser_download_url` values, and the token reaches an asset download only if it is on the feed's origin. The checksum protects against transfer errors, not against a feed that is itself compromised - authenticity is the trust in the feed's host.
- `--expect <version>` (only with `--latest`) pins the release the feed may announce: a different latest version is refused before anything is downloaded. Pass the version `wikitool version notes` was read for, in the dry run and in the real run alike.
- The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md`), plus the stamp itself, always rewritten.
- Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten.
- Locally changed files abort the run with the full list; the abort text names the three answers with the command line filled in, and none of them is the default.
@@ -2478,13 +2493,14 @@ Apply a stack update `dist export` produced - the write half of `version check`.
- Reports, but does not block on, a crossed compatibility boundary.
- The local preconditions - `VERSION`, the release stamp, `.wikitool-kb.json`, no outstanding migration, a clean working tree - are checked before the source is read. Not being a git repository at all is a WARN, not a refusal.
- Never touches git - no commit, no push.
- `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`.
- `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`. With `--latest` it still downloads and verifies the archive - that is the only way to classify - and removes it again.
- An interrupted write is not resumed automatically: compare the tree against the printed classification and finish or revert by hand.
- The closing report names `instructions/upgrade-instance.md`, which carries the order for everything after the swap and resumes at `instructions sync`.
**SEE ALSO**
- `wikitool version check` - finds out whether an update exists
- `wikitool version notes` - the notes of the release `--expect` should name
- `instructions/upgrade-instance.md` - the order after the swap
- `INSTALL.md` § "Version und Updates" - which release, whether to take it, where the tarball comes from
- `wikitool upstream merge` - the update path for a checkout with shared git history
@@ -2571,8 +2587,8 @@ Ask the origin's release feed whether a newer stack exists.
**NOTES**
- Asks the release feed for its latest release and compares it with `VERSION`: `state` is `current`, `update`, `migration` (the step crosses a compatibility boundary) or `ahead`.
- The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance.
- Never reached implicitly from another command, needs no key, and times out after `--timeout` seconds (default 10).
- The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance, and `dist upgrade --latest` asks it which release to download.
- Never reached implicitly from another command (`dist upgrade` asks only when passed `--latest`), needs no key, and times out after `--timeout` seconds (default 10).
- The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously.
- For `update` or `migration` it prints that applying the release is a separate, manual step (`INSTALL.md` § "Eine Instanz aktualisieren").
- Read-only; exempt from the Iteration Budget Gate.
+204 -28
View File
@@ -736,8 +736,8 @@ def run_export(target: Path, dry_run: bool = False, origin: Optional[Origin] = N
# modified/deleted file is never silently overwritten: the run aborts unless
# `--keep-local` keeps it or `--take-release <path>` names it, which is the
# difference between a file the instance means to carry and one that drifted.
# This never calls a release feed; the caller supplies an already-downloaded
# tree or archive.
# The classification never touches the network: the tree or archive it works
# on is either supplied by the caller or fetched by `--latest` beforehand.
@dataclass(frozen=True)
@@ -867,6 +867,87 @@ def _resolved_source(source: Path):
yield _extract_single_top_level_dir(source, Path(tmp))
def _version_gate(
new_version: "version_mod.Version",
local_version: "version_mod.Version",
allow_pre: bool,
origin: str,
) -> bool:
"""The three version refusals shared by a source tree and the feed: a
candidate without `--pre`, a downgrade, an equal version. `True` means go
on; `False` means the run ended successfully as a no-op. `origin` names
what reported the version, for the message."""
if new_version.is_prerelease and not allow_pre:
fail(
f"{origin} reports {new_version}, a running candidate (-beta.N). "
"`.gitea/workflows/release.yml` never publishes one, so it can only come from a "
"dev checkout or a hand-made release - pass --pre if that is deliberate."
)
if new_version < local_version:
fail(f"{origin} reports {new_version}, older than the installed {local_version} - refusing a downgrade.")
if new_version == local_version:
success(f"Already at {local_version}. Nothing to do.")
return False
return True
def _query_feed(
stamp: dict,
local_version: "version_mod.Version",
allow_pre: bool,
expect: Optional["version_mod.Version"],
) -> Optional[tuple["version_mod.LatestRelease", str, Optional[str]]]:
"""Ask the release feed which release is latest and decide, before any
download, whether there is anything to fetch. Returns the release with the
feed URL and token, or `None` when the installed version is already it."""
url = version_mod.update_url(stamp)
token = os.environ.get(version_mod.UPDATE_TOKEN_ENV, "").strip() or None
console.print(f"Asking the release feed {url}", highlight=False)
try:
release = version_mod.fetch_latest_assets(url, token)
except version_mod.VersionError as exc:
fail(str(exc))
if expect is not None and release.version != expect:
fail(
f"The feed's latest release is {release.version}, not the expected {expect} - "
"nothing was downloaded. The feed moved on since the notes were read, or "
f"the wrong version was named: read `wikitool version notes` for {release.version} "
"before deciding to go there instead."
)
if not _version_gate(release.version, local_version, allow_pre, "the release feed"):
return None
return release, url, token
@contextmanager
def _downloaded_source(release: "version_mod.LatestRelease", feed_url: str, token: Optional[str]):
"""Yield the directory of the release the feed announced: download its
archive and checksum into a scratch directory, hold the archive to the
checksum (a missing checksum is an error, never a WARN), and unpack it.
A release without both assets is refused before the first download. The
scratch directory is removed on every exit, `--dry-run` included."""
archive_url, sum_url = _asset_urls(release)
with tempfile.TemporaryDirectory(prefix="wikitool-upgrade-") as tmp:
archive = Path(tmp) / release.archive_name
checksum = archive.with_name(archive.name + version_mod.CHECKSUM_SUFFIX)
for url, dest in ((archive_url, archive), (sum_url, checksum)):
console.print(f"Downloading {url}", highlight=False)
try:
version_mod.download_asset(url, dest, version_mod.token_for_asset(feed_url, url, token))
except version_mod.VersionError as exc:
fail(str(exc))
_verify_sha256_sidecar(archive)
yield _extract_single_top_level_dir(archive, Path(tmp) / "unpacked")
def _asset_urls(release: "version_mod.LatestRelease") -> tuple[str, str]:
try:
return release.asset_urls()
except version_mod.VersionError as exc:
fail(str(exc))
raise # unreachable: fail() raises typer.Exit
def _git_working_tree_status() -> Optional[str]:
"""`git status --porcelain` for `config.ROOT`, or None if it is not a git
repository at all - which is a valid, if unprotected, state for a tarball
@@ -907,7 +988,7 @@ def _resolve_take_release(
def _refusal_for_blocked(
source: Path, undecided: list[str], classification: FileClassification
target: str, undecided: list[str], classification: FileClassification
) -> str:
"""The abort text for blocked paths no flag has answered for.
@@ -927,9 +1008,9 @@ def _refusal_for_blocked(
f"{len(undecided)} locally changed file(s) (listed above) would be silently "
f"overwritten. Nothing was written, and none of these three is the default:\n"
f" - take the release's version and discard the local change:\n"
f" dist upgrade {rel_path(source)} --take-release {paths}\n"
f" dist upgrade {target} --take-release {paths}\n"
f" - keep every local change and upgrade around them ({kept_again}):\n"
f" dist upgrade {rel_path(source)} --keep-local\n"
f" dist upgrade {target} --keep-local\n"
f" - reconcile them by hand first, then re-run."
)
@@ -984,8 +1065,8 @@ def _report_plan(
path="dist upgrade",
summary="Apply a stack update `dist export` produced - the write half of `version check`.",
synopsis=(cli_contract.Variant(
usage="dist upgrade <source> [--dry-run] [--keep-local] [--take-release <path>]... "
"[--prune] [--pre]",
usage="dist upgrade (<source> | --latest [--expect <version>]) [--dry-run] "
"[--keep-local] [--take-release <path>]... [--prune] [--pre]",
),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
@@ -994,12 +1075,28 @@ def _report_plan(
"sequential file copy with no partial-state cleanup: an interruption mid-copy "
"(killed process, disk full) can leave the tree part-old, part-new",
budget=cli_contract.Budget.COUNTED,
network=cli_contract.Network.YES,
),
notes=(
"Never downloads anything: `<source>` is an already-fetched export directory or "
"`.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one "
"is present (a missing one is a WARN, not a block) and must unpack to exactly one "
"top-level directory - the shape `.gitea/workflows/release.yml` packs.",
"Exactly one of `<source>` and `--latest` names the release. `<source>` is an "
"already-fetched export directory or `.tar.gz` release archive and touches no network: "
"the archive is verified against a sibling `.sha256` if one is present (a missing one "
"is a WARN, not a block) and must unpack to exactly one top-level directory - the "
"shape `.gitea/workflows/release.yml` packs.",
"`--latest` asks the release feed (`update_url` of the local stamp, overridable with "
"`$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is "
"latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, "
"already installed - before any download, then downloads the archive and its "
"`.sha256` into a scratch directory removed on every exit. The checksum is mandatory "
"here (a release without one, or an archive that fails it, is an error, not a WARN) "
"and the archive's own `VERSION` must equal the feed's version. The asset URLs are the "
"feed's own `browser_download_url` values, and the token reaches an asset download only "
"if it is on the feed's origin. The checksum protects against transfer errors, not "
"against a feed that is itself compromised - authenticity is the trust in the feed's "
"host.",
"`--expect <version>` (only with `--latest`) pins the release the feed may announce: a "
"different latest version is refused before anything is downloaded. Pass the version "
"`wikitool version notes` was read for, in the dry run and in the real run alike.",
"The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus "
"what an export re-seeds from a blank template every time (`kb/log.md`, "
"`raw/.gitkeep`) or seeds once and the instance owns from then on "
@@ -1029,7 +1126,8 @@ def _report_plan(
"Not being a git repository at all is a WARN, not a refusal.",
"Never touches git - no commit, no push.",
"`--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source "
"needs `--pre`.",
"needs `--pre`. With `--latest` it still downloads and verifies the archive - that is "
"the only way to classify - and removes it again.",
"An interrupted write is not resumed automatically: compare the tree against the "
"printed classification and finish or revert by hand.",
"The closing report names `instructions/upgrade-instance.md`, which carries the order "
@@ -1041,6 +1139,11 @@ def _report_plan(
reaction="",
code=0,
),
cli_contract.Failure(
cause="Both `<source>` and `--latest`, or neither; or `--expect` without `--latest`",
reaction="Not transient - name exactly one source, and pass `--expect` only with "
"`--latest`",
),
cli_contract.Failure(
cause="Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` "
"block",
@@ -1064,6 +1167,34 @@ def _report_plan(
"one top-level directory",
reaction="Fix the path or re-download the release archive, then retry",
),
cli_contract.Failure(
cause="`--latest`: the release feed cannot be reached, or answers with something "
"that is not a release",
reaction="Transient - retry once; then report the URL from the message, or take the "
"release page's archive by hand and pass it as `<source>`",
),
cli_contract.Failure(
cause="`--latest --expect`: the feed's latest release is another version than the "
"one expected; nothing was downloaded",
reaction="Not transient - read `wikitool version notes` for the version the message "
"names, then either expect that one or stop",
),
cli_contract.Failure(
cause="`--latest`: the release publishes no archive or no `.sha256` under the "
"expected name; nothing was downloaded",
reaction="Not transient - the message lists the assets present and the release page; "
"report it there rather than upgrading without the checksum",
),
cli_contract.Failure(
cause="`--latest`: an asset download fails, or the archive fails its `.sha256`",
reaction="Retry once; if it fails again, report the exact message - do not fall back "
"to an unchecked archive",
),
cli_contract.Failure(
cause="`--latest`: the downloaded archive's `VERSION` is not the version the feed "
"announced",
reaction="Not transient - an inconsistent release; report it against the release page",
),
cli_contract.Failure(
cause="The source carries no `VERSION`, `.wikitool-release.json` or `files` block",
reaction="Point `<source>` at a distribution export, then retry",
@@ -1091,12 +1222,14 @@ def _report_plan(
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run",
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz",
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md",
"tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run",
),
never=(
"Never treat any of the three answers to locally changed files as the default.",
),
see_also=(
"`wikitool version check` - finds out whether an update exists",
"`wikitool version notes` - the notes of the release `--expect` should name",
"`instructions/upgrade-instance.md` - the order after the swap",
"`INSTALL.md` § \"Version und Updates\" - which release, whether to take it, where "
"the tarball comes from",
@@ -1106,8 +1239,17 @@ def _report_plan(
))
@app.command("upgrade")
def upgrade_command(
source: Path = typer.Argument(
..., help="An extracted distribution directory, or a release .tar.gz archive"
source: Optional[Path] = typer.Argument(
None, help="An extracted distribution directory, or a release .tar.gz archive - or use --latest"
),
latest: bool = typer.Option(
False, "--latest",
help="Download the release the feed announces as latest instead of naming a <source>",
),
expect: Optional[str] = typer.Option(
None, "--expect",
help="With --latest: refuse, before downloading anything, unless the feed's latest release "
"is this version (the one `version notes` was read for)",
),
dry_run: bool = typer.Option(
False, "--dry-run", help="Classify and report, without writing anything"
@@ -1133,8 +1275,10 @@ def upgrade_command(
):
"""Apply a stack update `dist export` produced - the write half of `version check`.
\f
Never downloads anything: `source` is an already-fetched
export directory or `.tar.gz` archive. Writes exactly the new release
`source` is an already-fetched export directory or `.tar.gz` archive;
`--latest` instead downloads the archive and its checksum from the
release feed (`--expect <version>` pins which release that may be) and
holds it to the checksum before anything else. Writes exactly the new release
stamp's `files` block, minus what an export re-seeds every time
(`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from
then on (`.wikitool-kb.json`, `CHANGES.md`), classifying every candidate
@@ -1155,19 +1299,41 @@ def upgrade_command(
take_release=take_release,
prune=prune,
allow_pre=allow_pre,
latest=latest,
expect=expect,
)
def run_upgrade(
source: Path,
source: Optional[Path] = None,
dry_run: bool = False,
keep_local: bool = False,
take_release: Optional[Sequence[str]] = None,
prune: bool = False,
allow_pre: bool = False,
latest: bool = False,
expect: Optional[str] = None,
) -> None:
"""The upgrade itself, free of Typer's option objects - see `run_export`
for why this split exists."""
if latest == (source is not None):
fail(
"Name exactly one source: `<source>` (an export directory or a release .tar.gz "
"already on disk) or `--latest` (the release the feed announces). "
+ ("Both were given." if latest else "Neither was given.")
)
return
expected_version = None
if expect is not None:
if not latest:
fail("--expect only makes sense with --latest: it pins the release the feed may announce.")
return
try:
expected_version = version_mod.Version.parse(expect)
except version_mod.VersionError as exc:
fail(f"--expect: {exc}")
return
try:
local_version = version_mod.read_version()
except version_mod.VersionError as exc:
@@ -1223,7 +1389,19 @@ def run_upgrade(
)
return
with _resolved_source(source) as new_root:
feed = None
if latest:
feed = _query_feed(old_stamp, local_version, allow_pre, expected_version)
if feed is None:
return
release, feed_url, feed_token = feed
source_cm = _downloaded_source(release, feed_url, feed_token)
target = f"--latest --expect {release.version}"
else:
source_cm = _resolved_source(source)
target = rel_path(source)
with source_cm as new_root:
version_path = new_root / version_mod.VERSION_FILENAME
if not version_path.is_file():
fail(f"{rel_path(new_root)} has no VERSION - not a distribution export.")
@@ -1234,18 +1412,16 @@ def run_upgrade(
fail(str(exc))
return
if new_version.is_prerelease and not allow_pre:
if feed is not None and new_version != feed[0].version:
fail(
f"{new_version} is a running candidate (-beta.N). `.gitea/workflows/release.yml` "
"never publishes one, so a candidate tree can only come from a dev checkout by "
"hand - pass --pre if that is deliberate."
f"The downloaded archive's VERSION says {new_version}, but the feed announced "
f"{feed[0].version} (tag {feed[0].tag}) - this release is inconsistent, so "
"nothing was applied. Report it against the release page"
+ (f" {feed[0].html_url}" if feed[0].html_url else "")
+ "."
)
return
if new_version < local_version:
fail(f"{new_version} is older than the installed {local_version} - refusing a downgrade.")
return
if new_version == local_version:
success(f"Already at {local_version}. Nothing to do.")
if not _version_gate(new_version, local_version, allow_pre, "the source"):
return
stamp_path = new_root / version_mod.RELEASE_STAMP_FILENAME
@@ -1289,7 +1465,7 @@ def run_upgrade(
undecided = [path for path in classification.blocked if path not in taken]
if undecided and not keep_local:
fail(_refusal_for_blocked(source, undecided, classification))
fail(_refusal_for_blocked(target, undecided, classification))
return
to_write = sorted(classification.unchanged + classification.new + sorted(taken))
+7 -2
View File
@@ -29,6 +29,9 @@ number means, and `instructions/dev/version-parts.md` for the candidate model):
stderr before asking and takes `--offline`. Both need no key, both time
out, and a feed that cannot be reached is reported as an error rather than
silently answered as "up to date" or "no notes".
- `dist upgrade --latest` reaches the same feed as a third, equally explicit
caller: it asks which release is latest, then downloads that release's
assets. Only `version_mod` talks to the feed; this module is not involved.
"""
from __future__ import annotations
@@ -158,8 +161,10 @@ def show_command(
"is `current`, `update`, `migration` (the step crosses a compatibility boundary) or "
"`ahead`.",
"The only command whose whole job is the network call - `version notes` reaches the "
"same feed too, but only as a fallback on a distributed instance.",
"Never reached implicitly from another command, needs no key, and times out after "
"same feed too, but only as a fallback on a distributed instance, and `dist upgrade "
"--latest` asks it which release to download.",
"Never reached implicitly from another command (`dist upgrade` asks only when passed "
"`--latest`), needs no key, and times out after "
"`--timeout` seconds (default 10).",
"The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the "
"built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable "
+3 -1
View File
@@ -337,7 +337,8 @@ def test_every_gated_record_shows_its_re_run_after_exit_42():
def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkout():
"""Gitea #144: `network:` means *any* reach outside this checkout - an HTTP call
`wikitool` makes itself, or a git operation against a remote (fetch/ls-remote/push) -
not only the two `version_cmd.py` used to claim exclusivity for. Pinned as an explicit
not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list
since `--latest`, which asks the release feed and downloads from it. Pinned as an explicit
set so a command gaining or losing that reach is a deliberate edit here, not a silent
drift between the property and what the command actually does."""
expected = {
@@ -346,6 +347,7 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
"upstream merge",
"version check",
"version notes",
"dist upgrade",
"review",
"task new",
"task list",
+427
View File
@@ -4,9 +4,13 @@ execution, and every refusal before anything is written. See Gitea #7."""
from __future__ import annotations
import hashlib
import http.server
import json
import shutil
import subprocess
import tarfile
import tempfile
import threading
from pathlib import Path
import pytest
@@ -586,3 +590,426 @@ def test_tarball_sha256_sidecar_mismatch_is_refused(instance, tmp_path):
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(archive)
# --- --latest: the release feed ---------------------------------------------
#
# A real `http.server` on 127.0.0.1 rather than a fetcher stub: the download
# path is `urllib` streaming into a file, and the same-origin token rule is a
# statement about which requests carry which header - both only observable on
# the wire. Every server records `(path, Authorization)` for each request.
class FeedServer:
def __init__(self) -> None:
self.routes: dict[str, tuple[int, bytes]] = {}
self.requests: list[tuple[str, str | None]] = []
outer = self
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802 - http.server's name
outer.requests.append((self.path, self.headers.get("Authorization")))
status, body = outer.routes.get(self.path, (404, b"not found"))
self.send_response(status)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args): # silence the test output
pass
self.httpd = http.server.HTTPServer(("127.0.0.1", 0), Handler)
self.thread = threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True)
self.thread.start()
@property
def base(self) -> str:
return f"http://127.0.0.1:{self.httpd.server_address[1]}"
def stop(self) -> None:
self.httpd.shutdown()
self.httpd.server_close()
def paths(self) -> list[str]:
return [path for path, _ in self.requests]
def asset_paths(self) -> list[str]:
return [path for path in self.paths() if path.startswith("/assets/")]
@pytest.fixture
def feed(monkeypatch: pytest.MonkeyPatch):
server = FeedServer()
monkeypatch.setenv("WIKITOOL_UPDATE_URL", f"{server.base}/feed")
yield server
server.stop()
@pytest.fixture
def other_origin():
server = FeedServer()
yield server
server.stop()
@pytest.fixture
def scratch_tmp(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Redirect `tempfile` into a directory the test can inspect afterwards."""
scratch = tmp_path / "scratch-tmp"
scratch.mkdir()
monkeypatch.setattr(tempfile, "tempdir", str(scratch))
return scratch
def _publish(
feed: FeedServer,
tmp_path: Path,
version: str,
files: dict[str, str] | None = None,
*,
tag: str | None = None,
inner_version: str | None = None,
checksum: str | None = "ok",
archive: bool = True,
asset_server: FeedServer | None = None,
) -> str:
"""Serve `version` as the feed's latest release and return its archive name.
`inner_version` is what the archive's own VERSION says (default: the same),
`checksum` is `"ok"`, `"wrong"` or `None` for no `.sha256` asset."""
name = f"chemenu-stack-{version}"
tree = _release(
tmp_path, f"served-{name}", inner_version or version,
files or {"AGENTS.md": "core v2\n", "tools/wikitool": "#!/bin/sh\n"},
)
packed = tmp_path / f"{name}.tar.gz"
with tarfile.open(packed, "w:gz") as tf:
tf.add(tree, arcname=name)
data = packed.read_bytes()
digest = hashlib.sha256(data).hexdigest() if checksum == "ok" else "0" * 64
holder = asset_server or feed
assets = []
if archive:
holder.routes[f"/assets/{name}.tar.gz"] = (200, data)
assets.append({"name": f"{name}.tar.gz", "browser_download_url": f"{holder.base}/assets/{name}.tar.gz"})
if checksum is not None:
holder.routes[f"/assets/{name}.tar.gz.sha256"] = (200, f"{digest} {name}.tar.gz\n".encode())
assets.append({
"name": f"{name}.tar.gz.sha256",
"browser_download_url": f"{holder.base}/assets/{name}.tar.gz.sha256",
})
feed.routes["/feed"] = (
200,
json.dumps({
"tag_name": tag or f"v{version}",
"html_url": f"{feed.base}/releases/tag/v{version}",
"assets": assets,
}).encode(),
)
return name
def _stamp_version(instance: Path) -> str:
return json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["version"]
def test_latest_downloads_verifies_and_applies_the_feeds_release(instance, tmp_path, feed, scratch_tmp):
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True)
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core v2\n"
assert _stamp_version(instance) == "1.1.0"
assert feed.paths() == ["/feed", f"/assets/{name}.tar.gz", f"/assets/{name}.tar.gz.sha256"]
assert list(scratch_tmp.iterdir()) == []
recorded = json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["files"]
assert recorded
for relative, digest in recorded.items():
assert _digest((instance / relative).read_text(encoding="utf-8")) == digest
def test_latest_yields_the_same_tree_as_the_archive_it_downloaded(instance, tmp_path, feed, monkeypatch):
name = _publish(feed, tmp_path, "1.1.0")
served = tmp_path / f"{name}.tar.gz"
snapshot = tmp_path / "instance-copy"
shutil.copytree(instance, snapshot)
dist_cmd.run_upgrade(latest=True)
via_feed = {p.relative_to(instance): p.read_bytes() for p in instance.rglob("*") if p.is_file()}
monkeypatch.setattr(config, "ROOT", snapshot)
dist_cmd.run_upgrade(served)
via_archive = {p.relative_to(snapshot): p.read_bytes() for p in snapshot.rglob("*") if p.is_file()}
assert via_feed == via_archive
def test_latest_dry_run_downloads_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp):
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True, dry_run=True)
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n"
assert _stamp_version(instance) == "1.0.0"
assert f"/assets/{name}.tar.gz" in feed.paths()
assert list(scratch_tmp.iterdir()) == []
def test_latest_already_current_is_a_no_op_without_any_download(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.0.0")
dist_cmd.run_upgrade(latest=True)
assert feed.paths() == ["/feed"]
assert _stamp_version(instance) == "1.0.0"
def test_latest_older_than_installed_is_a_downgrade_refusal_before_any_download(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "0.9.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "downgrade" in capsys.readouterr().out
assert feed.asset_paths() == []
def test_latest_prerelease_needs_pre_and_is_refused_before_any_download(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.1.0-beta.1")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
assert feed.asset_paths() == []
dist_cmd.run_upgrade(latest=True, allow_pre=True)
assert _stamp_version(instance) == "1.1.0-beta.1"
@pytest.mark.parametrize("missing", ["checksum", "archive"])
def test_latest_release_without_an_asset_is_refused_before_any_download(
instance, tmp_path, feed, capsys, missing
):
_publish(
feed, tmp_path, "1.1.0",
checksum=None if missing == "checksum" else "ok",
archive=missing != "archive",
)
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "chemenu-stack-1.1.0.tar.gz" in out
assert "assets present" in out
assert "/releases/tag/v1.1.0" in out
assert feed.asset_paths() == []
assert _stamp_version(instance) == "1.0.0"
def test_latest_checksum_mismatch_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp, capsys):
_publish(feed, tmp_path, "1.1.0", checksum="wrong")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "does not match" in " ".join(capsys.readouterr().out.split())
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n"
assert _stamp_version(instance) == "1.0.0"
assert list(scratch_tmp.iterdir()) == []
def test_latest_archive_version_that_differs_from_the_feed_is_refused(instance, tmp_path, feed, scratch_tmp, capsys):
_publish(feed, tmp_path, "1.1.0", inner_version="1.2.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "1.2.0" in out and "1.1.0" in out
assert _stamp_version(instance) == "1.0.0"
assert list(scratch_tmp.iterdir()) == []
def test_latest_asset_download_error_is_reported_with_its_url(instance, tmp_path, feed, capsys):
name = _publish(feed, tmp_path, "1.1.0")
del feed.routes[f"/assets/{name}.tar.gz"]
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert f"/assets/{name}.tar.gz answered HTTP 404" in " ".join(capsys.readouterr().out.split())
assert _stamp_version(instance) == "1.0.0"
def test_latest_unreachable_feed_is_an_error_naming_the_url(instance, monkeypatch, capsys):
monkeypatch.setenv("WIKITOOL_UPDATE_URL", "http://127.0.0.1:1/feed")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "Could not reach http://127.0.0.1:1/feed" in " ".join(capsys.readouterr().out.split())
def test_latest_respects_the_local_preconditions_before_asking_the_feed(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.1.0")
(instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8")
subprocess.run(["git", "init", "-q"], cwd=instance, check=True)
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
assert feed.requests == []
def test_latest_blocked_refusal_names_the_pinned_latest_invocation(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "1.1.0")
(instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
out = " ".join(capsys.readouterr().out.split())
assert "dist upgrade --latest --expect 1.1.0 --take-release AGENTS.md" in out
assert "dist upgrade --latest --expect 1.1.0 --keep-local" in out
def test_source_and_latest_together_or_neither_is_refused_before_any_check(instance, tmp_path, feed):
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"})
with pytest.raises(typer.Exit) as both:
dist_cmd.run_upgrade(release, latest=True)
with pytest.raises(typer.Exit) as neither:
dist_cmd.run_upgrade()
assert both.value.exit_code == 1 and neither.value.exit_code == 1
assert feed.requests == []
# --- --expect ---------------------------------------------------------------
@pytest.mark.parametrize("spelling", ["1.1.0", "v1.1.0"])
def test_expect_matching_the_feed_proceeds(instance, tmp_path, feed, spelling):
_publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True, expect=spelling)
assert _stamp_version(instance) == "1.1.0"
def test_expect_mismatch_is_refused_before_any_download_naming_both_versions(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "1.2.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True, expect="1.1.0")
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "1.2.0" in out and "1.1.0" in out and "version notes" in out
assert feed.asset_paths() == []
assert _stamp_version(instance) == "1.0.0"
def test_expect_mismatch_wins_over_an_already_current_install(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.0.0")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True, expect="1.1.0")
def test_expect_without_latest_is_refused(instance, tmp_path, feed):
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"})
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(release, expect="1.1.0")
assert excinfo.value.exit_code == 1
assert feed.requests == []
def test_expect_that_is_not_a_version_is_refused(instance, feed):
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True, expect="latest")
assert feed.requests == []
# --- the token --------------------------------------------------------------
def test_token_goes_to_the_feed_and_same_origin_assets_only(instance, tmp_path, feed, other_origin, monkeypatch):
monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret")
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True)
assert {auth for _, auth in feed.requests} == {"token s3cret"}
assert f"/assets/{name}.tar.gz" in feed.paths()
assert other_origin.requests == []
def test_token_is_withheld_from_an_asset_on_another_origin(instance, tmp_path, feed, other_origin, monkeypatch):
monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret")
_publish(feed, tmp_path, "1.1.0", asset_server=other_origin)
dist_cmd.run_upgrade(latest=True)
assert _stamp_version(instance) == "1.1.0"
assert feed.requests == [("/feed", "token s3cret")]
assert len(other_origin.requests) == 2
assert {auth for _, auth in other_origin.requests} == {None}
@pytest.mark.parametrize(
"feed_url, asset_url, sent",
[
("https://git.example/api/v1/releases/latest", "https://git.example/dl/a.tar.gz", True),
("https://git.example/api/v1/releases/latest", "https://git.example:443/dl/a.tar.gz", True),
("https://git.example/api/v1/releases/latest", "http://git.example/dl/a.tar.gz", False),
("https://git.example/api/v1/releases/latest", "https://cdn.example/dl/a.tar.gz", False),
("http://git.example:3000/x", "http://git.example:3001/x", False),
],
)
def test_token_for_asset_compares_scheme_host_and_port(feed_url, asset_url, sent):
assert version_mod.token_for_asset(feed_url, asset_url, "t") == ("t" if sent else None)
assert version_mod.token_for_asset(feed_url, asset_url, None) is None
# --- the asset names --------------------------------------------------------
def test_release_asset_names_match_the_workflow():
workflow = Path(__file__).resolve().parents[3] / ".gitea" / "workflows" / "release.yml"
if not workflow.is_file():
pytest.skip("a distributed instance carries no release workflow")
text = workflow.read_text(encoding="utf-8")
archive = version_mod.ARCHIVE_NAME.format(version="${VERSION}")
stem = archive.removesuffix(".tar.gz")
assert f'name="{stem}"' in text
assert archive.replace(stem, "${name}") in text
assert (archive + version_mod.CHECKSUM_SUFFIX).replace(stem, "${name}") in text
def test_fetch_latest_assets_reads_urls_off_the_release_and_skips_malformed_entries():
payload = json.dumps({
"tag_name": "v1.1.0",
"assets": [
{"name": "a.tar.gz", "browser_download_url": "https://h/a"},
{"name": "no-url"},
"not an object",
],
}).encode()
release = version_mod.fetch_latest_assets("https://h/feed", fetcher=lambda u, t, to: payload)
assert release.version == version_mod.Version.parse("1.1.0")
assert release.tag == "v1.1.0"
assert release.assets == {"a.tar.gz": "https://h/a"}
with pytest.raises(version_mod.VersionError, match="assets present: a.tar.gz"):
release.asset_urls()
+124 -5
View File
@@ -37,7 +37,9 @@ import functools
import json
import os
import re
import shutil
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from pathlib import Path
@@ -349,12 +351,15 @@ def fetch_latest(
The network call sits behind `fetcher` so every caller above this line -
and every test - can run without a network. This is the one place that
talks to the **release feed**, and only two commands reach it: `version
check`, whose whole job it is, and `version notes` on a *distributed*
talks to the **release feed**, and only three commands reach it: `version
check`, whose whole job it is, `version notes` on a *distributed*
instance, whose local `CHANGES.md` is a stub with no entry to print (see
`fetch_latest_notes`). Neither is implicit - `check` exists for the call,
and `notes` announces the URL it is asking before it asks, on stderr, and
takes `--offline` for a caller that wants none of it.
`fetch_latest_notes`), and `dist upgrade --latest`, which asks the feed
which release to download (see `fetch_latest_assets`). None is implicit -
`check` exists for the call, `notes` announces the URL it is asking before
it asks, on stderr, and takes `--offline` for a caller that wants none of
it, and `upgrade` reaches the feed only when the operator passes
`--latest`.
"""
fetch = fetcher or _urlopen_fetch
try:
@@ -462,6 +467,120 @@ def fetch_latest_notes(
return version, body, (str(html_url) if html_url else None)
# --- release assets --------------------------------------------------------
#
# `dist upgrade --latest` downloads the two assets `.gitea/workflows/release.yml`
# attaches to every release. The names are that workflow's `name=` line plus
# `.tar.gz` / `.tar.gz.sha256`; `test_release_asset_names_match_the_workflow`
# ties this constant to the workflow so one cannot move without the other.
ARCHIVE_NAME = "chemenu-stack-{version}.tar.gz"
CHECKSUM_SUFFIX = ".sha256"
DOWNLOAD_TIMEOUT = 60.0
@dataclass(frozen=True)
class LatestRelease:
"""What the feed's latest release offers: its version, its tag as the feed
spells it, its own page, and every asset as `name -> download URL`."""
version: Version
tag: str
html_url: Optional[str]
assets: dict[str, str]
@property
def archive_name(self) -> str:
return ARCHIVE_NAME.format(version=self.version)
def asset_urls(self) -> tuple[str, str]:
"""`(archive URL, checksum URL)`, read off the release object and never
composed. Raises `VersionError` naming the release page when either is
missing - an upgrade without its checksum is refused, not downgraded to
an unchecked one."""
archive = self.archive_name
wanted = (archive, archive + CHECKSUM_SUFFIX)
missing = [name for name in wanted if name not in self.assets]
if missing:
present = ", ".join(sorted(self.assets)) or "none"
page = f" - see the release page {self.html_url}" if self.html_url else ""
raise VersionError(
f"release {self.tag} does not publish {' and '.join(missing)} "
f"(assets present: {present}){page}"
)
return self.assets[wanted[0]], self.assets[wanted[1]]
def fetch_latest_assets(
url: str,
token: Optional[str] = None,
timeout: float = 10.0,
fetcher: Optional[Fetcher] = None,
) -> LatestRelease:
"""`fetch_latest`'s version plus the release's assets, from the one
response - the asset URLs are the feed's own `browser_download_url`
values, never a path this code composes (AGENTS.md invariant 7)."""
version, captured = _fetch_latest_object(url, token, timeout, fetcher)
assets: dict[str, str] = {}
raw_assets = captured.get("assets")
if isinstance(raw_assets, list):
for entry in raw_assets:
if not isinstance(entry, dict):
continue
name = str(entry.get("name") or "").strip()
download = str(entry.get("browser_download_url") or "").strip()
if name and download:
assets[name] = download
html_url = captured.get("html_url") or captured.get("url")
return LatestRelease(
version=version,
tag=str(captured.get("tag_name") or "").strip(),
html_url=str(html_url) if html_url else None,
assets=assets,
)
def _origin(url: str) -> tuple[str, str, int]:
parts = urllib.parse.urlsplit(url)
scheme = parts.scheme.lower()
default = {"http": 80, "https": 443}.get(scheme, 0)
return scheme, (parts.hostname or "").lower(), parts.port or default
def token_for_asset(feed_url: str, asset_url: str, token: Optional[str]) -> Optional[str]:
"""The token to send with an asset download: `token` when the asset is on
the feed's own origin (scheme, host, port), `None` otherwise. An asset URL
is data from a JSON answer, so the credential does not follow it to
another host."""
return token if token and _origin(feed_url) == _origin(asset_url) else None
def download_asset(
url: str,
dest: Path,
token: Optional[str] = None,
timeout: float = DOWNLOAD_TIMEOUT,
) -> None:
"""Stream `url` into `dest`. `timeout` is a socket timeout per read, not a
deadline for the whole transfer. The token goes out as an *unredirected*
header, so a redirect to another host does not carry it along. Every
failure is a `VersionError` naming the URL."""
request = urllib.request.Request(url)
if token:
request.add_unredirected_header("Authorization", f"token {token}")
try:
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310 - URL from the feed
with dest.open("wb") as out:
shutil.copyfileobj(response, out)
except urllib.error.HTTPError as exc:
hint = ""
if exc.code in (401, 403):
hint = f" - the download needs authentication; set ${UPDATE_TOKEN_ENV}"
raise VersionError(f"{url} answered HTTP {exc.code}{hint}") from exc
except (urllib.error.URLError, OSError, TimeoutError) as exc:
raise VersionError(f"Could not download {url}: {exc}") from exc
# --- CHANGES.md ------------------------------------------------------------
#
# The changelog is prose and stays the author's job. What is mechanical is the