feat: dist upgrade --latest downloads and verifies the release from the feed, --expect pins the version (#161)
CI / verify (push) Successful in 1m42s
Release / release (push) Successful in 38s

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- instructions/upgrade-instance.md
- tools/CONTRACT.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/version_cmd.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_dist_upgrade.py
- tools/chemenu/version.py
This commit is contained in:
torben committed 2026-09-29 22:08:00 +02:00
1 parent dd885db625
commit cf892315e6
10 files changed
+888 -62

No files matched your search

+46 -1
View File
@@ -59,13 +59,14 @@ concern - readable here, never shipped as something to parse.
--- ---
## 7.1.0-beta.31 - 2026-09-26 - new_page/type_resolver comments no longer claim only entities declare a layout: ## 7.1.0-beta.32 - 2026-09-29 - dist upgrade --latest: one-command update from the release feed
**Author:** Torben Nehmer **Author:** Torben Nehmer
<!-- wikitool:bumps --> <!-- wikitool:bumps -->
**High impact** **High impact**
- wikitool: one data record per command - `-h`, index and CONTRACT.md render from cli_contract (Gitea #121 Phase 1) - wikitool: one data record per command - `-h`, index and CONTRACT.md render from cli_contract (Gitea #121 Phase 1)
- dist upgrade --latest: one-command update from the release feed
**Medium impact** **Medium impact**
- CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them - CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
@@ -102,6 +103,50 @@ concern - readable here, never shipped as something to parse.
- new_page/type_resolver comments no longer claim only entities declare a layout: - new_page/type_resolver comments no longer claim only entities declare a layout:
<!-- /wikitool:bumps --> <!-- /wikitool:bumps -->
### dist upgrade --latest: one-command update from the release feed (Gitea #161)
Updating a tarball instance took a manual detour: `version notes`, then fetching the `.tar.gz` and
its `.sha256` from the release page by hand, then `dist upgrade <tarball>`. `dist upgrade --latest`
now does the middle part. It asks the release feed (`update_url` from the stamp,
`$WIKITOOL_UPDATE_URL` and `$WIKITOOL_UPDATE_TOKEN` as before) which release is latest, downloads
the release's archive and checksum into a scratch directory, verifies the archive against the
checksum, and hands over to the existing upgrade path unchanged - classification,
`--keep-local`/`--take-release`, migration report and every refusal are the same code. `<source>`
becomes optional; exactly one of it and `--latest` must be given.
The order is what matters. The local preconditions run first, then the feed is asked, and the
version it reports is judged **before any download**: already installed is a success no-op, an
older version is a downgrade refusal, a `-beta.N` version needs `--pre`. Only then are the two
assets looked up - by exact name, `chemenu-stack-<version>.tar.gz` and its `.sha256`, in the
`assets` of the release object the version query already returned, using the feed's own
`browser_download_url` values; no URL is composed. A release missing either asset is refused
before the first download with the assets it does have and its release page named. The checksum
is mandatory on this path (a `<source>` archive without a sibling `.sha256` is still only a
WARN), and the archive's own `VERSION` must equal the feed's version, otherwise nothing is
applied. `--dry-run` downloads and verifies too - classification needs the tree - and removes
everything again; the scratch directory goes away on every exit.
`--expect <version>` (only with `--latest`) closes the gap between reading the notes and applying
the update: the feed only offers its *latest* release, so if a newer one appeared since
`version notes` was read, the run refuses before downloading anything and names both versions.
`instructions/upgrade-instance.md` passes the version from step 2 in the dry run and in the real
run. The comparison is on parsed versions, so a `v` prefix does not matter.
`$WIKITOOL_UPDATE_TOKEN` is sent to an asset download only when the asset URL has the feed's
scheme, host and port, and as an unredirected header, so a redirect cannot carry it elsewhere.
There is no https enforcement: the checksum comes from the same host as the archive and protects
against transfer errors, not against a compromised feed - `INSTALL.md` says so.
The asset names are a Python constant (`version.ARCHIVE_NAME`) that a test ties to
`.gitea/workflows/release.yml`, so renaming them in one place fails the suite instead of the next
upgrade. `dist upgrade` is now `network: yes` in its record and in the pinned set in `test_cli.py`,
the `version check` record's "never reached implicitly" note names the one explicit exception, the
`fetch_latest` docstring says "three commands", and `tools/CONTRACT.md` is regenerated. The tests
run against a local `http.server` that logs every request and its `Authorization` header, so the
token rule and the "no asset request before the checks pass" rule are asserted on the wire. The
`upstream merge` pointers in the `dist upgrade` record and failure message stay as they are; they
belong to the separate work on the clone path.
### new_page/type_resolver comments no longer claim only entities declare a layout: ### new_page/type_resolver comments no longer claim only entities declare a layout:
Two code comments - `new_page.py`'s module docstring and `TypeResolver.get_layout`'s - still said Two code comments - `new_page.py`'s module docstring and `TypeResolver.get_layout`'s - still said
+37 -12
View File
@@ -152,12 +152,14 @@ tools/wikitool version # was läuft hier, und woher kommt es
tools/wikitool version check # gibt es ein neueres Release? tools/wikitool version check # gibt es ein neueres Release?
``` ```
`version check` und `version notes` sind die einzigen Befehle, die ins Netz gehen, und beide `version check`, `version notes` und `dist upgrade --latest` sind die einzigen Befehle, die
fragen denselben Release-Feed der Ursprungs-Instanz (`$WIKITOOL_UPDATE_URL` überschreibt; sonst ins Netz gehen, und alle drei fragen denselben Release-Feed der Ursprungs-Instanz
der Wert aus dem Stamp). `version check` ist dafür da; `version notes` greift nur dann darauf (`$WIKITOOL_UPDATE_URL` überschreibt; sonst der Wert aus dem Stamp). `version check` ist dafür da;
zurück, wenn die lokale `CHANGES.md` den Eintrag nicht hat - auf einer Instanz also immer, siehe `version notes` greift nur dann darauf zurück, wenn die lokale `CHANGES.md` den Eintrag nicht
unten - und sagt vorher auf stderr, welche URL es fragt. Ein nicht erreichbarer Feed wird als hat - auf einer Instanz also immer, siehe unten - und sagt vorher auf stderr, welche URL es
Fehler gemeldet - **nie** als „aktuell" und nie als „keine Notes". fragt; `dist upgrade` fragt den Feed nur, wenn `--latest` dasteht, und lädt dann auch das Release
herunter (siehe „Eine Instanz aktualisieren"). Ein nicht erreichbarer Feed wird als Fehler
gemeldet - **nie** als „aktuell" und nie als „keine Notes".
**Was die Versionsnummer aussagt:** kompatibel ist, was in der *linkesten von Null **Was die Versionsnummer aussagt:** kompatibel ist, was in der *linkesten von Null
verschiedenen Stelle* übereinstimmt. `0.1.3 → 0.1.4` ist ein sicheres Update, `0.1.3 → 0.2.0` verschiedenen Stelle* übereinstimmt. `0.1.3 → 0.1.4` ist ein sicheres Update, `0.1.3 → 0.2.0`
@@ -203,9 +205,32 @@ Agent-Sitzung ausgeführt wird; eine zweite Fassung derselben Schrittfolge an di
genau die Kopie, die irgendwann auseinanderläuft. Wer den Lauf selbst fahren will, liest dieselbe genau die Kopie, die irgendwann auseinanderläuft. Wer den Lauf selbst fahren will, liest dieselbe
Datei. Datei.
Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, woher Was dieses Dokument beiträgt, ist die Entscheidung *davor* - welches Release, ob überhaupt, und
der Tarball kommt (§ „Version und Updates" und Weg A oben) - und der eine Sonderfall, den die wem die Instanz als Quelle vertraut - und zwei Sonderfälle, die die Instruktion nicht abdecken
Instruktion nicht abdecken kann, weil es sie dort noch nicht gibt: kann, weil es sie dort noch nicht gibt.
**Das Release kommt mit einem Befehl.** `tools/wikitool dist upgrade --latest --expect <version>`
fragt den Release-Feed, lädt Tarball und `.sha256` in ein Arbeitsverzeichnis, prüft den Tarball
gegen die Summe und wendet ihn an; das Arbeitsverzeichnis verschwindet bei jedem Ausgang wieder,
auch bei `--dry-run`. `<version>` ist die, die `version notes` gedruckt hat: der Feed kennt nur
sein *neuestes* Release, und `--expect` verweigert den Lauf **vor** dem Download, wenn inzwischen
ein neueres erschienen ist, statt es ungelesen einzuspielen. Beides zusammen entscheidet vor dem
Download über „schon aktuell", Downgrade und Vor-Release (`-beta.N` braucht `--pre`); fehlt dem
Release der Tarball oder die Summe, bricht der Befehl vor dem ersten Download ab und nennt die
Release-Seite. Wer offline arbeitet oder einen Tarball vom Betreiber bekommen hat, gibt statt
`--latest` weiter die Datei an (`dist upgrade <tarball>`).
**Was die Prüfsumme leistet - und was nicht.** Die `.sha256` liegt beim selben Feed wie der
Tarball. Sie schützt vor einer beschädigten Übertragung, nicht vor einem Feed, der selbst
kompromittiert ist: die Echtheit eines Releases beruht auf dem Vertrauen in den Host, dessen
Feed die Instanz fragt (`update_url` im Stamp). Es gibt keinen https-Zwang; wer einen
`http://`-Feed konfiguriert, tut das bewusst. `$WIKITOOL_UPDATE_TOKEN` geht nur an Downloads
auf demselben Host wie der Feed.
**Beim ersten Sprung auf ein Release, das `--latest` kennt, gibt es die Option in der Instanz
noch nicht** - die Instruktion, die dort steht, gehört zum Release, das die Instanz verlässt.
Dann den Tarball einmal von Hand holen (Weg A oben), prüfen und `dist upgrade <tarball>` geben;
ab dem Release danach trägt die Instanz `--latest` selbst.
**Beim ersten Sprung auf `4.5.0` oder höher gibt es `dist upgrade` in der Instanz noch nicht** - **Beim ersten Sprung auf `4.5.0` oder höher gibt es `dist upgrade` in der Instanz noch nicht** -
es kam erst mit `4.5.0`. Dann das Werkzeug aus dem entpackten *neuen* Tarball verwenden, gegen die es kam erst mit `4.5.0`. Dann das Werkzeug aus dem entpackten *neuen* Tarball verwenden, gegen die
@@ -233,9 +258,9 @@ sind. Einer Instanz, die älter ist als `.wikitool-kb.json`, fehlt die Datei gan
**Fallstricke.** Eine Instanz ohne lokale `.wikitool-release.json` (oder eine ohne `files`-Block, **Fallstricke.** Eine Instanz ohne lokale `.wikitool-release.json` (oder eine ohne `files`-Block,
aus der Zeit vor `4.5.0`) hat für `dist upgrade` keine Basis, gegen die es eine lokale Änderung aus der Zeit vor `4.5.0`) hat für `dist upgrade` keine Basis, gegen die es eine lokale Änderung
erkennen könnte, und verweigert den Tausch - dafür gibt es heute keine Reparatur. erkennen könnte, und verweigert den Tausch - dafür gibt es heute keine Reparatur.
Der Befehl lädt selbst nichts herunter: `<tarball-oder-verzeichnis>` muss vorher aus Weg A Mit `<tarball-oder-verzeichnis>` lädt der Befehl selbst nichts herunter; die Datei muss vorher
geholt werden, und ein Tarball muss genau ein Top-Level-Verzeichnis enthalten - die Form, in der aus Weg A geholt werden. Nur `--latest` lädt, und ein Tarball muss in beiden Fällen genau ein
`.gitea/workflows/release.yml` es baut. Top-Level-Verzeichnis enthalten - die Form, in der `.gitea/workflows/release.yml` es baut.
Vor `4.5.0` stand hier ein rein manueller Ablauf (Maschinerie von Hand kopieren, `kb/CONTRACT.md` Vor `4.5.0` stand hier ein rein manueller Ablauf (Maschinerie von Hand kopieren, `kb/CONTRACT.md`
eingeschlossen, sha256-Vergleich von Hand). `dist upgrade` ersetzt genau diesen Teil; wer ihn eingeschlossen, sha256-Vergleich von Hand). `dist upgrade` ersetzt genau diesen Teil; wer ihn
+1 -1
View File
@@ -1 +1 @@
7.1.0-beta.31 7.1.0-beta.32
+17 -6
View File
@@ -81,14 +81,25 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
step 12's, run against the migration documents this instance already has. An `offered` upgrade step 12's, run against the migration documents this instance already has. An `offered` upgrade
listed separately blocks nothing and is decided later, in step 12. listed separately blocks nothing and is decided later, in step 12.
4. **Fetch the tarball and verify it.** `dist upgrade` downloads nothing; the file has to be 4. **Nothing to fetch by hand.** `dist upgrade --latest` asks the release feed for the latest
there already. Take the `.tar.gz` and its `.sha256` from the release page found in step 2 and release, downloads its `.tar.gz` and `.sha256` into a scratch directory, checks the archive
check them before unpacking. A tarball must unpack to exactly one top-level directory. against the checksum and removes both again - all inside the calls of steps 5 to 7. Note the
version step 2's `version notes` printed: steps 5 to 7 pass it as `--expect`, so a release that
appeared in the meantime is refused before anything is downloaded, rather than applied unread.
The offline alternative is the tarball path: with the feed unreachable, or an archive the
operator supplies, take the `.tar.gz` and its `.sha256` from the release page named in step 2,
check the archive against the checksum before unpacking, and pass the file as `<tarball>`
where the steps below say `--latest --expect <version>`. A tarball must unpack to exactly one
top-level directory. The checksum comes from the same host as the archive, so it catches a
damaged transfer, not a compromised host - who is trusted to publish releases is the
operator's decision, made before this file starts ([INSTALL.md](../INSTALL.md) § "Version und
Updates").
5. **Dry-run the swap and read all four counts:** 5. **Dry-run the swap and read all four counts:**
```bash ```bash
tools/wikitool dist upgrade <tarball> --dry-run tools/wikitool dist upgrade --latest --expect <version from step 2> --dry-run
``` ```
`unchanged` / `new` / `locally changed` / `removed from the release`. `unchanged` needs no `unchanged` / `new` / `locally changed` / `removed from the release`. `unchanged` needs no
@@ -115,7 +126,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
file reset and another kept. Preview it before it writes: file reset and another kept. Preview it before it writes:
```bash ```bash
tools/wikitool dist upgrade <tarball> --dry-run --take-release <path> [--take-release <path>] tools/wikitool dist upgrade --latest --expect <version from step 2> --dry-run --take-release <path> [--take-release <path>]
``` ```
The preview marks every named path as one it would overwrite from the release, and a path that The preview marks every named path as one it would overwrite from the release, and a path that
@@ -134,7 +145,7 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
```bash ```bash
git rev-parse --short HEAD # the pre-swap commit; keep it git rev-parse --short HEAD # the pre-swap commit; keep it
tools/wikitool dist upgrade <tarball> [--take-release <path>] [--keep-local] tools/wikitool dist upgrade --latest --expect <version from step 2> [--take-release <path>] [--keep-local]
``` ```
It writes, and commits nothing. It writes, and commits nothing.
+22 -6
View File
@@ -2418,7 +2418,7 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**SYNOPSIS** **SYNOPSIS**
- `wikitool dist upgrade <source> [--dry-run] [--keep-local] [--take-release <path>]... [--prune] [--pre]` - `wikitool dist upgrade (<source> | --latest [--expect <version>]) [--dry-run] [--keep-local] [--take-release <path>]... [--prune] [--pre]`
**PROPERTIES** **PROPERTIES**
@@ -2426,23 +2426,30 @@ Apply a stack update `dist export` produced - the write half of `version check`.
- idempotent: no - idempotent: no
- atomic: Yes for every refusal - nothing is written. Once writing starts it is a plain sequential file copy with no partial-state cleanup: an interruption mid-copy (killed process, disk full) can leave the tree part-old, part-new - atomic: Yes for every refusal - nothing is written. Once writing starts it is a plain sequential file copy with no partial-state cleanup: an interruption mid-copy (killed process, disk full) can leave the tree part-old, part-new
- budget: counted - budget: counted
- network: no - network: yes
**EXAMPLES** **EXAMPLES**
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run` - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run`
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz` - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz`
- `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md` - `tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md`
- `tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run`
**EXIT STATUS** **EXIT STATUS**
- 0 success - 0 success
- 0 The source's version equals the installed one - a no-op success - 0 The source's version equals the installed one - a no-op success
- 1 Both `<source>` and `--latest`, or neither; or `--expect` without `--latest`
- 1 Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block - 1 Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block
- 1 `.wikitool-kb.json` is missing - 1 `.wikitool-kb.json` is missing
- 1 A migration is already outstanding against the *installed* machinery - 1 A migration is already outstanding against the *installed* machinery
- 1 The working tree is dirty - 1 The working tree is dirty
- 1 `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory - 1 `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory
- 1 `--latest`: the release feed cannot be reached, or answers with something that is not a release
- 1 `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded
- 1 `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded
- 1 `--latest`: an asset download fails, or the archive fails its `.sha256`
- 1 `--latest`: the downloaded archive's `VERSION` is not the version the feed announced
- 1 The source carries no `VERSION`, `.wikitool-release.json` or `files` block - 1 The source carries no `VERSION`, `.wikitool-release.json` or `files` block
- 1 The source's version is older than the installed one, or a pre-release without `--pre` - 1 The source's version is older than the installed one, or a pre-release without `--pre`
- 1 A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises - 1 A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises
@@ -2450,11 +2457,17 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**ON FAILURE** **ON FAILURE**
- Both `<source>` and `--latest`, or neither; or `--expect` without `--latest` -> Not transient - name exactly one source, and pass `--expect` only with `--latest`
- Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block -> Not transient - fix the named precondition and retry. A checkout with shared git history takes stack updates with `wikitool upstream merge` instead - Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` block -> Not transient - fix the named precondition and retry. A checkout with shared git history takes stack updates with `wikitool upstream merge` instead
- `.wikitool-kb.json` is missing -> Run `wikitool migrate baseline <version>`, then retry - `.wikitool-kb.json` is missing -> Run `wikitool migrate baseline <version>`, then retry
- A migration is already outstanding against the *installed* machinery -> Finish it first - `wikitool migrate status` names it - then retry - A migration is already outstanding against the *installed* machinery -> Finish it first - `wikitool migrate status` names it - then retry
- The working tree is dirty -> Commit or stash first, then retry - The working tree is dirty -> Commit or stash first, then retry
- `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory -> Fix the path or re-download the release archive, then retry - `<source>` does not exist, fails its `.sha256`, or does not unpack to exactly one top-level directory -> Fix the path or re-download the release archive, then retry
- `--latest`: the release feed cannot be reached, or answers with something that is not a release -> Transient - retry once; then report the URL from the message, or take the release page's archive by hand and pass it as `<source>`
- `--latest --expect`: the feed's latest release is another version than the one expected; nothing was downloaded -> Not transient - read `wikitool version notes` for the version the message names, then either expect that one or stop
- `--latest`: the release publishes no archive or no `.sha256` under the expected name; nothing was downloaded -> Not transient - the message lists the assets present and the release page; report it there rather than upgrading without the checksum
- `--latest`: an asset download fails, or the archive fails its `.sha256` -> Retry once; if it fails again, report the exact message - do not fall back to an unchecked archive
- `--latest`: the downloaded archive's `VERSION` is not the version the feed announced -> Not transient - an inconsistent release; report it against the release page
- The source carries no `VERSION`, `.wikitool-release.json` or `files` block -> Point `<source>` at a distribution export, then retry - The source carries no `VERSION`, `.wikitool-release.json` or `files` block -> Point `<source>` at a distribution export, then retry
- The source's version is older than the installed one, or a pre-release without `--pre` -> Not transient - choose another source, or pass `--pre` for a pre-release - The source's version is older than the installed one, or a pre-release without `--pre` -> Not transient - choose another source, or pass `--pre` for a pre-release
- A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises -> Correct it against the locally-changed list the refusal prints, then retry - A `--take-release` path this run does not classify as locally changed - the one refusal a `--dry-run` also raises -> Correct it against the locally-changed list the refusal prints, then retry
@@ -2466,7 +2479,9 @@ Apply a stack update `dist export` produced - the write half of `version check`.
**NOTES** **NOTES**
- Never downloads anything: `<source>` is an already-fetched export directory or `.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs. - Exactly one of `<source>` and `--latest` names the release. `<source>` is an already-fetched export directory or `.tar.gz` release archive and touches no network: the archive is verified against a sibling `.sha256` if one is present (a missing one is a WARN, not a block) and must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs.
- `--latest` asks the release feed (`update_url` of the local stamp, overridable with `$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, already installed - before any download, then downloads the archive and its `.sha256` into a scratch directory removed on every exit. The checksum is mandatory here (a release without one, or an archive that fails it, is an error, not a WARN) and the archive's own `VERSION` must equal the feed's version. The asset URLs are the feed's own `browser_download_url` values, and the token reaches an asset download only if it is on the feed's origin. The checksum protects against transfer errors, not against a feed that is itself compromised - authenticity is the trust in the feed's host.
- `--expect <version>` (only with `--latest`) pins the release the feed may announce: a different latest version is refused before anything is downloaded. Pass the version `wikitool version notes` was read for, in the dry run and in the real run alike.
- The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md`), plus the stamp itself, always rewritten. - The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md`), plus the stamp itself, always rewritten.
- Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten. - Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten.
- Locally changed files abort the run with the full list; the abort text names the three answers with the command line filled in, and none of them is the default. - Locally changed files abort the run with the full list; the abort text names the three answers with the command line filled in, and none of them is the default.
@@ -2478,13 +2493,14 @@ Apply a stack update `dist export` produced - the write half of `version check`.
- Reports, but does not block on, a crossed compatibility boundary. - Reports, but does not block on, a crossed compatibility boundary.
- The local preconditions - `VERSION`, the release stamp, `.wikitool-kb.json`, no outstanding migration, a clean working tree - are checked before the source is read. Not being a git repository at all is a WARN, not a refusal. - The local preconditions - `VERSION`, the release stamp, `.wikitool-kb.json`, no outstanding migration, a clean working tree - are checked before the source is read. Not being a git repository at all is a WARN, not a refusal.
- Never touches git - no commit, no push. - Never touches git - no commit, no push.
- `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`. - `--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source needs `--pre`. With `--latest` it still downloads and verifies the archive - that is the only way to classify - and removes it again.
- An interrupted write is not resumed automatically: compare the tree against the printed classification and finish or revert by hand. - An interrupted write is not resumed automatically: compare the tree against the printed classification and finish or revert by hand.
- The closing report names `instructions/upgrade-instance.md`, which carries the order for everything after the swap and resumes at `instructions sync`. - The closing report names `instructions/upgrade-instance.md`, which carries the order for everything after the swap and resumes at `instructions sync`.
**SEE ALSO** **SEE ALSO**
- `wikitool version check` - finds out whether an update exists - `wikitool version check` - finds out whether an update exists
- `wikitool version notes` - the notes of the release `--expect` should name
- `instructions/upgrade-instance.md` - the order after the swap - `instructions/upgrade-instance.md` - the order after the swap
- `INSTALL.md` § "Version und Updates" - which release, whether to take it, where the tarball comes from - `INSTALL.md` § "Version und Updates" - which release, whether to take it, where the tarball comes from
- `wikitool upstream merge` - the update path for a checkout with shared git history - `wikitool upstream merge` - the update path for a checkout with shared git history
@@ -2571,8 +2587,8 @@ Ask the origin's release feed whether a newer stack exists.
**NOTES** **NOTES**
- Asks the release feed for its latest release and compares it with `VERSION`: `state` is `current`, `update`, `migration` (the step crosses a compatibility boundary) or `ahead`. - Asks the release feed for its latest release and compares it with `VERSION`: `state` is `current`, `update`, `migration` (the step crosses a compatibility boundary) or `ahead`.
- The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance. - The only command whose whole job is the network call - `version notes` reaches the same feed too, but only as a fallback on a distributed instance, and `dist upgrade --latest` asks it which release to download.
- Never reached implicitly from another command, needs no key, and times out after `--timeout` seconds (default 10). - Never reached implicitly from another command (`dist upgrade` asks only when passed `--latest`), needs no key, and times out after `--timeout` seconds (default 10).
- The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously. - The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously.
- For `update` or `migration` it prints that applying the release is a separate, manual step (`INSTALL.md` § "Eine Instanz aktualisieren"). - For `update` or `migration` it prints that applying the release is a separate, manual step (`INSTALL.md` § "Eine Instanz aktualisieren").
- Read-only; exempt from the Iteration Budget Gate. - Read-only; exempt from the Iteration Budget Gate.
+204 -28
View File
@@ -736,8 +736,8 @@ def run_export(target: Path, dry_run: bool = False, origin: Optional[Origin] = N
# modified/deleted file is never silently overwritten: the run aborts unless # modified/deleted file is never silently overwritten: the run aborts unless
# `--keep-local` keeps it or `--take-release <path>` names it, which is the # `--keep-local` keeps it or `--take-release <path>` names it, which is the
# difference between a file the instance means to carry and one that drifted. # difference between a file the instance means to carry and one that drifted.
# This never calls a release feed; the caller supplies an already-downloaded # The classification never touches the network: the tree or archive it works
# tree or archive. # on is either supplied by the caller or fetched by `--latest` beforehand.
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -867,6 +867,87 @@ def _resolved_source(source: Path):
yield _extract_single_top_level_dir(source, Path(tmp)) yield _extract_single_top_level_dir(source, Path(tmp))
def _version_gate(
new_version: "version_mod.Version",
local_version: "version_mod.Version",
allow_pre: bool,
origin: str,
) -> bool:
"""The three version refusals shared by a source tree and the feed: a
candidate without `--pre`, a downgrade, an equal version. `True` means go
on; `False` means the run ended successfully as a no-op. `origin` names
what reported the version, for the message."""
if new_version.is_prerelease and not allow_pre:
fail(
f"{origin} reports {new_version}, a running candidate (-beta.N). "
"`.gitea/workflows/release.yml` never publishes one, so it can only come from a "
"dev checkout or a hand-made release - pass --pre if that is deliberate."
)
if new_version < local_version:
fail(f"{origin} reports {new_version}, older than the installed {local_version} - refusing a downgrade.")
if new_version == local_version:
success(f"Already at {local_version}. Nothing to do.")
return False
return True
def _query_feed(
stamp: dict,
local_version: "version_mod.Version",
allow_pre: bool,
expect: Optional["version_mod.Version"],
) -> Optional[tuple["version_mod.LatestRelease", str, Optional[str]]]:
"""Ask the release feed which release is latest and decide, before any
download, whether there is anything to fetch. Returns the release with the
feed URL and token, or `None` when the installed version is already it."""
url = version_mod.update_url(stamp)
token = os.environ.get(version_mod.UPDATE_TOKEN_ENV, "").strip() or None
console.print(f"Asking the release feed {url}", highlight=False)
try:
release = version_mod.fetch_latest_assets(url, token)
except version_mod.VersionError as exc:
fail(str(exc))
if expect is not None and release.version != expect:
fail(
f"The feed's latest release is {release.version}, not the expected {expect} - "
"nothing was downloaded. The feed moved on since the notes were read, or "
f"the wrong version was named: read `wikitool version notes` for {release.version} "
"before deciding to go there instead."
)
if not _version_gate(release.version, local_version, allow_pre, "the release feed"):
return None
return release, url, token
@contextmanager
def _downloaded_source(release: "version_mod.LatestRelease", feed_url: str, token: Optional[str]):
"""Yield the directory of the release the feed announced: download its
archive and checksum into a scratch directory, hold the archive to the
checksum (a missing checksum is an error, never a WARN), and unpack it.
A release without both assets is refused before the first download. The
scratch directory is removed on every exit, `--dry-run` included."""
archive_url, sum_url = _asset_urls(release)
with tempfile.TemporaryDirectory(prefix="wikitool-upgrade-") as tmp:
archive = Path(tmp) / release.archive_name
checksum = archive.with_name(archive.name + version_mod.CHECKSUM_SUFFIX)
for url, dest in ((archive_url, archive), (sum_url, checksum)):
console.print(f"Downloading {url}", highlight=False)
try:
version_mod.download_asset(url, dest, version_mod.token_for_asset(feed_url, url, token))
except version_mod.VersionError as exc:
fail(str(exc))
_verify_sha256_sidecar(archive)
yield _extract_single_top_level_dir(archive, Path(tmp) / "unpacked")
def _asset_urls(release: "version_mod.LatestRelease") -> tuple[str, str]:
try:
return release.asset_urls()
except version_mod.VersionError as exc:
fail(str(exc))
raise # unreachable: fail() raises typer.Exit
def _git_working_tree_status() -> Optional[str]: def _git_working_tree_status() -> Optional[str]:
"""`git status --porcelain` for `config.ROOT`, or None if it is not a git """`git status --porcelain` for `config.ROOT`, or None if it is not a git
repository at all - which is a valid, if unprotected, state for a tarball repository at all - which is a valid, if unprotected, state for a tarball
@@ -907,7 +988,7 @@ def _resolve_take_release(
def _refusal_for_blocked( def _refusal_for_blocked(
source: Path, undecided: list[str], classification: FileClassification target: str, undecided: list[str], classification: FileClassification
) -> str: ) -> str:
"""The abort text for blocked paths no flag has answered for. """The abort text for blocked paths no flag has answered for.
@@ -927,9 +1008,9 @@ def _refusal_for_blocked(
f"{len(undecided)} locally changed file(s) (listed above) would be silently " f"{len(undecided)} locally changed file(s) (listed above) would be silently "
f"overwritten. Nothing was written, and none of these three is the default:\n" f"overwritten. Nothing was written, and none of these three is the default:\n"
f" - take the release's version and discard the local change:\n" f" - take the release's version and discard the local change:\n"
f" dist upgrade {rel_path(source)} --take-release {paths}\n" f" dist upgrade {target} --take-release {paths}\n"
f" - keep every local change and upgrade around them ({kept_again}):\n" f" - keep every local change and upgrade around them ({kept_again}):\n"
f" dist upgrade {rel_path(source)} --keep-local\n" f" dist upgrade {target} --keep-local\n"
f" - reconcile them by hand first, then re-run." f" - reconcile them by hand first, then re-run."
) )
@@ -984,8 +1065,8 @@ def _report_plan(
path="dist upgrade", path="dist upgrade",
summary="Apply a stack update `dist export` produced - the write half of `version check`.", summary="Apply a stack update `dist export` produced - the write half of `version check`.",
synopsis=(cli_contract.Variant( synopsis=(cli_contract.Variant(
usage="dist upgrade <source> [--dry-run] [--keep-local] [--take-release <path>]... " usage="dist upgrade (<source> | --latest [--expect <version>]) [--dry-run] "
"[--prune] [--pre]", "[--keep-local] [--take-release <path>]... [--prune] [--pre]",
),), ),),
properties=cli_contract.Properties( properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE, effect=cli_contract.Effect.WRITE,
@@ -994,12 +1075,28 @@ def _report_plan(
"sequential file copy with no partial-state cleanup: an interruption mid-copy " "sequential file copy with no partial-state cleanup: an interruption mid-copy "
"(killed process, disk full) can leave the tree part-old, part-new", "(killed process, disk full) can leave the tree part-old, part-new",
budget=cli_contract.Budget.COUNTED, budget=cli_contract.Budget.COUNTED,
network=cli_contract.Network.YES,
), ),
notes=( notes=(
"Never downloads anything: `<source>` is an already-fetched export directory or " "Exactly one of `<source>` and `--latest` names the release. `<source>` is an "
"`.tar.gz` release archive. An archive is verified against a sibling `.sha256` if one " "already-fetched export directory or `.tar.gz` release archive and touches no network: "
"is present (a missing one is a WARN, not a block) and must unpack to exactly one " "the archive is verified against a sibling `.sha256` if one is present (a missing one "
"top-level directory - the shape `.gitea/workflows/release.yml` packs.", "is a WARN, not a block) and must unpack to exactly one top-level directory - the "
"shape `.gitea/workflows/release.yml` packs.",
"`--latest` asks the release feed (`update_url` of the local stamp, overridable with "
"`$WIKITOOL_UPDATE_URL`; `$WIKITOOL_UPDATE_TOKEN` is sent along) which release is "
"latest, checks that version - `--expect`, a downgrade, a pre-release without `--pre`, "
"already installed - before any download, then downloads the archive and its "
"`.sha256` into a scratch directory removed on every exit. The checksum is mandatory "
"here (a release without one, or an archive that fails it, is an error, not a WARN) "
"and the archive's own `VERSION` must equal the feed's version. The asset URLs are the "
"feed's own `browser_download_url` values, and the token reaches an asset download only "
"if it is on the feed's origin. The checksum protects against transfer errors, not "
"against a feed that is itself compromised - authenticity is the trust in the feed's "
"host.",
"`--expect <version>` (only with `--latest`) pins the release the feed may announce: a "
"different latest version is refused before anything is downloaded. Pass the version "
"`wikitool version notes` was read for, in the dry run and in the real run alike.",
"The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus " "The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus "
"what an export re-seeds from a blank template every time (`kb/log.md`, " "what an export re-seeds from a blank template every time (`kb/log.md`, "
"`raw/.gitkeep`) or seeds once and the instance owns from then on " "`raw/.gitkeep`) or seeds once and the instance owns from then on "
@@ -1029,7 +1126,8 @@ def _report_plan(
"Not being a git repository at all is a WARN, not a refusal.", "Not being a git repository at all is a WARN, not a refusal.",
"Never touches git - no commit, no push.", "Never touches git - no commit, no push.",
"`--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source " "`--dry-run` classifies and reports without writing; a pre-release (`-beta.N`) source "
"needs `--pre`.", "needs `--pre`. With `--latest` it still downloads and verifies the archive - that is "
"the only way to classify - and removes it again.",
"An interrupted write is not resumed automatically: compare the tree against the " "An interrupted write is not resumed automatically: compare the tree against the "
"printed classification and finish or revert by hand.", "printed classification and finish or revert by hand.",
"The closing report names `instructions/upgrade-instance.md`, which carries the order " "The closing report names `instructions/upgrade-instance.md`, which carries the order "
@@ -1041,6 +1139,11 @@ def _report_plan(
reaction="", reaction="",
code=0, code=0,
), ),
cli_contract.Failure(
cause="Both `<source>` and `--latest`, or neither; or `--expect` without `--latest`",
reaction="Not transient - name exactly one source, and pass `--expect` only with "
"`--latest`",
),
cli_contract.Failure( cli_contract.Failure(
cause="Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` " cause="Local `VERSION` missing, or no local `.wikitool-release.json` with a `files` "
"block", "block",
@@ -1064,6 +1167,34 @@ def _report_plan(
"one top-level directory", "one top-level directory",
reaction="Fix the path or re-download the release archive, then retry", reaction="Fix the path or re-download the release archive, then retry",
), ),
cli_contract.Failure(
cause="`--latest`: the release feed cannot be reached, or answers with something "
"that is not a release",
reaction="Transient - retry once; then report the URL from the message, or take the "
"release page's archive by hand and pass it as `<source>`",
),
cli_contract.Failure(
cause="`--latest --expect`: the feed's latest release is another version than the "
"one expected; nothing was downloaded",
reaction="Not transient - read `wikitool version notes` for the version the message "
"names, then either expect that one or stop",
),
cli_contract.Failure(
cause="`--latest`: the release publishes no archive or no `.sha256` under the "
"expected name; nothing was downloaded",
reaction="Not transient - the message lists the assets present and the release page; "
"report it there rather than upgrading without the checksum",
),
cli_contract.Failure(
cause="`--latest`: an asset download fails, or the archive fails its `.sha256`",
reaction="Retry once; if it fails again, report the exact message - do not fall back "
"to an unchecked archive",
),
cli_contract.Failure(
cause="`--latest`: the downloaded archive's `VERSION` is not the version the feed "
"announced",
reaction="Not transient - an inconsistent release; report it against the release page",
),
cli_contract.Failure( cli_contract.Failure(
cause="The source carries no `VERSION`, `.wikitool-release.json` or `files` block", cause="The source carries no `VERSION`, `.wikitool-release.json` or `files` block",
reaction="Point `<source>` at a distribution export, then retry", reaction="Point `<source>` at a distribution export, then retry",
@@ -1091,12 +1222,14 @@ def _report_plan(
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run", "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --dry-run",
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz", "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz",
"tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md", "tools/wikitool dist upgrade ../chemenu-7.1.0.tar.gz --take-release tools/README.md",
"tools/wikitool dist upgrade --latest --expect 8.0.0 --dry-run",
), ),
never=( never=(
"Never treat any of the three answers to locally changed files as the default.", "Never treat any of the three answers to locally changed files as the default.",
), ),
see_also=( see_also=(
"`wikitool version check` - finds out whether an update exists", "`wikitool version check` - finds out whether an update exists",
"`wikitool version notes` - the notes of the release `--expect` should name",
"`instructions/upgrade-instance.md` - the order after the swap", "`instructions/upgrade-instance.md` - the order after the swap",
"`INSTALL.md` § \"Version und Updates\" - which release, whether to take it, where " "`INSTALL.md` § \"Version und Updates\" - which release, whether to take it, where "
"the tarball comes from", "the tarball comes from",
@@ -1106,8 +1239,17 @@ def _report_plan(
)) ))
@app.command("upgrade") @app.command("upgrade")
def upgrade_command( def upgrade_command(
source: Path = typer.Argument( source: Optional[Path] = typer.Argument(
..., help="An extracted distribution directory, or a release .tar.gz archive" None, help="An extracted distribution directory, or a release .tar.gz archive - or use --latest"
),
latest: bool = typer.Option(
False, "--latest",
help="Download the release the feed announces as latest instead of naming a <source>",
),
expect: Optional[str] = typer.Option(
None, "--expect",
help="With --latest: refuse, before downloading anything, unless the feed's latest release "
"is this version (the one `version notes` was read for)",
), ),
dry_run: bool = typer.Option( dry_run: bool = typer.Option(
False, "--dry-run", help="Classify and report, without writing anything" False, "--dry-run", help="Classify and report, without writing anything"
@@ -1133,8 +1275,10 @@ def upgrade_command(
): ):
"""Apply a stack update `dist export` produced - the write half of `version check`. """Apply a stack update `dist export` produced - the write half of `version check`.
\f \f
Never downloads anything: `source` is an already-fetched `source` is an already-fetched export directory or `.tar.gz` archive;
export directory or `.tar.gz` archive. Writes exactly the new release `--latest` instead downloads the archive and its checksum from the
release feed (`--expect <version>` pins which release that may be) and
holds it to the checksum before anything else. Writes exactly the new release
stamp's `files` block, minus what an export re-seeds every time stamp's `files` block, minus what an export re-seeds every time
(`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from (`kb/log.md`, `raw/.gitkeep`) or seeds once and the instance owns from
then on (`.wikitool-kb.json`, `CHANGES.md`), classifying every candidate then on (`.wikitool-kb.json`, `CHANGES.md`), classifying every candidate
@@ -1155,19 +1299,41 @@ def upgrade_command(
take_release=take_release, take_release=take_release,
prune=prune, prune=prune,
allow_pre=allow_pre, allow_pre=allow_pre,
latest=latest,
expect=expect,
) )
def run_upgrade( def run_upgrade(
source: Path, source: Optional[Path] = None,
dry_run: bool = False, dry_run: bool = False,
keep_local: bool = False, keep_local: bool = False,
take_release: Optional[Sequence[str]] = None, take_release: Optional[Sequence[str]] = None,
prune: bool = False, prune: bool = False,
allow_pre: bool = False, allow_pre: bool = False,
latest: bool = False,
expect: Optional[str] = None,
) -> None: ) -> None:
"""The upgrade itself, free of Typer's option objects - see `run_export` """The upgrade itself, free of Typer's option objects - see `run_export`
for why this split exists.""" for why this split exists."""
if latest == (source is not None):
fail(
"Name exactly one source: `<source>` (an export directory or a release .tar.gz "
"already on disk) or `--latest` (the release the feed announces). "
+ ("Both were given." if latest else "Neither was given.")
)
return
expected_version = None
if expect is not None:
if not latest:
fail("--expect only makes sense with --latest: it pins the release the feed may announce.")
return
try:
expected_version = version_mod.Version.parse(expect)
except version_mod.VersionError as exc:
fail(f"--expect: {exc}")
return
try: try:
local_version = version_mod.read_version() local_version = version_mod.read_version()
except version_mod.VersionError as exc: except version_mod.VersionError as exc:
@@ -1223,7 +1389,19 @@ def run_upgrade(
) )
return return
with _resolved_source(source) as new_root: feed = None
if latest:
feed = _query_feed(old_stamp, local_version, allow_pre, expected_version)
if feed is None:
return
release, feed_url, feed_token = feed
source_cm = _downloaded_source(release, feed_url, feed_token)
target = f"--latest --expect {release.version}"
else:
source_cm = _resolved_source(source)
target = rel_path(source)
with source_cm as new_root:
version_path = new_root / version_mod.VERSION_FILENAME version_path = new_root / version_mod.VERSION_FILENAME
if not version_path.is_file(): if not version_path.is_file():
fail(f"{rel_path(new_root)} has no VERSION - not a distribution export.") fail(f"{rel_path(new_root)} has no VERSION - not a distribution export.")
@@ -1234,18 +1412,16 @@ def run_upgrade(
fail(str(exc)) fail(str(exc))
return return
if new_version.is_prerelease and not allow_pre: if feed is not None and new_version != feed[0].version:
fail( fail(
f"{new_version} is a running candidate (-beta.N). `.gitea/workflows/release.yml` " f"The downloaded archive's VERSION says {new_version}, but the feed announced "
"never publishes one, so a candidate tree can only come from a dev checkout by " f"{feed[0].version} (tag {feed[0].tag}) - this release is inconsistent, so "
"hand - pass --pre if that is deliberate." "nothing was applied. Report it against the release page"
+ (f" {feed[0].html_url}" if feed[0].html_url else "")
+ "."
) )
return return
if new_version < local_version: if not _version_gate(new_version, local_version, allow_pre, "the source"):
fail(f"{new_version} is older than the installed {local_version} - refusing a downgrade.")
return
if new_version == local_version:
success(f"Already at {local_version}. Nothing to do.")
return return
stamp_path = new_root / version_mod.RELEASE_STAMP_FILENAME stamp_path = new_root / version_mod.RELEASE_STAMP_FILENAME
@@ -1289,7 +1465,7 @@ def run_upgrade(
undecided = [path for path in classification.blocked if path not in taken] undecided = [path for path in classification.blocked if path not in taken]
if undecided and not keep_local: if undecided and not keep_local:
fail(_refusal_for_blocked(source, undecided, classification)) fail(_refusal_for_blocked(target, undecided, classification))
return return
to_write = sorted(classification.unchanged + classification.new + sorted(taken)) to_write = sorted(classification.unchanged + classification.new + sorted(taken))
+7 -2
View File
@@ -29,6 +29,9 @@ number means, and `instructions/dev/version-parts.md` for the candidate model):
stderr before asking and takes `--offline`. Both need no key, both time stderr before asking and takes `--offline`. Both need no key, both time
out, and a feed that cannot be reached is reported as an error rather than out, and a feed that cannot be reached is reported as an error rather than
silently answered as "up to date" or "no notes". silently answered as "up to date" or "no notes".
- `dist upgrade --latest` reaches the same feed as a third, equally explicit
caller: it asks which release is latest, then downloads that release's
assets. Only `version_mod` talks to the feed; this module is not involved.
""" """
from __future__ import annotations from __future__ import annotations
@@ -158,8 +161,10 @@ def show_command(
"is `current`, `update`, `migration` (the step crosses a compatibility boundary) or " "is `current`, `update`, `migration` (the step crosses a compatibility boundary) or "
"`ahead`.", "`ahead`.",
"The only command whose whole job is the network call - `version notes` reaches the " "The only command whose whole job is the network call - `version notes` reaches the "
"same feed too, but only as a fallback on a distributed instance.", "same feed too, but only as a fallback on a distributed instance, and `dist upgrade "
"Never reached implicitly from another command, needs no key, and times out after " "--latest` asks it which release to download.",
"Never reached implicitly from another command (`dist upgrade` asks only when passed "
"`--latest`), needs no key, and times out after "
"`--timeout` seconds (default 10).", "`--timeout` seconds (default 10).",
"The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the " "The feed is `--url`, else `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the "
"built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable " "built-in origin. `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable "
+3 -1
View File
@@ -337,7 +337,8 @@ def test_every_gated_record_shows_its_re_run_after_exit_42():
def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkout(): def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkout():
"""Gitea #144: `network:` means *any* reach outside this checkout - an HTTP call """Gitea #144: `network:` means *any* reach outside this checkout - an HTTP call
`wikitool` makes itself, or a git operation against a remote (fetch/ls-remote/push) - `wikitool` makes itself, or a git operation against a remote (fetch/ls-remote/push) -
not only the two `version_cmd.py` used to claim exclusivity for. Pinned as an explicit not only the two `version_cmd.py` used to claim exclusivity for. `dist upgrade` is on the list
since `--latest`, which asks the release feed and downloads from it. Pinned as an explicit
set so a command gaining or losing that reach is a deliberate edit here, not a silent set so a command gaining or losing that reach is a deliberate edit here, not a silent
drift between the property and what the command actually does.""" drift between the property and what the command actually does."""
expected = { expected = {
@@ -346,6 +347,7 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
"upstream merge", "upstream merge",
"version check", "version check",
"version notes", "version notes",
"dist upgrade",
"review", "review",
"task new", "task new",
"task list", "task list",
+427
View File
@@ -4,9 +4,13 @@ execution, and every refusal before anything is written. See Gitea #7."""
from __future__ import annotations from __future__ import annotations
import hashlib import hashlib
import http.server
import json import json
import shutil
import subprocess import subprocess
import tarfile import tarfile
import tempfile
import threading
from pathlib import Path from pathlib import Path
import pytest import pytest
@@ -586,3 +590,426 @@ def test_tarball_sha256_sidecar_mismatch_is_refused(instance, tmp_path):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(archive) dist_cmd.run_upgrade(archive)
# --- --latest: the release feed ---------------------------------------------
#
# A real `http.server` on 127.0.0.1 rather than a fetcher stub: the download
# path is `urllib` streaming into a file, and the same-origin token rule is a
# statement about which requests carry which header - both only observable on
# the wire. Every server records `(path, Authorization)` for each request.
class FeedServer:
def __init__(self) -> None:
self.routes: dict[str, tuple[int, bytes]] = {}
self.requests: list[tuple[str, str | None]] = []
outer = self
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802 - http.server's name
outer.requests.append((self.path, self.headers.get("Authorization")))
status, body = outer.routes.get(self.path, (404, b"not found"))
self.send_response(status)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args): # silence the test output
pass
self.httpd = http.server.HTTPServer(("127.0.0.1", 0), Handler)
self.thread = threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True)
self.thread.start()
@property
def base(self) -> str:
return f"http://127.0.0.1:{self.httpd.server_address[1]}"
def stop(self) -> None:
self.httpd.shutdown()
self.httpd.server_close()
def paths(self) -> list[str]:
return [path for path, _ in self.requests]
def asset_paths(self) -> list[str]:
return [path for path in self.paths() if path.startswith("/assets/")]
@pytest.fixture
def feed(monkeypatch: pytest.MonkeyPatch):
server = FeedServer()
monkeypatch.setenv("WIKITOOL_UPDATE_URL", f"{server.base}/feed")
yield server
server.stop()
@pytest.fixture
def other_origin():
server = FeedServer()
yield server
server.stop()
@pytest.fixture
def scratch_tmp(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Redirect `tempfile` into a directory the test can inspect afterwards."""
scratch = tmp_path / "scratch-tmp"
scratch.mkdir()
monkeypatch.setattr(tempfile, "tempdir", str(scratch))
return scratch
def _publish(
feed: FeedServer,
tmp_path: Path,
version: str,
files: dict[str, str] | None = None,
*,
tag: str | None = None,
inner_version: str | None = None,
checksum: str | None = "ok",
archive: bool = True,
asset_server: FeedServer | None = None,
) -> str:
"""Serve `version` as the feed's latest release and return its archive name.
`inner_version` is what the archive's own VERSION says (default: the same),
`checksum` is `"ok"`, `"wrong"` or `None` for no `.sha256` asset."""
name = f"chemenu-stack-{version}"
tree = _release(
tmp_path, f"served-{name}", inner_version or version,
files or {"AGENTS.md": "core v2\n", "tools/wikitool": "#!/bin/sh\n"},
)
packed = tmp_path / f"{name}.tar.gz"
with tarfile.open(packed, "w:gz") as tf:
tf.add(tree, arcname=name)
data = packed.read_bytes()
digest = hashlib.sha256(data).hexdigest() if checksum == "ok" else "0" * 64
holder = asset_server or feed
assets = []
if archive:
holder.routes[f"/assets/{name}.tar.gz"] = (200, data)
assets.append({"name": f"{name}.tar.gz", "browser_download_url": f"{holder.base}/assets/{name}.tar.gz"})
if checksum is not None:
holder.routes[f"/assets/{name}.tar.gz.sha256"] = (200, f"{digest} {name}.tar.gz\n".encode())
assets.append({
"name": f"{name}.tar.gz.sha256",
"browser_download_url": f"{holder.base}/assets/{name}.tar.gz.sha256",
})
feed.routes["/feed"] = (
200,
json.dumps({
"tag_name": tag or f"v{version}",
"html_url": f"{feed.base}/releases/tag/v{version}",
"assets": assets,
}).encode(),
)
return name
def _stamp_version(instance: Path) -> str:
return json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["version"]
def test_latest_downloads_verifies_and_applies_the_feeds_release(instance, tmp_path, feed, scratch_tmp):
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True)
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core v2\n"
assert _stamp_version(instance) == "1.1.0"
assert feed.paths() == ["/feed", f"/assets/{name}.tar.gz", f"/assets/{name}.tar.gz.sha256"]
assert list(scratch_tmp.iterdir()) == []
recorded = json.loads((instance / version_mod.RELEASE_STAMP_FILENAME).read_text())["files"]
assert recorded
for relative, digest in recorded.items():
assert _digest((instance / relative).read_text(encoding="utf-8")) == digest
def test_latest_yields_the_same_tree_as_the_archive_it_downloaded(instance, tmp_path, feed, monkeypatch):
name = _publish(feed, tmp_path, "1.1.0")
served = tmp_path / f"{name}.tar.gz"
snapshot = tmp_path / "instance-copy"
shutil.copytree(instance, snapshot)
dist_cmd.run_upgrade(latest=True)
via_feed = {p.relative_to(instance): p.read_bytes() for p in instance.rglob("*") if p.is_file()}
monkeypatch.setattr(config, "ROOT", snapshot)
dist_cmd.run_upgrade(served)
via_archive = {p.relative_to(snapshot): p.read_bytes() for p in snapshot.rglob("*") if p.is_file()}
assert via_feed == via_archive
def test_latest_dry_run_downloads_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp):
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True, dry_run=True)
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n"
assert _stamp_version(instance) == "1.0.0"
assert f"/assets/{name}.tar.gz" in feed.paths()
assert list(scratch_tmp.iterdir()) == []
def test_latest_already_current_is_a_no_op_without_any_download(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.0.0")
dist_cmd.run_upgrade(latest=True)
assert feed.paths() == ["/feed"]
assert _stamp_version(instance) == "1.0.0"
def test_latest_older_than_installed_is_a_downgrade_refusal_before_any_download(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "0.9.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "downgrade" in capsys.readouterr().out
assert feed.asset_paths() == []
def test_latest_prerelease_needs_pre_and_is_refused_before_any_download(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.1.0-beta.1")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
assert feed.asset_paths() == []
dist_cmd.run_upgrade(latest=True, allow_pre=True)
assert _stamp_version(instance) == "1.1.0-beta.1"
@pytest.mark.parametrize("missing", ["checksum", "archive"])
def test_latest_release_without_an_asset_is_refused_before_any_download(
instance, tmp_path, feed, capsys, missing
):
_publish(
feed, tmp_path, "1.1.0",
checksum=None if missing == "checksum" else "ok",
archive=missing != "archive",
)
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "chemenu-stack-1.1.0.tar.gz" in out
assert "assets present" in out
assert "/releases/tag/v1.1.0" in out
assert feed.asset_paths() == []
assert _stamp_version(instance) == "1.0.0"
def test_latest_checksum_mismatch_writes_nothing_and_cleans_up(instance, tmp_path, feed, scratch_tmp, capsys):
_publish(feed, tmp_path, "1.1.0", checksum="wrong")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "does not match" in " ".join(capsys.readouterr().out.split())
assert (instance / "AGENTS.md").read_text(encoding="utf-8") == "core\n"
assert _stamp_version(instance) == "1.0.0"
assert list(scratch_tmp.iterdir()) == []
def test_latest_archive_version_that_differs_from_the_feed_is_refused(instance, tmp_path, feed, scratch_tmp, capsys):
_publish(feed, tmp_path, "1.1.0", inner_version="1.2.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "1.2.0" in out and "1.1.0" in out
assert _stamp_version(instance) == "1.0.0"
assert list(scratch_tmp.iterdir()) == []
def test_latest_asset_download_error_is_reported_with_its_url(instance, tmp_path, feed, capsys):
name = _publish(feed, tmp_path, "1.1.0")
del feed.routes[f"/assets/{name}.tar.gz"]
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert f"/assets/{name}.tar.gz answered HTTP 404" in " ".join(capsys.readouterr().out.split())
assert _stamp_version(instance) == "1.0.0"
def test_latest_unreachable_feed_is_an_error_naming_the_url(instance, monkeypatch, capsys):
monkeypatch.setenv("WIKITOOL_UPDATE_URL", "http://127.0.0.1:1/feed")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True)
assert excinfo.value.exit_code == 1
assert "Could not reach http://127.0.0.1:1/feed" in " ".join(capsys.readouterr().out.split())
def test_latest_respects_the_local_preconditions_before_asking_the_feed(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.1.0")
(instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8")
subprocess.run(["git", "init", "-q"], cwd=instance, check=True)
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
assert feed.requests == []
def test_latest_blocked_refusal_names_the_pinned_latest_invocation(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "1.1.0")
(instance / "AGENTS.md").write_text("locally edited\n", encoding="utf-8")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True)
out = " ".join(capsys.readouterr().out.split())
assert "dist upgrade --latest --expect 1.1.0 --take-release AGENTS.md" in out
assert "dist upgrade --latest --expect 1.1.0 --keep-local" in out
def test_source_and_latest_together_or_neither_is_refused_before_any_check(instance, tmp_path, feed):
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"})
with pytest.raises(typer.Exit) as both:
dist_cmd.run_upgrade(release, latest=True)
with pytest.raises(typer.Exit) as neither:
dist_cmd.run_upgrade()
assert both.value.exit_code == 1 and neither.value.exit_code == 1
assert feed.requests == []
# --- --expect ---------------------------------------------------------------
@pytest.mark.parametrize("spelling", ["1.1.0", "v1.1.0"])
def test_expect_matching_the_feed_proceeds(instance, tmp_path, feed, spelling):
_publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True, expect=spelling)
assert _stamp_version(instance) == "1.1.0"
def test_expect_mismatch_is_refused_before_any_download_naming_both_versions(instance, tmp_path, feed, capsys):
_publish(feed, tmp_path, "1.2.0")
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(latest=True, expect="1.1.0")
assert excinfo.value.exit_code == 1
out = " ".join(capsys.readouterr().out.split())
assert "1.2.0" in out and "1.1.0" in out and "version notes" in out
assert feed.asset_paths() == []
assert _stamp_version(instance) == "1.0.0"
def test_expect_mismatch_wins_over_an_already_current_install(instance, tmp_path, feed):
_publish(feed, tmp_path, "1.0.0")
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True, expect="1.1.0")
def test_expect_without_latest_is_refused(instance, tmp_path, feed):
release = _release(tmp_path, "release", "1.1.0", {"AGENTS.md": "core\n"})
with pytest.raises(typer.Exit) as excinfo:
dist_cmd.run_upgrade(release, expect="1.1.0")
assert excinfo.value.exit_code == 1
assert feed.requests == []
def test_expect_that_is_not_a_version_is_refused(instance, feed):
with pytest.raises(typer.Exit):
dist_cmd.run_upgrade(latest=True, expect="latest")
assert feed.requests == []
# --- the token --------------------------------------------------------------
def test_token_goes_to_the_feed_and_same_origin_assets_only(instance, tmp_path, feed, other_origin, monkeypatch):
monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret")
name = _publish(feed, tmp_path, "1.1.0")
dist_cmd.run_upgrade(latest=True)
assert {auth for _, auth in feed.requests} == {"token s3cret"}
assert f"/assets/{name}.tar.gz" in feed.paths()
assert other_origin.requests == []
def test_token_is_withheld_from_an_asset_on_another_origin(instance, tmp_path, feed, other_origin, monkeypatch):
monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "s3cret")
_publish(feed, tmp_path, "1.1.0", asset_server=other_origin)
dist_cmd.run_upgrade(latest=True)
assert _stamp_version(instance) == "1.1.0"
assert feed.requests == [("/feed", "token s3cret")]
assert len(other_origin.requests) == 2
assert {auth for _, auth in other_origin.requests} == {None}
@pytest.mark.parametrize(
"feed_url, asset_url, sent",
[
("https://git.example/api/v1/releases/latest", "https://git.example/dl/a.tar.gz", True),
("https://git.example/api/v1/releases/latest", "https://git.example:443/dl/a.tar.gz", True),
("https://git.example/api/v1/releases/latest", "http://git.example/dl/a.tar.gz", False),
("https://git.example/api/v1/releases/latest", "https://cdn.example/dl/a.tar.gz", False),
("http://git.example:3000/x", "http://git.example:3001/x", False),
],
)
def test_token_for_asset_compares_scheme_host_and_port(feed_url, asset_url, sent):
assert version_mod.token_for_asset(feed_url, asset_url, "t") == ("t" if sent else None)
assert version_mod.token_for_asset(feed_url, asset_url, None) is None
# --- the asset names --------------------------------------------------------
def test_release_asset_names_match_the_workflow():
workflow = Path(__file__).resolve().parents[3] / ".gitea" / "workflows" / "release.yml"
if not workflow.is_file():
pytest.skip("a distributed instance carries no release workflow")
text = workflow.read_text(encoding="utf-8")
archive = version_mod.ARCHIVE_NAME.format(version="${VERSION}")
stem = archive.removesuffix(".tar.gz")
assert f'name="{stem}"' in text
assert archive.replace(stem, "${name}") in text
assert (archive + version_mod.CHECKSUM_SUFFIX).replace(stem, "${name}") in text
def test_fetch_latest_assets_reads_urls_off_the_release_and_skips_malformed_entries():
payload = json.dumps({
"tag_name": "v1.1.0",
"assets": [
{"name": "a.tar.gz", "browser_download_url": "https://h/a"},
{"name": "no-url"},
"not an object",
],
}).encode()
release = version_mod.fetch_latest_assets("https://h/feed", fetcher=lambda u, t, to: payload)
assert release.version == version_mod.Version.parse("1.1.0")
assert release.tag == "v1.1.0"
assert release.assets == {"a.tar.gz": "https://h/a"}
with pytest.raises(version_mod.VersionError, match="assets present: a.tar.gz"):
release.asset_urls()
+124 -5
View File
@@ -37,7 +37,9 @@ import functools
import json import json
import os import os
import re import re
import shutil
import urllib.error import urllib.error
import urllib.parse
import urllib.request import urllib.request
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@@ -349,12 +351,15 @@ def fetch_latest(
The network call sits behind `fetcher` so every caller above this line - The network call sits behind `fetcher` so every caller above this line -
and every test - can run without a network. This is the one place that and every test - can run without a network. This is the one place that
talks to the **release feed**, and only two commands reach it: `version talks to the **release feed**, and only three commands reach it: `version
check`, whose whole job it is, and `version notes` on a *distributed* check`, whose whole job it is, `version notes` on a *distributed*
instance, whose local `CHANGES.md` is a stub with no entry to print (see instance, whose local `CHANGES.md` is a stub with no entry to print (see
`fetch_latest_notes`). Neither is implicit - `check` exists for the call, `fetch_latest_notes`), and `dist upgrade --latest`, which asks the feed
and `notes` announces the URL it is asking before it asks, on stderr, and which release to download (see `fetch_latest_assets`). None is implicit -
takes `--offline` for a caller that wants none of it. `check` exists for the call, `notes` announces the URL it is asking before
it asks, on stderr, and takes `--offline` for a caller that wants none of
it, and `upgrade` reaches the feed only when the operator passes
`--latest`.
""" """
fetch = fetcher or _urlopen_fetch fetch = fetcher or _urlopen_fetch
try: try:
@@ -462,6 +467,120 @@ def fetch_latest_notes(
return version, body, (str(html_url) if html_url else None) return version, body, (str(html_url) if html_url else None)
# --- release assets --------------------------------------------------------
#
# `dist upgrade --latest` downloads the two assets `.gitea/workflows/release.yml`
# attaches to every release. The names are that workflow's `name=` line plus
# `.tar.gz` / `.tar.gz.sha256`; `test_release_asset_names_match_the_workflow`
# ties this constant to the workflow so one cannot move without the other.
ARCHIVE_NAME = "chemenu-stack-{version}.tar.gz"
CHECKSUM_SUFFIX = ".sha256"
DOWNLOAD_TIMEOUT = 60.0
@dataclass(frozen=True)
class LatestRelease:
"""What the feed's latest release offers: its version, its tag as the feed
spells it, its own page, and every asset as `name -> download URL`."""
version: Version
tag: str
html_url: Optional[str]
assets: dict[str, str]
@property
def archive_name(self) -> str:
return ARCHIVE_NAME.format(version=self.version)
def asset_urls(self) -> tuple[str, str]:
"""`(archive URL, checksum URL)`, read off the release object and never
composed. Raises `VersionError` naming the release page when either is
missing - an upgrade without its checksum is refused, not downgraded to
an unchecked one."""
archive = self.archive_name
wanted = (archive, archive + CHECKSUM_SUFFIX)
missing = [name for name in wanted if name not in self.assets]
if missing:
present = ", ".join(sorted(self.assets)) or "none"
page = f" - see the release page {self.html_url}" if self.html_url else ""
raise VersionError(
f"release {self.tag} does not publish {' and '.join(missing)} "
f"(assets present: {present}){page}"
)
return self.assets[wanted[0]], self.assets[wanted[1]]
def fetch_latest_assets(
url: str,
token: Optional[str] = None,
timeout: float = 10.0,
fetcher: Optional[Fetcher] = None,
) -> LatestRelease:
"""`fetch_latest`'s version plus the release's assets, from the one
response - the asset URLs are the feed's own `browser_download_url`
values, never a path this code composes (AGENTS.md invariant 7)."""
version, captured = _fetch_latest_object(url, token, timeout, fetcher)
assets: dict[str, str] = {}
raw_assets = captured.get("assets")
if isinstance(raw_assets, list):
for entry in raw_assets:
if not isinstance(entry, dict):
continue
name = str(entry.get("name") or "").strip()
download = str(entry.get("browser_download_url") or "").strip()
if name and download:
assets[name] = download
html_url = captured.get("html_url") or captured.get("url")
return LatestRelease(
version=version,
tag=str(captured.get("tag_name") or "").strip(),
html_url=str(html_url) if html_url else None,
assets=assets,
)
def _origin(url: str) -> tuple[str, str, int]:
parts = urllib.parse.urlsplit(url)
scheme = parts.scheme.lower()
default = {"http": 80, "https": 443}.get(scheme, 0)
return scheme, (parts.hostname or "").lower(), parts.port or default
def token_for_asset(feed_url: str, asset_url: str, token: Optional[str]) -> Optional[str]:
"""The token to send with an asset download: `token` when the asset is on
the feed's own origin (scheme, host, port), `None` otherwise. An asset URL
is data from a JSON answer, so the credential does not follow it to
another host."""
return token if token and _origin(feed_url) == _origin(asset_url) else None
def download_asset(
url: str,
dest: Path,
token: Optional[str] = None,
timeout: float = DOWNLOAD_TIMEOUT,
) -> None:
"""Stream `url` into `dest`. `timeout` is a socket timeout per read, not a
deadline for the whole transfer. The token goes out as an *unredirected*
header, so a redirect to another host does not carry it along. Every
failure is a `VersionError` naming the URL."""
request = urllib.request.Request(url)
if token:
request.add_unredirected_header("Authorization", f"token {token}")
try:
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310 - URL from the feed
with dest.open("wb") as out:
shutil.copyfileobj(response, out)
except urllib.error.HTTPError as exc:
hint = ""
if exc.code in (401, 403):
hint = f" - the download needs authentication; set ${UPDATE_TOKEN_ENV}"
raise VersionError(f"{url} answered HTTP {exc.code}{hint}") from exc
except (urllib.error.URLError, OSError, TimeoutError) as exc:
raise VersionError(f"Could not download {url}: {exc}") from exc
# --- CHANGES.md ------------------------------------------------------------ # --- CHANGES.md ------------------------------------------------------------
# #
# The changelog is prose and stays the author's job. What is mechanical is the # The changelog is prose and stays the author's job. What is mechanical is the