torben/chemenu
1
1 Fork 0
Code Issues 35 Pull Requests Actions Packages Releases 37 Activity

feat: preflight as a release asset - download, verify, unpack, then run the tree preflight (#151, C)
CI / verify (push) Successful in 2m26s
Details
CI / pwsh (push) Failing after 11s
Details
Release / release (push) Successful in 37s
Details

Browse Source 
Files changed:
- .gitea/workflows/release.yml
- CHANGES.md
- INSTALL.md
- README.md
- VERSION
- instructions/dev/testing-conventions.md
- instructions/preflight.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/preflight.ps1
- tools/preflight.sh
This commit is contained in:
torben committed 2026-10-01 13:39:56 +02:00
1 parent 210e0c8286
commit c33e8cdfb1
13 files changed
+1020 -88
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitea/workflows/release.yml
+26 -2
View File
@@ -3,7 +3,9 @@
# The release artifact is exactly a `dist export` tree, packed with a top-level
# directory: unpack it, run instructions/setup-instance.md, and there is a
# working wiki instance - no checkout of this repo required. CI already proved
# that path works before this workflow ever runs.
# that path works before this workflow ever runs. Beside it the release carries
# tools/preflight.sh and tools/preflight.ps1 as assets, which download and unpack
# that tarball themselves.
#
# The tag is created here, by CI, and never by an agent: AGENTS.md invariant 5
# ("never call raw git commit/push") stays intact because nothing in a session
@@ -149,6 +151,28 @@ jobs:
cat "${BUILD_DIR}/${name}.tar.gz.sha256"
echo "name=${name}" >> "$GITHUB_OUTPUT"
# The two preflight scripts are attached to the release as well: the first
# thing a new user runs, before there is any tree to run it from. Each copy
# is the tree's script with the download address of *this* release written
# into its two placeholder lines (the tree copy keeps them empty, which is
# how a script knows it is not a release asset). The address is the public
# one, for the same reason as the URLs in `dist export` above.
download="${PUBLIC_BASE_URL}/${GITHUB_REPOSITORY}/releases/download/${TAG}"
sed \
-e "s|^RELEASE_ARCHIVE_URL=''|RELEASE_ARCHIVE_URL='${download}/${name}.tar.gz'|" \
-e "s|^RELEASE_CHECKSUM_URL=''|RELEASE_CHECKSUM_URL='${download}/${name}.tar.gz.sha256'|" \
tools/preflight.sh > "${BUILD_DIR}/preflight.sh"
sed \
-e "s|^\$ReleaseArchiveUrl = ''|\$ReleaseArchiveUrl = '${download}/${name}.tar.gz'|" \
-e "s|^\$ReleaseChecksumUrl = ''|\$ReleaseChecksumUrl = '${download}/${name}.tar.gz.sha256'|" \
tools/preflight.ps1 > "${BUILD_DIR}/preflight.ps1"
# A placeholder that did not match would ship a script that refuses to run.
grep -qF "RELEASE_ARCHIVE_URL='${download}/${name}.tar.gz'" "${BUILD_DIR}/preflight.sh"
grep -qF "RELEASE_CHECKSUM_URL='${download}/${name}.tar.gz.sha256'" "${BUILD_DIR}/preflight.sh"
grep -qF "ReleaseArchiveUrl = '${download}/${name}.tar.gz'" "${BUILD_DIR}/preflight.ps1"
grep -qF "ReleaseChecksumUrl = '${download}/${name}.tar.gz.sha256'" "${BUILD_DIR}/preflight.ps1"
chmod +x "${BUILD_DIR}/preflight.sh"
- name: Publish the release
if: steps.version.outputs.skip != 'true'
env:
@@ -174,7 +198,7 @@ jobs:
id="$(printf '%s' "$release" | jq -r '.id')"
echo "Created release ${TAG} (id ${id})."
for asset in "${NAME}.tar.gz" "${NAME}.tar.gz.sha256"; do
for asset in "${NAME}.tar.gz" "${NAME}.tar.gz.sha256" preflight.sh preflight.ps1; do
curl -sS -f -X POST "${API}/releases/${id}/assets?name=${asset}" \
-H "Authorization: token ${TOKEN}" \
-F "attachment=@${BUILD_DIR}/${asset}" > /dev/null
CHANGES.md
+36 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.15 - 2026-10-01 - PowerShell 7 preflight and launcher: tools/preflight.ps1, tools/wikitool.ps1, doctor checks for execution policy and Mark of the Web
## 8.0.0-beta.16 - 2026-10-01 - Preflight as a release asset: download, verify and unpack the stack, then run the tree preflight
**Author:** Torben Nehmer
@@ -90,6 +90,7 @@ concern - readable here, never shipped as something to parse.
- publish: the gate lists the staged state; a missing or unreachable remote stops before the commit
- Path budget: a file's path stays at 160 characters or fewer so a Windows checkout works without long paths (#163)
- PowerShell 7 preflight and launcher: tools/preflight.ps1, tools/wikitool.ps1, doctor checks for execution policy and Mark of the Web
- Preflight as a release asset: download, verify and unpack the stack, then run the tree preflight
**Low impact**
- version bump no longer points at version release in its output
@@ -126,6 +127,40 @@ concern - readable here, never shipped as something to parse.
- raw/CONTRACT.md points at the path budget for a name accepted from incoming/
<!-- /wikitool:bumps -->
### Preflight as a release asset: download, verify and unpack the stack, then run the tree preflight
Section C of #151, the first install. Until now a new user had to get the stack onto the machine
before any preflight could run - which is exactly the step that fails on a machine without git
or a short enough path. Each release now also attaches `preflight.sh` and `preflight.ps1` as
assets, and a script without `tools/prerequisites.txt` beside it runs in **asset mode**: it
downloads the release tarball and its `.sha256`, stops with exit 1 unless the checksum matches,
reads the folder limit from `tools/prerequisites.txt` inside the archive, and unpacks into
`chemenu/` next to itself (`--into <path>` for another place). It unpacks into a temporary
sibling and renames, requires exactly one top-level folder, and refuses an existing target
without touching it. Then it runs the unpacked tree's own preflight, passing `--set` and the exit
code through, so everything after the unpack is the tree mode that already existed.
The asset copies are made by `release.yml`: it writes the same release's tarball and checksum
URLs into two placeholder lines of each script, checks that the substitution took, and uploads
the two files under exactly those names. The tree copies keep the placeholders empty; an asset
script with empty placeholders and no `--archive` exits 1 saying it does not come from a release.
`--archive <tarball>` uses a tarball already on disk (its `<tarball>.sha256` must sit beside it)
for a machine that cannot download, and is the entry point the tests use.
On POSIX asset mode needs `curl`, `tar` and `sha256sum` (or `shasum`) and stops with exit 42 and
the usual guidance block when one is missing; the PowerShell script uses what Windows ships plus
`tar`. On Windows with long paths off, the 95-character folder limit is judged at the final
target before anything is unpacked, so a too-long `--into` stops with exit 42 and the fix is a
shorter folder. Git Bash gets the target and archive converted with `cygpath -u`, because GNU
tar would read `C:` as a host.
The tests build a release tarball with a checksum and run both scripts against it, including a
local HTTP server for the PowerShell download, the 95/96 boundary, a failed checksum, two
top-level folders, an existing target, and a test that ties the placeholder lines to the `sed`
expressions in `release.yml`. `tools/CONTRACT.md`, `tools/README.md`, `README.md`,
`instructions/preflight.md` (a new decision point for the asset script),
`instructions/dev/testing-conventions.md` and `INSTALL.md` describe the first-install route.
### PowerShell 7 preflight and launcher: tools/preflight.ps1, tools/wikitool.ps1, doctor checks for execution policy and Mark of the Web
The PowerShell half of #151. Harnesses that run in PowerShell 7 on Windows (GitHub Copilot CLI,
INSTALL.md
+6
View File
@@ -46,6 +46,12 @@ Die Prüfsumme ist nicht Zierde: Sie ist das Einzige, was einen unterbrochenen D
einem vollständigen unterscheidet, und `sha256sum -c` muss `OK` sagen, bevor irgendetwas
entpackt wird.
Statt dieser Befehle von Hand trägt jedes Release auch den Preflight selbst als Datei
(`preflight.sh`, unter Windows `preflight.ps1`). In einen leeren Ordner geladen und dort
gestartet, lädt er das Release herunter, prüft die Prüfsumme, entpackt es nach `chemenu/` neben
sich (mit `--into <Pfad>` woandershin; ein vorhandenes Ziel wird nie angefasst) und führt dann
den Preflight im entpackten Baum aus, siehe [instructions/preflight.md](instructions/preflight.md).
Danach weiter mit Schritt 2 aus Weg B: den Agenten
[instructions/setup-instance.md](instructions/setup-instance.md) ausführen lassen. Der
entpackte Baum ist bereits eine Distribution - Schritt 1 (`dist export`) entfällt.
README.md
+3 -1
View File
@@ -39,7 +39,9 @@ Two starting points, depending on what you're doing - full walkthrough in [INSTA
```
`tools/wikitool` refuses to start (exit 42) until the preflight has passed; if it stops
instead, its output says what to install - `instructions/preflight.md`.
instead, its output says what to install - `instructions/preflight.md`. A release carries the
same two scripts as assets that download and unpack the stack themselves, for installing
without a clone.
That copies each `instructions/<name>/SKILL.md` into `.agents/skills/` (GitHub Copilot, Codex
CLI, Mistral Vibe) and `.claude/skills/` (Claude Code). Re-run it after changing a skill.
VERSION
+1 -1
View File
@@ -1 +1 @@
8.0.0-beta.15
8.0.0-beta.16
instructions/dev/testing-conventions.md
+5
View File
@@ -165,6 +165,11 @@ Whenever you add or change a test under `tools/chemenu/tests/`.
## Decision points
- **Testing a PowerShell script?** `test_preflight_pwsh.py` needs `pwsh` and skips silently
without it - a green run on a machine with no PowerShell 7 has not run those tests. Check
`command -v pwsh` before trusting the result; CI's `pwsh` job runs them in the image built from
`.gitea/pwsh-ci/Dockerfile`, and so can you (`docker run` that image with the checkout mounted
and `pytest tools/chemenu/tests/test_preflight_pwsh.py` as the command).
- **A test genuinely needs the developer's real environment?** There is no such test, and a new
one is a design problem rather than an exception: what it wants is a fixture that *builds*
the state it needs inside `tmp_path`. Building it is also the only version CI can run.
instructions/preflight.md
+22 -2
View File
@@ -68,7 +68,7 @@ It is safe to run at any time: a second run on a ready checkout changes nothing
|---|---|---|
| 0 | Everything is in place | Continue with the procedure that sent you here |
| 42 | The user has to act | Step 3 |
| 1 | Called wrongly, or `tools/prerequisites.txt` is missing next to the script | Report the exact command and output to the user; do not retry blindly |
| 1 | Called wrongly, a download or unpack failed (asset mode), or the stack tree next to the script is incomplete | Report the exact command and output to the user; do not retry blindly |
3. **On exit 42, show the output to the user exactly as it is, then stop and wait.** It is
written for someone without an IT background: each numbered block says what is missing, why
@@ -99,10 +99,30 @@ It is safe to run at any time: a second run on a ready checkout changes nothing
## Decision points
- **The script is a release asset, not a tree script** - there is no `tools/prerequisites.txt`
beside it, because the user downloaded `preflight.sh` or `preflight.ps1` from a release page
into an empty folder. That is the *first* install, and the script does one more thing before
the steps above: it downloads the release tarball and its `.sha256`, refuses unless the
checksum matches, and unpacks into a `chemenu/` folder next to itself; then it runs the
preflight of the unpacked tree, passing `--set` and its exit code through. Run it exactly as
in step 1 (the path is the downloaded file, not `tools/...`), and read the exit code the same
way. After it, every later run - including the retry after an exit 42 - is the tree's own
`tools/preflight.sh` or `tools/preflight.ps1`, run from inside `chemenu/`.
- `--into <path>` unpacks somewhere else; an existing target is refused with exit 1 and
nothing is touched, which is the user's decision to make, not yours to resolve by deleting.
- `--archive <tarball>` uses a tarball already on disk, with its `<tarball>.sha256` beside it,
when the machine cannot download.
- A checksum that does not match, a failed download, and a copy of the script that carries no
download address (it was not taken from a release) exit 1 with nothing unpacked; report the
message, and do not fetch the tarball by another route.
- On POSIX, `curl`, `tar` and `sha256sum` (or `shasum`) have to exist; when one does not, the
script stops with exit 42 like any other missing tool. The PowerShell script needs nothing
beyond what Windows ships.
- **The output names a folder that is too long.** Only on Windows with long paths off: the
install folder may be at most 95 characters, because every file of the wiki below it has to
stay within 259. Moving the wiki to a shorter folder is the user's step; do not try to shorten
paths inside the wiki instead.
paths inside the wiki instead. In asset mode the length is judged at the folder the stack
*would* be unpacked into, before anything is unpacked; the fix is a shorter `--into`.
- **The output names the PowerShell execution policy** (`Restricted` or `AllSigned`). The fix is a
line the user runs in a PowerShell 7 window; it changes a setting of their account, so it is
theirs to run. When a *group policy* sets it, nothing on this computer can override it: the
tools/CONTRACT.md
+5
View File
@@ -65,6 +65,11 @@ From PowerShell 7 on Windows, the twin: `pwsh -NoProfile -ExecutionPolicy Bypass
Until it has passed, every `tools/wikitool` call exits 42 and names it.
A release also carries both scripts as assets (`preflight.sh`, `preflight.ps1`) for the very
first install, before there is a tree: run from a folder with no `tools/prerequisites.txt` beside
them, they download and verify the release tarball, unpack it into `chemenu/` next to themselves
(or `--into <path>`), and run the preflight inside the unpacked tree.
## Usage
Run from the repo root:
tools/README.md
+12 -1
View File
@@ -32,6 +32,17 @@ it with `-m pip`. It is POSIX sh because it has to run before Python is known
to exist; exit 42 means the user has to act, and its output says how. The
procedure an agent follows around it is `instructions/preflight.md`.
Each release also attaches `preflight.sh` and `preflight.ps1` as assets, for the first install
before any tree exists. `release.yml` writes the download address of that same release into two
placeholder lines of the copy (`RELEASE_ARCHIVE_URL`/`RELEASE_CHECKSUM_URL` in the shell script,
`$ReleaseArchiveUrl`/`$ReleaseChecksumUrl` in the PowerShell one; the tree copy leaves them
empty). With no `prerequisites.txt` beside it, the script is in *asset mode*: it downloads the
tarball and its `.sha256`, refuses on a mismatch, reads the folder limit out of the archive,
unpacks into `<script folder>/chemenu` (`--into <path>` to choose, an existing target is refused),
and runs the tree copy of itself, passing `--set` and its exit code through. `--archive <tarball>`
takes a local tarball instead of a download (its `.sha256` has to lie next to it); that is also
how the tests drive it.
`tools/wikitool` refuses to start (exit 42) until the preflight has written a
*complete* `.wikitool-tools.json` and the venv exists. Past that, every `git`
and `rg` the package starts goes through `chemenu/toolpaths.py`, which reads
@@ -51,7 +62,7 @@ tools/
wikitool entry point (POSIX sh): stops with exit 42 until the preflight has passed
wikitool.ps1 the same entry point for PowerShell 7, which resolves `tools/wikitool` to this file first
run_wikitool.py what the launcher runs with the venv's Python - puts chemenu on sys.path without PYTHONPATH
preflight.sh checks prerequisites.txt, records .wikitool-tools.json, creates .venv (POSIX sh)
preflight.sh checks prerequisites.txt, records .wikitool-tools.json, creates .venv (POSIX sh); as the release asset, downloads and unpacks the stack first
preflight.ps1 the same for PowerShell 7; also checks the execution policy and the Mark of the Web
prerequisites.txt what the machine needs, one `|`-separated line per tool - read by the preflight and `doctor`
trace-hook what the harness hooks call: trace_ingest.py under the venv's Python
tools/chemenu/tests/test_preflight.py
+295 -3
View File
@@ -11,12 +11,14 @@ preflight for real.
"""
from __future__ import annotations
import hashlib
import json
import os
import shutil
import stat
import subprocess
import sys
import tarfile
from pathlib import Path
import pytest
@@ -28,7 +30,7 @@ TOOLS = config._PACKAGE_ROOT / "tools"
# What preflight.sh, the launcher and trace-hook call besides shell built-ins
# and the tools under test. `iconv` is optional in the script itself.
UTILITIES = ("dirname", "uname", "sed", "tr", "wc", "iconv", "tail", "cat", "mv", "head",
"cut", "grep", "mkdir", "cp", "rm")
"cut", "grep", "mkdir", "cp", "rm", "tar", "gzip", "sha256sum", "mktemp")
def _shells() -> list[str]:
@@ -93,6 +95,7 @@ class Machine:
self.base = base
self.root = base / root_name
self.tools = self.root / "tools"
self.script = self.tools / "preflight.sh"
self.tools.mkdir(parents=True)
for name in ("preflight.sh", "preflight.ps1", "prerequisites.txt", "wikitool", "wikitool.ps1",
"run_wikitool.py", "trace-hook"):
@@ -108,6 +111,7 @@ class Machine:
self.pip_log = base / "pip.log"
self.extra_env: dict[str, str] = {}
self.path_dirs: list[Path] = [self.stubs, self.sysbin]
self.cwd: Path | None = None
def stub(self, name: str, text: str, where: Path | None = None) -> Path:
return _executable((where or self.stubs) / name, text)
@@ -132,8 +136,8 @@ class Machine:
**self.extra_env,
}
return subprocess.run(
[shell, str(self.tools / "preflight.sh"), *args],
capture_output=True, text=True, env=env, timeout=60,
[shell, str(self.script), *args],
capture_output=True, text=True, env=env, timeout=60, cwd=self.cwd,
)
@property
@@ -354,6 +358,294 @@ def test_manifest_names_the_tools_the_stack_starts():
assert [t.name for t in manifest.tools_for("windows")] == ["python", "git", "rg", "pwsh"]
# --- asset mode (D33, D38-D41) ------------------------------------------------------
RELEASE_VERSION = "9.9.9"
DOWNLOAD_BASE = f"https://example.test/torben/chemenu/releases/download/v{RELEASE_VERSION}"
STACK_FILES = ("preflight.sh", "preflight.ps1", "prerequisites.txt", "wikitool", "wikitool.ps1",
"run_wikitool.py", "trace-hook")
# The two lines each script keeps for the release build to fill: (empty, filled).
PLACEHOLDERS = {
"preflight.sh": (
("RELEASE_ARCHIVE_URL=''", f"RELEASE_ARCHIVE_URL='{DOWNLOAD_BASE}/chemenu-stack-{RELEASE_VERSION}.tar.gz'"),
("RELEASE_CHECKSUM_URL=''", f"RELEASE_CHECKSUM_URL='{DOWNLOAD_BASE}/chemenu-stack-{RELEASE_VERSION}.tar.gz.sha256'"),
),
"preflight.ps1": (
("$ReleaseArchiveUrl = ''", f"$ReleaseArchiveUrl = '{DOWNLOAD_BASE}/chemenu-stack-{RELEASE_VERSION}.tar.gz'"),
("$ReleaseChecksumUrl = ''", f"$ReleaseChecksumUrl = '{DOWNLOAD_BASE}/chemenu-stack-{RELEASE_VERSION}.tar.gz.sha256'"),
),
}
CURL_STUB = """#!/bin/sh
# Serves the files of $RELEASE_DIR by the last part of the URL, and logs every request.
while [ $# -gt 0 ]; do
case "$1" in
-o) dest=$2; shift ;;
-*) ;;
*) url=$1 ;;
esac
shift
done
echo "$url" >> "$CURL_LOG"
src="$RELEASE_DIR/${url##*/}"
[ -f "$src" ] || exit 22
cp "$src" "$dest"
"""
def build_release(base: Path, *, limit: int | None = None, tops: tuple[str, ...] | None = None) -> Path:
"""A release tarball and its .sha256 as CI builds them; returns the tarball."""
name = f"chemenu-stack-{RELEASE_VERSION}"
stage = base / "stage"
for top in tops or (name,):
(stage / top / "tools").mkdir(parents=True)
for file in STACK_FILES:
shutil.copy2(TOOLS / file, stage / top / "tools" / file)
(stage / top / "tools" / "requirements.txt").write_text("PyYAML\n", encoding="utf-8")
if limit is not None:
manifest = stage / top / "tools" / "prerequisites.txt"
manifest.write_text(
manifest.read_text(encoding="utf-8").replace(
"limit|install_dir_max|95", f"limit|install_dir_max|{limit}"),
encoding="utf-8")
release = base / "release"
release.mkdir()
tarball = release / f"{name}.tar.gz"
with tarfile.open(tarball, "w:gz") as archive:
for top in tops or (name,):
archive.add(stage / top, arcname=top)
digest = hashlib.sha256(tarball.read_bytes()).hexdigest()
(release / f"{tarball.name}.sha256").write_text(f"{digest} {tarball.name}\n", encoding="utf-8")
return tarball
class AssetMachine(Machine):
"""The release asset on a machine with nothing unpacked yet: just the script, alone in a folder."""
def __init__(self, base: Path, *, filled: bool = False, **release):
super().__init__(base)
self.tarball = build_release(base, **release)
self.download = base / "download"
self.download.mkdir()
for name, pairs in PLACEHOLDERS.items():
text = (TOOLS / name).read_text(encoding="utf-8")
for empty, full in pairs:
assert empty in text, f"{name} lost its placeholder {empty}"
if filled:
text = text.replace(empty, full)
(self.download / name).write_text(text, encoding="utf-8")
self.script = self.download / "preflight.sh"
self.root = self.download / "chemenu"
self.tools = self.root / "tools"
self.cwd = self.download
self.extra_env.update({"RELEASE_DIR": str(self.tarball.parent), "CURL_LOG": str(base / "curl.log")})
self.standard()
def unpacked(self, root: Path | None = None) -> bool:
return (root or self.root).exists()
def leftovers(self) -> list[str]:
found = [str(path) for path in self.base.rglob(".chemenu-unpack.*")]
found += [str(path) for path in Path(os.environ.get("TMPDIR", "/tmp")).glob("chemenu-preflight.*")]
return found
@pytest.fixture
def asset(tmp_path: Path) -> AssetMachine:
return AssetMachine(tmp_path.resolve())
@pytest.mark.parametrize("shell", SHELLS)
def test_asset_unpacks_next_to_itself_and_runs_the_tree_copy(asset, shell):
result = asset.run("--archive", str(asset.tarball), shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
assert "sha256 OK" in result.stdout and "Preflight passed" in result.stdout
assert (asset.tools / "preflight.sh").is_file() and (asset.tools / "prerequisites.txt").is_file()
assert asset.recorded()["complete"] is True
assert (asset.tools / ".venv").is_dir()
assert not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
def test_into_chooses_the_target_and_creates_missing_parents(asset, shell):
target = asset.base / "deep" / "er" / "wiki"
result = asset.run("--archive", str(asset.tarball), "--into", str(target), shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
assert (target / "tools" / "preflight.sh").is_file()
assert (target / ".wikitool-tools.json").is_file()
assert not asset.unpacked()
@pytest.mark.parametrize("shell", SHELLS)
def test_a_relative_into_resolves_against_the_working_directory(asset, shell):
asset.cwd = asset.base
result = asset.run("--archive", str(asset.tarball), "--into", "here", shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
assert (asset.base / "here" / "tools" / "preflight.sh").is_file()
@pytest.mark.parametrize("shell", SHELLS)
def test_an_existing_target_is_refused_and_left_alone(asset, shell):
asset.root.mkdir()
(asset.root / "mine.txt").write_text("keep", encoding="utf-8")
result = asset.run("--archive", str(asset.tarball), shell=shell)
assert result.returncode == 1
assert "already exists" in result.stderr and "--into" in result.stderr
assert [path.name for path in asset.root.iterdir()] == ["mine.txt"]
assert not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
def test_a_wrong_sha256_exits_1_and_unpacks_nothing(asset, shell):
checksum = asset.tarball.parent / f"{asset.tarball.name}.sha256"
checksum.write_text("0" * 64 + f" {asset.tarball.name}\n", encoding="utf-8")
result = asset.run("--archive", str(asset.tarball), shell=shell)
assert result.returncode == 1
assert "does not match" in result.stderr and "nothing was unpacked" in result.stderr
assert not asset.unpacked() and not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
def test_archive_without_a_checksum_file_exits_1(asset, shell):
(asset.tarball.parent / f"{asset.tarball.name}.sha256").unlink()
result = asset.run("--archive", str(asset.tarball), shell=shell)
assert result.returncode == 1
assert ".sha256 is missing" in result.stderr
assert not asset.unpacked()
@pytest.mark.parametrize("shell", SHELLS)
def test_an_archive_with_two_top_level_folders_exits_1(tmp_path, shell):
asset = AssetMachine(tmp_path.resolve(), tops=(f"chemenu-stack-{RELEASE_VERSION}", "stray"))
result = asset.run("--archive", str(asset.tarball), shell=shell)
assert result.returncode == 1
assert "exactly one top-level folder" in result.stderr
assert not asset.unpacked() and not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
def test_a_copy_without_download_addresses_says_it_is_no_release_asset(asset, shell):
result = asset.run(shell=shell)
assert result.returncode == 1
assert "does not come from a release" in result.stderr
assert not asset.unpacked()
@pytest.mark.parametrize("shell", SHELLS)
def test_download_fetches_both_files_of_the_same_release(tmp_path, shell):
asset = AssetMachine(tmp_path.resolve(), filled=True)
asset.stub("curl", CURL_STUB)
result = asset.run(shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
requested = (asset.base / "curl.log").read_text(encoding="utf-8").split()
assert requested == [f"{DOWNLOAD_BASE}/{asset.tarball.name}", f"{DOWNLOAD_BASE}/{asset.tarball.name}.sha256"]
assert (asset.tools / "preflight.sh").is_file()
assert not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
def test_a_failed_download_exits_1_and_unpacks_nothing(tmp_path, shell):
asset = AssetMachine(tmp_path.resolve(), filled=True)
asset.stub("curl", CURL_STUB)
asset.extra_env["RELEASE_DIR"] = str(asset.base / "nothing-here")
result = asset.run(shell=shell)
assert result.returncode == 1
assert "the download failed" in result.stderr
assert not asset.unpacked() and not asset.leftovers()
@pytest.mark.parametrize("shell", SHELLS)
@pytest.mark.parametrize("missing", ["curl", "tar", "sha256sum"])
def test_a_missing_download_tool_is_a_stop_with_guidance(tmp_path, shell, missing):
asset = AssetMachine(tmp_path.resolve(), filled=True)
asset.stub("curl", CURL_STUB)
(asset.stubs / "curl").unlink() if missing == "curl" else (asset.sysbin / missing).unlink()
result = asset.run(shell=shell)
assert result.returncode == 42
assert_guidance(result.stdout, f"could not be found: {missing}")
assert not asset.unpacked() and not (asset.base / "curl.log").exists()
@pytest.mark.parametrize("shell", SHELLS)
def test_with_an_archive_no_download_tool_is_needed(asset, shell):
assert not (asset.stubs / "curl").exists() and not (asset.sysbin / "curl").exists()
assert asset.run("--archive", str(asset.tarball), shell=shell).returncode == 0
@pytest.mark.parametrize("shell", SHELLS)
def test_set_and_the_exit_code_pass_through_to_the_tree_copy(asset, shell):
refused = asset.run("--archive", str(asset.tarball), "--set", f"rg={asset.base / 'nowhere' / 'rg'}", shell=shell)
assert refused.returncode == 42
assert_guidance(refused.stdout, "The path given for ripgrep (rg) does not work")
assert (asset.tools / "preflight.sh").is_file() and not asset.tools_file.exists()
elsewhere = asset.stub("rg", "#!/bin/sh\necho 'ripgrep 14.1.1'\n", asset.base / "opt")
second = AssetMachine(asset.base / "second")
second.stub("rg", "#!/bin/sh\necho 'ripgrep 14.1.1'\n", asset.base / "opt")
ok = second.run("--archive", str(second.tarball), "--set", f"rg={elsewhere}", shell=shell)
assert ok.returncode == 0, ok.stdout
assert second.recorded()["tools"]["rg"] == str(elsewhere)
@pytest.mark.parametrize("shell", SHELLS)
@pytest.mark.parametrize("flag", ["--into", "--archive"])
def test_asset_options_are_a_usage_error_inside_a_tree(machine, shell, flag):
machine.standard()
result = machine.run(flag, str(machine.base), shell=shell)
assert result.returncode == 1
assert "belong to the release asset" in result.stderr
@pytest.mark.parametrize("shell", SHELLS)
@pytest.mark.parametrize("length, longpaths, limit, expected", [
(95, "0", None, 0),
(96, "0", None, 42),
(96, "1", None, 0),
(110, "0", 120, 0),
(121, "0", 120, 42),
])
def test_the_folder_limit_is_judged_at_the_final_target_from_the_archive(
tmp_path, shell, length, longpaths, limit, expected):
base = tmp_path.resolve()
name_length = length - len(str(base / "download")) - 1
assert name_length > 0, "tmp_path is too long for this test"
asset = AssetMachine(base, limit=limit)
asset.standard(pwsh=True)
asset.extra_env.update({"CHEMENU_PREFLIGHT_PLATFORM": "windows", "CHEMENU_PREFLIGHT_LONGPATHS": longpaths})
target = asset.download / ("t" * name_length)
assert len(str(target)) == length
result = asset.run("--archive", str(asset.tarball), "--into", str(target), shell=shell)
assert result.returncode == expected, result.stdout + result.stderr
if expected == 42:
assert_guidance(result.stdout, f"too long ({length} characters, at most {limit or 95})")
assert not asset.unpacked(target) and not asset.leftovers()
else:
assert (target / "tools" / "preflight.sh").is_file()
def test_the_release_workflow_fills_exactly_the_placeholders_the_scripts_keep():
workflow = Path(__file__).resolve().parents[3] / ".gitea" / "workflows" / "release.yml"
if not workflow.is_file():
pytest.skip("a distributed instance carries no release workflow")
text = workflow.read_text(encoding="utf-8")
def as_written_in_the_workflow(line: str) -> str:
return (line.replace("$Release", "\\$Release")
.replace(DOWNLOAD_BASE, "${download}")
.replace(f"chemenu-stack-{RELEASE_VERSION}", "${name}"))
for script, pairs in PLACEHOLDERS.items():
source = (TOOLS / script).read_text(encoding="utf-8")
assert f'tools/{script} > "${{BUILD_DIR}}/{script}"' in text
for empty, filled in pairs:
assert source.count(empty) == 1, f"{script} must hold {empty} exactly once"
expression = f"s|^{as_written_in_the_workflow(empty)}|{as_written_in_the_workflow(filled)}|"
assert expression in text, f"release.yml does not run {expression}"
assert 'download="${PUBLIC_BASE_URL}/${GITHUB_REPOSITORY}/releases/download/${TAG}"' in text
assert 'for asset in "${NAME}.tar.gz" "${NAME}.tar.gz.sha256" preflight.sh preflight.ps1; do' in text
# --- the launcher -----------------------------------------------------------------
tools/chemenu/tests/test_preflight_pwsh.py
+171 -1
View File
@@ -13,17 +13,21 @@ compares them byte for byte.
"""
from __future__ import annotations
import functools
import http.server
import json
import os
import shutil
import subprocess
import threading
from pathlib import Path
import pytest
from chemenu import prerequisites
from chemenu.tests.test_preflight import (
ECHO_PYTHON, SHELLS, TOOLS, Machine, _machine_with_root_of, assert_guidance,
DOWNLOAD_BASE, ECHO_PYTHON, SHELLS, TOOLS, AssetMachine, Machine, _machine_with_root_of,
assert_guidance,
)
PWSH = shutil.which("pwsh")
@@ -195,6 +199,172 @@ def test_install_folder_limit_at_the_boundary(tmp_path, length, longpaths, expec
assert not (machine.tools / ".venv").exists()
# --- asset mode (D33, D38-D41) ------------------------------------------------------
#
# The sh tests in test_preflight.py cover the same cases against preflight.sh; what is
# here is the twin's own wiring - Get-FileHash, tar.exe, Invoke-WebRequest - and the
# hand-over to the tree copy in a child pwsh.
def _asset_run(asset: AssetMachine, *args: str) -> subprocess.CompletedProcess:
env = {
"PATH": os.pathsep.join(str(d) for d in asset.path_dirs),
"HOME": str(asset.base),
"PIP_LOG": str(asset.pip_log),
"DOTNET_CLI_TELEMETRY_OPTOUT": "1",
"POWERSHELL_TELEMETRY_OPTOUT": "1",
**asset.extra_env,
}
return subprocess.run(
[PWSH, "-NoProfile", "-ExecutionPolicy", "Bypass", "-File", str(asset.download / "preflight.ps1"), *args],
capture_output=True, text=True, env=env, timeout=120, cwd=asset.cwd,
)
@pytest.fixture
def asset(tmp_path: Path) -> AssetMachine:
return AssetMachine(tmp_path.resolve())
def test_asset_unpacks_next_to_itself_and_runs_the_tree_copy(asset):
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 0, result.stdout + result.stderr
assert "sha256 OK" in result.stdout and "Preflight passed" in result.stdout
assert (asset.tools / "preflight.ps1").is_file()
assert asset.recorded()["complete"] is True
assert not asset.leftovers()
def test_into_chooses_the_target_and_a_relative_one_follows_the_working_directory(asset):
far = asset.base / "deep" / "er" / "wiki"
assert _asset_run(asset, "--archive", str(asset.tarball), "--into", str(far)).returncode == 0
assert (far / "tools" / "preflight.ps1").is_file() and not asset.unpacked()
asset.cwd = asset.base
assert _asset_run(asset, "--archive", str(asset.tarball), "--into", "here").returncode == 0
assert (asset.base / "here" / "tools" / "preflight.ps1").is_file()
def test_an_existing_target_is_refused_and_left_alone(asset):
asset.root.mkdir()
(asset.root / "mine.txt").write_text("keep", encoding="utf-8")
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 1
assert "already exists" in result.stderr and "--into" in result.stderr
assert [path.name for path in asset.root.iterdir()] == ["mine.txt"]
def test_a_wrong_sha256_exits_1_and_unpacks_nothing(asset):
checksum = asset.tarball.parent / f"{asset.tarball.name}.sha256"
checksum.write_text("0" * 64 + f" {asset.tarball.name}\n", encoding="utf-8")
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 1
assert "does not match" in result.stderr and "nothing was unpacked" in result.stderr
assert not asset.unpacked() and not asset.leftovers()
def test_archive_without_a_checksum_file_exits_1(asset):
(asset.tarball.parent / f"{asset.tarball.name}.sha256").unlink()
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 1 and ".sha256 is missing" in result.stderr
assert not asset.unpacked()
def test_an_archive_with_two_top_level_folders_exits_1(tmp_path):
asset = AssetMachine(tmp_path.resolve(), tops=("chemenu-stack-9.9.9", "stray"))
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 1 and "exactly one top-level folder" in result.stderr
assert not asset.unpacked() and not asset.leftovers()
def test_a_copy_without_download_addresses_says_it_is_no_release_asset(asset):
result = _asset_run(asset)
assert result.returncode == 1 and "does not come from a release" in result.stderr
assert not asset.unpacked()
def test_a_missing_tar_is_a_stop_with_guidance(asset):
(asset.sysbin / "tar").unlink()
result = _asset_run(asset, "--archive", str(asset.tarball))
assert result.returncode == 42
assert_guidance(result.stdout, "(tar) could not be found")
assert not asset.unpacked()
def test_set_and_the_exit_code_pass_through_to_the_tree_copy(asset):
refused = _asset_run(asset, "--archive", str(asset.tarball), "--set", f"rg={asset.base / 'nowhere' / 'rg'}")
assert refused.returncode == 42
assert_guidance(refused.stdout, "The path given for ripgrep (rg) does not work")
assert (asset.tools / "preflight.ps1").is_file() and not asset.tools_file.exists()
second = AssetMachine(asset.base / "second")
elsewhere = second.stub("rg", "#!/bin/sh\necho 'ripgrep 14.1.1'\n", asset.base / "opt")
ok = _asset_run(second, "--archive", str(second.tarball), "--set", f"rg={elsewhere}")
assert ok.returncode == 0, ok.stdout
assert second.recorded()["tools"]["rg"] == str(elsewhere)
@pytest.mark.parametrize("flag", ["--into", "--archive"])
def test_asset_options_are_a_usage_error_inside_a_tree(machine, flag):
result = _preflight(machine, flag, str(machine.base))
assert result.returncode == 1 and "belong to the release asset" in result.stderr
@pytest.mark.parametrize("length, longpaths, limit, expected", [
(95, "0", None, 0),
(96, "0", None, 42),
(96, "1", None, 0),
(110, "0", 120, 0),
(121, "0", 120, 42),
])
def test_the_folder_limit_is_judged_at_the_final_target_from_the_archive(
tmp_path, length, longpaths, limit, expected):
base = tmp_path.resolve()
name_length = length - len(str(base / "download")) - 1
assert name_length > 0, "tmp_path is too long for this test"
asset = AssetMachine(base, limit=limit)
asset.standard(pwsh=True)
asset.extra_env.update({"CHEMENU_PREFLIGHT_PLATFORM": "windows", "CHEMENU_PREFLIGHT_LONGPATHS": longpaths})
target = asset.download / ("t" * name_length)
result = _asset_run(asset, "--archive", str(asset.tarball), "--into", str(target))
assert result.returncode == expected, result.stdout + result.stderr
if expected == 42:
assert_guidance(result.stdout, f"too long ({length} characters, at most {limit or 95})")
assert not asset.unpacked(target) and not asset.leftovers()
else:
assert (target / "tools" / "preflight.ps1").is_file()
@pytest.fixture
def release_server(asset):
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(asset.tarball.parent))
handler.log_message = lambda *args: None
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
script = asset.download / "preflight.ps1"
script.write_text(
(TOOLS / "preflight.ps1").read_text(encoding="utf-8").replace(
"$ReleaseArchiveUrl = \'\'", f"$ReleaseArchiveUrl = \'http://127.0.0.1:{server.server_port}/{asset.tarball.name}\'").replace(
"$ReleaseChecksumUrl = \'\'", f"$ReleaseChecksumUrl = \'http://127.0.0.1:{server.server_port}/{asset.tarball.name}.sha256\'"),
encoding="utf-8")
yield asset
server.shutdown()
def test_download_fetches_both_files_and_unpacks(release_server):
result = _asset_run(release_server)
assert result.returncode == 0, result.stdout + result.stderr
assert "Downloading http://127.0.0.1" in result.stdout and "sha256 OK" in result.stdout
assert (release_server.tools / "preflight.ps1").is_file()
assert not release_server.leftovers()
def test_a_failed_download_exits_1_and_unpacks_nothing(release_server):
(release_server.tarball.parent / f"{release_server.tarball.name}.sha256").unlink()
result = _asset_run(release_server)
assert result.returncode == 1 and "the download failed" in result.stderr
assert not release_server.unpacked() and not release_server.leftovers()
# --- execution policy and Mark of the Web (D16, T6) --------------------------------
OPEN = "MachinePolicy=Undefined,UserPolicy=Undefined,Process=Undefined,CurrentUser=Undefined,LocalMachine=RemoteSigned"
tools/preflight.ps1
+239 -49
View File
@@ -5,6 +5,17 @@
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --set rg=C:\Tools\rg.exe
#
# As the release asset `preflight.ps1` - a copy with no tools/prerequisites.txt next
# to it - the script is the first install step instead: it downloads the stack
# release named in $ReleaseArchiveUrl / $ReleaseChecksumUrl, checks the sha256,
# unpacks it into <script folder>\chemenu (or --into <path>), and runs the copy of
# this script inside the unpacked tree, which does everything above.
#
# pwsh -NoProfile -ExecutionPolicy Bypass -File preflight.ps1 [--into <path>] [--archive <tarball>]
#
# --archive uses a local tarball instead of a download; <tarball>.sha256 has to lie
# next to it. Release builds fill the two URL lines below; a tree copy leaves them empty.
#
# Always start it that way (instructions/preflight.md): the bypass holds for this one
# process only and changes no setting, and it is what lets a script that carries a
# Mark of the Web start at all, so that it can report its own mark.
@@ -13,7 +24,8 @@
# Exit 42: the user has to act. The output says what, why, the command that fixes
# it and what happens next; show it verbatim and wait (AGENTS.md, Tool
# error contract). Never install anything on the user's behalf.
# Exit 1: this script was called wrongly, or the tree next to it is incomplete.
# Exit 1: this script was called wrongly, a download or unpack failed (nothing is
# unpacked then), or the tree next to it is incomplete.
#
# The counterpart of tools/preflight.sh, and the two answer the same questions from the
# same list, tools/prerequisites.txt. What only this one checks: the PowerShell execution
@@ -40,6 +52,10 @@ $Requirements = Join-Path $Dir 'requirements.txt'
$Stamp = Join-Path $Venv '.chemenu-requirements.sha256'
$Self = 'pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1'
# Filled in by the release build, in the copy attached to the release; empty in a tree.
$ReleaseArchiveUrl = ''
$ReleaseChecksumUrl = ''
$PyProbe = "import sys; print(str(sys.version_info[0]) + '.' + str(sys.version_info[1])); print(sys.executable)"
function Write-Line {
@@ -55,6 +71,8 @@ function Write-ErrorLine {
# --- arguments ----------------------------------------------------------------
$Sets = @{}
$Into = ''
$Archive = ''
$index = 0
while ($index -lt $args.Count) {
$arg = [string]$args[$index]
@@ -68,12 +86,30 @@ while ($index -lt $args.Count) {
$given = [string]$args[$index]
} elseif ($arg.StartsWith('--set=')) {
$given = $arg.Substring(6)
} elseif ($arg -eq '--into' -or $arg -eq '--archive') {
if ($index + 1 -ge $args.Count -or -not [string]$args[$index + 1]) {
Write-ErrorLine "preflight: $arg needs a path"
exit 1
}
$index++
if ($arg -eq '--into') {
$Into = [string]$args[$index]
} else {
$Archive = [string]$args[$index]
}
$index++
continue
} elseif ($arg -eq '-h' -or $arg -eq '--help') {
Write-Line "usage: $Self [--set <tool>=<path>]..."
Write-Line ' preflight.ps1 [--into <path>] [--archive <tarball>] [--set <tool>=<path>]...'
Write-Line ''
Write-Line 'Checks the tools this stack needs (tools/prerequisites.txt), records their paths'
Write-Line 'in .wikitool-tools.json and sets up tools/.venv. Exit 0 means ready; exit 42'
Write-Line 'means the user has to act - the output says how.'
Write-Line ''
Write-Line 'The second form is the release asset: it downloads the release (or takes'
Write-Line '--archive), checks its sha256, unpacks it into <script folder>\chemenu or --into,'
Write-Line 'and runs the preflight inside it.'
exit 0
} else {
Write-ErrorLine "preflight: unknown argument: $arg"
@@ -88,8 +124,9 @@ while ($index -lt $args.Count) {
$index++
}
if (-not (Test-Path -LiteralPath $Manifest -PathType Leaf)) {
Write-ErrorLine "preflight: $Manifest is missing - this script has to run from inside an unpacked stack tree."
$AssetMode = -not (Test-Path -LiteralPath $Manifest -PathType Leaf)
if (-not $AssetMode -and ($Into -or $Archive)) {
Write-ErrorLine 'preflight: --into and --archive belong to the release asset; inside a stack tree there is nothing to unpack.'
exit 1
}
@@ -132,6 +169,205 @@ function Exit-WithGuide {
exit 42
}
# Runs a program; its standard output joined by newlines, or $null when it did not
# start or did not exit 0.
function Invoke-Native {
param([string]$Path, [string[]]$Arguments = @())
try {
$output = & $Path @Arguments 2>$null
if ($LASTEXITCODE -ne 0) {
return $null
}
return (@($output | ForEach-Object { "$_".TrimEnd("`r") }) -join "`n")
} catch {
return $null
}
}
# Same, but with the error stream merged in, for the messages the user is shown.
function Invoke-NativeVerbose {
param([string]$Path, [string[]]$Arguments = @())
try {
$output = & $Path @Arguments 2>&1
return [pscustomobject]@{
Code = $LASTEXITCODE
Output = (@($output | ForEach-Object { "$_".TrimEnd("`r") }) -join "`n")
}
} catch {
return [pscustomobject]@{ Code = -1; Output = $_.Exception.Message }
}
}
# --- asset mode: the release asset unpacks the stack, then hands over ------------------
#
# Only when no tools/prerequisites.txt lies next to this script. Nothing here reads
# the tree; the folder limit comes out of the archive itself. The tools the tree copy
# checks (Python, git, rg) are not needed until that copy runs.
function Test-LongPathsEnabled {
if ($env:CHEMENU_PREFLIGHT_LONGPATHS) {
return $env:CHEMENU_PREFLIGHT_LONGPATHS -eq '1'
}
# An unreadable key counts as "off": the limit then protects a machine it did not
# have to.
if (-not $IsWindows) {
return $false
}
try {
$value = Get-ItemPropertyValue -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name LongPathsEnabled
return $value -eq 1
} catch {
return $false
}
}
function Stop-Asset {
param([string]$Message)
Write-ErrorLine "preflight: $Message"
exit 1
}
function Resolve-AssetPath {
param([string]$Path)
if (-not [IO.Path]::IsPathRooted($Path)) {
$Path = Join-Path (Get-Location).ProviderPath $Path
}
return [IO.Path]::GetFullPath($Path).TrimEnd('\', '/')
}
function Invoke-AssetMode {
$target = if ($Into) { Resolve-AssetPath $Into } else { Join-Path $Dir 'chemenu' }
if ($null -ne (Get-Item -LiteralPath $target -Force -ErrorAction SilentlyContinue)) {
Stop-Asset "$target already exists - nothing was unpacked. Choose another folder with --into <path>, or move the existing one away."
}
if (-not $Archive -and (-not $ReleaseArchiveUrl -or -not $ReleaseChecksumUrl)) {
Stop-Asset 'this copy of the script does not come from a release (it has no download address). Download preflight.ps1 from the release page, or pass --archive <tarball>.'
}
if ($Archive) {
$Archive = Resolve-AssetPath $Archive
if (-not (Test-Path -LiteralPath $Archive -PathType Leaf)) {
Stop-Asset "--archive: $Archive is not a file."
}
if (-not (Test-Path -LiteralPath "$Archive.sha256" -PathType Leaf)) {
Stop-Asset "--archive: $Archive.sha256 is missing - the checksum file has to lie next to the tarball."
}
}
$tar = Get-Command tar -CommandType Application -ErrorAction SilentlyContinue | Select-Object -First 1
if (-not $tar) {
Add-Problem 'The tool needed to unpack the stack (tar) could not be found.' `
'the first step downloads the release, checks its checksum and unpacks it' `
'Windows 10 and 11 ship tar.exe in C:\Windows\System32 - if it is missing, repair or update Windows.'
Exit-WithGuide
}
$work = Join-Path ([IO.Path]::GetTempPath()) "chemenu-preflight.$([guid]::NewGuid().ToString('N').Substring(0, 8))"
$unpack = ''
try {
$null = New-Item -ItemType Directory -Path $work
if ($Archive) {
$archivePath = $Archive
$checksumPath = "$Archive.sha256"
} else {
$archivePath = Join-Path $work ($ReleaseArchiveUrl.Split('/')[-1])
$checksumPath = Join-Path $work ($ReleaseChecksumUrl.Split('/')[-1])
Write-Line "Downloading $ReleaseArchiveUrl"
$ProgressPreference = 'SilentlyContinue'
foreach ($pair in @(@($ReleaseArchiveUrl, $archivePath), @($ReleaseChecksumUrl, $checksumPath))) {
try {
Invoke-WebRequest -Uri $pair[0] -OutFile $pair[1]
} catch {
Stop-Asset "the download failed: $($pair[0]) ($($_.Exception.Message)) - check the internet connection, then run this again."
}
}
}
$first = Get-Content -LiteralPath $checksumPath -TotalCount 1 -Encoding utf8
$match = [regex]::Match("$first", '^([0-9A-Fa-f]{64})')
if (-not $match.Success) {
Stop-Asset 'the checksum file holds no sha256 in its first line - nothing was unpacked.'
}
$want = $match.Groups[1].Value.ToLowerInvariant()
$have = (Get-FileHash -LiteralPath $archivePath -Algorithm SHA256).Hash.ToLowerInvariant()
if ($want -ne $have) {
Stop-Asset "the sha256 of the archive does not match its checksum file (expected $want, got $have) - nothing was unpacked. Delete the download and run this again."
}
Write-Line 'sha256 OK'
$listing = Invoke-Native $tar.Source @('-tzf', $archivePath)
if ($null -eq $listing) {
Stop-Asset 'the archive could not be read - nothing was unpacked.'
}
$tops = @(
$listing -split "`n" |
ForEach-Object { ($_ -replace '\\', '/' -replace '^\./', '').Split('/')[0] } |
Where-Object { $_ -and $_ -ne '.' } |
Select-Object -Unique
)
if ($tops.Count -ne 1) {
Stop-Asset 'the archive does not hold exactly one top-level folder - nothing was unpacked.'
}
$top = $tops[0]
$manifestText = Invoke-Native $tar.Source @('-xOzf', $archivePath, "$top/tools/prerequisites.txt")
if ($null -eq $manifestText) {
Stop-Asset "the archive holds no $top/tools/prerequisites.txt - nothing was unpacked."
}
$limit = 0
foreach ($line in ($manifestText -split "`n")) {
if ($line -match '^limit\|install_dir_max\|(\d+)') {
$limit = [int]$Matches[1]
break
}
}
if ($Platform -eq 'windows' -and $limit -gt 0 -and -not (Test-LongPathsEnabled)) {
$length = $target.Length
if ($length -gt $limit) {
Add-Problem "The folder this wiki would be installed in is too long ($length characters, at most $limit): $target" `
"Windows on this computer only allows paths of up to 259 characters, and the wiki's own files need the rest" `
"Run this again with a shorter folder, for example: --into C:\Chemenu`nAlternatively, someone with administrator rights can turn on long paths in Windows."
Exit-WithGuide
}
}
$parent = Split-Path -Parent $target
$null = New-Item -ItemType Directory -Path $parent -Force
$unpack = Join-Path $parent ".chemenu-unpack.$PID"
$null = New-Item -ItemType Directory -Path $unpack
$null = Invoke-NativeVerbose $tar.Source @('-xzf', $archivePath, '-C', $unpack)
if (-not (Test-Path -LiteralPath (Join-Path $unpack $top) -PathType Container)) {
Stop-Asset "unpacking failed - the target was not created."
}
Move-Item -LiteralPath (Join-Path $unpack $top) -Destination $target
} finally {
foreach ($leftover in @($work, $unpack)) {
if ($leftover -and (Test-Path -LiteralPath $leftover)) {
Remove-Item -LiteralPath $leftover -Recurse -Force -ErrorAction SilentlyContinue
}
}
}
$treeScript = Join-Path $target 'tools/preflight.ps1'
if (-not (Test-Path -LiteralPath $treeScript -PathType Leaf)) {
Stop-Asset 'the unpacked stack has no tools/preflight.ps1.'
}
Write-Line "Unpacked into $target."
Write-Line 'If a later step stops, run tools/preflight.ps1 from inside that folder.'
Write-Line ''
$passed = @()
foreach ($name in $Sets.Keys) {
$passed += "--set=$name=$($Sets[$name])"
}
& ([Environment]::ProcessPath) -NoProfile -ExecutionPolicy Bypass -File $treeScript @passed
exit $LASTEXITCODE
}
if ($AssetMode) {
Invoke-AssetMode
}
# --- the manifest ---------------------------------------------------------------
$ManifestLines = @(
@@ -299,35 +535,6 @@ function Find-OnPath {
return $found
}
# Runs a program; its standard output joined by newlines, or $null when it did not
# start or did not exit 0.
function Invoke-Native {
param([string]$Path, [string[]]$Arguments = @())
try {
$output = & $Path @Arguments 2>$null
if ($LASTEXITCODE -ne 0) {
return $null
}
return (@($output | ForEach-Object { "$_".TrimEnd("`r") }) -join "`n")
} catch {
return $null
}
}
# Same, but with the error stream merged in, for the messages the user is shown.
function Invoke-NativeVerbose {
param([string]$Path, [string[]]$Arguments = @())
try {
$output = & $Path @Arguments 2>&1
return [pscustomobject]@{
Code = $LASTEXITCODE
Output = (@($output | ForEach-Object { "$_".TrimEnd("`r") }) -join "`n")
}
} catch {
return [pscustomobject]@{ Code = -1; Output = $_.Exception.Message }
}
}
# The value recorded for <name> in .wikitool-tools.json.
function Get-RecordedPath {
param([string]$Name)
@@ -511,23 +718,6 @@ foreach ($tool in $Tools) {
# --- install folder length (Windows, long paths off) ------------------------------
function Test-LongPathsEnabled {
if ($env:CHEMENU_PREFLIGHT_LONGPATHS) {
return $env:CHEMENU_PREFLIGHT_LONGPATHS -eq '1'
}
# An unreadable key counts as "off": the limit then protects a machine it did not
# have to.
if (-not $IsWindows) {
return $false
}
try {
$value = Get-ItemPropertyValue -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name LongPathsEnabled
return $value -eq 1
} catch {
return $false
}
}
if ($Platform -eq 'windows' -and -not (Test-LongPathsEnabled)) {
$limit = [int](Get-ManifestField 'limit' 'install_dir_max' 3)
$length = $Root.Length
tools/preflight.sh
+199 -27
View File
@@ -5,11 +5,23 @@
# tools/preflight.sh check, record, set up
# tools/preflight.sh --set rg=/opt/rg/rg use a path the user named
#
# As the release asset `preflight.sh` - a copy with no tools/prerequisites.txt next
# to it - the script is the first install step instead: it downloads the stack
# release named in RELEASE_ARCHIVE_URL / RELEASE_CHECKSUM_URL, checks the sha256,
# unpacks it into <script folder>/chemenu (or --into <path>), and runs the copy of
# this script inside the unpacked tree, which does everything above.
#
# preflight.sh [--into <path>] [--archive <tarball>] [--set <tool>=<path>]...
#
# --archive uses a local tarball instead of a download; <tarball>.sha256 has to lie
# next to it. Release builds fill the two URL lines below; a tree copy leaves them empty.
#
# Exit 0: everything is in place and .wikitool-tools.json is complete.
# Exit 42: the user has to act. The output says what, why, the command that fixes
# it and what happens next; show it verbatim and wait (AGENTS.md § Tool
# error contract). Never install anything on the user's behalf.
# Exit 1: this script was called wrongly, or the tree next to it is incomplete.
# Exit 1: this script was called wrongly, a download or unpack failed (nothing is
# unpacked then), or the tree next to it is incomplete.
#
# POSIX sh on purpose: it has to run before Python is known to exist, under dash,
# bash and the Git Bash that Claude Code uses on Windows. The PowerShell
@@ -31,23 +43,41 @@ REQUIREMENTS="$DIR/requirements.txt"
STAMP="$VENV/.chemenu-requirements.sha256"
SELF="tools/preflight.sh"
# Filled in by the release build, in the copy attached to the release; empty in a tree.
RELEASE_ARCHIVE_URL=''
RELEASE_CHECKSUM_URL=''
PYPROBE='import sys; print("%d.%d" % sys.version_info[:2]); print(sys.executable)'
usage() {
cat <<'EOF'
usage: tools/preflight.sh [--set <tool>=<path>]...
preflight.sh [--into <path>] [--archive <tarball>] [--set <tool>=<path>]...
Checks the tools this stack needs (tools/prerequisites.txt), records their paths
in .wikitool-tools.json and sets up tools/.venv. Exit 0 means ready; exit 42
means the user has to act - the output says how.
The second form is the release asset: it downloads the release (or takes
--archive), checks its sha256, unpacks it into <script folder>/chemenu or --into,
and runs the preflight inside it.
EOF
}
# --- arguments ----------------------------------------------------------------
SETS=""
INTO="" ARCHIVE=""
while [ $# -gt 0 ]; do
case "$1" in
--into|--archive)
if [ $# -lt 2 ] || [ -z "$2" ]; then
echo "preflight: $1 needs a path" >&2
exit 1
fi
if [ "$1" = --into ]; then INTO=$2; else ARCHIVE=$2; fi
shift
;;
--set)
if [ $# -lt 2 ]; then
echo "preflight: --set needs <tool>=<path>" >&2
@@ -65,8 +95,11 @@ ${1#--set=}" ;;
shift
done
ASSET=0
if [ ! -f "$MANIFEST" ]; then
echo "preflight: $MANIFEST is missing - this script has to run from inside an unpacked stack tree." >&2
ASSET=1
elif [ -n "$INTO$ARCHIVE" ]; then
echo "preflight: --into and --archive belong to the release asset; inside a stack tree there is nothing to unpack." >&2
exit 1
fi
@@ -124,6 +157,170 @@ stop() {
exit 42
}
longpaths_enabled() {
if [ -n "${CHEMENU_PREFLIGHT_LONGPATHS:-}" ]; then
[ "$CHEMENU_PREFLIGHT_LONGPATHS" = 1 ]
return
fi
# MSYS would rewrite the `/v` into a path without the exclusion. An unreadable
# key counts as "off": the limit then protects a machine it did not have to.
answer=$(MSYS2_ARG_CONV_EXCL='*' reg query 'HKLM\SYSTEM\CurrentControlSet\Control\FileSystem' \
/v LongPathsEnabled 2>/dev/null) || return 1
case "$answer" in *0x1*) return 0 ;; esac
return 1
}
# Length in UTF-16 code units, which is what MAX_PATH counts.
utf16_length() {
if command -v iconv >/dev/null 2>&1; then
bytes=$(printf '%s' "$1" | iconv -f UTF-8 -t UTF-16LE 2>/dev/null | wc -c | tr -d ' ')
if [ "${bytes:-0}" -gt 0 ]; then
echo $((bytes / 2))
return
fi
fi
echo ${#1}
}
# --- asset mode: the release asset unpacks the stack, then hands over ------------------
#
# Only when no tools/prerequisites.txt lies next to this script. Nothing here reads
# the tree; the folder limit comes out of the archive itself. The tools the tree copy
# checks (Python, git, rg) are not needed until that copy runs.
WORK=""
cleanup() {
[ -z "$WORK" ] || rm -rf "$WORK"
[ -z "${UNPACK:-}" ] || rm -rf "$UNPACK"
}
asset_fail() { # <message>
echo "preflight: $1" >&2
exit 1
}
sha256_of() { # <file>
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | cut -d' ' -f1
else
shasum -a 256 "$1" | cut -d' ' -f1
fi
}
asset_mode() {
# GNU tar reads `C:` as a host name; Git Bash has to hand it `/c/...`.
if [ "$PLATFORM" = windows ] && command -v cygpath >/dev/null 2>&1; then
[ -z "$ARCHIVE" ] || ARCHIVE=$(cygpath -u "$ARCHIVE")
[ -z "$INTO" ] || INTO=$(cygpath -u "$INTO")
fi
if [ -n "$INTO" ]; then
case "$INTO" in /*) TARGET=$INTO ;; *) TARGET="$(pwd)/$INTO" ;; esac
else
TARGET="$DIR/chemenu"
fi
TARGET=${TARGET%/}
if [ -e "$TARGET" ] || [ -L "$TARGET" ]; then
asset_fail "$(native_path "$TARGET") already exists - nothing was unpacked. Choose another folder with --into <path>, or move the existing one away."
fi
if [ -z "$ARCHIVE" ] && { [ -z "$RELEASE_ARCHIVE_URL" ] || [ -z "$RELEASE_CHECKSUM_URL" ]; }; then
asset_fail "this copy of the script does not come from a release (it has no download address). Download preflight.sh from the release page, or pass --archive <tarball>."
fi
if [ -n "$ARCHIVE" ]; then
[ -f "$ARCHIVE" ] || asset_fail "--archive: $ARCHIVE is not a file."
[ -f "$ARCHIVE.sha256" ] || asset_fail "--archive: $ARCHIVE.sha256 is missing - the checksum file has to lie next to the tarball."
fi
# What has to be here for the download, tested before anything is fetched.
missing=""
if [ -z "$ARCHIVE" ] && ! command -v curl >/dev/null 2>&1; then missing="$missing curl"; fi
command -v tar >/dev/null 2>&1 || missing="$missing tar"
command -v sha256sum >/dev/null 2>&1 || command -v shasum >/dev/null 2>&1 || missing="$missing sha256sum"
if [ -n "$missing" ]; then
case "$PLATFORM" in
macos) hint="brew install$missing" ;;
windows) hint="Git for Windows (https://gitforwindows.org) provides all of them in Git Bash." ;;
*) hint="sudo apt install$(printf '%s' "$missing" | sed 's/ sha256sum/ coreutils/')" ;;
esac
problem "Tools needed to fetch and unpack the stack could not be found:$missing" \
"the first step downloads the release, checks its checksum and unpacks it" \
"$hint"
stop
fi
trap cleanup EXIT
WORK=$(mktemp -d "${TMPDIR:-/tmp}/chemenu-preflight.XXXXXX") || asset_fail "could not create a temporary folder."
if [ -n "$ARCHIVE" ]; then
archive=$ARCHIVE
checksum=$ARCHIVE.sha256
else
archive="$WORK/${RELEASE_ARCHIVE_URL##*/}"
checksum="$WORK/${RELEASE_CHECKSUM_URL##*/}"
echo "Downloading $RELEASE_ARCHIVE_URL"
curl -fsSL -o "$archive" "$RELEASE_ARCHIVE_URL" || asset_fail "the download failed: $RELEASE_ARCHIVE_URL (check the internet connection, then run this again)."
curl -fsSL -o "$checksum" "$RELEASE_CHECKSUM_URL" || asset_fail "the download failed: $RELEASE_CHECKSUM_URL (check the internet connection, then run this again)."
fi
want=$(sed -n '1s/^\([0-9A-Fa-f]\{64\}\).*/\1/p' "$checksum" | tr 'A-F' 'a-f')
[ -n "$want" ] || asset_fail "the checksum file holds no sha256 in its first line - nothing was unpacked."
have=$(sha256_of "$archive" | tr 'A-F' 'a-f')
if [ "$want" != "$have" ]; then
asset_fail "the sha256 of the archive does not match its checksum file (expected $want, got $have) - nothing was unpacked. Delete the download and run this again."
fi
echo "sha256 OK"
listing=$(tar -tzf "$archive" 2>/dev/null) || asset_fail "the archive could not be read - nothing was unpacked."
tops=$(printf '%s\n' "$listing" | sed 's|^\./||; s|/.*||; /^\.\{0,1\}$/d')
top=$(printf '%s\n' "$tops" | head -n 1)
other=$(printf '%s\n' "$tops" | grep -v -x -F -- "$top" | head -n 1)
if [ -z "$top" ] || [ -n "$other" ]; then
asset_fail "the archive does not hold exactly one top-level folder - nothing was unpacked."
fi
manifest=$(tar -xOzf "$archive" "$top/tools/prerequisites.txt" 2>/dev/null) \
|| asset_fail "the archive holds no $top/tools/prerequisites.txt - nothing was unpacked."
limit=$(printf '%s\n' "$manifest" | sed -n 's/^limit|install_dir_max|\([0-9][0-9]*\).*/\1/p' | head -n 1)
if [ "$PLATFORM" = windows ] && [ -n "$limit" ] && ! longpaths_enabled; then
folder=$(native_path "$TARGET")
length=$(utf16_length "$folder")
if [ "$length" -gt "$limit" ]; then
problem "The folder this wiki would be installed in is too long ($length characters, at most $limit): $folder" \
"Windows on this computer only allows paths of up to 259 characters, and the wiki's own files need the rest" \
"Run this again with a shorter folder, for example: --into C:\\\\Chemenu
Alternatively, someone with administrator rights can turn on long paths in Windows."
stop
fi
fi
parent=$(dirname -- "$TARGET")
mkdir -p "$parent" || asset_fail "could not create $parent."
UNPACK="$parent/.chemenu-unpack.$$"
mkdir "$UNPACK" || asset_fail "could not create a temporary folder next to the target."
tar -xzf "$archive" -C "$UNPACK" || asset_fail "unpacking failed - the target was not created."
[ -d "$UNPACK/$top" ] || asset_fail "unpacking produced no $top folder - the target was not created."
mv "$UNPACK/$top" "$TARGET" || asset_fail "could not move the unpacked stack to $(native_path "$TARGET")."
rm -rf "$UNPACK"
UNPACK=""
[ -f "$TARGET/tools/preflight.sh" ] || asset_fail "the unpacked stack has no tools/preflight.sh."
echo "Unpacked into $(native_path "$TARGET")."
echo "If a later step stops, run tools/preflight.sh from inside that folder."
echo ""
set --
while IFS= read -r entry; do
[ -n "$entry" ] || continue
set -- "$@" --set "$entry"
done <<EOT
$SETS
EOT
trap - EXIT
cleanup
exec /bin/sh "$TARGET/tools/preflight.sh" "$@"
}
[ "$ASSET" -eq 0 ] || asset_mode
# --- the manifest ---------------------------------------------------------------
manifest_field() { # <kind> <name> <field number, 1-based>
@@ -398,31 +595,6 @@ done
# --- install folder length (Windows, long paths off) ------------------------------
longpaths_enabled() {
if [ -n "${CHEMENU_PREFLIGHT_LONGPATHS:-}" ]; then
[ "$CHEMENU_PREFLIGHT_LONGPATHS" = 1 ]
return
fi
# MSYS would rewrite the `/v` into a path without the exclusion. An unreadable
# key counts as "off": the limit then protects a machine it did not have to.
answer=$(MSYS2_ARG_CONV_EXCL='*' reg query 'HKLM\SYSTEM\CurrentControlSet\Control\FileSystem' \
/v LongPathsEnabled 2>/dev/null) || return 1
case "$answer" in *0x1*) return 0 ;; esac
return 1
}
# Length in UTF-16 code units, which is what MAX_PATH counts.
utf16_length() {
if command -v iconv >/dev/null 2>&1; then
bytes=$(printf '%s' "$1" | iconv -f UTF-8 -t UTF-16LE 2>/dev/null | wc -c | tr -d ' ')
if [ "${bytes:-0}" -gt 0 ]; then
echo $((bytes / 2))
return
fi
fi
echo ${#1}
}
if [ "$PLATFORM" = windows ] && ! longpaths_enabled; then
limit=$(manifest_field limit install_dir_max 3)
folder=$(native_path "$ROOT")