torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8
CI / verify (push) Successful in 8m8s
Details
CI / pwsh (push) Successful in 2m40s
Details
Release / release (push) Successful in 35s
Details
Release / wiki-mirror (push) Skipped
Details

Browse Source 
Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-11 20:50:23 +02:00
1 parent aed846740b
commit 5994b2a88c
43 files changed
+2189 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGES.md
+6 -1
View File
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
---
## 10.0.0-beta.7 - 2026-10-11 - raw fetch --scope ist die Entscheidung des Nutzers: raw/CONTRACT.md und wiki-ingest sagen, wer es anstößt und warum es kein Folgen von Links ist
## 10.0.0-beta.8 - 2026-10-11 - raw fetch hinter einem Login: raw login speichert die Session des Nutzers als Secret, raw fetch holt damit; gerenderte Seiten über die optionale Browser-Komponente, externe Abrufer pro Host
**Author:** Torben Nehmer
@@ -31,6 +31,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
- Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen
- Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind
- raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/
- raw fetch hinter einem Login: raw login speichert die Session des Nutzers als Secret, raw fetch holt damit; gerenderte Seiten über die optionale Browser-Komponente, externe Abrufer pro Host
**Low impact**
- Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
@@ -52,6 +53,10 @@ Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, le
Bietet der Site-Handler einer Seite Unterseiten an (Baustein `subpages` in der Handler-Datei; der Modulindex von learn.microsoft.com bietet seine Cmdlet-Seiten an), nennt `raw fetch` das als eine `CHECK`-Zeile, und `raw fetch <url> --scope` holt Index und Unterseiten als einen Ordner `incoming/<name>/` mit einem Paar je Seite und dem Manifest `_scope.json`; `raw accept incoming/<name>` nimmt ihn unverändert an. Geholt wird nur, was der Handler benennt, nie von einem anderen Host. Vorher prüft der Befehl `robots.txt` und die Obergrenze `sites.max_pages` (Standard 200); jeder Fehler bricht ab, bevor etwas in `incoming/` liegt. Zwischen den Unterseiten wartet er `sites.pause_seconds` (Standard 1). Ob `--scope` läuft, entscheidet der Nutzer: `wiki-ingest` fragt ihn, wenn die `CHECK`-Zeile es anbietet.
### `raw fetch` hinter einem Login (#169)
Der Abschnitt `fetch` der Einstellungen ordnet einem Host einen Abrufweg zu. Bei `session` holt `raw fetch` mit der Session, die der Nutzer einmal mit `raw login <host>` im Browserfenster hinterlassen hat. Sie liegt als Secret außerhalb des Baums, ein Passwort wird nie gespeichert. Der Site-Handler nennt die Login-Seite und das Zeichen einer eingeloggten Seite. Fehlt die Session oder ist sie abgelaufen, endet der Abruf mit Exit 42 ohne Teaser. `render: true` holt über einen Browser (`rendered: true`). Ein anderer Weg nennt einen externen Abrufer aus `.wikitool-local.json`. Playwright ist optional (`preflight --with browser`), und `doctor` meldet es in der Zeile `browser`.
### Altlasten der Konfiguration bereinigt (#208)
Die vier Einzeldateien `.wikitool-remotes.json`, `.wikitool-telemetry.json`, `.wikitool-upload.json` und `.wikitool-tasks.json` gibt es nicht mehr; ihre Felder sind die Abschnitte `remotes`, `telemetry`, `upload`, `tasks`, `superproductivity` und `caldav` der drei Stufen. Zugangsdaten sind dort nur noch Verweise. Der Code liest die alten Dateien nicht mehr: ein Kommando, das die Einstellung braucht, endet mit Exit 42 und nennt `tools/wikitool config migrate`, `doctor` meldet FAIL, der Telemetrie-Hook bleibt still. `config migrate` läuft einmal je Checkout, ist wiederholbar und lässt bei jedem Abbruch die alten Dateien unverändert. `WIKITOOL_TASKS_CONFIG` zeigt auf ein Profil im Format von `.wikitool-local.json`. Nur `.wikitool-tools.json` bleibt eine eigene Datei.
INSTALL.md
+11
View File
@@ -33,6 +33,17 @@ Optional: the **1Password CLI (`op`)**, if you want secrets and private values i
come from 1Password ([use-1password-for-settings.md](docs/how-to/use-1password-for-settings.md)).
The preflight records it when it finds it and never asks for it.
Optional: the **browser component**, if you want the wiki to fetch pages of a website you are
subscribed to with your own login, or pages that only appear in a browser
([fetch-pages-behind-a-login.md](docs/how-to/fetch-pages-behind-a-login.md)). It is a Python
package plus a Chromium of its own, a few hundred megabytes, installed into the instance's
`tools/.venv` and nowhere else. You install it yourself, once the instance runs:
`tools/preflight.sh --with browser` (PowerShell 7:
`pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`). Every later
preflight run keeps it up to date. On Linux, the browser may need system libraries the preflight
cannot install; Playwright's message then names them, for you or whoever administers the machine.
Native Windows is not yet verified for it.
You also need:
- **An agent harness**: Claude Code, GitHub Copilot (in VS Code or as a CLI), Codex CLI or
VERSION
+1 -1
View File
@@ -1 +1 @@
10.0.0-beta.7
10.0.0-beta.8
docs/how-to/README.md
+2
View File
@@ -11,6 +11,8 @@ Working with the wiki:
- [Ingest a source](ingest-a-source.md) - a file, a folder, a web page, or a document submitted
through the MCP server
- [Write a site handler](write-a-site-handler.md) - teach `raw fetch` how a website serves its pages
- [Fetch pages behind a login](fetch-pages-behind-a-login.md) - with your own session, through a
browser, or through a fetch program of your own
- [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
- [Export guidelines to captured repositories](export-guidelines.md)
- [Query the wiki](query-the-wiki.md) - asking, and searching yourself
docs/how-to/fetch-pages-behind-a-login.md
+128
View File
@@ -0,0 +1,128 @@
# Fetch pages behind a login
Let `tools/wikitool raw fetch` fetch the articles of a website you are subscribed to, with your own
login: you sign in once in a browser window, and the wiki fetches that website's pages with the
session your sign-in left until it expires. No password is stored, and the agent never sees one.
The same page covers websites whose pages only appear in a browser, and handing a website to a
fetch program of your own.
Whether a website's terms allow you to fetch your subscription this way is yours to check; the
stack does not check it and offers no advice on it.
Without any of this, a page behind a login can still be saved from your browser and handed to
`raw fetch --html` ([Ingest a source](ingest-a-source.md)) - one page at a time, by hand.
<!-- wikitool:toc -->
## Contents
- [1. Install the browser component](#1-install-the-browser-component)
- [2. Describe the website's login](#2-describe-the-websites-login)
- [3. Route the website through your session](#3-route-the-website-through-your-session)
- [4. Sign in](#4-sign-in)
- [5. Ingest as usual](#5-ingest-as-usual)
- [Pages that only appear in a browser](#pages-that-only-appear-in-a-browser)
- [A fetch program of your own](#a-fetch-program-of-your-own)
- [Where the session is kept](#where-the-session-is-kept)
<!-- /wikitool:toc -->
## 1. Install the browser component
Signing in needs a real browser window, so it needs the optional browser component: a Python
package and a Chromium of its own, installed into the instance's `tools/.venv`. You install it -
the agent never does:
```bash
tools/preflight.sh --with browser
```
In PowerShell 7: `pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`.
Later preflight runs keep it up to date. `tools/wikitool doctor` reports it on its `browser` line.
## 2. Describe the website's login
The website needs a site handler that says where you sign in and how a page shows that it arrived
signed in. Add a `session` block to the website's handler in `sites/`, or write a handler just for
it ([Write a site handler](write-a-site-handler.md)):
```json
{
"host": "news.example.com",
"summary": "A subscription news site: articles need the subscriber's session.",
"session": {
"login": "https://news.example.com/login",
"signed_in": {"tag": "a", "class": "logout"},
"signed_out": {"class": "paywall-teaser"}
}
}
```
`signed_in` is something only a signed-in page carries - a logout link, an account menu.
`signed_out` is something only a page that was not carries - the teaser box that asks you to
subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element
that tells them apart. [sites/CONTRACT.md](../../sites/CONTRACT.md) has the exact format.
## 3. Route the website through your session
The handler alone changes nothing. Tell the agent:
> Route news.example.com through my session.
It runs `tools/wikitool config set fetch.routes '{"news.example.com": "session"}'`, keeping any
host already listed. The setting is committed with the instance, so every clone fetches the
website the same way - each with its own user's session.
## 4. Sign in
Run this yourself - it opens a browser window and waits for you:
```bash
tools/wikitool raw login news.example.com
```
Sign in as you always do, captcha and second factor included, then close the window. The session
is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.
## 5. Ingest as usual
Give the agent the article's URL as for any web page ([Ingest a source](ingest-a-source.md)).
The capture's header names the website whose session was used (`session: news.example.com`),
never the session itself.
When your session has expired, or the page came back as a teaser, the agent stops: `raw fetch`
writes nothing and asks you to run `raw login` again. Nothing is kept from a teaser in silence.
## Pages that only appear in a browser
Some websites send an empty frame and draw the text with JavaScript; `raw fetch` then captures
almost nothing. Add `"render": true` to the website's handler: its pages are then fetched through
the browser component, with your session where the website has one, and the header says
`rendered: true`. The `.html` is then the page as the browser drew it, not what the server sent -
which is why it happens only where a handler says so.
## A fetch program of your own
A website can instead go to a program you run - a fetch service on another machine, say. Add it to
`.wikitool-local.json` by hand, since it names a program to run and `config set` does not write
such a field:
```json
{
"schema": 1,
"fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
}
```
and route the host to it: `tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'`.
The program gets the URL as its last argument and prints the page's HTML; anything else - an exit
other than 0, no output - stops `raw fetch` with the program's error output, and the built-in
fetch is not tried instead. The `fetch` section is in
[docs/reference/configuration.md](../reference/configuration.md#fetch-routes).
## Where the session is kept
In the secrets file in your own configuration directory - `~/.config/chemenu/<instance_id>/secrets.json`,
`%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows - under `session_<host>`, readable by
you only, outside every working tree. It never reaches a commit or a `wikitool` output. It does
not keep an agent with a shell in your account from reading it, any more than any other secret
there: [configuration-tiers.md](../explanation/configuration-tiers.md) says what keeping secrets
out of the tree protects against and what it does not.
docs/how-to/ingest-a-source.md
+4 -1
View File
@@ -47,7 +47,10 @@ pick a directory:
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
text to derive - the tool warns, and there is no OCR.
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
agent derives the same text from that file with `raw fetch --html`.
agent derives the same text from that file with `raw fetch --html`. For a website you read often,
sign in once instead and let the wiki fetch it with your session:
[Fetch pages behind a login](fetch-pages-behind-a-login.md). When that session has expired the
agent stops and asks you to run `tools/wikitool raw login <host>` - it never signs in itself.
Where a file ends up in `raw/` is computed by `tools/wikitool raw accept`: a `YYYY/MM` shard for the
day it was accepted. So is whether several files of one source get bundled. You never choose either.
docs/reference/commands/README.md
+2 -1
View File
@@ -79,7 +79,8 @@ Full records: [provenance.md](provenance.md)
Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
- [`raw login`](raw-material-and-uploads.md#raw-login) - Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
- [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
- [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
- [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
docs/reference/commands/instance-health.md
+1
View File
@@ -44,6 +44,7 @@ tools/wikitool doctor --json
- Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
- Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
- Browser component (`browser`): whether the optional Playwright component and its browser are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is missing and the instance needs it: a host `fetch.routes` sends through the session (`raw login`), or a site handler that declares `render`. The user installs it with `tools/preflight.sh --with browser`.
- Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
- Read-only and exempt from the Iteration Budget Gate.
docs/reference/commands/raw-material-and-uploads.md
+65 -3
View File
@@ -6,6 +6,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
## Contents
- [`raw fetch`](#raw-fetch)
- [`raw login`](#raw-login)
- [`raw derive`](#raw-derive)
- [`raw capture`](#raw-capture)
- [`raw status`](#raw-status)
@@ -20,7 +21,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
<!-- wikitool:reference-commands -->
## `raw fetch`
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
**Synopsis**
@@ -64,7 +65,9 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
- `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
- Success prints the `raw accept` line for the written files and the `source_url` for the source page.
@@ -83,17 +86,76 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
- `1` - A target file already exists in `incoming/`
- `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
- `1` - The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
- `1` - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
- `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
- `42` - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
- `42` - The site handler declares `render` and the optional browser component is not installed
**See also**
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
- `sites/CONTRACT.md` - the instance's own handler files
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
- `wikitool raw login` - the user signs in once for a host routed through the session
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
## `raw login`
Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
**Synopsis**
- `wikitool raw login <host>`
**Options**
```text
Arguments:
host The website's host, such as www.example.org [required]
Options:
-h, --help Show this message and exit.
```
**Examples**
```bash
tools/wikitool raw login www.heise.de
```
**Behaviour**
- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
- Prints no part of the session. Signing in again replaces the stored session.
**Properties**
- Changes files: yes
- Idempotent: yes
- Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
- Counts toward the session call budget: yes
- Can reach the network: yes
**Exit codes**
- `0` - success
- `1` - The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
- `1` - The browser could not be started, or the window closed before any state could be read
- `42` - The optional browser component is not installed
**See also**
- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
- `sites/CONTRACT.md` - a handler's `session` block
- `docs/reference/configuration.md` - the `fetch` section
## `raw derive`
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
docs/reference/configuration.md
+59 -3
View File
@@ -29,6 +29,7 @@ something you run.
- [Profiles](#profiles)
- [MCP submissions](#mcp-submissions)
- [Site handlers](#site-handlers)
- [Fetch routes](#fetch-routes)
- [Files of earlier versions](#files-of-earlier-versions)
<!-- /wikitool:toc -->
@@ -90,7 +91,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
[upload](#mcp-submissions), [sites](#site-handlers) and [fetch](#fetch-routes). `config set` names the sections it knows, and `doctor` reports a
section it does not read as a `WARN`.
**One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -117,8 +118,9 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
**Never in the committed file:** a field that names a program to run or affects a gate - not even
as a reference. `config set` does not write such a field either (exit 42); the user edits
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` in the
tables below are these: the publish allowlist and the upload limits.
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
`names a command` in the tables below are these: the publish allowlist, the upload limits and the
external fetchers.
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
`settings-secrets` - and for the references only whether they resolve. `docs verify` holds a
@@ -477,6 +479,60 @@ All three fields are instance-tier: what a website needs cleaned is the same in
of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
with exit 1 before it fetches anything.
## Fetch routes
`tools/wikitool raw fetch` fetches every page the same way unless the `fetch` section says
otherwise for its host: anonymously, with no cookie and no browser. `routes` maps a host to another
way, and is instance-tier, so every clone fetches the host the same way:
- `"session"` fetches the host's pages with the session the user stored with
`tools/wikitool raw login <host>`, through the same built-in fetch. The host needs a site handler
with a `session` block - its login page and how a page shows it arrived signed in
([sites/CONTRACT.md](../../sites/CONTRACT.md)). No stored session, or a page without that sign,
stops the fetch with exit 42 for the user to sign in again. The session is a secret: it is
stored in the secrets file [above](#tiered-settings) under `session_<host>` (every run of other
characters as one `_`, so `www.example.org` is `session_www_example_org`), or comes from
`WIKITOOL_SECRET_SESSION_<HOST>`, and no settings field refers to it. It is never written to
1Password.
- Any other value names a fetcher of `fetchers`: a program of this checkout, run without a shell
with the URL appended. `fetchers` names a program to run, so it lives only in
`.wikitool-local.json`, written by hand - `config set` refuses it with exit 42, and a fetcher in
`wikitool.json` is an error. A route to a fetcher this checkout does not define stops a fetch of
that host with exit 1, so a clone without the program fails loudly instead of fetching
anonymously.
The whole procedure, with the browser component it needs for signing in:
[fetch-pages-behind-a-login.md](../how-to/fetch-pages-behind-a-login.md).
<!-- wikitool:reference-config-fetch -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `routes` | instance | object | no | `{}` | Host (exact, lower case) to route: `"session"` fetches with the user's session - the host needs a site handler that names its login page - and any other value names a fetcher of `fetchers`. A host not listed is fetched as before. |
| `fetchers` | checkout, names a command | object | no | `{}` | Fetcher name to command: an array of the program and its arguments, run without a shell with the URL appended. Exit 0 and the page's HTML on stdout is a fetch; anything else stops `raw fetch` with exit 1 and the program's stderr. |
- **Absent:** Every page is fetched by the built-in anonymous fetch, as before.
- **Malformed:** An error when `raw fetch` or `raw login` reads it: the command stops with exit 1 before fetching anything. A route to a fetcher this checkout does not define stops a fetch of that host the same way.
Example:
```json
{
"fetch": {
"routes": {
"www.heise.de": "session",
"intranet.example.org": "homelab"
},
"fetchers": {
"homelab": [
"/usr/local/bin/fetch-page",
"--html"
]
}
}
}
```
<!-- /wikitool:reference-config-fetch -->
## Files of earlier versions
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
docs/reference/exit-codes.md
+3
View File
@@ -36,6 +36,9 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
- [`task list`](commands/pages.md#task-list): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`task close`](commands/pages.md#task-close): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`review`](commands/finding-and-checking.md#review): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
- [`raw fetch`](commands/raw-material-and-uploads.md#raw-fetch): The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
- [`raw fetch`](commands/raw-material-and-uploads.md#raw-fetch): The site handler declares `render` and the optional browser component is not installed
- [`raw login`](commands/raw-material-and-uploads.md#raw-login): The optional browser component is not installed
- [`upload accept`](commands/raw-material-and-uploads.md#upload-accept): Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
- [`sync`](commands/git.md#sync): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file, generated files aside; the output lists the upstream commits, the overlapping files and their diff
- [`publish`](commands/git.md#publish): The remotes settings file of an earlier version is still there
instructions/preflight.md
+11
View File
@@ -24,6 +24,11 @@ Windows. The two answer the same questions from the same list and write the same
`ABSENT` and is never a reason to stop, nor to install anything;
- creates `tools/.venv` from the recorded Python and installs `tools/requirements.txt` into it.
One more thing it does only when the user asks for it: `--with browser` installs the optional
browser component - `tools/requirements-browser.txt` and the Chromium it drives - into
`tools/.venv`. Only `raw login` and a page a site handler fetches through a browser need it. Once
installed, every later run keeps it up to date; nothing else ever installs it.
<!-- wikitool:toc -->
## Contents
@@ -148,6 +153,12 @@ It is safe to run at any time: a second run on a ready checkout changes nothing
- **The venv or its libraries could not be installed.** The output carries the last lines of
what Python or pip said. A network, proxy or security-product cause is for the user - or
whoever administers their machine - to resolve; do not retry with other flags.
- **A `wikitool` command names `--with browser`.** `raw login` or a rendered `raw fetch` needs the
optional browser component, and exits 42 saying so. Never pass `--with browser` yourself: it
downloads a browser onto the user's machine, which is the user's decision like any other
install. Show the output and wait; once the user has run it, a plain run of the preflight - yours
or theirs - keeps the component. A failure while installing it stops the preflight with exit 42
like any other, but leaves `.wikitool-tools.json` complete: every other command still runs.
- **`.wikitool-tools.json` looks wrong.** Never edit it. Run the preflight again, with `--set` for
a path the user names; `doctor` reports whether the result holds.
instructions/wiki-ingest/SKILL.md
+9 -1
View File
@@ -124,6 +124,12 @@ validator complains - and the ticked list is the only record that they happened.
`robots.txt` forbids a subpage goes to the user as it is; never fetch the subpages one by one
instead.
**`raw fetch` exited 42 and names `raw login <host>` or `--with browser`?** The host is fetched
with the user's own session, and there is none or it has expired - or the page needs the
optional browser component. Show the output to the user verbatim and wait; nothing was
written. **Never run `raw login` yourself** - it opens a browser window and waits for a human -
and never install the component. Once the user says it is done, fetch again.
**Check that the text is the whole article.** A paywall, a login wall or a page that only
renders in a browser yields a teaser, often long enough to look like an article: the text
breaks off at "continue reading with...", a subscription offer or a login prompt. Stop there
@@ -136,7 +142,9 @@ validator complains - and the ticked list is the only record that they happened.
which derives the `.md` from that file without touching the network. It never overwrites, so
a teaser's `.md` still in `incoming/` under the same name makes it refuse: remove the teaser's
files first - they were never accepted, so nothing refers to them.
files first - they were never accepted, so nothing refers to them. For a website the user
reads often, mention the lasting way as well: fetching with their own session
(`docs/how-to/fetch-pages-behind-a-login.md`) - set up by the user, not by you.
**Check the size first, on both axes.** *Volume* - how many raw files this ingest covers -
and *breadth* - how many entities and concepts this one source would produce or update.
raw/CONTRACT.md
+45 -6
View File
@@ -217,7 +217,10 @@ derived_from: post.html
---
```
(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler.)
(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler;
`session:` and `rendered: true` follow `site:` on a page fetched with the user's session or drawn
by a browser, and a page from an external fetcher names it in `fetched_by:` and carries no
`http_status:` - all three below.)
The fields are capture metadata, not page frontmatter - `raw/` has no types, and nothing in the
stack reads the block back. There is deliberately no `author:`: HTML does not reliably say who
@@ -227,10 +230,45 @@ is stored exactly as received with no header and no derivation, because a header
added without changing the bytes; `url` and the retrieval time are in the command's output and
reach the source page as `source_url`.
**A paywall, a login or a page that only renders in a browser is not the tool's to get past.**
`raw fetch` sends no cookies, runs no JavaScript and holds no credentials - credentials have no
place in a working tree (below), and a login adapter per site is not a knowledge compiler's
maintenance to carry. Instead, the human saves the page from their own logged-in browser into
**A paywall, a login or a page that only renders in a browser is never got past with a
password.** `raw fetch` holds no credentials - they have no place in a working tree (below) - and
a login adapter per website is not a knowledge compiler's maintenance to carry. What it can use is
the user's own session, the way the user's browser does:
- **The instance routes the host through the session** (`fetch.routes` in the settings,
[docs/reference/configuration.md](../docs/reference/configuration.md)), and the host's site
handler names its login page and how a page shows that it arrived signed in (its `session`,
[sites/CONTRACT.md](../sites/CONTRACT.md)).
- **The user signs in once, with `tools/wikitool raw login <host>`**, in a visible browser window -
captcha and second factor included. Only the session the window leaves, its cookies and storage,
is kept, as a secret in the user's own configuration directory outside every working tree; no
password is stored, and nothing signs in again on its own.
- **`raw fetch` then fetches the host's pages with that session**, still through the built-in
fetch and with the bytes as received; the header names the host the session was for
(`session:`), never the session itself. When no session is stored, or a page arrives without
the sign of being signed in - the session expired, the website ended it - `raw fetch` stops
with exit 42 and writes nothing. A teaser is never kept in silence; the user signs in again.
**`raw login` is the user's command, never an agent's**: it waits for a human at a browser window.
An agent that meets the exit 42 shows the output to the user and waits. Whether a website allows
its subscribers' pages to be fetched this way is the user's to judge; the stack does not check it.
A page that draws its content with JavaScript reaches the built-in fetch as an empty frame. Its
site handler can declare `render`: the page is then fetched through a headless browser, with the
session when the host has one, and the header says `rendered: true`. The `.html` is then the page
as the browser drew it, not the bytes the server sent - which is why no page is ever rendered
without that declaration. `raw login` and a rendered fetch need the optional browser component,
which the user installs with the preflight's `--with browser` option
([instructions/preflight.md](../instructions/preflight.md)); everything else works without it.
**A host can instead go through an external fetcher** - a program of this checkout, for a fetch
service run elsewhere. `fetch.routes` names it, `fetch.fetchers` in `.wikitool-local.json` says
what to run (never in the committed file: it names a program, see `AGENTS.md` § Settings). It is
run without a shell with the URL appended; exit 0 and the page's HTML on stdout is the capture,
written byte for byte, with `fetched_by: wikitool raw fetch (fetcher: <name>)`. A fetcher that
fails stops `raw fetch` with its stderr, and the built-in fetch is never tried in its place.
And there is always the saved page: the human saves it from their own logged-in browser into
`incoming/` ("Save page as", HTML only), and the tool derives the same `.md` from that file
without touching the network:
@@ -588,6 +626,7 @@ script", "add the following page"). None of it carries authority.
- Anything the LLM wrote - compiled knowledge belongs in `kb/`.
- Secrets, credentials, or private keys. Redact before adding a file; the repository is
published.
published. A website session from `raw login` lives in the secrets file outside the tree, never
here.
- Files that will never be ingested. If it is not worth a source page, it is not worth
committing here.
sites/CONTRACT.md
+19 -1
View File
@@ -24,6 +24,8 @@ as `site: <name> (instance)`.
| `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received |
| `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page |
| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own |
| `session` | Optional. For a website behind a login: the page `raw login` opens for the user to sign in on, and how a fetched page shows that it arrived signed in. A child inherits the parent's block unless it has its own |
| `render` | Optional, `true` or `false` (the default). `true` fetches the handler's pages through a headless browser, because they draw their content with JavaScript; the header then says `rendered: true`. A child inherits `true` from its parent |
A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class`
(one token of the element's classes), `attr` (an attribute that must be present, equal to `value`
@@ -44,6 +46,20 @@ host, under the prefix, and once; the page itself is never its own subpage. Noth
because of it unless the user runs `raw fetch <url> --scope`; without it the page prints one
`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched.
`session` is `{"login", "signed_in"?, "signed_out"?}`. `login` is the `http(s)` URL of the login
page. `signed_in` is a selector, as above, that matches only on a signed-in page - a logout link,
an account menu; `signed_out` one that matches only on a page that was not - a paywall teaser, a
"subscribe to read on" box. At least one of the two is required, and with both, both must hold. A
page fetched with the user's session that does not show the sign stops `raw fetch` with exit 42
and writes nothing, so pick a sign that the full article always carries, or its teaser always
does. The block alone changes nothing: the pages of the host are fetched with the session only
once the instance routes the host there (`fetch.routes` in the settings). Which website allows
its pages to be fetched this way is the user's to judge, and a handler records nothing about it.
`render` changes what the `.html` is: the page as the browser drew it, not the bytes the server
sent. Declare it only for a website whose text is missing from the HTML it sends. `raw login` and
a rendered fetch need the optional browser component (`tools/preflight.sh --with browser`).
`tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is
refused.
@@ -70,4 +86,6 @@ host does not stop the fetch.
## Safety
A pattern runs only against the URL and its path, never against the page. A handler cannot run
code, read a file or reach another host: it is data, not a program.
code, read a file or reach another host: it is data, not a program. A `session` holds no
credential either - only where the user signs in and what a signed-in page looks like; the
session itself is stored outside the working tree by `raw login`.
tools/CONTRACT.md
+70 -4
View File
@@ -123,7 +123,8 @@ review read idempotent budget:exempt exit:0,1,
sources coverage read idempotent budget:counted exit:0 List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages.
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
raw fetch write non-idempotent budget:counted exit:0,1,42 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
raw login write idempotent budget:counted exit:0,1,42 Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
raw derive write non-idempotent budget:counted exit:0,1 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
raw capture write non-idempotent budget:counted exit:0,1 Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
raw status read idempotent budget:counted exit:0 Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
@@ -1420,7 +1421,7 @@ Regenerate the `kb/provenance.md` reverse index.
#### `raw fetch`
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
**SYNOPSIS**
@@ -1450,8 +1451,12 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- 1 The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
- 1 A target file already exists in `incoming/`
- 1 The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
- 1 The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
- 1 The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
- 1 raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
- 1 raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
- 42 The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
- 42 The site handler declares `render` and the optional browser component is not installed
**ON FAILURE**
@@ -1459,8 +1464,12 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once
- A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
- The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
- The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define -> Nothing was fetched or written. Fix the route with `tools/wikitool config set`, or have the user add the fetcher to `.wikitool-local.json` by hand - it names a program, so `config set` does not write it - then retry once
- The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB -> Nothing was written, and the built-in fetch was not tried. The message carries the fetcher's stderr - report it to the user; do not fetch the page another way
- raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget -> Nothing was written. Read the message: fetch the page without `--scope`, raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user
- raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
- The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired) -> Nothing was written. Show the user the output and stop: they sign in with `tools/wikitool raw login <host>`, never you. Retry once they say it is done
- The site handler declares `render` and the optional browser component is not installed -> Nothing was written. Show the user the output and stop: they install it with `tools/preflight.sh --with browser`. Retry once they say it is done
**NEVER**
@@ -1478,7 +1487,9 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
- `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
- Success prints the `raw accept` line for the written files and the `source_url` for the source page.
@@ -1486,10 +1497,64 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
- `sites/CONTRACT.md` - the instance's own handler files
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
- `wikitool raw login` - the user signs in once for a host routed through the session
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
#### `raw login`
Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
**SYNOPSIS**
- `wikitool raw login <host>`
**PROPERTIES**
- effect: write
- idempotent: yes
- atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
- budget: counted
- network: yes
**EXAMPLES**
- `tools/wikitool raw login www.heise.de`
**EXIT STATUS**
- 0 success
- 1 The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
- 1 The browser could not be started, or the window closed before any state could be read
- 42 The optional browser component is not installed
**ON FAILURE**
- The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed -> Nothing was stored. Route the host (`tools/wikitool config set fetch.routes '{"<host>": "session"}'`), add a `session` block to its handler (`sites/CONTRACT.md`), or fix the section, then retry once
- The browser could not be started, or the window closed before any state could be read -> Nothing was stored. Read the browser's message - on Linux it may name system libraries the browser needs, which whoever administers the machine installs - then retry once
- The optional browser component is not installed -> Nothing was stored. The user installs it with `tools/preflight.sh --with browser`, then runs this command again
**NEVER**
- Never run this command as an agent - it waits for a human at a browser window. When `raw fetch` asks for a sign-in, show the user its output and let them run it.
- Never read, print or copy the stored session.
**NOTES**
- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
- Prints no part of the session. Signing in again replaces the stored session.
**SEE ALSO**
- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
- `sites/CONTRACT.md` - a handler's `session` block
- `docs/reference/configuration.md` - the `fetch` section
#### `raw derive`
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
@@ -3849,6 +3914,7 @@ Check that this instance is correctly configured.
- Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
- Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
- Browser component (`browser`): whether the optional Playwright component and its browser are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is missing and the instance needs it: a host `fetch.routes` sends through the session (`raw login`), or a site handler that declares `render`. The user installs it with `tools/preflight.sh --with browser`.
- Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
- Read-only and exempt from the Iteration Budget Gate.
tools/README.md
+11 -1
View File
@@ -73,6 +73,13 @@ fix rather than degrading silently. `pypdf` and `olefile` are in `requirements.t
but are imported only when a PDF or an Outlook `.msg` is derived, so a missing one stops
that format alone; `doctor` reports each.
Playwright is the one package that is not in `requirements.txt`: it sits in
`requirements-browser.txt`, which the preflight installs - together with the Chromium it
drives, into `.venv/ms-playwright` - only when the user runs it with `--with browser`, and
keeps up to date from then on (`.venv/.chemenu-browser.sha256` remembers the choice). Only
`raw login` and a page a site handler declares `render` need it; `browser.py` imports it
inside the two functions that use it, and its absence is an exit 42 naming that option.
## Layout
```
@@ -118,7 +125,10 @@ tools/
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
corpus_diff.py invariant comparison of kb/ between two revisions
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
fetch_routes.py which way `raw fetch` takes for a host (`fetch.routes`): the built-in fetch, with the user's session, or an external fetcher run without a shell - and the check that a session page arrived signed in
web_session.py the user's session for a website behind a login, kept as a secret outside the tree: its name per host, storing it for `raw login`, the cookie jar `raw fetch` sends
browser.py the optional browser component (Playwright, `requirements-browser.txt`): `raw login`'s visible window and the headless fetch of a page a handler declares `render` - imported lazily, so a venv without it runs everything else
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, which links are subpages for `raw fetch --scope`, a website's login page and the sign of a signed-in page, and whether its pages need a browser - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
tools/chemenu/browser.py
+141
View File
@@ -0,0 +1,141 @@
"""The optional browser component: Playwright and the Chromium it drives.
Two things in the stack need a real browser, and nothing else does: `raw login`,
where the user signs in in a visible window - captcha and second factor
included - and a fetch of a page whose site handler declares `render`, because
the page draws its content with JavaScript. Everything else, a fetch with the
user's session included, goes through the built-in fetch.
So the component is optional. `tools/requirements-browser.txt` holds the
package, and the preflight installs it, together with the browser, only when the
user asks for it (`--with browser`); the agent installs nothing. Playwright is
imported inside the functions that use it, never at module level, so a venv
without it runs every other command unchanged. Missing, it is `BrowserMissing`
- exit 42, since only the user can install it.
The browser lives inside the venv (`browsers_path`), not in Playwright's default
cache in the home directory: the preflight sets everything it installs up inside
the install folder, and a second instance on the machine must not share or
update another's browser.
"""
from __future__ import annotations
import importlib.util
import os
import sys
import time
from pathlib import Path
from typing import Optional
from chemenu import toolpaths, web_capture
from chemenu.errors import BackendError, ChemenuError
PACKAGE = "playwright"
BROWSERS_ENV = "PLAYWRIGHT_BROWSERS_PATH"
LOGIN_WAIT_SECONDS = 15 * 60
PREFLIGHT_WITH = f"tools/preflight.sh --with browser (PowerShell 7: {toolpaths.PREFLIGHT_PWSH} --with browser)"
class BrowserMissing(ChemenuError):
"""The browser component is not installed. A human installs it - the
agent never does - so the CLI renders this as exit 42."""
def browsers_path() -> Path:
"""Where the preflight puts the browser: inside the venv the tool runs in."""
return Path(sys.prefix) / "ms-playwright"
def missing() -> Optional[str]:
"""Why the component cannot be used, or None when it can."""
if importlib.util.find_spec(PACKAGE) is None:
return f"the Python package `{PACKAGE}` is not installed in the venv"
directory = browsers_path()
if not directory.is_dir() or not any(p.name.startswith("chromium") for p in directory.iterdir()):
return f"its browser is not installed (looked in {directory})"
return None
def require(purpose: str) -> None:
"""Raise `BrowserMissing` naming `purpose` and the preflight option."""
why = missing()
if why is not None:
raise BrowserMissing(
f"{purpose} needs the optional browser component, and {why}. Ask the user to install it: "
f"{PREFLIGHT_WITH}. Only the user installs it; nothing was written."
)
def _environment() -> None:
os.environ.setdefault(BROWSERS_ENV, str(browsers_path()))
def login(url: str, wait_seconds: float = LOGIN_WAIT_SECONDS) -> dict:
"""Open `url` in a visible browser window and wait until the user closes
it, then return the browser state (cookies and storage). The state is read
once a second while the window is open, so closing the window - or the
whole browser - keeps what the last reading saw."""
require("`raw login`")
_environment()
from playwright.sync_api import Error, sync_playwright
state: Optional[dict] = None
try:
with sync_playwright() as driver:
browser = driver.chromium.launch(headless=False)
try:
context = browser.new_context()
context.new_page().goto(url)
deadline = time.monotonic() + wait_seconds
while time.monotonic() < deadline:
try:
state = context.storage_state()
except Error:
break
if not context.pages:
break
time.sleep(1)
finally:
try:
browser.close()
except Error:
pass
except Error as exc:
raise BackendError(f"The browser could not be started or could not open {url}: {exc}") from None
if state is None:
raise BackendError("The browser window closed before any state could be read - nothing was stored.")
return state
def render(
url: str, user_agent: str, state: Optional[dict] = None,
timeout: float = web_capture.TIMEOUT_SECONDS, max_bytes: int = web_capture.MAX_BYTES,
) -> web_capture.Response:
"""Load `url` in a headless browser - with `state`, the user's session,
when given - and return the page as the browser drew it: the serialised
DOM after the network went quiet, not the bytes the server sent."""
require(f"{url} is declared `render` by its site handler, so the fetch")
_environment()
from playwright.sync_api import Error, sync_playwright
web_capture.check_url(url)
try:
with sync_playwright() as driver:
browser = driver.chromium.launch(headless=True)
try:
context = browser.new_context(user_agent=user_agent, storage_state=state)
page = context.new_page()
response = page.goto(url, wait_until="networkidle", timeout=timeout * 1000)
body = page.content().encode("utf-8")
final_url = page.url
status = response.status if response is not None else 0
finally:
browser.close()
except Error as exc:
raise BackendError(f"The browser could not load {url}: {exc}") from None
web_capture.check_url(final_url)
if status >= 400:
raise web_capture.HttpError(f"{url} answered HTTP {status}.", status)
if len(body) > max_bytes:
raise BackendError(f"{url} is larger than {max_bytes // (1024 * 1024)} MiB as drawn by the browser.")
return web_capture.Response(url, final_url, status, "text/html; charset=utf-8", body)
tools/chemenu/cli_contract.py
+1 -1
View File
@@ -287,7 +287,7 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
"sources coverage", "sources trace", "sources rebuild-index",
)),
("Raw material and uploads", (
"raw fetch", "raw derive", "raw capture", "raw status", "raw pending", "raw accept",
"raw fetch", "raw login", "raw derive", "raw capture", "raw status", "raw pending", "raw accept",
"upload list", "upload show", "upload accept", "upload reject",
)),
("Git", (
tools/chemenu/commands/doctor.py
+32
View File
@@ -804,6 +804,32 @@ def check_site_handlers() -> Check:
return Check("site-handlers", "OK", f"{len(instance)} handler(s) of this instance: {detail}")
def check_browser() -> Check:
"""The optional browser component: OK whether or not it is installed, a
WARN only when this instance has something that needs it - a host routed
through the session (`raw login`) or a handler that declares `render`.
Never a FAIL: every other command runs without it."""
from chemenu import browser, settings, sites
why = browser.missing()
if why is None:
return Check("browser", "OK", f"available - Playwright in the venv, its browser in {rel_path(browser.browsers_path())}")
try:
routes = settings.load(only=("fetch",)).section("fetch")["routes"]
except ChemenuError:
routes = {} # a broken section is reported under settings-*
sessions = sorted(host for host, via in routes.items() if via == config_shapes.SESSION_ROUTE)
instance, _ = sites.read_instance()
rendered = sorted(site.name for site in sites.catalogue(instance) if site.render)
needs = ([f"`raw login` for {', '.join(sessions)}"] if sessions else []) + (
[f"the handler(s) {', '.join(rendered)}, which declare `render`"] if rendered else [])
if needs:
return Check("browser", "WARN", f"not available ({why}) - needed by {' and '.join(needs)}",
f"The user installs it: {browser.PREFLIGHT_WITH}")
return Check("browser", "OK", f"not installed - optional, needed only by `raw login` and handlers "
f"that declare `render`; the user installs it with {browser.PREFLIGHT_WITH}")
def check_session_id() -> Check:
"""Three-valued, not two: an explicit `WIKITOOL_SESSION_ID` and a
recognised harness variable (see `chemenu.session.HARNESS_ENV_VARS`) both
@@ -927,6 +953,7 @@ def run_doctor() -> list[Check]:
check_tasks_provider(),
*check_settings(),
check_site_handlers(),
check_browser(),
check_generated_files(),
check_session_id(),
check_telemetry(),
@@ -1007,6 +1034,11 @@ def run_doctor() -> list[Check]:
"schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, "
"naming the file and the reason, and `OK` listing the instance's handlers, each one that "
"replaces a stack handler (the same host and path) named as such.",
"Browser component (`browser`): whether the optional Playwright component and its browser "
"are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is "
"missing and the instance needs it: a host `fetch.routes` sends through the session "
"(`raw login`), or a site handler that declares `render`. The user installs it with "
"`tools/preflight.sh --with browser`.",
"Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a "
"fault.",
"Read-only and exempt from the Iteration Budget Gate.",
tools/chemenu/commands/raw/__init__.py
+2 -1
View File
@@ -7,13 +7,14 @@ subcommands are listed in `-h` and in the generated command reference.
"""
import typer
from chemenu.commands.raw import accept, pending, fetch, derive, capture
from chemenu.commands.raw import accept, pending, fetch, login, derive, capture
app = typer.Typer(help="Bring raw material into incoming/, and promote it from there into raw/.")
app.command("accept")(accept.raw_accept_command)
app.command("pending")(pending.raw_pending_command)
app.command("fetch")(fetch.raw_fetch_command)
app.command("login")(login.raw_login_command)
app.command("derive")(derive.raw_derive_command)
app.command("capture")(capture.raw_capture_command)
app.command("status")(capture.raw_status_command)
tools/chemenu/commands/raw/fetch.py
+129 -29
View File
@@ -20,8 +20,16 @@ from typing import Annotated, Optional
import typer
from rich.markup import escape
from chemenu import cli_contract, config, sites, web_capture
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
from chemenu import browser, cli_contract, config, fetch_routes, sites, web_capture
from chemenu.commands._util import (
clearance_on_settings,
fail,
load_settings,
needs_clearance,
path_budget_problem_for,
rel_path,
success,
)
from chemenu.errors import ChemenuError
from chemenu.version import VersionError, read_version
from chemenu.commands.raw._common import (
@@ -102,6 +110,50 @@ def _plan(url: str, *, canonical: bool, preferences: Optional[sites.Preferences]
fail(escape(str(exc)))
def _fetch_section():
"""The `fetch` settings section - which hosts go another way than the
built-in anonymous fetch."""
with clearance_on_settings():
return load_settings(only=("fetch",)).section("fetch")
def _route(plan: sites.Plan, preferences: sites.Preferences, section) -> fetch_routes.Route:
try:
return fetch_routes.route_for(plan, preferences, section)
except ChemenuError as exc:
fail(escape(str(exc)))
def _get(route: fetch_routes.Route, url: str, agent: str, context: str = "") -> web_capture.Response:
"""Fetch `url` along `route`: exit 42 when only the user can help (no
session, no browser component), exit 1 for everything else."""
try:
return fetch_routes.get(route, url, agent)
except (fetch_routes.SessionNeeded, browser.BrowserMissing) as exc:
needs_clearance(escape(str(exc)))
except ChemenuError as exc:
fail(escape(f"{exc}{context}"))
def _signed_in(route: fetch_routes.Route, response: web_capture.Response) -> None:
try:
fetch_routes.check_signed_in(route, response)
except fetch_routes.SessionNeeded as exc:
needs_clearance(escape(str(exc)))
def _route_entries(route: fetch_routes.Route) -> dict:
"""What `_scope.json` says about a route other than the built-in fetch."""
found: dict = {}
if route.kind == fetch_routes.SESSION:
found["session"] = route.host
if route.kind == fetch_routes.FETCHER:
found["fetcher"] = route.fetcher
if route.render:
found["rendered"] = True
return found
def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation, subpages: int = 0) -> None:
for line in plan.checks(derivation.decoded.text, subpages):
typer.echo(f"CHECK {line}")
@@ -202,22 +254,21 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
))
refuse_taken()
section = _fetch_section()
route = _route(plan, preferences, section)
agent = _user_agent()
try:
index = web_capture.fetch(
plan.url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
)
except ChemenuError as exc:
fail(escape(str(exc)))
index = _get(route, plan.url, agent)
index_retrieved = web_capture.utc_now()
if index.media_type not in web_capture.HTML_TYPES:
fail(escape(
f"{plan.url} answered {index.media_type or 'no content type'}, not an HTML page - "
"--scope bundles HTML pages. Nothing was written."
))
_signed_in(route, index)
index_derivation = web_capture.derive_document(
index.body, index.final_url, f"{stem}.html", response=index,
drop=plan.drop, site=plan.label, requested_url=plan.requested_url, derived_at=index_retrieved,
fetched_by=route.fetched_by, session=route.session_label, rendered=route.render,
)
urls = plan.subpages(index_derivation.decoded.text, index.final_url)
if not urls:
@@ -261,12 +312,9 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
for position, (sub_url, sub_name) in enumerate(zip(urls, names), start=1):
if preferences.pause_seconds:
time.sleep(preferences.pause_seconds)
try:
page = web_capture.fetch(
sub_url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
)
except ChemenuError as exc:
fail(escape(f"{exc} (subpage {position} of {len(urls)}) Nothing was written."))
sub_plan = _plan(sub_url, canonical=False, preferences=preferences)
sub_route = _route(sub_plan, preferences, section)
page = _get(sub_route, sub_url, agent, f" (subpage {position} of {len(urls)}) Nothing was written.")
retrieved = web_capture.utc_now()
if urllib.parse.urlsplit(page.final_url).netloc.lower() != host:
fail(escape(
@@ -278,10 +326,11 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
f"{sub_url} answered {page.media_type or 'no content type'}, not an HTML page - "
"--scope bundles HTML pages. Nothing was written."
))
sub_plan = _plan(sub_url, canonical=False, preferences=preferences)
_signed_in(sub_route, page)
derivation = web_capture.derive_document(
page.body, page.final_url, f"{sub_name}.html", response=page,
drop=sub_plan.drop, site=sub_plan.label, derived_at=retrieved,
fetched_by=sub_route.fetched_by, session=sub_route.session_label, rendered=sub_route.render,
)
(bundle / f"{sub_name}.html").write_bytes(page.body)
(bundle / f"{sub_name}.md").write_bytes(derivation.markdown)
@@ -293,6 +342,7 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
"url": plan.url,
**({"requested_url": plan.requested_url} if plan.requested_url else {}),
"retrieved": index_retrieved,
**_route_entries(route),
"pages": entries,
}
(bundle / SCOPE_MANIFEST).write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n")
@@ -317,7 +367,8 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
path="raw fetch",
summary="Capture a web page the user names into `incoming/`: the HTML as received plus a "
"derived text, for `raw accept` to promote. Pages of a known website are fetched in the "
"website's standard form and cleaned of its page furniture.",
"website's standard form and cleaned of its page furniture; a host the instance routes so is "
"fetched with the user's own session or through an external fetcher.",
synopsis=(
cli_contract.Variant(
usage="raw fetch <url> [--name <stem>]",
@@ -394,8 +445,27 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
"other non-HTML response (PDF, image, ...) as received under the extension of its "
"content type. Neither gets a derived file or a header.",
"Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at "
"most. No cookies, no JavaScript: a derived text under 200 characters is written, with "
"a warning to read it before accepting.",
"most. No cookies and no JavaScript unless the host is routed or its handler says so "
"(below): a derived text under 200 characters is written, with a warning to read it "
"before accepting.",
"Fetch routes: the `fetch` section of the settings sends a host's pages another way. "
"`session` fetches them with the cookies of the session the user stored with `raw login`, "
"still with the built-in fetch and the bytes as received; the header adds `session: "
"<host>` and never the session's content. The host's site handler names how a page shows "
"it arrived signed in: no stored session, or a page without that sign (an expired "
"session, a teaser), stops with exit 42 and writes nothing. Any other route names an "
"external fetcher of this checkout: run without a shell with the URL as its last "
"argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says "
"`fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A "
"fetcher that fails is never replaced by the built-in fetch. A host no route names is "
"fetched as before.",
"Rendered pages: a site handler that declares `render` has its pages fetched through a "
"headless browser - with the session, when the host has one - and the `.html` is the page "
"as the browser drew it, not the bytes the server sent; the header says `rendered: true`. "
"No page is rendered without that declaration, and an external fetcher's page never is. "
"It needs the optional browser component; without it the fetch stops with exit 42. "
"`--scope` fetches every page of the bundle the same way, and `_scope.json` names the "
"`session`, `fetcher` or `rendered` it used.",
"`--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser "
"- the way past a paywall or a script-rendered page. Its header carries `derived` (when "
"the text was derived) instead of `retrieved`, `final_url`, `http_status` and "
@@ -430,6 +500,35 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
"with the user; for a paywall or login, save the page in a browser and use "
"`--html`. An unreachable host or a timeout may be retried once",
),
cli_contract.Failure(
cause="The host is fetched with the user's session and none is stored, or the page "
"arrived without the sign of being signed in that its site handler names (the session "
"expired)",
reaction="Nothing was written. Show the user the output and stop: they sign in with "
"`tools/wikitool raw login <host>`, never you. Retry once they say it is done",
code=42,
),
cli_contract.Failure(
cause="The site handler declares `render` and the optional browser component is not "
"installed",
reaction="Nothing was written. Show the user the output and stop: they install it with "
"`tools/preflight.sh --with browser`. Retry once they say it is done",
code=42,
),
cli_contract.Failure(
cause="The `fetch` section is malformed, routes the host through the session though "
"no site handler of the host names a login page, or names a fetcher this checkout does "
"not define",
reaction="Nothing was fetched or written. Fix the route with `tools/wikitool config "
"set`, or have the user add the fetcher to `.wikitool-local.json` by hand - it names a "
"program, so `config set` does not write it - then retry once",
),
cli_contract.Failure(
cause="The external fetcher could not be started, timed out, ended with an exit other "
"than 0, printed nothing or more than 25 MiB",
reaction="Nothing was written, and the built-in fetch was not tried. The message carries "
"the fetcher's stderr - report it to the user; do not fetch the page another way",
),
cli_contract.Failure(
label="raw fetch --scope",
cause="No site handler offers subpages for the URL, the page offers none or more than "
@@ -462,7 +561,9 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
see_also=(
"`raw/CONTRACT.md` \"Getting a URL in: `raw fetch`\" - the rules and why",
"`sites/CONTRACT.md` - the instance's own handler files",
"`docs/reference/configuration.md` - the `sites` section, which tunes the site handlers",
"`docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, "
"and the `fetch` section, which routes a host",
"`wikitool raw login` - the user signs in once for a host routed through the session",
"`wikitool raw accept` - promotes the written files into `raw/`",
"`instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts",
),
@@ -508,32 +609,31 @@ def raw_fetch_command(
if scope:
_fetch_scope(url, name)
return
plan = _plan(url, canonical=True)
preferences = _preferences()
plan = _plan(url, canonical=True, preferences=preferences)
stem = name or web_capture.stem_for(plan.url)
try:
response = web_capture.fetch(
plan.url, _user_agent(), timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
)
except ChemenuError as exc:
fail(escape(str(exc)))
route = _route(plan, preferences, _fetch_section())
response = _get(route, plan.url, _user_agent())
incoming = _incoming_dir()
kind = response.media_type
if kind in web_capture.HTML_TYPES:
html_path, md_path = incoming / f"{stem}.html", incoming / f"{stem}.md"
_refuse_existing([html_path, md_path])
_signed_in(route, response)
derivation = web_capture.derive_document(
response.body, response.final_url, html_path.name, response=response,
drop=plan.drop, site=plan.label, requested_url=plan.requested_url,
fetched_by=route.fetched_by, session=route.session_label, rendered=route.render,
)
_write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
_warn(derivation, md_path)
_echo_checks(plan, derivation, len(plan.subpages(derivation.decoded.text, response.final_url)))
anonymous = route.kind == fetch_routes.PLAIN and not route.render
success(escape(
f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n"
+ _next_lines([html_path, md_path], plan.url) + "\n"
+ _TEASER_HINT.format(url=plan.url)
+ _next_lines([html_path, md_path], plan.url)
+ ("\n" + _TEASER_HINT.format(url=plan.url) if anonymous else "")
))
return
tools/chemenu/commands/raw/login.py
+141
View File
@@ -0,0 +1,141 @@
"""`wikitool raw login <host>` - the user signs in to a website once, in a visible
browser window, and the session it leaves is stored as a secret outside the
working tree for `raw fetch` to use. Only a human runs it: it waits for someone
at the window. The browser part is `chemenu.browser`, the storage
`chemenu.web_session`; this adapter checks the order of things and says what
happened.
"""
from __future__ import annotations
import typer
from rich.markup import escape
from chemenu import browser, cli_contract, config, sites, web_session
from chemenu.commands._util import clearance_on_settings, fail, load_settings, needs_clearance, success
from chemenu.config_shapes import SESSION_ROUTE
from chemenu.errors import ChemenuError
from chemenu.fetch_routes import host_of
def _host(value: str) -> str:
host = value.strip().lower()
if "://" in host:
host = host_of(host)
if not host or any(c in host for c in "/:?#@ "):
fail(escape(f"{value!r} is not a host name - pass the host alone, such as www.example.org."))
return host
@cli_contract.record(cli_contract.CommandRecord(
path="raw login",
summary="Let the user sign in to a website once, in a visible browser window, and store the "
"session it leaves as a secret outside the working tree, for `raw fetch` to fetch that "
"website's pages with. Only the user runs it.",
synopsis=(cli_contract.Variant(usage="raw login <host>"),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.YES,
atomic="Yes - the secrets file is replaced in one rename, with every other entry kept",
budget=cli_contract.Budget.COUNTED,
network=cli_contract.Network.YES,
),
notes=(
"Opens the login page the host's site handler names (its `session` block) in a visible "
"browser window and waits until the user has signed in - captcha and second factor "
"included - and closed the window, at most 15 minutes.",
"Stores only the browser state the window leaves (cookies and storage), never a "
"password, under `session_<host>` (every run of other characters as one `_`) in the "
"secrets file of the instance, outside the working tree and readable by its owner only. "
"`WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The "
"session is never written to 1Password.",
"Needs the host routed through the session first (`fetch.routes`, set with `wikitool "
"config set`), which also gives the instance the id its secrets file is found by.",
"Needs the optional browser component, which the user installs with the preflight's "
"`--with browser` option.",
"Prints no part of the session. Signing in again replaces the stored session.",
),
failures=(
cli_contract.Failure(
cause="The host is not routed through the session in `fetch.routes`, no enabled site "
"handler of the host names a login page, or the `fetch` settings are malformed",
reaction="Nothing was stored. Route the host (`tools/wikitool config set fetch.routes "
"'{\"<host>\": \"session\"}'`), add a `session` block to its handler (`sites/CONTRACT.md`), "
"or fix the section, then retry once",
),
cli_contract.Failure(
cause="The browser could not be started, or the window closed before any state could "
"be read",
reaction="Nothing was stored. Read the browser's message - on Linux it may name system "
"libraries the browser needs, which whoever administers the machine installs - then retry once",
),
cli_contract.Failure(
cause="The optional browser component is not installed",
reaction="Nothing was stored. The user installs it with `tools/preflight.sh --with "
"browser`, then runs this command again",
code=42,
),
),
examples=("tools/wikitool raw login www.heise.de",),
never=(
"Never run this command as an agent - it waits for a human at a browser window. When "
"`raw fetch` asks for a sign-in, show the user its output and let them run it.",
"Never read, print or copy the stored session.",
),
see_also=(
"`wikitool raw fetch` - fetches the pages of a routed host with the stored session",
"`raw/CONTRACT.md` \"Getting a URL in: `raw fetch`\" - fetching behind a login",
"`sites/CONTRACT.md` - a handler's `session` block",
"`docs/reference/configuration.md` - the `fetch` section",
),
))
def raw_login_command(
host: str = typer.Argument(..., help="The website's host, such as www.example.org"),
):
"""Let the user sign in to a website in a browser window and store the session.
Only a human runs this. See raw/CONTRACT.md "Getting a URL in: `raw fetch`"."""
host = _host(host)
with clearance_on_settings():
loaded = load_settings(only=("fetch",))
section = loaded.section("fetch")
route = section["routes"].get(host)
if route != SESSION_ROUTE:
fail(escape(
f"{host} is not fetched with a session - `fetch.routes` {'sends it to ' + repr(route) if route else 'does not list it'}. "
f"Route it first: tools/wikitool config set fetch.routes '{{\"{host}\": \"{SESSION_ROUTE}\"}}' "
"(keep the hosts already listed there)."
))
try:
preferences = sites.read_preferences()
except ChemenuError as exc:
fail(escape(str(exc)))
site = sites.login_site(host, preferences.disabled, preferences.instance)
if site is None or site.session is None:
fail(escape(
f"No enabled site handler of {host} names a login page - add a `session` block to a handler "
f"of {host} (sites/CONTRACT.md)."
))
if not loaded.instance_id:
fail(escape(
f"{config.INSTANCE_SETTINGS_FILENAME} has no instance_id, so there is no secrets file to store "
"the session in - set the route with `tools/wikitool config set`, which writes one."
))
try:
state = browser.login(site.session.login)
except browser.BrowserMissing as exc:
needs_clearance(escape(str(exc)))
except ChemenuError as exc:
fail(escape(f"{exc} Nothing was stored."))
try:
path = web_session.store(host, loaded.instance_id, state)
except (ChemenuError, OSError) as exc:
fail(escape(f"The session could not be stored: {exc}"))
cookies = len(state.get("cookies", [])) if isinstance(state, dict) else 0
shadow = web_session.overridden(host)
if shadow:
typer.echo(f"WARN {shadow} is set and answers before the secrets file - unset it, or the "
"session just stored is not used.")
success(escape(
f"Signed in to {host}: the session ({cookies} cookie(s)) is stored as "
f"{web_session.secret_name(host)} in {path}, outside the working tree. "
f"`raw fetch` uses it for {host} until it expires; then it asks for this command again."
))
tools/chemenu/config_shapes.py
+67 -1
View File
@@ -244,6 +244,38 @@ def site_notes_problem(value: list[dict]) -> Optional[str]:
return None
SESSION_ROUTE = "session"
_HOST = re.compile(r"[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*")
_FETCHER_NAME = re.compile(r"[a-z0-9]+(-[a-z0-9]+)*")
def object_value(value: Any) -> dict:
if not isinstance(value, dict):
raise TypeError("not an object")
return dict(value)
def fetch_routes_problem(value: dict) -> Optional[str]:
for host, route in value.items():
if not _HOST.fullmatch(host):
return f"has the key {host!r}, which is not a lower-case host name"
if not isinstance(route, str) or not _FETCHER_NAME.fullmatch(route):
return (f"maps {host} to {route!r} - a route is {SESSION_ROUTE!r} or the name of a fetcher "
"(lower-case letters, digits and hyphens)")
return None
def fetchers_problem(value: dict) -> Optional[str]:
for name, command in value.items():
if name == SESSION_ROUTE or not _FETCHER_NAME.fullmatch(name):
return (f"has the name {name!r} - a fetcher's name is lower-case letters, digits and "
f"hyphens, and not {SESSION_ROUTE!r}")
if (not isinstance(command, list) or not command
or not all(isinstance(part, str) and part for part in command)):
return f"{name} must be a non-empty array of non-empty strings: the program, then its arguments"
return None
def _positive(value: int) -> Optional[str]:
return None if value > 0 else "must be positive"
@@ -510,11 +542,45 @@ SITES = Shape(
"broken section is never read as \"no handlers\".",
)
FETCH = Shape(
key="fetch",
title="Fetch routes",
purpose="Sends the pages of a host another way than the built-in anonymous fetch of `raw "
"fetch`: with the session the user signed in with through `raw login`, or through an external "
"fetcher program of this checkout.",
fields=(
Field(
"routes", "object", f"Host (exact, lower case) to route: `\"{SESSION_ROUTE}\"` fetches "
"with the user's session - the host needs a site handler that names its login page - "
"and any other value names a fetcher of `fetchers`. A host not listed is fetched as "
"before.",
default={}, json_type="object", coerce=object_value, check=fetch_routes_problem,
tier=INSTANCE,
),
Field(
"fetchers", "object", "Fetcher name to command: an array of the program and its "
"arguments, run without a shell with the URL appended. Exit 0 and the page's HTML on "
"stdout is a fetch; anything else stops `raw fetch` with exit 1 and the program's "
"stderr.",
default={}, json_type="object", coerce=object_value, check=fetchers_problem,
tier=CHECKOUT, names_command=True,
),
),
examples=({
"routes": {"www.heise.de": SESSION_ROUTE, "intranet.example.org": "homelab"},
"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]},
},),
absent="Every page is fetched by the built-in anonymous fetch, as before.",
malformed="An error when `raw fetch` or `raw login` reads it: the command stops with exit 1 "
"before fetching anything. A route to a fetcher this checkout does not define stops a fetch "
"of that host the same way.",
)
# The sections of the tiered settings files, `wikitool.json` and
# `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
# stand in a section of its own.
SETTINGS_SECTIONS: tuple[Shape, ...] = (
TELEMETRY, REMOTES, UPLOAD, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES,
TELEMETRY, REMOTES, UPLOAD, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES, FETCH,
)
# The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
tools/chemenu/fetch_routes.py
+179
View File
@@ -0,0 +1,179 @@
"""Which way a page is fetched: the built-in anonymous fetch, the user's own
session, or an external fetcher - and the fetch itself.
The instance decides per host (`fetch.routes`, tier `instance`):
- not listed: the built-in fetch, exactly as before - no cookie, no browser;
- `session`: the built-in fetch with the cookies of the session the user signed
in with through `raw login` (`chemenu.web_session`). The host's site handler
names the login page and how a page shows that it arrived signed in; a page
that did not, or no stored session at all, is `SessionNeeded` - exit 42,
because only the user can sign in again. A teaser is never kept in silence;
- any other value: an external fetcher of this checkout (`fetch.fetchers`, tier
`checkout`, never `instance` - it names a program to run). It is run without
a shell with the URL appended; exit 0 and HTML on stdout is the page,
anything else is an error, and the built-in fetch is never tried instead.
A site handler that declares `render` sends a fetch through a headless browser
instead of the built-in fetch, with the session when the route has one
(`chemenu.browser`); an external fetcher is its own fetch and is never rendered.
No `typer`, no `rich`: errors are `ChemenuError`s, and the CLI decides what each
one means for the exit code.
"""
from __future__ import annotations
import subprocess
import urllib.parse
from dataclasses import dataclass
from typing import Optional
from chemenu import browser, config, sites, web_capture, web_session
from chemenu.config_shapes import SESSION_ROUTE
from chemenu.errors import BackendError, ChemenuError, ValidationError
PLAIN = "plain"
SESSION = "session"
FETCHER = "fetcher"
FETCHER_TIMEOUT_SECONDS = 120
_STDERR_TAIL_CHARS = 2000
class SessionNeeded(ChemenuError):
"""No usable session for the host: none stored, or the page arrived
without the sign of being signed in. Only the user can sign in, so the CLI
renders this as exit 42."""
@dataclass(frozen=True)
class Route:
"""How one page is fetched. `session` is the handler's description of
the login, for a `SESSION` route; `command` the program and arguments of a
`FETCHER` route; `render` whether the page goes through a browser."""
host: str
kind: str = PLAIN
fetcher: str = ""
command: tuple[str, ...] = ()
session: Optional[sites.Session] = None
render: bool = False
instance_id: Optional[str] = None
@property
def fetched_by(self) -> str:
if self.kind == FETCHER:
return f"wikitool raw fetch (fetcher: {self.fetcher})"
return "wikitool raw fetch"
@property
def session_label(self) -> str:
return self.host if self.kind == SESSION else ""
def host_of(url: str) -> str:
return (urllib.parse.urlsplit(url).hostname or "").lower()
def route_for(plan: sites.Plan, preferences: sites.Preferences, section) -> Route:
"""The route for the page `plan` describes, from the `fetch` settings
`section` (`chemenu.settings.Section`). Raises `ValidationError` for a route
that cannot work: `session` for a host no handler names a login page for,
or a fetcher this checkout does not define."""
host = host_of(plan.url)
render = bool(plan.site and plan.site.render)
via = section["routes"].get(host)
if via is None:
return Route(host, render=render)
if via == SESSION_ROUTE:
login = sites.login_site(host, preferences.disabled, preferences.instance)
if login is None:
raise ValidationError(
f"fetch.routes sends {host} through the user's session, and no enabled site handler of "
f"that host names a login page - add a `session` block to a handler of {host} "
"(sites/CONTRACT.md), or remove the route."
)
own = plan.site.session if plan.site and plan.site.session else login.session
return Route(host, SESSION, session=own, render=render, instance_id=section.instance_id)
command = section["fetchers"].get(via)
if command is None:
raise ValidationError(
f"fetch.routes sends {host} to the fetcher {via!r}, which this checkout does not define - "
f"the user adds `fetch.fetchers.{via}` to {config.LOCAL_SETTINGS_FILENAME} by hand "
"(it names a program to run, so `config set` does not write it)."
)
return Route(host, FETCHER, fetcher=via, command=tuple(command))
def _login_hint(host: str) -> str:
return (f"Ask the user to run `tools/wikitool raw login {host}` - it opens a browser window for "
"them to sign in; never run it yourself. Nothing was written.")
def get(route: Route, url: str, user_agent: str) -> web_capture.Response:
"""Fetch `url` the way `route` says. Raises `SessionNeeded` when no
session is stored, `browser.BrowserMissing` when a rendered fetch has no
browser, and `BackendError`/`ValidationError` for everything else."""
if route.kind == FETCHER:
return run_fetcher(route, url)
state = None
if route.kind == SESSION:
state = web_session.load(route.host, route.instance_id)
if state is None:
raise SessionNeeded(f"{route.host} is fetched with the user's session, and none is stored. "
+ _login_hint(route.host))
if route.render:
return browser.render(url, user_agent, state)
# The built-in fetch exactly as before when there is no session to send.
extra = {"cookies": web_session.cookie_jar(state)} if state is not None else {}
return web_capture.fetch(
url, user_agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES, **extra
)
def check_signed_in(route: Route, response: web_capture.Response) -> None:
"""Raise `SessionNeeded` when a page fetched with the session did not
arrive signed in - the session expired, or the website ended it."""
if route.kind != SESSION or route.session is None or response.media_type not in web_capture.HTML_TYPES:
return
text = web_capture.decode(response.body, response.content_type).text
if not route.session.is_signed_in(text):
raise SessionNeeded(
f"{response.final_url} arrived without a valid session - the page does not show the sign of "
f"being signed in that the site handler names, so the session for {route.host} has expired "
"or was ended. " + _login_hint(route.host)
)
def _tail(stderr: bytes) -> str:
text = stderr.decode("utf-8", errors="replace").strip()
if len(text) > _STDERR_TAIL_CHARS:
text = "..." + text[-_STDERR_TAIL_CHARS:]
return text or "(nothing on stderr)"
def run_fetcher(route: Route, url: str, timeout: float = FETCHER_TIMEOUT_SECONDS,
max_bytes: int = web_capture.MAX_BYTES) -> web_capture.Response:
"""Run the external fetcher of `route` for `url`: no shell, the URL as the
last argument, stdin closed. Its stdout is the page, byte for byte."""
web_capture.check_url(url)
where = f"the fetcher {route.fetcher!r}"
try:
result = subprocess.run(
[*route.command, url], capture_output=True, stdin=subprocess.DEVNULL, timeout=timeout, check=False,
)
except FileNotFoundError:
raise BackendError(f"{where} could not be started: {route.command[0]} does not exist. Nothing was written.") from None
except subprocess.TimeoutExpired:
raise BackendError(f"{where} did not finish within {timeout:g} s for {url}. Nothing was written.") from None
except OSError as exc:
raise BackendError(f"{where} could not be started ({exc}). Nothing was written.") from None
if result.returncode != 0:
raise BackendError(
f"{where} ended with exit {result.returncode} for {url} - nothing was written, and the built-in "
f"fetch was not tried instead. Its stderr:\n{_tail(result.stderr)}"
)
if not result.stdout:
raise BackendError(f"{where} ended with exit 0 for {url} but printed nothing. Nothing was written.")
if len(result.stdout) > max_bytes:
raise BackendError(f"{where} printed more than {max_bytes // (1024 * 1024)} MiB for {url}. Nothing was written.")
return web_capture.Response(url, url, 0, "text/html", result.stdout)
tools/chemenu/reference_doc.py
+1
View File
@@ -69,6 +69,7 @@ CONFIG_REGIONS = (
("reference-config-caldav", "CALDAV"),
("reference-config-upload", "UPLOAD"),
("reference-config-sites", "SITES"),
("reference-config-fetch", "FETCH"),
)
_SLUG_RE = re.compile(r"[^a-z0-9]+")
tools/chemenu/settings.py
+15
View File
@@ -558,6 +558,21 @@ def read_secrets_file(path: Path) -> Optional[dict]:
return data
def write_secret(instance_id: str, name: str, value: str) -> Path:
"""Put `value` under `name` into the secrets file of `instance_id`, keeping
every other entry, and return the file's path. The file is replaced in one
rename and created readable by its owner only (`write_file` goes through
`mkstemp`). Used by what writes a secret the user produced but never typed -
`raw login`'s session; `config set` refuses a secret value outright."""
if not _NAME_RE.match(name):
raise ValidationError(f"{name!r} is not a secret name: lower-case letters, digits and single underscores")
path = config.secrets_file(instance_id)
data = read_secrets_file(path) or {}
data[name] = value
write_file(path, data)
return path
def _resolve_named(ref: Ref, key: str, instance_id: Optional[str]) -> str:
value = os.environ.get(ref.env_var, "")
if value:
tools/chemenu/sites/__init__.py
+49 -2
View File
@@ -16,7 +16,10 @@ host and path the stack has a handler for replaces it; the `sites` section of th
settings still switches handlers off, sets the `locale` and adds hint lines
(`notes`). A handler may also offer the subpages of a page
(`Subpages`: the links to take, under a path prefix), which `raw fetch --scope`
fetches as one bundle. A handler may still be registered from a Python module in this
fetches as one bundle; name the login page of a website behind a login and how
a page shows it arrived signed in (`Session`, for `raw login` and a fetch with
the user's session); and declare that its pages are drawn by JavaScript
(`render`). A handler may still be registered from a Python module in this
package. The core knows neither `typer` nor `rich` and raises `ValidationError`.
"""
from __future__ import annotations
@@ -123,12 +126,44 @@ class Subpages:
return found
@dataclass(frozen=True)
class Session:
"""What a website behind a login needs from the stack: the page `raw login`
opens for the user to sign in on, and how a fetched page shows whether it
arrived signed in. `signed_in` matches only on a signed-in page (a logout
link), `signed_out` only on a page that was not (a paywall teaser); with
both, both must hold. Whether a host's pages are fetched with the session at
all is the instance's choice (`fetch.routes`), not the handler's."""
login: str
signed_in: Optional[Rule] = None
signed_out: Optional[Rule] = None
def __post_init__(self) -> None:
if self.signed_in is None and self.signed_out is None:
raise ValueError("a session needs a signed_in or a signed_out rule - nothing would tell a teaser from the article")
if urllib.parse.urlsplit(self.login).scheme.lower() not in web_capture.ALLOWED_SCHEMES:
raise ValueError(f"a session's login page is an http(s) URL, not {self.login!r}")
def is_signed_in(self, html: str) -> bool:
document = web_capture.parse(html)
if self.signed_in is not None and not web_capture.count_matches(document, self.signed_in):
return False
if self.signed_out is not None and web_capture.count_matches(document, self.signed_out):
return False
return True
@dataclass(frozen=True)
class Site:
"""One handler. `host` is matched exactly (lower case); `path` is a regular
expression matched against the start of the URL path, and the longest `path`
wins among the handlers of a host. `parent` names a handler this one
inherits from: its rewrites, rules and checks come first."""
inherits from: its rewrites, rules and checks come first.
`session` is what a fetch with the user's own login needs; `render` says the
page draws its content with JavaScript, so it is fetched through a browser
and the header says `rendered: true` - never without this declaration."""
name: str
host: str
@@ -139,6 +174,8 @@ class Site:
drop: tuple[Rule, ...] = ()
checks: tuple[Check, ...] = ()
subpages: Optional[Subpages] = None
session: Optional[Session] = None
render: bool = False
origin: str = "stack"
def __post_init__(self) -> None:
@@ -177,6 +214,8 @@ def inherit(candidate: Site, parent: Site) -> Site:
drop=parent.drop + candidate.drop,
checks=parent.checks + candidate.checks,
subpages=candidate.subpages or parent.subpages,
session=candidate.session or parent.session,
render=candidate.render or parent.render,
)
if len(flat.checks) > MAX_CHECKS:
raise ValueError(
@@ -239,6 +278,14 @@ def site_for(url: str, disabled: frozenset[str] = frozenset(), instance: tuple[S
return max(found, key=lambda s: len(s.path), default=None)
def login_site(host: str, disabled: frozenset[str] = frozenset(), instance: tuple[Site, ...] = ()) -> Optional[Site]:
"""The enabled handler of `host` whose session `raw login` opens: among
those that declare one, the one with the shortest path pattern - the
website's own, not a section's."""
found = [s for s in catalogue(instance) if s.name not in disabled and s.host == host and s.session]
return min(found, key=lambda s: len(s.path), default=None)
# --- the instance's part ----------------------------------------------------
tools/chemenu/sites/data.py
+14 -1
View File
@@ -15,7 +15,7 @@ from typing import Any, Optional
from jsonschema import Draft202012Validator
from chemenu.sites import Broken, Check, Rewrite, Rule, Site, Subpages, inherit
from chemenu.sites import Broken, Check, Rewrite, Rule, Session, Site, Subpages, inherit
SCHEMA_PATH = Path(__file__).with_name("site.schema.json")
STACK_DIRECTORY = Path(__file__).with_name("handlers")
@@ -77,6 +77,17 @@ def build(name: str, document: Any, origin: str) -> Site:
subpages = Subpages(_rule(entry["links"]), entry.get("prefix", "{page}"))
except ValueError as exc:
raise ValueError(f"subpages: {exc}") from None
session = None
if "session" in document:
entry = document["session"]
try:
session = Session(
entry["login"],
_rule(entry["signed_in"]) if "signed_in" in entry else None,
_rule(entry["signed_out"]) if "signed_out" in entry else None,
)
except ValueError as exc:
raise ValueError(f"session: {exc}") from None
try:
return Site(
name=name,
@@ -88,6 +99,8 @@ def build(name: str, document: Any, origin: str) -> Site:
drop=tuple(_rule(entry) for entry in document.get("drop", [])),
checks=tuple(checks),
subpages=subpages,
session=session,
render=document.get("render", False),
origin=origin,
)
except (ValueError, re.error) as exc:
tools/chemenu/sites/site.schema.json
+16 -1
View File
@@ -46,7 +46,22 @@
"links": {"$ref": "#/$defs/rule"},
"prefix": {"type": "string", "pattern": "^(\\{page\\}|/)", "maxLength": 200}
}
}
},
"session": {
"type": "object",
"additionalProperties": false,
"required": ["login"],
"properties": {
"login": {"type": "string", "pattern": "^https?://", "maxLength": 400},
"signed_in": {"$ref": "#/$defs/rule"},
"signed_out": {"$ref": "#/$defs/rule"}
},
"anyOf": [
{"required": ["signed_in"]},
{"required": ["signed_out"]}
]
},
"render": {"type": "boolean"}
},
"$defs": {
"rule": {
tools/chemenu/tests/test_cli.py
+3 -1
View File
@@ -341,11 +341,13 @@ def test_network_yes_is_exactly_the_commands_that_can_reach_outside_this_checkou
since `--latest`, which asks the release feed and downloads from it, and `raw fetch`
since it exists (Gitea #120) - its `--html` form stays offline, which does not turn the
command back to `no`. `raw capture` and `raw status` reach a git remote (Gitea #177), and so
does `export guidelines --push` (Gitea #179). Pinned as an explicit
does `export guidelines --push` (Gitea #179). `raw login` opens a website's login page in a
browser (Gitea #169). Pinned as an explicit
set so a command gaining or losing that reach is a deliberate edit here, not a silent
drift between the property and what the command actually does."""
expected = {
"raw fetch",
"raw login",
"raw capture",
"raw status",
"export guidelines",
tools/chemenu/tests/test_config_shapes.py
+1 -1
View File
@@ -78,7 +78,7 @@ def test_the_telemetry_policy_reads_the_record():
def test_one_record_per_section():
assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
"telemetry", "remotes", "upload", "tasks", "superproductivity", "caldav", "sites",
"telemetry", "remotes", "upload", "tasks", "superproductivity", "caldav", "sites", "fetch",
]
tools/chemenu/tests/test_doctor.py
+32
View File
@@ -740,3 +740,35 @@ def test_marked_scripts_fail_with_the_unblock_command(instance, monkeypatch):
def test_unmarked_scripts_are_ok(instance, monkeypatch):
_windows(monkeypatch)
assert _status(doctor.run_doctor(), "script-marks") == "OK"
# --- the optional browser component --------------------------------------------
def _without_browser(monkeypatch) -> None:
from chemenu import browser
monkeypatch.setattr(browser, "missing", lambda: "the Python package `playwright` is not installed in the venv")
def test_a_missing_browser_component_nothing_needs_is_ok(instance, monkeypatch):
_without_browser(monkeypatch)
checks = doctor.run_doctor()
assert _status(checks, "browser") == "OK"
assert "--with browser" in _detail(checks, "browser")
def test_a_missing_browser_component_a_session_route_needs_is_a_warning_not_a_failure(instance, monkeypatch):
_without_browser(monkeypatch)
write_settings(config.ROOT, monkeypatch, fetch={"routes": {"www.example.org": "session"}})
checks = doctor.run_doctor()
assert _status(checks, "browser") == "WARN"
assert "www.example.org" in _detail(checks, "browser")
assert "--with browser" in _fix(checks, "browser")
def test_an_installed_browser_component_is_ok(instance, monkeypatch):
from chemenu import browser
monkeypatch.setattr(browser, "missing", lambda: None)
assert _status(doctor.run_doctor(), "browser") == "OK"
tools/chemenu/tests/test_preflight.py
+77 -1
View File
@@ -78,11 +78,24 @@ case "$1" in
shift 2
[ "$1" = --version ] && exit 0
echo "$*" >> "$PIP_LOG"
if [ -n "${{FAKE_PIP_FAIL:-}}" ]; then
case "$*" in
*requirements-browser*) fail=${{FAKE_BROWSER_PIP_FAIL:-}} ;;
*) fail=${{FAKE_PIP_FAIL:-}} ;;
esac
if [ -n "$fail" ]; then
echo "ERROR: Could not find a version that satisfies the requirement (network unreachable)" >&2
exit 1
fi
exit 0 ;;
playwright)
shift 2
echo "playwright $* into $PLAYWRIGHT_BROWSERS_PATH" >> "$PIP_LOG"
if [ -n "${{FAKE_PLAYWRIGHT_FAIL:-}}" ]; then
echo "Host system is missing dependencies to run browsers" >&2
exit 1
fi
mkdir -p "$PLAYWRIGHT_BROWSERS_PATH/chromium-1200"
exit 0 ;;
esac ;;
esac
exit 1
@@ -319,6 +332,69 @@ def test_pip_failure_stops_and_is_retried_next_time(machine, shell):
assert machine.pip_log.read_text(encoding="utf-8").count("install") == 2
# --- the optional browser component -------------------------------------------------
def _browser_lines(machine) -> list[str]:
if not machine.pip_log.exists():
return []
return [line for line in machine.pip_log.read_text(encoding="utf-8").splitlines()
if "requirements-browser" in line or line.startswith("playwright ")]
@pytest.mark.parametrize("shell", SHELLS)
def test_without_with_browser_nothing_of_the_component_is_installed(machine, shell):
machine.standard()
(machine.tools / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
result = machine.run(shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
assert _browser_lines(machine) == []
assert not any(line.split()[1:2] == ["browser"] for line in result.stdout.splitlines())
@pytest.mark.parametrize("shell", SHELLS)
def test_with_browser_installs_the_component_into_the_venv_and_keeps_it(machine, shell):
machine.standard()
(machine.tools / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
result = machine.run("--with", "browser", shell=shell)
assert result.returncode == 0, result.stdout + result.stderr
venv = machine.tools / ".venv"
lines = _browser_lines(machine)
assert any("-r" in line and line.endswith("requirements-browser.txt") for line in lines)
assert f"playwright install chromium into {venv / 'ms-playwright'}" in lines
assert any(line.split()[:2] == ["OK", "browser"] for line in result.stdout.splitlines())
assert (venv / ".chemenu-browser.sha256").is_file()
# A later plain run keeps it, and reinstalls only when its requirements change.
assert machine.run(shell=shell).returncode == 0
assert len(_browser_lines(machine)) == 2
(machine.tools / "requirements-browser.txt").write_text("playwright>=2\n", encoding="utf-8")
result = machine.run(shell=shell)
assert result.returncode == 0 and len(_browser_lines(machine)) == 4
assert "OK browser" in result.stdout
@pytest.mark.parametrize("shell", SHELLS)
@pytest.mark.parametrize("failing", ["FAKE_BROWSER_PIP_FAIL", "FAKE_PLAYWRIGHT_FAIL"])
def test_a_component_that_fails_stops_but_leaves_wikitool_working(machine, shell, failing):
machine.standard()
(machine.tools / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
machine.extra_env[failing] = "1"
result = machine.run("--with=browser", shell=shell)
assert result.returncode == 42
assert_guidance(result.stdout, "optional browser component", "--with browser")
assert machine.recorded()["complete"] is True
assert not (machine.tools / ".venv" / ".chemenu-browser.sha256").exists()
@pytest.mark.parametrize("shell", SHELLS)
def test_an_unknown_component_is_refused(machine, shell):
machine.standard()
result = machine.run("--with", "spreadsheet", shell=shell)
assert result.returncode == 1
assert "not an optional component" in result.stderr
assert not machine.tools_file.exists()
# --- the store alias ----------------------------------------------------------------
tools/chemenu/tests/test_preflight_pwsh.py
+36
View File
@@ -183,6 +183,42 @@ def test_pip_failure_stops_and_is_retried_next_time(machine):
assert machine.recorded()["complete"] is True
def _browser_lines(machine: Machine) -> list[str]:
if not machine.pip_log.exists():
return []
return [line for line in machine.pip_log.read_text(encoding="utf-8").splitlines()
if "requirements-browser" in line or line.startswith("playwright ")]
def test_with_browser_installs_the_component_and_a_plain_run_keeps_it(machine):
(machine.tools / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
assert _preflight(machine).returncode == 0
assert _browser_lines(machine) == []
result = _preflight(machine, "--with", "browser")
assert result.returncode == 0, result.stdout + result.stderr
venv = machine.tools / ".venv"
assert f"playwright install chromium into {venv / 'ms-playwright'}" in _browser_lines(machine)
assert "OK browser" in result.stdout
assert _preflight(machine).returncode == 0
assert len(_browser_lines(machine)) == 2
@pytest.mark.parametrize("failing", ["FAKE_BROWSER_PIP_FAIL", "FAKE_PLAYWRIGHT_FAIL"])
def test_a_component_that_fails_stops_but_leaves_wikitool_working(machine, failing):
(machine.tools / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
machine.extra_env[failing] = "1"
result = _preflight(machine, "--with=browser")
assert result.returncode == 42
assert_guidance(result.stdout, "optional browser component", "--with browser")
assert machine.recorded()["complete"] is True
def test_an_unknown_component_is_refused(machine):
result = _preflight(machine, "--with", "spreadsheet")
assert result.returncode == 1
assert "not an optional component" in result.stderr
@pytest.mark.parametrize("platform, alias", [("linux", "python3"), ("windows", "python")])
def test_store_alias_is_never_run_and_never_recorded(machine, platform, alias):
marker = machine.base / "alias-was-run"
tools/chemenu/tests/test_raw_fetch_routes.py
+352
View File
@@ -0,0 +1,352 @@
"""`wikitool raw fetch` along a route: with the user's session, through an
external fetcher, or rendered by a browser (Gitea #169).
As in `test_raw_fetch.py`, every request goes to a real `http.server` on
127.0.0.1. The paid pages answer by the `Cookie` header they receive - the full
article to the right session, a teaser to anything else - so what is under test
is what a session actually sends, not what a stub was told. The browser is the
one thing replaced: Playwright is an optional component that CI does not
install, so `browser.render` and `browser.missing` are stood in for.
"""
from __future__ import annotations
import datetime
import json
import os
import sys
from pathlib import Path
import pytest
import typer
from chemenu import browser, config, config_shapes, fetch_routes, web_capture, web_session
from chemenu.commands.raw.fetch import raw_fetch_command
from chemenu.tests.conftest import write_settings
from chemenu.tests.test_raw_fetch import ARTICLE, Site
SECRET = "sid-6f3a-THIS-IS-THE-SESSION-VALUE"
FULL = (
b"<!doctype html><html><head><meta charset=\"utf-8\"><title>Paid</title></head><body><main>"
b"<h1>The paid article</h1><p>All of the text, " + b"long enough to be an article. " * 10 + b"</p>"
b"<a class=\"logout\" href=\"/logout\">Log out</a></main></body></html>"
)
TEASER = (
b"<!doctype html><html><head><meta charset=\"utf-8\"><title>Paid</title></head><body><main>"
b"<h1>The paid article</h1><p>The first sentence.</p>"
b"<div class=\"paywall\">Subscribe to read on</div></main></body></html>"
)
HANDLER = {
"host": "127.0.0.1",
"summary": "A test publisher behind a login.",
"session": {
"login": "http://127.0.0.1/login",
"signed_in": {"tag": "a", "class": "logout"},
"signed_out": {"class": "paywall"},
},
}
def _state(value: str = SECRET, expires: float = -1) -> dict:
return {
"cookies": [{
"name": "sid", "value": value, "domain": "127.0.0.1", "path": "/", "expires": expires,
"httpOnly": True, "secure": False, "sameSite": "Lax",
}],
"origins": [],
}
def _paid(server: Site, path: str = "/paid/article") -> str:
"""A page that answers by the cookie it receives."""
def answer(handler) -> None:
handler.wfile.write(FULL if f"sid={SECRET}" in (handler.headers.get("Cookie") or "") else TEASER)
server.routes[path] = (200, {"Content-Type": "text/html; charset=utf-8"}, answer)
return server.url(path)
@pytest.fixture
def site(monkeypatch):
for var in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY", "all_proxy", "ALL_PROXY"):
monkeypatch.delenv(var, raising=False)
server = Site()
yield server
server.stop()
@pytest.fixture
def tree(kb_dir, monkeypatch):
root = kb_dir.parent
(root / "raw").mkdir()
(root / "incoming").mkdir(exist_ok=True)
(root / "sites").mkdir()
return root
def _handler(root: Path, **changes) -> None:
(root / "sites/publisher.json").write_text(json.dumps({**HANDLER, **changes}), encoding="utf-8")
@pytest.fixture
def routed(tree, monkeypatch):
"""127.0.0.1 routed through the session, with a handler that names the login."""
_handler(tree)
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "session"}})
return tree
def _session_env(monkeypatch, state: dict) -> None:
monkeypatch.setenv(config.ENV_SECRET_PREFIX + web_session.secret_name("127.0.0.1").upper(), json.dumps(state))
def _fetch(url, name=None, scope=False):
return raw_fetch_command(url=url, html=None, source_url=None, name=name, scope=scope)
def _exit_code(url, **kwargs) -> int:
with pytest.raises(typer.Exit) as excinfo:
_fetch(url, **kwargs)
return excinfo.value.exit_code
def _header(md: Path) -> dict:
import yaml
content = md.read_text(encoding="utf-8")
return yaml.safe_load(content[4:].partition("\n---\n")[0])
def _snapshot(root: Path) -> dict[str, bytes]:
return {str(p.relative_to(root)): p.read_bytes() for p in sorted(root.rglob("*")) if p.is_file()}
# --- with the user's session ------------------------------------------------
def test_with_a_stored_session_the_full_text_arrives_byte_identical(routed, site, monkeypatch, capsys):
_session_env(monkeypatch, _state())
url = _paid(site)
_fetch(url)
out = capsys.readouterr().out
assert (routed / "incoming/article.html").read_bytes() == FULL
header = _header(routed / "incoming/article.md")
assert header["session"] == "127.0.0.1"
assert header["fetched_by"] == "wikitool raw fetch"
assert "rendered" not in header
assert SECRET not in (routed / "incoming/article.md").read_text(encoding="utf-8")
assert SECRET not in out
assert "raw fetch --html" not in out # the teaser hint is for the anonymous fetch
def test_the_session_is_read_from_the_secrets_file_outside_the_tree(routed, site, monkeypatch):
path = web_session.store("127.0.0.1", "testinstance", _state())
assert path == config.secrets_file("testinstance") # the user's configuration directory
if os.name != "nt":
assert path.stat().st_mode & 0o077 == 0
_fetch(_paid(site))
assert (routed / "incoming/article.html").read_bytes() == FULL
# The test's HOME lies inside its tmp_path; in it, the secrets file is the one place the value is.
holding = [p for p in routed.rglob("*") if p.is_file() and SECRET.encode() in p.read_bytes()]
assert holding == [path]
@pytest.mark.parametrize("state", [None, _state("stale"), _state(expires=1_000_000_000)],
ids=["no-session", "ended-by-the-website", "expired-cookie"])
def test_without_a_valid_session_exit_42_names_raw_login_and_keeps_no_teaser(routed, site, monkeypatch, capsys, state):
if state is not None:
_session_env(monkeypatch, state)
url = _paid(site)
before = _snapshot(routed / "incoming")
assert _exit_code(url) == 42
out = capsys.readouterr().out
assert "tools/wikitool raw login 127.0.0.1" in out
assert _snapshot(routed / "incoming") == before
if state is None:
assert site.requests == [] # nothing to send, so nothing is asked
def test_a_session_route_without_a_login_in_any_handler_is_refused_before_fetching(tree, site, monkeypatch, capsys):
(tree / "sites/publisher.json").write_text(
json.dumps({k: v for k, v in HANDLER.items() if k != "session"}), encoding="utf-8")
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "session"}})
assert _exit_code(_paid(site)) == 1
assert "names a login page" in capsys.readouterr().out
assert site.requests == []
def test_scope_sends_the_session_with_every_page_and_names_it(routed, site, monkeypatch):
_handler(routed, path="/paid/", subpages={"links": {"tag": "a", "inside": "td"}})
write_settings(routed, monkeypatch, sites={"pause_seconds": 0})
_session_env(monkeypatch, _state())
index = (b"<html><body><main><h1>Index</h1><table><tr><td><a href=\"one\">One</a></td></tr></table>"
b"<a class=\"logout\" href=\"/logout\">Log out</a></main></body></html>")
def answer(handler) -> None:
handler.wfile.write(index if f"sid={SECRET}" in (handler.headers.get("Cookie") or "") else TEASER)
site.routes["/paid/"] = (200, {"Content-Type": "text/html"}, answer)
_paid(site, "/paid/one")
_fetch(site.url("/paid/"), name="bundle", scope=True)
bundle = routed / "incoming/bundle"
assert (bundle / "one.html").read_bytes() == FULL
assert _header(bundle / "one.md")["session"] == "127.0.0.1"
assert json.loads((bundle / "_scope.json").read_text(encoding="utf-8"))["session"] == "127.0.0.1"
# --- rendered by a browser --------------------------------------------------
def _fake_render(calls: list):
def render(url, user_agent, state=None, **kwargs):
calls.append((url, state))
return web_capture.Response(url, url, 200, "text/html; charset=utf-8", FULL)
return render
def test_a_handler_declaring_render_gives_a_rendered_header(tree, site, monkeypatch):
_handler(tree, render=True)
calls: list = []
monkeypatch.setattr(browser, "render", _fake_render(calls))
url = site.html("/page", ARTICLE)
_fetch(url)
assert calls == [(url, None)]
assert site.requests == [] # the browser fetched it, not the built-in fetch
assert _header(tree / "incoming/page.md")["rendered"] is True
def test_render_with_a_session_hands_the_browser_the_session(routed, site, monkeypatch):
_handler(routed, render=True)
_session_env(monkeypatch, _state())
calls: list = []
monkeypatch.setattr(browser, "render", _fake_render(calls))
_fetch(site.url("/paid/article"))
assert calls[0][1] == _state()
header = _header(routed / "incoming/article.md")
assert header["rendered"] is True and header["session"] == "127.0.0.1"
def test_without_the_declaration_nothing_is_ever_rendered(routed, site, monkeypatch):
_session_env(monkeypatch, _state())
def refuse(*args, **kwargs):
raise AssertionError("rendered without a declaration")
monkeypatch.setattr(browser, "render", refuse)
_fetch(_paid(site))
assert "rendered" not in _header(routed / "incoming/article.md")
def test_render_without_the_browser_component_exits_42_with_the_preflight_option(tree, site, monkeypatch, capsys):
_handler(tree, render=True)
monkeypatch.setattr(browser, "missing", lambda: "the Python package `playwright` is not installed in the venv")
before = _snapshot(tree / "incoming")
assert _exit_code(site.html("/page", ARTICLE)) == 42
assert "tools/preflight.sh --with browser" in capsys.readouterr().out
assert _snapshot(tree / "incoming") == before
# --- an external fetcher ----------------------------------------------------
def _fetcher(tmp_path: Path, body: bytes = FULL, code: int = 0, stderr: str = "") -> list[str]:
script = tmp_path / "fetcher.py"
script.write_text(
"import sys\n"
f"sys.stdout.buffer.write({body!r})\n"
f"sys.stderr.write({stderr!r})\n"
f"open({str(tmp_path / 'fetcher.args')!r}, 'w').write(' '.join(sys.argv[1:]))\n"
f"sys.exit({code})\n",
encoding="utf-8",
)
return [sys.executable, str(script), "--html"]
def test_an_external_fetcher_writes_its_stdout_byte_identical(tree, site, monkeypatch, tmp_path):
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "homelab"},
"fetchers": {"homelab": _fetcher(tmp_path)}})
url = site.url("/elsewhere/post")
_fetch(url)
assert site.requests == [] # the built-in fetch never ran
assert (tmp_path / "fetcher.args").read_text(encoding="utf-8") == f"--html {url}"
assert (tree / "incoming/post.html").read_bytes() == FULL
header = _header(tree / "incoming/post.md")
assert header["fetched_by"] == "wikitool raw fetch (fetcher: homelab)"
assert "http_status" not in header
def test_a_failing_fetcher_exits_1_with_its_stderr_and_no_fallback(tree, site, monkeypatch, tmp_path, capsys):
write_settings(tree, monkeypatch, fetch={
"routes": {"127.0.0.1": "homelab"},
"fetchers": {"homelab": _fetcher(tmp_path, code=3, stderr="upstream says 503")},
})
url = site.html("/post", ARTICLE)
before = _snapshot(tree / "incoming")
assert _exit_code(url) == 1
out = capsys.readouterr().out
assert "exit 3" in out and "upstream says 503" in out
assert _snapshot(tree / "incoming") == before
assert site.requests == []
def test_a_route_to_a_fetcher_this_checkout_lacks_is_refused(tree, site, monkeypatch, capsys):
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "homelab"}})
assert _exit_code(site.html("/post", ARTICLE)) == 1
assert "fetch.fetchers.homelab" in capsys.readouterr().out
assert site.requests == []
def test_a_fetcher_command_in_the_committed_file_is_refused(tree, site, monkeypatch, capsys, tmp_path):
(tree / config.INSTANCE_SETTINGS_FILENAME).write_text(json.dumps({
"schema": 1, "instance_id": "testinstance",
"fetch": {"routes": {"127.0.0.1": "homelab"}, "fetchers": {"homelab": _fetcher(tmp_path)}},
}), encoding="utf-8")
assert _exit_code(site.html("/post", ARTICLE)) == 1
out = capsys.readouterr().out
assert "fetch.fetchers" in out and "checkout" in out
assert site.requests == []
assert config_shapes.FETCH.field("fetchers").names_command
with pytest.raises(ValueError):
config_shapes.Field("fetchers", "object", "", tier=config_shapes.INSTANCE, names_command=True)
# --- a host no route names --------------------------------------------------
def _without_time(md: Path) -> str:
return "\n".join(line for line in md.read_text(encoding="utf-8").splitlines() if not line.startswith("retrieved:"))
def test_a_host_no_route_names_is_fetched_exactly_as_before(tree, site, monkeypatch, capsys):
url = site.html("/post", ARTICLE)
_fetch(url, name="first")
first_out = capsys.readouterr().out
write_settings(tree, monkeypatch, fetch={"routes": {"other.example.org": "session"}})
_fetch(url, name="later")
later_out = capsys.readouterr().out
assert (tree / "incoming/later.html").read_bytes() == (tree / "incoming/first.html").read_bytes()
assert _without_time(tree / "incoming/later.md") == _without_time(tree / "incoming/first.md").replace(
"first.html", "later.html")
assert later_out.replace("later", "first") == first_out
assert [ua for _, ua in site.requests] == [site.requests[0][1]] * 2
# --- the pieces -------------------------------------------------------------
def test_secret_names_are_valid_reference_names():
assert web_session.secret_name("www.heise.de") == "session_www_heise_de"
assert web_session.secret_name("a--b.example") == "session_a_b_example"
def test_expired_cookies_are_not_sent(site):
jar = web_session.cookie_jar(_state(expires=(datetime.datetime.now() - datetime.timedelta(days=1)).timestamp()))
url = _paid(site)
response = web_capture.fetch(url, "test", cookies=jar)
assert response.body == TEASER
def test_a_route_names_a_session_label_only_for_the_session():
assert fetch_routes.Route("h", fetch_routes.SESSION).session_label == "h"
assert fetch_routes.Route("h", fetch_routes.FETCHER, fetcher="x").session_label == ""
assert fetch_routes.Route("h", fetch_routes.FETCHER, fetcher="x").fetched_by.endswith("(fetcher: x)")
tools/chemenu/tests/test_raw_login.py
+124
View File
@@ -0,0 +1,124 @@
"""`wikitool raw login <host>` - the user signs in once, and the session is stored
as a secret outside the working tree (Gitea #169).
The browser window itself needs a human and the optional Playwright component,
so `browser.login` and `browser.missing` are stood in for: what is under test is
the order of the checks, where the session lands, and that nothing prints it.
"""
from __future__ import annotations
import json
import os
import pytest
import typer
from chemenu import browser, config, web_session
from chemenu.commands.raw.fetch import raw_fetch_command
from chemenu.commands.raw.login import raw_login_command
from chemenu.tests.conftest import write_settings
from chemenu.tests.test_raw_fetch_routes import FULL, HANDLER, SECRET, _paid, _state, site, tree # noqa: F401
@pytest.fixture
def routed(tree, monkeypatch): # noqa: F811
(tree / "sites/publisher.json").write_text(json.dumps(HANDLER), encoding="utf-8")
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "session"}})
return tree
def _login(host: str = "127.0.0.1") -> int:
try:
raw_login_command(host=host)
except typer.Exit as exc:
return exc.exit_code
return 0
def test_login_stores_the_session_outside_the_tree_and_prints_none_of_it(routed, site, monkeypatch, capsys): # noqa: F811
opened = []
monkeypatch.setattr(browser, "login", lambda url: opened.append(url) or _state())
assert _login() == 0
out = capsys.readouterr()
assert opened == ["http://127.0.0.1/login"]
path = config.secrets_file("testinstance")
stored = json.loads(json.loads(path.read_text(encoding="utf-8"))["session_127_0_0_1"])
assert stored == _state()
if os.name != "nt":
assert path.stat().st_mode & 0o077 == 0
assert SECRET not in out.out + out.err
assert "session_127_0_0_1" in out.out
# ... and the next fetch uses it.
raw_fetch_command(url=_paid(site), html=None, source_url=None, name=None, scope=False)
assert (routed / "incoming/article.html").read_bytes() == FULL
def test_login_keeps_the_other_secrets(routed, monkeypatch):
path = config.secrets_file("testinstance")
path.parent.mkdir(parents=True)
path.write_text(json.dumps({"caldav_app_password": "other"}), encoding="utf-8")
path.chmod(0o600)
monkeypatch.setattr(browser, "login", lambda url: _state())
assert _login() == 0
assert json.loads(path.read_text(encoding="utf-8"))["caldav_app_password"] == "other"
def test_login_takes_a_url_for_its_host(routed, monkeypatch):
monkeypatch.setattr(browser, "login", lambda url: _state())
assert _login("http://127.0.0.1/some/article") == 0
assert web_session.load("127.0.0.1", "testinstance") == _state()
def test_login_without_the_browser_component_exits_42_with_the_preflight_option(routed, monkeypatch, capsys):
monkeypatch.setattr(browser, "missing", lambda: "the Python package `playwright` is not installed in the venv")
assert _login() == 42
out = capsys.readouterr().out
assert "tools/preflight.sh --with browser" in out
assert not config.secrets_file("testinstance").exists()
def test_login_for_a_host_not_routed_through_the_session_is_refused(tree, monkeypatch, capsys): # noqa: F811
(tree / "sites/publisher.json").write_text(json.dumps(HANDLER), encoding="utf-8")
monkeypatch.setattr(browser, "login", lambda url: pytest.fail("the browser was opened"))
assert _login() == 1
assert "config set fetch.routes" in capsys.readouterr().out
def test_login_for_a_host_whose_handler_names_no_login_page_is_refused(tree, monkeypatch, capsys): # noqa: F811
write_settings(tree, monkeypatch, fetch={"routes": {"127.0.0.1": "session"}})
monkeypatch.setattr(browser, "login", lambda url: pytest.fail("the browser was opened"))
assert _login() == 1
assert "names a login page" in capsys.readouterr().out
def test_login_warns_when_a_variable_answers_before_the_file(routed, monkeypatch, capsys):
monkeypatch.setenv("WIKITOOL_SECRET_SESSION_127_0_0_1", json.dumps(_state("older")))
monkeypatch.setattr(browser, "login", lambda url: _state())
assert _login() == 0
assert "WARN WIKITOOL_SECRET_SESSION_127_0_0_1 is set" in capsys.readouterr().out
def test_the_component_is_reported_missing_without_playwright(monkeypatch):
monkeypatch.setattr(browser.importlib.util, "find_spec", lambda name: None)
assert "playwright" in browser.missing()
with pytest.raises(browser.BrowserMissing, match="--with browser"):
browser.require("`raw login`")
def test_config_set_writes_a_route_and_leaves_a_fetcher_to_the_user(tree): # noqa: F811
from typer.testing import CliRunner
from chemenu.cli import app
runner = CliRunner()
result = runner.invoke(app, ["config", "set", "fetch.routes", '{"www.example.org": "session"}'])
assert result.exit_code == 0, result.output
data = json.loads((tree / config.INSTANCE_SETTINGS_FILENAME).read_text(encoding="utf-8"))
assert data["fetch"] == {"routes": {"www.example.org": "session"}} and data["instance_id"]
result = runner.invoke(app, ["config", "set", "fetch.fetchers", '{"homelab": ["/bin/true"]}'])
assert result.exit_code == 42
assert not (tree / config.LOCAL_SETTINGS_FILENAME).exists()
result = runner.invoke(app, ["config", "set", "fetch.routes", '{"www.example.org": "Session!"}'])
assert result.exit_code == 1
tools/chemenu/tests/test_site_files.py
+18
View File
@@ -190,6 +190,10 @@ BROKEN = {
"a placeholder without its field": {**EXAMPLE, "checks": [{"text": "see {count}"}]},
"a parent that does not exist": {**EXAMPLE, "parent": "nobody"},
"a parent for another host": {**EXAMPLE, "parent": "learn-microsoft"},
"a session without a sign of being signed in": {**EXAMPLE, "session": {"login": "https://docs.example.org/login"}},
"a session whose login page is not http(s)": {
**EXAMPLE, "session": {"login": "file:///etc/passwd", "signed_in": {"class": "logout"}}},
"a render that is not a boolean": {**EXAMPLE, "render": "yes"},
}
@@ -273,6 +277,20 @@ def test_a_symlink_is_not_followed(tree):
assert doctor.check_site_handlers().status == "FAIL"
def test_a_session_and_render_are_read_and_inherited(tree):
_handler(tree, "example", {**EXAMPLE, "render": True, "session": {
"login": "https://docs.example.org/login", "signed_out": {"class": "paywall"}}})
_handler(tree, "example-guide", {"host": "docs.example.org", "summary": "The guide.", "path": "/guide/",
"parent": "example"})
instance, broken = sites.read_instance(tree)
assert broken == ()
child = next(s for s in instance if s.name == "example-guide")
assert child.render and child.session.login == "https://docs.example.org/login"
assert child.session.is_signed_in("<p>article</p>")
assert not child.session.is_signed_in('<div class="paywall">subscribe</div>')
assert sites.login_site("docs.example.org", instance=instance).name == "example"
def test_doctor_is_quiet_without_a_sites_directory(tree):
assert doctor.check_site_handlers().status == "OK"
tools/chemenu/web_capture.py
+23 -6
View File
@@ -20,6 +20,7 @@ from __future__ import annotations
import codecs
import datetime
import email.message
import http.cookiejar
import json
import mimetypes
import re
@@ -109,11 +110,14 @@ def fetch(
user_agent: str,
timeout: float = TIMEOUT_SECONDS,
max_bytes: int = MAX_BYTES,
cookies: Optional[http.cookiejar.CookieJar] = None,
) -> Response:
"""GET `url` and return the body exactly as received.
No cookies (the opener carries no cookie processor) and no content
decoding - urllib asks for `identity`, so the bytes are the page, not a
No cookies unless `cookies` is given - the user's session for a website
behind a login (`chemenu.web_session`), sent by the jar's own domain, path,
`secure` and expiry rules, on every redirect hop too - and no content
decoding: urllib asks for `identity`, so the bytes are the page, not a
compressed transfer of it. `timeout` bounds the whole transfer, not only a
single read, so a server that trickles bytes cannot hold the call open.
Raises `ValidationError` for a URL outside `http`/`https` (also on a
@@ -121,7 +125,10 @@ def fetch(
error status, an unreachable host, a timeout, a body over `max_bytes`.
"""
check_url(url)
opener = urllib.request.build_opener(_HttpOnlyRedirects)
handlers: list = [_HttpOnlyRedirects]
if cookies is not None:
handlers.append(urllib.request.HTTPCookieProcessor(cookies))
opener = urllib.request.build_opener(*handlers)
request = urllib.request.Request(url, headers={"User-Agent": user_agent, "Accept": "*/*"})
deadline = time.monotonic() + timeout
try:
@@ -712,6 +719,9 @@ def derive_document(
drop: "tuple[Rule, ...]" = (),
site: str = "",
requested_url: str = "",
fetched_by: str = "wikitool raw fetch",
session: str = "",
rendered: bool = False,
) -> Derivation:
"""The `.md` beside a captured `.html`: the fixed header, then the derived
text. With `response` it describes a fetch; without, a page a human saved
@@ -719,22 +729,29 @@ def derive_document(
`drop`, `site` and `requested_url` come from a site handler
(`chemenu.sites`): its rules for the text, its name, and the URL the user
typed when the page was fetched under another one."""
typed when the page was fetched under another one. `fetched_by`, `session`
and `rendered` come from the route the fetch took (`chemenu.fetch_routes`):
an external fetcher names itself, a fetch with the user's session names the
host whose session it was - never its content - and a page drawn by a
browser says so. A response without an HTTP status (an external fetcher's)
leaves `http_status` out rather than inventing one."""
decoded = decode(body, response.content_type if response else None)
document_title = title_of(parse(decoded.text))
text = derive_text(decoded.text, base_url, drop)
charset = f"{decoded.charset} (from: {decoded.source})"
if response is not None:
fields = [
("fetched_by", "wikitool raw fetch"),
("fetched_by", fetched_by),
("url", response.url),
*([("requested_url", requested_url)] if requested_url else []),
("final_url", response.final_url),
("retrieved", derived_at or utc_now()),
("http_status", str(response.status)),
*([("http_status", str(response.status))] if response.status else []),
("content_type", response.content_type),
("charset", charset),
*([("site", site)] if site else []),
*([("session", session)] if session else []),
*([("rendered", "true")] if rendered else []),
("title", document_title),
("derived_from", html_name),
]
tools/chemenu/web_session.py
+115
View File
@@ -0,0 +1,115 @@
"""The user's own session for a website behind a login, kept as a secret.
`raw login <host>` lets the user sign in in a browser window and hands the
resulting browser state - cookies and storage, in Playwright's `storage_state`
form - to `store`. It lands in the secrets file of the instance
(`config.secrets_file`), outside every working tree, under a name derived from
the host (`secret_name`). `raw fetch` reads it back with `load` and sends its
cookies through `cookie_jar`, with the built-in fetch and no browser.
The session is never written to 1Password: it changes on every sign-in, and the
stack only ever reads from there. `WIKITOOL_SECRET_<NAME>` stands in for the
file as it does for any named secret, which is how a container gets a session
without a secrets file.
Nothing here prints a value. An error names the host, the secret's name and the
file, never what is in it.
"""
from __future__ import annotations
import http.cookiejar
import json
import os
import re
from pathlib import Path
from typing import Any, Optional
from chemenu import config, settings
from chemenu.errors import ValidationError
PREFIX = "session"
_NON_ALNUM = re.compile(r"[^a-z0-9]+")
def secret_name(host: str) -> str:
"""The name the session of `host` is stored under: `session_` and the host
with every run of other characters as one `_` - `www.heise.de` becomes
`session_www_heise_de`, readable from `WIKITOOL_SECRET_SESSION_WWW_HEISE_DE`."""
parts = [part for part in _NON_ALNUM.split(host.lower()) if part]
if not parts:
raise ValidationError(f"{host!r} is not a host name")
return "_".join([PREFIX, *parts])
def _env_var(host: str) -> str:
return config.ENV_SECRET_PREFIX + secret_name(host).upper()
def _parse(raw: str, where: str) -> dict:
try:
state = json.loads(raw)
except ValueError:
raise ValidationError(f"the session in {where} is not a browser state (not JSON) - sign in again") from None
if not isinstance(state, dict) or not isinstance(state.get("cookies", []), list):
raise ValidationError(f"the session in {where} is not a browser state - sign in again")
return state
def load(host: str, instance_id: Optional[str]) -> Optional[dict]:
"""The stored browser state for `host`, or None when there is none.
Raises `ValidationError` for one that cannot be read - a secrets file
others can read, or a value that is not a browser state."""
variable = _env_var(host)
raw = os.environ.get(variable, "")
if raw:
return _parse(raw, variable)
if not instance_id:
return None
path = config.secrets_file(instance_id)
raw = (settings.read_secrets_file(path) or {}).get(secret_name(host), "")
if not raw:
return None
return _parse(raw, f"{path} ({secret_name(host)})")
def store(host: str, instance_id: str, state: dict) -> Path:
"""Write the browser state for `host` into the secrets file; returns its path."""
return settings.write_secret(instance_id, secret_name(host), json.dumps(state, separators=(",", ":")))
def overridden(host: str) -> Optional[str]:
"""The variable that answers for `host` before the secrets file, when it is set."""
variable = _env_var(host)
return variable if os.environ.get(variable) else None
def _cookie(entry: Any) -> Optional[http.cookiejar.Cookie]:
if not isinstance(entry, dict):
return None
name, value, domain = entry.get("name"), entry.get("value"), entry.get("domain")
if not (isinstance(name, str) and isinstance(value, str) and isinstance(domain, str) and name and domain):
return None
expires = entry.get("expires", -1)
lasting = isinstance(expires, (int, float)) and not isinstance(expires, bool) and expires >= 0
dotted = domain.startswith(".")
return http.cookiejar.Cookie(
version=0, name=name, value=value, port=None, port_specified=False,
domain=domain, domain_specified=dotted, domain_initial_dot=dotted,
path=str(entry.get("path") or "/"), path_specified=True,
secure=bool(entry.get("secure", False)),
expires=int(expires) if lasting else None, discard=not lasting,
comment=None, comment_url=None,
rest={"HttpOnly": ""} if entry.get("httpOnly") else {},
)
def cookie_jar(state: dict) -> http.cookiejar.CookieJar:
"""A jar holding the cookies of `state`. Which of them a request carries
is the jar's decision - domain, path, `secure` and expiry - so an expired
cookie is simply not sent, and the page shows whether that mattered."""
jar = http.cookiejar.CookieJar()
for entry in state.get("cookies", []):
cookie = _cookie(entry)
if cookie is not None:
jar.set_cookie(cookie)
return jar
tools/preflight.ps1
+84 -4
View File
@@ -4,6 +4,12 @@
#
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --set rg=C:\Tools\rg.exe
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser
#
# --with browser installs tools/requirements-browser.txt into tools/.venv and the
# Chromium that Playwright drives into tools/.venv/ms-playwright - only on request,
# because it is a large download that only `raw login` and pages a site handler
# declares `render` need. Once installed, every later run keeps it up to date.
#
# As the release asset `preflight.ps1` - a copy with no tools/prerequisites.txt next
# to it - the script is the first install step instead: it downloads the stack
@@ -57,6 +63,9 @@ $ToolsFile = Join-Path $Root '.wikitool-tools.json'
$Venv = Join-Path $Dir '.venv'
$Requirements = Join-Path $Dir 'requirements.txt'
$Stamp = Join-Path $Venv '.chemenu-requirements.sha256'
$BrowserRequirements = Join-Path $Dir 'requirements-browser.txt'
$BrowserStamp = Join-Path $Venv '.chemenu-browser.sha256'
$Browsers = Join-Path $Venv 'ms-playwright'
$Self = 'pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1'
# Filled in by the release build, in the copy attached to the release; empty in a tree.
@@ -80,11 +89,30 @@ function Write-ErrorLine {
$Sets = @{}
$Into = ''
$Archive = ''
$With = ''
$index = 0
while ($index -lt $args.Count) {
$arg = [string]$args[$index]
$given = $null
if ($arg -eq '--set') {
if ($arg -eq '--with' -or $arg.StartsWith('--with=')) {
if ($arg -eq '--with') {
if ($index + 1 -ge $args.Count -or -not [string]$args[$index + 1]) {
Write-ErrorLine 'preflight: --with needs a component name (known: browser)'
exit 1
}
$index++
$component = [string]$args[$index]
} else {
$component = $arg.Substring(7)
}
if ($component -ne 'browser') {
Write-ErrorLine "preflight: --with names '$component', which is not an optional component (known: browser)"
exit 1
}
$With = $component
$index++
continue
} elseif ($arg -eq '--set') {
if ($index + 1 -ge $args.Count) {
Write-ErrorLine 'preflight: --set needs <tool>=<path>'
exit 1
@@ -107,12 +135,13 @@ while ($index -lt $args.Count) {
$index++
continue
} elseif ($arg -eq '-h' -or $arg -eq '--help') {
Write-Line "usage: $Self [--set <tool>=<path>]..."
Write-Line ' preflight.ps1 [--into <path>] [--archive <tarball>] [--set <tool>=<path>]...'
Write-Line "usage: $Self [--set <tool>=<path>]... [--with browser]"
Write-Line ' preflight.ps1 [--into <path>] [--archive <tarball>] [--set <tool>=<path>]... [--with browser]'
Write-Line ''
Write-Line 'Checks the tools this stack needs (tools/prerequisites.txt), records their paths'
Write-Line 'in .wikitool-tools.json and sets up tools/.venv. Exit 0 means ready; exit 42'
Write-Line 'means the user has to act - the output says how.'
Write-Line 'means the user has to act - the output says how. --with browser also installs'
Write-Line 'the optional browser component (Playwright and its Chromium) into tools/.venv.'
Write-Line ''
Write-Line 'The second form is the release asset: it downloads the release (or takes'
Write-Line '--archive), checks its sha256, unpacks it into its own folder or --into (empty,'
@@ -492,6 +521,9 @@ function Invoke-AssetMode {
foreach ($name in $Sets.Keys) {
$passed += "--set=$name=$($Sets[$name])"
}
if ($With) {
$passed += "--with=$With"
}
& ([Environment]::ProcessPath) -NoProfile -ExecutionPolicy Bypass -File $treeScript @passed
exit $LASTEXITCODE
}
@@ -1068,6 +1100,54 @@ if ($want -ne $have) {
Write-Line ('OK venv - {0}' -f $Venv)
Write-ToolsFile $true
# --- optional components: installed on request, then kept up to date ---------------
# After .wikitool-tools.json is complete on purpose: a component that fails to install
# stops this run with exit 42, but leaves every wikitool command working.
function Test-Browser {
if (-not (Test-Path -LiteralPath $Browsers -PathType Container)) {
return $false
}
return $null -ne (Get-ChildItem -LiteralPath $Browsers -Filter 'chromium*' -ErrorAction SilentlyContinue | Select-Object -First 1)
}
if ($With -eq 'browser' -or (Test-Path -LiteralPath $BrowserStamp -PathType Leaf)) {
if (-not (Test-Path -LiteralPath $BrowserRequirements -PathType Leaf)) {
Write-ErrorLine "preflight: $BrowserRequirements is missing - the stack tree next to this script is incomplete."
exit 1
}
$want = (Get-FileHash -LiteralPath $BrowserRequirements -Algorithm SHA256).Hash.ToLowerInvariant()
$have = ''
if (Test-Path -LiteralPath $BrowserStamp -PathType Leaf) {
$have = (Get-Content -Raw -LiteralPath $BrowserStamp).Trim()
}
if ($want -ne $have -or -not (Test-Browser)) {
$installed = Invoke-NativeVerbose -Path $venvPython -Arguments @('-m', 'pip', 'install', '--disable-pip-version-check', '--quiet', '-r', $BrowserRequirements)
if ($installed.Code -ne 0) {
Add-Problem 'The optional browser component could not be installed into tools/.venv.' `
'only raw login and pages a site handler fetches through a browser need it; everything else works without it' `
"Check that this computer can reach the internet (a proxy or a security program can block it; if so, ask whoever looks after this computer), then run: $Self --with browser`npip said:`n$(Get-LastLine $installed.Output)"
Exit-WithGuide
}
$previous = $env:PLAYWRIGHT_BROWSERS_PATH
$env:PLAYWRIGHT_BROWSERS_PATH = $Browsers
try {
$downloaded = Invoke-NativeVerbose -Path $venvPython -Arguments @('-m', 'playwright', 'install', 'chromium')
} finally {
$env:PLAYWRIGHT_BROWSERS_PATH = $previous
}
if ($downloaded.Code -ne 0) {
Add-Problem 'The browser of the optional browser component could not be downloaded.' `
'only raw login and pages a site handler fetches through a browser need it; everything else works without it' `
"Check that this computer can reach the internet, then run: $Self --with browser`nPlaywright said:`n$(Get-LastLine $downloaded.Output)"
Exit-WithGuide
}
[IO.File]::WriteAllText($BrowserStamp, "$want`n", [Text.UTF8Encoding]::new($false))
}
Write-Line ('OK browser - {0}' -f $Browsers)
}
Write-Line ''
Write-Line 'Preflight passed. Tool paths are recorded in .wikitool-tools.json; tools/wikitool is ready.'
exit 0
tools/preflight.sh
+73 -3
View File
@@ -4,6 +4,12 @@
#
# tools/preflight.sh check, record, set up
# tools/preflight.sh --set rg=/opt/rg/rg use a path the user named
# tools/preflight.sh --with browser also install the optional browser component
#
# --with browser installs tools/requirements-browser.txt into tools/.venv and the
# Chromium that Playwright drives into tools/.venv/ms-playwright - only on request,
# because it is a large download that only `raw login` and pages a site handler
# declares `render` need. Once installed, every later run keeps it up to date.
#
# As the release asset `preflight.sh` - a copy with no tools/prerequisites.txt next
# to it - the script is the first install step instead: it downloads the stack
@@ -48,6 +54,9 @@ TOOLS_FILE="$ROOT/.wikitool-tools.json"
VENV="$DIR/.venv"
REQUIREMENTS="$DIR/requirements.txt"
STAMP="$VENV/.chemenu-requirements.sha256"
BROWSER_REQUIREMENTS="$DIR/requirements-browser.txt"
BROWSER_STAMP="$VENV/.chemenu-browser.sha256"
BROWSERS="$VENV/ms-playwright"
SELF="tools/preflight.sh"
# Filled in by the release build, in the copy attached to the release; empty in a tree.
@@ -58,12 +67,13 @@ PYPROBE='import sys; print("%d.%d" % sys.version_info[:2]); print(sys.executable
usage() {
cat <<'EOF'
usage: tools/preflight.sh [--set <tool>=<path>]...
preflight.sh [--into <path>] [--archive <tarball>] [--set <tool>=<path>]...
usage: tools/preflight.sh [--set <tool>=<path>]... [--with browser]
preflight.sh [--into <path>] [--archive <tarball>] [--set <tool>=<path>]... [--with browser]
Checks the tools this stack needs (tools/prerequisites.txt), records their paths
in .wikitool-tools.json and sets up tools/.venv. Exit 0 means ready; exit 42
means the user has to act - the output says how.
means the user has to act - the output says how. --with browser also installs
the optional browser component (Playwright and its Chromium) into tools/.venv.
The second form is the release asset: it downloads the release (or takes
--archive), checks its sha256, unpacks it into its own folder or --into (empty,
@@ -75,8 +85,25 @@ EOF
SETS=""
INTO="" ARCHIVE=""
WITH=""
with_component() { # <name>
case "$1" in
browser) WITH=$1 ;;
*) echo "preflight: --with names '$1', which is not an optional component (known: browser)" >&2
exit 1 ;;
esac
}
while [ $# -gt 0 ]; do
case "$1" in
--with)
if [ $# -lt 2 ] || [ -z "$2" ]; then
echo "preflight: --with needs a component name (known: browser)" >&2
exit 1
fi
with_component "$2"
shift
;;
--with=*) with_component "${1#--with=}" ;;
--into|--archive)
if [ $# -lt 2 ] || [ -z "$2" ]; then
echo "preflight: $1 needs a path" >&2
@@ -426,6 +453,7 @@ ${entry##*/}"
done <<EOT
$SETS
EOT
[ -z "$WITH" ] || set -- "$@" --with "$WITH"
trap - EXIT
cleanup
exec /bin/sh "$TARGET/tools/preflight.sh" "$@"
@@ -842,6 +870,48 @@ fi
echo "OK venv - $(native_path "$VENV")"
write_tools_file true
# --- optional components: installed on request, then kept up to date ---------------
# After .wikitool-tools.json is complete on purpose: a component that fails to install
# stops this run with exit 42, but leaves every wikitool command working.
has_browser() {
for entry in "$BROWSERS"/chromium*; do
[ -e "$entry" ] && return 0
done
return 1
}
if [ "$WITH" = browser ] || [ -f "$BROWSER_STAMP" ]; then
if [ ! -f "$BROWSER_REQUIREMENTS" ]; then
echo "preflight: $BROWSER_REQUIREMENTS is missing - the stack tree next to this script is incomplete." >&2
exit 1
fi
want=$("$VPY" -c 'import hashlib, sys; print(hashlib.sha256(open(sys.argv[1], "rb").read()).hexdigest())' \
"$BROWSER_REQUIREMENTS" 2>/dev/null | tr -d '\r')
have=$(cat "$BROWSER_STAMP" 2>/dev/null) || have=""
if [ -z "$want" ] || [ "$want" != "$have" ] || ! has_browser; then
if ! output=$("$VPY" -m pip install --disable-pip-version-check --quiet -r "$BROWSER_REQUIREMENTS" 2>&1); then
problem "The optional browser component could not be installed into tools/.venv." \
"only raw login and pages a site handler fetches through a browser need it; everything else works without it" \
"Check that this computer can reach the internet (a proxy or a security program can block it; if so, ask whoever looks after this computer), then run: $SELF --with browser
pip said:
$(last_lines "$output")"
stop
fi
if ! output=$(PLAYWRIGHT_BROWSERS_PATH="$BROWSERS" "$VPY" -m playwright install chromium 2>&1); then
problem "The browser of the optional browser component could not be downloaded." \
"only raw login and pages a site handler fetches through a browser need it; everything else works without it" \
"Check that this computer can reach the internet, then run: $SELF --with browser
Playwright said:
$(last_lines "$output")"
stop
fi
printf '%s\n' "$want" > "$BROWSER_STAMP"
fi
echo "OK browser - $(native_path "$BROWSERS")"
fi
echo ""
echo "Preflight passed. Tool paths are recorded in .wikitool-tools.json; tools/wikitool is ready."
exit 0
tools/requirements-browser.txt
+17
View File
@@ -0,0 +1,17 @@
# The optional browser component, deliberately not in requirements.txt.
#
# Only two things need a real browser: `wikitool raw login`, where the user signs
# in to a website in a visible window, and a fetch of a page whose site handler
# declares `render`. Everything else - a fetch with the stored session included -
# runs without it, and an instance that never fetches behind a login should not
# download a browser to do it.
#
# The user installs it - the agent never does - with the preflight option, which
# also downloads the Chromium that Playwright drives into tools/.venv/ms-playwright:
#
# tools/preflight.sh --with browser
# pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser
#
# The origin repository's tests that need it skip without it; its CI does not
# install it.
playwright>=1.47