raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/reference/commands/raw-material-and-uploads.md

@@ -6,6 +6,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
 ## Contents
 
 - [`raw fetch`](#raw-fetch)
+- [`raw login`](#raw-login)
 - [`raw derive`](#raw-derive)
 - [`raw capture`](#raw-capture)
 - [`raw status`](#raw-status)
@@ -20,7 +21,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
 <!-- wikitool:reference-commands -->
 ## `raw fetch`
 
-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
 
 **Synopsis**
 
@@ -64,7 +65,9 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
-- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
+- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
+- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
+- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
 - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
 - Success prints the `raw accept` line for the written files and the `source_url` for the source page.
 
@@ -83,17 +86,76 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- `1` - The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
+- `1` - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
 - `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
+- `42` - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
+- `42` - The site handler declares `render` and the optional browser component is not installed
 
 **See also**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
 - `sites/CONTRACT.md` - the instance's own handler files
-- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
+- `wikitool raw login` - the user signs in once for a host routed through the session
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
 
+## `raw login`
+
+Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
+
+**Synopsis**
+
+- `wikitool raw login <host>`
+
+**Options**
+
+```text
+Arguments:
+  host  The website's host, such as www.example.org  [required]
+
+Options:
+  -h, --help  Show this message and exit.
+```
+
+**Examples**
+
+```bash
+tools/wikitool raw login www.heise.de
+```
+
+**Behaviour**
+
+- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
+- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
+- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
+- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
+- Prints no part of the session. Signing in again replaces the stored session.
+
+**Properties**
+
+- Changes files: yes
+- Idempotent: yes
+- Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
+- Counts toward the session call budget: yes
+- Can reach the network: yes
+
+**Exit codes**
+
+- `0` - success
+- `1` - The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
+- `1` - The browser could not be started, or the window closed before any state could be read
+- `42` - The optional browser component is not installed
+
+**See also**
+
+- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
+- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
+- `sites/CONTRACT.md` - a handler's `session` block
+- `docs/reference/configuration.md` - the `fetch` section
+
 ## `raw derive`
 
 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.