raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/reference/configuration.md

@@ -29,6 +29,7 @@ something you run.
   - [Profiles](#profiles)
 - [MCP submissions](#mcp-submissions)
 - [Site handlers](#site-handlers)
+- [Fetch routes](#fetch-routes)
 - [Files of earlier versions](#files-of-earlier-versions)
 <!-- /wikitool:toc -->
 
@@ -90,7 +91,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
 `instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
 the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
 [remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
-[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
+[upload](#mcp-submissions), [sites](#site-handlers) and [fetch](#fetch-routes). `config set` names the sections it knows, and `doctor` reports a
 section it does not read as a `WARN`.
 
 **One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -117,8 +118,9 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
 
 **Never in the committed file:** a field that names a program to run or affects a gate - not even
 as a reference. `config set` does not write such a field either (exit 42); the user edits
-`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` in the
-tables below are these: the publish allowlist and the upload limits.
+`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
+`names a command` in the tables below are these: the publish allowlist, the upload limits and the
+external fetchers.
 
 `wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
 `settings-secrets` - and for the references only whether they resolve. `docs verify` holds a
@@ -477,6 +479,60 @@ All three fields are instance-tier: what a website needs cleaned is the same in
 of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
 with exit 1 before it fetches anything.
 
+## Fetch routes
+
+`tools/wikitool raw fetch` fetches every page the same way unless the `fetch` section says
+otherwise for its host: anonymously, with no cookie and no browser. `routes` maps a host to another
+way, and is instance-tier, so every clone fetches the host the same way:
+
+- `"session"` fetches the host's pages with the session the user stored with
+  `tools/wikitool raw login <host>`, through the same built-in fetch. The host needs a site handler
+  with a `session` block - its login page and how a page shows it arrived signed in
+  ([sites/CONTRACT.md](../../sites/CONTRACT.md)). No stored session, or a page without that sign,
+  stops the fetch with exit 42 for the user to sign in again. The session is a secret: it is
+  stored in the secrets file [above](#tiered-settings) under `session_<host>` (every run of other
+  characters as one `_`, so `www.example.org` is `session_www_example_org`), or comes from
+  `WIKITOOL_SECRET_SESSION_<HOST>`, and no settings field refers to it. It is never written to
+  1Password.
+- Any other value names a fetcher of `fetchers`: a program of this checkout, run without a shell
+  with the URL appended. `fetchers` names a program to run, so it lives only in
+  `.wikitool-local.json`, written by hand - `config set` refuses it with exit 42, and a fetcher in
+  `wikitool.json` is an error. A route to a fetcher this checkout does not define stops a fetch of
+  that host with exit 1, so a clone without the program fails loudly instead of fetching
+  anonymously.
+
+The whole procedure, with the browser component it needs for signing in:
+[fetch-pages-behind-a-login.md](../how-to/fetch-pages-behind-a-login.md).
+
+<!-- wikitool:reference-config-fetch -->
+| Field | Tier | Type | Required | Default | Description |
+|-------|------|------|----------|---------|-------------|
+| `routes` | instance | object | no | `{}` | Host (exact, lower case) to route: `"session"` fetches with the user's session - the host needs a site handler that names its login page - and any other value names a fetcher of `fetchers`. A host not listed is fetched as before. |
+| `fetchers` | checkout, names a command | object | no | `{}` | Fetcher name to command: an array of the program and its arguments, run without a shell with the URL appended. Exit 0 and the page's HTML on stdout is a fetch; anything else stops `raw fetch` with exit 1 and the program's stderr. |
+
+- **Absent:** Every page is fetched by the built-in anonymous fetch, as before.
+- **Malformed:** An error when `raw fetch` or `raw login` reads it: the command stops with exit 1 before fetching anything. A route to a fetcher this checkout does not define stops a fetch of that host the same way.
+
+Example:
+
+```json
+{
+  "fetch": {
+    "routes": {
+      "www.heise.de": "session",
+      "intranet.example.org": "homelab"
+    },
+    "fetchers": {
+      "homelab": [
+        "/usr/local/bin/fetch-page",
+        "--html"
+      ]
+    }
+  }
+}
+```
+<!-- /wikitool:reference-config-fetch -->
+
 ## Files of earlier versions
 
 Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,