raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/fetch-pages-behind-a-login.md

@@ -0,0 +1,128 @@
+# Fetch pages behind a login
+
+Let `tools/wikitool raw fetch` fetch the articles of a website you are subscribed to, with your own
+login: you sign in once in a browser window, and the wiki fetches that website's pages with the
+session your sign-in left until it expires. No password is stored, and the agent never sees one.
+The same page covers websites whose pages only appear in a browser, and handing a website to a
+fetch program of your own.
+
+Whether a website's terms allow you to fetch your subscription this way is yours to check; the
+stack does not check it and offers no advice on it.
+
+Without any of this, a page behind a login can still be saved from your browser and handed to
+`raw fetch --html` ([Ingest a source](ingest-a-source.md)) - one page at a time, by hand.
+
+<!-- wikitool:toc -->
+## Contents
+
+- [1. Install the browser component](#1-install-the-browser-component)
+- [2. Describe the website's login](#2-describe-the-websites-login)
+- [3. Route the website through your session](#3-route-the-website-through-your-session)
+- [4. Sign in](#4-sign-in)
+- [5. Ingest as usual](#5-ingest-as-usual)
+- [Pages that only appear in a browser](#pages-that-only-appear-in-a-browser)
+- [A fetch program of your own](#a-fetch-program-of-your-own)
+- [Where the session is kept](#where-the-session-is-kept)
+<!-- /wikitool:toc -->
+
+## 1. Install the browser component
+
+Signing in needs a real browser window, so it needs the optional browser component: a Python
+package and a Chromium of its own, installed into the instance's `tools/.venv`. You install it -
+the agent never does:
+
+```bash
+tools/preflight.sh --with browser
+```
+
+In PowerShell 7: `pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`.
+Later preflight runs keep it up to date. `tools/wikitool doctor` reports it on its `browser` line.
+
+## 2. Describe the website's login
+
+The website needs a site handler that says where you sign in and how a page shows that it arrived
+signed in. Add a `session` block to the website's handler in `sites/`, or write a handler just for
+it ([Write a site handler](write-a-site-handler.md)):
+
+```json
+{
+  "host": "news.example.com",
+  "summary": "A subscription news site: articles need the subscriber's session.",
+  "session": {
+    "login": "https://news.example.com/login",
+    "signed_in": {"tag": "a", "class": "logout"},
+    "signed_out": {"class": "paywall-teaser"}
+  }
+}
+```
+
+`signed_in` is something only a signed-in page carries - a logout link, an account menu.
+`signed_out` is something only a page that was not carries - the teaser box that asks you to
+subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element
+that tells them apart. [sites/CONTRACT.md](../../sites/CONTRACT.md) has the exact format.
+
+## 3. Route the website through your session
+
+The handler alone changes nothing. Tell the agent:
+
+> Route news.example.com through my session.
+
+It runs `tools/wikitool config set fetch.routes '{"news.example.com": "session"}'`, keeping any
+host already listed. The setting is committed with the instance, so every clone fetches the
+website the same way - each with its own user's session.
+
+## 4. Sign in
+
+Run this yourself - it opens a browser window and waits for you:
+
+```bash
+tools/wikitool raw login news.example.com
+```
+
+Sign in as you always do, captcha and second factor included, then close the window. The session
+is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.
+
+## 5. Ingest as usual
+
+Give the agent the article's URL as for any web page ([Ingest a source](ingest-a-source.md)).
+The capture's header names the website whose session was used (`session: news.example.com`),
+never the session itself.
+
+When your session has expired, or the page came back as a teaser, the agent stops: `raw fetch`
+writes nothing and asks you to run `raw login` again. Nothing is kept from a teaser in silence.
+
+## Pages that only appear in a browser
+
+Some websites send an empty frame and draw the text with JavaScript; `raw fetch` then captures
+almost nothing. Add `"render": true` to the website's handler: its pages are then fetched through
+the browser component, with your session where the website has one, and the header says
+`rendered: true`. The `.html` is then the page as the browser drew it, not what the server sent -
+which is why it happens only where a handler says so.
+
+## A fetch program of your own
+
+A website can instead go to a program you run - a fetch service on another machine, say. Add it to
+`.wikitool-local.json` by hand, since it names a program to run and `config set` does not write
+such a field:
+
+```json
+{
+  "schema": 1,
+  "fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
+}
+```
+
+and route the host to it: `tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'`.
+The program gets the URL as its last argument and prints the page's HTML; anything else - an exit
+other than 0, no output - stops `raw fetch` with the program's error output, and the built-in
+fetch is not tried instead. The `fetch` section is in
+[docs/reference/configuration.md](../reference/configuration.md#fetch-routes).
+
+## Where the session is kept
+
+In the secrets file in your own configuration directory - `~/.config/chemenu/<instance_id>/secrets.json`,
+`%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows - under `session_<host>`, readable by
+you only, outside every working tree. It never reaches a commit or a `wikitool` output. It does
+not keep an agent with a shell in your account from reading it, any more than any other secret
+there: [configuration-tiers.md](../explanation/configuration-tiers.md) says what keeping secrets
+out of the tree protects against and what it does not.