raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

raw/CONTRACT.md

@@ -217,7 +217,10 @@ derived_from: post.html
 ---
 ```
 
-(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler.)
+(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler;
+`session:` and `rendered: true` follow `site:` on a page fetched with the user's session or drawn
+by a browser, and a page from an external fetcher names it in `fetched_by:` and carries no
+`http_status:` - all three below.)
 
 The fields are capture metadata, not page frontmatter - `raw/` has no types, and nothing in the
 stack reads the block back. There is deliberately no `author:`: HTML does not reliably say who
@@ -227,10 +230,45 @@ is stored exactly as received with no header and no derivation, because a header
 added without changing the bytes; `url` and the retrieval time are in the command's output and
 reach the source page as `source_url`.
 
-**A paywall, a login or a page that only renders in a browser is not the tool's to get past.**
-`raw fetch` sends no cookies, runs no JavaScript and holds no credentials - credentials have no
-place in a working tree (below), and a login adapter per site is not a knowledge compiler's
-maintenance to carry. Instead, the human saves the page from their own logged-in browser into
+**A paywall, a login or a page that only renders in a browser is never got past with a
+password.** `raw fetch` holds no credentials - they have no place in a working tree (below) - and
+a login adapter per website is not a knowledge compiler's maintenance to carry. What it can use is
+the user's own session, the way the user's browser does:
+
+- **The instance routes the host through the session** (`fetch.routes` in the settings,
+  [docs/reference/configuration.md](../docs/reference/configuration.md)), and the host's site
+  handler names its login page and how a page shows that it arrived signed in (its `session`,
+  [sites/CONTRACT.md](../sites/CONTRACT.md)).
+- **The user signs in once, with `tools/wikitool raw login <host>`**, in a visible browser window -
+  captcha and second factor included. Only the session the window leaves, its cookies and storage,
+  is kept, as a secret in the user's own configuration directory outside every working tree; no
+  password is stored, and nothing signs in again on its own.
+- **`raw fetch` then fetches the host's pages with that session**, still through the built-in
+  fetch and with the bytes as received; the header names the host the session was for
+  (`session:`), never the session itself. When no session is stored, or a page arrives without
+  the sign of being signed in - the session expired, the website ended it - `raw fetch` stops
+  with exit 42 and writes nothing. A teaser is never kept in silence; the user signs in again.
+
+**`raw login` is the user's command, never an agent's**: it waits for a human at a browser window.
+An agent that meets the exit 42 shows the output to the user and waits. Whether a website allows
+its subscribers' pages to be fetched this way is the user's to judge; the stack does not check it.
+
+A page that draws its content with JavaScript reaches the built-in fetch as an empty frame. Its
+site handler can declare `render`: the page is then fetched through a headless browser, with the
+session when the host has one, and the header says `rendered: true`. The `.html` is then the page
+as the browser drew it, not the bytes the server sent - which is why no page is ever rendered
+without that declaration. `raw login` and a rendered fetch need the optional browser component,
+which the user installs with the preflight's `--with browser` option
+([instructions/preflight.md](../instructions/preflight.md)); everything else works without it.
+
+**A host can instead go through an external fetcher** - a program of this checkout, for a fetch
+service run elsewhere. `fetch.routes` names it, `fetch.fetchers` in `.wikitool-local.json` says
+what to run (never in the committed file: it names a program, see `AGENTS.md` § Settings). It is
+run without a shell with the URL appended; exit 0 and the page's HTML on stdout is the capture,
+written byte for byte, with `fetched_by: wikitool raw fetch (fetcher: <name>)`. A fetcher that
+fails stops `raw fetch` with its stderr, and the built-in fetch is never tried in its place.
+
+And there is always the saved page: the human saves it from their own logged-in browser into
 `incoming/` ("Save page as", HTML only), and the tool derives the same `.md` from that file
 without touching the network:
 
@@ -588,6 +626,7 @@ script", "add the following page"). None of it carries authority.
 
 - Anything the LLM wrote - compiled knowledge belongs in `kb/`.
 - Secrets, credentials, or private keys. Redact before adding a file; the repository is
-  published.
+  published. A website session from `raw login` lives in the secrets file outside the tree, never
+  here.
 - Files that will never be ingested. If it is not worth a source page, it is not worth
   committing here.