raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8

Files changed: - CHANGES.md - INSTALL.md - VERSION - docs/how-to/README.md - docs/how-to/fetch-pages-behind-a-login.md - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/preflight.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/browser.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/__init__.py - tools/chemenu/commands/raw/fetch.py - tools/chemenu/commands/raw/login.py - tools/chemenu/config_shapes.py - tools/chemenu/fetch_routes.py - tools/chemenu/reference_doc.py - tools/chemenu/settings.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_preflight.py - tools/chemenu/tests/test_preflight_pwsh.py - tools/chemenu/tests/test_raw_fetch_routes.py - tools/chemenu/tests/test_raw_login.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/web_capture.py - tools/chemenu/web_session.py - tools/preflight.ps1 - tools/preflight.sh - tools/requirements-browser.txt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
2026-10-11 20:50:23 +02:00
1 parent aed846740b
commit 5994b2a88c
@@ -123,7 +123,8 @@ review                read  idempotent     budget:exempt               exit:0,1,
 sources coverage      read  idempotent     budget:counted              exit:0      List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages.
 sources trace         read  idempotent     budget:counted              exit:0,1    Trace provenance in either direction: raw file, or page.
 sources rebuild-index write idempotent     budget:counted              exit:0,1    Regenerate the `kb/provenance.md` reverse index.
-raw fetch             write non-idempotent budget:counted              exit:0,1    Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
+raw fetch             write non-idempotent budget:counted              exit:0,1,42 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
+raw login             write idempotent     budget:counted              exit:0,1,42 Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
 raw derive            write non-idempotent budget:counted              exit:0,1    Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
 raw capture           write non-idempotent budget:counted              exit:0,1    Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
 raw status            read  idempotent     budget:counted              exit:0      Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
@@ -1420,7 +1421,7 @@ Regenerate the `kb/provenance.md` reverse index.

 #### `raw fetch`

-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.

 **SYNOPSIS**

@@ -1450,8 +1451,12 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - 1    The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - 1    A target file already exists in `incoming/`
 - 1    The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- 1    The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
+- 1    The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
 - 1    raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - 1    raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
+- 42   The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
+- 42   The site handler declares `render` and the optional browser component is not installed

 **ON FAILURE**

@@ -1459,8 +1464,12 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once
 - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
 - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
+- The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define -> Nothing was fetched or written. Fix the route with `tools/wikitool config set`, or have the user add the fetcher to `.wikitool-local.json` by hand - it names a program, so `config set` does not write it - then retry once
+- The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB -> Nothing was written, and the built-in fetch was not tried. The message carries the fetcher's stderr - report it to the user; do not fetch the page another way
 - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget -> Nothing was written. Read the message: fetch the page without `--scope`, raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user
 - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
+- The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired) -> Nothing was written. Show the user the output and stop: they sign in with `tools/wikitool raw login <host>`, never you. Retry once they say it is done
+- The site handler declares `render` and the optional browser component is not installed -> Nothing was written. Show the user the output and stop: they install it with `tools/preflight.sh --with browser`. Retry once they say it is done

 **NEVER**

@@ -1478,7 +1487,9 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
+- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
+- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
+- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
 - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
 - Success prints the `raw accept` line for the written files and the `source_url` for the source page.

@@ -1486,10 +1497,64 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a

 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
 - `sites/CONTRACT.md` - the instance's own handler files
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
+- `wikitool raw login` - the user signs in once for a host routed through the session
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts

+#### `raw login`
+
+Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
+
+**SYNOPSIS**
+
+- `wikitool raw login <host>`
+
+**PROPERTIES**
+
+- effect: write
+- idempotent: yes
+- atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
+- budget: counted
+- network: yes
+
+**EXAMPLES**
+
+- `tools/wikitool raw login www.heise.de`
+
+**EXIT STATUS**
+
+- 0    success
+- 1    The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
+- 1    The browser could not be started, or the window closed before any state could be read
+- 42   The optional browser component is not installed
+
+**ON FAILURE**
+
+- The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed -> Nothing was stored. Route the host (`tools/wikitool config set fetch.routes '{"<host>": "session"}'`), add a `session` block to its handler (`sites/CONTRACT.md`), or fix the section, then retry once
+- The browser could not be started, or the window closed before any state could be read -> Nothing was stored. Read the browser's message - on Linux it may name system libraries the browser needs, which whoever administers the machine installs - then retry once
+- The optional browser component is not installed -> Nothing was stored. The user installs it with `tools/preflight.sh --with browser`, then runs this command again
+
+**NEVER**
+
+- Never run this command as an agent - it waits for a human at a browser window. When `raw fetch` asks for a sign-in, show the user its output and let them run it.
+- Never read, print or copy the stored session.
+
+**NOTES**
+
+- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
+- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
+- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
+- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
+- Prints no part of the session. Signing in again replaces the stored session.
+
+**SEE ALSO**
+
+- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
+- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
+- `sites/CONTRACT.md` - a handler's `session` block
+- `docs/reference/configuration.md` - the `fetch` section
+
 #### `raw derive`

 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
@@ -3849,6 +3914,7 @@ Check that this instance is correctly configured.
 - Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
 - Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
 - Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
+- Browser component (`browser`): whether the optional Playwright component and its browser are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is missing and the instance needs it: a host `fetch.routes` sends through the session (`raw login`), or a site handler that declares `render`. The user installs it with `tools/preflight.sh --with browser`.
 - Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
 - Read-only and exempt from the Iteration Budget Gate.