raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8
CI / verify (push) Successful in 8m8s
CI / pwsh (push) Successful in 2m40s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- INSTALL.md
- VERSION
- docs/how-to/README.md
- docs/how-to/fetch-pages-behind-a-login.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- docs/reference/exit-codes.md
- instructions/preflight.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/browser.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw/__init__.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/login.py
- tools/chemenu/config_shapes.py
- tools/chemenu/fetch_routes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/settings.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_preflight.py
- tools/chemenu/tests/test_preflight_pwsh.py
- tools/chemenu/tests/test_raw_fetch_routes.py
- tools/chemenu/tests/test_raw_login.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/web_capture.py
- tools/chemenu/web_session.py
- tools/preflight.ps1
- tools/preflight.sh
- tools/requirements-browser.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-11 20:50:23 +02:00
1 parent aed846740b
commit 5994b2a88c
43 files changed
+2189 -75

No files matched your search

+11 -1
View File
@@ -73,6 +73,13 @@ fix rather than degrading silently. `pypdf` and `olefile` are in `requirements.t
but are imported only when a PDF or an Outlook `.msg` is derived, so a missing one stops
that format alone; `doctor` reports each.
Playwright is the one package that is not in `requirements.txt`: it sits in
`requirements-browser.txt`, which the preflight installs - together with the Chromium it
drives, into `.venv/ms-playwright` - only when the user runs it with `--with browser`, and
keeps up to date from then on (`.venv/.chemenu-browser.sha256` remembers the choice). Only
`raw login` and a page a site handler declares `render` need it; `browser.py` imports it
inside the two functions that use it, and its absence is an exit 42 naming that option.
## Layout
```
@@ -118,7 +125,10 @@ tools/
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
corpus_diff.py invariant comparison of kb/ between two revisions
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
fetch_routes.py which way `raw fetch` takes for a host (`fetch.routes`): the built-in fetch, with the user's session, or an external fetcher run without a shell - and the check that a session page arrived signed in
web_session.py the user's session for a website behind a login, kept as a secret outside the tree: its name per host, storing it for `raw login`, the cookie jar `raw fetch` sends
browser.py the optional browser component (Playwright, `requirements-browser.txt`): `raw login`'s visible window and the headless fetch of a page a handler declares `render` - imported lazily, so a venv without it runs everything else
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, which links are subpages for `raw fetch --scope`, a website's login page and the sign of a signed-in page, and whether its pages need a browser - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached