raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8
Files changed: - CHANGES.md - INSTALL.md - VERSION - docs/how-to/README.md - docs/how-to/fetch-pages-behind-a-login.md - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/preflight.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/browser.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/__init__.py - tools/chemenu/commands/raw/fetch.py - tools/chemenu/commands/raw/login.py - tools/chemenu/config_shapes.py - tools/chemenu/fetch_routes.py - tools/chemenu/reference_doc.py - tools/chemenu/settings.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_preflight.py - tools/chemenu/tests/test_preflight_pwsh.py - tools/chemenu/tests/test_raw_fetch_routes.py - tools/chemenu/tests/test_raw_login.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/web_capture.py - tools/chemenu/web_session.py - tools/preflight.ps1 - tools/preflight.sh - tools/requirements-browser.txt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
aed846740b
commit
5994b2a88c
43 files changed
+2189
-75
No files matched your search
+11
-1
@@ -73,6 +73,13 @@ fix rather than degrading silently. `pypdf` and `olefile` are in `requirements.t
|
||||
but are imported only when a PDF or an Outlook `.msg` is derived, so a missing one stops
|
||||
that format alone; `doctor` reports each.
|
||||
|
||||
Playwright is the one package that is not in `requirements.txt`: it sits in
|
||||
`requirements-browser.txt`, which the preflight installs - together with the Chromium it
|
||||
drives, into `.venv/ms-playwright` - only when the user runs it with `--with browser`, and
|
||||
keeps up to date from then on (`.venv/.chemenu-browser.sha256` remembers the choice). Only
|
||||
`raw login` and a page a site handler declares `render` need it; `browser.py` imports it
|
||||
inside the two functions that use it, and its absence is an exit 42 naming that option.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
@@ -118,7 +125,10 @@ tools/
|
||||
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
|
||||
corpus_diff.py invariant comparison of kb/ between two revisions
|
||||
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
|
||||
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
|
||||
fetch_routes.py which way `raw fetch` takes for a host (`fetch.routes`): the built-in fetch, with the user's session, or an external fetcher run without a shell - and the check that a session page arrived signed in
|
||||
web_session.py the user's session for a website behind a login, kept as a secret outside the tree: its name per host, storing it for `raw login`, the cookie jar `raw fetch` sends
|
||||
browser.py the optional browser component (Playwright, `requirements-browser.txt`): `raw login`'s visible window and the headless fetch of a page a handler declares `render` - imported lazily, so a venv without it runs everything else
|
||||
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, which links are subpages for `raw fetch --scope`, a website's login page and the sign of a signed-in page, and whether its pages need a browser - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
|
||||
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
|
||||
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
|
||||
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
|
||||
|
||||
Reference in new issue
Block a user