raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8
Files changed: - CHANGES.md - INSTALL.md - VERSION - docs/how-to/README.md - docs/how-to/fetch-pages-behind-a-login.md - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/preflight.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/browser.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/__init__.py - tools/chemenu/commands/raw/fetch.py - tools/chemenu/commands/raw/login.py - tools/chemenu/config_shapes.py - tools/chemenu/fetch_routes.py - tools/chemenu/reference_doc.py - tools/chemenu/settings.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_preflight.py - tools/chemenu/tests/test_preflight_pwsh.py - tools/chemenu/tests/test_raw_fetch_routes.py - tools/chemenu/tests/test_raw_login.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/web_capture.py - tools/chemenu/web_session.py - tools/preflight.ps1 - tools/preflight.sh - tools/requirements-browser.txt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
aed846740b
commit
5994b2a88c
43 files changed
+2189
-75
No files matched your search
@@ -124,6 +124,12 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
`robots.txt` forbids a subpage goes to the user as it is; never fetch the subpages one by one
|
||||
instead.
|
||||
|
||||
**`raw fetch` exited 42 and names `raw login <host>` or `--with browser`?** The host is fetched
|
||||
with the user's own session, and there is none or it has expired - or the page needs the
|
||||
optional browser component. Show the output to the user verbatim and wait; nothing was
|
||||
written. **Never run `raw login` yourself** - it opens a browser window and waits for a human -
|
||||
and never install the component. Once the user says it is done, fetch again.
|
||||
|
||||
**Check that the text is the whole article.** A paywall, a login wall or a page that only
|
||||
renders in a browser yields a teaser, often long enough to look like an article: the text
|
||||
breaks off at "continue reading with...", a subscription offer or a login prompt. Stop there
|
||||
@@ -136,7 +142,9 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
which derives the `.md` from that file without touching the network. It never overwrites, so
|
||||
a teaser's `.md` still in `incoming/` under the same name makes it refuse: remove the teaser's
|
||||
files first - they were never accepted, so nothing refers to them.
|
||||
files first - they were never accepted, so nothing refers to them. For a website the user
|
||||
reads often, mention the lasting way as well: fetching with their own session
|
||||
(`docs/how-to/fetch-pages-behind-a-login.md`) - set up by the user, not by you.
|
||||
|
||||
**Check the size first, on both axes.** *Volume* - how many raw files this ingest covers -
|
||||
and *breadth* - how many entities and concepts this one source would produce or update.
|
||||
|
||||
Reference in new issue
Block a user