raw: fetch behind a login - raw login stores the user's session as a secret outside the tree, fetch.routes sends a host through it or to an external fetcher, a handler's render goes through the optional browser component (preflight --with browser) (#169); 10.0.0-beta.8
Files changed: - CHANGES.md - INSTALL.md - VERSION - docs/how-to/README.md - docs/how-to/fetch-pages-behind-a-login.md - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - docs/reference/exit-codes.md - instructions/preflight.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/browser.py - tools/chemenu/cli_contract.py - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw/__init__.py - tools/chemenu/commands/raw/fetch.py - tools/chemenu/commands/raw/login.py - tools/chemenu/config_shapes.py - tools/chemenu/fetch_routes.py - tools/chemenu/reference_doc.py - tools/chemenu/settings.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_preflight.py - tools/chemenu/tests/test_preflight_pwsh.py - tools/chemenu/tests/test_raw_fetch_routes.py - tools/chemenu/tests/test_raw_login.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/web_capture.py - tools/chemenu/web_session.py - tools/preflight.ps1 - tools/preflight.sh - tools/requirements-browser.txt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
aed846740b
commit
5994b2a88c
43 files changed
+2189
-75
No files matched your search
@@ -11,6 +11,8 @@ Working with the wiki:
|
||||
- [Ingest a source](ingest-a-source.md) - a file, a folder, a web page, or a document submitted
|
||||
through the MCP server
|
||||
- [Write a site handler](write-a-site-handler.md) - teach `raw fetch` how a website serves its pages
|
||||
- [Fetch pages behind a login](fetch-pages-behind-a-login.md) - with your own session, through a
|
||||
browser, or through a fetch program of your own
|
||||
- [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
|
||||
- [Export guidelines to captured repositories](export-guidelines.md)
|
||||
- [Query the wiki](query-the-wiki.md) - asking, and searching yourself
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# Fetch pages behind a login
|
||||
|
||||
Let `tools/wikitool raw fetch` fetch the articles of a website you are subscribed to, with your own
|
||||
login: you sign in once in a browser window, and the wiki fetches that website's pages with the
|
||||
session your sign-in left until it expires. No password is stored, and the agent never sees one.
|
||||
The same page covers websites whose pages only appear in a browser, and handing a website to a
|
||||
fetch program of your own.
|
||||
|
||||
Whether a website's terms allow you to fetch your subscription this way is yours to check; the
|
||||
stack does not check it and offers no advice on it.
|
||||
|
||||
Without any of this, a page behind a login can still be saved from your browser and handed to
|
||||
`raw fetch --html` ([Ingest a source](ingest-a-source.md)) - one page at a time, by hand.
|
||||
|
||||
<!-- wikitool:toc -->
|
||||
## Contents
|
||||
|
||||
- [1. Install the browser component](#1-install-the-browser-component)
|
||||
- [2. Describe the website's login](#2-describe-the-websites-login)
|
||||
- [3. Route the website through your session](#3-route-the-website-through-your-session)
|
||||
- [4. Sign in](#4-sign-in)
|
||||
- [5. Ingest as usual](#5-ingest-as-usual)
|
||||
- [Pages that only appear in a browser](#pages-that-only-appear-in-a-browser)
|
||||
- [A fetch program of your own](#a-fetch-program-of-your-own)
|
||||
- [Where the session is kept](#where-the-session-is-kept)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
## 1. Install the browser component
|
||||
|
||||
Signing in needs a real browser window, so it needs the optional browser component: a Python
|
||||
package and a Chromium of its own, installed into the instance's `tools/.venv`. You install it -
|
||||
the agent never does:
|
||||
|
||||
```bash
|
||||
tools/preflight.sh --with browser
|
||||
```
|
||||
|
||||
In PowerShell 7: `pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`.
|
||||
Later preflight runs keep it up to date. `tools/wikitool doctor` reports it on its `browser` line.
|
||||
|
||||
## 2. Describe the website's login
|
||||
|
||||
The website needs a site handler that says where you sign in and how a page shows that it arrived
|
||||
signed in. Add a `session` block to the website's handler in `sites/`, or write a handler just for
|
||||
it ([Write a site handler](write-a-site-handler.md)):
|
||||
|
||||
```json
|
||||
{
|
||||
"host": "news.example.com",
|
||||
"summary": "A subscription news site: articles need the subscriber's session.",
|
||||
"session": {
|
||||
"login": "https://news.example.com/login",
|
||||
"signed_in": {"tag": "a", "class": "logout"},
|
||||
"signed_out": {"class": "paywall-teaser"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`signed_in` is something only a signed-in page carries - a logout link, an account menu.
|
||||
`signed_out` is something only a page that was not carries - the teaser box that asks you to
|
||||
subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element
|
||||
that tells them apart. [sites/CONTRACT.md](../../sites/CONTRACT.md) has the exact format.
|
||||
|
||||
## 3. Route the website through your session
|
||||
|
||||
The handler alone changes nothing. Tell the agent:
|
||||
|
||||
> Route news.example.com through my session.
|
||||
|
||||
It runs `tools/wikitool config set fetch.routes '{"news.example.com": "session"}'`, keeping any
|
||||
host already listed. The setting is committed with the instance, so every clone fetches the
|
||||
website the same way - each with its own user's session.
|
||||
|
||||
## 4. Sign in
|
||||
|
||||
Run this yourself - it opens a browser window and waits for you:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw login news.example.com
|
||||
```
|
||||
|
||||
Sign in as you always do, captcha and second factor included, then close the window. The session
|
||||
is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.
|
||||
|
||||
## 5. Ingest as usual
|
||||
|
||||
Give the agent the article's URL as for any web page ([Ingest a source](ingest-a-source.md)).
|
||||
The capture's header names the website whose session was used (`session: news.example.com`),
|
||||
never the session itself.
|
||||
|
||||
When your session has expired, or the page came back as a teaser, the agent stops: `raw fetch`
|
||||
writes nothing and asks you to run `raw login` again. Nothing is kept from a teaser in silence.
|
||||
|
||||
## Pages that only appear in a browser
|
||||
|
||||
Some websites send an empty frame and draw the text with JavaScript; `raw fetch` then captures
|
||||
almost nothing. Add `"render": true` to the website's handler: its pages are then fetched through
|
||||
the browser component, with your session where the website has one, and the header says
|
||||
`rendered: true`. The `.html` is then the page as the browser drew it, not what the server sent -
|
||||
which is why it happens only where a handler says so.
|
||||
|
||||
## A fetch program of your own
|
||||
|
||||
A website can instead go to a program you run - a fetch service on another machine, say. Add it to
|
||||
`.wikitool-local.json` by hand, since it names a program to run and `config set` does not write
|
||||
such a field:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": 1,
|
||||
"fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
|
||||
}
|
||||
```
|
||||
|
||||
and route the host to it: `tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'`.
|
||||
The program gets the URL as its last argument and prints the page's HTML; anything else - an exit
|
||||
other than 0, no output - stops `raw fetch` with the program's error output, and the built-in
|
||||
fetch is not tried instead. The `fetch` section is in
|
||||
[docs/reference/configuration.md](../reference/configuration.md#fetch-routes).
|
||||
|
||||
## Where the session is kept
|
||||
|
||||
In the secrets file in your own configuration directory - `~/.config/chemenu/<instance_id>/secrets.json`,
|
||||
`%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows - under `session_<host>`, readable by
|
||||
you only, outside every working tree. It never reaches a commit or a `wikitool` output. It does
|
||||
not keep an agent with a shell in your account from reading it, any more than any other secret
|
||||
there: [configuration-tiers.md](../explanation/configuration-tiers.md) says what keeping secrets
|
||||
out of the tree protects against and what it does not.
|
||||
@@ -47,7 +47,10 @@ pick a directory:
|
||||
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
|
||||
text to derive - the tool warns, and there is no OCR.
|
||||
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
||||
agent derives the same text from that file with `raw fetch --html`.
|
||||
agent derives the same text from that file with `raw fetch --html`. For a website you read often,
|
||||
sign in once instead and let the wiki fetch it with your session:
|
||||
[Fetch pages behind a login](fetch-pages-behind-a-login.md). When that session has expired the
|
||||
agent stops and asks you to run `tools/wikitool raw login <host>` - it never signs in itself.
|
||||
|
||||
Where a file ends up in `raw/` is computed by `tools/wikitool raw accept`: a `YYYY/MM` shard for the
|
||||
day it was accepted. So is whether several files of one source get bundled. You never choose either.
|
||||
|
||||
@@ -79,7 +79,8 @@ Full records: [provenance.md](provenance.md)
|
||||
|
||||
Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
|
||||
|
||||
- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
|
||||
- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
|
||||
- [`raw login`](raw-material-and-uploads.md#raw-login) - Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
|
||||
- [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
|
||||
- [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
- [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
|
||||
|
||||
@@ -44,6 +44,7 @@ tools/wikitool doctor --json
|
||||
- Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
|
||||
- Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
|
||||
- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
|
||||
- Browser component (`browser`): whether the optional Playwright component and its browser are installed in the venv - `OK` either way, never a `FAIL`; a `WARN` only when it is missing and the instance needs it: a host `fetch.routes` sends through the session (`raw login`), or a site handler that declares `render`. The user installs it with `tools/preflight.sh --with browser`.
|
||||
- Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
|
||||
- Read-only and exempt from the Iteration Budget Gate.
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
|
||||
## Contents
|
||||
|
||||
- [`raw fetch`](#raw-fetch)
|
||||
- [`raw login`](#raw-login)
|
||||
- [`raw derive`](#raw-derive)
|
||||
- [`raw capture`](#raw-capture)
|
||||
- [`raw status`](#raw-status)
|
||||
@@ -20,7 +21,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
|
||||
<!-- wikitool:reference-commands -->
|
||||
## `raw fetch`
|
||||
|
||||
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
|
||||
Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
@@ -64,7 +65,9 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
|
||||
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
|
||||
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
|
||||
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
|
||||
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
|
||||
- Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
|
||||
- Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
|
||||
- `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
|
||||
- Success prints the `raw accept` line for the written files and the `source_url` for the source page.
|
||||
|
||||
@@ -83,17 +86,76 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
|
||||
- `1` - A target file already exists in `incoming/`
|
||||
- `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
|
||||
- `1` - The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
|
||||
- `1` - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
|
||||
- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
|
||||
- `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
|
||||
- `42` - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
|
||||
- `42` - The site handler declares `render` and the optional browser component is not installed
|
||||
|
||||
**See also**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
|
||||
- `sites/CONTRACT.md` - the instance's own handler files
|
||||
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
|
||||
- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host
|
||||
- `wikitool raw login` - the user signs in once for a host routed through the session
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
## `raw login`
|
||||
|
||||
Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
- `wikitool raw login <host>`
|
||||
|
||||
**Options**
|
||||
|
||||
```text
|
||||
Arguments:
|
||||
host The website's host, such as www.example.org [required]
|
||||
|
||||
Options:
|
||||
-h, --help Show this message and exit.
|
||||
```
|
||||
|
||||
**Examples**
|
||||
|
||||
```bash
|
||||
tools/wikitool raw login www.heise.de
|
||||
```
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
|
||||
- Stores only the browser state the window leaves (cookies and storage), never a password, under `session_<host>` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_<HOST>` answers before the file, as for any named secret. The session is never written to 1Password.
|
||||
- Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by.
|
||||
- Needs the optional browser component, which the user installs with the preflight's `--with browser` option.
|
||||
- Prints no part of the session. Signing in again replaces the stored session.
|
||||
|
||||
**Properties**
|
||||
|
||||
- Changes files: yes
|
||||
- Idempotent: yes
|
||||
- Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
|
||||
- Counts toward the session call budget: yes
|
||||
- Can reach the network: yes
|
||||
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed
|
||||
- `1` - The browser could not be started, or the window closed before any state could be read
|
||||
- `42` - The optional browser component is not installed
|
||||
|
||||
**See also**
|
||||
|
||||
- `wikitool raw fetch` - fetches the pages of a routed host with the stored session
|
||||
- `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login
|
||||
- `sites/CONTRACT.md` - a handler's `session` block
|
||||
- `docs/reference/configuration.md` - the `fetch` section
|
||||
|
||||
## `raw derive`
|
||||
|
||||
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
|
||||
|
||||
@@ -29,6 +29,7 @@ something you run.
|
||||
- [Profiles](#profiles)
|
||||
- [MCP submissions](#mcp-submissions)
|
||||
- [Site handlers](#site-handlers)
|
||||
- [Fetch routes](#fetch-routes)
|
||||
- [Files of earlier versions](#files-of-earlier-versions)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
@@ -90,7 +91,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
|
||||
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
|
||||
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
||||
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
||||
[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
|
||||
[upload](#mcp-submissions), [sites](#site-handlers) and [fetch](#fetch-routes). `config set` names the sections it knows, and `doctor` reports a
|
||||
section it does not read as a `WARN`.
|
||||
|
||||
**One place per value.** A value in the file its tier does not name is an error naming the file,
|
||||
@@ -117,8 +118,9 @@ for a human to unlock. Setting 1Password up: [use-1password-for-settings.md](../
|
||||
|
||||
**Never in the committed file:** a field that names a program to run or affects a gate - not even
|
||||
as a reference. `config set` does not write such a field either (exit 42); the user edits
|
||||
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` in the
|
||||
tables below are these: the publish allowlist and the upload limits.
|
||||
`.wikitool-local.json` by hand, and `doctor` checks it afterwards. The fields marked `gate` or
|
||||
`names a command` in the tables below are these: the publish allowlist, the upload limits and the
|
||||
external fetchers.
|
||||
|
||||
`wikitool doctor` reports one line per tier - `settings-instance`, `settings-checkout`,
|
||||
`settings-secrets` - and for the references only whether they resolve. `docs verify` holds a
|
||||
@@ -477,6 +479,60 @@ All three fields are instance-tier: what a website needs cleaned is the same in
|
||||
of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
|
||||
with exit 1 before it fetches anything.
|
||||
|
||||
## Fetch routes
|
||||
|
||||
`tools/wikitool raw fetch` fetches every page the same way unless the `fetch` section says
|
||||
otherwise for its host: anonymously, with no cookie and no browser. `routes` maps a host to another
|
||||
way, and is instance-tier, so every clone fetches the host the same way:
|
||||
|
||||
- `"session"` fetches the host's pages with the session the user stored with
|
||||
`tools/wikitool raw login <host>`, through the same built-in fetch. The host needs a site handler
|
||||
with a `session` block - its login page and how a page shows it arrived signed in
|
||||
([sites/CONTRACT.md](../../sites/CONTRACT.md)). No stored session, or a page without that sign,
|
||||
stops the fetch with exit 42 for the user to sign in again. The session is a secret: it is
|
||||
stored in the secrets file [above](#tiered-settings) under `session_<host>` (every run of other
|
||||
characters as one `_`, so `www.example.org` is `session_www_example_org`), or comes from
|
||||
`WIKITOOL_SECRET_SESSION_<HOST>`, and no settings field refers to it. It is never written to
|
||||
1Password.
|
||||
- Any other value names a fetcher of `fetchers`: a program of this checkout, run without a shell
|
||||
with the URL appended. `fetchers` names a program to run, so it lives only in
|
||||
`.wikitool-local.json`, written by hand - `config set` refuses it with exit 42, and a fetcher in
|
||||
`wikitool.json` is an error. A route to a fetcher this checkout does not define stops a fetch of
|
||||
that host with exit 1, so a clone without the program fails loudly instead of fetching
|
||||
anonymously.
|
||||
|
||||
The whole procedure, with the browser component it needs for signing in:
|
||||
[fetch-pages-behind-a-login.md](../how-to/fetch-pages-behind-a-login.md).
|
||||
|
||||
<!-- wikitool:reference-config-fetch -->
|
||||
| Field | Tier | Type | Required | Default | Description |
|
||||
|-------|------|------|----------|---------|-------------|
|
||||
| `routes` | instance | object | no | `{}` | Host (exact, lower case) to route: `"session"` fetches with the user's session - the host needs a site handler that names its login page - and any other value names a fetcher of `fetchers`. A host not listed is fetched as before. |
|
||||
| `fetchers` | checkout, names a command | object | no | `{}` | Fetcher name to command: an array of the program and its arguments, run without a shell with the URL appended. Exit 0 and the page's HTML on stdout is a fetch; anything else stops `raw fetch` with exit 1 and the program's stderr. |
|
||||
|
||||
- **Absent:** Every page is fetched by the built-in anonymous fetch, as before.
|
||||
- **Malformed:** An error when `raw fetch` or `raw login` reads it: the command stops with exit 1 before fetching anything. A route to a fetcher this checkout does not define stops a fetch of that host the same way.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"fetch": {
|
||||
"routes": {
|
||||
"www.heise.de": "session",
|
||||
"intranet.example.org": "homelab"
|
||||
},
|
||||
"fetchers": {
|
||||
"homelab": [
|
||||
"/usr/local/bin/fetch-page",
|
||||
"--html"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
<!-- /wikitool:reference-config-fetch -->
|
||||
|
||||
## Files of earlier versions
|
||||
|
||||
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
|
||||
|
||||
@@ -36,6 +36,9 @@ What each exit code means, and how a caller reacts to it, is [AGENTS.md § Tool
|
||||
- [`task list`](commands/pages.md#task-list): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
|
||||
- [`task close`](commands/pages.md#task-close): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
|
||||
- [`review`](commands/finding-and-checking.md#review): The tracker settings file of an earlier version is still there, or a secret source (1Password) needs unlocking
|
||||
- [`raw fetch`](commands/raw-material-and-uploads.md#raw-fetch): The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
|
||||
- [`raw fetch`](commands/raw-material-and-uploads.md#raw-fetch): The site handler declares `render` and the optional browser component is not installed
|
||||
- [`raw login`](commands/raw-material-and-uploads.md#raw-login): The optional browser component is not installed
|
||||
- [`upload accept`](commands/raw-material-and-uploads.md#upload-accept): Upload Review Gate: `--confirm` is absent or does not match the manifest's current token
|
||||
- [`sync`](commands/git.md#sync): Rebase-Review Gate: `<remote>/<branch>` moved and both sides changed the same file, generated files aside; the output lists the upstream commits, the overlapping files and their diff
|
||||
- [`publish`](commands/git.md#publish): The remotes settings file of an earlier version is still there
|
||||
|
||||
Reference in new issue
Block a user