chemenu/docs/how-to/fetch-pages-behind-a-login.md

Fetch pages behind a login

Let tools/wikitool raw fetch fetch the articles of a website you are subscribed to, with your own login: you sign in once in a browser window, and the wiki fetches that website's pages with the session your sign-in left until it expires. No password is stored, and the agent never sees one. The same page covers websites whose pages only appear in a browser, and handing a website to a fetch program of your own.

Whether a website's terms allow you to fetch your subscription this way is yours to check; the stack does not check it and offers no advice on it.

Without any of this, a page behind a login can still be saved from your browser and handed to raw fetch --html (Ingest a source) - one page at a time, by hand.

Contents

• 1. Install the browser component
• 2. Describe the website's login
• 3. Route the website through your session
• 4. Sign in
• 5. Ingest as usual
• Pages that only appear in a browser
• A fetch program of your own
• Where the session is kept

1. Install the browser component

Signing in needs a real browser window, so it needs the optional browser component: a Python package and a Chromium of its own, installed into the instance's tools/.venv. You install it - the agent never does:

tools/preflight.sh --with browser

In PowerShell 7: pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser. Later preflight runs keep it up to date. tools/wikitool doctor reports it on its browser line.

2. Describe the website's login

The website needs a site handler that says where you sign in and how a page shows that it arrived signed in. Add a session block to the website's handler in sites/, or write a handler just for it (Write a site handler):

{
  "host": "news.example.com",
  "summary": "A subscription news site: articles need the subscriber's session.",
  "session": {
    "login": "https://news.example.com/login",
    "signed_in": {"tag": "a", "class": "logout"},
    "signed_out": {"class": "paywall-teaser"}
  }
}

signed_in is something only a signed-in page carries - a logout link, an account menu. signed_out is something only a page that was not carries - the teaser box that asks you to subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element that tells them apart. sites/CONTRACT.md has the exact format.

3. Route the website through your session

The handler alone changes nothing. Tell the agent:

Route news.example.com through my session.

It runs tools/wikitool config set fetch.routes '{"news.example.com": "session"}', keeping any host already listed. The setting is committed with the instance, so every clone fetches the website the same way - each with its own user's session.

4. Sign in

Run this yourself - it opens a browser window and waits for you:

tools/wikitool raw login news.example.com

Sign in as you always do, captcha and second factor included, then close the window. The session is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.

5. Ingest as usual

Give the agent the article's URL as for any web page (Ingest a source). The capture's header names the website whose session was used (session: news.example.com), never the session itself.

When your session has expired, or the page came back as a teaser, the agent stops: raw fetch writes nothing and asks you to run raw login again. Nothing is kept from a teaser in silence.

Pages that only appear in a browser

Some websites send an empty frame and draw the text with JavaScript; raw fetch then captures almost nothing. Add "render": true to the website's handler: its pages are then fetched through the browser component, with your session where the website has one, and the header says rendered: true. The .html is then the page as the browser drew it, not what the server sent - which is why it happens only where a handler says so.

A fetch program of your own

A website can instead go to a program you run - a fetch service on another machine, say. Add it to .wikitool-local.json by hand, since it names a program to run and config set does not write such a field:

{
  "schema": 1,
  "fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
}

and route the host to it: tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'. The program gets the URL as its last argument and prints the page's HTML; anything else - an exit other than 0, no output - stops raw fetch with the program's error output, and the built-in fetch is not tried instead. The fetch section is in docs/reference/configuration.md.

Where the session is kept

In the secrets file in your own configuration directory - ~/.config/chemenu/<instance_id>/secrets.json, %APPDATA%\chemenu\<instance_id>\secrets.json on Windows - under session_<host>, readable by you only, outside every working tree. It never reaches a commit or a wikitool output. It does not keep an agent with a shell in your account from reading it, any more than any other secret there: configuration-tiers.md says what keeping secrets out of the tree protects against and what it does not.