chemenu/docs/how-to/fetch-pages-behind-a-login.md

# Fetch pages behind a login

Let `tools/wikitool raw fetch` fetch the articles of a website you are subscribed to, with your own
login: you sign in once in a browser window, and the wiki fetches that website's pages with the
session your sign-in left until it expires. No password is stored, and the agent never sees one.
The same page covers websites whose pages only appear in a browser, and handing a website to a
fetch program of your own.

Whether a website's terms allow you to fetch your subscription this way is yours to check; the
stack does not check it and offers no advice on it.

Without any of this, a page behind a login can still be saved from your browser and handed to
`raw fetch --html` ([Ingest a source](ingest-a-source.md)) - one page at a time, by hand.

<!-- wikitool:toc -->
## Contents

- [1. Install the browser component](#1-install-the-browser-component)
- [2. Describe the website's login](#2-describe-the-websites-login)
- [3. Route the website through your session](#3-route-the-website-through-your-session)
- [4. Sign in](#4-sign-in)
- [5. Ingest as usual](#5-ingest-as-usual)
- [Pages that only appear in a browser](#pages-that-only-appear-in-a-browser)
- [A fetch program of your own](#a-fetch-program-of-your-own)
- [Where the session is kept](#where-the-session-is-kept)
<!-- /wikitool:toc -->

## 1. Install the browser component

Signing in needs a real browser window, so it needs the optional browser component: a Python
package and a Chromium of its own, installed into the instance's `tools/.venv`. You install it -
the agent never does:

```bash
tools/preflight.sh --with browser
```

In PowerShell 7: `pwsh -NoProfile -ExecutionPolicy Bypass -File tools/preflight.ps1 --with browser`.
Later preflight runs keep it up to date. `tools/wikitool doctor` reports it on its `browser` line.

## 2. Describe the website's login

The website needs a site handler that says where you sign in and how a page shows that it arrived
signed in. Add a `session` block to the website's handler in `sites/`, or write a handler just for
it ([Write a site handler](write-a-site-handler.md)):

```json
{
  "host": "news.example.com",
  "summary": "A subscription news site: articles need the subscriber's session.",
  "session": {
    "login": "https://news.example.com/login",
    "signed_in": {"tag": "a", "class": "logout"},
    "signed_out": {"class": "paywall-teaser"}
  }
}
```

`signed_in` is something only a signed-in page carries - a logout link, an account menu.
`signed_out` is something only a page that was not carries - the teaser box that asks you to
subscribe. One of the two is enough; open a saved article and a saved teaser and pick an element
that tells them apart. [sites/CONTRACT.md](../../sites/CONTRACT.md) has the exact format.

## 3. Route the website through your session

The handler alone changes nothing. Tell the agent:

> Route news.example.com through my session.

It runs `tools/wikitool config set fetch.routes '{"news.example.com": "session"}'`, keeping any
host already listed. The setting is committed with the instance, so every clone fetches the
website the same way - each with its own user's session.

## 4. Sign in

Run this yourself - it opens a browser window and waits for you:

```bash
tools/wikitool raw login news.example.com
```

Sign in as you always do, captcha and second factor included, then close the window. The session
is stored outside the instance's folder. Sign in again the same way whenever the wiki asks.

## 5. Ingest as usual

Give the agent the article's URL as for any web page ([Ingest a source](ingest-a-source.md)).
The capture's header names the website whose session was used (`session: news.example.com`),
never the session itself.

When your session has expired, or the page came back as a teaser, the agent stops: `raw fetch`
writes nothing and asks you to run `raw login` again. Nothing is kept from a teaser in silence.

## Pages that only appear in a browser

Some websites send an empty frame and draw the text with JavaScript; `raw fetch` then captures
almost nothing. Add `"render": true` to the website's handler: its pages are then fetched through
the browser component, with your session where the website has one, and the header says
`rendered: true`. The `.html` is then the page as the browser drew it, not what the server sent -
which is why it happens only where a handler says so.

## A fetch program of your own

A website can instead go to a program you run - a fetch service on another machine, say. Add it to
`.wikitool-local.json` by hand, since it names a program to run and `config set` does not write
such a field:

```json
{
  "schema": 1,
  "fetch": {"fetchers": {"homelab": ["/usr/local/bin/fetch-page", "--html"]}}
}
```

and route the host to it: `tools/wikitool config set fetch.routes '{"intranet.example.org": "homelab"}'`.
The program gets the URL as its last argument and prints the page's HTML; anything else - an exit
other than 0, no output - stops `raw fetch` with the program's error output, and the built-in
fetch is not tried instead. The `fetch` section is in
[docs/reference/configuration.md](../reference/configuration.md#fetch-routes).

## Where the session is kept

In the secrets file in your own configuration directory - `~/.config/chemenu/<instance_id>/secrets.json`,
`%APPDATA%\chemenu\<instance_id>\secrets.json` on Windows - under `session_<host>`, readable by
you only, outside every working tree. It never reaches a commit or a `wikitool` output. It does
not keep an agent with a shell in your account from reading it, any more than any other secret
there: [configuration-tiers.md](../explanation/configuration-tiers.md) says what keeping secrets
out of the tree protects against and what it does not.