chemenu/sites/CONTRACT.md

# sites/ - Handlers of this Instance

One JSON file per website, read by `tools/wikitool raw fetch`: what this instance knows about how
a website serves its pages, beyond what the stack's own handlers know. This file is the stack's;
the handler files beside it are the instance's own, and no release touches them. They are
tracked like `kb/` and `raw/`, so every clone fetches a website the same way.

The stack's handlers are the same kind of file, under `tools/chemenu/sites/handlers/` - open one
to see a working example, copy it to `sites/<name>.json` to start from it.

## A handler file

The file name without `.json` is the handler's name: lower-case letters, digits and hyphens. It is
what `disabled` in the `sites` section of the settings names, and what the capture header shows
as `site: <name> (instance)`.

| Field | Meaning |
|-------|---------|
| `host` | Required. The website's host name, lower case, matched exactly |
| `summary` | Required. One sentence on what the handler does - what a reader of the file needs |
| `path` | Optional. A regular expression matched against the start of the URL path; among the handlers of a host, the longest matching `path` wins. Without it the handler takes every path of the host |
| `parent` | Optional. The name of a handler for the same host whose `rewrites`, `drop` rules and `checks` this one starts with |
| `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` |
| `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received |
| `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page |
| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own |
| `session` | Optional. For a website behind a login: the page `raw login` opens for the user to sign in on, and how a fetched page shows that it arrived signed in. A child inherits the parent's block unless it has its own |
| `render` | Optional, `true` or `false` (the default). `true` fetches the handler's pages through a headless browser, because they draw their content with JavaScript; the header then says `rendered: true`. A child inherits `true` from its parent |

A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class`
(one token of the element's classes), `attr` (an attribute that must be present, equal to `value`
when that is given) and `inside` (the name of an ancestor element). At least one of `tag`, `id`,
`class`, `attr` is required, and `value` needs `attr`; every field given must match.

A check is `{"text", "meta"?, "count"?}`. With `meta`, the line applies only when the page has a
`<meta name="...">` with that name and content, which replaces `{meta}` in the text. With `count`,
it applies only when at least one element matches the selector, and `{count}` is how many. With
neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have
the field that fills it.

`subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `<a>` elements whose
`href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the
path of the page itself, with a trailing slash, and a value starting with `/` is a literal path.
A link is resolved against the page's URL, loses its fragment and counts only on the page's own
host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched
because of it unless the user runs `raw fetch <url> --scope`; without it the page prints one
`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched.

`session` is `{"login", "signed_in"?, "signed_out"?}`. `login` is the `http(s)` URL of the login
page. `signed_in` is a selector, as above, that matches only on a signed-in page - a logout link,
an account menu; `signed_out` one that matches only on a page that was not - a paywall teaser, a
"subscribe to read on" box. At least one of the two is required, and with both, both must hold. A
page fetched with the user's session that does not show the sign stops `raw fetch` with exit 42
and writes nothing, so pick a sign that the full article always carries, or its teaser always
does. The block alone changes nothing: the pages of the host are fetched with the session only
once the instance routes the host there (`fetch.routes` in the settings). Which website allows
its pages to be fetched this way is the user's to judge, and a handler records nothing about it.

`render` changes what the `.html` is: the page as the browser drew it, not the bytes the server
sent. Declare it only for a website whose text is missing from the HTML it sends. `raw login` and
a rendered fetch need the optional browser component (`tools/preflight.sh --with browser`).

`tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is
refused.

## Which handler applies

A handler of this instance for the same `host` and `path` as a stack handler **replaces** it - the
stack's is not applied at all for that host and path, and `wikitool doctor` names the replacement.
Handlers of the stack that inherited from the replaced one keep what they inherited. To keep what
the stack knows and add to it, give the file a `parent` instead and a `path` of its own.

A file named like a stack handler but for another host or path is refused, and so are two files
with the same host and path. A `parent` is a handler of the stack or of this instance; a missing
one, or a cycle, refuses the file.

## A file that cannot be used

A file that is not valid JSON, breaks the schema, has a pattern that does not compile, or breaks
a rule above is not applied. `wikitool doctor` fails and names the file and the reason. A
`raw fetch` whose URL has the host such a file names - or any URL, when no host can be read from
it - exits 1 before it fetches or writes anything: a handler that silently did not apply would
leave furniture in the text and a locale-specific page in the source. A broken file for another
host does not stop the fetch.

## Safety

A pattern runs only against the URL and its path, never against the page. A handler cannot run
code, read a file or reach another host: it is data, not a program. A `session` holds no
credential either - only where the user signs in and what a signed-in page looks like; the
session itself is stored outside the working tree by `raw login`.