chemenu/sites/CONTRACT.md

sites/ - Handlers of this Instance

One JSON file per website, read by tools/wikitool raw fetch: what this instance knows about how a website serves its pages, beyond what the stack's own handlers know. This file is the stack's; the handler files beside it are the instance's own, and no release touches them. They are tracked like kb/ and raw/, so every clone fetches a website the same way.

The stack's handlers are the same kind of file, under tools/chemenu/sites/handlers/ - open one to see a working example, copy it to sites/<name>.json to start from it.

A handler file

The file name without .json is the handler's name: lower-case letters, digits and hyphens. It is what disabled in the sites section of the settings names, and what the capture header shows as site: <name> (instance).

Field	Meaning
host	Required. The website's host name, lower case, matched exactly
summary	Required. One sentence on what the handler does - what a reader of the file needs
path	Optional. A regular expression matched against the start of the URL path; among the handlers of a host, the longest matching path wins. Without it the handler takes every path of the host
parent	Optional. The name of a handler for the same host whose rewrites, drop rules and checks this one starts with
rewrites	Optional, at most 5. Each {"pattern", "replacement"} is applied to the URL once, in order, as re.sub(pattern, replacement, url, count=1); {locale} in a replacement is the locale of the sites section. The result is the standard form that is fetched; the URL as typed goes into the header as requested_url
drop	Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the .html stays as received
checks	Optional, at most 3 - counting the parent's. Hint lines printed as CHECK when they apply to the page
subpages	Optional. Which links of a page are its subpages, so raw fetch --scope can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own
session	Optional. For a website behind a login: the page raw login opens for the user to sign in on, and how a fetched page shows that it arrived signed in. A child inherits the parent's block unless it has its own
render	Optional, true or false (the default). true fetches the handler's pages through a headless browser, because they draw their content with JavaScript; the header then says rendered: true. A child inherits true from its parent

A selector in drop, and the count of a check, is an object with any of tag, id, class (one token of the element's classes), attr (an attribute that must be present, equal to value when that is given) and inside (the name of an ancestor element). At least one of tag, id, class, attr is required, and value needs attr; every field given must match.

A check is {"text", "meta"?, "count"?}. With meta, the line applies only when the page has a <meta name="..."> with that name and content, which replaces {meta} in the text. With count, it applies only when at least one element matches the selector, and {count} is how many. With neither, it applies to every page of the handler. A text that uses {meta} or {count} must have the field that fills it.

subpages is {"links", "prefix"?}. links is a selector, as above, for the <a> elements whose href names a subpage; prefix limits the path a subpage may have: {page} (the default) is the path of the page itself, with a trailing slash, and a value starting with / is a literal path. A link is resolved against the page's URL, loses its fragment and counts only on the page's own host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched because of it unless the user runs raw fetch <url> --scope; without it the page prints one CHECK line with the number of subpages. At most max_pages of the sites section are fetched.

session is {"login", "signed_in"?, "signed_out"?}. login is the http(s) URL of the login page. signed_in is a selector, as above, that matches only on a signed-in page - a logout link, an account menu; signed_out one that matches only on a page that was not - a paywall teaser, a "subscribe to read on" box. At least one of the two is required, and with both, both must hold. A page fetched with the user's session that does not show the sign stops raw fetch with exit 42 and writes nothing, so pick a sign that the full article always carries, or its teaser always does. The block alone changes nothing: the pages of the host are fetched with the session only once the instance routes the host there (fetch.routes in the settings). Which website allows its pages to be fetched this way is the user's to judge, and a handler records nothing about it.

render changes what the .html is: the page as the browser drew it, not the bytes the server sent. Declare it only for a website whose text is missing from the HTML it sends. raw login and a rendered fetch need the optional browser component (tools/preflight.sh --with browser).

tools/chemenu/sites/site.schema.json is the exact shape; every field outside the table above is refused.

Which handler applies

A handler of this instance for the same host and path as a stack handler replaces it - the stack's is not applied at all for that host and path, and wikitool doctor names the replacement. Handlers of the stack that inherited from the replaced one keep what they inherited. To keep what the stack knows and add to it, give the file a parent instead and a path of its own.

A file named like a stack handler but for another host or path is refused, and so are two files with the same host and path. A parent is a handler of the stack or of this instance; a missing one, or a cycle, refuses the file.

A file that cannot be used

A file that is not valid JSON, breaks the schema, has a pattern that does not compile, or breaks a rule above is not applied. wikitool doctor fails and names the file and the reason. A raw fetch whose URL has the host such a file names - or any URL, when no host can be read from it - exits 1 before it fetches or writes anything: a handler that silently did not apply would leave furniture in the text and a locale-specific page in the source. A broken file for another host does not stop the fetch.

Safety

A pattern runs only against the URL and its path, never against the page. A handler cannot run code, read a file or reach another host: it is data, not a program. A session holds no credential either - only where the user signs in and what a signed-in page looks like; the session itself is stored outside the working tree by raw login.