# sites/ - Handlers of this Instance One JSON file per website, read by `tools/wikitool raw fetch`: what this instance knows about how a website serves its pages, beyond what the stack's own handlers know. This file is the stack's; the handler files beside it are the instance's own, and no release touches them. They are tracked like `kb/` and `raw/`, so every clone fetches a website the same way. The stack's handlers are the same kind of file, under `tools/chemenu/sites/handlers/` - open one to see a working example, copy it to `sites/.json` to start from it. ## A handler file The file name without `.json` is the handler's name: lower-case letters, digits and hyphens. It is what `disabled` in the `sites` section of the settings names, and what the capture header shows as `site: (instance)`. | Field | Meaning | |-------|---------| | `host` | Required. The website's host name, lower case, matched exactly | | `summary` | Required. One sentence on what the handler does - what a reader of the file needs | | `path` | Optional. A regular expression matched against the start of the URL path; among the handlers of a host, the longest matching `path` wins. Without it the handler takes every path of the host | | `parent` | Optional. The name of a handler for the same host whose `rewrites`, `drop` rules and `checks` this one starts with | | `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` | | `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received | | `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page | | `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own | | `session` | Optional. For a website behind a login: the page `raw login` opens for the user to sign in on, and how a fetched page shows that it arrived signed in. A child inherits the parent's block unless it has its own | | `render` | Optional, `true` or `false` (the default). `true` fetches the handler's pages through a headless browser, because they draw their content with JavaScript; the header then says `rendered: true`. A child inherits `true` from its parent | A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class` (one token of the element's classes), `attr` (an attribute that must be present, equal to `value` when that is given) and `inside` (the name of an ancestor element). At least one of `tag`, `id`, `class`, `attr` is required, and `value` needs `attr`; every field given must match. A check is `{"text", "meta"?, "count"?}`. With `meta`, the line applies only when the page has a `` with that name and content, which replaces `{meta}` in the text. With `count`, it applies only when at least one element matches the selector, and `{count}` is how many. With neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have the field that fills it. `subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `` elements whose `href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the path of the page itself, with a trailing slash, and a value starting with `/` is a literal path. A link is resolved against the page's URL, loses its fragment and counts only on the page's own host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched because of it unless the user runs `raw fetch --scope`; without it the page prints one `CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched. `session` is `{"login", "signed_in"?, "signed_out"?}`. `login` is the `http(s)` URL of the login page. `signed_in` is a selector, as above, that matches only on a signed-in page - a logout link, an account menu; `signed_out` one that matches only on a page that was not - a paywall teaser, a "subscribe to read on" box. At least one of the two is required, and with both, both must hold. A page fetched with the user's session that does not show the sign stops `raw fetch` with exit 42 and writes nothing, so pick a sign that the full article always carries, or its teaser always does. The block alone changes nothing: the pages of the host are fetched with the session only once the instance routes the host there (`fetch.routes` in the settings). Which website allows its pages to be fetched this way is the user's to judge, and a handler records nothing about it. `render` changes what the `.html` is: the page as the browser drew it, not the bytes the server sent. Declare it only for a website whose text is missing from the HTML it sends. `raw login` and a rendered fetch need the optional browser component (`tools/preflight.sh --with browser`). `tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is refused. ## Which handler applies A handler of this instance for the same `host` and `path` as a stack handler **replaces** it - the stack's is not applied at all for that host and path, and `wikitool doctor` names the replacement. Handlers of the stack that inherited from the replaced one keep what they inherited. To keep what the stack knows and add to it, give the file a `parent` instead and a `path` of its own. A file named like a stack handler but for another host or path is refused, and so are two files with the same host and path. A `parent` is a handler of the stack or of this instance; a missing one, or a cycle, refuses the file. ## A file that cannot be used A file that is not valid JSON, breaks the schema, has a pattern that does not compile, or breaks a rule above is not applied. `wikitool doctor` fails and names the file and the reason. A `raw fetch` whose URL has the host such a file names - or any URL, when no host can be read from it - exits 1 before it fetches or writes anything: a handler that silently did not apply would leave furniture in the text and a locale-specific page in the source. A broken file for another host does not stop the fetch. ## Safety A pattern runs only against the URL and its path, never against the page. A handler cannot run code, read a file or reach another host: it is data, not a program. A `session` holds no credential either - only where the user signs in and what a signed-in page looks like; the session itself is stored outside the working tree by `raw login`.