torbenandClaude Code
21cdd787b7
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/how-to/write-a-site-handler.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_raw_fetch.py - tools/chemenu/tests/test_raw_fetch_scope.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
74 lines
5.0 KiB
Markdown
74 lines
5.0 KiB
Markdown
# sites/ - Handlers of this Instance
|
|
|
|
One JSON file per website, read by `tools/wikitool raw fetch`: what this instance knows about how
|
|
a website serves its pages, beyond what the stack's own handlers know. This file is the stack's;
|
|
the handler files beside it are the instance's own, and no release touches them. They are
|
|
tracked like `kb/` and `raw/`, so every clone fetches a website the same way.
|
|
|
|
The stack's handlers are the same kind of file, under `tools/chemenu/sites/handlers/` - open one
|
|
to see a working example, copy it to `sites/<name>.json` to start from it.
|
|
|
|
## A handler file
|
|
|
|
The file name without `.json` is the handler's name: lower-case letters, digits and hyphens. It is
|
|
what `disabled` in the `sites` section of the settings names, and what the capture header shows
|
|
as `site: <name> (instance)`.
|
|
|
|
| Field | Meaning |
|
|
|-------|---------|
|
|
| `host` | Required. The website's host name, lower case, matched exactly |
|
|
| `summary` | Required. One sentence on what the handler does - what a reader of the file needs |
|
|
| `path` | Optional. A regular expression matched against the start of the URL path; among the handlers of a host, the longest matching `path` wins. Without it the handler takes every path of the host |
|
|
| `parent` | Optional. The name of a handler for the same host whose `rewrites`, `drop` rules and `checks` this one starts with |
|
|
| `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` |
|
|
| `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received |
|
|
| `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page |
|
|
| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own |
|
|
|
|
A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class`
|
|
(one token of the element's classes), `attr` (an attribute that must be present, equal to `value`
|
|
when that is given) and `inside` (the name of an ancestor element). At least one of `tag`, `id`,
|
|
`class`, `attr` is required, and `value` needs `attr`; every field given must match.
|
|
|
|
A check is `{"text", "meta"?, "count"?}`. With `meta`, the line applies only when the page has a
|
|
`<meta name="...">` with that name and content, which replaces `{meta}` in the text. With `count`,
|
|
it applies only when at least one element matches the selector, and `{count}` is how many. With
|
|
neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have
|
|
the field that fills it.
|
|
|
|
`subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `<a>` elements whose
|
|
`href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the
|
|
path of the page itself, with a trailing slash, and a value starting with `/` is a literal path.
|
|
A link is resolved against the page's URL, loses its fragment and counts only on the page's own
|
|
host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched
|
|
because of it unless the user runs `raw fetch <url> --scope`; without it the page prints one
|
|
`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched.
|
|
|
|
`tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is
|
|
refused.
|
|
|
|
## Which handler applies
|
|
|
|
A handler of this instance for the same `host` and `path` as a stack handler **replaces** it - the
|
|
stack's is not applied at all for that host and path, and `wikitool doctor` names the replacement.
|
|
Handlers of the stack that inherited from the replaced one keep what they inherited. To keep what
|
|
the stack knows and add to it, give the file a `parent` instead and a `path` of its own.
|
|
|
|
A file named like a stack handler but for another host or path is refused, and so are two files
|
|
with the same host and path. A `parent` is a handler of the stack or of this instance; a missing
|
|
one, or a cycle, refuses the file.
|
|
|
|
## A file that cannot be used
|
|
|
|
A file that is not valid JSON, breaks the schema, has a pattern that does not compile, or breaks
|
|
a rule above is not applied. `wikitool doctor` fails and names the file and the reason. A
|
|
`raw fetch` whose URL has the host such a file names - or any URL, when no host can be read from
|
|
it - exits 1 before it fetches or writes anything: a handler that silently did not apply would
|
|
leave furniture in the text and a locale-specific page in the source. A broken file for another
|
|
host does not stop the fetch.
|
|
|
|
## Safety
|
|
|
|
A pattern runs only against the URL and its path, never against the page. A handler cannot run
|
|
code, read a file or reach another host: it is data, not a program.
|