torbenandClaude Code
21cdd787b7
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/how-to/write-a-site-handler.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_raw_fetch.py - tools/chemenu/tests/test_raw_fetch_scope.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
86 lines
3.3 KiB
Markdown
86 lines
3.3 KiB
Markdown
# Write a site handler
|
|
|
|
Teach `tools/wikitool raw fetch` how one website serves its pages: the standard form of its URLs,
|
|
the page furniture to leave out of the derived text, and a hint or two for whoever reads the
|
|
capture. A handler is a JSON file in `sites/` at the root of the instance - data, not a program -
|
|
so every clone fetches the website the same way.
|
|
|
|
The stack already ships handlers, for `learn.microsoft.com` among others. They are the same kind of
|
|
file, so the quickest start is to copy one:
|
|
|
|
```bash
|
|
cp tools/chemenu/sites/handlers/learn-microsoft.json sites/my-website.json
|
|
```
|
|
|
|
The file name without `.json` is the handler's name: lower-case letters, digits and hyphens.
|
|
|
|
## 1. Find out what the website does
|
|
|
|
Fetch a page of the website and read the derived `.md`:
|
|
|
|
```bash
|
|
tools/wikitool raw fetch <url>
|
|
```
|
|
|
|
Two things are worth a handler:
|
|
|
|
- **The same page under several URLs** - a language segment, a tracking parameter. A rewrite
|
|
turns every spelling into one standard form, so two people ingesting the page get the same source.
|
|
- **Page furniture in the derived text** - a sign-in notice, an action bar, a feedback footer. Open
|
|
the saved `.html`, find the element that holds it, and note its `id`, `class` or an attribute.
|
|
|
|
## 2. Write the file
|
|
|
|
A small handler that drops a cookie banner and a footer on one host:
|
|
|
|
```json
|
|
{
|
|
"host": "docs.example.org",
|
|
"summary": "Drops the cookie banner and the feedback footer.",
|
|
"drop": [
|
|
{"id": "cookie-banner"},
|
|
{"class": "feedback", "inside": "footer"}
|
|
]
|
|
}
|
|
```
|
|
|
|
`host` is required and matched exactly, in lower case. `path` narrows the handler to a part of the
|
|
website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
|
|
`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
|
|
|
|
A page that is an index - a module overview, a table of contents - can name its subpages, so that
|
|
`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
|
|
|
|
```json
|
|
"subpages": {"links": {"tag": "a", "inside": "td"}}
|
|
```
|
|
|
|
The links are the `<a>` elements inside a table cell; only those on the same host and under the
|
|
page's own path count. A page with such a handler prints a `CHECK` line with their number when it
|
|
is fetched without `--scope`.
|
|
|
|
## 3. Try it
|
|
|
|
Fetch a page again, or derive a saved one without a fetch:
|
|
|
|
```bash
|
|
tools/wikitool raw fetch <url>
|
|
tools/wikitool raw fetch --html <saved.html> --url <url>
|
|
```
|
|
|
|
The header of the derived text then says `site: <name> (instance)`, and the furniture is gone. A
|
|
check shows up as a `CHECK` line when it applies to the page.
|
|
|
|
`tools/wikitool doctor` lists the handlers of the instance and fails on a file that cannot be used,
|
|
naming the file and the reason. Such a file also stops a `raw fetch` of its own host with exit 1,
|
|
before anything is fetched or written, so a handler that did not apply never leaves furniture in a
|
|
source unnoticed.
|
|
|
|
## Replace or extend a stack handler
|
|
|
|
A handler of the instance for the same host and path as a stack handler **replaces** it. To keep
|
|
what the stack knows and add to it, name it as `parent` and give the new file a `path` of its own.
|
|
|
|
To switch a handler off without deleting it, list its name under `disabled` in the `sites` section
|
|
of the settings - see [Configuration](../reference/configuration.md#site-handlers).
|