chemenu/docs/how-to/write-a-site-handler.md

Write a site handler

Teach tools/wikitool raw fetch how one website serves its pages: the standard form of its URLs, the page furniture to leave out of the derived text, and a hint or two for whoever reads the capture. A handler is a JSON file in sites/ at the root of the instance - data, not a program - so every clone fetches the website the same way.

The stack already ships handlers, for learn.microsoft.com among others. They are the same kind of file, so the quickest start is to copy one:

cp tools/chemenu/sites/handlers/learn-microsoft.json sites/my-website.json

The file name without .json is the handler's name: lower-case letters, digits and hyphens.

1. Find out what the website does

Fetch a page of the website and read the derived .md:

tools/wikitool raw fetch <url>

Two things are worth a handler:

• The same page under several URLs - a language segment, a tracking parameter. A rewrite turns every spelling into one standard form, so two people ingesting the page get the same source.
• Page furniture in the derived text - a sign-in notice, an action bar, a feedback footer. Open the saved .html, find the element that holds it, and note its id, class or an attribute.

2. Write the file

A small handler that drops a cookie banner and a footer on one host:

{
  "host": "docs.example.org",
  "summary": "Drops the cookie banner and the feedback footer.",
  "drop": [
    {"id": "cookie-banner"},
    {"class": "feedback", "inside": "footer"}
  ]
}

host is required and matched exactly, in lower case. path narrows the handler to a part of the website; among the handlers of a host the longest matching path wins. The other fields - rewrites, checks, parent - and what each selector may say are in sites/CONTRACT.md.

A page that is an index - a module overview, a table of contents - can name its subpages, so that raw fetch <url> --scope fetches it and them as one bundle in incoming/<name>/:

"subpages": {"links": {"tag": "a", "inside": "td"}}

The links are the <a> elements inside a table cell; only those on the same host and under the page's own path count. A page with such a handler prints a CHECK line with their number when it is fetched without --scope.

3. Try it

Fetch a page again, or derive a saved one without a fetch:

tools/wikitool raw fetch <url>
tools/wikitool raw fetch --html <saved.html> --url <url>

The header of the derived text then says site: <name> (instance), and the furniture is gone. A check shows up as a CHECK line when it applies to the page.

tools/wikitool doctor lists the handlers of the instance and fails on a file that cannot be used, naming the file and the reason. Such a file also stops a raw fetch of its own host with exit 1, before anything is fetched or written, so a handler that did not apply never leaves furniture in a source unnoticed.

Replace or extend a stack handler

A handler of the instance for the same host and path as a stack handler replaces it. To keep what the stack knows and add to it, name it as parent and give the new file a path of its own.

To switch a handler off without deleting it, list its name under disabled in the sites section of the settings - see Configuration.