sites: the instance writes its own site handlers as JSON files under sites/, the stack's learn.microsoft.com handlers become data files in the same format (#210); 10.0.0-beta.4

Files changed:
- .gitignore
- AGENTS.md
- CHANGES.md
- VERSION
- docs/explanation/architecture.md
- docs/how-to/README.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- instructions/dev/doc-pull-through.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/ownership.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/handlers/learn-microsoft.json
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/toc.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/explanation/architecture.md

@@ -140,6 +140,8 @@ chemenu/
 │   └── CONTRACT.md     # Run keys, required files, how a run closes
 ├── reports/            # DERIVED: lint reports, traces, eval scores. Gitignored
 │   └── CONTRACT.md
+├── sites/              # The instance's own site handlers for `raw fetch`, one JSON file per website
+│   └── CONTRACT.md     # File format, which handler applies, what a broken file does
 └── tools/              # COMPILER: the wikitool CLI
     ├── CONTRACT.md     # Command records (generated) and index, maintenance schedule
     └── README.md       # How wikitool is built and how to change it

docs/how-to/README.md

@@ -10,6 +10,7 @@ Working with the wiki:
 
 - [Ingest a source](ingest-a-source.md) - a file, a folder, a web page, or a document submitted
   through the MCP server
+- [Write a site handler](write-a-site-handler.md) - teach `raw fetch` how a website serves its pages
 - [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current
 - [Export guidelines to captured repositories](export-guidelines.md)
 - [Query the wiki](query-the-wiki.md) - asking, and searching yourself

docs/how-to/write-a-site-handler.md

@@ -0,0 +1,74 @@
+# Write a site handler
+
+Teach `tools/wikitool raw fetch` how one website serves its pages: the standard form of its URLs,
+the page furniture to leave out of the derived text, and a hint or two for whoever reads the
+capture. A handler is a JSON file in `sites/` at the root of the instance - data, not a program -
+so every clone fetches the website the same way.
+
+The stack already ships handlers, for `learn.microsoft.com` among others. They are the same kind of
+file, so the quickest start is to copy one:
+
+```bash
+cp tools/chemenu/sites/handlers/learn-microsoft.json sites/my-website.json
+```
+
+The file name without `.json` is the handler's name: lower-case letters, digits and hyphens.
+
+## 1. Find out what the website does
+
+Fetch a page of the website and read the derived `.md`:
+
+```bash
+tools/wikitool raw fetch <url>
+```
+
+Two things are worth a handler:
+
+- **The same page under several URLs** - a language segment, a tracking parameter. A rewrite
+  turns every spelling into one standard form, so two people ingesting the page get the same source.
+- **Page furniture in the derived text** - a sign-in notice, an action bar, a feedback footer. Open
+  the saved `.html`, find the element that holds it, and note its `id`, `class` or an attribute.
+
+## 2. Write the file
+
+A small handler that drops a cookie banner and a footer on one host:
+
+```json
+{
+  "host": "docs.example.org",
+  "summary": "Drops the cookie banner and the feedback footer.",
+  "drop": [
+    {"id": "cookie-banner"},
+    {"class": "feedback", "inside": "footer"}
+  ]
+}
+```
+
+`host` is required and matched exactly, in lower case. `path` narrows the handler to a part of the
+website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
+`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
+
+## 3. Try it
+
+Fetch a page again, or derive a saved one without a fetch:
+
+```bash
+tools/wikitool raw fetch <url>
+tools/wikitool raw fetch --html <saved.html> --url <url>
+```
+
+The header of the derived text then says `site: <name> (instance)`, and the furniture is gone. A
+check shows up as a `CHECK` line when it applies to the page.
+
+`tools/wikitool doctor` lists the handlers of the instance and fails on a file that cannot be used,
+naming the file and the reason. Such a file also stops a `raw fetch` of its own host with exit 1,
+before anything is fetched or written, so a handler that did not apply never leaves furniture in a
+source unnoticed.
+
+## Replace or extend a stack handler
+
+A handler of the instance for the same host and path as a stack handler **replaces** it. To keep
+what the stack knows and add to it, name it as `parent` and give the new file a `path` of its own.
+
+To switch a handler off without deleting it, list its name under `disabled` in the `sites` section
+of the settings - see [Configuration](../reference/configuration.md#site-handlers).

docs/reference/commands/instance-health.md

@@ -43,6 +43,7 @@ tools/wikitool doctor --json
 - Settings (`settings-instance`, `settings-checkout`, `settings-secrets`): one line per tier. `wikitool.json` and `.wikitool-local.json` each present or absent - absent is `OK` - and a `FAIL` naming file, key and the right tier when a value sits in the wrong file, a secret or a committed private field holds a plain value, or a value fails its field; a section this version does not read is a `WARN`. For the references: how many resolve, never a value - an unresolvable one is a `FAIL`, one 1Password will not hand out until a human unlocks it a `WARN`.
 - Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
 - Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
+- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
 - Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
 - Read-only and exempt from the Iteration Budget Gate.
 

docs/reference/commands/raw-material-and-uploads.md

@@ -55,7 +55,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
-- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -74,7 +74,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - `0` - success
 - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
-- `1` - The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
+- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -82,6 +82,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **See also**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `sites/CONTRACT.md` - the instance's own handler files
 - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts

docs/reference/configuration.md

@@ -414,8 +414,9 @@ Size and quota are your deployment's decision, not defaults worth copying blindl
 derived text and prints hints about the page as `CHECK` lines; today there are two, both for
 `learn.microsoft.com`: `learn-microsoft` for every page of the host and
 `learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
-naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one.
-The `sites` section is what the instance decides about them, and it is optional: without it every
+naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/`
+(see [How to write a site handler](../how-to/write-a-site-handler.md)).
+The `sites` section is what the instance decides about the handlers, and it is optional: without it every
 handler applies, in the locale `en-us`.
 
 - `disabled` switches handlers off by name. A page of that website is then handled as any other: