sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3

Files changed:
- tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py
- tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

CHANGES.md

@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 ---
 
-## 10.0.0-beta.2 - 2026-10-11 - Die Einzeldateien .wikitool-remotes/-telemetry/-upload/-tasks.json sind in den drei Einstellungsstufen aufgegangen; neues Kommando config migrate
+## 10.0.0-beta.3 - 2026-10-11 - Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen
 
 **Author:** Torben Nehmer
 
@@ -28,6 +28,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 **Medium impact**
 - Einstellungen in drei Stufen: wikitool.json (Instanz), .wikitool-local.json (Checkout), Secrets nur per Verweis - auch über die 1Password-CLI; neues Kommando config set
+- Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen
 
 **Low impact**
 - Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
@@ -39,6 +40,10 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 Neue Einstellungen liegen in `wikitool.json` (eingecheckt, gleich in jedem Klon) oder `.wikitool-local.json` (gitignoriert, nur dieser Checkout); jedes Feld deklariert seine Stufe, ein Wert in der falschen Datei ist ein Fehler, keine Überschreibung. Ein Feld, das ein Programm benennt oder ein Gate betrifft, ist nie `instance`. Secrets stehen nur als Verweis in den Dateien: `{"ref": "<name>"}` über `WIKITOOL_SECRET_<NAME>` oder eine Secret-Datei im Nutzerverzeichnis, `{"ref": "op://..."}` über die 1Password-CLI, die der Preflight als optionales Werkzeug erfasst; gesperrtes 1Password ergibt exit 42. `config set` schreibt dorthin, wo die Stufe es verlangt, und verweigert Secret-Werte; `doctor` zeigt eine Zeile pro Stufe, `docs verify` prüft `wikitool.json` auf Klartext-Secrets. Noch liest kein Feature seine Einstellungen so.
 
+### Site-Handler für `raw fetch` (#200)
+
+Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt die Hinweise der Website als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind Code in `tools/chemenu/sites/` (ein Modul je Website, der längste Pfad gewinnt, ein spezieller erbt vom allgemeinen, eine doppelte Registrierung scheitert beim Import), ihre Bausteine sind auch als Daten ausdrückbar. Der Header nennt `site:` und, wenn die Standardform abweicht, die eingegebene URL als `requested_url`; das `.html` bleibt byte-genau. Erste Handler: `learn-microsoft` und `learn-microsoft-powershell-module`. Der Abschnitt `sites` der Einstellungen (Stufe `instance`) schaltet Handler ab, wählt das Gebietsschema und nimmt bis zu drei Hinweiszeilen je Host-Muster auf. URLs ohne Handler und `raw derive` bleiben unverändert.
+
 ### Altlasten der Konfiguration bereinigt (#208)
 
 Die vier Einzeldateien `.wikitool-remotes.json`, `.wikitool-telemetry.json`, `.wikitool-upload.json` und `.wikitool-tasks.json` gibt es nicht mehr; ihre Felder sind die Abschnitte `remotes`, `telemetry`, `upload`, `tasks`, `superproductivity` und `caldav` der drei Stufen. Zugangsdaten sind dort nur noch Verweise. Der Code liest die alten Dateien nicht mehr: ein Kommando, das die Einstellung braucht, endet mit Exit 42 und nennt `tools/wikitool config migrate`, `doctor` meldet FAIL, der Telemetrie-Hook bleibt still. `config migrate` läuft einmal je Checkout, ist wiederholbar und lässt bei jedem Abbruch die alten Dateien unverändert. `WIKITOOL_TASKS_CONFIG` zeigt auf ein Profil im Format von `.wikitool-local.json`. Nur `.wikitool-tools.json` bleibt eine eigene Datei.

VERSION

@@ -1 +1 @@
-10.0.0-beta.2
+10.0.0-beta.3

docs/how-to/ingest-a-source.md

@@ -16,6 +16,12 @@ pick a directory:
 - **A web page** needs no download of your own. Name the URL to the agent, and
   `tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
   text derived from it with a header recording where and when it was fetched.
+  For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the
+  same page in the website's standard form (the English one, whatever language your link was in),
+  leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints
+  `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
+  capture. Your instance can switch such a handler off, choose another language and add hint lines
+  of its own per website; see [the configuration reference](../reference/configuration.md).
 - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the

docs/reference/commands/README.md

@@ -79,7 +79,7 @@ Full records: [provenance.md](provenance.md)
 
 Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
 
-- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 - [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
 - [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
 - [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.

docs/reference/commands/raw-material-and-uploads.md

@@ -20,7 +20,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
 <!-- wikitool:reference-commands -->
 ## `raw fetch`
 
-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 
 **Synopsis**
 
@@ -52,9 +52,10 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **Behaviour**
 
 - Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.
-- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text.
+- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
+- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -73,6 +74,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - `0` - success
 - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
+- `1` - The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -80,6 +82,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **See also**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
 

docs/reference/configuration.md

@@ -28,6 +28,7 @@ something you run.
   - [CalDAV](#caldav)
   - [Profiles](#profiles)
 - [MCP submissions](#mcp-submissions)
+- [Site handlers](#site-handlers)
 - [Files of earlier versions](#files-of-earlier-versions)
 <!-- /wikitool:toc -->
 
@@ -89,7 +90,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
 `instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
 the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
 [remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
-[upload](#mcp-submissions). `config set` names the sections it knows, and `doctor` reports a
+[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
 section it does not read as a `WARN`.
 
 **One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -406,6 +407,68 @@ Example:
 
 Size and quota are your deployment's decision, not defaults worth copying blindly.
 
+## Site handlers
+
+`tools/wikitool raw fetch` knows how some websites serve their pages. For a URL of such a website a
+*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
+derived text and prints hints about the page as `CHECK` lines; today there are two, both for
+`learn.microsoft.com`: `learn-microsoft` for every page of the host and
+`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
+naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one.
+The `sites` section is what the instance decides about them, and it is optional: without it every
+handler applies, in the locale `en-us`.
+
+- `disabled` switches handlers off by name. A page of that website is then handled as any other:
+  fetched under the URL as typed, with its page furniture in the derived text. A name no handler has
+  is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the
+  general one of the same website in force.
+- `locale` is the language segment a handler for a multi-language website puts into the URL it
+  fetches. The header of the derived text keeps the URL as typed under `requested_url`.
+- `notes` adds hint lines of the instance's own, printed after a fetch or an `--html` derivation of
+  a page of that host, whether or not a handler exists for it. An entry names a `host` (exact, lower
+  case), optionally a `path` (a regular expression matched against the start of the URL path, after
+  the standard form was applied) and one to three single-line `lines`. A host and path pattern appears
+  once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
+  output as it is, so it is for a reminder to the reader of the capture, never for an instruction
+  the tool should carry out.
+
+<!-- wikitool:reference-config-sites -->
+| Field | Tier | Type | Required | Default | Description |
+|-------|------|------|----------|---------|-------------|
+| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
+| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
+| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
+
+- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
+- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
+
+Example:
+
+```json
+{
+  "sites": {
+    "disabled": [
+      "learn-microsoft-powershell-module"
+    ],
+    "locale": "en-us",
+    "notes": [
+      {
+        "host": "learn.microsoft.com",
+        "path": "^/[a-z-]+/azure/",
+        "lines": [
+          "Check the Azure retirement notice at the top of the page."
+        ]
+      }
+    ]
+  }
+}
+```
+<!-- /wikitool:reference-config-sites -->
+
+All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none
+of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
+with exit 1 before it fetches anything.
+
 ## Files of earlier versions
 
 Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,

raw/CONTRACT.md

@@ -217,6 +217,8 @@ derived_from: post.html
 ---
 ```
 
+(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler.)
+
 The fields are capture metadata, not page frontmatter - `raw/` has no types, and nothing in the
 stack reads the block back. There is deliberately no `author:`: HTML does not reliably say who
 wrote a page, and a guessed author would be a claim about the source. It goes on the source page
@@ -242,6 +244,28 @@ carries `derived:` - when the text was derived - instead of `retrieved:`, `final
 `http_status:` and `content_type:`: when the human saved the page, the tool does not know and does
 not claim.
 
+**A website the stack knows is fetched in its standard form and cleaned.** The same page of a
+website often exists under many URLs - in every language, with tracking parameters - and carries
+page furniture the browser hides but the HTML contains: an action bar, a sign-in notice, a
+feedback footer. Left in, it ends up in the derived text, and with it in every claim made from the
+page. A *site handler* - code in the stack, one module per website under `tools/chemenu/sites/`,
+chosen by host name and the longest matching path pattern - states what is known about that
+website: the standard form of its URLs, the page furniture to leave out of the derived text, and
+hints about what the derivation cannot do for the reader. For `learn.microsoft.com` that is the
+locale segment (`en-us`, so two people ingesting the German and the English page get the same
+source) with the `view=` of the URL kept, and a hint naming the Markdown source the page is built
+from, which is not fetched. A handler's hints are printed as `CHECK` lines, only when they apply
+to the page; act on one before accepting the capture.
+
+The standard form is what is fetched and what `url:` and the stem come from; the URL as typed is
+kept in the header as `requested_url:`, and `site:` names the handler. The `.html` is still the
+response body byte for byte - only the derived text is cleaned. `--html --url <url>` removes the
+same furniture from a saved page but keeps the URL as typed, since no fetch happened. A URL no
+handler claims behaves exactly as without handlers. The instance does not write handlers: its
+`sites` section in the settings ([docs/reference/configuration.md](../docs/reference/configuration.md))
+switches one off, sets the locale, and adds hint lines of its own per host and path pattern,
+printed as `CHECK` lines too - at most three per pattern.
+
 Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser
 can be longer than the 200 characters under which `raw fetch` warns. The session that reads the
 `.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a

tools/CONTRACT.md

@@ -123,7 +123,7 @@ review                read  idempotent     budget:exempt               exit:0,1,
 sources coverage      read  idempotent     budget:counted              exit:0      List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages.
 sources trace         read  idempotent     budget:counted              exit:0,1    Trace provenance in either direction: raw file, or page.
 sources rebuild-index write idempotent     budget:counted              exit:0,1    Regenerate the `kb/provenance.md` reverse index.
-raw fetch             write non-idempotent budget:counted              exit:0,1    Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+raw fetch             write non-idempotent budget:counted              exit:0,1    Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 raw derive            write non-idempotent budget:counted              exit:0,1    Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
 raw capture           write non-idempotent budget:counted              exit:0,1    Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
 raw status            read  idempotent     budget:counted              exit:0      Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
@@ -1420,7 +1420,7 @@ Regenerate the `kb/provenance.md` reverse index.
 
 #### `raw fetch`
 
-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 
 **SYNOPSIS**
 
@@ -1445,6 +1445,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 
 - 0    success
 - 1    The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
+- 1    The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
 - 1    A target file already exists in `incoming/`
 - 1    The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - 1    raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -1452,6 +1453,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **ON FAILURE**
 
 - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given -> Fix the call and retry once. A local file is dropped into `incoming/` by hand, not fetched
+- The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist -> Nothing was fetched or written. Fix the section named in the message - `tools/wikitool config set` or the settings file - then retry once
 - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
 - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
 - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
@@ -1464,9 +1466,10 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **NOTES**
 
 - Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.
-- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text.
+- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
+- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -1476,6 +1479,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **SEE ALSO**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
 

tools/README.md

@@ -118,6 +118,7 @@ tools/
     kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
     corpus_diff.py         invariant comparison of kb/ between two revisions
     web_capture.py         `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
+    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`) and one module per website, each registering the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text), standard library only
     filters/               the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
     repo_capture.py        `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
     guideline_export.py    `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached

tools/chemenu/commands/raw/fetch.py

@@ -13,7 +13,7 @@ from typing import Optional
 import typer
 from rich.markup import escape
 
-from chemenu import cli_contract, web_capture
+from chemenu import cli_contract, sites, web_capture
 from chemenu.commands._util import fail, rel_path, success
 from chemenu.errors import ChemenuError
 from chemenu.version import VersionError, read_version
@@ -73,6 +73,24 @@ def _warn(derivation: web_capture.Derivation, md_path: Path) -> None:
         )
 
 
+def _plan(url: str, *, canonical: bool) -> sites.Plan:
+    try:
+        return sites.plan_for(url, sites.read_preferences(), canonical=canonical)
+    except ChemenuError as exc:
+        fail(escape(str(exc)))
+
+
+def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation) -> None:
+    for line in plan.checks(derivation.decoded.text):
+        typer.echo(f"CHECK {line}")
+
+
+def _fetched(plan: sites.Plan) -> str:
+    if not plan.requested_url:
+        return plan.url
+    return f"{plan.url} (the {plan.name} standard form of {plan.requested_url})"
+
+
 def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     if source_url is None:
         fail("--html needs --url <url>: it goes into the header, resolves relative links, and "
@@ -81,6 +99,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         web_capture.check_url(source_url)
     except ChemenuError as exc:
         fail(escape(str(exc)))
+    plan = _plan(source_url, canonical=False)
     path = _resolve(html)
     _validate_under_incoming(path, _incoming_dir())
     if not path.is_file():
@@ -90,9 +109,12 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         fail(f"{rel_path(path)} is already a .md file - --html takes the saved HTML page.")
     _refuse_existing([md_path])
 
-    derivation = web_capture.derive_document(path.read_bytes(), source_url, path.name, url=source_url)
+    derivation = web_capture.derive_document(
+        path.read_bytes(), source_url, path.name, url=source_url, drop=plan.drop, site=plan.name
+    )
     _write_exclusive([(md_path, derivation.markdown)])
     _warn(derivation, md_path)
+    _echo_checks(plan, derivation)
     success(escape(
         f"Derived {rel_path(md_path)} from {rel_path(path)} (no network access).\n"
         + _next_lines([path, md_path], source_url)
@@ -102,7 +124,8 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
 @cli_contract.record(cli_contract.CommandRecord(
     path="raw fetch",
     summary="Capture a web page the user names into `incoming/`: the HTML as received plus a "
-    "derived text, for `raw accept` to promote.",
+    "derived text, for `raw accept` to promote. Pages of a known website are fetched in the "
+    "website's standard form and cleaned of its page furniture.",
     synopsis=(
         cli_contract.Variant(
             usage="raw fetch <url> [--name <stem>]",
@@ -127,7 +150,9 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         "afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.",
         "The `.html` holds the response body byte for byte. The `.md` starts with a fixed "
         "header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, "
-        "`content_type`, `charset`, `title`, `derived_from`) followed by the derived text.",
+        "`content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page "
+        "of a website with a site handler adds `site`, and `requested_url` when the URL fetched "
+        "is not the one typed.",
         "Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a "
         "`<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with "
         "U+FFFD and reported; the header names the charset and where it came from.",
@@ -135,6 +160,15 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         "`<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, "
         "`template` and `svg` are dropped below it. Headings, lists, code, tables and links "
         "(made absolute) become Markdown. The same bytes always give the same text.",
+        "Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), "
+        "`raw fetch <url>` fetches the standard form of the URL (the locale segment becomes "
+        "`en-us`, the `view=` query stays) and `<url>` as typed goes into the header as "
+        "`requested_url`; `url` is the form fetched. The handler also removes the page furniture "
+        "the website renders around the article from the derived text - never from the `.html` - "
+        "and prints hints that apply to this page as `CHECK` lines: act on one before accepting. "
+        "`--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no "
+        "handler claims is handled as before. The `sites` section of the settings switches handlers "
+        "off, sets the locale and adds hint lines of the instance's own per host.",
         "The stem is the URL's last path segment without its extension, else its host name - "
         "ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.",
         "A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any "
@@ -157,6 +191,12 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
             reaction="Fix the call and retry once. A local file is dropped into `incoming/` by "
             "hand, not fetched",
         ),
+        cli_contract.Failure(
+            cause="The `sites` section of the settings is malformed, or `disabled` names a handler "
+            "that does not exist",
+            reaction="Nothing was fetched or written. Fix the section named in the message - "
+            "`tools/wikitool config set` or the settings file - then retry once",
+        ),
         cli_contract.Failure(
             cause="A target file already exists in `incoming/`",
             reaction="Nothing was written or overwritten. Accept or remove what is there, or "
@@ -189,6 +229,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     ),
     see_also=(
         "`raw/CONTRACT.md` \"Getting a URL in: `raw fetch`\" - the rules and why",
+        "`docs/reference/configuration.md` - the `sites` section, which tunes the site handlers",
         "`wikitool raw accept` - promotes the written files into `raw/`",
         "`instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts",
     ),
@@ -222,11 +263,12 @@ def raw_fetch_command(
 
     if name is not None and (not name.strip() or name in (".", "..") or any(c in name for c in "/\\")):
         fail(f"--name {name!r} is not a usable file stem: no path separators, not empty.")
-    stem = name or web_capture.stem_for(url)
+    plan = _plan(url, canonical=True)
+    stem = name or web_capture.stem_for(plan.url)
 
     try:
         response = web_capture.fetch(
-            url, _user_agent(), timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
+            plan.url, _user_agent(), timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
         )
     except ChemenuError as exc:
         fail(escape(str(exc)))
@@ -237,14 +279,16 @@ def raw_fetch_command(
         html_path, md_path = incoming / f"{stem}.html", incoming / f"{stem}.md"
         _refuse_existing([html_path, md_path])
         derivation = web_capture.derive_document(
-            response.body, response.final_url, html_path.name, response=response
+            response.body, response.final_url, html_path.name, response=response,
+            drop=plan.drop, site=plan.name, requested_url=plan.requested_url,
         )
         _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
         _warn(derivation, md_path)
+        _echo_checks(plan, derivation)
         success(escape(
-            f"Fetched {url} to {rel_path(html_path)}, {rel_path(md_path)}\n"
-            + _next_lines([html_path, md_path], url) + "\n"
-            + _TEASER_HINT.format(url=url)
+            f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n"
+            + _next_lines([html_path, md_path], plan.url) + "\n"
+            + _TEASER_HINT.format(url=plan.url)
         ))
         return
 
@@ -252,7 +296,7 @@ def raw_fetch_command(
     _refuse_existing([target])
     _write_exclusive([(target, response.body)])
     success(escape(
-        f"Fetched {url} to {rel_path(target)} ({kind or 'no content type'}, stored as received - "
+        f"Fetched {_fetched(plan)} to {rel_path(target)} ({kind or 'no content type'}, stored as received - "
         f"no text derived; retrieved {web_capture.utc_now()}).\n"
-        + _next_lines([target], url)
+        + _next_lines([target], plan.url)
     ))

tools/chemenu/config_shapes.py

@@ -31,6 +31,7 @@ every tool call without the venv.
 from __future__ import annotations
 
 import json
+import re
 from dataclasses import dataclass, field as dc_field
 from typing import Any, Callable, Optional
 
@@ -187,6 +188,62 @@ def extensions(value: Any) -> tuple[str, ...]:
     return tuple(sorted({str(ext).lower() for ext in value}))
 
 
+SITE_NOTE_MAX_LINES = 3
+SITE_NOTE_MAX_CHARS = 300
+_LOCALE = re.compile(r"[a-z]{2,3}(-[a-z0-9]{2,8})+")
+_NOTE_KEYS = ("host", "path", "lines")
+
+
+def site_notes(value: Any) -> list[dict]:
+    if not isinstance(value, list) or not all(isinstance(item, dict) for item in value):
+        raise TypeError("not a list of objects")
+    return [dict(item) for item in value]
+
+
+def site_locale(value: str) -> Optional[str]:
+    if not _LOCALE.fullmatch(value):
+        return "must be a lower-case locale such as `en-us` or `de-de`"
+    return None
+
+
+def site_names(value: list[str]) -> Optional[str]:
+    if any(not name.strip() for name in value):
+        return "must not contain an empty name"
+    return None
+
+
+def site_notes_problem(value: list[dict]) -> Optional[str]:
+    seen = set()
+    for index, note in enumerate(value, start=1):
+        where = f"entry {index}"
+        extra = sorted(set(note) - set(_NOTE_KEYS))
+        if extra:
+            return f"{where} has unknown key(s) {', '.join(extra)}; the keys are {', '.join(_NOTE_KEYS)}"
+        host = note.get("host")
+        if not isinstance(host, str) or not host.strip() or host != host.strip().lower():
+            return f"{where}: `host` must be a non-empty lower-case host name"
+        path = note.get("path", "")
+        if not isinstance(path, str):
+            return f"{where}: `path` must be a string"
+        try:
+            re.compile(path)
+        except re.error as exc:
+            return f"{where}: `path` is not a regular expression ({exc})"
+        lines = note.get("lines")
+        if not isinstance(lines, list) or not lines or not all(isinstance(line, str) for line in lines):
+            return f"{where}: `lines` must be a non-empty list of strings"
+        if len(lines) > SITE_NOTE_MAX_LINES:
+            return f"{where}: `lines` holds {len(lines)} lines, at most {SITE_NOTE_MAX_LINES} are allowed"
+        for line in lines:
+            if not line.strip() or "\n" in line or "\r" in line or len(line) > SITE_NOTE_MAX_CHARS:
+                return (f"{where}: each line must be non-empty, a single line, and at most "
+                        f"{SITE_NOTE_MAX_CHARS} characters")
+        if (host, path) in seen:
+            return f"{where}: the pattern ({host}, {path or 'any path'}) appears twice"
+        seen.add((host, path))
+    return None
+
+
 def _positive(value: int) -> Optional[str]:
     return None if value > 0 else "must be positive"
 
@@ -392,10 +449,56 @@ CALDAV = Shape(
     absent="Read only when `tasks.provider` is `caldav`.",
 )
 
+SITES = Shape(
+    key="sites",
+    title="Site handlers",
+    purpose="Tunes the site handlers `raw fetch` applies to the pages of well-known websites: "
+    "switches one off, picks the locale a handler fetches, and adds hint lines of the "
+    "instance's own per host.",
+    fields=(
+        Field(
+            "disabled", "array of strings", "The names of the site handlers that do not apply. "
+            "`raw fetch` then treats a page of that website as it does any other.",
+            default=[], json_type="array", coerce=str_list, check=site_names, tier=INSTANCE,
+            lenient="A name no handler has is an error when `raw fetch` reads the section, so a "
+            "typo cannot leave a handler on.",
+        ),
+        Field(
+            "locale", "string", "The locale a handler for a multi-language website fetches, in "
+            "the website's own spelling and lower case.",
+            default="en-us", json_type="string", coerce=str, check=site_locale,
+            empty_is_missing=True, lenient="An empty value means the default.", tier=INSTANCE,
+        ),
+        Field(
+            "notes", "array of objects", "Hint lines of the instance's own, printed as `CHECK` "
+            "lines after a fetch or an `--html` derivation of a page of that host. Each entry has "
+            f"`host` (exact, lower case), an optional `path` (a regular expression matched "
+            f"against the start of the URL path) and `lines` (one to {SITE_NOTE_MAX_LINES} "
+            f"single-line strings of at most {SITE_NOTE_MAX_CHARS} characters). A host and path "
+            "pattern appears once.",
+            default=[], json_type="array", coerce=site_notes, check=site_notes_problem,
+            tier=INSTANCE,
+        ),
+    ),
+    examples=({
+        "disabled": ["learn-microsoft-powershell-module"],
+        "locale": "en-us",
+        "notes": [{
+            "host": "learn.microsoft.com", "path": "^/[a-z-]+/azure/",
+            "lines": ["Check the Azure retirement notice at the top of the page."],
+        }],
+    },),
+    absent="Every handler applies, with the locale `en-us` and no further hint lines.",
+    malformed="An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A "
+    "broken section is never read as \"no handlers\".",
+)
+
 # The sections of the tiered settings files, `wikitool.json` and
 # `.wikitool-local.json` (`chemenu.settings`), read at call time so a test can
 # stand in a section of its own.
-SETTINGS_SECTIONS: tuple[Shape, ...] = (TELEMETRY, REMOTES, UPLOAD, TASKS, SUPERPRODUCTIVITY, CALDAV)
+SETTINGS_SECTIONS: tuple[Shape, ...] = (
+    TELEMETRY, REMOTES, UPLOAD, TASKS, SUPERPRODUCTIVITY, CALDAV, SITES,
+)
 
 # The sections `WIKITOOL_TASKS_CONFIG` replaces as a whole: a profile names a
 # tracker, and everything that configures one.

tools/chemenu/reference_doc.py

@@ -68,6 +68,7 @@ CONFIG_REGIONS = (
     ("reference-config-superproductivity", "SUPERPRODUCTIVITY"),
     ("reference-config-caldav", "CALDAV"),
     ("reference-config-upload", "UPLOAD"),
+    ("reference-config-sites", "SITES"),
 )
 
 _SLUG_RE = re.compile(r"[^a-z0-9]+")

tools/chemenu/sites/__init__.py

@@ -0,0 +1,251 @@
+"""The site handlers behind `wikitool raw fetch`: one module per website, each
+registering what is known about how that website serves its pages.
+
+A handler turns the URL a user typed into the website's standard form
+(`canonical`), names the page furniture that is not part of the text (`drop`),
+and says what the derivation cannot do for the reader (`checks`, printed as
+`CHECK` lines). The registry is code in this stack - a website is a module here,
+a registration and tests; the command, the header and the skill stay as they are.
+The instance does not write handlers: it switches them off, parametrises them
+(`locale`) and adds hint lines of its own (`notes`), all in the `sites` section
+of the settings.
+
+Every building block is data, not a function: a canonical form is a list of
+regular-expression replacements, `drop` a list of selectors, a check a text line
+with an optional condition. That is what lets a handler of an instance's own be
+declared as data later without a second mechanism. The core knows neither
+`typer` nor `rich` and raises `ValidationError`.
+"""
+from __future__ import annotations
+
+import importlib
+import pkgutil
+import re
+import urllib.parse
+from dataclasses import dataclass, replace
+from pathlib import Path
+from typing import Optional
+
+from chemenu import config_shapes, settings, web_capture
+from chemenu.errors import ValidationError
+from chemenu.web_capture import Rule
+
+MAX_CHECKS = 3
+LOCALE_PLACEHOLDER = "{locale}"
+
+
+@dataclass(frozen=True)
+class Rewrite:
+    """One step of a canonical form: `re.sub(pattern, replacement, url)`.
+    `{locale}` in `replacement` is the instance's locale setting."""
+
+    pattern: str
+    replacement: str
+
+    def __post_init__(self) -> None:
+        re.compile(self.pattern)
+
+    def apply(self, url: str, locale: str) -> str:
+        return re.sub(self.pattern, self.replacement.replace(LOCALE_PLACEHOLDER, locale), url, count=1)
+
+
+@dataclass(frozen=True)
+class Check:
+    """A hint line, with the condition under which it applies.
+
+    `meta` names a `<meta name=...>` of the page: the line applies only when it
+    has content, which replaces `{meta}` in `text`. `count` is a selector: the
+    line applies only when it matches at least one element, and `{count}` is
+    how many. With neither, the line always applies."""
+
+    text: str
+    meta: str = ""
+    count: Optional[Rule] = None
+
+    def line(self, document: web_capture._Element) -> Optional[str]:
+        text = self.text
+        if self.meta:
+            value = web_capture.meta_content(document, self.meta)
+            if not value:
+                return None
+            text = text.replace("{meta}", value)
+        if self.count is not None:
+            number = web_capture.count_matches(document, self.count)
+            if not number:
+                return None
+            text = text.replace("{count}", str(number))
+        return text
+
+
+@dataclass(frozen=True)
+class Site:
+    """One handler. `host` is matched exactly (lower case); `path` is a regular
+    expression matched against the start of the URL path, and the longest `path`
+    wins among the handlers of a host. `parent` names a handler this one
+    inherits from: its rewrites, rules and checks come first."""
+
+    name: str
+    host: str
+    summary: str
+    path: str = ""
+    parent: str = ""
+    rewrites: tuple[Rewrite, ...] = ()
+    drop: tuple[Rule, ...] = ()
+    checks: tuple[Check, ...] = ()
+
+    def __post_init__(self) -> None:
+        if not self.name or not self.host or self.host != self.host.lower():
+            raise ValueError(f"site {self.name!r}: needs a name and a lower-case host")
+        re.compile(self.path)
+        if len(self.checks) > MAX_CHECKS:
+            raise ValueError(f"site {self.name!r}: a handler has at most {MAX_CHECKS} checks, got {len(self.checks)}")
+
+    def matches(self, host: str, path: str) -> bool:
+        return host == self.host and re.match(self.path, path) is not None
+
+    def canonical(self, url: str, locale: str = "en-us") -> str:
+        for step in self.rewrites:
+            url = step.apply(url, locale)
+        return url
+
+    def check_lines(self, html: str) -> list[str]:
+        if not self.checks:
+            return []
+        document = web_capture.parse(html)
+        return [line for line in (check.line(document) for check in self.checks) if line]
+
+
+_REGISTRY: list[Site] = []
+
+
+def register(candidate: Site) -> Site:
+    """Add a handler. One that names a `parent` is stored flattened - the
+    parent's rewrites, rules and checks first - so a lookup never walks a chain."""
+    for known in _REGISTRY:
+        if known.name == candidate.name:
+            raise ValueError(f"site {candidate.name!r} is already registered")
+        if (known.host, known.path) == (candidate.host, candidate.path):
+            raise ValueError(
+                f"site {candidate.name!r} has the same host and path pattern as {known.name!r}: "
+                f"{candidate.host} {candidate.path or '(any path)'}"
+            )
+    if candidate.parent:
+        parent = next((s for s in _REGISTRY if s.name == candidate.parent), None)
+        if parent is None:
+            raise ValueError(f"site {candidate.name!r}: parent {candidate.parent!r} is not registered (yet)")
+        if parent.host != candidate.host:
+            raise ValueError(f"site {candidate.name!r}: parent {parent.name!r} is for another host")
+        candidate = replace(
+            candidate,
+            rewrites=parent.rewrites + candidate.rewrites,
+            drop=parent.drop + candidate.drop,
+            checks=parent.checks + candidate.checks,
+        )
+        if len(candidate.checks) > MAX_CHECKS:
+            raise ValueError(
+                f"site {candidate.name!r}: with its parent's checks it has {len(candidate.checks)}, "
+                f"at most {MAX_CHECKS} are allowed"
+            )
+    _REGISTRY.append(candidate)
+    return candidate
+
+
+def all_sites() -> list[Site]:
+    return list(_REGISTRY)
+
+
+def _host_and_path(url: str) -> tuple[str, str]:
+    parts = urllib.parse.urlsplit(url)
+    return (parts.hostname or "").lower(), urllib.parse.unquote(parts.path) or "/"
+
+
+def site_for(url: str, disabled: frozenset[str] = frozenset()) -> Optional[Site]:
+    """The handler for `url`: the enabled one of its host whose path pattern is
+    the longest among those that match, else None."""
+    host, path = _host_and_path(url)
+    found = [s for s in _REGISTRY if s.name not in disabled and s.matches(host, path)]
+    return max(found, key=lambda s: len(s.path), default=None)
+
+
+# --- the instance's part ----------------------------------------------------
+
+
+@dataclass(frozen=True)
+class Note:
+    host: str
+    path: str
+    lines: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class Preferences:
+    disabled: frozenset[str] = frozenset()
+    locale: str = "en-us"
+    notes: tuple[Note, ...] = ()
+
+
+def read_preferences(root: "Path | str | None" = None) -> Preferences:
+    """The `sites` section of the settings, or the defaults when it is absent.
+    A malformed section, or a name in `disabled` that no handler has, is a
+    `ValidationError` - a typo must not leave a handler on."""
+    section = settings.load(root, only=(config_shapes.SITES.key,)).section(config_shapes.SITES.key)
+    disabled = frozenset(section["disabled"])
+    unknown = sorted(disabled - {s.name for s in _REGISTRY})
+    if unknown:
+        raise ValidationError(
+            f"sites.disabled names {', '.join(unknown)}, which no handler has. "
+            f"The handlers are: {', '.join(sorted(s.name for s in _REGISTRY))}."
+        )
+    notes = tuple(
+        Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"]
+    )
+    return Preferences(disabled, section["locale"], notes)
+
+
+@dataclass(frozen=True)
+class Plan:
+    """What applies to one URL: the handler (if any), the URL to fetch, the rules
+    for the text, and the instance's notes for it."""
+
+    url: str
+    requested_url: str = ""
+    site: Optional[Site] = None
+    notes: tuple[str, ...] = ()
+
+    @property
+    def name(self) -> str:
+        return self.site.name if self.site else ""
+
+    @property
+    def drop(self) -> tuple[Rule, ...]:
+        return self.site.drop if self.site else ()
+
+    def checks(self, html: str) -> list[str]:
+        """The `CHECK` lines for the page: the handler's, then the instance's notes."""
+        lines = self.site.check_lines(html) if self.site else []
+        return lines + list(self.notes)
+
+
+def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan:
+    """The plan for `url`. `canonical=False` keeps the URL as given - for a page
+    that is already on disk, where the standard form would be a fiction."""
+    site = site_for(url, preferences.disabled)
+    target = url
+    if site is not None and canonical:
+        target = site.canonical(url, preferences.locale)
+    host, path = _host_and_path(target)
+    notes = tuple(
+        line
+        for note in preferences.notes
+        if note.host == host and re.match(note.path, path)
+        for line in note.lines
+    )
+    return Plan(target, url if target != url else "", site, notes)
+
+
+def _load_modules() -> None:
+    for module in pkgutil.iter_modules(__path__):
+        importlib.import_module(f"{__name__}.{module.name}")
+
+
+_load_modules()

tools/chemenu/sites/learn_microsoft.py

@@ -0,0 +1,56 @@
+"""learn.microsoft.com: Microsoft Learn, the home of the Microsoft product documentation.
+
+The same page exists in many locales and in one version per `view=`; the page
+furniture around the article (the action bar, the sign-in notice, the feedback
+footer) is rendered into the HTML for the script that hides it.
+"""
+from __future__ import annotations
+
+import re
+
+from chemenu.sites import Check, Rewrite, Rule, Site, register
+
+HOST = "learn.microsoft.com"
+LOCALE = r"[a-z]{2,3}(?:-[a-z0-9]{2,8})+"
+
+register(Site(
+    name="learn-microsoft",
+    host=HOST,
+    summary="Fetches the page in the instance's locale (`en-us` unless set otherwise), keeps the "
+    "`view=` of the URL, drops the page furniture, and names the page's Markdown source.",
+    rewrites=(Rewrite(rf"^(https?://{re.escape(HOST)}/){LOCALE}(?=/)", r"\1{locale}"),),
+    drop=(
+        Rule(id="ms--content-header"),
+        Rule(id="action-panel"),
+        Rule(id="article-header"),
+        Rule(attr="unauthorized-private-section"),
+        Rule(cls="authentication-determined"),
+        Rule(id="site-user-feedback-footer"),
+        Rule(id="ms--inline-notifications"),
+        Rule(id="assertive-live-region"),
+        Rule(id="polite-live-region"),
+    ),
+    checks=(
+        Check(
+            "Microsoft keeps the Markdown source of this page at {meta} - not fetched; fetch it "
+            "separately if the source wording is needed.",
+            meta="original_content_git_url",
+        ),
+    ),
+))
+
+register(Site(
+    name="learn-microsoft-powershell-module",
+    host=HOST,
+    summary="A PowerShell module's overview page: names the cmdlet pages it lists, which are not "
+    "fetched with it.",
+    path=rf"/{LOCALE}/powershell/module/[^/]+/?$",
+    parent="learn-microsoft",
+    checks=(
+        Check(
+            "This module page lists {count} cmdlet page(s) as links; they are not fetched - "
+            "ingest the ones the topic needs, each with its own raw fetch.",
+            count=Rule(tag="a", attr="data-linktype", value="relative-path", inside="td"),
+        ),
+    ),
+))

tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html

@@ -0,0 +1,194 @@
+<!DOCTYPE html>
+<html class="layout layout-holy-grail show-table-of-contents reference" lang="en-us" dir="ltr" data-authenticated="false" data-auth-status-determined="false" data-target="docs">
+<head>
+<meta charset="utf-8" />
+<title>OSLicense Module | Fixture Learn</title>
+<meta name="locale" content="en-us" />
+<meta name="original_content_git_url" content="https://example.invalid/docs-source/blob/live/docset/winserver2025-ps/OSLicense/OSLicense.md" />
+<script>window.appInsights = "SCRIPT-TEXT";</script>
+</head>
+<body class="luna-theme">
+<div class="header-holder has-default-focus"><header class="header">HEADER-TEXT</header></div>
+<div class="mainContainer uhfMainContainer">
+<div class="columns has-large-gaps">
+<div id="main-column" class="column is-full is-8-desktop">
+<main id="main" role="main" class="content">
+	<div id="ms--content-header" class="content-header default-focus border-bottom-none">
+		<button class="button button-sm contents-button" type="button" data-bi-name="contents-expand"><span>Table of contents</span></button>
+		<button class="button button-sm ap-collapse-behavior" type="button" data-bi-name="editor-mode-collapse"><span>Exit editor mode</span></button>
+	</div>
+	<div id="action-panel" role="region" aria-label="Action Panel" class="action-panel"></div>
+	<div id="article-header" class="background-color-body-medium">
+		<details id="article-header-breadcrumbs-overflow-popover"><summary>Breadcrumb overflow</summary></details>
+		<bread-crumbs id="article-header-breadcrumbs"><ol><li><a href="/en-us/">BREADCRUMB-TEXT</a></li></ol></bread-crumbs>
+		<div id="article-header-page-actions">
+			<button type="button" data-bi-name="ask-learn-assistant-entry"><span>Ask Learn</span></button>
+			<button type="button" data-bi-name="ask-learn-assistant-entry"><span>Ask Learn</span></button>
+			<button type="button" data-bi-name="ask-learn-assistant-entry"><span>Ask Learn</span></button>
+			<details id="article-header-page-actions-overflow">
+				<summary>More actions</summary>
+				<div class="popover">
+					<button type="button" id="ms--focus-mode-button"><span>Reading mode</span></button>
+					<button type="button" class="contents-button"><span>Table of contents</span></button>
+					<a id="lang-link-overflow" href="https://example.invalid/en-us/">Read in English</a>
+					<button type="button" class="collection" data-list-type="collection"><span>Add</span></button>
+					<button type="button" data-list-type="plan"><span>Add to Plans</span></button>
+					<a href="https://example.invalid/docs-source/blob/main/docset/winserver2025-ps/OSLicense/OSLicense.md" data-contenteditbtn>Edit</a>
+					<hr>
+					<button type="button" data-copy-markdown><span>Copy Markdown</span></button>
+					<button type="button" data-print-page><span>Print</span></button>
+				</div>
+			</details>
+		</div>
+	</div>
+	<!-- privateUnauthorizedTemplate is hidden by default -->
+	<div unauthorized-private-section data-bi-name="permission-content-unauthorized-private" hidden>
+		<hr class="hr margin-top-xs margin-bottom-sm" />
+		<div class="notification notification-info">
+			<div class="notification-content">
+				<p class="margin-top-none notification-title"><span>Note</span></p>
+				<p class="margin-top-none authentication-determined not-authenticated">
+					Access to this page requires authorization. You can try <a class="docs-sign-in" href="#">signing in</a> or <a class="docs-change-directory">changing directories</a>.
+				</p>
+				<p class="margin-top-none authentication-determined authenticated">
+					Access to this page requires authorization. You can try <a class="docs-change-directory">changing directories</a>.
+				</p>
+			</div>
+		</div>
+	</div>
+	<div class="content"></div>
+	<div class="content"><h1 class="margin-bottom-xs" data-no-chunk="">OSLicense Module</h1>
+	<div class="margin-block-xxs"><ul class="metadata page-metadata" data-bi-name="page info"></ul></div>
+	<nav id="center-doc-outline" class="doc-outline">OUTLINE-TEXT</nav>
+	<div data-moniker-range-group><div data-moniker="windowsserver2025-ps">
+	<p>The <code>OSLicense</code> module provides commands to read and configure the activation of an operating system.</p>
+	<h2 id="oslicense">OSLicense</h2>
+	<table class="table" data-no-chunk="">
+		<thead>
+			<tr>
+				<th>Cmdlet</th>
+				<th>Description</th>
+			</tr>
+		</thead>
+		<tbody>
+					<tr>
+	<td>
+		<a href="get-adlicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Get-ADLicenseInfo</a>
+	</td>
+	<td>
+		<p>Reads Active Directory-based activation information.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="get-kmslicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Get-KmsLicenseInfo</a>
+	</td>
+	<td>
+		<p>Reads Key Management Service information.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="get-oslicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Get-OSLicenseInfo</a>
+	</td>
+	<td>
+		<p>Reads operating system licensing information.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="get-subscriptionlicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Get-SubscriptionLicenseInfo</a>
+	</td>
+	<td>
+		<p>Reads subscription activation information.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="invoke-adlicense?view=windowsserver2025-ps" data-linktype="relative-path">Invoke-ADLicense</a>
+	</td>
+	<td>
+		<p>Runs Active Directory-based activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="invoke-kmslicense?view=windowsserver2025-ps" data-linktype="relative-path">Invoke-KmsLicense</a>
+	</td>
+	<td>
+		<p>Runs Key Management Service activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="invoke-oslicense?view=windowsserver2025-ps" data-linktype="relative-path">Invoke-OSLicense</a>
+	</td>
+	<td>
+		<p>Runs operating system activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="invoke-subscriptionlicense?view=windowsserver2025-ps" data-linktype="relative-path">Invoke-SubscriptionLicense</a>
+	</td>
+	<td>
+		<p>Runs subscription activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="set-kmslicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Set-KmsLicenseInfo</a>
+	</td>
+	<td>
+		<p>Configures Key Management Service activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="set-oslicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Set-OSLicenseInfo</a>
+	</td>
+	<td>
+		<p>Configures operating system activation.</p>
+
+	</td>
+</tr>
+					<tr>
+	<td>
+		<a href="set-subscriptionlicenseinfo?view=windowsserver2025-ps" data-linktype="relative-path">Set-SubscriptionLicenseInfo</a>
+	</td>
+	<td>
+		<p>Configures subscription activation.</p>
+
+	</td>
+</tr>
+		</tbody>
+	</table>
+	</div></div>
+	</div>
+	<section id="site-user-feedback-footer" class="font-size-sm margin-top-md">
+		<h2 id="feedback" class="margin-top-sm-tablet">Feedback</h2>
+		<p>Was this page helpful?</p>
+		<button type="button"><span>Yes</span></button>
+		<button type="button"><span>No</span></button>
+		<details id="mobile-help-popover"><summary>Need help with this topic?</summary><p>Want to try using Ask Learn to clarify or guide you through this topic? Suggest a fix?</p></details>
+	</section>
+	<div id="ms--inline-notifications" class="margin-block-xs"></div>
+	<div id="assertive-live-region" role="alert" aria-live="assertive" class="visually-hidden">LIVE-REGION-TEXT</div>
+	<div id="polite-live-region" role="status" aria-live="polite" class="visually-hidden">LIVE-REGION-TEXT</div>
+</main>
+</div>
+<aside id="layout-body-aside" class="layout-body-aside">RIGHT-RAIL-TEXT</aside>
+</div>
+</div>
+<footer id="footer">FOOTER-TEXT</footer>
+</body>
+</html>

tools/chemenu/tests/test_config_shapes.py

@@ -78,7 +78,7 @@ def test_the_telemetry_policy_reads_the_record():
 
 def test_one_record_per_section():
     assert [shape.key for shape in config_shapes.SETTINGS_SECTIONS] == [
-        "telemetry", "remotes", "upload", "tasks", "superproductivity", "caldav",
+        "telemetry", "remotes", "upload", "tasks", "superproductivity", "caldav", "sites",
     ]
 
 

tools/chemenu/tests/test_sites.py

@@ -0,0 +1,365 @@
+"""The site handlers behind `raw fetch` (`chemenu.sites`).
+
+The page under `fixtures/learn_microsoft/` is a rebuild of the structure of a
+learn.microsoft.com module overview page, written for this suite. The handlers
+are keyed by host name, so a fetch of that host is stood in for by replacing
+`web_capture.fetch` - the network itself is covered by `test_raw_fetch.py`.
+"""
+from __future__ import annotations
+
+import importlib
+import sys
+from pathlib import Path
+
+import pytest
+import typer
+import yaml
+
+from chemenu import sites, web_capture
+from chemenu.commands.raw.fetch import raw_fetch_command
+from chemenu.tests.conftest import write_settings
+
+FIXTURE = Path(__file__).parent / "fixtures/learn_microsoft/oslicense-module.html"
+PAGE = FIXTURE.read_bytes()
+MODULE_URL = "https://learn.microsoft.com/de-de/powershell/module/oslicense/?view=windowsserver2025-ps"
+STANDARD_URL = "https://learn.microsoft.com/en-us/powershell/module/oslicense/?view=windowsserver2025-ps"
+CMDLETS = [
+    "Get-ADLicenseInfo", "Get-KmsLicenseInfo", "Get-OSLicenseInfo", "Get-SubscriptionLicenseInfo",
+    "Invoke-ADLicense", "Invoke-KmsLicense", "Invoke-OSLicense", "Invoke-SubscriptionLicense",
+    "Set-KmsLicenseInfo", "Set-OSLicenseInfo", "Set-SubscriptionLicenseInfo",
+]
+CHROME = [
+    "requires authorization", "Exit editor mode", "Add to Plans", "Reading mode", "Copy Markdown",
+    "Ask Learn", "BREADCRUMB-TEXT", "OUTLINE-TEXT", "HEADER-TEXT", "FOOTER-TEXT", "RIGHT-RAIL-TEXT",
+    "LIVE-REGION-TEXT", "Was this page helpful", "SCRIPT-TEXT",
+]
+SOURCE_URL = "https://example.invalid/docs-source/blob/live/docset/winserver2025-ps/OSLicense/OSLicense.md"
+
+
+@pytest.fixture
+def tree(kb_dir):
+    root = kb_dir.parent
+    (root / "raw").mkdir()
+    (root / "incoming").mkdir(exist_ok=True)
+    return root
+
+
+@pytest.fixture
+def served(monkeypatch):
+    """Every fetch answers with the fixture page; the URLs asked for are recorded."""
+    asked: list[str] = []
+
+    def fake(url, user_agent, timeout=0, max_bytes=0):
+        asked.append(url)
+        return web_capture.Response(url, url, 200, "text/html; charset=utf-8", PAGE)
+
+    monkeypatch.setattr(web_capture, "fetch", fake)
+    return asked
+
+
+def _fetch(url=None, html=None, source_url=None, name=None):
+    return raw_fetch_command(url=url, html=html, source_url=source_url, name=name)
+
+
+def _refused(**kwargs) -> None:
+    with pytest.raises(typer.Exit) as excinfo:
+        _fetch(**kwargs)
+    assert excinfo.value.exit_code == 1
+
+
+def _md(path: Path) -> tuple[dict, str]:
+    head, _, text = path.read_text(encoding="utf-8")[4:].partition("\n---\n")
+    return yaml.safe_load(head), text
+
+
+def _checks(capsys) -> list[str]:
+    return [line[len("CHECK "):] for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")]
+
+
+# --- the Learn handlers ------------------------------------------------------
+
+
+def test_the_page_furniture_is_dropped_and_the_article_is_kept(tree, served):
+    _fetch(MODULE_URL, name="oslicense")
+    _, text = _md(tree / "incoming/oslicense.md")
+
+    for chrome in CHROME:
+        assert chrome not in text, chrome
+    assert "# OSLicense Module" in text
+    assert "## OSLicense" in text
+    for cmdlet in CMDLETS:
+        assert f"| [{cmdlet}](https://learn.microsoft.com/en-us/powershell/module/oslicense/{cmdlet.lower()}?view=windowsserver2025-ps) |" in text
+
+
+def test_the_same_page_of_another_host_keeps_its_furniture(tree, served):
+    _fetch("https://example.org/page", name="other")
+    header, text = _md(tree / "incoming/other.md")
+
+    for chrome in ("requires authorization", "Exit editor mode", "Add to Plans"):
+        assert chrome in text
+    assert "site" not in header and "requested_url" not in header
+
+
+def test_the_html_is_stored_exactly_as_the_server_sent_it(tree, served):
+    _fetch(MODULE_URL, name="oslicense")
+    assert (tree / "incoming/oslicense.html").read_bytes() == PAGE
+
+
+def test_the_standard_form_is_fetched_and_the_header_names_both_urls(tree, served):
+    _fetch(MODULE_URL, name="oslicense")
+    header, _ = _md(tree / "incoming/oslicense.md")
+
+    assert served == [STANDARD_URL]
+    assert header["url"] == STANDARD_URL
+    assert header["requested_url"] == MODULE_URL
+    assert header["site"] == "learn-microsoft-powershell-module"
+    assert header["fetched_by"] == "wikitool raw fetch"
+
+
+def test_a_url_already_in_standard_form_has_no_requested_url(tree, served):
+    _fetch(STANDARD_URL, name="oslicense")
+    header, _ = _md(tree / "incoming/oslicense.md")
+
+    assert served == [STANDARD_URL]
+    assert "requested_url" not in header
+    assert header["site"] == "learn-microsoft-powershell-module"
+
+
+def test_the_stem_comes_from_the_standard_form(tree, served):
+    _fetch("https://learn.microsoft.com/de-de/powershell/module/oslicense/get-oslicenseinfo?view=x")
+    assert (tree / "incoming/get-oslicenseinfo.md").is_file()
+
+
+def test_the_checks_name_the_source_and_the_cmdlet_pages(tree, served, capsys):
+    _fetch(MODULE_URL, name="oslicense")
+    lines = _checks(capsys)
+
+    assert len(lines) == 2
+    assert SOURCE_URL in lines[0]
+    assert "11 cmdlet page(s)" in lines[1]
+
+
+def test_a_page_of_the_general_handler_has_only_the_source_check(tree, served, capsys):
+    _fetch("https://learn.microsoft.com/de-de/azure/overview?view=x", name="azure")
+    header, text = _md(tree / "incoming/azure.md")
+    lines = _checks(capsys)
+
+    assert header["site"] == "learn-microsoft"
+    assert len(lines) == 1 and SOURCE_URL in lines[0]
+    assert "Exit editor mode" not in text
+
+
+def test_a_page_without_the_meta_tag_has_no_source_check(tree, monkeypatch, capsys):
+    body = PAGE.replace(b'name="original_content_git_url"', b'name="something_else"')
+    monkeypatch.setattr(
+        web_capture, "fetch",
+        lambda url, ua, timeout=0, max_bytes=0: web_capture.Response(url, url, 200, "text/html", body),
+    )
+    _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure")
+    assert _checks(capsys) == []
+
+
+def test_two_runs_over_the_same_bytes_give_the_same_text(tree, served):
+    _fetch(MODULE_URL, name="first")
+    _fetch(MODULE_URL, name="second")
+
+    def stable(name: str) -> list[str]:
+        lines = (tree / f"incoming/{name}.md").read_text(encoding="utf-8").splitlines()
+        return [line.replace(name, "X") for line in lines if not line.startswith("retrieved:")]
+
+    assert stable("first") == stable("second")
+
+
+# --- --html -----------------------------------------------------------------
+
+
+def test_html_applies_the_rules_but_keeps_the_url_as_typed(tree, monkeypatch, capsys):
+    def no_network(*args, **kwargs):
+        raise AssertionError("--html must not touch the network")
+
+    monkeypatch.setattr(web_capture, "fetch", no_network)
+    saved = tree / "incoming/oslicense.html"
+    saved.write_bytes(PAGE)
+    _fetch(html=saved, source_url=MODULE_URL)
+
+    header, text = _md(tree / "incoming/oslicense.md")
+    assert saved.read_bytes() == PAGE
+    assert header["url"] == MODULE_URL
+    assert "requested_url" not in header
+    assert header["site"] == "learn-microsoft-powershell-module"
+    assert "Exit editor mode" not in text and "# OSLicense Module" in text
+    assert len(_checks(capsys)) == 2
+
+
+# --- the instance's part -----------------------------------------------------
+
+
+def test_a_disabled_handler_does_not_apply(tree, served, monkeypatch, capsys):
+    write_settings(tree, monkeypatch, sites={
+        "disabled": ["learn-microsoft", "learn-microsoft-powershell-module"],
+    })
+    _fetch(MODULE_URL, name="oslicense")
+    header, text = _md(tree / "incoming/oslicense.md")
+
+    assert served == [MODULE_URL]
+    assert header["url"] == MODULE_URL
+    assert "site" not in header and "requested_url" not in header
+    assert "Exit editor mode" in text
+    assert _checks(capsys) == []
+
+
+def test_switching_off_the_special_handler_leaves_the_general_one(tree, served, monkeypatch, capsys):
+    write_settings(tree, monkeypatch, sites={"disabled": ["learn-microsoft-powershell-module"]})
+    _fetch(MODULE_URL, name="oslicense")
+    header, _ = _md(tree / "incoming/oslicense.md")
+
+    assert header["site"] == "learn-microsoft"
+    assert [SOURCE_URL in line for line in _checks(capsys)] == [True]
+
+
+def test_the_locale_is_a_setting(tree, served, monkeypatch):
+    write_settings(tree, monkeypatch, sites={"locale": "de-de"})
+    _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure")
+    assert served == ["https://learn.microsoft.com/de-de/azure/overview"]
+
+
+def test_an_instance_note_is_printed_as_a_check(tree, served, monkeypatch, capsys):
+    write_settings(tree, monkeypatch, sites={"notes": [
+        {"host": "learn.microsoft.com", "path": "^/en-us/powershell/", "lines": ["First note.", "Second note."]},
+        {"host": "learn.microsoft.com", "path": "^/en-us/azure/", "lines": ["Azure only."]},
+    ]})
+    _fetch(MODULE_URL, name="oslicense")
+    lines = _checks(capsys)
+
+    assert lines[-2:] == ["First note.", "Second note."]
+    assert "Azure only." not in lines
+
+
+def test_a_note_applies_to_a_host_without_a_handler_and_to_html(tree, served, monkeypatch, capsys):
+    write_settings(tree, monkeypatch, sites={"notes": [{"host": "example.org", "lines": ["Mind the paywall."]}]})
+    _fetch("https://example.org/page", name="page")
+    assert _checks(capsys) == ["Mind the paywall."]
+
+    saved = tree / "incoming/saved.html"
+    saved.write_bytes(PAGE)
+    _fetch(html=saved, source_url="https://example.org/other")
+    assert _checks(capsys) == ["Mind the paywall."]
+
+
+@pytest.mark.parametrize("sites_section", [
+    {"disabled": ["no-such-handler"]},
+    {"locale": "Deutsch"},
+    {"notes": [{"host": "example.org", "lines": ["1", "2", "3", "4"]}]},
+    {"notes": [{"host": "example.org", "lines": ["a\nb"]}]},
+    {"notes": [{"host": "Example.org", "lines": ["a"]}]},
+    {"notes": [{"host": "example.org", "path": "(", "lines": ["a"]}]},
+    {"notes": [{"host": "example.org", "lines": ["a"]}, {"host": "example.org", "lines": ["b"]}]},
+    {"notes": [{"host": "example.org", "lines": ["a"], "extra": 1}]},
+])
+def test_a_broken_sites_section_stops_before_anything_is_fetched(tree, served, monkeypatch, sites_section):
+    write_settings(tree, monkeypatch, sites=sites_section)
+    _refused(url="https://example.org/page")
+    assert served == []
+    assert list((tree / "incoming").iterdir()) == []
+
+
+# --- the registry ------------------------------------------------------------
+
+
+@pytest.fixture
+def registry(monkeypatch):
+    monkeypatch.setattr(sites, "_REGISTRY", list(sites._REGISTRY))
+    return sites._REGISTRY
+
+
+def test_the_longest_path_pattern_wins():
+    url = "https://learn.microsoft.com/en-us/powershell/module/x/"
+    assert sites.site_for(url).name == "learn-microsoft-powershell-module"
+    assert sites.site_for("https://learn.microsoft.com/en-us/powershell/module/x/y").name == "learn-microsoft"
+    assert sites.site_for("https://Learn.Microsoft.com/en-us/azure").name == "learn-microsoft"
+    assert sites.site_for("https://example.org/en-us/powershell/module/x/") is None
+
+
+def test_a_special_handler_inherits_from_the_general_one():
+    general = next(s for s in sites.all_sites() if s.name == "learn-microsoft")
+    special = next(s for s in sites.all_sites() if s.name == "learn-microsoft-powershell-module")
+    assert special.drop == general.drop
+    assert special.rewrites == general.rewrites
+    assert special.checks[: len(general.checks)] == general.checks
+    assert len(special.checks) == len(general.checks) + 1
+
+
+def test_a_duplicate_host_and_path_is_a_registration_error(registry):
+    with pytest.raises(ValueError, match="same host and path pattern"):
+        sites.register(sites.Site(name="again", host="learn.microsoft.com", summary="x"))
+
+
+def test_a_duplicate_name_is_a_registration_error(registry):
+    with pytest.raises(ValueError, match="already registered"):
+        sites.register(sites.Site(name="learn-microsoft", host="example.org", summary="x"))
+
+
+def test_a_duplicate_in_a_module_fails_at_import(registry, tmp_path, monkeypatch):
+    (tmp_path / "duplicate.py").write_text(
+        "from chemenu.sites import Site, register\n"
+        "register(Site(name='dup', host='learn.microsoft.com', summary='x'))\n",
+        encoding="utf-8",
+    )
+    monkeypatch.setattr(sites, "__path__", [*sites.__path__, str(tmp_path)])
+    monkeypatch.delitem(sys.modules, "chemenu.sites.duplicate", raising=False)
+    with pytest.raises(ValueError, match="same host and path pattern"):
+        importlib.import_module("chemenu.sites.duplicate")
+
+
+def test_a_parent_must_exist_and_share_the_host(registry):
+    with pytest.raises(ValueError, match="not registered"):
+        sites.register(sites.Site(name="child", host="example.org", summary="x", parent="nobody"))
+    with pytest.raises(ValueError, match="another host"):
+        sites.register(sites.Site(name="child", host="example.org", summary="x", parent="learn-microsoft"))
+
+
+def test_a_handler_has_at_most_three_checks():
+    with pytest.raises(ValueError, match="at most 3"):
+        sites.Site(name="n", host="example.org", summary="x", checks=(sites.Check("a"),) * 4)
+
+
+def test_an_inherited_handler_stays_within_the_limit(registry):
+    sites.register(sites.Site(name="base", host="example.org", summary="x", checks=(sites.Check("a"),) * 3))
+    with pytest.raises(ValueError, match="at most 3"):
+        sites.register(sites.Site(name="sub", host="example.org", summary="x", path="/a/", parent="base",
+                                  checks=(sites.Check("b"),)))
+
+
+# --- the building blocks -----------------------------------------------------
+
+
+def test_a_rule_needs_something_to_match_on():
+    with pytest.raises(ValueError):
+        web_capture.Rule()
+    with pytest.raises(ValueError):
+        web_capture.Rule(tag="a", inside="td", value="x")
+
+
+def test_rules_match_by_tag_id_class_attribute_and_ancestor():
+    document = web_capture.parse(
+        '<div id="a" class="x y"><p class="y" data-k="v">one</p></div><table><tr><td><a>two</a></td></tr></table><a>three</a>'
+    )
+    count = web_capture.count_matches
+    assert count(document, web_capture.Rule(id="a")) == 1
+    assert count(document, web_capture.Rule(cls="y")) == 2
+    assert count(document, web_capture.Rule(cls="x y")) == 0
+    assert count(document, web_capture.Rule(attr="data-k")) == 1
+    assert count(document, web_capture.Rule(attr="data-k", value="w")) == 0
+    assert count(document, web_capture.Rule(tag="a")) == 2
+    assert count(document, web_capture.Rule(tag="a", inside="td")) == 1
+
+
+def test_derive_text_without_rules_is_unchanged():
+    html = "<main><h1>T</h1><p>x</p><div id='gone'>y</div></main>"
+    assert web_capture.derive_text(html, "https://e.org/") == "# T\n\nx\n\ny"
+    assert web_capture.derive_text(html, "https://e.org/", (web_capture.Rule(id="gone"),)) == "# T\n\nx"
+
+
+def test_a_rewrite_is_a_regular_expression_with_the_locale_filled_in():
+    step = sites.Rewrite(r"^(https://e\.org/)[a-z-]+(?=/)", r"\1{locale}")
+    assert step.apply("https://e.org/de-de/a/b?view=1", "fr-fr") == "https://e.org/fr-fr/a/b?view=1"

tools/chemenu/web_capture.py

@@ -543,10 +543,79 @@ def _join(blocks: list[_Block]) -> str:
     return "\n\n".join(block.text for block in blocks)
 
 
-def derive_text(html: str, base_url: str) -> str:
+@dataclass(frozen=True)
+class Rule:
+    """A selector over the parsed tree, written as data: every field that is
+    set must match. `tag` is the element name, `id` its `id`, `cls` one token
+    of its `class`, `attr` an attribute that must be present (with `value`,
+    equal to it), `inside` the name of an ancestor element."""
+
+    tag: str = ""
+    id: str = ""
+    cls: str = ""
+    attr: str = ""
+    value: Optional[str] = None
+    inside: str = ""
+
+    def __post_init__(self) -> None:
+        if not (self.tag or self.id or self.cls or self.attr):
+            raise ValueError("a rule needs at least one of tag, id, cls, attr - it would match everything")
+        if self.value is not None and not self.attr:
+            raise ValueError("a rule's value needs an attr")
+
+    def matches(self, element: _Element) -> bool:
+        if self.tag and element.tag != self.tag:
+            return False
+        if self.id and element.attrs.get("id") != self.id:
+            return False
+        if self.cls and self.cls not in element.attrs.get("class", "").split():
+            return False
+        if self.attr:
+            if self.attr not in element.attrs:
+                return False
+            if self.value is not None and element.attrs[self.attr] != self.value:
+                return False
+        if self.inside:
+            parent = element.parent
+            while parent is not None and parent.tag != self.inside:
+                parent = parent.parent
+            if parent is None:
+                return False
+        return True
+
+
+def prune(root: _Element, rules: "tuple[Rule, ...]") -> None:
+    """Remove every element a rule matches, with everything below it."""
+    if not rules:
+        return
+    kept = []
+    for child in root.children:
+        if isinstance(child, _Element):
+            if any(rule.matches(child) for rule in rules):
+                continue
+            prune(child, rules)
+        kept.append(child)
+    root.children = kept
+
+
+def count_matches(document: _Element, rule: Rule) -> int:
+    return sum(1 for element in document.iter() if rule.matches(element))
+
+
+def meta_content(document: _Element, name: str) -> str:
+    """The `content` of the first `<meta name=...>`, stripped, else `""`."""
+    for element in document.iter():
+        if element.tag == "meta" and element.attrs.get("name") == name:
+            return element.attrs.get("content", "").strip()
+    return ""
+
+
+def derive_text(html: str, base_url: str, drop: "tuple[Rule, ...]" = ()) -> str:
     """The reading text of an HTML document: the content root's blocks as
-    Markdown-like text, with links made absolute against `base_url`."""
+    Markdown-like text, with links made absolute against `base_url`. `drop`
+    removes the elements its rules match before anything is read."""
     document = parse(html)
+    prune(document, drop)
     renderer = _Renderer(base_url)
     root = content_root(document)
     blocks = renderer.block(root) if root.tag != "#document" else renderer.blocks(root)
@@ -600,23 +669,32 @@ def derive_document(
     response: Optional[Response] = None,
     url: Optional[str] = None,
     derived_at: Optional[str] = None,
+    drop: "tuple[Rule, ...]" = (),
+    site: str = "",
+    requested_url: str = "",
 ) -> Derivation:
     """The `.md` beside a captured `.html`: the fixed header, then the derived
     text. With `response` it describes a fetch; without, a page a human saved
-    (`url` and `derived_at` then fill the header instead)."""
+    (`url` and `derived_at` then fill the header instead).
+
+    `drop`, `site` and `requested_url` come from a site handler
+    (`chemenu.sites`): its rules for the text, its name, and the URL the user
+    typed when the page was fetched under another one."""
     decoded = decode(body, response.content_type if response else None)
     document_title = title_of(parse(decoded.text))
-    text = derive_text(decoded.text, base_url)
+    text = derive_text(decoded.text, base_url, drop)
     charset = f"{decoded.charset} (from: {decoded.source})"
     if response is not None:
         fields = [
             ("fetched_by", "wikitool raw fetch"),
             ("url", response.url),
+            *([("requested_url", requested_url)] if requested_url else []),
             ("final_url", response.final_url),
             ("retrieved", derived_at or utc_now()),
             ("http_status", str(response.status)),
             ("content_type", response.content_type),
             ("charset", charset),
+            *([("site", site)] if site else []),
             ("title", document_title),
             ("derived_from", html_name),
         ]
@@ -626,6 +704,7 @@ def derive_document(
             ("url", url or ""),
             ("derived", derived_at or utc_now()),
             ("charset", charset),
+            *([("site", site)] if site else []),
             ("title", document_title),
             ("derived_from", html_name),
         ]