sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3

Files changed:
- tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py
- tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/ingest-a-source.md

@@ -16,6 +16,12 @@ pick a directory:
 - **A web page** needs no download of your own. Name the URL to the agent, and
   `tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
   text derived from it with a header recording where and when it was fetched.
+  For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the
+  same page in the website's standard form (the English one, whatever language your link was in),
+  leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints
+  `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
+  capture. Your instance can switch such a handler off, choose another language and add hint lines
+  of its own per website; see [the configuration reference](../reference/configuration.md).
 - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the

docs/reference/commands/README.md

@@ -79,7 +79,7 @@ Full records: [provenance.md](provenance.md)
 
 Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
 
-- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 - [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
 - [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
 - [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.

docs/reference/commands/raw-material-and-uploads.md

@@ -20,7 +20,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
 <!-- wikitool:reference-commands -->
 ## `raw fetch`
 
-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 
 **Synopsis**
 
@@ -52,9 +52,10 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **Behaviour**
 
 - Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.
-- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text.
+- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
+- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -73,6 +74,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - `0` - success
 - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
+- `1` - The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -80,6 +82,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **See also**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
 

docs/reference/configuration.md

@@ -28,6 +28,7 @@ something you run.
   - [CalDAV](#caldav)
   - [Profiles](#profiles)
 - [MCP submissions](#mcp-submissions)
+- [Site handlers](#site-handlers)
 - [Files of earlier versions](#files-of-earlier-versions)
 <!-- /wikitool:toc -->
 
@@ -89,7 +90,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
 `instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
 the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
 [remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
-[upload](#mcp-submissions). `config set` names the sections it knows, and `doctor` reports a
+[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
 section it does not read as a `WARN`.
 
 **One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -406,6 +407,68 @@ Example:
 
 Size and quota are your deployment's decision, not defaults worth copying blindly.
 
+## Site handlers
+
+`tools/wikitool raw fetch` knows how some websites serve their pages. For a URL of such a website a
+*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
+derived text and prints hints about the page as `CHECK` lines; today there are two, both for
+`learn.microsoft.com`: `learn-microsoft` for every page of the host and
+`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
+naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one.
+The `sites` section is what the instance decides about them, and it is optional: without it every
+handler applies, in the locale `en-us`.
+
+- `disabled` switches handlers off by name. A page of that website is then handled as any other:
+  fetched under the URL as typed, with its page furniture in the derived text. A name no handler has
+  is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the
+  general one of the same website in force.
+- `locale` is the language segment a handler for a multi-language website puts into the URL it
+  fetches. The header of the derived text keeps the URL as typed under `requested_url`.
+- `notes` adds hint lines of the instance's own, printed after a fetch or an `--html` derivation of
+  a page of that host, whether or not a handler exists for it. An entry names a `host` (exact, lower
+  case), optionally a `path` (a regular expression matched against the start of the URL path, after
+  the standard form was applied) and one to three single-line `lines`. A host and path pattern appears
+  once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
+  output as it is, so it is for a reminder to the reader of the capture, never for an instruction
+  the tool should carry out.
+
+<!-- wikitool:reference-config-sites -->
+| Field | Tier | Type | Required | Default | Description |
+|-------|------|------|----------|---------|-------------|
+| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
+| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
+| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
+
+- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
+- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
+
+Example:
+
+```json
+{
+  "sites": {
+    "disabled": [
+      "learn-microsoft-powershell-module"
+    ],
+    "locale": "en-us",
+    "notes": [
+      {
+        "host": "learn.microsoft.com",
+        "path": "^/[a-z-]+/azure/",
+        "lines": [
+          "Check the Azure retirement notice at the top of the page."
+        ]
+      }
+    ]
+  }
+}
+```
+<!-- /wikitool:reference-config-sites -->
+
+All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none
+of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
+with exit 1 before it fetches anything.
+
 ## Files of earlier versions
 
 Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,