sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3

Files changed:
- tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py
- tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/reference/commands/README.md

@@ -79,7 +79,7 @@ Full records: [provenance.md](provenance.md)
 
 Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md)
 
-- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+- [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 - [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote.
 - [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
 - [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.

docs/reference/commands/raw-material-and-uploads.md

@@ -20,7 +20,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
 <!-- wikitool:reference-commands -->
 ## `raw fetch`
 
-Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
+Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture.
 
 **Synopsis**
 
@@ -52,9 +52,10 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **Behaviour**
 
 - Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw/<YYYY>/<MM>/<stem>/`.
-- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text.
+- The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
+- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -73,6 +74,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - `0` - success
 - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
+- `1` - The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -80,6 +82,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 **See also**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts