sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3
CI / verify (push) Successful in 7m52s
CI / pwsh (push) Successful in 2m26s
Release / release (push) Successful in 33s
Release / wiki-mirror (push) Skipped

Files changed:
- tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py
- tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-11 11:32:58 +02:00
1 parent 298b514119
commit 2213d68900
18 files changed
+1226 -27

No files matched your search

+64 -1
View File
@@ -28,6 +28,7 @@ something you run.
- [CalDAV](#caldav)
- [Profiles](#profiles)
- [MCP submissions](#mcp-submissions)
- [Site handlers](#site-handlers)
- [Files of earlier versions](#files-of-earlier-versions)
<!-- /wikitool:toc -->
@@ -89,7 +90,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
[upload](#mcp-submissions). `config set` names the sections it knows, and `doctor` reports a
[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
section it does not read as a `WARN`.
**One place per value.** A value in the file its tier does not name is an error naming the file,
@@ -406,6 +407,68 @@ Example:
Size and quota are your deployment's decision, not defaults worth copying blindly.
## Site handlers
`tools/wikitool raw fetch` knows how some websites serve their pages. For a URL of such a website a
*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
derived text and prints hints about the page as `CHECK` lines; today there are two, both for
`learn.microsoft.com`: `learn-microsoft` for every page of the host and
`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one.
The `sites` section is what the instance decides about them, and it is optional: without it every
handler applies, in the locale `en-us`.
- `disabled` switches handlers off by name. A page of that website is then handled as any other:
fetched under the URL as typed, with its page furniture in the derived text. A name no handler has
is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the
general one of the same website in force.
- `locale` is the language segment a handler for a multi-language website puts into the URL it
fetches. The header of the derived text keeps the URL as typed under `requested_url`.
- `notes` adds hint lines of the instance's own, printed after a fetch or an `--html` derivation of
a page of that host, whether or not a handler exists for it. An entry names a `host` (exact, lower
case), optionally a `path` (a regular expression matched against the start of the URL path, after
the standard form was applied) and one to three single-line `lines`. A host and path pattern appears
once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
output as it is, so it is for a reminder to the reader of the capture, never for an instruction
the tool should carry out.
<!-- wikitool:reference-config-sites -->
| Field | Tier | Type | Required | Default | Description |
|-------|------|------|----------|---------|-------------|
| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
Example:
```json
{
"sites": {
"disabled": [
"learn-microsoft-powershell-module"
],
"locale": "en-us",
"notes": [
{
"host": "learn.microsoft.com",
"path": "^/[a-z-]+/azure/",
"lines": [
"Check the Azure retirement notice at the top of the page."
]
}
]
}
}
```
<!-- /wikitool:reference-config-sites -->
All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none
of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
with exit 1 before it fetches anything.
## Files of earlier versions
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,