sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3
Files changed: - tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py - tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/reference_doc.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/learn_microsoft.py - tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
298b514119
commit
2213d68900
18 files changed
+1226
-27
No files matched your search
@@ -28,6 +28,7 @@ something you run.
|
||||
- [CalDAV](#caldav)
|
||||
- [Profiles](#profiles)
|
||||
- [MCP submissions](#mcp-submissions)
|
||||
- [Site handlers](#site-handlers)
|
||||
- [Files of earlier versions](#files-of-earlier-versions)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
@@ -89,7 +90,7 @@ Each file is a JSON object of sections, one per feature, beside two keys of its
|
||||
`instance_id` sits in `wikitool.json` only. The first `wikitool config set` creates the file and
|
||||
the id; the id names the directory of the secrets file below. The sections are [telemetry](#telemetry),
|
||||
[remotes](#allowed-push-urls), [tasks, superproductivity and caldav](#task-tracker) and
|
||||
[upload](#mcp-submissions). `config set` names the sections it knows, and `doctor` reports a
|
||||
[upload](#mcp-submissions) and [sites](#site-handlers). `config set` names the sections it knows, and `doctor` reports a
|
||||
section it does not read as a `WARN`.
|
||||
|
||||
**One place per value.** A value in the file its tier does not name is an error naming the file,
|
||||
@@ -406,6 +407,68 @@ Example:
|
||||
|
||||
Size and quota are your deployment's decision, not defaults worth copying blindly.
|
||||
|
||||
## Site handlers
|
||||
|
||||
`tools/wikitool raw fetch` knows how some websites serve their pages. For a URL of such a website a
|
||||
*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
|
||||
derived text and prints hints about the page as `CHECK` lines; today there are two, both for
|
||||
`learn.microsoft.com`: `learn-microsoft` for every page of the host and
|
||||
`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
|
||||
naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one.
|
||||
The `sites` section is what the instance decides about them, and it is optional: without it every
|
||||
handler applies, in the locale `en-us`.
|
||||
|
||||
- `disabled` switches handlers off by name. A page of that website is then handled as any other:
|
||||
fetched under the URL as typed, with its page furniture in the derived text. A name no handler has
|
||||
is an error, so a typo cannot leave a handler on. Switching off a special handler leaves the
|
||||
general one of the same website in force.
|
||||
- `locale` is the language segment a handler for a multi-language website puts into the URL it
|
||||
fetches. The header of the derived text keeps the URL as typed under `requested_url`.
|
||||
- `notes` adds hint lines of the instance's own, printed after a fetch or an `--html` derivation of
|
||||
a page of that host, whether or not a handler exists for it. An entry names a `host` (exact, lower
|
||||
case), optionally a `path` (a regular expression matched against the start of the URL path, after
|
||||
the standard form was applied) and one to three single-line `lines`. A host and path pattern appears
|
||||
once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
|
||||
output as it is, so it is for a reminder to the reader of the capture, never for an instruction
|
||||
the tool should carry out.
|
||||
|
||||
<!-- wikitool:reference-config-sites -->
|
||||
| Field | Tier | Type | Required | Default | Description |
|
||||
|-------|------|------|----------|---------|-------------|
|
||||
| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
|
||||
| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
|
||||
| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
|
||||
|
||||
- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
|
||||
- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"sites": {
|
||||
"disabled": [
|
||||
"learn-microsoft-powershell-module"
|
||||
],
|
||||
"locale": "en-us",
|
||||
"notes": [
|
||||
{
|
||||
"host": "learn.microsoft.com",
|
||||
"path": "^/[a-z-]+/azure/",
|
||||
"lines": [
|
||||
"Check the Azure retirement notice at the top of the page."
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
<!-- /wikitool:reference-config-sites -->
|
||||
|
||||
All three fields are instance-tier: what a website needs cleaned is the same in every clone, and none
|
||||
of them names a program or opens a gate. A section that does not read correctly stops `raw fetch`
|
||||
with exit 1 before it fetches anything.
|
||||
|
||||
## Files of earlier versions
|
||||
|
||||
Until version 10 every feature above had a file of its own at the root: `.wikitool-telemetry.json`,
|
||||
|
||||
Reference in new issue
Block a user