sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3
CI / verify (push) Successful in 7m52s
CI / pwsh (push) Successful in 2m26s
Release / release (push) Successful in 33s
Release / wiki-mirror (push) Skipped

Files changed:
- tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py
- tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/reference_doc.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-11 11:32:58 +02:00
1 parent 298b514119
commit 2213d68900
18 files changed
+1226 -27

No files matched your search

+24
View File
@@ -217,6 +217,8 @@ derived_from: post.html
---
```
(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler.)
The fields are capture metadata, not page frontmatter - `raw/` has no types, and nothing in the
stack reads the block back. There is deliberately no `author:`: HTML does not reliably say who
wrote a page, and a guessed author would be a claim about the source. It goes on the source page
@@ -242,6 +244,28 @@ carries `derived:` - when the text was derived - instead of `retrieved:`, `final
`http_status:` and `content_type:`: when the human saved the page, the tool does not know and does
not claim.
**A website the stack knows is fetched in its standard form and cleaned.** The same page of a
website often exists under many URLs - in every language, with tracking parameters - and carries
page furniture the browser hides but the HTML contains: an action bar, a sign-in notice, a
feedback footer. Left in, it ends up in the derived text, and with it in every claim made from the
page. A *site handler* - code in the stack, one module per website under `tools/chemenu/sites/`,
chosen by host name and the longest matching path pattern - states what is known about that
website: the standard form of its URLs, the page furniture to leave out of the derived text, and
hints about what the derivation cannot do for the reader. For `learn.microsoft.com` that is the
locale segment (`en-us`, so two people ingesting the German and the English page get the same
source) with the `view=` of the URL kept, and a hint naming the Markdown source the page is built
from, which is not fetched. A handler's hints are printed as `CHECK` lines, only when they apply
to the page; act on one before accepting the capture.
The standard form is what is fetched and what `url:` and the stem come from; the URL as typed is
kept in the header as `requested_url:`, and `site:` names the handler. The `.html` is still the
response body byte for byte - only the derived text is cleaned. `--html --url <url>` removes the
same furniture from a saved page but keeps the URL as typed, since no fetch happened. A URL no
handler claims behaves exactly as without handlers. The instance does not write handlers: its
`sites` section in the settings ([docs/reference/configuration.md](../docs/reference/configuration.md))
switches one off, sets the locale, and adds hint lines of its own per host and path pattern,
printed as `CHECK` lines too - at most three per pattern.
Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser
can be longer than the 200 characters under which `raw fetch` warns. The session that reads the
`.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a