sites: raw fetch gets site handlers - standard URL form, per-website cleanup and CHECK hints as a code registry, learn.microsoft.com first, plus a sites settings section (#200); 10.0.0-beta.3
Files changed: - tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py - tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/reference_doc.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/learn_microsoft.py - tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
298b514119
commit
2213d68900
18 files changed
+1226
-27
No files matched your search
@@ -217,6 +217,8 @@ derived_from: post.html
|
||||
---
|
||||
```
|
||||
|
||||
(`requested_url:` follows `url:` and `site:` follows `charset:` on a page with a site handler.)
|
||||
|
||||
The fields are capture metadata, not page frontmatter - `raw/` has no types, and nothing in the
|
||||
stack reads the block back. There is deliberately no `author:`: HTML does not reliably say who
|
||||
wrote a page, and a guessed author would be a claim about the source. It goes on the source page
|
||||
@@ -242,6 +244,28 @@ carries `derived:` - when the text was derived - instead of `retrieved:`, `final
|
||||
`http_status:` and `content_type:`: when the human saved the page, the tool does not know and does
|
||||
not claim.
|
||||
|
||||
**A website the stack knows is fetched in its standard form and cleaned.** The same page of a
|
||||
website often exists under many URLs - in every language, with tracking parameters - and carries
|
||||
page furniture the browser hides but the HTML contains: an action bar, a sign-in notice, a
|
||||
feedback footer. Left in, it ends up in the derived text, and with it in every claim made from the
|
||||
page. A *site handler* - code in the stack, one module per website under `tools/chemenu/sites/`,
|
||||
chosen by host name and the longest matching path pattern - states what is known about that
|
||||
website: the standard form of its URLs, the page furniture to leave out of the derived text, and
|
||||
hints about what the derivation cannot do for the reader. For `learn.microsoft.com` that is the
|
||||
locale segment (`en-us`, so two people ingesting the German and the English page get the same
|
||||
source) with the `view=` of the URL kept, and a hint naming the Markdown source the page is built
|
||||
from, which is not fetched. A handler's hints are printed as `CHECK` lines, only when they apply
|
||||
to the page; act on one before accepting the capture.
|
||||
|
||||
The standard form is what is fetched and what `url:` and the stem come from; the URL as typed is
|
||||
kept in the header as `requested_url:`, and `site:` names the handler. The `.html` is still the
|
||||
response body byte for byte - only the derived text is cleaned. `--html --url <url>` removes the
|
||||
same furniture from a saved page but keeps the URL as typed, since no fetch happened. A URL no
|
||||
handler claims behaves exactly as without handlers. The instance does not write handlers: its
|
||||
`sites` section in the settings ([docs/reference/configuration.md](../docs/reference/configuration.md))
|
||||
switches one off, sets the locale, and adds hint lines of its own per host and path pattern,
|
||||
printed as `CHECK` lines too - at most three per pattern.
|
||||
|
||||
Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser
|
||||
can be longer than the 200 characters under which `raw fetch` warns. The session that reads the
|
||||
`.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a
|
||||
|
||||
Reference in new issue
Block a user