sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/reference/commands/raw-material-and-uploads.md

@@ -25,6 +25,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **Synopsis**
 
 - `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
+- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
 - `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
 
 **Options**
@@ -38,6 +39,8 @@ Options:
                  a browser) instead of fetching - no network access
   --url <str>    With --html: the page's URL, for the header and relative links
   --name <str>   File stem in incoming/ instead of the one computed from the URL
+  --scope        Also fetch the subpages the site handler offers for this page,
+                 into the folder incoming/<stem>/
   -h, --help     Show this message and exit.
 ```
 
@@ -46,6 +49,7 @@ Options:
 ```bash
 tools/wikitool raw fetch https://example.org/blog/post
 tools/wikitool raw fetch https://example.org/ --name example-start
+tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
 tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post
 ```
 
@@ -56,6 +60,8 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
 - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
+- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -66,7 +72,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - Changes files: yes
 - Idempotent: no
-- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
+- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
 - Counts toward the session call budget: yes
 - Can reach the network: yes
 
@@ -77,6 +83,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
 
 **See also**