sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/ingest-a-source.md

@@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
 the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
 it and where it waits for you.
 
+<!-- wikitool:toc -->
+## Contents
+
+- [What you prepare](#what-you-prepare)
+- [What you say](#what-you-say)
+- [Where the agent asks you](#where-the-agent-asks-you)
+- [What you get back](#what-you-get-back)
+- [A document submitted from outside](#a-document-submitted-from-outside)
+- [Related](#related)
+<!-- /wikitool:toc -->
+
 ## What you prepare
 
 Put the material into `incoming/` at the root of the instance. You don't have to classify it or
@@ -22,6 +33,11 @@ pick a directory:
   `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
   capture. Your instance can switch such a handler off, choose another language and add hint lines
   of its own per website; see [the configuration reference](../reference/configuration.md).
+  Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
+  lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
+  fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
+  any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
+  page that offers more than the instance's `max_pages`.
 - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the

docs/how-to/write-a-site-handler.md

@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
 website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
 `checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
 
+A page that is an index - a module overview, a table of contents - can name its subpages, so that
+`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
+
+```json
+"subpages": {"links": {"tag": "a", "inside": "td"}}
+```
+
+The links are the `<a>` elements inside a table cell; only those on the same host and under the
+page's own path count. A page with such a handler prints a `CHECK` line with their number when it
+is fetched without `--scope`.
+
 ## 3. Try it
 
 Fetch a page again, or derive a saved one without a fetch:

docs/reference/commands/raw-material-and-uploads.md

@@ -25,6 +25,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **Synopsis**
 
 - `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
+- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
 - `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
 
 **Options**
@@ -38,6 +39,8 @@ Options:
                  a browser) instead of fetching - no network access
   --url <str>    With --html: the page's URL, for the header and relative links
   --name <str>   File stem in incoming/ instead of the one computed from the URL
+  --scope        Also fetch the subpages the site handler offers for this page,
+                 into the folder incoming/<stem>/
   -h, --help     Show this message and exit.
 ```
 
@@ -46,6 +49,7 @@ Options:
 ```bash
 tools/wikitool raw fetch https://example.org/blog/post
 tools/wikitool raw fetch https://example.org/ --name example-start
+tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
 tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post
 ```
 
@@ -56,6 +60,8 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
 - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
+- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -66,7 +72,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - Changes files: yes
 - Idempotent: no
-- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
+- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
 - Counts toward the session call budget: yes
 - Can reach the network: yes
 
@@ -77,6 +83,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
 
 **See also**

docs/reference/configuration.md

@@ -413,8 +413,8 @@ Size and quota are your deployment's decision, not defaults worth copying blindl
 *site handler* fetches the page in the website's standard form, leaves the page furniture out of the
 derived text and prints hints about the page as `CHECK` lines; today there are two, both for
 `learn.microsoft.com`: `learn-microsoft` for every page of the host and
-`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
-naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/`
+`learn-microsoft-powershell-module` for a PowerShell module's overview page, which offers the cmdlet
+pages it lists as subpages for `raw fetch --scope`. Handlers are JSON files: the stack's, and the instance's own under `sites/`
 (see [How to write a site handler](../how-to/write-a-site-handler.md)).
 The `sites` section is what the instance decides about the handlers, and it is optional: without it every
 handler applies, in the locale `en-us`.
@@ -432,6 +432,9 @@ handler applies, in the locale `en-us`.
   once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
   output as it is, so it is for a reminder to the reader of the capture, never for an instruction
   the tool should carry out.
+- `max_pages` and `pause_seconds` bound `raw fetch --scope`: the most subpages one page may offer
+  before the fetch is refused, and the pause before each subpage request, which keeps the load on
+  the website low.
 
 <!-- wikitool:reference-config-sites -->
 | Field | Tier | Type | Required | Default | Description |
@@ -439,6 +442,8 @@ handler applies, in the locale `en-us`.
 | `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
 | `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
 | `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
+| `max_pages` | instance | positive integer | no | `200` | The most subpages `raw fetch --scope` fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted. |
+| `pause_seconds` | instance | number | no | `1` | The pause, in seconds, before each subpage request of `raw fetch --scope`. `0` means none. A number written as a string is accepted. |
 
 - **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
 - **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
@@ -452,6 +457,8 @@ Example:
       "learn-microsoft-powershell-module"
     ],
     "locale": "en-us",
+    "max_pages": 200,
+    "pause_seconds": 1,
     "notes": [
       {
         "host": "learn.microsoft.com",