sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

CHANGES.md

@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 ---
 
-## 10.0.0-beta.5 - 2026-10-11 - doctor: die Beschreibung der Strukturprüfung nennt sites/CONTRACT.md, die doctor seit den Site-Handlern der Instanz verlangt
+## 10.0.0-beta.6 - 2026-10-11 - raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/
 
 **Author:** Torben Nehmer
 
@@ -30,6 +30,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 - Einstellungen in drei Stufen: wikitool.json (Instanz), .wikitool-local.json (Checkout), Secrets nur per Verweis - auch über die 1Password-CLI; neues Kommando config set
 - Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen
 - Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind
+- raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/
 
 **Low impact**
 - Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
@@ -46,6 +47,10 @@ Neue Einstellungen liegen in `wikitool.json` (eingecheckt, gleich in jedem Klon)
 
 Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt Hinweise als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind JSON-Dateien nach `tools/chemenu/sites/site.schema.json`: die des Stacks unter `tools/chemenu/sites/handlers/` (zuerst `learn-microsoft` und `learn-microsoft-powershell-module`), die der Instanz unter `sites/`; ein Handler der Instanz mit demselben Host und Pfad ersetzt den des Stacks. `doctor` meldet eine unbrauchbare Datei als FAIL, und `raw fetch` bricht für deren Host mit Exit 1 ab, bevor etwas geholt wird. Der Header nennt `site:` (`<name> (instance)` bei der Instanz) und `requested_url`; das `.html` bleibt byte-genau. Der Abschnitt `sites` der Einstellungen schaltet Handler ab, wählt das Gebietsschema und nimmt Hinweiszeilen je Host-Muster auf. Regeln: `sites/CONTRACT.md`.
 
+### `raw fetch --scope`: ein Bündel aus Index und Unterseiten
+
+Bietet der Site-Handler einer Seite Unterseiten an (Baustein `subpages` in der Handler-Datei; der Modulindex von learn.microsoft.com bietet seine Cmdlet-Seiten an), nennt `raw fetch` das als eine `CHECK`-Zeile, und `raw fetch <url> --scope` holt Index und Unterseiten als einen Ordner `incoming/<name>/` mit einem Paar je Seite und dem Manifest `_scope.json`; `raw accept incoming/<name>` nimmt ihn unverändert an. Geholt wird nur, was der Handler benennt, nie von einem anderen Host. Vorher prüft der Befehl `robots.txt` und die Obergrenze `sites.max_pages` (Standard 200); jeder Fehler bricht ab, bevor etwas in `incoming/` liegt. Zwischen den Unterseiten wartet er `sites.pause_seconds` (Standard 1).
+
 ### Altlasten der Konfiguration bereinigt (#208)
 
 Die vier Einzeldateien `.wikitool-remotes.json`, `.wikitool-telemetry.json`, `.wikitool-upload.json` und `.wikitool-tasks.json` gibt es nicht mehr; ihre Felder sind die Abschnitte `remotes`, `telemetry`, `upload`, `tasks`, `superproductivity` und `caldav` der drei Stufen. Zugangsdaten sind dort nur noch Verweise. Der Code liest die alten Dateien nicht mehr: ein Kommando, das die Einstellung braucht, endet mit Exit 42 und nennt `tools/wikitool config migrate`, `doctor` meldet FAIL, der Telemetrie-Hook bleibt still. `config migrate` läuft einmal je Checkout, ist wiederholbar und lässt bei jedem Abbruch die alten Dateien unverändert. `WIKITOOL_TASKS_CONFIG` zeigt auf ein Profil im Format von `.wikitool-local.json`. Nur `.wikitool-tools.json` bleibt eine eigene Datei.

VERSION

@@ -1 +1 @@
-10.0.0-beta.5
+10.0.0-beta.6

docs/how-to/ingest-a-source.md

@@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
 the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
 it and where it waits for you.
 
+<!-- wikitool:toc -->
+## Contents
+
+- [What you prepare](#what-you-prepare)
+- [What you say](#what-you-say)
+- [Where the agent asks you](#where-the-agent-asks-you)
+- [What you get back](#what-you-get-back)
+- [A document submitted from outside](#a-document-submitted-from-outside)
+- [Related](#related)
+<!-- /wikitool:toc -->
+
 ## What you prepare
 
 Put the material into `incoming/` at the root of the instance. You don't have to classify it or
@@ -22,6 +33,11 @@ pick a directory:
   `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
   capture. Your instance can switch such a handler off, choose another language and add hint lines
   of its own per website; see [the configuration reference](../reference/configuration.md).
+  Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
+  lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
+  fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
+  any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
+  page that offers more than the instance's `max_pages`.
 - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the

docs/how-to/write-a-site-handler.md

@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
 website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
 `checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
 
+A page that is an index - a module overview, a table of contents - can name its subpages, so that
+`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
+
+```json
+"subpages": {"links": {"tag": "a", "inside": "td"}}
+```
+
+The links are the `<a>` elements inside a table cell; only those on the same host and under the
+page's own path count. A page with such a handler prints a `CHECK` line with their number when it
+is fetched without `--scope`.
+
 ## 3. Try it
 
 Fetch a page again, or derive a saved one without a fetch:

docs/reference/commands/raw-material-and-uploads.md

@@ -25,6 +25,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **Synopsis**
 
 - `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
+- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
 - `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
 
 **Options**
@@ -38,6 +39,8 @@ Options:
                  a browser) instead of fetching - no network access
   --url <str>    With --html: the page's URL, for the header and relative links
   --name <str>   File stem in incoming/ instead of the one computed from the URL
+  --scope        Also fetch the subpages the site handler offers for this page,
+                 into the folder incoming/<stem>/
   -h, --help     Show this message and exit.
 ```
 
@@ -46,6 +49,7 @@ Options:
 ```bash
 tools/wikitool raw fetch https://example.org/blog/post
 tools/wikitool raw fetch https://example.org/ --name example-start
+tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
 tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post
 ```
 
@@ -56,6 +60,8 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
 - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
+- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -66,7 +72,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 
 - Changes files: yes
 - Idempotent: no
-- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
+- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
 - Counts toward the session call budget: yes
 - Can reach the network: yes
 
@@ -77,6 +83,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - `1` - A target file already exists in `incoming/`
 - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
 
 **See also**

docs/reference/configuration.md

@@ -413,8 +413,8 @@ Size and quota are your deployment's decision, not defaults worth copying blindl
 *site handler* fetches the page in the website's standard form, leaves the page furniture out of the
 derived text and prints hints about the page as `CHECK` lines; today there are two, both for
 `learn.microsoft.com`: `learn-microsoft` for every page of the host and
-`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
+`learn-microsoft-powershell-module` for a PowerShell module's overview page, which offers the cmdlet
-naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/`
+pages it lists as subpages for `raw fetch --scope`. Handlers are JSON files: the stack's, and the instance's own under `sites/`
 (see [How to write a site handler](../how-to/write-a-site-handler.md)).
 The `sites` section is what the instance decides about the handlers, and it is optional: without it every
 handler applies, in the locale `en-us`.
@@ -432,6 +432,9 @@ handler applies, in the locale `en-us`.
   once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
   output as it is, so it is for a reminder to the reader of the capture, never for an instruction
   the tool should carry out.
+- `max_pages` and `pause_seconds` bound `raw fetch --scope`: the most subpages one page may offer
+  before the fetch is refused, and the pause before each subpage request, which keeps the load on
+  the website low.
 
 <!-- wikitool:reference-config-sites -->
 | Field | Tier | Type | Required | Default | Description |
@@ -439,6 +442,8 @@ handler applies, in the locale `en-us`.
 | `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
 | `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
 | `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
+| `max_pages` | instance | positive integer | no | `200` | The most subpages `raw fetch --scope` fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted. |
+| `pause_seconds` | instance | number | no | `1` | The pause, in seconds, before each subpage request of `raw fetch --scope`. `0` means none. A number written as a string is accepted. |
 
 - **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
 - **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
@@ -452,6 +457,8 @@ Example:
       "learn-microsoft-powershell-module"
     ],
     "locale": "en-us",
+    "max_pages": 200,
+    "pause_seconds": 1,
     "notes": [
       {
         "host": "learn.microsoft.com",

raw/CONTRACT.md

@@ -270,6 +270,20 @@ resolution and what a broken file does are in [sites/CONTRACT.md](../sites/CONTR
 switches a handler off, sets the locale, and adds hint lines of its own per host and path pattern,
 printed as `CHECK` lines too - at most three per pattern.
 
+**A page that is an index can be fetched together with its subpages.** A handler may name which
+links of its page are subpages (the `subpages` field of [sites/CONTRACT.md](../sites/CONTRACT.md));
+for the module overview of `learn.microsoft.com` those are the cmdlet pages it lists. `raw fetch`
+then prints one `CHECK` line with their number, and nothing more is fetched. `raw fetch <url>
+--scope` fetches the page and its subpages as one bundle - a folder `incoming/<stem>/` holding a
+`.html`/`.md` pair per page, written as for a single page, and `_scope.json`, a manifest of the
+pages with their URL, retrieval time and the SHA-256 of the received HTML. `raw accept
+incoming/<stem>` takes the folder as one source. Only the URLs the handler names are fetched, on
+the host of the page and under the path the handler allows, never a link followed further. The
+`robots.txt` of the host is honoured for the subpages: one it forbids, or one that cannot be read,
+refuses the whole fetch. So do more subpages than `sites.max_pages`, a subpage that fails or is not
+HTML, and a folder of that name in `incoming/` or `raw/`; everything is checked before the first
+write, and a failure leaves `incoming/` as it was.
+
 Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser
 can be longer than the 200 characters under which `raw fetch` warns. The session that reads the
 `.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a

sites/CONTRACT.md

@@ -23,6 +23,7 @@ as `site: <name> (instance)`.
 | `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` |
 | `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received |
 | `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page |
+| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own |
 
 A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class`
 (one token of the element's classes), `attr` (an attribute that must be present, equal to `value`
@@ -35,6 +36,14 @@ it applies only when at least one element matches the selector, and `{count}` is
 neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have
 the field that fills it.
 
+`subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `<a>` elements whose
+`href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the
+path of the page itself, with a trailing slash, and a value starting with `/` is a literal path.
+A link is resolved against the page's URL, loses its fragment and counts only on the page's own
+host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched
+because of it unless the user runs `raw fetch <url> --scope`; without it the page prints one
+`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched.
+
 `tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is
 refused.
 

tools/CONTRACT.md

@@ -1425,13 +1425,14 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **SYNOPSIS**
 
 - `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
+- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
 - `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
 
 **PROPERTIES**
 
 - effect: write
 - idempotent: no
-- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
+- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
 - budget: counted
 - network: yes
 
@@ -1439,6 +1440,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 
 - `tools/wikitool raw fetch https://example.org/blog/post`
 - `tools/wikitool raw fetch https://example.org/ --name example-start`
+- `tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope`
 - `tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post`
 
 **EXIT STATUS**
@@ -1448,6 +1450,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - 1    The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - 1    A target file already exists in `incoming/`
 - 1    The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
+- 1    raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
 - 1    raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
 
 **ON FAILURE**
@@ -1456,6 +1459,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once
 - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
 - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
+- raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget -> Nothing was written. Read the message: fetch the page without `--scope`, raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user
 - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
 
 **NEVER**
@@ -1470,6 +1474,8 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
 - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
+- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.

tools/README.md

@@ -118,7 +118,7 @@ tools/
     kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
     corpus_diff.py         invariant comparison of kb/ between two revisions
     web_capture.py         `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
-    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
+    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
     filters/               the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
     repo_capture.py        `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
     guideline_export.py    `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached

tools/chemenu/commands/raw/fetch.py

@@ -7,23 +7,38 @@ the caller is told.
 """
 from __future__ import annotations
 
+import hashlib
+import json
+import os
+import shutil
+import tempfile
+import time
+import urllib.parse
 from pathlib import Path
-from typing import Optional
+from typing import Annotated, Optional
 
 import typer
 from rich.markup import escape
 
-from chemenu import cli_contract, sites, web_capture
+from chemenu import cli_contract, config, sites, web_capture
-from chemenu.commands._util import fail, rel_path, success
+from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
 from chemenu.errors import ChemenuError
 from chemenu.version import VersionError, read_version
 from chemenu.commands.raw._common import (
+    _folder_collision_message,
     _incoming_dir,
+    _occupied_stems,
     _resolve,
+    _shard_dir,
+    _unusable_name,
     _validate_under_incoming,
     _write_exclusive,
 )
 
+SCOPE_MANIFEST = "_scope.json"
+SCOPE_MANIFEST_VERSION = 1
+_LISTED_REFUSALS = 10
+
 
 _TEASER_HINT = (
     "  If the text stops at a teaser (paywall, login, script-rendered page): save the page from a\n"
@@ -73,15 +88,22 @@ def _warn(derivation: web_capture.Derivation, md_path: Path) -> None:
         )
 
 
-def _plan(url: str, *, canonical: bool) -> sites.Plan:
+def _preferences() -> sites.Preferences:
     try:
-        return sites.plan_for(url, sites.read_preferences(), canonical=canonical)
+        return sites.read_preferences()
     except ChemenuError as exc:
         fail(escape(str(exc)))
 
 
-def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation) -> None:
+def _plan(url: str, *, canonical: bool, preferences: Optional[sites.Preferences] = None) -> sites.Plan:
-    for line in plan.checks(derivation.decoded.text):
+    try:
+        return sites.plan_for(url, preferences or _preferences(), canonical=canonical)
+    except ChemenuError as exc:
+        fail(escape(str(exc)))
+
+
+def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation, subpages: int = 0) -> None:
+    for line in plan.checks(derivation.decoded.text, subpages):
         typer.echo(f"CHECK {line}")
 
 
@@ -121,6 +143,176 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     ))
 
 
+def _subpage_names(stem: str, urls: list[str]) -> list[str]:
+    """One file stem per subpage, from its URL, unique within the bundle and
+    never the overview's."""
+    taken = {stem}
+    names = []
+    for url in urls:
+        base = web_capture.stem_for(url)
+        name, number = base, 1
+        while name in taken:
+            number += 1
+            name = f"{base}-{number}"
+        taken.add(name)
+        names.append(name)
+    return names
+
+
+def _listed(urls: list[str]) -> str:
+    shown = "\n".join(f"  - {url}" for url in urls[:_LISTED_REFUSALS])
+    more = len(urls) - _LISTED_REFUSALS
+    return shown + (f"\n  ... and {more} more" if more > 0 else "")
+
+
+def _page_entry(name: str, role: str, response: web_capture.Response, retrieved: str) -> dict:
+    return {
+        "name": name,
+        "role": role,
+        "url": response.url,
+        "final_url": response.final_url,
+        "retrieved": retrieved,
+        "sha256": hashlib.sha256(response.body).hexdigest(),
+        "files": [f"{name}.html", f"{name}.md"],
+    }
+
+
+def _fetch_scope(url: str, name: Optional[str]) -> None:
+    preferences = _preferences()
+    plan = _plan(url, canonical=True, preferences=preferences)
+    stem = name or web_capture.stem_for(plan.url)
+    if _unusable_name(stem):
+        fail(escape(f"{stem!r} is not a usable folder name - choose another with --name."))
+    target = _incoming_dir() / stem
+
+    def refuse_taken() -> None:
+        if os.path.lexists(target):
+            fail(escape(
+                f"{rel_path(target)} already exists in incoming/ - raw fetch never overwrites. "
+                "Accept or remove what is there, or pass --name <stem>."
+            ))
+        holder = _occupied_stems(config.RAW_DIR).get(stem)
+        if holder is not None:
+            fail(escape(_folder_collision_message(stem, holder)))
+
+    if plan.site is None or plan.site.subpages is None:
+        fail(escape(
+            f"--scope needs a site handler that offers subpages, and none applies to {plan.url}. "
+            "Fetch the page without --scope."
+        ))
+    refuse_taken()
+
+    agent = _user_agent()
+    try:
+        index = web_capture.fetch(
+            plan.url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
+        )
+    except ChemenuError as exc:
+        fail(escape(str(exc)))
+    index_retrieved = web_capture.utc_now()
+    if index.media_type not in web_capture.HTML_TYPES:
+        fail(escape(
+            f"{plan.url} answered {index.media_type or 'no content type'}, not an HTML page - "
+            "--scope bundles HTML pages. Nothing was written."
+        ))
+    index_derivation = web_capture.derive_document(
+        index.body, index.final_url, f"{stem}.html", response=index,
+        drop=plan.drop, site=plan.label, requested_url=plan.requested_url, derived_at=index_retrieved,
+    )
+    urls = plan.subpages(index_derivation.decoded.text, index.final_url)
+    if not urls:
+        fail(escape(f"{index.final_url} offers no subpages - nothing was written. Fetch it without --scope."))
+    if len(urls) > preferences.max_pages:
+        fail(escape(
+            f"{index.final_url} offers {len(urls)} subpages, more than the {preferences.max_pages} "
+            "that `sites.max_pages` allows - nothing was fetched or written. Raise the setting "
+            "(`tools/wikitool config set sites.max_pages <n>`) if the whole family is wanted."
+        ))
+    try:
+        forbidden = web_capture.robots_refusals(urls, agent)
+    except ChemenuError as exc:
+        fail(escape(f"{exc} Nothing was written."))
+    if forbidden:
+        fail(escape(
+            f"robots.txt of {urllib.parse.urlsplit(index.final_url).netloc} forbids {len(forbidden)} "
+            f"of the {len(urls)} subpages - none was fetched, nothing was written:\n"
+            f"{_listed(forbidden)}"
+        ))
+
+    names = _subpage_names(stem, urls)
+    files = [f"{stem}.html", f"{stem}.md", SCOPE_MANIFEST]
+    files += [f"{n}{ext}" for n in names for ext in (".html", ".md")]
+    over = [p for rel in files if (p := path_budget_problem_for(_shard_dir() / stem / rel))]
+    if over:
+        fail(escape(
+            f"Cannot fetch - these files would be over the path budget at their place under raw/:\n"
+            f"{_listed(over)}\n  Choose a shorter --name. Nothing was written."
+        ))
+
+    work = Path(tempfile.mkdtemp(prefix="wikitool-scope-"))
+    try:
+        bundle = work / stem
+        bundle.mkdir()
+        host = urllib.parse.urlsplit(plan.url).netloc.lower()
+        (bundle / f"{stem}.html").write_bytes(index.body)
+        (bundle / f"{stem}.md").write_bytes(index_derivation.markdown)
+        entries = [_page_entry(stem, "index", index, index_retrieved)]
+        derivations = [(stem, index_derivation)]
+        for position, (sub_url, sub_name) in enumerate(zip(urls, names), start=1):
+            if preferences.pause_seconds:
+                time.sleep(preferences.pause_seconds)
+            try:
+                page = web_capture.fetch(
+                    sub_url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
+                )
+            except ChemenuError as exc:
+                fail(escape(f"{exc} (subpage {position} of {len(urls)}) Nothing was written."))
+            retrieved = web_capture.utc_now()
+            if urllib.parse.urlsplit(page.final_url).netloc.lower() != host:
+                fail(escape(
+                    f"{sub_url} was redirected to {page.final_url}, another host - --scope stays on "
+                    f"{host}. Nothing was written."
+                ))
+            if page.media_type not in web_capture.HTML_TYPES:
+                fail(escape(
+                    f"{sub_url} answered {page.media_type or 'no content type'}, not an HTML page - "
+                    "--scope bundles HTML pages. Nothing was written."
+                ))
+            sub_plan = _plan(sub_url, canonical=False, preferences=preferences)
+            derivation = web_capture.derive_document(
+                page.body, page.final_url, f"{sub_name}.html", response=page,
+                drop=sub_plan.drop, site=sub_plan.label, derived_at=retrieved,
+            )
+            (bundle / f"{sub_name}.html").write_bytes(page.body)
+            (bundle / f"{sub_name}.md").write_bytes(derivation.markdown)
+            entries.append(_page_entry(sub_name, "subpage", page, retrieved))
+            derivations.append((sub_name, derivation))
+        manifest = {
+            "scope": SCOPE_MANIFEST_VERSION,
+            "fetched_by": "wikitool raw fetch --scope",
+            "url": plan.url,
+            **({"requested_url": plan.requested_url} if plan.requested_url else {}),
+            "retrieved": index_retrieved,
+            "pages": entries,
+        }
+        (bundle / SCOPE_MANIFEST).write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n")
+        refuse_taken()
+        shutil.move(str(bundle), str(target))
+    except OSError as exc:
+        fail(escape(f"Could not write {rel_path(target)}: {exc}. Nothing was left in incoming/."))
+    finally:
+        shutil.rmtree(work, ignore_errors=True)
+
+    for page_name, derivation in derivations:
+        _warn(derivation, target / f"{page_name}.md")
+    _echo_checks(plan, index_derivation)
+    success(escape(
+        f"Fetched {_fetched(plan)} with {len(urls)} subpage(s) to {rel_path(target)}/ "
+        f"({len(entries)} page pairs and {SCOPE_MANIFEST}).\n"
+        + _next_lines([target], plan.url)
+    ))
+
+
 @cli_contract.record(cli_contract.CommandRecord(
     path="raw fetch",
     summary="Capture a web page the user names into `incoming/`: the HTML as received plus a "
@@ -131,6 +323,11 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
             usage="raw fetch <url> [--name <stem>]",
             notes="Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`",
         ),
+        cli_contract.Variant(
+            usage="raw fetch <url> --scope [--name <stem>]",
+            notes="Fetch the page and the subpages its site handler offers, and write the folder "
+            "`incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`",
+        ),
         cli_contract.Variant(
             usage="raw fetch --html incoming/<file>.html --url <url>",
             notes="Derive the `.md` from a page a human saved from their browser - no network "
@@ -141,7 +338,8 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         effect=cli_contract.Effect.WRITE,
         idempotent=cli_contract.Idempotent.NO,
         atomic="Yes for what it leaves behind - one or two new files in `incoming/`, created "
-        "exclusively; a failure on the second removes the first",
+        "exclusively; a failure on the second removes the first. With `--scope`: all of the folder "
+        "or none of it - the pages are fetched into a temporary place and moved in as a whole",
         budget=cli_contract.Budget.COUNTED,
         network=cli_contract.Network.YES,
     ),
@@ -173,6 +371,23 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         "URL's host or no readable host, stops the fetch before anything is fetched or written. "
         "The `sites` section of the settings switches handlers "
         "off, sets the locale and adds hint lines of the instance's own per host.",
+        "`--scope` is for a page whose site handler offers subpages - the stack's today: a "
+        "PowerShell module's overview page on `learn.microsoft.com`. The handler names them from "
+        "the page's own HTML: the links it selects that lead to the same host, under the path of "
+        "the page itself (or the prefix the handler sets); no other link is followed, and a "
+        "subpage's own links never are. Without `--scope` the one page is fetched as before and "
+        "a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the "
+        "page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response "
+        "byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - "
+        "and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each "
+        "page. `raw accept incoming/<stem>` takes the folder as one bundle.",
+        "`--scope` is bounded. The `robots.txt` of the host is honoured for the subpages "
+        "(not for the page itself): one it forbids refuses the whole fetch and names the URL. "
+        "More subpages than `sites.max_pages` (200) refuses it before the first subpage is "
+        "requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both "
+        "are settings of the `instance` tier. Everything is checked before the first write: "
+        "the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` "
+        "or `raw/`. A failing request leaves `incoming/` as it was.",
         "The stem is the URL's last path segment without its extension, else its host name - "
         "ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.",
         "A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any "
@@ -215,6 +430,16 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
             "with the user; for a paywall or login, save the page in a browser and use "
             "`--html`. An unreachable host or a timeout may be retried once",
         ),
+        cli_contract.Failure(
+            label="raw fetch --scope",
+            cause="No site handler offers subpages for the URL, the page offers none or more than "
+            "`sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot "
+            "be fetched, answers something other than HTML or leaves the host, the folder "
+            "`incoming/<stem>/` exists, or a path would be over the budget",
+            reaction="Nothing was written. Read the message: fetch the page without `--scope`, "
+            "raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then "
+            "retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user",
+        ),
         cli_contract.Failure(
             label="raw fetch --html",
             cause="`--url` is missing, the file is not directly in `incoming/` or does not "
@@ -226,6 +451,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     examples=(
         "tools/wikitool raw fetch https://example.org/blog/post",
         "tools/wikitool raw fetch https://example.org/ --name example-start",
+        "tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope",
         "tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post",
     ),
     never=(
@@ -255,6 +481,13 @@ def raw_fetch_command(
     name: Optional[str] = typer.Option(
         None, "--name", help="File stem in incoming/ instead of the one computed from the URL"
     ),
+    # `Annotated`, so that a direct call - the way the test suite calls this
+    # function - gets a real `False` here rather than a truthy `OptionInfo`.
+    scope: Annotated[bool, typer.Option(
+        "--scope",
+        help="Also fetch the subpages the site handler offers for this page, into the folder "
+        "incoming/<stem>/",
+    )] = False,
 ):
     """Capture a web page into incoming/ as received HTML plus derived text.
     See raw/CONTRACT.md "Getting a URL in: `raw fetch`"."""
@@ -263,6 +496,8 @@ def raw_fetch_command(
     if html is not None:
         if name is not None:
             fail("--name does not apply to --html - the stem is the saved file's own.")
+        if scope:
+            fail("--scope does not apply to --html - it fetches subpages, and --html makes no request.")
         _fetch_html_file(html, source_url)
         return
     if source_url is not None:
@@ -270,6 +505,9 @@ def raw_fetch_command(
 
     if name is not None and (not name.strip() or name in (".", "..") or any(c in name for c in "/\\")):
         fail(f"--name {name!r} is not a usable file stem: no path separators, not empty.")
+    if scope:
+        _fetch_scope(url, name)
+        return
     plan = _plan(url, canonical=True)
     stem = name or web_capture.stem_for(plan.url)
 
@@ -291,7 +529,7 @@ def raw_fetch_command(
         )
         _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
         _warn(derivation, md_path)
-        _echo_checks(plan, derivation)
+        _echo_checks(plan, derivation, len(plan.subpages(derivation.decoded.text, response.final_url)))
         success(escape(
             f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n"
             + _next_lines([html_path, md_path], plan.url) + "\n"

tools/chemenu/config_shapes.py

@@ -248,6 +248,10 @@ def _positive(value: int) -> Optional[str]:
     return None if value > 0 else "must be positive"
 
 
+def non_negative(value: float) -> Optional[str]:
+    return None if value >= 0 else "must not be negative"
+
+
 def dotted_extensions(value: tuple[str, ...]) -> Optional[str]:
     if not value:
         return "must not be empty"
@@ -453,8 +457,8 @@ SITES = Shape(
     key="sites",
     title="Site handlers",
     purpose="Tunes the site handlers `raw fetch` applies to the pages of well-known websites: "
-    "switches one off, picks the locale a handler fetches, and adds hint lines of the "
+    "switches one off, picks the locale a handler fetches, adds hint lines of the "
-    "instance's own per host.",
+    "instance's own per host, and bounds `--scope`, the fetch of a page with its subpages.",
     fields=(
         Field(
             "disabled", "array of strings", "The names of the site handlers that do not apply. "
@@ -479,10 +483,23 @@ SITES = Shape(
             default=[], json_type="array", coerce=site_notes, check=site_notes_problem,
             tier=INSTANCE,
         ),
+        positive_integer(
+            "max_pages", "The most subpages `raw fetch --scope` fetches for one page. A page that "
+            "offers more is refused before the first subpage is requested.",
+            default=200, tier=INSTANCE,
+        ),
+        Field(
+            "pause_seconds", "number", "The pause, in seconds, before each subpage request of "
+            "`raw fetch --scope`. `0` means none.",
+            default=1, json_type="number", coerce=float, check=non_negative,
+            lenient="A number written as a string is accepted.", tier=INSTANCE,
+        ),
     ),
     examples=({
         "disabled": ["learn-microsoft-powershell-module"],
         "locale": "en-us",
+        "max_pages": 200,
+        "pause_seconds": 1,
         "notes": [{
             "host": "learn.microsoft.com", "path": "^/[a-z-]+/azure/",
             "lines": ["Check the Azure retirement notice at the top of the page."],
@@ -515,6 +532,7 @@ _JSON_TYPES: dict[str, Callable[[Any], bool]] = {
     "string": lambda v: isinstance(v, str),
     "array": lambda v: isinstance(v, list),
     "object": lambda v: isinstance(v, dict),
+    "number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
 }
 
 _TYPE_NAMES = {bool: "a boolean", int: "an integer", float: "a number", str: "a string",

tools/chemenu/sites/__init__.py

@@ -14,7 +14,9 @@ with an optional condition. So a handler is a JSON file (`data.py` reads it,
 in `sites/` at its root, by the same building blocks. An instance handler for a
 host and path the stack has a handler for replaces it; the `sites` section of the
 settings still switches handlers off, sets the `locale` and adds hint lines
-(`notes`). A handler may still be registered from a Python module in this
+(`notes`). A handler may also offer the subpages of a page
+(`Subpages`: the links to take, under a path prefix), which `raw fetch --scope`
+fetches as one bundle. A handler may still be registered from a Python module in this
 package. The core knows neither `typer` nor `rich` and raises `ValidationError`.
 """
 from __future__ import annotations
@@ -33,6 +35,7 @@ from chemenu.web_capture import Rule
 
 MAX_CHECKS = 3
 LOCALE_PLACEHOLDER = "{locale}"
+PAGE_PLACEHOLDER = "{page}"
 INSTANCE_DIRECTORY = "sites"
 
 
@@ -79,6 +82,47 @@ class Check:
         return text
 
 
+@dataclass(frozen=True)
+class Subpages:
+    """The subpages a page offers: the links `links` selects, resolved against
+    the page's URL, that stay on its host and lie under `prefix`. `{page}` in
+    `prefix` is the path of the page itself with a trailing slash, which is
+    the default; the page is never its own subpage. Document order, each URL
+    once, the fragment dropped - and nothing else is followed."""
+
+    links: Rule
+    prefix: str = PAGE_PLACEHOLDER
+
+    def __post_init__(self) -> None:
+        if not (self.prefix.startswith(PAGE_PLACEHOLDER) or self.prefix.startswith("/")):
+            raise ValueError(f"a subpages prefix starts with {PAGE_PLACEHOLDER} or with /, not {self.prefix!r}")
+
+    def urls(self, html: str, page_url: str) -> list[str]:
+        page = urllib.parse.urlsplit(page_url)
+        own = urllib.parse.unquote(page.path) or "/"
+        own = own if own.endswith("/") else own + "/"
+        prefix = self.prefix.replace(PAGE_PLACEHOLDER, own)
+        found: list[str] = []
+        for element in web_capture.matching(web_capture.parse(html), self.links):
+            href = element.attrs.get("href", "").strip()
+            if not href:
+                continue
+            url = urllib.parse.urldefrag(urllib.parse.urljoin(page_url, href))[0]
+            parts = urllib.parse.urlsplit(url)
+            path = urllib.parse.unquote(parts.path)
+            if (
+                parts.scheme.lower() not in web_capture.ALLOWED_SCHEMES
+                or parts.netloc.lower() != page.netloc.lower()
+                or not path.startswith(prefix)
+                or path.rstrip("/") == own.rstrip("/")
+                or path == prefix
+                or url in found
+            ):
+                continue
+            found.append(url)
+        return found
+
+
 @dataclass(frozen=True)
 class Site:
     """One handler. `host` is matched exactly (lower case); `path` is a regular
@@ -94,6 +138,7 @@ class Site:
     rewrites: tuple[Rewrite, ...] = ()
     drop: tuple[Rule, ...] = ()
     checks: tuple[Check, ...] = ()
+    subpages: Optional[Subpages] = None
     origin: str = "stack"
 
     def __post_init__(self) -> None:
@@ -131,6 +176,7 @@ def inherit(candidate: Site, parent: Site) -> Site:
         rewrites=parent.rewrites + candidate.rewrites,
         drop=parent.drop + candidate.drop,
         checks=parent.checks + candidate.checks,
+        subpages=candidate.subpages or parent.subpages,
     )
     if len(flat.checks) > MAX_CHECKS:
         raise ValueError(
@@ -220,6 +266,8 @@ class Preferences:
     notes: tuple[Note, ...] = ()
     instance: tuple[Site, ...] = ()
     broken: tuple[Broken, ...] = ()
+    max_pages: int = 200
+    pause_seconds: float = 1.0
 
 
 def read_instance(root: "Path | str | None" = None) -> tuple[tuple[Site, ...], tuple[Broken, ...]]:
@@ -250,7 +298,9 @@ def read_preferences(root: "Path | str | None" = None) -> Preferences:
     notes = tuple(
         Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"]
     )
-    return Preferences(disabled, section["locale"], notes, instance, broken)
+    return Preferences(
+        disabled, section["locale"], notes, instance, broken, section["max_pages"], section["pause_seconds"]
+    )
 
 
 @dataclass(frozen=True)
@@ -278,11 +328,23 @@ class Plan:
     def drop(self) -> tuple[Rule, ...]:
         return self.site.drop if self.site else ()
 
-    def checks(self, html: str) -> list[str]:
+    def checks(self, html: str, subpages: int = 0) -> list[str]:
-        """The `CHECK` lines for the page: the handler's, then the instance's notes."""
+        """The `CHECK` lines for the page: the handler's, the offer of `subpages`
+        subpages when there are any, then the instance's notes."""
         lines = self.site.check_lines(html) if self.site else []
+        if subpages:
+            lines.append(
+                f"this page is an index with {subpages} subpage(s) - rerun with `--scope` to "
+                "fetch them as one bundle"
+            )
         return lines + list(self.notes)
 
+    def subpages(self, html: str, page_url: str) -> list[str]:
+        """The subpage URLs the handler offers for this page, else none."""
+        if self.site is None or self.site.subpages is None:
+            return []
+        return self.site.subpages.urls(html, page_url)
+
 
 def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan:
     """The plan for `url`. `canonical=False` keeps the URL as given - for a page

tools/chemenu/sites/data.py

@@ -15,7 +15,7 @@ from typing import Any, Optional
 
 from jsonschema import Draft202012Validator
 
-from chemenu.sites import Broken, Check, Rewrite, Rule, Site, inherit
+from chemenu.sites import Broken, Check, Rewrite, Rule, Site, Subpages, inherit
 
 SCHEMA_PATH = Path(__file__).with_name("site.schema.json")
 STACK_DIRECTORY = Path(__file__).with_name("handlers")
@@ -70,6 +70,13 @@ def build(name: str, document: Any, origin: str) -> Site:
         if "{count}" in text and "count" not in entry:
             raise ValueError("checks: a text with {count} needs a `count` rule")
         checks.append(Check(text, entry.get("meta", ""), _rule(entry["count"]) if "count" in entry else None))
+    subpages = None
+    if "subpages" in document:
+        entry = document["subpages"]
+        try:
+            subpages = Subpages(_rule(entry["links"]), entry.get("prefix", "{page}"))
+        except ValueError as exc:
+            raise ValueError(f"subpages: {exc}") from None
     try:
         return Site(
             name=name,
@@ -80,6 +87,7 @@ def build(name: str, document: Any, origin: str) -> Site:
             rewrites=tuple(rewrites),
             drop=tuple(_rule(entry) for entry in document.get("drop", [])),
             checks=tuple(checks),
+            subpages=subpages,
             origin=origin,
         )
     except (ValueError, re.error) as exc:

tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json

@@ -1,12 +1,9 @@
 {
   "host": "learn.microsoft.com",
-  "summary": "A PowerShell module's overview page: names the cmdlet pages it lists, which are not fetched with it.",
+  "summary": "A PowerShell module's overview page: offers the cmdlet pages it lists as its subpages, which `--scope` fetches with it as one bundle.",
   "path": "/[a-z]{2,3}(?:-[a-z0-9]{2,8})+/powershell/module/[^/]+/?$",
   "parent": "learn-microsoft",
-  "checks": [
+  "subpages": {
-    {
+    "links": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"}
-      "text": "This module page lists {count} cmdlet page(s) as links; they are not fetched - ingest the ones the topic needs, each with its own raw fetch.",
-      "count": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"}
   }
-  ]
 }

tools/chemenu/sites/site.schema.json

@@ -37,6 +37,15 @@
           "count": {"$ref": "#/$defs/rule"}
         }
       }
+    },
+    "subpages": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["links"],
+      "properties": {
+        "links": {"$ref": "#/$defs/rule"},
+        "prefix": {"type": "string", "pattern": "^(\\{page\\}|/)", "maxLength": 200}
+      }
     }
   },
   "$defs": {

tools/chemenu/tests/test_raw_fetch.py

@@ -100,8 +100,8 @@ def tree(kb_dir):
     return root
 
 
-def _fetch(url=None, html=None, source_url=None, name=None):
+def _fetch(url=None, html=None, source_url=None, name=None, scope=False):
-    return raw_fetch_command(url=url, html=html, source_url=source_url, name=name)
+    return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope)
 
 
 def _refused(**kwargs) -> None:

tools/chemenu/tests/test_raw_fetch_scope.py

@@ -0,0 +1,346 @@
+"""`wikitool raw fetch --scope` - a page and the subpages its site handler offers,
+fetched as one bundle.
+
+Like `test_raw_fetch.py`, every request goes to a real `http.server` on 127.0.0.1.
+The handler is an instance handler in `sites/`, because that is the form in which
+the subpages building block is data an instance can write.
+"""
+from __future__ import annotations
+
+import datetime
+import hashlib
+import json
+from pathlib import Path
+
+import pytest
+import typer
+
+from chemenu import sites, web_capture
+from chemenu.commands.raw.accept import raw_accept_command
+from chemenu.commands.raw.fetch import raw_fetch_command
+from chemenu.config_shapes import SITES, check
+from chemenu.tests.conftest import write_settings
+from chemenu.tests.test_raw_fetch import Site
+
+HANDLER = {
+    "host": "127.0.0.1",
+    "summary": "A test module: the links in its table are its subpages.",
+    "path": "/mod/",
+    "subpages": {"links": {"tag": "a", "inside": "td"}},
+}
+
+
+def _index(port: int) -> bytes:
+    return (
+        "<html><head><title>Module</title></head><body><main><h1>Module</h1><table>"
+        '<tr><td><a href="a?v=1#top">A</a></td></tr>'
+        '<tr><td><a href="b?v=1">B</a></td></tr>'
+        '<tr><td><a href="/mod/c">C</a></td></tr>'
+        '<tr><td><a href="a?v=1">A again</a></td></tr>'
+        '<tr><td><a href="/elsewhere/x">outside the prefix</a></td></tr>'
+        f'<tr><td><a href="http://localhost:{port}/mod/foreign">another host</a></td></tr>'
+        '<tr><td><a href="mailto:x@example.org">mail</a></td></tr>'
+        '<tr><td><a href="./">the page itself</a></td></tr>'
+        "</table>"
+        '<p><a href="/mod/hidden">a link outside the table</a></p>'
+        "</main></body></html>"
+    ).encode()
+
+
+def _sub(name: str) -> bytes:
+    return (
+        f"<html><head><title>{name}</title></head><body><main><h1>{name}</h1>"
+        f"<p>{name} text</p></main></body></html>"
+    ).encode()
+
+
+@pytest.fixture
+def site(monkeypatch):
+    for var in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY", "all_proxy", "ALL_PROXY"):
+        monkeypatch.delenv(var, raising=False)
+    server = Site()
+    yield server
+    server.stop()
+
+
+@pytest.fixture
+def tree(kb_dir, monkeypatch, site):
+    root = kb_dir.parent
+    (root / "raw").mkdir()
+    (root / "incoming").mkdir(exist_ok=True)
+    (root / "sites").mkdir()
+    (root / "sites/test-module.json").write_text(json.dumps(HANDLER), encoding="utf-8")
+    write_settings(root, monkeypatch, sites={"pause_seconds": 0})
+    return root
+
+
+@pytest.fixture
+def module(site):
+    """The index and the pages around it, served."""
+    site.html("/mod/", _index(site.httpd.server_address[1]))
+    for name in ("a?v=1", "b?v=1", "c", "hidden", "foreign"):
+        site.html(f"/mod/{name}", _sub(name.split("?")[0]))
+    return site
+
+
+def _fetch(url, name=None, scope=True):
+    return raw_fetch_command(url=url, html=None, source_url=None, name=name, scope=scope)
+
+
+def _refused(url, **kwargs) -> None:
+    with pytest.raises(typer.Exit) as excinfo:
+        _fetch(url, **kwargs)
+    assert excinfo.value.exit_code == 1
+
+
+def _snapshot(root: Path) -> list[str]:
+    return sorted(str(p.relative_to(root)) for p in root.rglob("*"))
+
+
+def _requested(site: Site) -> list[str]:
+    return [path for path, _ in site.requests]
+
+
+def _checks(capsys) -> list[str]:
+    return [line[len("CHECK "):] for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")]
+
+
+# --- without --scope --------------------------------------------------------
+
+
+def test_without_scope_the_page_is_fetched_alone_and_the_offer_is_one_check(tree, module, capsys):
+    _fetch(module.url("/mod/"), name="mod", scope=False)
+
+    assert sorted(p.name for p in (tree / "incoming").iterdir()) == ["mod.html", "mod.md"]
+    assert (tree / "incoming/mod.html").read_bytes() == module.routes["/mod/"][2]
+    assert _requested(module) == ["/mod/"]
+    lines = _checks(capsys)
+    assert len(lines) == 1
+    assert "3 subpage(s)" in lines[0] and "--scope" in lines[0]
+
+
+# --- with --scope -----------------------------------------------------------
+
+
+def test_scope_writes_a_folder_with_a_pair_per_page_and_the_manifest(tree, module):
+    before = datetime.datetime.now(datetime.timezone.utc)
+    _fetch(module.url("/mod/"), name="mod")
+
+    folder = tree / "incoming/mod"
+    assert sorted(p.name for p in folder.iterdir()) == [
+        "_scope.json", "a.html", "a.md", "b.html", "b.md", "c.html", "c.md", "mod.html", "mod.md",
+    ]
+    assert [p.name for p in (tree / "incoming").iterdir()] == ["mod"]
+    for name, path in (("mod", "/mod/"), ("a", "/mod/a?v=1"), ("b", "/mod/b?v=1"), ("c", "/mod/c")):
+        assert (folder / f"{name}.html").read_bytes() == module.routes[path][2]
+    text = (folder / "a.md").read_text(encoding="utf-8")
+    assert "a text" in text and "derived_from: a.html" in text
+
+    manifest = json.loads((folder / "_scope.json").read_text(encoding="utf-8"))
+    assert manifest["scope"] == 1 and manifest["url"] == module.url("/mod/")
+    assert [(p["name"], p["role"]) for p in manifest["pages"]] == [
+        ("mod", "index"), ("a", "subpage"), ("b", "subpage"), ("c", "subpage"),
+    ]
+    for page in manifest["pages"]:
+        html = (folder / f"{page['name']}.html").read_bytes()
+        assert page["sha256"] == hashlib.sha256(html).hexdigest()
+        assert page["files"] == [f"{page['name']}.html", f"{page['name']}.md"]
+        stamp = datetime.datetime.strptime(page["retrieved"], "%Y-%m-%dT%H:%M:%SZ").replace(
+            tzinfo=datetime.timezone.utc)
+        assert before - datetime.timedelta(seconds=2) <= stamp <= datetime.datetime.now(datetime.timezone.utc)
+    assert [p["url"] for p in manifest["pages"][1:]] == [
+        module.url("/mod/a?v=1"), module.url("/mod/b?v=1"), module.url("/mod/c"),
+    ]
+
+
+def test_only_the_urls_the_handler_names_are_requested(tree, module):
+    _fetch(module.url("/mod/"), name="mod")
+
+    paths = _requested(module)
+    assert sorted(paths) == sorted(["/mod/", "/robots.txt", "/mod/a?v=1", "/mod/b?v=1", "/mod/c"])
+    assert not {"/mod/hidden", "/mod/foreign", "/elsewhere/x"} & set(paths)
+
+
+def test_the_requests_identify_the_tool(tree, module):
+    _fetch(module.url("/mod/"), name="mod")
+    assert module.requests and all(agent.startswith("chemenu-wikitool/") for _, agent in module.requests)
+
+
+def test_the_folder_is_accepted_by_raw_accept_unchanged(tree, module):
+    _fetch(module.url("/mod/"), name="mod")
+    snapshot = {p.name: p.read_bytes() for p in (tree / "incoming/mod").iterdir()}
+
+    raw_accept_command(
+        files=[tree / "incoming/mod"], fidelity="published", authority="reporting",
+        page=None, replaces=None, dry_run=False,
+    )
+
+    accepted = [p for p in (tree / "raw").rglob("*") if p.is_dir() and p.name == "mod"]
+    assert len(accepted) == 1
+    assert {p.name: p.read_bytes() for p in accepted[0].iterdir()} == snapshot
+    assert not (tree / "incoming/mod").exists()
+
+
+def test_the_pause_is_taken_before_each_subpage(tree, module, monkeypatch):
+    write_settings(tree, monkeypatch, sites={"pause_seconds": 0.25})
+    slept: list[float] = []
+    monkeypatch.setattr("chemenu.commands.raw.fetch.time.sleep", slept.append)
+    _fetch(module.url("/mod/"), name="mod")
+    assert slept == [0.25, 0.25, 0.25]
+
+
+# --- refusals: nothing is written --------------------------------------------
+
+
+def test_robots_forbidding_a_subpage_refuses_the_fetch_and_names_it(tree, module, capsys):
+    module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /mod/b\n")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+
+    assert _snapshot(tree) == before
+    assert module.url("/mod/b?v=1") in capsys.readouterr().out.replace("\n", "")
+    assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module))
+
+
+def test_a_robots_txt_for_another_agent_does_not_apply(tree, module):
+    module.routes["/robots.txt"] = (
+        200, {"Content-Type": "text/plain"}, b"User-agent: somebodyelse\nDisallow: /\n",
+    )
+    _fetch(module.url("/mod/"), name="mod")
+    assert (tree / "incoming/mod/c.html").is_file()
+
+
+def test_a_missing_robots_txt_allows_everything(tree, module):
+    assert "/robots.txt" not in module.routes
+    _fetch(module.url("/mod/"), name="mod")
+    assert (tree / "incoming/mod/c.html").is_file()
+
+
+def test_a_robots_txt_that_cannot_be_read_refuses_the_fetch(tree, module, capsys):
+    module.routes["/robots.txt"] = (503, {}, b"later")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+    assert _snapshot(tree) == before
+    assert "robots.txt" in capsys.readouterr().out
+
+
+def test_robots_does_not_apply_to_the_page_without_scope(tree, module):
+    module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /\n")
+    _fetch(module.url("/mod/"), name="mod", scope=False)
+    assert (tree / "incoming/mod.html").is_file()
+    assert "/robots.txt" not in _requested(module)
+
+
+def test_a_failing_third_fetch_leaves_incoming_as_it_was(tree, module):
+    module.routes["/mod/b?v=1"] = (500, {}, b"broken")
+    (tree / "incoming/other.md").write_text("other", encoding="utf-8")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+
+    assert _snapshot(tree) == before
+    assert "/mod/b?v=1" in _requested(module)
+    assert "/mod/c" not in _requested(module)
+
+
+def test_a_subpage_that_is_not_html_refuses_the_fetch(tree, module):
+    module.routes["/mod/c"] = (200, {"Content-Type": "application/pdf"}, b"%PDF-1.7")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+    assert _snapshot(tree) == before
+
+
+def test_more_subpages_than_the_maximum_refuse_before_the_first_subpage(tree, module, monkeypatch, capsys):
+    write_settings(tree, monkeypatch, sites={"max_pages": 2})
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+
+    assert _snapshot(tree) == before
+    assert _requested(module) == ["/mod/"]
+    assert "max_pages" in " ".join(capsys.readouterr().out.split())
+
+
+def test_a_raised_maximum_takes_effect(tree, module, monkeypatch):
+    write_settings(tree, monkeypatch, sites={"max_pages": 4})
+    _fetch(module.url("/mod/"), name="mod")
+    assert (tree / "incoming/mod/c.md").is_file()
+
+
+def test_an_existing_folder_is_never_overwritten(tree, module):
+    (tree / "incoming/mod").mkdir()
+    (tree / "incoming/mod/mine.md").write_text("mine", encoding="utf-8")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+    assert _snapshot(tree) == before
+    assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module))
+
+
+def test_a_path_over_the_budget_is_refused_before_a_subpage_is_requested(tree, module):
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="m" * 140)
+    assert _snapshot(tree) == before
+    assert "/mod/a?v=1" not in _requested(module)
+
+
+def test_a_page_whose_handler_offers_no_subpages_is_refused(tree, module):
+    module.html("/plain", _sub("plain"))
+    before = _snapshot(tree)
+    _refused(module.url("/plain"), name="plain")
+    assert _snapshot(tree) == before
+
+
+def test_an_index_without_subpages_is_refused(tree, module):
+    module.html("/mod/", b"<html><body><main><p>No table at all, only text.</p></main></body></html>")
+    before = _snapshot(tree)
+    _refused(module.url("/mod/"), name="mod")
+    assert _snapshot(tree) == before
+
+
+def test_scope_does_not_go_with_html(tree):
+    saved = tree / "incoming/mod.html"
+    saved.write_bytes(b"<p>x</p>")
+    with pytest.raises(typer.Exit) as excinfo:
+        raw_fetch_command(url=None, html=saved, source_url="http://127.0.0.1/mod/", name=None, scope=True)
+    assert excinfo.value.exit_code == 1
+
+
+# --- the building block as data ---------------------------------------------
+
+
+def test_the_stack_handler_for_a_module_offers_its_cmdlet_pages_as_subpages():
+    special = next(s for s in sites.all_sites() if s.name == "learn-microsoft-powershell-module")
+    assert special.subpages is not None and special.subpages.prefix == "{page}"
+    html = (
+        '<table><tr><td><a data-linktype="relative-path" href="get-x?view=v">x</a></td></tr>'
+        '<tr><td><a data-linktype="absolute-path" href="/en-us/other">y</a></td></tr></table>'
+        '<a data-linktype="relative-path" href="get-y">outside the table</a>'
+    )
+    assert special.subpages.urls(html, "https://learn.microsoft.com/en-us/powershell/module/m/?view=v") == [
+        "https://learn.microsoft.com/en-us/powershell/module/m/get-x?view=v",
+    ]
+
+
+@pytest.mark.parametrize("subpages", [
+    {},
+    {"links": {}},
+    {"links": {"tag": "a"}, "prefix": "relative/"},
+    {"links": {"tag": "a"}, "extra": 1},
+])
+def test_a_malformed_subpages_block_is_a_rejected_file(subpages):
+    from chemenu.sites import data
+
+    with pytest.raises(ValueError):
+        data.build("x", {"host": "example.org", "summary": "s", "subpages": subpages}, "instance")
+
+
+def test_a_literal_prefix_replaces_the_page_path():
+    block = sites.Subpages(web_capture.Rule(tag="a"), "/docs/ref/")
+    html = '<a href="/docs/ref/one">1</a><a href="/docs/guide/two">2</a><a href="/docs/ref/">self</a>'
+    assert block.urls(html, "https://example.org/docs/index") == ["https://example.org/docs/ref/one"]
+
+
+def test_the_new_settings_have_their_defaults_and_refuse_nonsense():
+    assert check(SITES, {}).values["max_pages"] == 200
+    assert check(SITES, {}).values["pause_seconds"] == 1
+    assert check(SITES, {"max_pages": 0}).problems
+    assert check(SITES, {"pause_seconds": -1}).problems
+    assert not check(SITES, {"pause_seconds": 0}).problems

tools/chemenu/tests/test_site_files.py

@@ -60,7 +60,7 @@ def _handler(root: Path, name: str, content) -> Path:
 
 
 def _fetch(url, name=None, html=None):
-    return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name)
+    return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name, scope=False)
 
 
 def _refused(url, **kwargs) -> None:

tools/chemenu/tests/test_sites.py

@@ -57,8 +57,8 @@ def served(monkeypatch):
     return asked
 
 
-def _fetch(url=None, html=None, source_url=None, name=None):
+def _fetch(url=None, html=None, source_url=None, name=None, scope=False):
-    return raw_fetch_command(url=url, html=html, source_url=source_url, name=name)
+    return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope)
 
 
 def _refused(**kwargs) -> None:
@@ -130,13 +130,13 @@ def test_the_stem_comes_from_the_standard_form(tree, served):
     assert (tree / "incoming/get-oslicenseinfo.md").is_file()
 
 
-def test_the_checks_name_the_source_and_the_cmdlet_pages(tree, served, capsys):
+def test_the_checks_name_the_source_and_the_subpages(tree, served, capsys):
     _fetch(MODULE_URL, name="oslicense")
     lines = _checks(capsys)
 
     assert len(lines) == 2
     assert SOURCE_URL in lines[0]
-    assert "11 cmdlet page(s)" in lines[1]
+    assert "index with 11 subpage(s)" in lines[1] and "--scope" in lines[1]
 
 
 def test_a_page_of_the_general_handler_has_only_the_source_check(tree, served, capsys):
@@ -188,7 +188,7 @@ def test_html_applies_the_rules_but_keeps_the_url_as_typed(tree, monkeypatch, ca
     assert "requested_url" not in header
     assert header["site"] == "learn-microsoft-powershell-module"
     assert "Exit editor mode" not in text and "# OSLicense Module" in text
-    assert len(_checks(capsys)) == 2
+    assert len(_checks(capsys)) == 1
 
 
 # --- the instance's part -----------------------------------------------------
@@ -286,7 +286,8 @@ def test_a_special_handler_inherits_from_the_general_one():
     assert special.drop == general.drop
     assert special.rewrites == general.rewrites
     assert special.checks[: len(general.checks)] == general.checks
-    assert len(special.checks) == len(general.checks) + 1
+    assert special.checks == general.checks
+    assert special.subpages is not None and general.subpages is None
 
 
 def test_a_duplicate_host_and_path_is_a_registration_error(registry):

tools/chemenu/web_capture.py

@@ -25,6 +25,7 @@ import mimetypes
 import re
 import time
 import unicodedata
+import urllib.robotparser
 import urllib.error
 import urllib.parse
 import urllib.request
@@ -66,6 +67,14 @@ class Response:
         return media_type(self.content_type)
 
 
+class HttpError(BackendError):
+    """A server answered with an error status; `status` is the number."""
+
+    def __init__(self, message: str, status: int):
+        super().__init__(message)
+        self.status = status
+
+
 def media_type(content_type: str) -> str:
     if not content_type.strip():
         return ""
@@ -140,7 +149,7 @@ def fetch(
                 body=b"".join(chunks),
             )
     except urllib.error.HTTPError as exc:
-        raise BackendError(f"{url} answered HTTP {exc.code} {exc.reason}.") from exc
+        raise HttpError(f"{url} answered HTTP {exc.code} {exc.reason}.", exc.code) from exc
     except urllib.error.URLError as exc:
         if isinstance(exc.reason, TimeoutError):
             raise BackendError(f"{url} did not answer within {timeout:g} s.") from exc
@@ -598,8 +607,12 @@ def prune(root: _Element, rules: "tuple[Rule, ...]") -> None:
     root.children = kept
 
 
+def matching(document: _Element, rule: Rule) -> "list[_Element]":
+    return [element for element in document.iter() if rule.matches(element)]
+
+
 def count_matches(document: _Element, rule: Rule) -> int:
-    return sum(1 for element in document.iter() if rule.matches(element))
+    return len(matching(document, rule))
 
 
 def meta_content(document: _Element, name: str) -> str:
@@ -622,6 +635,33 @@ def derive_text(html: str, base_url: str, drop: "tuple[Rule, ...]" = ()) -> str:
     return _join(blocks)
 
 
+# --- robots.txt -------------------------------------------------------------
+
+
+def robots_refusals(
+    urls: "list[str]", user_agent: str, timeout: float = TIMEOUT_SECONDS, max_bytes: int = MAX_BYTES
+) -> "list[str]":
+    """The URLs among `urls` (all of one host) that the host's `robots.txt`
+    forbids this tool. A missing `robots.txt` (any 4xx) forbids nothing; one
+    that cannot be read at all (a 5xx, no answer) is a `BackendError`, because
+    "unknown" is not "allowed"."""
+    if not urls:
+        return []
+    parts = urllib.parse.urlsplit(urls[0])
+    robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
+    try:
+        body = fetch(robots_url, user_agent, timeout=timeout, max_bytes=max_bytes).body
+    except HttpError as exc:
+        if 400 <= exc.status < 500:
+            return []
+        raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc
+    except BackendError as exc:
+        raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc
+    parser = urllib.robotparser.RobotFileParser()
+    parser.parse(body.decode("utf-8", errors="replace").splitlines())
+    return [url for url in urls if not parser.can_fetch(user_agent, url)]
+
+
 # --- the capture file -------------------------------------------------------