sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/ingest-a-source.md

@@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
 the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
 it and where it waits for you.
 
+<!-- wikitool:toc -->
+## Contents
+
+- [What you prepare](#what-you-prepare)
+- [What you say](#what-you-say)
+- [Where the agent asks you](#where-the-agent-asks-you)
+- [What you get back](#what-you-get-back)
+- [A document submitted from outside](#a-document-submitted-from-outside)
+- [Related](#related)
+<!-- /wikitool:toc -->
+
 ## What you prepare
 
 Put the material into `incoming/` at the root of the instance. You don't have to classify it or
@@ -22,6 +33,11 @@ pick a directory:
   `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
   capture. Your instance can switch such a handler off, choose another language and add hint lines
   of its own per website; see [the configuration reference](../reference/configuration.md).
+  Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
+  lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
+  fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
+  any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
+  page that offers more than the instance's `max_pages`.
 - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the

docs/how-to/write-a-site-handler.md

@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
 website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
 `checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
 
+A page that is an index - a module overview, a table of contents - can name its subpages, so that
+`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
+
+```json
+"subpages": {"links": {"tag": "a", "inside": "td"}}
+```
+
+The links are the `<a>` elements inside a table cell; only those on the same host and under the
+page's own path count. A page with such a handler prints a `CHECK` line with their number when it
+is fetched without `--scope`.
+
 ## 3. Try it
 
 Fetch a page again, or derive a saved one without a fetch: