sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6
CI / verify (push) Successful in 8m33s
CI / pwsh (push) Successful in 2m35s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-11 18:36:10 +02:00
1 parent f25ee6a1ce
commit 21cdd787b7
21 files changed
+835 -41

No files matched your search

+11
View File
@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
A page that is an index - a module overview, a table of contents - can name its subpages, so that
`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
```json
"subpages": {"links": {"tag": "a", "inside": "td"}}
```
The links are the `<a>` elements inside a table cell; only those on the same host and under the
page's own path count. A page with such a handler prints a `CHECK` line with their number when it
is fetched without `--scope`.
## 3. Try it
Fetch a page again, or derive a saved one without a fetch: