sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/how-to/write-a-site-handler.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_raw_fetch.py - tools/chemenu/tests/test_raw_fetch_scope.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
f25ee6a1ce
commit
21cdd787b7
21 files changed
+835
-41
No files matched your search
@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
|
||||
website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
|
||||
`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
|
||||
|
||||
A page that is an index - a module overview, a table of contents - can name its subpages, so that
|
||||
`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
|
||||
|
||||
```json
|
||||
"subpages": {"links": {"tag": "a", "inside": "td"}}
|
||||
```
|
||||
|
||||
The links are the `<a>` elements inside a table cell; only those on the same host and under the
|
||||
page's own path count. A page with such a handler prints a `CHECK` line with their number when it
|
||||
is fetched without `--scope`.
|
||||
|
||||
## 3. Try it
|
||||
|
||||
Fetch a page again, or derive a saved one without a fetch:
|
||||
|
||||
Reference in new issue
Block a user