sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6
CI / verify (push) Successful in 8m33s
CI / pwsh (push) Successful in 2m35s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/config_shapes.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/tests/test_raw_fetch_scope.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/tests/test_sites.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-11 18:36:10 +02:00
1 parent f25ee6a1ce
commit 21cdd787b7
21 files changed
+835 -41

No files matched your search

+16
View File
@@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
it and where it waits for you.
<!-- wikitool:toc -->
## Contents
- [What you prepare](#what-you-prepare)
- [What you say](#what-you-say)
- [Where the agent asks you](#where-the-agent-asks-you)
- [What you get back](#what-you-get-back)
- [A document submitted from outside](#a-document-submitted-from-outside)
- [Related](#related)
<!-- /wikitool:toc -->
## What you prepare
Put the material into `incoming/` at the root of the instance. You don't have to classify it or
@@ -22,6 +33,11 @@ pick a directory:
`CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
capture. Your instance can switch such a handler off, choose another language and add hint lines
of its own per website; see [the configuration reference](../reference/configuration.md).
Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
page that offers more than the instance's `max_pages`.
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
Outlook `.msg`) or a ZIP archive - is
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the