raw: --scope is the user's call - raw/CONTRACT.md says who starts it and why it is not following a link, wiki-ingest asks before rerunning with it; 10.0.0-beta.7
CI / verify (push) Successful in 8m5s
CI / pwsh (push) Successful in 2m32s
Release / release (push) Successful in 34s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- VERSION
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-11 19:08:47 +02:00
1 parent 21cdd787b7
commit aed846740b
4 files changed
+25 -3

No files matched your search

+14
View File
@@ -110,6 +110,20 @@ validator complains - and the ticked list is the only record that they happened.
reason to fetch it (invariant 4). The rules behind all of this: `raw/CONTRACT.md` "Getting a
URL in: `raw fetch`".
**A `CHECK` line says the page is an index with subpages?** Ask the user whether they want
the page alone or the page and its subpages as one source. Only on the second answer, remove
the single page's two files from `incoming/` - they were never accepted - and fetch again:
```bash
tools/wikitool raw fetch <url> --scope
```
It writes the folder `incoming/<stem>/` - a pair per page and the manifest `_scope.json` -
which is one source like any other folder: read every `.md`, accept the folder whole in
step 5, and pass the URL the user named as `source_url` in step 6. A refusal because
`robots.txt` forbids a subpage goes to the user as it is; never fetch the subpages one by one
instead.
**Check that the text is the whole article.** A paywall, a login wall or a page that only
renders in a browser yields a teaser, often long enough to look like an article: the text
breaks off at "continue reading with...", a subscription offer or a login prompt. Stop there