raw: --scope is the user's call - raw/CONTRACT.md says who starts it and why it is not following a link, wiki-ingest asks before rerunning with it; 10.0.0-beta.7
Files changed: - CHANGES.md - VERSION - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
21cdd787b7
commit
aed846740b
4 files changed
+25
-3
No files matched your search
@@ -110,6 +110,20 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
reason to fetch it (invariant 4). The rules behind all of this: `raw/CONTRACT.md` "Getting a
|
||||
URL in: `raw fetch`".
|
||||
|
||||
**A `CHECK` line says the page is an index with subpages?** Ask the user whether they want
|
||||
the page alone or the page and its subpages as one source. Only on the second answer, remove
|
||||
the single page's two files from `incoming/` - they were never accepted - and fetch again:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw fetch <url> --scope
|
||||
```
|
||||
|
||||
It writes the folder `incoming/<stem>/` - a pair per page and the manifest `_scope.json` -
|
||||
which is one source like any other folder: read every `.md`, accept the folder whole in
|
||||
step 5, and pass the URL the user named as `source_url` in step 6. A refusal because
|
||||
`robots.txt` forbids a subpage goes to the user as it is; never fetch the subpages one by one
|
||||
instead.
|
||||
|
||||
**Check that the text is the whole article.** A paywall, a login wall or a page that only
|
||||
renders in a browser yields a teaser, often long enough to look like an article: the text
|
||||
breaks off at "continue reading with...", a subscription offer or a login prompt. Stop there
|
||||
|
||||
Reference in new issue
Block a user