raw derive/fetch/pending: Länge als dritte Größenachse beim Ingest - eine CHECK-Zeile nennt jeden geschriebenen Text über LONG_TEXT_CHARS (Zeichen der .md wie geschrieben), raw pending markiert eine lange .md/.txt mit long:; ingest-large-tree liest ihn in Teilen nach der Gliederung, je Teil nur den Zeilenbereich, Extrakt als Faktenliste mit Fundstelle und Modalität (#226); 10.1.0-beta.10
CI / verify (push) Successful in 7m55s
CI / pwsh (push) Successful in 2m41s
Converters / converters (push) Successful in 9m32s
Release / converters (push) Successful in 27s
Release / release (push) Successful in 37s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw/_common.py
- tools/chemenu/commands/raw/derive.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/pending.py
- tools/chemenu/tests/test_raw_derive.py
- tools/chemenu/tests/test_raw_fetch.py
- types/source.guidance.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-15 21:21:24 +02:00
1 parent 9cb981a4cd
commit eae4a427d2
15 files changed
+341 -37

No files matched your search

+19 -8
View File
@@ -42,7 +42,8 @@ validator complains - and the ticked list is the only record that they happened.
## Steps
1. **Read the source.** Read the file completely, wherever it currently sits - `incoming/` for
1. **Read the source.** Read the file completely - unless the length check below fires, which
it does before anything is read - wherever it currently sits: `incoming/` for
the normal path, or already under `raw/` when the run started there (a file `capture-session`
just wrote, for instance, which skips step 5 entirely). A PDF, mail, Office file or ZIP: run
`tools/wikitool raw derive <path>` first and read the `.md` - its output names what to check.
@@ -165,14 +166,19 @@ validator complains - and the ticked list is the only record that they happened.
reads often, mention the lasting way as well: fetching with their own session
(`docs/how-to/fetch-pages-behind-a-login.md`) - set up by the user, not by you.
**Check the size first, on both axes.** *Volume* - how many raw files this ingest covers -
and *breadth* - how many entities and concepts this one source would produce or update.
Either one past the thresholds in `instructions/ingest-large-tree.md` § When to
**Check the size first, on all three axes.** *Volume* - how many raw files this ingest
covers -, *breadth* - how many entities and concepts this one source would produce or update -
and *length* - whether one text is longer than one pass reads. Length is known before the
reading: `raw derive` and `raw fetch` print a `CHECK` line naming such a text, and
`raw pending` lists a lone `.md` or `.txt` with `long:`; a text file named outside
`incoming/` is measured with `wc -m <file>` against the value `tools/wikitool raw derive -h`
names. Any one past the thresholds in `instructions/ingest-large-tree.md` § When to
run is that procedure, not this one: stop and follow it. There, volume is cut into units;
breadth cannot be cut at all (`raw/` keeps a file whole, and one raw file has one owning
source page) and buys an extract pass instead, before any page is written. Skipping either
fails silently: an oversized source page drops most of what it read, and an over-broad one
leaves a cohort of stub pages behind.
breadth and length cannot be cut (`raw/` keeps a file whole, and one raw file has one owning
source page) - breadth buys an extract pass before any page is written, length a reading in
parts with a fact list per part. Skipping any of them fails silently: an oversized source
page drops most of what it read, an over-broad one leaves a cohort of stub pages behind, and
a long text read in one go loses its facts by leaving them out.
**A trigger firing here promotes now, ahead of step 4's commitment discussion below - the one
deliberate exception to this skill's ordering.** `ingest-large-tree.md`'s own step 2
@@ -482,6 +488,11 @@ validator complains - and the ticked list is the only record that they happened.
- **One source names far more subjects than usual?** That is breadth, not volume. It is not
split into several sources - it cannot be - and it does not get a page per name either:
`instructions/ingest-large-tree.md` § A broad source is not cut.
- **One source is longer than one pass reads?** That is length - the `CHECK` line of
`raw derive` or `raw fetch` says so, or `raw pending`'s `long:`. It is not split into several
sources and not summarised whole: it is read in parts along its outline, each part yielding a
fact list, and one source page is written from all of them -
`instructions/ingest-large-tree.md` § A long source is read in parts.
- **The entry is a new edition of a captured bundle?** Step 1's branch for updating the captured
repositories is how a run gets here: `raw status` reported it, `raw capture --update` wrote it,
and `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>` takes it in - the name refusal