raw derive/fetch/pending: Länge als dritte Größenachse beim Ingest - eine CHECK-Zeile nennt jeden geschriebenen Text über LONG_TEXT_CHARS (Zeichen der .md wie geschrieben), raw pending markiert eine lange .md/.txt mit long:; ingest-large-tree liest ihn in Teilen nach der Gliederung, je Teil nur den Zeilenbereich, Extrakt als Faktenliste mit Fundstelle und Modalität (#226); 10.1.0-beta.10
CI / verify (push) Successful in 7m55s
CI / pwsh (push) Successful in 2m41s
Converters / converters (push) Successful in 9m32s
Release / converters (push) Successful in 27s
Release / release (push) Successful in 37s
Release / wiki-mirror (push) Skipped
CI / verify (push) Successful in 7m55s
CI / pwsh (push) Successful in 2m41s
Converters / converters (push) Successful in 9m32s
Release / converters (push) Successful in 27s
Release / release (push) Successful in 37s
Release / wiki-mirror (push) Skipped
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/raw-material-and-uploads.md - instructions/ingest-large-tree.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/commands/raw/_common.py - tools/chemenu/commands/raw/derive.py - tools/chemenu/commands/raw/fetch.py - tools/chemenu/commands/raw/pending.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/tests/test_raw_fetch.py - types/source.guidance.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
9cb981a4cd
commit
eae4a427d2
15 files changed
+341
-37
No files matched your search
@@ -42,7 +42,8 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Read the source.** Read the file completely, wherever it currently sits - `incoming/` for
|
||||
1. **Read the source.** Read the file completely - unless the length check below fires, which
|
||||
it does before anything is read - wherever it currently sits: `incoming/` for
|
||||
the normal path, or already under `raw/` when the run started there (a file `capture-session`
|
||||
just wrote, for instance, which skips step 5 entirely). A PDF, mail, Office file or ZIP: run
|
||||
`tools/wikitool raw derive <path>` first and read the `.md` - its output names what to check.
|
||||
@@ -165,14 +166,19 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
reads often, mention the lasting way as well: fetching with their own session
|
||||
(`docs/how-to/fetch-pages-behind-a-login.md`) - set up by the user, not by you.
|
||||
|
||||
**Check the size first, on both axes.** *Volume* - how many raw files this ingest covers -
|
||||
and *breadth* - how many entities and concepts this one source would produce or update.
|
||||
Either one past the thresholds in `instructions/ingest-large-tree.md` § When to
|
||||
**Check the size first, on all three axes.** *Volume* - how many raw files this ingest
|
||||
covers -, *breadth* - how many entities and concepts this one source would produce or update -
|
||||
and *length* - whether one text is longer than one pass reads. Length is known before the
|
||||
reading: `raw derive` and `raw fetch` print a `CHECK` line naming such a text, and
|
||||
`raw pending` lists a lone `.md` or `.txt` with `long:`; a text file named outside
|
||||
`incoming/` is measured with `wc -m <file>` against the value `tools/wikitool raw derive -h`
|
||||
names. Any one past the thresholds in `instructions/ingest-large-tree.md` § When to
|
||||
run is that procedure, not this one: stop and follow it. There, volume is cut into units;
|
||||
breadth cannot be cut at all (`raw/` keeps a file whole, and one raw file has one owning
|
||||
source page) and buys an extract pass instead, before any page is written. Skipping either
|
||||
fails silently: an oversized source page drops most of what it read, and an over-broad one
|
||||
leaves a cohort of stub pages behind.
|
||||
breadth and length cannot be cut (`raw/` keeps a file whole, and one raw file has one owning
|
||||
source page) - breadth buys an extract pass before any page is written, length a reading in
|
||||
parts with a fact list per part. Skipping any of them fails silently: an oversized source
|
||||
page drops most of what it read, an over-broad one leaves a cohort of stub pages behind, and
|
||||
a long text read in one go loses its facts by leaving them out.
|
||||
|
||||
**A trigger firing here promotes now, ahead of step 4's commitment discussion below - the one
|
||||
deliberate exception to this skill's ordering.** `ingest-large-tree.md`'s own step 2
|
||||
@@ -482,6 +488,11 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
- **One source names far more subjects than usual?** That is breadth, not volume. It is not
|
||||
split into several sources - it cannot be - and it does not get a page per name either:
|
||||
`instructions/ingest-large-tree.md` § A broad source is not cut.
|
||||
- **One source is longer than one pass reads?** That is length - the `CHECK` line of
|
||||
`raw derive` or `raw fetch` says so, or `raw pending`'s `long:`. It is not split into several
|
||||
sources and not summarised whole: it is read in parts along its outline, each part yielding a
|
||||
fact list, and one source page is written from all of them -
|
||||
`instructions/ingest-large-tree.md` § A long source is read in parts.
|
||||
- **The entry is a new edition of a captured bundle?** Step 1's branch for updating the captured
|
||||
repositories is how a run gets here: `raw status` reported it, `raw capture --update` wrote it,
|
||||
and `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>` takes it in - the name refusal
|
||||
|
||||
Reference in new issue
Block a user