raw derive/fetch/pending: Länge als dritte Größenachse beim Ingest - eine CHECK-Zeile nennt jeden geschriebenen Text über LONG_TEXT_CHARS (Zeichen der .md wie geschrieben), raw pending markiert eine lange .md/.txt mit long:; ingest-large-tree liest ihn in Teilen nach der Gliederung, je Teil nur den Zeilenbereich, Extrakt als Faktenliste mit Fundstelle und Modalität (#226); 10.1.0-beta.10

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw/_common.py
- tools/chemenu/commands/raw/derive.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/commands/raw/pending.py
- tools/chemenu/tests/test_raw_derive.py
- tools/chemenu/tests/test_raw_fetch.py
- types/source.guidance.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

CHANGES.md

@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 ---
 
-## 10.1.0-beta.9 - 2026-10-15 - wiki-ingest step 1 sends the session to read-page-images.md before the discussion, so up to the threshold the page images are no question for the user
+## 10.1.0-beta.10 - 2026-10-15 - Länge als dritte Größenachse beim Ingest: raw derive und raw fetch melden einen Text über der Längenschwelle mit einer CHECK-Zeile, raw pending markiert eine lange Textdatei; ingest-large-tree liest ihn in Teilen nach der Gliederung, mit einer Faktenliste je Teil (#226)
 
 **Author:** Torben Nehmer
 
@@ -24,6 +24,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 - Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release
 - raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder
 - raw derive PDF: Kopf nennt die Bildseiten (images) und die nicht zugeordneten Seiten (unmatched); neue Anweisung read-page-images für den gezielten Blick auf diese Seiten
+- Länge als dritte Größenachse beim Ingest: raw derive und raw fetch melden einen Text über der Längenschwelle mit einer CHECK-Zeile, raw pending markiert eine lange Textdatei; ingest-large-tree liest ihn in Teilen nach der Gliederung, mit einer Faktenliste je Teil (#226)
 
 **Low impact**
 - instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
@@ -41,6 +42,10 @@ Der Testkorpus mit echten Dokumenten (PDF, Mail, Office, ZIP) und sein Test lieg
 
 Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`. Im Text stehen drei Auszeichnungen als gewöhnliches Markdown für das, was die Seite nur durch ihre Form sagt: `~~…~~` für Durchstreichung, `**…**` für Textmarker, Fettdruck oder eine deutlich andere Textfarbe, `*…*` für Kursiv. Die Ableitung bestimmt erst die Struktur (Fließtext, Code, Tabellengitter, wiederkehrende Vorlagen) und zeichnet nur aus, wo die Form eindeutig ist; Fett, Kursiv und Farbe zählen nur im Fließtext. Ohne die Auszeichnungen ist der Text derselbe wie zuvor. Das Feld `markings` ist die Legende mit höchstens drei Zeilen, kein Tag überschreitet eine Zeile oder Seitenmarke. Der CHECK nennt nur Seiten mit inhaltlichem Bild oder Diagramm. Das Layout liest `pdfminer.six`, neu in `requirements.txt`; vorhandene `derived.md` bleiben unberührt.
 
+### Lange Dokumente: Länge als dritte Größenachse (#226)
+
+Ein einzelnes langes Dokument mit wenigen Themen löste bisher keine der beiden Größenachsen aus und wurde in einem Zug gelesen. Jetzt melden `raw derive` und `raw fetch` jeden geschriebenen Text über der Längenschwelle (Zeichen, nicht Tokens; der Wert steht in `raw derive -h`) mit einer `CHECK`-Zeile, und `raw pending` markiert eine lange `.md`/`.txt` mit `long:` (JSON: `length`). `ingest-large-tree` liest einen solchen Text in Teilen entlang seiner Gliederung, je Teil nur dessen Zeilenbereich, und hält je Teil eine Faktenliste mit Fundstelle, Bedingung, Ausnahme und Modalität fest; die eine Source-Seite entsteht aus allen Listen, nicht aus Zusammenfassungen.
+
 ---
 
 ## 10.0.0 - 2026-10-13 - Einstellungen in drei Stufen, Eingangskanäle, raw fetch mit Site-Handlern und Login, Backup und Exporte

VERSION

@@ -1 +1 @@
-10.1.0-beta.9
+10.1.0-beta.10

docs/how-to/ingest-a-source.md

@@ -87,6 +87,12 @@ what is waiting and names the entry the next bare `Ingest` will take.
   estimate, proposes a selection, and you choose all, none or a selection. Pages nobody looked at
   are listed with the reason. Only whole pages are read; a harness that cannot show PDF pages as
   images gets the page numbers named instead.
+- **Before reading a long document in parts.** When the text `raw derive` or `raw fetch` wrote
+  is longer than one pass reads well, the tool says so in a `CHECK` line, and `raw pending` marks
+  a long text file the same way. The agent then does not read it in one go: it cuts it into parts
+  along its outline (or its pages), shows you the plan in a workshop under `work/`, and after your
+  answer reads one part at a time, noting each fact with where it stands. The document stays one
+  file in `raw/` and gets one source page, written from those notes.
 - **Before writing pages**, the agent presents the key takeaways and asks which points matter most
   and what to emphasise. The pages follow your answer.
 - **When it publishes**, a source that touches ten or more files stops at the Mass-Update Gate

docs/reference/commands/raw-material-and-uploads.md

@@ -70,6 +70,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
 - Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
 - Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
 - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
+- Length: a `CHECK` line names every `.md` this run wrote - and a `text/plain` or `text/markdown` response stored as received - that is longer than 50,000 characters, the threshold of `raw derive`: more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`).
 - Success prints the `raw accept` line for the written files and the `source_url` for the source page.
 
 **Properties**
@@ -198,6 +199,7 @@ tools/wikitool raw derive incoming/scans.zip
 - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
 - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
 - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
+- Length: one more `CHECK` line names every `.md` this run wrote that is longer than 50,000 characters, counted in the file as written, header included - more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). Characters, not tokens: they do not depend on a model's tokenizer. `raw fetch` and `raw pending` measure against the same value.
 - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original.
 
 **Properties**
@@ -371,9 +373,10 @@ tools/wikitool raw pending --json
 - Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
 - Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
 - The default is the first candidate `raw accept` would take, and the output names it.
-- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`.
+- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`, `length`.
 - A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is listed with the `raw derive` line for it, in text output and as `hint` in JSON (`null` for every other candidate). The hint changes nothing about the order or the default.
-- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
+- A lone `.md` or `.txt` file longer than the length threshold of `raw derive` and `raw fetch` is listed with its length in characters, in text output and as `length` in JSON (`null` for every other candidate) - that text reaches a session with no derivation in front of it to print the length `CHECK`. Like the hint, it changes nothing about the order or the default.
+- Reads directory listings and `lstat`, and a file's content only to measure a lone `.md`/`.txt` whose size in bytes is over the threshold - a UTF-8 text has no more characters than bytes; an empty `incoming/` is exit 0 with nothing to do.
 
 **Properties**
 

instructions/ingest-large-tree.md

@@ -1,7 +1,7 @@
 ---
 type: types/instruction.md
 name: ingest-large-tree
-description: Ingest a raw tree too large, or a single source too broad, for one pass - in planned units through a work/ workshop, instead of one oversized source page or a cohort of stub pages.
+description: Ingest a raw tree too large, or a single source too broad or too long, for one pass - in planned units through a work/ workshop, instead of one oversized source page, a cohort of stub pages or a text read in one go.
 ---
 # Ingest a large raw tree
 
@@ -10,7 +10,9 @@ read, promoted and published on its own. The plan and the intermediate extracts
 `work/` workshop, so the run survives across sessions and days instead of having to fit in one.
 
 A single source carrying too many subjects lands here too, and takes the workshop but not the
-cut - see [A broad source is not cut](#a-broad-source-is-not-cut).
+cut - see [A broad source is not cut](#a-broad-source-is-not-cut). So does a single source too
+long to read in one pass, which keeps its one raw file and is read in parts - see
+[A long source is read in parts](#a-long-source-is-read-in-parts).
 
 <!-- wikitool:toc -->
 ## Contents
@@ -18,6 +20,7 @@ cut - see [A broad source is not cut](#a-broad-source-is-not-cut).
 - [When to run](#when-to-run)
 - [Tiers](#tiers)
 - [A broad source is not cut](#a-broad-source-is-not-cut)
+- [A long source is read in parts](#a-long-source-is-read-in-parts)
 - [Steps](#steps)
 - [Decision points](#decision-points)
 - [Scope](#scope)
@@ -25,7 +28,7 @@ cut - see [A broad source is not cut](#a-broad-source-is-not-cut).
 
 ## When to run
 
-Any one of these is enough, on either axis.
+Any one of these is enough, on any of the three axes.
 
 **Volume** - more material than one pass can read:
 
@@ -40,9 +43,26 @@ Any one of these is enough, on either axis.
   concepts together**. Estimate it from the reading, before writing anything; on a finished page
   the same number is the length of `entities:` plus `concepts:`.
 
-The two axes take different routes through this procedure. Volume is cut: several files become
-several units, each its own source page. Breadth is not cut at all - it takes the workshop for
-the extract pass and nothing else (§ [A broad source is not cut](#a-broad-source-is-not-cut)).
+**Length** - one text longer than one pass reads well:
+
+- `raw derive` or `raw fetch` printed the `CHECK` line naming the text as longer than one ingest
+  reads in one pass, or `raw pending` listed the entry with `long:`. For a `.md` or `.txt` that
+  is not in `incoming/`, compare `wc -m <file>` with the value `tools/wikitool raw derive -h`
+  names.
+
+Length is an axis of its own because neither of the others fires for one long document on a
+few subjects, and reading it whole costs more than it seems. A session has already loaded a
+large fixed context before it opens the source - the harness's own, `AGENTS.md`, the skill, the
+contracts and conventions an ingest reads. Summaries of long input lose far more by leaving facts
+out than by getting them wrong, and small models lose recall well before their context window is
+full. Summarising the summaries of the parts does not recover it either. So the text is read in
+parts, and what each part yields is a list of facts, not a summary.
+
+The three axes take different routes through this procedure. Volume is cut: several files become
+several units, each its own source page. Breadth and length are not cut - one raw file keeps one
+source page. Breadth takes the workshop for the extract pass
+(§ [A broad source is not cut](#a-broad-source-is-not-cut)); length takes it for reading in parts
+and an extract per part (§ [A long source is read in parts](#a-long-source-is-read-in-parts)).
 
 Otherwise use `wiki-ingest` unchanged. This procedure costs a workshop and a planning round;
 a single, narrow document does not earn it.
@@ -54,6 +74,7 @@ a single, narrow document does not earn it.
 | Standard | One file, or a small folder | `wiki-ingest`, unchanged |
 | Tree | A volume trigger above | This instruction |
 | Broad | The breadth trigger above | This instruction, § A broad source is not cut |
+| Long | The length trigger above | This instruction, § A long source is read in parts |
 | Audited | A unit covering secrets, RBAC, ingress, disaster recovery, or an audit trail | This instruction plus step 5c |
 
 ## A broad source is not cut
@@ -91,6 +112,25 @@ produces when every name in the source is turned into a page: pages that restate
 pass `lint` (which measures structure, never substance), and read as covered ground to the next
 session.
 
+## A long source is read in parts
+
+A long source is one raw file, so the two rules of
+[A broad source is not cut](#a-broad-source-is-not-cut) hold for it unchanged: `raw/` keeps it
+whole, and it gets one source page. The cut is in the reading, never in the file. What the
+workshop buys:
+
+1. `tools/wikitool work new --input <the file, or the folder raw derive made>`, then one unit per
+   **part of the text** in `plan.md` - step 3 says where a long source is cut.
+2. Per unit, steps 5a and 5b: read the unit's line range and nothing else, and extract a fact
+   list.
+3. One source page, one publish: steps 5d-5e run once, after the last unit, over all the
+   extracts. The page is written from the fact lists of every unit - never from summaries of
+   the units, which is the summary of summaries this route exists to avoid.
+
+**Long and broad at once?** The same workshop, units by the text's parts. Which subjects earn a
+page - point 3 of the broad route - is decided over the collected extracts, after the last unit,
+not unit by unit.
+
 ## Steps
 
 1. **Survey the tree, do not read it yet.**
@@ -129,6 +169,17 @@ session.
    planned page title, and the reason for the cut. Record what is excluded from the run
    entirely, and why.
 
+   **A long source is cut by its outline**, not by subject: a unit is a page range, cut at the
+   highest outline level whose sections each stay under the length the `CHECK` line names. A
+   section longer than that - a long annex, a commentary part - is cut at the next level down,
+   or at page boundaries where it has none. A derived PDF's `outline` header field gives each
+   entry's page and the line of that page's marker, so the ranges come from the header, not from
+   reading the text. Without an outline the units are page ranges between the
+   `<!-- page N -->` markers (`grep -n '<!-- page ' <file>`); a text without those either - a
+   fetched page, a plain text - is cut at its own headings (`grep -n '^#' <file>`), else at line
+   numbers. Measure a range with `sed -n '<from>,<to>p' <file> | wc -m`. For each unit record its
+   title, its pages and its **line range** in `plan.md`: that range is all step 5a reads.
+
    Then fill the `README.md` checklist - one line per unit.
 
 4. **Agree the plan with the user.** This is the one decision checkpoint for the whole run:
@@ -144,11 +195,21 @@ session.
       (AGENTS.md invariant 4). A derived PDF whose header has `images:` or `unmatched:` is
       followed by `read-page-images.md`, whose threshold of 20 pages holds per unit.
 
+      **A long source:** read the unit's line range from `plan.md` and nothing else - the header
+      once, in the first unit. Reading the whole file again in every unit is the one pass this
+      route replaces, repeated. The page images are those of the unit's own pages.
+
    b. **Extract** into `work/<runkey>/extract-u<N>.md`: the hard facts (IPs, ports, versions,
       paths, commands, config values), each with the raw file it came from, plus what is
       *new* relative to what step 1's searches found. Write down what you are dropping and
       why - that becomes the page's `## Not Extracted` section.
 
+      **A long source:** the extract is a **fact list**, one entry per fact, each with where it
+      stands - the page, and the rule or section number where the text has one - and, where the
+      text states them, its condition, its exception, its deadline or number, and its modality
+      (*must*, *should*, *may*, *must not*). A normative text - rules, a regulation, a contract -
+      binds through exactly those parts, and they are what a summary drops first.
+
    c. **Audited tier only:** before touching any existing page, check the extract back against
       the raw files and record findings in `work/<runkey>/audit.md`, each as
       `Status: open` / `Status: resolved` with what changed. Do not promote while a finding is
@@ -157,8 +218,8 @@ session.
 
    d. **Promote** with `wiki-ingest` steps 4-10, using the extract as the input rather than the
       raw files - step 5 (promotion itself) is a no-op here, since the unit's raw file is
-      already under `raw/` (§ [When to run](#when-to-run) named the volume/breadth trigger that
-      put it there). Fill `## Not Extracted` from b.
+      already under `raw/` (§ [When to run](#when-to-run) named the trigger that put it
+      there). Fill `## Not Extracted` from b.
 
       This includes step 4's commitment question, asked once **per unit** rather than once for
       the whole tree: a unit is a subject the same way a single-file `wiki-ingest` source is one,
@@ -190,6 +251,10 @@ session.
 - **The breadth trigger fired, but the source reads narrower than it looked?** Close the
   workshop and run an ordinary `wiki-ingest`. The count is estimated before the reading, so
   being wrong about it is expected; carrying a workshop nobody needs is the avoidable half.
+- **The length trigger fired for a text that is mostly not prose** - a long register, tables of
+  figures, an annex of forms? The units still cut the reading. A unit with nothing to extract is
+  recorded as excluded in `plan.md`, with the reason, and that reason goes into
+  `## Not Extracted`.
 - **A unit turns out to be a duplicate of an existing page?** Update that page instead of
   creating a second one, and say so in `plan.md`. That is a result, not a failure.
 - **The plan changes mid-run?** Edit `plan.md` and the checklist, and say why in `README.md`.
@@ -199,7 +264,7 @@ session.
 
 ## Scope
 
-This is about *size*, on the two axes § When to run names: volume and breadth. It is not about
+This is about *size*, on the three axes § When to run names: volume, breadth and length. It is not about
 difficulty - a short but hard source, a specification that needs careful reading, is still an
 ordinary `wiki-ingest`. And nothing here changes what a page must contain:
 [kb/CONTRACT.md](../kb/CONTRACT.md) and the collection contracts still decide that.

instructions/wiki-ingest/SKILL.md

@@ -42,7 +42,8 @@ validator complains - and the ticked list is the only record that they happened.
 
 ## Steps
 
-1. **Read the source.** Read the file completely, wherever it currently sits - `incoming/` for
+1. **Read the source.** Read the file completely - unless the length check below fires, which
+   it does before anything is read - wherever it currently sits: `incoming/` for
    the normal path, or already under `raw/` when the run started there (a file `capture-session`
    just wrote, for instance, which skips step 5 entirely). A PDF, mail, Office file or ZIP: run
    `tools/wikitool raw derive <path>` first and read the `.md` - its output names what to check.
@@ -165,14 +166,19 @@ validator complains - and the ticked list is the only record that they happened.
    reads often, mention the lasting way as well: fetching with their own session
    (`docs/how-to/fetch-pages-behind-a-login.md`) - set up by the user, not by you.
 
-   **Check the size first, on both axes.** *Volume* - how many raw files this ingest covers -
-   and *breadth* - how many entities and concepts this one source would produce or update.
-   Either one past the thresholds in `instructions/ingest-large-tree.md` § When to
+   **Check the size first, on all three axes.** *Volume* - how many raw files this ingest
+   covers -, *breadth* - how many entities and concepts this one source would produce or update -
+   and *length* - whether one text is longer than one pass reads. Length is known before the
+   reading: `raw derive` and `raw fetch` print a `CHECK` line naming such a text, and
+   `raw pending` lists a lone `.md` or `.txt` with `long:`; a text file named outside
+   `incoming/` is measured with `wc -m <file>` against the value `tools/wikitool raw derive -h`
+   names. Any one past the thresholds in `instructions/ingest-large-tree.md` § When to
    run is that procedure, not this one: stop and follow it. There, volume is cut into units;
-   breadth cannot be cut at all (`raw/` keeps a file whole, and one raw file has one owning
-   source page) and buys an extract pass instead, before any page is written. Skipping either
-   fails silently: an oversized source page drops most of what it read, and an over-broad one
-   leaves a cohort of stub pages behind.
+   breadth and length cannot be cut (`raw/` keeps a file whole, and one raw file has one owning
+   source page) - breadth buys an extract pass before any page is written, length a reading in
+   parts with a fact list per part. Skipping any of them fails silently: an oversized source
+   page drops most of what it read, an over-broad one leaves a cohort of stub pages behind, and
+   a long text read in one go loses its facts by leaving them out.
 
    **A trigger firing here promotes now, ahead of step 4's commitment discussion below - the one
    deliberate exception to this skill's ordering.** `ingest-large-tree.md`'s own step 2
@@ -482,6 +488,11 @@ validator complains - and the ticked list is the only record that they happened.
 - **One source names far more subjects than usual?** That is breadth, not volume. It is not
   split into several sources - it cannot be - and it does not get a page per name either:
   `instructions/ingest-large-tree.md` § A broad source is not cut.
+- **One source is longer than one pass reads?** That is length - the `CHECK` line of
+  `raw derive` or `raw fetch` says so, or `raw pending`'s `long:`. It is not split into several
+  sources and not summarised whole: it is read in parts along its outline, each part yielding a
+  fact list, and one source page is written from all of them -
+  `instructions/ingest-large-tree.md` § A long source is read in parts.
 - **The entry is a new edition of a captured bundle?** Step 1's branch for updating the captured
   repositories is how a run gets here: `raw status` reported it, `raw capture --update` wrote it,
   and `raw accept incoming/<bundle> --replaces-bundle <raw-bundle>` takes it in - the name refusal

raw/CONTRACT.md

@@ -417,6 +417,15 @@ them is [read-page-images.md](../instructions/read-page-images.md). A PDF withou
 markings, images and unmatched pages is derived as before. Reading the layout needs `pdfminer.six` beside
 `pypdf`.
 
+**A text too long for one pass is named.** After a success `raw derive` prints one more `CHECK`
+line naming every `.md` it wrote that is longer than one ingest reads in one pass. Length is
+counted in characters of the file as written, header included, not in tokens: characters are the
+same for every model, tokens depend on its tokenizer. `raw fetch` prints the same line for the
+text it writes, and `raw pending` marks a lone `.md` or `.txt` in `incoming/` the same way, since
+no derivation stands in front of it to print one. The value is the tool's, and `raw derive -h`
+names it; what an ingest does with such a text is
+[ingest-large-tree.md](../instructions/ingest-large-tree.md).
+
 **The original stays.** The derivation is the tool's reading of it, never the source: with the
 original kept, a claim in `kb/` can still be checked against what the document said when a
 derivation dropped something, and a later version of the tool can derive it better. Both go into

tools/CONTRACT.md

@@ -1495,6 +1495,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: <host>` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: <name>)` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
 - Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used.
 - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched.
+- Length: a `CHECK` line names every `.md` this run wrote - and a `text/plain` or `text/markdown` response stored as received - that is longer than 50,000 characters, the threshold of `raw derive`: more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`).
 - Success prints the `raw accept` line for the written files and the `source_url` for the source page.
 
 **SEE ALSO**
@@ -1613,6 +1614,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
 - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
 - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
 - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
+- Length: one more `CHECK` line names every `.md` this run wrote that is longer than 50,000 characters, counted in the file as written, header included - more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). Characters, not tokens: they do not depend on a model's tokenizer. `raw fetch` and `raw pending` measure against the same value.
 - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original.
 
 **SEE ALSO**
@@ -1766,9 +1768,10 @@ List what waits in `incoming/`, oldest first, and name the entry an ingest witho
 - Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
 - Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
 - The default is the first candidate `raw accept` would take, and the output names it.
-- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`.
+- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`, `length`.
 - A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is listed with the `raw derive` line for it, in text output and as `hint` in JSON (`null` for every other candidate). The hint changes nothing about the order or the default.
-- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
+- A lone `.md` or `.txt` file longer than the length threshold of `raw derive` and `raw fetch` is listed with its length in characters, in text output and as `length` in JSON (`null` for every other candidate) - that text reaches a session with no derivation in front of it to print the length `CHECK`. Like the hint, it changes nothing about the order or the default.
+- Reads directory listings and `lstat`, and a file's content only to measure a lone `.md`/`.txt` whose size in bytes is over the threshold - a UTF-8 text has no more characters than bytes; an empty `incoming/` is exit 0 with nothing to do.
 
 **SEE ALSO**
 

tools/chemenu/commands/raw/_common.py

@@ -508,6 +508,56 @@ def _derive_hint(path: Path) -> Optional[str]:
     return f"tools/wikitool raw derive {rel_path(path)}"
 
 
+# The length axis of an ingest (Gitea #226): a text longer than this many characters
+# is read in parts, through `instructions/ingest-large-tree.md`, not in one pass.
+# Characters rather than tokens, because they do not depend on a model's tokenizer:
+# ~15 k tokens of German at ~3.3 characters a token, set early because a session's
+# fixed load already fills much of its context. The value stands here and nowhere
+# else - `raw derive`, `raw fetch` and `raw pending` all read it from this line.
+LONG_TEXT_CHARS = 50_000
+
+# The files `raw pending` measures: text a session reads as it is, with no derivation
+# - and so no `raw derive` or `raw fetch` - in front of it to print the length CHECK.
+_PLAIN_TEXT_SUFFIXES = (".md", ".txt")
+
+
+def _char_count(data: bytes) -> int:
+    """Characters of a UTF-8 text, the way a session reads it: an undecodable
+    byte counts as the one replacement character it becomes."""
+    return len(data.decode("utf-8", errors="replace"))
+
+
+def _long_text_check(texts: list[tuple[Path, bytes]]) -> Optional[str]:
+    """The one `CHECK` line for the texts longer than `LONG_TEXT_CHARS`, else None.
+    A UTF-8 text has no more characters than bytes, so a short one is not decoded."""
+    over = [
+        (path, count) for path, data in texts
+        if len(data) > LONG_TEXT_CHARS and (count := _char_count(data)) > LONG_TEXT_CHARS
+    ]
+    if not over:
+        return None
+    named = ", ".join(f"{rel_path(path)} ({count:,} characters)" for path, count in over)
+    return (
+        f"Longer than one ingest reads in one pass ({LONG_TEXT_CHARS:,} characters): {named}. "
+        "Read it in parts - instructions/ingest-large-tree.md, the length axis."
+    )
+
+
+def _plain_text_length(path: Path) -> Optional[int]:
+    """The characters of a lone `.md`/`.txt` file when they are over
+    `LONG_TEXT_CHARS`, else None. Reads the file only when its size could put
+    it over - `lstat` decides every other case."""
+    if path.suffix.lower() not in _PLAIN_TEXT_SUFFIXES or path.is_symlink() or not path.is_file():
+        return None
+    if path.lstat().st_size <= LONG_TEXT_CHARS:
+        return None
+    try:
+        count = _char_count(path.read_bytes())
+    except OSError:
+        return None
+    return count if count > LONG_TEXT_CHARS else None
+
+
 def _unusable_name(name: str) -> bool:
     """Not one plain directory name: empty, padded, hidden, a path, a control
     character, or the manifest's reserved name."""

tools/chemenu/commands/raw/derive.py

@@ -28,7 +28,9 @@ from chemenu.commands.raw._common import (
     _Refused,
     _check_directly_in_incoming,
     _folder_collision_message,
+    LONG_TEXT_CHARS,
     _incoming_dir,
+    _long_text_check,
     _occupied_stems,
     _refusals_fail,
     _resolve,
@@ -284,19 +286,36 @@ def _plan_derive_folder(folder: Path):
 
 
 def _echo_notes(
-    derivations: list[filters.Derivation], container: Optional[filters.Container] = None
+    derivations: list[filters.Derivation],
+    texts: list[tuple[Path, bytes]],
+    container: Optional[filters.Container] = None,
 ) -> None:
+    """The filters' warnings and CHECKs, then the length CHECK over `texts` -
+    every `.md` this run wrote, as written."""
     warnings = [w for d in derivations for w in d.warnings]
     checks = [c for d in derivations for c in d.checks]
     if container is not None:
         warnings += container.warnings
         checks += container.checks
+    long_text = _long_text_check(texts)
+    if long_text is not None:
+        checks.append(long_text)
     for warning in filters.unique(warnings):
         typer.echo(f"WARN {warning}")
     for line in filters.unique(checks):
         typer.echo(f"CHECK {line}")
 
 
+def _written_texts(plan: _FilePlan) -> list[tuple[Path, bytes]]:
+    """Every `.md` the single-file form writes: `derived.md`, and the `.md`
+    beside each member of a container that a filter read."""
+    texts = [(plan.derived, plan.markdown)] if plan.derived is not None else []
+    if plan.container is not None:
+        members = dict(plan.container.files)
+        texts += [(plan.folder / md, members[md]) for _src, md, _name in plan.container.derived]
+    return texts
+
+
 def _container_report(container: filters.Container) -> list[str]:
     lines = [f"  {src} -> {md} ({name})" for src, md, name in container.derived]
     if container.plain:
@@ -413,6 +432,11 @@ def _derive_next(target: Path) -> str:
         "After a success the output prints the warnings the filters raised, at most three "
         "`CHECK` lines per format that was derived in this run - what to look at before the "
         "text is trusted - and the `raw accept` line for the result.",
+        f"Length: one more `CHECK` line names every `.md` this run wrote that is longer than "
+        f"{LONG_TEXT_CHARS:,} characters, counted in the file as written, header included - "
+        "more than one ingest reads in one pass, so it is read in parts "
+        "(`instructions/ingest-large-tree.md`). Characters, not tokens: they do not depend on a "
+        "model's tokenizer. `raw fetch` and `raw pending` measure against the same value.",
         "Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original.",
     ),
     failures=(
@@ -479,7 +503,7 @@ def raw_derive_command(
         warning = _collision_warning(resolved.name)
         if warning:
             typer.echo("WARN " + warning)
-        _echo_notes([d.derivation for d in derived])
+        _echo_notes([d.derivation for d in derived], [(d.target, d.markdown) for d in derived])
         typer.echo(_derive_next(resolved))
         return
 
@@ -512,5 +536,5 @@ def raw_derive_command(
     warning = _collision_warning(plan.stem)
     if warning:
         typer.echo("WARN " + warning)
-    _echo_notes(plan.derivations, container)
+    _echo_notes(plan.derivations, _written_texts(plan), container)
     typer.echo(_derive_next(plan.folder))

tools/chemenu/commands/raw/fetch.py

@@ -34,7 +34,9 @@ from chemenu.errors import ChemenuError
 from chemenu.version import VersionError, read_version
 from chemenu.commands.raw._common import (
     _folder_collision_message,
+    LONG_TEXT_CHARS,
     _incoming_dir,
+    _long_text_check,
     _occupied_stems,
     _resolve,
     _shard_dir,
@@ -154,9 +156,19 @@ def _route_entries(route: fetch_routes.Route) -> dict:
     return found
 
 
-def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation, subpages: int = 0) -> None:
+def _echo_checks(
+    plan: sites.Plan,
+    derivation: web_capture.Derivation,
+    texts: list[tuple[Path, bytes]],
+    subpages: int = 0,
+) -> None:
+    """The site handler's CHECKs for the page, then the length CHECK over
+    `texts` - every `.md` this run wrote, as written."""
     for line in plan.checks(derivation.decoded.text, subpages):
         typer.echo(f"CHECK {line}")
+    long_text = _long_text_check(texts)
+    if long_text is not None:
+        typer.echo(f"CHECK {long_text}")
 
 
 def _fetched(plan: sites.Plan) -> str:
@@ -188,7 +200,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     )
     _write_exclusive([(md_path, derivation.markdown)])
     _warn(derivation, md_path)
-    _echo_checks(plan, derivation)
+    _echo_checks(plan, derivation, [(md_path, derivation.markdown)])
     success(escape(
         f"Derived {rel_path(md_path)} from {rel_path(path)} (no network access).\n"
         + _next_lines([path, md_path], source_url)
@@ -355,7 +367,7 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
 
     for page_name, derivation in derivations:
         _warn(derivation, target / f"{page_name}.md")
-    _echo_checks(plan, index_derivation)
+    _echo_checks(plan, index_derivation, [(target / f"{n}.md", d.markdown) for n, d in derivations])
     success(escape(
         f"Fetched {_fetched(plan)} with {len(urls)} subpage(s) to {rel_path(target)}/ "
         f"({len(entries)} page pairs and {SCOPE_MANIFEST}).\n"
@@ -470,6 +482,10 @@ def _fetch_scope(url: str, name: Optional[str]) -> None:
         "- the way past a paywall or a script-rendered page. Its header carries `derived` (when "
         "the text was derived) instead of `retrieved`, `final_url`, `http_status` and "
         "`content_type`, and the `.html` is left untouched.",
+        f"Length: a `CHECK` line names every `.md` this run wrote - and a `text/plain` or "
+        f"`text/markdown` response stored as received - that is longer than "
+        f"{LONG_TEXT_CHARS:,} characters, the threshold of `raw derive`: more than one ingest "
+        "reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`).",
         "Success prints the `raw accept` line for the written files and the `source_url` for the "
         "source page.",
     ),
@@ -628,7 +644,10 @@ def raw_fetch_command(
         )
         _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
         _warn(derivation, md_path)
-        _echo_checks(plan, derivation, len(plan.subpages(derivation.decoded.text, response.final_url)))
+        _echo_checks(
+            plan, derivation, [(md_path, derivation.markdown)],
+            len(plan.subpages(derivation.decoded.text, response.final_url)),
+        )
         anonymous = route.kind == fetch_routes.PLAIN and not route.render
         success(escape(
             f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n"
@@ -640,6 +659,10 @@ def raw_fetch_command(
     target = incoming / f"{stem}{web_capture.extension_for(response.content_type)}"
     _refuse_existing([target])
     _write_exclusive([(target, response.body)])
+    if kind in web_capture.PLAIN_TEXT_EXTENSIONS:
+        long_text = _long_text_check([(target, response.body)])
+        if long_text is not None:
+            typer.echo(f"CHECK {long_text}")
     success(escape(
         f"Fetched {_fetched(plan)} to {rel_path(target)} ({kind or 'no content type'}, stored as received - "
         f"no text derived; retrieved {web_capture.utc_now()}).\n"

tools/chemenu/commands/raw/pending.py

@@ -22,6 +22,7 @@ import typer
 from chemenu import cli_contract, repo_capture
 from chemenu.commands._util import rel_path
 from chemenu.commands.raw._common import (
+    LONG_TEXT_CHARS,
     _Refused,
     _check_files,
     _derive_hint,
@@ -29,6 +30,7 @@ from chemenu.commands.raw._common import (
     _plan_captured_folder,
     _plan_file_moves,
     _plan_folder,
+    _plain_text_length,
 )
 
 
@@ -40,6 +42,7 @@ class _Candidate:
     mtime_ns: int
     reason: Optional[str]  # None when `raw accept` would take it as it stands
     hint: Optional[str] = None  # the `raw derive` line for a lone file a filter reads
+    length: Optional[int] = None  # characters of a lone text file over the length threshold
 
     def label(self) -> str:
         return ", ".join(rel_path(p) + ("/" if self.kind == "folder" else "") for p in self.paths)
@@ -111,6 +114,7 @@ def _pending_candidates() -> list[_Candidate]:
             max(p.lstat().st_mtime_ns for p in paths),
             _refusal(_plain_files, paths),
             _derive_hint(paths[0]) if len(paths) == 1 else None,
+            _plain_text_length(paths[0]) if len(paths) == 1 else None,
         ))
     candidates.sort(key=lambda c: (c.mtime_ns, os.fsencode(c.paths[0].name)))
     return candidates
@@ -142,13 +146,19 @@ def _pending_candidates() -> list[_Candidate]:
         "human.",
         "The default is the first candidate `raw accept` would take, and the output names it.",
         "`--json` prints the same candidates in the same order: `kind`, `paths`, `files`, "
-        "`mtime`, `acceptable`, `reason`, `default`, `hint`.",
+        "`mtime`, `acceptable`, `reason`, `default`, `hint`, `length`.",
         "A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is "
         "listed with the `raw derive` line for it, in text output and as `hint` in JSON "
         "(`null` for every other candidate). The hint changes nothing about the order or the "
         "default.",
-        "Reads directory listings and `lstat` only, never a file's content; an empty "
-        "`incoming/` is exit 0 with nothing to do.",
+        "A lone `.md` or `.txt` file longer than the length threshold of `raw derive` and `raw "
+        "fetch` is listed with its length in characters, in text output and as `length` in "
+        "JSON (`null` for every other candidate) - that text reaches a session with no "
+        "derivation in front of it to print the length `CHECK`. Like the hint, it changes "
+        "nothing about the order or the default.",
+        "Reads directory listings and `lstat`, and a file's content only to measure a lone "
+        "`.md`/`.txt` whose size in bytes is over the threshold - a UTF-8 text has no more "
+        "characters than bytes; an empty `incoming/` is exit 0 with nothing to do.",
     ),
     failures=(),
     examples=(
@@ -180,6 +190,7 @@ def raw_pending_command(
                 "reason": c.reason,
                 "default": c is default,
                 "hint": c.hint,
+                "length": c.length,
             }
             for c in candidates
         ], indent=2, ensure_ascii=False))
@@ -199,6 +210,11 @@ def raw_pending_command(
                 typer.echo(f"       {line}")
         if c.hint is not None:
             typer.echo(f"       text: {c.hint}  (derives readable text, keeps the original)")
+        if c.length is not None:
+            typer.echo(
+                f"       long: {c.length:,} characters, over the {LONG_TEXT_CHARS:,} one ingest "
+                "reads in one pass - read it in parts (instructions/ingest-large-tree.md)"
+            )
     if default is None:
         typer.echo("No candidate can be accepted as it stands - each needs a human (reasons above).")
         return

tools/chemenu/tests/test_raw_derive.py

@@ -20,6 +20,7 @@ import typer
 import yaml
 
 from chemenu import filters, web_capture
+from chemenu.commands.raw._common import LONG_TEXT_CHARS
 from chemenu.commands.raw.accept import raw_accept_command
 from chemenu.commands.raw.derive import raw_derive_command
 from chemenu.commands.raw.pending import raw_pending_command
@@ -883,6 +884,71 @@ def test_header_renders_a_list_as_a_flow_sequence_and_a_string_as_before():
     assert yaml.safe_load(rendered.strip("-\n").replace("---", "")) == {"a": "plain", "b": ["x y", "z"], "c": []}
 
 
+# --- the length axis (Gitea #226) -------------------------------------------
+
+
+def _long_checks(out: str) -> list[str]:
+    return [line for line in out.splitlines() if line.startswith("CHECK Longer than one ingest")]
+
+
+def test_the_length_check_fires_one_character_past_the_constant_and_not_at_it(tree, capsys):
+    # one paragraph, the same file-name length: every header is as long as the probe's
+    _put(tree, "a.docx", make_docx(_p("x")))
+    _derive("incoming/a.docx")
+    overhead = len((tree / "incoming/a/derived.md").read_text(encoding="utf-8")) - 1
+    capsys.readouterr()
+
+    _put(tree, "b.docx", make_docx(_p("x" * (LONG_TEXT_CHARS - overhead))))
+    _derive("incoming/b.docx")
+    assert len((tree / "incoming/b/derived.md").read_text(encoding="utf-8")) == LONG_TEXT_CHARS
+    assert _long_checks(capsys.readouterr().out) == []
+
+    _put(tree, "c.docx", make_docx(_p("x" * (LONG_TEXT_CHARS - overhead + 1))))
+    _derive("incoming/c.docx")
+    [line] = _long_checks(capsys.readouterr().out)
+    assert f"incoming/c/derived.md ({LONG_TEXT_CHARS + 1:,} characters)" in line
+    assert "instructions/ingest-large-tree.md" in line
+
+
+def test_the_length_is_counted_in_characters_not_in_bytes(tree, capsys):
+    _put(tree, "umlaute.docx", make_docx(_p("ä" * (LONG_TEXT_CHARS // 2 + 1000))))
+    _derive("incoming/umlaute.docx")
+    assert (tree / "incoming/umlaute/derived.md").stat().st_size > LONG_TEXT_CHARS
+    assert _long_checks(capsys.readouterr().out) == []
+
+
+def test_the_length_check_names_each_long_md_of_a_folder_and_of_a_zip(tree, capsys):
+    folder = tree / "incoming/mappe"
+    folder.mkdir()
+    (folder / "lang.docx").write_bytes(make_docx(_p("x" * (LONG_TEXT_CHARS + 1))))
+    (folder / "kurz.docx").write_bytes(make_docx(_p("x")))
+    _derive("incoming/mappe")
+    [line] = _long_checks(capsys.readouterr().out)
+    assert "incoming/mappe/lang.md (" in line and "kurz.md" not in line
+
+    _put(tree, "paket.zip", _zip({"lang.docx": make_docx(_p("x" * (LONG_TEXT_CHARS + 1)))}))
+    _derive("incoming/paket.zip")
+    [line] = _long_checks(capsys.readouterr().out)
+    assert "incoming/paket/lang.md (" in line
+
+
+def test_pending_names_the_length_of_a_long_lone_text_file_and_of_nothing_else(tree, capsys):
+    _put(tree, "lang.md", b"x" * (LONG_TEXT_CHARS + 1))
+    _put(tree, "grenze.txt", b"x" * LONG_TEXT_CHARS)
+    _put(tree, "umlaute.md", ("ä" * (LONG_TEXT_CHARS // 2 + 1000)).encode("utf-8"))
+    _put(tree, "page.html", b"x" * (LONG_TEXT_CHARS + 1))
+    _put(tree, "page.md", b"x" * (LONG_TEXT_CHARS + 1))
+    raw_pending_command(json_out=True)
+    by_path = {c["paths"][0]: c for c in json.loads(capsys.readouterr().out)}
+    assert by_path["incoming/lang.md"]["length"] == LONG_TEXT_CHARS + 1
+    assert by_path["incoming/grenze.txt"]["length"] is None
+    assert by_path["incoming/umlaute.md"]["length"] is None
+    assert by_path["incoming/page.html"]["length"] is None  # a fetch pair: raw fetch printed it
+
+    raw_pending_command(json_out=False)
+    assert f"long: {LONG_TEXT_CHARS + 1:,} characters" in capsys.readouterr().out
+
+
 # --- the nets in raw pending and raw accept ---------------------------------
 
 

tools/chemenu/tests/test_raw_fetch.py

@@ -20,6 +20,7 @@ import typer
 import yaml
 
 from chemenu import web_capture
+from chemenu.commands.raw._common import LONG_TEXT_CHARS
 from chemenu.commands.raw.accept import raw_accept_command
 from chemenu.commands.raw.fetch import raw_fetch_command
 
@@ -330,6 +331,27 @@ def test_plain_text_is_stored_as_txt_unchanged(tree, site):
     assert (tree / "incoming/notes.txt").read_bytes() == b"line one\r\nline two\n"
 
 
+def _long_checks(out: str) -> list[str]:
+    return [line for line in out.splitlines() if line.startswith("CHECK Longer than one ingest")]
+
+
+def test_a_derived_page_over_the_length_threshold_gets_the_length_check(tree, site, capsys):
+    _fetch(site.html("/short", ARTICLE))
+    assert _long_checks(capsys.readouterr().out) == []
+
+    long_page = b"<html><body><main><p>" + b"x" * (LONG_TEXT_CHARS + 1) + b"</p></main></body></html>"
+    _fetch(site.html("/long", long_page))
+    [line] = _long_checks(capsys.readouterr().out)
+    assert "incoming/long.md (" in line and "instructions/ingest-large-tree.md" in line
+
+
+def test_plain_text_over_the_length_threshold_gets_the_length_check(tree, site, capsys):
+    site.routes["/notes"] = (200, {"Content-Type": "text/plain; charset=utf-8"}, b"x" * (LONG_TEXT_CHARS + 1))
+    _fetch(site.url("/notes"))
+    [line] = _long_checks(capsys.readouterr().out)
+    assert f"incoming/notes.txt ({LONG_TEXT_CHARS + 1:,} characters)" in line
+
+
 # --- --html: a page saved from a browser ------------------------------------
 
 

types/source.guidance.md

@@ -45,10 +45,11 @@ some source page - never whether anyone decided about its contents. Left unwritt
 source is renegotiated on every later pass.
 
 - Record every deliberate omission with a **reason**, not just a filename.
-- Mandatory where the ingest ran through `instructions/ingest-large-tree.md` - on both axes: for
-  a tree ingest the section records what was not taken from the tree; for a thematically broad
-  single source, which named subjects got no page of their own, and why. Optional for a single
-  small file - but an empty section still beats a missing one.
+- Mandatory where the ingest ran through `instructions/ingest-large-tree.md` - on every axis:
+  for a tree ingest the section records what was not taken from the tree; for a thematically
+  broad single source, which named subjects got no page of their own, and why; for a long
+  source, which of its parts or kinds of content no unit's fact list took over, and why. Optional
+  for a single small file - but an empty section still beats a missing one.
 - Belongs on the source page, not in `kb/log.md`: it is a statement about *this* source, and the
   log is chronological rather than per-source.