diff --git a/CHANGES.md b/CHANGES.md index 8381150..81534bc 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.1.0-beta.9 - 2026-10-15 - wiki-ingest step 1 sends the session to read-page-images.md before the discussion, so up to the threshold the page images are no question for the user +## 10.1.0-beta.10 - 2026-10-15 - Länge als dritte Größenachse beim Ingest: raw derive und raw fetch melden einen Text über der Längenschwelle mit einer CHECK-Zeile, raw pending markiert eine lange Textdatei; ingest-large-tree liest ihn in Teilen nach der Gliederung, mit einer Faktenliste je Teil (#226) **Author:** Torben Nehmer @@ -24,6 +24,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. - Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release - raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder - raw derive PDF: Kopf nennt die Bildseiten (images) und die nicht zugeordneten Seiten (unmatched); neue Anweisung read-page-images für den gezielten Blick auf diese Seiten +- Länge als dritte Größenachse beim Ingest: raw derive und raw fetch melden einen Text über der Längenschwelle mit einer CHECK-Zeile, raw pending markiert eine lange Textdatei; ingest-large-tree liest ihn in Teilen nach der Gliederung, mit einer Faktenliste je Teil (#226) **Low impact** - instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie @@ -41,6 +42,10 @@ Der Testkorpus mit echten Dokumenten (PDF, Mail, Office, ZIP) und sein Test lieg Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`. Im Text stehen drei Auszeichnungen als gewöhnliches Markdown für das, was die Seite nur durch ihre Form sagt: `~~…~~` für Durchstreichung, `**…**` für Textmarker, Fettdruck oder eine deutlich andere Textfarbe, `*…*` für Kursiv. Die Ableitung bestimmt erst die Struktur (Fließtext, Code, Tabellengitter, wiederkehrende Vorlagen) und zeichnet nur aus, wo die Form eindeutig ist; Fett, Kursiv und Farbe zählen nur im Fließtext. Ohne die Auszeichnungen ist der Text derselbe wie zuvor. Das Feld `markings` ist die Legende mit höchstens drei Zeilen, kein Tag überschreitet eine Zeile oder Seitenmarke. Der CHECK nennt nur Seiten mit inhaltlichem Bild oder Diagramm. Das Layout liest `pdfminer.six`, neu in `requirements.txt`; vorhandene `derived.md` bleiben unberührt. +### Lange Dokumente: Länge als dritte Größenachse (#226) + +Ein einzelnes langes Dokument mit wenigen Themen löste bisher keine der beiden Größenachsen aus und wurde in einem Zug gelesen. Jetzt melden `raw derive` und `raw fetch` jeden geschriebenen Text über der Längenschwelle (Zeichen, nicht Tokens; der Wert steht in `raw derive -h`) mit einer `CHECK`-Zeile, und `raw pending` markiert eine lange `.md`/`.txt` mit `long:` (JSON: `length`). `ingest-large-tree` liest einen solchen Text in Teilen entlang seiner Gliederung, je Teil nur dessen Zeilenbereich, und hält je Teil eine Faktenliste mit Fundstelle, Bedingung, Ausnahme und Modalität fest; die eine Source-Seite entsteht aus allen Listen, nicht aus Zusammenfassungen. + --- ## 10.0.0 - 2026-10-13 - Einstellungen in drei Stufen, Eingangskanäle, raw fetch mit Site-Handlern und Login, Backup und Exporte diff --git a/VERSION b/VERSION index 2dbea57..39c657a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.1.0-beta.9 +10.1.0-beta.10 diff --git a/docs/how-to/ingest-a-source.md b/docs/how-to/ingest-a-source.md index 17cb39b..d6c3023 100644 --- a/docs/how-to/ingest-a-source.md +++ b/docs/how-to/ingest-a-source.md @@ -87,6 +87,12 @@ what is waiting and names the entry the next bare `Ingest` will take. estimate, proposes a selection, and you choose all, none or a selection. Pages nobody looked at are listed with the reason. Only whole pages are read; a harness that cannot show PDF pages as images gets the page numbers named instead. +- **Before reading a long document in parts.** When the text `raw derive` or `raw fetch` wrote + is longer than one pass reads well, the tool says so in a `CHECK` line, and `raw pending` marks + a long text file the same way. The agent then does not read it in one go: it cuts it into parts + along its outline (or its pages), shows you the plan in a workshop under `work/`, and after your + answer reads one part at a time, noting each fact with where it stands. The document stays one + file in `raw/` and gets one source page, written from those notes. - **Before writing pages**, the agent presents the key takeaways and asks which points matter most and what to emphasise. The pages follow your answer. - **When it publishes**, a source that touches ten or more files stops at the Mass-Update Gate diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index 929deb7..fa984e0 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -70,6 +70,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: ` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: )` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before. - Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used. - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched. +- Length: a `CHECK` line names every `.md` this run wrote - and a `text/plain` or `text/markdown` response stored as received - that is longer than 50,000 characters, the threshold of `raw derive`: more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). - Success prints the `raw accept` line for the written files and the `source_url` for the source page. **Properties** @@ -198,6 +199,7 @@ tools/wikitool raw derive incoming/scans.zip - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. +- Length: one more `CHECK` line names every `.md` this run wrote that is longer than 50,000 characters, counted in the file as written, header included - more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). Characters, not tokens: they do not depend on a model's tokenizer. `raw fetch` and `raw pending` measure against the same value. - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original. **Properties** @@ -371,9 +373,10 @@ tools/wikitool raw pending --json - Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped. - Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human. - The default is the first candidate `raw accept` would take, and the output names it. -- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`. +- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`, `length`. - A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is listed with the `raw derive` line for it, in text output and as `hint` in JSON (`null` for every other candidate). The hint changes nothing about the order or the default. -- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do. +- A lone `.md` or `.txt` file longer than the length threshold of `raw derive` and `raw fetch` is listed with its length in characters, in text output and as `length` in JSON (`null` for every other candidate) - that text reaches a session with no derivation in front of it to print the length `CHECK`. Like the hint, it changes nothing about the order or the default. +- Reads directory listings and `lstat`, and a file's content only to measure a lone `.md`/`.txt` whose size in bytes is over the threshold - a UTF-8 text has no more characters than bytes; an empty `incoming/` is exit 0 with nothing to do. **Properties** diff --git a/instructions/ingest-large-tree.md b/instructions/ingest-large-tree.md index eedbdcd..80560c7 100644 --- a/instructions/ingest-large-tree.md +++ b/instructions/ingest-large-tree.md @@ -1,7 +1,7 @@ --- type: types/instruction.md name: ingest-large-tree -description: Ingest a raw tree too large, or a single source too broad, for one pass - in planned units through a work/ workshop, instead of one oversized source page or a cohort of stub pages. +description: Ingest a raw tree too large, or a single source too broad or too long, for one pass - in planned units through a work/ workshop, instead of one oversized source page, a cohort of stub pages or a text read in one go. --- # Ingest a large raw tree @@ -10,7 +10,9 @@ read, promoted and published on its own. The plan and the intermediate extracts `work/` workshop, so the run survives across sessions and days instead of having to fit in one. A single source carrying too many subjects lands here too, and takes the workshop but not the -cut - see [A broad source is not cut](#a-broad-source-is-not-cut). +cut - see [A broad source is not cut](#a-broad-source-is-not-cut). So does a single source too +long to read in one pass, which keeps its one raw file and is read in parts - see +[A long source is read in parts](#a-long-source-is-read-in-parts). ## Contents @@ -18,6 +20,7 @@ cut - see [A broad source is not cut](#a-broad-source-is-not-cut). - [When to run](#when-to-run) - [Tiers](#tiers) - [A broad source is not cut](#a-broad-source-is-not-cut) +- [A long source is read in parts](#a-long-source-is-read-in-parts) - [Steps](#steps) - [Decision points](#decision-points) - [Scope](#scope) @@ -25,7 +28,7 @@ cut - see [A broad source is not cut](#a-broad-source-is-not-cut). ## When to run -Any one of these is enough, on either axis. +Any one of these is enough, on any of the three axes. **Volume** - more material than one pass can read: @@ -40,9 +43,26 @@ Any one of these is enough, on either axis. concepts together**. Estimate it from the reading, before writing anything; on a finished page the same number is the length of `entities:` plus `concepts:`. -The two axes take different routes through this procedure. Volume is cut: several files become -several units, each its own source page. Breadth is not cut at all - it takes the workshop for -the extract pass and nothing else (§ [A broad source is not cut](#a-broad-source-is-not-cut)). +**Length** - one text longer than one pass reads well: + +- `raw derive` or `raw fetch` printed the `CHECK` line naming the text as longer than one ingest + reads in one pass, or `raw pending` listed the entry with `long:`. For a `.md` or `.txt` that + is not in `incoming/`, compare `wc -m ` with the value `tools/wikitool raw derive -h` + names. + +Length is an axis of its own because neither of the others fires for one long document on a +few subjects, and reading it whole costs more than it seems. A session has already loaded a +large fixed context before it opens the source - the harness's own, `AGENTS.md`, the skill, the +contracts and conventions an ingest reads. Summaries of long input lose far more by leaving facts +out than by getting them wrong, and small models lose recall well before their context window is +full. Summarising the summaries of the parts does not recover it either. So the text is read in +parts, and what each part yields is a list of facts, not a summary. + +The three axes take different routes through this procedure. Volume is cut: several files become +several units, each its own source page. Breadth and length are not cut - one raw file keeps one +source page. Breadth takes the workshop for the extract pass +(§ [A broad source is not cut](#a-broad-source-is-not-cut)); length takes it for reading in parts +and an extract per part (§ [A long source is read in parts](#a-long-source-is-read-in-parts)). Otherwise use `wiki-ingest` unchanged. This procedure costs a workshop and a planning round; a single, narrow document does not earn it. @@ -54,6 +74,7 @@ a single, narrow document does not earn it. | Standard | One file, or a small folder | `wiki-ingest`, unchanged | | Tree | A volume trigger above | This instruction | | Broad | The breadth trigger above | This instruction, § A broad source is not cut | +| Long | The length trigger above | This instruction, § A long source is read in parts | | Audited | A unit covering secrets, RBAC, ingress, disaster recovery, or an audit trail | This instruction plus step 5c | ## A broad source is not cut @@ -91,6 +112,25 @@ produces when every name in the source is turned into a page: pages that restate pass `lint` (which measures structure, never substance), and read as covered ground to the next session. +## A long source is read in parts + +A long source is one raw file, so the two rules of +[A broad source is not cut](#a-broad-source-is-not-cut) hold for it unchanged: `raw/` keeps it +whole, and it gets one source page. The cut is in the reading, never in the file. What the +workshop buys: + +1. `tools/wikitool work new --input `, then one unit per + **part of the text** in `plan.md` - step 3 says where a long source is cut. +2. Per unit, steps 5a and 5b: read the unit's line range and nothing else, and extract a fact + list. +3. One source page, one publish: steps 5d-5e run once, after the last unit, over all the + extracts. The page is written from the fact lists of every unit - never from summaries of + the units, which is the summary of summaries this route exists to avoid. + +**Long and broad at once?** The same workshop, units by the text's parts. Which subjects earn a +page - point 3 of the broad route - is decided over the collected extracts, after the last unit, +not unit by unit. + ## Steps 1. **Survey the tree, do not read it yet.** @@ -129,6 +169,17 @@ session. planned page title, and the reason for the cut. Record what is excluded from the run entirely, and why. + **A long source is cut by its outline**, not by subject: a unit is a page range, cut at the + highest outline level whose sections each stay under the length the `CHECK` line names. A + section longer than that - a long annex, a commentary part - is cut at the next level down, + or at page boundaries where it has none. A derived PDF's `outline` header field gives each + entry's page and the line of that page's marker, so the ranges come from the header, not from + reading the text. Without an outline the units are page ranges between the + `` markers (`grep -n '