raw derive PDF: Kopf nennt Bildseiten (images) und nicht zugeordnete Seiten (unmatched); neue Anweisung read-page-images für den gezielten Blick auf diese Seiten, Schwelle 20 Seiten (#234); 10.1.0-beta.7

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- instructions/ingest-large-tree.md
- instructions/read-page-images.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw/derive.py
- tools/chemenu/filters/pdf.py
- tools/chemenu/filters/pdf_layout.py
- tools/chemenu/tests/test_raw_derive_binary.py
- types/source.guidance.md

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

tools/CONTRACT.md

@@ -1608,7 +1608,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
 - A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
 - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
 - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
-- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before.
+- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document - and an `images` header field names the same pages, so that they outlast the output. An `unmatched` header field names the pages a WARN reports as left without markings because their text did not match the page layout. Both are written only when not empty, as page ranges (`6-8, 10`). A PDF without outline, markings, images and unmatched pages is derived as before.
 - A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
 - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
 - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.

tools/chemenu/commands/raw/derive.py

@@ -379,8 +379,12 @@ def _derive_next(target: Path) -> str:
         "most three lines: form, places, pages; the `**` line names the highlighter colours); "
         "removing the markings gives the text the PDF has without them, and a tag never starts "
         "or ends on whitespace. A CHECK names the pages with a picture or drawing the text does "
-        "not contain - not a background, a filler or a picture repeated across the document. A "
-        "PDF without outline, markings and images is derived as before.",
+        "not contain - not a background, a filler or a picture repeated across the document - "
+        "and an `images` header field names the same pages, so that they outlast the output. An "
+        "`unmatched` header field names the pages a WARN reports as left without markings "
+        "because their text did not match the page layout. Both are written only when not "
+        "empty, as page ranges (`6-8, 10`). A PDF without outline, markings, images and "
+        "unmatched pages is derived as before.",
         "A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder "
         "`<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` "
         "per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet "

tools/chemenu/filters/pdf.py

@@ -69,10 +69,13 @@ class PdfFilter(Filter):
                 fields.append(("markings", list(annotation.markings)))
             image_pages = list(annotation.image_pages)
             if image_pages:
+                fields.append(("images", pdf_layout.page_ranges(image_pages)))
                 checks.append(
                     f"Images on page(s) {pdf_layout.page_ranges(image_pages)} are not contained in the text - "
                     "check them against the original."
                 )
+            if annotation.unmatched_pages:
+                fields.append(("unmatched", pdf_layout.page_ranges(annotation.unmatched_pages)))
 
         blocks = [f"<!-- page {number} -->\n{text}".rstrip() for number, text in enumerate(pages, 1)]
         marker_lines, line = {}, 1

tools/chemenu/filters/pdf_layout.py

@@ -153,6 +153,7 @@ class Annotation:
     markings: tuple[str, ...]  # the legend
     warnings: tuple[str, ...]
     image_pages: tuple[int, ...]
+    unmatched_pages: tuple[int, ...]
 
 
 def colour_name(value: str) -> str:
@@ -816,4 +817,4 @@ def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation:
             f"Page(s) {page_ranges(unmatched)}: under {round(MIN_COVERAGE * 100)}% of the text could be matched to the "
             "page layout, so the markings (highlight, bold, colour, strike-through, italic) are left out there."
         )
-    return Annotation(tuple(result), tuple(markings), tuple(warnings), tuple(image_pages(layouts, ctx)))
+    return Annotation(tuple(result), tuple(markings), tuple(warnings), tuple(image_pages(layouts, ctx)), tuple(unmatched))

tools/chemenu/tests/test_raw_derive_binary.py

@@ -175,6 +175,7 @@ def test_a_pdf_without_outline_or_markings_has_neither_field_and_no_image_check(
     header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[LONG_LINE]])], title="T"))
     out = capsys.readouterr().out
     assert "outline" not in header and "markings" not in header
+    assert "images" not in header and "unmatched" not in header
     assert "*" not in text and "WARN" not in out and out.count("CHECK") == 1
 
 
@@ -330,8 +331,9 @@ def test_only_pictures_that_carry_content_are_named_in_the_check(tree, capsys):
         page(Picture(100, 400, 120, 90, pixels=8, gray=0x40)),  # 4: content
         page(ops=[f"0 G 0.5 w {80 + 8 * n} 400 m {84 + 8 * n} {430 + (n % 3) * 9} l S" for n in range(30)]),  # 5: a drawing
     ])
-    _derive_pdf(tree, pdf)
+    header, _text = _derive_pdf(tree, pdf)
     assert "CHECK Images on page(s) 4-5 are not contained in the text" in capsys.readouterr().out
+    assert header["images"] == "4-5"
 
 
 @needs_pypdf
@@ -414,9 +416,10 @@ def test_the_legend_is_at_most_three_lines(tree):
 @needs_pdfminer
 def test_pages_with_images_get_one_check_line(tree, capsys):
     pdf = build_styled_pdf([StyledPage([["a"]]), StyledPage([["b"]], image=True), StyledPage([["c"]], image=True), StyledPage([["d"]])])
-    _derive_pdf(tree, pdf)
+    header, _text = _derive_pdf(tree, pdf)
     out = capsys.readouterr().out
     assert out.count("CHECK") == 2 and "Images on page(s) 2-3 are not contained in the text" in out
+    assert header["images"] == "2-3"
 
 
 @needs_pypdf
@@ -436,6 +439,7 @@ def test_a_page_whose_text_does_not_match_the_layout_is_left_unmarked_with_a_war
     assert "highlighted" in text and "**highlighted" not in text
     assert "**fine**" in text
     assert "WARN" in out and "Page(s) 1" in out and "left out" in out
+    assert str(header["unmatched"]) == "1" and "images" not in header
 
 
 @needs_pypdf
@@ -448,6 +452,7 @@ def test_a_layout_that_cannot_be_read_costs_the_markings_not_the_derivation(monk
     header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[("kept", {"highlight": YELLOW})]])], outline=[("Top", 1, 0)]))
     out = capsys.readouterr().out
     assert "kept" in text and "**" not in text and header["outline"][0].startswith("Top | level 1 | page 1")
+    assert "images" not in header and "unmatched" not in header
     assert "WARN" in out and "boom" in out