raw derive PDF: Kopf nennt Bildseiten (images) und nicht zugeordnete Seiten (unmatched); neue Anweisung read-page-images für den gezielten Blick auf diese Seiten, Schwelle 20 Seiten (#234); 10.1.0-beta.7
CI / verify (push) Successful in 8m9s
CI / pwsh (push) Successful in 2m52s
Converters / converters (push) Successful in 9m27s
Release / converters (push) Successful in 25s
Release / release (push) Successful in 34s
Release / wiki-mirror (push) Skipped

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- instructions/ingest-large-tree.md
- instructions/read-page-images.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw/derive.py
- tools/chemenu/filters/pdf.py
- tools/chemenu/filters/pdf_layout.py
- tools/chemenu/tests/test_raw_derive_binary.py
- types/source.guidance.md

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-15 06:36:47 +02:00
1 parent cb1c9a2963
commit 128c7f5c59
14 files changed
+144 -14

No files matched your search

@@ -193,7 +193,7 @@ tools/wikitool raw derive incoming/scans.zip
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before.
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document - and an `images` header field names the same pages, so that they outlast the output. An `unmatched` header field names the pages a WARN reports as left without markings because their text did not match the page layout. Both are written only when not empty, as page ranges (`6-8, 10`). A PDF without outline, markings, images and unmatched pages is derived as before.
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.