diff --git a/CHANGES.md b/CHANGES.md index 358916d..f42135a 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.1.0-beta.4 - 2026-10-14 - instructions/dev: converter-comparison nennt harness/compare/README.md als Ort der Einrichtung +## 10.1.0-beta.5 - 2026-10-14 - raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder **Author:** Torben Nehmer @@ -22,6 +22,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. **Medium impact** - raw derive PDF: Gliederung mit Seite und Zeile im Kopf, Hinterlegungen, Durchstreichungen und abweichende Textfarben als Auszeichnung im Text, Bildseiten als CHECK - Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release +- raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder **Low impact** - instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie @@ -34,7 +35,7 @@ Der Testkorpus mit echten Dokumenten (PDF, Mail, Office, ZIP) und sein Test lieg ### Große PDFs: Gliederung und Auszeichnungen -Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`; so findet ein Leser einen Abschnitt, ohne das ganze Dokument zu lesen. Im Text stehen drei Auszeichnungen für das, was die Seite nur durch ihre Form sagt: `…` für eine Hinterlegung, `~~…~~` für eine Durchstreichung und `*…*` für eine vom Fließtext abweichende Textfarbe ohne Link. Sie geben das Aussehen wieder, nicht die Bedeutung; ohne sie ist der Text derselbe wie zuvor. Das Feld `markings` im Kopf ist die Legende mit Anzahl und Seiten, kein Tag überschreitet eine Zeile oder Seitenmarke, und Bildseiten melden einen CHECK. Das Layout liest `pdfminer.six`, das neu in `requirements.txt` steht; ein PDF ohne Gliederung und Auszeichnungen wird wie bisher abgeleitet, vorhandene `derived.md` bleiben unberührt. +Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`. Im Text stehen drei Auszeichnungen als gewöhnliches Markdown für das, was die Seite nur durch ihre Form sagt: `~~…~~` für Durchstreichung, `**…**` für Textmarker, Fettdruck oder eine deutlich andere Textfarbe, `*…*` für Kursiv. Die Ableitung bestimmt erst die Struktur (Fließtext, Code, Tabellengitter, wiederkehrende Vorlagen) und zeichnet nur aus, wo die Form eindeutig ist; Fett, Kursiv und Farbe zählen nur im Fließtext. Ohne die Auszeichnungen ist der Text derselbe wie zuvor. Das Feld `markings` ist die Legende mit höchstens drei Zeilen, kein Tag überschreitet eine Zeile oder Seitenmarke. Der CHECK nennt nur Seiten mit inhaltlichem Bild oder Diagramm. Das Layout liest `pdfminer.six`, neu in `requirements.txt`; vorhandene `derived.md` bleiben unberührt. --- diff --git a/VERSION b/VERSION index 164ecff..040658f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.1.0-beta.4 +10.1.0-beta.5 diff --git a/docs/how-to/ingest-a-source.md b/docs/how-to/ingest-a-source.md index 53e46b6..17f81f9 100644 --- a/docs/how-to/ingest-a-source.md +++ b/docs/how-to/ingest-a-source.md @@ -46,11 +46,12 @@ pick a directory: sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the - header of the text, each entry with its page and the line where that page starts; text over a - coloured background, in another colour than the running text, or struck through is marked in - the text (``, `*…*`, `~~…~~`) with a legend in the header, and what a - colour means is for the reader to take from the document itself. Pages with pictures are - named in a `CHECK` line, because the pictures are not in the text. + header of the text, each entry with its page and the line where that page starts; struck-through + text, highlighted, bold or clearly differently coloured text, and italic text are marked in the + text (`~~…~~`, `**…**`, `*…*`) with a legend in the header, and what a colour means is for the + reader to take from the document itself. The tool marks sparingly, in running text and + not in code or tables. Pages with pictures or drawings that carry content are named in a `CHECK` + line, because the pictures are not in the text. - **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The agent derives the same text from that file with `raw fetch --html`. For a website you read often, sign in once instead and let the wiki fetch it with your session: diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index 70596d0..9b3c590 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -193,7 +193,7 @@ tools/wikitool raw derive incoming/scans.zip - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment. -- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only - `…` for text over a coloured background, `*…*` for text in a colour other than the running text that is not under a link, `~~…~~` for strike-through. A `markings` header field is the legend (form, colour, places, pages); removing the markings gives the text the PDF has without them. A CHECK names the pages with images, which the text does not contain. A PDF without outline, markings and images is derived as before. +- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before. - A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index f4c7399..b3f3afb 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -394,15 +394,25 @@ derivation, accepted with it, and an existing one is never overwritten. **A PDF's outline and markings are in the derivation.** The header of `derived.md` lists the PDF's outline as `outline` entries - title, level, page and the line of that page's `` marker in the file - so a reader finds a section without reading the whole -text. In the text, three markings carry what the page shows by form alone: a background fill is -`…`, a strike-through is `~~…~~`, and text in a colour other than the -running text, without a link under it, is `*…*`. A marking records how the page looks, not what -it means: nothing in the tool decides that yellow is a deadline, and removing the markings gives -the same text as without them. The `markings` header field is the legend - each marking in use, -with its count and pages. No tag crosses a line or a page marker. A page whose text cannot be -matched to its layout gets no markings and a WARN; a PDF with images the text does not hold gets -a CHECK naming the pages, because the tool has no OCR. A PDF without outline or markings is -derived as before. Reading the layout needs `pdfminer.six` beside `pypdf`. +text. In the text, three markings carry what the page shows by form alone, as plain Markdown: +`~~…~~` is a strike-through, `**…**` is a highlighter stroke, bold type, or a colour clearly +unlike the running text (not under a link), and `*…*` is italic. The tool marks as little as it +can stand behind: it reads the structure first - running text, program text, ruled tables, and +shapes the document repeats on its pages - and marks only where the form is unambiguous. Bold, +italic and colour count inside running text only; highlighter and strike-through also in the +regulations that use them, but never in program text, a table's grid or a recurring template. A +line that is bold or italic from end to end is a heading or a quotation, not a marking. A marking +records how the page looks, not what it means: nothing in the tool decides that yellow is a +deadline, and removing the markings gives the same text as without them. The `markings` header +field is the legend, at most three lines, one per kind in use, with its count and pages; the +`**` line also names the highlighter colours with their pages. A tag never starts or ends on +whitespace and never crosses a line or a page marker. A page whose text cannot be matched to its +layout gets no markings and a WARN. A CHECK names the pages whose pictures the text does not +hold, because the tool has no OCR: a picture counts unless it is a background covering most of +the page, a filler a few pixels or points in size, or the same picture on a fifth of the pages +(a logo, a frame); a vector drawing with many paths and little text counts too. A PDF without +outline or markings is derived as before. Reading the layout needs `pdfminer.six` beside +`pypdf`. **The original stays.** The derivation is the tool's reading of it, never the source: with the original kept, a claim in `kb/` can still be checked against what the document said when a diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 777e7ea..a4732c7 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -1608,7 +1608,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment. -- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only - `…` for text over a coloured background, `*…*` for text in a colour other than the running text that is not under a link, `~~…~~` for strike-through. A `markings` header field is the legend (form, colour, places, pages); removing the markings gives the text the PDF has without them. A CHECK names the pages with images, which the text does not contain. A PDF without outline, markings and images is derived as before. +- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before. - A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. diff --git a/tools/chemenu/commands/raw/derive.py b/tools/chemenu/commands/raw/derive.py index 08d81b1..344fa31 100644 --- a/tools/chemenu/commands/raw/derive.py +++ b/tools/chemenu/commands/raw/derive.py @@ -371,13 +371,16 @@ def _derive_next(target: Path) -> str: "`multipart/alternative` is not an attachment.", "A PDF also gets, where the document has them: an `outline` header field (one entry per " "outline item: title, level, page and the line of `derived.md` where that page's " - "marker is), and markings in the text that follow the visible form only - `\">…` for text over a coloured background, `*…*` for text in a " - "colour other than the running text that is not under a link, `~~…~~` for strike-through. " - "A `markings` header field is the legend (form, colour, places, pages); removing the " - "markings gives the text the PDF has without them. A CHECK names the pages with images, " - "which the text does not contain. A PDF without outline, markings and images is derived " - "as before.", + "marker is), and markings in the text that follow the visible form only, in plain " + "Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a " + "colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, " + "italic and colour count in running text only, never in program text, a table's grid or " + "a shape the document repeats on its pages. A `markings` header field is the legend (at " + "most three lines: form, places, pages; the `**` line names the highlighter colours); " + "removing the markings gives the text the PDF has without them, and a tag never starts " + "or ends on whitespace. A CHECK names the pages with a picture or drawing the text does " + "not contain - not a background, a filler or a picture repeated across the document. A " + "PDF without outline, markings and images is derived as before.", "A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder " "`.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` " "per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet " diff --git a/tools/chemenu/filters/pdf.py b/tools/chemenu/filters/pdf.py index 8fe93cc..9714903 100644 --- a/tools/chemenu/filters/pdf.py +++ b/tools/chemenu/filters/pdf.py @@ -1,5 +1,5 @@ -"""PDF: the text and the outline are read with `pypdf`, the look of the characters (highlight, -colour, strike-through, images) with `pdfminer.six` in `pdf_layout`. Both are imported when a PDF is +"""PDF: the text and the outline are read with `pypdf`, the look of the page (strike-through, +highlight, bold, italic, pictures) with `pdfminer.six` in `pdf_layout`. Both are imported when a PDF is derived, never at start-up.""" from __future__ import annotations @@ -67,7 +67,7 @@ class PdfFilter(Filter): warnings.extend(annotation.warnings) if annotation.markings: fields.append(("markings", list(annotation.markings))) - image_pages = [number for number, layout in enumerate(layouts, 1) if layout.images] + image_pages = list(annotation.image_pages) if image_pages: checks.append( f"Images on page(s) {pdf_layout.page_ranges(image_pages)} are not contained in the text - " diff --git a/tools/chemenu/filters/pdf_layout.py b/tools/chemenu/filters/pdf_layout.py index 3390f3b..ce2b3a7 100644 --- a/tools/chemenu/filters/pdf_layout.py +++ b/tools/chemenu/filters/pdf_layout.py @@ -1,26 +1,78 @@ -"""What the PDF filter reads from the page layout: per character its text colour, the colour -filled behind it, a line through it and a link over it - and the mapping of those onto the text -`pypdf` extracted. Read with `pdfminer.six` (imported when `analyse` runs, never at start-up). +"""What the PDF filter reads from the page layout, and the markings it lays onto the text `pypdf` +extracted. Read with `pdfminer.six` (imported when `analyse` runs, never at start-up). `pypdf` stays the source of the text, so a PDF's wording does not change; `pdfminer.six` only says what each character looks like. The two are matched page by page over their non-space -characters, and the markings are laid onto the `pypdf` text. The markings follow the visible -form and nothing else: the filter has no model, and what a colour means is written in the -document itself.""" +characters, and the markings are laid onto the `pypdf` text. The filter has no model, so it marks +as little as it can stand behind: structure first (running text, code, tables, shapes that recur +on many pages), then three markings - `~~` struck, `**` highlighted, bold or in a clearly +different colour, `*` italic - and only where the form is unambiguous. A colour is mostly +decoration, so it is a marking only inside running text. Every threshold is a constant below.""" from __future__ import annotations import difflib +from bisect import bisect_left, bisect_right +import hashlib import io import itertools +import re import unicodedata +from collections import Counter from dataclasses import dataclass from typing import Iterable, Optional MIN_COVERAGE = 0.9 # a page whose text matches the layout less than this gets no markings -MAX_HIGHLIGHT_LINES = 2.5 # a fill taller than this many text heights is a panel, not a highlight -STRIKE_BAND = (0.3, 0.75) # where in the character's height a line counts as going through it + +# Structure: which characters count as running text. +FLOW_SIZE = (0.85, 1.20) # a line's type size, as a multiple of the body size +FLOW_MIN_CHARS = 15 # a line shorter than this (in non-space characters) is a label, not running text +FLOW_MIN_PLAIN = 0.60 # share of a line's characters that are plain running type +SEGMENT_GAP = 2.5 # a gap wider than this many type heights splits a line (columns, table cells) +LINE_TOLERANCE = 0.5 # characters whose baselines differ by less than this many type heights share a line +CODE_MIN_SIGNALS = 3 # a block of lines is code when this many of the five signals hold +CODE_MIN_COLOURS = 5 +CODE_OPERATOR_SHARE = 0.15 +CODE_END_SHARE = 0.30 # share of lines ending in ; { } ) , : +CODE_MONO_SHARE = 0.5 +CODE_BLOCK_GAP = 1.8 # lines closer than this many type heights belong to one block +MONO_NAMES = re.compile(r"courier|mono|consolas|menlo|typewriter|lucida console|andale|inconsolata|fixed", re.I) +OPERATORS = set("{}()[]<>=+-*/%&|!^~;:\"'\\#@$") +TEMPLATE_SHARE = 0.20 # a shape on this share of the pages (and on at least TEMPLATE_PAGES) is a template +TEMPLATE_PAGES = 3 +GRID_TOLERANCE = 3.0 # points: how close a horizontal and a vertical line must come to cross +GRID_MIN_LINES = 2 # a table region needs this many lines in each direction + +# Rule 2: strike-through. +STRIKE_BAND = (0.30, 0.75) # where in the character's height a line counts as going through it +STRIKE_MAX_WIDTH = 1.5 # points +STRIKE_MIN_CHARS = 2 +STRIKE_OVERHANG = 0.5 # type heights the line may reach past the first/last struck character +STRIKE_EDGE = 0.05 # share of the page that is edge FLAT = 1.5 # a filled shape this thin (points) is a line +# Rule 3: highlighter. +HIGHLIGHT_MAX_HEIGHT = 1.8 # type heights +HIGHLIGHT_MAX_EXTRA = 2.0 # type heights wider than the text it covers +HIGHLIGHT_MAX_AREA = 0.08 # share of the page +HIGHLIGHT_BRIGHT = 0xC0 # the lightest channel at least this +HIGHLIGHT_SPAN = 0x60 # the spread between the lightest and darkest channel at least this + +# Rule 4: bold, italic and a clearly different colour. +BOLD_NAMES = re.compile(r"bold|black|heavy|semibold|demi", re.I) +ITALIC_NAMES = re.compile(r"italic|oblique", re.I) +COLOUR_DISTANCE = 0x30 # per channel, from the running text colour +COLOUR_MAX_SHARE = 0.40 # share of a line that may be coloured + +# Rule 6: pictures. +IMAGE_MAX_PAGE_SHARE = 0.80 # a picture covering more of the page is a background +IMAGE_MIN_PIXELS = 4 # a source this size or smaller is a filler +IMAGE_MIN_POINTS = 10 # drawn smaller than this in either direction is a filler +VECTOR_MIN_PATHS = 20 +VECTOR_MIN_PAGE_SHARE = 0.02 +VECTOR_MAX_TEXT_PER_PATH = 2.0 # characters inside the cluster per path: more is a table, not a drawing +VECTOR_JOIN = 4.0 # points: paths this close belong to one drawing +MAX_PATHS = 20000 # a page with more paths than this is not scanned for drawings + NAMES = { "#ffff00": "gelb", "#00ff00": "grün", "#ff0000": "rot", "#0000ff": "blau", "#00ffff": "türkis", "#ff00ff": "pink", "#ffa500": "orange", "#ffc000": "orange", @@ -29,23 +81,70 @@ NAMES = { "#800080": "violett", "#800000": "dunkelrot", "#808000": "olivgelb", } -Flags = tuple # (background hex | None, True | None for struck, colour hex | None) +Box = tuple # (x0, y0, x1, y1) +Flags = tuple # (strong, struck, italic): True or None per layer LAYERS = 3 @dataclass(frozen=True) -class Glyph: +class Char: + """One non-space character as the page draws it.""" + text: str + box: Box + size: float # type height: the em of the font + font: str colour: Optional[str] - background: Optional[str] - struck: bool + direction: Optional[int] # direction of the baseline in degrees (0, 90, 180, 270); None for any other angle + order: int # position in the drawing order linked: bool +@dataclass(frozen=True) +class Fill: + order: int + box: Box + colour: str + + +@dataclass(frozen=True) +class Line: + """A thin line: `start` to `end` along its direction, `pos` across it, `width` its thickness.""" + + start: float + end: float + pos: float + width: float + + +@dataclass(frozen=True) +class ImageUse: + box: Box + pixels: tuple[int, int] + key: str # a digest of the image's content: the same picture drawn twice has the same key + + @dataclass(frozen=True) class PageLayout: - glyphs: tuple[Glyph, ...] - images: bool + width: float + height: float + chars: tuple[Char, ...] + fills: tuple[Fill, ...] + hlines: tuple[Line, ...] + vlines: tuple[Line, ...] + images: tuple[ImageUse, ...] + paths: tuple[Box, ...] + + +@dataclass(frozen=True) +class Glyph: + """What `annotate` needs per character: its text and the markings that survived the rules.""" + + text: str + strong: bool + struck: bool + italic: bool + highlight: Optional[str] = None # the highlighter colour behind a strong glyph, for the legend @dataclass(frozen=True) @@ -53,6 +152,7 @@ class Annotation: texts: tuple[str, ...] markings: tuple[str, ...] # the legend warnings: tuple[str, ...] + image_pages: tuple[int, ...] def colour_name(value: str) -> str: @@ -87,15 +187,36 @@ def _hex(value) -> Optional[str]: return "#%02x%02x%02x" % (round(red * 255), round(green * 255), round(blue * 255)) +def _channels(value: str) -> tuple[int, int, int]: + return int(value[1:3], 16), int(value[3:5], 16), int(value[5:7], 16) + + def _is_white(value: str) -> bool: - return all(int(value[i:i + 2], 16) >= 0xF8 for i in (1, 3, 5)) + return all(channel >= 0xF8 for channel in _channels(value)) + + +def _direction(matrix) -> Optional[int]: + a, b = matrix[0], matrix[1] + if abs(b) <= 0.01 * abs(a): + return 0 if a > 0 else 180 + if abs(a) <= 0.01 * abs(b): + return 90 if b > 0 else 270 + return None + + +def _image_key(item) -> str: + try: + raw = item.stream.get_rawdata() or b"" + except Exception: # a stream pdfminer cannot hand over is told apart by its object id alone + raw = b"" + return hashlib.blake2b(raw[:65536] + str(len(raw)).encode() + str(item.srcsize).encode(), digest_size=8).hexdigest() def analyse(data: bytes) -> list[PageLayout]: - """The glyphs and images of each page, in page order. Raises whatever `pdfminer.six` - raises on a document it cannot read.""" + """The characters, fills, lines, pictures and paths of each page, in page order. Raises + whatever `pdfminer.six` raises on a document it cannot read.""" from pdfminer.converter import PDFPageAggregator - from pdfminer.layout import LTChar, LTCurve, LTFigure, LTImage, LTLine + from pdfminer.layout import LTChar, LTCurve, LTFigure, LTImage, LTLine, LTRect from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager from pdfminer.pdfpage import PDFPage from pdfminer.pdftypes import resolve1 @@ -107,7 +228,7 @@ def analyse(data: bytes) -> list[PageLayout]: else: yield item - def link_boxes(page) -> Optional[list[tuple[float, float, float, float]]]: + def link_boxes(page) -> Optional[list[Box]]: """Link rectangles in the coordinates of the layout; None on a rotated page, where they cannot be told apart from the layout's.""" if page.rotate % 360: @@ -134,61 +255,441 @@ def analyse(data: bytes) -> list[PageLayout]: for page in PDFPage.get_pages(io.BytesIO(data)): interpreter.process_page(page) links = link_boxes(page) - chars, highlights, lines, images = [], [], [], False + width, height = page.mediabox[2] - page.mediabox[0], page.mediabox[3] - page.mediabox[1] + if page.rotate % 180: + width, height = height, width + chars, fills, hlines, vlines, images, paths = [], [], [], [], [], [] for order, item in enumerate(flatten(device.get_result())): if isinstance(item, LTChar): - chars.append((order, item)) + if item.get_text().isspace(): + continue + direction = _direction(item.matrix) + x0, y0, x1, y1 = item.bbox + size = (y1 - y0) if direction in (0, 180) else (x1 - x0) if direction in (90, 270) else item.size + centre_x, centre_y = (x0 + x1) / 2, (y0 + y1) / 2 + linked = links is None or any( + bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1 for bx0, by0, bx1, by1 in links + ) + chars.append(Char(item.get_text(), item.bbox, size, item.fontname, _hex(item.graphicstate.ncolor), direction, order, linked)) elif isinstance(item, LTImage): - images = True + images.append(ImageUse(item.bbox, tuple(item.srcsize or (0, 0)), _image_key(item))) elif isinstance(item, LTCurve): x0, y0, x1, y1 = item.bbox fill = _hex(item.non_stroking_color) if item.fill else None + stroke_width = max(item.linewidth or 0, 0) + if len(paths) < MAX_PATHS: + paths.append(item.bbox) if y1 - y0 <= FLAT and x1 - x0 >= 1 and (fill is not None or item.stroke or isinstance(item, LTLine)): - lines.append((x0, x1, (y0 + y1) / 2)) + hlines.append(Line(x0, x1, (y0 + y1) / 2, max(y1 - y0, stroke_width))) + elif x1 - x0 <= FLAT and y1 - y0 >= 1 and (fill is not None or item.stroke or isinstance(item, LTLine)): + vlines.append(Line(y0, y1, (x0 + x1) / 2, max(x1 - x0, stroke_width))) elif fill is not None and not _is_white(fill): - highlights.append((order, item.bbox, fill)) - glyphs = [] - for order, char in chars: - if char.get_text().isspace(): - continue - x0, y0, x1, y1 = char.bbox - centre_x, centre_y, height = (x0 + x1) / 2, (y0 + y1) / 2, y1 - y0 - background = next(( - colour for placed, (bx0, by0, bx1, by1), colour in reversed(highlights) - if placed < order and bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1 - and by1 - by0 <= MAX_HIGHLIGHT_LINES * height - ), None) - struck = any( - lx0 <= centre_x <= lx1 and y0 + STRIKE_BAND[0] * height <= ly <= y0 + STRIKE_BAND[1] * height - for lx0, lx1, ly in lines - ) - linked = links is not None and any( - bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1 for bx0, by0, bx1, by1 in links - ) - glyphs.append(Glyph( - char.get_text(), _hex(char.graphicstate.ncolor), background, struck, - linked or links is None, - )) - layouts.append(PageLayout(tuple(glyphs), images)) + fills.append(Fill(order, item.bbox, fill)) + elif isinstance(item, LTRect) and item.stroke and not item.fill and stroke_width <= FLAT: + hlines.extend((Line(x0, x1, y0, stroke_width), Line(x0, x1, y1, stroke_width))) + vlines.extend((Line(y0, y1, x0, stroke_width), Line(y0, y1, x1, stroke_width))) + layouts.append(PageLayout( + width, height, tuple(chars), tuple(fills), tuple(hlines), tuple(vlines), tuple(images), tuple(paths), + )) return layouts -def body_colour(layouts: Iterable[PageLayout]) -> Optional[str]: - """The fill colour with the most characters: the colour of the running text.""" - counts: dict[str, int] = {} +# --- templates: what the document repeats on its pages ------------------------------------------- + +def _snap(*values: float) -> tuple: + return tuple(round(value / 2) * 2 for value in values) + + +@dataclass(frozen=True) +class Templates: + """Shapes and pictures that recur on many pages: a letterhead, a frame, a logo. They are part + of the page, not of its content, so no rule reads them.""" + + lines: frozenset + fills: frozenset + paths: frozenset + images: frozenset + + +def _recurring(per_page: list[set]) -> frozenset: + if not per_page: + return frozenset() + counts = Counter(key for keys in per_page for key in keys) + needed = max(TEMPLATE_PAGES, TEMPLATE_SHARE * len(per_page)) + return frozenset(key for key, pages in counts.items() if pages >= needed) + + +def _line_key(kind: str, line: Line) -> tuple: + return (kind,) + _snap(line.start, line.end, line.pos) + + +def templates(layouts: list[PageLayout]) -> Templates: + return Templates( + _recurring([ + {_line_key("h", line) for line in layout.hlines} | {_line_key("v", line) for line in layout.vlines} + for layout in layouts + ]), + _recurring([{_snap(*fill.box) + (fill.colour,) for fill in layout.fills} for layout in layouts]), + _recurring([{_snap(*box) for box in layout.paths} for layout in layouts]), + _recurring([{image.key for image in layout.images} for layout in layouts]), + ) + + +# --- structure: table regions, line pieces, code ---------------------------------------------------- + +def _inside(box: Box, x: float, y: float, margin: float = 0.0) -> bool: + return box[0] - margin <= x <= box[2] + margin and box[1] - margin <= y <= box[3] + margin + + +def _centre(box: Box) -> tuple[float, float]: + return (box[0] + box[2]) / 2, (box[1] + box[3]) / 2 + + +@dataclass(frozen=True) +class Grid: + regions: tuple[Box, ...] # the extent of each table grid + h_used: frozenset # indices of the horizontal lines that belong to a grid + v_used: frozenset + + +def _grids(layout: PageLayout, template: Templates) -> Grid: + """Connected sets of crossing horizontal and vertical lines with at least GRID_MIN_LINES + of each: the ruling of a table.""" + hs = [i for i, line in enumerate(layout.hlines) if _line_key("h", line) not in template.lines] + vs = [i for i, line in enumerate(layout.vlines) if _line_key("v", line) not in template.lines] + if len(hs) < GRID_MIN_LINES or len(vs) < GRID_MIN_LINES: + return Grid((), frozenset(), frozenset()) + parent = {("h", i): ("h", i) for i in hs} | {("v", j): ("v", j) for j in vs} + + def find(node): + while parent[node] != node: + parent[node] = parent[parent[node]] + node = parent[node] + return node + + if len(hs) * len(vs) > 1_000_000: # too many lines to cross: one ruled area + for node in parent: + parent[find(node)] = find(("h", hs[0])) + else: + for i in hs: + h = layout.hlines[i] + for j in vs: + v = layout.vlines[j] + if h.start - GRID_TOLERANCE <= v.pos <= h.end + GRID_TOLERANCE and v.start - GRID_TOLERANCE <= h.pos <= v.end + GRID_TOLERANCE: + parent[find(("h", i))] = find(("v", j)) + members: dict[tuple, list] = {} + for node in parent: + members.setdefault(find(node), []).append(node) + regions, h_used, v_used = [], set(), set() + for nodes in members.values(): + h_nodes = [i for kind, i in nodes if kind == "h"] + v_nodes = [j for kind, j in nodes if kind == "v"] + if len(h_nodes) < GRID_MIN_LINES or len(v_nodes) < GRID_MIN_LINES: + continue + boxes = [(layout.hlines[i].start, layout.hlines[i].pos, layout.hlines[i].end, layout.hlines[i].pos) for i in h_nodes] + boxes += [(layout.vlines[j].pos, layout.vlines[j].start, layout.vlines[j].pos, layout.vlines[j].end) for j in v_nodes] + regions.append((min(b[0] for b in boxes), min(b[1] for b in boxes), max(b[2] for b in boxes), max(b[3] for b in boxes))) + h_used.update(h_nodes) + v_used.update(v_nodes) + return Grid(tuple(regions), frozenset(h_used), frozenset(v_used)) + + +@dataclass(frozen=True) +class Piece: + """A run of upright characters on one line with no wide gap: a line, or a cell of a line.""" + + members: tuple[int, ...] # indices into PageLayout.chars, left to right + row: int + + +def _pieces(layout: PageLayout) -> list[list[Piece]]: + """The upright characters as rows (top to bottom) of pieces (left to right).""" + upright = sorted((i for i, char in enumerate(layout.chars) if char.direction == 0), key=lambda i: -layout.chars[i].box[1]) + rows: list[list[int]] = [] + base = 0.0 + for i in upright: + char = layout.chars[i] + if not rows or abs(char.box[1] - base) > LINE_TOLERANCE * char.size: + rows.append([]) + base = char.box[1] + rows[-1].append(i) + out = [] + for number, row in enumerate(rows): + row.sort(key=lambda i: layout.chars[i].box[0]) + pieces, current = [], [row[0]] + for previous, i in zip(row, row[1:]): + gap = layout.chars[i].box[0] - layout.chars[previous].box[2] + if gap > SEGMENT_GAP * layout.chars[i].size: + pieces.append(Piece(tuple(current), number)) + current = [] + current.append(i) + pieces.append(Piece(tuple(current), number)) + out.append(pieces) + return out + + +def _is_bold(char: Char) -> bool: + return bool(BOLD_NAMES.search(char.font)) + + +def _is_italic(char: Char) -> bool: + return bool(ITALIC_NAMES.search(char.font)) + + +def _code_chars(layout: PageLayout, rows: list[list[Piece]]) -> set[int]: + """The characters of blocks of lines that read as program text: at least CODE_MIN_SIGNALS of + a monospaced font, many operators, syntax colours, varying indentation and statement endings.""" + found: set[int] = set() + + def judge(block: list[list[Piece]]) -> None: + if len(block) < 3: + return + chars = [layout.chars[i] for pieces in block for piece in pieces for i in piece.members] + lefts = {round(layout.chars[pieces[0].members[0]].box[0] / (0.5 * chars[0].size)) for pieces in block} + ends = [layout.chars[pieces[-1].members[-1]].text for pieces in block] + signals = sum([ + sum(bool(MONO_NAMES.search(char.font)) for char in chars) / len(chars) >= CODE_MONO_SHARE, + sum(char.text in OPERATORS for char in chars) / len(chars) >= CODE_OPERATOR_SHARE, + len({char.colour for char in chars if char.colour}) >= CODE_MIN_COLOURS, + len(lefts) >= 2, + sum(end in ";{}),:" for end in ends) / len(ends) >= CODE_END_SHARE, + ]) + if signals >= CODE_MIN_SIGNALS: + found.update(i for pieces in block for piece in pieces for i in piece.members) + + block: list[list[Piece]] = [] + previous = None + for pieces in rows: + char = layout.chars[pieces[0].members[0]] + if previous is not None and previous - char.box[1] > CODE_BLOCK_GAP * char.size: + judge(block) + block = [] + block.append(pieces) + previous = char.box[1] + judge(block) + return found + + +# --- the rules --------------------------------------------------------------------------------- + +@dataclass(frozen=True) +class Context: + body_size: Optional[float] + body_colour: Optional[str] + template: Templates + + +def context(layouts: list[PageLayout]) -> Context: + sizes = Counter(round(char.size * 2) / 2 for layout in layouts for char in layout.chars if char.direction == 0) + body_size = max(sizes, key=lambda size: (sizes[size], size)) if sizes else None + colours: Counter = Counter() for layout in layouts: - for glyph in layout.glyphs: - if glyph.colour is not None: - counts[glyph.colour] = counts.get(glyph.colour, 0) + 1 - return max(counts, key=lambda colour: (counts[colour], colour)) if counts else None + for char in layout.chars: + if char.direction == 0 and char.colour and body_size and abs(char.size - body_size) <= 0.5: + colours[char.colour] += 1 + body_colour = max(colours, key=lambda colour: (colours[colour], colour)) if colours else None + return Context(body_size, body_colour, templates(layouts)) -def _flags(glyph: Glyph, body: Optional[str]) -> Flags: - other = glyph.colour if glyph.colour is not None and glyph.colour != body and not glyph.linked else None - return (glyph.background, True if glyph.struck else None, other) +def _struck(layout: PageLayout, grid: Grid, template: Templates, table: set[int]) -> set[int]: + """Rule 2: a thin line through the middle of at least STRIKE_MIN_CHARS characters that does + not reach further than STRIKE_OVERHANG past them, measured in the characters' own frame + (so text turned by a quarter is struck by a vertical line). Not a table's ruling, not a + recurring shape, and not a line at the page edge.""" + found: set[int] = set() + for horizontal in (True, False): + lines = layout.hlines if horizontal else layout.vlines + used = grid.h_used if horizontal else grid.v_used + kind = "h" if horizontal else "v" + extent = layout.height if horizontal else layout.width + along = [ + (_centre(layout.chars[i].box)[0 if horizontal else 1], i) + for i in range(len(layout.chars)) + if layout.chars[i].direction in ((0, 180) if horizontal else (90, 270)) and i not in table + ] + along.sort() + keys = [key for key, _ in along] + for n, line in enumerate(lines): + if n in used or line.width > STRIKE_MAX_WIDTH or _line_key(kind, line) in template.lines: + continue + if line.pos < STRIKE_EDGE * extent or line.pos > (1 - STRIKE_EDGE) * extent: + continue + low, high = bisect_left(keys, line.start), bisect_right(keys, line.end) + hits = [] + for _, i in along[low:high]: + char = layout.chars[i] + x0, y0, x1, y1 = char.box + lo, hi = (y0, y1) if horizontal else (x0, x1) + if char.direction in (0, 270): + band = (lo + STRIKE_BAND[0] * char.size, lo + STRIKE_BAND[1] * char.size) + else: + band = (hi - STRIKE_BAND[1] * char.size, hi - STRIKE_BAND[0] * char.size) + if band[0] <= line.pos <= band[1]: + hits.append(i) + if len(hits) < STRIKE_MIN_CHARS: + continue + ends = [(layout.chars[i].box[0], layout.chars[i].box[2]) if horizontal else (layout.chars[i].box[1], layout.chars[i].box[3]) for i in hits] + reach = STRIKE_OVERHANG * max(layout.chars[i].size for i in hits) + if line.start < min(a for a, _ in ends) - reach or line.end > max(b for _, b in ends) + reach: + continue + found.update(hits) + return found -def _align(text: str, glyphs: tuple[Glyph, ...], flags: list[Flags]) -> tuple[dict[int, Flags], float]: +def _highlighted(layout: PageLayout, template: Templates, table: set[int]) -> dict[int, str]: + """Rule 3: a bright, saturated fill drawn under the text, about as high as a line and no + wider than the text it covers plus HIGHLIGHT_MAX_EXTRA type heights. The colour per character.""" + found: dict[int, str] = {} + page_area = layout.width * layout.height + for fill in layout.fills: + if _snap(*fill.box) + (fill.colour,) in template.fills: + continue + red, green, blue = _channels(fill.colour) + if max(red, green, blue) < HIGHLIGHT_BRIGHT or max(red, green, blue) - min(red, green, blue) < HIGHLIGHT_SPAN: + continue + x0, y0, x1, y1 = fill.box + if page_area and (x1 - x0) * (y1 - y0) > HIGHLIGHT_MAX_AREA * page_area: + continue + covered = [ + i for i, char in enumerate(layout.chars) + if char.order > fill.order and i not in table and _inside(fill.box, *_centre(char.box)) + ] + if not covered: + continue + size = max(layout.chars[i].size for i in covered) + left = min(layout.chars[i].box[0] for i in covered) + right = max(layout.chars[i].box[2] for i in covered) + if y1 - y0 > HIGHLIGHT_MAX_HEIGHT * size or (x1 - x0) - (right - left) > HIGHLIGHT_MAX_EXTRA * size: + continue + for i in covered: + found[i] = fill.colour + return found + + +def _different(colour: Optional[str], body: Optional[str]) -> bool: + if colour is None or body is None: + return False + return any(abs(a - b) > COLOUR_DISTANCE for a, b in zip(_channels(colour), _channels(body))) + + +def page_glyphs(layout: PageLayout, ctx: Context) -> list[Glyph]: + """One glyph per character of the page, in drawing order, with the markings the rules leave.""" + grid = _grids(layout, ctx.template) + table = { + i for i, char in enumerate(layout.chars) + if any(_inside(region, *_centre(char.box), margin=1.0) for region in grid.regions) + } + rows = _pieces(layout) + code = _code_chars(layout, rows) + strong = {i: None for i in range(len(layout.chars))} + italic: set[int] = set() + if ctx.body_size: + for pieces in rows: + for piece in pieces: + members = [i for i in piece.members if i not in table and i not in code] + chars = [layout.chars[i] for i in members] + if len(members) < FLOW_MIN_CHARS or len(members) != len(piece.members): + continue + if not all(FLOW_SIZE[0] * ctx.body_size <= char.size <= FLOW_SIZE[1] * ctx.body_size for char in chars): + continue + plain = sum(not _is_bold(char) and not _is_italic(char) for char in chars) / len(chars) + if plain < FLOW_MIN_PLAIN: + continue + coloured = [i for i in members if _different(layout.chars[i].colour, ctx.body_colour) and not layout.chars[i].linked] + if len(coloured) / len(members) > COLOUR_MAX_SHARE: + coloured = [] + for i in members: + char = layout.chars[i] + if _is_bold(char) and not _is_italic(char) or i in coloured: + strong[i] = "bold" + if _is_italic(char): + italic.add(i) + for i, colour in _highlighted(layout, ctx.template, table).items(): + strong[i] = colour + struck = _struck(layout, grid, ctx.template, table) + return [ + Glyph(char.text, strong[i] is not None, i in struck, i in italic, strong[i] if strong[i] not in (None, "bold") else None) + for i, char in enumerate(layout.chars) + ] + + +def image_pages(layouts: list[PageLayout], ctx: Context) -> list[int]: + """Rule 6: the pages with a picture or a drawing that carries content - not a background, a + filler, or something the document repeats on its pages.""" + pages = [] + for number, layout in enumerate(layouts, 1): + page_area = layout.width * layout.height or 1.0 + if any(_content_image(image, page_area, ctx.template) for image in layout.images) or _drawing(layout, ctx.template): + pages.append(number) + return pages + + +def _content_image(image: ImageUse, page_area: float, template: Templates) -> bool: + x0, y0, x1, y1 = image.box + width, height = x1 - x0, y1 - y0 + return not ( + image.key in template.images + or width * height > IMAGE_MAX_PAGE_SHARE * page_area + or image.pixels[0] * image.pixels[1] <= IMAGE_MIN_PIXELS + or min(width, height) < IMAGE_MIN_POINTS + ) + + +def _drawing(layout: PageLayout, template: Templates) -> bool: + """A cluster of at least VECTOR_MIN_PATHS paths, big enough, with little text inside, and + that is not a table's ruling.""" + if not layout.paths or len(layout.paths) >= MAX_PATHS: + return False + grid = _grids(layout, template) + page_area = layout.width * layout.height or 1.0 + boxes = [ + box for box in layout.paths + if _snap(*box) not in template.paths + and (box[2] - box[0]) * (box[3] - box[1]) <= IMAGE_MAX_PAGE_SHARE * page_area + and not any(_inside(region, *_centre(box), margin=1.0) for region in grid.regions) + ] + if len(boxes) < VECTOR_MIN_PATHS: + return False + parent = list(range(len(boxes))) + + def find(n: int) -> int: + while parent[n] != n: + parent[n] = parent[parent[n]] + n = parent[n] + return n + + cell = 32.0 + cells: dict[tuple[int, int], list[int]] = {} + for n, (x0, y0, x1, y1) in enumerate(boxes): + ex0, ey0, ex1, ey1 = x0 - VECTOR_JOIN, y0 - VECTOR_JOIN, x1 + VECTOR_JOIN, y1 + VECTOR_JOIN + for cx in range(int(ex0 // cell), int(ex1 // cell) + 1): + for cy in range(int(ey0 // cell), int(ey1 // cell) + 1): + for m in cells.setdefault((cx, cy), []): + mx0, my0, mx1, my1 = boxes[m] + if mx0 - VECTOR_JOIN <= x1 and x0 - VECTOR_JOIN <= mx1 and my0 - VECTOR_JOIN <= y1 and y0 - VECTOR_JOIN <= my1: + parent[find(n)] = find(m) + cells[(cx, cy)].append(n) + clusters: dict[int, list[int]] = {} + for n in range(len(boxes)): + clusters.setdefault(find(n), []).append(n) + for members in clusters.values(): + if len(members) < VECTOR_MIN_PATHS: + continue + x0 = min(boxes[n][0] for n in members) + y0 = min(boxes[n][1] for n in members) + x1 = max(boxes[n][2] for n in members) + y1 = max(boxes[n][3] for n in members) + if (x1 - x0) * (y1 - y0) < VECTOR_MIN_PAGE_SHARE * page_area: + continue + text = sum(1 for char in layout.chars if _inside((x0, y0, x1, y1), *_centre(char.box))) + if text <= VECTOR_MAX_TEXT_PER_PATH * len(members): + return True + return False + + +# --- laying the markings onto the text --------------------------------------------------------- + +def _align(text: str, glyphs: list[Glyph], flags: list[Flags]) -> tuple[dict[int, Flags], float]: """The flags of each character of `text` that has a counterpart among the glyphs, and the share of `text`'s non-space characters that has one.""" layout_chars, layout_flags = [], [] @@ -246,10 +747,7 @@ def _line_flags(line: str, found: dict[int, Flags], start: int) -> list[Optional return flags -def _open_close(layer: int, value) -> tuple[str, str]: - if layer == 0: - return f'', "" - return ("~~", "~~") if layer == 1 else ("*", "*") +MARKS = (("**", "**"), ("~~", "~~"), ("*", "*")) def _emit(items: list[tuple[str, Optional[Flags]]], layer: int, record) -> str: @@ -261,30 +759,35 @@ def _emit(items: list[tuple[str, Optional[Flags]]], layer: int, record) -> str: if value is None: out.append(inner) else: - opening, closing = _open_close(layer, value) - out.append(opening + inner + closing) - record(layer, value) + out.append(MARKS[layer][0] + inner + MARKS[layer][1]) + record(layer) return "".join(out) def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation: - """`texts` with the markings laid on, the legend and the warnings. A page without - markings comes back as it is.""" - body = body_colour(layouts) + """`texts` with the markings laid on, the legend (one line per kind of marking, at most + three), the warnings and the pages whose pictures need a look. A page without markings + comes back as it is.""" + ctx = context(layouts) result = list(texts) - places: dict[tuple[int, object], list[int]] = {} + places: dict[int, list[int]] = {} + highlights: dict[str, list[int]] = {} unmatched: list[int] = [] for number, (text, layout) in enumerate(zip(texts, layouts), 1): - flags = [_flags(glyph, body) for glyph in layout.glyphs] + glyphs = page_glyphs(layout, ctx) + flags: list[Flags] = [ + (True if glyph.strong else None, True if glyph.struck else None, True if glyph.italic else None) + for glyph in glyphs + ] if not any(any(value is not None for value in flag) for flag in flags): continue - found, coverage = _align(text, layout.glyphs, flags) + found, coverage = _align(text, glyphs, flags) if coverage < MIN_COVERAGE: unmatched.append(number) continue - def record(layer: int, value, page: int = number) -> None: - places.setdefault((layer, value if layer != 1 else True), []).append(page) + def record(layer: int, page: int = number) -> None: + places.setdefault(layer, []).append(page) lines, start = [], 0 for line in text.split("\n"): @@ -292,27 +795,25 @@ def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation: start += len(line) + 1 lines.append(line if not any(line_flags) else _emit(list(zip(line, line_flags)), 0, record)) result[number - 1] = "\n".join(lines) + for glyph in glyphs: + if glyph.highlight: + highlights.setdefault(glyph.highlight, []) + if number not in highlights[glyph.highlight]: + highlights[glyph.highlight].append(number) markings = [] - for (layer, value), pages in sorted(places.items(), key=lambda entry: (entry[0][0], str(entry[0][1]))): - if layer == 0: - opening, closing = _open_close(0, value) - markings.append(f"{opening}…{closing} | background {value} ({colour_name(value)}) | places {len(pages)} | pages {page_ranges(pages)}") - strike = places.get((1, True)) - if strike: - markings.append(f"~~…~~ | strike-through | places {len(strike)} | pages {page_ranges(strike)}") - colours = {value: pages for (layer, value), pages in places.items() if layer == 2} - if colours: - all_pages = [page for pages in colours.values() for page in pages] - names = ", ".join(sorted(colours)) - markings.append( - f"*…* | text colour other than the running text ({body}), not under a link: {names} | " - f"places {len(all_pages)} | pages {page_ranges(all_pages)}" - ) + if 0 in places: + shades = "; ".join(f"{colour_name(colour)} on page(s) {page_ranges(pages)}" for colour, pages in sorted(highlights.items())) + what = "highlighted" + (f" ({shades})" if shades else "") + ", bold, or in a colour clearly unlike the running text" + markings.append(f"**…** | {what} | places {len(places[0])} | pages {page_ranges(places[0])}") + if 1 in places: + markings.append(f"~~…~~ | struck through | places {len(places[1])} | pages {page_ranges(places[1])}") + if 2 in places: + markings.append(f"*…* | italic | places {len(places[2])} | pages {page_ranges(places[2])}") warnings = [] if unmatched: warnings.append( f"Page(s) {page_ranges(unmatched)}: under {round(MIN_COVERAGE * 100)}% of the text could be matched to the " - "page layout, so the markings (highlight, colour, strike-through) are left out there." + "page layout, so the markings (highlight, bold, colour, strike-through, italic) are left out there." ) - return Annotation(tuple(result), tuple(markings), tuple(warnings)) + return Annotation(tuple(result), tuple(markings), tuple(warnings), tuple(image_pages(layouts, ctx))) diff --git a/tools/chemenu/tests/binary_formats.py b/tools/chemenu/tests/binary_formats.py index aba3a27..a5d14ac 100644 --- a/tools/chemenu/tests/binary_formats.py +++ b/tools/chemenu/tests/binary_formats.py @@ -51,20 +51,42 @@ def build_pdf(pages: list[list[str]], *, title: Optional[str] = None) -> bytes: return bytes(out) -Run = Union[str, tuple[str, dict]] # text, or text and options: color, highlight, strike, link +Run = Union[str, tuple[str, dict]] # text, or text and options: color, highlight, strike, link, bold, italic, size, rotate + + +@dataclass(frozen=True) +class Picture: + """An image drawn at `x`, `y` (lower left) in `width` x `height` points from a + `pixels` x `pixels` raster; `gray` makes the pixel content, so equal grays are the same image.""" + + x: float + y: float + width: float + height: float + pixels: int = 8 + gray: int = 0x80 @dataclass class StyledPage: """Lines of runs. A run's options: `color` (a gray level, an RGB or a CMYK tuple), `highlight` (an RGB filled behind the text), `strike` (a line through it), `link` - (a /Link annotation over it).""" + (a /Link annotation over it), `bold`, `italic`, `size` (points, default 10) and + `rotate` (90: the text runs upwards; no highlight or strike then). + + `ops` are raw content-stream operators drawn before the text (rectangles, lines, + paths), `pictures` are drawn after it, `image` is the 1 x 1 pixel picture of the + first tests, and `size` is the page size in points.""" lines: list[list[Run]] = field(default_factory=list) image: bool = False + ops: list[str] = field(default_factory=list) + pictures: list[Picture] = field(default_factory=list) + size: tuple[float, float] = (612, 792) CHAR_WIDTH, LINE_HEIGHT, LEFT, TOP = 6, 14, 72, 720 # Courier 10 pt +FONTS = {(False, False): "Courier", (True, False): "Courier-Bold", (False, True): "Courier-Oblique", (True, True): "Courier-BoldOblique"} def _operator(color: "float | tuple[float, ...]", fill: bool) -> str: @@ -91,20 +113,38 @@ def build_styled_pdf( return number page_ids = [allocate() for _ in pages] - font_id, image_id = allocate(), allocate() + font_ids = {key: allocate() for key in FONTS} + font_names = {key: f"F{index}" for index, key in enumerate(FONTS, 1)} + image_ids: dict[tuple[int, int], int] = {} + + def image_object(pixels: int, gray: int) -> int: + if (pixels, gray) not in image_ids: + number = image_ids[(pixels, gray)] = allocate() + data = bytes([gray]) * (pixels * pixels) + objects[number] = ( + f"<< /Type /XObject /Subtype /Image /Width {pixels} /Height {pixels} /ColorSpace /DeviceGray " + f"/BitsPerComponent 8 /Length {len(data)} >>\nstream\n" + ).encode() + data + b"\nendstream" + return image_ids[(pixels, gray)] + for number, page in zip(page_ids, pages): - ops, annots = ["BT /F1 10 Tf ET"], [] + ops, annots, xobjects = list(page.ops), [], {} for row, line in enumerate(page.lines): y, x = TOP - LINE_HEIGHT * row, LEFT for run in line: text, options = (run, {}) if isinstance(run, str) else run - width = CHAR_WIDTH * len(text) + size = options.get("size", 10) + width = 0.6 * size * len(text) + font = font_names[(bool(options.get("bold")), bool(options.get("italic")))] if "highlight" in options: - ops.append(f"{_operator(options['highlight'], True)} {x} {y - 3} {width} 12 re f 0 g") + ops.append(f"{_operator(options['highlight'], True)} {x} {y - 3} {width} {size * 1.2} re f 0 g") if "color" in options: ops.append(_operator(options["color"], True)) escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") - ops.append(f"BT /F1 10 Tf {x} {y} Td ({escaped}) Tj ET") + if options.get("rotate") == 90: + ops.append(f"BT /{font} {size} Tf 0 1 -1 0 {x} {y} Tm ({escaped}) Tj ET") + else: + ops.append(f"BT /{font} {size} Tf {x} {y} Td ({escaped}) Tj ET") if "color" in options: ops.append("0 g") if options.get("strike"): @@ -115,18 +155,23 @@ def build_styled_pdf( "/A << /S /URI /URI (https://example.org/) >> >>" ) x += width + pictures = list(page.pictures) if page.image: - ops.append("q 40 0 0 40 72 400 cm /Im1 Do Q") + pictures.append(Picture(72, 400, 40, 40, pixels=8)) + for index, picture in enumerate(pictures): + xobjects[f"Im{index}"] = image_object(picture.pixels, picture.gray) + ops.append(f"q {picture.width} 0 0 {picture.height} {picture.x} {picture.y} cm /Im{index} Do Q") stream = "\n".join(ops).encode("latin-1") content_id = allocate() objects[content_id] = b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream" - xobject = f"/XObject << /Im1 {image_id} 0 R >>" if page.image else "" + xobject = "/XObject << " + " ".join(f"/{name} {ref} 0 R" for name, ref in xobjects.items()) + " >>" if xobjects else "" + fonts = " ".join(f"/{font_names[key]} {font_ids[key]} 0 R" for key in FONTS) objects[number] = ( - f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents {content_id} 0 R " - f"/Resources << /Font << /F1 {font_id} 0 R >> {xobject} >> /Annots [{' '.join(annots)}] >>" + f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {page.size[0]:g} {page.size[1]:g}] /Contents {content_id} 0 R " + f"/Resources << /Font << {fonts} >> {xobject} >> /Annots [{' '.join(annots)}] >>" ).encode() - objects[font_id] = b"<< /Type /Font /Subtype /Type1 /BaseFont /Courier >>" - objects[image_id] = b"<< /Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 1 >>\nstream\n\x80\nendstream" + for key, number in font_ids.items(): + objects[number] = f"<< /Type /Font /Subtype /Type1 /BaseFont /{FONTS[key]} >>".encode() objects[1] = b"<< /Type /Catalog /Pages 2 0 R >>" objects[2] = f"<< /Type /Pages /Kids [{' '.join(f'{n} 0 R' for n in page_ids)}] /Count {len(pages)} >>".encode() if outline: diff --git a/tools/chemenu/tests/test_raw_derive_binary.py b/tools/chemenu/tests/test_raw_derive_binary.py index 6537484..a472d4b 100644 --- a/tools/chemenu/tests/test_raw_derive_binary.py +++ b/tools/chemenu/tests/test_raw_derive_binary.py @@ -5,6 +5,7 @@ reading them here. The tests that need a package skip where it is not installed installs `tools/requirements.txt`, so they run there.""" from __future__ import annotations +import dataclasses import importlib.util import io import re @@ -15,7 +16,7 @@ import pytest from chemenu import filters from chemenu.commands import doctor from chemenu.filters import pdf_layout -from chemenu.tests.binary_formats import StyledPage, build_msg, build_pdf, build_styled_pdf +from chemenu.tests.binary_formats import Picture, StyledPage, build_msg, build_pdf, build_styled_pdf from chemenu.tests.test_raw_derive import ( # noqa: F401 (tree is a fixture) _derive, _header_and_text, @@ -140,7 +141,7 @@ def _derive_pdf(tree, pdf: bytes, name: str = "doc.pdf"): def _markings_stripped(text: str) -> str: - return re.sub(r'||~~|\*', "", text) + return re.sub(r"~~|\*", "", text) @needs_pypdf @@ -174,52 +175,52 @@ def test_a_pdf_without_outline_or_markings_has_neither_field_and_no_image_check( header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[LONG_LINE]])], title="T")) out = capsys.readouterr().out assert "outline" not in header and "markings" not in header - assert "old rule and new rule end' in text - assert 'second yellow' in text - legend = {entry.split(" | ")[0]: entry for entry in header["markings"]} - assert "places 2 | pages 1-2" in legend['…'] - assert "#ffff00" in legend['…'] - assert "places 1 | pages 1" in legend['…'] + assert "Intro **old rule** and **new rule** end" in text and "**second yellow**" in text + (entry,) = [e for e in header["markings"] if e.startswith("**…**")] + assert "places 3 | pages 1-2" in entry + assert "gelb on page(s) 1-2" in entry and "grün on page(s) 1" in entry @needs_pypdf @needs_pdfminer -def test_a_background_without_a_name_is_marked_with_its_hex_value(tree): - pdf = build_styled_pdf([StyledPage([[("odd", {"highlight": (0.2, 0.4, 0.6)})]])]) - _header, text = _derive_pdf(tree, pdf) - assert 'odd' in text - - -@needs_pypdf -@needs_pdfminer -def test_a_white_fill_is_not_a_highlight(tree): - pdf = build_styled_pdf([StyledPage([[("on white", {"highlight": (1, 1, 1)})]])]) +def test_only_a_bright_saturated_fill_about_as_wide_as_its_text_is_a_highlight(tree): + dark = "0.2 0.3 0.5 rg 72 700 200 14 re f 0 g" # no highlighter is dark + panel = "1 1 0 rg 60 650 480 40 re f 0 g" # a box far bigger than the line it holds + wide = "1 1 0 rg 60 612 480 12 re f 0 g" # as high as a line, but wider than the words + grey = "0.85 g 72 580 200 12 re f 0 g" # shading, not a pen + pdf = build_styled_pdf([StyledPage( + [[("on white", {"highlight": (1, 1, 1)})], ["on dark"], ["in a panel"], ["short"], ["on grey"]], ops=[dark, panel, wide, grey], + )]) header, text = _derive_pdf(tree, pdf) - assert "~~*two words*~~' in text + assert "**~~*two words*~~**" in text @needs_pypdf @@ -259,25 +351,65 @@ def test_a_marking_never_runs_over_a_line_or_a_page_marker(tree): ]) _header, text = _derive_pdf(tree, pdf) for line in text.splitlines(): - assert line.count("") and line.count("~~") % 2 == 0, line - assert 'first line\nsecond line' in text - assert "\n\n\n\n**" in text + + +def _tags_hug_their_text(text: str) -> bool: + """No opening tag is followed, and no closing tag preceded, by whitespace.""" + open_kinds: set[str] = set() + for match in re.finditer(r"\*\*|~~|\*", text): + kind = match.group() + if kind in open_kinds: + open_kinds.discard(kind) + if text[match.start() - 1].isspace(): + return False + else: + open_kinds.add(kind) + if text[match.end():match.end() + 1].isspace(): + return False + return True + + +@needs_pypdf +@needs_pdfminer +def test_no_tag_ever_sits_next_to_whitespace_on_its_inner_side(tree): + pdf = build_styled_pdf([StyledPage([ + ["Plain ", ("yellow ", {"highlight": YELLOW}), "and ", (" padded", {"highlight": GREEN}), " words ", ("gone ", {"strike": True}), "on."], + [("Running text that mixes ", {}), ("bold ", {"bold": True}), (" and ", {"bold": True}), ("more bold", {"bold": True}), " then plain text."], + [(" ", {"strike": True}), ("left", {"strike": True}), (" ", {"strike": True}), ("right", {"strike": True})], + ])]) + _header, text = _derive_pdf(tree, pdf) + assert "**" in text and "~~" in text + assert _tags_hug_their_text(text), text @needs_pypdf @needs_pdfminer def test_removing_the_markings_gives_the_text_the_pdf_has_without_them(tree): styled = build_styled_pdf([StyledPage([ - ["Plain ", ("yellow", {"highlight": YELLOW}), " ", ("red", {"color": RED}), " ", ("struck", {"strike": True}), " end"], + ["Plain ", ("yellow", {"highlight": YELLOW}), " ", ("red", {"color": RED}), " ", ("struck", {"strike": True}), " end of the line, ", ("bold", {"bold": True}), " ", ("slanted", {"italic": True}), "."], [("whole line", {"highlight": GREEN})], ])]) - unstyled = build_styled_pdf([StyledPage([["Plain yellow red struck end"], ["whole line"]])]) + unstyled = build_styled_pdf([StyledPage([["Plain yellow red struck end of the line, bold slanted."], ["whole line"]])]) _header, marked = _derive_pdf(tree, styled, "marked.pdf") _header, plain = _derive_pdf(tree, unstyled, "plain.pdf") assert marked != plain assert _markings_stripped(marked) == plain +@needs_pypdf +@needs_pdfminer +def test_the_legend_is_at_most_three_lines(tree): + pdf = build_styled_pdf([StyledPage([[ + "A line of running text with ", ("highlight", {"highlight": YELLOW}), ", ", ("struck", {"strike": True}), " and an ", + ("italic", {"italic": True}), " word, and ", ("bold", {"bold": True}), ".", + ]])]) + header, _text = _derive_pdf(tree, pdf) + assert [entry.split(" | ")[0] for entry in header["markings"]] == ["**…**", "~~…~~", "*…*"] + + @needs_pypdf @needs_pdfminer def test_pages_with_images_get_one_check_line(tree, capsys): @@ -295,14 +427,14 @@ def test_a_page_whose_text_does_not_match_the_layout_is_left_unmarked_with_a_war def disagree(data): first, second = real(data) - garbled = tuple(pdf_layout.Glyph(c, g.colour, g.background, g.struck, g.linked) for g, c in zip(first.glyphs, "zzzzzzzzzzz")) - return [pdf_layout.PageLayout(garbled, False), second] + garbled = tuple(dataclasses.replace(char, text="z") for char in first.chars) + return [dataclasses.replace(first, chars=garbled), second] monkeypatch.setattr(pdf_layout, "analyse", disagree) header, text = _derive_pdf(tree, pdf) out = capsys.readouterr().out - assert "highlighted" in text and 'highlighted' not in text - assert 'fine' in text + assert "highlighted" in text and "**highlighted" not in text + assert "**fine**" in text assert "WARN" in out and "Page(s) 1" in out and "left out" in out @@ -315,10 +447,11 @@ def test_a_layout_that_cannot_be_read_costs_the_markings_not_the_derivation(monk monkeypatch.setattr(pdf_layout, "analyse", broken) header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[("kept", {"highlight": YELLOW})]])], outline=[("Top", 1, 0)])) out = capsys.readouterr().out - assert "kept" in text and "= 4 else None, False, False) for i, c in enumerate("start ghost middle end")) - found, coverage = pdf_layout._align("start middle end", glyphs, [pdf_layout._flags(g, "#000000") for g in glyphs]) + flag = (True, None, None) + glyphs = [pdf_layout.Glyph(c, i >= 4, False, False) for i, c in enumerate("start ghost middle end")] + found, coverage = pdf_layout._align("start middle end", glyphs, [(True if g.strong else None, None, None) for g in glyphs]) assert coverage == 1.0 assert [found.get(i) for i in (0, 6, 7, 13)] == [(None, None, None), flag, flag, flag]