raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst (Fließtext, Code, Tabellengitter, Vorlagen), ~~ ** * nur wo die Form eindeutig ist, Bildseiten-CHECK nur für Inhaltsbilder (#231); 10.1.0-beta.5
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/raw-material-and-uploads.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/commands/raw/derive.py - tools/chemenu/filters/pdf.py - tools/chemenu/filters/pdf_layout.py - tools/chemenu/tests/binary_formats.py - tools/chemenu/tests/test_raw_derive_binary.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed
1 parent
01360e9045
commit
a581271165
11 files changed
+871
-177
No files matched your search
+3
-2
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10.1.0-beta.4 - 2026-10-14 - instructions/dev: converter-comparison nennt harness/compare/README.md als Ort der Einrichtung
|
## 10.1.0-beta.5 - 2026-10-14 - raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder
|
||||||
|
|
||||||
**Author:** Torben Nehmer
|
**Author:** Torben Nehmer
|
||||||
|
|
||||||
@@ -22,6 +22,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
**Medium impact**
|
**Medium impact**
|
||||||
- raw derive PDF: Gliederung mit Seite und Zeile im Kopf, Hinterlegungen, Durchstreichungen und abweichende Textfarben als Auszeichnung im Text, Bildseiten als CHECK
|
- raw derive PDF: Gliederung mit Seite und Zeile im Kopf, Hinterlegungen, Durchstreichungen und abweichende Textfarben als Auszeichnung im Text, Bildseiten als CHECK
|
||||||
- Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release
|
- Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release
|
||||||
|
- raw derive PDF: Auszeichnungen auf ein Minimum - Struktur zuerst, ~~ ** * nur wo eindeutig, Bildseiten-CHECK nur für Inhaltsbilder
|
||||||
|
|
||||||
**Low impact**
|
**Low impact**
|
||||||
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
||||||
@@ -34,7 +35,7 @@ Der Testkorpus mit echten Dokumenten (PDF, Mail, Office, ZIP) und sein Test lieg
|
|||||||
|
|
||||||
### Große PDFs: Gliederung und Auszeichnungen
|
### Große PDFs: Gliederung und Auszeichnungen
|
||||||
|
|
||||||
Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`; so findet ein Leser einen Abschnitt, ohne das ganze Dokument zu lesen. Im Text stehen drei Auszeichnungen für das, was die Seite nur durch ihre Form sagt: `<mark data-farbe="gelb">…</mark>` für eine Hinterlegung, `~~…~~` für eine Durchstreichung und `*…*` für eine vom Fließtext abweichende Textfarbe ohne Link. Sie geben das Aussehen wieder, nicht die Bedeutung; ohne sie ist der Text derselbe wie zuvor. Das Feld `markings` im Kopf ist die Legende mit Anzahl und Seiten, kein Tag überschreitet eine Zeile oder Seitenmarke, und Bildseiten melden einen CHECK. Das Layout liest `pdfminer.six`, das neu in `requirements.txt` steht; ein PDF ohne Gliederung und Auszeichnungen wird wie bisher abgeleitet, vorhandene `derived.md` bleiben unberührt.
|
Bei einem PDF schreibt `raw derive` die Gliederung als Feld `outline` in den Kopf, je Eintrag mit Ebene, Seite und der Zeile der Seitenmarke in `derived.md`. Im Text stehen drei Auszeichnungen als gewöhnliches Markdown für das, was die Seite nur durch ihre Form sagt: `~~…~~` für Durchstreichung, `**…**` für Textmarker, Fettdruck oder eine deutlich andere Textfarbe, `*…*` für Kursiv. Die Ableitung bestimmt erst die Struktur (Fließtext, Code, Tabellengitter, wiederkehrende Vorlagen) und zeichnet nur aus, wo die Form eindeutig ist; Fett, Kursiv und Farbe zählen nur im Fließtext. Ohne die Auszeichnungen ist der Text derselbe wie zuvor. Das Feld `markings` ist die Legende mit höchstens drei Zeilen, kein Tag überschreitet eine Zeile oder Seitenmarke. Der CHECK nennt nur Seiten mit inhaltlichem Bild oder Diagramm. Das Layout liest `pdfminer.six`, neu in `requirements.txt`; vorhandene `derived.md` bleiben unberührt.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -46,11 +46,12 @@ pick a directory:
|
|||||||
sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is
|
sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is
|
||||||
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
|
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
|
||||||
text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the
|
text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the
|
||||||
header of the text, each entry with its page and the line where that page starts; text over a
|
header of the text, each entry with its page and the line where that page starts; struck-through
|
||||||
coloured background, in another colour than the running text, or struck through is marked in
|
text, highlighted, bold or clearly differently coloured text, and italic text are marked in the
|
||||||
the text (`<mark data-farbe="gelb">`, `*…*`, `~~…~~`) with a legend in the header, and what a
|
text (`~~…~~`, `**…**`, `*…*`) with a legend in the header, and what a colour means is for the
|
||||||
colour means is for the reader to take from the document itself. Pages with pictures are
|
reader to take from the document itself. The tool marks sparingly, in running text and
|
||||||
named in a `CHECK` line, because the pictures are not in the text.
|
not in code or tables. Pages with pictures or drawings that carry content are named in a `CHECK`
|
||||||
|
line, because the pictures are not in the text.
|
||||||
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
||||||
agent derives the same text from that file with `raw fetch --html`. For a website you read often,
|
agent derives the same text from that file with `raw fetch --html`. For a website you read often,
|
||||||
sign in once instead and let the wiki fetch it with your session:
|
sign in once instead and let the wiki fetch it with your session:
|
||||||
|
|||||||
@@ -193,7 +193,7 @@ tools/wikitool raw derive incoming/scans.zip
|
|||||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
|
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
|
||||||
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only - `<mark data-farbe="<name>">…</mark>` for text over a coloured background, `*…*` for text in a colour other than the running text that is not under a link, `~~…~~` for strike-through. A `markings` header field is the legend (form, colour, places, pages); removing the markings gives the text the PDF has without them. A CHECK names the pages with images, which the text does not contain. A PDF without outline, markings and images is derived as before.
|
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before.
|
||||||
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
|
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
|
||||||
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
||||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||||
|
|||||||
+19
-9
@@ -394,15 +394,25 @@ derivation, accepted with it, and an existing one is never overwritten.
|
|||||||
**A PDF's outline and markings are in the derivation.** The header of `derived.md` lists the
|
**A PDF's outline and markings are in the derivation.** The header of `derived.md` lists the
|
||||||
PDF's outline as `outline` entries - title, level, page and the line of that page's
|
PDF's outline as `outline` entries - title, level, page and the line of that page's
|
||||||
`<!-- page N -->` marker in the file - so a reader finds a section without reading the whole
|
`<!-- page N -->` marker in the file - so a reader finds a section without reading the whole
|
||||||
text. In the text, three markings carry what the page shows by form alone: a background fill is
|
text. In the text, three markings carry what the page shows by form alone, as plain Markdown:
|
||||||
`<mark data-farbe="…">…</mark>`, a strike-through is `~~…~~`, and text in a colour other than the
|
`~~…~~` is a strike-through, `**…**` is a highlighter stroke, bold type, or a colour clearly
|
||||||
running text, without a link under it, is `*…*`. A marking records how the page looks, not what
|
unlike the running text (not under a link), and `*…*` is italic. The tool marks as little as it
|
||||||
it means: nothing in the tool decides that yellow is a deadline, and removing the markings gives
|
can stand behind: it reads the structure first - running text, program text, ruled tables, and
|
||||||
the same text as without them. The `markings` header field is the legend - each marking in use,
|
shapes the document repeats on its pages - and marks only where the form is unambiguous. Bold,
|
||||||
with its count and pages. No tag crosses a line or a page marker. A page whose text cannot be
|
italic and colour count inside running text only; highlighter and strike-through also in the
|
||||||
matched to its layout gets no markings and a WARN; a PDF with images the text does not hold gets
|
regulations that use them, but never in program text, a table's grid or a recurring template. A
|
||||||
a CHECK naming the pages, because the tool has no OCR. A PDF without outline or markings is
|
line that is bold or italic from end to end is a heading or a quotation, not a marking. A marking
|
||||||
derived as before. Reading the layout needs `pdfminer.six` beside `pypdf`.
|
records how the page looks, not what it means: nothing in the tool decides that yellow is a
|
||||||
|
deadline, and removing the markings gives the same text as without them. The `markings` header
|
||||||
|
field is the legend, at most three lines, one per kind in use, with its count and pages; the
|
||||||
|
`**` line also names the highlighter colours with their pages. A tag never starts or ends on
|
||||||
|
whitespace and never crosses a line or a page marker. A page whose text cannot be matched to its
|
||||||
|
layout gets no markings and a WARN. A CHECK names the pages whose pictures the text does not
|
||||||
|
hold, because the tool has no OCR: a picture counts unless it is a background covering most of
|
||||||
|
the page, a filler a few pixels or points in size, or the same picture on a fifth of the pages
|
||||||
|
(a logo, a frame); a vector drawing with many paths and little text counts too. A PDF without
|
||||||
|
outline or markings is derived as before. Reading the layout needs `pdfminer.six` beside
|
||||||
|
`pypdf`.
|
||||||
|
|
||||||
**The original stays.** The derivation is the tool's reading of it, never the source: with the
|
**The original stays.** The derivation is the tool's reading of it, never the source: with the
|
||||||
original kept, a claim in `kb/` can still be checked against what the document said when a
|
original kept, a claim in `kb/` can still be checked against what the document said when a
|
||||||
|
|||||||
+1
-1
@@ -1608,7 +1608,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
|
|||||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
|
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
|
||||||
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only - `<mark data-farbe="<name>">…</mark>` for text over a coloured background, `*…*` for text in a colour other than the running text that is not under a link, `~~…~~` for strike-through. A `markings` header field is the legend (form, colour, places, pages); removing the markings gives the text the PDF has without them. A CHECK names the pages with images, which the text does not contain. A PDF without outline, markings and images is derived as before.
|
- A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document. A PDF without outline, markings and images is derived as before.
|
||||||
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
|
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
|
||||||
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
||||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||||
|
|||||||
@@ -371,13 +371,16 @@ def _derive_next(target: Path) -> str:
|
|||||||
"`multipart/alternative` is not an attachment.",
|
"`multipart/alternative` is not an attachment.",
|
||||||
"A PDF also gets, where the document has them: an `outline` header field (one entry per "
|
"A PDF also gets, where the document has them: an `outline` header field (one entry per "
|
||||||
"outline item: title, level, page and the line of `derived.md` where that page's "
|
"outline item: title, level, page and the line of `derived.md` where that page's "
|
||||||
"marker is), and markings in the text that follow the visible form only - `<mark "
|
"marker is), and markings in the text that follow the visible form only, in plain "
|
||||||
"data-farbe=\"<name>\">…</mark>` for text over a coloured background, `*…*` for text in a "
|
"Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a "
|
||||||
"colour other than the running text that is not under a link, `~~…~~` for strike-through. "
|
"colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, "
|
||||||
"A `markings` header field is the legend (form, colour, places, pages); removing the "
|
"italic and colour count in running text only, never in program text, a table's grid or "
|
||||||
"markings gives the text the PDF has without them. A CHECK names the pages with images, "
|
"a shape the document repeats on its pages. A `markings` header field is the legend (at "
|
||||||
"which the text does not contain. A PDF without outline, markings and images is derived "
|
"most three lines: form, places, pages; the `**` line names the highlighter colours); "
|
||||||
"as before.",
|
"removing the markings gives the text the PDF has without them, and a tag never starts "
|
||||||
|
"or ends on whitespace. A CHECK names the pages with a picture or drawing the text does "
|
||||||
|
"not contain - not a background, a filler or a picture repeated across the document. A "
|
||||||
|
"PDF without outline, markings and images is derived as before.",
|
||||||
"A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder "
|
"A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder "
|
||||||
"`<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` "
|
"`<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` "
|
||||||
"per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet "
|
"per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet "
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
"""PDF: the text and the outline are read with `pypdf`, the look of the characters (highlight,
|
"""PDF: the text and the outline are read with `pypdf`, the look of the page (strike-through,
|
||||||
colour, strike-through, images) with `pdfminer.six` in `pdf_layout`. Both are imported when a PDF is
|
highlight, bold, italic, pictures) with `pdfminer.six` in `pdf_layout`. Both are imported when a PDF is
|
||||||
derived, never at start-up."""
|
derived, never at start-up."""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -67,7 +67,7 @@ class PdfFilter(Filter):
|
|||||||
warnings.extend(annotation.warnings)
|
warnings.extend(annotation.warnings)
|
||||||
if annotation.markings:
|
if annotation.markings:
|
||||||
fields.append(("markings", list(annotation.markings)))
|
fields.append(("markings", list(annotation.markings)))
|
||||||
image_pages = [number for number, layout in enumerate(layouts, 1) if layout.images]
|
image_pages = list(annotation.image_pages)
|
||||||
if image_pages:
|
if image_pages:
|
||||||
checks.append(
|
checks.append(
|
||||||
f"Images on page(s) {pdf_layout.page_ranges(image_pages)} are not contained in the text - "
|
f"Images on page(s) {pdf_layout.page_ranges(image_pages)} are not contained in the text - "
|
||||||
|
|||||||
@@ -1,26 +1,78 @@
|
|||||||
"""What the PDF filter reads from the page layout: per character its text colour, the colour
|
"""What the PDF filter reads from the page layout, and the markings it lays onto the text `pypdf`
|
||||||
filled behind it, a line through it and a link over it - and the mapping of those onto the text
|
extracted. Read with `pdfminer.six` (imported when `analyse` runs, never at start-up).
|
||||||
`pypdf` extracted. Read with `pdfminer.six` (imported when `analyse` runs, never at start-up).
|
|
||||||
|
|
||||||
`pypdf` stays the source of the text, so a PDF's wording does not change; `pdfminer.six` only
|
`pypdf` stays the source of the text, so a PDF's wording does not change; `pdfminer.six` only
|
||||||
says what each character looks like. The two are matched page by page over their non-space
|
says what each character looks like. The two are matched page by page over their non-space
|
||||||
characters, and the markings are laid onto the `pypdf` text. The markings follow the visible
|
characters, and the markings are laid onto the `pypdf` text. The filter has no model, so it marks
|
||||||
form and nothing else: the filter has no model, and what a colour means is written in the
|
as little as it can stand behind: structure first (running text, code, tables, shapes that recur
|
||||||
document itself."""
|
on many pages), then three markings - `~~` struck, `**` highlighted, bold or in a clearly
|
||||||
|
different colour, `*` italic - and only where the form is unambiguous. A colour is mostly
|
||||||
|
decoration, so it is a marking only inside running text. Every threshold is a constant below."""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import difflib
|
import difflib
|
||||||
|
from bisect import bisect_left, bisect_right
|
||||||
|
import hashlib
|
||||||
import io
|
import io
|
||||||
import itertools
|
import itertools
|
||||||
|
import re
|
||||||
import unicodedata
|
import unicodedata
|
||||||
|
from collections import Counter
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Iterable, Optional
|
from typing import Iterable, Optional
|
||||||
|
|
||||||
MIN_COVERAGE = 0.9 # a page whose text matches the layout less than this gets no markings
|
MIN_COVERAGE = 0.9 # a page whose text matches the layout less than this gets no markings
|
||||||
MAX_HIGHLIGHT_LINES = 2.5 # a fill taller than this many text heights is a panel, not a highlight
|
|
||||||
STRIKE_BAND = (0.3, 0.75) # where in the character's height a line counts as going through it
|
# Structure: which characters count as running text.
|
||||||
|
FLOW_SIZE = (0.85, 1.20) # a line's type size, as a multiple of the body size
|
||||||
|
FLOW_MIN_CHARS = 15 # a line shorter than this (in non-space characters) is a label, not running text
|
||||||
|
FLOW_MIN_PLAIN = 0.60 # share of a line's characters that are plain running type
|
||||||
|
SEGMENT_GAP = 2.5 # a gap wider than this many type heights splits a line (columns, table cells)
|
||||||
|
LINE_TOLERANCE = 0.5 # characters whose baselines differ by less than this many type heights share a line
|
||||||
|
CODE_MIN_SIGNALS = 3 # a block of lines is code when this many of the five signals hold
|
||||||
|
CODE_MIN_COLOURS = 5
|
||||||
|
CODE_OPERATOR_SHARE = 0.15
|
||||||
|
CODE_END_SHARE = 0.30 # share of lines ending in ; { } ) , :
|
||||||
|
CODE_MONO_SHARE = 0.5
|
||||||
|
CODE_BLOCK_GAP = 1.8 # lines closer than this many type heights belong to one block
|
||||||
|
MONO_NAMES = re.compile(r"courier|mono|consolas|menlo|typewriter|lucida console|andale|inconsolata|fixed", re.I)
|
||||||
|
OPERATORS = set("{}()[]<>=+-*/%&|!^~;:\"'\\#@$")
|
||||||
|
TEMPLATE_SHARE = 0.20 # a shape on this share of the pages (and on at least TEMPLATE_PAGES) is a template
|
||||||
|
TEMPLATE_PAGES = 3
|
||||||
|
GRID_TOLERANCE = 3.0 # points: how close a horizontal and a vertical line must come to cross
|
||||||
|
GRID_MIN_LINES = 2 # a table region needs this many lines in each direction
|
||||||
|
|
||||||
|
# Rule 2: strike-through.
|
||||||
|
STRIKE_BAND = (0.30, 0.75) # where in the character's height a line counts as going through it
|
||||||
|
STRIKE_MAX_WIDTH = 1.5 # points
|
||||||
|
STRIKE_MIN_CHARS = 2
|
||||||
|
STRIKE_OVERHANG = 0.5 # type heights the line may reach past the first/last struck character
|
||||||
|
STRIKE_EDGE = 0.05 # share of the page that is edge
|
||||||
FLAT = 1.5 # a filled shape this thin (points) is a line
|
FLAT = 1.5 # a filled shape this thin (points) is a line
|
||||||
|
|
||||||
|
# Rule 3: highlighter.
|
||||||
|
HIGHLIGHT_MAX_HEIGHT = 1.8 # type heights
|
||||||
|
HIGHLIGHT_MAX_EXTRA = 2.0 # type heights wider than the text it covers
|
||||||
|
HIGHLIGHT_MAX_AREA = 0.08 # share of the page
|
||||||
|
HIGHLIGHT_BRIGHT = 0xC0 # the lightest channel at least this
|
||||||
|
HIGHLIGHT_SPAN = 0x60 # the spread between the lightest and darkest channel at least this
|
||||||
|
|
||||||
|
# Rule 4: bold, italic and a clearly different colour.
|
||||||
|
BOLD_NAMES = re.compile(r"bold|black|heavy|semibold|demi", re.I)
|
||||||
|
ITALIC_NAMES = re.compile(r"italic|oblique", re.I)
|
||||||
|
COLOUR_DISTANCE = 0x30 # per channel, from the running text colour
|
||||||
|
COLOUR_MAX_SHARE = 0.40 # share of a line that may be coloured
|
||||||
|
|
||||||
|
# Rule 6: pictures.
|
||||||
|
IMAGE_MAX_PAGE_SHARE = 0.80 # a picture covering more of the page is a background
|
||||||
|
IMAGE_MIN_PIXELS = 4 # a source this size or smaller is a filler
|
||||||
|
IMAGE_MIN_POINTS = 10 # drawn smaller than this in either direction is a filler
|
||||||
|
VECTOR_MIN_PATHS = 20
|
||||||
|
VECTOR_MIN_PAGE_SHARE = 0.02
|
||||||
|
VECTOR_MAX_TEXT_PER_PATH = 2.0 # characters inside the cluster per path: more is a table, not a drawing
|
||||||
|
VECTOR_JOIN = 4.0 # points: paths this close belong to one drawing
|
||||||
|
MAX_PATHS = 20000 # a page with more paths than this is not scanned for drawings
|
||||||
|
|
||||||
NAMES = {
|
NAMES = {
|
||||||
"#ffff00": "gelb", "#00ff00": "grün", "#ff0000": "rot", "#0000ff": "blau",
|
"#ffff00": "gelb", "#00ff00": "grün", "#ff0000": "rot", "#0000ff": "blau",
|
||||||
"#00ffff": "türkis", "#ff00ff": "pink", "#ffa500": "orange", "#ffc000": "orange",
|
"#00ffff": "türkis", "#ff00ff": "pink", "#ffa500": "orange", "#ffc000": "orange",
|
||||||
@@ -29,23 +81,70 @@ NAMES = {
|
|||||||
"#800080": "violett", "#800000": "dunkelrot", "#808000": "olivgelb",
|
"#800080": "violett", "#800000": "dunkelrot", "#808000": "olivgelb",
|
||||||
}
|
}
|
||||||
|
|
||||||
Flags = tuple # (background hex | None, True | None for struck, colour hex | None)
|
Box = tuple # (x0, y0, x1, y1)
|
||||||
|
Flags = tuple # (strong, struck, italic): True or None per layer
|
||||||
LAYERS = 3
|
LAYERS = 3
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class Glyph:
|
class Char:
|
||||||
|
"""One non-space character as the page draws it."""
|
||||||
|
|
||||||
text: str
|
text: str
|
||||||
|
box: Box
|
||||||
|
size: float # type height: the em of the font
|
||||||
|
font: str
|
||||||
colour: Optional[str]
|
colour: Optional[str]
|
||||||
background: Optional[str]
|
direction: Optional[int] # direction of the baseline in degrees (0, 90, 180, 270); None for any other angle
|
||||||
struck: bool
|
order: int # position in the drawing order
|
||||||
linked: bool
|
linked: bool
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Fill:
|
||||||
|
order: int
|
||||||
|
box: Box
|
||||||
|
colour: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Line:
|
||||||
|
"""A thin line: `start` to `end` along its direction, `pos` across it, `width` its thickness."""
|
||||||
|
|
||||||
|
start: float
|
||||||
|
end: float
|
||||||
|
pos: float
|
||||||
|
width: float
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ImageUse:
|
||||||
|
box: Box
|
||||||
|
pixels: tuple[int, int]
|
||||||
|
key: str # a digest of the image's content: the same picture drawn twice has the same key
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class PageLayout:
|
class PageLayout:
|
||||||
glyphs: tuple[Glyph, ...]
|
width: float
|
||||||
images: bool
|
height: float
|
||||||
|
chars: tuple[Char, ...]
|
||||||
|
fills: tuple[Fill, ...]
|
||||||
|
hlines: tuple[Line, ...]
|
||||||
|
vlines: tuple[Line, ...]
|
||||||
|
images: tuple[ImageUse, ...]
|
||||||
|
paths: tuple[Box, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Glyph:
|
||||||
|
"""What `annotate` needs per character: its text and the markings that survived the rules."""
|
||||||
|
|
||||||
|
text: str
|
||||||
|
strong: bool
|
||||||
|
struck: bool
|
||||||
|
italic: bool
|
||||||
|
highlight: Optional[str] = None # the highlighter colour behind a strong glyph, for the legend
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -53,6 +152,7 @@ class Annotation:
|
|||||||
texts: tuple[str, ...]
|
texts: tuple[str, ...]
|
||||||
markings: tuple[str, ...] # the legend
|
markings: tuple[str, ...] # the legend
|
||||||
warnings: tuple[str, ...]
|
warnings: tuple[str, ...]
|
||||||
|
image_pages: tuple[int, ...]
|
||||||
|
|
||||||
|
|
||||||
def colour_name(value: str) -> str:
|
def colour_name(value: str) -> str:
|
||||||
@@ -87,15 +187,36 @@ def _hex(value) -> Optional[str]:
|
|||||||
return "#%02x%02x%02x" % (round(red * 255), round(green * 255), round(blue * 255))
|
return "#%02x%02x%02x" % (round(red * 255), round(green * 255), round(blue * 255))
|
||||||
|
|
||||||
|
|
||||||
|
def _channels(value: str) -> tuple[int, int, int]:
|
||||||
|
return int(value[1:3], 16), int(value[3:5], 16), int(value[5:7], 16)
|
||||||
|
|
||||||
|
|
||||||
def _is_white(value: str) -> bool:
|
def _is_white(value: str) -> bool:
|
||||||
return all(int(value[i:i + 2], 16) >= 0xF8 for i in (1, 3, 5))
|
return all(channel >= 0xF8 for channel in _channels(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _direction(matrix) -> Optional[int]:
|
||||||
|
a, b = matrix[0], matrix[1]
|
||||||
|
if abs(b) <= 0.01 * abs(a):
|
||||||
|
return 0 if a > 0 else 180
|
||||||
|
if abs(a) <= 0.01 * abs(b):
|
||||||
|
return 90 if b > 0 else 270
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _image_key(item) -> str:
|
||||||
|
try:
|
||||||
|
raw = item.stream.get_rawdata() or b""
|
||||||
|
except Exception: # a stream pdfminer cannot hand over is told apart by its object id alone
|
||||||
|
raw = b""
|
||||||
|
return hashlib.blake2b(raw[:65536] + str(len(raw)).encode() + str(item.srcsize).encode(), digest_size=8).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def analyse(data: bytes) -> list[PageLayout]:
|
def analyse(data: bytes) -> list[PageLayout]:
|
||||||
"""The glyphs and images of each page, in page order. Raises whatever `pdfminer.six`
|
"""The characters, fills, lines, pictures and paths of each page, in page order. Raises
|
||||||
raises on a document it cannot read."""
|
whatever `pdfminer.six` raises on a document it cannot read."""
|
||||||
from pdfminer.converter import PDFPageAggregator
|
from pdfminer.converter import PDFPageAggregator
|
||||||
from pdfminer.layout import LTChar, LTCurve, LTFigure, LTImage, LTLine
|
from pdfminer.layout import LTChar, LTCurve, LTFigure, LTImage, LTLine, LTRect
|
||||||
from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
|
from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
|
||||||
from pdfminer.pdfpage import PDFPage
|
from pdfminer.pdfpage import PDFPage
|
||||||
from pdfminer.pdftypes import resolve1
|
from pdfminer.pdftypes import resolve1
|
||||||
@@ -107,7 +228,7 @@ def analyse(data: bytes) -> list[PageLayout]:
|
|||||||
else:
|
else:
|
||||||
yield item
|
yield item
|
||||||
|
|
||||||
def link_boxes(page) -> Optional[list[tuple[float, float, float, float]]]:
|
def link_boxes(page) -> Optional[list[Box]]:
|
||||||
"""Link rectangles in the coordinates of the layout; None on a rotated page, where
|
"""Link rectangles in the coordinates of the layout; None on a rotated page, where
|
||||||
they cannot be told apart from the layout's."""
|
they cannot be told apart from the layout's."""
|
||||||
if page.rotate % 360:
|
if page.rotate % 360:
|
||||||
@@ -134,61 +255,441 @@ def analyse(data: bytes) -> list[PageLayout]:
|
|||||||
for page in PDFPage.get_pages(io.BytesIO(data)):
|
for page in PDFPage.get_pages(io.BytesIO(data)):
|
||||||
interpreter.process_page(page)
|
interpreter.process_page(page)
|
||||||
links = link_boxes(page)
|
links = link_boxes(page)
|
||||||
chars, highlights, lines, images = [], [], [], False
|
width, height = page.mediabox[2] - page.mediabox[0], page.mediabox[3] - page.mediabox[1]
|
||||||
|
if page.rotate % 180:
|
||||||
|
width, height = height, width
|
||||||
|
chars, fills, hlines, vlines, images, paths = [], [], [], [], [], []
|
||||||
for order, item in enumerate(flatten(device.get_result())):
|
for order, item in enumerate(flatten(device.get_result())):
|
||||||
if isinstance(item, LTChar):
|
if isinstance(item, LTChar):
|
||||||
chars.append((order, item))
|
if item.get_text().isspace():
|
||||||
|
continue
|
||||||
|
direction = _direction(item.matrix)
|
||||||
|
x0, y0, x1, y1 = item.bbox
|
||||||
|
size = (y1 - y0) if direction in (0, 180) else (x1 - x0) if direction in (90, 270) else item.size
|
||||||
|
centre_x, centre_y = (x0 + x1) / 2, (y0 + y1) / 2
|
||||||
|
linked = links is None or any(
|
||||||
|
bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1 for bx0, by0, bx1, by1 in links
|
||||||
|
)
|
||||||
|
chars.append(Char(item.get_text(), item.bbox, size, item.fontname, _hex(item.graphicstate.ncolor), direction, order, linked))
|
||||||
elif isinstance(item, LTImage):
|
elif isinstance(item, LTImage):
|
||||||
images = True
|
images.append(ImageUse(item.bbox, tuple(item.srcsize or (0, 0)), _image_key(item)))
|
||||||
elif isinstance(item, LTCurve):
|
elif isinstance(item, LTCurve):
|
||||||
x0, y0, x1, y1 = item.bbox
|
x0, y0, x1, y1 = item.bbox
|
||||||
fill = _hex(item.non_stroking_color) if item.fill else None
|
fill = _hex(item.non_stroking_color) if item.fill else None
|
||||||
|
stroke_width = max(item.linewidth or 0, 0)
|
||||||
|
if len(paths) < MAX_PATHS:
|
||||||
|
paths.append(item.bbox)
|
||||||
if y1 - y0 <= FLAT and x1 - x0 >= 1 and (fill is not None or item.stroke or isinstance(item, LTLine)):
|
if y1 - y0 <= FLAT and x1 - x0 >= 1 and (fill is not None or item.stroke or isinstance(item, LTLine)):
|
||||||
lines.append((x0, x1, (y0 + y1) / 2))
|
hlines.append(Line(x0, x1, (y0 + y1) / 2, max(y1 - y0, stroke_width)))
|
||||||
|
elif x1 - x0 <= FLAT and y1 - y0 >= 1 and (fill is not None or item.stroke or isinstance(item, LTLine)):
|
||||||
|
vlines.append(Line(y0, y1, (x0 + x1) / 2, max(x1 - x0, stroke_width)))
|
||||||
elif fill is not None and not _is_white(fill):
|
elif fill is not None and not _is_white(fill):
|
||||||
highlights.append((order, item.bbox, fill))
|
fills.append(Fill(order, item.bbox, fill))
|
||||||
glyphs = []
|
elif isinstance(item, LTRect) and item.stroke and not item.fill and stroke_width <= FLAT:
|
||||||
for order, char in chars:
|
hlines.extend((Line(x0, x1, y0, stroke_width), Line(x0, x1, y1, stroke_width)))
|
||||||
if char.get_text().isspace():
|
vlines.extend((Line(y0, y1, x0, stroke_width), Line(y0, y1, x1, stroke_width)))
|
||||||
continue
|
layouts.append(PageLayout(
|
||||||
x0, y0, x1, y1 = char.bbox
|
width, height, tuple(chars), tuple(fills), tuple(hlines), tuple(vlines), tuple(images), tuple(paths),
|
||||||
centre_x, centre_y, height = (x0 + x1) / 2, (y0 + y1) / 2, y1 - y0
|
|
||||||
background = next((
|
|
||||||
colour for placed, (bx0, by0, bx1, by1), colour in reversed(highlights)
|
|
||||||
if placed < order and bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1
|
|
||||||
and by1 - by0 <= MAX_HIGHLIGHT_LINES * height
|
|
||||||
), None)
|
|
||||||
struck = any(
|
|
||||||
lx0 <= centre_x <= lx1 and y0 + STRIKE_BAND[0] * height <= ly <= y0 + STRIKE_BAND[1] * height
|
|
||||||
for lx0, lx1, ly in lines
|
|
||||||
)
|
|
||||||
linked = links is not None and any(
|
|
||||||
bx0 <= centre_x <= bx1 and by0 <= centre_y <= by1 for bx0, by0, bx1, by1 in links
|
|
||||||
)
|
|
||||||
glyphs.append(Glyph(
|
|
||||||
char.get_text(), _hex(char.graphicstate.ncolor), background, struck,
|
|
||||||
linked or links is None,
|
|
||||||
))
|
))
|
||||||
layouts.append(PageLayout(tuple(glyphs), images))
|
|
||||||
return layouts
|
return layouts
|
||||||
|
|
||||||
|
|
||||||
def body_colour(layouts: Iterable[PageLayout]) -> Optional[str]:
|
# --- templates: what the document repeats on its pages -------------------------------------------
|
||||||
"""The fill colour with the most characters: the colour of the running text."""
|
|
||||||
counts: dict[str, int] = {}
|
def _snap(*values: float) -> tuple:
|
||||||
|
return tuple(round(value / 2) * 2 for value in values)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Templates:
|
||||||
|
"""Shapes and pictures that recur on many pages: a letterhead, a frame, a logo. They are part
|
||||||
|
of the page, not of its content, so no rule reads them."""
|
||||||
|
|
||||||
|
lines: frozenset
|
||||||
|
fills: frozenset
|
||||||
|
paths: frozenset
|
||||||
|
images: frozenset
|
||||||
|
|
||||||
|
|
||||||
|
def _recurring(per_page: list[set]) -> frozenset:
|
||||||
|
if not per_page:
|
||||||
|
return frozenset()
|
||||||
|
counts = Counter(key for keys in per_page for key in keys)
|
||||||
|
needed = max(TEMPLATE_PAGES, TEMPLATE_SHARE * len(per_page))
|
||||||
|
return frozenset(key for key, pages in counts.items() if pages >= needed)
|
||||||
|
|
||||||
|
|
||||||
|
def _line_key(kind: str, line: Line) -> tuple:
|
||||||
|
return (kind,) + _snap(line.start, line.end, line.pos)
|
||||||
|
|
||||||
|
|
||||||
|
def templates(layouts: list[PageLayout]) -> Templates:
|
||||||
|
return Templates(
|
||||||
|
_recurring([
|
||||||
|
{_line_key("h", line) for line in layout.hlines} | {_line_key("v", line) for line in layout.vlines}
|
||||||
|
for layout in layouts
|
||||||
|
]),
|
||||||
|
_recurring([{_snap(*fill.box) + (fill.colour,) for fill in layout.fills} for layout in layouts]),
|
||||||
|
_recurring([{_snap(*box) for box in layout.paths} for layout in layouts]),
|
||||||
|
_recurring([{image.key for image in layout.images} for layout in layouts]),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- structure: table regions, line pieces, code ----------------------------------------------------
|
||||||
|
|
||||||
|
def _inside(box: Box, x: float, y: float, margin: float = 0.0) -> bool:
|
||||||
|
return box[0] - margin <= x <= box[2] + margin and box[1] - margin <= y <= box[3] + margin
|
||||||
|
|
||||||
|
|
||||||
|
def _centre(box: Box) -> tuple[float, float]:
|
||||||
|
return (box[0] + box[2]) / 2, (box[1] + box[3]) / 2
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Grid:
|
||||||
|
regions: tuple[Box, ...] # the extent of each table grid
|
||||||
|
h_used: frozenset # indices of the horizontal lines that belong to a grid
|
||||||
|
v_used: frozenset
|
||||||
|
|
||||||
|
|
||||||
|
def _grids(layout: PageLayout, template: Templates) -> Grid:
|
||||||
|
"""Connected sets of crossing horizontal and vertical lines with at least GRID_MIN_LINES
|
||||||
|
of each: the ruling of a table."""
|
||||||
|
hs = [i for i, line in enumerate(layout.hlines) if _line_key("h", line) not in template.lines]
|
||||||
|
vs = [i for i, line in enumerate(layout.vlines) if _line_key("v", line) not in template.lines]
|
||||||
|
if len(hs) < GRID_MIN_LINES or len(vs) < GRID_MIN_LINES:
|
||||||
|
return Grid((), frozenset(), frozenset())
|
||||||
|
parent = {("h", i): ("h", i) for i in hs} | {("v", j): ("v", j) for j in vs}
|
||||||
|
|
||||||
|
def find(node):
|
||||||
|
while parent[node] != node:
|
||||||
|
parent[node] = parent[parent[node]]
|
||||||
|
node = parent[node]
|
||||||
|
return node
|
||||||
|
|
||||||
|
if len(hs) * len(vs) > 1_000_000: # too many lines to cross: one ruled area
|
||||||
|
for node in parent:
|
||||||
|
parent[find(node)] = find(("h", hs[0]))
|
||||||
|
else:
|
||||||
|
for i in hs:
|
||||||
|
h = layout.hlines[i]
|
||||||
|
for j in vs:
|
||||||
|
v = layout.vlines[j]
|
||||||
|
if h.start - GRID_TOLERANCE <= v.pos <= h.end + GRID_TOLERANCE and v.start - GRID_TOLERANCE <= h.pos <= v.end + GRID_TOLERANCE:
|
||||||
|
parent[find(("h", i))] = find(("v", j))
|
||||||
|
members: dict[tuple, list] = {}
|
||||||
|
for node in parent:
|
||||||
|
members.setdefault(find(node), []).append(node)
|
||||||
|
regions, h_used, v_used = [], set(), set()
|
||||||
|
for nodes in members.values():
|
||||||
|
h_nodes = [i for kind, i in nodes if kind == "h"]
|
||||||
|
v_nodes = [j for kind, j in nodes if kind == "v"]
|
||||||
|
if len(h_nodes) < GRID_MIN_LINES or len(v_nodes) < GRID_MIN_LINES:
|
||||||
|
continue
|
||||||
|
boxes = [(layout.hlines[i].start, layout.hlines[i].pos, layout.hlines[i].end, layout.hlines[i].pos) for i in h_nodes]
|
||||||
|
boxes += [(layout.vlines[j].pos, layout.vlines[j].start, layout.vlines[j].pos, layout.vlines[j].end) for j in v_nodes]
|
||||||
|
regions.append((min(b[0] for b in boxes), min(b[1] for b in boxes), max(b[2] for b in boxes), max(b[3] for b in boxes)))
|
||||||
|
h_used.update(h_nodes)
|
||||||
|
v_used.update(v_nodes)
|
||||||
|
return Grid(tuple(regions), frozenset(h_used), frozenset(v_used))
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Piece:
|
||||||
|
"""A run of upright characters on one line with no wide gap: a line, or a cell of a line."""
|
||||||
|
|
||||||
|
members: tuple[int, ...] # indices into PageLayout.chars, left to right
|
||||||
|
row: int
|
||||||
|
|
||||||
|
|
||||||
|
def _pieces(layout: PageLayout) -> list[list[Piece]]:
|
||||||
|
"""The upright characters as rows (top to bottom) of pieces (left to right)."""
|
||||||
|
upright = sorted((i for i, char in enumerate(layout.chars) if char.direction == 0), key=lambda i: -layout.chars[i].box[1])
|
||||||
|
rows: list[list[int]] = []
|
||||||
|
base = 0.0
|
||||||
|
for i in upright:
|
||||||
|
char = layout.chars[i]
|
||||||
|
if not rows or abs(char.box[1] - base) > LINE_TOLERANCE * char.size:
|
||||||
|
rows.append([])
|
||||||
|
base = char.box[1]
|
||||||
|
rows[-1].append(i)
|
||||||
|
out = []
|
||||||
|
for number, row in enumerate(rows):
|
||||||
|
row.sort(key=lambda i: layout.chars[i].box[0])
|
||||||
|
pieces, current = [], [row[0]]
|
||||||
|
for previous, i in zip(row, row[1:]):
|
||||||
|
gap = layout.chars[i].box[0] - layout.chars[previous].box[2]
|
||||||
|
if gap > SEGMENT_GAP * layout.chars[i].size:
|
||||||
|
pieces.append(Piece(tuple(current), number))
|
||||||
|
current = []
|
||||||
|
current.append(i)
|
||||||
|
pieces.append(Piece(tuple(current), number))
|
||||||
|
out.append(pieces)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _is_bold(char: Char) -> bool:
|
||||||
|
return bool(BOLD_NAMES.search(char.font))
|
||||||
|
|
||||||
|
|
||||||
|
def _is_italic(char: Char) -> bool:
|
||||||
|
return bool(ITALIC_NAMES.search(char.font))
|
||||||
|
|
||||||
|
|
||||||
|
def _code_chars(layout: PageLayout, rows: list[list[Piece]]) -> set[int]:
|
||||||
|
"""The characters of blocks of lines that read as program text: at least CODE_MIN_SIGNALS of
|
||||||
|
a monospaced font, many operators, syntax colours, varying indentation and statement endings."""
|
||||||
|
found: set[int] = set()
|
||||||
|
|
||||||
|
def judge(block: list[list[Piece]]) -> None:
|
||||||
|
if len(block) < 3:
|
||||||
|
return
|
||||||
|
chars = [layout.chars[i] for pieces in block for piece in pieces for i in piece.members]
|
||||||
|
lefts = {round(layout.chars[pieces[0].members[0]].box[0] / (0.5 * chars[0].size)) for pieces in block}
|
||||||
|
ends = [layout.chars[pieces[-1].members[-1]].text for pieces in block]
|
||||||
|
signals = sum([
|
||||||
|
sum(bool(MONO_NAMES.search(char.font)) for char in chars) / len(chars) >= CODE_MONO_SHARE,
|
||||||
|
sum(char.text in OPERATORS for char in chars) / len(chars) >= CODE_OPERATOR_SHARE,
|
||||||
|
len({char.colour for char in chars if char.colour}) >= CODE_MIN_COLOURS,
|
||||||
|
len(lefts) >= 2,
|
||||||
|
sum(end in ";{}),:" for end in ends) / len(ends) >= CODE_END_SHARE,
|
||||||
|
])
|
||||||
|
if signals >= CODE_MIN_SIGNALS:
|
||||||
|
found.update(i for pieces in block for piece in pieces for i in piece.members)
|
||||||
|
|
||||||
|
block: list[list[Piece]] = []
|
||||||
|
previous = None
|
||||||
|
for pieces in rows:
|
||||||
|
char = layout.chars[pieces[0].members[0]]
|
||||||
|
if previous is not None and previous - char.box[1] > CODE_BLOCK_GAP * char.size:
|
||||||
|
judge(block)
|
||||||
|
block = []
|
||||||
|
block.append(pieces)
|
||||||
|
previous = char.box[1]
|
||||||
|
judge(block)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
# --- the rules ---------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Context:
|
||||||
|
body_size: Optional[float]
|
||||||
|
body_colour: Optional[str]
|
||||||
|
template: Templates
|
||||||
|
|
||||||
|
|
||||||
|
def context(layouts: list[PageLayout]) -> Context:
|
||||||
|
sizes = Counter(round(char.size * 2) / 2 for layout in layouts for char in layout.chars if char.direction == 0)
|
||||||
|
body_size = max(sizes, key=lambda size: (sizes[size], size)) if sizes else None
|
||||||
|
colours: Counter = Counter()
|
||||||
for layout in layouts:
|
for layout in layouts:
|
||||||
for glyph in layout.glyphs:
|
for char in layout.chars:
|
||||||
if glyph.colour is not None:
|
if char.direction == 0 and char.colour and body_size and abs(char.size - body_size) <= 0.5:
|
||||||
counts[glyph.colour] = counts.get(glyph.colour, 0) + 1
|
colours[char.colour] += 1
|
||||||
return max(counts, key=lambda colour: (counts[colour], colour)) if counts else None
|
body_colour = max(colours, key=lambda colour: (colours[colour], colour)) if colours else None
|
||||||
|
return Context(body_size, body_colour, templates(layouts))
|
||||||
|
|
||||||
|
|
||||||
def _flags(glyph: Glyph, body: Optional[str]) -> Flags:
|
def _struck(layout: PageLayout, grid: Grid, template: Templates, table: set[int]) -> set[int]:
|
||||||
other = glyph.colour if glyph.colour is not None and glyph.colour != body and not glyph.linked else None
|
"""Rule 2: a thin line through the middle of at least STRIKE_MIN_CHARS characters that does
|
||||||
return (glyph.background, True if glyph.struck else None, other)
|
not reach further than STRIKE_OVERHANG past them, measured in the characters' own frame
|
||||||
|
(so text turned by a quarter is struck by a vertical line). Not a table's ruling, not a
|
||||||
|
recurring shape, and not a line at the page edge."""
|
||||||
|
found: set[int] = set()
|
||||||
|
for horizontal in (True, False):
|
||||||
|
lines = layout.hlines if horizontal else layout.vlines
|
||||||
|
used = grid.h_used if horizontal else grid.v_used
|
||||||
|
kind = "h" if horizontal else "v"
|
||||||
|
extent = layout.height if horizontal else layout.width
|
||||||
|
along = [
|
||||||
|
(_centre(layout.chars[i].box)[0 if horizontal else 1], i)
|
||||||
|
for i in range(len(layout.chars))
|
||||||
|
if layout.chars[i].direction in ((0, 180) if horizontal else (90, 270)) and i not in table
|
||||||
|
]
|
||||||
|
along.sort()
|
||||||
|
keys = [key for key, _ in along]
|
||||||
|
for n, line in enumerate(lines):
|
||||||
|
if n in used or line.width > STRIKE_MAX_WIDTH or _line_key(kind, line) in template.lines:
|
||||||
|
continue
|
||||||
|
if line.pos < STRIKE_EDGE * extent or line.pos > (1 - STRIKE_EDGE) * extent:
|
||||||
|
continue
|
||||||
|
low, high = bisect_left(keys, line.start), bisect_right(keys, line.end)
|
||||||
|
hits = []
|
||||||
|
for _, i in along[low:high]:
|
||||||
|
char = layout.chars[i]
|
||||||
|
x0, y0, x1, y1 = char.box
|
||||||
|
lo, hi = (y0, y1) if horizontal else (x0, x1)
|
||||||
|
if char.direction in (0, 270):
|
||||||
|
band = (lo + STRIKE_BAND[0] * char.size, lo + STRIKE_BAND[1] * char.size)
|
||||||
|
else:
|
||||||
|
band = (hi - STRIKE_BAND[1] * char.size, hi - STRIKE_BAND[0] * char.size)
|
||||||
|
if band[0] <= line.pos <= band[1]:
|
||||||
|
hits.append(i)
|
||||||
|
if len(hits) < STRIKE_MIN_CHARS:
|
||||||
|
continue
|
||||||
|
ends = [(layout.chars[i].box[0], layout.chars[i].box[2]) if horizontal else (layout.chars[i].box[1], layout.chars[i].box[3]) for i in hits]
|
||||||
|
reach = STRIKE_OVERHANG * max(layout.chars[i].size for i in hits)
|
||||||
|
if line.start < min(a for a, _ in ends) - reach or line.end > max(b for _, b in ends) + reach:
|
||||||
|
continue
|
||||||
|
found.update(hits)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
def _align(text: str, glyphs: tuple[Glyph, ...], flags: list[Flags]) -> tuple[dict[int, Flags], float]:
|
def _highlighted(layout: PageLayout, template: Templates, table: set[int]) -> dict[int, str]:
|
||||||
|
"""Rule 3: a bright, saturated fill drawn under the text, about as high as a line and no
|
||||||
|
wider than the text it covers plus HIGHLIGHT_MAX_EXTRA type heights. The colour per character."""
|
||||||
|
found: dict[int, str] = {}
|
||||||
|
page_area = layout.width * layout.height
|
||||||
|
for fill in layout.fills:
|
||||||
|
if _snap(*fill.box) + (fill.colour,) in template.fills:
|
||||||
|
continue
|
||||||
|
red, green, blue = _channels(fill.colour)
|
||||||
|
if max(red, green, blue) < HIGHLIGHT_BRIGHT or max(red, green, blue) - min(red, green, blue) < HIGHLIGHT_SPAN:
|
||||||
|
continue
|
||||||
|
x0, y0, x1, y1 = fill.box
|
||||||
|
if page_area and (x1 - x0) * (y1 - y0) > HIGHLIGHT_MAX_AREA * page_area:
|
||||||
|
continue
|
||||||
|
covered = [
|
||||||
|
i for i, char in enumerate(layout.chars)
|
||||||
|
if char.order > fill.order and i not in table and _inside(fill.box, *_centre(char.box))
|
||||||
|
]
|
||||||
|
if not covered:
|
||||||
|
continue
|
||||||
|
size = max(layout.chars[i].size for i in covered)
|
||||||
|
left = min(layout.chars[i].box[0] for i in covered)
|
||||||
|
right = max(layout.chars[i].box[2] for i in covered)
|
||||||
|
if y1 - y0 > HIGHLIGHT_MAX_HEIGHT * size or (x1 - x0) - (right - left) > HIGHLIGHT_MAX_EXTRA * size:
|
||||||
|
continue
|
||||||
|
for i in covered:
|
||||||
|
found[i] = fill.colour
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _different(colour: Optional[str], body: Optional[str]) -> bool:
|
||||||
|
if colour is None or body is None:
|
||||||
|
return False
|
||||||
|
return any(abs(a - b) > COLOUR_DISTANCE for a, b in zip(_channels(colour), _channels(body)))
|
||||||
|
|
||||||
|
|
||||||
|
def page_glyphs(layout: PageLayout, ctx: Context) -> list[Glyph]:
|
||||||
|
"""One glyph per character of the page, in drawing order, with the markings the rules leave."""
|
||||||
|
grid = _grids(layout, ctx.template)
|
||||||
|
table = {
|
||||||
|
i for i, char in enumerate(layout.chars)
|
||||||
|
if any(_inside(region, *_centre(char.box), margin=1.0) for region in grid.regions)
|
||||||
|
}
|
||||||
|
rows = _pieces(layout)
|
||||||
|
code = _code_chars(layout, rows)
|
||||||
|
strong = {i: None for i in range(len(layout.chars))}
|
||||||
|
italic: set[int] = set()
|
||||||
|
if ctx.body_size:
|
||||||
|
for pieces in rows:
|
||||||
|
for piece in pieces:
|
||||||
|
members = [i for i in piece.members if i not in table and i not in code]
|
||||||
|
chars = [layout.chars[i] for i in members]
|
||||||
|
if len(members) < FLOW_MIN_CHARS or len(members) != len(piece.members):
|
||||||
|
continue
|
||||||
|
if not all(FLOW_SIZE[0] * ctx.body_size <= char.size <= FLOW_SIZE[1] * ctx.body_size for char in chars):
|
||||||
|
continue
|
||||||
|
plain = sum(not _is_bold(char) and not _is_italic(char) for char in chars) / len(chars)
|
||||||
|
if plain < FLOW_MIN_PLAIN:
|
||||||
|
continue
|
||||||
|
coloured = [i for i in members if _different(layout.chars[i].colour, ctx.body_colour) and not layout.chars[i].linked]
|
||||||
|
if len(coloured) / len(members) > COLOUR_MAX_SHARE:
|
||||||
|
coloured = []
|
||||||
|
for i in members:
|
||||||
|
char = layout.chars[i]
|
||||||
|
if _is_bold(char) and not _is_italic(char) or i in coloured:
|
||||||
|
strong[i] = "bold"
|
||||||
|
if _is_italic(char):
|
||||||
|
italic.add(i)
|
||||||
|
for i, colour in _highlighted(layout, ctx.template, table).items():
|
||||||
|
strong[i] = colour
|
||||||
|
struck = _struck(layout, grid, ctx.template, table)
|
||||||
|
return [
|
||||||
|
Glyph(char.text, strong[i] is not None, i in struck, i in italic, strong[i] if strong[i] not in (None, "bold") else None)
|
||||||
|
for i, char in enumerate(layout.chars)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def image_pages(layouts: list[PageLayout], ctx: Context) -> list[int]:
|
||||||
|
"""Rule 6: the pages with a picture or a drawing that carries content - not a background, a
|
||||||
|
filler, or something the document repeats on its pages."""
|
||||||
|
pages = []
|
||||||
|
for number, layout in enumerate(layouts, 1):
|
||||||
|
page_area = layout.width * layout.height or 1.0
|
||||||
|
if any(_content_image(image, page_area, ctx.template) for image in layout.images) or _drawing(layout, ctx.template):
|
||||||
|
pages.append(number)
|
||||||
|
return pages
|
||||||
|
|
||||||
|
|
||||||
|
def _content_image(image: ImageUse, page_area: float, template: Templates) -> bool:
|
||||||
|
x0, y0, x1, y1 = image.box
|
||||||
|
width, height = x1 - x0, y1 - y0
|
||||||
|
return not (
|
||||||
|
image.key in template.images
|
||||||
|
or width * height > IMAGE_MAX_PAGE_SHARE * page_area
|
||||||
|
or image.pixels[0] * image.pixels[1] <= IMAGE_MIN_PIXELS
|
||||||
|
or min(width, height) < IMAGE_MIN_POINTS
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _drawing(layout: PageLayout, template: Templates) -> bool:
|
||||||
|
"""A cluster of at least VECTOR_MIN_PATHS paths, big enough, with little text inside, and
|
||||||
|
that is not a table's ruling."""
|
||||||
|
if not layout.paths or len(layout.paths) >= MAX_PATHS:
|
||||||
|
return False
|
||||||
|
grid = _grids(layout, template)
|
||||||
|
page_area = layout.width * layout.height or 1.0
|
||||||
|
boxes = [
|
||||||
|
box for box in layout.paths
|
||||||
|
if _snap(*box) not in template.paths
|
||||||
|
and (box[2] - box[0]) * (box[3] - box[1]) <= IMAGE_MAX_PAGE_SHARE * page_area
|
||||||
|
and not any(_inside(region, *_centre(box), margin=1.0) for region in grid.regions)
|
||||||
|
]
|
||||||
|
if len(boxes) < VECTOR_MIN_PATHS:
|
||||||
|
return False
|
||||||
|
parent = list(range(len(boxes)))
|
||||||
|
|
||||||
|
def find(n: int) -> int:
|
||||||
|
while parent[n] != n:
|
||||||
|
parent[n] = parent[parent[n]]
|
||||||
|
n = parent[n]
|
||||||
|
return n
|
||||||
|
|
||||||
|
cell = 32.0
|
||||||
|
cells: dict[tuple[int, int], list[int]] = {}
|
||||||
|
for n, (x0, y0, x1, y1) in enumerate(boxes):
|
||||||
|
ex0, ey0, ex1, ey1 = x0 - VECTOR_JOIN, y0 - VECTOR_JOIN, x1 + VECTOR_JOIN, y1 + VECTOR_JOIN
|
||||||
|
for cx in range(int(ex0 // cell), int(ex1 // cell) + 1):
|
||||||
|
for cy in range(int(ey0 // cell), int(ey1 // cell) + 1):
|
||||||
|
for m in cells.setdefault((cx, cy), []):
|
||||||
|
mx0, my0, mx1, my1 = boxes[m]
|
||||||
|
if mx0 - VECTOR_JOIN <= x1 and x0 - VECTOR_JOIN <= mx1 and my0 - VECTOR_JOIN <= y1 and y0 - VECTOR_JOIN <= my1:
|
||||||
|
parent[find(n)] = find(m)
|
||||||
|
cells[(cx, cy)].append(n)
|
||||||
|
clusters: dict[int, list[int]] = {}
|
||||||
|
for n in range(len(boxes)):
|
||||||
|
clusters.setdefault(find(n), []).append(n)
|
||||||
|
for members in clusters.values():
|
||||||
|
if len(members) < VECTOR_MIN_PATHS:
|
||||||
|
continue
|
||||||
|
x0 = min(boxes[n][0] for n in members)
|
||||||
|
y0 = min(boxes[n][1] for n in members)
|
||||||
|
x1 = max(boxes[n][2] for n in members)
|
||||||
|
y1 = max(boxes[n][3] for n in members)
|
||||||
|
if (x1 - x0) * (y1 - y0) < VECTOR_MIN_PAGE_SHARE * page_area:
|
||||||
|
continue
|
||||||
|
text = sum(1 for char in layout.chars if _inside((x0, y0, x1, y1), *_centre(char.box)))
|
||||||
|
if text <= VECTOR_MAX_TEXT_PER_PATH * len(members):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# --- laying the markings onto the text ---------------------------------------------------------
|
||||||
|
|
||||||
|
def _align(text: str, glyphs: list[Glyph], flags: list[Flags]) -> tuple[dict[int, Flags], float]:
|
||||||
"""The flags of each character of `text` that has a counterpart among the glyphs, and the
|
"""The flags of each character of `text` that has a counterpart among the glyphs, and the
|
||||||
share of `text`'s non-space characters that has one."""
|
share of `text`'s non-space characters that has one."""
|
||||||
layout_chars, layout_flags = [], []
|
layout_chars, layout_flags = [], []
|
||||||
@@ -246,10 +747,7 @@ def _line_flags(line: str, found: dict[int, Flags], start: int) -> list[Optional
|
|||||||
return flags
|
return flags
|
||||||
|
|
||||||
|
|
||||||
def _open_close(layer: int, value) -> tuple[str, str]:
|
MARKS = (("**", "**"), ("~~", "~~"), ("*", "*"))
|
||||||
if layer == 0:
|
|
||||||
return f'<mark data-farbe="{colour_name(value)}">', "</mark>"
|
|
||||||
return ("~~", "~~") if layer == 1 else ("*", "*")
|
|
||||||
|
|
||||||
|
|
||||||
def _emit(items: list[tuple[str, Optional[Flags]]], layer: int, record) -> str:
|
def _emit(items: list[tuple[str, Optional[Flags]]], layer: int, record) -> str:
|
||||||
@@ -261,30 +759,35 @@ def _emit(items: list[tuple[str, Optional[Flags]]], layer: int, record) -> str:
|
|||||||
if value is None:
|
if value is None:
|
||||||
out.append(inner)
|
out.append(inner)
|
||||||
else:
|
else:
|
||||||
opening, closing = _open_close(layer, value)
|
out.append(MARKS[layer][0] + inner + MARKS[layer][1])
|
||||||
out.append(opening + inner + closing)
|
record(layer)
|
||||||
record(layer, value)
|
|
||||||
return "".join(out)
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation:
|
def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation:
|
||||||
"""`texts` with the markings laid on, the legend and the warnings. A page without
|
"""`texts` with the markings laid on, the legend (one line per kind of marking, at most
|
||||||
markings comes back as it is."""
|
three), the warnings and the pages whose pictures need a look. A page without markings
|
||||||
body = body_colour(layouts)
|
comes back as it is."""
|
||||||
|
ctx = context(layouts)
|
||||||
result = list(texts)
|
result = list(texts)
|
||||||
places: dict[tuple[int, object], list[int]] = {}
|
places: dict[int, list[int]] = {}
|
||||||
|
highlights: dict[str, list[int]] = {}
|
||||||
unmatched: list[int] = []
|
unmatched: list[int] = []
|
||||||
for number, (text, layout) in enumerate(zip(texts, layouts), 1):
|
for number, (text, layout) in enumerate(zip(texts, layouts), 1):
|
||||||
flags = [_flags(glyph, body) for glyph in layout.glyphs]
|
glyphs = page_glyphs(layout, ctx)
|
||||||
|
flags: list[Flags] = [
|
||||||
|
(True if glyph.strong else None, True if glyph.struck else None, True if glyph.italic else None)
|
||||||
|
for glyph in glyphs
|
||||||
|
]
|
||||||
if not any(any(value is not None for value in flag) for flag in flags):
|
if not any(any(value is not None for value in flag) for flag in flags):
|
||||||
continue
|
continue
|
||||||
found, coverage = _align(text, layout.glyphs, flags)
|
found, coverage = _align(text, glyphs, flags)
|
||||||
if coverage < MIN_COVERAGE:
|
if coverage < MIN_COVERAGE:
|
||||||
unmatched.append(number)
|
unmatched.append(number)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
def record(layer: int, value, page: int = number) -> None:
|
def record(layer: int, page: int = number) -> None:
|
||||||
places.setdefault((layer, value if layer != 1 else True), []).append(page)
|
places.setdefault(layer, []).append(page)
|
||||||
|
|
||||||
lines, start = [], 0
|
lines, start = [], 0
|
||||||
for line in text.split("\n"):
|
for line in text.split("\n"):
|
||||||
@@ -292,27 +795,25 @@ def annotate(texts: list[str], layouts: list[PageLayout]) -> Annotation:
|
|||||||
start += len(line) + 1
|
start += len(line) + 1
|
||||||
lines.append(line if not any(line_flags) else _emit(list(zip(line, line_flags)), 0, record))
|
lines.append(line if not any(line_flags) else _emit(list(zip(line, line_flags)), 0, record))
|
||||||
result[number - 1] = "\n".join(lines)
|
result[number - 1] = "\n".join(lines)
|
||||||
|
for glyph in glyphs:
|
||||||
|
if glyph.highlight:
|
||||||
|
highlights.setdefault(glyph.highlight, [])
|
||||||
|
if number not in highlights[glyph.highlight]:
|
||||||
|
highlights[glyph.highlight].append(number)
|
||||||
|
|
||||||
markings = []
|
markings = []
|
||||||
for (layer, value), pages in sorted(places.items(), key=lambda entry: (entry[0][0], str(entry[0][1]))):
|
if 0 in places:
|
||||||
if layer == 0:
|
shades = "; ".join(f"{colour_name(colour)} on page(s) {page_ranges(pages)}" for colour, pages in sorted(highlights.items()))
|
||||||
opening, closing = _open_close(0, value)
|
what = "highlighted" + (f" ({shades})" if shades else "") + ", bold, or in a colour clearly unlike the running text"
|
||||||
markings.append(f"{opening}…{closing} | background {value} ({colour_name(value)}) | places {len(pages)} | pages {page_ranges(pages)}")
|
markings.append(f"**…** | {what} | places {len(places[0])} | pages {page_ranges(places[0])}")
|
||||||
strike = places.get((1, True))
|
if 1 in places:
|
||||||
if strike:
|
markings.append(f"~~…~~ | struck through | places {len(places[1])} | pages {page_ranges(places[1])}")
|
||||||
markings.append(f"~~…~~ | strike-through | places {len(strike)} | pages {page_ranges(strike)}")
|
if 2 in places:
|
||||||
colours = {value: pages for (layer, value), pages in places.items() if layer == 2}
|
markings.append(f"*…* | italic | places {len(places[2])} | pages {page_ranges(places[2])}")
|
||||||
if colours:
|
|
||||||
all_pages = [page for pages in colours.values() for page in pages]
|
|
||||||
names = ", ".join(sorted(colours))
|
|
||||||
markings.append(
|
|
||||||
f"*…* | text colour other than the running text ({body}), not under a link: {names} | "
|
|
||||||
f"places {len(all_pages)} | pages {page_ranges(all_pages)}"
|
|
||||||
)
|
|
||||||
warnings = []
|
warnings = []
|
||||||
if unmatched:
|
if unmatched:
|
||||||
warnings.append(
|
warnings.append(
|
||||||
f"Page(s) {page_ranges(unmatched)}: under {round(MIN_COVERAGE * 100)}% of the text could be matched to the "
|
f"Page(s) {page_ranges(unmatched)}: under {round(MIN_COVERAGE * 100)}% of the text could be matched to the "
|
||||||
"page layout, so the markings (highlight, colour, strike-through) are left out there."
|
"page layout, so the markings (highlight, bold, colour, strike-through, italic) are left out there."
|
||||||
)
|
)
|
||||||
return Annotation(tuple(result), tuple(markings), tuple(warnings))
|
return Annotation(tuple(result), tuple(markings), tuple(warnings), tuple(image_pages(layouts, ctx)))
|
||||||
@@ -51,20 +51,42 @@ def build_pdf(pages: list[list[str]], *, title: Optional[str] = None) -> bytes:
|
|||||||
return bytes(out)
|
return bytes(out)
|
||||||
|
|
||||||
|
|
||||||
Run = Union[str, tuple[str, dict]] # text, or text and options: color, highlight, strike, link
|
Run = Union[str, tuple[str, dict]] # text, or text and options: color, highlight, strike, link, bold, italic, size, rotate
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Picture:
|
||||||
|
"""An image drawn at `x`, `y` (lower left) in `width` x `height` points from a
|
||||||
|
`pixels` x `pixels` raster; `gray` makes the pixel content, so equal grays are the same image."""
|
||||||
|
|
||||||
|
x: float
|
||||||
|
y: float
|
||||||
|
width: float
|
||||||
|
height: float
|
||||||
|
pixels: int = 8
|
||||||
|
gray: int = 0x80
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class StyledPage:
|
class StyledPage:
|
||||||
"""Lines of runs. A run's options: `color` (a gray level, an RGB or a CMYK tuple),
|
"""Lines of runs. A run's options: `color` (a gray level, an RGB or a CMYK tuple),
|
||||||
`highlight` (an RGB filled behind the text), `strike` (a line through it), `link`
|
`highlight` (an RGB filled behind the text), `strike` (a line through it), `link`
|
||||||
(a /Link annotation over it)."""
|
(a /Link annotation over it), `bold`, `italic`, `size` (points, default 10) and
|
||||||
|
`rotate` (90: the text runs upwards; no highlight or strike then).
|
||||||
|
|
||||||
|
`ops` are raw content-stream operators drawn before the text (rectangles, lines,
|
||||||
|
paths), `pictures` are drawn after it, `image` is the 1 x 1 pixel picture of the
|
||||||
|
first tests, and `size` is the page size in points."""
|
||||||
|
|
||||||
lines: list[list[Run]] = field(default_factory=list)
|
lines: list[list[Run]] = field(default_factory=list)
|
||||||
image: bool = False
|
image: bool = False
|
||||||
|
ops: list[str] = field(default_factory=list)
|
||||||
|
pictures: list[Picture] = field(default_factory=list)
|
||||||
|
size: tuple[float, float] = (612, 792)
|
||||||
|
|
||||||
|
|
||||||
CHAR_WIDTH, LINE_HEIGHT, LEFT, TOP = 6, 14, 72, 720 # Courier 10 pt
|
CHAR_WIDTH, LINE_HEIGHT, LEFT, TOP = 6, 14, 72, 720 # Courier 10 pt
|
||||||
|
FONTS = {(False, False): "Courier", (True, False): "Courier-Bold", (False, True): "Courier-Oblique", (True, True): "Courier-BoldOblique"}
|
||||||
|
|
||||||
|
|
||||||
def _operator(color: "float | tuple[float, ...]", fill: bool) -> str:
|
def _operator(color: "float | tuple[float, ...]", fill: bool) -> str:
|
||||||
@@ -91,20 +113,38 @@ def build_styled_pdf(
|
|||||||
return number
|
return number
|
||||||
|
|
||||||
page_ids = [allocate() for _ in pages]
|
page_ids = [allocate() for _ in pages]
|
||||||
font_id, image_id = allocate(), allocate()
|
font_ids = {key: allocate() for key in FONTS}
|
||||||
|
font_names = {key: f"F{index}" for index, key in enumerate(FONTS, 1)}
|
||||||
|
image_ids: dict[tuple[int, int], int] = {}
|
||||||
|
|
||||||
|
def image_object(pixels: int, gray: int) -> int:
|
||||||
|
if (pixels, gray) not in image_ids:
|
||||||
|
number = image_ids[(pixels, gray)] = allocate()
|
||||||
|
data = bytes([gray]) * (pixels * pixels)
|
||||||
|
objects[number] = (
|
||||||
|
f"<< /Type /XObject /Subtype /Image /Width {pixels} /Height {pixels} /ColorSpace /DeviceGray "
|
||||||
|
f"/BitsPerComponent 8 /Length {len(data)} >>\nstream\n"
|
||||||
|
).encode() + data + b"\nendstream"
|
||||||
|
return image_ids[(pixels, gray)]
|
||||||
|
|
||||||
for number, page in zip(page_ids, pages):
|
for number, page in zip(page_ids, pages):
|
||||||
ops, annots = ["BT /F1 10 Tf ET"], []
|
ops, annots, xobjects = list(page.ops), [], {}
|
||||||
for row, line in enumerate(page.lines):
|
for row, line in enumerate(page.lines):
|
||||||
y, x = TOP - LINE_HEIGHT * row, LEFT
|
y, x = TOP - LINE_HEIGHT * row, LEFT
|
||||||
for run in line:
|
for run in line:
|
||||||
text, options = (run, {}) if isinstance(run, str) else run
|
text, options = (run, {}) if isinstance(run, str) else run
|
||||||
width = CHAR_WIDTH * len(text)
|
size = options.get("size", 10)
|
||||||
|
width = 0.6 * size * len(text)
|
||||||
|
font = font_names[(bool(options.get("bold")), bool(options.get("italic")))]
|
||||||
if "highlight" in options:
|
if "highlight" in options:
|
||||||
ops.append(f"{_operator(options['highlight'], True)} {x} {y - 3} {width} 12 re f 0 g")
|
ops.append(f"{_operator(options['highlight'], True)} {x} {y - 3} {width} {size * 1.2} re f 0 g")
|
||||||
if "color" in options:
|
if "color" in options:
|
||||||
ops.append(_operator(options["color"], True))
|
ops.append(_operator(options["color"], True))
|
||||||
escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
|
escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
|
||||||
ops.append(f"BT /F1 10 Tf {x} {y} Td ({escaped}) Tj ET")
|
if options.get("rotate") == 90:
|
||||||
|
ops.append(f"BT /{font} {size} Tf 0 1 -1 0 {x} {y} Tm ({escaped}) Tj ET")
|
||||||
|
else:
|
||||||
|
ops.append(f"BT /{font} {size} Tf {x} {y} Td ({escaped}) Tj ET")
|
||||||
if "color" in options:
|
if "color" in options:
|
||||||
ops.append("0 g")
|
ops.append("0 g")
|
||||||
if options.get("strike"):
|
if options.get("strike"):
|
||||||
@@ -115,18 +155,23 @@ def build_styled_pdf(
|
|||||||
"/A << /S /URI /URI (https://example.org/) >> >>"
|
"/A << /S /URI /URI (https://example.org/) >> >>"
|
||||||
)
|
)
|
||||||
x += width
|
x += width
|
||||||
|
pictures = list(page.pictures)
|
||||||
if page.image:
|
if page.image:
|
||||||
ops.append("q 40 0 0 40 72 400 cm /Im1 Do Q")
|
pictures.append(Picture(72, 400, 40, 40, pixels=8))
|
||||||
|
for index, picture in enumerate(pictures):
|
||||||
|
xobjects[f"Im{index}"] = image_object(picture.pixels, picture.gray)
|
||||||
|
ops.append(f"q {picture.width} 0 0 {picture.height} {picture.x} {picture.y} cm /Im{index} Do Q")
|
||||||
stream = "\n".join(ops).encode("latin-1")
|
stream = "\n".join(ops).encode("latin-1")
|
||||||
content_id = allocate()
|
content_id = allocate()
|
||||||
objects[content_id] = b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream"
|
objects[content_id] = b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream"
|
||||||
xobject = f"/XObject << /Im1 {image_id} 0 R >>" if page.image else ""
|
xobject = "/XObject << " + " ".join(f"/{name} {ref} 0 R" for name, ref in xobjects.items()) + " >>" if xobjects else ""
|
||||||
|
fonts = " ".join(f"/{font_names[key]} {font_ids[key]} 0 R" for key in FONTS)
|
||||||
objects[number] = (
|
objects[number] = (
|
||||||
f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents {content_id} 0 R "
|
f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {page.size[0]:g} {page.size[1]:g}] /Contents {content_id} 0 R "
|
||||||
f"/Resources << /Font << /F1 {font_id} 0 R >> {xobject} >> /Annots [{' '.join(annots)}] >>"
|
f"/Resources << /Font << {fonts} >> {xobject} >> /Annots [{' '.join(annots)}] >>"
|
||||||
).encode()
|
).encode()
|
||||||
objects[font_id] = b"<< /Type /Font /Subtype /Type1 /BaseFont /Courier >>"
|
for key, number in font_ids.items():
|
||||||
objects[image_id] = b"<< /Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 1 >>\nstream\n\x80\nendstream"
|
objects[number] = f"<< /Type /Font /Subtype /Type1 /BaseFont /{FONTS[key]} >>".encode()
|
||||||
objects[1] = b"<< /Type /Catalog /Pages 2 0 R >>"
|
objects[1] = b"<< /Type /Catalog /Pages 2 0 R >>"
|
||||||
objects[2] = f"<< /Type /Pages /Kids [{' '.join(f'{n} 0 R' for n in page_ids)}] /Count {len(pages)} >>".encode()
|
objects[2] = f"<< /Type /Pages /Kids [{' '.join(f'{n} 0 R' for n in page_ids)}] /Count {len(pages)} >>".encode()
|
||||||
if outline:
|
if outline:
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ reading them here. The tests that need a package skip where it is not installed
|
|||||||
installs `tools/requirements.txt`, so they run there."""
|
installs `tools/requirements.txt`, so they run there."""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
import importlib.util
|
import importlib.util
|
||||||
import io
|
import io
|
||||||
import re
|
import re
|
||||||
@@ -15,7 +16,7 @@ import pytest
|
|||||||
from chemenu import filters
|
from chemenu import filters
|
||||||
from chemenu.commands import doctor
|
from chemenu.commands import doctor
|
||||||
from chemenu.filters import pdf_layout
|
from chemenu.filters import pdf_layout
|
||||||
from chemenu.tests.binary_formats import StyledPage, build_msg, build_pdf, build_styled_pdf
|
from chemenu.tests.binary_formats import Picture, StyledPage, build_msg, build_pdf, build_styled_pdf
|
||||||
from chemenu.tests.test_raw_derive import ( # noqa: F401 (tree is a fixture)
|
from chemenu.tests.test_raw_derive import ( # noqa: F401 (tree is a fixture)
|
||||||
_derive,
|
_derive,
|
||||||
_header_and_text,
|
_header_and_text,
|
||||||
@@ -140,7 +141,7 @@ def _derive_pdf(tree, pdf: bytes, name: str = "doc.pdf"):
|
|||||||
|
|
||||||
|
|
||||||
def _markings_stripped(text: str) -> str:
|
def _markings_stripped(text: str) -> str:
|
||||||
return re.sub(r'<mark data-farbe="[^"]*">|</mark>|~~|\*', "", text)
|
return re.sub(r"~~|\*", "", text)
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@@ -174,52 +175,52 @@ def test_a_pdf_without_outline_or_markings_has_neither_field_and_no_image_check(
|
|||||||
header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[LONG_LINE]])], title="T"))
|
header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[LONG_LINE]])], title="T"))
|
||||||
out = capsys.readouterr().out
|
out = capsys.readouterr().out
|
||||||
assert "outline" not in header and "markings" not in header
|
assert "outline" not in header and "markings" not in header
|
||||||
assert "<mark" not in text and "WARN" not in out and out.count("CHECK") == 1
|
assert "*" not in text and "WARN" not in out and out.count("CHECK") == 1
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_a_highlight_behind_text_becomes_a_mark_per_colour_and_the_legend_counts_them(tree):
|
def test_a_highlight_behind_text_becomes_a_strong_mark_and_the_legend_names_the_colours(tree):
|
||||||
pdf = build_styled_pdf([
|
pdf = build_styled_pdf([
|
||||||
StyledPage([["Intro ", ("old rule", {"highlight": YELLOW}), " and ", ("new rule", {"highlight": GREEN}), " end"]]),
|
StyledPage([["Intro ", ("old rule", {"highlight": YELLOW}), " and ", ("new rule", {"highlight": GREEN}), " end"]]),
|
||||||
StyledPage([[("second yellow", {"highlight": YELLOW})]]),
|
StyledPage([[("second yellow", {"highlight": YELLOW})]]),
|
||||||
])
|
])
|
||||||
header, text = _derive_pdf(tree, pdf)
|
header, text = _derive_pdf(tree, pdf)
|
||||||
assert 'Intro <mark data-farbe="gelb">old rule</mark> and <mark data-farbe="grün">new rule</mark> end' in text
|
assert "Intro **old rule** and **new rule** end" in text and "**second yellow**" in text
|
||||||
assert '<mark data-farbe="gelb">second yellow</mark>' in text
|
(entry,) = [e for e in header["markings"] if e.startswith("**…**")]
|
||||||
legend = {entry.split(" | ")[0]: entry for entry in header["markings"]}
|
assert "places 3 | pages 1-2" in entry
|
||||||
assert "places 2 | pages 1-2" in legend['<mark data-farbe="gelb">…</mark>']
|
assert "gelb on page(s) 1-2" in entry and "grün on page(s) 1" in entry
|
||||||
assert "#ffff00" in legend['<mark data-farbe="gelb">…</mark>']
|
|
||||||
assert "places 1 | pages 1" in legend['<mark data-farbe="grün">…</mark>']
|
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_a_background_without_a_name_is_marked_with_its_hex_value(tree):
|
def test_only_a_bright_saturated_fill_about_as_wide_as_its_text_is_a_highlight(tree):
|
||||||
pdf = build_styled_pdf([StyledPage([[("odd", {"highlight": (0.2, 0.4, 0.6)})]])])
|
dark = "0.2 0.3 0.5 rg 72 700 200 14 re f 0 g" # no highlighter is dark
|
||||||
_header, text = _derive_pdf(tree, pdf)
|
panel = "1 1 0 rg 60 650 480 40 re f 0 g" # a box far bigger than the line it holds
|
||||||
assert '<mark data-farbe="#336699">odd</mark>' in text
|
wide = "1 1 0 rg 60 612 480 12 re f 0 g" # as high as a line, but wider than the words
|
||||||
|
grey = "0.85 g 72 580 200 12 re f 0 g" # shading, not a pen
|
||||||
|
pdf = build_styled_pdf([StyledPage(
|
||||||
@needs_pypdf
|
[[("on white", {"highlight": (1, 1, 1)})], ["on dark"], ["in a panel"], ["short"], ["on grey"]], ops=[dark, panel, wide, grey],
|
||||||
@needs_pdfminer
|
)])
|
||||||
def test_a_white_fill_is_not_a_highlight(tree):
|
|
||||||
pdf = build_styled_pdf([StyledPage([[("on white", {"highlight": (1, 1, 1)})]])])
|
|
||||||
header, text = _derive_pdf(tree, pdf)
|
header, text = _derive_pdf(tree, pdf)
|
||||||
assert "<mark" not in text and "markings" not in header
|
assert "*" not in text and "markings" not in header
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_colour_other_than_the_running_text_is_emphasised_unless_it_sits_under_a_link(tree):
|
def test_bold_and_a_different_colour_are_strong_but_only_inside_running_text(tree):
|
||||||
pdf = build_styled_pdf([StyledPage([[
|
pdf = build_styled_pdf([StyledPage([
|
||||||
"Running text in the body colour, ", ("alert", {"color": RED}), " and ",
|
["Running text in the body colour, ", ("alert", {"color": RED}), " and ", ("see page 3", {"color": BLUE, "link": True}), " and a ", ("bold", {"bold": True}), " word."],
|
||||||
("see page 3", {"color": BLUE, "link": True}), " and ", ("blue without a link", {"color": BLUE}), ".",
|
[("A whole line of heading text in bold", {"bold": True})],
|
||||||
]])])
|
["Label ", ("bold", {"bold": True})],
|
||||||
|
[("A line that is red from end to end, a banner", {"color": RED})],
|
||||||
|
])])
|
||||||
header, text = _derive_pdf(tree, pdf)
|
header, text = _derive_pdf(tree, pdf)
|
||||||
assert "colour, *alert* and see page 3 and *blue without a link*." in text
|
assert "colour, **alert** and see page 3 and a **bold** word." in text
|
||||||
(entry,) = [e for e in header["markings"] if e.startswith("*…*")]
|
assert "\nA whole line of heading text in bold\n" in text
|
||||||
assert "#ff0000" in entry and "#0000ff" in entry and "places 2" in entry and "(#000000)" in entry
|
assert "\nLabel bold\n" in text and "\nA line that is red from end to end, a banner\n" in text
|
||||||
|
(entry,) = [e for e in header["markings"] if e.startswith("**…**")]
|
||||||
|
assert "places 2" in entry and "bold" in entry and "colour" in entry
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@@ -227,10 +228,22 @@ def test_colour_other_than_the_running_text_is_emphasised_unless_it_sits_under_a
|
|||||||
def test_the_running_text_colour_is_the_one_with_most_characters_not_black(tree):
|
def test_the_running_text_colour_is_the_one_with_most_characters_not_black(tree):
|
||||||
navy, gray_level = (0, 0.18, 0.49), 0.4
|
navy, gray_level = (0, 0.18, 0.49), 0.4
|
||||||
pdf = build_styled_pdf([StyledPage([[
|
pdf = build_styled_pdf([StyledPage([[
|
||||||
("Body text that is set in navy, ", {"color": navy}), ("margin", {"color": gray_level}), (" more navy body text.", {"color": navy}),
|
("Body text that is set in navy, ", {"color": navy}), ("margin", {"color": (0.9, 0.2, 0.2)}), (" more navy body text.", {"color": navy}),
|
||||||
]])])
|
]])])
|
||||||
_header, text = _derive_pdf(tree, pdf)
|
_header, text = _derive_pdf(tree, pdf)
|
||||||
assert "Body text that is set in navy, *margin* more navy body text." in text
|
assert "Body text that is set in navy, **margin** more navy body text." in text
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_italic_is_marked_inside_running_text_and_not_when_the_whole_line_is_italic(tree):
|
||||||
|
pdf = build_styled_pdf([StyledPage([
|
||||||
|
["Running text with an ", ("emphasised", {"italic": True}), " word in the middle."],
|
||||||
|
[("A whole line of quotation in italic type", {"italic": True})],
|
||||||
|
])])
|
||||||
|
header, text = _derive_pdf(tree, pdf)
|
||||||
|
assert "an *emphasised* word" in text and "\nA whole line of quotation in italic type\n" in text
|
||||||
|
assert [e for e in header["markings"] if e.startswith("*…*")]
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@@ -242,12 +255,91 @@ def test_a_line_through_text_becomes_a_strike_through(tree):
|
|||||||
assert any(entry.startswith("~~…~~") and "places 1" in entry for entry in header["markings"])
|
assert any(entry.startswith("~~…~~") and "places 1" in entry for entry in header["markings"])
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_a_line_that_is_not_the_characters_own_is_no_strike_through(tree):
|
||||||
|
rule = "0 G 0.7 w 72 723 m 540 723 l S" # through the middle of "short", but running far past it
|
||||||
|
one = "0 G 0.7 w 72 709 m 78 709 l S" # reaches a single character
|
||||||
|
pdf = build_styled_pdf([StyledPage([["short"], ["x"], ["end"]], ops=[rule, one])])
|
||||||
|
header, text = _derive_pdf(tree, pdf)
|
||||||
|
assert "~~" not in text and "markings" not in header
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_text_in_a_ruled_table_is_left_alone_and_its_grid_strikes_nothing(tree):
|
||||||
|
rows = [[f"cell {n}a ", f"cell {n}b"] for n in range(3)]
|
||||||
|
grid = ["0 G 0.5 w"]
|
||||||
|
for y in (736, 722, 708, 694, 680, 666): # horizontal rules through the lines of text as well as between
|
||||||
|
grid.append(f"60 {y - 4} m 200 {y - 4} l S")
|
||||||
|
for x in (60, 130, 200):
|
||||||
|
grid.append(f"{x} 660 m {x} 740 l S")
|
||||||
|
shaded = "1 1 0 rg 60 716 140 14 re f 0 g"
|
||||||
|
pdf = build_styled_pdf([StyledPage([[("cell 0a", {"bold": True}), " cell 0b"], ["cell 1a cell 1b"], ["cell 2a cell 2b"]], ops=["\n".join(grid), shaded])])
|
||||||
|
header, text = _derive_pdf(tree, pdf)
|
||||||
|
assert "*" not in text and "~" not in text and "markings" not in header
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_text_turned_by_a_quarter_is_struck_by_a_line_in_its_own_direction_only(tree):
|
||||||
|
pdf = build_styled_pdf([StyledPage([[("turned words", {"rotate": 90})]], ops=[
|
||||||
|
"0 G 0.5 w 60 700 m 300 700 l S", # a horizontal line is no strike-through for it
|
||||||
|
"0 G 0.7 w 75 720 m 75 756 l S", # nor is a line that is not in its middle
|
||||||
|
], size=(612, 1000))])
|
||||||
|
header, text = _derive_pdf(tree, pdf)
|
||||||
|
assert "~~" not in text
|
||||||
|
struck_pdf = build_styled_pdf([StyledPage([[("turned words", {"rotate": 90})]], ops=["0 G 0.7 w 68.8 720 m 68.8 792 l S"], size=(612, 1000))])
|
||||||
|
_header, text = _derive_pdf(tree, struck_pdf, "struck.pdf")
|
||||||
|
assert "~~turned words~~" in text
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_program_text_is_left_alone(tree):
|
||||||
|
code = [
|
||||||
|
[("if", {"bold": True, "color": BLUE}), " (x == 1) {"],
|
||||||
|
[" ", ("return", {"bold": True, "color": BLUE}), " f(a, b);"],
|
||||||
|
[" y = g(x) + h[0];"],
|
||||||
|
["}"],
|
||||||
|
]
|
||||||
|
_header, text = _derive_pdf(tree, build_styled_pdf([StyledPage(code)]))
|
||||||
|
assert "*" not in text
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_shapes_that_recur_on_the_pages_are_part_of_the_page_not_of_the_text(tree, capsys):
|
||||||
|
rule = "0 G 0.7 w 72 723 m 160 723 l S"
|
||||||
|
banner = "1 1 0 rg 72 717 90 12 re f 0 g"
|
||||||
|
pages = [StyledPage([["Letterhead text"]], ops=[rule, banner], pictures=[Picture(400, 700, 40, 40, pixels=8)]) for _ in range(5)]
|
||||||
|
pages.append(StyledPage([["Last page"]], pictures=[Picture(300, 300, 60, 60, pixels=8, gray=0x40)]))
|
||||||
|
header, text = _derive_pdf(tree, build_styled_pdf(pages))
|
||||||
|
assert "*" not in text and "~" not in text and "markings" not in header
|
||||||
|
assert "Images on page(s) 6 are not" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_only_pictures_that_carry_content_are_named_in_the_check(tree, capsys):
|
||||||
|
page = lambda *pictures, ops=(): StyledPage([["x"]], pictures=list(pictures), ops=list(ops)) # noqa: E731
|
||||||
|
pdf = build_styled_pdf([
|
||||||
|
page(Picture(0, 0, 612, 792, pixels=8, gray=0x20)), # 1: a background
|
||||||
|
page(Picture(100, 400, 40, 40, pixels=2)), # 2: a filler a few pixels in size
|
||||||
|
page(Picture(100, 400, 6, 6, pixels=8)), # 3: a speck a few points in size
|
||||||
|
page(Picture(100, 400, 120, 90, pixels=8, gray=0x40)), # 4: content
|
||||||
|
page(ops=[f"0 G 0.5 w {80 + 8 * n} 400 m {84 + 8 * n} {430 + (n % 3) * 9} l S" for n in range(30)]), # 5: a drawing
|
||||||
|
])
|
||||||
|
_derive_pdf(tree, pdf)
|
||||||
|
assert "CHECK Images on page(s) 4-5 are not contained in the text" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_markings_nest_and_a_gap_between_two_equal_runs_is_inside_the_tag(tree):
|
def test_markings_nest_and_a_gap_between_two_equal_runs_is_inside_the_tag(tree):
|
||||||
pdf = build_styled_pdf([StyledPage([[("two words", {"highlight": YELLOW, "color": RED, "strike": True})], ["tail of the page in the running colour"]])])
|
pdf = build_styled_pdf([StyledPage([["Running text, ", ("two words", {"highlight": YELLOW, "strike": True, "italic": True}), " and the rest of the line."]])])
|
||||||
_header, text = _derive_pdf(tree, pdf)
|
_header, text = _derive_pdf(tree, pdf)
|
||||||
assert '<mark data-farbe="gelb">~~*two words*~~</mark>' in text
|
assert "**~~*two words*~~**" in text
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@@ -259,25 +351,65 @@ def test_a_marking_never_runs_over_a_line_or_a_page_marker(tree):
|
|||||||
])
|
])
|
||||||
_header, text = _derive_pdf(tree, pdf)
|
_header, text = _derive_pdf(tree, pdf)
|
||||||
for line in text.splitlines():
|
for line in text.splitlines():
|
||||||
assert line.count("<mark") == line.count("</mark>") and line.count("~~") % 2 == 0, line
|
assert line.count("**") % 2 == 0 and line.count("~~") % 2 == 0, line
|
||||||
assert '<mark data-farbe="gelb">first line</mark>\n<mark data-farbe="gelb">second line</mark>' in text
|
assert "**first line**\n**second line**" in text
|
||||||
assert "</mark>\n\n<!-- page 2 -->\n<mark" in text
|
assert "**\n\n<!-- page 2 -->\n**" in text
|
||||||
|
|
||||||
|
|
||||||
|
def _tags_hug_their_text(text: str) -> bool:
|
||||||
|
"""No opening tag is followed, and no closing tag preceded, by whitespace."""
|
||||||
|
open_kinds: set[str] = set()
|
||||||
|
for match in re.finditer(r"\*\*|~~|\*", text):
|
||||||
|
kind = match.group()
|
||||||
|
if kind in open_kinds:
|
||||||
|
open_kinds.discard(kind)
|
||||||
|
if text[match.start() - 1].isspace():
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
open_kinds.add(kind)
|
||||||
|
if text[match.end():match.end() + 1].isspace():
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_no_tag_ever_sits_next_to_whitespace_on_its_inner_side(tree):
|
||||||
|
pdf = build_styled_pdf([StyledPage([
|
||||||
|
["Plain ", ("yellow ", {"highlight": YELLOW}), "and ", (" padded", {"highlight": GREEN}), " words ", ("gone ", {"strike": True}), "on."],
|
||||||
|
[("Running text that mixes ", {}), ("bold ", {"bold": True}), (" and ", {"bold": True}), ("more bold", {"bold": True}), " then plain text."],
|
||||||
|
[(" ", {"strike": True}), ("left", {"strike": True}), (" ", {"strike": True}), ("right", {"strike": True})],
|
||||||
|
])])
|
||||||
|
_header, text = _derive_pdf(tree, pdf)
|
||||||
|
assert "**" in text and "~~" in text
|
||||||
|
assert _tags_hug_their_text(text), text
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_removing_the_markings_gives_the_text_the_pdf_has_without_them(tree):
|
def test_removing_the_markings_gives_the_text_the_pdf_has_without_them(tree):
|
||||||
styled = build_styled_pdf([StyledPage([
|
styled = build_styled_pdf([StyledPage([
|
||||||
["Plain ", ("yellow", {"highlight": YELLOW}), " ", ("red", {"color": RED}), " ", ("struck", {"strike": True}), " end"],
|
["Plain ", ("yellow", {"highlight": YELLOW}), " ", ("red", {"color": RED}), " ", ("struck", {"strike": True}), " end of the line, ", ("bold", {"bold": True}), " ", ("slanted", {"italic": True}), "."],
|
||||||
[("whole line", {"highlight": GREEN})],
|
[("whole line", {"highlight": GREEN})],
|
||||||
])])
|
])])
|
||||||
unstyled = build_styled_pdf([StyledPage([["Plain yellow red struck end"], ["whole line"]])])
|
unstyled = build_styled_pdf([StyledPage([["Plain yellow red struck end of the line, bold slanted."], ["whole line"]])])
|
||||||
_header, marked = _derive_pdf(tree, styled, "marked.pdf")
|
_header, marked = _derive_pdf(tree, styled, "marked.pdf")
|
||||||
_header, plain = _derive_pdf(tree, unstyled, "plain.pdf")
|
_header, plain = _derive_pdf(tree, unstyled, "plain.pdf")
|
||||||
assert marked != plain
|
assert marked != plain
|
||||||
assert _markings_stripped(marked) == plain
|
assert _markings_stripped(marked) == plain
|
||||||
|
|
||||||
|
|
||||||
|
@needs_pypdf
|
||||||
|
@needs_pdfminer
|
||||||
|
def test_the_legend_is_at_most_three_lines(tree):
|
||||||
|
pdf = build_styled_pdf([StyledPage([[
|
||||||
|
"A line of running text with ", ("highlight", {"highlight": YELLOW}), ", ", ("struck", {"strike": True}), " and an ",
|
||||||
|
("italic", {"italic": True}), " word, and ", ("bold", {"bold": True}), ".",
|
||||||
|
]])])
|
||||||
|
header, _text = _derive_pdf(tree, pdf)
|
||||||
|
assert [entry.split(" | ")[0] for entry in header["markings"]] == ["**…**", "~~…~~", "*…*"]
|
||||||
|
|
||||||
|
|
||||||
@needs_pypdf
|
@needs_pypdf
|
||||||
@needs_pdfminer
|
@needs_pdfminer
|
||||||
def test_pages_with_images_get_one_check_line(tree, capsys):
|
def test_pages_with_images_get_one_check_line(tree, capsys):
|
||||||
@@ -295,14 +427,14 @@ def test_a_page_whose_text_does_not_match_the_layout_is_left_unmarked_with_a_war
|
|||||||
|
|
||||||
def disagree(data):
|
def disagree(data):
|
||||||
first, second = real(data)
|
first, second = real(data)
|
||||||
garbled = tuple(pdf_layout.Glyph(c, g.colour, g.background, g.struck, g.linked) for g, c in zip(first.glyphs, "zzzzzzzzzzz"))
|
garbled = tuple(dataclasses.replace(char, text="z") for char in first.chars)
|
||||||
return [pdf_layout.PageLayout(garbled, False), second]
|
return [dataclasses.replace(first, chars=garbled), second]
|
||||||
|
|
||||||
monkeypatch.setattr(pdf_layout, "analyse", disagree)
|
monkeypatch.setattr(pdf_layout, "analyse", disagree)
|
||||||
header, text = _derive_pdf(tree, pdf)
|
header, text = _derive_pdf(tree, pdf)
|
||||||
out = capsys.readouterr().out
|
out = capsys.readouterr().out
|
||||||
assert "highlighted" in text and '<mark data-farbe="gelb">highlighted' not in text
|
assert "highlighted" in text and "**highlighted" not in text
|
||||||
assert '<mark data-farbe="gelb">fine</mark>' in text
|
assert "**fine**" in text
|
||||||
assert "WARN" in out and "Page(s) 1" in out and "left out" in out
|
assert "WARN" in out and "Page(s) 1" in out and "left out" in out
|
||||||
|
|
||||||
|
|
||||||
@@ -315,10 +447,11 @@ def test_a_layout_that_cannot_be_read_costs_the_markings_not_the_derivation(monk
|
|||||||
monkeypatch.setattr(pdf_layout, "analyse", broken)
|
monkeypatch.setattr(pdf_layout, "analyse", broken)
|
||||||
header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[("kept", {"highlight": YELLOW})]])], outline=[("Top", 1, 0)]))
|
header, text = _derive_pdf(tree, build_styled_pdf([StyledPage([[("kept", {"highlight": YELLOW})]])], outline=[("Top", 1, 0)]))
|
||||||
out = capsys.readouterr().out
|
out = capsys.readouterr().out
|
||||||
assert "kept" in text and "<mark" not in text and header["outline"][0].startswith("Top | level 1 | page 1")
|
assert "kept" in text and "**" not in text and header["outline"][0].startswith("Top | level 1 | page 1")
|
||||||
assert "WARN" in out and "boom" in out
|
assert "WARN" in out and "boom" in out
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_pdf_without_pdfminer_is_refused_by_package_name(tree, monkeypatch, capsys):
|
def test_a_pdf_without_pdfminer_is_refused_by_package_name(tree, monkeypatch, capsys):
|
||||||
monkeypatch.setattr(type(filters.filter_for("a.pdf")), "missing", lambda self: ["pdfminer.six"])
|
monkeypatch.setattr(type(filters.filter_for("a.pdf")), "missing", lambda self: ["pdfminer.six"])
|
||||||
_put(tree, "a.pdf", b"%PDF-1.4")
|
_put(tree, "a.pdf", b"%PDF-1.4")
|
||||||
@@ -361,9 +494,9 @@ def test_outline_lines_count_the_header_whatever_fields_it_has():
|
|||||||
|
|
||||||
|
|
||||||
def test_characters_are_matched_across_a_difference_in_the_middle():
|
def test_characters_are_matched_across_a_difference_in_the_middle():
|
||||||
flag = ("#ffff00", None, None)
|
flag = (True, None, None)
|
||||||
glyphs = tuple(pdf_layout.Glyph(c, "#000000", "#ffff00" if i >= 4 else None, False, False) for i, c in enumerate("start ghost middle end"))
|
glyphs = [pdf_layout.Glyph(c, i >= 4, False, False) for i, c in enumerate("start ghost middle end")]
|
||||||
found, coverage = pdf_layout._align("start middle end", glyphs, [pdf_layout._flags(g, "#000000") for g in glyphs])
|
found, coverage = pdf_layout._align("start middle end", glyphs, [(True if g.strong else None, None, None) for g in glyphs])
|
||||||
assert coverage == 1.0
|
assert coverage == 1.0
|
||||||
assert [found.get(i) for i in (0, 6, 7, 13)] == [(None, None, None), flag, flag, flag]
|
assert [found.get(i) for i in (0, 6, 7, 13)] == [(None, None, None), flag, flag, flag]
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user