From d393e2ffe110e5fa308b96bc2c680b66960f3f48 Mon Sep 17 00:00:00 2001 From: Torben Nehmer Date: Wed, 14 Oct 2026 18:07:51 +0200 Subject: [PATCH] =?UTF-8?q?instructions/dev:=20Konverter-Vergleich=20?= =?UTF-8?q?=E2=80=93=20eigene=20Ableitung=20gegen=20Referenzwerkzeuge=20am?= =?UTF-8?q?=20privaten=20Korpus,=20sieben=20Ma=C3=9Fe,=20Grundlinie=20nur?= =?UTF-8?q?=20bewusst=20fortgeschrieben=20(#233);=2010.1.0-beta.3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Files changed: - CHANGES.md - VERSION - docs/dev/development.md - instructions/dev/converter-comparison.md - instructions/dev/stack-mode.md - instructions/dev/testing-conventions.md Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2 --- CHANGES.md | 6 +- VERSION | 2 +- docs/dev/development.md | 8 ++ instructions/dev/converter-comparison.md | 169 +++++++++++++++++++++++ instructions/dev/stack-mode.md | 4 + instructions/dev/testing-conventions.md | 4 +- 6 files changed, 190 insertions(+), 3 deletions(-) create mode 100644 instructions/dev/converter-comparison.md diff --git a/CHANGES.md b/CHANGES.md index 8b60262..c52ed8a 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,13 +14,17 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.1.0-beta.2 - 2026-10-14 - Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release +## 10.1.0-beta.3 - 2026-10-14 - instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie **Author:** Torben Nehmer +**Medium impact** - raw derive PDF: Gliederung mit Seite und Zeile im Kopf, Hinterlegungen, Durchstreichungen und abweichende Textfarben als Auszeichnung im Text, Bildseiten als CHECK - Konverter-Testkorpus und Harness ziehen in das private Repository chemenu-testfiles; neuer Workflow Converters, Freigabe-Gate im Release + +**Low impact** +- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie ### Konverter-Tests gegen echte Dokumente im privaten Repository diff --git a/VERSION b/VERSION index 2386dcd..34befc4 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.1.0-beta.2 +10.1.0-beta.3 diff --git a/docs/dev/development.md b/docs/dev/development.md index 4b5ce84..110177e 100644 --- a/docs/dev/development.md +++ b/docs/dev/development.md @@ -207,6 +207,14 @@ works only because the private repository's own Actions settings grant `torben/c without that grant the checkout step fails with "Determining the default branch … not found", which reads like a missing repository rather than a missing permission. +That workflow shows that the corpus still derives, not that the derivation is good. The second +question is a session's: `harness/compare/` in the same private repository derives every corpus +file with the checkout's filters and with reference tools (pymupdf4llm, markitdown, pandoc, each in +a throwaway environment of its own, never in `tools/.venv`), measures wording fidelity, markup +overhead, missing semantics, structure, CHECK precision and runtime, and reports the difference to +a committed baseline. When a session runs it and what it does with the result is +[instructions/dev/converter-comparison.md](../../instructions/dev/converter-comparison.md). + Three more workflows carry the live tests of the tracker adapters (Super Productivity, CalDAV): `ci.yml` runs the CalDAV half in a step of its own against Radicale as a process, `tracker-live.yml` runs nightly and covers both providers, and `sp-live-image.yml` builds the diff --git a/instructions/dev/converter-comparison.md b/instructions/dev/converter-comparison.md new file mode 100644 index 0000000..c196190 --- /dev/null +++ b/instructions/dev/converter-comparison.md @@ -0,0 +1,169 @@ +--- +type: types/instruction.md +name: converter-comparison +description: How to measure whether raw derive's conversion of a non-text source (PDF, DOCX, PPTX, XLSX, mail, HTML) is good rather than merely working - running the stack's own derivation against reference tools over the private corpus, the seven measures, judging markings against the original, and when the baseline may move. +--- +# Compare the converters against reference tools on the real corpus + +The real-document tests in the private repository check that a derivation succeeds, is stable and +contains the text it should. They do not check whether it is **good**: whether a marking is right, +whether the converter adds noise, whether an available tool reads the format better. That takes a +judgment against the original, and only part of it is a number. Without this step, a PDF marking +change once shipped to a candidate before anyone saw that it added a quarter to a guide's length +and struck through 190 single letters along the grid lines of rotated table pages. + +The comparison runs in the private repository `chemenu-testfiles` (cloned into `testfiles/`, +[dev-setup.md](dev-setup.md) step 6): `harness/compare/compare.py` derives every corpus file with +this checkout's filters and with the reference tools, computes the measures, and reports the +difference to `harness/baseline.json`. Its README has the setup and every option; this file says +when to run it and what to do with the result. + + +## Contents + +- [When to run](#when-to-run) +- [What is compared](#what-is-compared) +- [The measures](#the-measures) +- [Steps](#steps) +- [Decision points](#decision-points) +- [Scope](#scope) + + +## When to run + +- **Before designing a converter change** - measure where it starts. The numbers go into the + design, and a difference to the baseline found here is explained before anything is built on it. +- **After building it, before the publish** - measure what it changed. Every line of the + difference is either intended or a defect. +- **When a file is added to the corpus** - it gets its first measures and its baseline entry. +- **When a format shows a new effect** - colour as decoration, shaded table cells, slide panels, + syntax colouring in code: measure it before deciding whether the converter should react. + +A change elsewhere in the stack does not need it; the `converters` workflow already proves that the +corpus still derives ([publish-and-ci.md](publish-and-ci.md) step 3). + +## What is compared + +Every format with a filter: PDF, DOCX, PPTX, XLSX/XLSM, EML/MSG and HTML. A ZIP is only a container +- its members are corpus files of their own - and is skipped. + +**Reference tools, by role.** The starting set is below; the versions are pinned in the harness +(`harness/compare/requirements-reference.txt`), not here. + +| Format | Reference | Oracle (read from the file directly) | +|---|---|---| +| PDF | pymupdf4llm | MuPDF style flags (`TEXT_COLLECT_STYLES`): strike-out, bold, italic, image and rotated pages | +| DOCX | markitdown, pandoc | the document XML: tracked insertions and deletions, comments, run formatting | +| PPTX | markitdown | python-pptx: slides and speaker notes | +| XLSX/XLSM | markitdown | openpyxl: sheets, hidden sheets, formula cells | +| MSG | markitdown | - | +| HTML | markitdown, pandoc | - | +| EML | none | - | + +Docling is the heavy comparison for when the starting set leaves a question open; it is installed +by hand into the same environment and is not part of the routine run. + +**A reference tool is never a dependency of the stack.** It lives in `harness/compare/.ref-venv`, +a throwaway environment the README builds, never in `tools/.venv` and never in +`tools/requirements.txt`. `compare.py` runs with `tools/.venv` (so it measures this checkout's +filters) and starts the reference tools as a separate process. + +## The measures + +Per file, for the stack's derivation and for each reference: + +1. **Wording fidelity.** For PDF: with all markings stripped, the derivation is the same text as + the derivation without the layout analysis (`pypdf` alone). Anything else is a defect - a + marking may never change the wording. For the other formats, which have no separate style pass: + the share of the reference's words the derivation also has (recall) and of its own words the + reference also has (precision). +2. **Markup overhead.** Markup characters per text character, in percent. For PDF exactly: what the + layout analysis adds, legend included. +3. **Precision of the markings.** A sample of the markings judged against the original - by page + image where the harness can render one (PDF). Session work: step 4. +4. **Semantics missing against a reference.** Strike-through, highlight, bold, italic; for DOCX + tracked changes and comments, for XLSX hidden sheets and formulas, for PPTX speaker notes. A kind + the reference or the oracle finds and the derivation does not is listed as missing. +5. **Structure.** Headings, outline entries (and whether each points at its page marker), tables, + lists, code blocks. +6. **CHECK precision.** The pages the image CHECK names against the pages whose images carry + content - neither a background over most of the page, nor a filler a few pixels or points in + size, nor the same image repeated as a template across the document; and the rotated pages, + whether turned by the page or by its text. +7. **Runtime.** + +**Precision before recall.** A wrong marking weighs more than a missing one: a reader trusts a +`~~` that is there, and does not miss one that never was. A change that adds correct markings at +the price of any wrong ones is not an improvement until the wrong ones are gone. + +## Steps + +1. **Have the clone and the reference environment.** The README of the private repository builds + the environment in one command; check it is there: + + ```bash + ls testfiles/harness/compare/.ref-venv/bin/python + ``` + + Without access to the private repository there is no comparison: say so in the issue body and + leave the corpus measures to a session that has it. + +2. **Run the comparison**, the whole corpus or the formats the change touches: + + ```bash + cd tools + .venv/bin/python ../testfiles/harness/compare/compare.py --render + .venv/bin/python ../testfiles/harness/compare/compare.py --format pdf --render + ``` + + The report lands in `testfiles/harness/compare/out/report.md` (gitignored there), next to + `measures.json`, every derived text under `derived/`, `samples.md` and the page images under + `pages/`. The whole corpus takes some minutes; the large books dominate. + +3. **Read the difference to the baseline first.** Before a change it must be empty - a run of the + unchanged checkout reproduces the baseline, runtime jitter aside. If it is not, the baseline is + stale or the corpus moved; find out which before measuring anything on top of it. After a change + every line is accounted for: intended, or a defect to fix in this build. + +4. **Judge the markings (measure 3).** `samples.md` lists, per file and kind of marking, lines + spread over the document with their page; the PNGs under `pages/` show those pages. For each + sample, look at the page and decide: is the marked text really struck, highlighted, bold or + italic there, and does the marking cover exactly that text? Count right and wrong per kind. A + wrong marking found this way is a defect even when every number is within its limit. + +5. **Record the findings** in the body of the issue being worked on - per format, the measures + that matter for it, the difference to the baseline, the precision judged in step 4. A finding + outside that issue's topic goes into the issue for that topic, or a new one if there is none. + +6. **Move the baseline only on purpose.** Once the change is published and the new numbers are + the accepted state, write them and commit in the private repository with the reason: + + ```bash + .venv/bin/python ../testfiles/harness/compare/compare.py --write-baseline + ``` + + `--write-baseline` together with `--only`/`--format` updates just those files' entries. The + commit message names the stack version and why the numbers moved ("baseline for 10.2.0-beta.3: + strike-through only where the line hugs the text"). A baseline that is rewritten to make a + difference disappear has stopped measuring anything. + +## Decision points + +- **A threshold in a converter looks wrong?** Change it only with a corpus run before and after; + a threshold tuned on one document moves the others, and only the full run shows by how much. +- **A reference tool does something better?** That is a finding, not an instruction to adopt the + tool: the stack takes no runtime dependency for it without a design decision. Record what it does + better and on which files. +- **A reference tool fails or disagrees with its oracle?** Record it and carry on - the reference + is a yardstick, not the truth. The oracle and the page image decide. +- **A measure itself is wrong?** Fix it in `harness/compare/metrics.py` with its test, rerun, and + rewrite the baseline in the same private-repository commit, naming the measure. + +## Scope + +For the converters behind `raw derive` (`tools/chemenu/filters/`) and only in this origin +repository with the private clone present. Whether a derivation succeeds and stays stable is the +real-document test beside it in `harness/`, run by the `converters` workflow; the synthetic filter +tests in `tools/chemenu/tests/` are [testing-conventions.md](testing-conventions.md). What of the +corpus may be copied out of the private repository - into an issue, too - is +[dev-setup.md](dev-setup.md) step 6; measures are numbers about a file, a sample line is its text. diff --git a/instructions/dev/stack-mode.md b/instructions/dev/stack-mode.md index 28c0604..cc0f101 100644 --- a/instructions/dev/stack-mode.md +++ b/instructions/dev/stack-mode.md @@ -88,6 +88,10 @@ slash command for the operator, and stops. a tracker of your own, the nightly workflow (which you dispatch yourself after touching the Super Productivity surface), what a red night means, and refreshing the recorded fixtures. Read it before changing an adapter under `tools/chemenu/tasks/`. +- [converter-comparison.md](converter-comparison.md) - whether a `raw derive` conversion is + *good*, not merely working: the converters against reference tools over the private corpus, + seven measures, markings judged against the original, a baseline moved only on purpose. Read it + before designing a converter change, after building one, and when the corpus gains a file. - [doc-pull-through.md](doc-pull-through.md) - which document makes a claim about a touched surface (a `wikitool` command, a stage's rules, an `AGENTS.md` rule/gate/invariant, a README-shaped human doc, a `docs/` page's reasoning) and therefore needs updating alongside diff --git a/instructions/dev/testing-conventions.md b/instructions/dev/testing-conventions.md index 5189e9f..294d82a 100644 --- a/instructions/dev/testing-conventions.md +++ b/instructions/dev/testing-conventions.md @@ -193,7 +193,9 @@ cloned into the gitignored `testfiles/` ([dev-setup.md](dev-setup.md) step 6) an checkout's `tools/.venv` by the `converters` workflow. A new file for the corpus is a case for that harness, not a fixture here; the synthetic filter tests (`test_raw_derive.py`, `test_raw_derive_binary.py`) stay and are what the suite relies on. The harness borrows this -suite's `conftest.py`, so a change to a fixture there reaches it too. +suite's `conftest.py`, so a change to a fixture there reaches it too. Whether a derivation is +*good* - right markings, no noise, nothing a reference tool reads better - is no test at all but +a measured comparison: [converter-comparison.md](converter-comparison.md). This file says nothing about what to test - the test/review expectations for a stack change are the local checks in [publish-and-ci.md](publish-and-ci.md)