instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, sieben Maße, Grundlinie nur bewusst fortgeschrieben (#233); 10.1.0-beta.3

Files changed:
- CHANGES.md
- VERSION
- docs/dev/development.md
- instructions/dev/converter-comparison.md
- instructions/dev/stack-mode.md
- instructions/dev/testing-conventions.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

instructions/dev/converter-comparison.md

@@ -0,0 +1,169 @@
+---
+type: types/instruction.md
+name: converter-comparison
+description: How to measure whether raw derive's conversion of a non-text source (PDF, DOCX, PPTX, XLSX, mail, HTML) is good rather than merely working - running the stack's own derivation against reference tools over the private corpus, the seven measures, judging markings against the original, and when the baseline may move.
+---
+# Compare the converters against reference tools on the real corpus
+
+The real-document tests in the private repository check that a derivation succeeds, is stable and
+contains the text it should. They do not check whether it is **good**: whether a marking is right,
+whether the converter adds noise, whether an available tool reads the format better. That takes a
+judgment against the original, and only part of it is a number. Without this step, a PDF marking
+change once shipped to a candidate before anyone saw that it added a quarter to a guide's length
+and struck through 190 single letters along the grid lines of rotated table pages.
+
+The comparison runs in the private repository `chemenu-testfiles` (cloned into `testfiles/`,
+[dev-setup.md](dev-setup.md) step 6): `harness/compare/compare.py` derives every corpus file with
+this checkout's filters and with the reference tools, computes the measures, and reports the
+difference to `harness/baseline.json`. Its README has the setup and every option; this file says
+when to run it and what to do with the result.
+
+<!-- wikitool:toc -->
+## Contents
+
+- [When to run](#when-to-run)
+- [What is compared](#what-is-compared)
+- [The measures](#the-measures)
+- [Steps](#steps)
+- [Decision points](#decision-points)
+- [Scope](#scope)
+<!-- /wikitool:toc -->
+
+## When to run
+
+- **Before designing a converter change** - measure where it starts. The numbers go into the
+  design, and a difference to the baseline found here is explained before anything is built on it.
+- **After building it, before the publish** - measure what it changed. Every line of the
+  difference is either intended or a defect.
+- **When a file is added to the corpus** - it gets its first measures and its baseline entry.
+- **When a format shows a new effect** - colour as decoration, shaded table cells, slide panels,
+  syntax colouring in code: measure it before deciding whether the converter should react.
+
+A change elsewhere in the stack does not need it; the `converters` workflow already proves that the
+corpus still derives ([publish-and-ci.md](publish-and-ci.md) step 3).
+
+## What is compared
+
+Every format with a filter: PDF, DOCX, PPTX, XLSX/XLSM, EML/MSG and HTML. A ZIP is only a container
+- its members are corpus files of their own - and is skipped.
+
+**Reference tools, by role.** The starting set is below; the versions are pinned in the harness
+(`harness/compare/requirements-reference.txt`), not here.
+
+| Format | Reference | Oracle (read from the file directly) |
+|---|---|---|
+| PDF | pymupdf4llm | MuPDF style flags (`TEXT_COLLECT_STYLES`): strike-out, bold, italic, image and rotated pages |
+| DOCX | markitdown, pandoc | the document XML: tracked insertions and deletions, comments, run formatting |
+| PPTX | markitdown | python-pptx: slides and speaker notes |
+| XLSX/XLSM | markitdown | openpyxl: sheets, hidden sheets, formula cells |
+| MSG | markitdown | - |
+| HTML | markitdown, pandoc | - |
+| EML | none | - |
+
+Docling is the heavy comparison for when the starting set leaves a question open; it is installed
+by hand into the same environment and is not part of the routine run.
+
+**A reference tool is never a dependency of the stack.** It lives in `harness/compare/.ref-venv`,
+a throwaway environment the README builds, never in `tools/.venv` and never in
+`tools/requirements.txt`. `compare.py` runs with `tools/.venv` (so it measures this checkout's
+filters) and starts the reference tools as a separate process.
+
+## The measures
+
+Per file, for the stack's derivation and for each reference:
+
+1. **Wording fidelity.** For PDF: with all markings stripped, the derivation is the same text as
+   the derivation without the layout analysis (`pypdf` alone). Anything else is a defect - a
+   marking may never change the wording. For the other formats, which have no separate style pass:
+   the share of the reference's words the derivation also has (recall) and of its own words the
+   reference also has (precision).
+2. **Markup overhead.** Markup characters per text character, in percent. For PDF exactly: what the
+   layout analysis adds, legend included.
+3. **Precision of the markings.** A sample of the markings judged against the original - by page
+   image where the harness can render one (PDF). Session work: step 4.
+4. **Semantics missing against a reference.** Strike-through, highlight, bold, italic; for DOCX
+   tracked changes and comments, for XLSX hidden sheets and formulas, for PPTX speaker notes. A kind
+   the reference or the oracle finds and the derivation does not is listed as missing.
+5. **Structure.** Headings, outline entries (and whether each points at its page marker), tables,
+   lists, code blocks.
+6. **CHECK precision.** The pages the image CHECK names against the pages whose images carry
+   content - neither a background over most of the page, nor a filler a few pixels or points in
+   size, nor the same image repeated as a template across the document; and the rotated pages,
+   whether turned by the page or by its text.
+7. **Runtime.**
+
+**Precision before recall.** A wrong marking weighs more than a missing one: a reader trusts a
+`~~` that is there, and does not miss one that never was. A change that adds correct markings at
+the price of any wrong ones is not an improvement until the wrong ones are gone.
+
+## Steps
+
+1. **Have the clone and the reference environment.** The README of the private repository builds
+   the environment in one command; check it is there:
+
+   ```bash
+   ls testfiles/harness/compare/.ref-venv/bin/python
+   ```
+
+   Without access to the private repository there is no comparison: say so in the issue body and
+   leave the corpus measures to a session that has it.
+
+2. **Run the comparison**, the whole corpus or the formats the change touches:
+
+   ```bash
+   cd tools
+   .venv/bin/python ../testfiles/harness/compare/compare.py --render
+   .venv/bin/python ../testfiles/harness/compare/compare.py --format pdf --render
+   ```
+
+   The report lands in `testfiles/harness/compare/out/report.md` (gitignored there), next to
+   `measures.json`, every derived text under `derived/`, `samples.md` and the page images under
+   `pages/`. The whole corpus takes some minutes; the large books dominate.
+
+3. **Read the difference to the baseline first.** Before a change it must be empty - a run of the
+   unchanged checkout reproduces the baseline, runtime jitter aside. If it is not, the baseline is
+   stale or the corpus moved; find out which before measuring anything on top of it. After a change
+   every line is accounted for: intended, or a defect to fix in this build.
+
+4. **Judge the markings (measure 3).** `samples.md` lists, per file and kind of marking, lines
+   spread over the document with their page; the PNGs under `pages/` show those pages. For each
+   sample, look at the page and decide: is the marked text really struck, highlighted, bold or
+   italic there, and does the marking cover exactly that text? Count right and wrong per kind. A
+   wrong marking found this way is a defect even when every number is within its limit.
+
+5. **Record the findings** in the body of the issue being worked on - per format, the measures
+   that matter for it, the difference to the baseline, the precision judged in step 4. A finding
+   outside that issue's topic goes into the issue for that topic, or a new one if there is none.
+
+6. **Move the baseline only on purpose.** Once the change is published and the new numbers are
+   the accepted state, write them and commit in the private repository with the reason:
+
+   ```bash
+   .venv/bin/python ../testfiles/harness/compare/compare.py --write-baseline
+   ```
+
+   `--write-baseline` together with `--only`/`--format` updates just those files' entries. The
+   commit message names the stack version and why the numbers moved ("baseline for 10.2.0-beta.3:
+   strike-through only where the line hugs the text"). A baseline that is rewritten to make a
+   difference disappear has stopped measuring anything.
+
+## Decision points
+
+- **A threshold in a converter looks wrong?** Change it only with a corpus run before and after;
+  a threshold tuned on one document moves the others, and only the full run shows by how much.
+- **A reference tool does something better?** That is a finding, not an instruction to adopt the
+  tool: the stack takes no runtime dependency for it without a design decision. Record what it does
+  better and on which files.
+- **A reference tool fails or disagrees with its oracle?** Record it and carry on - the reference
+  is a yardstick, not the truth. The oracle and the page image decide.
+- **A measure itself is wrong?** Fix it in `harness/compare/metrics.py` with its test, rerun, and
+  rewrite the baseline in the same private-repository commit, naming the measure.
+
+## Scope
+
+For the converters behind `raw derive` (`tools/chemenu/filters/`) and only in this origin
+repository with the private clone present. Whether a derivation succeeds and stays stable is the
+real-document test beside it in `harness/`, run by the `converters` workflow; the synthetic filter
+tests in `tools/chemenu/tests/` are [testing-conventions.md](testing-conventions.md). What of the
+corpus may be copied out of the private repository - into an issue, too - is
+[dev-setup.md](dev-setup.md) step 6; measures are numbers about a file, a sample line is its text.

instructions/dev/stack-mode.md

@@ -88,6 +88,10 @@ slash command for the operator, and stops.
   a tracker of your own, the nightly workflow (which you dispatch yourself after touching the
   Super Productivity surface), what a red night means, and refreshing the recorded fixtures.
   Read it before changing an adapter under `tools/chemenu/tasks/`.
+- [converter-comparison.md](converter-comparison.md) - whether a `raw derive` conversion is
+  *good*, not merely working: the converters against reference tools over the private corpus,
+  seven measures, markings judged against the original, a baseline moved only on purpose. Read it
+  before designing a converter change, after building one, and when the corpus gains a file.
 - [doc-pull-through.md](doc-pull-through.md) - which document makes a claim about a touched
   surface (a `wikitool` command, a stage's rules, an `AGENTS.md` rule/gate/invariant, a
   README-shaped human doc, a `docs/` page's reasoning) and therefore needs updating alongside

instructions/dev/testing-conventions.md

@@ -193,7 +193,9 @@ cloned into the gitignored `testfiles/` ([dev-setup.md](dev-setup.md) step 6) an
 checkout's `tools/.venv` by the `converters` workflow. A new file for the corpus is a case for
 that harness, not a fixture here; the synthetic filter tests (`test_raw_derive.py`,
 `test_raw_derive_binary.py`) stay and are what the suite relies on. The harness borrows this
-suite's `conftest.py`, so a change to a fixture there reaches it too.
+suite's `conftest.py`, so a change to a fixture there reaches it too. Whether a derivation is
+*good* - right markings, no noise, nothing a reference tool reads better - is no test at all but
+a measured comparison: [converter-comparison.md](converter-comparison.md).
 
 This file says nothing about what to test - the test/review
 expectations for a stack change are the local checks in [publish-and-ci.md](publish-and-ci.md)