chemenu/instructions/dev/converter-comparison.md

type, name, description

type	name	description
types/instruction.md	converter-comparison	How to measure whether raw derive's conversion of a non-text source (PDF, DOCX, PPTX, XLSX, mail, HTML) is good rather than merely working - running the stack's own derivation against reference tools over the private corpus, the seven measures, judging markings against the original, and when the baseline may move.

Compare the converters against reference tools on the real corpus

The real-document tests in the private repository check that a derivation succeeds, is stable and contains the text it should. They do not check whether it is good: whether a marking is right, whether the converter adds noise, whether an available tool reads the format better. That takes a judgment against the original, and only part of it is a number. Without this step, a PDF marking change once shipped to a candidate before anyone saw that it added a quarter to a guide's length and struck through 190 single letters along the grid lines of rotated table pages.

The comparison runs in the private repository chemenu-testfiles (cloned into testfiles/, dev-setup.md step 6): harness/compare/compare.py derives every corpus file with this checkout's filters and with the reference tools, computes the measures, and reports the difference to harness/baseline.json. harness/compare/README.md has the setup and every option; this file says when to run it and what to do with the result.

Contents

• When to run
• What is compared
• The measures
• Steps
• Decision points
• Scope

When to run

• Before designing a converter change - measure where it starts. The numbers go into the design, and a difference to the baseline found here is explained before anything is built on it.
• After building it, before the publish - measure what it changed. Every line of the difference is either intended or a defect.
• When a file is added to the corpus - it gets its first measures and its baseline entry.
• When a format shows a new effect - colour as decoration, shaded table cells, slide panels, syntax colouring in code: measure it before deciding whether the converter should react.

A change elsewhere in the stack does not need it; the converters workflow already proves that the corpus still derives (publish-and-ci.md step 3).

What is compared

Every format with a filter: PDF, DOCX, PPTX, XLSX/XLSM, EML/MSG and HTML. A ZIP is only a container

• its members are corpus files of their own - and is skipped.

Reference tools, by role. The starting set is below; the versions are pinned in the harness (harness/compare/requirements-reference.txt), not here.

Format	Reference	Oracle (read from the file directly)
PDF	pymupdf4llm	MuPDF style flags (TEXT_COLLECT_STYLES): strike-out, bold, italic, image and rotated pages
DOCX	markitdown, pandoc	the document XML: tracked insertions and deletions, comments, run formatting
PPTX	markitdown	python-pptx: slides and speaker notes
XLSX/XLSM	markitdown	openpyxl: sheets, hidden sheets, formula cells
MSG	markitdown	-
HTML	markitdown, pandoc	-
EML	none	-

Docling is the heavy comparison for when the starting set leaves a question open; it is installed by hand into the same environment and is not part of the routine run.

A reference tool is never a dependency of the stack. It lives in harness/compare/.ref-venv, a throwaway environment that harness/compare/README.md builds, never in tools/.venv and never in tools/requirements.txt. compare.py runs with tools/.venv (so it measures this checkout's filters) and starts the reference tools as a separate process.

The measures

Per file, for the stack's derivation and for each reference:

1. Wording fidelity. For PDF: with all markings stripped, the derivation is the same text as the derivation without the layout analysis (pypdf alone). Anything else is a defect - a marking may never change the wording. For the other formats, which have no separate style pass: the share of the reference's words the derivation also has (recall) and of its own words the reference also has (precision).
2. Markup overhead. Markup characters per text character, in percent. For PDF exactly: what the layout analysis adds, legend included.
3. Precision of the markings. A sample of the markings judged against the original - by page image where the harness can render one (PDF). Session work: step 4.
4. Semantics missing against a reference. Strike-through, highlight, bold, italic; for DOCX tracked changes and comments, for XLSX hidden sheets and formulas, for PPTX speaker notes. A kind the reference or the oracle finds and the derivation does not is listed as missing.
5. Structure. Headings, outline entries (and whether each points at its page marker), tables, lists, code blocks.
6. CHECK precision. The pages the image CHECK names against the pages whose images carry content - neither a background over most of the page, nor a filler a few pixels or points in size, nor the same image repeated as a template across the document; and the rotated pages, whether turned by the page or by its text.
7. Runtime.

Precision before recall. A wrong marking weighs more than a missing one: a reader trusts a ~~ that is there, and does not miss one that never was. A change that adds correct markings at the price of any wrong ones is not an improvement until the wrong ones are gone.

Steps

1. Have the clone and the reference environment. harness/compare/README.md builds the environment in one command; check it is there:

   ls testfiles/harness/compare/.ref-venv/bin/python
   

   Without access to the private repository there is no comparison: say so in the issue body and leave the corpus measures to a session that has it.

2. Run the comparison, the whole corpus or the formats the change touches:

   cd tools
   .venv/bin/python ../testfiles/harness/compare/compare.py --render
   .venv/bin/python ../testfiles/harness/compare/compare.py --format pdf --render
   

   The report lands in testfiles/harness/compare/out/report.md (gitignored there), next to measures.json, every derived text under derived/, samples.md and the page images under pages/. The whole corpus takes some minutes; the large books dominate.

3. Read the difference to the baseline first. Before a change it must be empty - a run of the unchanged checkout reproduces the baseline, runtime jitter aside. If it is not, the baseline is stale or the corpus moved; find out which before measuring anything on top of it. After a change every line is accounted for: intended, or a defect to fix in this build.

4. Judge the markings (measure 3). samples.md lists, per file and kind of marking, lines spread over the document with their page; the PNGs under pages/ show those pages. For each sample, look at the page and decide: is the marked text really struck, highlighted, bold or italic there, and does the marking cover exactly that text? Count right and wrong per kind. A wrong marking found this way is a defect even when every number is within its limit.

5. Record the findings in the body of the issue being worked on - per format, the measures that matter for it, the difference to the baseline, the precision judged in step 4. A finding outside that issue's topic goes into the issue for that topic, or a new one if there is none.

6. Move the baseline only on purpose. Once the change is published and the new numbers are the accepted state, write them and commit in the private repository with the reason:

   .venv/bin/python ../testfiles/harness/compare/compare.py --write-baseline
   

   --write-baseline together with --only/--format updates just those files' entries. The commit message names the stack version and why the numbers moved ("baseline for 10.2.0-beta.3: strike-through only where the line hugs the text"). A baseline that is rewritten to make a difference disappear has stopped measuring anything.

Decision points

• A threshold in a converter looks wrong? Change it only with a corpus run before and after; a threshold tuned on one document moves the others, and only the full run shows by how much.
• A reference tool does something better? That is a finding, not an instruction to adopt the tool: the stack takes no runtime dependency for it without a design decision. Record what it does better and on which files.
• A reference tool fails or disagrees with its oracle? Record it and carry on - the reference is a yardstick, not the truth. The oracle and the page image decide.
• A measure itself is wrong? Fix it in harness/compare/metrics.py with its test, rerun, and rewrite the baseline in the same private-repository commit, naming the measure.

Scope

For the converters behind raw derive (tools/chemenu/filters/) and only in this origin repository with the private clone present. Whether a derivation succeeds and stays stable is the real-document test beside it in harness/, run by the converters workflow; the synthetic filter tests in tools/chemenu/tests/ are testing-conventions.md. What of the corpus may be copied out of the private repository - into an issue, too - is dev-setup.md step 6; measures are numbers about a file, a sample line is its text.