---
type: types/instruction.md
name: converter-comparison
description: How to measure whether raw derive's conversion of a non-text source (PDF, DOCX, PPTX, XLSX, mail, HTML) is good rather than merely working - running the stack's own derivation against reference tools over the private corpus, the seven measures, judging markings against the original, and when the baseline may move.
---
# Compare the converters against reference tools on the real corpus

The real-document tests in the private repository check that a derivation succeeds, is stable and
contains the text it should. They do not check whether it is **good**: whether a marking is right,
whether the converter adds noise, whether an available tool reads the format better. That takes a
judgment against the original, and only part of it is a number. Without this step, a PDF marking
change once shipped to a candidate before anyone saw that it added a quarter to a guide's length
and struck through 190 single letters along the grid lines of rotated table pages.

The comparison runs in the private repository `chemenu-testfiles` (cloned into `testfiles/`,
[dev-setup.md](dev-setup.md) step 6): `harness/compare/compare.py` derives every corpus file with
this checkout's filters and with the reference tools, computes the measures, and reports the
difference to `harness/baseline.json`. `harness/compare/README.md` has the setup and every option; this file says
when to run it and what to do with the result.

<!-- wikitool:toc -->
## Contents

- [When to run](#when-to-run)
- [What is compared](#what-is-compared)
- [The measures](#the-measures)
- [Steps](#steps)
- [Decision points](#decision-points)
- [Scope](#scope)
<!-- /wikitool:toc -->

## When to run

- **Before designing a converter change** - measure where it starts. The numbers go into the
  design, and a difference to the baseline found here is explained before anything is built on it.
- **After building it, before the publish** - measure what it changed. Every line of the
  difference is either intended or a defect.
- **When a file is added to the corpus** - it gets its first measures and its baseline entry.
- **When a format shows a new effect** - colour as decoration, shaded table cells, slide panels,
  syntax colouring in code: measure it before deciding whether the converter should react.

A change elsewhere in the stack does not need it; the `converters` workflow already proves that the
corpus still derives ([publish-and-ci.md](publish-and-ci.md) step 3).

## What is compared

Every format with a filter: PDF, DOCX, PPTX, XLSX/XLSM, EML/MSG and HTML. A ZIP is only a container
- its members are corpus files of their own - and is skipped.

**Reference tools, by role.** The starting set is below; the versions are pinned in the harness
(`harness/compare/requirements-reference.txt`), not here.

| Format | Reference | Oracle (read from the file directly) |
|---|---|---|
| PDF | pymupdf4llm | MuPDF style flags (`TEXT_COLLECT_STYLES`): strike-out, bold, italic, image and rotated pages |
| DOCX | markitdown, pandoc | the document XML: tracked insertions and deletions, comments, run formatting |
| PPTX | markitdown | python-pptx: slides and speaker notes |
| XLSX/XLSM | markitdown | openpyxl: sheets, hidden sheets, formula cells |
| MSG | markitdown | - |
| HTML | markitdown, pandoc | - |
| EML | none | - |

Docling is the heavy comparison for when the starting set leaves a question open; it is installed
by hand into the same environment and is not part of the routine run.

**A reference tool is never a dependency of the stack.** It lives in `harness/compare/.ref-venv`,
a throwaway environment that `harness/compare/README.md` builds, never in `tools/.venv` and never in
`tools/requirements.txt`. `compare.py` runs with `tools/.venv` (so it measures this checkout's
filters) and starts the reference tools as a separate process.

## The measures

Per file, for the stack's derivation and for each reference:

1. **Wording fidelity.** For PDF: with all markings stripped, the derivation is the same text as
   the derivation without the layout analysis (`pypdf` alone). Anything else is a defect - a
   marking may never change the wording. For the other formats, which have no separate style pass:
   the share of the reference's words the derivation also has (recall) and of its own words the
   reference also has (precision).
2. **Markup overhead.** Markup characters per text character, in percent. For PDF exactly: what the
   layout analysis adds, legend included.
3. **Precision of the markings.** A sample of the markings judged against the original - by page
   image where the harness can render one (PDF). Session work: step 4.
4. **Semantics missing against a reference.** Strike-through, highlight, bold, italic; for DOCX
   tracked changes and comments, for XLSX hidden sheets and formulas, for PPTX speaker notes. A kind
   the reference or the oracle finds and the derivation does not is listed as missing.
5. **Structure.** Headings, outline entries (and whether each points at its page marker), tables,
   lists, code blocks.
6. **CHECK precision.** The pages the image CHECK names against the pages whose images carry
   content - neither a background over most of the page, nor a filler a few pixels or points in
   size, nor the same image repeated as a template across the document; and the rotated pages,
   whether turned by the page or by its text.
7. **Runtime.**

**Precision before recall.** A wrong marking weighs more than a missing one: a reader trusts a
`~~` that is there, and does not miss one that never was. A change that adds correct markings at
the price of any wrong ones is not an improvement until the wrong ones are gone.

## Steps

1. **Have the clone and the reference environment.** `harness/compare/README.md` builds the
   environment in one command; check it is there:

   ```bash
   ls testfiles/harness/compare/.ref-venv/bin/python
   ```

   Without access to the private repository there is no comparison: say so in the issue body and
   leave the corpus measures to a session that has it.

2. **Run the comparison**, the whole corpus or the formats the change touches:

   ```bash
   cd tools
   .venv/bin/python ../testfiles/harness/compare/compare.py --render
   .venv/bin/python ../testfiles/harness/compare/compare.py --format pdf --render
   ```

   The report lands in `testfiles/harness/compare/out/report.md` (gitignored there), next to
   `measures.json`, every derived text under `derived/`, `samples.md` and the page images under
   `pages/`. The whole corpus takes some minutes; the large books dominate.

3. **Read the difference to the baseline first.** Before a change it must be empty - a run of the
   unchanged checkout reproduces the baseline, runtime jitter aside. If it is not, the baseline is
   stale or the corpus moved; find out which before measuring anything on top of it. After a change
   every line is accounted for: intended, or a defect to fix in this build.

4. **Judge the markings (measure 3).** `samples.md` lists, per file and kind of marking, lines
   spread over the document with their page; the PNGs under `pages/` show those pages. For each
   sample, look at the page and decide: is the marked text really struck, highlighted, bold or
   italic there, and does the marking cover exactly that text? Count right and wrong per kind. A
   wrong marking found this way is a defect even when every number is within its limit.

5. **Record the findings** in the body of the issue being worked on - per format, the measures
   that matter for it, the difference to the baseline, the precision judged in step 4. A finding
   outside that issue's topic goes into the issue for that topic, or a new one if there is none.

6. **Move the baseline only on purpose.** Once the change is published and the new numbers are
   the accepted state, write them and commit in the private repository with the reason:

   ```bash
   .venv/bin/python ../testfiles/harness/compare/compare.py --write-baseline
   ```

   `--write-baseline` together with `--only`/`--format` updates just those files' entries. The
   commit message names the stack version and why the numbers moved ("baseline for 10.2.0-beta.3:
   strike-through only where the line hugs the text"). A baseline that is rewritten to make a
   difference disappear has stopped measuring anything.

## Decision points

- **A threshold in a converter looks wrong?** Change it only with a corpus run before and after;
  a threshold tuned on one document moves the others, and only the full run shows by how much.
- **A reference tool does something better?** That is a finding, not an instruction to adopt the
  tool: the stack takes no runtime dependency for it without a design decision. Record what it does
  better and on which files.
- **A reference tool fails or disagrees with its oracle?** Record it and carry on - the reference
  is a yardstick, not the truth. The oracle and the page image decide.
- **A measure itself is wrong?** Fix it in `harness/compare/metrics.py` with its test, rerun, and
  rewrite the baseline in the same private-repository commit, naming the measure.

## Scope

For the converters behind `raw derive` (`tools/chemenu/filters/`) and only in this origin
repository with the private clone present. Whether a derivation succeeds and stays stable is the
real-document test beside it in `harness/`, run by the `converters` workflow; the synthetic filter
tests in `tools/chemenu/tests/` are [testing-conventions.md](testing-conventions.md). What of the
corpus may be copied out of the private repository - into an issue, too - is
[dev-setup.md](dev-setup.md) step 6; measures are numbers about a file, a sample line is its text.