feat(raw): raw derive writes an Excel file's sheets as TSV files with a formula list (#202)

Each sheet becomes NN-<sheet>.tsv (RFC 4180, tab-delimited, A1-aligned) in a
<stem>.sheets/ folder beside the .md; sheets with formulas also get a
.formulas.tsv. derived.md is the overview. Adds testfiles/ with a CI test.

Files changed:
- .gitattributes
- CHANGES.md
- VERSION
- docs/dev/development.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- raw/CONTRACT.md
- testfiles/README.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/filters/__init__.py
- tools/chemenu/filters/archive.py
- tools/chemenu/filters/ooxml.py
- tools/chemenu/tests/test_raw_derive.py
- tools/chemenu/tests/test_raw_derive_testfiles.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

.gitattributes

@@ -11,3 +11,7 @@
 # and normalizing a CRLF source on `git add` would change it.
 /raw/** -text
 /incoming/** -text
+
+# Real documents for the `raw derive` tests are compared byte for byte (an .eml with CRLF
+# included), so nothing under testfiles/ is ever normalized.
+/testfiles/** -text

CHANGES.md

@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 
 ---
 
-## 9.0.0-beta.17 - 2026-10-08 - raw pending names only the raw fetch pair as a bundle; a derived document is a folder
+## 9.0.0-beta.18 - 2026-10-09 - raw derive writes an Excel file's sheets as tab-separated files with a formula list; derived.md becomes the overview (#202)
 
 **Author:** Torben Nehmer
 
@@ -47,6 +47,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
 - tools/README.md: Asset-Modus beschreibt Entpacken in den eigenen Ordner, Wiederholung und Zurückrollen
 - Gitea-Wiki als generierter Spiegel von docs/: Konverter, Push-Skript, Workflows
 - raw pending names only the raw fetch pair as a bundle; a derived document is a folder
+- raw derive writes an Excel file's sheets as tab-separated files with a formula list; derived.md becomes the overview (#202)
 <!-- /wikitool:bumps -->
 
 ### Tutorials: installation in English, and a first source to a first answer (#191)

VERSION

@@ -1 +1 @@
-9.0.0-beta.17
+9.0.0-beta.18

docs/dev/development.md

@@ -40,6 +40,9 @@ Then have the agent run `instructions/bootstrap.md`: the preflight, then
 and how `dist export` runs as a test tool - is
 [instructions/dev/dev-setup.md](../../instructions/dev/dev-setup.md).
 
+Real documents for `raw derive` live in [testfiles/](../../testfiles/README.md) and are
+committed, so CI derives them too. The repository is public: that README says what may go in.
+
 What a clone has and an instance does not:
 
 - **The demo corpus.** Some 170 pages documenting the stack itself: gates, lint, versioning,

docs/how-to/ingest-a-source.md

@@ -20,7 +20,8 @@ pick a directory:
   Outlook `.msg`) or a ZIP archive - is
   put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
   untouched original and a readable text beside it, with a header saying how it was made. The
-  folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is
+  folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's
+  sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is
   unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
   text to derive - the tool warns, and there is no OCR.
 - **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The

docs/reference/commands/raw-material-and-uploads.md

@@ -119,6 +119,7 @@ tools/wikitool raw derive incoming/scans.zip
 - A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
 - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
 - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read.
+- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
 - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
 - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
 - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
@@ -136,7 +137,7 @@ tools/wikitool raw derive incoming/scans.zip
 
 - `0` - success
 - `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension
-- `1` - The target folder or a `.md` already exists, or the path would be over the path budget
+- `1` - The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget
 - `1` - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed
 
 **See also**

raw/CONTRACT.md

@@ -289,6 +289,17 @@ output. A name that is taken or reserved (`original.*` and `derived.md` at the t
 folder, `_capture.json` anywhere) gets `-2`, `-3` before its extension, and the output lists
 every rename.
 
+**A workbook's cells are files, not text.** An Excel file becomes an overview in `derived.md` -
+one section per sheet with its range, its file names and a short preview - and, beside it, a
+folder `<stem>.sheets/` (`derived.sheets/` for a single file) with one tab-separated file per
+sheet. The format is RFC 4180 with a tab as the delimiter, so a cell holding a tab, a quote or a
+line break is quoted and a reader gets every cell back unchanged; row *n* of the file is row *n*
+of the sheet, so a cell address in the file is the address in Excel. A sheet with formulas gets
+a second file listing each cell, its formula and the value Excel last stored: the tool reads
+the file and does not recalculate it. Text that is data is read by a script, not by a model's
+eye; the overview is what tells the reader which file to open. The folder is part of the
+derivation, accepted with it, and an existing one is never overwritten.
+
 **The original stays.** The derivation is the tool's reading of it, never the source: with the
 original kept, a claim in `kb/` can still be checked against what the document said when a
 derivation dropped something, and a later version of the tool can derive it better. Both go into

testfiles/README.md

@@ -0,0 +1,47 @@
+# testfiles
+
+Real documents (PDF, mail, Office, ZIP) that CI feeds to `raw derive`, beyond the synthetic
+fixtures in `tools/chemenu/tests/binary_formats.py`. `tools/chemenu/tests/test_raw_derive_testfiles.py`
+picks up every file under this directory, so adding a file is adding a test case - no code change.
+The directory exists only in a development clone: `dist export` does not ship it.
+
+## What goes in
+
+**Everything here is committed to a public repository.** A file belongs here only when it can be
+public:
+
+- a document you wrote for the purpose, or one whose licence allows redistribution;
+- no real people, customers, employers, addresses or internal URLs - in the text *and* in the
+  metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
+- nothing that was received in confidence.
+
+Removing a file later does not remove it from the history.
+
+## Layout
+
+| Folder | For |
+|--------|-----|
+| `pdf/` | PDFs, including scanned ones without a text layer |
+| `mail/` | `.eml` and `.msg`, with and without attachments, with attached mails |
+| `office/` | `.docx`, `.pptx`, `.xlsx` (tracked changes, comments, hidden sheets) |
+| `zip/` | archives, including mail-in-ZIP cases |
+
+The folders are a convention; the test reads every file below this directory. Keep file names
+short and free of spaces - a name that is too long for the `raw/` path budget fails the test for
+the right reason, but it is a nuisance to find.
+
+## What the test checks
+
+For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for
+byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same
+bytes gives the same text apart from `derived:`; and `raw accept` takes the folder.
+
+A file may carry a sidecar `<file name>.expect` (`offer.pdf.expect` for `offer.pdf`): one line per
+piece of text that must appear in the derived text, empty lines and lines starting with `#`
+ignored. Without one the file only has to derive, be stable and be accepted. A document that is
+meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
+
+## Trying a file by hand
+
+`raw derive` moves what it is given, so work on a copy in a throwaway instance, never on
+`incoming/` of this clone.

tools/CONTRACT.md

@@ -1495,13 +1495,13 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
 
 - 0    success
 - 1    The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension
-- 1    The target folder or a `.md` already exists, or the path would be over the path budget
+- 1    The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget
 - 1    The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed
 
 **ON FAILURE**
 
 - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -> Nothing was written. Fix the argument and retry once - a file in a subfolder is moved up, or its folder is derived whole
-- The target folder or a `.md` already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name <shorter stem>`, then retry once. An existing `.md` is never overwritten
+- The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name <shorter stem>`, then retry once. An existing `.md` is never overwritten
 - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed -> Nothing was written. A missing package is reported with the preflight hint - show it to the user and stop, never install it. Any other filter keeps working
 
 **NEVER**
@@ -1516,6 +1516,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
 - A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
 - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
 - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read.
+- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
 - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
 - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
 - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.

tools/chemenu/commands/raw_cmd.py

@@ -1616,15 +1616,23 @@ def _write_exclusive(writes: list[tuple[Path, bytes]]) -> None:
     """Create every file or none: `x` mode refuses a file that appeared since
     the check above, and a failure part-way removes what this call wrote."""
     written: list[Path] = []
+    made: list[Path] = []
     try:
         for path, data in writes:
+            missing = [d for d in reversed(path.parents) if not d.exists()]
             path.parent.mkdir(parents=True, exist_ok=True)
+            made.extend(missing)
             with path.open("xb") as handle:
                 written.append(path)
                 handle.write(data)
     except OSError as exc:
         for path in written:
             path.unlink(missing_ok=True)
+        for directory in reversed(made):
+            try:
+                directory.rmdir()
+            except OSError:
+                pass
         fail(escape(f"Could not write {rel_path(path)}: {exc}. Nothing was left in incoming/."))
 
 
@@ -1851,6 +1859,7 @@ class _Derived:
     source: Path  # the file in incoming/
     markdown: bytes
     target: Path  # where the .md goes
+    extras: tuple[tuple[Path, bytes], ...] = ()  # the derivation's further files, beside the .md
 
 
 def _derive_hint(path: Path) -> Optional[str]:
@@ -1897,6 +1906,7 @@ class _FilePlan:
     markdown: bytes
     container: Optional[filters.Container]
     derivations: list[filters.Derivation]
+    extras: tuple[tuple[str, bytes], ...] = ()  # (path in the folder, bytes) of the derivation's further files
 
 
 def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan:
@@ -1942,15 +1952,19 @@ def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan:
         markdown = b""
         targets = [rel for rel, _data in container.files]
         derivations = list(container.derivations)
+        extras = ()
     else:
         original = folder / f"{ORIGINAL_STEM}{extension}"
         derived = folder / DERIVED_NAME
         derivation = container.text if container is not None else _run_filter(candidate, data, path)
         assert derivation is not None
+        companion = filters.companion_name(DERIVED_NAME) if derivation.files else None
         markdown = filters.render(
-            candidate, derivation, derived_from=original.name, original_name=path.name
+            candidate, derivation, derived_from=original.name, original_name=path.name,
+            companion=companion,
         )
-        targets = [original.name, derived.name] + (
+        extras = tuple((f"{companion}/{name}", content) for name, content in derivation.files)
+        targets = [original.name, derived.name] + [rel for rel, _data in extras] + (
             [rel for rel, _data in container.files] if container is not None else []
         )
         derivations = [derivation] + (list(container.derivations) if container is not None else [])
@@ -1961,7 +1975,7 @@ def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan:
                 f"Cannot write {problem}. The folder name comes from the file's name: pass "
                 "--name <shorter stem>."
             )
-    return _FilePlan(folder, stem, candidate, original, derived, markdown, container, derivations)
+    return _FilePlan(folder, stem, candidate, original, derived, markdown, container, derivations, extras)
 
 
 def _write_folder(source: Path, plan: _FilePlan) -> None:
@@ -1975,6 +1989,11 @@ def _write_folder(source: Path, plan: _FilePlan) -> None:
         if plan.derived is not None:
             with plan.derived.open("xb") as handle:
                 handle.write(plan.markdown)
+        for rel, data in plan.extras:
+            target = plan.folder / rel
+            target.parent.mkdir(parents=True, exist_ok=True)
+            with target.open("xb") as handle:
+                handle.write(data)
         if plan.container is not None:
             for rel, data in plan.container.files:
                 target = plan.folder / rel
@@ -2054,8 +2073,26 @@ def _plan_derive_folder(folder: Path):
     derived: list[_Derived] = []
     for file, candidate, md in plan:
         derivation = _run_filter(candidate, _read_for_derive(file), file)
-        markdown = filters.render(candidate, derivation, derived_from=file.name)
+        companion = md.with_name(filters.companion_name(md.name))
-        derived.append(_Derived(candidate, derivation, file, markdown, md))
+        extras: tuple[tuple[Path, bytes], ...] = ()
+        if derivation.files:
+            if os.path.lexists(companion):
+                raise _Refused(
+                    f"{rel_path(companion)} already exists - raw derive never overwrites. Remove it "
+                    "or rename it, then derive the folder again."
+                )
+            extras = tuple((companion / name, content) for name, content in derivation.files)
+            for target, _content in extras:
+                problem = path_budget_problem_for(_shard_dir() / folder.name / target.relative_to(folder))
+                if problem:
+                    raise _Refused(
+                        f"Cannot write {problem}. The path comes from the folder in incoming/: "
+                        "shorten the folder's name or the names inside it."
+                    )
+        markdown = filters.render(
+            candidate, derivation, derived_from=file.name, companion=companion.name
+        )
+        derived.append(_Derived(candidate, derivation, file, markdown, md, extras))
     return derived, untouched
 
 
@@ -2141,6 +2178,15 @@ def _derive_next(target: Path) -> str:
         "`<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is "
         "no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an "
         "embedded message inside it is named, not read.",
+        "A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder "
+        "`<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` "
+        "per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet "
+        "- and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and "
+        "last stored value. The `.md` is the overview: one section per sheet with its range, "
+        "the file names and a preview. Formulas are not recalculated; a shared formula is "
+        "written out per cell, and one that cannot be is named `shared:<master cell>` with a "
+        "WARN. The folder counts against the path budget and is accepted with the rest; an "
+        "existing one is never overwritten.",
         "A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder "
         "`incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` "
         "where a filter reads it; the ZIP is removed and the output names it with its sha256. A "
@@ -2170,7 +2216,7 @@ def _derive_next(target: Path) -> str:
             "subfolder is moved up, or its folder is derived whole",
         ),
         cli_contract.Failure(
-            cause="The target folder or a `.md` already exists, or the path would be over the "
+            cause="The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the "
             "path budget",
             reaction="Nothing was written. Remove what is there or pass `--name <shorter stem>`, "
             "then retry once. An existing `.md` is never overwritten",
@@ -2214,9 +2260,11 @@ def raw_derive_command(
         if name is not None:
             fail("--name applies to a single file; a folder keeps its own name.")
         derived, untouched = _refusals_fail(_plan_derive_folder, resolved)
-        _write_exclusive([(d.target, d.markdown) for d in derived])
+        _write_exclusive([write for d in derived for write in [(d.target, d.markdown), *d.extras]])
         lines = [f"Derived {len(derived)} file(s) in {rel_path(resolved)}/:"]
-        lines += [f"  {rel_path(d.source)} -> {rel_path(d.target)} ({d.filter.name})" for d in derived]
+        for d in derived:
+            more = f" + {len(d.extras)} file(s) in {rel_path(d.extras[0][0].parent)}/" if d.extras else ""
+            lines.append(f"  {rel_path(d.source)} -> {rel_path(d.target)} ({d.filter.name}){more}")
         if untouched:
             lines.append(
                 "  No filter for, left as they are: " + ", ".join(rel_path(p) for p in untouched)
@@ -2248,6 +2296,9 @@ def raw_derive_command(
             f"Derived {rel_path(plan.derived)} from {resolved.name} ({plan.candidate.name}); the "
             f"original is now {rel_path(plan.original)}."
         ))
+    if plan.extras:
+        lines = f"  + {len(plan.extras)} file(s) beside it in {rel_path(plan.folder)}/{plan.extras[0][0].rsplit('/', 1)[0]}/"
+        typer.echo(escape(lines))
     if container is not None:
         report = _container_report(container)
         if report:

tools/chemenu/filters/__init__.py

@@ -23,6 +23,9 @@ from chemenu.version import VersionError, read_version
 
 MAX_CHECKS = 3
 
+COMPANION_SUFFIX = ".sheets"
+COMPANION_DIR = "\x00companion\x00"  # NUL cannot occur in XML text: a safe placeholder inside `Derivation.text`
+
 PREFLIGHT_HINT = (
     "Run the preflight to install it (tools/preflight.sh, or tools/preflight.ps1 on Windows; "
     "instructions/preflight.md). Every other format keeps working."
@@ -36,6 +39,11 @@ class Derivation:
     `fields` are the format-specific header lines, after `title`; a value is a
     string or a list of strings. `checks` are the short English lines `raw derive`
     prints after a success - already narrowed to the case that occurred.
+
+    `files` are further files of the derivation (a spreadsheet's sheets), as
+    `(name, bytes)`. They are written into a folder beside the `.md`, named
+    `<md stem>.sheets`; the text refers to that folder as `COMPANION_DIR`, which
+    `render` replaces with the folder's real name.
     """
 
     text: str
@@ -43,6 +51,7 @@ class Derivation:
     fields: tuple[tuple[str, "str | list[str]"], ...] = ()
     warnings: tuple[str, ...] = ()
     checks: tuple[str, ...] = ()
+    files: tuple[tuple[str, bytes], ...] = ()
 
     def __post_init__(self) -> None:
         if len(self.checks) > MAX_CHECKS:
@@ -176,9 +185,11 @@ def render(
     derived_from: str,
     original_name: Optional[str] = None,
     derived_at: Optional[str] = None,
+    companion: Optional[str] = None,
 ) -> bytes:
     """The `.md` for one document: the fixed header, then the derived text.
-    `original_name` is left out for a file that was not renamed."""
+    `original_name` is left out for a file that was not renamed. `companion` is the
+    name of the folder holding `derivation.files`."""
     fields: list[tuple[str, "str | list[str]"]] = [
         ("derived_by", "wikitool raw derive"),
         ("filter", candidate.name),
@@ -191,10 +202,20 @@ def render(
     fields.append(("title", derivation.title))
     fields.extend(derivation.fields)
     text = derivation.text.strip("\n")
+    if COMPANION_DIR in text:
+        if companion is None:
+            raise ValueError("a derivation with files needs the name of the folder they go into")
+        text = text.replace(COMPANION_DIR, companion)
     content = web_capture.header(fields) + "\n" + (text + "\n" if text else "")
     return content.encode("utf-8")
 
 
+def companion_name(md_name: str) -> str:
+    """`derived.md` -> `derived.sheets`: the folder beside a `.md` for its `files`."""
+    stem = md_name[:-3] if md_name.lower().endswith(".md") else md_name
+    return stem + COMPANION_SUFFIX
+
+
 def unique(lines: Iterable[str]) -> list[str]:
     seen: set[str] = set()
     out = []

tools/chemenu/filters/archive.py

@@ -19,6 +19,7 @@ from chemenu.filters import (
     Container,
     Derivation,
     Filter,
+    companion_name,
     filter_for,
     register,
     render,
@@ -211,10 +212,19 @@ class Collector:
                 raise ValidationError(f"{path}: {exc}") from None
             stem = path[: len(path) - len(os.path.splitext(base)[1])]
             md = self._tree.claim(stem + ".md")
+            companion = ""
+            if derivation.files:
+                directory, _, md_base = md.rpartition("/")
+                companion = self._tree.claim_folder((directory + "/" if directory else "") + companion_name(md_base))
             files.append((
                 md,
-                render(candidate, derivation, derived_from=base, original_name=came_as if came_as != base else None),
+                render(
+                    candidate, derivation, derived_from=base,
+                    original_name=came_as if came_as != base else None,
+                    companion=companion.rsplit("/", 1)[-1],
+                ),
             ))
+            files.extend((f"{companion}/{name}", content) for name, content in derivation.files)
             derived.append((path, md, candidate.name))
             derivations.append(derivation)
         checks: list[str] = []

tools/chemenu/filters/ooxml.py

@@ -12,10 +12,11 @@ import posixpath
 import re
 import xml.etree.ElementTree as ET
 import zipfile
-from typing import Iterator, Optional
+from dataclasses import dataclass, field
+from typing import Iterable, Iterator, Optional
 
 from chemenu.errors import ValidationError
-from chemenu.filters import Derivation, Filter, register
+from chemenu.filters import COMPANION_DIR, Derivation, Filter, register
 
 PART_LIMIT = 100 * 1024 * 1024
 CELL_LIMIT = 2_000_000
@@ -596,11 +597,20 @@ class _Workbook:
             self.formulas_without_value += 1
         return text
 
-    def table(self, part: str) -> str:
+    def sheet(self, part: str) -> "_Sheet":
+        """The cells and formulas of one worksheet part, read as stored."""
         cells: dict[tuple[int, int], str] = {}
+        pending: list[_PendingFormula] = []
         row_no, last_col = 0, 0
-        for _event, el in ET.iterparse(io.BytesIO(self.pkg.read(part)), events=("end",)):
+        root_seen = False
+        chart = False
+        for event, el in ET.iterparse(io.BytesIO(self.pkg.read(part)), events=("start", "end")):
             tag = _local(el.tag)
+            if event == "start":
+                if not root_seen:
+                    root_seen = True
+                    chart = tag == "chartsheet"
+                continue
             if tag == "c":
                 ref = re.fullmatch(r"([A-Za-z]+)(\d+)", el.get("r") or "")
                 if ref:
@@ -609,25 +619,226 @@ class _Workbook:
                     col, row = last_col + 1, row_no + 1
                 last_col = col
                 text = self._value(el)
-                if text.strip():
+                if text != "":
                     cells[(row, col)] = text
+                f = _child(el, "f")
+                if f is not None:
+                    pending.append(_PendingFormula(
+                        row, col, _attr(f, "t") or "normal", _attr(f, "si") or "", _attr(f, "ref") or "",
+                        (f.text or "").strip(), (_attr(f, "r1") or "", _attr(f, "r2") or ""), text,
+                    ))
                 el.clear()
             elif tag == "row":
                 row_no = int(el.get("r") or row_no + 1)
                 last_col = 0
                 el.clear()
-        if not cells:
+        if chart:
-            return "*(empty sheet)*"
+            return _Sheet(chart=True)
-        rows = [r for r, _c in cells]
+        rows = max((r for r, _c in cells), default=0)
-        cols = [c for _r, c in cells]
+        cols = max((c for _r, c in cells), default=0)
-        min_r, max_r, min_c, max_c = min(rows), max(rows), min(cols), max(cols)
+        if rows * cols > CELL_LIMIT:
-        if (max_r - min_r + 1) * (max_c - min_c + 1) > CELL_LIMIT:
             raise ValidationError(
-                f"A sheet spans {max_r - min_r + 1} rows by {max_c - min_c + 1} columns - too large "
+                f"A sheet spans {rows} rows by {cols} columns from A1 - too large or too sparse "
-                "or too sparse to derive."
+                "to derive."
             )
-        grid = [[cells.get((r, c), "") for c in range(min_c, max_c + 1)] for r in range(min_r, max_r + 1)]
+        formulas, untranslated = _resolve_formulas(pending)
-        return _table(grid)
+        return _Sheet(cells=cells, rows=rows, cols=cols, formulas=formulas, untranslated=untranslated)
+
+
+@dataclass(frozen=True)
+class _PendingFormula:
+    row: int
+    col: int
+    kind: str  # normal, shared, array, dataTable
+    si: str
+    ref: str
+    text: str
+    inputs: tuple[str, str]
+    value: str
+
+
+@dataclass(frozen=True)
+class _Formula:
+    row: int
+    col: int
+    text: str
+    value: str
+
+
+@dataclass(frozen=True)
+class _Sheet:
+    chart: bool = False
+    cells: dict = field(default_factory=dict)
+    rows: int = 0
+    cols: int = 0
+    formulas: tuple[_Formula, ...] = ()
+    untranslated: int = 0
+
+
+class _Unsafe(Exception):
+    pass
+
+
+_MAX_COL, _MAX_ROW = 16384, 1048576
+_NOT_AFTER = r"(?<![\w.$])"
+_NOT_BEFORE = r"(?![\w(!])"
+_REFERENCE = re.compile(
+    "|".join((
+        r'"(?:[^"]|"")*"',
+        r"'(?:[^']|'')*'",
+        r"\[(?:[^\[\]]|\[[^\[\]]*\])*\]",
+        _NOT_AFTER + r"(?P<cells>\$?[A-Z]{1,3}\$?\d+(?::\$?[A-Z]{1,3}\$?\d+)?)" + _NOT_BEFORE,
+        _NOT_AFTER + r"(?P<cols>\$?[A-Z]{1,3}:\$?[A-Z]{1,3})" + _NOT_BEFORE,
+        _NOT_AFTER + r"(?P<rows>\$?\d+:\$?\d+)" + _NOT_BEFORE,
+    )),
+    re.IGNORECASE,
+)
+_CELL = re.compile(r"(\$?)([A-Z]{1,3})(\$?)(\d+)", re.IGNORECASE)
+
+
+def _col_letters(index: int) -> str:
+    letters = ""
+    while index:
+        index, rest = divmod(index - 1, 26)
+        letters = chr(65 + rest) + letters
+    return letters
+
+
+def _shift_formula(text: str, dr: int, dc: int) -> str:
+    """The formula of a shared-formula cell: the master's text with every relative
+    reference moved by the cell's offset. Strings, quoted sheet names and structured
+    references are left alone; a reference that would leave the sheet raises `_Unsafe`."""
+
+    def moved(index: int, absolute: str, limit: int, offset: int) -> int:
+        if absolute:
+            return index
+        index += offset
+        if not 1 <= index <= limit:
+            raise _Unsafe
+        return index
+
+    def cell(ref: str) -> str:
+        c_abs, letters, r_abs, digits = _CELL.fullmatch(ref).groups()
+        column, row = _col_index(letters.upper()), int(digits)
+        if not (1 <= column <= _MAX_COL and 1 <= row <= _MAX_ROW):
+            return ref
+        return (f"{c_abs}{_col_letters(moved(column, c_abs, _MAX_COL, dc))}"
+                f"{r_abs}{moved(row, r_abs, _MAX_ROW, dr)}")
+
+    def column(ref: str) -> str:
+        absolute, letters = re.fullmatch(r"(\$?)([A-Za-z]+)", ref).groups()
+        return f"{absolute}{_col_letters(moved(_col_index(letters.upper()), absolute, _MAX_COL, dc))}"
+
+    def row(ref: str) -> str:
+        absolute, digits = re.fullmatch(r"(\$?)(\d+)", ref).groups()
+        return f"{absolute}{moved(int(digits), absolute, _MAX_ROW, dr)}"
+
+    def replace(match: re.Match) -> str:
+        for group, shift in (("cells", cell), ("cols", column), ("rows", row)):
+            if match.group(group):
+                return ":".join(shift(part) for part in match.group(group).split(":"))
+        return match.group(0)
+
+    return _REFERENCE.sub(replace, text)
+
+
+def _a1(row: int, col: int) -> str:
+    return f"{_col_letters(col)}{row}"
+
+
+def _resolve_formulas(pending: list[_PendingFormula]) -> tuple[tuple[_Formula, ...], int]:
+    """The formula text of every formula cell. Excel stores a shared formula once, in
+    its master cell; the others get the master's text moved to their own position.
+    Returns the formulas in sheet order and how many could not be written out."""
+    masters = {f.si: f for f in pending if f.kind == "shared" and f.text}
+    out: list[_Formula] = []
+    untranslated = 0
+    for f in sorted(pending, key=lambda item: (item.row, item.col)):
+        if f.kind == "shared" and not f.text:
+            master = masters.get(f.si)
+            try:
+                if master is None:
+                    raise _Unsafe
+                text = "=" + _shift_formula(master.text, f.row - master.row, f.col - master.col)
+            except _Unsafe:
+                text = f"shared:{_a1(master.row, master.col)}" if master else f"shared:si={f.si}"
+                untranslated += 1
+        elif f.kind == "array":
+            text = f"={f.text}" + (f" (array formula, range {f.ref})" if f.ref else "")
+        elif f.kind == "dataTable":
+            text = f"=TABLE({f.inputs[0]},{f.inputs[1]})"
+        else:
+            text = "=" + f.text
+        out.append(_Formula(f.row, f.col, text, f.value))
+    return tuple(out), untranslated
+
+
+def _tsv(rows: Iterable[list[str]]) -> bytes:
+    """RFC 4180 with a tab as the delimiter: a field holding a tab, a quote or a line
+    break is quoted and its quotes doubled. A row of one empty field is written
+    quoted, or a reader would take the line for no row at all."""
+    lines = []
+    for row in rows:
+        fields = [
+            '"' + value.replace('"', '""') + '"' if any(ch in value for ch in '\t"\r\n') else value
+            for value in row
+        ]
+        if fields == [""]:
+            fields = ['""']
+        lines.append("\t".join(fields) + "\n")
+    return "".join(lines).encode("utf-8")
+
+
+PREVIEW_ROWS = 5
+PREVIEW_COLUMNS = 20
+PREVIEW_CELL = 60
+
+
+def _slug(name: str) -> str:
+    slug = re.sub(r"\W+", "-", name.lower()).strip("-")[:32].strip("-")
+    return slug or "sheet"
+
+
+def _fence(text: str) -> str:
+    longest = max((len(run) for run in re.findall(r"`+", text)), default=0)
+    return "`" * max(3, longest + 1)
+
+
+def _sheet_section(
+    number: int, total: int, width: int, name: str, state: str, sheet: _Sheet, files: list[tuple[str, bytes]]
+) -> str:
+    lines = [f"## {name}", "", f"- Sheet {number} of {total}, {state}"]
+    if sheet.chart:
+        return "\n".join(lines + ["", "*(chart sheet, no cell data)*"])
+    if not sheet.cells:
+        return "\n".join(lines + ["", "*(empty sheet)*"])
+    grid = [[sheet.cells.get((r, c), "") for c in range(1, sheet.cols + 1)] for r in range(1, sheet.rows + 1)]
+    stem = f"{number:0{width}d}-{_slug(name)}"
+    files.append((f"{stem}.tsv", _tsv(grid)))
+    lines.append(
+        f"- Range A1:{_a1(sheet.rows, sheet.cols)} ({sheet.rows} rows x {sheet.cols} columns), "
+        + (f"{len(sheet.formulas)} formula cell(s)" if sheet.formulas else "no formulas")
+    )
+    lines.append(f"- Cells: `{COMPANION_DIR}/{stem}.tsv`")
+    if sheet.formulas:
+        rows = [["cell", "formula", "value"]] + [[_a1(f.row, f.col), f.text, f.value] for f in sheet.formulas]
+        files.append((f"{stem}.formulas.tsv", _tsv(rows)))
+        lines.append(f"- Formulas: `{COMPANION_DIR}/{stem}.formulas.tsv`")
+    cut = sheet.cols > PREVIEW_COLUMNS or any(
+        len(value) > PREVIEW_CELL for row in grid[:PREVIEW_ROWS] for value in row[:PREVIEW_COLUMNS]
+    )
+    preview = _tsv(
+        [[v if len(v) <= PREVIEW_CELL else v[: PREVIEW_CELL - 1] + "…" for v in row[:PREVIEW_COLUMNS]]
+         for row in grid[:PREVIEW_ROWS]]
+    ).decode("utf-8")
+    fence = _fence(preview)
+    label = (
+        f"Preview (all {sheet.rows} rows)" if sheet.rows <= PREVIEW_ROWS
+        else f"Preview (first {PREVIEW_ROWS} of {sheet.rows} rows)"
+    )
+    if cut:
+        label += f" (columns past {PREVIEW_COLUMNS} and text past {PREVIEW_CELL} characters cut)"
+    return "\n".join(lines + ["", f"{label}:", "", fence + "tsv", preview.rstrip("\n"), fence])
 
 
 class XlsxFilter(Filter):
@@ -640,31 +851,49 @@ class XlsxFilter(Filter):
         sheets = book.sheets()
         if not sheets:
             raise ValidationError("The XLSX file lists no sheets.")
-        sections, hidden, warnings = [], [], []
+        width = max(2, len(str(len(sheets))))
-        for sheet_name, state, part in sheets:
+        sections, files, hidden, warnings = [], [], [], []
+        formulas = untranslated = 0
+        for number, (sheet_name, state, part) in enumerate(sheets, 1):
             if state != "visible":
                 hidden.append(sheet_name)
             if not part or not pkg.has(part):
                 warnings.append(f"Sheet '{sheet_name}' has no readable worksheet part and was left out.")
                 continue
-            if _child(pkg.xml(part), "sheetData") is None and _local(pkg.xml(part).tag) == "chartsheet":
+            sheet = book.sheet(part)
-                sections.append(f"## {sheet_name}\n\n*(chart sheet, no cell data)*")
+            formulas += len(sheet.formulas)
-                continue
+            untranslated += sheet.untranslated
-            sections.append(f"## {sheet_name}\n\n{book.table(part)}")
+            sections.append(_sheet_section(number, len(sheets), width, sheet_name, state, sheet, files))
         if book.formulas_without_value:
             warnings.append(
                 f"{book.formulas_without_value} formula cell(s) have no stored value (the file was "
                 "never calculated) and are empty."
             )
+        if untranslated:
+            warnings.append(
+                f"{untranslated} shared formula cell(s) could not be written out and show "
+                "'shared:<master cell>' - the master cell holds the formula."
+            )
+        intro = []
+        if files:
+            intro.append(
+                "Each sheet's cells are in a TSV file beside this one: RFC 4180 with a tab as the "
+                "delimiter, so a cell holding a tab, a quote or a line break is quoted. Read the "
+                'files with a CSV reader (`csv.reader(f, delimiter="\\t")`, `pandas.read_csv(sep="\\t")`), '
+                "not with `cut` or `awk`. Row n of a file is row n of the sheet, column k its k-th column."
+            )
         checks = ["Values are as last saved; formulas were not recalculated."]
+        if formulas:
+            checks.append("Each sheet with formulas has a .formulas.tsv beside its cells: cell, formula, stored value.")
         if hidden:
             checks.append("Hidden sheets are included and named in hidden_sheets.")
         return Derivation(
-            text="\n\n".join(sections),
+            text="\n\n".join(intro + sections),
             title=pkg.title(),
-            fields=(("sheets", str(len(sheets))), ("hidden_sheets", hidden)),
+            fields=(("sheets", str(len(sheets))), ("hidden_sheets", hidden), ("formulas", str(formulas))),
             warnings=tuple(warnings),
             checks=tuple(checks),
+            files=tuple(files),
         )
 
 

tools/chemenu/tests/test_raw_derive.py

@@ -5,9 +5,11 @@ readable in the test that uses it and no binary sits in the repository.
 """
 from __future__ import annotations
 
+import csv
 import hashlib
 import io
 import json
+import shutil
 import zipfile
 from email import message_from_bytes, policy
 from email.message import EmailMessage
@@ -156,6 +158,30 @@ def make_xlsx() -> bytes:
     })
 
 
+def make_book(sheets: list[tuple[str, str, str]]) -> bytes:
+    """A workbook of `(name, state, <row> elements)` with inline strings only."""
+    rels = "".join(
+        f'<Relationship Id="rId{n}" Type="{R}/worksheet" Target="worksheets/sheet{n}.xml"/>'
+        for n in range(1, len(sheets) + 1)
+    )
+    listed = "".join(
+        f'<sheet name="{name}" sheetId="{n}" state="{state}" r:id="rId{n}"/>'
+        for n, (name, state, _rows) in enumerate(sheets, 1)
+    )
+    parts = {
+        "_rels/.rels": _rels("xl/workbook.xml"),
+        "xl/workbook.xml": f'<workbook xmlns="{S}" xmlns:r="{R}"><sheets>{listed}</sheets></workbook>',
+        "xl/_rels/workbook.xml.rels": f'<Relationships xmlns="{REL}">{rels}</Relationships>',
+    }
+    for n, (_name, _state, rows) in enumerate(sheets, 1):
+        parts[f"xl/worksheets/sheet{n}.xml"] = f'<worksheet xmlns="{S}"><sheetData>{rows}</sheetData></worksheet>'
+    return _zip(parts)
+
+
+def _text_cell(ref: str, text: str) -> str:
+    return f'<c r="{ref}" t="inlineStr"><is><t xml:space="preserve">{text.replace(chr(13), "&#13;")}</t></is></c>'
+
+
 def make_eml(body: str = "Hallo Welt\n", *, html: str = "", attach: bool = False, inline: bool = False,
              subject: str = "Angebot") -> bytes:
     msg = EmailMessage()
@@ -374,7 +400,11 @@ def test_a_folder_gets_a_md_beside_each_file_with_a_filter(tree, capsys):
     _derive("incoming/mappe")
     out = capsys.readouterr().out
 
-    assert sorted(p.name for p in folder.iterdir()) == ["foto.jpg", "plan.docx", "plan.md", "zahlen.md", "zahlen.xlsx"]
+    assert sorted(p.name for p in folder.iterdir()) == ["foto.jpg", "plan.docx", "plan.md", "zahlen.md", "zahlen.sheets", "zahlen.xlsx"]
+    assert sorted(p.name for p in (folder / "zahlen.sheets").iterdir()) == [
+        "01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv",
+    ]
+    assert "`zahlen.sheets/01-budget.tsv`" in (folder / "zahlen.md").read_text(encoding="utf-8")
     header, _ = _header_and_text(folder / "plan.md")
     assert header["derived_from"] == "plan.docx"
     assert "original_name" not in header
@@ -487,20 +517,175 @@ def test_pptx_slides_notes_and_footer_placeholders(tree):
 # --- XLSX -------------------------------------------------------------------
 
 
-def test_xlsx_values_hidden_sheet_and_trimmed_edges(tree, capsys):
+def _tsv_rows(path: Path) -> list[list[str]]:
+    with path.open(newline="", encoding="utf-8") as handle:
+        return list(csv.reader(handle, delimiter="\t"))
+
+
+def test_xlsx_becomes_an_overview_and_one_tsv_per_sheet(tree, capsys):
     _put(tree, "zahlen.xlsx", make_xlsx())
     _derive("incoming/zahlen.xlsx")
     out = capsys.readouterr().out
-    header, text = _header_and_text(tree / "incoming/zahlen/derived.md")
+    folder = tree / "incoming/zahlen"
+    header, text = _header_and_text(folder / "derived.md")
 
-    assert header["sheets"] == 2 and header["hidden_sheets"] == ["Geheim"]
+    assert header["sheets"] == 2 and header["hidden_sheets"] == ["Geheim"] and header["formulas"] == 1
-    assert "## Budget" in text and "## Geheim" in text and "VERSTECKT-INHALT" in text
+    assert "## Budget" in text and "## Geheim" in text
-    assert "| Posten | Wert |" in text
+    assert "Sheet 2 of 2, hidden" in text
-    assert "| Miete | 1200 |" in text
+    assert "Range A1:C5 (5 rows x 3 columns), 1 formula cell(s)" in text
-    assert "2024-01-01" in text
+    assert "`derived.sheets/01-budget.tsv`" in text and "`derived.sheets/01-budget.formulas.tsv`" in text
-    assert "TRUE" in text
+    assert "`derived.sheets/02-geheim.tsv`" in text and "02-geheim.formulas" not in text
-    assert text.splitlines()[text.splitlines().index("## Budget") + 2].startswith("| Posten")
+    assert "Preview" in text and "Posten\tWert" in text
-    assert "Values are as last saved" in out and "Hidden sheets" in out
+
+    # Row n of the file is row n of the sheet, column k its k-th column: the sheet starts at B2.
+    assert _tsv_rows(folder / "derived.sheets/01-budget.tsv") == [
+        ["", "", ""],
+        ["", "Posten", "Wert"],
+        ["", "Miete", "1200"],
+        ["", "Stichtag", "2024-01-01"],
+        ["", "Fertig", "TRUE"],
+    ]
+    assert _tsv_rows(folder / "derived.sheets/01-budget.formulas.tsv") == [
+        ["cell", "formula", "value"],
+        ["C3", "=1200+0", "1200"],
+    ]
+    assert _tsv_rows(folder / "derived.sheets/02-geheim.tsv") == [["VERSTECKT-INHALT"]]
+    assert "Values are as last saved" in out and "Hidden sheets" in out and ".formulas.tsv" in out
+    assert "derived.sheets" in out
+
+
+def test_tsv_quotes_per_rfc_4180_and_reads_back_cell_for_cell(tree):
+    cells = ["a\tb", 'say "hi"', "two\nlines", "cr\rhere", " padded ", "plain", "="]
+    row = "".join(_text_cell(f"{chr(65 + i)}1", c) for i, c in enumerate(cells))
+    single = '<row r="1"><c r="A1" t="inlineStr"><is><t></t></is></c></row><row r="2">' + _text_cell("A2", "x") + "</row>"
+    _put(tree, "q.xlsx", make_book([("Quoting", "visible", f'<row r="1">{row}</row>'), ("One column", "visible", single)]))
+    _derive("incoming/q.xlsx")
+    sheets = tree / "incoming/q/derived.sheets"
+
+    assert _tsv_rows(sheets / "01-quoting.tsv") == [cells]
+    raw = (sheets / "01-quoting.tsv").read_bytes()
+    assert raw.endswith(b"\n") and b'"a\tb"' in raw and b'"say ""hi"""' in raw and b'"cr\rhere"' in raw
+    # A single-column sheet whose first row is empty: the empty row is still a row.
+    assert (sheets / "02-one-column.tsv").read_bytes() == b'""\nx\n'
+    only_empty_first = make_book([("S", "visible", '<row r="2">' + _text_cell("A2", "x") + "</row>")])
+    _put(tree, "e.xlsx", only_empty_first)
+    _derive("incoming/e.xlsx")
+    assert _tsv_rows(tree / "incoming/e/derived.sheets/01-s.tsv") == [[""], ["x"]]
+    assert (tree / "incoming/e/derived.sheets/01-s.tsv").read_bytes() == b'""\nx\n'
+
+
+def test_shared_array_and_plain_formulas_are_listed_with_their_stored_values(tree, capsys):
+    master = "A2*$B$1+B2+SUM(A$1:B$1)+LOG10(A2)&amp;\"A2\"&amp;'My Sheet'!A2"
+    rows = (
+        '<row r="1"><c r="A1"><v>10</v></c><c r="B1"><v>2</v></c></row>'
+        '<row r="2"><c r="A2"><v>20</v></c><c r="B2"><v>3</v></c>'
+        f'<c r="C2"><f t="shared" ref="C2:C4" si="0">{master}</f><v>1</v></c></row>'
+        '<row r="3"><c r="A3"><v>30</v></c><c r="C3"><f t="shared" si="0"/><v>2</v></c></row>'
+        '<row r="4"><c r="C4"><f t="shared" si="0"/><v>3</v></c>'
+        '<c r="D4"><f t="array" ref="D4:D5">SUM(A2:A4*B2:B4)</f><v>7</v></c></row>'
+    )
+    _put(tree, "f.xlsx", make_book([("Calc", "visible", rows)]))
+    _derive("incoming/f.xlsx")
+    formulas = _tsv_rows(tree / "incoming/f/derived.sheets/01-calc.formulas.tsv")
+
+    def shifted(n: int) -> str:
+        return f"=A{n}*$B$1+B{n}+SUM(A$1:B$1)+LOG10(A{n})&\"A2\"&'My Sheet'!A{n}"
+
+    assert formulas == [
+        ["cell", "formula", "value"],
+        ["C2", shifted(2), "1"],
+        ["C3", shifted(3), "2"],
+        ["C4", shifted(4), "3"],
+        ["D4", "=SUM(A2:A4*B2:B4) (array formula, range D4:D5)", "7"],
+    ]
+    assert "WARN" not in capsys.readouterr().out
+    assert _header_and_text(tree / "incoming/f/derived.md")[0]["formulas"] == 4
+
+
+def test_a_shared_formula_that_would_leave_the_sheet_is_named_by_its_master(tree, capsys):
+    rows = (
+        '<row r="5"><c r="B5"><f t="shared" ref="B5:B6" si="3">A4+1</f><v>1</v></c></row>'
+        '<row r="6"><c r="A6"><f t="shared" si="3"/><v>2</v></c><c r="C6"><f t="shared" si="9"/><v>3</v></c></row>'
+    )
+    _put(tree, "u.xlsx", make_book([("S", "visible", rows)]))
+    _derive("incoming/u.xlsx")
+    out = capsys.readouterr().out
+    formulas = _tsv_rows(tree / "incoming/u/derived.sheets/01-s.formulas.tsv")
+
+    assert formulas[1:] == [["B5", "=A4+1", "1"], ["A6", "shared:B5", "2"], ["C6", "shared:si=9", "3"]]
+    assert "WARN 2 shared formula cell(s) could not be written out" in out
+
+
+def test_xlsx_files_are_the_same_on_a_second_run(tree):
+    data = make_xlsx()
+    _put(tree, "a.xlsx", data)
+    _derive("incoming/a.xlsx")
+    first = {k: v for k, v in _snapshot(tree / "incoming").items() if k.endswith(".tsv")}
+    shutil.rmtree(tree / "incoming/a")
+    _put(tree, "a.xlsx", data)
+    _derive("incoming/a.xlsx")
+    assert {k: v for k, v in _snapshot(tree / "incoming").items() if k.endswith(".tsv")} == first
+    assert first
+
+
+def test_an_empty_and_a_chart_sheet_get_no_file(tree):
+    data = make_book([("Leer", "visible", ""), ("Daten", "visible", '<row r="1">' + _text_cell("A1", "x") + "</row>")])
+    _put(tree, "l.xlsx", data)
+    _derive("incoming/l.xlsx")
+    _header, text = _header_and_text(tree / "incoming/l/derived.md")
+    assert "*(empty sheet)*" in text
+    assert [p.name for p in (tree / "incoming/l/derived.sheets").iterdir()] == ["02-daten.tsv"]
+
+
+def test_sheet_files_count_against_the_path_budget_and_a_refusal_leaves_incoming_alone(tree, capsys):
+    stem = "s" * 120
+    _put(tree, f"{stem}.xlsx", make_xlsx())
+    before = _snapshot(tree / "incoming")
+    _refused(f"incoming/{stem}.xlsx")
+    assert "--name" in capsys.readouterr().out
+    assert _snapshot(tree / "incoming") == before
+    _derive(f"incoming/{stem}.xlsx", "kurz")
+    assert (tree / "incoming/kurz/derived.sheets/01-budget.tsv").is_file()
+
+
+def test_the_folder_form_never_overwrites_a_sheets_folder_and_changes_nothing(tree):
+    folder = tree / "incoming/mappe"
+    (folder / "zahlen.sheets").mkdir(parents=True)
+    (folder / "zahlen.xlsx").write_bytes(make_xlsx())
+    before = _snapshot(tree / "incoming")
+    (folder / "zahlen.sheets/x.txt").write_text("mine", encoding="utf-8")
+    before = _snapshot(tree / "incoming")
+    _refused("incoming/mappe")
+    assert _snapshot(tree / "incoming") == before
+
+
+def test_a_workbook_in_a_zip_or_a_mail_keeps_its_sheets_beside_its_md(tree):
+    _put(tree, "s.zip", make_zip({"sub/Budget 2024.xlsx": make_xlsx(), "Budget 2024.sheets": b"in the way"}))
+    _derive("incoming/s.zip")
+    folder = tree / "incoming/s"
+    assert (folder / "sub/Budget 2024.md").is_file()
+    assert sorted(p.name for p in (folder / "sub/Budget 2024.sheets").iterdir()) == [
+        "01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv",
+    ]
+    assert "`Budget 2024.sheets/01-budget.tsv`" in (folder / "sub/Budget 2024.md").read_text(encoding="utf-8")
+
+    _put(tree, "m.eml", make_mail([("derived.xlsx", make_xlsx(), "application/octet-stream")]))
+    _derive("incoming/m.eml")
+    mail = tree / "incoming/m"
+    assert (mail / "derived-2.md").is_file() and (mail / "derived-2.sheets/01-budget.tsv").is_file()
+    assert "`derived-2.sheets/01-budget.tsv`" in (mail / "derived-2.md").read_text(encoding="utf-8")
+
+
+def test_the_derived_folder_with_its_sheet_files_is_accepted_whole(tree):
+    _put(tree, "zahlen.xlsx", make_xlsx())
+    _derive("incoming/zahlen.xlsx")
+    raw_accept_command(
+        files=[Path("incoming/zahlen")], fidelity="verbatim", authority="reporting",
+        page=None, replaces=None, dry_run=False,
+    )
+    accepted = sorted(p for p in (tree / "raw").rglob("*.tsv"))
+    assert [p.name for p in accepted] == ["01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv"]
+    assert not (tree / "incoming/zahlen").exists()
 
 
 def test_xlsx_check_is_printed_only_for_the_formats_derived_in_the_run(tree, capsys):

tools/chemenu/tests/test_raw_derive_testfiles.py

@@ -0,0 +1,104 @@
+"""`wikitool raw derive` against the real documents in `testfiles/`.
+
+The synthetic fixtures in `binary_formats.py` and `test_raw_derive.py` prove the filters on
+documents built to be easy; the files in `testfiles/` are whatever the world produced. Every file
+there is derived, compared with itself on a second run and accepted into `raw/`, and an optional
+`<file>.expect` next to it names the text that must come out. A distributed instance has no
+`testfiles/`, so the whole module is then skipped.
+"""
+from __future__ import annotations
+
+import importlib.util
+import shutil
+from pathlib import Path
+
+import pytest
+
+from chemenu import filters
+from chemenu.commands.raw_cmd import raw_accept_command
+from chemenu.tests.test_raw_derive import (  # noqa: F401  (tree is a fixture)
+    _derive,
+    _header_and_text,
+    tree,
+)
+
+TESTFILES = Path(__file__).resolve().parents[3] / "testfiles"
+SIDECAR = ".expect"
+
+pytestmark = pytest.mark.skipif(
+    importlib.util.find_spec("pypdf") is None or importlib.util.find_spec("olefile") is None,
+    reason="pypdf and olefile are needed for real PDFs, MSGs and archives holding them",
+)
+
+
+def _documents() -> list[Path]:
+    if not TESTFILES.is_dir():
+        return []
+    return sorted(
+        path
+        for path in TESTFILES.rglob("*")
+        if path.is_file()
+        and not path.name.startswith(".")
+        and path.name != "README.md"
+        and path.suffix != SIDECAR
+    )
+
+
+def _id(path: Path) -> str:
+    return path.relative_to(TESTFILES).as_posix()
+
+
+def _texts(folder: Path) -> dict[str, str]:
+    texts = {}
+    for md in sorted(folder.rglob("*.md")):
+        lines = md.read_text(encoding="utf-8").splitlines()
+        texts[md.relative_to(folder).as_posix()] = "\n".join(
+            line for line in lines if not line.startswith("derived:")
+        )
+    return texts
+
+
+def _expected(document: Path) -> list[str]:
+    sidecar = document.with_name(document.name + SIDECAR)
+    if not sidecar.is_file():
+        return []
+    lines = sidecar.read_text(encoding="utf-8").splitlines()
+    return [line for line in lines if line.strip() and not line.startswith("#")]
+
+
+def _put_and_derive(tree: Path, document: Path) -> Path:
+    (tree / "incoming" / document.name).write_bytes(document.read_bytes())
+    _derive(f"incoming/{document.name}")
+    return tree / "incoming" / document.stem
+
+
+@pytest.mark.parametrize("document", _documents(), ids=_id)
+def test_a_real_document_derives_stably_and_is_accepted(tree, document):
+    assert filters.filter_for(document.name) is not None, (
+        f"no import filter takes {document.name}; a file in testfiles/ must be one `raw derive` reads"
+    )
+
+    folder = _put_and_derive(tree, document)
+    assert not (tree / "incoming" / document.name).exists()
+
+    if document.suffix.lower() == ".zip":
+        assert any(folder.rglob("*.md")), "an unpacked archive gets a .md beside its documents"
+    else:
+        original = folder / f"original{document.suffix.lower()}"
+        assert original.read_bytes() == document.read_bytes()
+        header, _text = _header_and_text(folder / "derived.md")
+        assert header["derived_by"] == "wikitool raw derive"
+        assert header["original_name"] == document.name
+
+    first = _texts(folder)
+    for needle in _expected(document):
+        assert any(needle in text for text in first.values()), f"{needle!r} not in the derived text"
+
+    shutil.rmtree(folder)
+    assert _texts(_put_and_derive(tree, document)) == first, "the same bytes gave a different text"
+
+    raw_accept_command(
+        files=[Path(f"incoming/{document.stem}")], fidelity="verbatim", authority="reporting",
+        page=None, replaces=None, dry_run=False,
+    )
+    assert not (tree / "incoming" / document.stem).exists()