feat(raw): raw derive writes an Excel file's sheets as TSV files with a formula list (#202)
CI / verify (push) Successful in 6m47s
CI / pwsh (push) Successful in 2m25s
Release / release (push) Successful in 34s
Release / wiki-mirror (push) Skipped

Each sheet becomes NN-<sheet>.tsv (RFC 4180, tab-delimited, A1-aligned) in a
<stem>.sheets/ folder beside the .md; sheets with formulas also get a
.formulas.tsv. derived.md is the overview. Adds testfiles/ with a CI test.

Files changed:
- .gitattributes
- CHANGES.md
- VERSION
- docs/dev/development.md
- docs/how-to/ingest-a-source.md
- docs/reference/commands/raw-material-and-uploads.md
- raw/CONTRACT.md
- testfiles/README.md
- tools/CONTRACT.md
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/filters/__init__.py
- tools/chemenu/filters/archive.py
- tools/chemenu/filters/ooxml.py
- tools/chemenu/tests/test_raw_derive.py
- tools/chemenu/tests/test_raw_derive_testfiles.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-09 13:35:03 +02:00
1 parent 8ea2795d88
commit 9868fce3d5
15 files changed
+719 -50

No files matched your search

+47
View File
@@ -0,0 +1,47 @@
# testfiles
Real documents (PDF, mail, Office, ZIP) that CI feeds to `raw derive`, beyond the synthetic
fixtures in `tools/chemenu/tests/binary_formats.py`. `tools/chemenu/tests/test_raw_derive_testfiles.py`
picks up every file under this directory, so adding a file is adding a test case - no code change.
The directory exists only in a development clone: `dist export` does not ship it.
## What goes in
**Everything here is committed to a public repository.** A file belongs here only when it can be
public:
- a document you wrote for the purpose, or one whose licence allows redistribution;
- no real people, customers, employers, addresses or internal URLs - in the text *and* in the
metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
- nothing that was received in confidence.
Removing a file later does not remove it from the history.
## Layout
| Folder | For |
|--------|-----|
| `pdf/` | PDFs, including scanned ones without a text layer |
| `mail/` | `.eml` and `.msg`, with and without attachments, with attached mails |
| `office/` | `.docx`, `.pptx`, `.xlsx` (tracked changes, comments, hidden sheets) |
| `zip/` | archives, including mail-in-ZIP cases |
The folders are a convention; the test reads every file below this directory. Keep file names
short and free of spaces - a name that is too long for the `raw/` path budget fails the test for
the right reason, but it is a nuisance to find.
## What the test checks
For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for
byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same
bytes gives the same text apart from `derived:`; and `raw accept` takes the folder.
A file may carry a sidecar `<file name>.expect` (`offer.pdf.expect` for `offer.pdf`): one line per
piece of text that must appear in the derived text, empty lines and lines starting with `#`
ignored. Without one the file only has to derive, be stable and be accepted. A document that is
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
## Trying a file by hand
`raw derive` moves what it is given, so work on a copy in a throwaway instance, never on
`incoming/` of this clone.