torbenandClaude Code
9868fce3d5
Each sheet becomes NN-<sheet>.tsv (RFC 4180, tab-delimited, A1-aligned) in a <stem>.sheets/ folder beside the .md; sheets with formulas also get a .formulas.tsv. derived.md is the overview. Adds testfiles/ with a CI test. Files changed: - .gitattributes - CHANGES.md - VERSION - docs/dev/development.md - docs/how-to/ingest-a-source.md - docs/reference/commands/raw-material-and-uploads.md - raw/CONTRACT.md - testfiles/README.md - tools/CONTRACT.md - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/archive.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/tests/test_raw_derive_testfiles.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
testfiles
Real documents (PDF, mail, Office, ZIP) that CI feeds to raw derive, beyond the synthetic
fixtures in tools/chemenu/tests/binary_formats.py. tools/chemenu/tests/test_raw_derive_testfiles.py
picks up every file under this directory, so adding a file is adding a test case - no code change.
The directory exists only in a development clone: dist export does not ship it.
What goes in
Everything here is committed to a public repository. A file belongs here only when it can be public:
- a document you wrote for the purpose, or one whose licence allows redistribution;
- no real people, customers, employers, addresses or internal URLs - in the text and in the metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
- nothing that was received in confidence.
Removing a file later does not remove it from the history.
Layout
| Folder | For |
|---|---|
pdf/ |
PDFs, including scanned ones without a text layer |
mail/ |
.eml and .msg, with and without attachments, with attached mails |
office/ |
.docx, .pptx, .xlsx (tracked changes, comments, hidden sheets) |
zip/ |
archives, including mail-in-ZIP cases |
The folders are a convention; the test reads every file below this directory. Keep file names
short and free of spaces - a name that is too long for the raw/ path budget fails the test for
the right reason, but it is a nuisance to find.
What the test checks
For each file, in a throwaway instance: raw derive succeeds; the original comes back byte for
byte (a ZIP is unpacked and gets a .md beside its documents instead); a second run over the same
bytes gives the same text apart from derived:; and raw accept takes the folder.
A file may carry a sidecar <file name>.expect (offer.pdf.expect for offer.pdf): one line per
piece of text that must appear in the derived text, empty lines and lines starting with #
ignored. Without one the file only has to derive, be stable and be accepted. A document that is
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
Trying a file by hand
raw derive moves what it is given, so work on a copy in a throwaway instance, never on
incoming/ of this clone.