torbenandClaude Code
be049217a9
Adds the nineteen real documents under testfiles/ (mail, Office, PDF, ZIP) with short ASCII names. Recipient addresses, names, host names, mailbox identifiers and recipient-bound tokens are replaced by filler of the same length, and the archive is rebuilt from the sanitized members. test_raw_derive_testfiles.py now compares the sheet TSVs too (with a direct test on _texts()), and a real workbook carries a formula needle that only a .formulas.tsv holds. testfiles/README.md gains the provenance table and the two new rules. Files changed: - CHANGES.md - VERSION - testfiles/README.md - testfiles/mail/github-outlook.msg - testfiles/mail/github-roundcube.eml - testfiles/mail/gog-outlook.msg - testfiles/mail/gog-roundcube.eml - testfiles/mail/tldr-outlook.msg - testfiles/mail/tldr-roundcube.eml - testfiles/office/excel-artificial.xlsx - testfiles/office/excel-business.xlsx - testfiles/office/excel-business.xlsx.expect - testfiles/office/excel-science.xlsx - testfiles/office/pptx-baseline.pptx - testfiles/office/pptx-master.pptx - testfiles/office/pptx-objects.pptx - testfiles/office/word-complex.docx - testfiles/office/word-simple.docx - testfiles/pdf/einstein-dissertation.pdf - testfiles/pdf/ifg.pdf - testfiles/pdf/nist-ai-rmf.pdf - testfiles/pdf/nist-ssdf-genai.pdf - testfiles/zip/archive.zip - tools/chemenu/tests/test_raw_derive_testfiles.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
testfiles
Real documents (PDF, mail, Office, ZIP) that CI feeds to raw derive, beyond the synthetic
fixtures in tools/chemenu/tests/binary_formats.py. tools/chemenu/tests/test_raw_derive_testfiles.py
picks up every file under this directory, so adding a file is adding a test case - no code change.
The directory exists only in a development clone: dist export does not ship it.
What goes in
Everything here is committed to a public repository. A file belongs here only when it can be public:
- a document you wrote for the purpose, or one whose licence allows redistribution;
- no real people, customers, employers, addresses or internal URLs - in the text and in the metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
- nothing that was received in confidence.
The author names of a published work (a NIST publication, a statute) are part of the work, not a real person in the sense of the second item. Links that address an account - unsubscribe, settings, tracking - count as personal data even when no name is in them: they let anyone holding the file reach the recipient's account at the sender. Replace such text with filler of the same length, so a mail keeps its structure.
Removing a file later does not remove it from the history.
Where the files come from
| Files | Origin and basis for publishing them |
|---|---|
pdf/nist-ai-rmf.pdf, pdf/nist-ssdf-genai.pdf |
NIST AI 600-1 and NIST SP 800-218A: works of the US federal government, public domain. The author in the PDF metadata is the author of the published work. |
pdf/ifg.pdf |
Informationsfreiheitsgesetz as published on gesetze-im-internet.de: an official work (§ 5 UrhG), free of copyright. |
pdf/einstein-dissertation.pdf |
Einstein's dissertation (1905), a public-domain work. The file is a scan from a web download; the digitizer's terms of use were not checked. Kept deliberately by the maintainer for the scanned pages (OCR coverage). |
mail/* (GitHub, GOG and TLDR newsletters, each as .eml and .msg) |
Mails received by the maintainer; promotional and informational content of the senders, released by the maintainer as test fixtures. Recipient address, names, host names and recipient-bound tokens are replaced by filler of the same length. |
office/*, zip/archive.zip |
Written by the maintainer for this purpose, with fictitious data; release confirmed by the maintainer. Author fields in the Office metadata are empty. The archive holds copies of six files of this directory (among them two mails) and is rebuilt from the sanitized copies. |
Layout
| Folder | For |
|---|---|
pdf/ |
PDFs, including scanned ones without a text layer |
mail/ |
.eml and .msg, with and without attachments, with attached mails |
office/ |
.docx, .pptx, .xlsx (tracked changes, comments, hidden sheets) |
zip/ |
archives, including mail-in-ZIP cases |
The folders are a convention; the test reads every file below this directory. Keep file names
short and free of spaces - a name that is too long for the raw/ path budget fails the test for
the right reason, but it is a nuisance to find.
What the test checks
For each file, in a throwaway instance: raw derive succeeds; the original comes back byte for
byte (a ZIP is unpacked and gets a .md beside its documents instead); a second run over the same
bytes gives the same text apart from derived:; and raw accept takes the folder. The text
compared is every .md of the result plus the sheet files of a workbook (<stem>.sheets/*.tsv,
cells and formula lists).
A file may carry a sidecar <file name>.expect (offer.pdf.expect for offer.pdf): one line per
piece of text that must appear in the derived text - in any .md or sheet TSV of the result -
empty lines and lines starting with # ignored. A formula is named by its bare text, such as
SUM(Sales!L6:L29), not by a tab-separated line. Without one the file only has to derive, be stable and be accepted. A document that is
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
Trying a file by hand
raw derive moves what it is given, so work on a copy in a throwaway instance, never on
incoming/ of this clone.