testfiles: real documents get short names and a source table, personal data is replaced by same-length filler (#203)
CI / verify (push) Successful in 7m43s
CI / pwsh (push) Successful in 2m27s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

Adds the nineteen real documents under testfiles/ (mail, Office, PDF, ZIP) with short ASCII names. Recipient addresses, names, host names, mailbox identifiers and recipient-bound tokens are replaced by filler of the same length, and the archive is rebuilt from the sanitized members. test_raw_derive_testfiles.py now compares the sheet TSVs too (with a direct test on _texts()), and a real workbook carries a formula needle that only a .formulas.tsv holds. testfiles/README.md gains the provenance table and the two new rules.

Files changed:
- CHANGES.md
- VERSION
- testfiles/README.md
- testfiles/mail/github-outlook.msg
- testfiles/mail/github-roundcube.eml
- testfiles/mail/gog-outlook.msg
- testfiles/mail/gog-roundcube.eml
- testfiles/mail/tldr-outlook.msg
- testfiles/mail/tldr-roundcube.eml
- testfiles/office/excel-artificial.xlsx
- testfiles/office/excel-business.xlsx
- testfiles/office/excel-business.xlsx.expect
- testfiles/office/excel-science.xlsx
- testfiles/office/pptx-baseline.pptx
- testfiles/office/pptx-master.pptx
- testfiles/office/pptx-objects.pptx
- testfiles/office/word-complex.docx
- testfiles/office/word-simple.docx
- testfiles/pdf/einstein-dissertation.pdf
- testfiles/pdf/ifg.pdf
- testfiles/pdf/nist-ai-rmf.pdf
- testfiles/pdf/nist-ssdf-genai.pdf
- testfiles/zip/archive.zip
- tools/chemenu/tests/test_raw_derive_testfiles.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-09 16:07:38 +02:00
1 parent 9868fce3d5
commit be049217a9
24 files changed
+2359 -5

No files matched your search

+22 -3
View File
@@ -15,8 +15,24 @@ public:
metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
- nothing that was received in confidence.
The author names of a published work (a NIST publication, a statute) are part of the work, not a
real person in the sense of the second item. Links that address an account - unsubscribe,
settings, tracking - count as personal data even when no name is in them: they let anyone holding
the file reach the recipient's account at the sender. Replace such text with filler of the same
length, so a mail keeps its structure.
Removing a file later does not remove it from the history.
## Where the files come from
| Files | Origin and basis for publishing them |
|-------|--------------------------------------|
| `pdf/nist-ai-rmf.pdf`, `pdf/nist-ssdf-genai.pdf` | NIST AI 600-1 and NIST SP 800-218A: works of the US federal government, public domain. The author in the PDF metadata is the author of the published work. |
| `pdf/ifg.pdf` | Informationsfreiheitsgesetz as published on gesetze-im-internet.de: an official work (§ 5 UrhG), free of copyright. |
| `pdf/einstein-dissertation.pdf` | Einstein's dissertation (1905), a public-domain work. The file is a scan from a web download; the digitizer's terms of use were not checked. Kept deliberately by the maintainer for the scanned pages (OCR coverage). |
| `mail/*` (GitHub, GOG and TLDR newsletters, each as `.eml` and `.msg`) | Mails received by the maintainer; promotional and informational content of the senders, released by the maintainer as test fixtures. Recipient address, names, host names and recipient-bound tokens are replaced by filler of the same length. |
| `office/*`, `zip/archive.zip` | Written by the maintainer for this purpose, with fictitious data; release confirmed by the maintainer. Author fields in the Office metadata are empty. The archive holds copies of six files of this directory (among them two mails) and is rebuilt from the sanitized copies. |
## Layout
| Folder | For |
@@ -34,11 +50,14 @@ the right reason, but it is a nuisance to find.
For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for
byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same
bytes gives the same text apart from `derived:`; and `raw accept` takes the folder.
bytes gives the same text apart from `derived:`; and `raw accept` takes the folder. The text
compared is every `.md` of the result plus the sheet files of a workbook (`<stem>.sheets/*.tsv`,
cells and formula lists).
A file may carry a sidecar `<file name>.expect` (`offer.pdf.expect` for `offer.pdf`): one line per
piece of text that must appear in the derived text, empty lines and lines starting with `#`
ignored. Without one the file only has to derive, be stable and be accepted. A document that is
piece of text that must appear in the derived text - in any `.md` or sheet TSV of the result -
empty lines and lines starting with `#` ignored. A formula is named by its bare text, such as
`SUM(Sales!L6:L29)`, not by a tab-separated line. Without one the file only has to derive, be stable and be accepted. A document that is
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
## Trying a file by hand