torbenandClaude Code
9868fce3d5
Each sheet becomes NN-<sheet>.tsv (RFC 4180, tab-delimited, A1-aligned) in a <stem>.sheets/ folder beside the .md; sheets with formulas also get a .formulas.tsv. derived.md is the overview. Adds testfiles/ with a CI test. Files changed: - .gitattributes - CHANGES.md - VERSION - docs/dev/development.md - docs/how-to/ingest-a-source.md - docs/reference/commands/raw-material-and-uploads.md - raw/CONTRACT.md - testfiles/README.md - tools/CONTRACT.md - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/archive.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/tests/test_raw_derive_testfiles.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
48 lines
2.2 KiB
Markdown
48 lines
2.2 KiB
Markdown
# testfiles
|
|
|
|
Real documents (PDF, mail, Office, ZIP) that CI feeds to `raw derive`, beyond the synthetic
|
|
fixtures in `tools/chemenu/tests/binary_formats.py`. `tools/chemenu/tests/test_raw_derive_testfiles.py`
|
|
picks up every file under this directory, so adding a file is adding a test case - no code change.
|
|
The directory exists only in a development clone: `dist export` does not ship it.
|
|
|
|
## What goes in
|
|
|
|
**Everything here is committed to a public repository.** A file belongs here only when it can be
|
|
public:
|
|
|
|
- a document you wrote for the purpose, or one whose licence allows redistribution;
|
|
- no real people, customers, employers, addresses or internal URLs - in the text *and* in the
|
|
metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
|
|
- nothing that was received in confidence.
|
|
|
|
Removing a file later does not remove it from the history.
|
|
|
|
## Layout
|
|
|
|
| Folder | For |
|
|
|--------|-----|
|
|
| `pdf/` | PDFs, including scanned ones without a text layer |
|
|
| `mail/` | `.eml` and `.msg`, with and without attachments, with attached mails |
|
|
| `office/` | `.docx`, `.pptx`, `.xlsx` (tracked changes, comments, hidden sheets) |
|
|
| `zip/` | archives, including mail-in-ZIP cases |
|
|
|
|
The folders are a convention; the test reads every file below this directory. Keep file names
|
|
short and free of spaces - a name that is too long for the `raw/` path budget fails the test for
|
|
the right reason, but it is a nuisance to find.
|
|
|
|
## What the test checks
|
|
|
|
For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for
|
|
byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same
|
|
bytes gives the same text apart from `derived:`; and `raw accept` takes the folder.
|
|
|
|
A file may carry a sidecar `<file name>.expect` (`offer.pdf.expect` for `offer.pdf`): one line per
|
|
piece of text that must appear in the derived text, empty lines and lines starting with `#`
|
|
ignored. Without one the file only has to derive, be stable and be accepted. A document that is
|
|
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
|
|
|
|
## Trying a file by hand
|
|
|
|
`raw derive` moves what it is given, so work on a copy in a throwaway instance, never on
|
|
`incoming/` of this clone.
|