feat(raw): raw derive writes an Excel file's sheets as TSV files with a formula list (#202)
Each sheet becomes NN-<sheet>.tsv (RFC 4180, tab-delimited, A1-aligned) in a <stem>.sheets/ folder beside the .md; sheets with formulas also get a .formulas.tsv. derived.md is the overview. Adds testfiles/ with a CI test. Files changed: - .gitattributes - CHANGES.md - VERSION - docs/dev/development.md - docs/how-to/ingest-a-source.md - docs/reference/commands/raw-material-and-uploads.md - raw/CONTRACT.md - testfiles/README.md - tools/CONTRACT.md - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/archive.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/tests/test_raw_derive_testfiles.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
8ea2795d88
commit
9868fce3d5
15 files changed
+719
-50
No files matched your search
@@ -0,0 +1,47 @@
|
||||
# testfiles
|
||||
|
||||
Real documents (PDF, mail, Office, ZIP) that CI feeds to `raw derive`, beyond the synthetic
|
||||
fixtures in `tools/chemenu/tests/binary_formats.py`. `tools/chemenu/tests/test_raw_derive_testfiles.py`
|
||||
picks up every file under this directory, so adding a file is adding a test case - no code change.
|
||||
The directory exists only in a development clone: `dist export` does not ship it.
|
||||
|
||||
## What goes in
|
||||
|
||||
**Everything here is committed to a public repository.** A file belongs here only when it can be
|
||||
public:
|
||||
|
||||
- a document you wrote for the purpose, or one whose licence allows redistribution;
|
||||
- no real people, customers, employers, addresses or internal URLs - in the text *and* in the
|
||||
metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names);
|
||||
- nothing that was received in confidence.
|
||||
|
||||
Removing a file later does not remove it from the history.
|
||||
|
||||
## Layout
|
||||
|
||||
| Folder | For |
|
||||
|--------|-----|
|
||||
| `pdf/` | PDFs, including scanned ones without a text layer |
|
||||
| `mail/` | `.eml` and `.msg`, with and without attachments, with attached mails |
|
||||
| `office/` | `.docx`, `.pptx`, `.xlsx` (tracked changes, comments, hidden sheets) |
|
||||
| `zip/` | archives, including mail-in-ZIP cases |
|
||||
|
||||
The folders are a convention; the test reads every file below this directory. Keep file names
|
||||
short and free of spaces - a name that is too long for the `raw/` path budget fails the test for
|
||||
the right reason, but it is a nuisance to find.
|
||||
|
||||
## What the test checks
|
||||
|
||||
For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for
|
||||
byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same
|
||||
bytes gives the same text apart from `derived:`; and `raw accept` takes the folder.
|
||||
|
||||
A file may carry a sidecar `<file name>.expect` (`offer.pdf.expect` for `offer.pdf`): one line per
|
||||
piece of text that must appear in the derived text, empty lines and lines starting with `#`
|
||||
ignored. Without one the file only has to derive, be stable and be accepted. A document that is
|
||||
meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals.
|
||||
|
||||
## Trying a file by hand
|
||||
|
||||
`raw derive` moves what it is given, so work on a copy in a throwaway instance, never on
|
||||
`incoming/` of this clone.
|
||||
Reference in new issue
Block a user