feat(raw): raw derive unpacks ZIP archives and mails with attachments into a folder
CI / verify (push) Successful in 6m36s
CI / pwsh (push) Successful in 2m20s
Release / release (push) Successful in 35s
Release / wiki-mirror (push) Skipped

A ZIP becomes incoming/<stem>/ with its files and a derived text beside each
document, the ZIP is removed and its sha256 named in the output. A mail with
attachments becomes a folder with original.eml, derived.md and the attachments;
a ZIP attached to a mail is unpacked once, an attached mail is quoted. Unsafe
entries, nested ZIPs and size limits are refused before anything is written.

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/README.md
- docs/reference/commands/raw-material-and-uploads.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/filters/__init__.py
- tools/chemenu/filters/archive.py
- tools/chemenu/filters/mail.py
- tools/chemenu/tests/test_portability.py
- tools/chemenu/tests/test_raw_derive.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-08 13:04:01 +02:00
1 parent 9c0bb6bfbf
commit 6659ca5409
14 files changed
+828 -94

No files matched your search

+13
View File
@@ -276,6 +276,19 @@ a stem of well over a hundred characters, where keeping the file's own name twic
as `original_name`. A folder already in `incoming/` is derived in place instead: each file with a
filter gets a `<stem>.md` beside it, and nothing is renamed.
**A container is unpacked into its folder.** A ZIP becomes the folder `incoming/<stem>/` holding
its files under their own names, each with a filter next to a `<stem>.md`, and the ZIP itself is
removed - the tool's output names it and its sha256, and naming it on the source page is a human's
step. A mail with attachments becomes a folder holding `original.eml`, `derived.md` and the
attachments under their own names; a ZIP attached to a mail is unpacked once into a folder of its
stem, and a mail attached to a mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP,
and any entry that is an absolute or `..` path, a symbolic link or encrypted, is refused, as is
an archive past the entry, file or total size limit that `raw derive -h` names. Hidden entries
(`__MACOSX/`, `.DS_Store`, anything starting with a dot) are not written but are named in the
output. A name that is taken or reserved (`original.*` and `derived.md` at the top of a mail
folder, `_capture.json` anywhere) gets `-2`, `-3` before its extension, and the output lists
every rename.
**The original stays.** The derivation is the tool's reading of it, never the source: with the
original kept, a claim in `kb/` can still be checked against what the document said when a
derivation dropped something, and a later version of the tool can derive it better. Both go into