feat(raw): raw derive unpacks ZIP archives and mails with attachments into a folder
A ZIP becomes incoming/<stem>/ with its files and a derived text beside each document, the ZIP is removed and its sha256 named in the output. A mail with attachments becomes a folder with original.eml, derived.md and the attachments; a ZIP attached to a mail is unpacked once, an attached mail is quoted. Unsafe entries, nested ZIPs and size limits are refused before anything is written. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/archive.py - tools/chemenu/filters/mail.py - tools/chemenu/tests/test_portability.py - tools/chemenu/tests/test_raw_derive.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
9c0bb6bfbf
commit
6659ca5409
14 files changed
+828
-94
No files matched your search
@@ -16,10 +16,11 @@ pick a directory:
|
||||
- **A web page** needs no download of your own. Name the URL to the agent, and
|
||||
`tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
|
||||
text derived from it with a header recording where and when it was fetched.
|
||||
- **A document that is not text** - a Word, PowerPoint or Excel file, or a mail - is put into
|
||||
`incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
|
||||
- **A document that is not text** - a Word, PowerPoint or Excel file, a mail or a ZIP archive - is
|
||||
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
|
||||
untouched original and a readable text beside it, with a header saying how it was made. The
|
||||
folder is accepted whole; the agent runs this itself when it meets such a file.
|
||||
folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is
|
||||
unpacked into a folder instead, and a mail's attachments land beside it.
|
||||
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
||||
agent derives the same text from that file with `raw fetch --html`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user