feat(raw): raw derive reads PDF and Outlook MSG; doctor shows each import filter
CI / verify (push) Successful in 6m41s
CI / pwsh (push) Successful in 2m24s
Release / release (push) Successful in 34s
Release / wiki-mirror (push) Skipped

PDF is read with pypdf (page markers, scan warning, encrypted files refused), Outlook .msg with olefile as a container like an .eml, and a .msg attached to a mail is quoted. Both packages are in tools/requirements.txt and imported only when such a file is derived, so a missing one stops that format alone. doctor reports one import-filter line per package-backed format.

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/ingest-a-source.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/filters/__init__.py
- tools/chemenu/filters/mail.py
- tools/chemenu/filters/msg.py
- tools/chemenu/filters/pdf.py
- tools/chemenu/tests/binary_formats.py
- tools/chemenu/tests/test_raw_derive_binary.py
- tools/requirements.txt

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed 2026-10-08 14:02:16 +02:00
1 parent 6659ca5409
commit 0e6a50aa98
16 files changed
+812 -17

No files matched your search

+4 -2
View File
@@ -16,11 +16,13 @@ pick a directory:
- **A web page** needs no download of your own. Name the URL to the agent, and
`tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
text derived from it with a header recording where and when it was fetched.
- **A document that is not text** - a Word, PowerPoint or Excel file, a mail or a ZIP archive - is
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
Outlook `.msg`) or a ZIP archive - is
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
untouched original and a readable text beside it, with a header saying how it was made. The
folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is
unpacked into a folder instead, and a mail's attachments land beside it.
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
text to derive - the tool warns, and there is no OCR.
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
agent derives the same text from that file with `raw fetch --html`.