feat(raw): raw derive reads PDF and Outlook MSG; doctor shows each import filter
PDF is read with pypdf (page markers, scan warning, encrypted files refused), Outlook .msg with olefile as a container like an .eml, and a .msg attached to a mail is quoted. Both packages are in tools/requirements.txt and imported only when such a file is derived, so a missing one stops that format alone. doctor reports one import-filter line per package-backed format. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/msg.py - tools/chemenu/filters/pdf.py - tools/chemenu/tests/binary_formats.py - tools/chemenu/tests/test_raw_derive_binary.py - tools/requirements.txt Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
6659ca5409
commit
0e6a50aa98
16 files changed
+812
-17
No files matched your search
@@ -16,11 +16,13 @@ pick a directory:
|
||||
- **A web page** needs no download of your own. Name the URL to the agent, and
|
||||
`tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
|
||||
text derived from it with a header recording where and when it was fetched.
|
||||
- **A document that is not text** - a Word, PowerPoint or Excel file, a mail or a ZIP archive - is
|
||||
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
|
||||
Outlook `.msg`) or a ZIP archive - is
|
||||
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
|
||||
untouched original and a readable text beside it, with a header saying how it was made. The
|
||||
folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is
|
||||
unpacked into a folder instead, and a mail's attachments land beside it.
|
||||
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
|
||||
text to derive - the tool warns, and there is no OCR.
|
||||
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
||||
agent derives the same text from that file with `raw fetch --html`.
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ tools/wikitool doctor --json
|
||||
**Behaviour**
|
||||
|
||||
- Checks dependencies (Python, ripgrep), author resolution, stack version, git identity/branch/remote, published skills, the kb/raw/reports/work/instructions structure, and generated files.
|
||||
- Import filters (`import-filter-<name>`): one line for each `raw derive` format that reads through a package (PDF, Outlook `.msg`) - a `WARN` naming the package when it is missing, since that costs only its own format; fixed by running `tools/preflight.sh` again.
|
||||
- Tool paths (`tool-paths`): `.wikitool-tools.json` written by a preflight that finished, every tool `tools/prerequisites.txt` names for this platform recorded, and every recorded path still there - a `FAIL` otherwise, fixed by running `tools/preflight.sh` again (PowerShell 7: `tools/preflight.ps1`).
|
||||
- Install folder (`install-dir`): on Windows with long paths off, a `FAIL` when the folder holding `tools/` is longer than `tools/prerequisites.txt` allows (95 characters) - the limit the preflight enforces before it sets anything up.
|
||||
- PowerShell (`execution-policy`, `script-marks`; Windows only, `OK` elsewhere): a `FAIL` when the effective execution policy is `Restricted` or `AllSigned` - the line then says whether a group policy sets it, which only whoever administers the computer can change - and a `FAIL` when a script under `tools/` carries a Mark of the Web from the internet zone, which a browser download unpacked in Explorer leaves behind and `Invoke-WebRequest` plus `tar` do not. `tools/preflight.ps1` checks the same two things before it stops.
|
||||
|
||||
@@ -118,7 +118,7 @@ tools/wikitool raw derive incoming/scans.zip
|
||||
- A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.<ext>` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`.
|
||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read.
|
||||
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||
- After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
|
||||
|
||||
Reference in new issue
Block a user