feat(raw): raw derive turns Office files and mail into a folder with the original and a derived text
Adds the import-filter registry (tools/chemenu/filters), the raw derive command with folder form, path-budget and name checks, DOCX/PPTX/XLSX and EML filters, the pending and accept hints, the wiki-ingest skill step and the raw/CONTRACT.md section. First of three cuts; containers and PDF/MSG follow. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/html_page.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
765ebe97d7
commit
9c0bb6bfbf
17 files changed
+2304
-17
No files matched your search
+61
-1
@@ -123,6 +123,7 @@ sources coverage read idempotent budget:counted exit:0
|
||||
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
|
||||
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
|
||||
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
|
||||
raw derive write non-idempotent budget:counted exit:0,1 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote.
|
||||
raw capture write non-idempotent budget:counted exit:0,1 Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
raw status read idempotent budget:counted exit:0 Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`.
|
||||
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
@@ -1466,6 +1467,64 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
#### `raw derive`
|
||||
|
||||
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw derive incoming/<file> [--name <stem>]` - Move the file into the new folder `incoming/<stem>/` as `original.<ext>` and write `incoming/<stem>/derived.md`
|
||||
- `wikitool raw derive incoming/<folder>` - Write a `<stem>.md` beside every file in the folder that has a filter
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: Yes for what it leaves behind - everything is read, derived and checked before the first write; the files are created exclusively, and a failure part-way puts back what this call changed
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool raw derive incoming/report.docx`
|
||||
- `tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz`
|
||||
- `tools/wikitool raw derive incoming/meeting-files`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension
|
||||
- 1 The target folder or a `.md` already exists, or the path would be over the path budget
|
||||
- 1 The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -> Nothing was written. Fix the argument and retry once - a file in a subfolder is moved up, or its folder is derived whole
|
||||
- The target folder or a `.md` already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name <shorter stem>`, then retry once. An existing `.md` is never overwritten
|
||||
- The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed -> Nothing was written. A missing package is reported with the preflight hint - show it to the user and stop, never install it. Any other filter keeps working
|
||||
|
||||
**NEVER**
|
||||
|
||||
- Never edit `derived.md` or its header by hand.
|
||||
- Never derive a file the user did not put into `incoming/` for this purpose.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- Writes into `incoming/` only, never into `raw/`, and reads no network: `raw accept` promotes the folder afterwards as one source under `raw/<YYYY>/<MM>/<stem>/`.
|
||||
- A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.<ext>` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`.
|
||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`) and e-mail (`.eml`) read with the standard library only. A mail with attachments is refused.
|
||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||
- After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
|
||||
- Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a document in: `raw derive`" - the rules and why
|
||||
- `wikitool raw accept` - promotes the folder into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a PDF, mail or Office file to ingest starts
|
||||
|
||||
#### `raw capture`
|
||||
|
||||
Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
@@ -1611,7 +1670,8 @@ List what waits in `incoming/`, oldest first, and name the entry an ingest witho
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`.
|
||||
- A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is listed with the `raw derive` line for it, in text output and as `hint` in JSON (`null` for every other candidate). The hint changes nothing about the order or the default.
|
||||
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
Reference in new issue
Block a user