feat(raw): raw derive turns Office files and mail into a folder with the original and a derived text
Adds the import-filter registry (tools/chemenu/filters), the raw derive command with folder form, path-budget and name checks, DOCX/PPTX/XLSX and EML filters, the pending and accept hints, the wiki-ingest skill step and the raw/CONTRACT.md section. First of three cuts; containers and PDF/MSG follow. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/html_page.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
765ebe97d7
commit
9c0bb6bfbf
17 files changed
+2304
-17
No files matched your search
@@ -6,6 +6,7 @@ Bringing new material into `raw/`: fetching and capturing it, seeing what is wai
|
||||
## Contents
|
||||
|
||||
- [`raw fetch`](#raw-fetch)
|
||||
- [`raw derive`](#raw-derive)
|
||||
- [`raw capture`](#raw-capture)
|
||||
- [`raw status`](#raw-status)
|
||||
- [`raw pending`](#raw-pending)
|
||||
@@ -82,6 +83,66 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
## `raw derive`
|
||||
|
||||
Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote.
|
||||
|
||||
**Synopsis**
|
||||
|
||||
- `wikitool raw derive incoming/<file> [--name <stem>]` - Move the file into the new folder `incoming/<stem>/` as `original.<ext>` and write `incoming/<stem>/derived.md`
|
||||
- `wikitool raw derive incoming/<folder>` - Write a `<stem>.md` beside every file in the folder that has a filter
|
||||
|
||||
**Options**
|
||||
|
||||
```text
|
||||
Arguments:
|
||||
path A file or a folder directly in incoming/ [required]
|
||||
|
||||
Options:
|
||||
--name <str> Folder stem for a single file instead of the file's own stem
|
||||
-h, --help Show this message and exit.
|
||||
```
|
||||
|
||||
**Examples**
|
||||
|
||||
```bash
|
||||
tools/wikitool raw derive incoming/report.docx
|
||||
tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz
|
||||
tools/wikitool raw derive incoming/meeting-files
|
||||
```
|
||||
|
||||
**Behaviour**
|
||||
|
||||
- Writes into `incoming/` only, never into `raw/`, and reads no network: `raw accept` promotes the folder afterwards as one source under `raw/<YYYY>/<MM>/<stem>/`.
|
||||
- A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.<ext>` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`.
|
||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`) and e-mail (`.eml`) read with the standard library only. A mail with attachments is refused.
|
||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||
- After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result.
|
||||
- Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original.
|
||||
|
||||
**Properties**
|
||||
|
||||
- Changes files: yes
|
||||
- Idempotent: no
|
||||
- Atomic: Yes for what it leaves behind - everything is read, derived and checked before the first write; the files are created exclusively, and a failure part-way puts back what this call changed
|
||||
- Counts toward the session call budget: yes
|
||||
- Can reach the network: no
|
||||
|
||||
**Exit codes**
|
||||
|
||||
- `0` - success
|
||||
- `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension
|
||||
- `1` - The target folder or a `.md` already exists, or the path would be over the path budget
|
||||
- `1` - The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed
|
||||
|
||||
**See also**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a document in: `raw derive`" - the rules and why
|
||||
- `wikitool raw accept` - promotes the folder into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a PDF, mail or Office file to ingest starts
|
||||
|
||||
## `raw capture`
|
||||
|
||||
Capture documentation from a git repository into `incoming/<bundle>/`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote.
|
||||
@@ -232,7 +293,8 @@ tools/wikitool raw pending --json
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`, `hint`.
|
||||
- A lone file that a `raw derive` filter reads (a PDF, mail, Office file or web page) is listed with the `raw derive` line for it, in text output and as `hint` in JSON (`null` for every other candidate). The hint changes nothing about the order or the default.
|
||||
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
|
||||
|
||||
**Properties**
|
||||
|
||||
Reference in new issue
Block a user