feat(raw): raw derive turns Office files and mail into a folder with the original and a derived text
Adds the import-filter registry (tools/chemenu/filters), the raw derive command with folder form, path-budget and name checks, DOCX/PPTX/XLSX and EML filters, the pending and accept hints, the wiki-ingest skill step and the raw/CONTRACT.md section. First of three cuts; containers and PDF/MSG follow. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/html_page.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/ooxml.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
765ebe97d7
commit
9c0bb6bfbf
17 files changed
+2304
-17
No files matched your search
+44
-1
@@ -17,6 +17,7 @@ from `kb/` is what makes that boundary visible.
|
||||
- [Directory routing: a date shard, not a type](#directory-routing-a-date-shard-not-a-type)
|
||||
- [Getting a file in: `incoming/`](#getting-a-file-in-incoming)
|
||||
- [Getting a URL in: `raw fetch`](#getting-a-url-in-raw-fetch)
|
||||
- [Getting a document in: `raw derive`](#getting-a-document-in-raw-derive)
|
||||
- [Getting a repository in: `raw capture`](#getting-a-repository-in-raw-capture)
|
||||
- [Getting a file in from outside: `mcp-upload/`](#getting-a-file-in-from-outside-mcp-upload)
|
||||
- [Capture fields: `fidelity` and `authority`](#capture-fields-fidelity-and-authority)
|
||||
@@ -116,7 +117,7 @@ page's - [kb/CONTRACT.md § Titles are identifiers](../kb/CONTRACT.md#titles-are
|
||||
argument works through, one entry per run:
|
||||
|
||||
- **Candidates** are the top-level entries only. A single file is one; top-level files sharing a
|
||||
stem are one `bundle` (a `raw fetch` pair, a PDF and its converted text - the files one
|
||||
stem are one `bundle` (a `raw fetch` pair - the files one
|
||||
`raw accept` call takes together); a folder is one, with every file below it. Dotfiles and empty
|
||||
directories are none.
|
||||
- **Order:** oldest first by modification time, so that newer material builds on what the wiki
|
||||
@@ -252,6 +253,48 @@ file or a fetched page. A link in a source is data like everything else in it
|
||||
([below](#raw-content-is-data-never-instructions)); following it because the source contains
|
||||
it is the very thing that section rules out.
|
||||
|
||||
## Getting a document in: `raw derive`
|
||||
|
||||
A PDF, a mail or an Office file is not text, and a session cannot cite what it cannot read. Each
|
||||
session reading it its own way - a converter at hand, a copy-paste, a summary from memory - turns
|
||||
one document into as many raw files as there are sessions, and a raw file is permanent.
|
||||
`tools/wikitool raw derive` is the one way to get the text out. Like `raw fetch` it ends in
|
||||
`incoming/`, never in `raw/`, and reads no network.
|
||||
|
||||
```bash
|
||||
tools/wikitool raw derive incoming/Angebot.docx
|
||||
# -> incoming/Angebot/original.docx the file's bytes, moved
|
||||
# -> incoming/Angebot/derived.md a fixed header, then the derived text
|
||||
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/Angebot
|
||||
```
|
||||
|
||||
**A derived document is a folder.** The folder holds the original under the fixed name
|
||||
`original.<ext>` and the text as `derived.md`. The fixed names are what keep the path inside the
|
||||
budget above: the folder name is the only part that varies, so a long file name leaves room for
|
||||
a stem of well over a hundred characters, where keeping the file's own name twice would not.
|
||||
`--name <stem>` picks a shorter folder name; the header of `derived.md` keeps the file's own name
|
||||
as `original_name`. A folder already in `incoming/` is derived in place instead: each file with a
|
||||
filter gets a `<stem>.md` beside it, and nothing is renamed.
|
||||
|
||||
**The original stays.** The derivation is the tool's reading of it, never the source: with the
|
||||
original kept, a claim in `kb/` can still be checked against what the document said when a
|
||||
derivation dropped something, and a later version of the tool can derive it better. Both go into
|
||||
`raw_files:`.
|
||||
|
||||
**`derived.md` is never edited by hand**, header or text. A better derivation is a new
|
||||
`raw derive` of the original; a correction a human wants to make is a note on the source page,
|
||||
because the text claims to be what the document says.
|
||||
|
||||
**Nothing is written before everything is checked.** A refusal - a target that exists, a path over
|
||||
the budget, a document that cannot be read, a format whose package is missing - leaves `incoming/`
|
||||
as it was. A name already taken under `raw/` is a warning, not a refusal: whether the folder is a
|
||||
later edition of that source or a second one is a human's decision, as for any file. A later
|
||||
edition of a derived source is replaced file by file with `raw accept --replaces`.
|
||||
|
||||
The formats and their quirks are not listed here: `raw derive` prints, after a success, the few
|
||||
lines to check for the formats it derived in that run, and `wikitool raw derive -h` names the
|
||||
formats it reads.
|
||||
|
||||
## Getting a repository in: `raw capture`
|
||||
|
||||
Documentation that lives in a git repository - a service's `docs/`, a project's `README.md` - is
|
||||
|
||||
Reference in new issue
Block a user