feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
Files changed: - CHANGES.md - README.md - VERSION - instructions/ingest-large-tree.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/tests/test_raw_cmd.py - tools/chemenu/tests/test_raw_fetch.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
c4dcff76e6
commit
59c06e5ddc
11 files changed
+1009
-182
No files matched your search
+42
-4
@@ -66,19 +66,57 @@ walks `raw/` recursively and works unchanged either way.
|
||||
|
||||
`raw/` is never chosen by hand. A file to be ingested is dropped into the top-level `incoming/`
|
||||
- gitignored content, so a fresh clone finds the directory itself already there but never
|
||||
anything dropped into it - flat: no subdirectory carries any classification any more. A
|
||||
subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
|
||||
never has to touch a caller), but it is **ignored**, never inspected:
|
||||
anything dropped into it - **directly**, not into a subdirectory of it:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
|
||||
# -> raw/2026/09/handbuch.pdf
|
||||
```
|
||||
|
||||
**A subdirectory of `incoming/` is a source of its own, accepted as a whole.** Several files
|
||||
that belong together - a folder of notes, an unpacked export - keep their structure:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/projekt-x
|
||||
# incoming/projekt-x/plan.md -> raw/2026/09/projekt-x/plan.md
|
||||
# incoming/projekt-x/docs/README.md -> raw/2026/09/projekt-x/docs/README.md
|
||||
```
|
||||
|
||||
The folder name is the bundle name, so the name rule below applies to it and not to the files
|
||||
inside: two `README.md` in different subfolders are no conflict. A folder is accepted alone, with
|
||||
one `--fidelity`/`--authority` pair for all of it, and `incoming/projekt-x` is gone afterwards.
|
||||
Every check runs before anything moves: an empty folder is refused, and so is one with a hidden
|
||||
entry (a name starting with `.`), a symlink or a special file anywhere below it - each is named.
|
||||
That is what keeps the clean-up safe: only directories the moves emptied are removed, so no file
|
||||
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
|
||||
both ways out - the whole folder, or the file moved up into `incoming/`.
|
||||
|
||||
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
|
||||
no type any more - the kind of source comes from its content, as `source_type:` on the source
|
||||
page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs
|
||||
together had no way in at all.
|
||||
|
||||
The file's name becomes part of its path under `raw/`, and that path has the same budget as a
|
||||
page's - [kb/CONTRACT.md § Titles are identifiers](../kb/CONTRACT.md#titles-are-identifiers).
|
||||
`raw accept` refuses a target over it before anything moves; the fix is a shorter name in
|
||||
`incoming/`.
|
||||
`incoming/` - for a folder, a shorter folder name or shorter names inside it.
|
||||
|
||||
**`incoming/` is a queue, and `tools/wikitool raw pending` reads it** - what an ingest without an
|
||||
argument works through, one entry per run:
|
||||
|
||||
- **Candidates** are the top-level entries only. A single file is one; top-level files sharing a
|
||||
stem are one `bundle` (a `raw fetch` pair, a PDF and its converted text - the files one
|
||||
`raw accept` call takes together); a folder is one, with every file below it. Dotfiles and empty
|
||||
directories are none.
|
||||
- **Order:** oldest first by modification time, so that newer material builds on what the wiki
|
||||
already took from older, or corrects it. A bundle or folder is as new as its newest file; a tie
|
||||
goes by name. The limit of this: the mtime is when a document last changed only if it was
|
||||
copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or
|
||||
a `raw fetch` it is merely when it was dropped. Reading each candidate for a date of its own
|
||||
would not be mechanical, and a name says nothing about age.
|
||||
- **The default** is the first candidate `raw accept` would take as it stands - the same checks,
|
||||
run without moving anything. One it would refuse is listed with the reason and skipped: it
|
||||
needs a human, not a guess.
|
||||
|
||||
**A bundle directory is created only from the second file onward.** One file promoted alone needs
|
||||
no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one
|
||||
|
||||
Reference in new issue
Block a user