feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
CI / verify (push) Successful in 5m24s
CI / pwsh (push) Successful in 1m58s
Release / release (push) Successful in 35s

Files changed:
- CHANGES.md
- README.md
- VERSION
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_raw_fetch.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-03 09:49:20 +02:00
1 parent c4dcff76e6
commit 59c06e5ddc
11 files changed
+1009 -182

No files matched your search

+42 -4
View File
@@ -66,19 +66,57 @@ walks `raw/` recursively and works unchanged either way.
`raw/` is never chosen by hand. A file to be ingested is dropped into the top-level `incoming/`
- gitignored content, so a fresh clone finds the directory itself already there but never
anything dropped into it - flat: no subdirectory carries any classification any more. A
subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
never has to touch a caller), but it is **ignored**, never inspected:
anything dropped into it - **directly**, not into a subdirectory of it:
```bash
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
# -> raw/2026/09/handbuch.pdf
```
**A subdirectory of `incoming/` is a source of its own, accepted as a whole.** Several files
that belong together - a folder of notes, an unpacked export - keep their structure:
```bash
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/projekt-x
# incoming/projekt-x/plan.md -> raw/2026/09/projekt-x/plan.md
# incoming/projekt-x/docs/README.md -> raw/2026/09/projekt-x/docs/README.md
```
The folder name is the bundle name, so the name rule below applies to it and not to the files
inside: two `README.md` in different subfolders are no conflict. A folder is accepted alone, with
one `--fidelity`/`--authority` pair for all of it, and `incoming/projekt-x` is gone afterwards.
Every check runs before anything moves: an empty folder is refused, and so is one with a hidden
entry (a name starting with `.`), a symlink or a special file anywhere below it - each is named.
That is what keeps the clean-up safe: only directories the moves emptied are removed, so no file
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
both ways out - the whole folder, or the file moved up into `incoming/`.
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
no type any more - the kind of source comes from its content, as `source_type:` on the source
page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs
together had no way in at all.
The file's name becomes part of its path under `raw/`, and that path has the same budget as a
page's - [kb/CONTRACT.md § Titles are identifiers](../kb/CONTRACT.md#titles-are-identifiers).
`raw accept` refuses a target over it before anything moves; the fix is a shorter name in
`incoming/`.
`incoming/` - for a folder, a shorter folder name or shorter names inside it.
**`incoming/` is a queue, and `tools/wikitool raw pending` reads it** - what an ingest without an
argument works through, one entry per run:
- **Candidates** are the top-level entries only. A single file is one; top-level files sharing a
stem are one `bundle` (a `raw fetch` pair, a PDF and its converted text - the files one
`raw accept` call takes together); a folder is one, with every file below it. Dotfiles and empty
directories are none.
- **Order:** oldest first by modification time, so that newer material builds on what the wiki
already took from older, or corrects it. A bundle or folder is as new as its newest file; a tie
goes by name. The limit of this: the mtime is when a document last changed only if it was
copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or
a `raw fetch` it is merely when it was dropped. Reading each candidate for a date of its own
would not be mechanical, and a name says nothing about age.
- **The default** is the first candidate `raw accept` would take as it stands - the same checks,
run without moving anything. One it would refuse is listed with the reason and skipped: it
needs a human, not a guess.
**A bundle directory is created only from the second file onward.** One file promoted alone needs
no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one