feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
Files changed: - CHANGES.md - README.md - VERSION - instructions/ingest-large-tree.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/tests/test_raw_cmd.py - tools/chemenu/tests/test_raw_fetch.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
c4dcff76e6
commit
59c06e5ddc
11 files changed
+1009
-182
No files matched your search
+37
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
||||
|
||||
---
|
||||
|
||||
## 8.0.0-beta.26 - 2026-10-02 - kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
## 8.0.0-beta.27 - 2026-10-03 - incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -69,6 +69,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- new, rename, move and raw accept refuse a target whose path below the instance root is over 160 characters (UTF-16 code units); lint reports existing files over it as Long Paths (advisory) - rename each affected page with tools/wikitool rename, and shorten an incoming/ file name before raw accept
|
||||
- tools/wikitool now refuses to start (exit 42) until tools/preflight.sh (PowerShell 7: tools/preflight.ps1) has passed in the checkout - after updating, run it once: it checks Python, git and ripgrep, records their paths in .wikitool-tools.json and sets up tools/.venv
|
||||
- An instance is installed only from a release, into an empty folder (instructions/setup-instance.md); dist export, a clone of the origin repo and a private clone with the origin as upstream are no install paths any more, and wikitool upstream merge, upstream verify and instructions/private-instance.md are gone - an instance built one of those ways is reinstalled from a release into an empty folder and its kb/, raw/ and personal files are copied over. The preflight release asset installs into its own folder, which must be empty apart from the script and a .git, instead of creating a chemenu/ subfolder
|
||||
- A file in a subdirectory of incoming/ is no longer accepted - raw accept and raw fetch --html refuse it; a subdirectory is now one source, accepted whole with raw accept incoming/<folder>. Drop files directly into incoming/ and adjust any script that writes to incoming/<type>/; files still waiting in such a subdirectory are moved up into incoming/, or, if they belong together, accepted as one folder
|
||||
|
||||
**Migration:** none required - No page format changes: the title and path rules only refuse names, each affected page is renamed with tools/wikitool rename, and the removed install paths touch no page
|
||||
|
||||
@@ -79,6 +80,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- Page titles must form valid, unique file names on Windows and macOS
|
||||
- Preflight: prerequisites checked and tool paths recorded before wikitool runs (#151, POSIX half)
|
||||
- Installation only from a release, into an empty folder; upstream merge/verify and the other install paths removed (#153)
|
||||
- incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Medium impact**
|
||||
- CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
|
||||
@@ -138,6 +140,40 @@ concern - readable here, never shipped as something to parse.
|
||||
- kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
<!-- /wikitool:bumps -->
|
||||
|
||||
### incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Breaking.** A subdirectory of `incoming/` was tolerated and ignored since the date shard
|
||||
replaced the type directories, so an old `incoming/documents/` habit kept working. It protected
|
||||
nothing - the kind of source comes from its content now, as `source_type:` - and it left a folder
|
||||
of files that belong together with no way in: `raw accept` took files at most one level down,
|
||||
bundled them flat under the first file's stem, lost the folder's name and left the emptied
|
||||
directory behind, and a folder with subfolders could not be accepted at all. Now a file argument
|
||||
of `raw accept` (also with `--replaces`) and of `raw fetch --html` must sit directly in
|
||||
`incoming/`; a file inside a subdirectory is refused, and the message names both ways out.
|
||||
|
||||
**A folder is one source.** `raw accept incoming/<folder> --fidelity ... --authority ...` moves
|
||||
every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same relative path and removes the
|
||||
directories left empty. The folder name is the bundle name and falls under the existing name
|
||||
rule; the files inside do not, so several `README.md` in one tree are no conflict. It is accepted
|
||||
alone (no other argument, no `--page`, no `--replaces`), and every check runs before anything
|
||||
moves: an empty folder, or a hidden entry, a symlink or a special file anywhere below it, is
|
||||
refused with each entry named - which is what guarantees that the clean-up, an `rmdir` per
|
||||
directory and never a recursive delete, cannot take a file with it. A folder that passes the
|
||||
large-tree thresholds continues with `work new --input raw/<YYYY>/<MM>/<folder>`, which the
|
||||
success message prints.
|
||||
|
||||
**`tools/wikitool raw pending`** reads `incoming/` as a queue without changing it: the
|
||||
top-level entries, with top-level files sharing a stem as one bundle (a `raw fetch` pair) and a
|
||||
folder as one candidate, oldest first by modification time - a bundle or folder as new as its
|
||||
newest file - each marked with whether `raw accept` would take it, by the same checks, and why
|
||||
not. The default is the first acceptable one. `wiki-ingest` uses it when the user names nothing:
|
||||
it announces the entry it took, ingests exactly that one, and says at the end how many still
|
||||
wait. `raw/CONTRACT.md` § Getting a file in describes candidates, order and the limit of an
|
||||
mtime, which is a document's last change only when it was copied with its timestamps kept.
|
||||
|
||||
No migration: no page changes and `raw/` is untouched. An instance whose scripts write to
|
||||
`incoming/<type>/` changes them to write directly into `incoming/`.
|
||||
|
||||
### kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
|
||||
`kb/CONTRACT.md` still told a source page for an external article to point `raw_files:` at "the
|
||||
|
||||
Reference in new issue
Block a user