feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
Files changed: - CHANGES.md - README.md - VERSION - instructions/ingest-large-tree.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/tests/test_raw_cmd.py - tools/chemenu/tests/test_raw_fetch.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed
1 parent
c4dcff76e6
commit
59c06e5ddc
11 files changed
+1009
-182
No files matched your search
+56
-8
@@ -115,7 +115,8 @@ sources coverage read idempotent budget:counted exit:0
|
||||
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
|
||||
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
|
||||
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
|
||||
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files from `incoming/` into `raw/`.
|
||||
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
upload show read idempotent budget:counted exit:0,1 Print one submission's manifest in full.
|
||||
upload accept write non-idempotent budget:counted exit:0,1,42 **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
|
||||
@@ -1446,20 +1447,61 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
#### `raw pending`
|
||||
|
||||
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw pending [--json]`
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: read
|
||||
- idempotent: yes
|
||||
- atomic: Read-only
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool raw pending`
|
||||
- `tools/wikitool raw pending --json`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
|
||||
**NOTES**
|
||||
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
|
||||
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool raw accept` - promotes the chosen candidate
|
||||
- `instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here
|
||||
- `raw/CONTRACT.md` "Getting a file in: incoming/" - candidates and order, and why
|
||||
|
||||
#### `raw accept`
|
||||
|
||||
Promote one or more files from `incoming/` into `raw/`.
|
||||
Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
|
||||
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
|
||||
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
@@ -1467,27 +1509,30 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
|
||||
- `tools/wikitool raw accept incoming/docker-cheatsheet.md --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
|
||||
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
||||
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
||||
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
||||
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
||||
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
|
||||
- 1 raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist
|
||||
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
|
||||
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once; for an occupied name, rename the folder in `incoming/` - there is no `--replaces` for a folder
|
||||
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
||||
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
|
||||
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
|
||||
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
|
||||
- raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
|
||||
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
|
||||
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
|
||||
|
||||
**NEVER**
|
||||
@@ -1497,7 +1542,9 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and ignored, not inspected - `raw/` does not address by type.
|
||||
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A file argument must sit directly in `incoming/`; a file inside a subdirectory is refused, since a subdirectory is a source of its own.
|
||||
- A folder argument (`incoming/<folder>`) is one source: every file below it moves to `raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the bundle name; two `README.md` in different subfolders are no conflict.
|
||||
- A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with one `--fidelity`/`--authority` pair for all of it. It is refused, before anything moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a special file sits anywhere below it; the refusal names each one.
|
||||
- One file promoted alone lands with no directory of its own; several files in one call nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.
|
||||
- `--fidelity`/`--authority` are required on a plain accept (`types describe source` lists the values); `unknown` is refused - it is backfill-only.
|
||||
- `--page "<Title>"` additionally extends that existing source page's `raw_files:` in the same call and writes both capture fields onto it - refused if it already carries a different value, since a capture field is fixed once.
|
||||
@@ -1513,6 +1560,7 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
**SEE ALSO**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
|
||||
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
|
||||
- `wikitool types describe source` - the capture field values
|
||||
- `wikitool new source` - the source page for a promoted file
|
||||
|
||||
|
||||
Reference in new issue
Block a user