feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
Files changed: - CHANGES.md - README.md - VERSION - instructions/ingest-large-tree.md - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/tests/test_raw_cmd.py - tools/chemenu/tests/test_raw_fetch.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
c4dcff76e6
commit
59c06e5ddc
11 files changed
+1009
-182
No files matched your search
+37
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
||||
|
||||
---
|
||||
|
||||
## 8.0.0-beta.26 - 2026-10-02 - kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
## 8.0.0-beta.27 - 2026-10-03 - incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -69,6 +69,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- new, rename, move and raw accept refuse a target whose path below the instance root is over 160 characters (UTF-16 code units); lint reports existing files over it as Long Paths (advisory) - rename each affected page with tools/wikitool rename, and shorten an incoming/ file name before raw accept
|
||||
- tools/wikitool now refuses to start (exit 42) until tools/preflight.sh (PowerShell 7: tools/preflight.ps1) has passed in the checkout - after updating, run it once: it checks Python, git and ripgrep, records their paths in .wikitool-tools.json and sets up tools/.venv
|
||||
- An instance is installed only from a release, into an empty folder (instructions/setup-instance.md); dist export, a clone of the origin repo and a private clone with the origin as upstream are no install paths any more, and wikitool upstream merge, upstream verify and instructions/private-instance.md are gone - an instance built one of those ways is reinstalled from a release into an empty folder and its kb/, raw/ and personal files are copied over. The preflight release asset installs into its own folder, which must be empty apart from the script and a .git, instead of creating a chemenu/ subfolder
|
||||
- A file in a subdirectory of incoming/ is no longer accepted - raw accept and raw fetch --html refuse it; a subdirectory is now one source, accepted whole with raw accept incoming/<folder>. Drop files directly into incoming/ and adjust any script that writes to incoming/<type>/; files still waiting in such a subdirectory are moved up into incoming/, or, if they belong together, accepted as one folder
|
||||
|
||||
**Migration:** none required - No page format changes: the title and path rules only refuse names, each affected page is renamed with tools/wikitool rename, and the removed install paths touch no page
|
||||
|
||||
@@ -79,6 +80,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- Page titles must form valid, unique file names on Windows and macOS
|
||||
- Preflight: prerequisites checked and tool paths recorded before wikitool runs (#151, POSIX half)
|
||||
- Installation only from a release, into an empty folder; upstream merge/verify and the other install paths removed (#153)
|
||||
- incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Medium impact**
|
||||
- CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
|
||||
@@ -138,6 +140,40 @@ concern - readable here, never shipped as something to parse.
|
||||
- kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
<!-- /wikitool:bumps -->
|
||||
|
||||
### incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
|
||||
|
||||
**Breaking.** A subdirectory of `incoming/` was tolerated and ignored since the date shard
|
||||
replaced the type directories, so an old `incoming/documents/` habit kept working. It protected
|
||||
nothing - the kind of source comes from its content now, as `source_type:` - and it left a folder
|
||||
of files that belong together with no way in: `raw accept` took files at most one level down,
|
||||
bundled them flat under the first file's stem, lost the folder's name and left the emptied
|
||||
directory behind, and a folder with subfolders could not be accepted at all. Now a file argument
|
||||
of `raw accept` (also with `--replaces`) and of `raw fetch --html` must sit directly in
|
||||
`incoming/`; a file inside a subdirectory is refused, and the message names both ways out.
|
||||
|
||||
**A folder is one source.** `raw accept incoming/<folder> --fidelity ... --authority ...` moves
|
||||
every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same relative path and removes the
|
||||
directories left empty. The folder name is the bundle name and falls under the existing name
|
||||
rule; the files inside do not, so several `README.md` in one tree are no conflict. It is accepted
|
||||
alone (no other argument, no `--page`, no `--replaces`), and every check runs before anything
|
||||
moves: an empty folder, or a hidden entry, a symlink or a special file anywhere below it, is
|
||||
refused with each entry named - which is what guarantees that the clean-up, an `rmdir` per
|
||||
directory and never a recursive delete, cannot take a file with it. A folder that passes the
|
||||
large-tree thresholds continues with `work new --input raw/<YYYY>/<MM>/<folder>`, which the
|
||||
success message prints.
|
||||
|
||||
**`tools/wikitool raw pending`** reads `incoming/` as a queue without changing it: the
|
||||
top-level entries, with top-level files sharing a stem as one bundle (a `raw fetch` pair) and a
|
||||
folder as one candidate, oldest first by modification time - a bundle or folder as new as its
|
||||
newest file - each marked with whether `raw accept` would take it, by the same checks, and why
|
||||
not. The default is the first acceptable one. `wiki-ingest` uses it when the user names nothing:
|
||||
it announces the entry it took, ingests exactly that one, and says at the end how many still
|
||||
wait. `raw/CONTRACT.md` § Getting a file in describes candidates, order and the limit of an
|
||||
mtime, which is a document's last change only when it was copied with its timestamps kept.
|
||||
|
||||
No migration: no page changes and `raw/` is untouched. An instance whose scripts write to
|
||||
`incoming/<type>/` changes them to write directly into `incoming/`.
|
||||
|
||||
### kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
|
||||
|
||||
`kb/CONTRACT.md` still told a source page for an external article to point `raw_files:` at "the
|
||||
|
||||
@@ -90,7 +90,7 @@ chemenu/
|
||||
├── mcp-upload/ # QUARANTINE (optional): the MCP `submit` tool's write path, gitignored -
|
||||
│ # read by no command in the ordinary pipeline; a human reviews it with
|
||||
│ # `wikitool upload list/show/accept/reject`
|
||||
├── incoming/ # INBOX: flat, content gitignored - drop a file here, `raw accept` promotes it
|
||||
├── incoming/ # INBOX: content gitignored - drop a file or one folder per source here, `raw accept` promotes it
|
||||
├── raw/ # INPUT: immutable, untrusted source material
|
||||
│ ├── CONTRACT.md # Date shard, capture fields, immutability, untrusted content
|
||||
│ ├── 2026/09/ # Where `raw accept` puts a file: the month it was accepted
|
||||
@@ -185,12 +185,14 @@ working *with* it.
|
||||
|
||||
### Adding Knowledge (Ingest)
|
||||
|
||||
1. Drop a file into `incoming/` - flat, no classification to make. Everything past that
|
||||
(the destination in `raw/`, which is a `YYYY/MM` shard of the day it was accepted,
|
||||
and whether several files of one source get bundled) is computed by
|
||||
`tools/wikitool raw accept`, never chosen by hand
|
||||
2. Tell the LLM: `Ingest incoming/my-article.md`. It will ask you two things before
|
||||
promoting: how faithful the capture is (`fidelity`) and what the material may claim
|
||||
1. Drop a file into `incoming/` - directly, no classification to make. Files that belong
|
||||
together go into one folder there instead: a folder is one source, accepted whole with
|
||||
its structure kept. Everything past that (the destination in `raw/`, which is a `YYYY/MM`
|
||||
shard of the day it was accepted, and whether several files of one source get bundled)
|
||||
is computed by `tools/wikitool raw accept`, never chosen by hand
|
||||
2. Tell the LLM: `Ingest incoming/my-article.md` - or just `Ingest`, which takes the oldest
|
||||
entry waiting in `incoming/` (`tools/wikitool raw pending` lists them). It will ask you
|
||||
two things before promoting: how faithful the capture is (`fidelity`) and what the material may claim
|
||||
about its subject (`authority`). Both are recorded once and never guessed - they are
|
||||
knowable now and unrecoverable later
|
||||
3. The LLM will:
|
||||
@@ -285,7 +287,8 @@ tools/wikitool types describe entity
|
||||
|
||||
### For You (Human)
|
||||
|
||||
1. **Curate sources** - Drop files you want processed into `incoming/` (flat)
|
||||
1. **Curate sources** - Drop files you want processed into `incoming/` - directly, or one folder
|
||||
per source
|
||||
2. **Ask questions** - Query the wiki naturally
|
||||
3. **Review changes** - Check `kb/log.md` and `kb/index.md`
|
||||
4. **Direct the LLM** - Guide it on what to emphasize or investigate
|
||||
|
||||
@@ -109,7 +109,15 @@ session.
|
||||
tools/wikitool work new --input <input path>
|
||||
```
|
||||
|
||||
This derives the run key, refuses a collision instead of working around it, and writes
|
||||
`--input` must lie under `raw/`, so a tree still waiting in `incoming/` is promoted first, as
|
||||
one source and with its structure kept - `raw accept` prints the path to pass on:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/<folder>
|
||||
tools/wikitool work new --input raw/<YYYY>/<MM>/<folder>
|
||||
```
|
||||
|
||||
`work new` derives the run key, refuses a collision instead of working around it, and writes
|
||||
`README.md` + `plan.md`. Never create the directory by hand -
|
||||
[work/CONTRACT.md](../work/CONTRACT.md) explains why the run key is not a free choice.
|
||||
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
---
|
||||
name: wiki-ingest
|
||||
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki".
|
||||
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file or folder into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki" - or just "ingest" with nothing named, which takes the oldest entry waiting in incoming/.
|
||||
---
|
||||
|
||||
# Wiki Ingest
|
||||
|
||||
**Purpose:** Process a new source file and integrate its knowledge into the wiki.
|
||||
|
||||
**Trigger:** User drops a file into `incoming/` (the normal path - see step 5) or directly into
|
||||
`raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or explicitly requests
|
||||
ingestion.
|
||||
**Trigger:** User drops a file or a folder into `incoming/` (the normal path - see step 5) or
|
||||
directly into `raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or
|
||||
explicitly requests ingestion - with or without naming what (step 1 picks the entry when nothing
|
||||
is named). **One run is one source:** one file, one bundle or one folder.
|
||||
|
||||
**Before the first `wikitool` call:** `instructions/session-setup.md`.
|
||||
|
||||
@@ -45,6 +46,24 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
just wrote, for instance, which skips step 5 entirely). If it is binary or an image, note its
|
||||
presence and what it shows.
|
||||
|
||||
**The user named nothing** ("ingest", "process the inbox")? Pick the entry from the queue:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw pending
|
||||
```
|
||||
|
||||
It lists what waits in `incoming/`, oldest first, and marks the default - the oldest entry
|
||||
`raw accept` would take as it stands. **Announce it and carry on with it:** which entry, why
|
||||
this one (the oldest that can be accepted), and how many wait after it. Ask nothing here -
|
||||
step 4 is the halt before anything is written. One run takes exactly that one entry. If
|
||||
nothing acceptable is waiting, the run ends here: say so, and name each entry the listing
|
||||
marked as not acceptable, with its reason - those need the user, not a guess.
|
||||
|
||||
**The source is a folder** (`incoming/<folder>/`, named or picked)? It is one source - read
|
||||
every file in it. Whether it is ingested here or by the large-tree procedure is decided by
|
||||
the size check below, by its thresholds, not by its being a folder: three notes in a folder
|
||||
do not earn a workshop.
|
||||
|
||||
**The user named a URL instead of a file?** Fetch it into `incoming/` first - never with
|
||||
`curl` or the harness's own web fetch, which returns a model's summary rather than the page:
|
||||
|
||||
@@ -202,8 +221,14 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown)
|
||||
in the same call, so they land bundled together rather than as two independent promotions. A
|
||||
file already in `raw/` skips this step entirely. A subdirectory under `incoming/` (an old
|
||||
`incoming/<type>/` habit) is tolerated and ignored - it carries no meaning any more.
|
||||
file already in `raw/` skips this step entirely. A folder is accepted as a whole, on its own:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/<folder>
|
||||
```
|
||||
|
||||
A file inside a subdirectory of `incoming/` is refused on its own - the subdirectory is the
|
||||
source.
|
||||
|
||||
**A file that arrived through the MCP `submit` tool is not yet in `incoming/`** - it sits in
|
||||
`mcp-upload/<id>/`, a quarantine no command in this step reads. A reviewer promotes it first
|
||||
@@ -343,6 +368,9 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
deterministic count behind the "every 10 sources" cadence. If the threshold is reached,
|
||||
tell the user a full lint is due and offer to run `wiki-lint` next.
|
||||
|
||||
Then say how many entries still wait in `incoming/` (`tools/wikitool raw pending`), so the
|
||||
user knows whether another run is due.
|
||||
|
||||
## Decision points
|
||||
|
||||
- **Subject already has a page?** Update it (step 7, `touch`) instead of creating a second one.
|
||||
@@ -374,7 +402,7 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
## wikitool commands used
|
||||
|
||||
`raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
|
||||
`raw pending`, `raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
|
||||
`new project`, `new source`, `new entity`, `new concept`, `touch`, `cite add`, `xref add`,
|
||||
`xref link-source`, `sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`,
|
||||
`log status`, `publish`
|
||||
@@ -383,4 +411,5 @@ validator complains - and the ticked list is the only record that they happened.
|
||||
|
||||
Updated wiki with the source's knowledge integrated, published to `origin/main`.
|
||||
|
||||
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post"
|
||||
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post",
|
||||
"Ingest incoming/projekt-x", "Ingest" (the oldest entry waiting in `incoming/`)
|
||||
+42
-4
@@ -66,19 +66,57 @@ walks `raw/` recursively and works unchanged either way.
|
||||
|
||||
`raw/` is never chosen by hand. A file to be ingested is dropped into the top-level `incoming/`
|
||||
- gitignored content, so a fresh clone finds the directory itself already there but never
|
||||
anything dropped into it - flat: no subdirectory carries any classification any more. A
|
||||
subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
|
||||
never has to touch a caller), but it is **ignored**, never inspected:
|
||||
anything dropped into it - **directly**, not into a subdirectory of it:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
|
||||
# -> raw/2026/09/handbuch.pdf
|
||||
```
|
||||
|
||||
**A subdirectory of `incoming/` is a source of its own, accepted as a whole.** Several files
|
||||
that belong together - a folder of notes, an unpacked export - keep their structure:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/projekt-x
|
||||
# incoming/projekt-x/plan.md -> raw/2026/09/projekt-x/plan.md
|
||||
# incoming/projekt-x/docs/README.md -> raw/2026/09/projekt-x/docs/README.md
|
||||
```
|
||||
|
||||
The folder name is the bundle name, so the name rule below applies to it and not to the files
|
||||
inside: two `README.md` in different subfolders are no conflict. A folder is accepted alone, with
|
||||
one `--fidelity`/`--authority` pair for all of it, and `incoming/projekt-x` is gone afterwards.
|
||||
Every check runs before anything moves: an empty folder is refused, and so is one with a hidden
|
||||
entry (a name starting with `.`), a symlink or a special file anywhere below it - each is named.
|
||||
That is what keeps the clean-up safe: only directories the moves emptied are removed, so no file
|
||||
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
|
||||
both ways out - the whole folder, or the file moved up into `incoming/`.
|
||||
|
||||
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
|
||||
no type any more - the kind of source comes from its content, as `source_type:` on the source
|
||||
page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs
|
||||
together had no way in at all.
|
||||
|
||||
The file's name becomes part of its path under `raw/`, and that path has the same budget as a
|
||||
page's - [kb/CONTRACT.md § Titles are identifiers](../kb/CONTRACT.md#titles-are-identifiers).
|
||||
`raw accept` refuses a target over it before anything moves; the fix is a shorter name in
|
||||
`incoming/`.
|
||||
`incoming/` - for a folder, a shorter folder name or shorter names inside it.
|
||||
|
||||
**`incoming/` is a queue, and `tools/wikitool raw pending` reads it** - what an ingest without an
|
||||
argument works through, one entry per run:
|
||||
|
||||
- **Candidates** are the top-level entries only. A single file is one; top-level files sharing a
|
||||
stem are one `bundle` (a `raw fetch` pair, a PDF and its converted text - the files one
|
||||
`raw accept` call takes together); a folder is one, with every file below it. Dotfiles and empty
|
||||
directories are none.
|
||||
- **Order:** oldest first by modification time, so that newer material builds on what the wiki
|
||||
already took from older, or corrects it. A bundle or folder is as new as its newest file; a tie
|
||||
goes by name. The limit of this: the mtime is when a document last changed only if it was
|
||||
copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or
|
||||
a `raw fetch` it is merely when it was dropped. Reading each candidate for a date of its own
|
||||
would not be mechanical, and a name says nothing about age.
|
||||
- **The default** is the first candidate `raw accept` would take as it stands - the same checks,
|
||||
run without moving anything. One it would refuse is listed with the reason and skipped: it
|
||||
needs a human, not a guess.
|
||||
|
||||
**A bundle directory is created only from the second file onward.** One file promoted alone needs
|
||||
no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one
|
||||
|
||||
+56
-8
@@ -115,7 +115,8 @@ sources coverage read idempotent budget:counted exit:0
|
||||
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
|
||||
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
|
||||
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
|
||||
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files from `incoming/` into `raw/`.
|
||||
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
|
||||
upload show read idempotent budget:counted exit:0,1 Print one submission's manifest in full.
|
||||
upload accept write non-idempotent budget:counted exit:0,1,42 **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
|
||||
@@ -1446,20 +1447,61 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- `wikitool raw accept` - promotes the written files into `raw/`
|
||||
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
|
||||
|
||||
#### `raw pending`
|
||||
|
||||
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw pending [--json]`
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: read
|
||||
- idempotent: yes
|
||||
- atomic: Read-only
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
**EXAMPLES**
|
||||
|
||||
- `tools/wikitool raw pending`
|
||||
- `tools/wikitool raw pending --json`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
|
||||
**NOTES**
|
||||
|
||||
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
|
||||
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
|
||||
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
|
||||
- The default is the first candidate `raw accept` would take, and the output names it.
|
||||
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
|
||||
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
|
||||
|
||||
**SEE ALSO**
|
||||
|
||||
- `wikitool raw accept` - promotes the chosen candidate
|
||||
- `instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here
|
||||
- `raw/CONTRACT.md` "Getting a file in: incoming/" - candidates and order, and why
|
||||
|
||||
#### `raw accept`
|
||||
|
||||
Promote one or more files from `incoming/` into `raw/`.
|
||||
Promote one or more files, or one folder, from `incoming/` into `raw/`.
|
||||
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
|
||||
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
|
||||
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
|
||||
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
|
||||
- budget: counted
|
||||
- network: no
|
||||
|
||||
@@ -1467,27 +1509,30 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
|
||||
- `tools/wikitool raw accept incoming/docker-cheatsheet.md --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
|
||||
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
|
||||
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
|
||||
|
||||
**EXIT STATUS**
|
||||
|
||||
- 0 success
|
||||
- 1 raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
|
||||
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
|
||||
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
|
||||
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
|
||||
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
|
||||
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
|
||||
- 1 raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist
|
||||
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
|
||||
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
|
||||
|
||||
**ON FAILURE**
|
||||
|
||||
- raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
|
||||
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once; for an occupied name, rename the folder in `incoming/` - there is no `--replaces` for a folder
|
||||
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
|
||||
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
|
||||
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
|
||||
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
|
||||
- raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
|
||||
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
|
||||
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
|
||||
|
||||
**NEVER**
|
||||
@@ -1497,7 +1542,9 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
|
||||
**NOTES**
|
||||
|
||||
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and ignored, not inspected - `raw/` does not address by type.
|
||||
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A file argument must sit directly in `incoming/`; a file inside a subdirectory is refused, since a subdirectory is a source of its own.
|
||||
- A folder argument (`incoming/<folder>`) is one source: every file below it moves to `raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the bundle name; two `README.md` in different subfolders are no conflict.
|
||||
- A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with one `--fidelity`/`--authority` pair for all of it. It is refused, before anything moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a special file sits anywhere below it; the refusal names each one.
|
||||
- One file promoted alone lands with no directory of its own; several files in one call nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.
|
||||
- `--fidelity`/`--authority` are required on a plain accept (`types describe source` lists the values); `unknown` is refused - it is backfill-only.
|
||||
- `--page "<Title>"` additionally extends that existing source page's `raw_files:` in the same call and writes both capture fields onto it - refused if it already carries a different value, since a capture field is fixed once.
|
||||
@@ -1513,6 +1560,7 @@ Promote one or more files from `incoming/` into `raw/`.
|
||||
**SEE ALSO**
|
||||
|
||||
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
|
||||
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
|
||||
- `wikitool types describe source` - the capture field values
|
||||
- `wikitool new source` - the source page for a promoted file
|
||||
|
||||
|
||||
@@ -259,7 +259,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
"sources coverage", "sources trace", "sources rebuild-index",
|
||||
)),
|
||||
("Raw material and uploads", (
|
||||
"raw fetch", "raw accept", "upload list", "upload show", "upload accept", "upload reject",
|
||||
"raw fetch", "raw pending", "raw accept",
|
||||
"upload list", "upload show", "upload accept", "upload reject",
|
||||
)),
|
||||
("Git", (
|
||||
"sync", "publish",
|
||||
|
||||
+506
-111
@@ -2,17 +2,22 @@
|
||||
`raw/`, with the destination computed rather than chosen by hand (Gitea #58),
|
||||
sharded by the accept date rather than by a hand-picked type (Gitea #67).
|
||||
|
||||
A human no longer classifies a file at all: `incoming/` is flat, and a
|
||||
subdirectory dropped under it (an old `incoming/documents/` habit, a script
|
||||
that still writes one) is accepted and ignored rather than inspected -
|
||||
promoting `raw/` from a routing decision to an address computed purely from
|
||||
*when* the file was accepted:
|
||||
A human no longer classifies a file at all - `raw/` is an address computed
|
||||
purely from *when* the file was accepted, and the kind of source comes from
|
||||
its content (`source_type:` on the source page). So a subdirectory of
|
||||
`incoming/` carries no type any more, and since Gitea #112 it is not tolerated
|
||||
as one either: it **is** a source, accepted as a whole. A file argument must
|
||||
sit directly in `incoming/`:
|
||||
|
||||
- **Single file, no bundle.** One file promoted alone lands as
|
||||
`raw/<YYYY>/<MM>/<name>` - no directory of its own.
|
||||
- **Bundle from the second file on.** Several files of one source promoted in
|
||||
the same call land under `raw/<YYYY>/<MM>/<stem>/`, named after the first
|
||||
file's stem.
|
||||
- **A folder is one source** (Gitea #112). `raw accept incoming/<folder>`
|
||||
moves every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same
|
||||
relative path, then removes the directories that are left empty. The folder
|
||||
name is the bundle name.
|
||||
- **Growing an existing single file into a bundle.** `--page` extends an
|
||||
existing source page's `raw_files:`. If that raises the page from one file
|
||||
to more than one, the file it already had is folded into a bundle at its
|
||||
@@ -20,6 +25,11 @@ promoting `raw/` from a routing decision to an address computed purely from
|
||||
today's shard, so a bundle never mixes an old capture date with today's
|
||||
(Gitea #67 decision, "Datums-Shard" § "Bündelort").
|
||||
|
||||
`raw pending` reads the same queue without changing it: the candidates in
|
||||
`incoming/` oldest first, each judged by the very checks `raw accept` runs
|
||||
before it moves anything (`_Refused` is how those checks report without
|
||||
exiting).
|
||||
|
||||
Existing files under `raw/` are never moved by this change (Gitea #67
|
||||
"Altbestand bleibt stehen"): `raw/articles/`, `raw/documents/`, `raw/notes/`
|
||||
and `raw/assets/` keep whatever they already held, and stay valid promotion
|
||||
@@ -65,7 +75,10 @@ and names both routes rather than choosing one (Gitea #64 decision 2).
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
@@ -73,7 +86,7 @@ import typer
|
||||
from rich.markup import escape
|
||||
|
||||
from chemenu import cli_contract, config, web_capture
|
||||
from chemenu.commands._util import check_path_budget, fail, rel_path, success
|
||||
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
|
||||
from chemenu.errors import ChemenuError
|
||||
from chemenu.frontmatter_io import write_page
|
||||
from chemenu.kb_scan import load_kb_pages
|
||||
@@ -107,33 +120,51 @@ def _shard_dir(today: Optional[datetime.date] = None) -> Path:
|
||||
return config.RAW_DIR / f"{d.year:04d}" / f"{d.month:02d}"
|
||||
|
||||
|
||||
def _validate_under_incoming(path: Path, incoming: Path) -> None:
|
||||
"""`path` must sit directly in `incoming/`, or exactly one level below it.
|
||||
class _Refused(Exception):
|
||||
"""A check that runs before anything moves has failed; the message is what
|
||||
`fail()` prints. Raised rather than failing on the spot so `raw pending`
|
||||
can ask the same checks of every candidate without exiting - one set of
|
||||
checks, two callers (Gitea #112)."""
|
||||
|
||||
Unlike before #67, that one optional level carries no meaning any more -
|
||||
it is accepted and ignored, kept only so an old `incoming/<type>/` habit
|
||||
or script does not have to change to keep working (the MINOR condition
|
||||
named in Gitea #67 "Versionsteil"). Nesting deeper than that is still
|
||||
refused: it was never meaningful and silently accepting it would hide a
|
||||
typo'd path.
|
||||
|
||||
def _refusals_fail(fn, *args):
|
||||
try:
|
||||
return fn(*args)
|
||||
except _Refused as exc:
|
||||
fail(escape(str(exc)))
|
||||
|
||||
|
||||
def _check_directly_in_incoming(path: Path, incoming: Path) -> None:
|
||||
"""`path` must sit directly in `incoming/`.
|
||||
|
||||
Up to Gitea #112 one subdirectory level was tolerated and ignored, so an
|
||||
old `incoming/<type>/` habit kept working after #67 stopped reading the
|
||||
type from it. A subdirectory is a source of its own now - accepted as a
|
||||
whole - so a file inside one is refused with both ways out named.
|
||||
"""
|
||||
try:
|
||||
rel = path.relative_to(incoming)
|
||||
except ValueError:
|
||||
fail(
|
||||
raise _Refused(
|
||||
f"{rel_path(path)} is not under incoming/ - `raw accept` and `raw fetch --html` "
|
||||
"only take files from there. See raw/CONTRACT.md."
|
||||
)
|
||||
) from None
|
||||
if len(rel.parts) < 1:
|
||||
fail(f"incoming/{rel.as_posix()} names no file.")
|
||||
if len(rel.parts) > 2:
|
||||
fail(
|
||||
f"incoming/{rel.as_posix()} is nested more than one level below incoming/ - place it "
|
||||
"directly in incoming/, or in at most one subdirectory of it (the "
|
||||
"subdirectory itself is ignored, see raw/CONTRACT.md)."
|
||||
raise _Refused(f"incoming/{rel.as_posix()} names no file.")
|
||||
if len(rel.parts) > 1:
|
||||
top = rel.parts[0]
|
||||
raise _Refused(
|
||||
f"incoming/{rel.as_posix()} lies in a subdirectory of incoming/ - a file is taken "
|
||||
"only from directly inside incoming/, and a subdirectory is a source of its own. "
|
||||
f"Either accept the whole folder as one source (raw accept incoming/{top}), or move "
|
||||
"the file up into incoming/ and accept it there. See raw/CONTRACT.md."
|
||||
)
|
||||
|
||||
|
||||
def _validate_under_incoming(path: Path, incoming: Path) -> None:
|
||||
_refusals_fail(_check_directly_in_incoming, path, incoming)
|
||||
|
||||
|
||||
_YEAR_DIR = re.compile(r"\d{4}")
|
||||
|
||||
|
||||
@@ -185,6 +216,180 @@ def _stem_collision_message(claimed_name: str, holder: Path) -> str:
|
||||
)
|
||||
|
||||
|
||||
def _check_files(resolved: list[Path]) -> None:
|
||||
incoming = _incoming_dir()
|
||||
for path in resolved:
|
||||
if not path.is_file():
|
||||
raise _Refused(f"{rel_path(path)} does not exist or is not a file.")
|
||||
_check_directly_in_incoming(path, incoming)
|
||||
names = [path.name for path in resolved]
|
||||
if len(names) != len(set(names)):
|
||||
raise _Refused("Two files share a filename; rename one before promoting.")
|
||||
|
||||
|
||||
def _check_target(dst: Path, remedy: str) -> None:
|
||||
problem = path_budget_problem_for(dst)
|
||||
if problem:
|
||||
raise _Refused(f"Cannot write {problem}. {remedy}")
|
||||
if dst.exists():
|
||||
raise _Refused(f"Cannot promote: {rel_path(dst)} already exists.")
|
||||
|
||||
|
||||
def _plan_file_moves(
|
||||
resolved: list[Path], existing_raw_paths: list[Path], page: Optional[str]
|
||||
) -> list[tuple[Path, Path]]:
|
||||
"""Where each file goes - the bundle decision, the path budget, an existing
|
||||
target and the name rule, all checked before anything moves. `page` and
|
||||
`existing_raw_paths` are the `--page` case; `raw pending` asks with
|
||||
neither."""
|
||||
# A bundle directory forms once two or more files belong to the source
|
||||
# (Gitea #58 decision 3): from the second file on, never before. Whenever
|
||||
# --page targets an existing page it always has >=1 raw file already
|
||||
# (types/source.md requires raw_files:), so bundling always applies there.
|
||||
total = len(existing_raw_paths) + len(resolved)
|
||||
bundle_dir: Optional[Path] = None
|
||||
if len(existing_raw_paths) >= 2:
|
||||
parents = {p.parent for p in existing_raw_paths}
|
||||
if len(parents) != 1:
|
||||
raise _Refused(
|
||||
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
|
||||
"(see `sources coverage`)."
|
||||
)
|
||||
bundle_dir = parents.pop()
|
||||
elif total >= 2:
|
||||
if existing_raw_paths:
|
||||
# Growing a bundle out of a single already-promoted file (Gitea
|
||||
# #67 decision): the bundle forms at that file's own parent
|
||||
# directory, never at today's shard - the file's capture date is
|
||||
# whatever it always was, and a bundle mixing an old and a new
|
||||
# shard would have no single correct address.
|
||||
primary = existing_raw_paths[0]
|
||||
bundle_dir = primary.parent / primary.stem
|
||||
else:
|
||||
primary = resolved[0]
|
||||
bundle_dir = _shard_dir() / primary.stem
|
||||
|
||||
moves: list[tuple[Path, Path]] = []
|
||||
for existing in existing_raw_paths:
|
||||
if bundle_dir is not None and existing.parent != bundle_dir:
|
||||
moves.append((existing, bundle_dir / existing.name))
|
||||
for new_path in resolved:
|
||||
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
|
||||
moves.append((new_path, dst))
|
||||
|
||||
for _src, dst in moves:
|
||||
_check_target(
|
||||
dst,
|
||||
"The name comes from the file in incoming/: rename it there to something shorter "
|
||||
"and accept it again.",
|
||||
)
|
||||
|
||||
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
|
||||
# call is about to claim - the bundle's name, or the lone file's stem when
|
||||
# no bundle forms - must not already belong to something this call does
|
||||
# not itself own. "Owns" means: one of the page's already-registered raw
|
||||
# files (the pitfall from the module docstring - a single file growing
|
||||
# into a bundle of its own name momentarily still occupies that name), or,
|
||||
# once a bundle already has >=2 registered files, the bundle directory
|
||||
# itself.
|
||||
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
|
||||
occupied = _occupied_stems(config.RAW_DIR)
|
||||
owned = set(existing_raw_paths)
|
||||
if len(existing_raw_paths) >= 2:
|
||||
owned.add(bundle_dir)
|
||||
holder = occupied.get(claimed_name)
|
||||
if holder is not None and holder not in owned:
|
||||
raise _Refused(_stem_collision_message(claimed_name, holder))
|
||||
return moves
|
||||
|
||||
|
||||
def _walk_folder(folder: Path) -> tuple[list[Path], list[Path], list[str]]:
|
||||
"""Every file below `folder`, every directory (`folder` included), and every
|
||||
entry `raw accept` refuses to take - hidden or a symlink anywhere, or
|
||||
neither a file nor a directory. Never descends into a refused entry, and
|
||||
never follows a link."""
|
||||
files: list[Path] = []
|
||||
dirs: list[Path] = [folder]
|
||||
refused: list[str] = []
|
||||
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
|
||||
base = Path(dirpath)
|
||||
descend = []
|
||||
for name in sorted(dirnames + filenames):
|
||||
entry = base / name
|
||||
if name.startswith("."):
|
||||
refused.append(f"{rel_path(entry)} (hidden)")
|
||||
elif entry.is_symlink():
|
||||
refused.append(f"{rel_path(entry)} (symlink)")
|
||||
elif entry.is_dir():
|
||||
dirs.append(entry)
|
||||
descend.append(name)
|
||||
elif entry.is_file():
|
||||
files.append(entry)
|
||||
else:
|
||||
refused.append(f"{rel_path(entry)} (neither a file nor a directory)")
|
||||
dirnames[:] = descend
|
||||
return sorted(files), dirs, refused
|
||||
|
||||
|
||||
def _folder_collision_message(name: str, holder: Path) -> str:
|
||||
return (
|
||||
f'{rel_path(holder)} already claims the name "{name}" under raw/.\n'
|
||||
" Rename the folder in incoming/ (add a distinguishing suffix) and accept it again.\n"
|
||||
" A folder has no --replaces: a later edition of one file in it is replaced file by file."
|
||||
)
|
||||
|
||||
|
||||
def _plan_folder(folder: Path) -> tuple[Path, list[tuple[Path, Path]], list[Path]]:
|
||||
"""The bundle directory for `raw accept incoming/<folder>`, every move into
|
||||
it, and the directories to remove afterwards, deepest first - or
|
||||
`_Refused`, before anything moves.
|
||||
|
||||
The invariant the cleanup rests on (Gitea #112): after the moves no file
|
||||
is left below `folder`, so removing the emptied directories can never
|
||||
take one with it. That holds only because every entry the move would
|
||||
skip - hidden, a symlink, a special file - is refused here, up front."""
|
||||
incoming = _incoming_dir()
|
||||
try:
|
||||
rel = folder.relative_to(incoming)
|
||||
except ValueError:
|
||||
raise _Refused(
|
||||
f"{rel_path(folder)} is not under incoming/ - `raw accept` only takes a folder from "
|
||||
"there. See raw/CONTRACT.md."
|
||||
) from None
|
||||
if len(rel.parts) != 1:
|
||||
raise _Refused(
|
||||
f"incoming/{rel.as_posix()} is not directly in incoming/ - a folder is accepted only "
|
||||
f"as a whole, from the top: raw accept incoming/{rel.parts[0]}"
|
||||
)
|
||||
if folder.name.startswith(".") or folder.is_symlink():
|
||||
raise _Refused(f"{rel_path(folder)} is hidden or a symlink - raw accept does not take it.")
|
||||
|
||||
files, dirs, refused = _walk_folder(folder)
|
||||
if refused:
|
||||
listed = "\n".join(f" - {entry}" for entry in refused)
|
||||
raise _Refused(
|
||||
f"{rel_path(folder)}/ holds entries raw accept does not take - hidden entries, "
|
||||
f"symlinks and special files are refused, so nothing is left behind:\n{listed}\n"
|
||||
" Remove or replace them in incoming/, then accept the folder again."
|
||||
)
|
||||
if not files:
|
||||
raise _Refused(f"{rel_path(folder)}/ holds no file - nothing to accept.")
|
||||
|
||||
holder = _occupied_stems(config.RAW_DIR).get(folder.name)
|
||||
if holder is not None:
|
||||
raise _Refused(_folder_collision_message(folder.name, holder))
|
||||
|
||||
bundle_dir = _shard_dir() / folder.name
|
||||
moves = [(src, bundle_dir / src.relative_to(folder)) for src in files]
|
||||
for _src, dst in moves:
|
||||
_check_target(
|
||||
dst,
|
||||
"The path comes from the folder in incoming/: shorten the folder's name or the "
|
||||
"names inside it, and accept it again.",
|
||||
)
|
||||
return bundle_dir, moves, sorted(dirs, key=lambda d: len(d.parts), reverse=True)
|
||||
|
||||
|
||||
def _capture_choices() -> tuple[list[str], list[str]]:
|
||||
"""Allowed `--fidelity`/`--authority` values, straight from the schema
|
||||
(single source of truth) - `unknown` excluded, since it is backfill-only
|
||||
@@ -312,9 +517,55 @@ def _replace(
|
||||
success("Review them in this same run: the replacement and their update belong in one commit.")
|
||||
|
||||
|
||||
def _accept_folder(folder: Path, fidelity: str, authority: str, dry_run: bool) -> None:
|
||||
"""`raw accept incoming/<folder>` - one folder, one source (Gitea #112).
|
||||
|
||||
Not atomic: one move per file, then the emptied directories. A failure
|
||||
part-way leaves a half-accepted folder, which is reported, not resumed -
|
||||
the call is not repeated (tool error contract, case 4)."""
|
||||
bundle_dir, moves, dirs = _refusals_fail(_plan_folder, folder)
|
||||
|
||||
if dry_run:
|
||||
for src, dst in moves:
|
||||
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
|
||||
typer.echo(f"[dry-run] would remove {rel_path(folder)}/ once it is empty")
|
||||
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
|
||||
return
|
||||
|
||||
for src, dst in moves:
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
src.rename(dst)
|
||||
typer.echo(f" moved {rel_path(src)} -> {rel_path(dst)}")
|
||||
# rmdir, never a recursive delete: it refuses a directory that still holds
|
||||
# anything, so this step can only ever remove what the moves emptied.
|
||||
left = []
|
||||
for directory in dirs:
|
||||
try:
|
||||
directory.rmdir()
|
||||
except OSError:
|
||||
left.append(directory)
|
||||
if left:
|
||||
typer.echo(
|
||||
f"WARN {', '.join(rel_path(d) for d in left)} could not be removed - something appeared "
|
||||
"in it during the move. Every file of the folder was moved; look at what is left."
|
||||
)
|
||||
|
||||
raw_files_arg = ",".join(rel_path(dst) for _src, dst in moves)
|
||||
success(
|
||||
f"Promoted {rel_path(folder)}/ ({len(moves)} file(s)) to {rel_path(bundle_dir)}/.\n"
|
||||
" Next:\n"
|
||||
" tools/wikitool new source --name \"<Title>\" \\\n"
|
||||
f" --set raw_files={raw_files_arg} \\\n"
|
||||
f" --set fidelity={fidelity} --set authority={authority} \\\n"
|
||||
" --set source_type=<category>\n"
|
||||
" Past the thresholds in instructions/ingest-large-tree.md, continue there instead:\n"
|
||||
f" tools/wikitool work new --input {rel_path(bundle_dir)}"
|
||||
)
|
||||
|
||||
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="raw accept",
|
||||
summary="Promote one or more files from `incoming/` into `raw/`.",
|
||||
summary="Promote one or more files, or one folder, from `incoming/` into `raw/`.",
|
||||
synopsis=(
|
||||
cli_contract.Variant(
|
||||
usage='raw accept <file> [<file> ...] --fidelity <v> --authority <v> '
|
||||
@@ -322,6 +573,11 @@ def _replace(
|
||||
notes="Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` "
|
||||
"shard",
|
||||
),
|
||||
cli_contract.Variant(
|
||||
usage="raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]",
|
||||
notes="Promote a whole folder as one source, its structure kept, into "
|
||||
"`raw/<YYYY>/<MM>/<folder>/`",
|
||||
),
|
||||
cli_contract.Variant(
|
||||
usage='raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] '
|
||||
"[--dry-run]",
|
||||
@@ -332,14 +588,23 @@ def _replace(
|
||||
effect=cli_contract.Effect.WRITE,
|
||||
idempotent=cli_contract.Idempotent.NO,
|
||||
atomic="`raw accept`: No - one filesystem move per file, then (with `--page`) one page "
|
||||
"write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if "
|
||||
"`--fidelity`/`--authority` was given) one page write",
|
||||
"write. With a folder: No - one move per file, then one `rmdir` per emptied directory; "
|
||||
"a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + "
|
||||
"one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write",
|
||||
budget=cli_contract.Budget.COUNTED,
|
||||
),
|
||||
notes=(
|
||||
"Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept "
|
||||
"date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and "
|
||||
"ignored, not inspected - `raw/` does not address by type.",
|
||||
"date rather than chosen by hand. A file argument must sit directly in `incoming/`; a "
|
||||
"file inside a subdirectory is refused, since a subdirectory is a source of its own.",
|
||||
"A folder argument (`incoming/<folder>`) is one source: every file below it moves to "
|
||||
"`raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty "
|
||||
"are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the "
|
||||
"bundle name; two `README.md` in different subfolders are no conflict.",
|
||||
"A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with "
|
||||
"one `--fidelity`/`--authority` pair for all of it. It is refused, before anything "
|
||||
"moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a "
|
||||
"special file sits anywhere below it; the refusal names each one.",
|
||||
"One file promoted alone lands with no directory of its own; several files in one call "
|
||||
"nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.",
|
||||
"`--fidelity`/`--authority` are required on a plain accept (`types describe source` "
|
||||
@@ -373,13 +638,23 @@ def _replace(
|
||||
failures=(
|
||||
cli_contract.Failure(
|
||||
label="raw accept",
|
||||
cause="A file does not exist, is not under `incoming/`, or is nested more than one "
|
||||
"level below it; two files in one call share a filename; a target path already "
|
||||
"exists; or a target path would be over the path budget (160 UTF-16 code units "
|
||||
"below the instance root)",
|
||||
reaction="Fix the named argument and retry once. For a path over the budget, "
|
||||
"rename the file in `incoming/` to something shorter - the refusal comes before "
|
||||
"anything moves, so `incoming/` and `raw/` are unchanged",
|
||||
cause="A file does not exist or is not directly in `incoming/`; two files in one "
|
||||
"call share a filename; a target path already exists; or a target path would be "
|
||||
"over the path budget (160 UTF-16 code units below the instance root)",
|
||||
reaction="Fix the named argument and retry once. A file inside a subdirectory of "
|
||||
"`incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or "
|
||||
"moved up into `incoming/` first. For a path over the budget, rename the file in "
|
||||
"`incoming/` to something shorter - the refusal comes before anything moves, so "
|
||||
"`incoming/` and `raw/` are unchanged",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
label="raw accept incoming/<folder>",
|
||||
cause="The folder is not directly in `incoming/`, is combined with another argument, "
|
||||
"`--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a "
|
||||
"special file; a target path is over the budget; or the folder name is already "
|
||||
"occupied under `raw/`",
|
||||
reaction="Nothing moved. Fix what the message names and retry once; for an occupied "
|
||||
"name, rename the folder in `incoming/` - there is no `--replaces` for a folder",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
label="raw accept",
|
||||
@@ -411,9 +686,9 @@ def _replace(
|
||||
),
|
||||
cli_contract.Failure(
|
||||
label="raw accept --replaces",
|
||||
cause="The incoming file does not exist or is not under `incoming/` (or is nested "
|
||||
"more than one level below it), its filename differs from the target's, or the "
|
||||
"target does not lie under `raw/` or does not exist",
|
||||
cause="The incoming file does not exist or is not directly in `incoming/`, its "
|
||||
"filename differs from the target's, or the target does not lie under `raw/` or "
|
||||
"does not exist",
|
||||
reaction="Fix the named argument and retry once - every check runs before the "
|
||||
"filesystem is touched, so both files are exactly as they were",
|
||||
),
|
||||
@@ -429,6 +704,7 @@ def _replace(
|
||||
"--authority reporting",
|
||||
'tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting '
|
||||
'--page "Source - Docker Cheatsheet"',
|
||||
"tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting",
|
||||
"tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md",
|
||||
),
|
||||
never=(
|
||||
@@ -439,6 +715,7 @@ def _replace(
|
||||
),
|
||||
see_also=(
|
||||
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - the rules and why",
|
||||
"`wikitool raw pending` - what is waiting in `incoming/`, and which entry is next",
|
||||
"`wikitool types describe source` - the capture field values",
|
||||
"`wikitool new source` - the source page for a promoted file",
|
||||
),
|
||||
@@ -447,7 +724,8 @@ def _replace(
|
||||
def raw_accept_command(
|
||||
files: list[Path] = typer.Argument(
|
||||
...,
|
||||
help="One or more files under incoming/, all belonging to the same source",
|
||||
help="One or more files directly in incoming/, all belonging to the same source - or "
|
||||
"one folder in incoming/, accepted whole as one source",
|
||||
),
|
||||
fidelity: Optional[str] = typer.Option(
|
||||
None,
|
||||
@@ -476,13 +754,19 @@ def raw_accept_command(
|
||||
),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
|
||||
):
|
||||
"""Promote file(s) from incoming/ into raw/, computing the destination
|
||||
(date shard, bundle or not, bundle name) instead of taking it as an
|
||||
argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
|
||||
"""Promote file(s) or one folder from incoming/ into raw/, computing the
|
||||
destination (date shard, bundle or not, bundle name) instead of taking it
|
||||
as an argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
|
||||
if not files:
|
||||
fail("Pass at least one file to promote.")
|
||||
|
||||
incoming = _incoming_dir()
|
||||
resolved = [_resolve(f) for f in files]
|
||||
folders = [p for p in resolved if p.is_dir()]
|
||||
if folders and (len(files) != 1 or page is not None or replaces is not None):
|
||||
fail(escape(
|
||||
f"{rel_path(folders[0])} is a folder, and a folder is accepted alone: one source, "
|
||||
"with no other argument, no --page and no --replaces. Accept it in a call of its own."
|
||||
))
|
||||
|
||||
if replaces is not None:
|
||||
_replace(files, replaces, page, fidelity, authority, dry_run)
|
||||
@@ -498,16 +782,11 @@ def raw_accept_command(
|
||||
_check_capture_value("fidelity", fidelity, fidelity_choices)
|
||||
_check_capture_value("authority", authority, authority_choices)
|
||||
|
||||
resolved = [_resolve(f) for f in files]
|
||||
if folders:
|
||||
_accept_folder(folders[0], fidelity, authority, dry_run)
|
||||
return
|
||||
|
||||
for path in resolved:
|
||||
if not path.is_file():
|
||||
fail(f"{rel_path(path)} does not exist or is not a file.")
|
||||
_validate_under_incoming(path, incoming)
|
||||
|
||||
names = [path.name for path in resolved]
|
||||
if len(names) != len(set(names)):
|
||||
fail("Two files share a filename; rename one before promoting.")
|
||||
_refusals_fail(_check_files, resolved)
|
||||
|
||||
pages = None
|
||||
target_page = None
|
||||
@@ -532,66 +811,7 @@ def raw_accept_command(
|
||||
"before promoting more."
|
||||
)
|
||||
|
||||
# A bundle directory forms once two or more files belong to the source
|
||||
# (Gitea #58 decision 3): from the second file on, never before. Whenever
|
||||
# --page targets an existing page it always has >=1 raw file already
|
||||
# (types/source.md requires raw_files:), so bundling always applies there.
|
||||
total = len(existing_raw_paths) + len(resolved)
|
||||
bundle_dir: Optional[Path] = None
|
||||
if len(existing_raw_paths) >= 2:
|
||||
parents = {p.parent for p in existing_raw_paths}
|
||||
if len(parents) != 1:
|
||||
fail(
|
||||
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
|
||||
"(see `sources coverage`)."
|
||||
)
|
||||
bundle_dir = parents.pop()
|
||||
elif total >= 2:
|
||||
if existing_raw_paths:
|
||||
# Growing a bundle out of a single already-promoted file (Gitea
|
||||
# #67 decision): the bundle forms at that file's own parent
|
||||
# directory, never at today's shard - the file's capture date is
|
||||
# whatever it always was, and a bundle mixing an old and a new
|
||||
# shard would have no single correct address.
|
||||
primary = existing_raw_paths[0]
|
||||
bundle_dir = primary.parent / primary.stem
|
||||
else:
|
||||
primary = resolved[0]
|
||||
bundle_dir = _shard_dir() / primary.stem
|
||||
|
||||
moves: list[tuple[Path, Path]] = []
|
||||
for existing in existing_raw_paths:
|
||||
if bundle_dir is not None and existing.parent != bundle_dir:
|
||||
moves.append((existing, bundle_dir / existing.name))
|
||||
for new_path in resolved:
|
||||
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
|
||||
moves.append((new_path, dst))
|
||||
|
||||
for _src, dst in moves:
|
||||
check_path_budget(
|
||||
dst,
|
||||
"The name comes from the file in incoming/: rename it there to something shorter "
|
||||
"and accept it again.",
|
||||
)
|
||||
if dst.exists():
|
||||
fail(f"Cannot promote: {rel_path(dst)} already exists.")
|
||||
|
||||
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
|
||||
# call is about to claim - the bundle's name, or the lone file's stem when
|
||||
# no bundle forms - must not already belong to something this call does
|
||||
# not itself own. "Owns" means: one of the page's already-registered raw
|
||||
# files (the pitfall from the module docstring - a single file growing
|
||||
# into a bundle of its own name momentarily still occupies that name), or,
|
||||
# once a bundle already has >=2 registered files, the bundle directory
|
||||
# itself.
|
||||
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
|
||||
occupied = _occupied_stems(config.RAW_DIR)
|
||||
owned = set(existing_raw_paths)
|
||||
if len(existing_raw_paths) >= 2:
|
||||
owned.add(bundle_dir)
|
||||
holder = occupied.get(claimed_name)
|
||||
if holder is not None and holder not in owned:
|
||||
fail(_stem_collision_message(claimed_name, holder))
|
||||
moves = _refusals_fail(_plan_file_moves, resolved, existing_raw_paths, page)
|
||||
|
||||
moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
|
||||
if moving_existing:
|
||||
@@ -663,6 +883,181 @@ def raw_accept_command(
|
||||
)
|
||||
|
||||
|
||||
# --- raw pending --------------------------------------------------------------
|
||||
#
|
||||
# `incoming/` read as a queue (Gitea #112): what `wiki-ingest` without an
|
||||
# argument works through, one candidate per run, oldest first so that newer
|
||||
# material builds on - or corrects - what the wiki already took from older.
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Candidate:
|
||||
kind: str # file | bundle | folder
|
||||
paths: tuple[Path, ...]
|
||||
files: int
|
||||
mtime_ns: int
|
||||
reason: Optional[str] # None when `raw accept` would take it as it stands
|
||||
|
||||
def label(self) -> str:
|
||||
return ", ".join(rel_path(p) + ("/" if self.kind == "folder" else "") for p in self.paths)
|
||||
|
||||
def mtime_iso(self) -> str:
|
||||
stamp = datetime.datetime.fromtimestamp(self.mtime_ns / 1e9).astimezone()
|
||||
return stamp.isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _refusal(fn, *args) -> Optional[str]:
|
||||
try:
|
||||
fn(*args)
|
||||
except _Refused as exc:
|
||||
return str(exc)
|
||||
return None
|
||||
|
||||
|
||||
def _folder_contents(folder: Path) -> tuple[int, int]:
|
||||
"""How many non-directory entries sit below `folder`, and the newest
|
||||
mtime among them - `lstat` only, no content read and no link followed."""
|
||||
count, newest = 0, 0
|
||||
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
|
||||
base = Path(dirpath)
|
||||
for name in filenames + [d for d in dirnames if (base / d).is_symlink()]:
|
||||
count += 1
|
||||
newest = max(newest, (base / name).lstat().st_mtime_ns)
|
||||
return count, newest
|
||||
|
||||
|
||||
def _plain_files(paths: list[Path]) -> None:
|
||||
_check_files(paths)
|
||||
_plan_file_moves(paths, [], None)
|
||||
|
||||
|
||||
def _pending_candidates() -> list[_Candidate]:
|
||||
"""The candidates in `incoming/`, oldest first (Gitea #112 E1/E5).
|
||||
|
||||
Only top-level entries count; dotfiles and empty directories are none.
|
||||
Top-level files sharing a stem are one `bundle` - a `raw fetch` pair, a PDF
|
||||
and its converted text. A top-level folder is one `folder`, with every
|
||||
file below it. A unit is as new as its newest part, so a bundle or folder
|
||||
sorts by the newest mtime it holds; a tie goes by the path's bytes."""
|
||||
incoming = _incoming_dir()
|
||||
if not incoming.is_dir():
|
||||
return []
|
||||
candidates: list[_Candidate] = []
|
||||
by_stem: dict[str, list[Path]] = {}
|
||||
for entry in incoming.iterdir():
|
||||
if entry.name.startswith("."):
|
||||
continue
|
||||
if entry.is_dir():
|
||||
if entry.is_symlink():
|
||||
count, newest = 1, entry.lstat().st_mtime_ns
|
||||
else:
|
||||
count, newest = _folder_contents(entry)
|
||||
if count:
|
||||
candidates.append(_Candidate(
|
||||
"folder", (entry,), count, newest, _refusal(_plan_folder, entry)
|
||||
))
|
||||
else:
|
||||
by_stem.setdefault(entry.stem, []).append(entry)
|
||||
for paths in by_stem.values():
|
||||
paths.sort(key=lambda p: os.fsencode(p.name))
|
||||
candidates.append(_Candidate(
|
||||
"file" if len(paths) == 1 else "bundle",
|
||||
tuple(paths),
|
||||
len(paths),
|
||||
max(p.lstat().st_mtime_ns for p in paths),
|
||||
_refusal(_plain_files, paths),
|
||||
))
|
||||
candidates.sort(key=lambda c: (c.mtime_ns, os.fsencode(c.paths[0].name)))
|
||||
return candidates
|
||||
|
||||
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="raw pending",
|
||||
summary="List what waits in `incoming/`, oldest first, and name the entry an ingest without "
|
||||
"an argument takes next.",
|
||||
synopsis=(cli_contract.Variant(usage="raw pending [--json]"),),
|
||||
properties=cli_contract.Properties(
|
||||
effect=cli_contract.Effect.READ,
|
||||
idempotent=cli_contract.Idempotent.YES,
|
||||
atomic="Read-only",
|
||||
budget=cli_contract.Budget.COUNTED,
|
||||
),
|
||||
notes=(
|
||||
"A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of "
|
||||
"top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or "
|
||||
"a `folder` with every file below it. Dotfiles, empty directories and their contents "
|
||||
"are none; `mcp-upload/` is outside `incoming/` and never listed.",
|
||||
"Order: oldest first by modification time. A bundle or folder counts as new as its "
|
||||
"newest file; a tie goes by name. The mtime is when a document last changed only if "
|
||||
"it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - "
|
||||
"for a download or a `raw fetch` it is merely when it was dropped.",
|
||||
"Each candidate shows its path(s), kind, file count and mtime, and whether `raw "
|
||||
"accept` would take it as it stands - the same checks, minus `--fidelity`/"
|
||||
"`--authority`. One it would refuse is listed with the reason and skipped: it needs a "
|
||||
"human.",
|
||||
"The default is the first candidate `raw accept` would take, and the output names it.",
|
||||
"`--json` prints the same candidates in the same order: `kind`, `paths`, `files`, "
|
||||
"`mtime`, `acceptable`, `reason`, `default`.",
|
||||
"Reads directory listings and `lstat` only, never a file's content; an empty "
|
||||
"`incoming/` is exit 0 with nothing to do.",
|
||||
),
|
||||
failures=(),
|
||||
examples=(
|
||||
"tools/wikitool raw pending",
|
||||
"tools/wikitool raw pending --json",
|
||||
),
|
||||
see_also=(
|
||||
"`wikitool raw accept` - promotes the chosen candidate",
|
||||
"`instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here",
|
||||
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - candidates and order, and why",
|
||||
),
|
||||
))
|
||||
@app.command("pending")
|
||||
def raw_pending_command(
|
||||
json_out: bool = typer.Option(False, "--json", help="Print the candidates as JSON"),
|
||||
):
|
||||
"""List the candidates waiting in incoming/, oldest first, and name the
|
||||
default. See raw/CONTRACT.md "Getting a file in: incoming/"."""
|
||||
candidates = _pending_candidates()
|
||||
default = next((c for c in candidates if c.reason is None), None)
|
||||
|
||||
if json_out:
|
||||
typer.echo(json.dumps([
|
||||
{
|
||||
"kind": c.kind,
|
||||
"paths": [rel_path(p) for p in c.paths],
|
||||
"files": c.files,
|
||||
"mtime": c.mtime_iso(),
|
||||
"acceptable": c.reason is None,
|
||||
"reason": c.reason,
|
||||
"default": c is default,
|
||||
}
|
||||
for c in candidates
|
||||
], indent=2, ensure_ascii=False))
|
||||
return
|
||||
|
||||
if not candidates:
|
||||
typer.echo("Nothing is waiting in incoming/.")
|
||||
return
|
||||
typer.echo(f"{len(candidates)} candidate(s) in incoming/, oldest first:")
|
||||
for number, c in enumerate(candidates, 1):
|
||||
marker = "*" if c is default else " "
|
||||
typer.echo(f"{marker} {number}. {c.kind:<6} {c.label()} ({c.files} file(s), {c.mtime_iso()})")
|
||||
if c.reason is not None:
|
||||
first, *rest = c.reason.splitlines()
|
||||
typer.echo(f" not acceptable: {first}")
|
||||
for line in rest:
|
||||
typer.echo(f" {line}")
|
||||
if default is None:
|
||||
typer.echo("No candidate can be accepted as it stands - each needs a human (reasons above).")
|
||||
return
|
||||
waiting = len(candidates) - 1
|
||||
typer.echo(
|
||||
f"Default (*): {default.label()} - the oldest candidate raw accept would take; "
|
||||
f"{waiting} more waiting after it."
|
||||
)
|
||||
|
||||
|
||||
# --- raw fetch ----------------------------------------------------------------
|
||||
#
|
||||
# The sanctioned intake for a URL the user names. It ends in `incoming/`, never
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
import pytest
|
||||
import typer
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.commands.raw_cmd import raw_accept_command
|
||||
from chemenu.commands.raw_cmd import raw_accept_command, raw_pending_command
|
||||
from chemenu.frontmatter_io import read_page, write_page
|
||||
from chemenu.provenance import uncovered_raw_files
|
||||
from chemenu.kb_scan import load_kb_pages
|
||||
@@ -85,40 +88,21 @@ def test_file_directly_in_incoming_is_accepted(tree):
|
||||
assert not src.exists()
|
||||
|
||||
|
||||
def test_subdirectory_under_incoming_is_ignored_not_inspected(tree):
|
||||
"""Gitea #67: a subdirectory of incoming/ - old habit, old script - is
|
||||
tolerated and ignored rather than read as a type classification. This is
|
||||
the MINOR condition named in the issue's Versionsteil."""
|
||||
(tree / "incoming/videos").mkdir()
|
||||
src = tree / "incoming/videos/clip.mp4"
|
||||
src.write_bytes(b"x")
|
||||
_accept(src)
|
||||
assert (tree / _shard() / "clip.mp4").is_file()
|
||||
assert not src.exists()
|
||||
|
||||
|
||||
def test_nested_too_deep_is_rejected(tree):
|
||||
nested = tree / "incoming/documents/sub"
|
||||
nested.mkdir(parents=True)
|
||||
src = nested / "deep.pdf"
|
||||
@pytest.mark.parametrize("rel", ["documents/a.pdf", "documents/sub/a.pdf"])
|
||||
def test_file_in_a_subdirectory_is_refused_naming_both_routes(tree, capsys, rel):
|
||||
"""Gitea #112: a subdirectory of incoming/ is a source of its own, no
|
||||
longer a tolerated, ignored type directory - a file inside one is refused,
|
||||
and the message names accepting the folder and moving the file up."""
|
||||
src = tree / "incoming" / rel
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_bytes(b"x")
|
||||
before = _tree_files(tree)
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(src)
|
||||
assert src.exists()
|
||||
|
||||
|
||||
def test_files_from_different_ignored_subdirs_bundle_together(tree):
|
||||
"""No more per-call type agreement to enforce (Gitea #67): which ignored
|
||||
subdirectory each file happened to sit under is irrelevant now."""
|
||||
a = tree / "incoming/documents/a.pdf"
|
||||
b = tree / "incoming/notes/a.md"
|
||||
a.parent.mkdir(parents=True)
|
||||
b.parent.mkdir(parents=True)
|
||||
a.write_bytes(b"a")
|
||||
b.write_text("b", encoding="utf-8")
|
||||
_accept(a, b)
|
||||
bundle = tree / _shard() / "a"
|
||||
assert (bundle / "a.pdf").exists() and (bundle / "a.md").exists()
|
||||
assert _tree_files(tree) == before
|
||||
out = " ".join(capsys.readouterr().out.split())
|
||||
assert "raw accept incoming/documents" in out
|
||||
assert "move the file up into incoming/" in out
|
||||
|
||||
|
||||
def test_same_file_passed_twice_is_rejected(tree):
|
||||
@@ -573,15 +557,13 @@ def test_replaces_rejects_filename_mismatch(tree):
|
||||
assert new.exists()
|
||||
|
||||
|
||||
def test_replaces_across_legacy_directories_is_now_allowed(tree):
|
||||
"""Gitea #67 removes the type-directory-match check `--replaces` used to
|
||||
enforce: a subdirectory of incoming/ carries no meaning any more, so
|
||||
replacing a raw/notes/ file with an incoming file dropped under an
|
||||
unrelated incoming/documents/ works exactly like one dropped flat."""
|
||||
def test_replaces_writes_back_into_a_legacy_directory(tree):
|
||||
"""Gitea #67 removed the type-directory-match check `--replaces` used to
|
||||
enforce: a raw/notes/ file is replaced by a file dropped flat into
|
||||
incoming/ like any other."""
|
||||
target = tree / "raw/notes/handbuch.md"
|
||||
target.write_text("old", encoding="utf-8")
|
||||
new = tree / "incoming/documents/handbuch.md"
|
||||
new.parent.mkdir(parents=True)
|
||||
new = tree / "incoming/handbuch.md"
|
||||
new.write_text("new", encoding="utf-8")
|
||||
|
||||
_accept(new, replaces=target, fidelity=None, authority=None)
|
||||
@@ -590,6 +572,21 @@ def test_replaces_across_legacy_directories_is_now_allowed(tree):
|
||||
assert not new.exists()
|
||||
|
||||
|
||||
def test_replaces_refuses_an_incoming_file_in_a_subdirectory(tree):
|
||||
"""Gitea #112: the subdirectory tolerance is gone for --replaces too."""
|
||||
target = tree / "raw/notes/handbuch.md"
|
||||
target.write_text("old", encoding="utf-8")
|
||||
new = tree / "incoming/documents/handbuch.md"
|
||||
new.parent.mkdir(parents=True)
|
||||
new.write_text("new", encoding="utf-8")
|
||||
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(new, replaces=target, fidelity=None, authority=None)
|
||||
|
||||
assert target.read_text(encoding="utf-8") == "old"
|
||||
assert new.read_text(encoding="utf-8") == "new"
|
||||
|
||||
|
||||
def test_replaces_rejects_nonexistent_target(tree):
|
||||
new = tree / "incoming/handbuch.md"
|
||||
new.write_text("new", encoding="utf-8")
|
||||
@@ -654,3 +651,263 @@ def test_accept_takes_a_target_at_the_path_budget(tree):
|
||||
_accept(src)
|
||||
assert (tree / _shard() / src.name).exists()
|
||||
assert not src.exists()
|
||||
|
||||
|
||||
# --- A folder is one source (Gitea #112) ---
|
||||
|
||||
|
||||
def _make_folder(tree, name, files):
|
||||
folder = tree / "incoming" / name
|
||||
for rel, data in files.items():
|
||||
(folder / rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
(folder / rel).write_bytes(data)
|
||||
return folder
|
||||
|
||||
|
||||
_BAUM = {"a.md": b"A", "sub/b.md": b"B", "sub/deep/README.md": b"R"}
|
||||
|
||||
|
||||
def test_folder_moves_every_file_byte_identical_and_leaves_no_folder(tree):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
_accept(folder)
|
||||
bundle = tree / _shard() / "baum"
|
||||
assert {str(p.relative_to(bundle)): p.read_bytes() for p in bundle.rglob("*") if p.is_file()} == _BAUM
|
||||
assert not folder.exists()
|
||||
assert sorted(p.name for p in (tree / "incoming").iterdir()) == []
|
||||
|
||||
|
||||
def test_folder_takes_same_named_files_in_different_subfolders(tree):
|
||||
folder = _make_folder(tree, "doku", {"x/README.md": b"x", "y/README.md": b"y"})
|
||||
_accept(folder)
|
||||
bundle = tree / _shard() / "doku"
|
||||
assert (bundle / "x/README.md").read_bytes() == b"x"
|
||||
assert (bundle / "y/README.md").read_bytes() == b"y"
|
||||
|
||||
|
||||
def test_folder_prints_raw_files_and_capture_pair(tree, capsys):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
_accept(folder, fidelity="secondhand", authority="opinion")
|
||||
out = "".join(capsys.readouterr().out.split()) # success() wraps long lines
|
||||
assert f"--setraw_files={_shard()}/baum/a.md,{_shard()}/baum/sub/b.md,{_shard()}/baum/sub/deep/README.md" in out
|
||||
assert "--setfidelity=secondhand--setauthority=opinion" in out
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra", ["sub/.hidden", ".git/config", "link"])
|
||||
def test_folder_with_hidden_entry_or_symlink_is_refused_and_named(tree, capsys, extra):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
if extra == "link":
|
||||
(folder / "sub/link").symlink_to(folder / "a.md")
|
||||
named = "baum/sub/link"
|
||||
else:
|
||||
(folder / extra).parent.mkdir(parents=True, exist_ok=True)
|
||||
(folder / extra).write_bytes(b"h")
|
||||
named = f"baum/{extra.split('/')[0]}" if extra.startswith(".") else f"baum/{extra}"
|
||||
before = _tree_files(tree)
|
||||
links_before = sorted(str(p) for p in tree.rglob("*") if p.is_symlink())
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(folder)
|
||||
assert _tree_files(tree) == before
|
||||
assert sorted(str(p) for p in tree.rglob("*") if p.is_symlink()) == links_before
|
||||
assert named in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_empty_folder_is_refused(tree):
|
||||
(tree / "incoming/leer/sub").mkdir(parents=True)
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(tree / "incoming/leer")
|
||||
assert (tree / "incoming/leer/sub").is_dir()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("combination", ["other-file", "page", "replaces"])
|
||||
def test_folder_with_anything_else_is_refused(tree, combination):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
other = tree / "incoming/other.md"
|
||||
other.write_text("o", encoding="utf-8")
|
||||
(tree / "raw/notes/a.md").write_bytes(b"old")
|
||||
_write_source(tree / "kb", "Source - Old", ["raw/notes/a.md"])
|
||||
before = _tree_files(tree)
|
||||
with pytest.raises(typer.Exit):
|
||||
if combination == "other-file":
|
||||
_accept(folder, other)
|
||||
elif combination == "page":
|
||||
_accept(folder, page="Source - Old")
|
||||
else:
|
||||
_accept(folder, replaces=tree / "raw/notes/a.md", fidelity=None, authority=None)
|
||||
assert _tree_files(tree) == before
|
||||
|
||||
|
||||
def test_folder_needs_both_capture_fields(tree):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
with pytest.raises(typer.Exit):
|
||||
raw_accept_command(files=[folder], fidelity=None, authority="reporting", page=None, replaces=None, dry_run=False)
|
||||
assert folder.is_dir()
|
||||
|
||||
|
||||
def test_folder_name_already_occupied_is_refused_with_the_rename_route(tree, capsys):
|
||||
(tree / "raw/documents/baum.pdf").write_bytes(b"pdf")
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
before = _tree_files(tree)
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(folder)
|
||||
assert _tree_files(tree) == before
|
||||
out = " ".join(capsys.readouterr().out.split())
|
||||
assert "Rename the folder in incoming/" in out
|
||||
assert "--replaces raw/" not in out
|
||||
|
||||
|
||||
def test_folder_target_over_the_path_budget_is_refused_before_any_move(tree):
|
||||
long_name = "n" * (161 - len(f"{_shard()}/baum/sub/") - len(".md")) + ".md"
|
||||
folder = _make_folder(tree, "baum", {"a.md": b"A", f"sub/{long_name}": b"L"})
|
||||
before = _tree_files(tree)
|
||||
with pytest.raises(typer.Exit):
|
||||
_accept(folder)
|
||||
assert _tree_files(tree) == before
|
||||
|
||||
|
||||
def test_folder_dry_run_lists_every_move_and_the_folder_and_changes_nothing(tree, capsys):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
before = _tree_files(tree)
|
||||
_accept(folder, dry_run=True)
|
||||
assert _tree_files(tree) == before
|
||||
out = capsys.readouterr().out
|
||||
for rel in _BAUM:
|
||||
assert f"incoming/baum/{rel} -> {_shard()}/baum/{rel}" in out
|
||||
assert "would remove incoming/baum/" in out
|
||||
|
||||
|
||||
def test_folder_files_are_uncovered_not_broken_afterwards(tree):
|
||||
folder = _make_folder(tree, "baum", _BAUM)
|
||||
_accept(folder)
|
||||
from chemenu.provenance import broken_raw_refs
|
||||
|
||||
pages = load_kb_pages(tree / "kb")
|
||||
assert sorted(f for f in uncovered_raw_files(tree / "raw", pages) if "/baum/" in f) == sorted(
|
||||
f"{_shard()}/baum/{rel}" for rel in _BAUM
|
||||
)
|
||||
assert [r for r in broken_raw_refs(pages) if "baum" in str(r)] == []
|
||||
|
||||
|
||||
def test_file_and_bundle_at_the_top_still_work(tree):
|
||||
single = tree / "incoming/a.pdf"
|
||||
single.write_bytes(b"a")
|
||||
_accept(single)
|
||||
assert (tree / _shard() / "a.pdf").read_bytes() == b"a"
|
||||
html, md = tree / "incoming/post.html", tree / "incoming/post.md"
|
||||
html.write_bytes(b"h")
|
||||
md.write_bytes(b"m")
|
||||
_accept(html, md)
|
||||
assert sorted(p.name for p in (tree / _shard() / "post").iterdir()) == ["post.html", "post.md"]
|
||||
|
||||
|
||||
# --- raw pending (Gitea #112) ---
|
||||
|
||||
|
||||
def _pending(capsys):
|
||||
capsys.readouterr()
|
||||
raw_pending_command(json_out=True)
|
||||
return json.loads(capsys.readouterr().out)
|
||||
|
||||
|
||||
def _age(path, seconds):
|
||||
for p in [path, *path.rglob("*")] if path.is_dir() else [path]:
|
||||
os.utime(p, ns=(seconds * 10**9, seconds * 10**9), follow_symlinks=False)
|
||||
|
||||
|
||||
def test_pending_on_an_empty_inbox_is_nothing(tree, capsys):
|
||||
(tree / "incoming/.gitkeep").write_bytes(b"")
|
||||
(tree / "incoming/documents").mkdir()
|
||||
(tree / "incoming/notes/sub").mkdir(parents=True)
|
||||
assert _pending(capsys) == []
|
||||
raw_pending_command(json_out=False)
|
||||
assert "Nothing is waiting" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_pending_groups_a_fetch_pair_into_one_bundle(tree, capsys):
|
||||
(tree / "incoming/post.html").write_bytes(b"h")
|
||||
(tree / "incoming/post.md").write_bytes(b"m")
|
||||
[candidate] = _pending(capsys)
|
||||
assert candidate["kind"] == "bundle"
|
||||
assert candidate["paths"] == ["incoming/post.html", "incoming/post.md"]
|
||||
assert candidate["files"] == 2
|
||||
|
||||
|
||||
def test_pending_counts_a_folder_once_with_all_its_files(tree, capsys):
|
||||
_make_folder(tree, "baum", _BAUM)
|
||||
[candidate] = _pending(capsys)
|
||||
assert candidate["kind"] == "folder"
|
||||
assert candidate["paths"] == ["incoming/baum"]
|
||||
assert candidate["files"] == 3
|
||||
|
||||
|
||||
def test_pending_orders_oldest_first_by_the_newest_part_then_by_name(tree, capsys):
|
||||
a = tree / "incoming/a.md"
|
||||
a.write_bytes(b"a")
|
||||
_age(a, 3000)
|
||||
b_html, b_md = tree / "incoming/b.html", tree / "incoming/b.md"
|
||||
b_html.write_bytes(b"h")
|
||||
b_md.write_bytes(b"m")
|
||||
_age(b_html, 1000)
|
||||
_age(b_md, 4000) # the bundle is as new as its newest part
|
||||
folder = _make_folder(tree, "c", {"x.md": b"x", "y/z.md": b"z"})
|
||||
_age(folder, 1000)
|
||||
_age(folder / "y/z.md", 2000)
|
||||
tie = tree / "incoming/0-tie.md"
|
||||
tie.write_bytes(b"t")
|
||||
_age(tie, 2000)
|
||||
order = [c["paths"][0] for c in _pending(capsys)]
|
||||
assert order == ["incoming/0-tie.md", "incoming/c", "incoming/a.md", "incoming/b.html"]
|
||||
|
||||
|
||||
def test_pending_marks_what_raw_accept_would_refuse_and_defaults_to_the_first_acceptable(tree, capsys):
|
||||
hidden = _make_folder(tree, "hidden", {"a.md": b"a", ".DS_Store": b"x"})
|
||||
_age(hidden, 1000)
|
||||
(tree / "raw/documents/taken.pdf").write_bytes(b"pdf")
|
||||
taken = tree / "incoming/taken.md"
|
||||
taken.write_bytes(b"t")
|
||||
_age(taken, 2000)
|
||||
long = tree / "incoming" / _incoming_name_for(161)
|
||||
long.write_bytes(b"l")
|
||||
_age(long, 3000)
|
||||
fine = tree / "incoming/fine.md"
|
||||
fine.write_bytes(b"f")
|
||||
_age(fine, 4000)
|
||||
|
||||
candidates = _pending(capsys)
|
||||
assert [c["acceptable"] for c in candidates] == [False, False, False, True]
|
||||
assert ".DS_Store" in candidates[0]["reason"]
|
||||
assert "already claims" in candidates[1]["reason"]
|
||||
assert "160" in candidates[2]["reason"] or "budget" in candidates[2]["reason"].lower()
|
||||
assert [c["default"] for c in candidates] == [False, False, False, True]
|
||||
|
||||
raw_pending_command(json_out=False)
|
||||
out = capsys.readouterr().out
|
||||
assert "Default (*): incoming/fine.md" in out
|
||||
assert "not acceptable" in out
|
||||
|
||||
|
||||
def test_pending_changes_nothing_not_even_mtimes(tree, capsys):
|
||||
_make_folder(tree, "baum", _BAUM)
|
||||
(tree / "incoming/post.html").write_bytes(b"h")
|
||||
(tree / "raw/documents/old.pdf").write_bytes(b"o")
|
||||
|
||||
def snapshot():
|
||||
return {
|
||||
str(p.relative_to(tree)): (p.read_bytes() if p.is_file() else None, p.lstat().st_mtime_ns)
|
||||
for root in ("incoming", "raw", "kb") for p in sorted((tree / root).rglob("*"))
|
||||
}
|
||||
|
||||
before = snapshot()
|
||||
_pending(capsys)
|
||||
raw_pending_command(json_out=False)
|
||||
assert snapshot() == before
|
||||
|
||||
|
||||
def test_pending_json_and_text_list_the_same_candidates_in_the_same_order(tree, capsys):
|
||||
for name, age in (("a.md", 3000), ("b.md", 1000), ("c.md", 2000)):
|
||||
(tree / "incoming" / name).write_bytes(b"x")
|
||||
_age(tree / "incoming" / name, age)
|
||||
paths = [c["paths"][0] for c in _pending(capsys)]
|
||||
raw_pending_command(json_out=False)
|
||||
text = capsys.readouterr().out
|
||||
listed = [m.group(1) for m in re.finditer(r"^[* ] \d+\. \w+\s+(\S+)", text, re.MULTILINE)]
|
||||
assert listed == paths
|
||||
@@ -371,6 +371,18 @@ def test_html_outside_incoming_is_refused(tree):
|
||||
assert list((tree / "incoming").iterdir()) == []
|
||||
|
||||
|
||||
def test_html_in_a_subdirectory_of_incoming_is_refused(tree, capsys):
|
||||
"""A subdirectory of incoming/ is a source of its own (Gitea #112), so a
|
||||
saved page inside one is not derived in place."""
|
||||
saved = tree / "incoming/articles/post.html"
|
||||
saved.parent.mkdir()
|
||||
saved.write_bytes(ARTICLE)
|
||||
before = _snapshot(tree)
|
||||
_refused(html=saved, source_url="https://example.org/post")
|
||||
assert _snapshot(tree) == before
|
||||
assert "raw accept incoming/articles" in " ".join(capsys.readouterr().out.split())
|
||||
|
||||
|
||||
def test_html_without_url_is_refused(tree):
|
||||
saved = tree / "incoming/post.html"
|
||||
saved.write_bytes(ARTICLE)
|
||||
|
||||
Reference in new issue
Block a user