feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
CI / verify (push) Successful in 5m24s
CI / pwsh (push) Successful in 1m58s
Release / release (push) Successful in 35s

Files changed:
- CHANGES.md
- README.md
- VERSION
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_raw_fetch.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-03 09:49:20 +02:00
1 parent c4dcff76e6
commit 59c06e5ddc
11 files changed
+1009 -182

No files matched your search

+37 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.26 - 2026-10-02 - kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
## 8.0.0-beta.27 - 2026-10-03 - incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
**Author:** Torben Nehmer
@@ -69,6 +69,7 @@ concern - readable here, never shipped as something to parse.
- new, rename, move and raw accept refuse a target whose path below the instance root is over 160 characters (UTF-16 code units); lint reports existing files over it as Long Paths (advisory) - rename each affected page with tools/wikitool rename, and shorten an incoming/ file name before raw accept
- tools/wikitool now refuses to start (exit 42) until tools/preflight.sh (PowerShell 7: tools/preflight.ps1) has passed in the checkout - after updating, run it once: it checks Python, git and ripgrep, records their paths in .wikitool-tools.json and sets up tools/.venv
- An instance is installed only from a release, into an empty folder (instructions/setup-instance.md); dist export, a clone of the origin repo and a private clone with the origin as upstream are no install paths any more, and wikitool upstream merge, upstream verify and instructions/private-instance.md are gone - an instance built one of those ways is reinstalled from a release into an empty folder and its kb/, raw/ and personal files are copied over. The preflight release asset installs into its own folder, which must be empty apart from the script and a .git, instead of creating a chemenu/ subfolder
- A file in a subdirectory of incoming/ is no longer accepted - raw accept and raw fetch --html refuse it; a subdirectory is now one source, accepted whole with raw accept incoming/<folder>. Drop files directly into incoming/ and adjust any script that writes to incoming/<type>/; files still waiting in such a subdirectory are moved up into incoming/, or, if they belong together, accepted as one folder
**Migration:** none required - No page format changes: the title and path rules only refuse names, each affected page is renamed with tools/wikitool rename, and the removed install paths touch no page
@@ -79,6 +80,7 @@ concern - readable here, never shipped as something to parse.
- Page titles must form valid, unique file names on Windows and macOS
- Preflight: prerequisites checked and tool paths recorded before wikitool runs (#151, POSIX half)
- Installation only from a release, into an empty folder; upstream merge/verify and the other install paths removed (#153)
- incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
**Medium impact**
- CalDAV task-tracker provider (Nextcloud Tasks, iOS Reminders); review reports unknown-value findings instead of skipping them
@@ -138,6 +140,40 @@ concern - readable here, never shipped as something to parse.
- kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
<!-- /wikitool:bumps -->
### incoming/ as a queue: raw pending picks the next entry, raw accept takes a whole folder
**Breaking.** A subdirectory of `incoming/` was tolerated and ignored since the date shard
replaced the type directories, so an old `incoming/documents/` habit kept working. It protected
nothing - the kind of source comes from its content now, as `source_type:` - and it left a folder
of files that belong together with no way in: `raw accept` took files at most one level down,
bundled them flat under the first file's stem, lost the folder's name and left the emptied
directory behind, and a folder with subfolders could not be accepted at all. Now a file argument
of `raw accept` (also with `--replaces`) and of `raw fetch --html` must sit directly in
`incoming/`; a file inside a subdirectory is refused, and the message names both ways out.
**A folder is one source.** `raw accept incoming/<folder> --fidelity ... --authority ...` moves
every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same relative path and removes the
directories left empty. The folder name is the bundle name and falls under the existing name
rule; the files inside do not, so several `README.md` in one tree are no conflict. It is accepted
alone (no other argument, no `--page`, no `--replaces`), and every check runs before anything
moves: an empty folder, or a hidden entry, a symlink or a special file anywhere below it, is
refused with each entry named - which is what guarantees that the clean-up, an `rmdir` per
directory and never a recursive delete, cannot take a file with it. A folder that passes the
large-tree thresholds continues with `work new --input raw/<YYYY>/<MM>/<folder>`, which the
success message prints.
**`tools/wikitool raw pending`** reads `incoming/` as a queue without changing it: the
top-level entries, with top-level files sharing a stem as one bundle (a `raw fetch` pair) and a
folder as one candidate, oldest first by modification time - a bundle or folder as new as its
newest file - each marked with whether `raw accept` would take it, by the same checks, and why
not. The default is the first acceptable one. `wiki-ingest` uses it when the user names nothing:
it announces the entry it took, ingests exactly that one, and says at the end how many still
wait. `raw/CONTRACT.md` § Getting a file in describes candidates, order and the limit of an
mtime, which is a document's last change only when it was copied with its timestamps kept.
No migration: no page changes and `raw/` is untouched. An instance whose scripts write to
`incoming/<type>/` changes them to write directly into `incoming/`.
### kb/CONTRACT.md: an external article's raw_files point under raw/, a raw fetch capture names both files
`kb/CONTRACT.md` still told a source page for an external article to point `raw_files:` at "the
+11 -8
View File
@@ -90,7 +90,7 @@ chemenu/
├── mcp-upload/ # QUARANTINE (optional): the MCP `submit` tool's write path, gitignored -
│ # read by no command in the ordinary pipeline; a human reviews it with
│ # `wikitool upload list/show/accept/reject`
├── incoming/ # INBOX: flat, content gitignored - drop a file here, `raw accept` promotes it
├── incoming/ # INBOX: content gitignored - drop a file or one folder per source here, `raw accept` promotes it
├── raw/ # INPUT: immutable, untrusted source material
│ ├── CONTRACT.md # Date shard, capture fields, immutability, untrusted content
│ ├── 2026/09/ # Where `raw accept` puts a file: the month it was accepted
@@ -185,12 +185,14 @@ working *with* it.
### Adding Knowledge (Ingest)
1. Drop a file into `incoming/` - flat, no classification to make. Everything past that
(the destination in `raw/`, which is a `YYYY/MM` shard of the day it was accepted,
and whether several files of one source get bundled) is computed by
`tools/wikitool raw accept`, never chosen by hand
2. Tell the LLM: `Ingest incoming/my-article.md`. It will ask you two things before
promoting: how faithful the capture is (`fidelity`) and what the material may claim
1. Drop a file into `incoming/` - directly, no classification to make. Files that belong
together go into one folder there instead: a folder is one source, accepted whole with
its structure kept. Everything past that (the destination in `raw/`, which is a `YYYY/MM`
shard of the day it was accepted, and whether several files of one source get bundled)
is computed by `tools/wikitool raw accept`, never chosen by hand
2. Tell the LLM: `Ingest incoming/my-article.md` - or just `Ingest`, which takes the oldest
entry waiting in `incoming/` (`tools/wikitool raw pending` lists them). It will ask you
two things before promoting: how faithful the capture is (`fidelity`) and what the material may claim
about its subject (`authority`). Both are recorded once and never guessed - they are
knowable now and unrecoverable later
3. The LLM will:
@@ -285,7 +287,8 @@ tools/wikitool types describe entity
### For You (Human)
1. **Curate sources** - Drop files you want processed into `incoming/` (flat)
1. **Curate sources** - Drop files you want processed into `incoming/` - directly, or one folder
per source
2. **Ask questions** - Query the wiki naturally
3. **Review changes** - Check `kb/log.md` and `kb/index.md`
4. **Direct the LLM** - Guide it on what to emphasize or investigate
+1 -1
View File
@@ -1 +1 @@
8.0.0-beta.26
8.0.0-beta.27
+9 -1
View File
@@ -109,7 +109,15 @@ session.
tools/wikitool work new --input <input path>
```
This derives the run key, refuses a collision instead of working around it, and writes
`--input` must lie under `raw/`, so a tree still waiting in `incoming/` is promoted first, as
one source and with its structure kept - `raw accept` prints the path to pass on:
```bash
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/<folder>
tools/wikitool work new --input raw/<YYYY>/<MM>/<folder>
```
`work new` derives the run key, refuses a collision instead of working around it, and writes
`README.md` + `plan.md`. Never create the directory by hand -
[work/CONTRACT.md](../work/CONTRACT.md) explains why the run key is not a free choice.
+37 -8
View File
@@ -1,15 +1,16 @@
---
name: wiki-ingest
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki".
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file or folder into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki" - or just "ingest" with nothing named, which takes the oldest entry waiting in incoming/.
---
# Wiki Ingest
**Purpose:** Process a new source file and integrate its knowledge into the wiki.
**Trigger:** User drops a file into `incoming/` (the normal path - see step 5) or directly into
`raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or explicitly requests
ingestion.
**Trigger:** User drops a file or a folder into `incoming/` (the normal path - see step 5) or
directly into `raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or
explicitly requests ingestion - with or without naming what (step 1 picks the entry when nothing
is named). **One run is one source:** one file, one bundle or one folder.
**Before the first `wikitool` call:** `instructions/session-setup.md`.
@@ -45,6 +46,24 @@ validator complains - and the ticked list is the only record that they happened.
just wrote, for instance, which skips step 5 entirely). If it is binary or an image, note its
presence and what it shows.
**The user named nothing** ("ingest", "process the inbox")? Pick the entry from the queue:
```bash
tools/wikitool raw pending
```
It lists what waits in `incoming/`, oldest first, and marks the default - the oldest entry
`raw accept` would take as it stands. **Announce it and carry on with it:** which entry, why
this one (the oldest that can be accepted), and how many wait after it. Ask nothing here -
step 4 is the halt before anything is written. One run takes exactly that one entry. If
nothing acceptable is waiting, the run ends here: say so, and name each entry the listing
marked as not acceptable, with its reason - those need the user, not a guess.
**The source is a folder** (`incoming/<folder>/`, named or picked)? It is one source - read
every file in it. Whether it is ingested here or by the large-tree procedure is decided by
the size check below, by its thresholds, not by its being a folder: three notes in a folder
do not earn a workshop.
**The user named a URL instead of a file?** Fetch it into `incoming/` first - never with
`curl` or the harness's own web fetch, which returns a model's summary rather than the page:
@@ -202,8 +221,14 @@ validator complains - and the ticked list is the only record that they happened.
List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown)
in the same call, so they land bundled together rather than as two independent promotions. A
file already in `raw/` skips this step entirely. A subdirectory under `incoming/` (an old
`incoming/<type>/` habit) is tolerated and ignored - it carries no meaning any more.
file already in `raw/` skips this step entirely. A folder is accepted as a whole, on its own:
```bash
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/<folder>
```
A file inside a subdirectory of `incoming/` is refused on its own - the subdirectory is the
source.
**A file that arrived through the MCP `submit` tool is not yet in `incoming/`** - it sits in
`mcp-upload/<id>/`, a quarantine no command in this step reads. A reviewer promotes it first
@@ -343,6 +368,9 @@ validator complains - and the ticked list is the only record that they happened.
deterministic count behind the "every 10 sources" cadence. If the threshold is reached,
tell the user a full lint is due and offer to run `wiki-lint` next.
Then say how many entries still wait in `incoming/` (`tools/wikitool raw pending`), so the
user knows whether another run is due.
## Decision points
- **Subject already has a page?** Update it (step 7, `touch`) instead of creating a second one.
@@ -374,7 +402,7 @@ validator complains - and the ticked list is the only record that they happened.
## wikitool commands used
`raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
`raw pending`, `raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
`new project`, `new source`, `new entity`, `new concept`, `touch`, `cite add`, `xref add`,
`xref link-source`, `sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`,
`log status`, `publish`
@@ -383,4 +411,5 @@ validator complains - and the ticked list is the only record that they happened.
Updated wiki with the source's knowledge integrated, published to `origin/main`.
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post"
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post",
"Ingest incoming/projekt-x", "Ingest" (the oldest entry waiting in `incoming/`)
+42 -4
View File
@@ -66,19 +66,57 @@ walks `raw/` recursively and works unchanged either way.
`raw/` is never chosen by hand. A file to be ingested is dropped into the top-level `incoming/`
- gitignored content, so a fresh clone finds the directory itself already there but never
anything dropped into it - flat: no subdirectory carries any classification any more. A
subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
never has to touch a caller), but it is **ignored**, never inspected:
anything dropped into it - **directly**, not into a subdirectory of it:
```bash
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
# -> raw/2026/09/handbuch.pdf
```
**A subdirectory of `incoming/` is a source of its own, accepted as a whole.** Several files
that belong together - a folder of notes, an unpacked export - keep their structure:
```bash
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/projekt-x
# incoming/projekt-x/plan.md -> raw/2026/09/projekt-x/plan.md
# incoming/projekt-x/docs/README.md -> raw/2026/09/projekt-x/docs/README.md
```
The folder name is the bundle name, so the name rule below applies to it and not to the files
inside: two `README.md` in different subfolders are no conflict. A folder is accepted alone, with
one `--fidelity`/`--authority` pair for all of it, and `incoming/projekt-x` is gone afterwards.
Every check runs before anything moves: an empty folder is refused, and so is one with a hidden
entry (a name starting with `.`), a symlink or a special file anywhere below it - each is named.
That is what keeps the clean-up safe: only directories the moves emptied are removed, so no file
can go with them. A file *inside* a subdirectory is never accepted on its own; the refusal names
both ways out - the whole folder, or the file moved up into `incoming/`.
A subdirectory used to be tolerated and ignored, for the old `incoming/<type>/` habit. It carries
no type any more - the kind of source comes from its content, as `source_type:` on the source
page (§ Directory routing above) - so the tolerance protected nothing, and a folder that belongs
together had no way in at all.
The file's name becomes part of its path under `raw/`, and that path has the same budget as a
page's - [kb/CONTRACT.md § Titles are identifiers](../kb/CONTRACT.md#titles-are-identifiers).
`raw accept` refuses a target over it before anything moves; the fix is a shorter name in
`incoming/`.
`incoming/` - for a folder, a shorter folder name or shorter names inside it.
**`incoming/` is a queue, and `tools/wikitool raw pending` reads it** - what an ingest without an
argument works through, one entry per run:
- **Candidates** are the top-level entries only. A single file is one; top-level files sharing a
stem are one `bundle` (a `raw fetch` pair, a PDF and its converted text - the files one
`raw accept` call takes together); a folder is one, with every file below it. Dotfiles and empty
directories are none.
- **Order:** oldest first by modification time, so that newer material builds on what the wiki
already took from older, or corrects it. A bundle or folder is as new as its newest file; a tie
goes by name. The limit of this: the mtime is when a document last changed only if it was
copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or
a `raw fetch` it is merely when it was dropped. Reading each candidate for a date of its own
would not be mechanical, and a name says nothing about age.
- **The default** is the first candidate `raw accept` would take as it stands - the same checks,
run without moving anything. One it would refuse is listed with the reason and skipped: it
needs a human, not a guess.
**A bundle directory is created only from the second file onward.** One file promoted alone needs
no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one
+56 -8
View File
@@ -115,7 +115,8 @@ sources coverage read idempotent budget:counted exit:0
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files from `incoming/` into `raw/`.
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
upload show read idempotent budget:counted exit:0,1 Print one submission's manifest in full.
upload accept write non-idempotent budget:counted exit:0,1,42 **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
@@ -1446,20 +1447,61 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
#### `raw pending`
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
**SYNOPSIS**
- `wikitool raw pending [--json]`
**PROPERTIES**
- effect: read
- idempotent: yes
- atomic: Read-only
- budget: counted
- network: no
**EXAMPLES**
- `tools/wikitool raw pending`
- `tools/wikitool raw pending --json`
**EXIT STATUS**
- 0 success
**NOTES**
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
- The default is the first candidate `raw accept` would take, and the output names it.
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
**SEE ALSO**
- `wikitool raw accept` - promotes the chosen candidate
- `instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here
- `raw/CONTRACT.md` "Getting a file in: incoming/" - candidates and order, and why
#### `raw accept`
Promote one or more files from `incoming/` into `raw/`.
Promote one or more files, or one folder, from `incoming/` into `raw/`.
**SYNOPSIS**
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
**PROPERTIES**
- effect: write
- idempotent: no
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
- budget: counted
- network: no
@@ -1467,27 +1509,30 @@ Promote one or more files from `incoming/` into `raw/`.
- `tools/wikitool raw accept incoming/docker-cheatsheet.md --fidelity verbatim --authority reporting`
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
**EXIT STATUS**
- 0 success
- 1 raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
- 1 raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
**ON FAILURE**
- raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once; for an occupied name, rename the folder in `incoming/` - there is no `--replaces` for a folder
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
- raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
**NEVER**
@@ -1497,7 +1542,9 @@ Promote one or more files from `incoming/` into `raw/`.
**NOTES**
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and ignored, not inspected - `raw/` does not address by type.
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A file argument must sit directly in `incoming/`; a file inside a subdirectory is refused, since a subdirectory is a source of its own.
- A folder argument (`incoming/<folder>`) is one source: every file below it moves to `raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the bundle name; two `README.md` in different subfolders are no conflict.
- A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with one `--fidelity`/`--authority` pair for all of it. It is refused, before anything moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a special file sits anywhere below it; the refusal names each one.
- One file promoted alone lands with no directory of its own; several files in one call nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.
- `--fidelity`/`--authority` are required on a plain accept (`types describe source` lists the values); `unknown` is refused - it is backfill-only.
- `--page "<Title>"` additionally extends that existing source page's `raw_files:` in the same call and writes both capture fields onto it - refused if it already carries a different value, since a capture field is fixed once.
@@ -1513,6 +1560,7 @@ Promote one or more files from `incoming/` into `raw/`.
**SEE ALSO**
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
- `wikitool types describe source` - the capture field values
- `wikitool new source` - the source page for a promoted file
+2 -1
View File
@@ -259,7 +259,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
"sources coverage", "sources trace", "sources rebuild-index",
)),
("Raw material and uploads", (
"raw fetch", "raw accept", "upload list", "upload show", "upload accept", "upload reject",
"raw fetch", "raw pending", "raw accept",
"upload list", "upload show", "upload accept", "upload reject",
)),
("Git", (
"sync", "publish",
+506 -111
View File
@@ -2,17 +2,22 @@
`raw/`, with the destination computed rather than chosen by hand (Gitea #58),
sharded by the accept date rather than by a hand-picked type (Gitea #67).
A human no longer classifies a file at all: `incoming/` is flat, and a
subdirectory dropped under it (an old `incoming/documents/` habit, a script
that still writes one) is accepted and ignored rather than inspected -
promoting `raw/` from a routing decision to an address computed purely from
*when* the file was accepted:
A human no longer classifies a file at all - `raw/` is an address computed
purely from *when* the file was accepted, and the kind of source comes from
its content (`source_type:` on the source page). So a subdirectory of
`incoming/` carries no type any more, and since Gitea #112 it is not tolerated
as one either: it **is** a source, accepted as a whole. A file argument must
sit directly in `incoming/`:
- **Single file, no bundle.** One file promoted alone lands as
`raw/<YYYY>/<MM>/<name>` - no directory of its own.
- **Bundle from the second file on.** Several files of one source promoted in
the same call land under `raw/<YYYY>/<MM>/<stem>/`, named after the first
file's stem.
- **A folder is one source** (Gitea #112). `raw accept incoming/<folder>`
moves every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same
relative path, then removes the directories that are left empty. The folder
name is the bundle name.
- **Growing an existing single file into a bundle.** `--page` extends an
existing source page's `raw_files:`. If that raises the page from one file
to more than one, the file it already had is folded into a bundle at its
@@ -20,6 +25,11 @@ promoting `raw/` from a routing decision to an address computed purely from
today's shard, so a bundle never mixes an old capture date with today's
(Gitea #67 decision, "Datums-Shard" § "Bündelort").
`raw pending` reads the same queue without changing it: the candidates in
`incoming/` oldest first, each judged by the very checks `raw accept` runs
before it moves anything (`_Refused` is how those checks report without
exiting).
Existing files under `raw/` are never moved by this change (Gitea #67
"Altbestand bleibt stehen"): `raw/articles/`, `raw/documents/`, `raw/notes/`
and `raw/assets/` keep whatever they already held, and stay valid promotion
@@ -65,7 +75,10 @@ and names both routes rather than choosing one (Gitea #64 decision 2).
from __future__ import annotations
import datetime
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@@ -73,7 +86,7 @@ import typer
from rich.markup import escape
from chemenu import cli_contract, config, web_capture
from chemenu.commands._util import check_path_budget, fail, rel_path, success
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
from chemenu.errors import ChemenuError
from chemenu.frontmatter_io import write_page
from chemenu.kb_scan import load_kb_pages
@@ -107,33 +120,51 @@ def _shard_dir(today: Optional[datetime.date] = None) -> Path:
return config.RAW_DIR / f"{d.year:04d}" / f"{d.month:02d}"
def _validate_under_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`, or exactly one level below it.
class _Refused(Exception):
"""A check that runs before anything moves has failed; the message is what
`fail()` prints. Raised rather than failing on the spot so `raw pending`
can ask the same checks of every candidate without exiting - one set of
checks, two callers (Gitea #112)."""
Unlike before #67, that one optional level carries no meaning any more -
it is accepted and ignored, kept only so an old `incoming/<type>/` habit
or script does not have to change to keep working (the MINOR condition
named in Gitea #67 "Versionsteil"). Nesting deeper than that is still
refused: it was never meaningful and silently accepting it would hide a
typo'd path.
def _refusals_fail(fn, *args):
try:
return fn(*args)
except _Refused as exc:
fail(escape(str(exc)))
def _check_directly_in_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`.
Up to Gitea #112 one subdirectory level was tolerated and ignored, so an
old `incoming/<type>/` habit kept working after #67 stopped reading the
type from it. A subdirectory is a source of its own now - accepted as a
whole - so a file inside one is refused with both ways out named.
"""
try:
rel = path.relative_to(incoming)
except ValueError:
fail(
raise _Refused(
f"{rel_path(path)} is not under incoming/ - `raw accept` and `raw fetch --html` "
"only take files from there. See raw/CONTRACT.md."
)
) from None
if len(rel.parts) < 1:
fail(f"incoming/{rel.as_posix()} names no file.")
if len(rel.parts) > 2:
fail(
f"incoming/{rel.as_posix()} is nested more than one level below incoming/ - place it "
"directly in incoming/, or in at most one subdirectory of it (the "
"subdirectory itself is ignored, see raw/CONTRACT.md)."
raise _Refused(f"incoming/{rel.as_posix()} names no file.")
if len(rel.parts) > 1:
top = rel.parts[0]
raise _Refused(
f"incoming/{rel.as_posix()} lies in a subdirectory of incoming/ - a file is taken "
"only from directly inside incoming/, and a subdirectory is a source of its own. "
f"Either accept the whole folder as one source (raw accept incoming/{top}), or move "
"the file up into incoming/ and accept it there. See raw/CONTRACT.md."
)
def _validate_under_incoming(path: Path, incoming: Path) -> None:
_refusals_fail(_check_directly_in_incoming, path, incoming)
_YEAR_DIR = re.compile(r"\d{4}")
@@ -185,6 +216,180 @@ def _stem_collision_message(claimed_name: str, holder: Path) -> str:
)
def _check_files(resolved: list[Path]) -> None:
incoming = _incoming_dir()
for path in resolved:
if not path.is_file():
raise _Refused(f"{rel_path(path)} does not exist or is not a file.")
_check_directly_in_incoming(path, incoming)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
raise _Refused("Two files share a filename; rename one before promoting.")
def _check_target(dst: Path, remedy: str) -> None:
problem = path_budget_problem_for(dst)
if problem:
raise _Refused(f"Cannot write {problem}. {remedy}")
if dst.exists():
raise _Refused(f"Cannot promote: {rel_path(dst)} already exists.")
def _plan_file_moves(
resolved: list[Path], existing_raw_paths: list[Path], page: Optional[str]
) -> list[tuple[Path, Path]]:
"""Where each file goes - the bundle decision, the path budget, an existing
target and the name rule, all checked before anything moves. `page` and
`existing_raw_paths` are the `--page` case; `raw pending` asks with
neither."""
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
# --page targets an existing page it always has >=1 raw file already
# (types/source.md requires raw_files:), so bundling always applies there.
total = len(existing_raw_paths) + len(resolved)
bundle_dir: Optional[Path] = None
if len(existing_raw_paths) >= 2:
parents = {p.parent for p in existing_raw_paths}
if len(parents) != 1:
raise _Refused(
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
"(see `sources coverage`)."
)
bundle_dir = parents.pop()
elif total >= 2:
if existing_raw_paths:
# Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
_check_target(
dst,
"The name comes from the file in incoming/: rename it there to something shorter "
"and accept it again.",
)
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# call is about to claim - the bundle's name, or the lone file's stem when
# no bundle forms - must not already belong to something this call does
# not itself own. "Owns" means: one of the page's already-registered raw
# files (the pitfall from the module docstring - a single file growing
# into a bundle of its own name momentarily still occupies that name), or,
# once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2:
owned.add(bundle_dir)
holder = occupied.get(claimed_name)
if holder is not None and holder not in owned:
raise _Refused(_stem_collision_message(claimed_name, holder))
return moves
def _walk_folder(folder: Path) -> tuple[list[Path], list[Path], list[str]]:
"""Every file below `folder`, every directory (`folder` included), and every
entry `raw accept` refuses to take - hidden or a symlink anywhere, or
neither a file nor a directory. Never descends into a refused entry, and
never follows a link."""
files: list[Path] = []
dirs: list[Path] = [folder]
refused: list[str] = []
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
base = Path(dirpath)
descend = []
for name in sorted(dirnames + filenames):
entry = base / name
if name.startswith("."):
refused.append(f"{rel_path(entry)} (hidden)")
elif entry.is_symlink():
refused.append(f"{rel_path(entry)} (symlink)")
elif entry.is_dir():
dirs.append(entry)
descend.append(name)
elif entry.is_file():
files.append(entry)
else:
refused.append(f"{rel_path(entry)} (neither a file nor a directory)")
dirnames[:] = descend
return sorted(files), dirs, refused
def _folder_collision_message(name: str, holder: Path) -> str:
return (
f'{rel_path(holder)} already claims the name "{name}" under raw/.\n'
" Rename the folder in incoming/ (add a distinguishing suffix) and accept it again.\n"
" A folder has no --replaces: a later edition of one file in it is replaced file by file."
)
def _plan_folder(folder: Path) -> tuple[Path, list[tuple[Path, Path]], list[Path]]:
"""The bundle directory for `raw accept incoming/<folder>`, every move into
it, and the directories to remove afterwards, deepest first - or
`_Refused`, before anything moves.
The invariant the cleanup rests on (Gitea #112): after the moves no file
is left below `folder`, so removing the emptied directories can never
take one with it. That holds only because every entry the move would
skip - hidden, a symlink, a special file - is refused here, up front."""
incoming = _incoming_dir()
try:
rel = folder.relative_to(incoming)
except ValueError:
raise _Refused(
f"{rel_path(folder)} is not under incoming/ - `raw accept` only takes a folder from "
"there. See raw/CONTRACT.md."
) from None
if len(rel.parts) != 1:
raise _Refused(
f"incoming/{rel.as_posix()} is not directly in incoming/ - a folder is accepted only "
f"as a whole, from the top: raw accept incoming/{rel.parts[0]}"
)
if folder.name.startswith(".") or folder.is_symlink():
raise _Refused(f"{rel_path(folder)} is hidden or a symlink - raw accept does not take it.")
files, dirs, refused = _walk_folder(folder)
if refused:
listed = "\n".join(f" - {entry}" for entry in refused)
raise _Refused(
f"{rel_path(folder)}/ holds entries raw accept does not take - hidden entries, "
f"symlinks and special files are refused, so nothing is left behind:\n{listed}\n"
" Remove or replace them in incoming/, then accept the folder again."
)
if not files:
raise _Refused(f"{rel_path(folder)}/ holds no file - nothing to accept.")
holder = _occupied_stems(config.RAW_DIR).get(folder.name)
if holder is not None:
raise _Refused(_folder_collision_message(folder.name, holder))
bundle_dir = _shard_dir() / folder.name
moves = [(src, bundle_dir / src.relative_to(folder)) for src in files]
for _src, dst in moves:
_check_target(
dst,
"The path comes from the folder in incoming/: shorten the folder's name or the "
"names inside it, and accept it again.",
)
return bundle_dir, moves, sorted(dirs, key=lambda d: len(d.parts), reverse=True)
def _capture_choices() -> tuple[list[str], list[str]]:
"""Allowed `--fidelity`/`--authority` values, straight from the schema
(single source of truth) - `unknown` excluded, since it is backfill-only
@@ -312,9 +517,55 @@ def _replace(
success("Review them in this same run: the replacement and their update belong in one commit.")
def _accept_folder(folder: Path, fidelity: str, authority: str, dry_run: bool) -> None:
"""`raw accept incoming/<folder>` - one folder, one source (Gitea #112).
Not atomic: one move per file, then the emptied directories. A failure
part-way leaves a half-accepted folder, which is reported, not resumed -
the call is not repeated (tool error contract, case 4)."""
bundle_dir, moves, dirs = _refusals_fail(_plan_folder, folder)
if dry_run:
for src, dst in moves:
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
typer.echo(f"[dry-run] would remove {rel_path(folder)}/ once it is empty")
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
return
for src, dst in moves:
dst.parent.mkdir(parents=True, exist_ok=True)
src.rename(dst)
typer.echo(f" moved {rel_path(src)} -> {rel_path(dst)}")
# rmdir, never a recursive delete: it refuses a directory that still holds
# anything, so this step can only ever remove what the moves emptied.
left = []
for directory in dirs:
try:
directory.rmdir()
except OSError:
left.append(directory)
if left:
typer.echo(
f"WARN {', '.join(rel_path(d) for d in left)} could not be removed - something appeared "
"in it during the move. Every file of the folder was moved; look at what is left."
)
raw_files_arg = ",".join(rel_path(dst) for _src, dst in moves)
success(
f"Promoted {rel_path(folder)}/ ({len(moves)} file(s)) to {rel_path(bundle_dir)}/.\n"
" Next:\n"
" tools/wikitool new source --name \"<Title>\" \\\n"
f" --set raw_files={raw_files_arg} \\\n"
f" --set fidelity={fidelity} --set authority={authority} \\\n"
" --set source_type=<category>\n"
" Past the thresholds in instructions/ingest-large-tree.md, continue there instead:\n"
f" tools/wikitool work new --input {rel_path(bundle_dir)}"
)
@cli_contract.record(cli_contract.CommandRecord(
path="raw accept",
summary="Promote one or more files from `incoming/` into `raw/`.",
summary="Promote one or more files, or one folder, from `incoming/` into `raw/`.",
synopsis=(
cli_contract.Variant(
usage='raw accept <file> [<file> ...] --fidelity <v> --authority <v> '
@@ -322,6 +573,11 @@ def _replace(
notes="Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` "
"shard",
),
cli_contract.Variant(
usage="raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]",
notes="Promote a whole folder as one source, its structure kept, into "
"`raw/<YYYY>/<MM>/<folder>/`",
),
cli_contract.Variant(
usage='raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] '
"[--dry-run]",
@@ -332,14 +588,23 @@ def _replace(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.NO,
atomic="`raw accept`: No - one filesystem move per file, then (with `--page`) one page "
"write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if "
"`--fidelity`/`--authority` was given) one page write",
"write. With a folder: No - one move per file, then one `rmdir` per emptied directory; "
"a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + "
"one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept "
"date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and "
"ignored, not inspected - `raw/` does not address by type.",
"date rather than chosen by hand. A file argument must sit directly in `incoming/`; a "
"file inside a subdirectory is refused, since a subdirectory is a source of its own.",
"A folder argument (`incoming/<folder>`) is one source: every file below it moves to "
"`raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty "
"are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the "
"bundle name; two `README.md` in different subfolders are no conflict.",
"A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with "
"one `--fidelity`/`--authority` pair for all of it. It is refused, before anything "
"moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a "
"special file sits anywhere below it; the refusal names each one.",
"One file promoted alone lands with no directory of its own; several files in one call "
"nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.",
"`--fidelity`/`--authority` are required on a plain accept (`types describe source` "
@@ -373,13 +638,23 @@ def _replace(
failures=(
cli_contract.Failure(
label="raw accept",
cause="A file does not exist, is not under `incoming/`, or is nested more than one "
"level below it; two files in one call share a filename; a target path already "
"exists; or a target path would be over the path budget (160 UTF-16 code units "
"below the instance root)",
reaction="Fix the named argument and retry once. For a path over the budget, "
"rename the file in `incoming/` to something shorter - the refusal comes before "
"anything moves, so `incoming/` and `raw/` are unchanged",
cause="A file does not exist or is not directly in `incoming/`; two files in one "
"call share a filename; a target path already exists; or a target path would be "
"over the path budget (160 UTF-16 code units below the instance root)",
reaction="Fix the named argument and retry once. A file inside a subdirectory of "
"`incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or "
"moved up into `incoming/` first. For a path over the budget, rename the file in "
"`incoming/` to something shorter - the refusal comes before anything moves, so "
"`incoming/` and `raw/` are unchanged",
),
cli_contract.Failure(
label="raw accept incoming/<folder>",
cause="The folder is not directly in `incoming/`, is combined with another argument, "
"`--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a "
"special file; a target path is over the budget; or the folder name is already "
"occupied under `raw/`",
reaction="Nothing moved. Fix what the message names and retry once; for an occupied "
"name, rename the folder in `incoming/` - there is no `--replaces` for a folder",
),
cli_contract.Failure(
label="raw accept",
@@ -411,9 +686,9 @@ def _replace(
),
cli_contract.Failure(
label="raw accept --replaces",
cause="The incoming file does not exist or is not under `incoming/` (or is nested "
"more than one level below it), its filename differs from the target's, or the "
"target does not lie under `raw/` or does not exist",
cause="The incoming file does not exist or is not directly in `incoming/`, its "
"filename differs from the target's, or the target does not lie under `raw/` or "
"does not exist",
reaction="Fix the named argument and retry once - every check runs before the "
"filesystem is touched, so both files are exactly as they were",
),
@@ -429,6 +704,7 @@ def _replace(
"--authority reporting",
'tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting '
'--page "Source - Docker Cheatsheet"',
"tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting",
"tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md",
),
never=(
@@ -439,6 +715,7 @@ def _replace(
),
see_also=(
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - the rules and why",
"`wikitool raw pending` - what is waiting in `incoming/`, and which entry is next",
"`wikitool types describe source` - the capture field values",
"`wikitool new source` - the source page for a promoted file",
),
@@ -447,7 +724,8 @@ def _replace(
def raw_accept_command(
files: list[Path] = typer.Argument(
...,
help="One or more files under incoming/, all belonging to the same source",
help="One or more files directly in incoming/, all belonging to the same source - or "
"one folder in incoming/, accepted whole as one source",
),
fidelity: Optional[str] = typer.Option(
None,
@@ -476,13 +754,19 @@ def raw_accept_command(
),
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
):
"""Promote file(s) from incoming/ into raw/, computing the destination
(date shard, bundle or not, bundle name) instead of taking it as an
argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
"""Promote file(s) or one folder from incoming/ into raw/, computing the
destination (date shard, bundle or not, bundle name) instead of taking it
as an argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
if not files:
fail("Pass at least one file to promote.")
incoming = _incoming_dir()
resolved = [_resolve(f) for f in files]
folders = [p for p in resolved if p.is_dir()]
if folders and (len(files) != 1 or page is not None or replaces is not None):
fail(escape(
f"{rel_path(folders[0])} is a folder, and a folder is accepted alone: one source, "
"with no other argument, no --page and no --replaces. Accept it in a call of its own."
))
if replaces is not None:
_replace(files, replaces, page, fidelity, authority, dry_run)
@@ -498,16 +782,11 @@ def raw_accept_command(
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
resolved = [_resolve(f) for f in files]
if folders:
_accept_folder(folders[0], fidelity, authority, dry_run)
return
for path in resolved:
if not path.is_file():
fail(f"{rel_path(path)} does not exist or is not a file.")
_validate_under_incoming(path, incoming)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
fail("Two files share a filename; rename one before promoting.")
_refusals_fail(_check_files, resolved)
pages = None
target_page = None
@@ -532,66 +811,7 @@ def raw_accept_command(
"before promoting more."
)
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
# --page targets an existing page it always has >=1 raw file already
# (types/source.md requires raw_files:), so bundling always applies there.
total = len(existing_raw_paths) + len(resolved)
bundle_dir: Optional[Path] = None
if len(existing_raw_paths) >= 2:
parents = {p.parent for p in existing_raw_paths}
if len(parents) != 1:
fail(
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
"(see `sources coverage`)."
)
bundle_dir = parents.pop()
elif total >= 2:
if existing_raw_paths:
# Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
check_path_budget(
dst,
"The name comes from the file in incoming/: rename it there to something shorter "
"and accept it again.",
)
if dst.exists():
fail(f"Cannot promote: {rel_path(dst)} already exists.")
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# call is about to claim - the bundle's name, or the lone file's stem when
# no bundle forms - must not already belong to something this call does
# not itself own. "Owns" means: one of the page's already-registered raw
# files (the pitfall from the module docstring - a single file growing
# into a bundle of its own name momentarily still occupies that name), or,
# once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2:
owned.add(bundle_dir)
holder = occupied.get(claimed_name)
if holder is not None and holder not in owned:
fail(_stem_collision_message(claimed_name, holder))
moves = _refusals_fail(_plan_file_moves, resolved, existing_raw_paths, page)
moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
if moving_existing:
@@ -663,6 +883,181 @@ def raw_accept_command(
)
# --- raw pending --------------------------------------------------------------
#
# `incoming/` read as a queue (Gitea #112): what `wiki-ingest` without an
# argument works through, one candidate per run, oldest first so that newer
# material builds on - or corrects - what the wiki already took from older.
@dataclass(frozen=True)
class _Candidate:
kind: str # file | bundle | folder
paths: tuple[Path, ...]
files: int
mtime_ns: int
reason: Optional[str] # None when `raw accept` would take it as it stands
def label(self) -> str:
return ", ".join(rel_path(p) + ("/" if self.kind == "folder" else "") for p in self.paths)
def mtime_iso(self) -> str:
stamp = datetime.datetime.fromtimestamp(self.mtime_ns / 1e9).astimezone()
return stamp.isoformat(timespec="seconds")
def _refusal(fn, *args) -> Optional[str]:
try:
fn(*args)
except _Refused as exc:
return str(exc)
return None
def _folder_contents(folder: Path) -> tuple[int, int]:
"""How many non-directory entries sit below `folder`, and the newest
mtime among them - `lstat` only, no content read and no link followed."""
count, newest = 0, 0
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
base = Path(dirpath)
for name in filenames + [d for d in dirnames if (base / d).is_symlink()]:
count += 1
newest = max(newest, (base / name).lstat().st_mtime_ns)
return count, newest
def _plain_files(paths: list[Path]) -> None:
_check_files(paths)
_plan_file_moves(paths, [], None)
def _pending_candidates() -> list[_Candidate]:
"""The candidates in `incoming/`, oldest first (Gitea #112 E1/E5).
Only top-level entries count; dotfiles and empty directories are none.
Top-level files sharing a stem are one `bundle` - a `raw fetch` pair, a PDF
and its converted text. A top-level folder is one `folder`, with every
file below it. A unit is as new as its newest part, so a bundle or folder
sorts by the newest mtime it holds; a tie goes by the path's bytes."""
incoming = _incoming_dir()
if not incoming.is_dir():
return []
candidates: list[_Candidate] = []
by_stem: dict[str, list[Path]] = {}
for entry in incoming.iterdir():
if entry.name.startswith("."):
continue
if entry.is_dir():
if entry.is_symlink():
count, newest = 1, entry.lstat().st_mtime_ns
else:
count, newest = _folder_contents(entry)
if count:
candidates.append(_Candidate(
"folder", (entry,), count, newest, _refusal(_plan_folder, entry)
))
else:
by_stem.setdefault(entry.stem, []).append(entry)
for paths in by_stem.values():
paths.sort(key=lambda p: os.fsencode(p.name))
candidates.append(_Candidate(
"file" if len(paths) == 1 else "bundle",
tuple(paths),
len(paths),
max(p.lstat().st_mtime_ns for p in paths),
_refusal(_plain_files, paths),
))
candidates.sort(key=lambda c: (c.mtime_ns, os.fsencode(c.paths[0].name)))
return candidates
@cli_contract.record(cli_contract.CommandRecord(
path="raw pending",
summary="List what waits in `incoming/`, oldest first, and name the entry an ingest without "
"an argument takes next.",
synopsis=(cli_contract.Variant(usage="raw pending [--json]"),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.READ,
idempotent=cli_contract.Idempotent.YES,
atomic="Read-only",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of "
"top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or "
"a `folder` with every file below it. Dotfiles, empty directories and their contents "
"are none; `mcp-upload/` is outside `incoming/` and never listed.",
"Order: oldest first by modification time. A bundle or folder counts as new as its "
"newest file; a tie goes by name. The mtime is when a document last changed only if "
"it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - "
"for a download or a `raw fetch` it is merely when it was dropped.",
"Each candidate shows its path(s), kind, file count and mtime, and whether `raw "
"accept` would take it as it stands - the same checks, minus `--fidelity`/"
"`--authority`. One it would refuse is listed with the reason and skipped: it needs a "
"human.",
"The default is the first candidate `raw accept` would take, and the output names it.",
"`--json` prints the same candidates in the same order: `kind`, `paths`, `files`, "
"`mtime`, `acceptable`, `reason`, `default`.",
"Reads directory listings and `lstat` only, never a file's content; an empty "
"`incoming/` is exit 0 with nothing to do.",
),
failures=(),
examples=(
"tools/wikitool raw pending",
"tools/wikitool raw pending --json",
),
see_also=(
"`wikitool raw accept` - promotes the chosen candidate",
"`instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here",
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - candidates and order, and why",
),
))
@app.command("pending")
def raw_pending_command(
json_out: bool = typer.Option(False, "--json", help="Print the candidates as JSON"),
):
"""List the candidates waiting in incoming/, oldest first, and name the
default. See raw/CONTRACT.md "Getting a file in: incoming/"."""
candidates = _pending_candidates()
default = next((c for c in candidates if c.reason is None), None)
if json_out:
typer.echo(json.dumps([
{
"kind": c.kind,
"paths": [rel_path(p) for p in c.paths],
"files": c.files,
"mtime": c.mtime_iso(),
"acceptable": c.reason is None,
"reason": c.reason,
"default": c is default,
}
for c in candidates
], indent=2, ensure_ascii=False))
return
if not candidates:
typer.echo("Nothing is waiting in incoming/.")
return
typer.echo(f"{len(candidates)} candidate(s) in incoming/, oldest first:")
for number, c in enumerate(candidates, 1):
marker = "*" if c is default else " "
typer.echo(f"{marker} {number}. {c.kind:<6} {c.label()} ({c.files} file(s), {c.mtime_iso()})")
if c.reason is not None:
first, *rest = c.reason.splitlines()
typer.echo(f" not acceptable: {first}")
for line in rest:
typer.echo(f" {line}")
if default is None:
typer.echo("No candidate can be accepted as it stands - each needs a human (reasons above).")
return
waiting = len(candidates) - 1
typer.echo(
f"Default (*): {default.label()} - the oldest candidate raw accept would take; "
f"{waiting} more waiting after it."
)
# --- raw fetch ----------------------------------------------------------------
#
# The sanctioned intake for a URL the user names. It ends in `incoming/`, never
+296 -39
View File
@@ -1,10 +1,13 @@
import datetime
import json
import os
import re
import pytest
import typer
from chemenu import config
from chemenu.commands.raw_cmd import raw_accept_command
from chemenu.commands.raw_cmd import raw_accept_command, raw_pending_command
from chemenu.frontmatter_io import read_page, write_page
from chemenu.provenance import uncovered_raw_files
from chemenu.kb_scan import load_kb_pages
@@ -85,40 +88,21 @@ def test_file_directly_in_incoming_is_accepted(tree):
assert not src.exists()
def test_subdirectory_under_incoming_is_ignored_not_inspected(tree):
"""Gitea #67: a subdirectory of incoming/ - old habit, old script - is
tolerated and ignored rather than read as a type classification. This is
the MINOR condition named in the issue's Versionsteil."""
(tree / "incoming/videos").mkdir()
src = tree / "incoming/videos/clip.mp4"
src.write_bytes(b"x")
_accept(src)
assert (tree / _shard() / "clip.mp4").is_file()
assert not src.exists()
def test_nested_too_deep_is_rejected(tree):
nested = tree / "incoming/documents/sub"
nested.mkdir(parents=True)
src = nested / "deep.pdf"
@pytest.mark.parametrize("rel", ["documents/a.pdf", "documents/sub/a.pdf"])
def test_file_in_a_subdirectory_is_refused_naming_both_routes(tree, capsys, rel):
"""Gitea #112: a subdirectory of incoming/ is a source of its own, no
longer a tolerated, ignored type directory - a file inside one is refused,
and the message names accepting the folder and moving the file up."""
src = tree / "incoming" / rel
src.parent.mkdir(parents=True)
src.write_bytes(b"x")
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
def test_files_from_different_ignored_subdirs_bundle_together(tree):
"""No more per-call type agreement to enforce (Gitea #67): which ignored
subdirectory each file happened to sit under is irrelevant now."""
a = tree / "incoming/documents/a.pdf"
b = tree / "incoming/notes/a.md"
a.parent.mkdir(parents=True)
b.parent.mkdir(parents=True)
a.write_bytes(b"a")
b.write_text("b", encoding="utf-8")
_accept(a, b)
bundle = tree / _shard() / "a"
assert (bundle / "a.pdf").exists() and (bundle / "a.md").exists()
assert _tree_files(tree) == before
out = " ".join(capsys.readouterr().out.split())
assert "raw accept incoming/documents" in out
assert "move the file up into incoming/" in out
def test_same_file_passed_twice_is_rejected(tree):
@@ -573,15 +557,13 @@ def test_replaces_rejects_filename_mismatch(tree):
assert new.exists()
def test_replaces_across_legacy_directories_is_now_allowed(tree):
"""Gitea #67 removes the type-directory-match check `--replaces` used to
enforce: a subdirectory of incoming/ carries no meaning any more, so
replacing a raw/notes/ file with an incoming file dropped under an
unrelated incoming/documents/ works exactly like one dropped flat."""
def test_replaces_writes_back_into_a_legacy_directory(tree):
"""Gitea #67 removed the type-directory-match check `--replaces` used to
enforce: a raw/notes/ file is replaced by a file dropped flat into
incoming/ like any other."""
target = tree / "raw/notes/handbuch.md"
target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md"
new.parent.mkdir(parents=True)
new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8")
_accept(new, replaces=target, fidelity=None, authority=None)
@@ -590,6 +572,21 @@ def test_replaces_across_legacy_directories_is_now_allowed(tree):
assert not new.exists()
def test_replaces_refuses_an_incoming_file_in_a_subdirectory(tree):
"""Gitea #112: the subdirectory tolerance is gone for --replaces too."""
target = tree / "raw/notes/handbuch.md"
target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md"
new.parent.mkdir(parents=True)
new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old"
assert new.read_text(encoding="utf-8") == "new"
def test_replaces_rejects_nonexistent_target(tree):
new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8")
@@ -654,3 +651,263 @@ def test_accept_takes_a_target_at_the_path_budget(tree):
_accept(src)
assert (tree / _shard() / src.name).exists()
assert not src.exists()
# --- A folder is one source (Gitea #112) ---
def _make_folder(tree, name, files):
folder = tree / "incoming" / name
for rel, data in files.items():
(folder / rel).parent.mkdir(parents=True, exist_ok=True)
(folder / rel).write_bytes(data)
return folder
_BAUM = {"a.md": b"A", "sub/b.md": b"B", "sub/deep/README.md": b"R"}
def test_folder_moves_every_file_byte_identical_and_leaves_no_folder(tree):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder)
bundle = tree / _shard() / "baum"
assert {str(p.relative_to(bundle)): p.read_bytes() for p in bundle.rglob("*") if p.is_file()} == _BAUM
assert not folder.exists()
assert sorted(p.name for p in (tree / "incoming").iterdir()) == []
def test_folder_takes_same_named_files_in_different_subfolders(tree):
folder = _make_folder(tree, "doku", {"x/README.md": b"x", "y/README.md": b"y"})
_accept(folder)
bundle = tree / _shard() / "doku"
assert (bundle / "x/README.md").read_bytes() == b"x"
assert (bundle / "y/README.md").read_bytes() == b"y"
def test_folder_prints_raw_files_and_capture_pair(tree, capsys):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder, fidelity="secondhand", authority="opinion")
out = "".join(capsys.readouterr().out.split()) # success() wraps long lines
assert f"--setraw_files={_shard()}/baum/a.md,{_shard()}/baum/sub/b.md,{_shard()}/baum/sub/deep/README.md" in out
assert "--setfidelity=secondhand--setauthority=opinion" in out
@pytest.mark.parametrize("extra", ["sub/.hidden", ".git/config", "link"])
def test_folder_with_hidden_entry_or_symlink_is_refused_and_named(tree, capsys, extra):
folder = _make_folder(tree, "baum", _BAUM)
if extra == "link":
(folder / "sub/link").symlink_to(folder / "a.md")
named = "baum/sub/link"
else:
(folder / extra).parent.mkdir(parents=True, exist_ok=True)
(folder / extra).write_bytes(b"h")
named = f"baum/{extra.split('/')[0]}" if extra.startswith(".") else f"baum/{extra}"
before = _tree_files(tree)
links_before = sorted(str(p) for p in tree.rglob("*") if p.is_symlink())
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
assert sorted(str(p) for p in tree.rglob("*") if p.is_symlink()) == links_before
assert named in capsys.readouterr().out
def test_empty_folder_is_refused(tree):
(tree / "incoming/leer/sub").mkdir(parents=True)
with pytest.raises(typer.Exit):
_accept(tree / "incoming/leer")
assert (tree / "incoming/leer/sub").is_dir()
@pytest.mark.parametrize("combination", ["other-file", "page", "replaces"])
def test_folder_with_anything_else_is_refused(tree, combination):
folder = _make_folder(tree, "baum", _BAUM)
other = tree / "incoming/other.md"
other.write_text("o", encoding="utf-8")
(tree / "raw/notes/a.md").write_bytes(b"old")
_write_source(tree / "kb", "Source - Old", ["raw/notes/a.md"])
before = _tree_files(tree)
with pytest.raises(typer.Exit):
if combination == "other-file":
_accept(folder, other)
elif combination == "page":
_accept(folder, page="Source - Old")
else:
_accept(folder, replaces=tree / "raw/notes/a.md", fidelity=None, authority=None)
assert _tree_files(tree) == before
def test_folder_needs_both_capture_fields(tree):
folder = _make_folder(tree, "baum", _BAUM)
with pytest.raises(typer.Exit):
raw_accept_command(files=[folder], fidelity=None, authority="reporting", page=None, replaces=None, dry_run=False)
assert folder.is_dir()
def test_folder_name_already_occupied_is_refused_with_the_rename_route(tree, capsys):
(tree / "raw/documents/baum.pdf").write_bytes(b"pdf")
folder = _make_folder(tree, "baum", _BAUM)
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
out = " ".join(capsys.readouterr().out.split())
assert "Rename the folder in incoming/" in out
assert "--replaces raw/" not in out
def test_folder_target_over_the_path_budget_is_refused_before_any_move(tree):
long_name = "n" * (161 - len(f"{_shard()}/baum/sub/") - len(".md")) + ".md"
folder = _make_folder(tree, "baum", {"a.md": b"A", f"sub/{long_name}": b"L"})
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
def test_folder_dry_run_lists_every_move_and_the_folder_and_changes_nothing(tree, capsys):
folder = _make_folder(tree, "baum", _BAUM)
before = _tree_files(tree)
_accept(folder, dry_run=True)
assert _tree_files(tree) == before
out = capsys.readouterr().out
for rel in _BAUM:
assert f"incoming/baum/{rel} -> {_shard()}/baum/{rel}" in out
assert "would remove incoming/baum/" in out
def test_folder_files_are_uncovered_not_broken_afterwards(tree):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder)
from chemenu.provenance import broken_raw_refs
pages = load_kb_pages(tree / "kb")
assert sorted(f for f in uncovered_raw_files(tree / "raw", pages) if "/baum/" in f) == sorted(
f"{_shard()}/baum/{rel}" for rel in _BAUM
)
assert [r for r in broken_raw_refs(pages) if "baum" in str(r)] == []
def test_file_and_bundle_at_the_top_still_work(tree):
single = tree / "incoming/a.pdf"
single.write_bytes(b"a")
_accept(single)
assert (tree / _shard() / "a.pdf").read_bytes() == b"a"
html, md = tree / "incoming/post.html", tree / "incoming/post.md"
html.write_bytes(b"h")
md.write_bytes(b"m")
_accept(html, md)
assert sorted(p.name for p in (tree / _shard() / "post").iterdir()) == ["post.html", "post.md"]
# --- raw pending (Gitea #112) ---
def _pending(capsys):
capsys.readouterr()
raw_pending_command(json_out=True)
return json.loads(capsys.readouterr().out)
def _age(path, seconds):
for p in [path, *path.rglob("*")] if path.is_dir() else [path]:
os.utime(p, ns=(seconds * 10**9, seconds * 10**9), follow_symlinks=False)
def test_pending_on_an_empty_inbox_is_nothing(tree, capsys):
(tree / "incoming/.gitkeep").write_bytes(b"")
(tree / "incoming/documents").mkdir()
(tree / "incoming/notes/sub").mkdir(parents=True)
assert _pending(capsys) == []
raw_pending_command(json_out=False)
assert "Nothing is waiting" in capsys.readouterr().out
def test_pending_groups_a_fetch_pair_into_one_bundle(tree, capsys):
(tree / "incoming/post.html").write_bytes(b"h")
(tree / "incoming/post.md").write_bytes(b"m")
[candidate] = _pending(capsys)
assert candidate["kind"] == "bundle"
assert candidate["paths"] == ["incoming/post.html", "incoming/post.md"]
assert candidate["files"] == 2
def test_pending_counts_a_folder_once_with_all_its_files(tree, capsys):
_make_folder(tree, "baum", _BAUM)
[candidate] = _pending(capsys)
assert candidate["kind"] == "folder"
assert candidate["paths"] == ["incoming/baum"]
assert candidate["files"] == 3
def test_pending_orders_oldest_first_by_the_newest_part_then_by_name(tree, capsys):
a = tree / "incoming/a.md"
a.write_bytes(b"a")
_age(a, 3000)
b_html, b_md = tree / "incoming/b.html", tree / "incoming/b.md"
b_html.write_bytes(b"h")
b_md.write_bytes(b"m")
_age(b_html, 1000)
_age(b_md, 4000) # the bundle is as new as its newest part
folder = _make_folder(tree, "c", {"x.md": b"x", "y/z.md": b"z"})
_age(folder, 1000)
_age(folder / "y/z.md", 2000)
tie = tree / "incoming/0-tie.md"
tie.write_bytes(b"t")
_age(tie, 2000)
order = [c["paths"][0] for c in _pending(capsys)]
assert order == ["incoming/0-tie.md", "incoming/c", "incoming/a.md", "incoming/b.html"]
def test_pending_marks_what_raw_accept_would_refuse_and_defaults_to_the_first_acceptable(tree, capsys):
hidden = _make_folder(tree, "hidden", {"a.md": b"a", ".DS_Store": b"x"})
_age(hidden, 1000)
(tree / "raw/documents/taken.pdf").write_bytes(b"pdf")
taken = tree / "incoming/taken.md"
taken.write_bytes(b"t")
_age(taken, 2000)
long = tree / "incoming" / _incoming_name_for(161)
long.write_bytes(b"l")
_age(long, 3000)
fine = tree / "incoming/fine.md"
fine.write_bytes(b"f")
_age(fine, 4000)
candidates = _pending(capsys)
assert [c["acceptable"] for c in candidates] == [False, False, False, True]
assert ".DS_Store" in candidates[0]["reason"]
assert "already claims" in candidates[1]["reason"]
assert "160" in candidates[2]["reason"] or "budget" in candidates[2]["reason"].lower()
assert [c["default"] for c in candidates] == [False, False, False, True]
raw_pending_command(json_out=False)
out = capsys.readouterr().out
assert "Default (*): incoming/fine.md" in out
assert "not acceptable" in out
def test_pending_changes_nothing_not_even_mtimes(tree, capsys):
_make_folder(tree, "baum", _BAUM)
(tree / "incoming/post.html").write_bytes(b"h")
(tree / "raw/documents/old.pdf").write_bytes(b"o")
def snapshot():
return {
str(p.relative_to(tree)): (p.read_bytes() if p.is_file() else None, p.lstat().st_mtime_ns)
for root in ("incoming", "raw", "kb") for p in sorted((tree / root).rglob("*"))
}
before = snapshot()
_pending(capsys)
raw_pending_command(json_out=False)
assert snapshot() == before
def test_pending_json_and_text_list_the_same_candidates_in_the_same_order(tree, capsys):
for name, age in (("a.md", 3000), ("b.md", 1000), ("c.md", 2000)):
(tree / "incoming" / name).write_bytes(b"x")
_age(tree / "incoming" / name, age)
paths = [c["paths"][0] for c in _pending(capsys)]
raw_pending_command(json_out=False)
text = capsys.readouterr().out
listed = [m.group(1) for m in re.finditer(r"^[* ] \d+\. \w+\s+(\S+)", text, re.MULTILINE)]
assert listed == paths
+12
View File
@@ -371,6 +371,18 @@ def test_html_outside_incoming_is_refused(tree):
assert list((tree / "incoming").iterdir()) == []
def test_html_in_a_subdirectory_of_incoming_is_refused(tree, capsys):
"""A subdirectory of incoming/ is a source of its own (Gitea #112), so a
saved page inside one is not derived in place."""
saved = tree / "incoming/articles/post.html"
saved.parent.mkdir()
saved.write_bytes(ARTICLE)
before = _snapshot(tree)
_refused(html=saved, source_url="https://example.org/post")
assert _snapshot(tree) == before
assert "raw accept incoming/articles" in " ".join(capsys.readouterr().out.split())
def test_html_without_url_is_refused(tree):
saved = tree / "incoming/post.html"
saved.write_bytes(ARTICLE)