feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
CI / verify (push) Successful in 5m24s
CI / pwsh (push) Successful in 1m58s
Release / release (push) Successful in 35s

Files changed:
- CHANGES.md
- README.md
- VERSION
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_raw_fetch.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-03 09:49:20 +02:00
1 parent c4dcff76e6
commit 59c06e5ddc
11 files changed
+1009 -182

No files matched your search

+56 -8
View File
@@ -115,7 +115,8 @@ sources coverage read idempotent budget:counted exit:0
sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page.
sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index.
raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote.
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files from `incoming/` into `raw/`.
raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
raw accept write non-idempotent budget:counted exit:0,1 Promote one or more files, or one folder, from `incoming/` into `raw/`.
upload list read idempotent budget:counted exit:0 List every MCP submission currently waiting in the quarantine (`mcp-upload/`).
upload show read idempotent budget:counted exit:0,1 Print one submission's manifest in full.
upload accept write non-idempotent budget:counted exit:0,1,42 **Upload Review Gate:** promote a submission's file from quarantine into `incoming/`.
@@ -1446,20 +1447,61 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
- `wikitool raw accept` - promotes the written files into `raw/`
- `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
#### `raw pending`
List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next.
**SYNOPSIS**
- `wikitool raw pending [--json]`
**PROPERTIES**
- effect: read
- idempotent: yes
- atomic: Read-only
- budget: counted
- network: no
**EXAMPLES**
- `tools/wikitool raw pending`
- `tools/wikitool raw pending --json`
**EXIT STATUS**
- 0 success
**NOTES**
- A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or a `folder` with every file below it. Dotfiles, empty directories and their contents are none; `mcp-upload/` is outside `incoming/` and never listed.
- Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - for a download or a `raw fetch` it is merely when it was dropped.
- Each candidate shows its path(s), kind, file count and mtime, and whether `raw accept` would take it as it stands - the same checks, minus `--fidelity`/`--authority`. One it would refuse is listed with the reason and skipped: it needs a human.
- The default is the first candidate `raw accept` would take, and the output names it.
- `--json` prints the same candidates in the same order: `kind`, `paths`, `files`, `mtime`, `acceptable`, `reason`, `default`.
- Reads directory listings and `lstat` only, never a file's content; an empty `incoming/` is exit 0 with nothing to do.
**SEE ALSO**
- `wikitool raw accept` - promotes the chosen candidate
- `instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here
- `raw/CONTRACT.md` "Getting a file in: incoming/" - candidates and order, and why
#### `raw accept`
Promote one or more files from `incoming/` into `raw/`.
Promote one or more files, or one folder, from `incoming/` into `raw/`.
**SYNOPSIS**
- `wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run]` - Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` shard
- `wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]` - Promote a whole folder as one source, its structure kept, into `raw/<YYYY>/<MM>/<folder>/`
- `wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run]` - Overwrite one existing raw file in place with a new edition
**PROPERTIES**
- effect: write
- idempotent: no
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
- atomic: `raw accept`: No - one filesystem move per file, then (with `--page`) one page write. With a folder: No - one move per file, then one `rmdir` per emptied directory; a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write
- budget: counted
- network: no
@@ -1467,27 +1509,30 @@ Promote one or more files from `incoming/` into `raw/`.
- `tools/wikitool raw accept incoming/docker-cheatsheet.md --fidelity verbatim --authority reporting`
- `tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"`
- `tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting`
- `tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md`
**EXIT STATUS**
- 0 success
- 1 raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
- 1 raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/`
- 1 raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum
- 1 raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own
- 1 raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value
- 1 raw accept --replaces: More than one incoming file, or `--page` also given
- 1 raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist
- 1 raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist
- 1 raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page
**ON FAILURE**
- raw accept: A file does not exist, is not under `incoming/`, or is nested more than one level below it; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept: A file does not exist or is not directly in `incoming/`; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root) -> Fix the named argument and retry once. A file inside a subdirectory of `incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or moved up into `incoming/` first. For a path over the budget, rename the file in `incoming/` to something shorter - the refusal comes before anything moves, so `incoming/` and `raw/` are unchanged
- raw accept incoming/<folder>: The folder is not directly in `incoming/`, is combined with another argument, `--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under `raw/` -> Nothing moved. Fix what the message names and retry once; for an occupied name, rename the folder in `incoming/` - there is no `--replaces` for a folder
- raw accept: `--fidelity`/`--authority` is missing, or names `unknown` or a value outside the schema's enum -> Pass both with a valid value, then retry once
- raw accept: The target name is already occupied anywhere under `raw/` by something the call does not own -> Not fixed by retrying: the refusal names `--replaces` (same source, new edition) and renaming in `incoming/` (a separate source) as the two routes, and neither is the tool's to pick. Show the message to the user and wait
- raw accept: `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk, a file to be moved has more than one owning page, or `--page` would overwrite an already-set `fidelity`/`authority` with a different value -> Fix the named argument and retry once; a different capture value on an existing page is a new edition - `--replaces`
- raw accept --replaces: More than one incoming file, or `--page` also given -> A replacement is one file for one file - fix the call and retry once
- raw accept --replaces: The incoming file does not exist or is not under `incoming/` (or is nested more than one level below it), its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
- raw accept --replaces: The incoming file does not exist or is not directly in `incoming/`, its filename differs from the target's, or the target does not lie under `raw/` or does not exist -> Fix the named argument and retry once - every check runs before the filesystem is touched, so both files are exactly as they were
- raw accept --replaces: `--fidelity`/`--authority` names `unknown` or a value outside the schema's enum, or the target has more than one owning source page -> Fix the named argument and retry once; nothing was touched
**NEVER**
@@ -1497,7 +1542,9 @@ Promote one or more files from `incoming/` into `raw/`.
**NOTES**
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and ignored, not inspected - `raw/` does not address by type.
- Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept date rather than chosen by hand. A file argument must sit directly in `incoming/`; a file inside a subdirectory is refused, since a subdirectory is a source of its own.
- A folder argument (`incoming/<folder>`) is one source: every file below it moves to `raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the bundle name; two `README.md` in different subfolders are no conflict.
- A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with one `--fidelity`/`--authority` pair for all of it. It is refused, before anything moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a special file sits anywhere below it; the refusal names each one.
- One file promoted alone lands with no directory of its own; several files in one call nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.
- `--fidelity`/`--authority` are required on a plain accept (`types describe source` lists the values); `unknown` is refused - it is backfill-only.
- `--page "<Title>"` additionally extends that existing source page's `raw_files:` in the same call and writes both capture fields onto it - refused if it already carries a different value, since a capture field is fixed once.
@@ -1513,6 +1560,7 @@ Promote one or more files from `incoming/` into `raw/`.
**SEE ALSO**
- `raw/CONTRACT.md` "Getting a file in: incoming/" - the rules and why
- `wikitool raw pending` - what is waiting in `incoming/`, and which entry is next
- `wikitool types describe source` - the capture field values
- `wikitool new source` - the source page for a promoted file
+2 -1
View File
@@ -259,7 +259,8 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
"sources coverage", "sources trace", "sources rebuild-index",
)),
("Raw material and uploads", (
"raw fetch", "raw accept", "upload list", "upload show", "upload accept", "upload reject",
"raw fetch", "raw pending", "raw accept",
"upload list", "upload show", "upload accept", "upload reject",
)),
("Git", (
"sync", "publish",
+506 -111
View File
@@ -2,17 +2,22 @@
`raw/`, with the destination computed rather than chosen by hand (Gitea #58),
sharded by the accept date rather than by a hand-picked type (Gitea #67).
A human no longer classifies a file at all: `incoming/` is flat, and a
subdirectory dropped under it (an old `incoming/documents/` habit, a script
that still writes one) is accepted and ignored rather than inspected -
promoting `raw/` from a routing decision to an address computed purely from
*when* the file was accepted:
A human no longer classifies a file at all - `raw/` is an address computed
purely from *when* the file was accepted, and the kind of source comes from
its content (`source_type:` on the source page). So a subdirectory of
`incoming/` carries no type any more, and since Gitea #112 it is not tolerated
as one either: it **is** a source, accepted as a whole. A file argument must
sit directly in `incoming/`:
- **Single file, no bundle.** One file promoted alone lands as
`raw/<YYYY>/<MM>/<name>` - no directory of its own.
- **Bundle from the second file on.** Several files of one source promoted in
the same call land under `raw/<YYYY>/<MM>/<stem>/`, named after the first
file's stem.
- **A folder is one source** (Gitea #112). `raw accept incoming/<folder>`
moves every file below it to `raw/<YYYY>/<MM>/<folder>/` at the same
relative path, then removes the directories that are left empty. The folder
name is the bundle name.
- **Growing an existing single file into a bundle.** `--page` extends an
existing source page's `raw_files:`. If that raises the page from one file
to more than one, the file it already had is folded into a bundle at its
@@ -20,6 +25,11 @@ promoting `raw/` from a routing decision to an address computed purely from
today's shard, so a bundle never mixes an old capture date with today's
(Gitea #67 decision, "Datums-Shard" § "Bündelort").
`raw pending` reads the same queue without changing it: the candidates in
`incoming/` oldest first, each judged by the very checks `raw accept` runs
before it moves anything (`_Refused` is how those checks report without
exiting).
Existing files under `raw/` are never moved by this change (Gitea #67
"Altbestand bleibt stehen"): `raw/articles/`, `raw/documents/`, `raw/notes/`
and `raw/assets/` keep whatever they already held, and stay valid promotion
@@ -65,7 +75,10 @@ and names both routes rather than choosing one (Gitea #64 decision 2).
from __future__ import annotations
import datetime
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@@ -73,7 +86,7 @@ import typer
from rich.markup import escape
from chemenu import cli_contract, config, web_capture
from chemenu.commands._util import check_path_budget, fail, rel_path, success
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
from chemenu.errors import ChemenuError
from chemenu.frontmatter_io import write_page
from chemenu.kb_scan import load_kb_pages
@@ -107,33 +120,51 @@ def _shard_dir(today: Optional[datetime.date] = None) -> Path:
return config.RAW_DIR / f"{d.year:04d}" / f"{d.month:02d}"
def _validate_under_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`, or exactly one level below it.
class _Refused(Exception):
"""A check that runs before anything moves has failed; the message is what
`fail()` prints. Raised rather than failing on the spot so `raw pending`
can ask the same checks of every candidate without exiting - one set of
checks, two callers (Gitea #112)."""
Unlike before #67, that one optional level carries no meaning any more -
it is accepted and ignored, kept only so an old `incoming/<type>/` habit
or script does not have to change to keep working (the MINOR condition
named in Gitea #67 "Versionsteil"). Nesting deeper than that is still
refused: it was never meaningful and silently accepting it would hide a
typo'd path.
def _refusals_fail(fn, *args):
try:
return fn(*args)
except _Refused as exc:
fail(escape(str(exc)))
def _check_directly_in_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`.
Up to Gitea #112 one subdirectory level was tolerated and ignored, so an
old `incoming/<type>/` habit kept working after #67 stopped reading the
type from it. A subdirectory is a source of its own now - accepted as a
whole - so a file inside one is refused with both ways out named.
"""
try:
rel = path.relative_to(incoming)
except ValueError:
fail(
raise _Refused(
f"{rel_path(path)} is not under incoming/ - `raw accept` and `raw fetch --html` "
"only take files from there. See raw/CONTRACT.md."
)
) from None
if len(rel.parts) < 1:
fail(f"incoming/{rel.as_posix()} names no file.")
if len(rel.parts) > 2:
fail(
f"incoming/{rel.as_posix()} is nested more than one level below incoming/ - place it "
"directly in incoming/, or in at most one subdirectory of it (the "
"subdirectory itself is ignored, see raw/CONTRACT.md)."
raise _Refused(f"incoming/{rel.as_posix()} names no file.")
if len(rel.parts) > 1:
top = rel.parts[0]
raise _Refused(
f"incoming/{rel.as_posix()} lies in a subdirectory of incoming/ - a file is taken "
"only from directly inside incoming/, and a subdirectory is a source of its own. "
f"Either accept the whole folder as one source (raw accept incoming/{top}), or move "
"the file up into incoming/ and accept it there. See raw/CONTRACT.md."
)
def _validate_under_incoming(path: Path, incoming: Path) -> None:
_refusals_fail(_check_directly_in_incoming, path, incoming)
_YEAR_DIR = re.compile(r"\d{4}")
@@ -185,6 +216,180 @@ def _stem_collision_message(claimed_name: str, holder: Path) -> str:
)
def _check_files(resolved: list[Path]) -> None:
incoming = _incoming_dir()
for path in resolved:
if not path.is_file():
raise _Refused(f"{rel_path(path)} does not exist or is not a file.")
_check_directly_in_incoming(path, incoming)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
raise _Refused("Two files share a filename; rename one before promoting.")
def _check_target(dst: Path, remedy: str) -> None:
problem = path_budget_problem_for(dst)
if problem:
raise _Refused(f"Cannot write {problem}. {remedy}")
if dst.exists():
raise _Refused(f"Cannot promote: {rel_path(dst)} already exists.")
def _plan_file_moves(
resolved: list[Path], existing_raw_paths: list[Path], page: Optional[str]
) -> list[tuple[Path, Path]]:
"""Where each file goes - the bundle decision, the path budget, an existing
target and the name rule, all checked before anything moves. `page` and
`existing_raw_paths` are the `--page` case; `raw pending` asks with
neither."""
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
# --page targets an existing page it always has >=1 raw file already
# (types/source.md requires raw_files:), so bundling always applies there.
total = len(existing_raw_paths) + len(resolved)
bundle_dir: Optional[Path] = None
if len(existing_raw_paths) >= 2:
parents = {p.parent for p in existing_raw_paths}
if len(parents) != 1:
raise _Refused(
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
"(see `sources coverage`)."
)
bundle_dir = parents.pop()
elif total >= 2:
if existing_raw_paths:
# Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
_check_target(
dst,
"The name comes from the file in incoming/: rename it there to something shorter "
"and accept it again.",
)
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# call is about to claim - the bundle's name, or the lone file's stem when
# no bundle forms - must not already belong to something this call does
# not itself own. "Owns" means: one of the page's already-registered raw
# files (the pitfall from the module docstring - a single file growing
# into a bundle of its own name momentarily still occupies that name), or,
# once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2:
owned.add(bundle_dir)
holder = occupied.get(claimed_name)
if holder is not None and holder not in owned:
raise _Refused(_stem_collision_message(claimed_name, holder))
return moves
def _walk_folder(folder: Path) -> tuple[list[Path], list[Path], list[str]]:
"""Every file below `folder`, every directory (`folder` included), and every
entry `raw accept` refuses to take - hidden or a symlink anywhere, or
neither a file nor a directory. Never descends into a refused entry, and
never follows a link."""
files: list[Path] = []
dirs: list[Path] = [folder]
refused: list[str] = []
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
base = Path(dirpath)
descend = []
for name in sorted(dirnames + filenames):
entry = base / name
if name.startswith("."):
refused.append(f"{rel_path(entry)} (hidden)")
elif entry.is_symlink():
refused.append(f"{rel_path(entry)} (symlink)")
elif entry.is_dir():
dirs.append(entry)
descend.append(name)
elif entry.is_file():
files.append(entry)
else:
refused.append(f"{rel_path(entry)} (neither a file nor a directory)")
dirnames[:] = descend
return sorted(files), dirs, refused
def _folder_collision_message(name: str, holder: Path) -> str:
return (
f'{rel_path(holder)} already claims the name "{name}" under raw/.\n'
" Rename the folder in incoming/ (add a distinguishing suffix) and accept it again.\n"
" A folder has no --replaces: a later edition of one file in it is replaced file by file."
)
def _plan_folder(folder: Path) -> tuple[Path, list[tuple[Path, Path]], list[Path]]:
"""The bundle directory for `raw accept incoming/<folder>`, every move into
it, and the directories to remove afterwards, deepest first - or
`_Refused`, before anything moves.
The invariant the cleanup rests on (Gitea #112): after the moves no file
is left below `folder`, so removing the emptied directories can never
take one with it. That holds only because every entry the move would
skip - hidden, a symlink, a special file - is refused here, up front."""
incoming = _incoming_dir()
try:
rel = folder.relative_to(incoming)
except ValueError:
raise _Refused(
f"{rel_path(folder)} is not under incoming/ - `raw accept` only takes a folder from "
"there. See raw/CONTRACT.md."
) from None
if len(rel.parts) != 1:
raise _Refused(
f"incoming/{rel.as_posix()} is not directly in incoming/ - a folder is accepted only "
f"as a whole, from the top: raw accept incoming/{rel.parts[0]}"
)
if folder.name.startswith(".") or folder.is_symlink():
raise _Refused(f"{rel_path(folder)} is hidden or a symlink - raw accept does not take it.")
files, dirs, refused = _walk_folder(folder)
if refused:
listed = "\n".join(f" - {entry}" for entry in refused)
raise _Refused(
f"{rel_path(folder)}/ holds entries raw accept does not take - hidden entries, "
f"symlinks and special files are refused, so nothing is left behind:\n{listed}\n"
" Remove or replace them in incoming/, then accept the folder again."
)
if not files:
raise _Refused(f"{rel_path(folder)}/ holds no file - nothing to accept.")
holder = _occupied_stems(config.RAW_DIR).get(folder.name)
if holder is not None:
raise _Refused(_folder_collision_message(folder.name, holder))
bundle_dir = _shard_dir() / folder.name
moves = [(src, bundle_dir / src.relative_to(folder)) for src in files]
for _src, dst in moves:
_check_target(
dst,
"The path comes from the folder in incoming/: shorten the folder's name or the "
"names inside it, and accept it again.",
)
return bundle_dir, moves, sorted(dirs, key=lambda d: len(d.parts), reverse=True)
def _capture_choices() -> tuple[list[str], list[str]]:
"""Allowed `--fidelity`/`--authority` values, straight from the schema
(single source of truth) - `unknown` excluded, since it is backfill-only
@@ -312,9 +517,55 @@ def _replace(
success("Review them in this same run: the replacement and their update belong in one commit.")
def _accept_folder(folder: Path, fidelity: str, authority: str, dry_run: bool) -> None:
"""`raw accept incoming/<folder>` - one folder, one source (Gitea #112).
Not atomic: one move per file, then the emptied directories. A failure
part-way leaves a half-accepted folder, which is reported, not resumed -
the call is not repeated (tool error contract, case 4)."""
bundle_dir, moves, dirs = _refusals_fail(_plan_folder, folder)
if dry_run:
for src, dst in moves:
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
typer.echo(f"[dry-run] would remove {rel_path(folder)}/ once it is empty")
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
return
for src, dst in moves:
dst.parent.mkdir(parents=True, exist_ok=True)
src.rename(dst)
typer.echo(f" moved {rel_path(src)} -> {rel_path(dst)}")
# rmdir, never a recursive delete: it refuses a directory that still holds
# anything, so this step can only ever remove what the moves emptied.
left = []
for directory in dirs:
try:
directory.rmdir()
except OSError:
left.append(directory)
if left:
typer.echo(
f"WARN {', '.join(rel_path(d) for d in left)} could not be removed - something appeared "
"in it during the move. Every file of the folder was moved; look at what is left."
)
raw_files_arg = ",".join(rel_path(dst) for _src, dst in moves)
success(
f"Promoted {rel_path(folder)}/ ({len(moves)} file(s)) to {rel_path(bundle_dir)}/.\n"
" Next:\n"
" tools/wikitool new source --name \"<Title>\" \\\n"
f" --set raw_files={raw_files_arg} \\\n"
f" --set fidelity={fidelity} --set authority={authority} \\\n"
" --set source_type=<category>\n"
" Past the thresholds in instructions/ingest-large-tree.md, continue there instead:\n"
f" tools/wikitool work new --input {rel_path(bundle_dir)}"
)
@cli_contract.record(cli_contract.CommandRecord(
path="raw accept",
summary="Promote one or more files from `incoming/` into `raw/`.",
summary="Promote one or more files, or one folder, from `incoming/` into `raw/`.",
synopsis=(
cli_contract.Variant(
usage='raw accept <file> [<file> ...] --fidelity <v> --authority <v> '
@@ -322,6 +573,11 @@ def _replace(
notes="Promote one or more files from `incoming/` into today's `raw/<YYYY>/<MM>/` "
"shard",
),
cli_contract.Variant(
usage="raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run]",
notes="Promote a whole folder as one source, its structure kept, into "
"`raw/<YYYY>/<MM>/<folder>/`",
),
cli_contract.Variant(
usage='raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] '
"[--dry-run]",
@@ -332,14 +588,23 @@ def _replace(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.NO,
atomic="`raw accept`: No - one filesystem move per file, then (with `--page`) one page "
"write. `raw accept --replaces`: No - one `unlink()` + one `rename()`, plus (if "
"`--fidelity`/`--authority` was given) one page write",
"write. With a folder: No - one move per file, then one `rmdir` per emptied directory; "
"a half-accepted folder is not resumed. `raw accept --replaces`: No - one `unlink()` + "
"one `rename()`, plus (if `--fidelity`/`--authority` was given) one page write",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"Promotes files from `incoming/` into `raw/<YYYY>/<MM>/`, computed from the accept "
"date rather than chosen by hand. A subdirectory under `incoming/` is tolerated and "
"ignored, not inspected - `raw/` does not address by type.",
"date rather than chosen by hand. A file argument must sit directly in `incoming/`; a "
"file inside a subdirectory is refused, since a subdirectory is a source of its own.",
"A folder argument (`incoming/<folder>`) is one source: every file below it moves to "
"`raw/<YYYY>/<MM>/<folder>/` at the same relative path, and the directories left empty "
"are removed - `incoming/<folder>` no longer exists afterwards. The folder name is the "
"bundle name; two `README.md` in different subfolders are no conflict.",
"A folder is accepted alone - no other argument, no `--page`, no `--replaces` - with "
"one `--fidelity`/`--authority` pair for all of it. It is refused, before anything "
"moves, if it is empty, or if a hidden entry (name starting with `.`), a symlink or a "
"special file sits anywhere below it; the refusal names each one.",
"One file promoted alone lands with no directory of its own; several files in one call "
"nest under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's stem.",
"`--fidelity`/`--authority` are required on a plain accept (`types describe source` "
@@ -373,13 +638,23 @@ def _replace(
failures=(
cli_contract.Failure(
label="raw accept",
cause="A file does not exist, is not under `incoming/`, or is nested more than one "
"level below it; two files in one call share a filename; a target path already "
"exists; or a target path would be over the path budget (160 UTF-16 code units "
"below the instance root)",
reaction="Fix the named argument and retry once. For a path over the budget, "
"rename the file in `incoming/` to something shorter - the refusal comes before "
"anything moves, so `incoming/` and `raw/` are unchanged",
cause="A file does not exist or is not directly in `incoming/`; two files in one "
"call share a filename; a target path already exists; or a target path would be "
"over the path budget (160 UTF-16 code units below the instance root)",
reaction="Fix the named argument and retry once. A file inside a subdirectory of "
"`incoming/` is accepted with its whole folder (`raw accept incoming/<folder>`) or "
"moved up into `incoming/` first. For a path over the budget, rename the file in "
"`incoming/` to something shorter - the refusal comes before anything moves, so "
"`incoming/` and `raw/` are unchanged",
),
cli_contract.Failure(
label="raw accept incoming/<folder>",
cause="The folder is not directly in `incoming/`, is combined with another argument, "
"`--page` or `--replaces`, holds no file, or holds a hidden entry, a symlink or a "
"special file; a target path is over the budget; or the folder name is already "
"occupied under `raw/`",
reaction="Nothing moved. Fix what the message names and retry once; for an occupied "
"name, rename the folder in `incoming/` - there is no `--replaces` for a folder",
),
cli_contract.Failure(
label="raw accept",
@@ -411,9 +686,9 @@ def _replace(
),
cli_contract.Failure(
label="raw accept --replaces",
cause="The incoming file does not exist or is not under `incoming/` (or is nested "
"more than one level below it), its filename differs from the target's, or the "
"target does not lie under `raw/` or does not exist",
cause="The incoming file does not exist or is not directly in `incoming/`, its "
"filename differs from the target's, or the target does not lie under `raw/` or "
"does not exist",
reaction="Fix the named argument and retry once - every check runs before the "
"filesystem is touched, so both files are exactly as they were",
),
@@ -429,6 +704,7 @@ def _replace(
"--authority reporting",
'tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting '
'--page "Source - Docker Cheatsheet"',
"tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting",
"tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md",
),
never=(
@@ -439,6 +715,7 @@ def _replace(
),
see_also=(
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - the rules and why",
"`wikitool raw pending` - what is waiting in `incoming/`, and which entry is next",
"`wikitool types describe source` - the capture field values",
"`wikitool new source` - the source page for a promoted file",
),
@@ -447,7 +724,8 @@ def _replace(
def raw_accept_command(
files: list[Path] = typer.Argument(
...,
help="One or more files under incoming/, all belonging to the same source",
help="One or more files directly in incoming/, all belonging to the same source - or "
"one folder in incoming/, accepted whole as one source",
),
fidelity: Optional[str] = typer.Option(
None,
@@ -476,13 +754,19 @@ def raw_accept_command(
),
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
):
"""Promote file(s) from incoming/ into raw/, computing the destination
(date shard, bundle or not, bundle name) instead of taking it as an
argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
"""Promote file(s) or one folder from incoming/ into raw/, computing the
destination (date shard, bundle or not, bundle name) instead of taking it
as an argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
if not files:
fail("Pass at least one file to promote.")
incoming = _incoming_dir()
resolved = [_resolve(f) for f in files]
folders = [p for p in resolved if p.is_dir()]
if folders and (len(files) != 1 or page is not None or replaces is not None):
fail(escape(
f"{rel_path(folders[0])} is a folder, and a folder is accepted alone: one source, "
"with no other argument, no --page and no --replaces. Accept it in a call of its own."
))
if replaces is not None:
_replace(files, replaces, page, fidelity, authority, dry_run)
@@ -498,16 +782,11 @@ def raw_accept_command(
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
resolved = [_resolve(f) for f in files]
if folders:
_accept_folder(folders[0], fidelity, authority, dry_run)
return
for path in resolved:
if not path.is_file():
fail(f"{rel_path(path)} does not exist or is not a file.")
_validate_under_incoming(path, incoming)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
fail("Two files share a filename; rename one before promoting.")
_refusals_fail(_check_files, resolved)
pages = None
target_page = None
@@ -532,66 +811,7 @@ def raw_accept_command(
"before promoting more."
)
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
# --page targets an existing page it always has >=1 raw file already
# (types/source.md requires raw_files:), so bundling always applies there.
total = len(existing_raw_paths) + len(resolved)
bundle_dir: Optional[Path] = None
if len(existing_raw_paths) >= 2:
parents = {p.parent for p in existing_raw_paths}
if len(parents) != 1:
fail(
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
"(see `sources coverage`)."
)
bundle_dir = parents.pop()
elif total >= 2:
if existing_raw_paths:
# Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
check_path_budget(
dst,
"The name comes from the file in incoming/: rename it there to something shorter "
"and accept it again.",
)
if dst.exists():
fail(f"Cannot promote: {rel_path(dst)} already exists.")
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# call is about to claim - the bundle's name, or the lone file's stem when
# no bundle forms - must not already belong to something this call does
# not itself own. "Owns" means: one of the page's already-registered raw
# files (the pitfall from the module docstring - a single file growing
# into a bundle of its own name momentarily still occupies that name), or,
# once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2:
owned.add(bundle_dir)
holder = occupied.get(claimed_name)
if holder is not None and holder not in owned:
fail(_stem_collision_message(claimed_name, holder))
moves = _refusals_fail(_plan_file_moves, resolved, existing_raw_paths, page)
moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
if moving_existing:
@@ -663,6 +883,181 @@ def raw_accept_command(
)
# --- raw pending --------------------------------------------------------------
#
# `incoming/` read as a queue (Gitea #112): what `wiki-ingest` without an
# argument works through, one candidate per run, oldest first so that newer
# material builds on - or corrects - what the wiki already took from older.
@dataclass(frozen=True)
class _Candidate:
kind: str # file | bundle | folder
paths: tuple[Path, ...]
files: int
mtime_ns: int
reason: Optional[str] # None when `raw accept` would take it as it stands
def label(self) -> str:
return ", ".join(rel_path(p) + ("/" if self.kind == "folder" else "") for p in self.paths)
def mtime_iso(self) -> str:
stamp = datetime.datetime.fromtimestamp(self.mtime_ns / 1e9).astimezone()
return stamp.isoformat(timespec="seconds")
def _refusal(fn, *args) -> Optional[str]:
try:
fn(*args)
except _Refused as exc:
return str(exc)
return None
def _folder_contents(folder: Path) -> tuple[int, int]:
"""How many non-directory entries sit below `folder`, and the newest
mtime among them - `lstat` only, no content read and no link followed."""
count, newest = 0, 0
for dirpath, dirnames, filenames in os.walk(folder, followlinks=False):
base = Path(dirpath)
for name in filenames + [d for d in dirnames if (base / d).is_symlink()]:
count += 1
newest = max(newest, (base / name).lstat().st_mtime_ns)
return count, newest
def _plain_files(paths: list[Path]) -> None:
_check_files(paths)
_plan_file_moves(paths, [], None)
def _pending_candidates() -> list[_Candidate]:
"""The candidates in `incoming/`, oldest first (Gitea #112 E1/E5).
Only top-level entries count; dotfiles and empty directories are none.
Top-level files sharing a stem are one `bundle` - a `raw fetch` pair, a PDF
and its converted text. A top-level folder is one `folder`, with every
file below it. A unit is as new as its newest part, so a bundle or folder
sorts by the newest mtime it holds; a tie goes by the path's bytes."""
incoming = _incoming_dir()
if not incoming.is_dir():
return []
candidates: list[_Candidate] = []
by_stem: dict[str, list[Path]] = {}
for entry in incoming.iterdir():
if entry.name.startswith("."):
continue
if entry.is_dir():
if entry.is_symlink():
count, newest = 1, entry.lstat().st_mtime_ns
else:
count, newest = _folder_contents(entry)
if count:
candidates.append(_Candidate(
"folder", (entry,), count, newest, _refusal(_plan_folder, entry)
))
else:
by_stem.setdefault(entry.stem, []).append(entry)
for paths in by_stem.values():
paths.sort(key=lambda p: os.fsencode(p.name))
candidates.append(_Candidate(
"file" if len(paths) == 1 else "bundle",
tuple(paths),
len(paths),
max(p.lstat().st_mtime_ns for p in paths),
_refusal(_plain_files, paths),
))
candidates.sort(key=lambda c: (c.mtime_ns, os.fsencode(c.paths[0].name)))
return candidates
@cli_contract.record(cli_contract.CommandRecord(
path="raw pending",
summary="List what waits in `incoming/`, oldest first, and name the entry an ingest without "
"an argument takes next.",
synopsis=(cli_contract.Variant(usage="raw pending [--json]"),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.READ,
idempotent=cli_contract.Idempotent.YES,
atomic="Read-only",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"A candidate is a top-level entry of `incoming/`: a single `file`, a `bundle` of "
"top-level files sharing a stem (a `raw fetch` pair, a PDF and its converted text), or "
"a `folder` with every file below it. Dotfiles, empty directories and their contents "
"are none; `mcp-upload/` is outside `incoming/` and never listed.",
"Order: oldest first by modification time. A bundle or folder counts as new as its "
"newest file; a tie goes by name. The mtime is when a document last changed only if "
"it was copied with its timestamps kept (`cp -p`, `rsync -a`, an unpacked archive) - "
"for a download or a `raw fetch` it is merely when it was dropped.",
"Each candidate shows its path(s), kind, file count and mtime, and whether `raw "
"accept` would take it as it stands - the same checks, minus `--fidelity`/"
"`--authority`. One it would refuse is listed with the reason and skipped: it needs a "
"human.",
"The default is the first candidate `raw accept` would take, and the output names it.",
"`--json` prints the same candidates in the same order: `kind`, `paths`, `files`, "
"`mtime`, `acceptable`, `reason`, `default`.",
"Reads directory listings and `lstat` only, never a file's content; an empty "
"`incoming/` is exit 0 with nothing to do.",
),
failures=(),
examples=(
"tools/wikitool raw pending",
"tools/wikitool raw pending --json",
),
see_also=(
"`wikitool raw accept` - promotes the chosen candidate",
"`instructions/wiki-ingest/SKILL.md` - ingest without an argument starts here",
"`raw/CONTRACT.md` \"Getting a file in: incoming/\" - candidates and order, and why",
),
))
@app.command("pending")
def raw_pending_command(
json_out: bool = typer.Option(False, "--json", help="Print the candidates as JSON"),
):
"""List the candidates waiting in incoming/, oldest first, and name the
default. See raw/CONTRACT.md "Getting a file in: incoming/"."""
candidates = _pending_candidates()
default = next((c for c in candidates if c.reason is None), None)
if json_out:
typer.echo(json.dumps([
{
"kind": c.kind,
"paths": [rel_path(p) for p in c.paths],
"files": c.files,
"mtime": c.mtime_iso(),
"acceptable": c.reason is None,
"reason": c.reason,
"default": c is default,
}
for c in candidates
], indent=2, ensure_ascii=False))
return
if not candidates:
typer.echo("Nothing is waiting in incoming/.")
return
typer.echo(f"{len(candidates)} candidate(s) in incoming/, oldest first:")
for number, c in enumerate(candidates, 1):
marker = "*" if c is default else " "
typer.echo(f"{marker} {number}. {c.kind:<6} {c.label()} ({c.files} file(s), {c.mtime_iso()})")
if c.reason is not None:
first, *rest = c.reason.splitlines()
typer.echo(f" not acceptable: {first}")
for line in rest:
typer.echo(f" {line}")
if default is None:
typer.echo("No candidate can be accepted as it stands - each needs a human (reasons above).")
return
waiting = len(candidates) - 1
typer.echo(
f"Default (*): {default.label()} - the oldest candidate raw accept would take; "
f"{waiting} more waiting after it."
)
# --- raw fetch ----------------------------------------------------------------
#
# The sanctioned intake for a URL the user names. It ends in `incoming/`, never
+296 -39
View File
@@ -1,10 +1,13 @@
import datetime
import json
import os
import re
import pytest
import typer
from chemenu import config
from chemenu.commands.raw_cmd import raw_accept_command
from chemenu.commands.raw_cmd import raw_accept_command, raw_pending_command
from chemenu.frontmatter_io import read_page, write_page
from chemenu.provenance import uncovered_raw_files
from chemenu.kb_scan import load_kb_pages
@@ -85,40 +88,21 @@ def test_file_directly_in_incoming_is_accepted(tree):
assert not src.exists()
def test_subdirectory_under_incoming_is_ignored_not_inspected(tree):
"""Gitea #67: a subdirectory of incoming/ - old habit, old script - is
tolerated and ignored rather than read as a type classification. This is
the MINOR condition named in the issue's Versionsteil."""
(tree / "incoming/videos").mkdir()
src = tree / "incoming/videos/clip.mp4"
src.write_bytes(b"x")
_accept(src)
assert (tree / _shard() / "clip.mp4").is_file()
assert not src.exists()
def test_nested_too_deep_is_rejected(tree):
nested = tree / "incoming/documents/sub"
nested.mkdir(parents=True)
src = nested / "deep.pdf"
@pytest.mark.parametrize("rel", ["documents/a.pdf", "documents/sub/a.pdf"])
def test_file_in_a_subdirectory_is_refused_naming_both_routes(tree, capsys, rel):
"""Gitea #112: a subdirectory of incoming/ is a source of its own, no
longer a tolerated, ignored type directory - a file inside one is refused,
and the message names accepting the folder and moving the file up."""
src = tree / "incoming" / rel
src.parent.mkdir(parents=True)
src.write_bytes(b"x")
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
def test_files_from_different_ignored_subdirs_bundle_together(tree):
"""No more per-call type agreement to enforce (Gitea #67): which ignored
subdirectory each file happened to sit under is irrelevant now."""
a = tree / "incoming/documents/a.pdf"
b = tree / "incoming/notes/a.md"
a.parent.mkdir(parents=True)
b.parent.mkdir(parents=True)
a.write_bytes(b"a")
b.write_text("b", encoding="utf-8")
_accept(a, b)
bundle = tree / _shard() / "a"
assert (bundle / "a.pdf").exists() and (bundle / "a.md").exists()
assert _tree_files(tree) == before
out = " ".join(capsys.readouterr().out.split())
assert "raw accept incoming/documents" in out
assert "move the file up into incoming/" in out
def test_same_file_passed_twice_is_rejected(tree):
@@ -573,15 +557,13 @@ def test_replaces_rejects_filename_mismatch(tree):
assert new.exists()
def test_replaces_across_legacy_directories_is_now_allowed(tree):
"""Gitea #67 removes the type-directory-match check `--replaces` used to
enforce: a subdirectory of incoming/ carries no meaning any more, so
replacing a raw/notes/ file with an incoming file dropped under an
unrelated incoming/documents/ works exactly like one dropped flat."""
def test_replaces_writes_back_into_a_legacy_directory(tree):
"""Gitea #67 removed the type-directory-match check `--replaces` used to
enforce: a raw/notes/ file is replaced by a file dropped flat into
incoming/ like any other."""
target = tree / "raw/notes/handbuch.md"
target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md"
new.parent.mkdir(parents=True)
new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8")
_accept(new, replaces=target, fidelity=None, authority=None)
@@ -590,6 +572,21 @@ def test_replaces_across_legacy_directories_is_now_allowed(tree):
assert not new.exists()
def test_replaces_refuses_an_incoming_file_in_a_subdirectory(tree):
"""Gitea #112: the subdirectory tolerance is gone for --replaces too."""
target = tree / "raw/notes/handbuch.md"
target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md"
new.parent.mkdir(parents=True)
new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old"
assert new.read_text(encoding="utf-8") == "new"
def test_replaces_rejects_nonexistent_target(tree):
new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8")
@@ -654,3 +651,263 @@ def test_accept_takes_a_target_at_the_path_budget(tree):
_accept(src)
assert (tree / _shard() / src.name).exists()
assert not src.exists()
# --- A folder is one source (Gitea #112) ---
def _make_folder(tree, name, files):
folder = tree / "incoming" / name
for rel, data in files.items():
(folder / rel).parent.mkdir(parents=True, exist_ok=True)
(folder / rel).write_bytes(data)
return folder
_BAUM = {"a.md": b"A", "sub/b.md": b"B", "sub/deep/README.md": b"R"}
def test_folder_moves_every_file_byte_identical_and_leaves_no_folder(tree):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder)
bundle = tree / _shard() / "baum"
assert {str(p.relative_to(bundle)): p.read_bytes() for p in bundle.rglob("*") if p.is_file()} == _BAUM
assert not folder.exists()
assert sorted(p.name for p in (tree / "incoming").iterdir()) == []
def test_folder_takes_same_named_files_in_different_subfolders(tree):
folder = _make_folder(tree, "doku", {"x/README.md": b"x", "y/README.md": b"y"})
_accept(folder)
bundle = tree / _shard() / "doku"
assert (bundle / "x/README.md").read_bytes() == b"x"
assert (bundle / "y/README.md").read_bytes() == b"y"
def test_folder_prints_raw_files_and_capture_pair(tree, capsys):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder, fidelity="secondhand", authority="opinion")
out = "".join(capsys.readouterr().out.split()) # success() wraps long lines
assert f"--setraw_files={_shard()}/baum/a.md,{_shard()}/baum/sub/b.md,{_shard()}/baum/sub/deep/README.md" in out
assert "--setfidelity=secondhand--setauthority=opinion" in out
@pytest.mark.parametrize("extra", ["sub/.hidden", ".git/config", "link"])
def test_folder_with_hidden_entry_or_symlink_is_refused_and_named(tree, capsys, extra):
folder = _make_folder(tree, "baum", _BAUM)
if extra == "link":
(folder / "sub/link").symlink_to(folder / "a.md")
named = "baum/sub/link"
else:
(folder / extra).parent.mkdir(parents=True, exist_ok=True)
(folder / extra).write_bytes(b"h")
named = f"baum/{extra.split('/')[0]}" if extra.startswith(".") else f"baum/{extra}"
before = _tree_files(tree)
links_before = sorted(str(p) for p in tree.rglob("*") if p.is_symlink())
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
assert sorted(str(p) for p in tree.rglob("*") if p.is_symlink()) == links_before
assert named in capsys.readouterr().out
def test_empty_folder_is_refused(tree):
(tree / "incoming/leer/sub").mkdir(parents=True)
with pytest.raises(typer.Exit):
_accept(tree / "incoming/leer")
assert (tree / "incoming/leer/sub").is_dir()
@pytest.mark.parametrize("combination", ["other-file", "page", "replaces"])
def test_folder_with_anything_else_is_refused(tree, combination):
folder = _make_folder(tree, "baum", _BAUM)
other = tree / "incoming/other.md"
other.write_text("o", encoding="utf-8")
(tree / "raw/notes/a.md").write_bytes(b"old")
_write_source(tree / "kb", "Source - Old", ["raw/notes/a.md"])
before = _tree_files(tree)
with pytest.raises(typer.Exit):
if combination == "other-file":
_accept(folder, other)
elif combination == "page":
_accept(folder, page="Source - Old")
else:
_accept(folder, replaces=tree / "raw/notes/a.md", fidelity=None, authority=None)
assert _tree_files(tree) == before
def test_folder_needs_both_capture_fields(tree):
folder = _make_folder(tree, "baum", _BAUM)
with pytest.raises(typer.Exit):
raw_accept_command(files=[folder], fidelity=None, authority="reporting", page=None, replaces=None, dry_run=False)
assert folder.is_dir()
def test_folder_name_already_occupied_is_refused_with_the_rename_route(tree, capsys):
(tree / "raw/documents/baum.pdf").write_bytes(b"pdf")
folder = _make_folder(tree, "baum", _BAUM)
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
out = " ".join(capsys.readouterr().out.split())
assert "Rename the folder in incoming/" in out
assert "--replaces raw/" not in out
def test_folder_target_over_the_path_budget_is_refused_before_any_move(tree):
long_name = "n" * (161 - len(f"{_shard()}/baum/sub/") - len(".md")) + ".md"
folder = _make_folder(tree, "baum", {"a.md": b"A", f"sub/{long_name}": b"L"})
before = _tree_files(tree)
with pytest.raises(typer.Exit):
_accept(folder)
assert _tree_files(tree) == before
def test_folder_dry_run_lists_every_move_and_the_folder_and_changes_nothing(tree, capsys):
folder = _make_folder(tree, "baum", _BAUM)
before = _tree_files(tree)
_accept(folder, dry_run=True)
assert _tree_files(tree) == before
out = capsys.readouterr().out
for rel in _BAUM:
assert f"incoming/baum/{rel} -> {_shard()}/baum/{rel}" in out
assert "would remove incoming/baum/" in out
def test_folder_files_are_uncovered_not_broken_afterwards(tree):
folder = _make_folder(tree, "baum", _BAUM)
_accept(folder)
from chemenu.provenance import broken_raw_refs
pages = load_kb_pages(tree / "kb")
assert sorted(f for f in uncovered_raw_files(tree / "raw", pages) if "/baum/" in f) == sorted(
f"{_shard()}/baum/{rel}" for rel in _BAUM
)
assert [r for r in broken_raw_refs(pages) if "baum" in str(r)] == []
def test_file_and_bundle_at_the_top_still_work(tree):
single = tree / "incoming/a.pdf"
single.write_bytes(b"a")
_accept(single)
assert (tree / _shard() / "a.pdf").read_bytes() == b"a"
html, md = tree / "incoming/post.html", tree / "incoming/post.md"
html.write_bytes(b"h")
md.write_bytes(b"m")
_accept(html, md)
assert sorted(p.name for p in (tree / _shard() / "post").iterdir()) == ["post.html", "post.md"]
# --- raw pending (Gitea #112) ---
def _pending(capsys):
capsys.readouterr()
raw_pending_command(json_out=True)
return json.loads(capsys.readouterr().out)
def _age(path, seconds):
for p in [path, *path.rglob("*")] if path.is_dir() else [path]:
os.utime(p, ns=(seconds * 10**9, seconds * 10**9), follow_symlinks=False)
def test_pending_on_an_empty_inbox_is_nothing(tree, capsys):
(tree / "incoming/.gitkeep").write_bytes(b"")
(tree / "incoming/documents").mkdir()
(tree / "incoming/notes/sub").mkdir(parents=True)
assert _pending(capsys) == []
raw_pending_command(json_out=False)
assert "Nothing is waiting" in capsys.readouterr().out
def test_pending_groups_a_fetch_pair_into_one_bundle(tree, capsys):
(tree / "incoming/post.html").write_bytes(b"h")
(tree / "incoming/post.md").write_bytes(b"m")
[candidate] = _pending(capsys)
assert candidate["kind"] == "bundle"
assert candidate["paths"] == ["incoming/post.html", "incoming/post.md"]
assert candidate["files"] == 2
def test_pending_counts_a_folder_once_with_all_its_files(tree, capsys):
_make_folder(tree, "baum", _BAUM)
[candidate] = _pending(capsys)
assert candidate["kind"] == "folder"
assert candidate["paths"] == ["incoming/baum"]
assert candidate["files"] == 3
def test_pending_orders_oldest_first_by_the_newest_part_then_by_name(tree, capsys):
a = tree / "incoming/a.md"
a.write_bytes(b"a")
_age(a, 3000)
b_html, b_md = tree / "incoming/b.html", tree / "incoming/b.md"
b_html.write_bytes(b"h")
b_md.write_bytes(b"m")
_age(b_html, 1000)
_age(b_md, 4000) # the bundle is as new as its newest part
folder = _make_folder(tree, "c", {"x.md": b"x", "y/z.md": b"z"})
_age(folder, 1000)
_age(folder / "y/z.md", 2000)
tie = tree / "incoming/0-tie.md"
tie.write_bytes(b"t")
_age(tie, 2000)
order = [c["paths"][0] for c in _pending(capsys)]
assert order == ["incoming/0-tie.md", "incoming/c", "incoming/a.md", "incoming/b.html"]
def test_pending_marks_what_raw_accept_would_refuse_and_defaults_to_the_first_acceptable(tree, capsys):
hidden = _make_folder(tree, "hidden", {"a.md": b"a", ".DS_Store": b"x"})
_age(hidden, 1000)
(tree / "raw/documents/taken.pdf").write_bytes(b"pdf")
taken = tree / "incoming/taken.md"
taken.write_bytes(b"t")
_age(taken, 2000)
long = tree / "incoming" / _incoming_name_for(161)
long.write_bytes(b"l")
_age(long, 3000)
fine = tree / "incoming/fine.md"
fine.write_bytes(b"f")
_age(fine, 4000)
candidates = _pending(capsys)
assert [c["acceptable"] for c in candidates] == [False, False, False, True]
assert ".DS_Store" in candidates[0]["reason"]
assert "already claims" in candidates[1]["reason"]
assert "160" in candidates[2]["reason"] or "budget" in candidates[2]["reason"].lower()
assert [c["default"] for c in candidates] == [False, False, False, True]
raw_pending_command(json_out=False)
out = capsys.readouterr().out
assert "Default (*): incoming/fine.md" in out
assert "not acceptable" in out
def test_pending_changes_nothing_not_even_mtimes(tree, capsys):
_make_folder(tree, "baum", _BAUM)
(tree / "incoming/post.html").write_bytes(b"h")
(tree / "raw/documents/old.pdf").write_bytes(b"o")
def snapshot():
return {
str(p.relative_to(tree)): (p.read_bytes() if p.is_file() else None, p.lstat().st_mtime_ns)
for root in ("incoming", "raw", "kb") for p in sorted((tree / root).rglob("*"))
}
before = snapshot()
_pending(capsys)
raw_pending_command(json_out=False)
assert snapshot() == before
def test_pending_json_and_text_list_the_same_candidates_in_the_same_order(tree, capsys):
for name, age in (("a.md", 3000), ("b.md", 1000), ("c.md", 2000)):
(tree / "incoming" / name).write_bytes(b"x")
_age(tree / "incoming" / name, age)
paths = [c["paths"][0] for c in _pending(capsys)]
raw_pending_command(json_out=False)
text = capsys.readouterr().out
listed = [m.group(1) for m in re.finditer(r"^[* ] \d+\. \w+\s+(\S+)", text, re.MULTILINE)]
assert listed == paths
+12
View File
@@ -371,6 +371,18 @@ def test_html_outside_incoming_is_refused(tree):
assert list((tree / "incoming").iterdir()) == []
def test_html_in_a_subdirectory_of_incoming_is_refused(tree, capsys):
"""A subdirectory of incoming/ is a source of its own (Gitea #112), so a
saved page inside one is not derived in place."""
saved = tree / "incoming/articles/post.html"
saved.parent.mkdir()
saved.write_bytes(ARTICLE)
before = _snapshot(tree)
_refused(html=saved, source_url="https://example.org/post")
assert _snapshot(tree) == before
assert "raw accept incoming/articles" in " ".join(capsys.readouterr().out.split())
def test_html_without_url_is_refused(tree):
saved = tree / "incoming/post.html"
saved.write_bytes(ARTICLE)