torben/chemenu
1
1 Fork 0
Code Issues 35 Pull Requests Actions Packages Releases 37 Activity

raw accept: Datums-Shard statt Typverzeichnis, fidelity/authority am Drop-Punkt (Teil 1/3, #67)
CI / verify (push) Successful in 52s
Details
Release / release (push) Successful in 35s
Details

Browse Source 
Files changed:
- .gitignore
- CHANGES.md
- VERSION
- instructions/bootstrap.md
- instructions/wiki-ingest/SKILL.md
- kb/CONTRACT.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/new_page.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/commands/touch.py
- tools/chemenu/lint_core.py
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_docs_verify.py
- tools/chemenu/tests/test_lint.py
- tools/chemenu/tests/test_new_page.py
- tools/chemenu/tests/test_provenance.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_touch.py
- tools/chemenu/tests/test_type_resolver.py
- tools/chemenu/type_resolver.py
- types/source.md
- types/source.schema.yaml
This commit is contained in:
torben committed 2026-09-08 21:42:27 +02:00
1 parent f2a093bc8b
commit f4353ccfb3
25 files changed
+1167 -344
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/chemenu/commands/raw_cmd.py
+248 -82
View File
@@ -1,34 +1,61 @@
"""`wikitool raw accept` - promote one or more files from `incoming/` into
`raw/`, with the destination computed rather than chosen by hand (Gitea #58).
`raw/`, with the destination computed rather than chosen by hand (Gitea #58),
sharded by the accept date rather than by a hand-picked type (Gitea #67).
A human classifies a file only by which type subdirectory of `incoming/` they
drop it into - `incoming/articles/`, `incoming/documents/`, `incoming/notes/`,
`incoming/assets/`, mirroring raw/CONTRACT.md's routing table. Everything past
that is this command's job:
A human no longer classifies a file at all: `incoming/` is flat, and a
subdirectory dropped under it (an old `incoming/documents/` habit, a script
that still writes one) is accepted and ignored rather than inspected -
promoting `raw/` from a routing decision to an address computed purely from
*when* the file was accepted:
- **Single file, no bundle.** One file promoted alone lands as
`raw/<type>/<name>` - no directory of its own.
`raw/<YYYY>/<MM>/<name>` - no directory of its own.
- **Bundle from the second file on.** Several files of one source promoted in
the same call land under `raw/<type>/<stem>/`, named after the first file's
stem.
the same call land under `raw/<YYYY>/<MM>/<stem>/`, named after the first
file's stem.
- **Growing an existing single file into a bundle.** `--page` extends an
existing source page's `raw_files:`. If that raises the page from one file
to more than one, the file it already had is folded into the new bundle
alongside the ones just promoted, in the same call - at no point does
`raw_files:` point at a path that does not exist.
to more than one, the file it already had is folded into a bundle at its
own parent directory - `raw/<its-existing-location>/<stem>/` - not at
today's shard, so a bundle never mixes an old capture date with today's
(Gitea #67 decision, "Datums-Shard" § "Bündelort").
Existing files under `raw/` are never moved by this change (Gitea #67
"Altbestand bleibt stehen"): `raw/articles/`, `raw/documents/`, `raw/notes/`
and `raw/assets/` keep whatever they already held, and stay valid promotion
targets for `--replaces`.
**Every promotion now also carries `--fidelity` and `--authority`** (Gitea
#67): how faithfully the material was captured, and what it is entitled to
claim about its subject. Neither has a default and neither may be `unknown`
here - that value is backfill-only, written only by `wikitool touch` on a
page predating this rule. Given `--page`, both are written straight onto the
target page (once - fill-once, like `touch`, see `touch._capture_field_or_fail`);
without `--page` there is no page yet to write them onto (`wiki-ingest`
creates it afterwards), so this command instead prints the exact
`wikitool new source --set fidelity=... --set authority=...` follow-up line,
and `new source` itself refuses to scaffold a source page without both.
`--replaces` is the one path where both become *optional*: passing them there
is the sanctioned way to correct an already-set capture value on a later
edition, the same way `touch`'s own refusal points back to `--replaces`.
Multi-owner raw files (`provenance.duplicate_raw_file_owners`) are refused
rather than silently moved: relocating a file another page also claims would
break that page's `raw_files:` without it ever being consulted.
**Stem uniqueness at `raw/<type>/` level** (Gitea #64) closes the gap this
leaves open: without it, a second, unrelated source whose primary file happens
not to collide on the exact filename slips silently into an existing bundle,
because the per-file `dst.exists()` check above never looks at the bundle
directory itself. The set of names occupied at `raw/<type>/` level - file
stems and bundle directory names alike - must stay unique; `_occupied_stems()`
and the check built on it enforce that, while still allowing a call to grow a
bundle it already owns (via `--page`, or by continuing an existing bundle).
**Stem uniqueness across the whole of `raw/`** (Gitea #64, widened by #67):
without it, a second, unrelated source whose primary file happens not to
collide on the exact filename slips silently into an existing bundle, because
the per-file `dst.exists()` check above never looks at the bundle directory
itself. The set of names occupied anywhere under `raw/` - file stems and
bundle directory names alike, at whichever level directly holds files - must
stay unique; `_occupied_stems()` and the check built on it enforce that,
while still allowing a call to grow a bundle it already owns (via `--page`,
or by continuing an existing bundle). It used to be scoped to one type
directory; #67 removes type directories from the addressing scheme entirely,
so uniqueness now has to span old-style flat directories and the new date
shard together, or `--replaces` on a bygone stem would be ambiguous across
shards.
**`--replaces <raw-path>`** is the only sanctioned way past that rule: whether
a new file is a later edition of an existing source or a second, separate one
@@ -37,6 +64,8 @@ and names both routes rather than choosing one (Gitea #64 decision 2).
"""
from __future__ import annotations
import datetime
import re
from pathlib import Path
from typing import Optional
@@ -44,13 +73,20 @@ import typer
from chemenu import config
from chemenu.commands._util import fail, rel_path, success
from chemenu.commands.dist_cmd import RAW_SUBDIRS
from chemenu.frontmatter_io import write_page
from chemenu.kb_scan import load_kb_pages
from chemenu.provenance import citing_pages, source_pages_by_raw_file, source_raw_files
from chemenu.type_resolver import resolver
app = typer.Typer(help="Promote raw material out of incoming/ into raw/.")
# The type this command always promotes into eventually - hardcoded rather
# than derived from --page, because there is no page yet in the common case
# (see module docstring): the follow-up hint and the capture-field/enum
# validation below both need *a* type to read from, and `source` is the only
# one `raw accept` has ever scaffolded towards.
_SOURCE_TYPE_PATH = "types/source.md"
def _incoming_dir() -> Path:
return config.ROOT / "incoming"
@@ -60,9 +96,24 @@ def _resolve(raw: Path) -> Path:
return raw if raw.is_absolute() else config.ROOT / raw
def _classify(path: Path, incoming: Path) -> str:
"""The type subdirectory `path` (already resolved, absolute) declares by
where it sits under `incoming/`, or fail with the reason it doesn't."""
def _shard_dir(today: Optional[datetime.date] = None) -> Path:
"""`raw/<YYYY>/<MM>/`, computed from the accept date - never from a count,
so it can never rebalance and break a `[^cite-id]` anchor (Gitea #67).
"""
d = today or datetime.date.today()
return config.RAW_DIR / f"{d.year:04d}" / f"{d.month:02d}"
def _validate_under_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`, or exactly one level below it.
Unlike before #67, that one optional level carries no meaning any more -
it is accepted and ignored, kept only so an old `incoming/<type>/` habit
or script does not have to change to keep working (the MINOR condition
named in Gitea #67 "Versionsteil"). Nesting deeper than that is still
refused: it was never meaningful and silently accepting it would hide a
typo'd path.
"""
try:
rel = path.relative_to(incoming)
except ValueError:
@@ -70,37 +121,58 @@ def _classify(path: Path, incoming: Path) -> str:
f"{rel_path(path)} is not under incoming/ - `raw accept` only promotes files "
"from there. See raw/CONTRACT.md."
)
allowed = ", ".join(f"incoming/{s}/" for s in RAW_SUBDIRS)
if len(rel.parts) < 2:
fail(
f"incoming/{rel} declares no type - place it inside one of {allowed} instead "
"of directly in incoming/."
)
sub = rel.parts[0]
if sub not in RAW_SUBDIRS:
fail(f"incoming/{rel} lies under an unknown type directory 'incoming/{sub}/'. Allowed: {allowed}.")
if len(rel.parts) < 1:
fail(f"incoming/{rel} names no file.")
if len(rel.parts) > 2:
fail(f"incoming/{rel} is nested below its type directory - place it directly in incoming/{sub}/.")
return sub
fail(
f"incoming/{rel} is nested more than one level below incoming/ - place it "
"directly in incoming/, or in at most one subdirectory of it (the "
"subdirectory itself is ignored, see raw/CONTRACT.md)."
)
def _occupied_stems(raw_sub_dir: Path) -> dict[str, Path]:
"""Every name occupied at `raw/<type>/` level: file stems and bundle
directory names alike, one level below `raw_sub_dir` only."""
_YEAR_DIR = re.compile(r"\d{4}")
def _occupied_stems(raw_dir: Path) -> dict[str, Path]:
"""Every name occupied anywhere under `raw/`: file stems and bundle
directory names alike, at whichever level directly holds files.
Distinguishes the pre-#67 flat layout (`raw/<name>/<file-or-bundle>`)
from the #67 date shard (`raw/<YYYY>/<MM>/<file-or-bundle>`) structurally,
by whether a top-level directory name is a 4-digit year - not from a
hardcoded list of legacy type names, so this keeps working unchanged if a
legacy directory is ever renamed by hand.
"""
occupied: dict[str, Path] = {}
if raw_sub_dir.is_dir():
for entry in raw_sub_dir.iterdir():
occupied[entry.stem if entry.is_file() else entry.name] = entry
if not raw_dir.is_dir():
return occupied
for top in raw_dir.iterdir():
if not top.is_dir():
continue # raw/CONTRACT.md
if _YEAR_DIR.fullmatch(top.name):
for month in top.iterdir():
if month.is_dir():
occupied.update(_leaf_group_entries(month))
else:
occupied.update(_leaf_group_entries(top))
return occupied
def _stem_collision_message(sub: str, claimed_name: str, holder: Path) -> str:
def _leaf_group_entries(group_dir: Path) -> dict[str, Path]:
"""The addressable entries directly inside one leaf-group directory
(a legacy type dir, or one `raw/<YYYY>/<MM>/`): a lone file occupies its
stem, a bundle directory occupies its own name."""
return {entry.stem if entry.is_file() else entry.name: entry for entry in group_dir.iterdir()}
def _stem_collision_message(claimed_name: str, holder: Path) -> str:
example_target = holder
if holder.is_dir():
children = sorted(holder.iterdir())
example_target = children[0] if children else holder
return (
f'{rel_path(holder)} already claims the stem "{claimed_name}" in raw/{sub}/.\n'
f'{rel_path(holder)} already claims the stem "{claimed_name}" under raw/.\n'
" These are two different intents and only you can tell them apart:\n"
f" Same source, new edition -> tools/wikitool raw accept "
f"--replaces {rel_path(example_target)} <incoming file>\n"
@@ -110,10 +182,53 @@ def _stem_collision_message(sub: str, claimed_name: str, holder: Path) -> str:
)
def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bool) -> None:
def _capture_choices() -> tuple[list[str], list[str]]:
"""Allowed `--fidelity`/`--authority` values, straight from the schema
(single source of truth) - `unknown` excluded, since it is backfill-only
and neither flag may write it (Gitea #67)."""
fidelity = [v for v in resolver.get_enum(_SOURCE_TYPE_PATH, "fidelity") if v != "unknown"]
authority = [v for v in resolver.get_enum(_SOURCE_TYPE_PATH, "authority") if v != "unknown"]
return fidelity, authority
def _check_capture_value(field: str, value: Optional[str], allowed: list[str]) -> None:
if value is None:
return
if value == "unknown" or value not in allowed:
fail(
f"--{field} must be one of: {', '.join(allowed)}. 'unknown' is backfill-only - only "
"`wikitool touch` may write it, on a page predating this rule."
)
def _overwrite_capture_fields_on_page(page, fidelity: Optional[str], authority: Optional[str]) -> bool:
"""Write `fidelity`/`authority` onto `page.frontmatter`, unconditionally -
only called from `_replace`, the one sanctioned way to correct an
already-set capture value (Gitea #67). Returns whether anything changed,
so the caller only writes the page back when it needs to."""
changed = False
for field, value in (("fidelity", fidelity), ("authority", authority)):
if value is not None and page.frontmatter.get(field) != value:
page.frontmatter[field] = value
changed = True
return changed
def _replace(
files: list[Path],
replaces: Path,
page: Optional[str],
fidelity: Optional[str],
authority: Optional[str],
dry_run: bool,
) -> None:
"""`raw accept --replaces <target> <incoming file>` - overwrite an
existing raw/ file wholesale with a later edition, in place, with
`raw_files:` on every source page left untouched (Gitea #64 decision 2).
`--fidelity`/`--authority` are optional here, and are the one sanctioned
way to correct an already-set capture value (Gitea #67) - passed, they
overwrite the owning page's value; omitted, the page's capture fields are
left exactly as they were.
Every check below runs before any write, so a failure leaves both the
target and the incoming file exactly as they were - the invariant the
@@ -124,24 +239,22 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
if len(files) != 1:
fail("--replaces takes exactly one incoming file - a replacement is one file for one file.")
fidelity_choices, authority_choices = _capture_choices()
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
incoming_path = _resolve(files[0])
if not incoming_path.is_file():
fail(f"{rel_path(incoming_path)} does not exist or is not a file.")
incoming_sub = _classify(incoming_path, _incoming_dir())
_validate_under_incoming(incoming_path, _incoming_dir())
target = _resolve(replaces)
try:
target_rel = target.relative_to(config.RAW_DIR)
target.relative_to(config.RAW_DIR)
except ValueError:
fail(f"--replaces target {rel_path(target)} does not lie under raw/.")
if not target.is_file():
fail(f"--replaces target {rel_path(target)} does not exist or is not a file.")
target_sub = target_rel.parts[0]
if target_sub != incoming_sub:
fail(
f"--replaces target is under raw/{target_sub}/, but {rel_path(incoming_path)} is "
f"under incoming/{incoming_sub}/ - a replacement stays within one type directory."
)
if target.name != incoming_path.name:
fail(
f"--replaces target {rel_path(target)} has a different filename than "
@@ -157,13 +270,25 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
f"({', '.join(owners)}). Resolve the multiple ownership first (see `wikitool sources coverage`)."
)
owner_page = pages.get(owners[0]) if owners else None
will_update_capture = owner_page is not None and _overwrite_capture_fields_on_page(
owner_page, fidelity, authority
)
if dry_run:
typer.echo(f"[dry-run] would replace {rel_path(target)} with {rel_path(incoming_path)}. No files written.")
if will_update_capture:
typer.echo(
f"[dry-run] would set fidelity={fidelity!r}, authority={authority!r} on '{owners[0]}'"
)
return
target.unlink()
incoming_path.rename(target)
if will_update_capture:
write_page(owner_page.path, owner_page.frontmatter, owner_page.body)
if not owners:
success(
f"Replaced {rel_path(target)} (previous version stays in git history). "
@@ -188,7 +313,20 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
def raw_accept_command(
files: list[Path] = typer.Argument(
...,
help="One or more files under incoming/<type>/, all belonging to the same source",
help="One or more files under incoming/, all belonging to the same source",
),
fidelity: Optional[str] = typer.Option(
None,
"--fidelity",
help="How faithful the capture is (see `wikitool types describe source`). Required unless "
"--replaces is given, where it is the optional, sanctioned way to correct an already-set value",
),
authority: Optional[str] = typer.Option(
None,
"--authority",
help="What the material may claim about its subject (see `wikitool types describe source`). "
"Required unless --replaces is given, where it is the optional, sanctioned way to correct "
"an already-set value",
),
page: Optional[str] = typer.Option(
None,
@@ -205,7 +343,7 @@ def raw_accept_command(
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
):
"""Promote file(s) from incoming/ into raw/, computing the destination
(type directory, bundle or not, bundle name) instead of taking it as an
(date shard, bundle or not, bundle name) instead of taking it as an
argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
if not files:
fail("Pass at least one file to promote.")
@@ -213,27 +351,30 @@ def raw_accept_command(
incoming = _incoming_dir()
if replaces is not None:
_replace(files, replaces, page, dry_run)
_replace(files, replaces, page, fidelity, authority, dry_run)
return
fidelity_choices, authority_choices = _capture_choices()
if fidelity is None or authority is None:
missing = ", ".join(n for n, v in (("--fidelity", fidelity), ("--authority", authority)) if v is None)
fail(
f"{missing} required - raw accept does not guess how faithful a capture is or what it "
f"may claim. --fidelity: {', '.join(fidelity_choices)}. --authority: {', '.join(authority_choices)}."
)
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
resolved = [_resolve(f) for f in files]
for path in resolved:
if not path.is_file():
fail(f"{rel_path(path)} does not exist or is not a file.")
subs = {_classify(path, incoming) for path in resolved}
if len(subs) > 1:
allowed = ", ".join(sorted(f"incoming/{s}/" for s in subs))
fail(f"All files in one `raw accept` call must share one type directory; got {allowed}.")
sub = subs.pop()
_validate_under_incoming(path, incoming)
names = [path.name for path in resolved]
if len(names) != len(set(names)):
fail("Two files share a filename; rename one before promoting.")
raw_sub_dir = config.RAW_DIR / sub
pages = None
target_page = None
existing_raw_paths: list[Path] = []
@@ -256,12 +397,6 @@ def raw_accept_command(
f"{', '.join(rel_path(p) for p in missing)}. Fix raw_files: (see `sources coverage`) "
"before promoting more."
)
existing_subs = {p.relative_to(config.RAW_DIR).parts[0] for p in existing_raw_paths}
if existing_subs != {sub}:
fail(
f"'{page}' already claims file(s) under {', '.join(sorted(f'raw/{s}/' for s in existing_subs))}, "
f"not raw/{sub}/. A bundle is one type directory; promote separately."
)
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
@@ -278,36 +413,46 @@ def raw_accept_command(
)
bundle_dir = parents.pop()
elif total >= 2:
primary = existing_raw_paths[0] if existing_raw_paths else resolved[0]
bundle_dir = raw_sub_dir / primary.stem
if existing_raw_paths:
# Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (raw_sub_dir / new_path.name)
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
if dst.exists():
fail(f"Cannot promote: {rel_path(dst)} already exists.")
# Stem uniqueness at raw/<type>/ level (Gitea #64): the name this call is
# about to claim there - the bundle's name, or the lone file's stem when no
# bundle forms - must not already belong to something this call does not
# itself own. "Owns" means: one of the page's already-registered raw files
# (the pitfall from the module docstring - a single file growing into a
# bundle of its own name momentarily still occupies that name), or, once a
# bundle already has >=2 registered files, the bundle directory itself.
# Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# call is about to claim - the bundle's name, or the lone file's stem when
# no bundle forms - must not already belong to something this call does
# not itself own. "Owns" means: one of the page's already-registered raw
# files (the pitfall from the module docstring - a single file growing
# into a bundle of its own name momentarily still occupies that name), or,
# once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(raw_sub_dir)
occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2:
owned.add(bundle_dir)
holder = occupied.get(claimed_name)
if holder is not None and holder not in owned:
fail(_stem_collision_message(sub, claimed_name, holder))
fail(_stem_collision_message(claimed_name, holder))
moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
if moving_existing:
@@ -325,6 +470,19 @@ def raw_accept_command(
"Resolve the multiple ownership first (see `wikitool sources coverage`)."
)
if target_page is not None:
current_fidelity = target_page.frontmatter.get("fidelity")
current_authority = target_page.frontmatter.get("authority")
for field, value, current in (
("fidelity", fidelity, current_fidelity),
("authority", authority, current_authority),
):
if current not in (None, "") and current != value:
fail(
f"'{page}' already has {field}={current!r} - a capture field is fixed once. "
"Use `raw accept --replaces` to correct it."
)
if dry_run:
for src, dst in moves:
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
@@ -332,6 +490,7 @@ def raw_accept_command(
moved_map = dict(moves)
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
typer.echo(f"[dry-run] would set raw_files: on '{page}' to {[rel_path(p) for p in final]}")
typer.echo(f"[dry-run] would set fidelity={fidelity!r}, authority={authority!r} on '{page}'")
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
return
@@ -344,6 +503,8 @@ def raw_accept_command(
if target_page is not None:
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
target_page.frontmatter["raw_files"] = [rel_path(p) for p in final]
target_page.frontmatter["fidelity"] = fidelity
target_page.frontmatter["authority"] = authority
write_page(target_page.path, target_page.frontmatter, target_page.body)
success(
f"Promoted {len(resolved)} file(s); updated raw_files: on '{page}' "
@@ -352,7 +513,12 @@ def raw_accept_command(
return
promoted = ", ".join(rel_path(moved_map[p]) for p in resolved)
raw_files_arg = ",".join(rel_path(moved_map[p]) for p in resolved)
success(
f"Promoted {len(resolved)} file(s) to {promoted}. "
"Run `wikitool new source --set raw_files=...` (or `touch --set` on an existing page) next."
f"Promoted {len(resolved)} file(s) to {promoted}.\n"
" Next:\n"
" tools/wikitool new source --name \"<Title>\" \\\n"
f" --set raw_files={raw_files_arg} \\\n"
f" --set fidelity={fidelity} --set authority={authority} \\\n"
" --set source_type=<category>"
)