raw accept: Datums-Shard statt Typverzeichnis, fidelity/authority am Drop-Punkt (Teil 1/3, #67)
CI / verify (push) Successful in 52s
Release / release (push) Successful in 35s

Files changed:
- .gitignore
- CHANGES.md
- VERSION
- instructions/bootstrap.md
- instructions/wiki-ingest/SKILL.md
- kb/CONTRACT.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/new_page.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/commands/touch.py
- tools/chemenu/lint_core.py
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_docs_verify.py
- tools/chemenu/tests/test_lint.py
- tools/chemenu/tests/test_new_page.py
- tools/chemenu/tests/test_provenance.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_touch.py
- tools/chemenu/tests/test_type_resolver.py
- tools/chemenu/type_resolver.py
- types/source.md
- types/source.schema.yaml
This commit is contained in:
2026-09-08 21:42:27 +02:00
parent f2a093bc8b
commit f4353ccfb3
25 changed files with 1167 additions and 344 deletions
+8 -7
View File
@@ -134,13 +134,14 @@ npm-debug.log*
/.agents/skills/ /.agents/skills/
/.claude/skills/ /.claude/skills/
# Ingest inbox (see raw/CONTRACT.md and Gitea #58). A human drops a file here # Ingest inbox (see raw/CONTRACT.md and Gitea #58, flattened by #67). A human
# under its declared type subdirectory; `wikitool raw accept` promotes it into # drops a file here - no subdirectory carries any meaning any more, an old
# `raw/`, computing the directory and any bundle from where it sits and how # one is merely tolerated and ignored; `wikitool raw accept` promotes it into
# many files move together. Unlike raw/ itself this must NEVER be committed - # `raw/`, computing the date-sharded directory and any bundle from what was
# the promotion is what makes a file immutable, not the drop - so this is the # passed in one call. Unlike raw/ itself this must NEVER be committed - the
# one directory pattern in this file that is deliberately not anchored back # promotion is what makes a file immutable, not the drop - so this is the one
# open by the content backstop below. # directory pattern in this file that is deliberately not anchored back open
# by the content backstop below.
/incoming/ /incoming/
# Content backstop - keep this block last. Nothing under raw/, kb/ or work/ may # Content backstop - keep this block last. Nothing under raw/, kb/ or work/ may
+82 -1
View File
@@ -35,7 +35,7 @@ dev-checkout concern - readable here, never shipped as something to parse.
--- ---
## 4.8.0-beta.7 - 2026-09-08 - source_type: Default streichen, unclassified als sichtbares Fach, layout: fuer source (schliesst #66) ## 4.8.0-beta.8 - 2026-09-08 - raw accept: Datums-Shard statt Typverzeichnis, fidelity/authority am Drop-Punkt (schliesst #67)
**Author:** Torben Nehmer **Author:** Torben Nehmer
@@ -48,6 +48,7 @@ dev-checkout concern - readable here, never shipped as something to parse.
- update entity naming conventions to use singular form for consistency - update entity naming conventions to use singular form for consistency
- kb/concepts/ bekommt Areas: layout: fuer concept, Area-Titel aus jedem Type-Spec, Schwellen-Empfehlung im lint (schliesst #59) - kb/concepts/ bekommt Areas: layout: fuer concept, Area-Titel aus jedem Type-Spec, Schwellen-Empfehlung im lint (schliesst #59)
- source_type: Default streichen, unclassified als sichtbares Fach, layout: fuer source (schliesst #66) - source_type: Default streichen, unclassified als sichtbares Fach, layout: fuer source (schliesst #66)
- raw accept: Datums-Shard statt Typverzeichnis, fidelity/authority am Drop-Punkt (schliesst #67)
<!-- /wikitool:bumps --> <!-- /wikitool:bumps -->
Das Label `status/incoming` gibt es seit heute in Gitea: der Mensch legt einen Das Label `status/incoming` gibt es seit heute in Gitea: der Mensch legt einen
@@ -418,6 +419,86 @@ maschinengelesenes Format. Kein `--breaking`, kein Migrationsdokument.
Schließt #66. Schließt #66.
**`raw accept`: Datums-Shard statt Typverzeichnis, `fidelity`/`authority` am Drop-Punkt** (#67,
Paket B von vier — A ist #66 oben, C ist #68, D ist #69): zwei unabhängige Befunde, ein
Codepfad.
Befund 1: `raw/CONTRACT.md`s vier Typverzeichnisse (`articles/`, `documents/`, `notes/`,
`assets/`) lösten keinen der drei Gründe ein, die einen Verzeichnis-Split rechtfertigen —
`raw/` wird nie durchgeblättert, keine Klausel dieser Datei galt je pro Verzeichnis, alle vier
verrotten gleich (unveränderlich, nie gelöscht). Der Split kostete real: der Mensch trifft beim
Ablegen in `incoming/<typ>/` eine Routing-Entscheidung, die später blind nach `source_type:`
abgeschrieben wird — genau darüber entstand der in #66 korrigierte Bias (`raw/notes/` hielt laut
altem Contract-Text „Gesprächsprotokolle", 16 der 25 Dateien dort waren tatsächlich Transkripte).
Befund 2: was am Drop-Punkt bekannt ist und danach nirgends mehr — wie treu eine Erfassung ist
und was das Material über seinen Gegenstand behaupten darf. Zwei neue, unabhängige Achsen auf
`types/source.md`: `fidelity` (`verbatim`/`published`/`secondhand`/`nontextual`) und `authority`
(`normative`/`reporting`/`opinion`), beide mit `unknown` als backfill-only-Wert.
Umgesetzt:
- `raw accept` adressiert eine Datei jetzt über `raw/<YYYY>/<MM>/`, berechnet aus dem
Annahmedatum — eine reine Funktion von etwas Unveränderlichem, kann also nie rebalancieren und
keinen `[^cite-id]`-Anker brechen. `incoming/` wird flach; ein Unterverzeichnis wird toleriert
und ignoriert statt inspiziert (alte `incoming/<typ>/`-Skripte laufen unverändert weiter).
Bestandsdateien in `raw/articles|documents|notes|assets/` bleiben unbewegt und weiter gültige
`--replaces`-Ziele — das Layout war nirgends versioniert, es gibt also keine „zwei Korpusformen".
Wächst eine bereits promotete Einzeldatei zum Bündel, entsteht das Bündel an ihrem eigenen
Speicherort, nie im heutigen Shard — ein Bündel aus altem und neuem Datum hätte keine
eindeutig richtige Adresse.
- Stem-Eindeutigkeit (#64) gilt jetzt global über `raw/`, nicht mehr pro Typverzeichnis —
ohne Typverzeichnisse als Grenze wäre die Prüfung sonst wirkungslos gegen ein Bündel in einem
anderen Shard oder einem Alt-Verzeichnis.
- `fidelity`/`authority`: neu in `types/source.schema.yaml`, ohne `default:` und bewusst
**nicht** in `required:` (sonst bricht jede bestehende Instanz an der Validierung — die
MINOR-Einstufung unten hängt daran). Erzwungen stattdessen im Werkzeug: `raw accept` verlangt
beide Flags immer; trägt der Aufruf `--page`, schreibt es sie direkt auf die Zielseite, sonst
druckt es die fertige `new source --set fidelity=... --set authority=...`-Folgezeile, und
`new source` verweigert seinerseits ohne beide Werte. `unknown` ist backfill-only — weder
`raw accept` noch `new source` dürfen es schreiben.
- Capture-Felder sind **fill-once**, nicht auf `touch.py`s `UNSETTABLE`-Denylist: eine Denylist
hätte auch den ersten (Backfill-)Schreibzugriff verboten, den der Migrationslauf braucht.
`touch --set <feld>=<wert>` schreibt nur, solange das Feld fehlt, und verweist danach auf
`raw accept --replaces` als einzigen Korrekturweg — der einzige Aufruf, der einen bereits
gesetzten Capture-Wert überschreiben darf, weil eine korrigierte Erfassung eine neue Edition
der Quelle ist, keine Bearbeitung der Seite. `types/source.md` deklariert die Feldliste selbst
(`capture_fields:`), gelesen über `TypeResolver.get_capture_fields` statt an drei Stellen
hartkodiert.
- `lint` bekommt einen neuen beratenden Befund, *Confidence Above Source Standing*: die
Autoritätsbewertung, die `kb/CONVENTIONS.md`s Confidence-Rubrik seit je verlangt („+0.1 für
offizielle Doku"), aber nirgends festhielt. Eine stackseitige Obergrenzentabelle in
`kb/CONTRACT.md` (`reporting` 0.8, `opinion` 0.6, `secondhand`/`nontextual` 0.7, `normative`/
`verbatim`/`published`/`unknown` ohne Obergrenze) begrenzt `confidence_base`, ersetzt es aber
nicht — eine Formel hätte zwei widersprechende Ableitungen derselben Zahl, und Autorität ist
eine Obergrenze, kein Determinant. Nicht in `HARD_ERROR_KEYS`. Auf dieser Instanz meldet der
Befund aktuell nichts: kein Bestand trägt die neuen Felder, das ist erwartet, nicht geprüft.
- Aufgeräumt: `dist_cmd.RAW_SUBDIRS` und der darauf laufende `docs verify`-Check
(`check_raw_subdirs`) entfallen ersatzlos, `raw/CONTRACT.md`s Routing-Tabelle beschreibt
stattdessen den Shard, die Ignore-Kanarie wandert von `incoming/documents/probe.pdf` auf
`incoming/probe.pdf`.
**MINOR**, `4.8.0-beta.8` desselben Kandidaten — drei geprüfte Bedingungen: `raw accept` nimmt
weiterhin Dateien aus `incoming/<irgendwas>/` an, statt sie zu verweigern; `fidelity`/`authority`
stehen nicht in `required:`; die zwei neuen Pflichtflags an `raw accept` sind eine
Verhaltensänderung, aber dieselbe Einstufung, die #66s `new source`-Verweigerung im selben
Kandidaten schon bekam. Kein `--breaking`, kein Migrationsdokument nötig — kein Bestand wird
durch diesen Bump ungültig.
Geändert: `raw/CONTRACT.md`, `types/source.md`, `types/source.schema.yaml`,
`tools/chemenu/commands/raw_cmd.py` (Neufassung), `tools/chemenu/commands/new_page.py`
(Capture-Feld-Pflicht), `tools/chemenu/commands/touch.py` (`_capture_field_or_fail`,
Fill-once), `tools/chemenu/type_resolver.py` (`get_capture_fields`),
`tools/chemenu/commands/dist_cmd.py`/`docs_verify.py` (Aufräumen), `tools/chemenu/lint_core.py`
(neuer Befund), `kb/CONTRACT.md` (Obergrenzentabelle), `tools/CONTRACT.md`, `.gitignore`,
`instructions/bootstrap.md`, `instructions/wiki-ingest/SKILL.md` (Schritt 1 und 6), zugehörige
Tests.
Absichtlich nicht in diesem Publish: der Backfill über die 31 bestehenden Source-Seiten (eigener
`work/`-Lauf nach `instructions/migrate-corpus.md`, eigenes Mass-Update-Gate) und die
Beleg-/Fundzeile im Changelog dazu — #67 bleibt bis dahin offen, dieser Bump deckt Schritt 1 der
im Issue festgehaltenen Drei-Publish-Abfolge.
--- ---
## 4.7.4 - 2026-09-04 - bootstrap.md nennt den session-id-WARN nach frischem Bootstrap explizit als erwartet ## 4.7.4 - 2026-09-04 - bootstrap.md nennt den session-id-WARN nach frischem Bootstrap explizit als erwartet
+1 -1
View File
@@ -1 +1 @@
4.8.0-beta.7 4.8.0-beta.8
+3 -3
View File
@@ -29,11 +29,11 @@ they are published: the agent harness will not offer `wiki-ingest`, `wiki-query`
``` ```
2. **Create the ingest inbox.** `incoming/` (raw/CONTRACT.md "Getting a file in") is gitignored, 2. **Create the ingest inbox.** `incoming/` (raw/CONTRACT.md "Getting a file in") is gitignored,
so a fresh clone has none of its type subdirectories - unlike `raw/` itself, which is so a fresh clone does not have it at all - unlike `raw/` itself, which is committed and
committed and present immediately: present immediately. It is flat since Gitea #67 (no type subdirectory to create):
```bash ```bash
mkdir -p incoming/{articles,documents,notes,assets} mkdir -p incoming
``` ```
`tools/wikitool doctor` only reports a missing one; it never creates it, so this is a one-off `tools/wikitool doctor` only reports a missing one; it never creates it, so this is a one-off
+27 -6
View File
@@ -19,20 +19,32 @@ pages should never have cost the concept contract. Field-level requirements alwa
## Steps ## Steps
1. **Promote from `incoming/` if that is where the file sits.** Read 1. **Promote from `incoming/` if that is where the file sits.** Read
[raw/CONTRACT.md](../../raw/CONTRACT.md) "Getting a file in" if you have not this session - [raw/CONTRACT.md](../../raw/CONTRACT.md) "Getting a file in" and "Capture fields" if you have
the directory and any bundling are computed, never chosen by hand: not this session - the directory and any bundling are computed, never chosen by hand, but the
two capture flags are not: `raw accept` refuses without them.
**Ask the user for `--fidelity` and `--authority` before this call, rather than guessing from
a quick look at the file.** This is the one place in the whole skill where the asymmetry from
`source_type` (step 6 below) does not hold: a wrong `source_type` sits in a visible,
correctable catalog slot (`unclassified`), but a guessed capture value is not "unknown" - it
is a claim about the capture that cannot be corrected later, because the knowledge exists only
at this drop point. If it is genuinely unclear how faithful the capture is or what the
material may claim about its subject, say so to the user and ask directly - do not fall back
to a plausible-looking default.
```bash ```bash
tools/wikitool raw accept incoming/<type>/<file> [incoming/<type>/<other-file> ...] tools/wikitool raw accept --fidelity <value> --authority <value> \
incoming/<file> [incoming/<other-file> ...]
``` ```
List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown) List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown)
in the same call, so they land bundled together rather than as two independent promotions. A in the same call, so they land bundled together rather than as two independent promotions. A
file already in `raw/` skips this step entirely. file already in `raw/` skips this step entirely. A subdirectory under `incoming/` (an old
`incoming/<type>/` habit) is tolerated and ignored - it carries no meaning any more.
**If this refuses because the name is already claimed** (a file stem or a bundle directory **If this refuses because the name is already claimed** (a file stem or a bundle directory
already occupies it at `raw/<type>/` level, Gitea #64), that is not this session's call to already occupies the name anywhere under `raw/`, Gitea #64/#67), that is not this session's
make: whether the incoming file is a later edition of the existing source or a second, call to make: whether the incoming file is a later edition of the existing source or a second,
separate one is a judgment about the world, and the command's message names both routes - separate one is a judgment about the world, and the command's message names both routes -
`--replaces` and renaming in `incoming/` - without recommending either. Show the message to `--replaces` and renaming in `incoming/` - without recommending either. Show the message to
the human and wait, the same way a session halts at an exit-42 gate (AGENTS.md invariant 6), the human and wait, the same way a session halts at an exit-42 gate (AGENTS.md invariant 6),
@@ -73,6 +85,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
tools/wikitool new source --name "<Title>" \ tools/wikitool new source --name "<Title>" \
--set source_type=<category> \ --set source_type=<category> \
--set raw_files=<path1>,<path2>,... \ --set raw_files=<path1>,<path2>,... \
--set fidelity=<value> --set authority=<value> \
--set source_language=<ISO 639-1 code of the raw material> \ --set source_language=<ISO 639-1 code of the raw material> \
--set entities=A,B,C --set concepts=D,E --set entities=A,B,C --set concepts=D,E
``` ```
@@ -85,6 +98,14 @@ pages should never have cost the concept contract. Field-level requirements alwa
`lint` finding, not a silent default, and `wikitool touch --set source_type=<value>` corrects `lint` finding, not a silent default, and `wikitool touch --set source_type=<value>` corrects
it later without moving or renaming the page. it later without moving or renaming the page.
`fidelity` and `authority` have no default either, and `new source` refuses without them the
same way (Gitea #67) - but unlike `source_type` there is no catalog slot to fall back on. If
step 1 already ran `raw accept` without `--page`, its success message printed the exact
`--set fidelity=... --set authority=...` pair to reuse here verbatim; if it did not (the file
was already in `raw/`), ask the user the same question step 1 asks before promoting, rather
than inferring an answer from the file's content now. Never pass `unknown` here - that value
is backfill-only, written only by `wikitool touch` on a page predating this rule.
List **every** raw file this ingest covers - a folder of related documents becomes one List **every** raw file this ingest covers - a folder of related documents becomes one
source page with all its files in `raw_files:`, not one page per file. For an external source page with all its files in `raw_files:`, not one page per file. For an external
article also pass `--set source_url=<upstream URL>`; `raw_files:` must still point at the article also pass `--set source_url=<upstream URL>`; `raw_files:` must still point at the
+34
View File
@@ -262,6 +262,40 @@ What the number *means* - the base score, what raises it and by how much, and ho
prose below a threshold - is a rubric rather than a mechanism, so it is prose below a threshold - is a rubric rather than a mechanism, so it is
[kb/CONVENTIONS.md § Confidence rubric](CONVENTIONS.md#confidence-rubric)'s. [kb/CONVENTIONS.md § Confidence rubric](CONVENTIONS.md#confidence-rubric)'s.
### Confidence against source standing
The rubric already asks for an authority judgment ("+0.1 for official documentation, +0.05 for a
reputable secondary source") without anywhere to record it - source count lives in `sources:`,
recency in `modified:`, agreement is readable from the prose itself, but the authority term is
thought, applied, and forgotten; the next person re-derives it from scratch. `raw/CONTRACT.md`'s
capture fields (`fidelity`, `authority`, Gitea #67) make it a recorded fact instead: `lint`
compares a page's `confidence_base` against the weakest capture standing of its cited sources and
reports one when the page claims more than its sources can carry.
**A ceiling, not a formula.** `confidence_base` stays hand-set (see above); this only bounds it.
The reasons are the same two, together: `confidence` is already derived from `confidence_base` by
one mechanism (`confidence decay`), and a second derivation of the same number would leave two
disagreeing answers to "what should this be" - and authority is a ceiling a page may fall under by
independent verification, never a determinant a formula could compute outright. The check is
therefore advisory, in the same class as `unsharded_collections`: it names a mismatch and asks a
human to look, it never fails a build.
The ceiling is stack vocabulary (`fidelity`/`authority` are - see `raw/CONTRACT.md`), not an
instance setting; an instance that disagrees with a specific finding overrides its judgment on
that one page, rather than reconfiguring the rubric:
| worst `authority` among a page's sources | ceiling | worst `fidelity` among a page's sources | ceiling |
|---|---|---|---|
| `normative` | (none) | `verbatim` | (none) |
| `reporting` | 0.8 | `published` | (none) |
| `opinion` | 0.6 | `secondhand` | 0.7 |
| `unknown` | no finding | `nontextual` | 0.7 |
| | | `unknown` | no finding |
The effective ceiling is the minimum of both columns. `unknown` never triggers a finding on its
own axis: a backfilled "we don't know" is not a claim about the source, and firing on it would
report every page citing a pre-#67 source at once, which is noise, not a finding.
## What does not belong here ## What does not belong here
- Raw source material - it stays immutable under `raw/`. - Raw source material - it stays immutable under `raw/`.
+111 -42
View File
@@ -11,62 +11,87 @@ checked against what was actually said.
wikilinks, no provenance and no confidence. They are untrusted input, and the top-level split wikilinks, no provenance and no confidence. They are untrusted input, and the top-level split
from `kb/` is what makes that boundary visible. from `kb/` is what makes that boundary visible.
## Directory routing ## Directory routing: a date shard, not a type
| Directory | Holds | `raw/` addresses a file by **when it was accepted**, never by what kind of document it is
|-----------|-------| (Gitea #67). A promotion lands under `raw/<YYYY>/<MM>/`, computed from the calendar month of the
| `articles/` | Web articles and blog posts (keep a local copy even when `source_url:` is set) | `raw accept` call that promoted it - a pure function of something immutable, so it can never
| `documents/` | PDFs, specifications, manuals, exported documentation sets | rebalance: an overflowing bucket would move files and break every `[^cite-id]` anchor pointing at
| `notes/` | Personal notes, meeting notes, conversation transcripts | them, and only a function of a fixed, past fact (the accept date) rules that out entirely. The
| `assets/` | Images, diagrams, configuration files, and other binaries | month it happened is also the one thing about a source that was previously recorded nowhere but
`git log`.
This table is read by `tools/wikitool docs verify` against `dist_cmd.RAW_SUBDIRS`, the single A mechanical shard is **an address, not a claim**. It cannot drift, cannot become wrong, cannot
place the four names are declared in code - the two are kept in sync mechanically rather than by turn into a collection bucket the way a hand-picked type directory did (see below) - which is
convention (Gitea #58). exactly why it is allowed to live on the total, exclusive surface (the directory), while the
semantic classification of what a source *is* moves to the portable one (frontmatter,
`source_type:` on the covering source page - see `types/source.md`).
**Why the type directories this replaced didn't earn their keep.** `articles/`, `documents/`,
`notes/` and `assets/` used to route every promotion, and none of the three reasons a directory
split is worth its cost ever applied to them: `raw/` is never browsed (access always goes through
`raw_files:`, `sources trace`, or `sources coverage` - the browsable surface is the generated
`kb/sources/INDEX.md`), the rules in this file never varied per directory, and nothing in them
ever decayed at a different rate (all four were equally immutable, never deleted). What the split
did cost was real: a human choosing `incoming/notes/` at drop time, then a later pass copying that
choice into `source_type:` by hand - which is exactly how a bias took hold. `raw/notes/` held
"personal notes, meeting notes, conversation transcripts" by this file's old wording; of the 25
files that landed there, 16 turned out to be transcripts, 4 LLM analyses, 2 tracker exports, and
only 3 actual notes. The catch-all formed in the raw layer and was carried straight into `kb/`.
**Pre-#67 files are not moved.** `raw/`'s directory layout was never versioned anywhere -
`.wikitool-kb.json` describes `kb/`'s shape, and `corpus_diff.py` compares `raw_files:` as a
*value*, never as directory structure - so there is no "two corpus forms at once" to reconcile,
only a shape that was simply never described. `raw/articles/`, `raw/documents/`, `raw/notes/` and
`raw/assets/` keep holding whatever they already held, indefinitely: `raw accept --replaces`
writes back to a file's existing location (the path is the identifier, Gitea #64), so a legacy
directory stays a valid promotion target as long as anything still lives there. `sources coverage`
walks `raw/` recursively and works unchanged either way.
## Getting a file in: `incoming/` ## Getting a file in: `incoming/`
`raw/` is never chosen by hand. A file to be ingested is dropped into the gitignored top-level `raw/` is never chosen by hand. A file to be ingested is dropped into the gitignored top-level
`incoming/`, under the subdirectory naming its type - `incoming/articles/`, `incoming/documents/`, `incoming/` - flat, since Gitea #67: no subdirectory carries any classification any more. A
`incoming/notes/`, `incoming/assets/`, mirroring the table above. That placement is the only subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
classification a human makes: **what kind of document this is**, not where it ends up on disk. never has to touch a caller), but it is **ignored**, never inspected:
Everything past it - the target directory, whether a bundle directory is needed, its name, the
move itself - is computed by `tools/wikitool raw accept`.
```bash ```bash
tools/wikitool raw accept incoming/documents/handbuch.pdf tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
# -> raw/documents/handbuch.pdf # -> raw/2026/09/handbuch.pdf
``` ```
**A bundle directory is created only from the second file onward.** One file promoted alone needs **A bundle directory is created only from the second file onward.** One file promoted alone needs
no directory of its own and lands as `raw/<type>/<name>`; promoting several files of one source in no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one
the same call nests them under `raw/<type>/<stem>/`, named after the first file's stem: source in the same call nests them under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's
stem:
```bash ```bash
tools/wikitool raw accept incoming/documents/handbuch.pdf incoming/documents/handbuch.md tools/wikitool raw accept --fidelity verbatim --authority reporting \
# -> raw/documents/handbuch/handbuch.pdf incoming/handbuch.pdf incoming/handbuch.md
# -> raw/documents/handbuch/handbuch.md # -> raw/2026/09/handbuch/handbuch.pdf
# -> raw/2026/09/handbuch/handbuch.md
``` ```
This is why the 29 files already in `raw/` sit directly under their type directory rather than **Growing an existing single file into a bundle forms it at that file's own location, never at
each in its own bundle: an eindateiige Quelle is already in the form the rule produces, not an today's shard.** `raw accept --page "Source - X" ...` extends an existing source page's
exception to it - nothing was reorganised to reach this state. `raw_files:` in the same call; if that raises the page past one file, its already-promoted file is
folded into the new bundle alongside the one(s) just accepted, at `<its-existing-parent>/<stem>/` -
the file that started single does not stay single once a second one belongs beside it, but its
capture date is whatever it always was, and a bundle mixing an old and a new shard would have no
single correct address.
A bundle's type directory is the **source's** type, not any one file's - a diagram that belongs to **The names occupied anywhere under `raw/` - file stems and bundle directory names alike - are
a `documents/` source is promoted from `incoming/documents/`, not `incoming/assets/`; `assets/` is unique** (Gitea #64, widened globally by #67 once type directories stopped bounding the check).
for a source that is itself an asset. Within a bundle, `handbuch.pdf` and `handbuch.md` sit side by side as always; the rule bites one
level up, so a second, unrelated source cannot promote quietly into a bundle it does not belong to
**The names occupied at `raw/<type>/` level - file stems and bundle directory names alike - are just because its own filename happens not to collide - nor into a same-named bundle sitting in a
unique** (Gitea #64). Within a bundle, `handbuch.pdf` and `handbuch.md` sit side by side as always; different shard, or in a pre-#67 type directory. A promote whose target name is already occupied
the rule bites one level up, so a second, unrelated source cannot promote quietly into a bundle it is refused, naming both sanctioned ways past it without recommending either:
does not belong to just because its own filename happens not to collide. A promote whose target
name is already occupied is refused, naming both sanctioned ways past it without recommending
either:
``` ```
ERROR raw/documents/cluster.md already claims the stem "cluster" in raw/documents/. ERROR raw/documents/cluster.md already claims the stem "cluster" under raw/.
These are two different intents and only you can tell them apart: These are two different intents and only you can tell them apart:
Same source, new edition -> tools/wikitool raw accept --replaces raw/documents/cluster.md incoming/documents/cluster.md Same source, new edition -> tools/wikitool raw accept --replaces raw/documents/cluster.md incoming/cluster.md
A second, separate source -> rename it in incoming/ (cluster-netzplan.md, cluster-2026-09.md) and accept it normally A second, separate source -> rename it in incoming/ (cluster-netzplan.md, cluster-2026-09.md) and accept it normally
raw accept does not guess which one this is. raw accept does not guess which one this is.
``` ```
@@ -76,16 +101,60 @@ to the human and waits, the same way it would for an exit-42 gate (AGENTS.md inv
though no gate fires here: the tool cannot ask the question itself, so the session passes it on though no gate fires here: the tool cannot ask the question itself, so the session passes it on
instead of answering it. instead of answering it.
`tools/wikitool raw accept --page "Source - X" ...` additionally extends an existing source page's
`raw_files:` in the same call. If that raises the page past one file, its already-promoted file is
folded into the new bundle alongside the one(s) just accepted - the file that started single does
not stay single once a second one belongs beside it.
`incoming/` is read by an ingest session, never by `sources coverage` or `lint`: both walk `raw/` `incoming/` is read by an ingest session, never by `sources coverage` or `lint`: both walk `raw/`
only, so a file waiting there is not yet a finding. It is also never committed - proven, not only, so a file waiting there is not yet a finding. It is also never committed - proven, not
merely asserted, by `docs verify`'s ignore-rule canaries - which is what makes accepting a file the merely asserted, by `docs verify`'s ignore-rule canaries - which is what makes accepting a file the
moment its immutability under the rules below begins, not the moment it was dropped. moment its immutability under the rules below begins, not the moment it was dropped.
## Capture fields: `fidelity` and `authority`
Two things are knowable at the moment a file is accepted and at no point afterwards: **how
faithful the capture is** to what was actually said or shown, and **what the material is entitled
to claim** about its subject. A model's own analysis of a system can be guessed at months later;
whether the sender of an archived thread was actually in a position to speak for its subject
cannot. Both are recorded once, on the covering source page (`types/source.md`), as **capture
fields** - fixed at capture time, never freely re-editable afterwards.
| Field | Question | Values |
|---|---|---|
| `fidelity` | How faithful is the *capture*? | `verbatim`, `published`, `secondhand`, `nontextual`, (`unknown`) |
| `authority` | What may the material claim about its *subject*? | `normative`, `reporting`, `opinion`, (`unknown`) |
The two move independently: a chat transcript is `verbatim` + `reporting`; an LLM's own analysis
of the same subject is `secondhand` + `opinion`; official system documentation and a web article
about the same system are both `published`, but `normative` against `reporting`.
Both are **required, with no default**, at the point a source page first exists - the same
no-guessing posture `source_type:` has (Gitea #66), but without that field's escape hatch: there
is no `unclassified` catalog slot for a capture field, because a guessed value here would not read
as "unknown", it would read as a claim about the capture that cannot be corrected later (the
knowledge exists only at the drop point). `raw accept --fidelity <value> --authority <value>`
refuses without both; if the call also carries `--page`, both are written straight onto that page.
Without `--page` there is no page yet to write them onto - `wiki-ingest` creates the source page
afterwards - so `raw accept` instead prints the exact follow-up line, and `wikitool new source`
itself refuses to scaffold a source page without both:
```
OK Promoted 1 file(s) to raw/2026/09/handbuch.pdf.
Next:
tools/wikitool new source --name "<Title>" \
--set raw_files=raw/2026/09/handbuch.pdf \
--set fidelity=verbatim --set authority=reporting \
--set source_type=<category>
```
**Fixed once, correctable only as a new edition.** `wikitool touch --set fidelity=<value>` writes
a capture field only while it is absent; once set, it refuses and points at the one sanctioned way
to correct it - `raw accept --replaces`, which alone may pass `--fidelity`/`--authority` to
overwrite an already-set value, because a corrected capture *is* a new edition of the source, not
an edit of the page describing it.
**`unknown` is backfill-only.** Neither `raw accept` nor `new source` may ever write it - only
`wikitool touch`, on a page that predates this rule (the same construction `source_language`
already has: "absent on pages predating the rule"). A capture value written as `unknown` by the
tool that captures it would not be an honest "we don't know", it would be indistinguishable from a
value nobody ever thought about.
## Rules ## Rules
- **Immutable.** Never edit, reformat, summarize, or "clean up" a file after it lands here. - **Immutable.** Never edit, reformat, summarize, or "clean up" a file after it lands here.
+1 -1
View File
@@ -71,7 +71,7 @@ tools/wikitool <command> --help
| `instructions sync [--force]` | Publish every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and delete published skills whose source is gone. Both targets are gitignored, so a fresh clone runs this once - see `instructions/bootstrap.md`. Re-running is also how a drifted copy is repaired: the source always wins. `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it) | | `instructions sync [--force]` | Publish every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and delete published skills whose source is gone. Both targets are gitignored, so a fresh clone runs this once - see `instructions/bootstrap.md`. Re-running is also how a drifted copy is repaired: the source always wins. `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it) |
| `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout | | `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout |
| `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only | | `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only |
| `docs verify` | Check the docs that mirror the code: every CLI command documented here (and vice versa), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `raw/CONTRACT.md`'s routing table naming the same subdirectories as `dist_cmd.RAW_SUBDIRS`, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, and the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks | | `docs verify` | Check the docs that mirror the code: every CLI command documented here (and vice versa), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, and the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks |
| `eval sessions [--json]` | List the sessions that have a trace under `reports/telemetry/`, most recent first. Read-only and exempt from the Iteration Budget Gate | | `eval sessions [--json]` | List the sessions that have a trace under `reports/telemetry/`, most recent first. Read-only and exempt from the Iteration Budget Gate |
| `eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]` | Score one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see [../EVALS.md](../EVALS.md) | | `eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]` | Score one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see [../EVALS.md](../EVALS.md) |
| `dist export <target> [--dry-run] [--source-repo U] [--source-commit SHA] [--release-url U] [--update-url U]` | Write a contentless, distributable copy of this repo's machinery into an empty `<target>` directory: `AGENTS.md`/`README.md`/`EVALS.md` with any `<!-- dist:strip-start -->...<!-- dist:strip-end -->` region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs and their schemas re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas), empty `raw/{articles,documents,notes,assets}/` and the | `dist export <target> [--dry-run] [--source-repo U] [--source-commit SHA] [--release-url U] [--update-url U]` | Write a contentless, distributable copy of this repo's machinery into an empty `<target>` directory: `AGENTS.md`/`README.md`/`EVALS.md` with any `<!-- dist:strip-start -->...<!-- dist:strip-end -->` region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs and their schemas re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas), empty `raw/{articles,documents,notes,assets}/` and the
+8 -16
View File
@@ -133,14 +133,6 @@ def _is_coverage_output(filename: str) -> bool:
# reconstructs it afterwards, unlike the marker-block content below. # reconstructs it afterwards, unlike the marker-block content below.
INSTRUCTIONS_EXCLUDE_DIRS = {"dev"} INSTRUCTIONS_EXCLUDE_DIRS = {"dev"}
# Fixed by raw/CONTRACT.md's routing table, unlike kb/'s areas (which are
# organic - see kb/CONTRACT.md - so `export` does not manufacture them).
# `docs verify` (check_raw_subdirs) holds the table to this tuple in both
# directions, and `incoming/` (raw/CONTRACT.md "Getting a file in",
# raw_cmd.py) mirrors it as the set of type subdirectories a human may drop a
# file into - so this is the one place all three read the list from.
RAW_SUBDIRS = ("articles", "documents", "notes", "assets")
# Stage contracts that are not collections and carry no pages: copied as a # Stage contracts that are not collections and carry no pages: copied as a
# single file each, nothing else from their directory. `kb/` is excluded here # single file each, nothing else from their directory. `kb/` is excluded here
# - it is a content stage too, but it has collections underneath it, so its # - it is a content stage too, but it has collections underneath it, so its
@@ -416,14 +408,14 @@ def build_plan(origin: Optional[Origin] = None) -> dict[str, PlannedFile]:
if source.is_file(): if source.is_file():
plan[relative] = _read_planned_file(source, relative) plan[relative] = _read_planned_file(source, relative)
for sub in RAW_SUBDIRS: # `raw/` and `incoming/` are both flat now (Gitea #67 removes type
plan[f"raw/{sub}/.gitkeep"] = PlannedFile("") # subdirectories from the addressing scheme entirely - a file's location
# `incoming/` mirrors raw/'s type subdirectories (raw/CONTRACT.md # under `raw/` is a date shard computed by `raw accept`, never a hand-picked
# "Getting a file in") - seeded the same way, though `.gitignore` # type). `.gitignore` (also exported, see ROOT_FILES) excludes `incoming/`
# (also exported, see ROOT_FILES) excludes the whole directory again # again once the instance is a git repo, which is why bootstrap.md
# once the instance is a git repo, which is why bootstrap.md re-creates # re-creates it for a plain clone that never had this export step at all.
# it for a plain clone that never had this export step at all. plan["raw/.gitkeep"] = PlannedFile("")
plan[f"incoming/{sub}/.gitkeep"] = PlannedFile("") plan["incoming/.gitkeep"] = PlannedFile("")
plan["kb/log.md"] = PlannedFile((DIST_TEMPLATES_DIR / "log.md").read_text(encoding="utf-8")) plan["kb/log.md"] = PlannedFile((DIST_TEMPLATES_DIR / "log.md").read_text(encoding="utf-8"))
plan["CHANGES.md"] = PlannedFile((DIST_TEMPLATES_DIR / "CHANGES.md").read_text(encoding="utf-8")) plan["CHANGES.md"] = PlannedFile((DIST_TEMPLATES_DIR / "CHANGES.md").read_text(encoding="utf-8"))
+3 -37
View File
@@ -38,7 +38,6 @@ from typing import Optional
import typer import typer
from chemenu import config, conventions, kb_collections, version as version_mod from chemenu import config, conventions, kb_collections, version as version_mod
from chemenu.commands import dist_cmd
from chemenu.commands._util import fail, rel_path, success from chemenu.commands._util import fail, rel_path, success
app = typer.Typer(help="Verify documentation that mirrors the code or repo layout.") app = typer.Typer(help="Verify documentation that mirrors the code or repo layout.")
@@ -113,8 +112,9 @@ REQUIRED_IGNORE_CANARIES = (
# raw/ itself, a file here must never be committed - promotion via # raw/ itself, a file here must never be committed - promotion via
# `wikitool raw accept` is what makes it immutable, not the drop - so this # `wikitool raw accept` is what makes it immutable, not the drop - so this
# is the one canary in this tuple asserting the *opposite* of raw/'s own # is the one canary in this tuple asserting the *opposite* of raw/'s own
# backstop a few lines above. # backstop a few lines above. Flat since Gitea #67 - incoming/ no longer
"incoming/documents/probe.pdf", # has type subdirectories, so the probe sits directly in it.
"incoming/probe.pdf",
) )
REQUIRED_TRACKED_PATHS = ( REQUIRED_TRACKED_PATHS = (
"reports/CONTRACT.md", "reports/CONTRACT.md",
@@ -165,11 +165,6 @@ LEGACY_TYPE_RE = re.compile(r"^type:\s*(entity|concept|source|comparison)\s*$",
# First backticked cell of a markdown table row, e.g. "| `xref add --a ...` | ... |" # First backticked cell of a markdown table row, e.g. "| `xref add --a ...` | ... |"
TABLE_CELL_RE = re.compile(r"^\|\s*`([^`]+)`", re.MULTILINE) TABLE_CELL_RE = re.compile(r"^\|\s*`([^`]+)`", re.MULTILINE)
# A raw/CONTRACT.md routing-table cell naming a bare type subdirectory, e.g.
# "| `articles/` | ... |" - deliberately narrower than TABLE_CELL_RE, which
# would also match a command example elsewhere on the page.
RAW_DIR_CELL_RE = re.compile(r"^\|\s*`([a-zA-Z0-9_-]+)/`\s*\|", re.MULTILINE)
def registered_commands() -> set[str]: def registered_commands() -> set[str]:
"""Every command path the CLI exposes, e.g. {'new', 'xref add', ...}. """Every command path the CLI exposes, e.g. {'new', 'xref add', ...}.
@@ -329,34 +324,6 @@ def check_stack_required_types() -> list[str]:
return issues return issues
def documented_raw_subdirs(text: str) -> list[str]:
return [match.group(1) for match in RAW_DIR_CELL_RE.finditer(text)]
def check_raw_subdirs() -> list[str]:
"""`raw/CONTRACT.md`'s routing table and `dist_cmd.RAW_SUBDIRS` must name
the same set of type subdirectories (Gitea #58) - the table is meant to
read as behaviour derived from the tuple, not as a second place the list
could drift (AGENTS.md invariant 8). Skipped if the contract itself is
missing; `check_collection_contracts` already reports that.
"""
contract_path = config.ROOT / "raw" / "CONTRACT.md"
if not contract_path.exists():
return []
documented = set(documented_raw_subdirs(contract_path.read_text(encoding="utf-8")))
declared = set(dist_cmd.RAW_SUBDIRS)
issues = [
f"raw/CONTRACT.md's routing table is missing `{missing}/` - dist_cmd.RAW_SUBDIRS names it"
for missing in sorted(declared - documented)
]
issues += [
f"raw/CONTRACT.md's routing table lists `{extra}/`, but dist_cmd.RAW_SUBDIRS does not - "
"the two must name the same set"
for extra in sorted(documented - declared)
]
return issues
def check_legacy_type_blocks() -> list[str]: def check_legacy_type_blocks() -> list[str]:
issues = [] issues = []
guarded = [ guarded = [
@@ -625,7 +592,6 @@ def verify():
check_cli_readme() check_cli_readme()
+ check_readmes_have_no_command_table() + check_readmes_have_no_command_table()
+ check_collection_contracts() + check_collection_contracts()
+ check_raw_subdirs()
+ check_legacy_type_blocks() + check_legacy_type_blocks()
+ check_ignored_content() + check_ignored_content()
+ check_version_changelog() + check_version_changelog()
+27
View File
@@ -298,6 +298,33 @@ def new_page_command(
frontmatter = _build_frontmatter(type_path, schema, today, explicit) frontmatter = _build_frontmatter(type_path, schema, today, explicit)
# Capture fields (Gitea #67, e.g. `fidelity`/`authority` on a source page)
# are deliberately absent from `required:` - putting them there would make
# every existing instance's source pages stop validating, a
# boundary-crossing change (version-parts.md). The requirement is instead
# enforced here, in the tool, exactly like `source_type`'s no-default
# refusal (#66) reads from the schema alone: without a `default:` and
# omitted from `explicit`, a capture field simply never lands in
# `frontmatter`, so its absence has to be caught before it is silently
# written as a page with no capture record at all.
try:
capture_fields = resolver.get_capture_fields(type_path)
except ValueError as exc:
fail(str(exc))
missing_capture = [f for f in capture_fields if not frontmatter.get(f)]
if missing_capture:
fail(
f"Type {type_path} requires capture field(s) {', '.join(missing_capture)} - pass them "
"explicitly, e.g. --set fidelity=verbatim --set authority=reporting. "
"new does not guess them (see raw/CONTRACT.md)."
)
guessed_unknown = [f for f in capture_fields if frontmatter.get(f) == "unknown"]
if guessed_unknown:
fail(
f"Capture field(s) {', '.join(guessed_unknown)} cannot be set to 'unknown' here - that "
"value is backfill-only, written only by `wikitool touch` on a page predating this rule."
)
target_dir = _target_dir(type_path, frontmatter) target_dir = _target_dir(type_path, frontmatter)
_type_spec, template = _load_type_or_fail(type_path, target_dir) _type_spec, template = _load_type_or_fail(type_path, target_dir)
_validate_or_fail(frontmatter, type_path, target_dir) _validate_or_fail(frontmatter, type_path, target_dir)
+248 -82
View File
@@ -1,34 +1,61 @@
"""`wikitool raw accept` - promote one or more files from `incoming/` into """`wikitool raw accept` - promote one or more files from `incoming/` into
`raw/`, with the destination computed rather than chosen by hand (Gitea #58). `raw/`, with the destination computed rather than chosen by hand (Gitea #58),
sharded by the accept date rather than by a hand-picked type (Gitea #67).
A human classifies a file only by which type subdirectory of `incoming/` they A human no longer classifies a file at all: `incoming/` is flat, and a
drop it into - `incoming/articles/`, `incoming/documents/`, `incoming/notes/`, subdirectory dropped under it (an old `incoming/documents/` habit, a script
`incoming/assets/`, mirroring raw/CONTRACT.md's routing table. Everything past that still writes one) is accepted and ignored rather than inspected -
that is this command's job: promoting `raw/` from a routing decision to an address computed purely from
*when* the file was accepted:
- **Single file, no bundle.** One file promoted alone lands as - **Single file, no bundle.** One file promoted alone lands as
`raw/<type>/<name>` - no directory of its own. `raw/<YYYY>/<MM>/<name>` - no directory of its own.
- **Bundle from the second file on.** Several files of one source promoted in - **Bundle from the second file on.** Several files of one source promoted in
the same call land under `raw/<type>/<stem>/`, named after the first file's the same call land under `raw/<YYYY>/<MM>/<stem>/`, named after the first
stem. file's stem.
- **Growing an existing single file into a bundle.** `--page` extends an - **Growing an existing single file into a bundle.** `--page` extends an
existing source page's `raw_files:`. If that raises the page from one file existing source page's `raw_files:`. If that raises the page from one file
to more than one, the file it already had is folded into the new bundle to more than one, the file it already had is folded into a bundle at its
alongside the ones just promoted, in the same call - at no point does own parent directory - `raw/<its-existing-location>/<stem>/` - not at
`raw_files:` point at a path that does not exist. today's shard, so a bundle never mixes an old capture date with today's
(Gitea #67 decision, "Datums-Shard" § "Bündelort").
Existing files under `raw/` are never moved by this change (Gitea #67
"Altbestand bleibt stehen"): `raw/articles/`, `raw/documents/`, `raw/notes/`
and `raw/assets/` keep whatever they already held, and stay valid promotion
targets for `--replaces`.
**Every promotion now also carries `--fidelity` and `--authority`** (Gitea
#67): how faithfully the material was captured, and what it is entitled to
claim about its subject. Neither has a default and neither may be `unknown`
here - that value is backfill-only, written only by `wikitool touch` on a
page predating this rule. Given `--page`, both are written straight onto the
target page (once - fill-once, like `touch`, see `touch._capture_field_or_fail`);
without `--page` there is no page yet to write them onto (`wiki-ingest`
creates it afterwards), so this command instead prints the exact
`wikitool new source --set fidelity=... --set authority=...` follow-up line,
and `new source` itself refuses to scaffold a source page without both.
`--replaces` is the one path where both become *optional*: passing them there
is the sanctioned way to correct an already-set capture value on a later
edition, the same way `touch`'s own refusal points back to `--replaces`.
Multi-owner raw files (`provenance.duplicate_raw_file_owners`) are refused Multi-owner raw files (`provenance.duplicate_raw_file_owners`) are refused
rather than silently moved: relocating a file another page also claims would rather than silently moved: relocating a file another page also claims would
break that page's `raw_files:` without it ever being consulted. break that page's `raw_files:` without it ever being consulted.
**Stem uniqueness at `raw/<type>/` level** (Gitea #64) closes the gap this **Stem uniqueness across the whole of `raw/`** (Gitea #64, widened by #67):
leaves open: without it, a second, unrelated source whose primary file happens without it, a second, unrelated source whose primary file happens not to
not to collide on the exact filename slips silently into an existing bundle, collide on the exact filename slips silently into an existing bundle, because
because the per-file `dst.exists()` check above never looks at the bundle the per-file `dst.exists()` check above never looks at the bundle directory
directory itself. The set of names occupied at `raw/<type>/` level - file itself. The set of names occupied anywhere under `raw/` - file stems and
stems and bundle directory names alike - must stay unique; `_occupied_stems()` bundle directory names alike, at whichever level directly holds files - must
and the check built on it enforce that, while still allowing a call to grow a stay unique; `_occupied_stems()` and the check built on it enforce that,
bundle it already owns (via `--page`, or by continuing an existing bundle). while still allowing a call to grow a bundle it already owns (via `--page`,
or by continuing an existing bundle). It used to be scoped to one type
directory; #67 removes type directories from the addressing scheme entirely,
so uniqueness now has to span old-style flat directories and the new date
shard together, or `--replaces` on a bygone stem would be ambiguous across
shards.
**`--replaces <raw-path>`** is the only sanctioned way past that rule: whether **`--replaces <raw-path>`** is the only sanctioned way past that rule: whether
a new file is a later edition of an existing source or a second, separate one a new file is a later edition of an existing source or a second, separate one
@@ -37,6 +64,8 @@ and names both routes rather than choosing one (Gitea #64 decision 2).
""" """
from __future__ import annotations from __future__ import annotations
import datetime
import re
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
@@ -44,13 +73,20 @@ import typer
from chemenu import config from chemenu import config
from chemenu.commands._util import fail, rel_path, success from chemenu.commands._util import fail, rel_path, success
from chemenu.commands.dist_cmd import RAW_SUBDIRS
from chemenu.frontmatter_io import write_page from chemenu.frontmatter_io import write_page
from chemenu.kb_scan import load_kb_pages from chemenu.kb_scan import load_kb_pages
from chemenu.provenance import citing_pages, source_pages_by_raw_file, source_raw_files from chemenu.provenance import citing_pages, source_pages_by_raw_file, source_raw_files
from chemenu.type_resolver import resolver
app = typer.Typer(help="Promote raw material out of incoming/ into raw/.") app = typer.Typer(help="Promote raw material out of incoming/ into raw/.")
# The type this command always promotes into eventually - hardcoded rather
# than derived from --page, because there is no page yet in the common case
# (see module docstring): the follow-up hint and the capture-field/enum
# validation below both need *a* type to read from, and `source` is the only
# one `raw accept` has ever scaffolded towards.
_SOURCE_TYPE_PATH = "types/source.md"
def _incoming_dir() -> Path: def _incoming_dir() -> Path:
return config.ROOT / "incoming" return config.ROOT / "incoming"
@@ -60,9 +96,24 @@ def _resolve(raw: Path) -> Path:
return raw if raw.is_absolute() else config.ROOT / raw return raw if raw.is_absolute() else config.ROOT / raw
def _classify(path: Path, incoming: Path) -> str: def _shard_dir(today: Optional[datetime.date] = None) -> Path:
"""The type subdirectory `path` (already resolved, absolute) declares by """`raw/<YYYY>/<MM>/`, computed from the accept date - never from a count,
where it sits under `incoming/`, or fail with the reason it doesn't.""" so it can never rebalance and break a `[^cite-id]` anchor (Gitea #67).
"""
d = today or datetime.date.today()
return config.RAW_DIR / f"{d.year:04d}" / f"{d.month:02d}"
def _validate_under_incoming(path: Path, incoming: Path) -> None:
"""`path` must sit directly in `incoming/`, or exactly one level below it.
Unlike before #67, that one optional level carries no meaning any more -
it is accepted and ignored, kept only so an old `incoming/<type>/` habit
or script does not have to change to keep working (the MINOR condition
named in Gitea #67 "Versionsteil"). Nesting deeper than that is still
refused: it was never meaningful and silently accepting it would hide a
typo'd path.
"""
try: try:
rel = path.relative_to(incoming) rel = path.relative_to(incoming)
except ValueError: except ValueError:
@@ -70,37 +121,58 @@ def _classify(path: Path, incoming: Path) -> str:
f"{rel_path(path)} is not under incoming/ - `raw accept` only promotes files " f"{rel_path(path)} is not under incoming/ - `raw accept` only promotes files "
"from there. See raw/CONTRACT.md." "from there. See raw/CONTRACT.md."
) )
allowed = ", ".join(f"incoming/{s}/" for s in RAW_SUBDIRS) if len(rel.parts) < 1:
if len(rel.parts) < 2: fail(f"incoming/{rel} names no file.")
fail(
f"incoming/{rel} declares no type - place it inside one of {allowed} instead "
"of directly in incoming/."
)
sub = rel.parts[0]
if sub not in RAW_SUBDIRS:
fail(f"incoming/{rel} lies under an unknown type directory 'incoming/{sub}/'. Allowed: {allowed}.")
if len(rel.parts) > 2: if len(rel.parts) > 2:
fail(f"incoming/{rel} is nested below its type directory - place it directly in incoming/{sub}/.") fail(
return sub f"incoming/{rel} is nested more than one level below incoming/ - place it "
"directly in incoming/, or in at most one subdirectory of it (the "
"subdirectory itself is ignored, see raw/CONTRACT.md)."
)
def _occupied_stems(raw_sub_dir: Path) -> dict[str, Path]: _YEAR_DIR = re.compile(r"\d{4}")
"""Every name occupied at `raw/<type>/` level: file stems and bundle
directory names alike, one level below `raw_sub_dir` only."""
def _occupied_stems(raw_dir: Path) -> dict[str, Path]:
"""Every name occupied anywhere under `raw/`: file stems and bundle
directory names alike, at whichever level directly holds files.
Distinguishes the pre-#67 flat layout (`raw/<name>/<file-or-bundle>`)
from the #67 date shard (`raw/<YYYY>/<MM>/<file-or-bundle>`) structurally,
by whether a top-level directory name is a 4-digit year - not from a
hardcoded list of legacy type names, so this keeps working unchanged if a
legacy directory is ever renamed by hand.
"""
occupied: dict[str, Path] = {} occupied: dict[str, Path] = {}
if raw_sub_dir.is_dir(): if not raw_dir.is_dir():
for entry in raw_sub_dir.iterdir(): return occupied
occupied[entry.stem if entry.is_file() else entry.name] = entry for top in raw_dir.iterdir():
if not top.is_dir():
continue # raw/CONTRACT.md
if _YEAR_DIR.fullmatch(top.name):
for month in top.iterdir():
if month.is_dir():
occupied.update(_leaf_group_entries(month))
else:
occupied.update(_leaf_group_entries(top))
return occupied return occupied
def _stem_collision_message(sub: str, claimed_name: str, holder: Path) -> str: def _leaf_group_entries(group_dir: Path) -> dict[str, Path]:
"""The addressable entries directly inside one leaf-group directory
(a legacy type dir, or one `raw/<YYYY>/<MM>/`): a lone file occupies its
stem, a bundle directory occupies its own name."""
return {entry.stem if entry.is_file() else entry.name: entry for entry in group_dir.iterdir()}
def _stem_collision_message(claimed_name: str, holder: Path) -> str:
example_target = holder example_target = holder
if holder.is_dir(): if holder.is_dir():
children = sorted(holder.iterdir()) children = sorted(holder.iterdir())
example_target = children[0] if children else holder example_target = children[0] if children else holder
return ( return (
f'{rel_path(holder)} already claims the stem "{claimed_name}" in raw/{sub}/.\n' f'{rel_path(holder)} already claims the stem "{claimed_name}" under raw/.\n'
" These are two different intents and only you can tell them apart:\n" " These are two different intents and only you can tell them apart:\n"
f" Same source, new edition -> tools/wikitool raw accept " f" Same source, new edition -> tools/wikitool raw accept "
f"--replaces {rel_path(example_target)} <incoming file>\n" f"--replaces {rel_path(example_target)} <incoming file>\n"
@@ -110,10 +182,53 @@ def _stem_collision_message(sub: str, claimed_name: str, holder: Path) -> str:
) )
def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bool) -> None: def _capture_choices() -> tuple[list[str], list[str]]:
"""Allowed `--fidelity`/`--authority` values, straight from the schema
(single source of truth) - `unknown` excluded, since it is backfill-only
and neither flag may write it (Gitea #67)."""
fidelity = [v for v in resolver.get_enum(_SOURCE_TYPE_PATH, "fidelity") if v != "unknown"]
authority = [v for v in resolver.get_enum(_SOURCE_TYPE_PATH, "authority") if v != "unknown"]
return fidelity, authority
def _check_capture_value(field: str, value: Optional[str], allowed: list[str]) -> None:
if value is None:
return
if value == "unknown" or value not in allowed:
fail(
f"--{field} must be one of: {', '.join(allowed)}. 'unknown' is backfill-only - only "
"`wikitool touch` may write it, on a page predating this rule."
)
def _overwrite_capture_fields_on_page(page, fidelity: Optional[str], authority: Optional[str]) -> bool:
"""Write `fidelity`/`authority` onto `page.frontmatter`, unconditionally -
only called from `_replace`, the one sanctioned way to correct an
already-set capture value (Gitea #67). Returns whether anything changed,
so the caller only writes the page back when it needs to."""
changed = False
for field, value in (("fidelity", fidelity), ("authority", authority)):
if value is not None and page.frontmatter.get(field) != value:
page.frontmatter[field] = value
changed = True
return changed
def _replace(
files: list[Path],
replaces: Path,
page: Optional[str],
fidelity: Optional[str],
authority: Optional[str],
dry_run: bool,
) -> None:
"""`raw accept --replaces <target> <incoming file>` - overwrite an """`raw accept --replaces <target> <incoming file>` - overwrite an
existing raw/ file wholesale with a later edition, in place, with existing raw/ file wholesale with a later edition, in place, with
`raw_files:` on every source page left untouched (Gitea #64 decision 2). `raw_files:` on every source page left untouched (Gitea #64 decision 2).
`--fidelity`/`--authority` are optional here, and are the one sanctioned
way to correct an already-set capture value (Gitea #67) - passed, they
overwrite the owning page's value; omitted, the page's capture fields are
left exactly as they were.
Every check below runs before any write, so a failure leaves both the Every check below runs before any write, so a failure leaves both the
target and the incoming file exactly as they were - the invariant the target and the incoming file exactly as they were - the invariant the
@@ -124,24 +239,22 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
if len(files) != 1: if len(files) != 1:
fail("--replaces takes exactly one incoming file - a replacement is one file for one file.") fail("--replaces takes exactly one incoming file - a replacement is one file for one file.")
fidelity_choices, authority_choices = _capture_choices()
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
incoming_path = _resolve(files[0]) incoming_path = _resolve(files[0])
if not incoming_path.is_file(): if not incoming_path.is_file():
fail(f"{rel_path(incoming_path)} does not exist or is not a file.") fail(f"{rel_path(incoming_path)} does not exist or is not a file.")
incoming_sub = _classify(incoming_path, _incoming_dir()) _validate_under_incoming(incoming_path, _incoming_dir())
target = _resolve(replaces) target = _resolve(replaces)
try: try:
target_rel = target.relative_to(config.RAW_DIR) target.relative_to(config.RAW_DIR)
except ValueError: except ValueError:
fail(f"--replaces target {rel_path(target)} does not lie under raw/.") fail(f"--replaces target {rel_path(target)} does not lie under raw/.")
if not target.is_file(): if not target.is_file():
fail(f"--replaces target {rel_path(target)} does not exist or is not a file.") fail(f"--replaces target {rel_path(target)} does not exist or is not a file.")
target_sub = target_rel.parts[0]
if target_sub != incoming_sub:
fail(
f"--replaces target is under raw/{target_sub}/, but {rel_path(incoming_path)} is "
f"under incoming/{incoming_sub}/ - a replacement stays within one type directory."
)
if target.name != incoming_path.name: if target.name != incoming_path.name:
fail( fail(
f"--replaces target {rel_path(target)} has a different filename than " f"--replaces target {rel_path(target)} has a different filename than "
@@ -157,13 +270,25 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
f"({', '.join(owners)}). Resolve the multiple ownership first (see `wikitool sources coverage`)." f"({', '.join(owners)}). Resolve the multiple ownership first (see `wikitool sources coverage`)."
) )
owner_page = pages.get(owners[0]) if owners else None
will_update_capture = owner_page is not None and _overwrite_capture_fields_on_page(
owner_page, fidelity, authority
)
if dry_run: if dry_run:
typer.echo(f"[dry-run] would replace {rel_path(target)} with {rel_path(incoming_path)}. No files written.") typer.echo(f"[dry-run] would replace {rel_path(target)} with {rel_path(incoming_path)}. No files written.")
if will_update_capture:
typer.echo(
f"[dry-run] would set fidelity={fidelity!r}, authority={authority!r} on '{owners[0]}'"
)
return return
target.unlink() target.unlink()
incoming_path.rename(target) incoming_path.rename(target)
if will_update_capture:
write_page(owner_page.path, owner_page.frontmatter, owner_page.body)
if not owners: if not owners:
success( success(
f"Replaced {rel_path(target)} (previous version stays in git history). " f"Replaced {rel_path(target)} (previous version stays in git history). "
@@ -188,7 +313,20 @@ def _replace(files: list[Path], replaces: Path, page: Optional[str], dry_run: bo
def raw_accept_command( def raw_accept_command(
files: list[Path] = typer.Argument( files: list[Path] = typer.Argument(
..., ...,
help="One or more files under incoming/<type>/, all belonging to the same source", help="One or more files under incoming/, all belonging to the same source",
),
fidelity: Optional[str] = typer.Option(
None,
"--fidelity",
help="How faithful the capture is (see `wikitool types describe source`). Required unless "
"--replaces is given, where it is the optional, sanctioned way to correct an already-set value",
),
authority: Optional[str] = typer.Option(
None,
"--authority",
help="What the material may claim about its subject (see `wikitool types describe source`). "
"Required unless --replaces is given, where it is the optional, sanctioned way to correct "
"an already-set value",
), ),
page: Optional[str] = typer.Option( page: Optional[str] = typer.Option(
None, None,
@@ -205,7 +343,7 @@ def raw_accept_command(
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"), dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
): ):
"""Promote file(s) from incoming/ into raw/, computing the destination """Promote file(s) from incoming/ into raw/, computing the destination
(type directory, bundle or not, bundle name) instead of taking it as an (date shard, bundle or not, bundle name) instead of taking it as an
argument. See raw/CONTRACT.md "Getting a file in: incoming/".""" argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
if not files: if not files:
fail("Pass at least one file to promote.") fail("Pass at least one file to promote.")
@@ -213,27 +351,30 @@ def raw_accept_command(
incoming = _incoming_dir() incoming = _incoming_dir()
if replaces is not None: if replaces is not None:
_replace(files, replaces, page, dry_run) _replace(files, replaces, page, fidelity, authority, dry_run)
return return
fidelity_choices, authority_choices = _capture_choices()
if fidelity is None or authority is None:
missing = ", ".join(n for n, v in (("--fidelity", fidelity), ("--authority", authority)) if v is None)
fail(
f"{missing} required - raw accept does not guess how faithful a capture is or what it "
f"may claim. --fidelity: {', '.join(fidelity_choices)}. --authority: {', '.join(authority_choices)}."
)
_check_capture_value("fidelity", fidelity, fidelity_choices)
_check_capture_value("authority", authority, authority_choices)
resolved = [_resolve(f) for f in files] resolved = [_resolve(f) for f in files]
for path in resolved: for path in resolved:
if not path.is_file(): if not path.is_file():
fail(f"{rel_path(path)} does not exist or is not a file.") fail(f"{rel_path(path)} does not exist or is not a file.")
_validate_under_incoming(path, incoming)
subs = {_classify(path, incoming) for path in resolved}
if len(subs) > 1:
allowed = ", ".join(sorted(f"incoming/{s}/" for s in subs))
fail(f"All files in one `raw accept` call must share one type directory; got {allowed}.")
sub = subs.pop()
names = [path.name for path in resolved] names = [path.name for path in resolved]
if len(names) != len(set(names)): if len(names) != len(set(names)):
fail("Two files share a filename; rename one before promoting.") fail("Two files share a filename; rename one before promoting.")
raw_sub_dir = config.RAW_DIR / sub
pages = None pages = None
target_page = None target_page = None
existing_raw_paths: list[Path] = [] existing_raw_paths: list[Path] = []
@@ -256,12 +397,6 @@ def raw_accept_command(
f"{', '.join(rel_path(p) for p in missing)}. Fix raw_files: (see `sources coverage`) " f"{', '.join(rel_path(p) for p in missing)}. Fix raw_files: (see `sources coverage`) "
"before promoting more." "before promoting more."
) )
existing_subs = {p.relative_to(config.RAW_DIR).parts[0] for p in existing_raw_paths}
if existing_subs != {sub}:
fail(
f"'{page}' already claims file(s) under {', '.join(sorted(f'raw/{s}/' for s in existing_subs))}, "
f"not raw/{sub}/. A bundle is one type directory; promote separately."
)
# A bundle directory forms once two or more files belong to the source # A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever # (Gitea #58 decision 3): from the second file on, never before. Whenever
@@ -278,36 +413,46 @@ def raw_accept_command(
) )
bundle_dir = parents.pop() bundle_dir = parents.pop()
elif total >= 2: elif total >= 2:
primary = existing_raw_paths[0] if existing_raw_paths else resolved[0] if existing_raw_paths:
bundle_dir = raw_sub_dir / primary.stem # Growing a bundle out of a single already-promoted file (Gitea
# #67 decision): the bundle forms at that file's own parent
# directory, never at today's shard - the file's capture date is
# whatever it always was, and a bundle mixing an old and a new
# shard would have no single correct address.
primary = existing_raw_paths[0]
bundle_dir = primary.parent / primary.stem
else:
primary = resolved[0]
bundle_dir = _shard_dir() / primary.stem
moves: list[tuple[Path, Path]] = [] moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths: for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir: if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name)) moves.append((existing, bundle_dir / existing.name))
for new_path in resolved: for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (raw_sub_dir / new_path.name) dst = (bundle_dir / new_path.name) if bundle_dir is not None else (_shard_dir() / new_path.name)
moves.append((new_path, dst)) moves.append((new_path, dst))
for _src, dst in moves: for _src, dst in moves:
if dst.exists(): if dst.exists():
fail(f"Cannot promote: {rel_path(dst)} already exists.") fail(f"Cannot promote: {rel_path(dst)} already exists.")
# Stem uniqueness at raw/<type>/ level (Gitea #64): the name this call is # Stem uniqueness across raw/ (Gitea #64, widened by #67): the name this
# about to claim there - the bundle's name, or the lone file's stem when no # call is about to claim - the bundle's name, or the lone file's stem when
# bundle forms - must not already belong to something this call does not # no bundle forms - must not already belong to something this call does
# itself own. "Owns" means: one of the page's already-registered raw files # not itself own. "Owns" means: one of the page's already-registered raw
# (the pitfall from the module docstring - a single file growing into a # files (the pitfall from the module docstring - a single file growing
# bundle of its own name momentarily still occupies that name), or, once a # into a bundle of its own name momentarily still occupies that name), or,
# bundle already has >=2 registered files, the bundle directory itself. # once a bundle already has >=2 registered files, the bundle directory
# itself.
claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem claimed_name = bundle_dir.name if bundle_dir is not None else resolved[0].stem
occupied = _occupied_stems(raw_sub_dir) occupied = _occupied_stems(config.RAW_DIR)
owned = set(existing_raw_paths) owned = set(existing_raw_paths)
if len(existing_raw_paths) >= 2: if len(existing_raw_paths) >= 2:
owned.add(bundle_dir) owned.add(bundle_dir)
holder = occupied.get(claimed_name) holder = occupied.get(claimed_name)
if holder is not None and holder not in owned: if holder is not None and holder not in owned:
fail(_stem_collision_message(sub, claimed_name, holder)) fail(_stem_collision_message(claimed_name, holder))
moving_existing = [src for src, _dst in moves if src in existing_raw_paths] moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
if moving_existing: if moving_existing:
@@ -325,6 +470,19 @@ def raw_accept_command(
"Resolve the multiple ownership first (see `wikitool sources coverage`)." "Resolve the multiple ownership first (see `wikitool sources coverage`)."
) )
if target_page is not None:
current_fidelity = target_page.frontmatter.get("fidelity")
current_authority = target_page.frontmatter.get("authority")
for field, value, current in (
("fidelity", fidelity, current_fidelity),
("authority", authority, current_authority),
):
if current not in (None, "") and current != value:
fail(
f"'{page}' already has {field}={current!r} - a capture field is fixed once. "
"Use `raw accept --replaces` to correct it."
)
if dry_run: if dry_run:
for src, dst in moves: for src, dst in moves:
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}") typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
@@ -332,6 +490,7 @@ def raw_accept_command(
moved_map = dict(moves) moved_map = dict(moves)
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved] final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
typer.echo(f"[dry-run] would set raw_files: on '{page}' to {[rel_path(p) for p in final]}") typer.echo(f"[dry-run] would set raw_files: on '{page}' to {[rel_path(p) for p in final]}")
typer.echo(f"[dry-run] would set fidelity={fidelity!r}, authority={authority!r} on '{page}'")
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.") typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
return return
@@ -344,6 +503,8 @@ def raw_accept_command(
if target_page is not None: if target_page is not None:
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved] final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
target_page.frontmatter["raw_files"] = [rel_path(p) for p in final] target_page.frontmatter["raw_files"] = [rel_path(p) for p in final]
target_page.frontmatter["fidelity"] = fidelity
target_page.frontmatter["authority"] = authority
write_page(target_page.path, target_page.frontmatter, target_page.body) write_page(target_page.path, target_page.frontmatter, target_page.body)
success( success(
f"Promoted {len(resolved)} file(s); updated raw_files: on '{page}' " f"Promoted {len(resolved)} file(s); updated raw_files: on '{page}' "
@@ -352,7 +513,12 @@ def raw_accept_command(
return return
promoted = ", ".join(rel_path(moved_map[p]) for p in resolved) promoted = ", ".join(rel_path(moved_map[p]) for p in resolved)
raw_files_arg = ",".join(rel_path(moved_map[p]) for p in resolved)
success( success(
f"Promoted {len(resolved)} file(s) to {promoted}. " f"Promoted {len(resolved)} file(s) to {promoted}.\n"
"Run `wikitool new source --set raw_files=...` (or `touch --set` on an existing page) next." " Next:\n"
" tools/wikitool new source --name \"<Title>\" \\\n"
f" --set raw_files={raw_files_arg} \\\n"
f" --set fidelity={fidelity} --set authority={authority} \\\n"
" --set source_type=<category>"
) )
+20
View File
@@ -128,6 +128,23 @@ def _settable_or_fail(field: str, schema: Optional[Dict[str, Any]], type_path: s
return properties[field] return properties[field]
def _capture_field_or_fail(field: str, value: Any, frontmatter: Dict[str, Any]) -> None:
"""Refuse to overwrite a capture field (Gitea #67) that already carries a
value - fill-once, not a denylist entry: `UNSETTABLE` would also forbid
the *first* write, which is exactly the write the backfill needs. A
capture field is fixed at `raw accept`/`new source` time; the only
sanctioned way to change an already-set value is a new edition of the
raw material (`raw accept --replaces`), never a second `touch`.
"""
current = frontmatter.get(field)
if current not in (None, "") and current != value:
fail(
f"`{field}` is a capture field: fixed once, at `raw accept`/`new source` time, and "
f"already reads {current!r}. A corrected capture is a new edition of the source, not "
f"a touch -> `wikitool raw accept --replaces <raw path> <incoming file>`."
)
def _apply_set(frontmatter: Dict[str, Any], field: str, value: Any) -> Optional[str]: def _apply_set(frontmatter: Dict[str, Any], field: str, value: Any) -> Optional[str]:
if frontmatter.get(field) == value: if frontmatter.get(field) == value:
return None return None
@@ -233,6 +250,7 @@ def touch_command(
try: try:
schema = resolver.get_schema(type_path, page.path) schema = resolver.get_schema(type_path, page.path)
capture_fields = set(resolver.get_capture_fields(type_path, page.path))
except ValueError as exc: except ValueError as exc:
fail(str(exc)) fail(str(exc))
@@ -290,6 +308,8 @@ def touch_command(
parsed = parse_set_fields(values, schema, flag=flag) parsed = parse_set_fields(values, schema, flag=flag)
for field, value in parsed.items(): for field, value in parsed.items():
_settable_or_fail(field, schema, type_path) _settable_or_fail(field, schema, type_path)
if field in capture_fields:
_capture_field_or_fail(field, value, frontmatter)
change = apply(frontmatter, field, value) change = apply(frontmatter, field, value)
touched.add(field) touched.add(field)
if change: if change:
+69
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
from datetime import date from datetime import date
from pathlib import Path from pathlib import Path
from typing import Optional
from collections import Counter from collections import Counter
@@ -143,6 +144,59 @@ def unclassified_source_pages(pages: dict[str, Page]) -> list[dict]:
] ]
# Capture-field ceilings (Gitea #67, kb/CONTRACT.md § "Confidence against
# source standing"): the weakest `authority`/`fidelity` among a page's cited
# sources bounds how high its `confidence_base` may honestly sit. Stack
# vocabulary, not instance configuration - `fidelity`/`authority` are defined
# by `raw/CONTRACT.md`, not by `kb/CONVENTIONS.md`. `unknown` and every axis
# value not listed here carry no ceiling: a backfilled "we don't know" is not
# a claim about the source, and `normative`/`verbatim`/`published` are simply
# not weaker than hand-set confidence gets to be.
_AUTHORITY_CEILING = {"reporting": 0.8, "opinion": 0.6}
_FIDELITY_CEILING = {"secondhand": 0.7, "nontextual": 0.7}
def _capture_ceiling(source_pages: list[Page]) -> Optional[float]:
"""The tightest ceiling implied by `source_pages`' capture fields, or
None if none of them carry a value with a ceiling at all."""
ceilings = [
ceiling
for src in source_pages
for field_map, value in (
(_AUTHORITY_CEILING, src.frontmatter.get("authority")),
(_FIDELITY_CEILING, src.frontmatter.get("fidelity")),
)
if (ceiling := field_map.get(value)) is not None
]
return min(ceilings) if ceilings else None
def confidence_exceeds_source_standing(pages: dict[str, Page]) -> list[dict]:
"""Pages whose `confidence_base` sits above what their cited sources'
capture standing can honestly carry.
Advisory, not a formula (kb/CONTRACT.md § "Confidence against source
standing" has the reasoning): `confidence_base` stays a human judgment,
and authority is a ceiling a page may sit under by independent
verification, not a value a formula could compute outright.
"""
findings: list[dict] = []
for title, page in sorted(pages.items()):
if page.kind not in ("entity", "concept"):
continue
confidence_base = page.frontmatter.get("confidence_base")
source_titles = page.frontmatter.get("sources") or []
if confidence_base is None or not source_titles:
continue
source_pages = [
pages[t] for t in source_titles if t in pages and pages[t].kind == "source"
]
ceiling = _capture_ceiling(source_pages)
if ceiling is not None and confidence_base > ceiling:
findings.append({"page": title, "confidence_base": confidence_base, "ceiling": ceiling})
return findings
def nested_pages(kb_dir: Path, pages: dict[str, Page]) -> list[dict]: def nested_pages(kb_dir: Path, pages: dict[str, Page]) -> list[dict]:
"""Report form of `find_nested_pages`: `{"page", "at", "depth"}` per """Report form of `find_nested_pages`: `{"page", "at", "depth"}` per
finding. finding.
@@ -491,6 +545,7 @@ def run_lint(kb_dir: Path) -> dict:
"nested_pages": nested, "nested_pages": nested,
"unsharded_collections": unsharded_collections(kb_dir, pages), "unsharded_collections": unsharded_collections(kb_dir, pages),
"unclassified_source_pages": unclassified_source_pages(pages), "unclassified_source_pages": unclassified_source_pages(pages),
"confidence_exceeds_source_standing": confidence_exceeds_source_standing(pages),
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages), "uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
"broken_raw_refs": find_broken_raw_refs(pages), "broken_raw_refs": find_broken_raw_refs(pages),
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages), "duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
@@ -590,6 +645,13 @@ def render_markdown(report: dict) -> str:
lambda i: f"[[{i['page']}]] - `wikitool touch --set source_type=<value>` once its " lambda i: f"[[{i['page']}]] - `wikitool touch --set source_type=<value>` once its "
"category is known", "category is known",
) )
_section(
lines, "Confidence Above Source Standing (ceiling, not a formula) - recommendation, not an error",
report.get("confidence_exceeds_source_standing", []),
lambda i: f"[[{i['page']}]] confidence_base={i['confidence_base']} exceeds the "
f"{i['ceiling']} ceiling its cited sources' fidelity/authority carry - "
"kb/CONTRACT.md § \"Confidence against source standing\"",
)
_section( _section(
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"], lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
lambda i: f"`{i}`", lambda i: f"`{i}`",
@@ -757,6 +819,13 @@ def default_report_path(report: dict) -> Path:
# would penalise the honest "I don't know yet" that the slot exists to allow, # would penalise the honest "I don't know yet" that the slot exists to allow,
# where the old silent `default: notes` hid the same uncertainty for free. # where the old silent `default: notes` hid the same uncertainty for free.
# #
# `confidence_exceeds_source_standing` is advisory by construction, like
# `unsharded_collections` above: `confidence_base` stays a hand-set judgment
# call (kb/CONTRACT.md § Confidence), and a source's capture standing is a
# ceiling a page may sit under by independent verification, not a value a
# formula could compute outright - see kb/CONTRACT.md § "Confidence against
# source standing" (Gitea #67).
#
# `malformed_edges` and `unbalanced_markers` are hard from the start: neither # `malformed_edges` and `unbalanced_markers` are hard from the start: neither
# describes an unconverted page, only a broken one. # describes an unconverted page, only a broken one.
# #
+7 -13
View File
@@ -364,21 +364,15 @@ def test_page_type_specs_ship_as_templates_and_stack_types_do_not(repo, monkeypa
assert "types/instruction.md.template" not in plan assert "types/instruction.md.template" not in plan
def test_plan_creates_empty_raw_subdirs_not_real_content(repo): def test_plan_creates_empty_raw_and_incoming_not_real_content(repo):
"""Both flat since Gitea #67: `raw/` addresses a file by its accept date,
never by a hand-picked type, so there is nothing left to seed per type."""
plan = dist_cmd.build_plan() plan = dist_cmd.build_plan()
for sub in ("articles", "documents", "notes", "assets"): assert "raw/.gitkeep" in plan
assert f"raw/{sub}/.gitkeep" in plan assert "incoming/.gitkeep" in plan
assert not any("personal-note" in relative for relative in plan) assert not any("personal-note" in relative for relative in plan)
def test_plan_creates_matching_incoming_subdirs(repo):
"""The ingest inbox (Gitea #58) mirrors raw/'s type subdirectories one for
one - both come from the same `RAW_SUBDIRS` tuple."""
plan = dist_cmd.build_plan()
for sub in ("articles", "documents", "notes", "assets"):
assert f"incoming/{sub}/.gitkeep" in plan
def test_plan_seeds_log_and_changes_from_templates(repo): def test_plan_seeds_log_and_changes_from_templates(repo):
plan = dist_cmd.build_plan() plan = dist_cmd.build_plan()
assert "Wiki Log" in plan["kb/log.md"].content assert "Wiki Log" in plan["kb/log.md"].content
@@ -496,8 +490,8 @@ def test_export_into_a_fresh_directory_works(repo, tmp_path):
dist_cmd.run_export(target, dry_run=False) dist_cmd.run_export(target, dry_run=False)
assert (target / "AGENTS.md").is_file() assert (target / "AGENTS.md").is_file()
assert (target / "kb" / "entities" / "COLLECTION.md.template").is_file() assert (target / "kb" / "entities" / "COLLECTION.md.template").is_file()
assert (target / "raw" / "notes" / ".gitkeep").is_file() assert (target / "raw" / ".gitkeep").is_file()
assert (target / "incoming" / "notes" / ".gitkeep").is_file() assert (target / "incoming" / ".gitkeep").is_file()
def test_unbalanced_markers_fail_loudly(repo): def test_unbalanced_markers_fail_loudly(repo):
-25
View File
@@ -134,31 +134,6 @@ def test_legacy_type_regex_matches_pre_migration_form():
assert not docs_verify.LEGACY_TYPE_RE.search("---\ntype: types/comparison.md\n---") assert not docs_verify.LEGACY_TYPE_RE.search("---\ntype: types/comparison.md\n---")
def test_this_repos_raw_subdirs_are_documented():
assert docs_verify.check_raw_subdirs() == []
def test_raw_subdirs_mismatch_is_reported(tmp_path, monkeypatch):
"""The regression this guards: `dist_cmd.RAW_SUBDIRS` and raw/CONTRACT.md's
routing table are two places naming the same set (AGENTS.md invariant 8),
so either one drifting from the other must be caught in both directions."""
(tmp_path / "raw").mkdir()
(tmp_path / "raw" / "CONTRACT.md").write_text(
"| Directory | Holds |\n|---|---|\n| `articles/` | ... |\n| `videos/` | ... |\n",
encoding="utf-8",
)
monkeypatch.setattr(config, "ROOT", tmp_path)
monkeypatch.setattr(docs_verify.dist_cmd, "RAW_SUBDIRS", ("articles", "documents"))
issues = docs_verify.check_raw_subdirs()
assert any("missing `documents/`" in i for i in issues)
assert any("lists `videos/`" in i for i in issues)
def test_raw_subdirs_check_is_skipped_without_a_contract(tmp_path, monkeypatch):
monkeypatch.setattr(config, "ROOT", tmp_path)
assert docs_verify.check_raw_subdirs() == []
def test_no_content_is_gitignored(): def test_no_content_is_gitignored():
"""The regression guard for the 2026-08-13 `.gitignore` rewrite: patterns """The regression guard for the 2026-08-13 `.gitignore` rewrite: patterns
like `*temp*` and `bin/` were silently excluding files under raw/, so the like `*temp*` and `bin/` were silently excluding files under raw/, so the
+125
View File
@@ -892,3 +892,128 @@ def test_redundant_see_also_reaches_the_rendered_report_and_the_summary(kb_dir):
summary = render_summary(report) summary = render_summary(report)
assert "Redundant see-also" in summary assert "Redundant see-also" in summary
assert "[[nearside]]" in summary and "depends-on" in summary assert "[[nearside]]" in summary and "depends-on" in summary
# --- Confidence against source standing (Gitea #67) --------------------------
def _write_capped_source(kb_dir, title, *, fidelity=None, authority=None, raw_path=None):
"""A distinct, never-colliding raw_files: path per title by default - the
fixture's own 'Source - Aurora' already claims raw/notes/Aurora.md, and
two of these in one test must not claim the same path either."""
if raw_path is None:
raw_path = f"raw/notes/{title.replace(' ', '-')}.md"
frontmatter = {
"type": "types/source.md", "source_type": "notes", "author": "Torben",
"raw_files": [raw_path], "date": "2026-09-01",
"tags": [], "entities": [], "concepts": [], "summary": "Test source.",
}
if fidelity is not None:
frontmatter["fidelity"] = fidelity
if authority is not None:
frontmatter["authority"] = authority
write_page(kb_dir / "sources" / f"{title}.md", frontmatter, f"\n# {title}\n\n## Summary\n\nTest.\n")
def test_confidence_exceeds_source_standing_flags_a_high_confidence_opinion_source(kb_dir):
_write_capped_source(kb_dir, "Source - Weak", fidelity="secondhand", authority="opinion")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Weak"],
"confidence": 0.9, "confidence_base": 0.9},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert {"page": "Modbus", "confidence_base": 0.9, "ceiling": 0.6} in report["confidence_exceeds_source_standing"]
# Advisory, not a hard error - unlike the fixture's own baseline issues,
# this finding on its own must never appear in `hard_error_keys()`.
assert "confidence_exceeds_source_standing" not in HARD_ERROR_KEYS
def test_confidence_exceeds_source_standing_is_silent_at_or_under_the_ceiling(kb_dir):
_write_capped_source(kb_dir, "Source - Weak", fidelity="secondhand", authority="opinion")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Weak"],
"confidence": 0.6, "confidence_base": 0.6},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert report["confidence_exceeds_source_standing"] == []
def test_confidence_exceeds_source_standing_is_silent_on_normative_verbatim_sources(kb_dir):
"""`normative`/`verbatim` carry no ceiling at all - a page may sit as
confident as its own hand-set judgment allows."""
_write_capped_source(kb_dir, "Source - Strong", fidelity="verbatim", authority="normative")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Strong"],
"confidence": 0.99, "confidence_base": 0.99},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert report["confidence_exceeds_source_standing"] == []
def test_confidence_exceeds_source_standing_is_silent_when_capture_fields_are_unknown(kb_dir):
"""A backfilled `unknown` is not a claim about the source - firing on it
would report every page citing a pre-#67 source at once."""
_write_capped_source(kb_dir, "Source - Backfilled", fidelity="unknown", authority="unknown")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Backfilled"],
"confidence": 0.99, "confidence_base": 0.99},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert report["confidence_exceeds_source_standing"] == []
def test_confidence_exceeds_source_standing_is_silent_without_capture_fields_at_all(kb_dir):
"""A source page predating Gitea #67 carries neither field yet - same
silence as the explicit `unknown` case, not a finding by omission."""
_write_capped_source(kb_dir, "Source - Predates 67")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Predates 67"],
"confidence": 0.99, "confidence_base": 0.99},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert report["confidence_exceeds_source_standing"] == []
def test_confidence_exceeds_source_standing_takes_the_tightest_ceiling_among_several_sources(kb_dir):
_write_capped_source(kb_dir, "Source - A", fidelity="published", authority="reporting")
_write_capped_source(kb_dir, "Source - B", fidelity="secondhand", authority="opinion")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - A", "Source - B"],
"confidence": 0.7, "confidence_base": 0.7},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert {"page": "Modbus", "confidence_base": 0.7, "ceiling": 0.6} in report["confidence_exceeds_source_standing"]
def test_confidence_exceeds_source_standing_reaches_the_rendered_report_and_the_summary(kb_dir):
_write_capped_source(kb_dir, "Source - Weak", fidelity="secondhand", authority="opinion")
write_page(
kb_dir / "concepts/protocols/Modbus.md",
{"type": "types/concept.md", "concept_type": "protocol", "tags": [], "created": "2026-07-25",
"modified": "2026-07-25", "related": [], "sources": ["Source - Weak"],
"confidence": 0.9, "confidence_base": 0.9},
"\n# Modbus\n\n## Definition\n\nx.\n",
)
report = run_lint(kb_dir)
assert "Confidence Above Source Standing" in render_markdown(report)
summary = render_summary(report)
assert "Confidence Above Source Standing" in summary
assert "[[Modbus]]" in summary
+38
View File
@@ -195,6 +195,7 @@ def test_new_source_prefixes_title_and_prefills_related_entities(monkeypatch, kb
"new", "source", "--name", "gateway.example.net", "new", "source", "--name", "gateway.example.net",
"--set", "source_type=notes", "--set", "source_type=notes",
"--set", "raw_files=raw/notes/gateway.example.net.md", "--set", "raw_files=raw/notes/gateway.example.net.md",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
"--set", "entities=aurora,Borealis", "--set", "entities=aurora,Borealis",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
@@ -221,6 +222,7 @@ def test_new_source_rejects_missing_source_type(monkeypatch, kb_dir):
result = _invoke_new(monkeypatch, kb_dir, [ result = _invoke_new(monkeypatch, kb_dir, [
"new", "source", "--name", "no-source-type", "new", "source", "--name", "no-source-type",
"--set", "raw_files=raw/notes/gateway.example.net.md", "--set", "raw_files=raw/notes/gateway.example.net.md",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
]) ])
assert result.exit_code != 0 assert result.exit_code != 0
assert "source_type" in result.output assert "source_type" in result.output
@@ -229,6 +231,39 @@ def test_new_source_rejects_missing_source_type(monkeypatch, kb_dir):
assert not list(kb_dir.glob("sources/**/Source - no-source-type.md")) assert not list(kb_dir.glob("sources/**/Source - no-source-type.md"))
def test_new_source_rejects_missing_capture_fields(monkeypatch, kb_dir):
"""Gitea #67: `fidelity`/`authority` have no `default:` and are not in
`required:` either (a required field would be boundary-crossing, see
version-parts.md) - the refusal is enforced by `new` itself instead."""
monkeypatch.setenv("WIKI_AUTHOR", "Torben")
_fixture_raw_file(monkeypatch, kb_dir, "raw/notes/gateway.example.net.md")
result = _invoke_new(monkeypatch, kb_dir, [
"new", "source", "--name", "no-capture-fields",
"--set", "source_type=notes",
"--set", "raw_files=raw/notes/gateway.example.net.md",
])
assert result.exit_code != 0
assert "fidelity" in result.output
assert "authority" in result.output
assert not list(kb_dir.glob("sources/**/Source - no-capture-fields.md"))
def test_new_source_rejects_unknown_as_a_capture_value(monkeypatch, kb_dir):
"""`unknown` is backfill-only - only `wikitool touch` may write it, on a
page predating this rule (Gitea #67)."""
monkeypatch.setenv("WIKI_AUTHOR", "Torben")
_fixture_raw_file(monkeypatch, kb_dir, "raw/notes/gateway.example.net.md")
result = _invoke_new(monkeypatch, kb_dir, [
"new", "source", "--name", "unknown-capture",
"--set", "source_type=notes",
"--set", "raw_files=raw/notes/gateway.example.net.md",
"--set", "fidelity=unknown", "--set", "authority=reporting",
])
assert result.exit_code != 0
assert "unknown" in result.output
assert not list(kb_dir.glob("sources/**/Source - unknown-capture.md"))
def test_new_source_author_falls_back_to_git_config(monkeypatch, kb_dir): def test_new_source_author_falls_back_to_git_config(monkeypatch, kb_dir):
"""No WIKI_AUTHOR set - default_author() falls back to `git config """No WIKI_AUTHOR set - default_author() falls back to `git config
user.name`, run with cwd=config.ROOT. user.name`, run with cwd=config.ROOT.
@@ -251,6 +286,7 @@ def test_new_source_author_falls_back_to_git_config(monkeypatch, kb_dir):
"new", "source", "--name", "git-config-author", "new", "source", "--name", "git-config-author",
"--set", "source_type=notes", "--set", "source_type=notes",
"--set", "raw_files=raw/notes/gateway.example.net.md", "--set", "raw_files=raw/notes/gateway.example.net.md",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
fm, _body = read_page(kb_dir / "sources/notes/Source - git-config-author.md") fm, _body = read_page(kb_dir / "sources/notes/Source - git-config-author.md")
@@ -363,6 +399,7 @@ def test_raw_files_error_points_at_the_comma_split(monkeypatch, kb_dir, raw_dir)
"new", "source", "--name", "Split Path", "new", "source", "--name", "Split Path",
"--set", "source_type=notes", "--set", "source_type=notes",
"--set", "raw_files=raw/notes/Versioning, CI-CD.md", "--set", "raw_files=raw/notes/Versioning, CI-CD.md",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
]) ])
assert result.exit_code == 1 assert result.exit_code == 1
assert "splitting the value on commas" in result.output assert "splitting the value on commas" in result.output
@@ -382,6 +419,7 @@ def test_source_page_accepts_a_raw_file_whose_name_has_a_comma(monkeypatch, kb_d
"new", "source", "--name", "Comma Source", "new", "source", "--name", "Comma Source",
"--set", r"raw_files=raw/notes/Versioning\, CI-CD.md", "--set", r"raw_files=raw/notes/Versioning\, CI-CD.md",
"--set", "source_type=notes", "--set", "source_type=notes",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
frontmatter, _ = read_page(kb_dir / "sources/notes/Source - Comma Source.md") frontmatter, _ = read_page(kb_dir / "sources/notes/Source - Comma Source.md")
+1
View File
@@ -402,6 +402,7 @@ def test_new_source_with_multiple_raw_files(kb_dir, raw_dir, monkeypatch):
"new", "source", "--name", "Multi", "new", "source", "--name", "Multi",
"--set", "source_type=notes", "--set", "source_type=notes",
"--set", "raw_files=raw/notes/Aurora.md,raw/notes/Second.md", "--set", "raw_files=raw/notes/Aurora.md,raw/notes/Second.md",
"--set", "fidelity=verbatim", "--set", "authority=reporting",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
fm, _body = read_page(kb_dir / "sources/notes/Source - Multi.md") fm, _body = read_page(kb_dir / "sources/notes/Source - Multi.md")
+255 -107
View File
@@ -1,3 +1,5 @@
import datetime
import pytest import pytest
import typer import typer
@@ -8,75 +10,96 @@ from chemenu.provenance import uncovered_raw_files
from chemenu.kb_scan import load_kb_pages from chemenu.kb_scan import load_kb_pages
def _shard() -> str:
"""Today's `raw/<YYYY>/<MM>` shard, as a relative-path string - matches
`raw_cmd._shard_dir()` without importing its internals, the same way the
rest of the suite asserts against real dates (see test_touch.py)."""
today = datetime.date.today()
return f"raw/{today.year:04d}/{today.month:02d}"
@pytest.fixture @pytest.fixture
def tree(kb_dir): def tree(kb_dir):
"""kb_dir already repoints config.ROOT at tmp_path; add raw/ and """kb_dir already repoints config.ROOT at tmp_path; add raw/ and
incoming/ beside it, each with the four type subdirectories incoming/ beside it. raw/ keeps two pre-#67 legacy type directories (not
dist_cmd.RAW_SUBDIRS declares.""" all four - #67 stops treating them as a declared, exhaustive set), so
tests can exercise both the legacy flat layout and the date shard.
incoming/ is flat now (Gitea #67): no subdirectory is required for a
promotion to succeed."""
root = kb_dir.parent root = kb_dir.parent
for sub in ("articles", "documents", "notes", "assets"): (root / "raw" / "documents").mkdir(parents=True)
(root / "raw" / sub).mkdir(parents=True) (root / "raw" / "notes").mkdir(parents=True)
(root / "incoming" / sub).mkdir(parents=True) (root / "incoming").mkdir(parents=True, exist_ok=True)
return root return root
def _accept(*files, page=None, replaces=None, dry_run=False): def _accept(*files, page=None, replaces=None, dry_run=False, fidelity="verbatim", authority="reporting"):
return raw_accept_command(files=list(files), page=page, replaces=replaces, dry_run=dry_run) return raw_accept_command(
files=list(files), fidelity=fidelity, authority=authority,
page=page, replaces=replaces, dry_run=dry_run,
def _write_source(kb_dir, title, raw_files):
write_page(
kb_dir / "sources" / f"{title}.md",
{
"type": "types/source.md", "source_type": "document", "author": "Torben",
"raw_files": list(raw_files), "date": "2026-09-01",
"tags": [], "entities": [], "concepts": [], "summary": "Test source.",
},
f"\n# {title}\n\n## Summary\n\nTest.\n",
) )
def _write_source(kb_dir, title, raw_files, fidelity=None, authority=None):
frontmatter = {
"type": "types/source.md", "source_type": "document", "author": "Torben",
"raw_files": list(raw_files), "date": "2026-09-01",
"tags": [], "entities": [], "concepts": [], "summary": "Test source.",
}
if fidelity is not None:
frontmatter["fidelity"] = fidelity
if authority is not None:
frontmatter["authority"] = authority
write_page(kb_dir / "sources" / f"{title}.md", frontmatter, f"\n# {title}\n\n## Summary\n\nTest.\n")
def test_single_file_needs_no_bundle(tree): def test_single_file_needs_no_bundle(tree):
src = tree / "incoming/documents/handbuch.pdf" src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"%PDF-1.4 fake\n") src.write_bytes(b"%PDF-1.4 fake\n")
_accept(src) _accept(src)
dst = tree / "raw/documents/handbuch.pdf" dst = tree / _shard() / "handbuch.pdf"
assert dst.is_file() assert dst.is_file()
assert dst.read_bytes() == b"%PDF-1.4 fake\n" assert dst.read_bytes() == b"%PDF-1.4 fake\n"
assert not src.exists() assert not src.exists()
def test_two_files_bundle_under_the_first_files_stem(tree): def test_two_files_bundle_under_the_first_files_stem(tree):
pdf = tree / "incoming/documents/handbuch.pdf" pdf = tree / "incoming/handbuch.pdf"
md = tree / "incoming/documents/handbuch.md" md = tree / "incoming/handbuch.md"
pdf.write_bytes(b"pdf-bytes") pdf.write_bytes(b"pdf-bytes")
md.write_text("# converted\n", encoding="utf-8") md.write_text("# converted\n", encoding="utf-8")
_accept(pdf, md) _accept(pdf, md)
assert (tree / "raw/documents/handbuch/handbuch.pdf").read_bytes() == b"pdf-bytes" bundle = tree / _shard() / "handbuch"
assert (tree / "raw/documents/handbuch/handbuch.md").read_text(encoding="utf-8") == "# converted\n" assert (bundle / "handbuch.pdf").read_bytes() == b"pdf-bytes"
assert (bundle / "handbuch.md").read_text(encoding="utf-8") == "# converted\n"
assert not pdf.exists() and not md.exists() assert not pdf.exists() and not md.exists()
def test_file_directly_in_incoming_is_rejected(tree): def test_file_directly_in_incoming_is_accepted(tree):
"""Gitea #67: incoming/ is flat, so a file directly in it is the normal
case now, not a rejection - unlike the pre-#67 behaviour this replaces."""
src = tree / "incoming/stray.md" src = tree / "incoming/stray.md"
src.write_text("x\n", encoding="utf-8") src.write_text("x\n", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(src) _accept(src)
assert src.exists() assert (tree / _shard() / "stray.md").is_file()
assert not src.exists()
def test_unknown_type_subdir_is_rejected(tree): def test_subdirectory_under_incoming_is_ignored_not_inspected(tree):
"""Gitea #67: a subdirectory of incoming/ - old habit, old script - is
tolerated and ignored rather than read as a type classification. This is
the MINOR condition named in the issue's Versionsteil."""
(tree / "incoming/videos").mkdir() (tree / "incoming/videos").mkdir()
src = tree / "incoming/videos/clip.mp4" src = tree / "incoming/videos/clip.mp4"
src.write_bytes(b"x") src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src) _accept(src)
assert src.exists() assert (tree / _shard() / "clip.mp4").is_file()
assert not src.exists()
def test_nested_too_deep_is_rejected(tree): def test_nested_too_deep_is_rejected(tree):
nested = tree / "incoming/documents/sub" nested = tree / "incoming/documents/sub"
nested.mkdir() nested.mkdir(parents=True)
src = nested / "deep.pdf" src = nested / "deep.pdf"
src.write_bytes(b"x") src.write_bytes(b"x")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
@@ -84,75 +107,116 @@ def test_nested_too_deep_is_rejected(tree):
assert src.exists() assert src.exists()
def test_mixed_type_subdirs_in_one_call_is_rejected(tree): def test_files_from_different_ignored_subdirs_bundle_together(tree):
"""No more per-call type agreement to enforce (Gitea #67): which ignored
subdirectory each file happened to sit under is irrelevant now."""
a = tree / "incoming/documents/a.pdf" a = tree / "incoming/documents/a.pdf"
b = tree / "incoming/notes/b.md" b = tree / "incoming/notes/a.md"
a.parent.mkdir(parents=True)
b.parent.mkdir(parents=True)
a.write_bytes(b"a") a.write_bytes(b"a")
b.write_text("b", encoding="utf-8") b.write_text("b", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(a, b) _accept(a, b)
assert a.exists() and b.exists() bundle = tree / _shard() / "a"
assert (bundle / "a.pdf").exists() and (bundle / "a.md").exists()
def test_same_file_passed_twice_is_rejected(tree): def test_same_file_passed_twice_is_rejected(tree):
src = tree / "incoming/documents/handbuch.pdf" src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x") src.write_bytes(b"x")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(src, src) _accept(src, src)
assert src.exists() assert src.exists()
def test_collision_with_existing_raw_file_is_rejected(tree): def test_collision_with_existing_file_at_same_shard_path_is_rejected(tree):
(tree / "raw/documents/handbuch.pdf").write_bytes(b"already there") dst_dir = tree / _shard()
src = tree / "incoming/documents/handbuch.pdf" dst_dir.mkdir(parents=True)
(dst_dir / "handbuch.pdf").write_bytes(b"already there")
src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"new") src.write_bytes(b"new")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(src) _accept(src)
assert src.exists() assert src.exists()
assert (tree / "raw/documents/handbuch.pdf").read_bytes() == b"already there" assert (dst_dir / "handbuch.pdf").read_bytes() == b"already there"
def test_missing_file_is_rejected(tree): def test_missing_file_is_rejected(tree):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(tree / "incoming/documents/absent.pdf") _accept(tree / "incoming/absent.pdf")
def test_missing_fidelity_is_rejected(tree):
src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
raw_accept_command(files=[src], fidelity=None, authority="reporting", page=None, replaces=None, dry_run=False)
assert src.exists()
def test_missing_authority_is_rejected(tree):
src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
raw_accept_command(files=[src], fidelity="verbatim", authority=None, page=None, replaces=None, dry_run=False)
assert src.exists()
def test_unknown_fidelity_is_rejected(tree):
src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src, fidelity="unknown")
assert src.exists()
def test_invalid_fidelity_value_is_rejected(tree):
src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src, fidelity="made-up")
assert src.exists()
def test_dry_run_moves_nothing(tree): def test_dry_run_moves_nothing(tree):
src = tree / "incoming/documents/handbuch.pdf" src = tree / "incoming/handbuch.pdf"
src.write_bytes(b"x") src.write_bytes(b"x")
_accept(src, dry_run=True) _accept(src, dry_run=True)
assert src.exists() assert src.exists()
assert not (tree / "raw/documents/handbuch.pdf").exists() assert not (tree / _shard() / "handbuch.pdf").exists()
def test_promoted_file_in_incoming_is_never_reported_uncovered(tree): def test_promoted_file_in_incoming_is_never_reported_uncovered(tree):
"""AC: incoming/ is invisible to sources coverage / lint until accepted - """AC: incoming/ is invisible to sources coverage / lint until accepted -
both walk raw/ only.""" both walk raw/ only."""
(tree / "incoming/notes/not-yet-promoted.md").write_text("draft\n", encoding="utf-8") (tree / "incoming/not-yet-promoted.md").write_text("draft\n", encoding="utf-8")
pages = load_kb_pages(tree / "kb") pages = load_kb_pages(tree / "kb")
assert "raw/notes/not-yet-promoted.md" not in uncovered_raw_files(tree / "raw", pages) assert "raw/notes/not-yet-promoted.md" not in uncovered_raw_files(tree / "raw", pages)
assert uncovered_raw_files(tree / "raw", pages) == [] assert uncovered_raw_files(tree / "raw", pages) == []
def test_without_page_flag_only_moves_and_prints_target(tree, capsys): def test_without_page_flag_prints_target_and_new_source_followup(tree, capsys):
src = tree / "incoming/notes/meeting.md" src = tree / "incoming/meeting.md"
src.write_text("notes\n", encoding="utf-8") src.write_text("notes\n", encoding="utf-8")
_accept(src) _accept(src, fidelity="secondhand", authority="opinion")
out = capsys.readouterr().out out = capsys.readouterr().out
assert "raw/notes/meeting.md" in out assert f"{_shard()}/meeting.md" in out
assert "new source" in out assert "new source" in out
assert "fidelity=secondhand" in out
assert "authority=opinion" in out
def test_page_flag_extends_raw_files_for_a_single_new_file(tree): def test_page_flag_extends_raw_files_and_sets_capture_fields_for_a_single_new_file(tree):
"""No growth case: the page already has >=1 file, so this always bundles - """No growth case: the page already has >=1 file, so this always bundles -
see test_growth_case below for the interesting path.""" see test_growth_case below for the interesting path. The pre-existing page
(tree / "raw/documents").mkdir(exist_ok=True) predates #67 and carries no capture fields yet, so this is also the first
write of them."""
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first") (tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/documents/handbuch-appendix.pdf" new_file = tree / "incoming/handbuch-appendix.pdf"
new_file.write_bytes(b"second") new_file.write_bytes(b"second")
_accept(new_file, page="Source - Handbuch") _accept(new_file, page="Source - Handbuch", fidelity="published", authority="normative")
bundle = tree / "raw/documents/handbuch" bundle = tree / "raw/documents/handbuch"
assert (bundle / "handbuch.pdf").read_bytes() == b"first" assert (bundle / "handbuch.pdf").read_bytes() == b"first"
@@ -163,12 +227,49 @@ def test_page_flag_extends_raw_files_for_a_single_new_file(tree):
assert sorted(frontmatter["raw_files"]) == sorted( assert sorted(frontmatter["raw_files"]) == sorted(
["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch-appendix.pdf"] ["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch-appendix.pdf"]
) )
assert frontmatter["fidelity"] == "published"
assert frontmatter["authority"] == "normative"
def test_page_flag_rejects_when_capture_field_already_set_differently(tree):
_write_source(
tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"],
fidelity="verbatim", authority="reporting",
)
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/handbuch-appendix.pdf"
new_file.write_bytes(b"second")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Handbuch", fidelity="secondhand", authority="reporting")
assert new_file.exists()
assert (tree / "raw/documents/handbuch.pdf").exists()
assert not (tree / "raw/documents/handbuch").exists()
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert frontmatter["fidelity"] == "verbatim"
def test_page_flag_is_idempotent_when_capture_field_already_set_the_same(tree):
_write_source(
tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"],
fidelity="verbatim", authority="reporting",
)
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/handbuch-appendix.pdf"
new_file.write_bytes(b"second")
_accept(new_file, page="Source - Handbuch", fidelity="verbatim", authority="reporting")
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert frontmatter["fidelity"] == "verbatim"
assert frontmatter["authority"] == "reporting"
def test_growth_case_no_broken_or_uncovered_refs_afterwards(tree): def test_growth_case_no_broken_or_uncovered_refs_afterwards(tree):
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first") (tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/documents/handbuch.md" new_file = tree / "incoming/handbuch.md"
new_file.write_text("converted", encoding="utf-8") new_file.write_text("converted", encoding="utf-8")
_accept(new_file, page="Source - Handbuch") _accept(new_file, page="Source - Handbuch")
@@ -189,11 +290,12 @@ def test_adding_to_an_already_bundled_source_joins_the_existing_bundle(tree):
_write_source( _write_source(
tree / "kb", "Source - Handbuch", tree / "kb", "Source - Handbuch",
["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch.md"], ["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch.md"],
fidelity="verbatim", authority="reporting",
) )
extra = tree / "incoming/documents/handbuch-notes.md" extra = tree / "incoming/handbuch-notes.md"
extra.write_text("c", encoding="utf-8") extra.write_text("c", encoding="utf-8")
_accept(extra, page="Source - Handbuch") _accept(extra, page="Source - Handbuch", fidelity="verbatim", authority="reporting")
assert (tree / "raw/documents/handbuch/handbuch-notes.md").read_text(encoding="utf-8") == "c" assert (tree / "raw/documents/handbuch/handbuch-notes.md").read_text(encoding="utf-8") == "c"
# Nothing already-bundled was moved a second time. # Nothing already-bundled was moved a second time.
@@ -210,7 +312,7 @@ def test_growth_case_rejects_a_multi_owner_raw_file(tree):
(tree / "raw/documents/handbuch.pdf").write_bytes(b"shared") (tree / "raw/documents/handbuch.pdf").write_bytes(b"shared")
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
_write_source(tree / "kb", "Source - Also Handbuch", ["raw/documents/handbuch.pdf"]) _write_source(tree / "kb", "Source - Also Handbuch", ["raw/documents/handbuch.pdf"])
new_file = tree / "incoming/documents/handbuch.md" new_file = tree / "incoming/handbuch.md"
new_file.write_text("x", encoding="utf-8") new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
@@ -222,7 +324,7 @@ def test_growth_case_rejects_a_multi_owner_raw_file(tree):
def test_page_not_found_is_rejected(tree): def test_page_not_found_is_rejected(tree):
new_file = tree / "incoming/documents/handbuch.md" new_file = tree / "incoming/handbuch.md"
new_file.write_text("x", encoding="utf-8") new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Nonexistent") _accept(new_file, page="Source - Nonexistent")
@@ -239,37 +341,29 @@ def test_page_with_no_raw_files_is_rejected(tree):
}, },
"\n# Source - Empty\n", "\n# Source - Empty\n",
) )
new_file = tree / "incoming/documents/handbuch.md" new_file = tree / "incoming/handbuch.md"
new_file.write_text("x", encoding="utf-8") new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Empty") _accept(new_file, page="Source - Empty")
assert new_file.exists() assert new_file.exists()
def test_type_directory_mismatch_with_existing_raw_files_is_rejected(tree): # --- Stem uniqueness across raw/ (Gitea #64, widened globally by #67) ---
(tree / "raw/notes/handbuch.md").write_bytes(b"first")
_write_source(tree / "kb", "Source - Handbuch", ["raw/notes/handbuch.md"])
new_file = tree / "incoming/documents/handbuch.pdf"
new_file.write_bytes(b"second")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Handbuch")
assert new_file.exists()
# --- Stem uniqueness at raw/<type>/ level (Gitea #64) --- def test_second_source_does_not_silently_join_an_existing_legacy_bundle(tree):
"""The exact repro from #64, now crossing the legacy/shard boundary: a
bundle exists in the pre-#67 flat layout, and a second, unrelated source's
def test_second_source_does_not_silently_join_an_existing_bundle(tree): files (no --page) land in today's date shard - they must still not slip
"""The exact repro from #64: a bundle exists for one source, and a second, into the existing bundle just because the two live in different
unrelated source's files (no --page) happen not to collide on filename - directories now."""
they must not slip into the existing bundle."""
bundle = tree / "raw/documents/handbuch" bundle = tree / "raw/documents/handbuch"
bundle.mkdir() bundle.mkdir()
(bundle / "handbuch.pdf").write_bytes(b"pdf") (bundle / "handbuch.pdf").write_bytes(b"pdf")
(bundle / "handbuch.md").write_text("md", encoding="utf-8") (bundle / "handbuch.md").write_text("md", encoding="utf-8")
txt = tree / "incoming/documents/handbuch.txt" txt = tree / "incoming/handbuch.txt"
anhang = tree / "incoming/documents/anhang.md" anhang = tree / "incoming/anhang.md"
txt.write_text("txt", encoding="utf-8") txt.write_text("txt", encoding="utf-8")
anhang.write_text("anhang", encoding="utf-8") anhang.write_text("anhang", encoding="utf-8")
@@ -280,11 +374,11 @@ def test_second_source_does_not_silently_join_an_existing_bundle(tree):
assert sorted(p.name for p in bundle.iterdir()) == ["handbuch.md", "handbuch.pdf"] assert sorted(p.name for p in bundle.iterdir()) == ["handbuch.md", "handbuch.pdf"]
def test_flat_promote_rejected_when_stem_matches_an_existing_bundle(tree): def test_flat_promote_rejected_when_stem_matches_an_existing_legacy_bundle(tree):
(tree / "raw/documents/handbuch").mkdir() (tree / "raw/documents/handbuch").mkdir()
(tree / "raw/documents/handbuch/handbuch.pdf").write_bytes(b"pdf") (tree / "raw/documents/handbuch/handbuch.pdf").write_bytes(b"pdf")
txt = tree / "incoming/documents/handbuch.txt" txt = tree / "incoming/handbuch.txt"
txt.write_text("txt", encoding="utf-8") txt.write_text("txt", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
@@ -294,12 +388,12 @@ def test_flat_promote_rejected_when_stem_matches_an_existing_bundle(tree):
assert (tree / "raw/documents/handbuch/handbuch.pdf").exists() assert (tree / "raw/documents/handbuch/handbuch.pdf").exists()
def test_flat_promote_rejected_when_stem_matches_an_existing_flat_file(tree): def test_flat_promote_rejected_when_stem_matches_an_existing_legacy_flat_file(tree):
"""Same stem, different extension: not caught by the old per-path """Same stem, different extension: not caught by the old per-path
dst.exists() check, which is the whole gap #64 closes.""" dst.exists() check, which is the whole gap #64 closes."""
(tree / "raw/documents/handbuch.pdf").write_bytes(b"pdf") (tree / "raw/documents/handbuch.pdf").write_bytes(b"pdf")
md = tree / "incoming/documents/handbuch.md" md = tree / "incoming/handbuch.md"
md.write_text("md", encoding="utf-8") md.write_text("md", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
@@ -307,12 +401,26 @@ def test_flat_promote_rejected_when_stem_matches_an_existing_flat_file(tree):
assert md.exists() assert md.exists()
assert (tree / "raw/documents/handbuch.pdf").read_bytes() == b"pdf" assert (tree / "raw/documents/handbuch.pdf").read_bytes() == b"pdf"
assert not (tree / "raw/documents/handbuch.md").exists()
def test_second_file_with_same_stem_in_same_shard_is_rejected(tree):
"""Uniqueness also holds within one date shard, not only against the
legacy layout."""
first = tree / "incoming/foo.md"
first.write_text("first", encoding="utf-8")
_accept(first)
assert (tree / _shard() / "foo.md").is_file()
second = tree / "incoming/foo.txt"
second.write_text("second", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(second)
assert second.exists()
def test_stem_collision_message_names_both_routes_without_recommending_one(tree, capsys): def test_stem_collision_message_names_both_routes_without_recommending_one(tree, capsys):
(tree / "raw/documents/handbuch.pdf").write_bytes(b"pdf") (tree / "raw/documents/handbuch.pdf").write_bytes(b"pdf")
md = tree / "incoming/documents/handbuch.md" md = tree / "incoming/handbuch.md"
md.write_text("md", encoding="utf-8") md.write_text("md", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
@@ -324,22 +432,58 @@ def test_stem_collision_message_names_both_routes_without_recommending_one(tree,
assert "does not guess" in err assert "does not guess" in err
# --- --replaces (Gitea #64 decision 2) --- # --- --replaces (Gitea #64 decision 2, --fidelity/--authority the correction
# path for a capture field per Gitea #67) ---
def test_replaces_swaps_bytes_and_leaves_raw_files_untouched(tree): def test_replaces_swaps_bytes_and_leaves_raw_files_untouched(tree):
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"])
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old edition", encoding="utf-8") target.write_text("old edition", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new edition", encoding="utf-8") new.write_text("new edition", encoding="utf-8")
_accept(new, replaces=target) _accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "new edition" assert target.read_text(encoding="utf-8") == "new edition"
assert not new.exists() assert not new.exists()
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md") frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert frontmatter["raw_files"] == ["raw/documents/handbuch.md"] assert frontmatter["raw_files"] == ["raw/documents/handbuch.md"]
assert "fidelity" not in frontmatter
def test_replaces_with_fidelity_overwrites_the_owning_pages_capture_field(tree):
"""The one sanctioned correction path (Gitea #67): --replaces may pass
--fidelity/--authority to fix an already-set capture value, unlike
`touch`, which fill-once refuses to."""
_write_source(
tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"],
fidelity="secondhand", authority="opinion",
)
target = tree / "raw/documents/handbuch.md"
target.write_text("old edition", encoding="utf-8")
new = tree / "incoming/handbuch.md"
new.write_text("new edition", encoding="utf-8")
_accept(new, replaces=target, fidelity="verbatim", authority=None)
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert frontmatter["fidelity"] == "verbatim"
assert frontmatter["authority"] == "opinion"
def test_replaces_rejects_unknown_fidelity(tree):
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"])
target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8")
new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new, replaces=target, fidelity="unknown", authority=None)
assert target.read_text(encoding="utf-8") == "old"
assert new.exists()
def test_replaces_reports_source_and_both_citing_pages(tree, capsys): def test_replaces_reports_source_and_both_citing_pages(tree, capsys):
@@ -365,10 +509,10 @@ def test_replaces_reports_source_and_both_citing_pages(tree, capsys):
) )
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
_accept(new, replaces=target) _accept(new, replaces=target, fidelity=None, authority=None)
out = capsys.readouterr().out out = capsys.readouterr().out
assert "Source - Handbuch" in out assert "Source - Handbuch" in out
@@ -379,10 +523,10 @@ def test_replaces_reports_source_and_both_citing_pages(tree, capsys):
def test_replaces_succeeds_with_no_owner_and_reports_it(tree, capsys): def test_replaces_succeeds_with_no_owner_and_reports_it(tree, capsys):
target = tree / "raw/documents/orphan.md" target = tree / "raw/documents/orphan.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/orphan.md" new = tree / "incoming/orphan.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
_accept(new, replaces=target) _accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "new" assert target.read_text(encoding="utf-8") == "new"
out = capsys.readouterr().out out = capsys.readouterr().out
@@ -392,10 +536,10 @@ def test_replaces_succeeds_with_no_owner_and_reports_it(tree, capsys):
def test_replaces_dry_run_writes_nothing(tree): def test_replaces_dry_run_writes_nothing(tree):
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
_accept(new, replaces=target, dry_run=True) _accept(new, replaces=target, dry_run=True, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "old"
assert new.exists() assert new.exists()
@@ -404,13 +548,13 @@ def test_replaces_dry_run_writes_nothing(tree):
def test_replaces_rejects_more_than_one_incoming_file(tree): def test_replaces_rejects_more_than_one_incoming_file(tree):
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
a = tree / "incoming/documents/handbuch.md" a = tree / "incoming/handbuch.md"
b = tree / "incoming/documents/extra.md" b = tree / "incoming/extra.md"
a.write_text("a", encoding="utf-8") a.write_text("a", encoding="utf-8")
b.write_text("b", encoding="utf-8") b.write_text("b", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(a, b, replaces=target) _accept(a, b, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "old"
assert a.exists() and b.exists() assert a.exists() and b.exists()
@@ -419,35 +563,39 @@ def test_replaces_rejects_more_than_one_incoming_file(tree):
def test_replaces_rejects_filename_mismatch(tree): def test_replaces_rejects_filename_mismatch(tree):
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch-v2.md" new = tree / "incoming/handbuch-v2.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new, replaces=target) _accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "old"
assert new.exists() assert new.exists()
def test_replaces_rejects_type_directory_mismatch(tree): def test_replaces_across_legacy_directories_is_now_allowed(tree):
"""Gitea #67 removes the type-directory-match check `--replaces` used to
enforce: a subdirectory of incoming/ carries no meaning any more, so
replacing a raw/notes/ file with an incoming file dropped under an
unrelated incoming/documents/ works exactly like one dropped flat."""
target = tree / "raw/notes/handbuch.md" target = tree / "raw/notes/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/documents/handbuch.md"
new.parent.mkdir(parents=True)
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit): _accept(new, replaces=target, fidelity=None, authority=None)
_accept(new, replaces=target)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "new"
assert new.exists() assert not new.exists()
def test_replaces_rejects_nonexistent_target(tree): def test_replaces_rejects_nonexistent_target(tree):
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new, replaces=tree / "raw/documents/handbuch.md") _accept(new, replaces=tree / "raw/documents/handbuch.md", fidelity=None, authority=None)
assert new.exists() assert new.exists()
@@ -457,11 +605,11 @@ def test_replaces_rejects_multi_owner_target(tree):
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"])
_write_source(tree / "kb", "Source - Also Handbuch", ["raw/documents/handbuch.md"]) _write_source(tree / "kb", "Source - Also Handbuch", ["raw/documents/handbuch.md"])
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new, replaces=target) _accept(new, replaces=target, fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "old"
assert new.exists() assert new.exists()
@@ -471,11 +619,11 @@ def test_replaces_rejects_combination_with_page(tree):
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"]) _write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.md"])
target = tree / "raw/documents/handbuch.md" target = tree / "raw/documents/handbuch.md"
target.write_text("old", encoding="utf-8") target.write_text("old", encoding="utf-8")
new = tree / "incoming/documents/handbuch.md" new = tree / "incoming/handbuch.md"
new.write_text("new", encoding="utf-8") new.write_text("new", encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_accept(new, replaces=target, page="Source - Handbuch") _accept(new, replaces=target, page="Source - Handbuch", fidelity=None, authority=None)
assert target.read_text(encoding="utf-8") == "old" assert target.read_text(encoding="utf-8") == "old"
assert new.exists() assert new.exists()
+33
View File
@@ -230,3 +230,36 @@ def test_dry_run_covers_set_too(touch_wiki):
before = path.read_text(encoding="utf-8") before = path.read_text(encoding="utf-8")
_touch(page_title="aurora", set_fields=["tags=nope"], dry_run=True) _touch(page_title="aurora", set_fields=["tags=nope"], dry_run=True)
assert path.read_text(encoding="utf-8") == before assert path.read_text(encoding="utf-8") == before
# --- Capture fields: fill-once, not a UNSETTABLE denylist (Gitea #67) --------
def test_capture_field_is_writable_while_absent(touch_wiki):
"""The fixture 'Source - Aurora' predates #67 and carries no `fidelity:`
yet - the backfill path `touch --set fidelity=unknown` (or a real value)
must succeed exactly once."""
_touch(page_title="Source - Aurora", set_fields=["fidelity=unknown"])
frontmatter, _ = read_page(touch_wiki / "sources/notes/Source - Aurora.md")
assert frontmatter["fidelity"] == "unknown"
def test_capture_field_is_refused_once_already_set(touch_wiki, capsys):
_touch(page_title="Source - Aurora", set_fields=["authority=reporting"])
with pytest.raises(typer.Exit):
_touch(page_title="Source - Aurora", set_fields=["authority=opinion"])
out = capsys.readouterr().out
assert "capture field" in out
assert "raw accept --replaces" in out
frontmatter, _ = read_page(touch_wiki / "sources/notes/Source - Aurora.md")
assert frontmatter["authority"] == "reporting"
def test_capture_field_set_to_the_same_value_again_is_a_no_op(touch_wiki):
"""Not idempotent by accident: `_apply_set` already no-ops an unchanged
value before the write, and the fill-once refusal only fires on an actual
*change* - re-running the exact same backfill value must not error."""
_touch(page_title="Source - Aurora", set_fields=["fidelity=verbatim"])
_touch(page_title="Source - Aurora", set_fields=["fidelity=verbatim"])
frontmatter, _ = read_page(touch_wiki / "sources/notes/Source - Aurora.md")
assert frontmatter["fidelity"] == "verbatim"
+19
View File
@@ -64,6 +64,25 @@ def test_get_page_ref_fields_defaults_to_empty():
assert resolver.get_page_ref_fields("types/type-spec.md") == [] assert resolver.get_page_ref_fields("types/type-spec.md") == []
def test_get_capture_fields_reads_the_type_spec():
"""`fidelity`/`authority` are fixed once, at capture time (Gitea #67) -
`raw accept`, `new source` and `touch` all read the field list from here
instead of hardcoding it three times over."""
assert resolver.get_capture_fields("types/source.md") == ["fidelity", "authority"]
def test_capture_fields_exist_in_the_type_schema():
properties = resolver.get_schema("types/source.md")["properties"]
for field in resolver.get_capture_fields("types/source.md"):
assert field in properties, f"types/source.md declares unknown capture field {field}"
assert "unknown" in properties[field]["enum"]
assert "default" not in properties[field]
def test_get_capture_fields_defaults_to_empty():
assert resolver.get_capture_fields("types/entity.md") == []
def test_get_layout_reads_entity_type_specs_own_layout_field(): def test_get_layout_reads_entity_type_specs_own_layout_field():
"""new_page.py's directory placement and index_build.py's section """new_page.py's directory placement and index_build.py's section
titles/order derive from here instead of hand-maintained titles/order derive from here instead of hand-maintained
+22
View File
@@ -459,6 +459,28 @@ class TypeResolver:
type_spec = self.load_type_spec(type_path, source_file) type_spec = self.load_type_spec(type_path, source_file)
return list(type_spec['frontmatter'].get('page_ref_fields') or []) return list(type_spec['frontmatter'].get('page_ref_fields') or [])
def get_capture_fields(self, type_path: str, source_file: Path = None) -> list:
"""Return the frontmatter fields that are fixed at capture time and
never correctable afterwards except by re-capturing the source (e.g.
`['fidelity', 'authority']` for a source page), as declared by the
type-spec's own `capture_fields:` frontmatter (Gitea #67).
Three commands read this instead of hardcoding the field names
separately (AGENTS.md invariant 8 - one rule, one place):
`raw accept` and `new source` refuse to proceed without a value for
each; `touch --set` writes one only while it is still absent
(fill-once) and refuses once it holds a value, pointing at
`raw accept --replaces` for a genuine correction.
Returns an empty list for a type that declares none.
Raises:
ValueError: If the type path cannot be resolved (propagated from
`load_type_spec`)
"""
type_spec = self.load_type_spec(type_path, source_file)
return list(type_spec['frontmatter'].get('capture_fields') or [])
def list_type_specs(self) -> list: def list_type_specs(self) -> list:
"""Return every type-spec document under types/ as a list of """Return every type-spec document under types/ as a list of
`(type_path, frontmatter)` tuples, sorted by path. `(type_path, frontmatter)` tuples, sorted by path.
+4
View File
@@ -7,6 +7,7 @@ subtype_field: source_type
base_dir: sources base_dir: sources
title_prefix: "Source - " title_prefix: "Source - "
page_ref_fields: [entities, concepts] page_ref_fields: [entities, concepts]
capture_fields: [fidelity, authority]
layout: layout:
transcript: {dir: transcripts, title: Transkripte} transcript: {dir: transcripts, title: Transkripte}
analysis: {dir: analyses, title: Analysen} analysis: {dir: analyses, title: Analysen}
@@ -42,6 +43,8 @@ layout:
| `source_type` | Ja | Eines von: transcript, analysis, article, document, notes, tracker, unclassified - kein Default, siehe unten | | `source_type` | Ja | Eines von: transcript, analysis, article, document, notes, tracker, unclassified - kein Default, siehe unten |
| `author` | Ja | Urheber des Quellmaterials | | `author` | Ja | Urheber des Quellmaterials |
| `raw_files` | Ja | Raw-Dateipfade, die diese Quelle **besitzt** - siehe "Eine Raw-Datei, ein Besitzer" unten | | `raw_files` | Ja | Raw-Dateipfade, die diese Quelle **besitzt** - siehe "Eine Raw-Datei, ein Besitzer" unten |
| `fidelity` | Ja (im Werkzeug, nicht im Schema) | Wie treu die *Erfassung* ist: `verbatim`, `published`, `secondhand`, `nontextual` - Capture-Feld, siehe unten |
| `authority` | Ja (im Werkzeug, nicht im Schema) | Was das Material über seinen *Gegenstand* behaupten darf: `normative`, `reporting`, `opinion` - Capture-Feld, siehe unten |
| `source_url` | Nein | Ursprungs-URL bei externen Quellen | | `source_url` | Nein | Ursprungs-URL bei externen Quellen |
| `source_language` | Nein | ISO-639-1-Code der Sprache des Rohmaterials, z. B. `de`, `en`, `fr` | | `source_language` | Nein | ISO-639-1-Code der Sprache des Rohmaterials, z. B. `de`, `en`, `fr` |
| `date` | Ja | Veröffentlichungs- oder Erstellungsdatum (YYYY-MM-DD) | | `date` | Ja | Veröffentlichungs- oder Erstellungsdatum (YYYY-MM-DD) |
@@ -55,6 +58,7 @@ layout:
- Der Titel beginnt mit "Source - ", gefolgt vom Namen der Quelle - Der Titel beginnt mit "Source - ", gefolgt vom Namen der Quelle
- `source_type` hat **keinen Default** - `wikitool new source` verweigert ohne einen expliziten Wert. Ist die Kategorie unklar, `unclassified` setzen statt zu raten; das ist ein sichtbares Katalogfach mit beratendem `lint`-Befund, kein Sammelbecken. Was `analysis` von `document` trennt und welche Area welchen Wert hält: [kb/sources/COLLECTION.md](../kb/sources/COLLECTION.md) - `source_type` hat **keinen Default** - `wikitool new source` verweigert ohne einen expliziten Wert. Ist die Kategorie unklar, `unclassified` setzen statt zu raten; das ist ein sichtbares Katalogfach mit beratendem `lint`-Befund, kein Sammelbecken. Was `analysis` von `document` trennt und welche Area welchen Wert hält: [kb/sources/COLLECTION.md](../kb/sources/COLLECTION.md)
- `raw_files` listet jede Raw-Datei, die diese Quelle abdeckt (eine Source-Seite pro logischer Quelle, nicht pro Datei) - `raw_files` listet jede Raw-Datei, die diese Quelle abdeckt (eine Source-Seite pro logischer Quelle, nicht pro Datei)
- `fidelity` und `authority` sind **Capture-Felder** (`capture_fields:` oben): am Drop-Punkt erhoben, danach nicht mehr frei änderbar. `wikitool raw accept --fidelity <wert> --authority <wert>` verweigert ohne beide; ohne `--page` druckt es stattdessen die fertige `new source --set fidelity=... --set authority=...`-Zeile, die `new source` seinerseits ohne beide Werte verweigert. `wikitool touch --set fidelity=<wert>` schreibt nur, solange das Feld fehlt - steht bereits ein Wert, verweigert es und verweist auf `raw accept --replaces` als einzigen Korrekturweg (eine korrigierte Erfassung ist eine neue Edition, keine Bearbeitung). `unknown` ist backfill-only: nur `wikitool touch` darf es schreiben, nie `raw accept` oder `new source` - dieselbe Konstruktion wie bei `source_language` für Seiten, die vor dieser Regel entstanden sind. Was die Werte bedeuten und wie sie sich unterscheiden: [raw/CONTRACT.md](../raw/CONTRACT.md#getting-a-file-in-incoming)
- Bei externen Artikeln immer `source_url` auf die Ursprungs-URL setzen - Bei externen Artikeln immer `source_url` auf die Ursprungs-URL setzen
- `source_language` auf die Sprache des Rohmaterials setzen, nicht auf die der Seite - `source_language` auf die Sprache des Rohmaterials setzen, nicht auf die der Seite
- Die Seite wird in der KB-Sprache geschrieben, unabhängig von der Sprache der Quelle; wörtliche Passagen werden im Original zitiert (`kb/CONVENTIONS.md` § "Language") - Die Seite wird in der KB-Sprache geschrieben, unabhängig von der Sprache der Quelle; wörtliche Passagen werden im Original zitiert (`kb/CONVENTIONS.md` § "Language")
+18
View File
@@ -23,6 +23,24 @@ properties:
type: string type: string
description: List of raw file paths this source covers description: List of raw file paths this source covers
minItems: 1 minItems: 1
fidelity:
type: string
enum: [verbatim, published, secondhand, nontextual, unknown]
description: >-
How faithful the capture is to what was actually said or shown. No
default - a capture field, fixed at `raw accept`/`new source` time and
never guessed (Gitea #67). `unknown` is backfill-only: `raw accept` and
`new source` refuse it, only `wikitool touch` (on a page predating this
rule) may write it. Deliberately absent from `required:` so an existing
instance's source pages keep validating without it.
authority:
type: string
enum: [normative, reporting, opinion, unknown]
description: >-
What the material is entitled to claim about its subject - independent
of `fidelity`, which asks how faithfully it was captured. Same capture,
no-default, backfill-only-`unknown` rules as `fidelity` (Gitea #67).
Deliberately absent from `required:` for the same reason.
source_url: source_url:
type: [string, "null"] type: [string, "null"]
description: Upstream URL for external sources description: Upstream URL for external sources