raw accept: Datums-Shard statt Typverzeichnis, fidelity/authority am Drop-Punkt (Teil 1/3, #67)
Files changed: - .gitignore - CHANGES.md - VERSION - instructions/bootstrap.md - instructions/wiki-ingest/SKILL.md - kb/CONTRACT.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/chemenu/commands/dist_cmd.py - tools/chemenu/commands/docs_verify.py - tools/chemenu/commands/new_page.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/commands/touch.py - tools/chemenu/lint_core.py - tools/chemenu/tests/test_dist_cmd.py - tools/chemenu/tests/test_docs_verify.py - tools/chemenu/tests/test_lint.py - tools/chemenu/tests/test_new_page.py - tools/chemenu/tests/test_provenance.py - tools/chemenu/tests/test_raw_cmd.py - tools/chemenu/tests/test_touch.py - tools/chemenu/tests/test_type_resolver.py - tools/chemenu/type_resolver.py - types/source.md - types/source.schema.yaml
This commit is contained in:
+111
-42
@@ -11,62 +11,87 @@ checked against what was actually said.
|
||||
wikilinks, no provenance and no confidence. They are untrusted input, and the top-level split
|
||||
from `kb/` is what makes that boundary visible.
|
||||
|
||||
## Directory routing
|
||||
## Directory routing: a date shard, not a type
|
||||
|
||||
| Directory | Holds |
|
||||
|-----------|-------|
|
||||
| `articles/` | Web articles and blog posts (keep a local copy even when `source_url:` is set) |
|
||||
| `documents/` | PDFs, specifications, manuals, exported documentation sets |
|
||||
| `notes/` | Personal notes, meeting notes, conversation transcripts |
|
||||
| `assets/` | Images, diagrams, configuration files, and other binaries |
|
||||
`raw/` addresses a file by **when it was accepted**, never by what kind of document it is
|
||||
(Gitea #67). A promotion lands under `raw/<YYYY>/<MM>/`, computed from the calendar month of the
|
||||
`raw accept` call that promoted it - a pure function of something immutable, so it can never
|
||||
rebalance: an overflowing bucket would move files and break every `[^cite-id]` anchor pointing at
|
||||
them, and only a function of a fixed, past fact (the accept date) rules that out entirely. The
|
||||
month it happened is also the one thing about a source that was previously recorded nowhere but
|
||||
`git log`.
|
||||
|
||||
This table is read by `tools/wikitool docs verify` against `dist_cmd.RAW_SUBDIRS`, the single
|
||||
place the four names are declared in code - the two are kept in sync mechanically rather than by
|
||||
convention (Gitea #58).
|
||||
A mechanical shard is **an address, not a claim**. It cannot drift, cannot become wrong, cannot
|
||||
turn into a collection bucket the way a hand-picked type directory did (see below) - which is
|
||||
exactly why it is allowed to live on the total, exclusive surface (the directory), while the
|
||||
semantic classification of what a source *is* moves to the portable one (frontmatter,
|
||||
`source_type:` on the covering source page - see `types/source.md`).
|
||||
|
||||
**Why the type directories this replaced didn't earn their keep.** `articles/`, `documents/`,
|
||||
`notes/` and `assets/` used to route every promotion, and none of the three reasons a directory
|
||||
split is worth its cost ever applied to them: `raw/` is never browsed (access always goes through
|
||||
`raw_files:`, `sources trace`, or `sources coverage` - the browsable surface is the generated
|
||||
`kb/sources/INDEX.md`), the rules in this file never varied per directory, and nothing in them
|
||||
ever decayed at a different rate (all four were equally immutable, never deleted). What the split
|
||||
did cost was real: a human choosing `incoming/notes/` at drop time, then a later pass copying that
|
||||
choice into `source_type:` by hand - which is exactly how a bias took hold. `raw/notes/` held
|
||||
"personal notes, meeting notes, conversation transcripts" by this file's old wording; of the 25
|
||||
files that landed there, 16 turned out to be transcripts, 4 LLM analyses, 2 tracker exports, and
|
||||
only 3 actual notes. The catch-all formed in the raw layer and was carried straight into `kb/`.
|
||||
|
||||
**Pre-#67 files are not moved.** `raw/`'s directory layout was never versioned anywhere -
|
||||
`.wikitool-kb.json` describes `kb/`'s shape, and `corpus_diff.py` compares `raw_files:` as a
|
||||
*value*, never as directory structure - so there is no "two corpus forms at once" to reconcile,
|
||||
only a shape that was simply never described. `raw/articles/`, `raw/documents/`, `raw/notes/` and
|
||||
`raw/assets/` keep holding whatever they already held, indefinitely: `raw accept --replaces`
|
||||
writes back to a file's existing location (the path is the identifier, Gitea #64), so a legacy
|
||||
directory stays a valid promotion target as long as anything still lives there. `sources coverage`
|
||||
walks `raw/` recursively and works unchanged either way.
|
||||
|
||||
## Getting a file in: `incoming/`
|
||||
|
||||
`raw/` is never chosen by hand. A file to be ingested is dropped into the gitignored top-level
|
||||
`incoming/`, under the subdirectory naming its type - `incoming/articles/`, `incoming/documents/`,
|
||||
`incoming/notes/`, `incoming/assets/`, mirroring the table above. That placement is the only
|
||||
classification a human makes: **what kind of document this is**, not where it ends up on disk.
|
||||
Everything past it - the target directory, whether a bundle directory is needed, its name, the
|
||||
move itself - is computed by `tools/wikitool raw accept`.
|
||||
`incoming/` - flat, since Gitea #67: no subdirectory carries any classification any more. A
|
||||
subdirectory is still tolerated if one is used out of habit or by an older script (so an upgrade
|
||||
never has to touch a caller), but it is **ignored**, never inspected:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept incoming/documents/handbuch.pdf
|
||||
# -> raw/documents/handbuch.pdf
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting incoming/handbuch.pdf
|
||||
# -> raw/2026/09/handbuch.pdf
|
||||
```
|
||||
|
||||
**A bundle directory is created only from the second file onward.** One file promoted alone needs
|
||||
no directory of its own and lands as `raw/<type>/<name>`; promoting several files of one source in
|
||||
the same call nests them under `raw/<type>/<stem>/`, named after the first file's stem:
|
||||
no directory of its own and lands as `raw/<YYYY>/<MM>/<name>`; promoting several files of one
|
||||
source in the same call nests them under `raw/<YYYY>/<MM>/<stem>/`, named after the first file's
|
||||
stem:
|
||||
|
||||
```bash
|
||||
tools/wikitool raw accept incoming/documents/handbuch.pdf incoming/documents/handbuch.md
|
||||
# -> raw/documents/handbuch/handbuch.pdf
|
||||
# -> raw/documents/handbuch/handbuch.md
|
||||
tools/wikitool raw accept --fidelity verbatim --authority reporting \
|
||||
incoming/handbuch.pdf incoming/handbuch.md
|
||||
# -> raw/2026/09/handbuch/handbuch.pdf
|
||||
# -> raw/2026/09/handbuch/handbuch.md
|
||||
```
|
||||
|
||||
This is why the 29 files already in `raw/` sit directly under their type directory rather than
|
||||
each in its own bundle: an eindateiige Quelle is already in the form the rule produces, not an
|
||||
exception to it - nothing was reorganised to reach this state.
|
||||
**Growing an existing single file into a bundle forms it at that file's own location, never at
|
||||
today's shard.** `raw accept --page "Source - X" ...` extends an existing source page's
|
||||
`raw_files:` in the same call; if that raises the page past one file, its already-promoted file is
|
||||
folded into the new bundle alongside the one(s) just accepted, at `<its-existing-parent>/<stem>/` -
|
||||
the file that started single does not stay single once a second one belongs beside it, but its
|
||||
capture date is whatever it always was, and a bundle mixing an old and a new shard would have no
|
||||
single correct address.
|
||||
|
||||
A bundle's type directory is the **source's** type, not any one file's - a diagram that belongs to
|
||||
a `documents/` source is promoted from `incoming/documents/`, not `incoming/assets/`; `assets/` is
|
||||
for a source that is itself an asset.
|
||||
|
||||
**The names occupied at `raw/<type>/` level - file stems and bundle directory names alike - are
|
||||
unique** (Gitea #64). Within a bundle, `handbuch.pdf` and `handbuch.md` sit side by side as always;
|
||||
the rule bites one level up, so a second, unrelated source cannot promote quietly into a bundle it
|
||||
does not belong to just because its own filename happens not to collide. A promote whose target
|
||||
name is already occupied is refused, naming both sanctioned ways past it without recommending
|
||||
either:
|
||||
**The names occupied anywhere under `raw/` - file stems and bundle directory names alike - are
|
||||
unique** (Gitea #64, widened globally by #67 once type directories stopped bounding the check).
|
||||
Within a bundle, `handbuch.pdf` and `handbuch.md` sit side by side as always; the rule bites one
|
||||
level up, so a second, unrelated source cannot promote quietly into a bundle it does not belong to
|
||||
just because its own filename happens not to collide - nor into a same-named bundle sitting in a
|
||||
different shard, or in a pre-#67 type directory. A promote whose target name is already occupied
|
||||
is refused, naming both sanctioned ways past it without recommending either:
|
||||
|
||||
```
|
||||
ERROR raw/documents/cluster.md already claims the stem "cluster" in raw/documents/.
|
||||
ERROR raw/documents/cluster.md already claims the stem "cluster" under raw/.
|
||||
These are two different intents and only you can tell them apart:
|
||||
Same source, new edition -> tools/wikitool raw accept --replaces raw/documents/cluster.md incoming/documents/cluster.md
|
||||
Same source, new edition -> tools/wikitool raw accept --replaces raw/documents/cluster.md incoming/cluster.md
|
||||
A second, separate source -> rename it in incoming/ (cluster-netzplan.md, cluster-2026-09.md) and accept it normally
|
||||
raw accept does not guess which one this is.
|
||||
```
|
||||
@@ -76,16 +101,60 @@ to the human and waits, the same way it would for an exit-42 gate (AGENTS.md inv
|
||||
though no gate fires here: the tool cannot ask the question itself, so the session passes it on
|
||||
instead of answering it.
|
||||
|
||||
`tools/wikitool raw accept --page "Source - X" ...` additionally extends an existing source page's
|
||||
`raw_files:` in the same call. If that raises the page past one file, its already-promoted file is
|
||||
folded into the new bundle alongside the one(s) just accepted - the file that started single does
|
||||
not stay single once a second one belongs beside it.
|
||||
|
||||
`incoming/` is read by an ingest session, never by `sources coverage` or `lint`: both walk `raw/`
|
||||
only, so a file waiting there is not yet a finding. It is also never committed - proven, not
|
||||
merely asserted, by `docs verify`'s ignore-rule canaries - which is what makes accepting a file the
|
||||
moment its immutability under the rules below begins, not the moment it was dropped.
|
||||
|
||||
## Capture fields: `fidelity` and `authority`
|
||||
|
||||
Two things are knowable at the moment a file is accepted and at no point afterwards: **how
|
||||
faithful the capture is** to what was actually said or shown, and **what the material is entitled
|
||||
to claim** about its subject. A model's own analysis of a system can be guessed at months later;
|
||||
whether the sender of an archived thread was actually in a position to speak for its subject
|
||||
cannot. Both are recorded once, on the covering source page (`types/source.md`), as **capture
|
||||
fields** - fixed at capture time, never freely re-editable afterwards.
|
||||
|
||||
| Field | Question | Values |
|
||||
|---|---|---|
|
||||
| `fidelity` | How faithful is the *capture*? | `verbatim`, `published`, `secondhand`, `nontextual`, (`unknown`) |
|
||||
| `authority` | What may the material claim about its *subject*? | `normative`, `reporting`, `opinion`, (`unknown`) |
|
||||
|
||||
The two move independently: a chat transcript is `verbatim` + `reporting`; an LLM's own analysis
|
||||
of the same subject is `secondhand` + `opinion`; official system documentation and a web article
|
||||
about the same system are both `published`, but `normative` against `reporting`.
|
||||
|
||||
Both are **required, with no default**, at the point a source page first exists - the same
|
||||
no-guessing posture `source_type:` has (Gitea #66), but without that field's escape hatch: there
|
||||
is no `unclassified` catalog slot for a capture field, because a guessed value here would not read
|
||||
as "unknown", it would read as a claim about the capture that cannot be corrected later (the
|
||||
knowledge exists only at the drop point). `raw accept --fidelity <value> --authority <value>`
|
||||
refuses without both; if the call also carries `--page`, both are written straight onto that page.
|
||||
Without `--page` there is no page yet to write them onto - `wiki-ingest` creates the source page
|
||||
afterwards - so `raw accept` instead prints the exact follow-up line, and `wikitool new source`
|
||||
itself refuses to scaffold a source page without both:
|
||||
|
||||
```
|
||||
OK Promoted 1 file(s) to raw/2026/09/handbuch.pdf.
|
||||
Next:
|
||||
tools/wikitool new source --name "<Title>" \
|
||||
--set raw_files=raw/2026/09/handbuch.pdf \
|
||||
--set fidelity=verbatim --set authority=reporting \
|
||||
--set source_type=<category>
|
||||
```
|
||||
|
||||
**Fixed once, correctable only as a new edition.** `wikitool touch --set fidelity=<value>` writes
|
||||
a capture field only while it is absent; once set, it refuses and points at the one sanctioned way
|
||||
to correct it - `raw accept --replaces`, which alone may pass `--fidelity`/`--authority` to
|
||||
overwrite an already-set value, because a corrected capture *is* a new edition of the source, not
|
||||
an edit of the page describing it.
|
||||
|
||||
**`unknown` is backfill-only.** Neither `raw accept` nor `new source` may ever write it - only
|
||||
`wikitool touch`, on a page that predates this rule (the same construction `source_language`
|
||||
already has: "absent on pages predating the rule"). A capture value written as `unknown` by the
|
||||
tool that captures it would not be an honest "we don't know", it would be indistinguishable from a
|
||||
value nobody ever thought about.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Immutable.** Never edit, reformat, summarize, or "clean up" a file after it lands here.
|
||||
|
||||
Reference in New Issue
Block a user