feat: raw accept - incoming/ als abgeleiteter Rohablage-Eingang (schliesst #58)
CI / verify (push) Successful in 56s
Release / release (push) Successful in 36s

Files changed:
- .gitignore
- CHANGES.md
- README.md
- VERSION
- docs/pipeline-rationale.md
- instructions/bootstrap.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/cli.py
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_docs_verify.py
- tools/chemenu/tests/test_raw_cmd.py
This commit is contained in:
2026-09-05 07:43:43 +02:00
parent 1f0ad7f9f3
commit 36d2128f29
16 changed files with 753 additions and 46 deletions
+9
View File
@@ -134,6 +134,15 @@ npm-debug.log*
/.agents/skills/ /.agents/skills/
/.claude/skills/ /.claude/skills/
# Ingest inbox (see raw/CONTRACT.md and Gitea #58). A human drops a file here
# under its declared type subdirectory; `wikitool raw accept` promotes it into
# `raw/`, computing the directory and any bundle from where it sits and how
# many files move together. Unlike raw/ itself this must NEVER be committed -
# the promotion is what makes a file immutable, not the drop - so this is the
# one directory pattern in this file that is deliberately not anchored back
# open by the content backstop below.
/incoming/
# Content backstop - keep this block last. Nothing under raw/, kb/ or work/ may # Content backstop - keep this block last. Nothing under raw/, kb/ or work/ may
# be excluded by a pattern above; see the header note for why directory patterns # be excluded by a pattern above; see the header note for why directory patterns
# still have to be anchored rather than relying on these negations. # still have to be anchored rather than relying on these negations.
+42 -1
View File
@@ -35,7 +35,7 @@ dev-checkout concern - readable here, never shipped as something to parse.
--- ---
## 4.8.0-beta.2 - 2026-09-04 - kb/CONTRACT.md: Tiefe 1 als Grenze - Katalog liest nur eine Area-Ebene ## 4.8.0-beta.3 - 2026-09-05 - raw accept: incoming/ als abgeleiteter Rohablage-Eingang (schliesst #58)
**Author:** Torben Nehmer **Author:** Torben Nehmer
@@ -43,6 +43,7 @@ dev-checkout concern - readable here, never shipped as something to parse.
- status/incoming: menschliche Stubs werden ausgearbeitet, nie so umgesetzt - status/incoming: menschliche Stubs werden ausgearbeitet, nie so umgesetzt
- page move: eine kb-Seite folgt ihrem Subtype ins Verzeichnis, das ihr Type-Spec berechnet - page move: eine kb-Seite folgt ihrem Subtype ins Verzeichnis, das ihr Type-Spec berechnet
- kb/CONTRACT.md: Tiefe 1 als Grenze - Katalog liest nur eine Area-Ebene - kb/CONTRACT.md: Tiefe 1 als Grenze - Katalog liest nur eine Area-Ebene
- raw accept: incoming/ als abgeleiteter Rohablage-Eingang (schliesst #58)
<!-- /wikitool:bumps --> <!-- /wikitool:bumps -->
Das Label `status/incoming` gibt es seit heute in Gitea: der Mensch legt einen Das Label `status/incoming` gibt es seit heute in Gitea: der Mensch legt einen
@@ -217,6 +218,46 @@ ausgelieferter Inhalt.
Schließt #57. Schließt #57.
**`raw accept`** (#58): `raw/CONTRACT.md`s Routing-Tabelle war bislang eine Regel für Menschen —
wer eine Datei ablegt, wählt `articles/`/`documents/`/`notes/`/`assets/` selbst, und mehrere
Dateien einer logischen Quelle waren im Dateisystem nicht als zusammengehörig erkennbar. Neu ist
ein gitignorierter Eingang `incoming/`, der dieselben vier Typverzeichnisse spiegelt: der Mensch
klassifiziert nur, indem er dort ablegt, `tools/wikitool raw accept <datei> [<datei> ...]
[--page "<Titel>"]` berechnet die Beförderung nach `raw/`.
Zwei Entscheidungen, gegen die ursprüngliche Skizze im Issue: ein Bundle-Verzeichnis
(`raw/<typ>/<stamm>/`, benannt nach der ersten Datei) entsteht erst ab der zweiten Datei, nie
einheitlich — damit sind die 29 heute flach liegenden Bestandsdateien keine Ausnahme, sondern
bereits die Regelform, und die Frage „was passiert mit dem Bestand" beantwortet sich von selbst.
Und der Typ wird über das Eingangs-Unterverzeichnis deklariert, nicht über ein `--type`-Flag: die
Erklärung wird abgegeben, wenn der Mensch die Datei in der Hand hat, statt im Moment des
`accept`-Aufrufs neu geraten werden zu müssen.
`--page` deckt den Wachstumsfall ab: erweitert `raw_files:` einer bestehenden Source-Seite und
faltet deren schon abgelegte Einzeldatei ins neue Bundle, sobald das die Seite über eine Datei
hinaus wachsen lässt — ohne ein Fenster, in dem `raw_files:` ins Leere zeigt. Die dafür nötige
Rückwärtssuche und der Mehrfach-Owner-Schutz sind keine neue Mechanik, sondern
`provenance.source_pages_by_raw_file`, das `lint` schon für `duplicate_raw_file_owners` benutzt —
ein Owner-Konflikt lehnt die Beförderung ab, statt eine andere Seite unbemerkt zu brechen.
`incoming/` ist für `sources coverage` und `lint` unsichtbar (beide laufen ausschließlich über
`config.iter_raw_files(config.RAW_DIR)`), und dass keine Datei dort je committet werden kann, ist
über `docs_verify.REQUIRED_IGNORE_CANARIES` bewiesen, nicht nur zugesichert. `RAW_SUBDIRS`
(`dist_cmd.py`) bleibt die einzige Quelle der Vier-Verzeichnis-Liste: `docs verify`
(`check_raw_subdirs`) hält `raw/CONTRACT.md`s Tabelle jetzt in beiden Richtungen dagegen, und
`dist export` sät `incoming/<typ>/.gitkeep` neben `raw/<typ>/.gitkeep`; `instructions/bootstrap.md`
legt den Eingang für einen bestehenden Klon nach, da er dort nie aus git kommt.
Geändert: `tools/chemenu/commands/raw_cmd.py` (neu, `raw accept`), `tools/chemenu/cli.py`,
`tools/chemenu/commands/dist_cmd.py` (`RAW_SUBDIRS`-Kommentar, `incoming/*/.gitkeep`),
`tools/chemenu/commands/docs_verify.py` (`check_raw_subdirs`, `incoming/`-Ignore-Kanarie),
`.gitignore`, `raw/CONTRACT.md`, `tools/CONTRACT.md`, `instructions/bootstrap.md`,
`instructions/wiki-ingest/SKILL.md`. MINOR: eine Umsortierung des Bestands wäre die Grenze
gewesen, findet aber unter der Bundle-erst-ab-zwei-Regel nicht statt — der Bestand bleibt
unangetastet, keine fremde Instanz muss migrieren, vorwärts wie rückwärts reines Überkopieren.
Schließt #58.
--- ---
## 4.7.4 - 2026-09-04 - bootstrap.md nennt den session-id-WARN nach frischem Bootstrap explizit als erwartet ## 4.7.4 - 2026-09-04 - bootstrap.md nennt den session-id-WARN nach frischem Bootstrap explizit als erwartet
+19 -9
View File
@@ -61,7 +61,8 @@ chemenu/
├── SOUL.md # How this instance sounds. AGENTS.md always wins over it ├── SOUL.md # How this instance sounds. AGENTS.md always wins over it
├── ENVIRONMENT.md # Optional, gitignored: this checkout's harness, MCP servers, remotes ├── ENVIRONMENT.md # Optional, gitignored: this checkout's harness, MCP servers, remotes
├── *.md.template # Unfilled USER/SOUL/ENVIRONMENT - what a distribution ships instead ├── *.md.template # Unfilled USER/SOUL/ENVIRONMENT - what a distribution ships instead
├── .gitignore # Anchored so nothing under raw/, kb/ or work/ is ever excluded ├── .gitignore # Anchored so nothing under raw/, kb/ or work/ is ever excluded;
│ # incoming/ is the one directory excluded the other way round
├── .github/hooks/ # Copilot CLI hooks - session tracing ├── .github/hooks/ # Copilot CLI hooks - session tracing
├── .vibe/ # Mistral Vibe hooks + the repo's telemetry policy ├── .vibe/ # Mistral Vibe hooks + the repo's telemetry policy
├── instructions/ # CONTROL: everything an agent is told to do ├── instructions/ # CONTROL: everything an agent is told to do
@@ -74,8 +75,13 @@ chemenu/
│ ├── publish-cycle.md │ ├── publish-cycle.md
│ ├── ingest-large-tree.md │ ├── ingest-large-tree.md
│ └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/ │ └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/
├── incoming/ # INBOX: gitignored - drop a file here, `raw accept` promotes it
│ ├── articles/ # Same four type directories as raw/, mirrored
│ ├── documents/
│ ├── notes/
│ └── assets/
├── raw/ # INPUT: immutable, untrusted source material ├── raw/ # INPUT: immutable, untrusted source material
│ ├── CONTRACT.md # Routing, immutability, untrusted content │ ├── CONTRACT.md # Routing, immutability, untrusted content, incoming/
│ ├── articles/ # Web articles, blog posts │ ├── articles/ # Web articles, blog posts
│ ├── documents/ # PDFs, specs, manuals │ ├── documents/ # PDFs, specs, manuals
│ ├── notes/ # Personal notes, transcriptions │ ├── notes/ # Personal notes, transcriptions
@@ -133,9 +139,13 @@ working *on* that layer, the contract is what binds an agent working *with* it.
### Adding Knowledge (Ingest) ### Adding Knowledge (Ingest)
1. Drop a file into `raw/` (articles, documents, notes, or assets) 1. Drop a file into `incoming/<type>/` (`articles`, `documents`, `notes`, or `assets`) -
2. Tell the LLM: `Ingest raw/articles/my-article.md` the directory you choose is the only classification you make; everything past it
(the destination in `raw/`, whether several files of one source get bundled) is
computed by `tools/wikitool raw accept`, never chosen by hand
2. Tell the LLM: `Ingest incoming/articles/my-article.md`
3. The LLM will: 3. The LLM will:
- Promote it into `raw/` with `raw accept`
- Read and summarize the source - Read and summarize the source
- Create a source page in `kb/sources/` - Create a source page in `kb/sources/`
- Create or update relevant entity pages - Create or update relevant entity pages
@@ -186,7 +196,7 @@ tools/wikitool types describe entity
### For You (Human) ### For You (Human)
1. **Curate sources** - Add files to `raw/` that you want processed 1. **Curate sources** - Drop files you want processed into `incoming/<type>/`
2. **Ask questions** - Query the wiki naturally 2. **Ask questions** - Query the wiki naturally
3. **Review changes** - Check `kb/log.md` and `kb/index.md` 3. **Review changes** - Check `kb/log.md` and `kb/index.md`
4. **Direct the LLM** - Guide it on what to emphasize or investigate 4. **Direct the LLM** - Guide it on what to emphasize or investigate
@@ -200,7 +210,7 @@ themselves live as independently-discoverable skills under `.agents/skills/`
| Skill | Purpose | | Skill | Purpose |
|-------|---------| |-------|---------|
| `wiki-ingest` | Process a new `raw/` source into the wiki: source summary, entity/concept pages, cross-references, index/log, publish | | `wiki-ingest` | Promote a new source from `incoming/` into `raw/`, then process it into the wiki: source summary, entity/concept pages, cross-references, index/log, publish |
| `wiki-query` | Answer a question from the compiled wiki; read-only, can optionally file a valuable answer back as a new page | | `wiki-query` | Answer a question from the compiled wiki; read-only, can optionally file a valuable answer back as a new page |
| `wiki-lint` | Health-check the wiki: structural scan, raw coverage, semantic review, confidence decay | | `wiki-lint` | Health-check the wiki: structural scan, raw coverage, semantic review, confidence decay |
| `wiki-manage` | Create a new entity/concept/source/comparison page, or update an existing page with new information | | `wiki-manage` | Create a new entity/concept/source/comparison page, or update an existing page with new information |
@@ -216,7 +226,7 @@ decay math, publishing) is delegated to `tools/wikitool` - never hand-edited.
1. Read `AGENTS.md` - the control plane (invariants, routing, gates) - then the stage contract 1. Read `AGENTS.md` - the control plane (invariants, routing, gates) - then the stage contract
for whichever of `raw/`, `types/` or `kb/` you are working in, and, inside `kb/`, the for whichever of `raw/`, `types/` or `kb/` you are working in, and, inside `kb/`, the
`COLLECTION.md` of the collection you are writing to `COLLECTION.md` of the collection you are writing to
2. Add your first source to `raw/` 2. Add your first source to `incoming/<type>/`
3. Run: `Ingest <your-file>` 3. Run: `Ingest <your-file>`
4. Review the created pages 4. Review the created pages
5. Ask your first query 5. Ask your first query
@@ -225,11 +235,11 @@ decay math, publishing) is delegated to `tools/wikitool` - never hand-edited.
```bash ```bash
# Add a source # Add a source
cp ~/Downloads/my-notes.md raw/notes/my-notes.md cp ~/Downloads/my-notes.md incoming/notes/my-notes.md
# Tell the LLM to process it # Tell the LLM to process it
# (in your LLM agent) # (in your LLM agent)
Ingest raw/notes/my-notes.md Ingest incoming/notes/my-notes.md
``` ```
## Tips ## Tips
+1 -1
View File
@@ -1 +1 @@
4.8.0-beta.2 4.8.0-beta.3
+4
View File
@@ -14,6 +14,10 @@ thing later claims get checked against is no longer the thing that was actually
immutable `raw/` means a citation always resolves to the original, not to somebody's tidied immutable `raw/` means a citation always resolves to the original, not to somebody's tidied
memory of it. It also draws a trust boundary in one place instead of scattering it - everything memory of it. It also draws a trust boundary in one place instead of scattering it - everything
past `raw/` can be treated as reviewed, because nothing upstream of it silently already was. past `raw/` can be treated as reviewed, because nothing upstream of it silently already was.
`incoming/` (Gitea #58) sits entirely on the near side of that boundary: a file waiting there is
not yet reviewed and not yet a citation target, so its being gitignored and readable by an
ingest session does not weaken anything - the boundary is the promotion into `raw/` itself, not
the moment a human happened to drop a file somewhere.
## Why extraction happens once, through a schema ## Why extraction happens once, through a schema
+18 -7
View File
@@ -28,13 +28,24 @@ they are published: the agent harness will not offer `wiki-ingest`, `wiki-query`
cd .. cd ..
``` ```
2. **Publish the skills:** 2. **Create the ingest inbox.** `incoming/` (raw/CONTRACT.md "Getting a file in") is gitignored,
so a fresh clone has none of its type subdirectories - unlike `raw/` itself, which is
committed and present immediately:
```bash
mkdir -p incoming/{articles,documents,notes,assets}
```
`tools/wikitool doctor` only reports a missing one; it never creates it, so this is a one-off
catch-up here the same way step 5 below is for personalization.
3. **Publish the skills:**
```bash ```bash
tools/wikitool instructions sync tools/wikitool instructions sync
``` ```
3. **Verify:** 4. **Verify:**
```bash ```bash
tools/wikitool instructions verify tools/wikitool instructions verify
@@ -43,14 +54,14 @@ they are published: the agent harness will not offer `wiki-ingest`, `wiki-query`
Expected: `OK`. If it reports drift, re-run `sync` - the source under `instructions/` always Expected: `OK`. If it reports drift, re-run `sync` - the source under `instructions/` always
wins, and a copy is never edited directly. wins, and a copy is never edited directly.
4. **Check for personalization.** A clone predating the personalization files has no 5. **Check for personalization.** A clone predating the personalization files has no
`USER.md`/`SOUL.md`, and `tools/wikitool doctor` reports `personalization: FAIL` for it. `USER.md`/`SOUL.md`, and `tools/wikitool doctor` reports `personalization: FAIL` for it.
That is a one-off catch-up, not a bootstrap step that repeats: run **only** the That is a one-off catch-up, not a bootstrap step that repeats: run **only** the
Personalization step (6) of [setup-instance.md](setup-instance.md), not the whole Personalization step (6) of [setup-instance.md](setup-instance.md), not the whole
procedure - this clone already has its git repo, author identity and content. A clone that procedure - this clone already has its git repo, author identity and content. A clone that
already carries both files needs nothing here. already carries both files needs nothing here.
5. **Offer to record the environment.** `ENVIRONMENT.md` is gitignored, so a fresh clone never 6. **Offer to record the environment.** `ENVIRONMENT.md` is gitignored, so a fresh clone never
has one, and every session in it re-asks which harness is in use, which MCP servers are has one, and every session in it re-asks which harness is in use, which MCP servers are
reachable, and which remote `publish` talks to. Copy `ENVIRONMENT.md.template` to reachable, and which remote `publish` talks to. Copy `ENVIRONMENT.md.template` to
`ENVIRONMENT.md`, fill in what is already known from this clone (`git remote -v`, the `ENVIRONMENT.md`, fill in what is already known from this clone (`git remote -v`, the
@@ -62,15 +73,15 @@ they are published: the agent harness will not offer `wiki-ingest`, `wiki-query`
session pays for it again. Never guess an entry: a wrong remote or an MCP server that is not session pays for it again. Never guess an entry: a wrong remote or an MCP server that is not
there is worse than the empty section it replaced, because it gets believed. there is worse than the empty section it replaced, because it gets believed.
6. **Restart the agent session** if it was already running. Harnesses read the skill 7. **Restart the agent session** if it was already running. Harnesses read the skill
directories at startup, so skills published mid-session are not picked up. directories at startup, so skills published mid-session are not picked up.
7. **Expect a lingering `session-id` WARN.** A `tools/wikitool doctor` run at this point reports 8. **Expect a lingering `session-id` WARN.** A `tools/wikitool doctor` run at this point reports
`OK` throughout except `session-id: WARN` - that check is scoped to the working session, not `OK` throughout except `session-id: WARN` - that check is scoped to the working session, not
the clone, so a freshly bootstrapped checkout with no `WIKITOOL_SESSION_ID` exported yet the clone, so a freshly bootstrapped checkout with no `WIKITOOL_SESSION_ID` exported yet
always shows it. This is expected, not a Bootstrap gap: exporting it here would only be true always shows it. This is expected, not a Bootstrap gap: exporting it here would only be true
for this one-off setup run, not for whichever session picks up the actual work next, in a new for this one-off setup run, not for whichever session picks up the actual work next, in a new
shell after step 6's restart. Run [session-setup.md](session-setup.md) at the start of that shell after step 7's restart. Run [session-setup.md](session-setup.md) at the start of that
session instead. session instead.
## Scope ## Scope
+38 -26
View File
@@ -1,13 +1,14 @@
--- ---
name: wiki-ingest name: wiki-ingest
description: Process a new source file into the LLM wiki - extract entities and concepts, create a source summary page, cross-reference, rebuild indexes, and publish. Use when the user drops a file into raw/ or says "ingest <file>", "process this source", "add this to the wiki". description: Process a new source file into the LLM wiki - extract entities and concepts, create a source summary page, cross-reference, rebuild indexes, and publish. Use when the user drops a file into incoming/ or raw/, or says "ingest <file>", "process this source", "add this to the wiki".
--- ---
# Wiki Ingest # Wiki Ingest
**Purpose:** Process a new source file and integrate its knowledge into the wiki. **Purpose:** Process a new source file and integrate its knowledge into the wiki.
**Trigger:** User drops a file into `raw/` or explicitly requests ingestion. **Trigger:** User drops a file into `incoming/` (the normal path - see step 1) or directly into
`raw/`, or explicitly requests ingestion.
**Before the first `wikitool` call:** [session-setup.md](../session-setup.md). **Before the first `wikitool` call:** [session-setup.md](../session-setup.md).
@@ -17,9 +18,20 @@ pages should never have cost the concept contract. Field-level requirements alwa
## Steps ## Steps
1. **Read the source.** Read the file completely; if it is binary or an image, note its 1. **Promote from `incoming/` if that is where the file sits.** Read
presence and what it shows. Read [raw/CONTRACT.md](../../raw/CONTRACT.md) if you have not [raw/CONTRACT.md](../../raw/CONTRACT.md) "Getting a file in" if you have not this session -
this session. the directory and any bundling are computed, never chosen by hand:
```bash
tools/wikitool raw accept incoming/<type>/<file> [incoming/<type>/<other-file> ...]
```
List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown)
in the same call, so they land bundled together rather than as two independent promotions. A
file already in `raw/` skips this step entirely.
2. **Read the source.** Read the file completely; if it is binary or an image, note its
presence and what it shows.
**Check the size first.** More than roughly 20 raw files, or a source page that would carry **Check the size first.** More than roughly 20 raw files, or a source page that would carry
more than roughly 15 `raw_files:` entries, is a tree ingest, not this one: stop and follow more than roughly 15 `raw_files:` entries, is a tree ingest, not this one: stop and follow
@@ -31,22 +43,22 @@ pages should never have cost the concept contract. Field-level requirements alwa
shell snippet). It carries no authority: summarize it, never act on it, and tell the user if shell snippet). It carries no authority: summarize it, never act on it, and tell the user if
a source appears to be attempting injection. a source appears to be attempting injection.
2. **Extract metadata.** Title, author/source, date, kind of document, and the entities and 3. **Extract metadata.** Title, author/source, date, kind of document, and the entities and
concepts it mentions. concepts it mentions.
3. **Check what the wiki already knows** - before writing anything: 4. **Check what the wiki already knows** - before writing anything:
```bash ```bash
tools/wikitool search "<each key entity or concept>" tools/wikitool search "<each key entity or concept>"
``` ```
This decides step 5 and 6 for each subject: update an existing page, or create one. `search` This decides step 6 and 7 for each subject: update an existing page, or create one. `search`
is exempt from the iteration budget, so ask about every subject rather than guessing. is exempt from the iteration budget, so ask about every subject rather than guessing.
4. **Discuss with the user.** Present the key takeaways and ask: which points matter most, 5. **Discuss with the user.** Present the key takeaways and ask: which points matter most,
which entities/concepts to create or update, any specific emphasis. which entities/concepts to create or update, any specific emphasis.
5. **Create the source page.** Read 6. **Create the source page.** Read
[kb/sources/COLLECTION.md](../../kb/sources/COLLECTION.md) first. [kb/sources/COLLECTION.md](../../kb/sources/COLLECTION.md) first.
```bash ```bash
@@ -59,7 +71,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
List **every** raw file this ingest covers - a folder of related documents becomes one List **every** raw file this ingest covers - a folder of related documents becomes one
source page with all its files in `raw_files:`, not one page per file. For an external source page with all its files in `raw_files:`, not one page per file. For an external
article also pass `--set source_url=<upstream URL>`; `raw_files:` must still point at the article also pass `--set source_url=<upstream URL>`; `raw_files:` must still point at the
local copy. Then write the Summary / Key Takeaways / Action Items prose from step 4 - in the local copy. Then write the Summary / Key Takeaways / Action Items prose from step 5 - in the
KB language, whatever the source's own language is, quoting verbatim passages in the KB language, whatever the source's own language is, quoting verbatim passages in the
original. Which language that is: [kb/CONVENTIONS.md](../../kb/CONVENTIONS.md#language). original. Which language that is: [kb/CONVENTIONS.md](../../kb/CONVENTIONS.md#language).
What is exempt from it, in any language: What is exempt from it, in any language:
@@ -69,7 +81,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
with the reason. Nothing in the repository can re-derive that judgment, and without it the with the reason. Nothing in the repository can re-derive that judgment, and without it the
same source gets re-litigated on the next pass. same source gets re-litigated on the next pass.
6. **Create or update entity pages.** Read 7. **Create or update entity pages.** Read
[kb/entities/COLLECTION.md](../../kb/entities/COLLECTION.md) and [kb/entities/COLLECTION.md](../../kb/entities/COLLECTION.md) and
[kb/CONTRACT.md](../../kb/CONTRACT.md) plus [kb/CONTRACT.md](../../kb/CONTRACT.md) plus
[kb/CONVENTIONS.md](../../kb/CONVENTIONS.md) first - the second is where provenance and [kb/CONVENTIONS.md](../../kb/CONVENTIONS.md) first - the second is where provenance and
@@ -98,7 +110,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
`[^cite-id]`, upserts its Footnotes definition, and adds the source to `sources:`; paste the `[^cite-id]`, upserts its Footnotes definition, and adds the source to `sources:`; paste the
marker it prints at the fact. marker it prints at the fact.
7. **Create or update concept pages** - only if the source produced any. Same pattern, reading 8. **Create or update concept pages** - only if the source produced any. Same pattern, reading
[kb/concepts/COLLECTION.md](../../kb/concepts/COLLECTION.md) first: [kb/concepts/COLLECTION.md](../../kb/concepts/COLLECTION.md) first:
```bash ```bash
@@ -106,7 +118,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
--set concept_type=<architecture|pattern|protocol|workflow|decision|problem> --set concept_type=<architecture|pattern|protocol|workflow|decision|problem>
``` ```
8. **Cross-reference.** 9. **Cross-reference.**
```bash ```bash
tools/wikitool xref add --a "<A>" --b "<B>" --rel-a "<label>" --rel-b "<label>" tools/wikitool xref add --a "<A>" --b "<B>" --rel-a "<label>" --rel-b "<label>"
@@ -115,19 +127,19 @@ pages should never have cost the concept contract. Field-level requirements alwa
The second links the new source to everything it backs in one pass. The second links the new source to everything it backs in one pass.
9. **Check coverage.** 10. **Check coverage.**
```bash ```bash
tools/wikitool sources coverage tools/wikitool sources coverage
``` ```
The new raw file(s) must no longer be listed as uncovered, and no `raw_files:` entry may be The new raw file(s) must no longer be listed as uncovered, and no `raw_files:` entry may be
broken. broken.
10. **Close out.** Follow [publish-cycle.md](../publish-cycle.md) with `--op ingest` and a 11. **Close out.** Follow [publish-cycle.md](../publish-cycle.md) with `--op ingest` and a
message of the form `ingest: <raw path>`. message of the form `ingest: <raw path>`.
11. **Check the lint cadence.** 12. **Check the lint cadence.**
```bash ```bash
tools/wikitool log status tools/wikitool log status
@@ -139,7 +151,7 @@ pages should never have cost the concept contract. Field-level requirements alwa
## Decision points ## Decision points
- **Subject already has a page?** Update it (step 6, `touch`) instead of creating a second one. - **Subject already has a page?** Update it (step 7, `touch`) instead of creating a second one.
Two pages on one subject is the failure this step exists to prevent. Two pages on one subject is the failure this step exists to prevent.
- **No raw file backs a claim you want to write?** Leave it out, or mark the page - **No raw file backs a claim you want to write?** Leave it out, or mark the page
`provenance: mixed` and put it under `## General Guidance (unsourced)`. `provenance: mixed` and put it under `## General Guidance (unsourced)`.
@@ -153,9 +165,9 @@ pages should never have cost the concept contract. Field-level requirements alwa
## wikitool commands used ## wikitool commands used
`search`, `new source`, `new entity`, `new concept`, `touch`, `xref add`, `xref link-source`, `raw accept`, `search`, `new source`, `new entity`, `new concept`, `touch`, `xref add`,
`sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`, `log status`, `xref link-source`, `sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`,
`publish` `log status`, `publish`
## Output ## Output
+46
View File
@@ -20,6 +20,52 @@ from `kb/` is what makes that boundary visible.
| `notes/` | Personal notes, meeting notes, conversation transcripts | | `notes/` | Personal notes, meeting notes, conversation transcripts |
| `assets/` | Images, diagrams, configuration files, and other binaries | | `assets/` | Images, diagrams, configuration files, and other binaries |
This table is read by `tools/wikitool docs verify` against `dist_cmd.RAW_SUBDIRS`, the single
place the four names are declared in code - the two are kept in sync mechanically rather than by
convention (Gitea #58).
## Getting a file in: `incoming/`
`raw/` is never chosen by hand. A file to be ingested is dropped into the gitignored top-level
`incoming/`, under the subdirectory naming its type - `incoming/articles/`, `incoming/documents/`,
`incoming/notes/`, `incoming/assets/`, mirroring the table above. That placement is the only
classification a human makes: **what kind of document this is**, not where it ends up on disk.
Everything past it - the target directory, whether a bundle directory is needed, its name, the
move itself - is computed by `tools/wikitool raw accept`.
```bash
tools/wikitool raw accept incoming/documents/handbuch.pdf
# -> raw/documents/handbuch.pdf
```
**A bundle directory is created only from the second file onward.** One file promoted alone needs
no directory of its own and lands as `raw/<type>/<name>`; promoting several files of one source in
the same call nests them under `raw/<type>/<stem>/`, named after the first file's stem:
```bash
tools/wikitool raw accept incoming/documents/handbuch.pdf incoming/documents/handbuch.md
# -> raw/documents/handbuch/handbuch.pdf
# -> raw/documents/handbuch/handbuch.md
```
This is why the 29 files already in `raw/` sit directly under their type directory rather than
each in its own bundle: an eindateiige Quelle is already in the form the rule produces, not an
exception to it - nothing was reorganised to reach this state.
A bundle's type directory is the **source's** type, not any one file's - a diagram that belongs to
a `documents/` source is promoted from `incoming/documents/`, not `incoming/assets/`; `assets/` is
for a source that is itself an asset.
`tools/wikitool raw accept --page "Source - X" ...` additionally extends an existing source page's
`raw_files:` in the same call. If that raises the page past one file, its already-promoted file is
folded into the new bundle alongside the one(s) just accepted - the file that started single does
not stay single once a second one belongs beside it.
`incoming/` is read by an ingest session, never by `sources coverage` or `lint`: both walk `raw/`
only, so a file waiting there is not yet a finding. It is also never committed - proven, not
merely asserted, by `docs verify`'s ignore-rule canaries - which is what makes accepting a file the
moment its immutability under the rules below begins, not the moment it was dropped.
## Rules ## Rules
- **Immutable.** Never edit, reformat, summarize, or "clean up" a file after it lands here. - **Immutable.** Never edit, reformat, summarize, or "clean up" a file after it lands here.
+7 -2
View File
@@ -58,6 +58,7 @@ tools/wikitool <command> --help
| `sources coverage [--json]` | List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages | | `sources coverage [--json]` | List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages |
| `sources trace --raw <path>` \| `--page "<Title>"` | Trace provenance in either direction: raw file -> source page(s) -> citing pages, or page -> its sources -> their raw files | | `sources trace --raw <path>` \| `--page "<Title>"` | Trace provenance in either direction: raw file -> source page(s) -> citing pages, or page -> its sources -> their raw files |
| `sources rebuild-index [--dry-run]` | Regenerate the `kb/provenance.md` reverse index (raw file -> source page -> citing pages) | | `sources rebuild-index [--dry-run]` | Regenerate the `kb/provenance.md` reverse index (raw file -> source page -> citing pages) |
| `raw accept <file> [<file> ...] [--page "<Title>"] [--dry-run]` | Promote one or more files from `incoming/<type>/` into `raw/<type>/`, computing the destination instead of taking it as an argument (`raw/CONTRACT.md` "Getting a file in", Gitea #58): the type subdirectory comes from where the file sits under `incoming/`, a bundle directory (`raw/<type>/<stem>/`, named after the first file's stem) forms only from the second file on, and one file promoted alone gets none. `--page "<Title>"` additionally extends that existing source page's `raw_files:` in the same call; if that raises the page past one file, its already-promoted file is folded into the new bundle alongside the one(s) just accepted, after checking it has no other owner (`provenance.duplicate_raw_file_owners`) - moving a file another page also claims would break that page's `raw_files:` unconsulted |
| `sync [--remote origin] [--branch main] [--confirm-rebase TOKEN]` | Fetch `<remote>/<branch>` and bring the local branch up to date with it: fast-forward when the remote is simply ahead, rebase local commit(s) on top when both sides moved but touch disjoint files (a content conflict is then impossible by construction), and exit **42** for review when they touch the same file (the **rebase-review gate** - see `publish` below). Never commits, never pushes, never force-anything - no remote configured, or one that cannot be reached, is reported and skipped, not a failure. Meant to run once at the start of a writing session (`instructions/session-setup.md`) so the rest of it works against a current tree instead of discovering the drift at the final `publish` | | `sync [--remote origin] [--branch main] [--confirm-rebase TOKEN]` | Fetch `<remote>/<branch>` and bring the local branch up to date with it: fast-forward when the remote is simply ahead, rebase local commit(s) on top when both sides moved but touch disjoint files (a content conflict is then impossible by construction), and exit **42** for review when they touch the same file (the **rebase-review gate** - see `publish` below). Never commits, never pushes, never force-anything - no remote configured, or one that cannot be reached, is reported and skipped, not a failure. Meant to run once at the start of a writing session (`instructions/session-setup.md`) so the rest of it works against a current tree instead of discovering the drift at the final `publish` |
| `publish --message "<op>: <desc>" [--no-push] [--confirm TOKEN] [--confirm-rebase TOKEN] [--threshold N] [--remote origin] [--branch main] [--path P ...]` | Reconcile with `<remote>/<branch>` exactly like `sync` (skipped for `--no-push`), then stage all changes, commit, and push. If the reconcile step found a still-unpushed local commit and there is nothing new to stage, that commit is pushed anyway - a previous `publish` whose push failed no longer strands it. If the push is rejected despite the pre-check (a genuine race - something landed on the remote in between), one more reconcile-and-retry is attempted before giving up; never more than one. **Mass-Update Gate:** when >= `--threshold` (default 10) *counted* files would be committed, exits **42 (`EXIT_NEEDS_CLEARANCE`)** instead of publishing - a third outcome distinct from success (0) and a validation error (1) - and prints a review report: a scale line (file count, total lines added/removed, status breakdown), only-what-applies attention notes (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn, generated files split out as needing no review. The token digests each counted path **and its contents** plus the publish target, so a clearance carries neither to a different file list nor to edited contents; a wrong, invented or superseded token exits 42 again with the current state. Two kinds of path are committed but never counted and never shown for approval: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md`) - each is recomputable from the tree, so approving it decides nothing, and a routine ingest rebuilds five or six of them. The refusal line accounts for both, by reason. The gate is evaluated *before* anything is staged, so a refused publish leaves the working tree untouched. **Publish-Remote Gate:** when this checkout carries a `.wikitool-remotes.json` and the resolved push URL of `--remote` is not listed in it, exits **42** before the reconcile step even fetches - the URL is read from `git remote get-url --push`, so a repointed remote does not pass on its name. Unlike the other two gates it has **no token and no flag**: the way past it is the user adding the URL to that file, and an agent editing it to get past a refusal is opening a gate on its own initiative. Absent file means unrestricted; a malformed one is an error, not permission. See [instructions/gates.md](../instructions/gates.md) `--yes`/`-y` are gone and now fail with an explicit error. `--path` (repeatable) scopes the whole operation - gate count, staging, and commit - to a subtree. **Stack-machinery note:** after a successful commit/push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md`, or a path ending in `CONTRACT.md` - roughly the scope a stack version bump covers, deliberately a shade broader than CI's version gate, which matches only a `CONTRACT.md` one segment deep - prints one reminder line that the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. Not a gate: no exit code change, nothing to clear, silent for an ordinary content publish | | `publish --message "<op>: <desc>" [--no-push] [--confirm TOKEN] [--confirm-rebase TOKEN] [--threshold N] [--remote origin] [--branch main] [--path P ...]` | Reconcile with `<remote>/<branch>` exactly like `sync` (skipped for `--no-push`), then stage all changes, commit, and push. If the reconcile step found a still-unpushed local commit and there is nothing new to stage, that commit is pushed anyway - a previous `publish` whose push failed no longer strands it. If the push is rejected despite the pre-check (a genuine race - something landed on the remote in between), one more reconcile-and-retry is attempted before giving up; never more than one. **Mass-Update Gate:** when >= `--threshold` (default 10) *counted* files would be committed, exits **42 (`EXIT_NEEDS_CLEARANCE`)** instead of publishing - a third outcome distinct from success (0) and a validation error (1) - and prints a review report: a scale line (file count, total lines added/removed, status breakdown), only-what-applies attention notes (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn, generated files split out as needing no review. The token digests each counted path **and its contents** plus the publish target, so a clearance carries neither to a different file list nor to edited contents; a wrong, invented or superseded token exits 42 again with the current state. Two kinds of path are committed but never counted and never shown for approval: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md`) - each is recomputable from the tree, so approving it decides nothing, and a routine ingest rebuilds five or six of them. The refusal line accounts for both, by reason. The gate is evaluated *before* anything is staged, so a refused publish leaves the working tree untouched. **Publish-Remote Gate:** when this checkout carries a `.wikitool-remotes.json` and the resolved push URL of `--remote` is not listed in it, exits **42** before the reconcile step even fetches - the URL is read from `git remote get-url --push`, so a repointed remote does not pass on its name. Unlike the other two gates it has **no token and no flag**: the way past it is the user adding the URL to that file, and an agent editing it to get past a refusal is opening a gate on its own initiative. Absent file means unrestricted; a malformed one is an error, not permission. See [instructions/gates.md](../instructions/gates.md) `--yes`/`-y` are gone and now fail with an explicit error. `--path` (repeatable) scopes the whole operation - gate count, staging, and commit - to a subtree. **Stack-machinery note:** after a successful commit/push whose changed files include `tools/`, `types/`, `instructions/`, `AGENTS.md`, or a path ending in `CONTRACT.md` - roughly the scope a stack version bump covers, deliberately a shade broader than CI's version gate, which matches only a `CONTRACT.md` one segment deep - prints one reminder line that the phase past this point (an issue-body rewrite, `docs/` staleness, a changelog entry's accuracy) is not covered by `docs verify`, `instructions verify` or `pytest`. Not a gate: no exit code change, nothing to clear, silent for an ordinary content publish |
| `work new (--input <raw path> \| --key <run key>) [--again] [--dry-run]` | Scaffold `work/<runkey>/` for one workshop run: refuses a collision instead of suffixing it, and writes the required `README.md` + `plan.md`. `--input` derives the run key from the path below `raw/` (an ingest); `--key` names it outright for a run with no raw input - a migration or a sweep across `kb/` - and may not start with `ingest-`, which stays reserved for derived keys. Exactly one of the two. `--again` opens a dated second pass over a tree that has itself changed. See [work/CONTRACT.md](../work/CONTRACT.md) | | `work new (--input <raw path> \| --key <run key>) [--again] [--dry-run]` | Scaffold `work/<runkey>/` for one workshop run: refuses a collision instead of suffixing it, and writes the required `README.md` + `plan.md`. `--input` derives the run key from the path below `raw/` (an ingest); `--key` names it outright for a run with no raw input - a migration or a sweep across `kb/` - and may not start with `ingest-`, which stays reserved for derived keys. Exactly one of the two. `--again` opens a dated second pass over a tree that has itself changed. See [work/CONTRACT.md](../work/CONTRACT.md) |
@@ -69,10 +70,13 @@ tools/wikitool <command> --help
| `instructions sync [--force]` | Publish every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and delete published skills whose source is gone. Both targets are gitignored, so a fresh clone runs this once - see `instructions/bootstrap.md`. Re-running is also how a drifted copy is repaired: the source always wins. `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it) | | `instructions sync [--force]` | Publish every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and delete published skills whose source is gone. Both targets are gitignored, so a fresh clone runs this once - see `instructions/bootstrap.md`. Re-running is also how a drifted copy is repaired: the source always wins. `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it) |
| `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout | | `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout |
| `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only | | `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only |
| `docs verify` | Check the docs that mirror the code: every CLI command documented here (and vice versa), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, and the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, everything ignored under `reports/` and the published skill directories). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks | | `docs verify` | Check the docs that mirror the code: every CLI command documented here (and vice versa), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `raw/CONTRACT.md`'s routing table naming the same subdirectories as `dist_cmd.RAW_SUBDIRS`, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, and the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks |
| `eval sessions [--json]` | List the sessions that have a trace under `reports/telemetry/`, most recent first. Read-only and exempt from the Iteration Budget Gate | | `eval sessions [--json]` | List the sessions that have a trace under `reports/telemetry/`, most recent first. Read-only and exempt from the Iteration Budget Gate |
| `eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]` | Score one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see [../EVALS.md](../EVALS.md) | | `eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]` | Score one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see [../EVALS.md](../EVALS.md) |
| `dist export <target> [--dry-run] [--source-repo U] [--source-commit SHA] [--release-url U] [--update-url U]` | Write a contentless, distributable copy of this repo's machinery into an empty `<target>` directory: `AGENTS.md`/`README.md`/`EVALS.md` with any `<!-- dist:strip-start -->...<!-- dist:strip-end -->` region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs and their schemas re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas), empty `raw/{articles,documents,notes,assets}/`, `VERSION`, `USER.md.template`/`SOUL.md.template` plus `kb/CONVENTIONS.md.template` and each collection's contract re-keyed as `kb/<name>/COLLECTION.md.template` (the templates ship; the filled `USER.md`/`SOUL.md`/`kb/CONVENTIONS.md`/`kb/<name>/COLLECTION.md`/`types/<page-type>.md` never do - all of them bind their instance and none are the stack's to decide, and `find_leaks` refuses a plan carrying one), and a generated `.wikitool-release.json` stamp (version, export date, origin, and a sha256 per exported file - the base a later upgrade would compare against). The four origin options only fill stamp fields: `export` never calls git and cannot discover them. Refuses a non-empty target, and a tree with no `VERSION`. See `instructions/setup-instance.md`. One-way: there is no command that reconstructs a distributed instance into a dev instance - work on the stack in the origin repo (or a new dev instance exported from it) instead | | `dist export <target> [--dry-run] [--source-repo U] [--source-commit SHA] [--release-url U] [--update-url U]` | Write a contentless, distributable copy of this repo's machinery into an empty `<target>` directory: `AGENTS.md`/`README.md`/`EVALS.md` with any `<!-- dist:strip-start -->...<!-- dist:strip-end -->` region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs and their schemas re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas), empty `raw/{articles,documents,notes,assets}/` and the
matching `incoming/{articles,documents,notes,assets}/` (gitignored again the moment the export
becomes a git repository, so `instructions/bootstrap.md` re-creates it for a plain clone that
never ran this step), `VERSION`, `USER.md.template`/`SOUL.md.template` plus `kb/CONVENTIONS.md.template` and each collection's contract re-keyed as `kb/<name>/COLLECTION.md.template` (the templates ship; the filled `USER.md`/`SOUL.md`/`kb/CONVENTIONS.md`/`kb/<name>/COLLECTION.md`/`types/<page-type>.md` never do - all of them bind their instance and none are the stack's to decide, and `find_leaks` refuses a plan carrying one), and a generated `.wikitool-release.json` stamp (version, export date, origin, and a sha256 per exported file - the base a later upgrade would compare against). The four origin options only fill stamp fields: `export` never calls git and cannot discover them. Refuses a non-empty target, and a tree with no `VERSION`. See `instructions/setup-instance.md`. One-way: there is no command that reconstructs a distributed instance into a dev instance - work on the stack in the origin repo (or a new dev instance exported from it) instead |
| `dist upgrade <source> [--dry-run] [--keep-local] [--prune] [--pre]` | Apply a stack update `dist export` produced - the write half of `version check`. Never downloads anything: `<source>` is an already-fetched export directory or `.tar.gz` release archive (verified against a sibling `.sha256` if one is present; WARNs, does not block, if it is absent), which must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs. The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/*/.gitkeep` - `chemenu.ownership.is_export_stub`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md` - `chemenu.ownership.is_upgrade_preserved`), plus the stamp itself, always rewritten. Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest for it: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten - the run aborts with the full list unless `--keep-local` says to proceed and leave every one of them untouched. After a `--keep-local` run the new stamp is still written whole, so it records the release's digest for files that were deliberately *not* written: the stamp is the baseline for the next comparison, not a literal inventory of what is on disk. That is what keeps a skipped file diverging - and therefore reported - on every later run, rather than quietly reading as current once it has been skipped once. A path in the old stamp but not the new one is reported as no longer part of the release and left alone unless `--prune` is passed, which removes it only if it is still unchanged since installation. Reports the migration chain the new machinery would owe (`chemenu.kb_state.chain` over the *new* tree's `instructions/migrations/`, read via a `directory` argument to `load_migrations`) but never runs any of it - there is no `migrate run`. Refuses before touching the source at all when: `VERSION` or `.wikitool-release.json` (with a `files` block) is missing locally, `.wikitool-kb.json` is missing, a migration is already outstanding against the *installed* machinery, or the working tree is dirty (not being a git repository at all is a WARN, not a refusal). Refuses after reading the source when: it carries no `VERSION`/`.wikitool-release.json`/`files` block, its version is older than or equal to the installed one (equal is a no-op success), or it is a pre-release (`-beta.N`) without `--pre`. Reports, but does not block on, a crossed compatibility boundary. Never touches git - no commit, no push (invariant 5). See Gitea #7 and `INSTALL.md` § "Eine Instanz aktualisieren" | | `dist upgrade <source> [--dry-run] [--keep-local] [--prune] [--pre]` | Apply a stack update `dist export` produced - the write half of `version check`. Never downloads anything: `<source>` is an already-fetched export directory or `.tar.gz` release archive (verified against a sibling `.sha256` if one is present; WARNs, does not block, if it is absent), which must unpack to exactly one top-level directory - the shape `.gitea/workflows/release.yml` packs. The write set is exactly the *new* `.wikitool-release.json`'s `files` block, minus what an export re-seeds from a blank template every time (`kb/log.md`, `raw/*/.gitkeep` - `chemenu.ownership.is_export_stub`) or seeds once and the instance owns from then on (`.wikitool-kb.json`, `CHANGES.md` - `chemenu.ownership.is_upgrade_preserved`), plus the stamp itself, always rewritten. Every candidate path is classified against the *local* `.wikitool-release.json`'s recorded digest for it: unchanged is overwritten silently, absent from the old stamp is created, and locally modified or locally deleted is **never** silently overwritten - the run aborts with the full list unless `--keep-local` says to proceed and leave every one of them untouched. After a `--keep-local` run the new stamp is still written whole, so it records the release's digest for files that were deliberately *not* written: the stamp is the baseline for the next comparison, not a literal inventory of what is on disk. That is what keeps a skipped file diverging - and therefore reported - on every later run, rather than quietly reading as current once it has been skipped once. A path in the old stamp but not the new one is reported as no longer part of the release and left alone unless `--prune` is passed, which removes it only if it is still unchanged since installation. Reports the migration chain the new machinery would owe (`chemenu.kb_state.chain` over the *new* tree's `instructions/migrations/`, read via a `directory` argument to `load_migrations`) but never runs any of it - there is no `migrate run`. Refuses before touching the source at all when: `VERSION` or `.wikitool-release.json` (with a `files` block) is missing locally, `.wikitool-kb.json` is missing, a migration is already outstanding against the *installed* machinery, or the working tree is dirty (not being a git repository at all is a WARN, not a refusal). Refuses after reading the source when: it carries no `VERSION`/`.wikitool-release.json`/`files` block, its version is older than or equal to the installed one (equal is a no-op success), or it is a pre-release (`-beta.N`) without `--pre`. Reports, but does not block on, a crossed compatibility boundary. Never touches git - no commit, no push (invariant 5). See Gitea #7 and `INSTALL.md` § "Eine Instanz aktualisieren" |
| `version show [--json]` | Print this instance's stack version and where it came from (development tree, or a distribution with its export date and origin). Bare `wikitool version` is an alias for this. Read-only, offline, and **exempt from the Iteration Budget Gate** | | `version show [--json]` | Print this instance's stack version and where it came from (development tree, or a distribution with its export date and origin). Bare `wikitool version` is an alias for this. Read-only, offline, and **exempt from the Iteration Budget Gate** |
| `version check [--url U] [--timeout S] [--json]` | Ask the origin's release feed whether a newer stack exists, and whether the step crosses a compatibility boundary (`state: current\|update\|migration\|ahead`). **The only command in `wikitool` that makes a network call** - never reached implicitly from another command, needs no key, times out, and reports an unreachable feed as an error rather than as "up to date". The feed is `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin; `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously. Read-only and exempt from the budget gate | | `version check [--url U] [--timeout S] [--json]` | Ask the origin's release feed whether a newer stack exists, and whether the step crosses a compatibility boundary (`state: current\|update\|migration\|ahead`). **The only command in `wikitool` that makes a network call** - never reached implicitly from another command, needs no key, times out, and reports an unreachable feed as an error rather than as "up to date". The feed is `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin; `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously. Read-only and exempt from the budget gate |
@@ -191,6 +195,7 @@ is atomic, and whether a retry is safe.
| `work new` | Neither or both of `--input`/`--key` given, `--input` outside `raw/`, a `--key` that is empty or starts with `ingest-`, or the workshop already exists | Yes - one directory with two files | A collision is not transient: resume the existing run instead, or pass `--again` if the tree itself changed. Never create a numbered variant by hand | | `work new` | Neither or both of `--input`/`--key` given, `--input` outside `raw/`, a `--key` that is empty or starts with `ingest-`, or the workshop already exists | Yes - one directory with two files | A collision is not transient: resume the existing run instead, or pass `--again` if the tree itself changed. Never create a numbered variant by hand |
| `work close` | Unknown run key, or `--yes` was not passed | No - a recursive delete | For "not confirmed": check the listed files are no longer needed, confirm the conclusions are in `kb/`, then re-run with `--yes` | | `work close` | Unknown run key, or `--yes` was not passed | No - a recursive delete | For "not confirmed": check the listed files are no longer needed, confirm the conclusions are in `kb/`, then re-run with `--yes` |
| `sources coverage` / `sources trace` | Bad arguments (e.g. neither or both of `--raw`/`--page`) | Read-only | Fix the argument and retry | | `sources coverage` / `sources trace` | Bad arguments (e.g. neither or both of `--raw`/`--page`) | Read-only | Fix the argument and retry |
| `raw accept` | A file does not exist, is not under `incoming/`, sits directly in `incoming/` or nested below its type directory, files in one call disagree on type or share a name, a target path already exists, `--page` names an unknown page or one with no `raw_files:` yet, an existing `raw_files:` entry is missing on disk or under a different type directory, or a file to be moved has more than one owning page | No - one filesystem move per file, then (with `--page`) one page write | Fix the named argument and retry once. Safe to retry as-is once the cause is fixed: a file already at its computed destination is what "already exists" reports, not a partial prior run to resume. Never choose the destination by hand instead - that is the decision this command exists to take away |
| `types list` / `types describe` | Unknown type name | Read-only | Fix the name and retry | | `types list` / `types describe` | Unknown type name | Read-only | Fix the name and retry |
| `instructions sync` / `verify` / `list` | Nothing found under `instructions/`, a malformed instruction or `SKILL.md`, a published copy that drifted from its source, an instruction nothing references (or, for `manual: true`, one that IS linked from AGENTS.md or a skill and so risks running implicitly), something under `instructions/dev/` referenced from outside it and outside a dist:strip block, or (sync) a target directory that is not a published skill and `--force` was not passed | `sync` rewrites one directory per target (idempotent); `verify`/`list` are read-only | Fix the flagged file, then re-run. For drift, re-run `sync`: the source under `instructions/` always wins, and a published copy is never edited directly | | `instructions sync` / `verify` / `list` | Nothing found under `instructions/`, a malformed instruction or `SKILL.md`, a published copy that drifted from its source, an instruction nothing references (or, for `manual: true`, one that IS linked from AGENTS.md or a skill and so risks running implicitly), something under `instructions/dev/` referenced from outside it and outside a dist:strip block, or (sync) a target directory that is not a published skill and `--force` was not passed | `sync` rewrites one directory per target (idempotent); `verify`/`list` are read-only | Fix the flagged file, then re-run. For drift, re-run `sync`: the source under `instructions/` always wins, and a published copy is never edited directly |
| `docs verify` | A command, contract, or type-form mismatch was found | Read-only | Fix the documentation it names, then re-run | | `docs verify` | A command, contract, or type-form mismatch was found | Read-only | Fix the documentation it names, then re-run |
+2
View File
@@ -27,6 +27,7 @@ try:
new_page, new_page,
page_ops, page_ops,
provenance_cmd, provenance_cmd,
raw_cmd,
run_budget, run_budget,
search as search_module, search as search_module,
touch as touch_module, touch as touch_module,
@@ -61,6 +62,7 @@ app.add_typer(index_build.app, name="index")
app.add_typer(log_append.app, name="log") app.add_typer(log_append.app, name="log")
app.add_typer(confidence_decay.app, name="confidence") app.add_typer(confidence_decay.app, name="confidence")
app.add_typer(provenance_cmd.app, name="sources") app.add_typer(provenance_cmd.app, name="sources")
app.add_typer(raw_cmd.app, name="raw")
app.add_typer(instructions_cmd.app, name="instructions") app.add_typer(instructions_cmd.app, name="instructions")
app.add_typer(run_budget.app, name="budget") app.add_typer(run_budget.app, name="budget")
app.add_typer(types_cmd.app, name="types") app.add_typer(types_cmd.app, name="types")
+10
View File
@@ -135,6 +135,10 @@ INSTRUCTIONS_EXCLUDE_DIRS = {"dev"}
# Fixed by raw/CONTRACT.md's routing table, unlike kb/'s areas (which are # Fixed by raw/CONTRACT.md's routing table, unlike kb/'s areas (which are
# organic - see kb/CONTRACT.md - so `export` does not manufacture them). # organic - see kb/CONTRACT.md - so `export` does not manufacture them).
# `docs verify` (check_raw_subdirs) holds the table to this tuple in both
# directions, and `incoming/` (raw/CONTRACT.md "Getting a file in",
# raw_cmd.py) mirrors it as the set of type subdirectories a human may drop a
# file into - so this is the one place all three read the list from.
RAW_SUBDIRS = ("articles", "documents", "notes", "assets") RAW_SUBDIRS = ("articles", "documents", "notes", "assets")
# Stage contracts that are not collections and carry no pages: copied as a # Stage contracts that are not collections and carry no pages: copied as a
@@ -414,6 +418,12 @@ def build_plan(origin: Optional[Origin] = None) -> dict[str, PlannedFile]:
for sub in RAW_SUBDIRS: for sub in RAW_SUBDIRS:
plan[f"raw/{sub}/.gitkeep"] = PlannedFile("") plan[f"raw/{sub}/.gitkeep"] = PlannedFile("")
# `incoming/` mirrors raw/'s type subdirectories (raw/CONTRACT.md
# "Getting a file in") - seeded the same way, though `.gitignore`
# (also exported, see ROOT_FILES) excludes the whole directory again
# once the instance is a git repo, which is why bootstrap.md re-creates
# it for a plain clone that never had this export step at all.
plan[f"incoming/{sub}/.gitkeep"] = PlannedFile("")
plan["kb/log.md"] = PlannedFile((DIST_TEMPLATES_DIR / "log.md").read_text(encoding="utf-8")) plan["kb/log.md"] = PlannedFile((DIST_TEMPLATES_DIR / "log.md").read_text(encoding="utf-8"))
plan["CHANGES.md"] = PlannedFile((DIST_TEMPLATES_DIR / "CHANGES.md").read_text(encoding="utf-8")) plan["CHANGES.md"] = PlannedFile((DIST_TEMPLATES_DIR / "CHANGES.md").read_text(encoding="utf-8"))
+41
View File
@@ -38,6 +38,7 @@ from typing import Optional
import typer import typer
from chemenu import config, conventions, kb_collections, version as version_mod from chemenu import config, conventions, kb_collections, version as version_mod
from chemenu.commands import dist_cmd
from chemenu.commands._util import fail, rel_path, success from chemenu.commands._util import fail, rel_path, success
app = typer.Typer(help="Verify documentation that mirrors the code or repo layout.") app = typer.Typer(help="Verify documentation that mirrors the code or repo layout.")
@@ -108,6 +109,12 @@ REQUIRED_IGNORE_CANARIES = (
"ENVIRONMENT.md", "ENVIRONMENT.md",
"tools/coverage.xml", "tools/coverage.xml",
"tools/htmlcov/index.html", "tools/htmlcov/index.html",
# The ingest inbox (Gitea #58, raw/CONTRACT.md "Getting a file in"). Unlike
# raw/ itself, a file here must never be committed - promotion via
# `wikitool raw accept` is what makes it immutable, not the drop - so this
# is the one canary in this tuple asserting the *opposite* of raw/'s own
# backstop a few lines above.
"incoming/documents/probe.pdf",
) )
REQUIRED_TRACKED_PATHS = ( REQUIRED_TRACKED_PATHS = (
"reports/CONTRACT.md", "reports/CONTRACT.md",
@@ -158,6 +165,11 @@ LEGACY_TYPE_RE = re.compile(r"^type:\s*(entity|concept|source|comparison)\s*$",
# First backticked cell of a markdown table row, e.g. "| `xref add --a ...` | ... |" # First backticked cell of a markdown table row, e.g. "| `xref add --a ...` | ... |"
TABLE_CELL_RE = re.compile(r"^\|\s*`([^`]+)`", re.MULTILINE) TABLE_CELL_RE = re.compile(r"^\|\s*`([^`]+)`", re.MULTILINE)
# A raw/CONTRACT.md routing-table cell naming a bare type subdirectory, e.g.
# "| `articles/` | ... |" - deliberately narrower than TABLE_CELL_RE, which
# would also match a command example elsewhere on the page.
RAW_DIR_CELL_RE = re.compile(r"^\|\s*`([a-zA-Z0-9_-]+)/`\s*\|", re.MULTILINE)
def registered_commands() -> set[str]: def registered_commands() -> set[str]:
"""Every command path the CLI exposes, e.g. {'new', 'xref add', ...}. """Every command path the CLI exposes, e.g. {'new', 'xref add', ...}.
@@ -317,6 +329,34 @@ def check_stack_required_types() -> list[str]:
return issues return issues
def documented_raw_subdirs(text: str) -> list[str]:
return [match.group(1) for match in RAW_DIR_CELL_RE.finditer(text)]
def check_raw_subdirs() -> list[str]:
"""`raw/CONTRACT.md`'s routing table and `dist_cmd.RAW_SUBDIRS` must name
the same set of type subdirectories (Gitea #58) - the table is meant to
read as behaviour derived from the tuple, not as a second place the list
could drift (AGENTS.md invariant 8). Skipped if the contract itself is
missing; `check_collection_contracts` already reports that.
"""
contract_path = config.ROOT / "raw" / "CONTRACT.md"
if not contract_path.exists():
return []
documented = set(documented_raw_subdirs(contract_path.read_text(encoding="utf-8")))
declared = set(dist_cmd.RAW_SUBDIRS)
issues = [
f"raw/CONTRACT.md's routing table is missing `{missing}/` - dist_cmd.RAW_SUBDIRS names it"
for missing in sorted(declared - documented)
]
issues += [
f"raw/CONTRACT.md's routing table lists `{extra}/`, but dist_cmd.RAW_SUBDIRS does not - "
"the two must name the same set"
for extra in sorted(documented - declared)
]
return issues
def check_legacy_type_blocks() -> list[str]: def check_legacy_type_blocks() -> list[str]:
issues = [] issues = []
guarded = [ guarded = [
@@ -585,6 +625,7 @@ def verify():
check_cli_readme() check_cli_readme()
+ check_readmes_have_no_command_table() + check_readmes_have_no_command_table()
+ check_collection_contracts() + check_collection_contracts()
+ check_raw_subdirs()
+ check_legacy_type_blocks() + check_legacy_type_blocks()
+ check_ignored_content() + check_ignored_content()
+ check_version_changelog() + check_version_changelog()
+217
View File
@@ -0,0 +1,217 @@
"""`wikitool raw accept` - promote one or more files from `incoming/` into
`raw/`, with the destination computed rather than chosen by hand (Gitea #58).
A human classifies a file only by which type subdirectory of `incoming/` they
drop it into - `incoming/articles/`, `incoming/documents/`, `incoming/notes/`,
`incoming/assets/`, mirroring raw/CONTRACT.md's routing table. Everything past
that is this command's job:
- **Single file, no bundle.** One file promoted alone lands as
`raw/<type>/<name>` - no directory of its own.
- **Bundle from the second file on.** Several files of one source promoted in
the same call land under `raw/<type>/<stem>/`, named after the first file's
stem.
- **Growing an existing single file into a bundle.** `--page` extends an
existing source page's `raw_files:`. If that raises the page from one file
to more than one, the file it already had is folded into the new bundle
alongside the ones just promoted, in the same call - at no point does
`raw_files:` point at a path that does not exist.
Multi-owner raw files (`provenance.duplicate_raw_file_owners`) are refused
rather than silently moved: relocating a file another page also claims would
break that page's `raw_files:` without it ever being consulted.
"""
from __future__ import annotations
from pathlib import Path
from typing import Optional
import typer
from chemenu import config
from chemenu.commands._util import fail, rel_path, success
from chemenu.commands.dist_cmd import RAW_SUBDIRS
from chemenu.frontmatter_io import write_page
from chemenu.kb_scan import load_kb_pages
from chemenu.provenance import source_pages_by_raw_file, source_raw_files
app = typer.Typer(help="Promote raw material out of incoming/ into raw/.")
def _incoming_dir() -> Path:
return config.ROOT / "incoming"
def _resolve(raw: Path) -> Path:
return raw if raw.is_absolute() else config.ROOT / raw
def _classify(path: Path, incoming: Path) -> str:
"""The type subdirectory `path` (already resolved, absolute) declares by
where it sits under `incoming/`, or fail with the reason it doesn't."""
try:
rel = path.relative_to(incoming)
except ValueError:
fail(
f"{rel_path(path)} is not under incoming/ - `raw accept` only promotes files "
"from there. See raw/CONTRACT.md."
)
allowed = ", ".join(f"incoming/{s}/" for s in RAW_SUBDIRS)
if len(rel.parts) < 2:
fail(
f"incoming/{rel} declares no type - place it inside one of {allowed} instead "
"of directly in incoming/."
)
sub = rel.parts[0]
if sub not in RAW_SUBDIRS:
fail(f"incoming/{rel} lies under an unknown type directory 'incoming/{sub}/'. Allowed: {allowed}.")
if len(rel.parts) > 2:
fail(f"incoming/{rel} is nested below its type directory - place it directly in incoming/{sub}/.")
return sub
@app.command("accept")
def raw_accept_command(
files: list[Path] = typer.Argument(
...,
help="One or more files under incoming/<type>/, all belonging to the same source",
),
page: Optional[str] = typer.Option(
None,
"--page",
help="Extend this existing source page's raw_files: with the promoted file(s), "
"folding in its already-promoted file if this raises it past one",
),
dry_run: bool = typer.Option(False, "--dry-run", help="List what would move without writing"),
):
"""Promote file(s) from incoming/ into raw/, computing the destination
(type directory, bundle or not, bundle name) instead of taking it as an
argument. See raw/CONTRACT.md "Getting a file in: incoming/"."""
if not files:
fail("Pass at least one file to promote.")
incoming = _incoming_dir()
resolved = [_resolve(f) for f in files]
for path in resolved:
if not path.is_file():
fail(f"{rel_path(path)} does not exist or is not a file.")
subs = {_classify(path, incoming) for path in resolved}
if len(subs) > 1:
allowed = ", ".join(sorted(f"incoming/{s}/" for s in subs))
fail(f"All files in one `raw accept` call must share one type directory; got {allowed}.")
sub = subs.pop()
names = [path.name for path in resolved]
if len(names) != len(set(names)):
fail("Two files share a filename; rename one before promoting.")
raw_sub_dir = config.RAW_DIR / sub
pages = None
target_page = None
existing_raw_paths: list[Path] = []
if page is not None:
pages = load_kb_pages(config.KB_DIR)
target_page = pages.get(page)
if target_page is None:
fail(f"No page titled '{page}' found under wiki/. Create it first, or omit --page.")
existing_rel = source_raw_files(target_page)
if not existing_rel:
fail(
f"'{page}' has no raw_files: yet - omit --page and run "
"`wikitool new source --set raw_files=...` for a page's first raw file."
)
existing_raw_paths = [config.ROOT / p for p in existing_rel]
missing = [p for p in existing_raw_paths if not p.is_file()]
if missing:
fail(
f"'{page}' claims raw file(s) that do not exist on disk: "
f"{', '.join(rel_path(p) for p in missing)}. Fix raw_files: (see `sources coverage`) "
"before promoting more."
)
existing_subs = {p.relative_to(config.RAW_DIR).parts[0] for p in existing_raw_paths}
if existing_subs != {sub}:
fail(
f"'{page}' already claims file(s) under {', '.join(sorted(f'raw/{s}/' for s in existing_subs))}, "
f"not raw/{sub}/. A bundle is one type directory; promote separately."
)
# A bundle directory forms once two or more files belong to the source
# (Gitea #58 decision 3): from the second file on, never before. Whenever
# --page targets an existing page it always has >=1 raw file already
# (types/source.md requires raw_files:), so bundling always applies there.
total = len(existing_raw_paths) + len(resolved)
bundle_dir: Optional[Path] = None
if len(existing_raw_paths) >= 2:
parents = {p.parent for p in existing_raw_paths}
if len(parents) != 1:
fail(
f"'{page}' raw_files: are not all in one directory - fix them by hand first "
"(see `sources coverage`)."
)
bundle_dir = parents.pop()
elif total >= 2:
primary = existing_raw_paths[0] if existing_raw_paths else resolved[0]
bundle_dir = raw_sub_dir / primary.stem
moves: list[tuple[Path, Path]] = []
for existing in existing_raw_paths:
if bundle_dir is not None and existing.parent != bundle_dir:
moves.append((existing, bundle_dir / existing.name))
for new_path in resolved:
dst = (bundle_dir / new_path.name) if bundle_dir is not None else (raw_sub_dir / new_path.name)
moves.append((new_path, dst))
for _src, dst in moves:
if dst.exists():
fail(f"Cannot promote: {rel_path(dst)} already exists.")
moving_existing = [src for src, _dst in moves if src in existing_raw_paths]
if moving_existing:
by_raw = source_pages_by_raw_file(pages)
conflicts = [
(rel_path(src), sorted(set(by_raw.get(rel_path(src), [])) - {page}))
for src in moving_existing
]
conflicts = [(p, owners) for p, owners in conflicts if owners]
if conflicts:
listed = "\n".join(f" - {p}: also claimed by {', '.join(o)}" for p, o in conflicts)
fail(
"Cannot bundle: the following already-covered raw file(s) have more than one "
f"owner, so moving them would break the other page(s)' raw_files::\n{listed}\n"
"Resolve the multiple ownership first (see `wikitool sources coverage`)."
)
if dry_run:
for src, dst in moves:
typer.echo(f"[dry-run] would move {rel_path(src)} -> {rel_path(dst)}")
if target_page is not None:
moved_map = dict(moves)
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
typer.echo(f"[dry-run] would set raw_files: on '{page}' to {[rel_path(p) for p in final]}")
typer.echo(f"[dry-run] would move {len(moves)} file(s). No files written.")
return
for src, dst in moves:
dst.parent.mkdir(parents=True, exist_ok=True)
src.rename(dst)
typer.echo(f" moved {rel_path(src)} -> {rel_path(dst)}")
moved_map = dict(moves)
if target_page is not None:
final = [moved_map.get(p, p) for p in existing_raw_paths] + [moved_map[p] for p in resolved]
target_page.frontmatter["raw_files"] = [rel_path(p) for p in final]
write_page(target_page.path, target_page.frontmatter, target_page.body)
success(
f"Promoted {len(resolved)} file(s); updated raw_files: on '{page}' "
f"({len(final)} file(s) total)."
)
return
promoted = ", ".join(rel_path(moved_map[p]) for p in resolved)
success(
f"Promoted {len(resolved)} file(s) to {promoted}. "
"Run `wikitool new source --set raw_files=...` (or `touch --set` on an existing page) next."
)
+9
View File
@@ -371,6 +371,14 @@ def test_plan_creates_empty_raw_subdirs_not_real_content(repo):
assert not any("personal-note" in relative for relative in plan) assert not any("personal-note" in relative for relative in plan)
def test_plan_creates_matching_incoming_subdirs(repo):
"""The ingest inbox (Gitea #58) mirrors raw/'s type subdirectories one for
one - both come from the same `RAW_SUBDIRS` tuple."""
plan = dist_cmd.build_plan()
for sub in ("articles", "documents", "notes", "assets"):
assert f"incoming/{sub}/.gitkeep" in plan
def test_plan_seeds_log_and_changes_from_templates(repo): def test_plan_seeds_log_and_changes_from_templates(repo):
plan = dist_cmd.build_plan() plan = dist_cmd.build_plan()
assert "Wiki Log" in plan["kb/log.md"].content assert "Wiki Log" in plan["kb/log.md"].content
@@ -489,6 +497,7 @@ def test_export_into_a_fresh_directory_works(repo, tmp_path):
assert (target / "AGENTS.md").is_file() assert (target / "AGENTS.md").is_file()
assert (target / "kb" / "entities" / "COLLECTION.md.template").is_file() assert (target / "kb" / "entities" / "COLLECTION.md.template").is_file()
assert (target / "raw" / "notes" / ".gitkeep").is_file() assert (target / "raw" / "notes" / ".gitkeep").is_file()
assert (target / "incoming" / "notes" / ".gitkeep").is_file()
def test_unbalanced_markers_fail_loudly(repo): def test_unbalanced_markers_fail_loudly(repo):
+34
View File
@@ -1,6 +1,7 @@
import pytest import pytest
import typer import typer
from chemenu import config
from chemenu.commands import docs_verify from chemenu.commands import docs_verify
@@ -133,6 +134,31 @@ def test_legacy_type_regex_matches_pre_migration_form():
assert not docs_verify.LEGACY_TYPE_RE.search("---\ntype: types/comparison.md\n---") assert not docs_verify.LEGACY_TYPE_RE.search("---\ntype: types/comparison.md\n---")
def test_this_repos_raw_subdirs_are_documented():
assert docs_verify.check_raw_subdirs() == []
def test_raw_subdirs_mismatch_is_reported(tmp_path, monkeypatch):
"""The regression this guards: `dist_cmd.RAW_SUBDIRS` and raw/CONTRACT.md's
routing table are two places naming the same set (AGENTS.md invariant 8),
so either one drifting from the other must be caught in both directions."""
(tmp_path / "raw").mkdir()
(tmp_path / "raw" / "CONTRACT.md").write_text(
"| Directory | Holds |\n|---|---|\n| `articles/` | ... |\n| `videos/` | ... |\n",
encoding="utf-8",
)
monkeypatch.setattr(config, "ROOT", tmp_path)
monkeypatch.setattr(docs_verify.dist_cmd, "RAW_SUBDIRS", ("articles", "documents"))
issues = docs_verify.check_raw_subdirs()
assert any("missing `documents/`" in i for i in issues)
assert any("lists `videos/`" in i for i in issues)
def test_raw_subdirs_check_is_skipped_without_a_contract(tmp_path, monkeypatch):
monkeypatch.setattr(config, "ROOT", tmp_path)
assert docs_verify.check_raw_subdirs() == []
def test_no_content_is_gitignored(): def test_no_content_is_gitignored():
"""The regression guard for the 2026-08-13 `.gitignore` rewrite: patterns """The regression guard for the 2026-08-13 `.gitignore` rewrite: patterns
like `*temp*` and `bin/` were silently excluding files under raw/, so the like `*temp*` and `bin/` were silently excluding files under raw/, so the
@@ -151,6 +177,14 @@ def test_a_swallowed_canary_is_reported():
assert swallowed == ["tools/.wikitool_session/budget.json"] assert swallowed == ["tools/.wikitool_session/budget.json"]
def test_incoming_inbox_is_ignored():
"""Unlike raw/, a file under incoming/ must never be committed - promotion
via `wikitool raw accept` is what makes it immutable, not the drop."""
assert docs_verify.ignored_canaries(("incoming/documents/probe.pdf",)) == [
"incoming/documents/probe.pdf"
]
def test_the_environment_note_is_ignored_but_its_template_is_not(): def test_the_environment_note_is_ignored_but_its_template_is_not():
"""The pattern has to split a file from its own template. `ENVIRONMENT.md` """The pattern has to split a file from its own template. `ENVIRONMENT.md`
describes one checkout and must never be committed; `ENVIRONMENT.md.template` describes one checkout and must never be committed; `ENVIRONMENT.md.template`
+256
View File
@@ -0,0 +1,256 @@
import pytest
import typer
from chemenu import config
from chemenu.commands.raw_cmd import raw_accept_command
from chemenu.frontmatter_io import read_page, write_page
from chemenu.provenance import uncovered_raw_files
from chemenu.kb_scan import load_kb_pages
@pytest.fixture
def tree(kb_dir):
"""kb_dir already repoints config.ROOT at tmp_path; add raw/ and
incoming/ beside it, each with the four type subdirectories
dist_cmd.RAW_SUBDIRS declares."""
root = kb_dir.parent
for sub in ("articles", "documents", "notes", "assets"):
(root / "raw" / sub).mkdir(parents=True)
(root / "incoming" / sub).mkdir(parents=True)
return root
def _accept(*files, page=None, dry_run=False):
return raw_accept_command(files=list(files), page=page, dry_run=dry_run)
def _write_source(kb_dir, title, raw_files):
write_page(
kb_dir / "sources" / f"{title}.md",
{
"type": "types/source.md", "source_type": "document", "author": "Torben",
"raw_files": list(raw_files), "date": "2026-09-01",
"tags": [], "entities": [], "concepts": [], "summary": "Test source.",
},
f"\n# {title}\n\n## Summary\n\nTest.\n",
)
def test_single_file_needs_no_bundle(tree):
src = tree / "incoming/documents/handbuch.pdf"
src.write_bytes(b"%PDF-1.4 fake\n")
_accept(src)
dst = tree / "raw/documents/handbuch.pdf"
assert dst.is_file()
assert dst.read_bytes() == b"%PDF-1.4 fake\n"
assert not src.exists()
def test_two_files_bundle_under_the_first_files_stem(tree):
pdf = tree / "incoming/documents/handbuch.pdf"
md = tree / "incoming/documents/handbuch.md"
pdf.write_bytes(b"pdf-bytes")
md.write_text("# converted\n", encoding="utf-8")
_accept(pdf, md)
assert (tree / "raw/documents/handbuch/handbuch.pdf").read_bytes() == b"pdf-bytes"
assert (tree / "raw/documents/handbuch/handbuch.md").read_text(encoding="utf-8") == "# converted\n"
assert not pdf.exists() and not md.exists()
def test_file_directly_in_incoming_is_rejected(tree):
src = tree / "incoming/stray.md"
src.write_text("x\n", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
def test_unknown_type_subdir_is_rejected(tree):
(tree / "incoming/videos").mkdir()
src = tree / "incoming/videos/clip.mp4"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
def test_nested_too_deep_is_rejected(tree):
nested = tree / "incoming/documents/sub"
nested.mkdir()
src = nested / "deep.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
def test_mixed_type_subdirs_in_one_call_is_rejected(tree):
a = tree / "incoming/documents/a.pdf"
b = tree / "incoming/notes/b.md"
a.write_bytes(b"a")
b.write_text("b", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(a, b)
assert a.exists() and b.exists()
def test_same_file_passed_twice_is_rejected(tree):
src = tree / "incoming/documents/handbuch.pdf"
src.write_bytes(b"x")
with pytest.raises(typer.Exit):
_accept(src, src)
assert src.exists()
def test_collision_with_existing_raw_file_is_rejected(tree):
(tree / "raw/documents/handbuch.pdf").write_bytes(b"already there")
src = tree / "incoming/documents/handbuch.pdf"
src.write_bytes(b"new")
with pytest.raises(typer.Exit):
_accept(src)
assert src.exists()
assert (tree / "raw/documents/handbuch.pdf").read_bytes() == b"already there"
def test_missing_file_is_rejected(tree):
with pytest.raises(typer.Exit):
_accept(tree / "incoming/documents/absent.pdf")
def test_dry_run_moves_nothing(tree):
src = tree / "incoming/documents/handbuch.pdf"
src.write_bytes(b"x")
_accept(src, dry_run=True)
assert src.exists()
assert not (tree / "raw/documents/handbuch.pdf").exists()
def test_promoted_file_in_incoming_is_never_reported_uncovered(tree):
"""AC: incoming/ is invisible to sources coverage / lint until accepted -
both walk raw/ only."""
(tree / "incoming/notes/not-yet-promoted.md").write_text("draft\n", encoding="utf-8")
pages = load_kb_pages(tree / "kb")
assert "raw/notes/not-yet-promoted.md" not in uncovered_raw_files(tree / "raw", pages)
assert uncovered_raw_files(tree / "raw", pages) == []
def test_without_page_flag_only_moves_and_prints_target(tree, capsys):
src = tree / "incoming/notes/meeting.md"
src.write_text("notes\n", encoding="utf-8")
_accept(src)
out = capsys.readouterr().out
assert "raw/notes/meeting.md" in out
assert "new source" in out
def test_page_flag_extends_raw_files_for_a_single_new_file(tree):
"""No growth case: the page already has >=1 file, so this always bundles -
see test_growth_case below for the interesting path."""
(tree / "raw/documents").mkdir(exist_ok=True)
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/documents/handbuch-appendix.pdf"
new_file.write_bytes(b"second")
_accept(new_file, page="Source - Handbuch")
bundle = tree / "raw/documents/handbuch"
assert (bundle / "handbuch.pdf").read_bytes() == b"first"
assert (bundle / "handbuch-appendix.pdf").read_bytes() == b"second"
assert not (tree / "raw/documents/handbuch.pdf").exists()
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert sorted(frontmatter["raw_files"]) == sorted(
["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch-appendix.pdf"]
)
def test_growth_case_no_broken_or_uncovered_refs_afterwards(tree):
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
(tree / "raw/documents/handbuch.pdf").write_bytes(b"first")
new_file = tree / "incoming/documents/handbuch.md"
new_file.write_text("converted", encoding="utf-8")
_accept(new_file, page="Source - Handbuch")
from chemenu.provenance import broken_raw_refs
pages = load_kb_pages(tree / "kb")
# kb_dir ships an unrelated "Source - Aurora" fixture page whose raw file
# this tree never had - filter to what this test actually changed.
assert [r for r in broken_raw_refs(pages) if r["page"] == "Source - Handbuch"] == []
assert [f for f in uncovered_raw_files(tree / "raw", pages) if "handbuch" in f] == []
def test_adding_to_an_already_bundled_source_joins_the_existing_bundle(tree):
(tree / "raw/documents/handbuch").mkdir(parents=True)
(tree / "raw/documents/handbuch/handbuch.pdf").write_bytes(b"a")
(tree / "raw/documents/handbuch/handbuch.md").write_text("b", encoding="utf-8")
_write_source(
tree / "kb", "Source - Handbuch",
["raw/documents/handbuch/handbuch.pdf", "raw/documents/handbuch/handbuch.md"],
)
extra = tree / "incoming/documents/handbuch-notes.md"
extra.write_text("c", encoding="utf-8")
_accept(extra, page="Source - Handbuch")
assert (tree / "raw/documents/handbuch/handbuch-notes.md").read_text(encoding="utf-8") == "c"
# Nothing already-bundled was moved a second time.
assert (tree / "raw/documents/handbuch/handbuch.pdf").read_bytes() == b"a"
frontmatter, _ = read_page(tree / "kb/sources/Source - Handbuch.md")
assert set(frontmatter["raw_files"]) == {
"raw/documents/handbuch/handbuch.pdf",
"raw/documents/handbuch/handbuch.md",
"raw/documents/handbuch/handbuch-notes.md",
}
def test_growth_case_rejects_a_multi_owner_raw_file(tree):
(tree / "raw/documents/handbuch.pdf").write_bytes(b"shared")
_write_source(tree / "kb", "Source - Handbuch", ["raw/documents/handbuch.pdf"])
_write_source(tree / "kb", "Source - Also Handbuch", ["raw/documents/handbuch.pdf"])
new_file = tree / "incoming/documents/handbuch.md"
new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Handbuch")
assert new_file.exists()
assert (tree / "raw/documents/handbuch.pdf").exists()
assert not (tree / "raw/documents/handbuch").exists()
def test_page_not_found_is_rejected(tree):
new_file = tree / "incoming/documents/handbuch.md"
new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Nonexistent")
assert new_file.exists()
def test_page_with_no_raw_files_is_rejected(tree):
write_page(
tree / "kb/sources/Source - Empty.md",
{
"type": "types/source.md", "source_type": "document", "author": "Torben",
"raw_files": [], "date": "2026-09-01",
"tags": [], "entities": [], "concepts": [], "summary": "Empty.",
},
"\n# Source - Empty\n",
)
new_file = tree / "incoming/documents/handbuch.md"
new_file.write_text("x", encoding="utf-8")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Empty")
assert new_file.exists()
def test_type_directory_mismatch_with_existing_raw_files_is_rejected(tree):
(tree / "raw/notes/handbuch.md").write_bytes(b"first")
_write_source(tree / "kb", "Source - Handbuch", ["raw/notes/handbuch.md"])
new_file = tree / "incoming/documents/handbuch.pdf"
new_file.write_bytes(b"second")
with pytest.raises(typer.Exit):
_accept(new_file, page="Source - Handbuch")
assert new_file.exists()