Compare commits

...

10 Commits

Author SHA1 Message Date
torben 576df2cddd feat: MCP-Leseserver, Bibliotheksgrenze, Haertung des Lesepfads, Publish-Remote-Gate scharf (2.4.0)
CI / verify (push) Successful in 52s
Release / release (push) Successful in 37s
Files changed:
- .gitea/workflows/ci.yml
- CHANGES.md
- README.md
- VERSION
- instructions/mcp-read-server.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/api.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/lint.py
- tools/chemenu/commands/search.py
- tools/chemenu/commands/types_cmd.py
- tools/chemenu/config.py
- tools/chemenu/corpus_cache.py
- tools/chemenu/errors.py
- tools/chemenu/frontmatter_io.py
- tools/chemenu/lint_core.py
- tools/chemenu/mcp/__init__.py
- tools/chemenu/mcp/__main__.py
- tools/chemenu/mcp/server.py
- tools/chemenu/page.py
- tools/chemenu/search/filters.py
- tools/chemenu/search/registry.py
- tools/chemenu/search/ripgrep.py
- tools/chemenu/search/service.py
- tools/chemenu/tests/conftest.py
- tools/chemenu/tests/test_api.py
- tools/chemenu/tests/test_corpus_cache.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_frontmatter_io.py
- tools/chemenu/tests/test_instructions_cmd.py
- tools/chemenu/tests/test_mcp_server.py
- tools/chemenu/tests/test_new_page.py
- tools/chemenu/tests/test_search.py
- tools/chemenu/type_resolver.py
- tools/chemenu/types_core.py
- tools/requirements-mcp.txt
2026-09-02 07:19:32 +02:00
torben d1cf2e0327 ingest: Private-Instance Merge Correction and Issue 30 Session 2026-09-01
Files changed:
- kb/concepts/Delete Rather Than Anonymize.md
- kb/concepts/Publish-Remote Gate.md
- kb/entities/projects/Chemenu.md
- kb/entities/tools/wikitool.md
- kb/index.md
- kb/log.md
- kb/provenance.md
- kb/sources/INDEX.md
- kb/sources/Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
2026-09-01 20:36:01 +02:00
torben df7ea93060 ingest: Publish-Remote Gate and Issue Triage Session 2026-09-01
Files changed:
- kb/concepts/INDEX.md
- kb/concepts/Mass-Update Gate.md
- kb/concepts/Publish-Remote Gate.md
- kb/entities/projects/Chemenu.md
- kb/entities/tools/wikitool.md
- kb/index.md
- kb/log.md
- kb/provenance.md
- kb/sources/INDEX.md
- kb/sources/Source - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
2026-09-01 20:34:21 +02:00
torben 00c2cf6ffe ingest: Public Release, Corpus Purge and History Squash Session 2026-09-01
Files changed:
- kb/concepts/Delete Rather Than Anonymize.md
- kb/concepts/Dual Licensing by File Plan.md
- kb/concepts/INDEX.md
- kb/entities/projects/Chemenu.md
- kb/entities/tools/wikitool.md
- kb/index.md
- kb/log.md
- kb/provenance.md
- kb/sources/INDEX.md
- kb/sources/Source - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
- raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
- raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
- raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
2026-09-01 20:32:05 +02:00
torben 32a9b8eb3f docs: Publish-Remote Gate im Werkzeugvertrag, Projektseite auf oeffentlich (2.2.2)
CI / verify (push) Successful in 45s
Release / release (push) Successful in 37s
Files changed:
- CHANGES.md
- VERSION
- instructions/gates.md
- kb/entities/INDEX.md
- kb/entities/projects/Chemenu.md
- kb/log.md
- tools/CONTRACT.md
2026-09-01 19:15:36 +02:00
torben b2f7dec122 fix: private-instance - Demo-Korpus wandert beim Merge mit, Prozedur korrigiert (2.2.1)
CI / verify (push) Successful in 43s
Release / release (push) Successful in 38s
Files changed:
- CHANGES.md
- VERSION
- instructions/private-instance.md
2026-09-01 19:03:23 +02:00
torben 29063f511b docs: INSTALL.md auf das oeffentliche Repo umgestellt, Weg D fuer die private Instanz
CI / verify (push) Successful in 45s
Files changed:
- INSTALL.md
2026-09-01 18:14:08 +02:00
torben 7263f85936 feat: Publish-Remote Gate und die Anleitung fuer eine private Instanz (2.2.0)
CI / verify (push) Successful in 44s
Release / release (push) Successful in 36s
Files changed:
- .gitignore
- AGENTS.md
- CHANGES.md
- VERSION
- instructions/gates.md
- instructions/private-instance.md
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/config.py
- tools/chemenu/tests/test_git_publish.py
2026-09-01 18:06:55 +02:00
torben fb97d46888 docs: Formulierung im 2.1.0-Eintrag geglaettet
CI / verify (push) Successful in 43s
Files changed:
- CHANGES.md
2026-09-01 17:07:18 +02:00
torben f63a72cb24 fix: raw_dir-Fixture kappt config.ROOT, private Fixture-Namen ersetzt (2.1.1)
CI / verify (push) Successful in 42s
Release / release (push) Successful in 37s
Files changed:
- CHANGES.md
- VERSION
- tools/chemenu/tests/conftest.py
- tools/chemenu/tests/test_corpus_diff.py
- tools/chemenu/tests/test_index_build.py
- tools/chemenu/tests/test_kb_collections.py
- tools/chemenu/tests/test_lint.py
- tools/chemenu/tests/test_migrate_cmd.py
- tools/chemenu/tests/test_new_page.py
- tools/chemenu/tests/test_page_ops.py
- tools/chemenu/tests/test_provenance.py
- tools/chemenu/tests/test_search.py
- tools/chemenu/tests/test_touch.py
2026-09-01 17:03:51 +02:00
70 changed files with 4948 additions and 755 deletions
+4
View File
@@ -101,6 +101,10 @@ jobs:
# instance does not measure this suite. Installed beside pytest for # instance does not measure this suite. Installed beside pytest for
# the same reason pytest itself is. # the same reason pytest itself is.
tools/.venv/bin/pip install --quiet pytest pytest-cov tools/.venv/bin/pip install --quiet pytest pytest-cov
# The MCP server's dependency is optional for an instance but not for
# CI: its tests skip without it, and a skipped golden test is exactly
# how the server's output and the CLI's would drift apart unnoticed.
tools/.venv/bin/pip install --quiet -r tools/requirements-mcp.txt
- name: Tests - name: Tests
# Not run with WIKI_TRACE=0: two telemetry tests assert that a trace is # Not run with WIKI_TRACE=0: two telemetry tests assert that a trace is
+7
View File
@@ -110,6 +110,13 @@ npm-debug.log*
# `dist export`; this anchored pattern deliberately does not match it. # `dist export`; this anchored pattern deliberately does not match it.
/ENVIRONMENT.md /ENVIRONMENT.md
# Publish-Remote Gate allowlist (see instructions/gates.md). Names the push
# URLs *this* checkout may publish to, so it is per-checkout for exactly the
# reason ENVIRONMENT.md above is: a committed copy would tell a private clone
# that the public upstream is a legitimate target for its own content. Absent
# means unrestricted; `doctor` reports which.
/.wikitool-remotes.json
# Coverage output from `pytest --cov` (see .gitea/workflows/ci.yml). Derived, # Coverage output from `pytest --cov` (see .gitea/workflows/ci.yml). Derived,
# like reports/: recomputable from any commit, and `publish` runs `git add -A`, # like reports/: recomputable from any commit, and `publish` runs `git add -A`,
# so an unignored htmlcov/ would commit itself on the next content publish. # so an unignored htmlcov/ would commit itself on the next content publish.
+8 -3
View File
@@ -20,7 +20,9 @@ tools/wikitool instructions sync
Full procedure, including the tool environment: [instructions/bootstrap.md](instructions/bootstrap.md). Full procedure, including the tool environment: [instructions/bootstrap.md](instructions/bootstrap.md).
Setting up a brand-new, empty instance instead of cloning this one: `tools/wikitool dist export` Setting up a brand-new, empty instance instead of cloning this one: `tools/wikitool dist export`
and [instructions/setup-instance.md](instructions/setup-instance.md) - see and [instructions/setup-instance.md](instructions/setup-instance.md) - see
[INSTALL.md](INSTALL.md). [INSTALL.md](INSTALL.md). A *private* instance that keeps taking stack updates from a public
upstream is a third shape, with a safeguard the other two do not need:
[instructions/private-instance.md](instructions/private-instance.md).
## Invariants ## Invariants
@@ -177,16 +179,19 @@ tools/wikitool search --field entity_type=system --field 'confidence<0.6'
## Gates ## Gates
Two limits are enforced in code rather than by instruction, because a prompt-level limit is Three limits are enforced in code rather than by instruction, because a prompt-level limit is
one an agent can talk itself past. one an agent can talk itself past.
- **Mass-Update Gate.** `publish` exits **42** on a change touching too many files, printing - **Mass-Update Gate.** `publish` exits **42** on a change touching too many files, printing
the file list and the `--confirm <token>` line that publishes it once the user approves. The the file list and the `--confirm <token>` line that publishes it once the user approves. The
threshold and the rule live in [instructions/gates.md](instructions/gates.md). threshold and the rule live in [instructions/gates.md](instructions/gates.md).
- **Publish-Remote Gate.** `publish` exits **42** on a push to a URL this checkout has not
declared in `.wikitool-remotes.json`. It has no token and no flag: the way past it is a
deliberate edit by the user, never by an agent.
- **Iteration Budget Gate / Loop-Breaker.** Past 60 `wikitool` calls in a session, or after 3 - **Iteration Budget Gate / Loop-Breaker.** Past 60 `wikitool` calls in a session, or after 3
identical calls in a row, further calls are refused. identical calls in a row, further calls are refused.
Both refuse with exit 1. **Do not retry, and do not open the gate.** Stop, summarize the The last refuses with exit 1. **Do not retry, and do not open a gate.** Stop, summarize the
situation to the user, and get explicit approval. The full procedure - including why situation to the user, and get explicit approval. The full procedure - including why
`budget reset` is not the escape hatch - is [instructions/gates.md](instructions/gates.md). `budget reset` is not the escape hatch - is [instructions/gates.md](instructions/gates.md).
+418 -2
View File
@@ -20,6 +20,423 @@ their date-only headings.
--- ---
## 2.4.0 - 2026-09-02 - MCP-Leseserver: zweiter Konsument auf demselben Kern
**Author:** Torben Nehmer
Letzter Schritt der Sequenz aus #36, inhaltlich Issue #19. `chemenu` bekommt einen zweiten
Konsumenten: `search`, `types`, `describe_type`, `lint` und `status` über MCP. Kein CLI mit
angeschraubter Netzwerkschnittstelle — CLI und Server sind zwei Adapter auf dem Kern, den 2.3.0
freigelegt hat.
**`tools/chemenu/mcp/`**, im Repo statt als eigenes Artefakt. Der Golden-Test, der die
Serverantworten gegen die `--json`-Formen der CLI hält, läuft nur mit beiden Seiten in einer
Testsuite; getrennt würde aus einem Contract eine Versionsabsprache. Der Test ruft `wikitool` als
Subprozess gegen denselben Baum auf, über `CHEMENU_ROOT` — womit er nebenbei die Root-Auflösung
von außen mitprüft.
**Zwei Transports.** `stdio` zum Entwickeln und Testen ohne Netz, `streamable-http` für die
Auslieferung — der einzige, vor den sich die Authentifizierungs-Middleware überhaupt setzen kann,
weil sie ein HTTP-Reverse-Proxy ist. `sse` ist über das SDK erreichbar und wird bewusst nicht
angeboten: der abgelöste Remote-Transport, jetzt darauf zu bauen verschiebt den Wechsel nur.
`--host`/`--port` gibt es, weil der Default auf Loopback bindet und ein Container hinter einem
Proxy eine Adresse braucht, die der Proxy erreicht — eine Eigenschaft der Software, nicht einer
Installation. Beide Transports sind gegen den echten Korpus gegengeprüft.
**Kein Schreibpfad, strukturell.** Weder der Server noch `chemenu.api` importiert irgendetwas
unter `chemenu.commands`, also existieren `new`, `touch`, `xref`, `cite`, `publish`, `migrate`
und `version bump` in dieser Reichweite gar nicht, statt aus einer Liste gefiltert zu werden. Ein
Test importiert das Servermodul in einem frischen Interpreter und sieht in `sys.modules` nach;
ein zweiter ruft alle fünf Tools auf und vergleicht den Dateibaum, `HEAD` und
`git status --porcelain` vorher/nachher.
**Jede Antwort trägt ihren Commit.** `commit` und `as_of` in jedem Payload; `null` heißt, der
bediente Baum hat uncommittete Änderungen und die Antwort entspricht keiner Revision. Der Stempel
ist die Revision, aus der die Seiten *tatsächlich* gelesen wurden — zwischen Laden und Stempeln
kann der Baum sich bewegen, deshalb reicht der Ladepfad seine Revision durch, statt noch einmal
zu fragen. Das war beim ersten Durchlauf falsch: `types`/`lint`/`status` lasen die zuletzt
*gecachte* Revision und stempelten `null`, obwohl der Baum sauber war.
**Telemetrie in den bedienten Baum wird beim Start verweigert**, nicht stillschweigend
umgeleitet. Tracing ist per Default an und schreibt nach `reports/telemetry/` im Repo — genau das
Verzeichnis, das der Sync per `git reset --hard` wegräumen darf. `WIKI_TRACE=0` oder
`WIKI_TRACE_DIR` außerhalb des Korpus. Heute schreibt auf diesem Pfad nichts (der Emitter hängt an
`cli.main()` und den Gates), die Sperre ist gegen später.
**Fehler an der Protokollgrenze.** Ein `ChemenuError` wird zum `ToolError` des SDK — eine
absichtliche Ablehnung, deren Text den Aufrufer erreicht. Alles andere bleibt ein Absturz, dessen
Text auf dem Server bleibt. Ein kaputtes Prädikat ist das Argument des Aufrufers, also muss die
Zeile mitreisen, die sagt, was stattdessen zu schreiben ist.
**Bewusst nicht enthalten:** Authentifizierung und Rate Limiting (Middleware vor dem Prozess),
Deployment (private Infrastruktur), der Iteration Budget Gate — er begrenzt eine Agenten-Session
und nicht einen Nutzer, weshalb Retrieval von ihm befreit ist; ihn hier als Rate Limiter zu
benutzen würde ihn dazu verwässern.
**Die Abhängigkeit ist optional** (`tools/requirements-mcp.txt`): eine Instanz, die nur die CLI
benutzt, soll dafür nicht pydantic, starlette, uvicorn und cryptography installieren müssen. CI
installiert sie, denn ein übersprungener Golden-Test ist genau der Weg, auf dem Server und CLI
unbemerkt auseinanderlaufen.
Betrieb und Sync-Mechanismus: [instructions/mcp-read-server.md](instructions/mcp-read-server.md).
Polling (`git fetch && git reset --hard`) statt Webhook — kein eingehender Endpunkt, keine
Signaturprüfung. `reset --hard` ist dort tragend und keine Bequemlichkeit: ein abgedrifteter Baum
antwortet zwar richtig, parst aber bei jeder Anfrage neu und stempelt jede Antwort mit `null`.
**Dateien:** `chemenu/mcp/` (neu: `server.py`, `__main__.py`), `chemenu/api.py`,
`tools/requirements-mcp.txt` (neu), `instructions/mcp-read-server.md` (neu), `tools/CONTRACT.md`,
`tools/README.md`, `.gitea/workflows/ci.yml`, `tests/test_mcp_server.py` (neu).
---
## 2.3.0 - 2026-09-02 - Bibliotheksgrenze: chemenu laesst sich auf einen Korpus zeigen
**Author:** Torben Nehmer
Dritter Schritt der Sequenz aus #36, inhaltlich Issue #31. Der Schritt, der `chemenu` von einem
Skript, das in seinem eigenen Verzeichnis lebt, zu einer Bibliothek macht, auf die man einen
Korpus *zeigen* kann. Rückwärtskompatibel: ohne gesetzte Variable verhält sich `tools/wikitool`
unverändert.
**Root-Auflösung nach Präzedenz statt nach Dateilage.** `config.resolve_root()`: expliziter
Parameter → `$CHEMENU_ROOT` → Walk-up wie bisher. Der Walk-up bleibt Default. Wichtiger als die
neue Fähigkeit ist die beseitigte Fehlerklasse: `ROOT` und alles darunter waren
Modulkonstanten, also **zur Importzeit gebunden**, und
`monkeypatch.setattr(config, "ROOT", ziel)` zeigte `ROOT` um, aber nicht `KB_DIR`/`RAW_DIR`. Wer
sich darauf verließ, baute etwas, das scheinbar auf einem Zielbaum arbeitete und in Wahrheit aus
dem Entwickler-Checkout antwortete. Die abgeleiteten Pfade werden jetzt bei jedem Zugriff
aufgelöst (PEP 562 `__getattr__`) und folgen `ROOT` — der halb-umgezeigte Zustand ist nicht mehr
konstruierbar. `CHEMENU_ROOT` ist in `_WIKITOOL_ENV` registriert, #23-konform.
**`config.reset()` und `config.rooted()`.** `reset()` nimmt Zuweisungen auf die verwalteten
Pfadnamen zurück; die Testsuite ruft es zwischen Tests, und das ist dort nicht optional:
`monkeypatch` merkt sich den alten Wert, indem es ihn *liest* — also auflöst — und schreibt ihn
beim Aufräumen als echtes Attribut zurück. Genau die stale Bindung, die der Umbau unmöglich
machen sollte, vom Cleanup wieder aufgebaut. `rooted(root)` setzt den Root für die Dauer eines
Blocks, prozessweit und damit nicht thread-sicher — der Aufrufer hält das Lock, dieselbe
Disziplin wie beim Korpus-Cache. Nötig, weil nicht alles einen Root als Argument nimmt: der
modulglobale `TypeResolver` muss `types/` finden, und ohne ihn läse ein fremder Korpus mit den
Type-Specs *dieses* Checkouts.
**Die Naht ist gezogen.** `run_search`/`run_lint`/`types` lagen in Modulen, die `typer` auf
Modulebene importieren und über `_util` auch `rich` — wer sie in-process aufrief, zog den
kompletten CLI-Kopf mit. Der reine Kern liegt jetzt in `search/service.py`, `lint_core.py` und
`types_core.py`; `commands/` sind die Terminal-Adapter darüber und re-exportieren die Namen, damit
kein bestehender Import bricht. Ein Test importiert `chemenu.api` in einem frischen Interpreter
und prüft, dass weder `chemenu.commands.*` noch `typer`/`rich`/`click` geladen werden.
**`chemenu.api.Corpus` als In-Process-Einstieg.** Nimmt einen Root, liefert exakt die
`--json`-Formen der CLI zurück — ein Wire-Contract statt zwei — und stempelt jede Antwort mit dem
Commit-SHA und einem Zeitstempel (`commit`, `as_of`), so dass aus einer stillen veralteten Antwort
eine sichtbare wird. `search`/`lint`/`types`/`describe_type`/`status`; `status` ist bewusst
serverseitig **komponiert** und kein Wrapper, weil es kein `wikitool status` gibt. Lesend
strukturell: nichts unter `chemenu.commands` wird importiert, die Schreibfunktionen existieren in
dieser Oberfläche also gar nicht, statt gefiltert zu werden. Das ist die Grenze, auf der #19
aufsetzt.
**Exceptions statt Exit-Codes an der Grenze.** `chemenu/errors.py`: `ChemenuError` mit
`ValidationError` (Eingabe abgelehnt) und `BackendError` (Abhängigkeit fehlt oder scheitert).
`PredicateError`, `FrontmatterError`, `UnknownBackend` und die beiden `Ripgrep*` hängen jetzt
darunter; `ValidationError` erbt zusätzlich von `ValueError`, weil `PredicateError` vorher eines
war und Aufrufer es so fangen. Das CLI-Verhalten ist unverändert: `fail()``ERROR`-Zeile,
Exit 1, Budget-Refund.
**`resolve()` reicht den Root an das Backend durch.** Vorher konnte ein Aufrufer `run_search` einen
Korpus übergeben, während `RipgrepBackend` weiter `config.KB_DIR` durchlief — die Anfrage aus dem
einen Baum beantwortet, die Seiten aus dem anderen gelesen, ohne dass irgendetwas das gesagt
hätte.
**Zwei Abhängigkeiten, die durch Zufall hielten, stehen jetzt da.** `TypeResolver.repo_root`
folgt `ROOT`, statt beim Import zu binden — womit Fixtures, die `ROOT` auf einen tmp-Baum zeigen,
die mitgelieferten Type-Specs ausdrücklich benennen müssen (`use_shipped_type_specs`). Dieselbe
Form wie das Loch, für das `raw_dir` geschrieben wurde, eine Ebene tiefer. Und
`types describe --json` trug `root:` nicht im Payload, obwohl `types list --json` es tut:
`types/instruction.md` deklariert `root: repo`, die Renderer-Zeile las es direkt aus dem
Frontmatter. Jetzt im Payload, in derselben Form wie bei `list`.
**Dateien:** `config.py`, `errors.py` (neu), `api.py` (neu), `search/service.py` (neu),
`lint_core.py` (neu), `types_core.py` (neu), `search/registry.py`, `type_resolver.py`,
`commands/search.py`, `commands/lint.py`, `commands/types_cmd.py`, `tools/CONTRACT.md`, dazu
`tests/conftest.py`, `tests/test_api.py` (neu), `tests/test_new_page.py`,
`tests/test_instructions_cmd.py`.
---
## 2.2.4 - 2026-09-02 - Haertung des Lesepfads: ReDoS, Subprozess-Timeout, YAML-Alias-Budget, Korpus-Cache
**Author:** Torben Nehmer
Zweiter Schritt der Sequenz aus #36, inhaltlich Issue #33. Limits vor der Exposition: der
Lesepfad bekommt mit dem MCP-Server (#19) einen Konsumenten, der nicht der Operator ist. Alle
sechs Befunde waren reproduziert, nicht vermutet; jeder hat jetzt eine Regression.
**ReDoS über `--regex` beseitigt.** `search/ripgrep.py:_contains` gab nutzergesteuerten Regex an
Pythons Backtracking-Engine: `(\w+\s?)+$` gegen 114 Zeichen gewöhnlichen Seiteninhalts terminiert
nicht in acht Sekunden, ein deterministisch scheiterndes Muster braucht 0,2 ms — der Unterschied
ist das Muster, nicht der Heuhaufen. `build_hit` ruft die Funktion je Treffer zweimal auf, und
`\w` matcht jede Seite: eine Anfrage kaufte zwei unbegrenzte Suchen pro Korpusseite. Der Zweig ist
**gelöscht**, nicht begrenzt. `rg` hat das Muster mit einer linearen Engine längst angewendet, wenn
die Funktion überhaupt läuft; verloren geht nur der zusätzliche Titel-/Summary-Bonus für
nicht-literale Muster, und Summary wie H1 sind selbst Zeilen in der Datei, die `rg` zählt.
**Subprozess-Timeout.** `rg` wird nach 30 s abgeräumt und über den vorhandenen
`RipgrepFailed`-Pfad gemeldet. Kein Performance-Budget — eine Fixed-String-Suche kostet hier 6 ms —
sondern ein Hänger-Abbruch, damit ein Aufruf als Fehler endet statt den Aufrufer offenzuhalten,
während seine Ausgabe in den Heap puffert.
**YAML-Anchors und -Aliases im Frontmatter werden verweigert, nicht budgetiert.** Gemessen:
267 Byte werden in 0,2 ms zu 672.603 Knoten beim Traversal, Wachstum 9ⁿ bei konstanter Parse-Zeit
— ein Größenlimit fasst das nicht an, weil die Eingabe klein bleibt. Die Prüfung läuft auf dem
*Event*-Strom (`yaml.parse`), der nichts auflöst, kostet also O(Text) und löst nie aus, wonach sie
fragt; `*` ist in jedem Alias-Knoten notwendig, seine Abwesenheit beweist Abwesenheit ohne jeden
Parse — der Weg, den jede echte Seite nimmt. Dazu ein Größenlimit von 64 KiB und ein Abfangen von
`RecursionError` (PyYAML komponiert rekursiv, tiefe Verschachtelung ist kein `YAMLError`). Heute
nicht erreichbar, weil `kb/` der Operator committet; erreichbar mit der Ingest-Queue (#32).
**`CSafeLoader` statt `SafeLoader`, mit Fallback.** Gemessen über diesen Korpus (176 Seiten,
best of 5): **265 ms → 54 ms**. Kein Mikro-Tuning — der Korpus-Parse war der größte Einzelposten
eines `search`-Aufrufs und skaliert linear mit der Korpusgröße. End-to-end fällt ein
`wikitool search` damit von 593 ms auf **347 ms**; die verbleibenden 262 ms sind Modulimport und
entfallen erst im residenten Prozess (#19).
**Stiller Frontmatter-Verlust wird gemeldet.** Entschieden: der Lesepfad *nennt* die Seite, statt
sie zu schlucken. Kaputtes YAML wird weiterhin zu `{}` — Massenoperationen dürfen an einer Seite
nicht scheitern —, aber der Grund wird mitgeführt (`Page.frontmatter_error`) und ausgegeben:
`search --json` trägt immer eine `unreadable`-Liste aus `{path, reason}`, die Tabellenform
schreibt dieselben Zeilen nach stderr. Das war nötig, weil so eine Seite weder `confidence` noch
`kind` hat und damit aus jedem positiven `--field`-Prädikat fällt — ausgerechnet aus der
Low-Confidence-Suche, die Seiten in genau diesem Zustand finden soll — und dabei aussieht wie eine
Seite, die nicht gematcht hat. Für Frontmatter, das diese Instanz nicht selbst geschrieben hat,
steht `read_page_strict()` bereit: die Quarantäne aus #32 muss strikt lesen, wo ein
unlesbares Dokument das Dokument stoppen und nicht leeren soll.
**Ein Parser statt zwei.** `read_page()` und `frontmatter_error()` liefen bisher getrennt durch
`safe_load` — so konnte der permissive Weg zu `{}` degradieren aus einem Grund, den der strikte
Weg anders beschrieb, und jeder Aufrufer, der beide Antworten wollte, las die Datei zweimal.
Beide gehen jetzt durch `_load_frontmatter()`; ein Test hält sie gegeneinander.
**Korpus-Cache am Commit-SHA** (`chemenu/corpus_cache.py`), als Objekt, das ein Aufrufer *hält*
kein Modul-Dict, das sich hinter allen einschaltet. Die CLI hält keins und verhält sich unverändert
(ein Aufruf pro Prozess, nichts wiederzuverwenden); der residente Prozess aus #19 hält eins.
Entscheidend ist nicht die Geschwindigkeit, sondern dass nichts veraltet: **ein schmutziger
Arbeitsbaum wird nie gecacht**, sonst bekäme eine Sitzung, die eine Seite schreibt und danach
sucht, die Analyse von vor dem Schreiben — bei unverändertem SHA. Kann git nicht antworten, gilt
der Baum als schmutzig. Derselbe SHA ist der Antwort-Stempel aus #19, per Konstruktion also die
Revision, aus der die Antwort auch wirklich berechnet wurde.
**Erhalten geblieben** ist die Eigenschaft, die kein Befund war: kein `shell=True`,
`--fixed-strings` als Default, `--`-Terminator. Der Modul-Docstring führt sie jetzt als drei
tragende Zusagen statt zwei — die dritte ist, dass nutzergesteuerte Muster ausschließlich `rg`
sieht.
**Dateien:** `frontmatter_io.py`, `search/ripgrep.py`, `commands/search.py`, `page.py`,
`corpus_cache.py` (neu), `tools/CONTRACT.md`, dazu `tests/test_frontmatter_io.py`,
`tests/test_search.py`, `tests/test_corpus_cache.py` (neu).
---
## 2.2.3 - 2026-09-02 - Publish-Remote-Gate in diesem Checkout scharf, doctor benennt den Zustand
**Author:** Torben Nehmer
Erster Schritt der Sequenz aus Issue #36 (Weg zum MCP-Leseserver), inhaltlich Issue #34. Das
Gate war seit 2.2.0 implementiert und in diesem Checkout **inert**: `.wikitool-remotes.json`
fehlte, und eine fehlende Datei heißt unbeschränkt. `ENVIRONMENT.md` beschrieb `origin`
gleichzeitig als einziges Publish-Ziel, `AGENTS.md` führt das Gate unter den drei in Code
erzwungenen Grenzen. Eine dokumentierte Sicherung, die nicht greift, ist schlechter als eine
fehlende — sie erzeugt genau das Vertrauen, das sie nicht verdient.
**Die Datei ist angelegt** (gitignored, per Checkout, reist nicht mit) und gegengeprüft: ein
`publish --remote` auf ein nicht gelistetes Ziel verweigert mit Exit 42, bevor irgendetwas
gestaged wird, und der Arbeitsbaum bleibt unberührt. Damit steht die Sicherung **vor** dem Klonen
der privaten Instanz (#30) — nachträglich gesetzt ließe sie genau das Fenster offen, das sie
schließt.
**`doctor` sagt jetzt, ob das Gate scharf ist, nicht nur ob die Datei da ist.** Alle drei
Zustände beginnen mit `Gate armed:` bzw. `Gate not armed:`; der einzelne Remote ohne Allowlist
bleibt `OK` (er hat nichts zu schützen, und ein FAIL machte die Datei durch die Hintertür
verpflichtend), sagt aber ausdrücklich, dass jedes Push-Ziel durchkommt. Der Fall, der wirklich
beißt — mehrere Remotes ohne Allowlist — bleibt `WARN`. Der Check hatte bislang **keine Tests**;
drei sind dazugekommen, einer je Zustand.
**Dateien:** `.wikitool-remotes.json` (neu, nicht committet), `doctor.check_publish_remotes()`,
`tools/chemenu/tests/test_doctor.py`.
---
## 2.2.2 - 2026-09-01 - Doku-Verdrahtung: Publish-Remote Gate im Werkzeugvertrag, Projektseite auf oeffentlich
**Author:** Torben Nehmer
Nachziehen dessen, was 2.2.0 und die Veröffentlichung offen gelassen haben. Gefunden durch eine
Durchsicht auf lose Enden, nicht durch einen Fehlschlag — `docs verify` deckt den Fall nicht ab,
weil es Kommando-*Namen* gegeneinander prüft, nicht ob ein neuer Fehlerfall beschrieben ist.
**`tools/CONTRACT.md` kannte das Publish-Remote-Gate nicht.** Der Werkzeugvertrag ist die Stelle,
an der pro Kommando steht, was ein Fehlschlag bedeutet und ob ein Retry sicher ist — und
`publish` hatte seit 2.2.0 einen dritten Weg zu Exit 42, der dort nirgends stand. Ergänzt in
beiden Richtungen: in der Kommandozeile (URL statt Remote-Name, `pushurl` wird gelesen, fehlende
Datei heißt unbeschränkt, kaputte Datei ist ein Fehler) und im Fehlerkontrakt, wo der
entscheidende Unterschied zu den anderen beiden Gates steht — es gibt **keine** `--confirm`-Zeile,
die der Agent nachreichen könnte.
**`instructions/gates.md` verwies nicht auf die Prozedur, für die das Gate gebaut wurde.**
Jetzt verlinkt, mit dem Hinweis auf Schritt 4: Das Gate gehört vor den ersten `publish`, später
hinzugefügt schützt es das Fenster nicht, das es schließen soll.
**Die Projektseite beschrieb sich selbst falsch.** `kb/entities/projects/Chemenu.md` nannte
Chemenu ein „persönliches IT-Wissens-Wiki" mit dem Zweck „Persönliche IT-Wissensbasis" — seit
der Veröffentlichung schlicht unzutreffend, und es ist die Seite, die ein Fremder über das
Projekt liest. Neu gefasst: deterministischer Wissenskompiler, diese Instanz seit 2026-09-01
öffentlich als Testbett und Demo, Lizenz benannt.
Die historische Aussage über die monolithische `AGENTS.md` (~30 KB) **bleibt** — sie war zu ihrer
Zeit korrekt und ist belegt. Sie ist jetzt als Ausgangspunkt datiert statt als Gegenwart
formuliert, dieselbe Unterscheidung, die #29 für die Issue-Texte trifft: ein Pfad als Wegweiser
wird nachgezogen, ein Pfad als Beleg bleibt stehen und bekommt ein Datum.
**Dateien:** `tools/CONTRACT.md`, `instructions/gates.md`,
`kb/entities/projects/Chemenu.md`, `kb/entities/INDEX.md` (generiert).
---
## 2.2.1 - 2026-09-01 - private-instance: der Demo-Korpus wandert beim Merge doch mit - Prozedur korrigiert
**Author:** Torben Nehmer
`instructions/private-instance.md` behauptete in 2.2.0, ein `git merge upstream/main` löse
Änderungen am Demo-Korpus stillschweigend auf, weil die private Instanz ihn einmal gelöscht hat:
*deleted-in-ours, unmodified-in-theirs*. Das war **nicht gemessen, sondern angenommen** — und es
ist falsch. Ein Nachbau mit einem Upstream, der seinen Korpus bewegt, zeigt drei verschiedene
Verhalten:
| Upstream tut | `git merge upstream/main` tut |
|---|---|
| ändert eine Seite, die du gelöscht hast | `CONFLICT (modify/delete)` — und **lässt die Upstream-Fassung im Arbeitsbaum liegen**. Ein `git add -A` beim Auflösen holt die Demo-Seite zurück. |
| legt eine neue Seite an | staged sie **stillschweigend**. Kein Konflikt, keine Meldung. |
| löscht eine Seite, die du auch gelöscht hast | nichts. Der einzige harmlose Fall. |
Die mittlere Zeile ist die gefährliche, weil nichts sie ankündigt. Ein Upstream, der einen
Demo-Korpus ausliefert **und** ihn als Testbett benutzt, legt Seiten an — und jede einzelne
landet in der privaten Instanz und taucht dort in `lint`, `index`, `search` und
`confidence decay` auf. Genau diese Doppelnutzung beschreibt Issue #28.
**Korrigiert:** Die Update-Prozedur hält den Merge mit `--no-commit` offen, zwingt `kb/` und
`raw/` danach auf den eigenen Stand zurück (`git rm --cached`, `rm -rf`, `git checkout HEAD --`)
und schließt ihn erst dann. Solange der Merge offen ist, zeigt `HEAD` noch auf den Commit davor,
und genau das macht den Schritt sauber. Anschließend eine Kontrolle, die man nicht überlesen
kann:
```bash
git diff --name-only $BEFORE HEAD -- kb raw # muss leer sein
```
Das Rezept ist wörtlich so nachgespielt worden, wie es jetzt in der Datei steht — mit einem
Upstream, der gleichzeitig eine Seite ändert, eine anlegt, eine löscht und dasselbe unter
`raw/` tut. Ergebnis: Stack aktualisiert, nur eigener Inhalt übrig, Kontrolle leer,
Arbeitsbaum sauber.
**Auch die Decision Points korrigiert.** „Konflikt in `kb/` per Hand als *keep deleted*
auflösen" stand vorher da und ist der Rat, der in die Falle führt: `git add -A` committet die
Fassung, die git im Arbeitsbaum liegen gelassen hat.
**Dateien:** `instructions/private-instance.md`.
---
## 2.2.0 - 2026-09-01 - Publish-Remote Gate: publish schreibt nur an erklaerte Ziele
**Author:** Torben Nehmer
Der Stack bekommt sein drittes Gate. Die beiden bestehenden fragen, ob eine Änderung zu groß
ist und ob ein Rebase gefährlich ist. Dieses fragt, was darunter liegt: **ob das überhaupt das
richtige Repository ist.**
**Das Problem entsteht erst durch die private Instanz.** Ein Checkout mit eigenem Inhalt hat
typischerweise zwei Remotes — sein eigenes und das öffentliche Upstream, von dem er
Stack-Updates zieht. Git unterscheidet die beim Push nicht, also legt ein falsches `--remote`
einen privaten Korpus auf ein öffentliches Repository. Das ist nicht billig rückholbar, und
zwar nachweislich: Beim Veröffentlichen dieses Repos blieb die gesamte alte History nach dem
Force-Push per SHA abrufbar, bis auf dem Server die Reflogs verfielen und `git gc --prune=now`
lief. Ein Force-Push bewegt den Branch, nicht die Objekte.
**`.wikitool-remotes.json` nennt die erlaubten Push-URLs.** Nicht die Remote-*Namen*: Eine
Namensliste ließe ein `publish` durch, dessen `origin` umgebogen wurde, und genau das ist der
Fall, den das Gate fangen soll. Gelesen wird die `pushurl`, wenn der Remote eine setzt, denn
dorthin schreibt `git push` tatsächlich.
**Pro Checkout und gitignored**, aus demselben Grund wie `ENVIRONMENT.md`: Zwei Klone pushen an
zwei verschiedene Orte, eine committete Kopie würde einem privaten Klon also mitteilen, das
öffentliche Upstream sei ein legitimes Ziel für seinen eigenen Inhalt. **Fehlt die Datei, gilt
keine Beschränkung** — ein Checkout mit einem Remote und ohne Privates hat nichts zu schützen,
und eine Pflichtdatei würde aus einer Sicherung Papierkram machen. Eine *kaputte* Datei ist
dagegen ein Fehler und kein „keine Beschränkung": Eine beschädigte Sicherung darf sich nicht
wie eine abgeschaltete verhalten.
**Kein Token, keine Flagge.** Die anderen beiden Gates lösen sich mit einem `--confirm <token>`,
weil ihre Frage („ist diese Änderung richtig?") für genau ein Changeset beantwortbar ist. Dieses
fragt „gehört dieser Inhalt in jenes Repository?", und das ist eine stehende Eigenschaft des
Checkouts, kein Einzelfallurteil. Der Weg daran vorbei ist ein bewusster Edit des Nutzers.
Ein Agent, der die Datei anfasst, um an einer Verweigerung vorbeizukommen, öffnet ein Gate aus
eigenem Antrieb — Invariante 6.
**`doctor` meldet den Zustand** statt ihn zu erzwingen: OK mit Anzahl der Ziele, OK bei
Abwesenheit mit einem Remote, und WARN bei mehr als einem Remote ohne Allowlist — also genau in
der Form, die eine private Instanz annimmt, sobald sie das Upstream hinzufügt.
**Und die Prozedur, für die das Gate gebaut wurde.** `instructions/private-instance.md` (neu)
beschreibt die dritte Instanz-Form neben „frisch aufsetzen" und „Repo klonen": eine private
Arbeitsinstanz, die Stack-Updates von einem öffentlichen Upstream per `git merge` zieht und
deren eigener Inhalt nie zurückwandert. Der Grund, warum das dem Tarball-Weg vorzuziehen ist,
steht dort ausformuliert — `cp -r` hat keinen Drei-Wege-Merge und keine Konflikterkennung.
Schritt 4 der Anleitung ist das Gate, und zwar ausdrücklich **vor** dem ersten `publish`:
später hinzugefügt schützt es das Fenster nicht, das es schließen soll.
**Dateien:** `config.PUBLISH_REMOTES_FILENAME`, `git_publish.read_allowed_push_urls()`,
`push_url_for()`, `publish_remote_refusal()` und die Prüfung vor dem Reconcile-Schritt,
`doctor.check_publish_remotes()`, `.gitignore`, `instructions/gates.md`,
`instructions/private-instance.md` (neu), `AGENTS.md` (Gate-Liste und Bootstrap-Routing),
12 neue Tests in `test_git_publish.py`.
---
## 2.1.1 - 2026-09-01 - raw_dir-Fixture kappt config.ROOT; letzte private Fixture-Namen ersetzt
**Author:** Torben Nehmer
Der erste CI-Lauf nach der Korpus-Bereinigung wurde rot, während dieselbe Suite lokal grün
blieb: `test_legacy_source_pages_flags_url_and_directory` scheiterte mit
`KeyError: 'Source - DirBacked'`.
**Die Ursache ist keine Testschwäche, sondern eine Umgebungsabhängigkeit.** Der Test legt eine
Quellenseite mit `source: raw/documents` an und erwartet, dass `legacy_source_pages()` sie als
„directory, not a file" meldet. Die Prüfung dort ist `(config.ROOT / legacy).is_dir()` — sie
löst also gegen das *echte* Repo-Wurzelverzeichnis auf, während die `raw_dir`-Fixture ihren
Baum unter `tmp_path` baut. Der Test war nie hermetisch; er bestand, weil dieser Checkout
zufällig ein `raw/documents/` besaß.
Die Korpus-Bereinigung hat dieses Verzeichnis geleert. Git verfolgt keine leeren Verzeichnisse,
also verschwand es im CI-Checkout und blieb im lokalen Arbeitsbaum stehen — grün hier, rot
dort, ohne dass sich am Test etwas geändert hätte.
**Geschlossen in der Fixture, nicht im Test.** `raw_dir` setzt jetzt `config.ROOT` auf
`tmp_path`. Das ist dieselbe Bauart und dasselbe Argument wie bei `hermetic_environment`
(Gitea #8): Wenn eine Umgebungsabhängigkeit lokal unsichtbar ist, gehört sie an der Stelle
geschlossen, die alle Tests erreicht, statt bei dem einen, der zufällig darüber stolpert.
Nachgewiesen, indem `raw/documents/` lokal entfernt und die Suite erneut gefahren wurde — 38
Provenance-Tests grün ohne das Verzeichnis, vorher rot.
**Fixture-Namen.** Die Suite benutzte weiterhin reale Systemnamen der Ursprungsinstanz als
Fixture-Bezeichner. In einem
öffentlichen Repo beschreiben sie nichts, verraten aber die Namensgebung einer privaten
Umgebung. Ersetzt durch `aurora`, `almanac`, `Borealis`, `gateway`, `docstore`. `gdeploy`
bleibt: die Seite existiert im öffentlichen Korpus.
**Dateien:** `tools/chemenu/tests/conftest.py` (`raw_dir`), sämtliche Testdateien mit
Fixture-Namen.
---
## 2.1.0 - 2026-09-01 - Dual-Lizenz AGPL-3.0/CC-BY-4.0, und ein Leak-Canary fuer dist export ## 2.1.0 - 2026-09-01 - Dual-Lizenz AGPL-3.0/CC-BY-4.0, und ein Leak-Canary fuer dist export
**Author:** Torben Nehmer **Author:** Torben Nehmer
@@ -65,8 +482,7 @@ oder IP-Literale wurde erwogen und verworfen: Der eigene Host steht legitim in `
oder bei jedem Export Fehlalarm geben. oder bei jedem Export Fehlalarm geben.
**Beispiele neutralisiert.** Hilfetexte, Docstrings und Contract-Beispiele nannten reale **Beispiele neutralisiert.** Hilfetexte, Docstrings und Contract-Beispiele nannten reale
Seiten und Pfade der Ursprungsinstanz (Quellenseiten und Pfade Quellenseiten und Pfade der Ursprungsinstanz. In einer
aus dem privaten Korpus). In einer
Distribution ist das nicht nur unpassend, sondern verrät den Inhalt der Instanz, die sie gebaut Distribution ist das nicht nur unpassend, sondern verrät den Inhalt der Instanz, die sie gebaut
hat. Ersetzt durch Beispiele aus dem verbleibenden öffentlichen Korpus. hat. Ersetzt durch Beispiele aus dem verbleibenden öffentlichen Korpus.
+51 -21
View File
@@ -1,10 +1,11 @@
# Installation # Installation
Dieses Dokument richtet sich an Menschen. Es gibt drei Wege: ein **Release herunterladen** Dieses Dokument richtet sich an Menschen. Es gibt vier Wege: ein **Release herunterladen**
(der normale Weg zu einer neuen Instanz), eine Distribution **selbst exportieren**, oder (der normale Weg zu einer neuen Instanz), eine Distribution **selbst exportieren**, **dieses
**dieses Repo klonen** (Torbens persönliche Wiki, samt Inhalt). Der agent-seitige Ablauf steckt Repo klonen** (Testbett und Demo, samt Beispielkorpus), oder eine **private Instanz mit diesem
in `instructions/`; hier stehen nur die menschlichen Teile - für die vollständige Repo als Upstream** aufsetzen. Der agent-seitige Ablauf steckt in `instructions/`; hier stehen
Kommandoreferenz siehe [tools/CONTRACT.md](tools/CONTRACT.md). nur die menschlichen Teile - für die vollständige Kommandoreferenz siehe
[tools/CONTRACT.md](tools/CONTRACT.md).
## Voraussetzungen ## Voraussetzungen
@@ -16,19 +17,22 @@ Kommandoreferenz siehe [tools/CONTRACT.md](tools/CONTRACT.md).
## Weg A: Release herunterladen ## Weg A: Release herunterladen
Der kürzeste Weg zu einer eigenen Instanz - kein Checkout dieses Repos nötig. Jedes Release Der kürzeste Weg zu einer eigenen Instanz - kein Checkout dieses Repos nötig. Jedes Release
trägt genau einen `dist export`-Baum plus eine Prüfsumme. Das Repo ist derzeit privat, der trägt genau einen `dist export`-Baum plus eine Prüfsumme. Das Repo ist öffentlich, der Download
Download braucht also ein Gitea-Token mit Lesezugriff (siehe braucht also weder Konto noch Token:
[Konfiguration](#konfiguration)):
```bash ```bash
BASE=https://gitea.nehmer.net/torben/chemenu/releases/download/v<version> BASE=https://gitea.nehmer.net/torben/chemenu/releases/download/v<version>
curl -LO -H "Authorization: token $WIKITOOL_UPDATE_TOKEN" $BASE/chemenu-stack-<version>.tar.gz curl -LO $BASE/chemenu-stack-<version>.tar.gz
curl -LO -H "Authorization: token $WIKITOOL_UPDATE_TOKEN" $BASE/chemenu-stack-<version>.tar.gz.sha256 curl -LO $BASE/chemenu-stack-<version>.tar.gz.sha256
sha256sum -c chemenu-stack-<version>.tar.gz.sha256 sha256sum -c chemenu-stack-<version>.tar.gz.sha256
tar xzf chemenu-stack-<version>.tar.gz tar xzf chemenu-stack-<version>.tar.gz
cd chemenu-stack-<version> cd chemenu-stack-<version>
``` ```
Die Prüfsumme ist nicht Zierde: Sie ist das Einzige, was einen unterbrochenen Download von
einem vollständigen unterscheidet, und `sha256sum -c` muss `OK` sagen, bevor irgendetwas
entpackt wird.
Danach weiter mit Schritt 2 aus Weg B: den Agenten Danach weiter mit Schritt 2 aus Weg B: den Agenten
[instructions/setup-instance.md](instructions/setup-instance.md) ausführen lassen. Der [instructions/setup-instance.md](instructions/setup-instance.md) ausführen lassen. Der
entpackte Baum ist bereits eine Distribution - Schritt 1 (`dist export`) entfällt. entpackte Baum ist bereits eine Distribution - Schritt 1 (`dist export`) entfällt.
@@ -82,10 +86,14 @@ Zwei Schritte, von denen nur der erste rein menschlich ist:
## Weg C: Dieses Repo klonen ## Weg C: Dieses Repo klonen
Für Torbens Instanz selbst, oder einen Fork davon samt Inhalt: Für die Arbeit am Stack selbst, oder um sich den mitgelieferten Korpus als begehbares Beispiel
anzusehen. Was hier liegt, ist ein **Testbett und eine Demo**, keine produktive Wissensbasis:
rund 170 Seiten, die den Stack selbst dokumentieren - Gates, Lint, Versionierung, Suche, das
Wiki-Muster. Wer eigenes Wissen sammeln will, nimmt Weg A oder B und fängt mit einem leeren
`kb/` an.
```bash ```bash
git clone <repo-url> git clone https://gitea.nehmer.net/torben/chemenu.git
cd chemenu cd chemenu
``` ```
@@ -102,6 +110,21 @@ Checkout* beschreibt und nicht das Repo. Sie ist optional; wer sie anlegt, spart
folgenden Session die Fragen nach Harness, MCP-Servern und Remote. Vorlage: folgenden Session die Fragen nach Harness, MCP-Servern und Remote. Vorlage:
`ENVIRONMENT.md.template`, Ablauf: Schritt 5 in `instructions/bootstrap.md`. `ENVIRONMENT.md.template`, Ablauf: Schritt 5 in `instructions/bootstrap.md`.
## Weg D: Private Instanz mit diesem Repo als Upstream
Die Kombination aus A und C: eine eigene, nicht öffentliche Instanz, die weiterhin
Stack-Updates von hier zieht - per `git merge` statt per Tarball, also mit echtem
Drei-Wege-Merge statt `cp -r`.
Das ist der Weg mit dem höchsten Einsatz, weil ein Checkout dann zwei Remotes hat und git beim
Push nicht unterscheidet, welcher welcher ist. Ein falsches `--remote` legt privaten Inhalt auf
ein öffentliches Repo, und ein Force-Push holt das nicht zurück - die Objekte bleiben per SHA
abrufbar, bis auf dem Server die Reflogs verfallen.
Dagegen gibt es das **Publish-Remote-Gate**, und die Anleitung setzt es an die Stelle, an der
es wirkt: *vor* dem ersten `publish`. Vollständiges Vorgehen:
[instructions/private-instance.md](instructions/private-instance.md).
## Version und Updates ## Version und Updates
Jede Instanz trägt die Version des **Stacks** (Werkzeuge, Typen, Instruktionen, Contracts) - Jede Instanz trägt die Version des **Stacks** (Werkzeuge, Typen, Instruktionen, Contracts) -
@@ -199,22 +222,23 @@ behält Schema und Shape.
| `WIKI_AUTHOR` | Override für den Autornamen neuer Source-Seiten | `git config user.name` - fehlt beides, bricht `new` mit `ERROR` ab | | `WIKI_AUTHOR` | Override für den Autornamen neuer Source-Seiten | `git config user.name` - fehlt beides, bricht `new` mit `ERROR` ab |
| `WIKITOOL_SESSION_ID` | Scopt das Iteration-Budget-Gate auf eine Aufgabe statt auf ein Terminal | Parent-Process-ID (siehe [instructions/session-setup.md](instructions/session-setup.md)) | | `WIKITOOL_SESSION_ID` | Scopt das Iteration-Budget-Gate auf eine Aufgabe statt auf ein Terminal | Parent-Process-ID (siehe [instructions/session-setup.md](instructions/session-setup.md)) |
| `WIKITOOL_UPDATE_URL` | Release-Feed, den `version check` abfragt | Wert aus `.wikitool-release.json`, sonst der Feed der Ursprungs-Instanz | | `WIKITOOL_UPDATE_URL` | Release-Feed, den `version check` abfragt | Wert aus `.wikitool-release.json`, sonst der Feed der Ursprungs-Instanz |
| `WIKITOOL_UPDATE_TOKEN` | Gitea-Token für den Release-Feed | keiner - **aber das Ursprungs-Repo ist derzeit privat, also wird ein Token gebraucht** (siehe unten) | | `WIKITOOL_UPDATE_TOKEN` | Gitea-Token für den Release-Feed | keiner - gegen `torben/chemenu` nicht nötig, nur für einen privaten Fork (siehe unten) |
**Privates Ursprungs-Repo.** `torben/chemenu` ist nicht öffentlich lesbar. Gitea **Gegen das Ursprungs-Repo braucht es kein Token.** `torben/chemenu` ist öffentlich lesbar;
antwortet anonymen Aufrufern für ein unsichtbares Repo mit demselben `404` wie für ein gar `version check` und der Download in Weg A funktionieren ohne Konfiguration.
nicht existierendes - ein fehlendes Release und ein fehlender Zugriff sehen also identisch aus.
Für `version check` (und für den Download in Weg A) braucht es deshalb ein Gitea-Token mit **Für einen privaten Fork schon.** Wer den Stack in ein eigenes, nicht öffentliches Repo legt
Lesezugriff: und `WIKITOOL_UPDATE_URL` auf dessen Feed zeigen lässt, stößt auf eine Eigenheit, die man
kennen sollte: Gitea antwortet anonymen Aufrufern für ein unsichtbares Repo mit demselben
`404` wie für ein gar nicht existierendes. Ein fehlendes Release und ein fehlender Zugriff
sehen dann identisch aus - „kein Update gefunden" wäre in dem Fall schlicht gelogen. Dagegen
hilft ein Gitea-Token mit Lesezugriff:
```bash ```bash
export WIKITOOL_UPDATE_TOKEN="<gitea-token>" export WIKITOOL_UPDATE_TOKEN="<gitea-token>"
tools/wikitool version check tools/wikitool version check
``` ```
Wird das Repo öffentlich geschaltet, entfällt das Token ersatzlos - der Feed ist dann anonym
lesbar und `version check` funktioniert ohne Konfiguration.
## Verifikation ## Verifikation
```bash ```bash
@@ -259,6 +283,12 @@ tools/wikitool instructions verify
sondern die Aufforderung, die Ausgabe einem Menschen zu zeigen: sie enthält die vollständige sondern die Aufforderung, die Ausgabe einem Menschen zu zeigen: sie enthält die vollständige
Dateiliste und die exakte `--confirm <token>`-Zeile, die nach Freigabe veröffentlicht. Dateiliste und die exakte `--confirm <token>`-Zeile, die nach Freigabe veröffentlicht.
Details: [instructions/gates.md](instructions/gates.md). Details: [instructions/gates.md](instructions/gates.md).
- **`publish` endet mit Exit-Code 42 (Publish-Remote-Gate)** - dieser Checkout hat eine
`.wikitool-remotes.json`, und das angesteuerte Remote steht nicht darin. Ebenfalls kein
Fehler: Die Ausgabe nennt die Push-URL, an die geschrieben würde, und die erlaubten. Anders
als beim Mass-Update-Gate gibt es hier **keinen Token und keine Flagge** - stimmt das Ziel
wirklich, trägt der Mensch dessen URL selbst in die Datei ein. Ein Agent, der die Datei
anfasst, um an der Verweigerung vorbeizukommen, öffnet ein Gate aus eigenem Antrieb.
- **Ich will am Tool-Stack selbst weiterarbeiten (nicht nur Wiki-Inhalt betreiben)** - eine neue - **Ich will am Tool-Stack selbst weiterarbeiten (nicht nur Wiki-Inhalt betreiben)** - eine neue
Instanz hat dafür keinen Weg: `dist export` lässt `instructions/dev/` (Stack-Entwicklung, Instanz hat dafür keinen Weg: `dist export` lässt `instructions/dev/` (Stack-Entwicklung,
inkl. der vendorten `commonplace/`-Wissensbasis) bewusst und dauerhaft weg, ohne inkl. der vendorten `commonplace/`-Wissensbasis) bewusst und dauerhaft weg, ohne
+25
View File
@@ -342,6 +342,31 @@ under `instructions/dev/` (never present in a distributed instance - `tools/CONT
explains why). explains why).
<!-- dist:strip-end --> <!-- dist:strip-end -->
### MCP read server (optional)
The terminal is not the only way in. `tools/chemenu/mcp/` serves the same wiki read-only over
MCP - `search`, `types`, `describe_type`, `lint` and `status` - so a consumer that is not a
shell on this machine can ask the same questions and get the same answers. Literally the same:
the CLI and the server are two adapters over one core, and a golden test holds their output
together rather than trusting that it agrees.
There is no tool that writes, and not because one is filtered out of a list: the server imports
nothing under `chemenu/commands/`, so `new`, `publish` and the rest are unreachable from it.
Every answer carries the commit it was computed from, so a checkout that has fallen behind
produces a visibly stale answer instead of a confident wrong one.
```bash
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp # stdio
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp \
--transport streamable-http --host 0.0.0.0 --port 8000 # deployed
```
The dependency is deliberately not in `requirements.txt`: an instance that only uses the CLI
should not have to install a web stack to do it. Running it, keeping its checkout current, and
where authentication belongs (in front of the process, not in it) are in
[`instructions/mcp-read-server.md`](instructions/mcp-read-server.md).
### Obsidian ### Obsidian
Open this directory in Obsidian for: Open this directory in Obsidian for:
+1 -1
View File
@@ -1 +1 @@
2.1.0 2.4.0
+45 -4
View File
@@ -22,10 +22,11 @@ Read the exit code first - it says which of these applies:
## Exit 42: user clearance required ## Exit 42: user clearance required
A `wikitool` command that exits **42** is not reporting an error. It is refusing to act until a A `wikitool` command that exits **42** is not reporting an error. It is refusing to act until a
human has *read its output*. Two gates use it today - the Mass-Update Gate (`publish`, on a human has *read its output*. Three gates use it today - the Mass-Update Gate (`publish`, on a
change touching 10 or more counted files) and the rebase-review gate (`sync` and `publish`, on change touching 10 or more counted files), the rebase-review gate (`sync` and `publish`, on
a rebase whose incoming commits touch a file this session is also changing) - but the rule is a rebase whose incoming commits touch a file this session is also changing), and the
about the exit code, not the command: Publish-Remote Gate (`publish`, on a push to a target this checkout has not declared) - but the
rule is about the exit code, not the command:
> **Copy the command's output into your reply - the substance of it, not a description of it - > **Copy the command's output into your reply - the substance of it, not a description of it -
> and stop.** Run no further commands in that turn. > and stop.** Run no further commands in that turn.
@@ -62,6 +63,46 @@ clearance.
Background: [[Mass-Update Gate]] (`kb/concepts/Mass-Update Gate.md`). Background: [[Mass-Update Gate]] (`kb/concepts/Mass-Update Gate.md`).
### Publish-Remote Gate
The Mass-Update Gate asks whether a change is too large to publish. This one asks the question
underneath it: **whether this is the right repository to publish to at all.**
A checkout that holds private content usually has two remotes - its own, and the public upstream
it takes stack updates from. Git does not distinguish them at push time, so one wrong `--remote`
puts a private corpus on a public repository. That is not cheaply reversible: a force-push moves
the branch, but the objects stay fetchable by SHA until someone expires the server's reflogs and
runs `git gc --prune=now` on the bare repo.
`.wikitool-remotes.json` names the push URLs a checkout permits:
```json
{ "schema": 1, "allowed_push_urls": ["ssh://git@example.net:22/you/your-wiki.git"] }
```
It pins **URLs, not remote names** - a name-based list would wave through a `publish` whose
`origin` had been repointed, which is the failure it exists to catch. It reads the remote's
`pushurl` when one is set, because that is where `git push` actually writes.
The file is per-checkout and gitignored, for the same reason `ENVIRONMENT.md` is: two clones push
to two different places, so a committed copy would tell a private clone that the public upstream
is a legitimate target for its own content. **Absent means unrestricted** - a single-remote
checkout with nothing private in it has nothing to protect, and `doctor` reports which state a
checkout is in, WARNing only when there is more than one remote and no allowlist. A malformed
file is an error rather than "no restriction": a corrupted safeguard must not read as a disabled
one.
**This gate has no `--confirm` token, on purpose.** The other two clear with a token because the
question they ask ("is this change right?") is one the agent can put to the user and the user can
answer for that one changeset. This one asks "does this content belong to that repository?", which
is a standing property of the checkout, not a per-push judgment. The way past it is for the user
to add the URL to the file. **An agent must never edit `.wikitool-remotes.json` to get past a
refusal** - that is opening a gate on your own initiative, which AGENTS.md invariant 6 forbids.
The setup this gate exists for - a private instance that takes stack updates from a public
upstream - is [private-instance.md](private-instance.md). Step 4 there arms it, deliberately
*before* the first `publish`: added afterwards it leaves open exactly the window it closes.
## Iteration Budget Gate and loop-breaker ## Iteration Budget Gate and loop-breaker
Every `wikitool` call is counted per session. Calls are refused past **60 in a session**, or Every `wikitool` call is counted per session. Calls are refused past **60 in a session**, or
+108
View File
@@ -0,0 +1,108 @@
---
type: types/instruction.md
name: mcp-read-server
description: Run and keep current the MCP read server that serves this wiki to a second consumer
---
# Running the MCP read server
Chemenu has a second consumer. `search`, `types`, `describe_type`, `lint` and `status` are
served over MCP to callers that are not this terminal - the CLI and the server are two adapters
over one core (`chemenu.api.Corpus`), not a CLI with a network interface bolted on.
This document is about *operating* it: how to start it, what has to be true of the checkout it
serves, and how that checkout stays current. What it exposes and why is in
[tools/CONTRACT.md](../tools/CONTRACT.md) and in the module's own docstring
(`tools/chemenu/mcp/server.py`).
**Deployment is deliberately not here.** Which cluster, which ingress host, where the credential
lives - that is private infrastructure and this is a public repository. What is here is
everything an operator needs that is *true of the software* rather than of one installation.
## When to run
- Standing something up for a consumer that is not a terminal on this machine.
- Diagnosing an answer that looks stale, or one that disagrees with the CLI.
- Before pointing a new consumer at an existing server.
## Steps
1. **Install the server's dependency.** It is deliberately not in `requirements.txt`: an
instance that only uses the CLI should not be made to install pydantic, starlette, uvicorn
and cryptography to do it.
```bash
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
```
2. **Decide which checkout it serves.** The root resolves by precedence - an explicit `--root`,
then `$CHEMENU_ROOT`, then the checkout the package lives in. A deployment points at its
corpus with one variable and no code:
```bash
export CHEMENU_ROOT=/srv/chemenu
```
3. **Take tracing out of the served tree.** The server refuses to start otherwise, and the
refusal is the point: telemetry defaults to on and writes under `reports/telemetry/` inside
the repo, which step 5's sync is entitled to wipe. Either is fine:
```bash
export WIKI_TRACE=0 # off
export WIKI_TRACE_DIR=/var/log/chemenu # or elsewhere, outside the corpus
```
4. **Start it on the transport that matches what is in front of it.**
```bash
tools/.venv/bin/python -m chemenu.mcp # stdio
tools/.venv/bin/python -m chemenu.mcp --transport streamable-http # deployed
```
`stdio` is for developing and testing without a network - one process per consumer, started
locally. `streamable-http` is what a deployed instance speaks, and the only one the
authentication middleware can sit in front of, because that middleware is an HTTP reverse
proxy. `sse` is reachable through the SDK and deliberately not offered: it is the superseded
remote transport, and building on it now only moves the migration later.
5. **Keep the checkout current by polling, and keep it clean.**
```bash
git -C "$CHEMENU_ROOT" fetch --quiet origin && \
git -C "$CHEMENU_ROOT" reset --hard --quiet origin/main
```
Every few minutes, from a timer beside the server. Polling rather than a webhook on purpose:
it needs no inbound endpoint and no signature checking, which is a smaller surface than the
thing it would optimize. A webhook is a later optimization, not a starting point.
`reset --hard` is load-bearing, not a convenience. The corpus cache reuses a parse while the
commit is unchanged and **refuses to cache a dirty tree at all**, so a checkout that has
drifted answers correctly but reparses on every request - and every answer it gives is
stamped `"commit": null`, because a dirty tree corresponds to no revision.
## Decision points
- **An answer looks stale?** Read `commit` in the response. If it names an old revision, the
sync is not running. If it is `null`, the served tree has uncommitted changes - something is
writing into the corpus that should not be.
- **The server disagrees with `wikitool` on the same query?** That is a defect, not a
configuration difference: the two go through the same functions and a golden test holds their
output together (`tools/chemenu/tests/test_mcp_server.py`). Check first that both are pointed
at the same root - `CHEMENU_ROOT` is easy to set for one and not the other.
- **Asked to expose a write tool?** There is none, and the way to add one is not a flag. The
server imports nothing under `chemenu.commands`, so `new`, `touch`, `xref`, `cite`, `publish`
and `migrate` are unreachable from it rather than filtered out of a list. Submitting documents
from outside is a different design with a quarantine in it - Gitea #32 - not a tool added
here.
- **Asked to rate-limit inside the server?** Rate limiting belongs in the middleware in front of
the process, next to authentication. Not the Iteration Budget Gate: that exists to stop an
agent *session* from iterating unnoticed over the wiki's state, which is why retrieval is
exempt from it, and using it as a rate limiter would dilute it into one.
## Scope
Not for setting up an instance ([setup-instance.md](setup-instance.md)) or a fresh clone
([bootstrap.md](bootstrap.md)). Not for the authentication or rate-limiting middleware, which is
infrastructure configuration rather than part of this repository. Not a write path: see the
decision point above.
+171
View File
@@ -0,0 +1,171 @@
---
type: types/instruction.md
name: private-instance
description: Set up a private working instance as a clone of a public upstream, so stack updates arrive by merge instead of by copying a tarball over the tree.
---
# Set up a private instance against a public upstream
The distribution path in [setup-instance.md](setup-instance.md) builds an instance from a
`dist export` tarball, with no git ancestry in common with the repo it came from. That is the
right shape for someone who only ever *consumes* the stack.
This is the other shape: a private instance that keeps taking stack changes from a public
upstream, and whose own content must never travel back. It costs one safeguard to set up and
saves the whole update procedure afterwards.
**Read this before, not after, the first `publish`.** The gate in step 4 is the thing that makes
the arrangement safe, and adding it later means the window it closes was open in between.
## Why a clone rather than a tarball
`INSTALL.md`'s "Eine Instanz aktualisieren" is `cp -r` as an upgrade strategy: copy `tools/`,
`types/`, `instructions/`, `AGENTS.md`, `VERSION` over the existing tree. It has no three-way
merge, so it cannot notice that the receiving instance changed a file, and it has no conflict
surface, so nobody learns when upstream and local both touched the same one. It overwrites
silently.
A clone gets all of that from git. Stack changes land as real merges, with real conflicts where
they conflict.
**What a plain `git merge` does *not* give you is protection from the upstream's content.** The
private `main` deletes the demo corpus once, but that deletion does not make later upstream
changes to those paths go away. Measured, not assumed:
| Upstream does | `git merge upstream/main` does |
|---|---|
| modifies a page you deleted | `CONFLICT (modify/delete)` - and **leaves the upstream version in your working tree**. Resolve it with `git add -A` and the demo page is back. |
| adds a new page | stages it **silently**. No conflict, no prompt, no mention. |
| deletes a page you also deleted | nothing. The only harmless case. |
The middle row is the one that matters, because nothing announces it. An upstream that ships a
demo corpus *and* uses it as a test bed will add pages, and each one arrives in your instance
and starts showing up in your `lint`, your `index`, your `search` and your `confidence decay`.
So the merge has to be scoped. That is the procedure below, and it is not optional.
## Steps
1. **Clone, and name the two remotes for what they are.**
```bash
git clone <private-repo-url> my-wiki
cd my-wiki
git remote add upstream <public-repo-url>
```
`origin` is yours and is the only thing you ever push to. `upstream` is where stack updates
come from and is fetch-only.
2. **Make the fetch-only half fetch-only in git, too.**
```bash
git remote set-url --push upstream no_push
```
git refuses to push to a URL it cannot resolve. This is a convenience, not the safeguard -
step 4 is the safeguard.
3. **Delete the upstream's demo corpus once, on your own `main`.**
Everything under `kb/` and `raw/` that came with the clone is the upstream's content, not
yours. Remove it with `wikitool rm --page` (never `rm -rf`: `rm` de-links each page from the
rest of the wiki, and a plain delete leaves dead wikilinks and broken citations behind), then
`index rebuild`, `sources rebuild-index`, `lint`.
This is a one-time cut. Afterwards the upstream corpus is frozen from your side, which is
what makes later merges content-free.
4. **Arm the Publish-Remote Gate — before the first `publish`.**
```bash
cat > .wikitool-remotes.json <<'EOF'
{ "schema": 1, "allowed_push_urls": ["<your-private-push-url>"] }
EOF
```
Use the URL `git remote get-url --push origin` prints, exactly. `publish` refuses with exit
42 for anything else, and there is no flag that opens it - see [gates.md](gates.md).
The file is gitignored, so it stays with this checkout and never travels to the upstream.
`wikitool doctor` reports whether the gate is armed, and WARNs at more than one remote
without it.
5. **Take away the write credential, if you can.** A token or deploy key for `origin` only,
with no write access to the upstream, is the one control that holds even if everything above
is misconfigured. Belt and braces.
6. **Personalize and bootstrap.** `USER.md`, `SOUL.md` and optionally `ENVIRONMENT.md` are
yours and unrelated to the upstream's - see the Personalization step of
[setup-instance.md](setup-instance.md), then [bootstrap.md](bootstrap.md) for the venv and
the skills.
## Taking a stack update
Take the machinery, never the content. The merge is held open, the content stages are forced
back to your own state, and only then does it close:
```bash
BEFORE=$(git rev-parse HEAD)
git fetch upstream
# --no-commit holds the merge open; it may report conflicts under kb/ or raw/,
# which the next three lines are about to make irrelevant.
git merge --no-commit --no-ff upstream/main || true
# Whatever the merge did to the content stages, undo it. HEAD is still your
# pre-merge commit while the merge is open, so this restores exactly your side.
git rm -rq --cached --ignore-unmatch kb raw
rm -rf kb raw
git checkout HEAD -- kb raw
git commit --no-edit
```
Then **check that it worked**, rather than trusting that it did:
```bash
git diff --name-only $BEFORE HEAD -- kb raw # must print nothing
```
An empty result is the proof that the update touched machinery only. A non-empty one means a
path slipped through - inspect it before going further.
Then, as after any stack change: `doctor`, `docs verify`, `instructions verify`, `migrate status`,
`lint`. A `migrate status` with outstanding links means the update crossed a compatibility
boundary - follow [migrate-corpus.md](migrate-corpus.md) before doing anything else.
**Why not just `git merge upstream/main`?** Because of the table above: a page the upstream
*adds* arrives with no conflict and no message. You would find out when `lint` starts reporting
pages you never wrote - if you noticed at all.
## Where stack development happens
**In the public repo, not here.** That is not a preference; the stack is built that way. The
development-only half of the instruction layer is pruned from a distribution one-way, with no
command that reconstructs it, so an instance built this way has no tool-development mode to
switch into in the first place.
When a tool bug blocks real content work here - and it will - file the issue against the public
repo (an MCP server or the web UI reaches it from any session; no shared history needed), fix it
there where the tests, `docs verify` and CI's version gate live, and take the fix back with the
merge above. Nothing is lost by the detour: the fix has to pass that CI either way.
## Decision points
- **Merge conflict in `kb/` or `raw/`?** Expected, and already handled: the update procedure
above overwrites those stages with your own afterwards, so the conflict resolves itself.
Never resolve one by hand with `git add -A` - that is exactly how the upstream version, which
git left sitting in your working tree, gets committed into your instance.
- **`git diff` after the merge shows something under `kb/` or `raw/`?** Stop. The scoping step
did not take. Do not publish; find out which path came through and where from.
- **Conflict in `tools/`, `types/` or `instructions/`?** You changed the stack locally, which
step "Where stack development happens" says not to do. Take the upstream side and re-file the
change as an issue there.
## Scope
Not for a first instance with no upstream - that is [setup-instance.md](setup-instance.md). Not
for a fresh clone of a repo you already own and develop in - that is
[bootstrap.md](bootstrap.md). This is specifically the two-remote case, where the cost of a
mistaken push is disclosure rather than inconvenience.
@@ -0,0 +1,73 @@
---
type: types/concept.md
concept_type: decision
tags: []
created: 2026-09-01
modified: 2026-09-01
related: [Chemenu]
sources: ['Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]
confidence: 0.90
confidence_base: 0.70
provenance: sourced
summary: 'Private Korpusinhalte per Loeschung entfernen statt zu anonymisieren: ein Seitentitel ist der einzige Identifier eines Wikis, Umbenennen ist die volle page-lifecycle-Prozedur je Seite, Loeschen ist ein unterstuetztes Kommando.'
---
# Delete Rather Than Anonymize
**Typ:** Decision
## Definition
Wenn private oder sensible Inhalte aus einem Wiki entfernt werden müssen, ist Löschen einer
zugehörigen Seite in der Regel dem Anonymisieren (Umbenennen, Ersetzen sensibler Details bei
sonst unverändertem Inhalt) vorzuziehen - wenn ein unterstütztes Löschkommando existiert.
## Kernpunkte
- Ein Seitentitel ist in einem verlinkten Wiki oft der **einzige Identifier** einer Seite: Er
lebt in Wikilinks, Zitatmarkern und Frontmatter-Arrays jeder referenzierenden Seite. Ihn zu
ändern (Anonymisieren durch Umbenennen) verlangt deshalb eine vollständige Rename-Prozedur pro
betroffener Seite - bei mehreren zusammenhängenden Seiten multipliziert sich der Aufwand.
- Löschen dagegen ist ein einzelner, unterstützter Vorgang, der eine Seite mechanisch aus dem
Rest des Wikis de-linkt (bekannte Referenzarten: Frontmatter-Felder, ganzzeilige
Verweis-Aufzählungen). Er ist damit für strukturelle Bereinigung **schneller und weniger
fehleranfällig** als Anonymisierung.
- Bei Inhalten, die eine reale Topologie beschreiben (z. B. eine Infrastrukturdokumentation),
entschärft Anonymisieren einzelner Bezeichner (Hostnamen, IP-Adressen) die eigentliche
Preisgabe nicht: Die Struktur - welche Systeme wie zusammenhängen - bleibt erhalten, auch wenn
die Namen ausgetauscht sind.
- **Grenze der Methode:** Ein mechanisches Löschkommando entfernt typischerweise nur
strukturelle Referenzen (Frontmatter, Aufzählungen), nicht zwingend Erwähnungen im Fließtext
einer anderen Seite. Nach der Löschung ist eine gezielte Nachkontrolle nötig, ob der entfernte
Name noch im Klartext irgendwo im Wiki steht.
## Wann zu verwenden
- Der zu entfernende Inhalt ist als eigenständige Seite oder eigenständige Seitengruppe
abgrenzbar.
- Ein Löschkommando existiert, das Referenzen mechanisch bereinigt (nicht ein bloßes Entfernen
der Datei, das tote Links hinterlässt).
- Der Inhalt beschreibt eine reale, zusammenhängende Struktur (Infrastruktur, ein Netzwerk, eine
Organisation), bei der einzelne Bezeichner austauschen die eigentliche Preisgabe nicht behebt.
## Wann NICHT zu verwenden
- Wenn nur ein einzelner sensibler Wert innerhalb einer sonst wertvollen, generischen Seite
steht (z. B. ein Firmenname als Beispiel in einer sonst allgemeingültigen Anleitung) - dort ist
gezieltes Redigieren der Seite treffender als sie komplett zu verwerfen.
- Wenn die Seite Beziehungen trägt, die für sich genommen wertvoll und nicht sensibel sind - dann
kann eine Neufassung mit generischem Beispiel sinnvoller sein als Löschung.
## Verwandte Concepts
- [[Mass-Update Gate]]
## Beziehungen
- **gilt fuer:** [[Chemenu]]
## Siehe auch
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
- [[Chemenu]]
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
@@ -0,0 +1,71 @@
---
type: types/concept.md
concept_type: decision
tags: []
created: 2026-09-01
modified: 2026-09-01
related: [Chemenu]
sources: ['Source - Public Release, Corpus Purge and History Squash Session 2026-09-01']
confidence: 0.50
confidence_base: 0.70
provenance: sourced
summary: Ein Repo mit Code- und Inhaltsanteil erhaelt zwei Lizenzen; die Grenze zwischen ihnen ist kein zweiter, gepflegter Pfadkatalog, sondern der ohnehin vorhandene Dateiplan des Distributionswerkzeugs.
---
# Dual Licensing by File Plan
**Typ:** Decision
## Definition
Ein Repository, das sowohl Werkzeug-Code als auch inhaltliches Material (Dokumentation, Daten,
kompiliertes Wissen) enthält, bekommt zwei Lizenzdateien statt einer - eine für den Code, eine
für den Inhalt. Welche Datei zu welcher Lizenz gehört, wird nicht in einer eigenen, zweiten
Liste festgehalten, sondern aus dem Dateiplan abgeleitet, den ein vorhandenes
Distributions-/Build-Werkzeug ohnehin pflegt.
## Kernpunkte
- Der naheliegende Fehler ist, die Grenze zwischen „Code" und „Inhalt" als eigene, gepflegte
Aufzählung von Pfaden in der Lizenzdatei selbst festzuschreiben. Das ist eine zweite Kopie
einer Regel, die bereits an anderer Stelle existiert (dem Dateiplan des Build-/
Distributionswerkzeugs) - und die Kopie, die driftet, wenn sich Verzeichnisse verschieben.
- Stattdessen verweist die Lizenz-Notiz auf den bestehenden Plan (z. B. eine Funktion, die
berechnet, was in eine Distribution exportiert wird und was nicht) als **einzige** Quelle der
Wahrheit für die Grenze.
- Welche der beiden Lizenzen den generischen Dateinamen `LICENSE` trägt, ist keine
Nebensächlichkeit: Es sollte die Lizenz sein, die ein Code-Hosting-Dienst (Forge) für das
Repository insgesamt meldet - typischerweise die restriktivere/Copyleft-Lizenz. Ein Leser, der
eine Copyleft-Pflicht übersieht, wird dadurch geschädigt; wer eine Pflicht zu viel annimmt,
nicht.
- Ein Distributions-Export, der Code unter einer Copyleft-Lizenz ausliefert, muss die
zugehörige Lizenzdatei zwingend mitliefern (nicht optional, nicht still übersprungen, wenn sie
fehlt) - sonst ist die exportierte Instanz eine Lizenzverletzung, sobald sie veröffentlicht
wird.
## Wann zu verwenden
- Ein Repository trägt sowohl Software-/Werkzeugcode als auch Inhalt mit eigenem
Urheberrechtscharakter (Dokumentation, Wissensbasis, Daten), für die unterschiedliche Lizenzen
angemessen sind.
- Es existiert bereits ein Werkzeug, das programmatisch entscheidet, welche Dateien zu welcher
Kategorie gehören (z. B. für einen Export- oder Build-Schritt).
## Wann NICHT zu verwenden
- Bei einem Repository, dessen Inhalt untrennbar mit dem Code verwoben ist und für das keine
separate, maschinell nachvollziehbare Grenze existiert - dort wäre die Lizenz-Zuordnung selbst
wieder eine unabhängige, drift-anfällige Liste.
## Verwandte Concepts
- [[Chemenu]]
## Beziehungen
- **gilt fuer:** [[Chemenu]]
## Siehe auch
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
- [[Chemenu]]
+4 -1
View File
@@ -2,7 +2,7 @@
# kb/concepts/ - Index # kb/concepts/ - Index
76 page(s). Regenerated by `wikitool index rebuild`. 79 page(s). Regenerated by `wikitool index rebuild`.
## All ## All
@@ -25,9 +25,11 @@
| [[CPPC]] | protocol | Hardwareschnittstelle Collaborative Processor Performance Control für feingranulares CPU-Power-Management zwischen Betriebssystem und AMD-Prozessor. | 2026-08-29 | | [[CPPC]] | protocol | Hardwareschnittstelle Collaborative Processor Performance Control für feingranulares CPU-Power-Management zwischen Betriebssystem und AMD-Prozessor. | 2026-08-29 |
| [[Cross-platform Agent Skills]] | architecture | Architektur fuer Agent-Skills, die ueber mehrere LLM-Werkzeuge hinweg funktionieren; in Chemenu selbst am 2026-08-04 umgesetzt und ueberprueft | 2026-09-01 | | [[Cross-platform Agent Skills]] | architecture | Architektur fuer Agent-Skills, die ueber mehrere LLM-Werkzeuge hinweg funktionieren; in Chemenu selbst am 2026-08-04 umgesetzt und ueberprueft | 2026-09-01 |
| [[Crystallization]] | workflow | Verdichten abgeschlossener Erkundungen, Debugging-Sitzungen und Recherchen zu strukturierten Wiki-Auszügen als eigenständige Wissensquellen. | 2026-08-29 | | [[Crystallization]] | workflow | Verdichten abgeschlossener Erkundungen, Debugging-Sitzungen und Recherchen zu strukturierten Wiki-Auszügen als eigenständige Wissensquellen. | 2026-08-29 |
| [[Delete Rather Than Anonymize]] | decision | Private Korpusinhalte per Loeschung entfernen statt zu anonymisieren: ein Seitentitel ist der einzige Identifier eines Wikis, Umbenennen ist die volle page-lifecycle-Prozedur je Seite, Loeschen ist ein unterstuetztes Kommando. | 2026-09-01 |
| [[Denylist over Allowlist]] | decision | Entscheidung, schreibbare Felder als Schema minus kurzer Sperrliste zu bestimmen statt als gepflegte Positivliste, weil die Positivliste eine zweite Kopie des Schemas waere | 2026-08-31 | | [[Denylist over Allowlist]] | decision | Entscheidung, schreibbare Felder als Schema minus kurzer Sperrliste zu bestimmen statt als gepflegte Positivliste, weil die Positivliste eine zweite Kopie des Schemas waere | 2026-08-31 |
| [[Detect-Repair Asymmetry]] | problem | Werkzeugluecke, in der ein Check einen Defekt zuverlaessig meldet, aber kein Befehl ihn behebt - womit die Handeditierung der einzige verbleibende Ausweg ist | 2026-08-31 | | [[Detect-Repair Asymmetry]] | problem | Werkzeugluecke, in der ein Check einen Defekt zuverlaessig meldet, aber kein Befehl ihn behebt - womit die Handeditierung der einzige verbleibende Ausweg ist | 2026-08-31 |
| [[Diff-Reviewable Agent Edits]] | decision | Entscheidung, Dateiaenderungen ueber Edit/Write statt ueber Shell-Heredocs zu fahren, weil nur das erste eine pruefbare Diff hinterlaesst | 2026-08-31 | | [[Diff-Reviewable Agent Edits]] | decision | Entscheidung, Dateiaenderungen ueber Edit/Write statt ueber Shell-Heredocs zu fahren, weil nur das erste eine pruefbare Diff hinterlaesst | 2026-08-31 |
| [[Dual Licensing by File Plan]] | decision | Ein Repo mit Code- und Inhaltsanteil erhaelt zwei Lizenzen; die Grenze zwischen ihnen ist kein zweiter, gepflegter Pfadkatalog, sondern der ohnehin vorhandene Dateiplan des Distributionswerkzeugs. | 2026-09-01 |
| [[Entity Extraction]] | pattern | Erkennen und Strukturieren von Entities (Personen, Projekte, Bibliotheken, Concepts, Dateien, Entscheidungen, Systeme, Werkzeuge) samt typspezifischer Attribute aus Rohquellen. | 2026-08-29 | | [[Entity Extraction]] | pattern | Erkennen und Strukturieren von Entities (Personen, Projekte, Bibliotheken, Concepts, Dateien, Entscheidungen, Systeme, Werkzeuge) samt typspezifischer Attribute aus Rohquellen. | 2026-08-29 |
| [[Episodic Memory]] | architecture | Speicherschicht für verdichtete Sitzungszusammenfassungen und Befunde; Brücke zwischen rohem Working Memory und langlebigem Semantic Memory. | 2026-08-29 | | [[Episodic Memory]] | architecture | Speicherschicht für verdichtete Sitzungszusammenfassungen und Befunde; Brücke zwischen rohem Working Memory und langlebigem Semantic Memory. | 2026-08-29 |
| [[Event-Driven Automation]] | workflow | Muster, das automatische Auslöser an Wiki-Lebenszyklusereignisse hängt, um manuellen Pflegeaufwand und das Risiko der Verwahrlosung zu senken. | 2026-08-29 | | [[Event-Driven Automation]] | workflow | Muster, das automatische Auslöser an Wiki-Lebenszyklusereignisse hängt, um manuellen Pflegeaufwand und das Risiko der Verwahrlosung zu senken. | 2026-08-29 |
@@ -58,6 +60,7 @@
| [[Personalization Plane]] | architecture | Schicht fuer Instanz-Identitaet: USER.md/SOUL.md werden als Template ausgeliefert, im Setup-Interview woertlich befuellt und vom doctor-Check auf fehlend wie unbefuellt geprueft | 2026-08-31 | | [[Personalization Plane]] | architecture | Schicht fuer Instanz-Identitaet: USER.md/SOUL.md werden als Template ausgeliefert, im Setup-Interview woertlich befuellt und vom doctor-Check auf fehlend wie unbefuellt geprueft | 2026-08-31 |
| [[Privacy and Governance]] | workflow | Rahmenwerk zur Absicherung von Wiki-Inhalten über Datenfilterung beim Ingest, Audit-Trail-Protokollierung und umkehrbare Massenoperationen. | 2026-08-29 | | [[Privacy and Governance]] | workflow | Rahmenwerk zur Absicherung von Wiki-Inhalten über Datenfilterung beim Ingest, Audit-Trail-Protokollierung und umkehrbare Massenoperationen. | 2026-08-29 |
| [[Procedural Memory]] | architecture | Langlebigste Speicherschicht für Abläufe, Muster, bewährte Vorgehensweisen und Rezepte, gewonnen aus wiederholten semantischen Beobachtungen. | 2026-08-29 | | [[Procedural Memory]] | architecture | Langlebigste Speicherschicht für Abläufe, Muster, bewährte Vorgehensweisen und Rezepte, gewonnen aus wiederholten semantischen Beobachtungen. | 2026-08-29 |
| [[Publish-Remote Gate]] | workflow | Drittes, im Code durchgesetztes Gate: publish bricht mit Exit 42 ab, wenn die aufgeloeste Push-URL nicht in einer optionalen, gitignoreten Allowlist steht; anders als die anderen Gates gibt es keinen Freigabe-Token. | 2026-09-01 |
| [[Quality and Self-Correction]] | workflow | Automatische Qualitätssicherung für Wikis mit Inhaltsbewertung, Selbstheilung und Widerspruchserkennung. | 2026-08-29 | | [[Quality and Self-Correction]] | workflow | Automatische Qualitätssicherung für Wikis mit Inhaltsbewertung, Selbstheilung und Widerspruchserkennung. | 2026-08-29 |
| [[Quality Scoring]] | pattern | Quantitative Bewertung aller vom LLM geschriebenen Inhalte nach struktureller Qualität, Vollständigkeit der Quellenangaben, Konsistenz mit dem Wiki und Themenabdeckung. | 2026-08-29 | | [[Quality Scoring]] | pattern | Quantitative Bewertung aller vom LLM geschriebenen Inhalte nach struktureller Qualität, Vollständigkeit der Quellenangaben, Konsistenz mit dem Wiki und Themenabdeckung. | 2026-08-29 |
| [[RAG]] | architecture | Architekturmuster, bei dem LLMs die Generierung um Dokumente aus einer Wissensbasis anreichern. | 2026-08-29 | | [[RAG]] | architecture | Architekturmuster, bei dem LLMs die Generierung um Dokumente aus einer Wissensbasis anreichern. | 2026-08-29 |
+3 -1
View File
@@ -4,7 +4,7 @@ concept_type: workflow
tags: [gate, safety, mass-update, confirmation] tags: [gate, safety, mass-update, confirmation]
created: 2026-08-03 created: 2026-08-03
modified: 2026-09-01 modified: 2026-09-01
related: [Content Quality Control, wikitool, Iteration and Cost Limits, Structural Enforcement over Documented Rule, Bulk Operations] related: [Content Quality Control, wikitool, Iteration and Cost Limits, Structural Enforcement over Documented Rule, Bulk Operations, Publish-Remote Gate]
sources: [Source - LLM Improvements Sonnet Analysis, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - LLM Improvements Codex Analysis] sources: [Source - LLM Improvements Sonnet Analysis, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - LLM Improvements Codex Analysis]
confidence: 0.88 confidence: 0.88
confidence_base: 0.88 confidence_base: 0.88
@@ -77,6 +77,7 @@ Das Mass-Update Gate ist ein Sicherheitsmechanismus, der die Ausführung pausier
- **wird gespiegelt durch:** [[Iteration and Cost Limits]] - **wird gespiegelt durch:** [[Iteration and Cost Limits]]
- **wendet an:** [[Structural Enforcement over Documented Rule]] - **wendet an:** [[Structural Enforcement over Documented Rule]]
- **grenzt ab gegen:** [[Bulk Operations]] - **grenzt ab gegen:** [[Bulk Operations]]
- **verwandtes Gate:** [[Publish-Remote Gate]]
## Siehe auch ## Siehe auch
@@ -86,6 +87,7 @@ Das Mass-Update Gate ist ein Sicherheitsmechanismus, der die Ausführung pausier
- [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]] - [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]]
- [[Structural Enforcement over Documented Rule]] - [[Structural Enforcement over Documented Rule]]
- [[Bulk Operations]] - [[Bulk Operations]]
- [[Publish-Remote Gate]]
## Fußnoten ## Fußnoten
+106
View File
@@ -0,0 +1,106 @@
---
type: types/concept.md
concept_type: workflow
tags: []
created: 2026-09-01
modified: 2026-09-01
related: [Mass-Update Gate, Chemenu]
sources: [Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]
confidence: 0.50
confidence_base: 0.70
provenance: sourced
summary: 'Drittes, im Code durchgesetztes Gate: publish bricht mit Exit 42 ab, wenn die aufgeloeste Push-URL nicht in einer optionalen, gitignoreten Allowlist steht; anders als die anderen Gates gibt es keinen Freigabe-Token.'
---
# Publish-Remote Gate
**Typ:** Workflow
## Definition
Ein im Code durchgesetztes Gate, das einen Schreibvorgang (hier: `publish`) auf ein
deklariertes, per Datei zugelassenes Ziel beschränkt. Der Vorgang bricht ab, wenn das
aufgelöste Push-Ziel nicht in der Allowlist steht - unabhängig davon, unter welchem Namen der
Remote lokal konfiguriert ist.
## Kernpunkte
- **Geprüft wird die aufgelöste URL, nicht der Remote-Name.** Ein namensbasiertes Gate würde
ein `publish` durchlassen, dessen `origin` zwischenzeitlich auf ein anderes Ziel umgebogen
wurde - genau der Fall, den das Gate abfangen soll.
- **Kein Freigabe-Token, anders als vergleichbare Gates.** Ein Gate, dessen Frage per
Änderungssatz beantwortbar ist ("ist diese konkrete Änderung richtig?"), kann sich mit einem
Token lösen, den ein Mensch einmalig ausstellt. Ein Gate, dessen Frage eine stehende
Eigenschaft des Checkouts ist ("gehört dieser Inhalt grundsätzlich in dieses Ziel?"), sollte
keinen Token haben - der einzige Weg daran vorbei ist ein bewusster Edit der Konfigurationsdatei
durch den Menschen, nie ein automatisierter Bypass.
- **Die Allowlist-Datei ist per Checkout, nicht Teil des versionierten Inhalts.** Sie
beschreibt, wohin *dieser* Checkout schreiben darf - eine committete Kopie würde jedem Klon
dieselbe Erlaubnis unterschieben, unabhängig davon, ob sie für ihn zutrifft.
- **Fehlende Datei bedeutet unbeschränkt, kaputte Datei bedeutet Fehler.** Diese Unterscheidung
ist wichtig: Ein Checkout ohne Beschränkungsbedarf soll nicht gezwungen sein, eine leere
Konfigurationsdatei zu pflegen; eine beschädigte Datei darf aber nicht wie eine abwesende
behandelt werden, sonst wird eine defekte Sicherung zu einer stillschweigend abgeschalteten.
## Wann zu verwenden
- Ein Checkout kann an mehr als ein Remote-Ziel schreiben, und ein Schreibvorgang an das
falsche Ziel ist teuer oder nicht rückgängig zu machen (z. B. weil das Ziel öffentlich ist).
- Die Menge der zulässigen Ziele ist eine stabile Eigenschaft des Checkouts, keine
Einzelfallentscheidung pro Vorgang.
## Wann NICHT zu verwenden
- Wenn nur ein Remote existiert und kein Risiko einer Zielverwechslung besteht - dort ist die
Allowlist reine Formalität ohne Schutzwirkung.
- Für Entscheidungen, die tatsächlich pro Änderungssatz getroffen werden sollen (dafür ist ein
Token-basiertes Gate wie das Mass-Update-Gate das richtige Muster).
## Was das Gate nicht abdeckt: Inhalt, der über einen Merge hereinkommt
Das Gate schützt den **Push**, nicht den **Merge**. Ein Setup, bei dem eine private Instanz
Maschinerie von einem öffentlichen Upstream per `git merge upstream/main` zieht, hat ein
eigenes, empirisch geprüftes Problem: Ein einfacher Merge übernimmt Upstream-Änderungen an
bereits gelöschten Inhaltsseiten nicht sauber.
Gemessen an einem Wegwerf-Repo-Paar, bei dem der Upstream nach der einmaligen Löschung des
Demo-Korpus eine Seite ändert, eine neue anlegt und eine dritte löscht:
- Eine **geänderte** Seite erzeugt einen `modify/delete`-Konflikt und lässt die
Upstream-Fassung im Arbeitsbaum liegen - ein naives Auflösen mit `git add -A` holt sie zurück.
- Eine **neu angelegte** Seite wird **stillschweigend** übernommen, ohne Konflikt und ohne
Meldung.
- Eine beidseitig gelöschte Seite verursacht nichts - der einzige Fall, der ohne Weiteres
funktioniert.
Ein naheliegender Fix (`.gitattributes` mit `merge=ours` für die betroffenen Verzeichnisse)
wurde ebenfalls gemessen und verworfen: Der Treiber wirkt nur bei Inhaltskonflikten auf
beidseitig vorhandenen Dateien, nicht bei modify/delete-Paaren oder Neuanlagen.
Die funktionierende Prozedur hält den Merge mit `--no-commit` offen, erzwingt die
Inhaltsverzeichnisse zurück auf den Stand vor dem Merge, solange `HEAD` noch dorthin zeigt, und
prüft danach explizit (`git diff --name-only $BEFORE HEAD -- kb raw` muss leer sein) - eine
Kontrolle, die nicht stillschweigend übersprungen werden kann, anders als eine bloße Behauptung
im Text[^s-private-instance-merge-correction-and-issue-30-session-2026-09-01]. Details, ein
getestetes Skript und zwei Architekturvorschläge (das Verfahren als `wikitool`-Kommando bauen,
oder den Demo-Korpus grundsätzlich von dem Branch fernhalten, von dem private Instanzen ihre
Maschinerie ziehen) stehen in Gitea-Issue #30.
## Verwandte Concepts
- [[Mass-Update Gate]]
## Beziehungen
- **gilt fuer:** [[Chemenu]]
- **verwandtes Gate:** [[Mass-Update Gate]]
## Siehe auch
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
- [[Chemenu]]
- [[Mass-Update Gate]]
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
## Fußnoten
[^s-private-instance-merge-correction-and-issue-30-session-2026-09-01]: [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
+1 -1
View File
@@ -19,7 +19,7 @@
|------|------|---------|----------------| |------|------|---------|----------------|
| [[andybalholm-edl]] | project | Go-basierte EDL-Bibliothek für die Kommunikation mit eingebetteten Geräten. | 2026-08-29 | | [[andybalholm-edl]] | project | Go-basierte EDL-Bibliothek für die Kommunikation mit eingebetteten Geräten. | 2026-08-29 |
| [[BCDModule]] | project | Go-Modul, das die Entscheidungslogik für die Batterieladung umsetzt. | 2026-08-29 | | [[BCDModule]] | project | Go-Modul, das die Entscheidungslogik für die Batterieladung umsetzt. | 2026-08-29 |
| [[Chemenu]] | project | Persoenliches IT-Wissenswiki, gepflegt von LLM-Agenten; seit 1.0.0 versioniert, seit 1.1.0 Personalization Plane, seit 1.8.0 ENVIRONMENT.md, seit 2.0.0 unter dem Namen Chemenu (vorher llm-wiki-test1) mit dem Python-Paket chemenu | 2026-09-01 | | [[Chemenu]] | project | Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich als Testbett und Demo unter AGPL-3.0/CC-BY-4.0 | 2026-09-01 |
| [[goresponsiveness]] | project | Go-Werkzeug zur Messung von Anwendungsleistung und Responsiveness. | 2026-08-29 | | [[goresponsiveness]] | project | Go-Werkzeug zur Messung von Anwendungsleistung und Responsiveness. | 2026-08-29 |
| [[ha-core]] | project | Kern-Integrationsbibliothek für Home-Assistant-E3DC-Systeme; stellt die E3DC-Kommunikationsprotokolle und den Home-Assistant-Integrationscode bereit. | 2026-08-29 | | [[ha-core]] | project | Kern-Integrationsbibliothek für Home-Assistant-E3DC-Systeme; stellt die E3DC-Kommunikationsprotokolle und den Home-Assistant-Integrationscode bereit. | 2026-08-29 |
| [[hacs-e3dc]] | project | Home Assistant Custom Component zur Überwachung von E3DC-Energiesystemen. | 2026-08-29 | | [[hacs-e3dc]] | project | Home Assistant Custom Component zur Überwachung von E3DC-Energiesystemen. | 2026-08-29 |
+29 -5
View File
@@ -4,12 +4,12 @@ entity_type: project
tags: [wiki, llm, knowledge-base] tags: [wiki, llm, knowledge-base]
created: 2026-08-04 created: 2026-08-04
modified: 2026-09-01 modified: 2026-09-01
related: [Personalization Plane, Issue Label Scheme, Optional Instance Context File] related: [Personalization Plane, Issue Label Scheme, Optional Instance Context File, Delete Rather Than Anonymize, Dual Licensing by File Plan, Publish-Remote Gate]
sources: [Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31] sources: [Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, 'Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]
confidence: 0.90 confidence: 0.90
confidence_base: 0.90 confidence_base: 0.90
provenance: mixed provenance: mixed
summary: Persoenliches IT-Wissenswiki, gepflegt von LLM-Agenten; seit 1.0.0 versioniert, seit 1.1.0 Personalization Plane, seit 1.8.0 ENVIRONMENT.md, seit 2.0.0 unter dem Namen Chemenu (vorher llm-wiki-test1) mit dem Python-Paket chemenu summary: Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich als Testbett und Demo unter AGPL-3.0/CC-BY-4.0
--- ---
# Chemenu # Chemenu
@@ -17,7 +17,21 @@ summary: Persoenliches IT-Wissenswiki, gepflegt von LLM-Agenten; seit 1.0.0 vers
## Beschreibung ## Beschreibung
Chemenu ist das persönliche IT-Wissens-Wiki-Repository, das Gegenstand der Skill-Umstrukturierung ist, die in den Copilot Skill Restructure Instructions beschrieben wird[^s-copilot-skill-restructure-instructions]. Es hat derzeit eine monolithische `AGENTS.md`-Datei (~30KB), die fünf Workflows (INGEST, QUERY, LINT, CREATE, UPDATE), das Frontmatter-Schema des Wikis, Entity/Concept-Typ-Tabellen, Provenance/Citation-Regeln, Naming-Konventionen und eine Konfidenz-Scoring-Formel beschreibt[^s-copilot-skill-restructure-instructions]. Chemenu ist ein deterministischer Wissenskompiler: Rohnotizen unter `raw/` werden über eine
Schema- und Werkzeugschicht zu einem verlinkten, quellengebundenen Wiki unter `kb/` verdichtet.
Was mechanisch ist, macht `tools/wikitool`; was Urteil braucht, macht ein Agent unter Contracts,
deren Grenzen in Code durchgesetzt sind statt im Prompt.
**Diese Instanz ist seit dem 2026-09-01 öffentlich** und dient als Testbett und Demo, nicht als
produktive Wissensbasis. Ihr Korpus beschreibt überwiegend den Stack selbst - Gates, Lint,
Versionierung, Suche, das Wiki-Muster -, sodass die Distribution sich mit ihren eigenen Mitteln
dokumentiert. Lizenz: AGPL-3.0 für den Stack, CC-BY-4.0 für die Inhalte.
**Ausgangspunkt war eine monolithische Struktur.** Zum Zeitpunkt der Copilot Skill Restructure
Instructions lag alles in einer einzigen `AGENTS.md` (~30 KB): fünf Workflows (INGEST, QUERY,
LINT, CREATE, UPDATE), das Frontmatter-Schema, Entity- und Concept-Typtabellen,
Provenance- und Zitatregeln, Namenskonventionen und die Konfidenz-Formel[^s-copilot-skill-restructure-instructions].
Die Aufteilung in Skills und Contracts, die daraus folgte, ist seit 2026-08-04 abgeschlossen.
**Der Name.** Bis zum 2026-09-01 hieß das Projekt `llm-wiki-test1` - ein Arbeitstitel mit einer **Der Name.** Bis zum 2026-09-01 hieß das Projekt `llm-wiki-test1` - ein Arbeitstitel mit einer
Ordnungszahl darin, kein Name. **Chemenu** ist die deutsche Wikipedia-Schreibweise des Ordnungszahl darin, kein Name. **Chemenu** ist die deutsche Wikipedia-Schreibweise des
@@ -30,9 +44,10 @@ Das Repository hat bereits ein deterministisches CLI, `tools/wikitool` (Python,
## Kerndaten ## Kerndaten
- **Zweck:** Persönliche IT-Wissensbasis - **Zweck:** Deterministischer Wissenskompiler; diese Instanz ist Testbett und öffentliche Demo
- **Status:** Aktiv - Skill-Umstrukturierung abgeschlossen 2026-08-04 - **Status:** Aktiv - Skill-Umstrukturierung abgeschlossen 2026-08-04
- **Verantwortlich:** Torben - **Verantwortlich:** Torben
- **Lizenz:** AGPL-3.0 (Stack: `tools/`, `types/`), CC-BY-4.0 (Inhalte)
- **Repository:** `torben/chemenu` auf gitea.nehmer.net; bis 2026-09-01 `torben/llm-wiki-test1` - **Repository:** `torben/chemenu` auf gitea.nehmer.net; bis 2026-09-01 `torben/llm-wiki-test1`
- **Architektur:** Dreilagig: raw/ (Quelle), wiki/ (Wissen), tools/ (deterministisches CLI) - **Architektur:** Dreilagig: raw/ (Quelle), wiki/ (Wissen), tools/ (deterministisches CLI)
@@ -45,6 +60,9 @@ Das Repository hat bereits ein deterministisches CLI, `tools/wikitool` (Python,
- **verwendet:** [[Personalization Plane]] - **verwendet:** [[Personalization Plane]]
- **verwendet:** [[Issue Label Scheme]] - **verwendet:** [[Issue Label Scheme]]
- **verwendet:** [[Optional Instance Context File]] - **verwendet:** [[Optional Instance Context File]]
- **wendet an:** [[Delete Rather Than Anonymize]]
- **wendet an:** [[Dual Licensing by File Plan]]
- **setzt um:** [[Publish-Remote Gate]]
## Details ## Details
@@ -195,6 +213,12 @@ Repositorys selbst und keine Aussagen aus einer Rohdatenquelle; der Änderungsda
- [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]] - [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]]
- [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]] - [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]]
- [[Optional Instance Context File]] - [[Optional Instance Context File]]
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
- [[Delete Rather Than Anonymize]]
- [[Dual Licensing by File Plan]]
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
- [[Publish-Remote Gate]]
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
## Fußnoten ## Fußnoten
+21 -1
View File
@@ -5,7 +5,7 @@ tags: [cli, automation, deterministic, wiki-management]
created: 2026-08-03 created: 2026-08-03
modified: 2026-09-01 modified: 2026-09-01
related: [Semantic Lint Automation, Session Orientation, Iteration and Cost Limits, KB Stack Versioning, KB Migration, Personalization Plane, Detect-Repair Asymmetry, Write-Once Frontmatter Fields, Denylist over Allowlist, Command Round-Trip Integrity, Green Suite Blind Spot, Ambient Environment Dependency, Structural Enforcement over Documented Rule, Optional Instance Context File] related: [Semantic Lint Automation, Session Orientation, Iteration and Cost Limits, KB Stack Versioning, KB Migration, Personalization Plane, Detect-Repair Asymmetry, Write-Once Frontmatter Fields, Denylist over Allowlist, Command Round-Trip Integrity, Green Suite Blind Spot, Ambient Environment Dependency, Structural Enforcement over Documented Rule, Optional Instance Context File]
sources: [Source - LLM Improvements Codex Analysis, Source - LLM Improvements Sonnet Analysis, Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Auto Mode and Tool Choice Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31] sources: [Source - LLM Improvements Codex Analysis, Source - LLM Improvements Sonnet Analysis, Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Auto Mode and Tool Choice Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31, 'Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]
confidence: 0.90 confidence: 0.90
confidence_base: 0.90 confidence_base: 0.90
provenance: sourced provenance: sourced
@@ -147,6 +147,21 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
## Historie ## Historie
- 2026-09-01 - `2.1.0`-`2.2.2`: Vorbereitung und Durchführung der Veröffentlichung. `dist export`
bekommt `REQUIRED_ROOT_FILES` (fehlende Lizenzdateien lassen den Export scheitern statt still
weiterzulaufen) und `find_leaks()` (strukturelle Prüfung des fertigen Export-Plans gegen
Personalisierungsdateien, `instructions/dev/`, `kb/`- und `raw/`-Inhalte - bewusst kein
Text-Muster-Scan, weil der eigene Hostname legitim in `INSTALL.md`/`version.py` steht). `publish`
bekommt ein drittes Gate: das **Publish-Remote-Gate** prüft die aufgelöste Push-URL (nicht den
Remote-Namen) gegen eine optionale, gitignorete `.wikitool-remotes.json` und hat als einziges
der drei Gates keinen `--confirm`-Token - der Weg daran vorbei ist ein bewusster Edit der
Datei durch den Nutzer, nie durch einen Agenten. `doctor` bekommt den `publish-remotes`-Check.
Ein `raw_dir`-Testfixture löste `config.ROOT` gegen das echte Repo-Root statt die Fixture auf
und bestand nur, weil zufällig ein Verzeichnis existierte, das der Korpus-Schnitt entfernte -
in der Fixture geschlossen, nicht im einzelnen Test, aus demselben Grund wie Gitea #8.
Siehe [[Chemenu]], [[Mass-Update Gate]], [[Publish-Remote Gate]],
[[Delete Rather Than Anonymize]],
[[Dual Licensing by File Plan]][^s-public-release-corpus-purge-and-history-squash-session-2026-09-01][^s-publish-remote-gate-and-issue-triage-session-2026-09-01]
- 2026-09-01 - `2.0.0` (Commit `9a7abe6`, 730 Tests grün): das Python-Paket heißt `chemenu` - 2026-09-01 - `2.0.0` (Commit `9a7abe6`, 730 Tests grün): das Python-Paket heißt `chemenu`
statt `wiki_tools`, **das Kommando bleibt `wikitool`**. Der Import-Name eines Pakets ist ein statt `wiki_tools`, **das Kommando bleibt `wikitool`**. Der Import-Name eines Pakets ist ein
flacher globaler Namensraum ohne Kollisionsschutz, `wiki_tools` war dafür zu generisch; flacher globaler Namensraum ohne Kollisionsschutz, `wiki_tools` war dafür zu generisch;
@@ -249,6 +264,9 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
- [[Structural Enforcement over Documented Rule]] - [[Structural Enforcement over Documented Rule]]
- [[Optional Instance Context File]] - [[Optional Instance Context File]]
- [[Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31]] - [[Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31]]
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
## Fußnoten ## Fußnoten
@@ -261,3 +279,5 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
[^s-conversation-agents-md-skill-restructuring-session-2026-08-04]: [[Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04]] [^s-conversation-agents-md-skill-restructuring-session-2026-08-04]: [[Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04]]
[^s-conversation-hardening-the-test-suite-against-silent-environment-dependencies-session-2026-08-31]: [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]] [^s-conversation-hardening-the-test-suite-against-silent-environment-dependencies-session-2026-08-31]: [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]]
[^s-llm-improvements-codex-analysis]: [[Source - LLM Improvements Codex Analysis]] [^s-llm-improvements-codex-analysis]: [[Source - LLM Improvements Codex Analysis]]
[^s-public-release-corpus-purge-and-history-squash-session-2026-09-01]: [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
[^s-publish-remote-gate-and-issue-triage-session-2026-09-01]: [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
+5 -5
View File
@@ -13,11 +13,11 @@ The page tables live in a generated `INDEX.md` inside each collection, linked be
## Statistics ## Statistics
- **Total Pages:** 170 - **Total Pages:** 176
- **Comparisons:** 1 - **Comparisons:** 1
- **Concepts:** 76 - **Concepts:** 79
- **Entities:** 72 - **Entities:** 72
- **Sources:** 21 - **Sources:** 24
- **Last Updated:** 2026-09-01 - **Last Updated:** 2026-09-01
--- ---
@@ -27,9 +27,9 @@ The page tables live in a generated `INDEX.md` inside each collection, linked be
| Collection | Pages | Index | | Collection | Pages | Index |
|------------|------:|-------| |------------|------:|-------|
| `comparisons/` | 1 | [comparisons/INDEX.md](comparisons/INDEX.md) | | `comparisons/` | 1 | [comparisons/INDEX.md](comparisons/INDEX.md) |
| `concepts/` | 76 | [concepts/INDEX.md](concepts/INDEX.md) | | `concepts/` | 79 | [concepts/INDEX.md](concepts/INDEX.md) |
| `entities/` | 72 | [entities/INDEX.md](entities/INDEX.md) | | `entities/` | 72 | [entities/INDEX.md](entities/INDEX.md) |
| `sources/` | 21 | [sources/INDEX.md](sources/INDEX.md) | | `sources/` | 24 | [sources/INDEX.md](sources/INDEX.md) |
### entities/ ### entities/
+24
View File
@@ -49,3 +49,27 @@ gesetzt. Dieser Eintrag ist der erste des neuen Logs.
und `doctor` grün. und `doctor` grün.
--- ---
## [2026-09-01] update | Chemenu-Projektseite auf den oeffentlichen Stand gebracht
Die Seite beschrieb sich als persoenliches IT-Wissenswiki; seit der Veroeffentlichung ist diese Instanz Testbett und oeffentliche Demo. Beschreibung, Zweck und Lizenz nachgezogen, die historische Aussage ueber die monolithische AGENTS.md als Ausgangspunkt datiert statt geloescht.
---
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
Source-Seite angelegt, Chemenu und wikitool aktualisiert, zwei neue Concept-Seiten (Delete Rather Than Anonymize, Dual Licensing by File Plan), quer verlinkt.
---
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
Source-Seite angelegt, neue Concept-Seite Publish-Remote Gate, wikitool-Historie ergaenzt, quer verlinkt mit Mass-Update Gate und Chemenu.
---
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
Source-Seite angelegt, Publish-Remote-Gate-Seite um die gemessene Merge-Semantik erweitert, quer verlinkt.
---
+17 -2
View File
@@ -8,8 +8,8 @@ inline `[^cite-id]` footnote).
## Coverage Summary ## Coverage Summary
- **Total raw files:** 21 - **Total raw files:** 24
- **Covered:** 21 - **Covered:** 24
- **Uncovered:** 0 - **Uncovered:** 0
--- ---
@@ -76,6 +76,21 @@ inline `[^cite-id]` footnote).
- Covered by: [[Source - Conversation - Nightly Drift-Check Workflow and doctor's Bootstrap Gap Session 2026-08-31]] - Covered by: [[Source - Conversation - Nightly Drift-Check Workflow and doctor's Bootstrap Gap Session 2026-08-31]]
- Cited by: [[CI Integration]], [[Gitea Actions]] - Cited by: [[CI Integration]], [[Gitea Actions]]
### `raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md`
- Covered by: [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
- Cited by: [[Chemenu]], [[Delete Rather Than Anonymize]], [[Publish-Remote Gate]], [[wikitool]]
### `raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md`
- Covered by: [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
- Cited by: [[Chemenu]], [[Delete Rather Than Anonymize]], [[Dual Licensing by File Plan]], [[wikitool]]
### `raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md`
- Covered by: [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
- Cited by: [[Chemenu]], [[Publish-Remote Gate]], [[wikitool]]
### `raw/notes/Conversation Transcript - Two Round-Trip Defects Found by an Ingest Session 2026-08-31.md` ### `raw/notes/Conversation Transcript - Two Round-Trip Defects Found by an Ingest Session 2026-08-31.md`
- Covered by: [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]] - Covered by: [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]]
+4 -1
View File
@@ -2,7 +2,7 @@
# kb/sources/ - Index # kb/sources/ - Index
21 page(s). Regenerated by `wikitool index rebuild`. 24 page(s). Regenerated by `wikitool index rebuild`.
## All ## All
@@ -28,5 +28,8 @@
| [[Source - LLM Improvements Sonnet Analysis]] | notes | Sonnet-Analyse, die AGENTS.md und wikitool mit Farzas Gist und awesome-llm-wiki vergleicht und die Codex-Analyse um konkrete Empfehlungen zu Qualitätsschwellen, Stilrichtlinie, Auditrhythmus und Skalierung ergänzt | 2026-08-03 | | [[Source - LLM Improvements Sonnet Analysis]] | notes | Sonnet-Analyse, die AGENTS.md und wikitool mit Farzas Gist und awesome-llm-wiki vergleicht und die Codex-Analyse um konkrete Empfehlungen zu Qualitätsschwellen, Stilrichtlinie, Auditrhythmus und Skalierung ergänzt | 2026-08-03 |
| [[Source - LLM Wiki Pattern]] | article | Grundlegendes Muster für persönliche Wissensbasen mit LLMs: ein dauerhaftes Wiki schrittweise pflegen, statt es aus den Quellen neu herzuleiten. | 2026-07-26 | | [[Source - LLM Wiki Pattern]] | article | Grundlegendes Muster für persönliche Wissensbasen mit LLMs: ein dauerhaftes Wiki schrittweise pflegen, statt es aus den Quellen neu herzuleiten. | 2026-07-26 |
| [[Source - LLM Wiki v2]] | article | Erweitertes LLM-Wiki-Muster mit Praxiserfahrungen aus agentmemory zu Memory Lifecycle, Confidence Scoring, Wissensgraph und Automatisierung. | 2026-07-26 | | [[Source - LLM Wiki v2]] | article | Erweitertes LLM-Wiki-Muster mit Praxiserfahrungen aus agentmemory zu Memory Lifecycle, Confidence Scoring, Wissensgraph und Automatisierung. | 2026-07-26 |
| [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]] | notes | Sitzung, die eine ungeprueft niedergeschriebene Merge-Behauptung in private-instance.md durch einen empirischen Test widerlegt, die Prozedur korrigiert (2.2.1) und Issue #30 mit einem getesteten Skript sowie zwei Architekturvorschlaegen anlegt. | 2026-09-01 |
| [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]] | notes | Sitzung, die den Chemenu-Stack von einer privaten Testinstanz in ein oeffentliches Repo ueberfuehrt: Korpus geloescht statt anonymisiert, Git-History auf einen Commit gesquashed, AGPL-3.0/CC-BY-4.0-Dual-Lizenz gewaehlt, dist export um einen Leak-Canary gehaertet. | 2026-09-01 |
| [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]] | notes | Sitzung, die ein drittes, Token-loses Gate fuer publish baut, instructions/private-instance.md schreibt, sechs Gitea-Issues auf den Rename und die neue Architektur nachzieht und die Actions-Run-Historie entfernen laesst. | 2026-09-01 |
| [[Source - Wine]] | notes | Wine-Konfiguration für Arch Linux: pacman-NoExtract-Einstellungen und Bottles-Runtime-Optionen einschließlich Proton- und Lutris-Varianten. | 2026-08-01 | | [[Source - Wine]] | notes | Wine-Konfiguration für Arch Linux: pacman-NoExtract-Einstellungen und Bottles-Runtime-Optionen einschließlich Proton- und Lutris-Varianten. | 2026-08-01 |
@@ -0,0 +1,83 @@
---
type: types/source.md
source_type: notes
author: Torben
raw_files: [raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md]
source_language: de
date: 2026-09-01
tags: []
entities: [Chemenu, wikitool]
concepts: [Publish-Remote Gate, Delete Rather Than Anonymize]
summary: 'Sitzung, die eine ungeprueft niedergeschriebene Merge-Behauptung in private-instance.md durch einen empirischen Test widerlegt, die Prozedur korrigiert (2.2.1) und Issue #30 mit einem getesteten Skript sowie zwei Architekturvorschlaegen anlegt.'
---
# Source: Private-Instance Merge Correction and Issue 30 Session 2026-09-01
**Autor:** Torben
**Datum:** 2026-09-01
**Raw-Dateien:** raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
**Typ:** Notes
## Zusammenfassung
Torben fragte, was in der privaten Instanz nach dem beschriebenen Schema passiert, wenn Upstream
den Demo-Korpus ändert - eine Frage, die eine am selben Tag geschriebene, aber nie getestete
Behauptung in `instructions/private-instance.md` traf. Statt die bestehende Textstelle zu
verteidigen, wurde sie an einem Wegwerf-Repo-Paar empirisch geprüft: Ein `git merge
upstream/main` löst eine geänderte, gelöschte Demo-Seite **nicht** still auf, sondern erzeugt
einen `modify/delete`-Konflikt und lässt die Upstream-Fassung im Arbeitsbaum liegen; eine neu
angelegte Demo-Seite wird dagegen **stillschweigend** gestaged, ohne Konflikt und ohne Meldung.
Ein zweiter, naheliegender Fix (`.gitattributes` mit `merge=ours` für die Inhaltsverzeichnisse)
wurde ebenfalls getestet und ebenfalls widerlegt - der Treiber wirkt nur bei Inhaltskonflikten
auf beidseitig vorhandenen Dateien, nicht bei modify/delete oder Neuanlage.
Die Instruktion wurde korrigiert (Version 2.2.1): Der Merge wird mit `--no-commit` offengehalten,
die Inhaltsverzeichnisse werden auf den Stand vor dem Merge zurückgezwungen, solange `HEAD` noch
dorthin zeigt, erst dann wird committet - gefolgt von einer Kontrolle
(`git diff --name-only $BEFORE HEAD -- kb raw` muss leer sein), die nicht stillschweigend
übersprungen werden kann. Ein eigenständiges, getestetes Skript wurde daraus abgeleitet und in
Issue #30 hinterlegt, zusammen mit zwei Architekturvorschlägen: das Verfahren als `wikitool`-
Kommando statt als Shell-Rezept, oder - als eigentliche Ursachenbehebung - den Demo-Korpus
grundsätzlich von dem Branch fernzuhalten, von dem private Instanzen ihre Maschinerie ziehen.
Anschließend wurde Issue #3 (Chemenu-Rebranding) gegen seine eigenen Abnahmekriterien geprüft
und für sauber befunden, und eine Verdrahtungslücke geschlossen (`tools/CONTRACT.md` kannte das
Publish-Remote-Gate nicht, `gates.md` verlinkte nicht auf die Prozedur, die Chemenu-Projektseite
beschrieb sich noch als privates Wiki) - veröffentlicht als 2.2.2.
## Kernaussagen
- Eine Behauptung über Git-Merge-Verhalten ist erst nach einem Test eine Tatsache. Am selben Tag
geschrieben zu haben ist kein Beleg für Richtigkeit.
- `git merge` behandelt "eine Datei geändert" und "eine Datei neu angelegt" unterschiedlich: Nur
Ersteres erzeugt einen sichtbaren Konflikt. Eine Prozedur, die nur den Konfliktfall bedenkt,
übersieht die stille Neuanlage.
- `.gitattributes`-Merge-Treiber wie `merge=ours` wirken nur auf Inhaltskonflikte zwischen
beidseitig vorhandenen Dateiversionen, nicht auf modify/delete-Paare oder Neuanlagen.
- Eine Kontrollprüfung, die ein Mensch lesen und verstehen muss, um einen Fehler zu bemerken, ist
schwächer als eine, die bei einem Fehler selbst nicht-null zurückgibt.
- Wiederkehrende Symptome bei jeder Downstream-Instanz sind oft ein Zeichen, dass die eigentliche
Ursache stromaufwärts liegt und dort einmalig behoben werden sollte.
## Aufgaben
- [x] `private-instance.md` korrigiert und als 2.2.1 veröffentlicht
- [x] Issue #30 angelegt (Skript + zwei Architekturvorschläge, hängt an #28)
- [x] Issue #3 gegen Abnahmekriterien geprüft, sauber
- [ ] Issue #30 selbst ist offen - siehe dort für den Entscheidungsstand
## Nicht übernommen
- Die im Transkript vollständig gezeigten Testskript-Läufe (Shell-Ausgabe der Wegwerf-Repos)
sind hier nicht wiederholt - die Kernaussage (welches Verhalten gemessen wurde) ist auf die
Concept-Seite [[Publish-Remote Gate]] und in Issue #30 übernommen, der Beleg bleibt im
Transkript.
## Verwandte Entities
- [[Chemenu]]
- [[wikitool]]
## Verwandte Concepts
- [[Publish-Remote Gate]]
- [[Delete Rather Than Anonymize]]
@@ -0,0 +1,87 @@
---
type: types/source.md
source_type: notes
author: Torben
raw_files: ['raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md']
source_language: de
date: 2026-09-01
tags: []
entities: [Chemenu, wikitool]
concepts: [Mass-Update Gate, Delete Rather Than Anonymize, Dual Licensing by File Plan]
summary: 'Sitzung, die den Chemenu-Stack von einer privaten Testinstanz in ein oeffentliches Repo ueberfuehrt: Korpus geloescht statt anonymisiert, Git-History auf einen Commit gesquashed, AGPL-3.0/CC-BY-4.0-Dual-Lizenz gewaehlt, dist export um einen Leak-Canary gehaertet.'
---
# Source: Public Release, Corpus Purge and History Squash Session 2026-09-01
**Autor:** Torben
**Datum:** 2026-09-01
**Raw-Dateien:** raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
**Typ:** Notes
## Zusammenfassung
Torben bat darum, den Chemenu-Korpus zu bereinigen und das Repo zu veröffentlichen, mit
ausdrücklichem Wunsch nach Teufels-Advokat-Modus und einer kleinen Multi-Agent-Debatte. Drei
parallele Explore-Agenten inventarisierten private Daten, den Distributionsmechanismus und die
Git-History; die History-Prüfung erklärte den Baum fälschlich für „safe to publish", ohne
`USER.md` je zu öffnen — ein Befund, der später den Ausschlag für den vollständigen History-Schnitt
gab. Drei geforkte Debattierer (harte Trennung, geteiltes Upstream, Teufels-Advokat) argumentierten
gegeneinander; die Synthese übernahm „löschen statt anonymisieren" und „History squashen" von der
harten Position, das Clone-mit-Upstream-Modell von der geteilten Position, aber unter der
Bedingung, dass ein Remote-Gate zuerst existiert — eine Bedingung, die der Teufels-Advokat mit
seinem Leck-Argument erzwang.
Nutzer traf vier Entscheidungen über `AskUserQuestion`: Gitea öffentlich schalten (kleinste
Änderung), Korpus chirurgisch löschen, private Instanz als Clone mit Upstream und Gate zuerst,
und als Lizenz AGPL-3.0 (Stack) + CC-BY-4.0 (Inhalte) — die Affero-Variante bewusst wegen Issue
#19 (MCP-Frontend als Netzdienst).
Ausgeführt wurde: Lizenzdateien (AGPL-Text von gnu.org geholt, nicht aus dem Gedächtnis
rekonstruiert), ein Leak-Canary in `dist export` (`find_leaks()`, strukturell statt textbasiert,
weil ein Muster-Scan den eigenen legitimen Host mit ausschließen müsste), die Korpus-Löschung
(108 Seiten statt der im Plan geschätzten ~35, weil 40 Seiten mit generischen Titeln tatsächlich
um die private Infrastruktur herum geschrieben waren), der History-Squash auf einen Commit, und
die Veröffentlichung selbst.
Zwei Annahmen wurden durch Messung widerlegt und korrigiert: Ein Force-Push allein reicht nicht —
der alte HEAD blieb per SHA abrufbar, bis Reflogs auf dem Server verfielen und `gc --prune=now`
lief. Und `rg`-Scans ohne `--hidden` übersehen `.gitea/`, `.github/`, `.vibe/` — ein zweiter
Fund (private Referenzen in `ci.yml`) kam erst über `git grep` zum Vorschein.
## Kernaussagen
- Ein Seitentitel ist der einzige Identifier des Wikis; Löschen (`wikitool rm`) ist dafür billiger
und sicherer als Anonymisieren, das die volle `page-lifecycle`-Prozedur pro Seite verlangt.
- Der eigentliche Preisgeber bei einem Infrastruktur-Handbuch ist die Topologie, nicht der
Hostname — gefälschte IPs entschärfen keine Angriffskarte.
- Ein Force-Push macht alte Commits unreferenziert, aber nicht unerreichbar: Sie bleiben per SHA
fetchbar, bis Server-Reflogs verfallen sind und `git gc --prune=now` gelaufen ist.
- `dist export`s Lizenz-Dateien mussten zur Pflicht werden (`REQUIRED_ROOT_FILES`), weil das
übliche `if source.is_file()`-Muster eine fehlende Lizenz still überspringen würde — bei AGPL
eine Verletzung, sobald eine Instanz öffentlich landet.
- Ein Text-Muster-Scan für Leaks scheitert an legitimen Vorkommen des eigenen Hostnamens; ein
struktureller Scan (welche Pfade/Dateien dürfen nie im Plan stehen) umgeht das.
## Aufgaben
- [x] Korpus bereinigt, History gesquasht, Lizenzen gesetzt, Repo veröffentlicht (in dieser
Sitzung erledigt)
- [ ] Siehe Issue #27 (Decay-/Lint-Ausschluss für mitgelieferte Seiten) und #28 (Demo-Korpus als
eigene Fixture) für Folgearbeit aus dieser Sitzung
## Nicht übernommen
- Die konkreten Namen und Details der gelöschten privaten Infrastruktur (Hostnamen, IP-Bereiche,
persönliche Angaben) sind bewusst nicht in diese Source-Seite übernommen — sie zu wiederholen
widerspräche dem Zweck der Sitzung. Wo sie als Beispiel dienen mussten, steht hier nur die Art
der Information (z. B. "eine Homelab-Cluster-Dokumentation"), nie der Wortlaut.
- Der volle Wortlaut der drei Debattenpositionen ist nicht übernommen - nur ihre tragenden
Argumente und was aus ihnen in die Synthese einging. Der vollständige Text steht im Transkript.
## Verwandte Entities
- [[Chemenu]]
- [[wikitool]]
## Verwandte Concepts
- [[Mass-Update Gate]]
@@ -0,0 +1,71 @@
---
type: types/source.md
source_type: notes
author: Torben
raw_files: [raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md]
source_language: de
date: 2026-09-01
tags: []
entities: [Chemenu, wikitool]
concepts: [Publish-Remote Gate, Mass-Update Gate, Issue Label Scheme]
summary: Sitzung, die ein drittes, Token-loses Gate fuer publish baut, instructions/private-instance.md schreibt, sechs Gitea-Issues auf den Rename und die neue Architektur nachzieht und die Actions-Run-Historie entfernen laesst.
---
# Source: Publish-Remote Gate and Issue Triage Session 2026-09-01
**Autor:** Torben
**Datum:** 2026-09-01
**Raw-Dateien:** raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
**Typ:** Notes
## Zusammenfassung
Direkte Fortsetzung der Veröffentlichungssitzung: Torben bat um eine Remote-Allowlist für
`publish`, eine Bitte, vor dem Öffentlich-Schalten von Gitea zu warten und einen Spickzettel
dafür, und ein Kurz-Howto für die lokale Dev-Umgebung - dazu, nach eigenem Ermessen, Issues zu
pflegen. Gebaut wurde das **Publish-Remote-Gate**: Es prüft die aufgelöste Push-URL (nicht den
Remote-Namen, weil ein umgebogener `origin` sonst durchrutschen würde) gegen eine optionale,
gitignorete Allowlist-Datei und hat als einziges der drei Gates keinen Freigabe-Token - der Weg
daran vorbei ist ein bewusster Edit der Datei durch den Menschen. `instructions/private-instance.md`
beschreibt seither das Clone-mit-Upstream-Setup, mit dem Gate als Schritt vor dem ersten `publish`.
Sechs offene Issues wurden auf den Chemenu-Rename hin durchgesehen; mehrere trugen noch den
alten Paketpfad. Drei neue Issues entstanden aus Punkten, die im Tagesverlauf entschieden und
dann zurückgestellt worden waren (Handbuch-Vorbedingung, Demo-vs-Testbett-Konflikt, veraltete
Issue-Texte). Nach dem Öffentlich-Schalten wurde anonym end-to-end geprüft (Klon, Release-Feed,
Distributionsweg), `INSTALL.md` von "Repo ist privat" auf den öffentlichen Zustand umgestellt,
und auf Bitte des Nutzers die Gitea-Actions-Run-Historie über die REST-API entfernt, nachdem
sich herausstellte, dass weder das MCP-Werkzeug noch eine sichtbare UI-Schaltfläche das können.
## Kernaussagen
- Ein Allowlist-Gate muss die **aufgelöste Push-URL** prüfen, nicht den Remote-Namen - sonst
schützt es nicht vor einem umbenannten oder umgebogenen Remote.
- Ein Gate, dessen Frage eine stehende Eigenschaft des Checkouts ist (nicht ein einzelnes
Changeset), braucht keinen Freigabe-Token - der Mensch löst es durch einen bewussten
Datei-Edit, nie ein Agent durch einen Bypass.
- Ein Werkzeugvertrag (`tools/CONTRACT.md`) muss jeden Fehlerfall eines Kommandos nennen; ein
neuer Exit-42-Pfad, der dort fehlt, ist eine Lücke, die kein automatischer Check findet.
- Ein MCP-Server kann weniger können als die zugrunde liegende API - hier: Actions-Runs
anzeigen/erneut starten, aber nicht löschen, obwohl die REST-API die Route hat.
## Aufgaben
- [x] Publish-Remote-Gate, `private-instance.md`, Issue-Pflege in dieser Sitzung erledigt
- [ ] #4, #5 tragen laut #29 noch veraltete Pfade und sind noch nicht nachgezogen
## Nicht übernommen
- Der vollständige Wortlaut des Gitea-Spickzettels ist nicht auf diese Seite übernommen - er
steht im Transkript und in der Chat-Antwort an den Nutzer, ist aber keine dauerhafte
Wiki-Aussage, sondern eine einmalige Handlungsanweisung.
## Verwandte Entities
- [[Chemenu]]
- [[wikitool]]
## Verwandte Concepts
- [[Publish-Remote Gate]]
- [[Mass-Update Gate]]
- [[Issue Label Scheme]]
@@ -0,0 +1,128 @@
# Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
> Collected: 2026-09-01
> Participant: Torben
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's questions and
> instructions are quoted verbatim; the assistant's reasoning is condensed. All shell output
> shown was produced by real, disposable test repositories built in a scratch directory during
> this session (not the chemenu repository itself) and is reproduced as run.
> No credentials appeared in the session.
> Third of three transcripts cut from this session; the first two cover the corpus/history work
> and the Publish-Remote Gate this one corrects a claim about.
Covers Torben asking what actually happens to the demo corpus in a private instance when the
upstream changes it - which exposed an unverified, incorrect claim the assistant had just
written into `instructions/private-instance.md` the same day. The claim was tested empirically,
found wrong, corrected in a released patch (2.2.1), and turned into a properly scoped, tested
merge procedure plus a new issue (#30) proposing the fix belongs in the tool itself, or that the
upstream should stop shipping content on the branch private instances pull machinery from.
---
## Turn 1 - "was passiert in der privaten instanz nach diesen schema wenn sich upstream der demo-korpus ändert?"
The assistant treated this as a request to verify, not answer from the existing text - the
previous transcript's `private-instance.md` had said a `git merge upstream/main` would see
*deleted-in-ours, unmodified-in-theirs* and resolve silently, which was written as an assumption
and never tested.
Built a throwaway upstream/private pair of git repos in the scratch directory to test it: private
instance deletes the demo corpus once, upstream then modifies one demo page, adds a new one, and
deletes another. Ran a plain `git merge upstream/main`. Result, measured rather than assumed:
- The upstream's **modified** page produced `CONFLICT (modify/delete)` and left the upstream's
version sitting in the working tree - resolving it the obvious way (`git add -A`) would commit
that version, bringing the deleted demo page back.
- The upstream's **new** page was staged silently. No conflict, no message.
- The upstream's **deleted** page (matching a page already deleted on the private side) produced
nothing - the one case that actually worked as claimed.
The middle case was the one the original claim missed entirely, and it is the dangerous one
because nothing announces it - a demo page the upstream adds simply arrives and starts showing
up in the private instance's own `lint`, `index`, `search` and `confidence decay`.
A second candidate fix was tested and also failed: a `.gitattributes` `merge=ours` driver for
`kb/**`/`raw/**`. Same three-scenario test showed the modify/delete conflict still occurred and
the new page still landed - the driver only resolves content conflicts on files both sides
touch, not modify/delete pairs or additions.
## Turn 2 - Fixing the instruction, in a released patch
`instructions/private-instance.md` was rewritten: the "Why a clone rather than a tarball"
section now states the measured three-case table instead of the wrong assumption. The update
procedure was rewritten to hold the merge open (`git merge --no-commit --no-ff`), force the
content stages (`kb/`, `raw/`) back to the pre-merge state while `HEAD` still points at the
commit before the merge, commit, and then run an explicit check
(`git diff --name-only $BEFORE HEAD -- kb raw` must be empty) - a check the reader cannot silently
skip past, unlike a claim they might trust. The bad advice in "Decision points" ("resolve a
conflict under kb/ as keep-deleted") was replaced, since that advice is exactly what leads
someone to `git add -A` the leaked content.
The corrected procedure was itself run against the same three-scenario test plus a fourth
(`raw/` alongside `kb/`), and against error paths (dirty working tree rejected, a second run
with nothing new to pull is a no-op) - all passing before publishing.
Version bumped `2.2.0 -> 2.2.1`. The changelog entry states plainly that the earlier text was
"nicht gemessen, sondern angenommen" (not measured, assumed) rather than framing it as a minor
wording fix. Published; tests green (752, unchanged in count - this was a documentation fix).
## Turn 3 - "stelle mal ein sauberes skripting hier exemplarisch dar. gibt es alternative setups? könnten wir die demo/test-seiten aus dem repo main irgendwie heraushalten? erstelle aus dem ganzen thema ein issue... aktualisiere/prüfe, ob #3 im gitea sauber ist. prüfe, ob wir sonst alle offenen enden verdrahtet haben."
Four asks in one message. Handled in order:
**A standalone script**, written and tested (not just described) against the same scenario:
rejects a dirty working tree, rejects a concurrent merge/rebase, holds the merge open the same
way the instruction's procedure now does, is a no-op on a second run with nothing new, and exits
non-zero with a rollback command if the post-merge diff check ever finds leaked content - the
check the manual procedure relies on a human to run, made unskippable.
**Alternative setups**, and the negative result from Turn 1 (`merge=ours` does not work) was kept
rather than omitted, since a rejected alternative is exactly the kind of finding this capture
procedure is meant to preserve.
**Whether the demo corpus can be kept out of `main` entirely** - assessed as the better fix in
principle: the private-instance script (and the instruction's manual procedure) treats a symptom
on every downstream instance, repeatedly; keeping content off the branch machinery is pulled from
would remove the need for either, by construction, for any new instance. Named as two variants
(a separate `demo` branch, or a separate demo repository) with the real cost stated rather than
glossed over - `main` becomes a content-free shell, which was the reason the corpus was kept in
the first place (as a walkable example), and `nightly.yml`'s corpus lint would need to move.
**Issue #30 opened**, carrying the measured table, the working script, both alternatives with
their tradeoffs, and the explicit note that it is not a duplicate of #28 but depends on it - #28
is why the corpus moves at all (demo and testbed sharing one `kb/`), #30 is what that movement
does to a downstream clone.
**Issue #3 checked against its own acceptance criteria** (all five, from the closed issue's own
text) rather than assumed closed-and-fine: a grep for the pre-rename name outside history/
changelog/transcripts, presence of a project entity page, the "Thoth" persona note in both
`SOUL.md` and its template, and no active technical identifier still using the old name. Found
clean - the handful of remaining mentions (`INSTALL.md`, two `kb/` pages) were all phrased as
history ("bis 2026-09-01", "damals"), which the criteria explicitly exempt.
**The wiring audit** turned up the two gaps described in the second transcript's closing turn
(`tools/CONTRACT.md` missing the new gate, `gates.md` not linking to `private-instance.md`) and
the stale project-page description - all three fixed in the same pass and published as 2.2.2,
verified with `docs verify`, `instructions verify`, `doctor` (16 checks OK), `lint` (clean but for
one pre-existing orphan page unrelated to this work), and the full test suite.
## Turn 4 - This capture
Torben asked for this session to be captured via `instructions/capture-session.md`, with an
explicit instruction to keep private and confidential material out - consistent with, not
separate from, everything the day's work had been about. Handled by generalizing every
infrastructure and personal-data reference that appeared during the day's audits rather than
reproducing the original identifiers, cutting the session into three topic transcripts (this
being the third), and filing this transcript's own existence as evidence rather than as new
open work - nothing in this turn required a new issue.
## Outcome
- **Version:** 2.2.0 -> 2.2.1 -> 2.2.2
- **Commits:** `private-instance.md` correction (2.2.1), documentation wiring + project-page
rewrite (2.2.2, shared with the second transcript's closing turn)
- **Tests:** 752, green throughout
- **Issues:** #30 opened; #3 verified against its own acceptance criteria and confirmed clean,
no action needed
- **CI:** green after each publish in this transcript's scope
@@ -0,0 +1,208 @@
# Conversation Transcript - Public Release, Corpus Purge and History Squash Session
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
> Collected: 2026-09-01
> Participant: Torben
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instructions and
> decisions are quoted verbatim; the assistant's reasoning and the exploration agents' findings
> are condensed. Command outputs shown are real, but every private hostname, IP address and
> personal detail that appeared during the audit has been **generalized rather than repeated** -
> reproducing them here would undo the point of the session. No secret values (tokens, keys)
> ever appeared.
> **Several passages are second-hand.** Three `Explore` subagents (private-data inventory,
> distribution mechanism, git-history audit) and three forked debate agents (hard-separation,
> shared-upstream, devil's-advocate positions) ran in the background; their findings are
> summarized here, not reproduced verbatim, and are named as such at each point they inform a
> decision.
> One of three transcripts cut from the same session; the other two cover building the
> Publish-Remote Gate and the private-instance procedure, and correcting a bug found in that
> procedure afterward (issue #30).
Covers turning the chemenu wiki stack from a private testbed into a publishable repository:
deciding what to delete versus anonymize, squashing the git history, choosing a license, and
verifying the result anonymously end to end. Resulted in the corpus purge (108 pages, ~90 raw
files removed), a single-commit public history, `LICENSE`/`LICENSE-CONTENT`/`NOTICE`, the
`dist export` leak canary (2.1.0), a CI fixture fix (2.1.1), and the repository going public on
Gitea.
---
## Turn 1 - "ich möchte die kb hier bereinigen, es soll nur noch test content sein, ich möchte das repo veröffentlichen..."
Torben's instruction, in substance: clean the KB down to test content, publish the repository,
falsify or remove IP addresses, keep the repo usable as a demo and a testbed going forward; he
would rebuild his real private instance separately. He asked for tips on three specific
questions - how a private fork could pull upstream fixes back, how to set up that private copy
and where to develop the stack, and whether a `kb/manual/` should ship with the distribution so
it documents itself - explicitly requested "mindestens ... teufels anwalt modus" and a small
multi-agent debate, and flagged himself as new to packaging/distributing an LLM application.
The session started in plan mode. Three `Explore` subagents ran in parallel:
- **Private-data inventory** - catalogued what in `raw/` and `kb/` was hard-private (a real
home-lab cluster's bootstrap manual, several personal notes, a 2.46 MB conversation transcript
full of infrastructure references), what was anonymizable (internal hostnames, private IP
ranges), and what was legitimately generic (the stack's own concepts, technologies, tools).
- **Distribution mechanism** - read `dist_cmd.py`, `INSTALL.md` and the version-check machinery;
confirmed `dist export` already ships no `kb/`/`raw/` content and that the update feed's
default URL pointed at the (then-private) origin repo.
- **Git-history audit** - measured `.git` size, largest blobs, deleted-file history, and
concluded the tree was "safe to publish" - a conclusion the session later disproved, see Turn 4.
## Turn 2 - The debate
Three forked debaters argued from the same context:
- **A (hard separation):** delete and rebuild the corpus rather than anonymize, because a page
title is the wiki's only identifier and lives in wikilinks/citations/frontmatter across the
corpus - renaming is the full page-lifecycle procedure, deleting is a supported command.
Squash history rather than selective `filter-repo`, because the history audit itself had just
been shown to under-report risk. No shared git ancestry between the public repo and the
private instance, ever.
- **B (shared upstream):** clone-with-`upstream`-remote instead, because the tarball update path
in `INSTALL.md` is `cp -r` with no three-way merge; argued the leak risk from shared history is
a mechanism problem (a remote allowlist), not a discipline problem, and that most of the
corpus (the ~190 generic pages) is worth keeping as real test data rather than deleting it all.
- **C (devil's advocate):** attacked both positions' shared premises - "just a testbed" was
false (`USER.md` is tracked and carries real personal data), "publish the repo" was
underspecified (which host, and what breaks), there was no `LICENSE` at all, a shipped manual
would decay under `confidence_decay.py` and mislead a new instance, and demo/testbed are
conflicting goals for one `kb/`.
The assistant's synthesis kept A's "delete, don't anonymize" and "squash the history" (C's
finding that the audit under-reported risk made this decisive), adopted B's clone-with-`upstream`
model but **conditioned it on a remote-allowlist gate existing first** (addressing C's leak
concern with code, not discipline), rejected the shipped manual per C's decay argument, and
decided the surviving corpus should lean toward the stack's own self-documentation rather than
Torben's real infrastructure - which the actual corpus math later validated (Turn 5).
## Turn 3 - Clarifying questions and plan approval
The assistant asked Torben to choose among concrete options (`AskUserQuestion`) rather than
picking on his behalf:
- **Publish target:** "Gitea öffentlich schalten" (chosen) over mirroring to GitHub or keeping
Gitea private with a GitHub mirror - smallest change, CI/release feed stay correct as-is.
- **Corpus:** "Chirurgisch löschen (Empfehlung)" (chosen) over full synthetic rebuild or
anonymization.
- **Private instance:** "Clone mit upstream, Gate zuerst (Empfehlung)" (chosen) over hard
separation or a later migration.
- **License:** first pass "Was wäre CC-BY-4 für content und eine GPL oder LGPL fürs Tooling?
Berate mich hier noch mal" - the assistant advised against LGPL (no library-linking use case
here to justify the fetch-only variant), and named the real choice as GPL vs. **AGPL**, because
issue #19 (an MCP frontend for `wikitool`) points toward running the stack as a network
service, which is exactly the gap GPL leaves and AGPL closes. Torben chose **AGPL-3.0 +
CC-BY-4.0**.
The written plan covered: publish blockers (license, `dist export` allowlist, `INSTALL.md`
token wording), the corpus purge, the history squash, the private-instance model with the gate
as a precondition, why the manual idea was declined, and a leak-canary for `dist export`. Torben
approved it via `ExitPlanMode` without further changes.
## Turn 4 - Executing the license and leak-canary work (2.1.0)
Before touching content: a full repo bundle backup outside the tree, verified by `git bundle
verify` and a commit-count comparison (150 commits, all 15 tags present).
- Fetched the AGPL-3.0 text from `gnu.org` (not reconstructed from memory - a license must be
the authoritative text) and copied the CC-BY-4.0 text from the vendored `commonplace`
submodule, per the license each already carries.
- Added `LICENSE` (AGPL - deliberately the AGPL rather than a separate `LICENSE-CODE`, because
that is the file a forge reports for the repository, and under-noticing a copyleft obligation
harms a reader in a way over-noticing does not), `LICENSE-CONTENT` (CC-BY), `NOTICE` (the
license boundary and the `commonplace` attribution CC-BY requires).
- `dist_cmd.py`: added the three license files to `ROOT_FILES`, and a separate
`REQUIRED_ROOT_FILES` check that **fails the export** if they are missing - every other
`ROOT_FILES` entry is copied `if source.is_file()` and silently skipped otherwise, which is
wrong for a license (a distribution shipping AGPL code with no license text is a violation the
moment it is published).
- Added `find_leaks()`: a structural (not text-pattern) check of the finished export plan against
personalization files, `instructions/dev/`, `kb/` pages and `raw/` sources - rejected a
hostname/IP text-scan approach, because the project's own host legitimately appears in
`INSTALL.md` and `version.py`, so such a scan would either whitelist the string it's looking
for or false-positive on every export.
- Six new tests in `test_dist_cmd.py`; full suite green; version bumped `2.0.0 -> 2.1.0`.
## Turn 5 - The corpus purge
A workshop (`work/publish-cleanup`) was opened per `work/CONTRACT.md`, because ~103 planned
`wikitool rm` calls exceed a single iteration-budget unit.
Analysis found the plan's estimate wrong in a way worth recording: not ~35 pages would become
sourceless after deleting the obviously-private ones, but **75**. Splitting them: 28 were the
user's own infrastructure (unambiguous), and **40 carried generic technology titles but were
written entirely around the user's own cluster** (a storage page explained itself via one
specific storage class name, a Kubernetes page via the user's own network ranges) - not
reusable pages with a private example, but private documentation with a generic heading. The
remaining 7 were genuinely clean in body text but sourced from the user's personal
document-processing pipeline.
Presented to Torben as a three-way choice; he chose **"Alle 75 löschen"** over keeping the
clean 7 (via provenance reclassification) or rewriting the 40 as vendor-neutral pages. The
result: 152 pages plus 25 sources remained by design - the stack's own self-documentation
(gates, lint, versioning, search, the wiki pattern itself), which the earlier debate had argued
for without knowing this would be the actual outcome.
Execution: `wikitool rm --page <title> --yes` per page across three units (27 sources, 28
infra, 48 entangled), 90 raw files removed including the full cluster bootstrap tree and the
2.46 MB transcript. `wikitool rm` was found to de-link only mechanically (frontmatter refs and
whole-line link bullets) and to deliberately leave inline prose wikilinks and plain-text
mentions standing - roughly 20 pages needed manual follow-up to actually remove the remaining
references, which `lint` and a targeted `rg` scan surfaced.
Also reset: `kb/log.md` (116 private references, no regenerator - reset to the
`dist_templates/log.md` starting state, a deliberate one-time stack-dev operation, not a
hand-edit of a generated file), `USER.md`/`SOUL.md` rewritten as an explicit demo-operator
persona rather than Torben's real profile.
## Turn 6 - Squashing the history
Backup re-verified before the destructive step. An orphan root commit was built from the purged
working tree, `main` reset onto it, all 15 tags deleted (locally, then individually on the
remote - a batch refspec push failed silently), `git reflog expire --expire=now --all` and
`git gc --prune=now --aggressive`, then a `--force` push.
**Verified rather than trusted:** a fresh clone attempted `git fetch --depth=1 origin
<old-head-sha>` immediately afterward and succeeded - the objects were still reachable through
Gitea's own reflog on the bare repository, contradicting the earlier belief that force-pushing
was sufficient. The rejected shortcut here was accepting Gitea's own scheduled cleanup jobs as
proof; Torben ran them once, and the same fetch test still succeeded afterward. Only a direct
`reflog expire` + `gc --prune=now` run on the bare repository itself (which Torben ran, having
shell access the assistant did not) closed it - reverified by the same fetch test returning
`not our ref`.
A second leak was found by scanning **hidden directories** (`.gitea/`, `.github/`, `.vibe/`),
which the earlier `rg` sweeps had silently skipped without `--hidden` - two references to an
internal CI branch name in `.gitea/workflows/ci.yml`, fixed and amended into the squashed
commit before the first push.
## Turn 7 - Public switch, CI fixture bug, and verification
Torben switched the Gitea repository to public. Verified anonymously (no token, no SSH key):
repository API reports `private: false`, the release feed serves the latest tag, and a clean
`git clone` over HTTPS from an empty directory succeeds. A further leak scan of that anonymous
clone found one remaining hit: the 2.1.1 changelog entry (see below) had *listed* the private
fixture names it replaced, which put them back into the public history it was announcing the
removal from. Corrected in the same commit that introduced it.
Separately, the first CI run on the squashed history failed on a test unrelated to the corpus
purge in content but caused by it: `test_legacy_source_pages_flags_url_and_directory` checks
`(config.ROOT / legacy).is_dir()` against the **real** repository root rather than the test
fixture's own tree, and had only ever passed because this checkout happened to have a
`raw/documents/` directory - which the corpus purge had just emptied. Git does not track empty
directories, so the directory vanished from CI's checkout and stayed in the local one: green
here, red there. Reproduced locally by removing the directory and rerunning; fixed in the
`raw_dir` fixture (`conftest.py`) rather than the one test, matching the reasoning already
recorded for a prior, similar case (Gitea #8). Several other test fixtures still used the
user's real system names and were renamed to generic, unrelated placeholders
alongside. Published as 2.1.1 after Mass-Update Gate clearance from Torben.
## Outcome
- **Version:** 2.0.0 -> 2.1.1
- **Commits:** license/leak-canary work, corpus purge (multiple), history squashed to one root
commit, CI fixture fix
- **Tests:** 630 -> 752, all green
- **Issues:** none opened in this transcript's scope (see the other two transcripts for #27-#30)
- **CI:** green on the squashed history after the fixture fix; release `v2.1.1` built and
anonymously downloadable
@@ -0,0 +1,154 @@
# Conversation Transcript - Publish-Remote Gate and Issue Triage Session
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
> Collected: 2026-09-01
> Participant: Torben
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instructions are
> quoted verbatim; the assistant's design reasoning is condensed. Command outputs and test
> results shown are real. No credentials appeared in the session; where a private hostname
> could have been used as a code example, a generic placeholder is used instead, matching what
> was actually committed.
> Continues directly from "Public Release, Corpus Purge and History Squash Session" (same day).
> Two of three transcripts cut from this session; the third covers a bug found afterward in the
> procedure this one builds (issue #30).
Covers building the Publish-Remote Gate (a third code-enforced gate, alongside the Mass-Update
and rebase-review gates), writing `instructions/private-instance.md`, updating six open Gitea
issues for the rename and the new architecture, opening three new issues (#27, #28, #29), and
deleting the Gitea Actions run history at Torben's request. Resulted in version 2.2.0, then a
documentation-only 2.2.2.
---
## Turn 1 - "1. remote allowlist. 2. unterbreche und warte auf mich, wenn ich gitea scharf schalten soll... 3. setup a) meiner lokalen dev-umgebung"
Torben asked for three things in one message, plus: "parallel issuen anlgen und aktualisieren
nach eigenem ermessen, prüfe auch vorhandene, evtl. müssen wir hier auch noch tätig werden?"
**The gate.** Designed to pin the **resolved push URL**, not the remote name - a name-based
allowlist would pass a `publish` whose `origin` had been repointed, which is the exact failure
it exists to catch. `git remote get-url --push <remote>` is read at check time so a `pushurl`
override is respected. The file (`.wikitool-remotes.json`) is gitignored and per-checkout, for
the same reason `ENVIRONMENT.md` is: two clones push to two different places, and a committed
copy would tell a private clone the public upstream is a legitimate target for its own content.
Absence means unrestricted, matching the pattern of the other optional per-checkout files;
`doctor` reports the state and WARNs only when a checkout has more than one remote and no
allowlist.
**Deliberately no `--confirm` token**, unlike the other two gates. Their question ("is this
change right?") is answerable per changeset; this gate's question ("does this content belong in
that repository?") is a standing property of the checkout, so the only way past it is the user
editing the file themselves - an agent editing it to clear a refusal would be opening a gate on
its own initiative, which the repository's own rules forbid.
Implementation: `config.PUBLISH_REMOTES_FILENAME`, `read_allowed_push_urls()` /
`push_url_for()` / `publish_remote_refusal()` in `git_publish.py`, checked before the reconcile
step in `publish_command` (before any network contact, so a refused publish never even fetches
from the wrong place), `doctor.check_publish_remotes()`, twelve new tests covering the URL-vs-name
distinction, `pushurl` precedence, a broken/missing/empty allowlist, and that no flag exists to
bypass it. `instructions/gates.md` and `AGENTS.md` updated to describe a third gate. Version
bumped `2.1.1 -> 2.2.0`; the Mass-Update Gate itself fired at 10 files and Torben cleared it with
the printed token.
**`instructions/private-instance.md`** (new): the procedure for cloning with the public repo as
`upstream`, arming the gate *before* the first `publish` (not after - the assistant stressed this
ordering explicitly, since a gate added later leaves the earlier window open), taking the write
credential away from the private clone as a second, independent control, and where stack
development happens ("in the public repo, not here" - not a preference but a structural fact,
since `instructions/dev/` does not survive `dist export`). **This file's description of what a
`git merge upstream/main` actually does to the content stages was wrong as first written** - see
the third transcript for the correction.
**The Gitea admin cheat-sheet** (delivered as a reply, not committed): what to check before
flipping the repo public (`DISABLE_REGISTRATION`, `REQUIRE_SIGNIN_VIEW`, rate limits, the
Actions runner's network exposure), and the follow-up steps (`INSTALL.md`, anonymous release
check, a clone test). Also flagged that the open issues would go public with the repo, which is
what motivated the issue-triage pass below.
## Turn 2 - Issue triage
Read the labels (`prio/1..3`, `size/XS..L`) and the open issues. Several pre-dated the
`llm-wiki-test1` -> `chemenu` rename (issue #3) and still named the old package path
(`tools/wiki_tools/...`) or the old repository name in code examples:
- **#6** title corrected in place (`wiki_tools` -> `chemenu` path).
- **#7** (`dist upgrade`) commented: the "origin repo is private" fallback it describes no
longer applies once the repo is public, and its urgency for *this* instance specifically
dropped, because the private instance now takes updates via `git merge upstream/main` (real
three-way merge) rather than the tarball-copy path the issue was written against - it remains
the right design for any instance without shared git history.
- **#10** (coverage reporting) commented: step 1 is done (1.8.1 shipped `pytest-cov` without a
failure threshold), the test count referenced is stale (630 -> 752, and the gap is itself
evidence for the issue's own argument - the raw_dir fixture bug from the first transcript), and
one code example named a private CI branch and needs neutralizing before being read publicly.
- **#4, #5** flagged as needing the same path correction but not rewritten in this pass.
**Three new issues opened**, each traced to a decision made and then set aside earlier in the
day rather than invented fresh:
- **#27** - the shipped-manual idea the debate rejected (first transcript, Turn 2) needs a
decay/lint exemption for distributed pages before it becomes buildable at all; names the exact
blocker (`confidence_decay.py` has no exemption path) and the two things still undecided (how
a shipped page is recognized, what exactly is exempted).
- **#28** - the demo-vs-testbed tension the debate argued through (assistant recommended
testbed, C objected) is not resolved, only decided for now; proposes a curated fixture corpus
as the eventual answer and names why (`kb_dir`/`raw_dir` fixtures already cover the small case,
this is about the dense, interconnected one).
- **#29** - catalogs which open issues still carry pre-rename paths and a private hostname
(#10's CI example), separating "pointer, fix it" references from "citation of what was true
then, date it" references rather than blanket-editing everything.
## Turn 3 - Public verification and the actions-history request
After Torben confirmed the repository was public, verification ran anonymously: repo API
(`private: false`), release feed (latest tag served without auth), a clean HTTPS clone from an
empty directory (three commits, two tags, all three license files present, 170 pages, no leak
hits including in hidden files), and the full `INSTALL.md` release-tarball path end to end -
checksum verified, license files present in the unpacked tree, `kb/`/`raw/` empty as designed,
`instructions/dev/` absent.
Torben: "an der Stelle: wenn du per mcp die action runner historie löschen kannst, leere sie
bitte komplett, da finde ich keinen knopf für und dort dürften noch alte tarballs herumgeistern."
The assistant checked: the Gitea MCP server exposes `dispatch`/`cancel`/`rerun` for Actions runs
but no delete, and the delete route Gitea's own API defines (`DELETE
/api/v1/repos/{owner}/{repo}/actions/runs/{run}`) requires a token the assistant did not hold.
Findings reported instead of silently working around the missing capability: **zero artifacts**
existed already (confirmed via the anonymous artifacts API), but an anonymous SPA data endpoint
the run pages call was readable and returned old commit titles from the squashed-away history -
no hostnames or personal data among the six reachable runs, but development history the squash
was meant to remove. A `curl` loop using a write-scoped token was handed to Torben to run
himself, rather than asking for a token in the session.
Torben ran it and reported back ("die action-historie habe ich jetzt per api gelöscht") in the
same message as "korrigiere die install.md wie vorgeschlagen." Both handled: the anonymous SPA
endpoint was re-tested across all previously-reachable run numbers and returned nothing for any
of them, closing the loop; `INSTALL.md`'s "repo is currently private, you need a token" language
was removed from three places (the download command, the configuration table, a dedicated
paragraph - the paragraph rewritten to explain the token is still needed **for a private fork**,
not for this repository), "Weg C" was reworded from "Torben's personal wiki" to a description of
the testbed/demo nature of the corpus, and a new "Weg D" documented the private-instance path
with a pointer to `private-instance.md`. No version bump (prose only).
## Turn 4 - Wiring audit (from the third transcript's closing turn, included here for the gate/issue thread)
A later pass (opened by a different question, documented fully in the third transcript) checked
whether the gate work here was actually cross-referenced: `tools/CONTRACT.md` did not mention
the Publish-Remote Gate at all despite it being `publish`'s third exit-42 path - fixed in both
the command-table entry and the error-contract entry, since a tool's error contract is exactly
where a caller learns what a given exit code means and whether retrying is safe. `gates.md` did
not link to `private-instance.md`, the procedure it exists for - fixed. The project's own KB
page (`kb/entities/projects/Chemenu.md`) still described the instance as a personal wiki with no
mention of the license or its public, testbed-and-demo status - rewritten, keeping the
historical note about the pre-restructuring `AGENTS.md` but dating it explicitly rather than
stating it as a current fact. Published as 2.2.2.
## Outcome
- **Version:** 2.1.1 -> 2.2.0 -> 2.2.2
- **Commits:** Publish-Remote Gate + `private-instance.md` (2.2.0, Mass-Update Gate cleared),
`INSTALL.md` correction (unversioned prose), documentation wiring + project-page rewrite
(2.2.2)
- **Tests:** 752, all green throughout (12 new for the gate)
- **Issues:** #6 title fixed; #7, #10 commented; #27, #28, #29 opened
- **CI:** green on every push in this transcript's scope; Gitea Actions run history removed by
Torben via the API, verified anonymously empty afterward
+32 -6
View File
@@ -50,14 +50,14 @@ tools/wikitool <command> --help
| `log append --op ingest\|query\|lint\|create\|update\|delete\|rename --title "..." [--body "..."\|--body-file path]` | Append a formatted entry to `kb/log.md` | | `log append --op ingest\|query\|lint\|create\|update\|delete\|rename --title "..." [--body "..."\|--body-file path]` | Append a formatted entry to `kb/log.md` |
| `log status` | Read-only: count `ingest` entries logged since the last `lint` entry - the deterministic trigger behind the Maintenance Schedule's "every 10 sources" full-lint cadence | | `log status` | Read-only: count `ingest` entries logged since the last `lint` entry - the deterministic trigger behind the Maintenance Schedule's "every 10 sources" full-lint cadence |
| `lint [--json] [--markdown out.md] [--full] [--fail-on-error]` | Structural + provenance checks: broken wikilinks, dangling frontmatter references, orphan pages, index drift, schema gaps, duplicate titles, title mismatches, uncovered raw files, broken `raw_files:` refs, raw files claimed by more than one source page, unmarked provenance, citation/frontmatter drift, quote-limit overages (>2 blockquoted lines/page, advisory only). Prints only the sections that found something and always writes the full report to `reports/Lint Report <date>.md` (or `--markdown`), naming the path - `--full` prints everything, `--json` prints the findings and writes nothing | | `lint [--json] [--markdown out.md] [--full] [--fail-on-error]` | Structural + provenance checks: broken wikilinks, dangling frontmatter references, orphan pages, index drift, schema gaps, duplicate titles, title mismatches, uncovered raw files, broken `raw_files:` refs, raw files claimed by more than one source page, unmarked provenance, citation/frontmatter drift, quote-limit overages (>2 blockquoted lines/page, advisory only). Prints only the sections that found something and always writes the full report to `reports/Lint Report <date>.md` (or `--markdown`), naming the path - `--full` prints everything, `--json` prints the findings and writes nothing |
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary/confidence so a hit can be judged without opening the page. Read-only, and **exempt from the Iteration Budget Gate** | | `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary/confidence so a hit can be judged without opening the page. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
| `confidence decay [--apply]` | Recompute every page's derived `confidence` as `confidence_base * (1 - 0.01/month)`, floored at 0.2; dry-run by default | | `confidence decay [--apply]` | Recompute every page's derived `confidence` as `confidence_base * (1 - 0.01/month)`, floored at 0.2; dry-run by default |
| `confidence init-base [--apply]` | One-time backfill: set `confidence_base` from the current `confidence` on pages that predate the derived-confidence model | | `confidence init-base [--apply]` | One-time backfill: set `confidence_base` from the current `confidence` on pages that predate the derived-confidence model |
| `sources coverage [--json]` | List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages | | `sources coverage [--json]` | List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages |
| `sources trace --raw <path>` \| `--page "<Title>"` | Trace provenance in either direction: raw file -> source page(s) -> citing pages, or page -> its sources -> their raw files | | `sources trace --raw <path>` \| `--page "<Title>"` | Trace provenance in either direction: raw file -> source page(s) -> citing pages, or page -> its sources -> their raw files |
| `sources rebuild-index [--dry-run]` | Regenerate the `kb/provenance.md` reverse index (raw file -> source page -> citing pages) | | `sources rebuild-index [--dry-run]` | Regenerate the `kb/provenance.md` reverse index (raw file -> source page -> citing pages) |
| `sync [--remote origin] [--branch main] [--confirm-rebase TOKEN]` | Fetch `<remote>/<branch>` and bring the local branch up to date with it: fast-forward when the remote is simply ahead, rebase local commit(s) on top when both sides moved but touch disjoint files (a content conflict is then impossible by construction), and exit **42** for review when they touch the same file (the **rebase-review gate** - see `publish` below). Never commits, never pushes, never force-anything - no remote configured, or one that cannot be reached, is reported and skipped, not a failure. Meant to run once at the start of a writing session (`instructions/session-setup.md`) so the rest of it works against a current tree instead of discovering the drift at the final `publish` | | `sync [--remote origin] [--branch main] [--confirm-rebase TOKEN]` | Fetch `<remote>/<branch>` and bring the local branch up to date with it: fast-forward when the remote is simply ahead, rebase local commit(s) on top when both sides moved but touch disjoint files (a content conflict is then impossible by construction), and exit **42** for review when they touch the same file (the **rebase-review gate** - see `publish` below). Never commits, never pushes, never force-anything - no remote configured, or one that cannot be reached, is reported and skipped, not a failure. Meant to run once at the start of a writing session (`instructions/session-setup.md`) so the rest of it works against a current tree instead of discovering the drift at the final `publish` |
| `publish --message "<op>: <desc>" [--no-push] [--confirm TOKEN] [--confirm-rebase TOKEN] [--threshold N] [--remote origin] [--branch main] [--path P ...]` | Reconcile with `<remote>/<branch>` exactly like `sync` (skipped for `--no-push`), then stage all changes, commit, and push. If the reconcile step found a still-unpushed local commit and there is nothing new to stage, that commit is pushed anyway - a previous `publish` whose push failed no longer strands it. If the push is rejected despite the pre-check (a genuine race - something landed on the remote in between), one more reconcile-and-retry is attempted before giving up; never more than one. **Mass-Update Gate:** when >= `--threshold` (default 10) *counted* files would be committed, exits **42 (`EXIT_NEEDS_CLEARANCE`)** instead of publishing - a third outcome distinct from success (0) and a validation error (1) - and prints a review report: a scale line (file count, total lines added/removed, status breakdown), only-what-applies attention notes (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn, generated files split out as needing no review. The token digests each counted path **and its contents** plus the publish target, so a clearance carries neither to a different file list nor to edited contents; a wrong, invented or superseded token exits 42 again with the current state. Two kinds of path are committed but never counted and never shown for approval: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md`) - each is recomputable from the tree, so approving it decides nothing, and a routine ingest rebuilds five or six of them. The refusal line accounts for both, by reason. The gate is evaluated *before* anything is staged, so a refused publish leaves the working tree untouched. `--yes`/`-y` are gone and now fail with an explicit error. `--path` (repeatable) scopes the whole operation - gate count, staging, and commit - to a subtree | | `publish --message "<op>: <desc>" [--no-push] [--confirm TOKEN] [--confirm-rebase TOKEN] [--threshold N] [--remote origin] [--branch main] [--path P ...]` | Reconcile with `<remote>/<branch>` exactly like `sync` (skipped for `--no-push`), then stage all changes, commit, and push. If the reconcile step found a still-unpushed local commit and there is nothing new to stage, that commit is pushed anyway - a previous `publish` whose push failed no longer strands it. If the push is rejected despite the pre-check (a genuine race - something landed on the remote in between), one more reconcile-and-retry is attempted before giving up; never more than one. **Mass-Update Gate:** when >= `--threshold` (default 10) *counted* files would be committed, exits **42 (`EXIT_NEEDS_CLEARANCE`)** instead of publishing - a third outcome distinct from success (0) and a validation error (1) - and prints a review report: a scale line (file count, total lines added/removed, status breakdown), only-what-applies attention notes (deletions by name, control-plane and harness-config touches, published pages, the largest single change, binaries), and every counted path grouped by area with its status and churn, generated files split out as needing no review. The token digests each counted path **and its contents** plus the publish target, so a clearance carries neither to a different file list nor to edited contents; a wrong, invented or superseded token exits 42 again with the current state. Two kinds of path are committed but never counted and never shown for approval: anything under `work/`, and the files `wikitool` generates itself (`kb/index.md`, `kb/log.md`, `kb/provenance.md`, every `INDEX.md`) - each is recomputable from the tree, so approving it decides nothing, and a routine ingest rebuilds five or six of them. The refusal line accounts for both, by reason. The gate is evaluated *before* anything is staged, so a refused publish leaves the working tree untouched. **Publish-Remote Gate:** when this checkout carries a `.wikitool-remotes.json` and the resolved push URL of `--remote` is not listed in it, exits **42** before the reconcile step even fetches - the URL is read from `git remote get-url --push`, so a repointed remote does not pass on its name. Unlike the other two gates it has **no token and no flag**: the way past it is the user adding the URL to that file, and an agent editing it to get past a refusal is opening a gate on its own initiative. Absent file means unrestricted; a malformed one is an error, not permission. See [instructions/gates.md](../instructions/gates.md) `--yes`/`-y` are gone and now fail with an explicit error. `--path` (repeatable) scopes the whole operation - gate count, staging, and commit - to a subtree |
| `work new (--input <raw path> \| --key <run key>) [--again] [--dry-run]` | Scaffold `work/<runkey>/` for one workshop run: refuses a collision instead of suffixing it, and writes the required `README.md` + `plan.md`. `--input` derives the run key from the path below `raw/` (an ingest); `--key` names it outright for a run with no raw input - a migration or a sweep across `kb/` - and may not start with `ingest-`, which stays reserved for derived keys. Exactly one of the two. `--again` opens a dated second pass over a tree that has itself changed. See [work/CONTRACT.md](../work/CONTRACT.md) | | `work new (--input <raw path> \| --key <run key>) [--again] [--dry-run]` | Scaffold `work/<runkey>/` for one workshop run: refuses a collision instead of suffixing it, and writes the required `README.md` + `plan.md`. `--input` derives the run key from the path below `raw/` (an ingest); `--key` names it outright for a run with no raw input - a migration or a sweep across `kb/` - and may not start with `ingest-`, which stays reserved for derived keys. Exactly one of the two. `--again` opens a dated second pass over a tree that has itself changed. See [work/CONTRACT.md](../work/CONTRACT.md) |
| `work close --run-key <name> [--yes] [--dry-run]` | Delete a finished workshop. Lists what would be lost and requires `--yes`, because nothing in it is recoverable from the rest of the repo - the durable conclusions must already be in `kb/` | | `work close --run-key <name> [--yes] [--dry-run]` | Delete a finished workshop. Lists what would be lost and requires `--yes`, because nothing in it is recoverable from the rest of the repo - the durable conclusions must already be in `kb/` |
| `budget status` | Show the current session's `wikitool` call count and recent command history (never counted against the budget) | | `budget status` | Show the current session's `wikitool` call count and recent command history (never counted against the budget) |
@@ -84,8 +84,34 @@ tools/wikitool <command> --help
## Design notes ## Design notes
- All commands operate on the real repo (paths resolved relative to this - All commands operate on the real repo, so they can be run from any working
file's location), so they can be run from any working directory. directory. The root is resolved by precedence - an explicit argument, then
`$CHEMENU_ROOT`, then a walk up from the package's own location - and the
walk-up is the default, so `tools/wikitool` with no variable set behaves
exactly as it always has. Nothing under the root is bound at import time:
`KB_DIR`, `RAW_DIR` and the rest follow whatever `ROOT` currently is, which
is what makes the half-repointed state (a moved `ROOT` with a stale `KB_DIR`)
unconstructible rather than merely discouraged.
- **The library boundary.** `chemenu.api.Corpus` is the in-process entry point:
it takes a corpus root, returns the same structures the `--json` forms print,
and raises `ChemenuError` where the CLI prints `ERROR` and exits 1. It is
read-only *structurally* - nothing under `chemenu.commands` is imported from
it, so `new`, `publish` and the rest are not reachable, rather than filtered.
The cores it calls (`search/service.py`, `lint_core.py`, `types_core.py`)
import no `typer` and no `rich`; the modules under `commands/` are the
terminal adapters over them. A second consumer is therefore a second adapter,
not a second implementation.
- **The MCP read server** (`chemenu/mcp/`) is that second adapter: `search`,
`types`, `describe_type`, `lint` and `status` over `chemenu.api`, on `stdio`
or `streamable-http`. Its dependency is optional and lives in
`requirements-mcp.txt`, so a CLI-only instance does not install it. There is
no write tool - nothing under `commands/` is importable from it, so the write
functions are unreachable rather than filtered - and every response carries
the commit it was computed from. Running it, and keeping its checkout
current, is [instructions/mcp-read-server.md](../instructions/mcp-read-server.md).
Authentication and rate limiting are middleware in front of the process, not
code here; the Iteration Budget Gate is deliberately not applied, because it
bounds an agent session rather than a user.
- `new`/`xref`/`log append` only produce structurally-correct frontmatter and - `new`/`xref`/`log append` only produce structurally-correct frontmatter and
body skeletons/edits - the prose (Description, Summary, judgment calls body skeletons/edits - the prose (Description, Summary, judgment calls
about relationships) is still written by the LLM afterwards. about relationships) is still written by the LLM afterwards.
@@ -151,10 +177,10 @@ is atomic, and whether a retry is safe.
| `index rebuild` / `sources rebuild-index` | Rare I/O error only | Yes - the file is regenerated from scratch | Safe to retry freely || `log append` | Invalid `--op` or unreadable `--body-file` | Yes - single append | **Not idempotent.** If the previous run's outcome is uncertain, check the tail of `kb/log.md` before retrying | | `index rebuild` / `sources rebuild-index` | Rare I/O error only | Yes - the file is regenerated from scratch | Safe to retry freely || `log append` | Invalid `--op` or unreadable `--body-file` | Yes - single append | **Not idempotent.** If the previous run's outcome is uncertain, check the tail of `kb/log.md` before retrying |
| `log status` | Never fails (reports 0 if `kb/log.md` is missing or empty) | Read-only | Safe to retry freely | | `log status` | Never fails (reports 0 if `kb/log.md` is missing or empty) | Read-only | Safe to retry freely |
| `lint` | Only with `--fail-on-error`: hard findings exist | Writes one report file (single atomic write) unless `--json` | Safe to retry freely, but re-run it to re-*measure*, never to re-read: the printed path holds the full report. Exit 1 means "act on the findings", not "the tool is broken" | | `lint` | Only with `--fail-on-error`: hard findings exist | Writes one report file (single atomic write) unless `--json` | Safe to retry freely, but re-run it to re-*measure*, never to re-read: the printed path holds the full report. Exit 1 means "act on the findings", not "the tool is broken" |
| `search` | `rg` is not installed, a malformed `--field` predicate, an unknown field name, or an unknown `--backend` | Read-only | Fix the argument and retry. An unknown field name is reported with the list of fields that do exist - it is never answered with an empty result, because that would read as "no such pages" | | `search` | `rg` is not installed or did not finish within 30 s, a malformed `--field` predicate, an unknown field name, or an unknown `--backend` | Read-only | Fix the argument and retry. A timeout is a pathological pattern or an unresponsive corpus directory, not a slow answer - narrow the query or drop `--regex` rather than retrying it unchanged. An unknown field name is reported with the list of fields that do exist - it is never answered with an empty result, because that would read as "no such pages" |
| `confidence decay --apply` / `init-base --apply` | Rare I/O error mid-loop | No - one write per page | Safe to retry freely; both recompute from `confidence_base` and never compound | | `confidence decay --apply` / `init-base --apply` | Rare I/O error mid-loop | No - one write per page | Safe to retry freely; both recompute from `confidence_base` and never compound |
| `sync` | The automatic rebase hit a real conflict (git failed) | No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure | For a conflict: **do not retry, do not force** - resolve manually and re-run. **Exit 42, not 1**, when the rebase-review gate needs clearance: show the user the command's full output verbatim (upstream commits, the overlapping files, their diff) and stop; re-running with `--confirm-rebase <token>` clears it, and a wrong, invented, or superseded token exits 42 again with the current state. No remote configured, or one that cannot be reached, is not a failure - reported and skipped | | `sync` | The automatic rebase hit a real conflict (git failed) | No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure | For a conflict: **do not retry, do not force** - resolve manually and re-run. **Exit 42, not 1**, when the rebase-review gate needs clearance: show the user the command's full output verbatim (upstream commits, the overlapping files, their diff) and stop; re-running with `--confirm-rebase <token>` clears it, and a wrong, invented, or superseded token exits 42 again with the current state. No remote configured, or one that cannot be reached, is not a failure - reported and skipped |
| `publish` | git failed, **or** `--yes`/`-y` was passed. **Exit 42, not 1**, when the Mass-Update Gate or the rebase-review gate (raised by the same reconcile `sync` performs) needs clearance | No - sequential git operations, but both gates run before staging | For git failures: **do not retry, do not force** - report and ask the user (the reconcile step already retried the push once on its own, if a rebase resolved the rejection). For exit 42: show the user the command's full output verbatim and stop; it names the evidence and the `--confirm <token>` or `--confirm-rebase <token>` line to re-run, and re-running without it exits 42 again | | `publish` | git failed, **or** `--yes`/`-y` was passed. **Exit 42, not 1**, when the Mass-Update Gate, the rebase-review gate (raised by the same reconcile `sync` performs), or the Publish-Remote Gate refuses | No - sequential git operations, but both gates run before staging | For git failures: **do not retry, do not force** - report and ask the user (the reconcile step already retried the push once on its own, if a rebase resolved the rejection). For exit 42: show the user the command's full output verbatim and stop; it names the evidence and the `--confirm <token>` or `--confirm-rebase <token>` line to re-run, and re-running without it exits 42 again. The Publish-Remote Gate is the exception with no such line: it names the push URL that would have been written to and the ones this checkout allows, and only the user resolves it |
| `work new` | Neither or both of `--input`/`--key` given, `--input` outside `raw/`, a `--key` that is empty or starts with `ingest-`, or the workshop already exists | Yes - one directory with two files | A collision is not transient: resume the existing run instead, or pass `--again` if the tree itself changed. Never create a numbered variant by hand | | `work new` | Neither or both of `--input`/`--key` given, `--input` outside `raw/`, a `--key` that is empty or starts with `ingest-`, or the workshop already exists | Yes - one directory with two files | A collision is not transient: resume the existing run instead, or pass `--again` if the tree itself changed. Never create a numbered variant by hand |
| `work close` | Unknown run key, or `--yes` was not passed | No - a recursive delete | For "not confirmed": check the listed files are no longer needed, confirm the conclusions are in `kb/`, then re-run with `--yes` | | `work close` | Unknown run key, or `--yes` was not passed | No - a recursive delete | For "not confirmed": check the listed files are no longer needed, confirm the conclusions are in `kb/`, then re-run with `--yes` |
| `sources coverage` / `sources trace` | Bad arguments (e.g. neither or both of `--raw`/`--page`) | Read-only | Fix the argument and retry | | `sources coverage` / `sources trace` | Bad arguments (e.g. neither or both of `--raw`/`--page`) | Read-only | Fix the argument and retry |
+23 -3
View File
@@ -34,20 +34,40 @@ tools/
wikitool entry point wikitool entry point
chemenu/ chemenu/
cli.py Typer app: registers every command, runs the budget gate cli.py Typer app: registers every command, runs the budget gate
config.py repo layout constants (ROOT, RAW_DIR, KB_DIR, WORK_DIR, ...) config.py repo layout: root resolution and every path under it
api.py the in-process entry point - point Chemenu at a corpus and read it
errors.py ChemenuError / ValidationError / BackendError
corpus_cache.py one parsed corpus per commit, never cached while the tree is dirty
kb_scan.py page iteration/loading over kb/ kb_scan.py page iteration/loading over kb/
kb_collections.py collection discovery (a directory with COLLECTION.md) kb_collections.py collection discovery (a directory with COLLECTION.md)
type_resolver.py type-spec loading and schema resolution type_resolver.py type-spec loading and schema resolution
lint_core.py the lint checks and the report, with no CLI attached
types_core.py type-spec listing/description, with no CLI attached
sections.py the section headings the tool reads and writes in a page body sections.py the section headings the tool reads and writes in a page body
markdown_code.py masks code spans/fences so a page may show wiki notation, not only use it markdown_code.py masks code spans/fences so a page may show wiki notation, not only use it
version.py the stack version: VERSION, the release stamp, the compatibility rule version.py the stack version: VERSION, the release stamp, the compatibility rule
kb_state.py the KB version (.wikitool-kb.json) and the migration chain kb_state.py the KB version (.wikitool-kb.json) and the migration chain
corpus_diff.py invariant comparison of kb/ between two revisions corpus_diff.py invariant comparison of kb/ between two revisions
search/ pluggable search backends (base protocol, ripgrep, filters, fuse) search/ pluggable search backends, plus service.py - the search core
commands/ one module per command or command group commands/ one module per command or command group: the terminal adapters
tests/ pytest suite tests/ pytest suite
``` ```
**Two consumers, one core.** The CLI is not the only caller any more. The cores
(`search/service.py`, `lint_core.py`, `types_core.py`) hold what decides an
answer and import no `typer` and no `rich`; the modules under `commands/` turn
those values into terminal output and those exceptions into exit codes.
`api.Corpus` is the in-process entry point over the same functions - it takes a
corpus root, returns exactly the structures the `--json` forms print, and
raises instead of exiting. A second consumer is therefore a second adapter
rather than a second implementation, and the write commands are unreachable
from `api` because nothing under `commands/` is imported there.
**Which corpus.** The root resolves by precedence: an explicit argument, then
`$CHEMENU_ROOT`, then a walk up from the package's own location. The walk-up is
the default, so the CLI is unaffected by any of this. Nothing under the root is
bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.
## Adding a command ## Adding a command
1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()` 1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
+202
View File
@@ -0,0 +1,202 @@
"""The in-process entry point: point Chemenu at a corpus and read from it.
This is the seam the MCP server (Gitea #19) is built on, and the reason it is
worth having as its own module rather than as "call the command functions
yourself": it fixes the two things that made an in-process caller a second-class
one.
**A corpus you name, not the one this package happens to sit in.** `Corpus`
holds the root, threads it into the loader and the search backend, and points
`config` at it for the duration of each call so the parts that reach for
`config` directly - the `TypeResolver` singleton, which has to find `types/` -
follow too. A test proves no path of the developer's checkout is read while a
foreign root is set.
**Values and exceptions, not exit codes.** The functions return the same
structures the CLI's `--json` forms print - one wire contract, with the CLI as
its executable specification - and raise `ChemenuError` where the CLI would
print `ERROR` and leave through `typer.Exit(1)`.
Read-only, structurally: nothing under `chemenu.commands` is imported, so `new`,
`touch`, `xref`, `cite`, `publish`, `migrate` and `version bump` are not
reachable from here at all. That is the property #19 asks for - the write
functions do not exist in this surface rather than being filtered out of it.
Import cost is the whole read core and nothing else: `yaml` and `jsonschema`,
plus the standard library. No `typer`, no `rich`.
"""
from __future__ import annotations
import datetime
from pathlib import Path
from typing import Any, Iterable, Optional
from chemenu import config
from chemenu.corpus_cache import CorpusCache
from chemenu.errors import BackendError, ChemenuError, ValidationError
from chemenu.lint_core import run_lint
from chemenu.search import filters
from chemenu.search.registry import resolve
from chemenu.search.service import run_search, unreadable_pages
from chemenu.search.types import Predicate, SearchQuery
from chemenu.types_core import describe_type, list_types
# Distinguishes "the caller did not pass a revision" from "the caller passed
# None", which is itself a meaningful answer: no commit, because the tree is
# dirty or is not a checkout.
_UNREAD = object()
__all__ = [
"Corpus",
"ChemenuError",
"ValidationError",
"BackendError",
]
class Corpus:
"""One corpus tree, read repeatedly.
`root` follows `config.resolve_root()`: an explicit path, else
`$CHEMENU_ROOT`, else the checkout this package lives in. `kb_dir` defaults
to `<root>/kb` and is separate only because the search backend already
distinguishes the two.
Holds a `CorpusCache`, so a long-lived caller parses the corpus once per
commit instead of once per request - and never answers from a superseded
parse, because a dirty tree is not cached. Not thread-safe: a server serving
concurrent requests holds the lock, for the reason given in
`chemenu/corpus_cache.py`.
"""
def __init__(self, root: Optional[Path | str] = None, kb_dir: Optional[Path | str] = None):
self.root = config.resolve_root(root)
self.kb_dir = Path(kb_dir) if kb_dir is not None else self.root / "kb"
self._cache = CorpusCache(self.kb_dir, self.root)
@property
def revision(self) -> Optional[str]:
"""The commit every answer from this corpus is stamped with, or None
when the tree is dirty or is not a git checkout - in which case the
answer corresponds to no commit, and says so."""
return self._cache.current_revision()
def _rooted(self):
"""Point `config` at this corpus for the duration of one call.
Threading a root through every argument gets the search backend and the
corpus loader, and misses the module-level `TypeResolver` singleton -
which resolves `types/` and is what `Page.kind` goes through. Without
this, a foreign corpus is read with *this* checkout's type specs, and
the page's `kind` is an answer about the wrong instance.
Process-wide while open, so `Corpus` inherits `config.rooted()`'s
thread-safety constraint: one lock per process, held by the caller.
"""
return config.rooted(self.root)
def _stamp(
self, payload: dict[str, Any], revision: Optional[str] = _UNREAD
) -> dict[str, Any]:
"""Every response carries the revision it was computed from.
A stale checkout otherwise answers confidently and wrongly, which is the
failure `SOUL.md` names as the cardinal one. The stamp turns a silent
stale answer into a visible one.
A caller that loaded the corpus passes the revision it got back, rather
than letting this ask again: between the load and the stamp the tree can
move, and the honest answer is the revision the pages actually came
from. Callers that read no pages (`types`) ask for the current one.
"""
if revision is _UNREAD:
revision = self._cache.current_revision()
payload["commit"] = revision
payload["as_of"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
return payload
def search(
self,
text: Optional[str] = None,
predicates: Iterable[str] = (),
regex: bool = False,
limit: int = 20,
sort: Optional[str] = None,
backend: Optional[str] = None,
) -> dict[str, Any]:
"""`wikitool search --json`, as a value.
`predicates` takes the raw `--field` strings, so the CLI and this share
one parser and cannot drift on what `confidence<0.6` means.
"""
raw = list(predicates)
if not text and not raw:
raise ValidationError(
"Nothing to search for: give a query, or at least one predicate."
)
parsed: tuple[Predicate, ...] = tuple(filters.parse_predicate(p) for p in raw)
backends = resolve(backend, self.kb_dir, self.root)
query = SearchQuery(
text=text, predicates=parsed, regex=regex, limit=limit, sort=sort
)
with self._rooted():
pages, revision = self._cache.load()
hits = run_search(query, pages, backends, self.kb_dir)
return self._stamp({
"query": text,
"predicates": [p.render() for p in parsed],
"backend": ",".join(b.name for b in backends),
"count": len(hits),
"results": [hit.as_dict() for hit in hits],
"unreadable": unreadable_pages(pages),
}, revision)
def lint(self) -> dict[str, Any]:
"""`wikitool lint --json`, as a value.
The JSON form only - `lint` without a flag writes a report into
`reports/`, and a read surface does not write into the tree it is
reading.
"""
with self._rooted():
return self._stamp(run_lint(self.kb_dir))
def types(self) -> dict[str, Any]:
"""`wikitool types list --json`, as a value."""
with self._rooted():
return self._stamp({"types": list_types()})
def describe_type(self, name: str) -> dict[str, Any]:
"""`wikitool types describe <name> --json`, as a value. Raises
`UnknownType` (a `ValidationError`) for a name that does not exist."""
with self._rooted():
return self._stamp(describe_type(name))
def status(self) -> dict[str, Any]:
"""A composed snapshot: how big the corpus is and what lint says about
it, without the full report.
Composed here on purpose. There is no `wikitool status` to wrap -
`wiki-status` is a *skill* that assembles `kb/index.md`, `lint` and
`kb/log.md` - so this is a new surface, and saying so is what keeps
anyone from looking for the CLI command it mirrors.
"""
with self._rooted():
pages, revision = self._cache.load()
report = run_lint(self.kb_dir)
collections: dict[str, int] = {}
for page in pages.values():
name = filters.collection_of(page.path, self.kb_dir)
if name:
collections[name] = collections.get(name, 0) + 1
return self._stamp({
"pages": len(pages),
"collections": dict(sorted(collections.items())),
"findings": {
key: len(value)
for key, value in sorted(report.items())
if isinstance(value, list)
},
"unreadable": unreadable_pages(pages),
}, revision)
+47 -1
View File
@@ -21,7 +21,7 @@ import typer
from rich.console import Console from rich.console import Console
from chemenu import config, kb_collections, version as version_mod from chemenu import config, kb_collections, version as version_mod
from chemenu.commands import instructions_cmd from chemenu.commands import git_publish, instructions_cmd
from chemenu.commands._util import rel_path from chemenu.commands._util import rel_path
from chemenu.session import ENV_VAR as SESSION_ENV_VAR from chemenu.session import ENV_VAR as SESSION_ENV_VAR
@@ -242,6 +242,51 @@ def check_environment() -> Check:
return Check("environment", "OK", f"{config.ENVIRONMENT_FILE} present and filled") return Check("environment", "OK", f"{config.ENVIRONMENT_FILE} present and filled")
def check_publish_remotes() -> Check:
"""Whether the Publish-Remote Gate is armed in this checkout.
Absent is a legitimate state, not a fault: a checkout with a single remote
and nothing private in it has nothing to protect, and making the file
mandatory would turn a safeguard into paperwork. So this never FAILs - it
reports, the way `environment` does.
It does WARN for the case that actually bites: more than one remote
configured and no allowlist. That is the shape a private instance has after
it adds the public upstream, and it is exactly when a wrong `--remote`
stops being a typo and starts being a disclosure.
Both absent states say **armed** or **not armed** rather than only naming
the file. AGENTS.md lists this among the three limits enforced in code, so a
line that reports the file's absence and leaves the reader to infer what
that means about the gate is how a checkout ends up trusting a safeguard
that is not running - which is worse than having none.
"""
urls = git_publish.read_allowed_push_urls()
if urls is not None:
return Check(
"publish-remotes", "OK",
f"Gate armed: {len(urls)} allowed push target(s) in "
f"{config.PUBLISH_REMOTES_FILENAME}",
)
result = subprocess.run(
["git", "remote"], cwd=config.ROOT, capture_output=True, text=True
)
remotes = [r for r in result.stdout.split() if r]
if len(remotes) > 1:
return Check(
"publish-remotes", "WARN",
f"Gate not armed: {len(remotes)} remotes ({', '.join(remotes)}) and no "
f"{config.PUBLISH_REMOTES_FILENAME} - every one of them is a legal publish target",
f"Create {config.PUBLISH_REMOTES_FILENAME} naming the push URL this checkout "
"may publish to - see instructions/gates.md",
)
return Check(
"publish-remotes", "OK",
f"Gate not armed: no {config.PUBLISH_REMOTES_FILENAME} - any push target passes "
"(1 remote configured, nothing to confuse it with)",
)
def check_generated_files() -> Check: def check_generated_files() -> Check:
missing = [ missing = [
rel_path(path) rel_path(path)
@@ -355,6 +400,7 @@ def run_doctor() -> list[Check]:
check_structure(), check_structure(),
check_personalization(), check_personalization(),
check_environment(), check_environment(),
check_publish_remotes(),
check_generated_files(), check_generated_files(),
check_session_id(), check_session_id(),
] ]
+100
View File
@@ -80,6 +80,86 @@ def _run(args: list[str]) -> subprocess.CompletedProcess:
return subprocess.run(args, cwd=config.ROOT, capture_output=True, text=True) return subprocess.run(args, cwd=config.ROOT, capture_output=True, text=True)
# --- Publish-Remote Gate -----------------------------------------------------
#
# The Mass-Update Gate asks "is this too much to publish?". This one asks the
# question underneath it: "is this the right place to publish to at all?".
#
# A checkout holding private content typically has two remotes - its own, and
# the public upstream it takes stack updates from. Nothing in git distinguishes
# them at push time, so a single wrong `--remote` puts a private corpus on a
# public repository, where a force-push does not take it back: the objects stay
# fetchable by SHA until someone expires the server's reflogs.
#
# Like the other two gates this refuses with exit 42 and has **no flag that
# opens it**. The way past it is to name the URL in the file, which is an edit
# the user makes deliberately rather than something an agent can decide mid-run.
def read_allowed_push_urls() -> Optional[list[str]]:
"""The URLs this checkout permits `publish` to push to, or None when the
file is absent (unrestricted - see config.PUBLISH_REMOTES_FILENAME)."""
path = config.ROOT / config.PUBLISH_REMOTES_FILENAME
if not path.is_file():
return None
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
fail(
f"{config.PUBLISH_REMOTES_FILENAME} is unreadable ({exc}). It decides where "
"`publish` may push, so a broken one is not treated as 'no restriction' - "
"fix the file or delete it deliberately."
)
return None
urls = data.get("allowed_push_urls")
if not isinstance(urls, list) or not all(isinstance(u, str) for u in urls):
fail(
f"{config.PUBLISH_REMOTES_FILENAME} has no usable `allowed_push_urls` list of "
"strings. Expected: {\"schema\": 1, \"allowed_push_urls\": [\"<url>\"]}"
)
return None
return urls
def push_url_for(remote: str) -> Optional[str]:
"""The URL `git push <remote>` would actually write to - `pushurl` when the
remote sets one, otherwise its fetch URL. Reading the resolved value rather
than the name is the whole point: a repointed `origin` must not pass."""
result = _run(["git", "remote", "get-url", "--push", remote])
if result.returncode != 0:
return None
return result.stdout.strip() or None
def publish_remote_refusal(remote: str, branch: str) -> Optional[str]:
"""The gate's message when `remote` is not an allowed push target, else None."""
allowed = read_allowed_push_urls()
if allowed is None:
return None
url = push_url_for(remote)
if url is None:
return (
f"Publish-Remote Gate: '{remote}' resolves to no push URL, so this publish "
f"cannot be checked against {config.PUBLISH_REMOTES_FILENAME}. Nothing was "
"committed or pushed."
)
if url in allowed:
return None
listed = "\n".join(f" - {u}" for u in allowed) or " (the list is empty)"
return (
f"Publish-Remote Gate: this checkout does not allow publishing to '{remote}'.\n\n"
f" would push to: {url}\n"
f" allowed here:\n{listed}\n\n"
"Nothing was committed or pushed. This checkout holds content that belongs to it "
"alone, and a push to the wrong remote is not cheaply reversible - the objects stay "
"fetchable by SHA even after a force-push, until the server's reflogs are expired.\n\n"
"THE USER CANNOT SEE THIS OUTPUT. It went to your context, not to their screen.\n"
"Show them the two lines above and stop. There is no flag that opens this gate: if "
f"the target really is right, the user adds its URL to {config.PUBLISH_REMOTES_FILENAME} "
"themselves. Do not edit that file to get past this."
)
def parse_porcelain_entries(stdout: str) -> list[tuple[str, str]]: def parse_porcelain_entries(stdout: str) -> list[tuple[str, str]]:
"""Parse `git status --porcelain -z` output into (status_code, path) pairs. """Parse `git status --porcelain -z` output into (status_code, path) pairs.
@@ -919,6 +999,26 @@ def publish_command(
if push and checked_out != branch: if push and checked_out != branch:
fail(branch_mismatch_message(checked_out, branch)) fail(branch_mismatch_message(checked_out, branch))
# Before the reconcile below, which is the first thing that talks to the
# remote at all: a publish aimed at the wrong repository should not even
# fetch from it, and the refusal message promises that nothing was
# committed or pushed.
if push:
refusal = publish_remote_refusal(remote, branch)
if refusal:
emit(
"wikitool",
"gate.refused",
{
"gate": "publish-remote",
"reason": "remote-not-allowed",
"remote": remote,
"branch": branch,
"url": push_url_for(remote),
},
)
needs_clearance(refusal)
# Pull against the remote before anything else, to minimise the window in which this # Pull against the remote before anything else, to minimise the window in which this
# publish could diverge from it - and, as a side effect, to finally publish a commit left # publish could diverge from it - and, as a side effect, to finally publish a commit left
# stranded by a previous push that failed (see `_local_ahead_of_remote` below). Skipped # stranded by a previous push that failed (see `_local_ahead_of_remote` below). Skipped
+28 -416
View File
@@ -1,16 +1,12 @@
"""Deterministic structural health checks for the wiki. """`wikitool lint` - the terminal adapter over `chemenu.lint_core`.
This intentionally covers only what can be computed mechanically: broken The checks, the report and the hard-error rule live in `chemenu/lint_core.py`,
wikilinks, orphan pages, index/page drift, frontmatter schema gaps, and which imports no CLI machinery. This module owns only what a terminal needs:
filename/title mismatches. Semantic judgment (contradictions, staleness, the flags, where the report file lands, and the exit code.
what's worth writing about next) stays with the LLM - this report gives it a
verified factual foundation instead of requiring it to re-derive these facts
by reading every page.
""" """
from __future__ import annotations from __future__ import annotations
import json import json
from datetime import date
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
@@ -18,416 +14,32 @@ import typer
from chemenu import config from chemenu import config
from chemenu.commands._util import rel_path, success from chemenu.commands._util import rel_path, success
from chemenu.frontmatter_io import frontmatter_error from chemenu.lint_core import (
from chemenu.markdown_code import strip_code_spans HARD_ERROR_KEYS,
from chemenu.provenance import broken_raw_refs as find_broken_raw_refs MOST_LINKED_COUNT,
from chemenu.provenance import duplicate_raw_file_owners as find_duplicate_raw_file_owners QUOTE_LIMIT,
from chemenu.provenance import extract_inline_cites count_quote_blocks,
from chemenu.provenance import legacy_citation_markers as find_legacy_citation_markers default_report_path,
from chemenu.provenance import legacy_source_pages as find_legacy_source_pages has_hard_errors,
from chemenu.provenance import orphan_footnote_defs as find_orphan_footnote_defs render_markdown,
from chemenu.provenance import uncovered_raw_files as find_uncovered_raw_files render_summary,
from chemenu.provenance import undefined_footnote_refs as find_undefined_footnote_refs run_lint,
from chemenu.kb_scan import (
GENERATED_INDEX,
WIKILINK_RE,
build_link_graph,
find_duplicate_title_paths,
inbound_links,
load_kb_pages,
)
from chemenu.type_resolver import resolver
# Style guide's one mechanically-checkable rule (hard oracle: a plain count).
# The rest of the style guide (tone, AI-phrase avoidance) is a soft/proxy judgment
# and stays with the LLM - see wiki-manage/wiki-ingest skill guidance, not lint.
#
# The unit is a quote, not a `>` line. It used to be the line, which measured
# the wrap width the rule has no opinion about: one quotation written long
# counted 1 and the same quotation wrapped at 100 columns counted 4. An author
# who took the finding seriously made the page harder to read to quiet it.
QUOTE_LIMIT = 2
# How many hub pages `most_linked` reports. Purely informational (wiki-status
# surfaces it); not a finding, so the cutoff only bounds report size.
MOST_LINKED_COUNT = 10
def count_quote_blocks(body: str) -> int:
"""How many distinct blockquotes `body` carries.
A run of consecutive `>` lines is one quote; a blank line or any
non-quoted line ends it. Code is masked out first, so a `>` inside a
fenced shell transcript is a prompt, not a quotation.
Lazy continuation - a quote whose wrapped lines drop the `>` - reads here
as two quotes rather than one. That over-counts in the direction the limit
already errs on, and the corpus prefixes every line, so the alternative
(tracking paragraph state) buys nothing.
"""
count, in_quote = 0, False
for line in strip_code_spans(body).splitlines():
is_quote = line.lstrip().startswith(">")
if is_quote and not in_quote:
count += 1
in_quote = is_quote
return count
def run_lint(kb_dir: Path) -> dict:
pages = load_kb_pages(kb_dir)
duplicate_titles = find_duplicate_title_paths(kb_dir, config.ROOT)
# Pages whose frontmatter can't be parsed read back as `{}` everywhere
# else, which would let them slip past every frontmatter-driven check
# below with no finding at all - so they are detected explicitly.
frontmatter_errors = []
for title, page in sorted(pages.items()):
reason = frontmatter_error(page.path)
if reason is None and not page.frontmatter.get("type"):
reason = "missing `type:` field"
if reason is not None:
frontmatter_errors.append({"page": title, "error": reason})
graph = build_link_graph(pages)
broken_links = [
{"page": title, "target": target}
for title, targets in graph.items()
for target in sorted(targets)
if target not in pages
]
inbound = inbound_links({t: v for t, v in graph.items() if t != "index"})
orphan_pages = sorted(
title
for title, sources in inbound.items()
if not sources
and title not in ("index", "log")
# comparison pages are not linked to by design; index.md is sufficient coverage
and pages[title].kind != "comparison"
)
# Same link graph, opposite end: the most-linked-to pages are the wiki's
# hubs. Reported (not judged) so `wiki-status` can show them without
# re-deriving the graph.
inbound_counts = {title: len(sources) for title, sources in inbound.items()}
most_linked = [
{"page": title, "inbound": count}
for title, count in sorted(inbound_counts.items(), key=lambda kv: (-kv[1], kv[0]))
if count > 0
][:MOST_LINKED_COUNT]
# The catalog is sharded: `kb/index.md` is a map carrying counts and links,
# and the page rows live in a generated INDEX.md per collection/area. Both
# halves have to be read, or every page reads as missing from the index.
index_text = "".join(
path.read_text(encoding="utf-8")
for path in [kb_dir / "index.md", *sorted(kb_dir.rglob(GENERATED_INDEX))]
if path.exists()
)
index_links = {m.group(1).strip() for m in WIKILINK_RE.finditer(index_text)}
missing_from_index = sorted(set(pages) - index_links - {"index", "log"})
dangling_index_entries = sorted(index_links - set(pages))
title_mismatches = []
for title, page in sorted(pages.items()):
if page.kind not in ("entity", "concept"):
continue
h1 = page.h1_title
if h1 is not None and h1 != title:
title_mismatches.append({"page": title, "h1": h1})
unmarked_provenance = []
for title, page in sorted(pages.items()):
if page.kind not in ("entity", "concept"):
continue
sources_list = page.frontmatter.get("sources") or []
if not sources_list and page.frontmatter.get("provenance") != "general":
unmarked_provenance.append(title)
citation_frontmatter_drift = []
for title, page in sorted(pages.items()):
sources_list = set(page.frontmatter.get("sources") or [])
cited = {cited_title for cited_title, _file in extract_inline_cites(page.body)}
cited.discard(title) # a source page citing itself for a specific file within it is not drift
for missing_source in sorted(cited - sources_list):
citation_frontmatter_drift.append({"page": title, "cited_but_not_in_sources": missing_source})
legacy_citation_markers = find_legacy_citation_markers(pages)
undefined_footnote_refs = find_undefined_footnote_refs(pages)
orphan_footnote_defs = find_orphan_footnote_defs(pages)
# The frontmatter half of the link graph. `broken_links` above only walks
# `[[wikilinks]]` in page *bodies*, so a `related:`/`sources:`/`entities:`
# entry naming a page that does not exist - a rename that was not
# propagated, a deleted page, or a URL pasted where a title belongs - used
# to pass every check. Which fields hold page titles is declared by each
# type-spec's `page_ref_fields:`, not hardcoded here.
dangling_frontmatter_refs = []
for title, page in sorted(pages.items()):
type_path = page.frontmatter.get("type")
if not type_path:
continue
try:
ref_fields = resolver.get_page_ref_fields(type_path, page.path)
except ValueError:
continue # unresolvable type is already reported as type_resolution_errors
for field in ref_fields:
for target in page.frontmatter.get(field) or []:
if target not in pages:
dangling_frontmatter_refs.append(
{"page": title, "field": field, "target": target}
)
quote_limit_violations = []
for title, page in sorted(pages.items()):
quote_count = count_quote_blocks(page.body)
if quote_count > QUOTE_LIMIT:
quote_limit_violations.append({"page": title, "quote_count": quote_count})
# Type system validation. Lint reports are not validated here: they are
# written to `reports/` outside kb/ and are never pages, so nothing this
# loop scans can be one.
invalid_type_paths = []
type_resolution_errors = []
schema_validation_errors = []
for title, page in sorted(pages.items()):
type_path = page.frontmatter.get("type")
if not type_path:
continue
# Check if type path is valid
if not type_path.endswith('.md'):
invalid_type_paths.append({"page": title, "type": type_path, "error": "Type path must end with .md"})
continue
# Try to resolve and validate the type
try:
resolver.load_type_spec(type_path, page.path)
# Try schema validation
try:
resolver.validate_frontmatter(page.frontmatter, type_path, page.path)
except ValueError as schema_error:
schema_validation_errors.append({"page": title, "type": type_path, "error": str(schema_error)})
except ValueError as resolution_error:
type_resolution_errors.append({"page": title, "type": type_path, "error": str(resolution_error)})
return {
"generated": date.today().isoformat(),
"page_count": len(pages),
"frontmatter_errors": frontmatter_errors,
"broken_links": broken_links,
"orphan_pages": orphan_pages,
"most_linked": most_linked,
"inbound_counts": inbound_counts,
"missing_from_index": missing_from_index,
"dangling_index_entries": dangling_index_entries,
"title_mismatches": title_mismatches,
"duplicate_titles": duplicate_titles,
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
"broken_raw_refs": find_broken_raw_refs(pages),
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
"legacy_source_pages": find_legacy_source_pages(pages),
"unmarked_provenance": unmarked_provenance,
"citation_frontmatter_drift": citation_frontmatter_drift,
"legacy_citation_markers": legacy_citation_markers,
"undefined_footnote_refs": undefined_footnote_refs,
"orphan_footnote_defs": orphan_footnote_defs,
"dangling_frontmatter_refs": dangling_frontmatter_refs,
"quote_limit_violations": quote_limit_violations,
"invalid_type_paths": invalid_type_paths,
"type_resolution_errors": type_resolution_errors,
"schema_validation_errors": schema_validation_errors,
}
def _section(lines: list[str], title: str, items: list, formatter) -> None:
lines.append(f"## {title}")
lines.append("")
if not items:
lines.append("None found.")
else:
for item in items:
lines.append(f"- {formatter(item)}")
lines.append("")
def render_markdown(report: dict) -> str:
lines = [f"# Structural Lint Report ({report['generated']})", ""]
lines.append(f"Scanned {report['page_count']} pages under `wiki/`. This report covers only")
lines.append("mechanically-verifiable structural issues; see the Semantic Review section")
lines.append("below for judgment calls the LLM should complete.")
lines.append("")
_section(
lines, "Unreadable Frontmatter", report["frontmatter_errors"],
lambda i: f"[[{i['page']}]] - {i['error']}",
)
_section(
lines, "Broken Wikilinks", report["broken_links"],
lambda i: f"[[{i['page']}]] links to missing [[{i['target']}]]",
)
_section(lines, "Orphan Pages (no inbound links)", report["orphan_pages"], lambda i: f"[[{i}]]")
_section(
lines, f"Most-Linked Pages (top {MOST_LINKED_COUNT} hubs)", report["most_linked"],
lambda i: f"[[{i['page']}]] - {i['inbound']} inbound link(s)",
)
_section(lines, "Pages Missing from index.md", report["missing_from_index"], lambda i: f"[[{i}]]")
_section(lines, "Dangling index.md Entries", report["dangling_index_entries"], lambda i: f"[[{i}]]")
_section(
lines, "Duplicate Titles (naming collisions)", report["duplicate_titles"],
lambda i: f"`{i['stem']}` -> {', '.join(f'`{p}`' for p in i['paths'])}",
)
_section(
lines, "Filename / H1 Title Mismatches", report["title_mismatches"],
lambda i: f"[[{i['page']}]] H1 is '{i['h1']}'",
)
_section(
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
lambda i: f"`{i}`",
)
_section(
lines, "Broken raw_files References", report["broken_raw_refs"],
lambda i: f"[[{i['page']}]] -> `{i['raw_path']}` (does not exist)",
)
_section(
lines, "Raw Files With More Than One Owner", report["duplicate_raw_file_owners"],
lambda i: f"`{i['raw_file']}` is claimed by " + ", ".join(f"[[{t}]]" for t in i["owners"]),
)
_section(
lines, "Legacy source: Field (not yet migrated to raw_files:)", report["legacy_source_pages"],
lambda i: f"[[{i['page']}]] source: `{i['source']}` ({i['reason']})",
)
_section(
lines, "Pages Missing provenance: general Marker", report["unmarked_provenance"],
lambda i: f"[[{i}]] has no sources and is not marked `provenance: general`",
)
_section(
lines, "Citation / Frontmatter Drift", report["citation_frontmatter_drift"],
lambda i: f"[[{i['page']}]] cites [[{i['cited_but_not_in_sources']}]] inline but it is missing from frontmatter `sources:`",
)
_section(
lines, "Legacy Citation Markers (pre-migration `^[[...]]`)", report["legacy_citation_markers"],
lambda i: f"[[{i['page']}]] still has `{i['marker']}` - run `wikitool cite add` and replace it with the `[^cite-id]` it prints",
)
_section(
lines, "Undefined Footnote References", report["undefined_footnote_refs"],
lambda i: f"[[{i['page']}]] references `[^{i['ref']}]`, which has no `[^{i['ref']}]: [[...]]` definition",
)
_section(
lines, "Orphan Footnote Definitions", report["orphan_footnote_defs"],
lambda i: f"[[{i['page']}]] defines `[^{i['id']}]` (-> [[{i['source']}]]) but nothing references it - run `wikitool cite sync`",
)
_section(
lines, "Dangling Frontmatter References", report["dangling_frontmatter_refs"],
lambda i: f"[[{i['page']}]] `{i['field']}:` names `{i['target']}`, which is not a page",
)
_section(
lines, "Invalid Type Paths", report["invalid_type_paths"],
lambda i: f"[[{i['page']}]] has type: `{i['type']}` - {i['error']}",
)
_section(
lines, "Type Resolution Errors", report["type_resolution_errors"],
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
)
_section(
lines, "Schema Validation Errors", report["schema_validation_errors"],
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
)
_section(
lines, f"Pages Exceeding Quote Limit (>{QUOTE_LIMIT}/page)", report["quote_limit_violations"],
lambda i: f"[[{i['page']}]] has {i['quote_count']} quotes - trim or confirm they're load-bearing",
)
lines.append("## Semantic Review (LLM to complete)")
lines.append("")
lines.append("- Contradictions across pages: TODO")
lines.append("- Stale claims (unconfirmed >6 months): TODO")
lines.append("- Suggested new pages / missing cross-references: TODO")
lines.append("")
return "\n".join(lines)
# Sections that always carry content but are not findings, so the summary
# handles them separately: a hub list is a statistic, and the semantic review
# is the checklist that follows the report rather than part of it.
INFORMATIONAL_SECTIONS = ("Most-Linked Pages",)
SEMANTIC_REVIEW_SECTION = "Semantic Review"
def _split_sections(markdown: str) -> tuple[str, list[tuple[str, str]]]:
"""Cut a rendered report into its preamble and (title, body) sections."""
preamble, *rest = markdown.split("\n## ")
sections = []
for part in rest:
title, _, body = part.partition("\n")
sections.append((title.strip(), body.strip()))
return preamble.rstrip(), sections
def render_summary(report: dict) -> str:
"""The same report with the empty sections removed.
On a healthy corpus the full report is better than 90% "None found.", so
reading it in the terminal means paging past the answer. The file on disk
stays complete - this is what gets printed, and the written path underneath
it is how the rest is reached without running lint a second time.
"""
preamble, sections = _split_sections(render_markdown(report))
findings, trailing = [], []
for title, body in sections:
if title.startswith(SEMANTIC_REVIEW_SECTION):
trailing.append((title, body))
elif body != "None found." and not title.startswith(INFORMATIONAL_SECTIONS):
findings.append((title, body))
lines = [preamble, ""]
if not findings:
lines += ["No structural findings.", ""]
for title, body in findings + trailing:
lines += [f"## {title}", "", body, ""]
return "\n".join(lines)
def default_report_path(report: dict) -> Path:
"""Where a report goes when the caller names no path.
`reports/` is derived and gitignored ([reports/CONTRACT.md]), so writing
here by default costs the tree nothing.
"""
return config.ROOT / "reports" / f"Lint Report {report['generated']}.md"
# Findings that make a tree structurally wrong rather than merely untidy.
# `orphan_pages` is deliberately absent: many pages are validly reachable
# through the index or navigation only. `quote_limit_violations` is advisory
# too - it flags a habit, not a broken tree.
#
# One definition, used by `lint --fail-on-error` and by the eval scorecard: if
# the two disagreed, a run could pass its score while lint refused it.
HARD_ERROR_KEYS = (
"frontmatter_errors",
"broken_links",
"dangling_index_entries",
"duplicate_titles",
"broken_raw_refs",
"duplicate_raw_file_owners",
"legacy_source_pages",
"citation_frontmatter_drift",
"legacy_citation_markers",
"undefined_footnote_refs",
"orphan_footnote_defs",
"dangling_frontmatter_refs",
"invalid_type_paths",
"type_resolution_errors",
"schema_validation_errors",
) )
# Re-exported: `from chemenu.commands.lint import run_lint` still resolves, and
def has_hard_errors(report: dict) -> bool: # so does every other name the tests and sibling commands already import.
return any(report.get(key) for key in HARD_ERROR_KEYS) __all__ = [
"HARD_ERROR_KEYS",
"MOST_LINKED_COUNT",
"QUOTE_LIMIT",
"count_quote_blocks",
"default_report_path",
"has_hard_errors",
"render_markdown",
"render_summary",
"run_lint",
"lint_command",
]
def lint_command( def lint_command(
+30 -71
View File
@@ -21,90 +21,38 @@ Scope is `kb/` only. `instructions/` is discovered through
from __future__ import annotations from __future__ import annotations
import json import json
from pathlib import Path
import typer import typer
from chemenu import config
from chemenu.commands._util import fail, today_iso from chemenu.commands._util import fail, today_iso
from chemenu.frontmatter_io import read_page
from chemenu.kb_scan import iter_kb_pages
from chemenu.page import Page
from chemenu.search import filters from chemenu.search import filters
from chemenu.search.base import page_key
from chemenu.search.filters import PredicateError from chemenu.search.filters import PredicateError
from chemenu.search.fuse import reciprocal_rank_fusion
from chemenu.search.registry import UnknownBackend, resolve from chemenu.search.registry import UnknownBackend, resolve
from chemenu.search.ripgrep import RipgrepFailed, RipgrepMissing, build_hit from chemenu.search.ripgrep import RipgrepFailed, RipgrepMissing
from chemenu.search.service import (
load_pages_by_path,
run_search,
sort_hits,
unreadable_pages,
)
from chemenu.search.types import Predicate, SearchHit, SearchQuery from chemenu.search.types import Predicate, SearchHit, SearchQuery
# Re-exported so `from chemenu.commands.search import run_search` keeps
# resolving. The core lives in `chemenu/search/service.py`, which imports no
# CLI machinery; this module is the terminal adapter over it.
__all__ = [
"load_pages_by_path",
"run_search",
"sort_hits",
"unreadable_pages",
"render_table",
"search_command",
]
TITLE_WIDTH = 34 TITLE_WIDTH = 34
SUMMARY_WIDTH = 84 SUMMARY_WIDTH = 84
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
"""Every page under `kb/`, keyed by repo-relative path.
Path-keyed rather than title-keyed on purpose: `load_kb_pages()` drops one
of two pages sharing a stem, and search should still find both - a
duplicate title is a lint finding, not a reason to hide a page.
"""
kb_dir = kb_dir or config.KB_DIR
root = root or config.ROOT
pages: dict[str, Page] = {}
for path in iter_kb_pages(kb_dir):
frontmatter, body = read_page(path)
pages[page_key(path, root)] = Page(path=path, frontmatter=frontmatter, body=body)
return pages
def _sort_key(hit: SearchHit, field: str):
value = hit.as_dict().get(field)
if value is None:
# Missing values sort last in either direction rather than crashing on
# a None comparison.
return (1, "")
if isinstance(value, (int, float)):
return (0, value)
return (0, str(value).lower())
def sort_hits(hits: list[SearchHit], sort: str | None) -> list[SearchHit]:
"""Sort by a hit field. A leading `-` reverses, e.g. `--sort -confidence`."""
if not sort:
return hits
descending = sort.startswith("-")
field = sort.lstrip("-")
ordered = sorted(hits, key=lambda h: _sort_key(h, field), reverse=descending)
return ordered
def run_search(
query: SearchQuery,
pages: dict[str, Page],
backends: list,
kb_dir: Path | None = None,
) -> list[SearchHit]:
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
filters.validate_fields(query.predicates, pages)
if query.text:
rankings = [backend.search(query, pages) for backend in backends]
hits = rankings[0] if len(rankings) == 1 else reciprocal_rank_fusion(rankings)
allowed = filters.apply_predicates(pages, query.predicates, kb_dir)
hits = [hit for hit in hits if hit.path in allowed]
else:
selected = filters.apply_predicates(pages, query.predicates, kb_dir)
hits = [
build_hit(page, key, [], query, backend="frontmatter", kb_dir=kb_dir)
for key, page in selected.items()
]
hits.sort(key=lambda h: h.title.lower())
hits = sort_hits(hits, query.sort)
return hits[: query.limit] if query.limit else hits
def _truncate(text: str, width: int) -> str: def _truncate(text: str, width: int) -> str:
text = " ".join(text.split()) text = " ".join(text.split())
return text if len(text) <= width else text[: width - 1] + "\u2026" return text if len(text) <= width else text[: width - 1] + "\u2026"
@@ -206,6 +154,8 @@ def search_command(
except RipgrepFailed as exc: except RipgrepFailed as exc:
fail(str(exc)) fail(str(exc))
unreadable = unreadable_pages(pages)
if json_out: if json_out:
payload = { payload = {
"generated": today_iso(), "generated": today_iso(),
@@ -214,8 +164,17 @@ def search_command(
"backend": ",".join(b.name for b in backends), "backend": ",".join(b.name for b in backends),
"count": len(hits), "count": len(hits),
"results": [hit.as_dict() for hit in hits], "results": [hit.as_dict() for hit in hits],
# Always present, usually empty. A caller that has to look for the
# key to learn whether it should worry will not look.
"unreadable": unreadable,
} }
typer.echo(json.dumps(payload, indent=2)) typer.echo(json.dumps(payload, indent=2))
return return
typer.echo(render_table(hits, show_matches)) typer.echo(render_table(hits, show_matches))
for entry in unreadable:
typer.echo(
f"WARN unreadable frontmatter: {entry['path']} ({entry['reason']}) - "
"this page cannot match any --field predicate",
err=True,
)
+25 -56
View File
@@ -7,35 +7,29 @@ into context on every skill invocation. A type-spec's own frontmatter
declared `.schema.yaml` are the single source of truth; this command only declared `.schema.yaml` are the single source of truth; this command only
formats what `TypeResolver` already resolves - it does not duplicate or formats what `TypeResolver` already resolves - it does not duplicate or
re-derive any type knowledge. re-derive any type knowledge.
The resolving half lives in `chemenu/types_core.py`, which imports no CLI
machinery. This module is the terminal adapter over it.
""" """
from __future__ import annotations from __future__ import annotations
import json import json
from typing import Any, Dict
import typer import typer
from chemenu.commands._util import fail from chemenu.commands._util import fail
from chemenu.type_resolver import resolver from chemenu.types_core import UnknownType, describe_type, list_types
app = typer.Typer(help="Discover and describe Chemenu type-spec contracts.") app = typer.Typer(help="Discover and describe Chemenu type-spec contracts.")
@app.command("list") @app.command("list")
def list_types(json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON")): def list_types_command(
json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON")
):
"""List every type-spec under types/, with its name, schema, subtype """List every type-spec under types/, with its name, schema, subtype
field (if any), base directory, and description.""" field (if any), base directory, and description."""
rows: list[Dict[str, Any]] = [] rows = list_types()
for type_path, frontmatter in resolver.list_type_specs():
rows.append({
"name": frontmatter.get("name"),
"type_path": type_path,
"schema": frontmatter.get("schema"),
"subtype_field": frontmatter.get("subtype_field"),
"root": frontmatter.get("root") or "kb",
"base_dir": frontmatter.get("base_dir"),
"description": frontmatter.get("description"),
})
if json_out: if json_out:
typer.echo(json.dumps(rows, indent=2)) typer.echo(json.dumps(rows, indent=2))
@@ -53,7 +47,7 @@ def list_types(json_out: bool = typer.Option(False, "--json", help="Print raw fi
@app.command("describe") @app.command("describe")
def describe_type( def describe_type_command(
name: str = typer.Argument(..., help="Type name, e.g. 'entity' (see `types list`)"), name: str = typer.Argument(..., help="Type name, e.g. 'entity' (see `types list`)"),
json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON"), json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON"),
): ):
@@ -61,51 +55,26 @@ def describe_type(
with enums where declared), its subtype field if any, and its authoring with enums where declared), its subtype field if any, and its authoring
body - the same information an LLM would otherwise gather by reading the body - the same information an LLM would otherwise gather by reading the
raw type-spec and `.schema.yaml` files directly.""" raw type-spec and `.schema.yaml` files directly."""
type_path = resolver.find_type_by_name(name) try:
if type_path is None: described = describe_type(name)
available = sorted(fm.get("name") for _, fm in resolver.list_type_specs()) except UnknownType as exc:
fail(f"No type-spec named '{name}'. Available: {', '.join(available)}") fail(str(exc))
return # unreachable; keeps type-checkers happy about `type_path` below return # unreachable; keeps type-checkers happy about `described` below
type_spec = resolver.load_type_spec(type_path)
frontmatter = type_spec["frontmatter"]
body = type_spec["body"]
schema = resolver.get_schema(type_path)
fields: list[Dict[str, Any]] = []
if schema is not None:
required = set(schema.get("required", []))
for field_name, field_schema in schema.get("properties", {}).items():
fields.append({
"field": field_name,
"required": field_name in required,
"type": field_schema.get("type"),
"enum": field_schema.get("enum"),
})
if json_out: if json_out:
typer.echo(json.dumps({ typer.echo(json.dumps(described, indent=2))
"name": frontmatter.get("name"),
"type_path": type_path,
"description": frontmatter.get("description"),
"schema": frontmatter.get("schema"),
"subtype_field": frontmatter.get("subtype_field"),
"base_dir": frontmatter.get("base_dir"),
"title_prefix": frontmatter.get("title_prefix"),
"fields": fields,
"body": body.strip(),
}, indent=2))
return return
typer.echo(f"# {frontmatter.get('name')} ({type_path})") fields = described["fields"]
typer.echo(frontmatter.get("description", "")) typer.echo(f"# {described['name']} ({described['type_path']})")
typer.echo(described["description"] or "")
typer.echo("") typer.echo("")
if frontmatter.get("subtype_field"): if described["subtype_field"]:
typer.echo(f"subtype_field: {frontmatter['subtype_field']}") typer.echo(f"subtype_field: {described['subtype_field']}")
if frontmatter.get("base_dir"): if described["base_dir"]:
typer.echo(f"base_dir: {frontmatter.get('root') or 'kb'}/{frontmatter['base_dir']}") typer.echo(f"base_dir: {described['root']}/{described['base_dir']}")
if frontmatter.get("title_prefix"): if described["title_prefix"]:
typer.echo(f"title_prefix: {frontmatter['title_prefix']!r}") typer.echo(f"title_prefix: {described['title_prefix']!r}")
typer.echo("") typer.echo("")
if not fields: if not fields:
@@ -119,4 +88,4 @@ def describe_type(
typer.echo("") typer.echo("")
typer.echo("## Authoring guidance") typer.echo("## Authoring guidance")
typer.echo(body.strip()) typer.echo(described["body"])
+167 -19
View File
@@ -4,32 +4,164 @@ The repo is a pipeline: `raw/` (untrusted input) -> `types/` + `tools/` (schema
compiler) -> `kb/` (compiled knowledge) -> `reports/` (derived output). Only `kb/` is compiler) -> `kb/` (compiled knowledge) -> `reports/` (derived output). Only `kb/` is
divided into collections; the other three stages are single-purpose directories. divided into collections; the other three stages are single-purpose directories.
Repo root is resolved by walking up from this file's location (tools/chemenu/config.py Repo root is resolved in three steps - an explicit argument to `resolve_root()`, then
-> tools/ -> repo root), which is stable regardless of the caller's current working `$CHEMENU_ROOT`, then a walk up from this file's location (tools/chemenu/config.py ->
directory. tools/ -> repo root). The walk-up stays the default, so `tools/wikitool` behaves exactly
as it always has; the two steps in front of it are what lets an in-process caller point
this package at a corpus it does not itself live inside.
**Nothing below is bound at import time.** `ROOT` and every path derived from it are
resolved on each attribute access, through the module `__getattr__` at the bottom. They
used to be module constants, which had a failure mode worse than the limitation itself:
`monkeypatch.setattr(config, "ROOT", other)` repointed `ROOT` and left `KB_DIR` and
`RAW_DIR` aimed at wherever this file happens to sit, so a caller that believed it was
working on a target tree was in fact answering out of the developer's checkout. Resolving
on access makes the derived paths follow whatever `ROOT` currently is - including a
monkeypatched one - so the half-repointed state cannot be constructed.
""" """
import os import os
import subprocess import subprocess
from contextlib import contextmanager
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] # The last resort, and the default every existing caller gets: the checkout this
# file is part of.
_PACKAGE_ROOT = Path(__file__).resolve().parents[2]
RAW_DIR = ROOT / "raw" # Points this package at a corpus other than its own checkout. Registered in
KB_DIR = ROOT / "kb" # `_WIKITOOL_ENV` (tools/chemenu/tests/conftest.py), so the suite runs with it
TYPES_DIR = ROOT / "types" # cleared and a test that wants it sets it itself.
REPORTS_DIR = ROOT / "reports" ENV_ROOT = "CHEMENU_ROOT"
WORK_DIR = ROOT / "work"
INSTRUCTIONS_DIR = ROOT / "instructions"
# Generated copies of the skill directories under `instructions/`. Both are
# gitignored: they are build output, and a fresh clone publishes them with
# `wikitool instructions sync` (see instructions/bootstrap.md).
AGENTS_SKILLS_DIR = ROOT / ".agents" / "skills"
CLAUDE_SKILLS_DIR = ROOT / ".claude" / "skills"
INDEX_FILE = KB_DIR / "index.md" def resolve_root(explicit: "Path | str | None" = None) -> Path:
LOG_FILE = KB_DIR / "log.md" """The repo root, by the documented precedence: argument, then
PROVENANCE_FILE = KB_DIR / "provenance.md" `$CHEMENU_ROOT`, then the checkout this package lives in.
An explicit argument wins because a caller serving two corpora cannot use a
process-wide variable to tell them apart; the variable exists for the case
where the caller is a whole process (a server, a CI job) and there is
nothing to pass it through.
"""
if explicit is not None:
return Path(explicit).expanduser().resolve()
from_env = os.environ.get(ENV_ROOT, "").strip()
if from_env:
return Path(from_env).expanduser().resolve()
return _PACKAGE_ROOT
def _root() -> Path:
"""`ROOT` as it stands right now, honouring an assignment onto this module.
Reads the module dict directly rather than `resolve_root()` so that a test
(or any caller) setting `config.ROOT` is what the derived paths follow.
That assignment is why the derived paths are computed here at all.
"""
assigned = globals().get("ROOT")
return Path(assigned) if assigned is not None else resolve_root()
# Everything under the root, as a name -> relative-path table rather than as
# assignments. One place to read, and the only place that has to know a derived
# path exists at all.
_DERIVED = {
"RAW_DIR": ("raw",),
"KB_DIR": ("kb",),
"TYPES_DIR": ("types",),
"REPORTS_DIR": ("reports",),
"WORK_DIR": ("work",),
"INSTRUCTIONS_DIR": ("instructions",),
# Generated copies of the skill directories under `instructions/`. Both are
# gitignored: they are build output, and a fresh clone publishes them with
# `wikitool instructions sync` (see instructions/bootstrap.md).
"AGENTS_SKILLS_DIR": (".agents", "skills"),
"CLAUDE_SKILLS_DIR": (".claude", "skills"),
}
# The generated files, derived from `KB_DIR` rather than from the root: a
# caller that repoints only the corpus directory must not be left with a log
# and a catalog belonging to a different tree.
_KB_DERIVED = {
"INDEX_FILE": "index.md",
"LOG_FILE": "log.md",
"PROVENANCE_FILE": "provenance.md",
}
# Every name this module resolves rather than stores. Assigning one is
# supported - that is what makes the paths repointable at all - but the
# assignment has to be taken back afterwards, or it outlives the caller that
# made it. See `reset()`.
MANAGED_PATHS = ("ROOT", *_DERIVED, *_KB_DERIVED)
@contextmanager
def rooted(root: "Path | str"):
"""Resolve every managed path under `root` for the duration of the block.
Some things below the read core reach for `config` directly rather than
taking a root - the module-level `TypeResolver` singleton, which has to
find `types/`, is the one that matters - so pointing this package at
another corpus means pointing `config` at it, not only the functions that
accept an argument.
**Process-wide while it is open, and therefore not thread-safe.** A caller
serving several corpora at once holds a lock around it, the same discipline
`CorpusCache` documents. That is a real constraint and not a hidden one:
`$CHEMENU_ROOT` is process-wide for the same reason, and the server this
exists for (Gitea #19) serves one checkout that a `git reset --hard` keeps
clean.
Restores exactly what was there, including "nothing was assigned" - it must
not leave `ROOT` bound behind it, or it recreates the stale-binding bug in
the shape `reset()` describes.
"""
previous = {name: globals()[name] for name in MANAGED_PATHS if name in globals()}
reset()
globals()["ROOT"] = Path(root)
try:
yield Path(root)
finally:
reset()
globals().update(previous)
def reset() -> None:
"""Drop every assignment onto a managed path name, back to resolution.
The test suite calls this between tests, and it is not optional there.
`monkeypatch.setattr(config, "KB_DIR", tmp)` records the old value by
*reading* it - which resolves it - and its undo then writes that resolved
path back as a real attribute. The name is bound from then on, so the next
caller to repoint only `ROOT` gets a `KB_DIR` still aimed at the previous
tree: exactly the half-repointed state this module was rewritten to make
unconstructible, rebuilt by the cleanup rather than by the test.
"""
for name in MANAGED_PATHS:
globals().pop(name, None)
def __getattr__(name: str):
"""Resolve `ROOT` and the paths under it on access (PEP 562).
Only reached for names *not* in the module dict, so an explicit assignment
- `monkeypatch.setattr(config, "ROOT", tmp)` - keeps working and now also
carries the derived paths with it, which is the bug this replaces.
"""
if name == "ROOT":
return resolve_root()
if name in _DERIVED:
return _root().joinpath(*_DERIVED[name])
if name in _KB_DERIVED:
kb_dir = globals().get("KB_DIR")
base = Path(kb_dir) if kb_dir is not None else _root() / "kb"
return base / _KB_DERIVED[name]
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
def __dir__() -> list[str]:
return sorted([*globals(), "ROOT", *_DERIVED, *_KB_DERIVED])
# Files/patterns to ignore when scanning raw/ for ingest coverage. # Files/patterns to ignore when scanning raw/ for ingest coverage.
# CONTRACT.md is the layer's source contract, not source material. # CONTRACT.md is the layer's source contract, not source material.
@@ -70,6 +202,22 @@ ENVIRONMENT_TEMPLATE = f"{ENVIRONMENT_FILE}.template"
# `dist export` already computes (AGENTS.md invariant 8). See NOTICE. # `dist export` already computes (AGENTS.md invariant 8). See NOTICE.
LICENSE_FILES = ("LICENSE", "LICENSE-CONTENT", "NOTICE") LICENSE_FILES = ("LICENSE", "LICENSE-CONTENT", "NOTICE")
# Which push targets `publish` may write to, for a checkout that says so. The
# danger this addresses is one checkout's content reaching another checkout's
# remote - a private instance pushing its own `kb/` to a public upstream, where
# it cannot be taken back.
#
# It pins **URLs, not remote names**: a name-based list would pass a `publish`
# whose `origin` had been repointed, which is the failure it exists to catch.
#
# Per-checkout and gitignored, like `ENVIRONMENT.md` and for the same reason:
# two clones of this repo push to two different places, so a committed copy
# would hand the second one an answer that is wrong rather than missing. Absent
# means unrestricted - `doctor` reports it, and the Publish-Remote Gate simply
# does not apply. A checkout that holds private content should have one; see
# instructions/gates.md.
PUBLISH_REMOTES_FILENAME = ".wikitool-remotes.json"
def default_author() -> str | None: def default_author() -> str | None:
"""The author to stamp a new source page with, per instance. """The author to stamp a new source page with, per instance.
@@ -86,7 +234,7 @@ def default_author() -> str | None:
try: try:
result = subprocess.run( result = subprocess.run(
["git", "config", "user.name"], ["git", "config", "user.name"],
cwd=ROOT, cwd=_root(),
capture_output=True, capture_output=True,
text=True, text=True,
timeout=5, timeout=5,
+123
View File
@@ -0,0 +1,123 @@
"""Load the corpus once per revision instead of once per query.
`load_pages_by_path()` reads and parses every page under `kb/` on every call.
For a CLI that is the right shape - one call per process, nothing to reuse, and
a cache would only add a way to answer from a tree that has since changed. For
a long-lived reader (the MCP server) it is the opposite: the same corpus is
reparsed for every request, and the cost grows linearly with the corpus.
So the cache is an object a caller holds, not a module-level dict that switches
itself on behind everyone's back. The CLI holds none and behaves exactly as
before; a resident process holds one.
**The key is the commit SHA, and a dirty tree is never cached.** The SHA alone
would be a correctness bug in any checkout someone edits: a session that writes
a page and searches for it would be answered from the parse taken before the
write, with nothing about the SHA having changed. A clean tree is the state the
server actually runs in - it is kept that way by `git fetch && git reset
--hard` - so the fast path is the one that holds there, and every other tree
falls back to reloading.
That same SHA is what a response is stamped with, which is deliberate: the
revision a caller is told about is by construction the revision its answer was
computed from.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
from typing import Optional
from chemenu import config
from chemenu.page import Page
def head_commit(root: Optional[Path] = None) -> Optional[str]:
"""The full SHA of `HEAD`, or None outside a git checkout."""
result = _git(["rev-parse", "HEAD"], root)
if result is None or result.returncode != 0:
return None
return result.stdout.strip() or None
def is_dirty(root: Optional[Path] = None, path: Optional[Path] = None) -> bool:
"""Whether the working tree has uncommitted changes under `path`.
Errs toward dirty: if git cannot answer, the answer is "assume it changed".
A cache that treats "unknown" as clean serves stale pages, which is the one
outcome this module exists to prevent.
"""
root = root or config.ROOT
target = path or config.KB_DIR
try:
relative = Path(target).resolve().relative_to(Path(root).resolve()).as_posix()
except ValueError:
return True
result = _git(["status", "--porcelain", "--", relative], root)
if result is None or result.returncode != 0:
return True
return bool(result.stdout.strip())
def _git(args: list[str], root: Optional[Path] = None):
try:
return subprocess.run(
["git", *args],
cwd=root or config.ROOT,
capture_output=True,
text=True,
timeout=10,
check=False,
)
except (OSError, subprocess.SubprocessError):
return None
class CorpusCache:
"""One parsed corpus, reused while the checkout stays on the same commit.
Not thread-safe by itself: a caller serving concurrent requests holds the
lock. Kept out of here because the locking discipline belongs to whoever
owns the request loop, and a lock hidden in a cache is one nobody can see
when they need to reason about it.
"""
def __init__(self, kb_dir: Optional[Path] = None, root: Optional[Path] = None):
self.kb_dir = kb_dir
self.root = root
self._pages: Optional[dict[str, Page]] = None
self._revision: Optional[str] = None
@property
def revision(self) -> Optional[str]:
"""The commit the cached corpus was read at, or None if nothing is
cached (including because the tree was dirty)."""
return self._revision
def current_revision(self) -> Optional[str]:
"""The commit this corpus would be cached under right now: `HEAD` on a
clean tree, None on a dirty one or outside git. None means uncacheable,
which is why it is also what a caller should report as "no revision" -
an answer read out of a dirty tree does not correspond to any commit."""
root = self.root or config.ROOT
if is_dirty(root, self.kb_dir or config.KB_DIR):
return None
return head_commit(root)
def load(self) -> tuple[dict[str, Page], Optional[str]]:
"""(pages, revision). Reparses whenever the revision is not the cached
one, and on every call while the tree is dirty."""
# Imported here rather than at module level: `commands.search` pulls in
# typer, and this module is meant to be importable by a library caller
# that has no CLI. Removing that edge properly is the library-boundary
# work, not this file's job.
from chemenu.commands.search import load_pages_by_path
revision = self.current_revision()
if revision is None or revision != self._revision or self._pages is None:
pages = load_pages_by_path(self.kb_dir, self.root)
if revision is None:
self._pages, self._revision = None, None
return pages, None
self._pages, self._revision = pages, revision
return self._pages, self._revision
+31
View File
@@ -0,0 +1,31 @@
"""The exception contract at the library boundary.
The CLI reports a bad argument by printing an `ERROR` line and leaving through
`typer.Exit(1)`. That is the right answer for a terminal and the wrong one for
an in-process caller, which gets an exit code where it expected a value, plus
module-global state (`_util._declined`) surviving into its next call.
So the core raises, and the CLI adapter translates. `ChemenuError` is the one
class a library caller has to know; the two below it separate "your input was
wrong, a different argument would work" from "the machinery underneath failed",
which is the same distinction the CLI's exit codes draw.
`ValidationError` also inherits `ValueError`. Not for elegance: `PredicateError`
was a `ValueError` before this existed, and callers catch it that way.
"""
from __future__ import annotations
class ChemenuError(Exception):
"""Base for every error this package raises deliberately."""
class ValidationError(ChemenuError, ValueError):
"""The caller's input was rejected. Re-running unchanged fails identically;
the CLI renders this as its exit-1 `ERROR` line."""
class BackendError(ChemenuError, RuntimeError):
"""A dependency the core relies on was missing or failed - `rg` absent, a
search that had to be killed. Not the caller's argument, and not
necessarily permanent."""
+124 -29
View File
@@ -16,29 +16,137 @@ from typing import Any
import yaml import yaml
from chemenu.errors import ValidationError
try: # pragma: no cover - which branch runs depends on the host's libyaml
from yaml import CSafeLoader as _Loader
except ImportError: # pragma: no cover
from yaml import SafeLoader as _Loader
FRONTMATTER_RE = re.compile(r"\A---\n(.*?)\n---\n?(.*)\Z", re.DOTALL) FRONTMATTER_RE = re.compile(r"\A---\n(.*?)\n---\n?(.*)\Z", re.DOTALL)
# Frontmatter is a flat block of scalars and one-line lists. Real pages sit
# well under a kilobyte, so this is not a budget anyone writes against - it is
# the bound that keeps parse cost proportional to the corpus rather than to
# whatever a single file claims to be. It matters because the parser is on the
# read path, and the read path is what gets exposed.
MAX_FRONTMATTER_BYTES = 64 * 1024
# YAML anchors and aliases are refused outright rather than budgeted. A page's
# frontmatter has no use for them, and alias expansion is where a small
# document becomes an enormous object graph: 267 bytes of nested aliases
# compose in 0.2 ms into 672,603 nodes, growing 9**n with nesting depth at
# constant parse time. A size limit alone does not touch that, because the
# input stays small - see `ALIAS_BOMB` in the tests.
#
# The check runs on the *event* stream (`yaml.parse`), which is streaming and
# resolves nothing - so asking the question costs O(text) and never triggers
# the expansion it is asking about. `"*"` is a necessary character in any alias
# node, so its absence proves absence without parsing at all, which is the case
# every real page takes.
_ALIAS_HINT = "*"
class FrontmatterError(ValidationError):
"""Frontmatter that cannot be used: missing, malformed, oversized, or
refused by a limit. Raised only by the strict entry points - the permissive
ones report it as a string instead."""
def _load_frontmatter(fm_text: str) -> tuple[dict[str, Any] | None, str | None]:
"""Parse one frontmatter block into (mapping, error). Exactly one is None.
The single parser behind both `read_page()` and `frontmatter_error()`. They
used to have one each, which is how the permissive path could degrade to
`{}` for a reason the strict path described differently - and how every
caller wanting both answers read and parsed the file twice.
"""
encoded = len(fm_text.encode("utf-8"))
if encoded > MAX_FRONTMATTER_BYTES:
return None, (
f"frontmatter is {encoded} bytes, over the {MAX_FRONTMATTER_BYTES}-byte limit"
)
if _ALIAS_HINT in fm_text:
try:
for event in yaml.parse(fm_text, Loader=_Loader):
if isinstance(event, yaml.AliasEvent):
return None, (
"frontmatter uses a YAML alias (`*"
f"{event.anchor}`); anchors and aliases are not allowed here"
)
except yaml.YAMLError as exc:
return None, f"invalid YAML frontmatter: {_first_line(exc)}"
except RecursionError:
return None, "frontmatter is nested too deeply to parse"
try:
parsed = yaml.load(fm_text, Loader=_Loader)
except yaml.YAMLError as exc:
return None, f"invalid YAML frontmatter: {_first_line(exc)}"
except RecursionError:
# PyYAML composes recursively, so deep nesting exhausts the interpreter
# stack rather than raising a YAMLError. Unbounded nesting is bounded by
# MAX_FRONTMATTER_BYTES; this catches what fits under it.
return None, "frontmatter is nested too deeply to parse"
if parsed is None:
return None, "empty frontmatter block"
if not isinstance(parsed, dict):
return None, f"frontmatter is {type(parsed).__name__}, expected a mapping"
return parsed, None
def _first_line(exc: Exception) -> str:
text = str(exc)
return text.splitlines()[0] if text else exc.__class__.__name__
def read_page_with_error(path: Path) -> tuple[dict[str, Any], str, str | None]:
"""(frontmatter, body, error) - the permissive read, with the reason it was
permissive handed back instead of dropped.
A page whose YAML is broken reads as `{}`, and a `{}` page then has no
`confidence` and no `kind`: it drops out of `--field confidence<0.6` -
precisely the query meant to find pages in bad shape - while looking to the
caller like a page that simply did not match. Returning the reason is what
lets a caller say so instead of losing the page quietly.
"""
text = path.read_text(encoding="utf-8")
match = FRONTMATTER_RE.match(text)
if not match:
return {}, text, "no `---` frontmatter block"
fm_text, body = match.group(1), match.group(2)
parsed, error = _load_frontmatter(fm_text)
if error == "empty frontmatter block":
# An empty block is a legitimate shape for the permissive read - it
# carries no fields, and there is nothing to lose. Only `lint` treats
# it as a finding.
return {}, body, error
return (parsed or {}), body, error
def read_page(path: Path) -> tuple[dict[str, Any], str]: def read_page(path: Path) -> tuple[dict[str, Any], str]:
"""Return (frontmatter_dict, body) for a markdown file. If the file has no """Return (frontmatter_dict, body) for a markdown file. If the file has no
frontmatter block, returns ({}, full_text). frontmatter block, returns ({}, full_text).
Deliberately permissive: malformed YAML degrades to an empty dict so bulk Deliberately permissive: malformed YAML degrades to an empty dict so bulk
operations never crash on one bad page. Use `frontmatter_error()` (which operations never crash on one bad page. Use `read_page_with_error()` (search
`wikitool lint` does) to surface those pages instead of losing them does) or `frontmatter_error()` (`wikitool lint` does) to surface those pages
silently. instead of losing them silently.
""" """
text = path.read_text(encoding="utf-8") frontmatter, body, _ = read_page_with_error(path)
match = FRONTMATTER_RE.match(text) return frontmatter, body
if not match:
return {}, text
fm_text, body = match.group(1), match.group(2) def read_page_strict(path: Path) -> tuple[dict[str, Any], str]:
try: """`read_page()` that raises `FrontmatterError` instead of degrading.
frontmatter = yaml.safe_load(fm_text) or {}
except yaml.YAMLError: For any path that ingests frontmatter this instance did not write itself.
frontmatter = {} The permissive read is right for bulk operations over a corpus the operator
if not isinstance(frontmatter, dict): committed; it is wrong the moment the document arrives from outside, where
frontmatter = {} "unparseable" must stop the document rather than empty it.
"""
frontmatter, body, error = read_page_with_error(path)
if error is not None:
raise FrontmatterError(f"{path}: {error}")
return frontmatter, body return frontmatter, body
@@ -50,20 +158,7 @@ def frontmatter_error(path: Path) -> str | None:
YAML is malformed (or whose frontmatter block is missing entirely) reads YAML is malformed (or whose frontmatter block is missing entirely) reads
back as `{}` and then quietly slips past every frontmatter-driven check. back as `{}` and then quietly slips past every frontmatter-driven check.
""" """
text = path.read_text(encoding="utf-8") return read_page_with_error(path)[2]
match = FRONTMATTER_RE.match(text)
if not match:
return "no `---` frontmatter block"
try:
parsed = yaml.safe_load(match.group(1))
except yaml.YAMLError as exc:
reason = str(exc).splitlines()[0] if str(exc) else exc.__class__.__name__
return f"invalid YAML frontmatter: {reason}"
if parsed is None:
return "empty frontmatter block"
if not isinstance(parsed, dict):
return f"frontmatter is {type(parsed).__name__}, expected a mapping"
return None
def _format_scalar(value: Any, flow: bool = False) -> str: def _format_scalar(value: Any, flow: bool = False) -> str:
@@ -132,7 +227,7 @@ def _round_trips_as_string(text: str, flow: bool = False) -> bool:
""" """
probe, expected = (f"[{text}]", [text]) if flow else (text, text) probe, expected = (f"[{text}]", [text]) if flow else (text, text)
try: try:
return yaml.safe_load(probe) == expected return yaml.load(probe, Loader=_Loader) == expected
except yaml.YAMLError: except yaml.YAMLError:
# Unparseable bare - quoting is exactly the fix. # Unparseable bare - quoting is exactly the fix.
return False return False
+431
View File
@@ -0,0 +1,431 @@
"""The lint core, with no CLI attached.
Split out of `commands/lint.py` for the reason given in
`chemenu/search/service.py`: `run_lint()` is a pure function over a corpus
directory, and it was sitting in a module that imports `typer` and `rich`, so
no in-process caller could reach it without the CLI head.
Everything that decides *findings* lives here. Everything that decides *how a
terminal sees them* - the report file, the exit code, the flags - stays in
`commands/lint.py`. `render_markdown()` and `render_summary()` are on this side
of the line because the markdown report is a data product (it is what
`reports/` holds and what `kb/log.md` refers to), not terminal formatting.
"""
from __future__ import annotations
from datetime import date
from pathlib import Path
from chemenu import config
from chemenu.frontmatter_io import frontmatter_error
from chemenu.markdown_code import strip_code_spans
from chemenu.provenance import broken_raw_refs as find_broken_raw_refs
from chemenu.provenance import duplicate_raw_file_owners as find_duplicate_raw_file_owners
from chemenu.provenance import extract_inline_cites
from chemenu.provenance import legacy_citation_markers as find_legacy_citation_markers
from chemenu.provenance import legacy_source_pages as find_legacy_source_pages
from chemenu.provenance import orphan_footnote_defs as find_orphan_footnote_defs
from chemenu.provenance import uncovered_raw_files as find_uncovered_raw_files
from chemenu.provenance import undefined_footnote_refs as find_undefined_footnote_refs
from chemenu.kb_scan import (
GENERATED_INDEX,
WIKILINK_RE,
build_link_graph,
find_duplicate_title_paths,
inbound_links,
load_kb_pages,
)
from chemenu.type_resolver import resolver
# Style guide's one mechanically-checkable rule (hard oracle: a plain count).
# The rest of the style guide (tone, AI-phrase avoidance) is a soft/proxy judgment
# and stays with the LLM - see wiki-manage/wiki-ingest skill guidance, not lint.
#
# The unit is a quote, not a `>` line. It used to be the line, which measured
# the wrap width the rule has no opinion about: one quotation written long
# counted 1 and the same quotation wrapped at 100 columns counted 4. An author
# who took the finding seriously made the page harder to read to quiet it.
QUOTE_LIMIT = 2
# How many hub pages `most_linked` reports. Purely informational (wiki-status
# surfaces it); not a finding, so the cutoff only bounds report size.
MOST_LINKED_COUNT = 10
def count_quote_blocks(body: str) -> int:
"""How many distinct blockquotes `body` carries.
A run of consecutive `>` lines is one quote; a blank line or any
non-quoted line ends it. Code is masked out first, so a `>` inside a
fenced shell transcript is a prompt, not a quotation.
Lazy continuation - a quote whose wrapped lines drop the `>` - reads here
as two quotes rather than one. That over-counts in the direction the limit
already errs on, and the corpus prefixes every line, so the alternative
(tracking paragraph state) buys nothing.
"""
count, in_quote = 0, False
for line in strip_code_spans(body).splitlines():
is_quote = line.lstrip().startswith(">")
if is_quote and not in_quote:
count += 1
in_quote = is_quote
return count
def run_lint(kb_dir: Path) -> dict:
pages = load_kb_pages(kb_dir)
duplicate_titles = find_duplicate_title_paths(kb_dir, config.ROOT)
# Pages whose frontmatter can't be parsed read back as `{}` everywhere
# else, which would let them slip past every frontmatter-driven check
# below with no finding at all - so they are detected explicitly.
frontmatter_errors = []
for title, page in sorted(pages.items()):
reason = frontmatter_error(page.path)
if reason is None and not page.frontmatter.get("type"):
reason = "missing `type:` field"
if reason is not None:
frontmatter_errors.append({"page": title, "error": reason})
graph = build_link_graph(pages)
broken_links = [
{"page": title, "target": target}
for title, targets in graph.items()
for target in sorted(targets)
if target not in pages
]
inbound = inbound_links({t: v for t, v in graph.items() if t != "index"})
orphan_pages = sorted(
title
for title, sources in inbound.items()
if not sources
and title not in ("index", "log")
# comparison pages are not linked to by design; index.md is sufficient coverage
and pages[title].kind != "comparison"
)
# Same link graph, opposite end: the most-linked-to pages are the wiki's
# hubs. Reported (not judged) so `wiki-status` can show them without
# re-deriving the graph.
inbound_counts = {title: len(sources) for title, sources in inbound.items()}
most_linked = [
{"page": title, "inbound": count}
for title, count in sorted(inbound_counts.items(), key=lambda kv: (-kv[1], kv[0]))
if count > 0
][:MOST_LINKED_COUNT]
# The catalog is sharded: `kb/index.md` is a map carrying counts and links,
# and the page rows live in a generated INDEX.md per collection/area. Both
# halves have to be read, or every page reads as missing from the index.
index_text = "".join(
path.read_text(encoding="utf-8")
for path in [kb_dir / "index.md", *sorted(kb_dir.rglob(GENERATED_INDEX))]
if path.exists()
)
index_links = {m.group(1).strip() for m in WIKILINK_RE.finditer(index_text)}
missing_from_index = sorted(set(pages) - index_links - {"index", "log"})
dangling_index_entries = sorted(index_links - set(pages))
title_mismatches = []
for title, page in sorted(pages.items()):
if page.kind not in ("entity", "concept"):
continue
h1 = page.h1_title
if h1 is not None and h1 != title:
title_mismatches.append({"page": title, "h1": h1})
unmarked_provenance = []
for title, page in sorted(pages.items()):
if page.kind not in ("entity", "concept"):
continue
sources_list = page.frontmatter.get("sources") or []
if not sources_list and page.frontmatter.get("provenance") != "general":
unmarked_provenance.append(title)
citation_frontmatter_drift = []
for title, page in sorted(pages.items()):
sources_list = set(page.frontmatter.get("sources") or [])
cited = {cited_title for cited_title, _file in extract_inline_cites(page.body)}
cited.discard(title) # a source page citing itself for a specific file within it is not drift
for missing_source in sorted(cited - sources_list):
citation_frontmatter_drift.append({"page": title, "cited_but_not_in_sources": missing_source})
legacy_citation_markers = find_legacy_citation_markers(pages)
undefined_footnote_refs = find_undefined_footnote_refs(pages)
orphan_footnote_defs = find_orphan_footnote_defs(pages)
# The frontmatter half of the link graph. `broken_links` above only walks
# `[[wikilinks]]` in page *bodies*, so a `related:`/`sources:`/`entities:`
# entry naming a page that does not exist - a rename that was not
# propagated, a deleted page, or a URL pasted where a title belongs - used
# to pass every check. Which fields hold page titles is declared by each
# type-spec's `page_ref_fields:`, not hardcoded here.
dangling_frontmatter_refs = []
for title, page in sorted(pages.items()):
type_path = page.frontmatter.get("type")
if not type_path:
continue
try:
ref_fields = resolver.get_page_ref_fields(type_path, page.path)
except ValueError:
continue # unresolvable type is already reported as type_resolution_errors
for field in ref_fields:
for target in page.frontmatter.get(field) or []:
if target not in pages:
dangling_frontmatter_refs.append(
{"page": title, "field": field, "target": target}
)
quote_limit_violations = []
for title, page in sorted(pages.items()):
quote_count = count_quote_blocks(page.body)
if quote_count > QUOTE_LIMIT:
quote_limit_violations.append({"page": title, "quote_count": quote_count})
# Type system validation. Lint reports are not validated here: they are
# written to `reports/` outside kb/ and are never pages, so nothing this
# loop scans can be one.
invalid_type_paths = []
type_resolution_errors = []
schema_validation_errors = []
for title, page in sorted(pages.items()):
type_path = page.frontmatter.get("type")
if not type_path:
continue
# Check if type path is valid
if not type_path.endswith('.md'):
invalid_type_paths.append({"page": title, "type": type_path, "error": "Type path must end with .md"})
continue
# Try to resolve and validate the type
try:
resolver.load_type_spec(type_path, page.path)
# Try schema validation
try:
resolver.validate_frontmatter(page.frontmatter, type_path, page.path)
except ValueError as schema_error:
schema_validation_errors.append({"page": title, "type": type_path, "error": str(schema_error)})
except ValueError as resolution_error:
type_resolution_errors.append({"page": title, "type": type_path, "error": str(resolution_error)})
return {
"generated": date.today().isoformat(),
"page_count": len(pages),
"frontmatter_errors": frontmatter_errors,
"broken_links": broken_links,
"orphan_pages": orphan_pages,
"most_linked": most_linked,
"inbound_counts": inbound_counts,
"missing_from_index": missing_from_index,
"dangling_index_entries": dangling_index_entries,
"title_mismatches": title_mismatches,
"duplicate_titles": duplicate_titles,
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
"broken_raw_refs": find_broken_raw_refs(pages),
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
"legacy_source_pages": find_legacy_source_pages(pages),
"unmarked_provenance": unmarked_provenance,
"citation_frontmatter_drift": citation_frontmatter_drift,
"legacy_citation_markers": legacy_citation_markers,
"undefined_footnote_refs": undefined_footnote_refs,
"orphan_footnote_defs": orphan_footnote_defs,
"dangling_frontmatter_refs": dangling_frontmatter_refs,
"quote_limit_violations": quote_limit_violations,
"invalid_type_paths": invalid_type_paths,
"type_resolution_errors": type_resolution_errors,
"schema_validation_errors": schema_validation_errors,
}
def _section(lines: list[str], title: str, items: list, formatter) -> None:
lines.append(f"## {title}")
lines.append("")
if not items:
lines.append("None found.")
else:
for item in items:
lines.append(f"- {formatter(item)}")
lines.append("")
def render_markdown(report: dict) -> str:
lines = [f"# Structural Lint Report ({report['generated']})", ""]
lines.append(f"Scanned {report['page_count']} pages under `wiki/`. This report covers only")
lines.append("mechanically-verifiable structural issues; see the Semantic Review section")
lines.append("below for judgment calls the LLM should complete.")
lines.append("")
_section(
lines, "Unreadable Frontmatter", report["frontmatter_errors"],
lambda i: f"[[{i['page']}]] - {i['error']}",
)
_section(
lines, "Broken Wikilinks", report["broken_links"],
lambda i: f"[[{i['page']}]] links to missing [[{i['target']}]]",
)
_section(lines, "Orphan Pages (no inbound links)", report["orphan_pages"], lambda i: f"[[{i}]]")
_section(
lines, f"Most-Linked Pages (top {MOST_LINKED_COUNT} hubs)", report["most_linked"],
lambda i: f"[[{i['page']}]] - {i['inbound']} inbound link(s)",
)
_section(lines, "Pages Missing from index.md", report["missing_from_index"], lambda i: f"[[{i}]]")
_section(lines, "Dangling index.md Entries", report["dangling_index_entries"], lambda i: f"[[{i}]]")
_section(
lines, "Duplicate Titles (naming collisions)", report["duplicate_titles"],
lambda i: f"`{i['stem']}` -> {', '.join(f'`{p}`' for p in i['paths'])}",
)
_section(
lines, "Filename / H1 Title Mismatches", report["title_mismatches"],
lambda i: f"[[{i['page']}]] H1 is '{i['h1']}'",
)
_section(
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
lambda i: f"`{i}`",
)
_section(
lines, "Broken raw_files References", report["broken_raw_refs"],
lambda i: f"[[{i['page']}]] -> `{i['raw_path']}` (does not exist)",
)
_section(
lines, "Raw Files With More Than One Owner", report["duplicate_raw_file_owners"],
lambda i: f"`{i['raw_file']}` is claimed by " + ", ".join(f"[[{t}]]" for t in i["owners"]),
)
_section(
lines, "Legacy source: Field (not yet migrated to raw_files:)", report["legacy_source_pages"],
lambda i: f"[[{i['page']}]] source: `{i['source']}` ({i['reason']})",
)
_section(
lines, "Pages Missing provenance: general Marker", report["unmarked_provenance"],
lambda i: f"[[{i}]] has no sources and is not marked `provenance: general`",
)
_section(
lines, "Citation / Frontmatter Drift", report["citation_frontmatter_drift"],
lambda i: f"[[{i['page']}]] cites [[{i['cited_but_not_in_sources']}]] inline but it is missing from frontmatter `sources:`",
)
_section(
lines, "Legacy Citation Markers (pre-migration `^[[...]]`)", report["legacy_citation_markers"],
lambda i: f"[[{i['page']}]] still has `{i['marker']}` - run `wikitool cite add` and replace it with the `[^cite-id]` it prints",
)
_section(
lines, "Undefined Footnote References", report["undefined_footnote_refs"],
lambda i: f"[[{i['page']}]] references `[^{i['ref']}]`, which has no `[^{i['ref']}]: [[...]]` definition",
)
_section(
lines, "Orphan Footnote Definitions", report["orphan_footnote_defs"],
lambda i: f"[[{i['page']}]] defines `[^{i['id']}]` (-> [[{i['source']}]]) but nothing references it - run `wikitool cite sync`",
)
_section(
lines, "Dangling Frontmatter References", report["dangling_frontmatter_refs"],
lambda i: f"[[{i['page']}]] `{i['field']}:` names `{i['target']}`, which is not a page",
)
_section(
lines, "Invalid Type Paths", report["invalid_type_paths"],
lambda i: f"[[{i['page']}]] has type: `{i['type']}` - {i['error']}",
)
_section(
lines, "Type Resolution Errors", report["type_resolution_errors"],
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
)
_section(
lines, "Schema Validation Errors", report["schema_validation_errors"],
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
)
_section(
lines, f"Pages Exceeding Quote Limit (>{QUOTE_LIMIT}/page)", report["quote_limit_violations"],
lambda i: f"[[{i['page']}]] has {i['quote_count']} quotes - trim or confirm they're load-bearing",
)
lines.append("## Semantic Review (LLM to complete)")
lines.append("")
lines.append("- Contradictions across pages: TODO")
lines.append("- Stale claims (unconfirmed >6 months): TODO")
lines.append("- Suggested new pages / missing cross-references: TODO")
lines.append("")
return "\n".join(lines)
# Sections that always carry content but are not findings, so the summary
# handles them separately: a hub list is a statistic, and the semantic review
# is the checklist that follows the report rather than part of it.
INFORMATIONAL_SECTIONS = ("Most-Linked Pages",)
SEMANTIC_REVIEW_SECTION = "Semantic Review"
def _split_sections(markdown: str) -> tuple[str, list[tuple[str, str]]]:
"""Cut a rendered report into its preamble and (title, body) sections."""
preamble, *rest = markdown.split("\n## ")
sections = []
for part in rest:
title, _, body = part.partition("\n")
sections.append((title.strip(), body.strip()))
return preamble.rstrip(), sections
def render_summary(report: dict) -> str:
"""The same report with the empty sections removed.
On a healthy corpus the full report is better than 90% "None found.", so
reading it in the terminal means paging past the answer. The file on disk
stays complete - this is what gets printed, and the written path underneath
it is how the rest is reached without running lint a second time.
"""
preamble, sections = _split_sections(render_markdown(report))
findings, trailing = [], []
for title, body in sections:
if title.startswith(SEMANTIC_REVIEW_SECTION):
trailing.append((title, body))
elif body != "None found." and not title.startswith(INFORMATIONAL_SECTIONS):
findings.append((title, body))
lines = [preamble, ""]
if not findings:
lines += ["No structural findings.", ""]
for title, body in findings + trailing:
lines += [f"## {title}", "", body, ""]
return "\n".join(lines)
def default_report_path(report: dict) -> Path:
"""Where a report goes when the caller names no path.
`reports/` is derived and gitignored ([reports/CONTRACT.md]), so writing
here by default costs the tree nothing.
"""
return config.ROOT / "reports" / f"Lint Report {report['generated']}.md"
# Findings that make a tree structurally wrong rather than merely untidy.
# `orphan_pages` is deliberately absent: many pages are validly reachable
# through the index or navigation only. `quote_limit_violations` is advisory
# too - it flags a habit, not a broken tree.
#
# One definition, used by `lint --fail-on-error` and by the eval scorecard: if
# the two disagreed, a run could pass its score while lint refused it.
HARD_ERROR_KEYS = (
"frontmatter_errors",
"broken_links",
"dangling_index_entries",
"duplicate_titles",
"broken_raw_refs",
"duplicate_raw_file_owners",
"legacy_source_pages",
"citation_frontmatter_drift",
"legacy_citation_markers",
"undefined_footnote_refs",
"orphan_footnote_defs",
"dangling_frontmatter_refs",
"invalid_type_paths",
"type_resolution_errors",
"schema_validation_errors",
)
def has_hard_errors(report: dict) -> bool:
return any(report.get(key) for key in HARD_ERROR_KEYS)
+8
View File
@@ -0,0 +1,8 @@
"""The MCP read server: Chemenu's second consumer.
The CLI and this server are two adapters over one core (`chemenu.api`), not a
CLI with a network interface bolted on. See `server.py`.
"""
from chemenu.mcp.server import build_server
__all__ = ["build_server"]
+79
View File
@@ -0,0 +1,79 @@
"""`python -m chemenu.mcp` - start the read server.
Deliberately argparse and not typer. This process is the one place that must
not pull the CLI head in: the whole point of the library boundary is that a
second consumer costs `yaml`, `jsonschema` and the MCP SDK, and nothing else.
python -m chemenu.mcp # stdio
python -m chemenu.mcp --transport streamable-http # behind the proxy
CHEMENU_ROOT=/srv/wiki python -m chemenu.mcp --transport streamable-http \
--host 0.0.0.0 --port 8000
`--host`/`--port` apply to `streamable-http` only. They are here because the
default binds loopback, and a server in a container with a reverse proxy in
front of it has to bind an interface the proxy can reach - that is a property
of the software, not of one installation. *Which* host and port a given
deployment picks is infrastructure and stays out of this repository.
"""
from __future__ import annotations
import argparse
import sys
from chemenu.mcp.server import TRANSPORTS, TraceWouldWriteIntoCorpus, serve
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m chemenu.mcp",
description="Serve a Chemenu wiki read-only over MCP.",
)
parser.add_argument(
"--transport",
choices=TRANSPORTS,
default="stdio",
help="stdio for local use and testing; streamable-http for a deployed "
"instance behind the Traefik middleware (default: stdio)",
)
parser.add_argument(
"--root",
default=None,
help="The corpus to serve. Defaults to $CHEMENU_ROOT, then the checkout "
"this package lives in.",
)
parser.add_argument(
"--host",
default="127.0.0.1",
help="Interface to bind, streamable-http only. The loopback default is "
"deliberate; a container behind a reverse proxy needs 0.0.0.0.",
)
parser.add_argument(
"--port",
type=int,
default=8000,
help="Port to bind, streamable-http only (default: 8000)",
)
args = parser.parse_args(argv)
# Passed only for the transport they mean anything to: `run_stdio_async`
# takes no host or port, and handing it one is a TypeError rather than a
# harmless no-op.
bind = (
{"host": args.host, "port": args.port}
if args.transport == "streamable-http"
else {}
)
try:
serve(transport=args.transport, root=args.root, **bind)
except TraceWouldWriteIntoCorpus as exc:
# Refused before binding anything: the message names both fixes, and a
# server that silently relocated the operator's telemetry instead would
# be a surprise buried in a log.
print(f"ERROR {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__": # pragma: no cover - process entry point
raise SystemExit(main())
+234
View File
@@ -0,0 +1,234 @@
"""The MCP read server: `search`, `types`, `lint` and `status` over `kb/`.
Chemenu's second consumer. The CLI and this are two adapters over one core -
`chemenu.api.Corpus` - so a question answered here and the same question asked
at a terminal go through the same code, and a golden test holds the two
outputs against each other rather than trusting that they agree.
**There is no write path, structurally.** Nothing under `chemenu.commands` is
imported here or in `chemenu.api`, so `new`, `touch`, `xref`, `cite`,
`publish`, `migrate` and `version bump` are not reachable - the functions do
not exist in this process's reach, rather than being filtered out of a list. A
test asserts it by importing this module in a clean interpreter and looking at
`sys.modules`.
**Authentication and rate limiting are not here.** Both are Traefik middleware
in front of the process, per the operator's decision of 2026-09-01: a request
that is not cleanly authenticated does not reach Python at all. What *is* here
is the resource protection that middleware cannot give - the search timeout and
the frontmatter limits - because those exist against an authenticated consumer
damaging itself, which is a different problem from an unauthenticated one.
**The Iteration Budget Gate is deliberately absent.** It exists to stop an
agent *session* from iterating unnoticed over the state of the wiki, which is
why retrieval is exempt from it in the first place. A user who searches too
often is a resource problem - different instrument, different purpose - and
using the gate as a rate limiter would dilute it into one.
**Every response carries the commit it was computed from.** `chemenu.api`
stamps `commit` and `as_of`; a stale checkout otherwise answers confidently and
wrongly. Keeping the checkout current is a `git fetch && git reset --hard`
poll outside this process - see `instructions/mcp-read-server.md` - which needs
no inbound endpoint and no signature checking.
"""
from __future__ import annotations
import os
from pathlib import Path
from typing import Any, Optional
from mcp.server.mcpserver import MCPServer
from mcp.server.mcpserver.exceptions import ToolError
from chemenu import config
from chemenu.api import Corpus
from chemenu.errors import ChemenuError
SERVER_NAME = "chemenu"
# Transports this server will start on. `stdio` is for developing and testing
# it without a network; `streamable-http` is what a deployed instance speaks,
# and the only one the Traefik middleware can sit in front of, because Traefik
# is an HTTP reverse proxy. `sse` is reachable through the SDK but not offered:
# it is the superseded remote transport, and building on it now only moves the
# migration later.
TRANSPORTS = ("stdio", "streamable-http")
class TraceWouldWriteIntoCorpus(RuntimeError):
"""Raised at startup when telemetry would land inside the served tree."""
def check_trace_destination(root: Path) -> None:
"""Refuse to start if a trace would be written into the corpus.
Telemetry defaults to *on* and writes under `reports/telemetry/` in the
repo. Today nothing on this path emits - the writer is wired into
`cli.main()` and the two gates, none of which run here - so this is a guard
against the future rather than a fix for the present. It is worth having
anyway: the sync that keeps this checkout current is `git reset --hard`, so
a trace written into the tree is both a per-request write into a directory
something else is entitled to wipe, and a silent way for the server to
dirty the tree its own cache keys on.
Turn tracing off (`WIKI_TRACE=0`) or point it somewhere else
(`WIKI_TRACE_DIR`). Refusing rather than correcting it: a server that
quietly relocates the operator's telemetry is a surprise waiting in a log
nobody reads.
"""
if os.environ.get("WIKI_TRACE", "1") == "0":
return
destination = os.environ.get("WIKI_TRACE_DIR")
if destination is None:
raise TraceWouldWriteIntoCorpus(
"Telemetry is on and would write into the served checkout "
f"({root / 'reports' / 'telemetry'}). The sync that keeps this checkout "
"current is `git reset --hard`, which is entitled to wipe that directory. "
"Set WIKI_TRACE=0, or point WIKI_TRACE_DIR outside the corpus."
)
resolved = Path(destination).expanduser().resolve()
try:
resolved.relative_to(Path(root).resolve())
except ValueError:
return
raise TraceWouldWriteIntoCorpus(
f"WIKI_TRACE_DIR ({resolved}) is inside the served checkout ({root}). "
"Point it outside, or set WIKI_TRACE=0."
)
def build_server(
root: Optional[Path | str] = None, check_trace: bool = True
) -> MCPServer:
"""Assemble the server over one corpus.
`root` follows `config.resolve_root()` - argument, then `$CHEMENU_ROOT`,
then the checkout the package lives in - so a deployment points at its
corpus with one environment variable and no code.
"""
corpus = Corpus(root)
if check_trace:
check_trace_destination(corpus.root)
server = MCPServer(
name=SERVER_NAME,
instructions=(
"Read access to a Chemenu wiki: compiled, sourced knowledge under kb/. "
"Every answer carries the commit it was computed from ('commit') and "
"when it was produced ('as_of'); a null commit means the served tree "
"has uncommitted changes and the answer corresponds to no revision. "
"This server is read-only - there is no tool that writes."
),
)
@server.tool(
name="search",
description=(
"Find pages in kb/ by text, by frontmatter, or by both. Returns "
"title, path, kind, summary and confidence per hit, so a result can "
"be judged without fetching the page. Prefer this over listing "
"files: the answer is a few hundred tokens instead of a whole index."
),
)
def search(
query: str | None = None,
predicates: list[str] | None = None,
regex: bool = False,
limit: int = 20,
sort: str | None = None,
) -> dict[str, Any]:
"""Search the wiki.
`predicates` are frontmatter filters in the CLI's own `--field` syntax,
ANDed: `confidence<0.6`, `entity_type=system`, `tags~k8s`, `source_url:*`
(present), `!source_url` (absent). With no `query` this is a pure
structured query over frontmatter.
`regex` applies the pattern with ripgrep's linear engine. It is off by
default, so an accidental `.*` is a literal.
"""
return _guard(
lambda: corpus.search(
text=query,
predicates=predicates or (),
regex=regex,
limit=limit,
sort=sort,
)
)
@server.tool(
name="types",
description=(
"List the page types this wiki declares - what kinds of page exist, "
"where each lives, and what its schema is. Read this before "
"interpreting a page's `kind`."
),
)
def types() -> dict[str, Any]:
return _guard(corpus.types)
@server.tool(
name="describe_type",
description=(
"One page type's full contract: its frontmatter fields with "
"required/optional and any enums, its subtype field, and its "
"authoring guidance."
),
)
def describe_type(name: str) -> dict[str, Any]:
"""`name` is a short type name as listed by `types`, e.g. 'entity'."""
return _guard(lambda: corpus.describe_type(name))
@server.tool(
name="lint",
description=(
"The wiki's structural health: broken wikilinks, orphan pages, "
"index drift, schema gaps, provenance gaps. Findings only - the "
"JSON form writes no report file."
),
)
def lint() -> dict[str, Any]:
return _guard(corpus.lint)
@server.tool(
name="status",
description=(
"A snapshot: how many pages the wiki holds, how they split across "
"collections, and how many findings of each kind lint reports. "
"Cheaper to read than the full lint output."
),
)
def status() -> dict[str, Any]:
return _guard(corpus.status)
return server
def _guard(call):
"""Turn a `ChemenuError` into a plain message for the protocol layer.
A bad predicate is the caller's argument, not a server fault, and it should
arrive as a tool error the model can act on, carrying the message that says
what to do differently. The SDK draws exactly this line: a `ToolError` is a
deliberate refusal and its text reaches the caller, while anything else is a
crash whose text stays on the server. Only `ChemenuError` is caught -
everything else is a genuine fault and belongs in the log, unswallowed.
"""
try:
return call()
except ChemenuError as exc:
raise ToolError(str(exc)) from exc
def serve(
transport: str = "stdio",
root: Optional[Path | str] = None,
**kwargs: Any,
) -> None:
"""Start the server. `transport` is one of `TRANSPORTS`."""
if transport not in TRANSPORTS:
raise ValueError(
f"unknown transport {transport!r}. Available: {', '.join(TRANSPORTS)}"
)
build_server(root).run(transport=transport, **kwargs)
+8
View File
@@ -17,6 +17,14 @@ class Page:
frontmatter: dict[str, Any] = field(default_factory=dict) frontmatter: dict[str, Any] = field(default_factory=dict)
body: str = "" body: str = ""
# Why the frontmatter above is empty, when it is empty for a reason. A page
# whose YAML does not parse reads back as `{}`, and a `{}` page has no
# `confidence` and no `kind`: it then drops out of `--field confidence<0.6`
# - the query whose whole purpose is to find pages in bad shape - looking
# exactly like a page that did not match. Loaders that know the reason put
# it here so a caller can report the page instead of losing it.
frontmatter_error: Optional[str] = None
@property @property
def title(self) -> str: def title(self) -> str:
"""The page's canonical title: the filename without extension. """The page's canonical title: the filename without extension.
+2 -1
View File
@@ -14,6 +14,7 @@ from pathlib import Path
from typing import Any, Optional from typing import Any, Optional
from chemenu import config from chemenu import config
from chemenu.errors import ValidationError
from chemenu.page import Page from chemenu.page import Page
from chemenu.search.types import Predicate from chemenu.search.types import Predicate
@@ -26,7 +27,7 @@ _COMPARISON_OPS = (">=", "<=", ">", "<", "~", "=")
VIRTUAL_FIELDS = ("title", "kind", "subtype", "collection") VIRTUAL_FIELDS = ("title", "kind", "subtype", "collection")
class PredicateError(ValueError): class PredicateError(ValidationError):
"""Raised for a malformed `--field` argument or an unknown field name.""" """Raised for a malformed `--field` argument or an unknown field name."""
+16 -5
View File
@@ -7,28 +7,39 @@ the output shape. Selecting several at once fuses them through RRF.
from __future__ import annotations from __future__ import annotations
import os import os
from typing import Callable from pathlib import Path
from typing import Callable, Optional
from chemenu.errors import ValidationError
from chemenu.search.base import SearchBackend from chemenu.search.base import SearchBackend
from chemenu.search.ripgrep import RipgrepBackend from chemenu.search.ripgrep import RipgrepBackend
DEFAULT_BACKEND = "rg" DEFAULT_BACKEND = "rg"
ENV_VAR = "WIKITOOL_SEARCH_BACKEND" ENV_VAR = "WIKITOOL_SEARCH_BACKEND"
BACKENDS: dict[str, Callable[[], SearchBackend]] = { BACKENDS: dict[str, Callable[..., SearchBackend]] = {
"rg": RipgrepBackend, "rg": RipgrepBackend,
} }
class UnknownBackend(ValueError): class UnknownBackend(ValidationError):
pass pass
def resolve(spec: str | None = None) -> list[SearchBackend]: def resolve(
spec: str | None = None,
kb_dir: Optional[Path] = None,
root: Optional[Path] = None,
) -> list[SearchBackend]:
"""Resolve a backend spec into instances. """Resolve a backend spec into instances.
Precedence: explicit argument, then `WIKITOOL_SEARCH_BACKEND`, then the Precedence: explicit argument, then `WIKITOOL_SEARCH_BACKEND`, then the
default. A comma-separated spec selects several and fuses their rankings. default. A comma-separated spec selects several and fuses their rankings.
`kb_dir`/`root` are handed to the backend rather than left to its own
defaults. Without them a caller could pass a corpus to `run_search` and
still have the backend walk `config.KB_DIR` - the query answered from one
tree and the pages read from another, with nothing saying so.
""" """
raw = spec or os.environ.get(ENV_VAR) or DEFAULT_BACKEND raw = spec or os.environ.get(ENV_VAR) or DEFAULT_BACKEND
names = [n.strip() for n in raw.split(",") if n.strip()] names = [n.strip() for n in raw.split(",") if n.strip()]
@@ -38,4 +49,4 @@ def resolve(spec: str | None = None) -> list[SearchBackend]:
f"unknown search backend(s): {', '.join(unknown)}. " f"unknown search backend(s): {', '.join(unknown)}. "
f"Available: {', '.join(sorted(BACKENDS))}" f"Available: {', '.join(sorted(BACKENDS))}"
) )
return [BACKENDS[name]() for name in names] return [BACKENDS[name](kb_dir, root) for name in names]
+49 -11
View File
@@ -4,7 +4,7 @@ Why shell out instead of scanning in Python: `rg` is already the retrieval
layer the agent instructions point at, it handles large trees fast, and its layer the agent instructions point at, it handles large trees fast, and its
`--json` mode gives line numbers and matched text without reparsing files. `--json` mode gives line numbers and matched text without reparsing files.
Two safety properties are load-bearing and must survive any edit here: Three safety properties are load-bearing and must survive any edit here:
1. The query is passed as an *argv element*, never through a shell. There is 1. The query is passed as an *argv element*, never through a shell. There is
no `shell=True` anywhere in this module, so a query containing `;`, `$(...)` no `shell=True` anywhere in this module, so a query containing `;`, `$(...)`
@@ -12,16 +12,19 @@ Two safety properties are load-bearing and must survive any edit here:
2. `--fixed-strings` is the default. A user-supplied regex is opt-in via 2. `--fixed-strings` is the default. A user-supplied regex is opt-in via
`--regex`, so an accidental `.*` in a search term is a literal, and a `--regex`, so an accidental `.*` in a search term is a literal, and a
pathological pattern cannot be introduced without asking for one. pathological pattern cannot be introduced without asking for one.
3. A user-supplied pattern is evaluated **only** by `rg`, whose engine is
linear in the input. Nothing here hands it to Python's `re`, which
backtracks - see `_contains`.
""" """
from __future__ import annotations from __future__ import annotations
import json import json
import re
import subprocess import subprocess
from pathlib import Path from pathlib import Path
from typing import Iterable from typing import Iterable
from chemenu import config from chemenu import config
from chemenu.errors import BackendError
from chemenu.page import Page from chemenu.page import Page
from chemenu.search.base import page_key from chemenu.search.base import page_key
from chemenu.search.filters import collection_of from chemenu.search.filters import collection_of
@@ -41,13 +44,22 @@ TITLE_WEIGHT = 5.0
SUMMARY_WEIGHT = 3.0 SUMMARY_WEIGHT = 3.0
LINE_WEIGHT = 1.0 LINE_WEIGHT = 1.0
# A hang-breaker, not a performance budget. A fixed-string search over this
# corpus costs 6 ms and a deliberately broad regex 1.4 s, so nothing legitimate
# comes near this; it exists so that a pathological pattern, a corpus on a
# stalled network mount, or an `rg` that never returns fails as an error
# instead of holding the caller open forever while its output buffers into the
# heap. The caller sees the ordinary `RipgrepFailed` path.
RIPGREP_TIMEOUT_SECONDS = 30.0
class RipgrepMissing(RuntimeError):
class RipgrepMissing(BackendError):
"""Raised when the `rg` executable is not on PATH.""" """Raised when the `rg` executable is not on PATH."""
class RipgrepFailed(RuntimeError): class RipgrepFailed(BackendError):
"""Raised when `rg` exits with an error (exit code 2 or above).""" """Raised when `rg` exits with an error (exit code 2 or above), or had to
be killed for running past `RIPGREP_TIMEOUT_SECONDS`."""
def build_argv(query: SearchQuery, root: Path) -> list[str]: def build_argv(query: SearchQuery, root: Path) -> list[str]:
@@ -99,7 +111,20 @@ class RipgrepBackend:
argv = build_argv(query, self.search_root) argv = build_argv(query, self.search_root)
try: try:
proc = subprocess.run(argv, capture_output=True, text=True, check=False) proc = subprocess.run(
argv,
capture_output=True,
text=True,
check=False,
timeout=RIPGREP_TIMEOUT_SECONDS,
)
except subprocess.TimeoutExpired as exc:
raise RipgrepFailed(
f"rg did not finish within {RIPGREP_TIMEOUT_SECONDS:g}s and was killed. "
"A search that takes this long is a pathological pattern or an "
"unresponsive corpus directory, not a slow answer - narrow the query "
"or use a fixed string instead of --regex."
) from exc
except FileNotFoundError as exc: # pragma: no cover - depends on host except FileNotFoundError as exc: # pragma: no cover - depends on host
raise RipgrepMissing( raise RipgrepMissing(
"ripgrep (rg) is not installed or not on PATH. It is the search " "ripgrep (rg) is not installed or not on PATH. It is the search "
@@ -141,13 +166,26 @@ class RipgrepBackend:
def _contains(haystack: str, query: SearchQuery) -> bool: def _contains(haystack: str, query: SearchQuery) -> bool:
"""Literal containment, used only for the title and summary ranking boosts.
It never evaluates `query.text` as a regex, even when `query.regex` is set.
It used to, via `re.search`, and Python's engine backtracks: `(\\w+\\s?)+$`
against 114 characters of ordinary page text does not terminate in eight
seconds, while a pattern that fails deterministically takes 0.2 ms - the
difference is the pattern, not the haystack. `build_hit` calls this twice
per hit, and a pattern as cheap as `\\w` matches every page, so one request
bought two unbounded searches per page in the corpus.
Deleting the branch rather than bounding it is the right trade: `rg` has
already applied the pattern with a linear engine by the time we get here,
and the page is a hit *because* of that. What is lost is only the extra
weight a regex hit in the title would have scored - and since a summary and
an H1 are themselves lines in the file, `rg` still counts them. A pattern
that is mostly literal (`longhorn`) still earns its boost through the test
below; one that is not gets ranked by match count alone.
"""
if not query.text: if not query.text:
return False return False
if query.regex:
try:
return re.search(query.text, haystack, re.IGNORECASE) is not None
except re.error:
return False
return query.text.lower() in haystack.lower() return query.text.lower() in haystack.lower()
+112
View File
@@ -0,0 +1,112 @@
"""The search core, with no CLI attached.
`run_search()` and the corpus loader used to live in `commands/search.py`,
which imports `typer` at module level and `rich` through `_util`. Any
in-process caller therefore dragged the whole CLI head in behind it - which
made the "two third-party packages" the read core actually needs (`yaml`,
`jsonschema`) an accounting fiction rather than a fact about the import graph.
Nothing here imports `typer`, `rich`, or anything under `chemenu.commands`.
That is the boundary, and it is worth keeping: `commands/search.py` is now the
adapter that turns these values into terminal output and these exceptions into
exit codes, and the MCP server (Gitea #19) is a second adapter over the same
functions rather than a second implementation of them.
Errors are raised, never printed: `PredicateError` for a bad `--field`,
`RipgrepMissing`/`RipgrepFailed` for the backend. All of them are
`ChemenuError` - see `chemenu/errors.py`.
"""
from __future__ import annotations
from pathlib import Path
from chemenu import config
from chemenu.frontmatter_io import read_page_with_error
from chemenu.kb_scan import iter_kb_pages
from chemenu.page import Page
from chemenu.search import filters
from chemenu.search.base import page_key
from chemenu.search.fuse import reciprocal_rank_fusion
from chemenu.search.ripgrep import build_hit
from chemenu.search.types import SearchHit, SearchQuery
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
"""Every page under `kb/`, keyed by repo-relative path.
Path-keyed rather than title-keyed on purpose: `load_kb_pages()` drops one
of two pages sharing a stem, and search should still find both - a
duplicate title is a lint finding, not a reason to hide a page.
"""
kb_dir = kb_dir or config.KB_DIR
root = root or config.ROOT
pages: dict[str, Page] = {}
for path in iter_kb_pages(kb_dir):
frontmatter, body, error = read_page_with_error(path)
pages[page_key(path, root)] = Page(
path=path, frontmatter=frontmatter, body=body, frontmatter_error=error
)
return pages
def unreadable_pages(pages: dict[str, Page]) -> list[dict[str, str]]:
"""The pages whose frontmatter could not be used, as `{path, reason}`.
Reported rather than swallowed. Such a page has no `confidence` and no
`kind`, so it silently drops out of every positive `--field` predicate -
including the low-confidence sweep that exists to find pages in exactly
that state. Saying nothing makes it look like a page that did not match;
an empty block is excluded, because a page can legitimately carry one.
"""
return [
{"path": key, "reason": page.frontmatter_error}
for key, page in sorted(pages.items())
if page.frontmatter_error and page.frontmatter_error != "empty frontmatter block"
]
def _sort_key(hit: SearchHit, field: str):
value = hit.as_dict().get(field)
if value is None:
# Missing values sort last in either direction rather than crashing on
# a None comparison.
return (1, "")
if isinstance(value, (int, float)):
return (0, value)
return (0, str(value).lower())
def sort_hits(hits: list[SearchHit], sort: str | None) -> list[SearchHit]:
"""Sort by a hit field. A leading `-` reverses, e.g. `--sort -confidence`."""
if not sort:
return hits
descending = sort.startswith("-")
field = sort.lstrip("-")
ordered = sorted(hits, key=lambda h: _sort_key(h, field), reverse=descending)
return ordered
def run_search(
query: SearchQuery,
pages: dict[str, Page],
backends: list,
kb_dir: Path | None = None,
) -> list[SearchHit]:
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
filters.validate_fields(query.predicates, pages)
if query.text:
rankings = [backend.search(query, pages) for backend in backends]
hits = rankings[0] if len(rankings) == 1 else reciprocal_rank_fusion(rankings)
allowed = filters.apply_predicates(pages, query.predicates, kb_dir)
hits = [hit for hit in hits if hit.path in allowed]
else:
selected = filters.apply_predicates(pages, query.predicates, kb_dir)
hits = [
build_hit(page, key, [], query, backend="frontmatter", kb_dir=kb_dir)
for key, page in selected.items()
]
hits.sort(key=lambda h: h.title.lower())
hits = sort_hits(hits, query.sort)
return hits[: query.limit] if query.limit else hits
+56 -9
View File
@@ -3,7 +3,9 @@ from pathlib import Path
import pytest import pytest
from chemenu import config
from chemenu.frontmatter_io import write_page from chemenu.frontmatter_io import write_page
from chemenu.type_resolver import resolver
# Environment the tool reads for its own behaviour. Cleared for every test, so # Environment the tool reads for its own behaviour. Cleared for every test, so
# that a test which needs one sets it itself and the rest run against the # that a test which needs one sets it itself and the rest run against the
@@ -17,6 +19,7 @@ _WIKITOOL_ENV = (
"WIKITOOL_SESSION_ID", "WIKITOOL_SESSION_ID",
"WIKITOOL_UPDATE_URL", "WIKITOOL_UPDATE_URL",
"WIKITOOL_UPDATE_TOKEN", "WIKITOOL_UPDATE_TOKEN",
"CHEMENU_ROOT",
) )
# Environment git reads for identity or for where its repo lives. A stray # Environment git reads for identity or for where its repo lives. A stray
@@ -35,7 +38,7 @@ _GIT_ENV = (
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
"""Cut every test off from the machine it runs on. """Cut every test off from the machine it runs on.
The suite was green for months while silently depending on whoever ran it: The suite was green for months while silently depending on whoever ran it:
@@ -67,7 +70,34 @@ def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Pat
for name in (*_WIKITOOL_ENV, *_GIT_ENV): for name in (*_WIKITOOL_ENV, *_GIT_ENV):
monkeypatch.delenv(name, raising=False) monkeypatch.delenv(name, raising=False)
return home
# The same hole as the environment above, one layer in: `config` resolves
# its paths on access, and `monkeypatch.setattr(config, "KB_DIR", ...)`
# undoes itself by writing the *resolved* old path back as a real
# attribute. That binding outlives the test and hands the next one a
# corpus directory belonging to the previous tree. Cleared on both sides,
# so neither a leak from before nor one from this test can be inherited.
config.reset()
yield home
config.reset()
def use_shipped_type_specs(monkeypatch: pytest.MonkeyPatch) -> None:
"""Keep the shipped `types/` reachable for a test that repoints `ROOT`.
`TYPES_DIR` and the `TypeResolver`'s own root both follow `ROOT` now, which
is the whole point of making the resolution lazy - but it means a fixture
tree without a `types/` resolves no page kind at all, and a check that
depends on a page being a `source` silently stops finding one. These tests
do want the real schema: a synthetic type-spec would prove the command
works against a fixture rather than against what it ships with.
So the dependency is declared instead of inherited. While both were bound at
import time it held by accident, which is the same shape as the hole
`raw_dir` was written to close, one layer down.
"""
monkeypatch.setattr(config, "TYPES_DIR", config._PACKAGE_ROOT / "types")
monkeypatch.setattr(resolver, "_repo_root", config._PACKAGE_ROOT)
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
@@ -92,10 +122,27 @@ def isolated_trace_dir(
@pytest.fixture @pytest.fixture
def raw_dir(tmp_path: Path) -> Path: def raw_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""A small fake raw/ tree sitting next to the kb_dir fixture (same """A small fake raw/ tree sitting next to the kb_dir fixture (same
tmp_path), for provenance-coverage tests. One file is deliberately left tmp_path), for provenance-coverage tests. One file is deliberately left
uncovered by any source page.""" uncovered by any source page.
`config.ROOT` is repointed at `tmp_path` for the same reason
`hermetic_environment` clears the environment: a `raw/...` path in a
fixture page is resolved against the *repository* root by the code under
test (`provenance.legacy_source_pages`, for one), so without this the
fixture builds a raw tree that the code never looks at and answers from
the developer's own `raw/` instead.
That was not theoretical. `test_legacy_source_pages_flags_url_and_directory`
passed for months only because this checkout happened to have a
`raw/documents/` directory; the run that emptied it turned the test red in
CI while it stayed green locally, because git does not track empty
directories and the local one survived. Same shape as Gitea #8, closed the
same way - in the fixture, not in the one test that happened to trip.
"""
monkeypatch.setattr(config, "ROOT", tmp_path)
use_shipped_type_specs(monkeypatch)
raw = tmp_path / "raw" raw = tmp_path / "raw"
(raw / "notes").mkdir(parents=True) (raw / "notes").mkdir(parents=True)
(raw / "notes" / "Aurora.md").write_text("# Aurora raw notes\n", encoding="utf-8") (raw / "notes" / "Aurora.md").write_text("# Aurora raw notes\n", encoding="utf-8")
@@ -126,19 +173,19 @@ def kb_dir(tmp_path: Path) -> Path:
{ {
"type": "types/entity.md", "entity_type": "system", "type": "types/entity.md", "entity_type": "system",
"tags": ["server"], "created": "2026-07-31", "modified": "2026-07-31", "tags": ["server"], "created": "2026-07-31", "modified": "2026-07-31",
"related": ["Nathan"], "sources": [], "confidence": 0.9, "related": ["Borealis"], "sources": [], "confidence": 0.9,
"summary": "Server hosting DocStore with ZFS storage", "summary": "Server hosting DocStore with ZFS storage",
}, },
"\n# aurora\n\n## Description\n\nHosts things.\n\n## Relationships\n\n- **Related to:** [[Nathan]]\n\n## See Also\n\n- [[Nathan]]\n", "\n# aurora\n\n## Description\n\nHosts things.\n\n## Relationships\n\n- **Related to:** [[Borealis]]\n\n## See Also\n\n- [[Borealis]]\n",
) )
write_page( write_page(
kb / "entities/systems/Nathan.md", kb / "entities/systems/Borealis.md",
{ {
"type": "types/entity.md", "entity_type": "system", "type": "types/entity.md", "entity_type": "system",
"tags": ["workstation"], "created": "2026-08-02", "modified": "2026-08-02", "tags": ["workstation"], "created": "2026-08-02", "modified": "2026-08-02",
"related": ["aurora"], "sources": [], "confidence": 0.9, "related": ["aurora"], "sources": [], "confidence": 0.9,
}, },
"\n# Nathan\n\n## Description\n\nA workstation.\n\n## Relationships\n\n- **Related to:** [[aurora]]\n", "\n# Borealis\n\n## Description\n\nA workstation.\n\n## Relationships\n\n- **Related to:** [[aurora]]\n",
) )
write_page( write_page(
kb / "entities/tools/gdeploy.md", kb / "entities/tools/gdeploy.md",
@@ -169,7 +216,7 @@ def kb_dir(tmp_path: Path) -> Path:
) )
(kb / "index.md").write_text( (kb / "index.md").write_text(
"# Wiki Index\n\n[[aurora]] [[Nathan]] [[gdeploy]] [[Modbus]] [[Source - Aurora]]\n", "# Wiki Index\n\n[[aurora]] [[Borealis]] [[gdeploy]] [[Modbus]] [[Source - Aurora]]\n",
encoding="utf-8", encoding="utf-8",
) )
(kb / "log.md").write_text("# Wiki Log\n", encoding="utf-8") (kb / "log.md").write_text("# Wiki Log\n", encoding="utf-8")
+184
View File
@@ -0,0 +1,184 @@
"""Tests for the in-process library boundary (Gitea #31).
Two properties, and neither is about the values coming back:
1. A caller can point Chemenu at a corpus tree and **no path of the checkout
this package lives in is read**. That is what "library" means here, and it
is what could not be asserted before: `config.KB_DIR` was bound at import
time, so a caller repointing `ROOT` was still answered out of the developer's
own `kb/`.
2. The surface is read-only **structurally**. `chemenu.api` imports nothing
under `chemenu.commands`, so `new`, `publish` and the rest are not reachable
from it - not filtered out of it.
"""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import pytest
from chemenu import config
from chemenu.api import Corpus
from chemenu.errors import ChemenuError, ValidationError
from chemenu.types_core import UnknownType
@pytest.fixture
def foreign_corpus(tmp_path: Path) -> Path:
"""A corpus tree that is not this checkout, with one findable page."""
root = tmp_path / "elsewhere"
kb = root / "kb" / "entities"
kb.mkdir(parents=True)
(root / "kb" / "entities" / "COLLECTION.md").write_text("# entities\n", encoding="utf-8")
(kb / "Peregrine.md").write_text(
"---\ntype: types/entity.md\nentity_type: system\nconfidence: 0.42\n"
"summary: A system that exists only in this fixture.\n---\n\n"
"# Peregrine\n\nPeregrine is the fixture's own system.\n",
encoding="utf-8",
)
return root
def test_a_corpus_can_be_named_and_is_the_one_that_answers(foreign_corpus):
result = Corpus(foreign_corpus).search("Peregrine")
assert [hit["title"] for hit in result["results"]] == ["Peregrine"]
assert result["results"][0]["path"] == "kb/entities/Peregrine.md"
def test_no_path_of_this_checkout_is_read_while_a_foreign_root_is_set(foreign_corpus):
"""The acceptance criterion, asserted rather than argued.
`Path.read_text` and `Path.rglob` are the two ways a page reaches the
reader; both are watched, and any access under the real repository root
fails the test. Before the root resolution was made lazy this test could not
pass: `config.KB_DIR` was already bound to this checkout's `kb/`.
"""
package_root = config._PACKAGE_ROOT
trespasses: list[str] = []
real_read_text = Path.read_text
real_rglob = Path.rglob
def watched_read_text(self, *args, **kwargs):
_note(self)
return real_read_text(self, *args, **kwargs)
def watched_rglob(self, *args, **kwargs):
_note(self)
return real_rglob(self, *args, **kwargs)
def _note(path: Path) -> None:
try:
path.resolve().relative_to(package_root)
except ValueError:
return
trespasses.append(str(path))
monkey = pytest.MonkeyPatch()
monkey.setattr(Path, "read_text", watched_read_text)
monkey.setattr(Path, "rglob", watched_rglob)
try:
Corpus(foreign_corpus).search("Peregrine")
Corpus(foreign_corpus).search(predicates=["confidence<0.6"])
finally:
monkey.undo()
assert trespasses == []
def test_the_env_var_points_the_default_corpus(foreign_corpus, monkeypatch):
"""`CHEMENU_ROOT` exists for a caller that *is* a whole process and has
nothing to pass an argument through."""
monkeypatch.setenv(config.ENV_ROOT, str(foreign_corpus))
assert Corpus().root == foreign_corpus.resolve()
assert Corpus().kb_dir == foreign_corpus.resolve() / "kb"
def test_without_the_env_var_the_root_is_this_checkout(monkeypatch):
"""The default has to be unchanged, or `tools/wikitool` moves under
everyone's feet."""
monkeypatch.delenv(config.ENV_ROOT, raising=False)
assert config.resolve_root() == config._PACKAGE_ROOT
def test_an_explicit_argument_beats_the_env_var(foreign_corpus, tmp_path, monkeypatch):
"""A caller serving two corpora cannot tell them apart with a process-wide
variable, so the argument has to win."""
monkeypatch.setenv(config.ENV_ROOT, str(tmp_path / "somewhere-else"))
assert Corpus(foreign_corpus).root == foreign_corpus.resolve()
def test_derived_paths_follow_the_root_instead_of_lagging_behind(foreign_corpus, monkeypatch):
"""The failure that made the old shape worse than the limitation: `ROOT`
moved and `KB_DIR` did not, so a caller believed it was working on the
target tree while reading this one."""
monkeypatch.setattr(config, "ROOT", foreign_corpus)
assert config.KB_DIR == foreign_corpus / "kb"
assert config.RAW_DIR == foreign_corpus / "raw"
assert config.INDEX_FILE == foreign_corpus / "kb" / "index.md"
def test_validation_errors_are_raised_not_exited(foreign_corpus):
corpus = Corpus(foreign_corpus)
with pytest.raises(ValidationError):
corpus.search()
with pytest.raises(ValidationError):
corpus.search("x", predicates=["not a predicate"])
with pytest.raises(UnknownType):
corpus.describe_type("no-such-type")
# One class to catch, whatever went wrong.
with pytest.raises(ChemenuError):
corpus.describe_type("no-such-type")
def test_the_read_surface_cannot_reach_a_write_command():
"""Structural, not filtered: import `chemenu.api` in a clean interpreter and
nothing under `chemenu.commands` is loaded, so there is no `publish` to
call. Run out-of-process because this suite has already imported the CLI."""
code = (
"import sys, chemenu.api;"
"print([m for m in sys.modules if m.startswith('chemenu.commands')]);"
"print([m for m in sys.modules if m in ('typer', 'rich', 'click')])"
)
result = subprocess.run(
[sys.executable, "-c", code],
cwd=config._PACKAGE_ROOT / "tools",
capture_output=True,
text=True,
check=True,
)
commands_loaded, cli_loaded = result.stdout.strip().splitlines()
assert commands_loaded == "[]", f"api pulled in command modules: {commands_loaded}"
assert cli_loaded == "[]", f"api pulled in the CLI head: {cli_loaded}"
def test_every_answer_carries_the_revision_it_was_computed_from(foreign_corpus):
"""A stale checkout answers confidently and wrongly otherwise. Outside git
there is no commit, and the stamp says so rather than inventing one."""
corpus = Corpus(foreign_corpus)
result = corpus.search("Peregrine")
assert result["commit"] is None and result["as_of"]
subprocess.run(["git", "init", "-b", "main"], cwd=foreign_corpus, check=True,
capture_output=True)
for key, value in (("user.name", "Fixture"), ("user.email", "f@example.com")):
subprocess.run(["git", "config", key, value], cwd=foreign_corpus, check=True,
capture_output=True)
subprocess.run(["git", "add", "-A"], cwd=foreign_corpus, check=True, capture_output=True)
subprocess.run(["git", "commit", "-m", "corpus"], cwd=foreign_corpus, check=True,
capture_output=True)
stamped = Corpus(foreign_corpus).search("Peregrine")
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=foreign_corpus,
capture_output=True, text=True, check=True).stdout.strip()
assert stamped["commit"] == head
def test_lint_and_status_answer_from_the_named_corpus(foreign_corpus):
corpus = Corpus(foreign_corpus)
assert corpus.lint()["page_count"] == 1
status = corpus.status()
assert status["pages"] == 1
assert status["collections"] == {"entities": 1}
+98
View File
@@ -0,0 +1,98 @@
"""Tests for the revision-keyed corpus cache (Gitea #33, feeding #19).
The property under test is not "it is fast" but "it is never stale". A cache on
this path that answers from a superseded parse is worse than the reparse it
replaces: the caller gets a confident answer about a corpus that no longer
exists, which is the failure `SOUL.md` names as the cardinal one.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import pytest
from chemenu import config, corpus_cache
from chemenu.corpus_cache import CorpusCache
def _git(root: Path, *args: str) -> None:
subprocess.run(["git", *args], cwd=root, check=True, capture_output=True)
@pytest.fixture
def repo(kb_dir: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""The `kb_dir` fixture tree, committed - so there is a revision to key on."""
_git(tmp_path, "init", "-b", "main")
_git(tmp_path, "config", "user.name", "Fixture Author")
_git(tmp_path, "config", "user.email", "fixture@example.com")
_git(tmp_path, "add", "-A")
_git(tmp_path, "commit", "-m", "fixture corpus")
monkeypatch.setattr(config, "ROOT", tmp_path)
monkeypatch.setattr(config, "KB_DIR", kb_dir)
return tmp_path
def test_a_clean_tree_is_cached_and_reused(repo, kb_dir):
cache = CorpusCache(kb_dir, repo)
first, revision = cache.load()
second, again = cache.load()
assert revision is not None and again == revision
# Identity, not equality: an equal-but-rebuilt dict would mean the reparse
# still happened, which is the whole cost this exists to remove.
assert first is second
def test_a_new_commit_invalidates_the_cache(repo, kb_dir):
cache = CorpusCache(kb_dir, repo)
first, first_revision = cache.load()
(kb_dir / "entities" / "Latecomer.md").write_text(
"---\ntype: types/entity.md\n---\n\n# Latecomer\n", encoding="utf-8"
)
_git(repo, "add", "-A")
_git(repo, "commit", "-m", "one more page")
second, second_revision = cache.load()
assert second_revision != first_revision
assert len(second) == len(first) + 1
def test_a_dirty_tree_is_never_cached(repo, kb_dir):
"""The correctness case. A session that writes a page and then searches for
it must not be answered from the parse taken before the write - and nothing
about the commit SHA changed in between."""
cache = CorpusCache(kb_dir, repo)
first, revision = cache.load()
assert revision is not None
(kb_dir / "entities" / "Uncommitted.md").write_text(
"---\ntype: types/entity.md\n---\n\n# Uncommitted\n", encoding="utf-8"
)
second, dirty_revision = cache.load()
assert dirty_revision is None
assert len(second) == len(first) + 1
assert cache.revision is None
def test_a_tree_outside_git_reloads_every_time(kb_dir, tmp_path):
cache = CorpusCache(kb_dir, tmp_path)
pages, revision = cache.load()
assert revision is None and pages
assert cache.load()[0] is not pages
def test_an_unanswerable_git_counts_as_dirty(repo, kb_dir, monkeypatch):
"""Errs toward reloading: a cache that reads "unknown" as "clean" serves
stale pages, which is the one outcome this module exists to prevent."""
monkeypatch.setattr(corpus_cache, "_git", lambda *a, **k: None)
assert corpus_cache.is_dirty(repo, kb_dir) is True
assert CorpusCache(kb_dir, repo).load()[1] is None
def test_head_commit_is_the_revision_a_response_gets_stamped_with(repo):
expected = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=repo, capture_output=True, text=True, check=True
).stdout.strip()
assert corpus_cache.head_commit(repo) == expected
+6 -6
View File
@@ -15,8 +15,8 @@ from chemenu.page import Page
BODY = """ BODY = """
# Aurora # Aurora
Runs [[DocStore]] and talks to [[Nathan]].[^src-notes] Runs [[DocStore]] and talks to [[Borealis]].[^src-notes]
The second mention of [[Nathan]] is what a careless rewrite loses. The second mention of [[Borealis]] is what a careless rewrite loses.
## Fußnoten ## Fußnoten
@@ -31,7 +31,7 @@ def page(body: str, **frontmatter) -> Page:
"created": "2026-07-31", "created": "2026-07-31",
"provenance": "sourced", "provenance": "sourced",
"confidence_base": 0.9, "confidence_base": 0.9,
"related": ["Nathan"], "related": ["Borealis"],
"sources": ["Source - Aurora"], "sources": ["Source - Aurora"],
} }
base.update(frontmatter) base.update(frontmatter)
@@ -57,7 +57,7 @@ def kinds(diff) -> list[str]:
def test_a_dropped_occurrence_is_caught_even_though_the_link_set_is_unchanged(): def test_a_dropped_occurrence_is_caught_even_though_the_link_set_is_unchanged():
after = BODY.replace("The second mention of [[Nathan]] is", "Das zweite Vorkommen ist") after = BODY.replace("The second mention of [[Borealis]] is", "Das zweite Vorkommen ist")
before_shapes, after_shapes = shapes(BODY, after) before_shapes, after_shapes = shapes(BODY, after)
# Precondition: the sets really are identical, so a set-based check passes. # Precondition: the sets really are identical, so a set-based check passes.
@@ -67,7 +67,7 @@ def test_a_dropped_occurrence_is_caught_even_though_the_link_set_is_unchanged():
diff = corpus_diff.compare(before_shapes, after_shapes) diff = corpus_diff.compare(before_shapes, after_shapes)
assert kinds(diff) == ["wikilinks"] assert kinds(diff) == ["wikilinks"]
assert "'Nathan' 2->1" in diff.findings[0].detail assert "'Borealis' 2->1" in diff.findings[0].detail
assert not diff.ok assert not diff.ok
@@ -85,7 +85,7 @@ def test_pure_prose_change_is_not_reported():
def test_a_dropped_citation_is_caught(): def test_a_dropped_citation_is_caught():
after = BODY.replace("[[Nathan]].[^src-notes]", "[[Nathan]].") after = BODY.replace("[[Borealis]].[^src-notes]", "[[Borealis]].")
diff = corpus_diff.compare(*shapes(BODY, after)) diff = corpus_diff.compare(*shapes(BODY, after))
assert "cite-refs" in kinds(diff) assert "cite-refs" in kinds(diff)
+34
View File
@@ -220,6 +220,40 @@ def test_a_renamed_but_unfilled_environment_template_warns(instance):
assert "template" in next(c.detail for c in checks if c.name == "environment") assert "template" in next(c.detail for c in checks if c.name == "environment")
def _remotes_detail(checks) -> str:
return next(c.detail for c in checks if c.name == "publish-remotes")
def test_publish_remotes_says_not_armed_when_the_file_is_absent(instance):
"""Absence stays OK - a single-remote checkout has nothing to protect - but
the line has to say the gate is off. AGENTS.md lists it among the limits
enforced in code, so "no .wikitool-remotes.json" alone leaves a reader
trusting a safeguard that is not running."""
checks = doctor.run_doctor()
assert _status(checks, "publish-remotes") == "OK"
assert "not armed" in _remotes_detail(checks)
def test_publish_remotes_says_armed_when_the_file_lists_a_target(instance):
(config.ROOT / config.PUBLISH_REMOTES_FILENAME).write_text(
'{ "schema": 1, "allowed_push_urls": ["ssh://git@example.net/one.git"] }',
encoding="utf-8",
)
checks = doctor.run_doctor()
assert _status(checks, "publish-remotes") == "OK"
detail = _remotes_detail(checks)
assert "armed" in detail and "not armed" not in detail
def test_publish_remotes_warns_on_several_remotes_without_an_allowlist(instance):
"""The shape a private instance has once it adds the public upstream."""
_git(config.ROOT, "remote", "add", "origin", "ssh://git@example.net/mine.git")
_git(config.ROOT, "remote", "add", "upstream", "ssh://git@example.net/theirs.git")
checks = doctor.run_doctor()
assert _status(checks, "publish-remotes") == "WARN"
assert "not armed" in _remotes_detail(checks)
def test_missing_generated_file_fails(instance): def test_missing_generated_file_fails(instance):
config.LOG_FILE.unlink() config.LOG_FILE.unlink()
checks = doctor.run_doctor() checks = doctor.run_doctor()
+102 -1
View File
@@ -1,4 +1,17 @@
from chemenu.frontmatter_io import dump_frontmatter, read_page, write_page import pytest
import yaml
from chemenu.frontmatter_io import (
MAX_FRONTMATTER_BYTES,
FrontmatterError,
_Loader,
dump_frontmatter,
frontmatter_error,
read_page,
read_page_strict,
read_page_with_error,
write_page,
)
@@ -82,3 +95,91 @@ def test_scalar_quoting_is_unchanged_by_the_flow_fix(tmp_path):
assert dump_frontmatter({"year": "1945"}) == "year: '1945'" assert dump_frontmatter({"year": "1945"}) == "year: '1945'"
assert dump_frontmatter({"summary": "He said hi"}) == "summary: He said hi" assert dump_frontmatter({"summary": "He said hi"}) == "summary: He said hi"
assert dump_frontmatter({"confidence": 0.85}) == "confidence: 0.85" assert dump_frontmatter({"confidence": 0.85}) == "confidence: 0.85"
# --- Read-path limits (Gitea #33) -------------------------------------------
#
# These are regressions against parser properties, not against a caller. Each
# one is reproducible on an unpatched tree, and each one becomes reachable the
# moment frontmatter this instance did not write itself reaches the parser.
# 267 bytes that compose into 672,603 nodes on traversal, parsed in 0.2 ms:
# every level references the one below it nine times, so the object graph grows
# 9**n while the input stays tiny and the parse time stays constant. A size
# limit does not touch this, which is why there is a second check.
ALIAS_BOMB = """\
a: &a ["x", "x", "x", "x", "x", "x", "x", "x", "x"]
b: &b [*a, *a, *a, *a, *a, *a, *a, *a, *a]
c: &c [*b, *b, *b, *b, *b, *b, *b, *b, *b]
d: &d [*c, *c, *c, *c, *c, *c, *c, *c, *c]
e: &e [*d, *d, *d, *d, *d, *d, *d, *d, *d]
f: &f [*e, *e, *e, *e, *e, *e, *e, *e, *e]
"""
def _page(tmp_path, frontmatter_text):
path = tmp_path / "Bomb.md"
path.write_text(f"---\n{frontmatter_text}---\n\n# Bomb\n", encoding="utf-8")
return path
def test_an_alias_bomb_is_refused_rather_than_expanded(tmp_path):
path = _page(tmp_path, ALIAS_BOMB)
frontmatter, _, error = read_page_with_error(path)
assert frontmatter == {}
assert "alias" in error
def test_the_alias_check_costs_nothing_on_a_page_that_has_none(tmp_path):
"""`*` is a necessary character in an alias node, so its absence proves
absence - which is the path every real page takes."""
path = _page(tmp_path, "type: types/entity.md\ntags: [k8s, storage]\n")
frontmatter, _, error = read_page_with_error(path)
assert error is None
assert frontmatter["tags"] == ["k8s", "storage"]
def test_oversized_frontmatter_is_refused(tmp_path):
path = _page(tmp_path, "summary: " + ("x" * (MAX_FRONTMATTER_BYTES + 1)) + "\n")
_, _, error = read_page_with_error(path)
assert "over the" in error and "limit" in error
def test_a_page_at_the_size_limit_still_parses(tmp_path):
filler = "x" * (MAX_FRONTMATTER_BYTES - len("summary: \n"))
path = _page(tmp_path, f"summary: {filler}\n")
frontmatter, _, error = read_page_with_error(path)
assert error is None and frontmatter["summary"] == filler
def test_the_permissive_read_hands_back_the_reason_it_was_permissive(tmp_path):
path = _page(tmp_path, "type: [unclosed\n")
frontmatter, _, error = read_page_with_error(path)
assert frontmatter == {}
assert error and "invalid YAML" in error
# The historical behaviour - the reason dropped on the floor - is what let
# a broken page slip past every --field predicate looking like a miss.
assert read_page(path) == ({}, "\n# Bomb\n")
def test_the_strict_read_raises_instead_of_emptying(tmp_path):
path = _page(tmp_path, "type: [unclosed\n")
with pytest.raises(FrontmatterError):
read_page_strict(path)
def test_frontmatter_error_and_the_permissive_read_agree(tmp_path):
"""One parser behind both, so they cannot describe the same file
differently - and a caller wanting both answers reads the file once."""
for text in (ALIAS_BOMB, "type: [unclosed\n", "type: types/entity.md\n", "\n"):
path = _page(tmp_path, text)
assert frontmatter_error(path) == read_page_with_error(path)[2]
def test_the_fast_loader_is_used_when_libyaml_is_available():
"""The corpus parse is the largest single cost of a search and scales with
the corpus, so the factor-3 loader is not micro-tuning."""
if hasattr(yaml, "CSafeLoader"):
assert _Loader is yaml.CSafeLoader
else: # pragma: no cover - depends on the host
assert _Loader is yaml.SafeLoader
+113
View File
@@ -1,3 +1,4 @@
import json
import subprocess import subprocess
import pytest import pytest
@@ -905,3 +906,115 @@ def test_numstat_survives_a_non_ascii_filename(repo):
change = next(c for c in collect_changes([]) if c.path == name) change = next(c for c in collect_changes([]) if c.path == name)
assert change.status == "modified" assert change.status == "modified"
assert (change.added, change.removed) == (1, 2) assert (change.added, change.removed) == (1, 2)
# --- Publish-Remote Gate -----------------------------------------------------
def _allowlist(root, *urls):
(root / config.PUBLISH_REMOTES_FILENAME).write_text(
json.dumps({"schema": 1, "allowed_push_urls": list(urls)}), encoding="utf-8"
)
def test_no_allowlist_means_unrestricted(repo):
"""Absence is a legitimate state: a checkout with nothing private in it
should not have to declare anything to publish at all."""
assert git_publish.read_allowed_push_urls() is None
assert git_publish.publish_remote_refusal("origin", "main") is None
def test_allowed_url_passes_the_gate(repo):
_allowlist(repo, git_publish.push_url_for("origin"))
assert git_publish.publish_remote_refusal("origin", "main") is None
def test_gate_refuses_a_remote_not_on_the_list(repo):
_git(repo, "remote", "add", "upstream", "https://example.com/public.git")
_allowlist(repo, git_publish.push_url_for("origin"))
refusal = git_publish.publish_remote_refusal("upstream", "main")
assert refusal is not None
assert "https://example.com/public.git" in refusal
assert "Nothing was committed or pushed" in refusal
def test_gate_matches_the_url_not_the_remote_name(repo):
"""A name-based list would pass a repointed `origin`, which is the failure
this gate exists to catch."""
_allowlist(repo, "ssh://git@example.com/only-this.git")
assert git_publish.publish_remote_refusal("origin", "main") is not None
def test_gate_reads_pushurl_when_the_remote_sets_one(repo):
"""`git push` writes to `pushurl` when present, so that is the value that
has to be checked - not the fetch URL beside it."""
_git(repo, "remote", "set-url", "--push", "origin", "https://example.com/elsewhere.git")
_allowlist(repo, "https://example.com/elsewhere.git")
assert git_publish.push_url_for("origin") == "https://example.com/elsewhere.git"
assert git_publish.publish_remote_refusal("origin", "main") is None
def test_gate_refuses_when_the_fetch_url_is_listed_but_the_pushurl_is_not(repo):
fetch_url = git_publish.push_url_for("origin")
_git(repo, "remote", "set-url", "--push", "origin", "https://example.com/elsewhere.git")
_allowlist(repo, fetch_url)
assert git_publish.publish_remote_refusal("origin", "main") is not None
def test_publish_exits_42_and_commits_nothing_when_the_remote_is_refused(repo):
_git(repo, "remote", "add", "upstream", "https://example.com/public.git")
_allowlist(repo, git_publish.push_url_for("origin"))
before = _git(repo, "rev-parse", "HEAD").stdout.strip()
(repo / "kb" / "secret.md").write_text("private\n", encoding="utf-8")
with pytest.raises(typer.Exit) as excinfo:
_publish(remote="upstream")
assert excinfo.value.exit_code == EXIT_NEEDS_CLEARANCE
# Nothing committed, and the file is still sitting there unstaged - the
# refusal message promises both. (`git status --porcelain` collapses the
# wholly-untracked `kb/` to one entry, so check the index directly.)
assert _git(repo, "rev-parse", "HEAD").stdout.strip() == before
assert (repo / "kb" / "secret.md").exists()
assert "secret.md" not in _git(repo, "ls-files").stdout
def test_no_push_skips_the_gate(repo):
"""`--no-push` publishes nowhere, so there is no wrong target to protect
against - and a local commit must stay possible."""
_allowlist(repo, "ssh://git@example.com/only-this.git")
(repo / "kb" / "page.md").write_text("local\n", encoding="utf-8")
_publish(push=False)
assert "page.md" in _git(repo, "show", "--name-only", "HEAD").stdout
def test_unreadable_allowlist_fails_instead_of_falling_open(repo):
"""A broken file must not be read as 'no restriction' - that would turn a
corrupted safeguard into a silently disabled one."""
(repo / config.PUBLISH_REMOTES_FILENAME).write_text("{not json", encoding="utf-8")
with pytest.raises(typer.Exit):
git_publish.read_allowed_push_urls()
def test_allowlist_without_a_usable_list_fails(repo):
(repo / config.PUBLISH_REMOTES_FILENAME).write_text(
json.dumps({"schema": 1, "allowed_push_urls": "not-a-list"}), encoding="utf-8"
)
with pytest.raises(typer.Exit):
git_publish.read_allowed_push_urls()
def test_empty_allowlist_refuses_everything(repo):
"""An empty list is a deliberate 'publish nowhere', not an oversight that
should behave like an absent file."""
_allowlist(repo)
assert git_publish.publish_remote_refusal("origin", "main") is not None
def test_gate_has_no_flag_that_opens_it(repo):
"""The other two gates clear with a token; this one deliberately does not,
because the right fix is a deliberate edit by the user."""
import inspect
params = inspect.signature(publish_command).parameters
assert not any("remote" in name and "confirm" in name for name in params)
+2 -2
View File
@@ -99,13 +99,13 @@ def test_map_deep_links_an_inlined_area_by_anchor(kb_dir):
def test_collection_shard_holds_the_page_rows(plan, kb_dir): def test_collection_shard_holds_the_page_rows(plan, kb_dir):
entities = _shard(plan, kb_dir, "entities") entities = _shard(plan, kb_dir, "entities")
assert "[[aurora]]" in entities assert "[[aurora]]" in entities
assert "[[Nathan]]" in entities assert "[[Borealis]]" in entities
assert "[[gdeploy]]" in entities assert "[[gdeploy]]" in entities
def test_pages_are_grouped_by_area_and_sorted_case_insensitively(plan, kb_dir): def test_pages_are_grouped_by_area_and_sorted_case_insensitively(plan, kb_dir):
systems = _section(_shard(plan, kb_dir, "entities"), f"## {_area_title('system')}") systems = _section(_shard(plan, kb_dir, "entities"), f"## {_area_title('system')}")
assert systems.index("[[aurora]]") < systems.index("[[Nathan]]") assert systems.index("[[aurora]]") < systems.index("[[Borealis]]")
assert "[[gdeploy]]" not in systems assert "[[gdeploy]]" not in systems
@@ -11,6 +11,7 @@ import pytest
import typer import typer
from chemenu import config from chemenu import config
from chemenu.tests.conftest import use_shipped_type_specs
from chemenu.commands import instructions_cmd from chemenu.commands import instructions_cmd
@@ -37,6 +38,10 @@ def layer(tmp_path: Path, monkeypatch):
monkeypatch.setattr(config, "INSTRUCTIONS_DIR", instructions) monkeypatch.setattr(config, "INSTRUCTIONS_DIR", instructions)
monkeypatch.setattr(config, "AGENTS_SKILLS_DIR", root / ".agents" / "skills") monkeypatch.setattr(config, "AGENTS_SKILLS_DIR", root / ".agents" / "skills")
monkeypatch.setattr(config, "CLAUDE_SKILLS_DIR", root / ".claude" / "skills") monkeypatch.setattr(config, "CLAUDE_SKILLS_DIR", root / ".claude" / "skills")
# `verify` resolves each instruction's `type: types/instruction.md`, and the
# resolver's root follows `ROOT` now - so the shipped specs have to be named
# rather than inherited. See `use_shipped_type_specs`.
use_shipped_type_specs(monkeypatch)
return root return root
+1 -1
View File
@@ -31,7 +31,7 @@ def test_directory_without_a_contract_is_not_a_collection(repo):
def test_area_resolves_to_its_enclosing_collection(repo): def test_area_resolves_to_its_enclosing_collection(repo):
page = repo / "kb" / "entities" / "systems" / "hermes.md" page = repo / "kb" / "entities" / "systems" / "gateway.md"
assert kb_collections.kb_collection_of(page) == repo / "kb" / "entities" assert kb_collections.kb_collection_of(page) == repo / "kb" / "entities"
+2 -2
View File
@@ -128,9 +128,9 @@ def test_lint_detects_orphan_page(kb_dir):
) )
report = run_lint(kb_dir) report = run_lint(kb_dir)
assert "isolated" in report["orphan_pages"] assert "isolated" in report["orphan_pages"]
# aurora and Nathan link to each other, so they must not be reported as orphans. # aurora and Borealis link to each other, so they must not be reported as orphans.
assert "aurora" not in report["orphan_pages"] assert "aurora" not in report["orphan_pages"]
assert "Nathan" not in report["orphan_pages"] assert "Borealis" not in report["orphan_pages"]
def test_lint_detects_missing_frontmatter_fields(kb_dir): def test_lint_detects_missing_frontmatter_fields(kb_dir):
+274
View File
@@ -0,0 +1,274 @@
"""Tests for the MCP read server (Gitea #19).
The acceptance criteria that are not about a value coming back:
- the wire format **is** the CLI's `--json` form, held together by a golden
test rather than by intention;
- no path of the server writes into `kb/`, `reports/` or git;
- there is no write tool, because nothing under `chemenu.commands` is
importable from it - structural, not filtered;
- every response carries the commit it was computed from;
- telemetry cannot land inside the served checkout.
"""
from __future__ import annotations
import asyncio
import json
import shutil
import subprocess
import sys
from pathlib import Path
import pytest
from chemenu import config
pytest.importorskip("mcp", reason="the MCP server's dependency is optional; see "
"tools/requirements-mcp.txt")
from chemenu.mcp.server import ( # noqa: E402 - after the skip guard
TRANSPORTS,
TraceWouldWriteIntoCorpus,
build_server,
check_trace_destination,
serve,
)
@pytest.fixture
def corpus(tmp_path: Path) -> Path:
"""A committed instance that is not this checkout.
Carries its own `types/`, because a served corpus is a whole instance: the
page kinds and `describe_type` are answered from the instance's own schema,
not from whichever checkout the package happens to sit in. Copied rather
than linked - a link would let a test that writes to `<root>/types/...`
write into this repository, which is not hypothetical.
"""
root = tmp_path / "served"
entities = root / "kb" / "entities"
entities.mkdir(parents=True)
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
(root / "kb" / "entities" / "COLLECTION.md").write_text("# entities\n", encoding="utf-8")
(entities / "Kingfisher.md").write_text(
"---\ntype: types/entity.md\nentity_type: system\nconfidence: 0.55\n"
"summary: The fixture's own system.\n---\n\n"
"# Kingfisher\n\nKingfisher is the system this fixture is about.\n",
encoding="utf-8",
)
subprocess.run(["git", "init", "-b", "main"], cwd=root, check=True, capture_output=True)
for key, value in (("user.name", "Fixture"), ("user.email", "f@example.com")):
subprocess.run(["git", "config", key, value], cwd=root, check=True, capture_output=True)
subprocess.run(["git", "add", "-A"], cwd=root, check=True, capture_output=True)
subprocess.run(["git", "commit", "-m", "corpus"], cwd=root, check=True, capture_output=True)
return root
def _call(server, name: str, arguments: dict | None = None) -> dict:
result = asyncio.run(server.call_tool(name, arguments or {}))
if isinstance(result, tuple):
result = result[1]
assert not result.is_error, result.content
return result.structured_content
def _tree(root: Path) -> dict[str, tuple[int, bytes]]:
"""Every file under `root` with its size and contents, for a before/after
comparison that catches a rewrite with the same length."""
return {
str(path.relative_to(root)): (path.stat().st_size, path.read_bytes())
for path in sorted(root.rglob("*"))
if path.is_file()
}
def test_the_four_tools_are_there_and_nothing_that_writes(corpus):
server = build_server(corpus, check_trace=False)
names = {tool.name for tool in asyncio.run(server.list_tools())}
assert names == {"search", "types", "describe_type", "lint", "status"}
# Named rather than pattern-matched: the point is that these are absent
# because the functions are unreachable, and a test that only looked for
# "no tool called publish" would pass on a filtered list too.
assert names.isdisjoint({"new", "touch", "xref", "cite", "publish", "migrate", "rm"})
def test_no_tool_writes_anything_into_the_corpus_or_git(corpus):
server = build_server(corpus, check_trace=False)
before = _tree(corpus)
head_before = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
).stdout
_call(server, "search", {"query": "Kingfisher"})
_call(server, "search", {"predicates": ["confidence<0.6"]})
_call(server, "types")
_call(server, "describe_type", {"name": "entity"})
_call(server, "lint")
_call(server, "status")
assert _tree(corpus) == before
head_after = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
).stdout
assert head_after == head_before
# `lint` without `--json` writes a report into `reports/`; the server must
# only ever reach the JSON form.
assert not (corpus / "reports").exists()
assert subprocess.run(
["git", "status", "--porcelain"], cwd=corpus, capture_output=True, text=True, check=True
).stdout == ""
def test_the_wire_format_is_the_clis_json_form(corpus):
"""The golden test. One contract, with the CLI as its executable
specification - so the two cannot drift while both look correct.
The CLI is run as a subprocess against the same tree, pointed at it with
`CHEMENU_ROOT` - which is also an end-to-end check that the root resolution
works from outside the process.
"""
server = build_server(corpus, check_trace=False)
served = _call(server, "search", {"query": "Kingfisher", "limit": 5})
import os
environment = {**os.environ, config.ENV_ROOT: str(corpus), "WIKI_TRACE": "0"}
completed = subprocess.run(
[str(config._PACKAGE_ROOT / "tools" / "wikitool"),
"search", "Kingfisher", "--limit", "5", "--json"],
capture_output=True, text=True, check=True, env=environment,
)
from_cli = json.loads(completed.stdout)
# `generated` is the CLI's date stamp and `commit`/`as_of` are the server's
# revision stamp - two answers to "when", neither of them a finding. What
# has to match is everything that describes the *corpus*.
shared = ("query", "predicates", "backend", "count", "results", "unreadable")
assert {key: served[key] for key in shared} == {key: from_cli[key] for key in shared}
def test_types_and_describe_match_the_cli_too(corpus):
import os
server = build_server(corpus, check_trace=False)
environment = {**os.environ, config.ENV_ROOT: str(corpus), "WIKI_TRACE": "0"}
wikitool = str(config._PACKAGE_ROOT / "tools" / "wikitool")
from_cli = json.loads(subprocess.run(
[wikitool, "types", "list", "--json"],
capture_output=True, text=True, check=True, env=environment,
).stdout)
assert _call(server, "types")["types"] == from_cli
from_cli = json.loads(subprocess.run(
[wikitool, "types", "describe", "entity", "--json"],
capture_output=True, text=True, check=True, env=environment,
).stdout)
served = _call(server, "describe_type", {"name": "entity"})
assert {k: v for k, v in served.items() if k not in ("commit", "as_of")} == from_cli
def test_every_response_carries_the_commit_it_was_computed_from(corpus):
server = build_server(corpus, check_trace=False)
head = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
).stdout.strip()
for name in ("types", "lint", "status"):
answer = _call(server, name)
assert answer["commit"] == head
assert answer["as_of"]
def test_a_dirty_tree_is_stamped_with_no_commit_rather_than_a_wrong_one(corpus):
"""The answer no longer corresponds to any revision, and says so instead of
naming the commit it is no longer about."""
(corpus / "kb" / "entities" / "Latecomer.md").write_text(
"---\ntype: types/entity.md\n---\n\n# Latecomer\n", encoding="utf-8"
)
assert _call(build_server(corpus, check_trace=False), "status")["commit"] is None
def test_a_bad_predicate_comes_back_as_a_tool_error_not_a_traceback(corpus):
"""The SDK draws the line this depends on: a `ToolError` is a deliberate
refusal whose text reaches the caller, while any other exception is a crash
whose text stays on the server. A bad predicate is the caller's argument, so
the message that says what to write instead has to travel."""
from mcp.server.mcpserver.exceptions import ToolError, UnexpectedToolError
server = build_server(corpus, check_trace=False)
with pytest.raises(ToolError) as excinfo:
asyncio.run(server.call_tool("search", {"predicates": ["not a predicate"]}))
assert not isinstance(excinfo.value, UnexpectedToolError)
assert "field=value" in str(excinfo.value)
def test_the_server_module_cannot_reach_a_write_command():
"""Structural: import the server in a clean interpreter and nothing under
`chemenu.commands` is loaded, so there is no `publish` to expose."""
result = subprocess.run(
[sys.executable, "-c",
"import sys, chemenu.mcp.server;"
"print([m for m in sys.modules if m.startswith('chemenu.commands')])"],
cwd=config._PACKAGE_ROOT / "tools", capture_output=True, text=True, check=True,
)
assert result.stdout.strip() == "[]"
def test_tracing_into_the_served_checkout_is_refused(corpus, monkeypatch):
"""The sync that keeps this checkout current is `git reset --hard`, which is
entitled to wipe `reports/`. A per-request trace written there is both lost
work and a silent way to dirty the tree the cache keys on."""
monkeypatch.setenv("WIKI_TRACE", "1")
monkeypatch.delenv("WIKI_TRACE_DIR", raising=False)
with pytest.raises(TraceWouldWriteIntoCorpus):
check_trace_destination(corpus)
monkeypatch.setenv("WIKI_TRACE_DIR", str(corpus / "reports" / "telemetry"))
with pytest.raises(TraceWouldWriteIntoCorpus):
check_trace_destination(corpus)
def test_tracing_outside_the_corpus_or_switched_off_is_accepted(corpus, tmp_path, monkeypatch):
monkeypatch.setenv("WIKI_TRACE", "0")
check_trace_destination(corpus)
monkeypatch.setenv("WIKI_TRACE", "1")
monkeypatch.setenv("WIKI_TRACE_DIR", str(tmp_path / "traces"))
check_trace_destination(corpus)
def test_only_the_two_chosen_transports_are_offered():
"""`sse` is reachable through the SDK and deliberately not offered: it is
the superseded remote transport, and building on it now only moves the
migration later."""
assert TRANSPORTS == ("stdio", "streamable-http")
with pytest.raises(ValueError, match="unknown transport"):
serve(transport="sse")
def test_bind_arguments_reach_only_the_transport_that_takes_them(monkeypatch):
"""`run_stdio_async` accepts no host or port; handing it one is a TypeError,
not a harmless no-op. And the loopback default has to be overridable, or a
server in a container behind a reverse proxy binds an interface the proxy
cannot reach."""
from chemenu.mcp import __main__ as entry
seen: dict = {}
monkeypatch.setattr(entry, "serve", lambda **kwargs: seen.update(kwargs))
entry.main([])
assert seen == {"transport": "stdio", "root": None}
seen.clear()
entry.main(["--transport", "streamable-http", "--host", "0.0.0.0", "--port", "9001"])
assert seen["transport"] == "streamable-http"
assert seen["host"] == "0.0.0.0" and seen["port"] == 9001
def test_a_refused_trace_destination_stops_the_process_with_a_message(monkeypatch, capsys):
from chemenu.mcp import __main__ as entry
monkeypatch.setenv("WIKI_TRACE", "1")
monkeypatch.delenv("WIKI_TRACE_DIR", raising=False)
assert entry.main([]) == 1
assert "WIKI_TRACE" in capsys.readouterr().err
+4 -4
View File
@@ -176,7 +176,7 @@ provenance: sourced
# Aurora # Aurora
Links to [[Nathan]] and again to [[Nathan]]. Links to [[Borealis]] and again to [[Borealis]].
""" """
@@ -214,19 +214,19 @@ def test_verify_is_clean_when_only_prose_moved(git_instance, capsys):
def test_verify_catches_a_dropped_link_against_history(git_instance, capsys): def test_verify_catches_a_dropped_link_against_history(git_instance, capsys):
page = git_instance / "kb" / "entities" / "Aurora.md" page = git_instance / "kb" / "entities" / "Aurora.md"
page.write_text(PAGE.replace(" and again to [[Nathan]]", ""), encoding="utf-8") page.write_text(PAGE.replace(" and again to [[Borealis]]", ""), encoding="utf-8")
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
migrate_cmd.verify_command( migrate_cmd.verify_command(
from_rev="HEAD", path=None, expect_body_change=False, json_out=True, fail_on_error=True from_rev="HEAD", path=None, expect_body_change=False, json_out=True, fail_on_error=True
) )
result = json.loads(capsys.readouterr().out) result = json.loads(capsys.readouterr().out)
assert result["findings"][0]["kind"] == "wikilinks" assert result["findings"][0]["kind"] == "wikilinks"
assert "'Nathan' 2->1" in result["findings"][0]["detail"] assert "'Borealis' 2->1" in result["findings"][0]["detail"]
def test_verify_scopes_to_a_subtree(git_instance, capsys): def test_verify_scopes_to_a_subtree(git_instance, capsys):
page = git_instance / "kb" / "entities" / "Aurora.md" page = git_instance / "kb" / "entities" / "Aurora.md"
page.write_text(PAGE.replace(" and again to [[Nathan]]", ""), encoding="utf-8") page.write_text(PAGE.replace(" and again to [[Borealis]]", ""), encoding="utf-8")
migrate_cmd.verify_command( migrate_cmd.verify_command(
from_rev="HEAD", from_rev="HEAD",
path=["kb/concepts"], path=["kb/concepts"],
+11 -8
View File
@@ -60,7 +60,7 @@ def test_new_entity_creates_page_with_expected_frontmatter(monkeypatch, kb_dir):
result = _invoke_new(monkeypatch, kb_dir, [ result = _invoke_new(monkeypatch, kb_dir, [
"new", "entity", "--name", "gateway.example.net", "new", "entity", "--name", "gateway.example.net",
"--set", "entity_type=system", "--set", "tags=gateway,firewall", "--set", "entity_type=system", "--set", "tags=gateway,firewall",
"--set", "related=Nathan", "--set", "confidence=0.9", "--set", "related=Borealis", "--set", "confidence=0.9",
"--set", "provenance=general", "--set", "provenance=general",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
@@ -70,7 +70,7 @@ def test_new_entity_creates_page_with_expected_frontmatter(monkeypatch, kb_dir):
assert fm["type"] == "types/entity.md" assert fm["type"] == "types/entity.md"
assert fm["entity_type"] == "system" assert fm["entity_type"] == "system"
assert fm["tags"] == ["gateway", "firewall"] assert fm["tags"] == ["gateway", "firewall"]
assert fm["related"] == ["Nathan"] assert fm["related"] == ["Borealis"]
assert fm["confidence"] == 0.9 assert fm["confidence"] == 0.9
assert "# gateway.example.net" in body assert "# gateway.example.net" in body
@@ -160,7 +160,10 @@ def _fixture_raw_file(monkeypatch, kb_dir, relative: str) -> None:
distribution does not have.""" distribution does not have."""
import chemenu.config as config import chemenu.config as config
from chemenu.tests.conftest import use_shipped_type_specs
monkeypatch.setattr(config, "ROOT", kb_dir.parent) monkeypatch.setattr(config, "ROOT", kb_dir.parent)
use_shipped_type_specs(monkeypatch)
path = kb_dir.parent / relative path = kb_dir.parent / relative
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("raw fixture content\n", encoding="utf-8") path.write_text("raw fixture content\n", encoding="utf-8")
@@ -172,18 +175,18 @@ def test_new_source_prefixes_title_and_prefills_related_entities(monkeypatch, kb
result = _invoke_new(monkeypatch, kb_dir, [ result = _invoke_new(monkeypatch, kb_dir, [
"new", "source", "--name", "gateway.example.net", "new", "source", "--name", "gateway.example.net",
"--set", "raw_files=raw/notes/gateway.example.net.md", "--set", "raw_files=raw/notes/gateway.example.net.md",
"--set", "entities=aurora,Nathan", "--set", "entities=aurora,Borealis",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
path = kb_dir / "sources/Source - gateway.example.net.md" path = kb_dir / "sources/Source - gateway.example.net.md"
assert path.exists() assert path.exists()
fm, body = read_page(path) fm, body = read_page(path)
assert fm["type"] == "types/source.md" assert fm["type"] == "types/source.md"
assert fm["entities"] == ["aurora", "Nathan"] assert fm["entities"] == ["aurora", "Borealis"]
assert fm["author"] == "Torben" # WIKI_AUTHOR override assert fm["author"] == "Torben" # WIKI_AUTHOR override
assert fm["source_type"] == "notes" # schema default assert fm["source_type"] == "notes" # schema default
assert "[[aurora]]" in body assert "[[aurora]]" in body
assert "[[Nathan]]" in body assert "[[Borealis]]" in body
def test_new_source_author_falls_back_to_git_config(monkeypatch, kb_dir): def test_new_source_author_falls_back_to_git_config(monkeypatch, kb_dir):
@@ -264,13 +267,13 @@ def test_new_concept_rejects_invalid_provenance(monkeypatch, kb_dir):
def test_new_comparison_renders_table_columns(monkeypatch, kb_dir): def test_new_comparison_renders_table_columns(monkeypatch, kb_dir):
result = _invoke_new(monkeypatch, kb_dir, [ result = _invoke_new(monkeypatch, kb_dir, [
"new", "comparison", "--name", "A vs B", "--set", "entities=aurora,Nathan", "new", "comparison", "--name", "A vs B", "--set", "entities=aurora,Borealis",
]) ])
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
fm, body = read_page(kb_dir / "comparisons/A vs B.md") fm, body = read_page(kb_dir / "comparisons/A vs B.md")
assert fm["entities"] == ["aurora", "Nathan"] assert fm["entities"] == ["aurora", "Borealis"]
# Table columns are rendered by the generic table_* template filters. # Table columns are rendered by the generic table_* template filters.
assert "[[aurora]] | [[Nathan]]" in body assert "[[aurora]] | [[Borealis]]" in body
def test_new_comparison_rejects_single_entity(monkeypatch, kb_dir): def test_new_comparison_rejects_single_entity(monkeypatch, kb_dir):
+45 -45
View File
@@ -78,49 +78,49 @@ def test_retarget_cite_ids_is_noop_without_a_footnotes_block():
def test_rename_refreshes_stale_slug_derived_cite_id(patched_wiki): def test_rename_refreshes_stale_slug_derived_cite_id(patched_wiki):
old_id = cite_id("Nathan") old_id = cite_id("Borealis")
write_page( write_page(
patched_wiki / "concepts/uses-nathan.md", patched_wiki / "concepts/uses-borealis.md",
{ {
"type": "types/concept.md", "concept_type": "protocol", "type": "types/concept.md", "concept_type": "protocol",
"tags": [], "created": "2026-07-25", "modified": "2026-07-25", "tags": [], "created": "2026-07-25", "modified": "2026-07-25",
"related": [], "sources": [], "confidence": 0.7, "related": [], "sources": [], "confidence": 0.7,
}, },
f"\n# uses-nathan\n\nRuns on it [^{old_id}].\n\n## Footnotes\n\n[^{old_id}]: [[Nathan]]\n", f"\n# uses-borealis\n\nRuns on it [^{old_id}].\n\n## Footnotes\n\n[^{old_id}]: [[Borealis]]\n",
) )
page_ops.rename_command(old="Nathan", new="nathan-ws", dry_run=False) page_ops.rename_command(old="Borealis", new="borealis-ws", dry_run=False)
_frontmatter, body = read_page(patched_wiki / "concepts/uses-nathan.md") _frontmatter, body = read_page(patched_wiki / "concepts/uses-borealis.md")
new_id = cite_id("nathan-ws") new_id = cite_id("borealis-ws")
assert f"[^{new_id}]: [[nathan-ws]]" in body assert f"[^{new_id}]: [[borealis-ws]]" in body
assert f"[^{old_id}]" not in body assert f"[^{old_id}]" not in body
def test_rename_updates_body_links_and_frontmatter(patched_wiki): def test_rename_updates_body_links_and_frontmatter(patched_wiki):
page_ops.rename_command(old="Nathan", new="nathan-ws", dry_run=False) page_ops.rename_command(old="Borealis", new="borealis-ws", dry_run=False)
assert (patched_wiki / "entities/systems/nathan-ws.md").exists() assert (patched_wiki / "entities/systems/borealis-ws.md").exists()
assert not (patched_wiki / "entities/systems/Nathan.md").exists() assert not (patched_wiki / "entities/systems/Borealis.md").exists()
frontmatter, body = read_page(patched_wiki / "entities/systems/aurora.md") frontmatter, body = read_page(patched_wiki / "entities/systems/aurora.md")
assert frontmatter["related"] == ["nathan-ws"] assert frontmatter["related"] == ["borealis-ws"]
assert "[[nathan-ws]]" in body assert "[[borealis-ws]]" in body
assert "[[Nathan]]" not in body assert "[[Borealis]]" not in body
def test_rename_rewrites_the_pages_own_h1(patched_wiki): def test_rename_rewrites_the_pages_own_h1(patched_wiki):
page_ops.rename_command(old="Nathan", new="nathan-ws", dry_run=False) page_ops.rename_command(old="Borealis", new="borealis-ws", dry_run=False)
_frontmatter, body = read_page(patched_wiki / "entities/systems/nathan-ws.md") _frontmatter, body = read_page(patched_wiki / "entities/systems/borealis-ws.md")
assert "# nathan-ws" in body assert "# borealis-ws" in body
assert "# Nathan\n" not in body assert "# Borealis\n" not in body
def test_rename_dry_run_writes_nothing(patched_wiki): def test_rename_dry_run_writes_nothing(patched_wiki):
page_ops.rename_command(old="Nathan", new="nathan-ws", dry_run=True) page_ops.rename_command(old="Borealis", new="borealis-ws", dry_run=True)
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
frontmatter, _body = read_page(patched_wiki / "entities/systems/aurora.md") frontmatter, _body = read_page(patched_wiki / "entities/systems/aurora.md")
assert frontmatter["related"] == ["Nathan"] assert frontmatter["related"] == ["Borealis"]
def test_rename_rejects_missing_page(patched_wiki): def test_rename_rejects_missing_page(patched_wiki):
@@ -136,15 +136,15 @@ def test_rename_repoints_references_to_an_existing_page(patched_wiki):
patched_wiki / "entities/tools/gdeploy.md", patched_wiki / "entities/tools/gdeploy.md",
{"type": "types/entity.md", "entity_type": "tool", "tags": [], {"type": "types/entity.md", "entity_type": "tool", "tags": [],
"created": "2026-07-25", "modified": "2026-07-25", "created": "2026-07-25", "modified": "2026-07-25",
"related": ["nathan"], "sources": [], "confidence": 0.8}, "related": ["borealis"], "sources": [], "confidence": 0.8},
"\n# gdeploy\n\n## See Also\n\n- [[nathan]]\n", "\n# gdeploy\n\n## See Also\n\n- [[borealis]]\n",
) )
page_ops.rename_command(old="nathan", new="Nathan", dry_run=False) page_ops.rename_command(old="borealis", new="Borealis", dry_run=False)
frontmatter, body = read_page(patched_wiki / "entities/tools/gdeploy.md") frontmatter, body = read_page(patched_wiki / "entities/tools/gdeploy.md")
assert frontmatter["related"] == ["Nathan"] assert frontmatter["related"] == ["Borealis"]
assert "[[Nathan]]" in body assert "[[Borealis]]" in body
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
def test_rename_reference_only_mode_requires_the_target_to_exist(patched_wiki): def test_rename_reference_only_mode_requires_the_target_to_exist(patched_wiki):
@@ -161,12 +161,12 @@ def test_rename_reference_only_mode_requires_the_target_to_exist(patched_wiki):
def test_rename_rejects_existing_target(patched_wiki): def test_rename_rejects_existing_target(patched_wiki):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
page_ops.rename_command(old="Nathan", new="aurora", dry_run=False) page_ops.rename_command(old="Borealis", new="aurora", dry_run=False)
def test_rename_rejects_identical_titles(patched_wiki): def test_rename_rejects_identical_titles(patched_wiki):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
page_ops.rename_command(old="Nathan", new="Nathan", dry_run=False) page_ops.rename_command(old="Borealis", new="Borealis", dry_run=False)
def test_rename_fixes_a_dangling_source_reference(patched_wiki): def test_rename_fixes_a_dangling_source_reference(patched_wiki):
@@ -201,24 +201,24 @@ def test_rename_stops_before_renaming_the_file_if_a_reference_write_fails(patche
monkeypatch.setattr(page_ops_module, "write_page", flaky_write_page) monkeypatch.setattr(page_ops_module, "write_page", flaky_write_page)
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
page_ops.rename_command(old="Nathan", new="nathan-ws", dry_run=False) page_ops.rename_command(old="Borealis", new="borealis-ws", dry_run=False)
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
assert not (patched_wiki / "entities/systems/nathan-ws.md").exists() assert not (patched_wiki / "entities/systems/borealis-ws.md").exists()
def test_rm_refuses_referenced_page_without_yes(patched_wiki): def test_rm_refuses_referenced_page_without_yes(patched_wiki):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
page_ops.rm_command(page_title="Nathan", yes=False, dry_run=False) page_ops.rm_command(page_title="Borealis", yes=False, dry_run=False)
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
def test_rm_deletes_and_delinks_with_yes(patched_wiki): def test_rm_deletes_and_delinks_with_yes(patched_wiki):
page_ops.rm_command(page_title="Nathan", yes=True, dry_run=False) page_ops.rm_command(page_title="Borealis", yes=True, dry_run=False)
assert not (patched_wiki / "entities/systems/Nathan.md").exists() assert not (patched_wiki / "entities/systems/Borealis.md").exists()
frontmatter, body = read_page(patched_wiki / "entities/systems/aurora.md") frontmatter, body = read_page(patched_wiki / "entities/systems/aurora.md")
assert frontmatter["related"] == [] assert frontmatter["related"] == []
assert "[[Nathan]]" not in body assert "[[Borealis]]" not in body
def test_rm_stops_before_deleting_the_file_if_a_delink_write_fails(patched_wiki, monkeypatch): def test_rm_stops_before_deleting_the_file_if_a_delink_write_fails(patched_wiki, monkeypatch):
@@ -234,9 +234,9 @@ def test_rm_stops_before_deleting_the_file_if_a_delink_write_fails(patched_wiki,
monkeypatch.setattr(page_ops_module, "write_page", flaky_write_page) monkeypatch.setattr(page_ops_module, "write_page", flaky_write_page)
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
page_ops.rm_command(page_title="Nathan", yes=True, dry_run=False) page_ops.rm_command(page_title="Borealis", yes=True, dry_run=False)
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
def test_rm_of_unreferenced_page_needs_no_confirmation(patched_wiki): def test_rm_of_unreferenced_page_needs_no_confirmation(patched_wiki):
@@ -259,20 +259,20 @@ def test_rm_leaves_prose_references_and_reports_them(patched_wiki, capsys):
{"type": "types/entity.md", "entity_type": "tool", "tags": [], {"type": "types/entity.md", "entity_type": "tool", "tags": [],
"created": "2026-07-25", "modified": "2026-07-25", "related": [], "created": "2026-07-25", "modified": "2026-07-25", "related": [],
"sources": [], "confidence": 0.8}, "sources": [], "confidence": 0.8},
"\n# gdeploy\n\n## Description\n\nRuns on [[Nathan]] nightly.\n", "\n# gdeploy\n\n## Description\n\nRuns on [[Borealis]] nightly.\n",
) )
page_ops.rm_command(page_title="Nathan", yes=True, dry_run=False) page_ops.rm_command(page_title="Borealis", yes=True, dry_run=False)
_frontmatter, body = read_page(patched_wiki / "entities/tools/gdeploy.md") _frontmatter, body = read_page(patched_wiki / "entities/tools/gdeploy.md")
assert "Runs on [[Nathan]] nightly." in body assert "Runs on [[Borealis]] nightly." in body
assert "remaining [[Nathan]] reference(s)" in capsys.readouterr().out assert "remaining [[Borealis]] reference(s)" in capsys.readouterr().out
def test_rm_dry_run_writes_nothing(patched_wiki): def test_rm_dry_run_writes_nothing(patched_wiki):
page_ops.rm_command(page_title="Nathan", yes=True, dry_run=True) page_ops.rm_command(page_title="Borealis", yes=True, dry_run=True)
assert (patched_wiki / "entities/systems/Nathan.md").exists() assert (patched_wiki / "entities/systems/Borealis.md").exists()
frontmatter, _body = read_page(patched_wiki / "entities/systems/aurora.md") frontmatter, _body = read_page(patched_wiki / "entities/systems/aurora.md")
assert frontmatter["related"] == ["Nathan"] assert frontmatter["related"] == ["Borealis"]
def test_inbound_pages_sees_frontmatter_only_references(patched_wiki): def test_inbound_pages_sees_frontmatter_only_references(patched_wiki):
+4 -4
View File
@@ -161,7 +161,7 @@ def test_the_block_is_re_emitted_last_so_the_layout_self_heals():
body = ( body = (
"# Page\n\nProse [^s-aurora].\n\n" "# Page\n\nProse [^s-aurora].\n\n"
+ render_cite_block(definitions) + render_cite_block(definitions)
+ "\n## Siehe auch\n\n- [[Nathan]]\n" + "\n## Siehe auch\n\n- [[Borealis]]\n"
) )
rebuilt = render_page_body(*split_cite_block(body)) rebuilt = render_page_body(*split_cite_block(body))
assert rebuilt.index("## Siehe auch") < rebuilt.index("[^s-aurora]:") assert rebuilt.index("## Siehe auch") < rebuilt.index("[^s-aurora]:")
@@ -173,7 +173,7 @@ def test_repeated_round_trips_are_stable():
body = ( body = (
"# Page\n\nProse [^s-aurora].\n\n" "# Page\n\nProse [^s-aurora].\n\n"
+ render_cite_block(definitions) + render_cite_block(definitions)
+ "\n## Siehe auch\n\n- [[Nathan]]\n" + "\n## Siehe auch\n\n- [[Borealis]]\n"
) )
once = render_page_body(*split_cite_block(body)) once = render_page_body(*split_cite_block(body))
twice = render_page_body(*split_cite_block(once)) twice = render_page_body(*split_cite_block(once))
@@ -200,7 +200,7 @@ def test_a_citation_used_in_a_rescued_section_still_resolves():
"# Page\n\nProse.\n\n" "# Page\n\nProse.\n\n"
"## Fußnoten\n\n" "## Fußnoten\n\n"
"[^s-aurora]: [[Source - Aurora]]\n\n" "[^s-aurora]: [[Source - Aurora]]\n\n"
"## Beziehungen\n\n- **belegt durch:** [[Nathan]] [^s-aurora]\n" "## Beziehungen\n\n- **belegt durch:** [[Borealis]] [^s-aurora]\n"
) )
assert extract_inline_cites(body) == {("Source - Aurora", None)} assert extract_inline_cites(body) == {("Source - Aurora", None)}
@@ -368,7 +368,7 @@ def test_sources_trace_by_raw_and_by_page(kb_dir, raw_dir, monkeypatch):
kb_dir / "entities/systems/aurora.md", kb_dir / "entities/systems/aurora.md",
{ {
"type": "entity", "entity_type": "system", "tags": ["server"], "type": "entity", "entity_type": "system", "tags": ["server"],
"created": "2026-07-31", "modified": "2026-07-31", "related": ["Nathan"], "created": "2026-07-31", "modified": "2026-07-31", "related": ["Borealis"],
"sources": ["Source - Aurora"], "confidence": 0.9, "sources": ["Source - Aurora"], "confidence": 0.9,
}, },
"\n# aurora\n\n## Description\n\nHosts things.\n", "\n# aurora\n\n## Description\n\nHosts things.\n",
+96 -5
View File
@@ -1,9 +1,18 @@
import subprocess
import time
from pathlib import Path from pathlib import Path
import pytest import pytest
from chemenu.commands.search import load_pages_by_path, render_table, run_search, sort_hits from chemenu.commands.search import (
from chemenu.search import filters load_pages_by_path,
render_table,
run_search,
sort_hits,
unreadable_pages,
)
from chemenu.search import filters, ripgrep
from chemenu.search.base import page_key
from chemenu.search.filters import PredicateError, parse_predicate from chemenu.search.filters import PredicateError, parse_predicate
from chemenu.search.fuse import reciprocal_rank_fusion from chemenu.search.fuse import reciprocal_rank_fusion
from chemenu.search.registry import UnknownBackend, resolve from chemenu.search.registry import UnknownBackend, resolve
@@ -74,7 +83,7 @@ def test_parse_predicate_rejects_malformed(raw):
def test_exact_match_on_frontmatter_field(search): def test_exact_match_on_frontmatter_field(search):
assert _titles(search(_q("entity_type=system"))) == ["aurora", "Nathan"] assert _titles(search(_q("entity_type=system"))) == ["aurora", "Borealis"]
def test_membership_on_list_field(search): def test_membership_on_list_field(search):
@@ -86,13 +95,13 @@ def test_substring_match_is_case_insensitive(search):
def test_numeric_comparison(search): def test_numeric_comparison(search):
assert _titles(search(_q("confidence>=0.9"))) == ["aurora", "Nathan"] assert _titles(search(_q("confidence>=0.9"))) == ["aurora", "Borealis"]
def test_date_comparison_handles_yaml_date_objects(search): def test_date_comparison_handles_yaml_date_objects(search):
"""PyYAML parses `modified: 2026-08-02` into a date, not a string - the """PyYAML parses `modified: 2026-08-02` into a date, not a string - the
comparison has to normalise it or it never matches.""" comparison has to normalise it or it never matches."""
assert _titles(search(_q("modified>=2026-08-01"))) == ["Nathan"] assert _titles(search(_q("modified>=2026-08-01"))) == ["Borealis"]
def test_exists_and_absent_are_complementary(search, pages): def test_exists_and_absent_are_complementary(search, pages):
@@ -238,3 +247,85 @@ def test_known_fields_includes_virtual_and_real(pages):
fields = filters.known_fields(pages) fields = filters.known_fields(pages)
assert {"title", "kind", "subtype", "collection"} <= fields assert {"title", "kind", "subtype", "collection"} <= fields
assert {"entity_type", "confidence", "tags"} <= fields assert {"entity_type", "confidence", "tags"} <= fields
# --- Read-path limits (Gitea #33) -------------------------------------------
def test_a_user_regex_never_reaches_pythons_backtracking_engine():
"""Regression for the ReDoS. `(\\w+\\s?)+$` against 114 characters of
ordinary page text does not terminate in eight seconds under `re`; the
ranking helper must not evaluate it as a pattern at all.
Asserted by time *and* by outcome: a bound alone would pass if the branch
came back with a cheaper engine, and the outcome alone would pass while the
call still hung on a different pattern."""
haystack = (
"Longhorn is the distributed block storage layer for Kubernetes that "
"this cluster runs, replicated across three nodes and backed up nightly"
)
query = SearchQuery(text=r"(\w+\s?)+$", regex=True)
started = time.perf_counter()
assert ripgrep._contains(haystack, query) is False
assert time.perf_counter() - started < 0.5
def test_a_mostly_literal_regex_still_earns_its_title_boost():
"""What the deleted branch cost, and what it did not: the common case of a
pattern that happens to be plain text keeps ranking as before."""
assert ripgrep._contains("Longhorn", SearchQuery(text="longhorn", regex=True)) is True
def test_ripgrep_is_called_with_a_timeout(monkeypatch, kb_dir, tmp_path):
seen = {}
def fake_run(argv, **kwargs):
seen.update(kwargs)
return subprocess.CompletedProcess(argv, 1, "", "")
monkeypatch.setattr(ripgrep.subprocess, "run", fake_run)
RipgrepBackend(kb_dir, tmp_path).search(SearchQuery(text="x"), {})
assert seen["timeout"] == ripgrep.RIPGREP_TIMEOUT_SECONDS
def test_a_hanging_ripgrep_is_reported_as_a_failure_not_a_hang(monkeypatch, kb_dir, tmp_path):
def fake_run(argv, **kwargs):
raise subprocess.TimeoutExpired(argv, kwargs["timeout"])
monkeypatch.setattr(ripgrep.subprocess, "run", fake_run)
with pytest.raises(ripgrep.RipgrepFailed) as excinfo:
RipgrepBackend(kb_dir, tmp_path).search(SearchQuery(text="x"), {})
assert "did not finish" in str(excinfo.value)
def test_a_page_with_broken_frontmatter_is_reported_not_lost(kb_dir, tmp_path):
"""It matches no positive predicate - including the low-confidence sweep
meant to find pages in exactly that state - so silence reads as 'did not
match'. The page has to be nameable."""
broken = kb_dir / "entities" / "Broken.md"
broken.write_text("---\ntype: [unclosed\n---\n\n# Broken\n", encoding="utf-8")
pages = load_pages_by_path(kb_dir, tmp_path)
key = page_key(broken, tmp_path)
assert pages[key].frontmatter == {}
assert filters.apply_predicates(pages, (parse_predicate("confidence<0.6"),)) .get(key) is None
reported = unreadable_pages(pages)
assert [entry["path"] for entry in reported] == [key]
assert "invalid YAML" in reported[0]["reason"]
def test_an_empty_frontmatter_block_is_not_reported_as_unreadable(kb_dir, tmp_path):
"""A page may legitimately carry an empty block - there are no fields to
lose, so there is nothing the caller was not told about. `lint` still has
an opinion about it; search does not."""
(kb_dir / "entities" / "Bare.md").write_text("---\n\n---\n\n# Bare\n", encoding="utf-8")
assert unreadable_pages(load_pages_by_path(kb_dir, tmp_path)) == []
def test_a_page_with_no_frontmatter_at_all_is_reported(kb_dir, tmp_path):
"""Unlike an empty block, this page has no `type:` either - it cannot match
a predicate, and nothing else would say so."""
(kb_dir / "entities" / "Naked.md").write_text("# Naked\n\nProse only.\n", encoding="utf-8")
reported = unreadable_pages(load_pages_by_path(kb_dir, tmp_path))
assert [entry["path"] for entry in reported] == ["kb/entities/Naked.md"]
+1 -1
View File
@@ -172,7 +172,7 @@ def test_page_reference_fields_are_refused_and_name_xref(touch_wiki, capsys):
frontmatter write would leave the other half stale.""" frontmatter write would leave the other half stale."""
for field in ("related", "sources", "entities", "concepts"): for field in ("related", "sources", "entities", "concepts"):
with pytest.raises(typer.Exit): with pytest.raises(typer.Exit):
_touch(page_title="aurora", set_fields=[f"{field}=Nathan"]) _touch(page_title="aurora", set_fields=[f"{field}=Borealis"])
assert "xref" in capsys.readouterr().out assert "xref" in capsys.readouterr().out
+9 -1
View File
@@ -26,11 +26,19 @@ class TypeResolver:
"""Resolves and validates type paths against type-spec files.""" """Resolves and validates type paths against type-spec files."""
def __init__(self, repo_root: Path = None): def __init__(self, repo_root: Path = None):
self.repo_root = repo_root or config.ROOT # None means "whatever the root currently resolves to" - see the
# `repo_root` property. Binding it here is what made the module-level
# `resolver` singleton answer about this checkout even when the caller
# had pointed everything else at another tree.
self._repo_root = Path(repo_root) if repo_root is not None else None
self.type_cache: Dict[str, Dict[str, Any]] = {} self.type_cache: Dict[str, Dict[str, Any]] = {}
self.schema_cache: Dict[str, Dict[str, Any]] = {} self.schema_cache: Dict[str, Dict[str, Any]] = {}
self.validator_cache: Dict[str, Any] = {} self.validator_cache: Dict[str, Any] = {}
@property
def repo_root(self) -> Path:
return self._repo_root if self._repo_root is not None else config.ROOT
def resolve_type_path(self, type_path: str, source_file: Path = None) -> Path: def resolve_type_path(self, type_path: str, source_file: Path = None) -> Path:
"""Resolve a type path to an absolute, validated path. """Resolve a type path to an absolute, validated path.
+90
View File
@@ -0,0 +1,90 @@
"""The type-spec surface, with no CLI attached.
Same split as `chemenu/search/service.py` and `chemenu/lint_core.py`: these two
functions produce the values, `commands/types_cmd.py` renders them and turns
`UnknownType` into an `ERROR` line and exit 1.
They return exactly the structures the CLI's `--json` forms print, because that
is the wire contract the MCP server (Gitea #19) is specified against - one
contract, not two, with the CLI as its executable specification.
"""
from __future__ import annotations
from typing import Any, Dict
from chemenu.errors import ValidationError
from chemenu.type_resolver import resolver
class UnknownType(ValidationError):
"""No type-spec by that name. Carries the available names, because an
unknown type is almost always a near-miss and the list is the answer."""
def __init__(self, name: str, available: list[str]):
self.name = name
self.available = available
super().__init__(f"No type-spec named '{name}'. Available: {', '.join(available)}")
def available_type_names() -> list[str]:
return sorted(fm.get("name") for _, fm in resolver.list_type_specs())
def list_types() -> list[Dict[str, Any]]:
"""Every type-spec under `types/`, in the shape `types list --json` prints."""
return [
{
"name": frontmatter.get("name"),
"type_path": type_path,
"schema": frontmatter.get("schema"),
"subtype_field": frontmatter.get("subtype_field"),
"root": frontmatter.get("root") or "kb",
"base_dir": frontmatter.get("base_dir"),
"description": frontmatter.get("description"),
}
for type_path, frontmatter in resolver.list_type_specs()
]
def describe_type(name: str) -> Dict[str, Any]:
"""One type's full contract, in the shape `types describe --json` prints.
Raises `UnknownType` rather than exiting: the caller may be a server, for
which an exit code is not an answer.
"""
type_path = resolver.find_type_by_name(name)
if type_path is None:
raise UnknownType(name, available_type_names())
type_spec = resolver.load_type_spec(type_path)
frontmatter = type_spec["frontmatter"]
schema = resolver.get_schema(type_path)
fields: list[Dict[str, Any]] = []
if schema is not None:
required = set(schema.get("required", []))
for field_name, field_schema in schema.get("properties", {}).items():
fields.append({
"field": field_name,
"required": field_name in required,
"type": field_schema.get("type"),
"enum": field_schema.get("enum"),
})
return {
"name": frontmatter.get("name"),
"type_path": type_path,
"description": frontmatter.get("description"),
"schema": frontmatter.get("schema"),
"subtype_field": frontmatter.get("subtype_field"),
# `root` was previously read straight from the frontmatter by the
# renderer and left out of the JSON payload, so `describe --json` could
# not tell you where a type's pages go while `list --json` could -
# `types/instruction.md` declares `root: repo`. Carried here, in the
# same shape `list` uses.
"root": frontmatter.get("root") or "kb",
"base_dir": frontmatter.get("base_dir"),
"title_prefix": frontmatter.get("title_prefix"),
"fields": fields,
"body": type_spec["body"].strip(),
}
+13
View File
@@ -0,0 +1,13 @@
# The MCP read server's dependency, deliberately not in requirements.txt.
#
# `requirements.txt` describes what an *instance* needs to run `wikitool`, and
# it is what `dist export` ships. The server is an optional second consumer:
# an instance that only ever uses the CLI should not be made to install
# pydantic, starlette, uvicorn and cryptography to do it.
#
# tools/.venv/bin/pip install -r tools/requirements-mcp.txt
# WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp --transport stdio
#
# The tests under `chemenu/tests/test_mcp_server.py` skip without it; CI
# installs it, so they do run.
mcp>=2.0