Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 torben
|
31662dc3ff | ||
|
torben
|
7fbb9a99df | ||
|
torben
|
778764de4d | ||
|
torben
|
f7597b209c | ||
|
torben
|
83018fcc7d | ||
|
torben
|
576df2cddd | ||
|
torben
|
d1cf2e0327 | ||
|
torben
|
df7ea93060 | ||
|
torben
|
00c2cf6ffe |
@@ -101,6 +101,10 @@ jobs:
|
||||
# instance does not measure this suite. Installed beside pytest for
|
||||
# the same reason pytest itself is.
|
||||
tools/.venv/bin/pip install --quiet pytest pytest-cov
|
||||
# The MCP server's dependency is optional for an instance but not for
|
||||
# CI: its tests skip without it, and a skipped golden test is exactly
|
||||
# how the server's output and the CLI's would drift apart unnoticed.
|
||||
tools/.venv/bin/pip install --quiet -r tools/requirements-mcp.txt
|
||||
|
||||
- name: Tests
|
||||
# Not run with WIKI_TRACE=0: two telemetry tests assert that a trace is
|
||||
|
||||
+323
@@ -20,6 +20,329 @@ their date-only headings.
|
||||
|
||||
---
|
||||
|
||||
## 2.5.0 - 2026-09-02 - Versionsstelle: Kompatibilitaet statt Inhaltsmigration, Breaking-Change-Vermerk erzwungen
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Die Doku des Stacks führte für die Wahl der Versionsstelle zwei Fragen zusammen, die nicht
|
||||
dieselbe sind: *muss der Korpus migriert werden?* und *ist die neue Version ein
|
||||
Drop-in-Ersatz?* An allen drei Stellen, die einer Session vor dem Bump begegnen - die Tabelle
|
||||
in `stack-dev`, der Modul-Docstring in `version.py`, `INSTALL.md` § Version und Updates -
|
||||
stand nur die erste. Nur `version bump --help` formulierte es richtig, also ausgerechnet die
|
||||
Stelle, die man erst liest, wenn die Entscheidung gefallen ist. Bei `2.0.0` hat genau diese
|
||||
Lücke zu `1.9.0` geführt (Gitea #26; der Eintrag zu `2.0.0` trägt den Fall im Volltext).
|
||||
|
||||
**Die Regel, jetzt an einer Stelle.** Neu ist
|
||||
`instructions/dev/version-parts.md`: der Drop-in-Test in zwei Hälften - vorwärts (kopieren,
|
||||
fertig, keine Handarbeit durch Nutzer oder Migrationsskript) und rückwärts (die alte Version
|
||||
lässt sich noch zurückinstallieren) -, der Katalog der Brüche, die die Grenze bei völlig
|
||||
unangetastetem `kb/` überschreiten (Update-Pfad, Artefaktname, Import-Name, Flags, Envvars,
|
||||
Shape maschinengelesener Dateien), und der Fall `2.0.0` als Fallbeispiel. `instructions/dev/`,
|
||||
also nicht in einer ausgelieferten Instanz - die Datei entscheidet, wer den Stack entwickelt.
|
||||
Die ausgelieferten Artefakte tragen deshalb die Kurzform der Regel, aber keinen Verweis auf
|
||||
sie: `instructions verify` meldet einen solchen Verweis, weil er nach `dist export` ins Leere
|
||||
zeigen würde.
|
||||
|
||||
**Breaking Changes sind teuer, also genehmigungspflichtig.** `stack-dev` hat einen
|
||||
Entscheidungspunkt bekommen: stellt sich eine Änderung als nicht drop-in heraus, wird nicht
|
||||
aus eigener Initiative über die Grenze gebumpt. Vorgelegt werden der konkrete Bruch, was jede
|
||||
bestehende Instanz von Hand tun muss, und die Alternativen - Bruch per Shim vermeiden,
|
||||
aufschieben und mit dem nächsten bündeln, oder in einen kompatiblen und einen brechenden Teil
|
||||
mit Deprecation-Fenster spalten -, dazu eine Empfehlung. Dann Freigabe abwarten.
|
||||
|
||||
**`--breaking` ist neu und bei jedem Grenzübertritt Pflicht.** Es schreibt eine
|
||||
`**Breaking Change:**`-Zeile in den Eintrag, vor die Migrationszeile: das ist, was der Operator
|
||||
einer bestehenden Instanz beim Lesen der Release-Notes zuerst braucht. `docs verify` prüft den
|
||||
neuesten grenzüberschreitenden Eintrag darauf, analog zu `**Migration:** none required` und aus
|
||||
demselben Grund - eine Prosa-Regel driftet, ein Marker nicht. Auf einem kompatiblen Bump wird
|
||||
`--breaking` verweigert, damit es keine Gewohnheit wird. Beide Prüfungen bleiben getrennt:
|
||||
`check_migration_for_boundary` fragt, ob der Inhalt sich bewegt, `check_breaking_change_for_boundary`,
|
||||
ob dem Operator gesagt wurde, dass der Tausch keiner zum Drüberkopieren ist. Ein Rebranding
|
||||
besteht die erste und fällt durch die zweite.
|
||||
|
||||
**Was ausdrücklich nicht passiert:** `docs verify` prüft weiterhin nicht, *ob die Stelle
|
||||
richtig gewählt* wurde. Kein Validator unterscheidet ein umbenanntes Flag von einem neuen; das
|
||||
bleibt Urteil, und dieser Eintrag existiert, weil das Urteil eine Anleitung braucht, keinen
|
||||
Automaten.
|
||||
|
||||
Berührt: `instructions/dev/version-parts.md` (neu), `instructions/dev/stack-dev/SKILL.md`,
|
||||
`tools/chemenu/version.py`, `tools/chemenu/commands/version_cmd.py`,
|
||||
`tools/chemenu/commands/docs_verify.py`, `tools/CONTRACT.md`, `INSTALL.md`. Sechs neue Tests.
|
||||
|
||||
---
|
||||
|
||||
## 2.4.1 - 2026-09-02 - INSTALL-MCP.md: Menschendoku fuer den Leseserver, und sie wird mit ausgeliefert
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Nachtrag zu 2.4.0. Der Server war ausgeliefert und für Menschen nur über `tools/CONTRACT.md` und
|
||||
eine Instruktion beschrieben — beide für Agenten geschrieben. `INSTALL-MCP.md` schließt das,
|
||||
neben `INSTALL.md` und in derselben Form: Voraussetzungen, sechs Schritte, Verifikation,
|
||||
Troubleshooting.
|
||||
|
||||
**Was drinsteht, das sonst nirgends zusammen stand:** die Client-Konfiguration als JSON-Block;
|
||||
dass `WIKI_TRACE=0` nicht optional ist und wie die Startverweigerung aussieht; dass
|
||||
`--host 0.0.0.0` bewusst nicht der Default ist; und der Verweis auf die
|
||||
Traefik-ForwardAuth-Middleware unter <https://gitea.nehmer.net/torben/gitea-mcp-forward-auth>
|
||||
samt dem, was sie erwartet (Bearer-Token gegen SHA-256-Hashes, `AUTH_PROXY_TOKEN_HASHES_DIR`).
|
||||
Der Verifikationsabschnitt ist ein lauffähiges Skript — wörtlich so ausgeführt, bevor es hier
|
||||
landete.
|
||||
|
||||
**`dist export` liefert die Datei mit.** `ROOT_FILES` ist eine Allowlist, also hätte sie sonst
|
||||
keine Instanz erreicht: ein ausgelieferter Server, dessen Bedienungsanleitung im Ursprungs-Repo
|
||||
zurückbleibt, ist einer, den der Empfänger durch Quelltextlesen findet. `INSTALL.md` bekommt
|
||||
zusätzlich `CHEMENU_ROOT` und `WIKI_TRACE`/`WIKI_TRACE_DIR` in seine Konfigurationstabelle —
|
||||
beide waren seit 2.3.0 bzw. seit jeher wirksam und dort nicht aufgeführt.
|
||||
|
||||
**Dateien:** `INSTALL-MCP.md` (neu), `INSTALL.md`, `README.md`,
|
||||
`tools/chemenu/commands/dist_cmd.py`.
|
||||
|
||||
---
|
||||
|
||||
## 2.4.0 - 2026-09-02 - MCP-Leseserver: zweiter Konsument auf demselben Kern
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Letzter Schritt der Sequenz aus #36, inhaltlich Issue #19. `chemenu` bekommt einen zweiten
|
||||
Konsumenten: `search`, `types`, `describe_type`, `lint` und `status` über MCP. Kein CLI mit
|
||||
angeschraubter Netzwerkschnittstelle — CLI und Server sind zwei Adapter auf dem Kern, den 2.3.0
|
||||
freigelegt hat.
|
||||
|
||||
**`tools/chemenu/mcp/`**, im Repo statt als eigenes Artefakt. Der Golden-Test, der die
|
||||
Serverantworten gegen die `--json`-Formen der CLI hält, läuft nur mit beiden Seiten in einer
|
||||
Testsuite; getrennt würde aus einem Contract eine Versionsabsprache. Der Test ruft `wikitool` als
|
||||
Subprozess gegen denselben Baum auf, über `CHEMENU_ROOT` — womit er nebenbei die Root-Auflösung
|
||||
von außen mitprüft.
|
||||
|
||||
**Zwei Transports.** `stdio` zum Entwickeln und Testen ohne Netz, `streamable-http` für die
|
||||
Auslieferung — der einzige, vor den sich die Authentifizierungs-Middleware überhaupt setzen kann,
|
||||
weil sie ein HTTP-Reverse-Proxy ist. `sse` ist über das SDK erreichbar und wird bewusst nicht
|
||||
angeboten: der abgelöste Remote-Transport, jetzt darauf zu bauen verschiebt den Wechsel nur.
|
||||
`--host`/`--port` gibt es, weil der Default auf Loopback bindet und ein Container hinter einem
|
||||
Proxy eine Adresse braucht, die der Proxy erreicht — eine Eigenschaft der Software, nicht einer
|
||||
Installation. Beide Transports sind gegen den echten Korpus gegengeprüft.
|
||||
|
||||
**Kein Schreibpfad, strukturell.** Weder der Server noch `chemenu.api` importiert irgendetwas
|
||||
unter `chemenu.commands`, also existieren `new`, `touch`, `xref`, `cite`, `publish`, `migrate`
|
||||
und `version bump` in dieser Reichweite gar nicht, statt aus einer Liste gefiltert zu werden. Ein
|
||||
Test importiert das Servermodul in einem frischen Interpreter und sieht in `sys.modules` nach;
|
||||
ein zweiter ruft alle fünf Tools auf und vergleicht den Dateibaum, `HEAD` und
|
||||
`git status --porcelain` vorher/nachher.
|
||||
|
||||
**Jede Antwort trägt ihren Commit.** `commit` und `as_of` in jedem Payload; `null` heißt, der
|
||||
bediente Baum hat uncommittete Änderungen und die Antwort entspricht keiner Revision. Der Stempel
|
||||
ist die Revision, aus der die Seiten *tatsächlich* gelesen wurden — zwischen Laden und Stempeln
|
||||
kann der Baum sich bewegen, deshalb reicht der Ladepfad seine Revision durch, statt noch einmal
|
||||
zu fragen. Das war beim ersten Durchlauf falsch: `types`/`lint`/`status` lasen die zuletzt
|
||||
*gecachte* Revision und stempelten `null`, obwohl der Baum sauber war.
|
||||
|
||||
**Telemetrie in den bedienten Baum wird beim Start verweigert**, nicht stillschweigend
|
||||
umgeleitet. Tracing ist per Default an und schreibt nach `reports/telemetry/` im Repo — genau das
|
||||
Verzeichnis, das der Sync per `git reset --hard` wegräumen darf. `WIKI_TRACE=0` oder
|
||||
`WIKI_TRACE_DIR` außerhalb des Korpus. Heute schreibt auf diesem Pfad nichts (der Emitter hängt an
|
||||
`cli.main()` und den Gates), die Sperre ist gegen später.
|
||||
|
||||
**Fehler an der Protokollgrenze.** Ein `ChemenuError` wird zum `ToolError` des SDK — eine
|
||||
absichtliche Ablehnung, deren Text den Aufrufer erreicht. Alles andere bleibt ein Absturz, dessen
|
||||
Text auf dem Server bleibt. Ein kaputtes Prädikat ist das Argument des Aufrufers, also muss die
|
||||
Zeile mitreisen, die sagt, was stattdessen zu schreiben ist.
|
||||
|
||||
**Bewusst nicht enthalten:** Authentifizierung und Rate Limiting (Middleware vor dem Prozess),
|
||||
Deployment (private Infrastruktur), der Iteration Budget Gate — er begrenzt eine Agenten-Session
|
||||
und nicht einen Nutzer, weshalb Retrieval von ihm befreit ist; ihn hier als Rate Limiter zu
|
||||
benutzen würde ihn dazu verwässern.
|
||||
|
||||
**Die Abhängigkeit ist optional** (`tools/requirements-mcp.txt`): eine Instanz, die nur die CLI
|
||||
benutzt, soll dafür nicht pydantic, starlette, uvicorn und cryptography installieren müssen. CI
|
||||
installiert sie, denn ein übersprungener Golden-Test ist genau der Weg, auf dem Server und CLI
|
||||
unbemerkt auseinanderlaufen.
|
||||
|
||||
Betrieb und Sync-Mechanismus: [instructions/mcp-read-server.md](instructions/mcp-read-server.md).
|
||||
Polling (`git fetch && git reset --hard`) statt Webhook — kein eingehender Endpunkt, keine
|
||||
Signaturprüfung. `reset --hard` ist dort tragend und keine Bequemlichkeit: ein abgedrifteter Baum
|
||||
antwortet zwar richtig, parst aber bei jeder Anfrage neu und stempelt jede Antwort mit `null`.
|
||||
|
||||
**Dateien:** `chemenu/mcp/` (neu: `server.py`, `__main__.py`), `chemenu/api.py`,
|
||||
`tools/requirements-mcp.txt` (neu), `instructions/mcp-read-server.md` (neu), `tools/CONTRACT.md`,
|
||||
`tools/README.md`, `.gitea/workflows/ci.yml`, `tests/test_mcp_server.py` (neu).
|
||||
|
||||
---
|
||||
|
||||
## 2.3.0 - 2026-09-02 - Bibliotheksgrenze: chemenu laesst sich auf einen Korpus zeigen
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Dritter Schritt der Sequenz aus #36, inhaltlich Issue #31. Der Schritt, der `chemenu` von einem
|
||||
Skript, das in seinem eigenen Verzeichnis lebt, zu einer Bibliothek macht, auf die man einen
|
||||
Korpus *zeigen* kann. Rückwärtskompatibel: ohne gesetzte Variable verhält sich `tools/wikitool`
|
||||
unverändert.
|
||||
|
||||
**Root-Auflösung nach Präzedenz statt nach Dateilage.** `config.resolve_root()`: expliziter
|
||||
Parameter → `$CHEMENU_ROOT` → Walk-up wie bisher. Der Walk-up bleibt Default. Wichtiger als die
|
||||
neue Fähigkeit ist die beseitigte Fehlerklasse: `ROOT` und alles darunter waren
|
||||
Modulkonstanten, also **zur Importzeit gebunden**, und
|
||||
`monkeypatch.setattr(config, "ROOT", ziel)` zeigte `ROOT` um, aber nicht `KB_DIR`/`RAW_DIR`. Wer
|
||||
sich darauf verließ, baute etwas, das scheinbar auf einem Zielbaum arbeitete und in Wahrheit aus
|
||||
dem Entwickler-Checkout antwortete. Die abgeleiteten Pfade werden jetzt bei jedem Zugriff
|
||||
aufgelöst (PEP 562 `__getattr__`) und folgen `ROOT` — der halb-umgezeigte Zustand ist nicht mehr
|
||||
konstruierbar. `CHEMENU_ROOT` ist in `_WIKITOOL_ENV` registriert, #23-konform.
|
||||
|
||||
**`config.reset()` und `config.rooted()`.** `reset()` nimmt Zuweisungen auf die verwalteten
|
||||
Pfadnamen zurück; die Testsuite ruft es zwischen Tests, und das ist dort nicht optional:
|
||||
`monkeypatch` merkt sich den alten Wert, indem es ihn *liest* — also auflöst — und schreibt ihn
|
||||
beim Aufräumen als echtes Attribut zurück. Genau die stale Bindung, die der Umbau unmöglich
|
||||
machen sollte, vom Cleanup wieder aufgebaut. `rooted(root)` setzt den Root für die Dauer eines
|
||||
Blocks, prozessweit und damit nicht thread-sicher — der Aufrufer hält das Lock, dieselbe
|
||||
Disziplin wie beim Korpus-Cache. Nötig, weil nicht alles einen Root als Argument nimmt: der
|
||||
modulglobale `TypeResolver` muss `types/` finden, und ohne ihn läse ein fremder Korpus mit den
|
||||
Type-Specs *dieses* Checkouts.
|
||||
|
||||
**Die Naht ist gezogen.** `run_search`/`run_lint`/`types` lagen in Modulen, die `typer` auf
|
||||
Modulebene importieren und über `_util` auch `rich` — wer sie in-process aufrief, zog den
|
||||
kompletten CLI-Kopf mit. Der reine Kern liegt jetzt in `search/service.py`, `lint_core.py` und
|
||||
`types_core.py`; `commands/` sind die Terminal-Adapter darüber und re-exportieren die Namen, damit
|
||||
kein bestehender Import bricht. Ein Test importiert `chemenu.api` in einem frischen Interpreter
|
||||
und prüft, dass weder `chemenu.commands.*` noch `typer`/`rich`/`click` geladen werden.
|
||||
|
||||
**`chemenu.api.Corpus` als In-Process-Einstieg.** Nimmt einen Root, liefert exakt die
|
||||
`--json`-Formen der CLI zurück — ein Wire-Contract statt zwei — und stempelt jede Antwort mit dem
|
||||
Commit-SHA und einem Zeitstempel (`commit`, `as_of`), so dass aus einer stillen veralteten Antwort
|
||||
eine sichtbare wird. `search`/`lint`/`types`/`describe_type`/`status`; `status` ist bewusst
|
||||
serverseitig **komponiert** und kein Wrapper, weil es kein `wikitool status` gibt. Lesend
|
||||
strukturell: nichts unter `chemenu.commands` wird importiert, die Schreibfunktionen existieren in
|
||||
dieser Oberfläche also gar nicht, statt gefiltert zu werden. Das ist die Grenze, auf der #19
|
||||
aufsetzt.
|
||||
|
||||
**Exceptions statt Exit-Codes an der Grenze.** `chemenu/errors.py`: `ChemenuError` mit
|
||||
`ValidationError` (Eingabe abgelehnt) und `BackendError` (Abhängigkeit fehlt oder scheitert).
|
||||
`PredicateError`, `FrontmatterError`, `UnknownBackend` und die beiden `Ripgrep*` hängen jetzt
|
||||
darunter; `ValidationError` erbt zusätzlich von `ValueError`, weil `PredicateError` vorher eines
|
||||
war und Aufrufer es so fangen. Das CLI-Verhalten ist unverändert: `fail()` → `ERROR`-Zeile,
|
||||
Exit 1, Budget-Refund.
|
||||
|
||||
**`resolve()` reicht den Root an das Backend durch.** Vorher konnte ein Aufrufer `run_search` einen
|
||||
Korpus übergeben, während `RipgrepBackend` weiter `config.KB_DIR` durchlief — die Anfrage aus dem
|
||||
einen Baum beantwortet, die Seiten aus dem anderen gelesen, ohne dass irgendetwas das gesagt
|
||||
hätte.
|
||||
|
||||
**Zwei Abhängigkeiten, die durch Zufall hielten, stehen jetzt da.** `TypeResolver.repo_root`
|
||||
folgt `ROOT`, statt beim Import zu binden — womit Fixtures, die `ROOT` auf einen tmp-Baum zeigen,
|
||||
die mitgelieferten Type-Specs ausdrücklich benennen müssen (`use_shipped_type_specs`). Dieselbe
|
||||
Form wie das Loch, für das `raw_dir` geschrieben wurde, eine Ebene tiefer. Und
|
||||
`types describe --json` trug `root:` nicht im Payload, obwohl `types list --json` es tut:
|
||||
`types/instruction.md` deklariert `root: repo`, die Renderer-Zeile las es direkt aus dem
|
||||
Frontmatter. Jetzt im Payload, in derselben Form wie bei `list`.
|
||||
|
||||
**Dateien:** `config.py`, `errors.py` (neu), `api.py` (neu), `search/service.py` (neu),
|
||||
`lint_core.py` (neu), `types_core.py` (neu), `search/registry.py`, `type_resolver.py`,
|
||||
`commands/search.py`, `commands/lint.py`, `commands/types_cmd.py`, `tools/CONTRACT.md`, dazu
|
||||
`tests/conftest.py`, `tests/test_api.py` (neu), `tests/test_new_page.py`,
|
||||
`tests/test_instructions_cmd.py`.
|
||||
|
||||
---
|
||||
|
||||
## 2.2.4 - 2026-09-02 - Haertung des Lesepfads: ReDoS, Subprozess-Timeout, YAML-Alias-Budget, Korpus-Cache
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Zweiter Schritt der Sequenz aus #36, inhaltlich Issue #33. Limits vor der Exposition: der
|
||||
Lesepfad bekommt mit dem MCP-Server (#19) einen Konsumenten, der nicht der Operator ist. Alle
|
||||
sechs Befunde waren reproduziert, nicht vermutet; jeder hat jetzt eine Regression.
|
||||
|
||||
**ReDoS über `--regex` beseitigt.** `search/ripgrep.py:_contains` gab nutzergesteuerten Regex an
|
||||
Pythons Backtracking-Engine: `(\w+\s?)+$` gegen 114 Zeichen gewöhnlichen Seiteninhalts terminiert
|
||||
nicht in acht Sekunden, ein deterministisch scheiterndes Muster braucht 0,2 ms — der Unterschied
|
||||
ist das Muster, nicht der Heuhaufen. `build_hit` ruft die Funktion je Treffer zweimal auf, und
|
||||
`\w` matcht jede Seite: eine Anfrage kaufte zwei unbegrenzte Suchen pro Korpusseite. Der Zweig ist
|
||||
**gelöscht**, nicht begrenzt. `rg` hat das Muster mit einer linearen Engine längst angewendet, wenn
|
||||
die Funktion überhaupt läuft; verloren geht nur der zusätzliche Titel-/Summary-Bonus für
|
||||
nicht-literale Muster, und Summary wie H1 sind selbst Zeilen in der Datei, die `rg` zählt.
|
||||
|
||||
**Subprozess-Timeout.** `rg` wird nach 30 s abgeräumt und über den vorhandenen
|
||||
`RipgrepFailed`-Pfad gemeldet. Kein Performance-Budget — eine Fixed-String-Suche kostet hier 6 ms —
|
||||
sondern ein Hänger-Abbruch, damit ein Aufruf als Fehler endet statt den Aufrufer offenzuhalten,
|
||||
während seine Ausgabe in den Heap puffert.
|
||||
|
||||
**YAML-Anchors und -Aliases im Frontmatter werden verweigert, nicht budgetiert.** Gemessen:
|
||||
267 Byte werden in 0,2 ms zu 672.603 Knoten beim Traversal, Wachstum 9ⁿ bei konstanter Parse-Zeit
|
||||
— ein Größenlimit fasst das nicht an, weil die Eingabe klein bleibt. Die Prüfung läuft auf dem
|
||||
*Event*-Strom (`yaml.parse`), der nichts auflöst, kostet also O(Text) und löst nie aus, wonach sie
|
||||
fragt; `*` ist in jedem Alias-Knoten notwendig, seine Abwesenheit beweist Abwesenheit ohne jeden
|
||||
Parse — der Weg, den jede echte Seite nimmt. Dazu ein Größenlimit von 64 KiB und ein Abfangen von
|
||||
`RecursionError` (PyYAML komponiert rekursiv, tiefe Verschachtelung ist kein `YAMLError`). Heute
|
||||
nicht erreichbar, weil `kb/` der Operator committet; erreichbar mit der Ingest-Queue (#32).
|
||||
|
||||
**`CSafeLoader` statt `SafeLoader`, mit Fallback.** Gemessen über diesen Korpus (176 Seiten,
|
||||
best of 5): **265 ms → 54 ms**. Kein Mikro-Tuning — der Korpus-Parse war der größte Einzelposten
|
||||
eines `search`-Aufrufs und skaliert linear mit der Korpusgröße. End-to-end fällt ein
|
||||
`wikitool search` damit von 593 ms auf **347 ms**; die verbleibenden 262 ms sind Modulimport und
|
||||
entfallen erst im residenten Prozess (#19).
|
||||
|
||||
**Stiller Frontmatter-Verlust wird gemeldet.** Entschieden: der Lesepfad *nennt* die Seite, statt
|
||||
sie zu schlucken. Kaputtes YAML wird weiterhin zu `{}` — Massenoperationen dürfen an einer Seite
|
||||
nicht scheitern —, aber der Grund wird mitgeführt (`Page.frontmatter_error`) und ausgegeben:
|
||||
`search --json` trägt immer eine `unreadable`-Liste aus `{path, reason}`, die Tabellenform
|
||||
schreibt dieselben Zeilen nach stderr. Das war nötig, weil so eine Seite weder `confidence` noch
|
||||
`kind` hat und damit aus jedem positiven `--field`-Prädikat fällt — ausgerechnet aus der
|
||||
Low-Confidence-Suche, die Seiten in genau diesem Zustand finden soll — und dabei aussieht wie eine
|
||||
Seite, die nicht gematcht hat. Für Frontmatter, das diese Instanz nicht selbst geschrieben hat,
|
||||
steht `read_page_strict()` bereit: die Quarantäne aus #32 muss strikt lesen, wo ein
|
||||
unlesbares Dokument das Dokument stoppen und nicht leeren soll.
|
||||
|
||||
**Ein Parser statt zwei.** `read_page()` und `frontmatter_error()` liefen bisher getrennt durch
|
||||
`safe_load` — so konnte der permissive Weg zu `{}` degradieren aus einem Grund, den der strikte
|
||||
Weg anders beschrieb, und jeder Aufrufer, der beide Antworten wollte, las die Datei zweimal.
|
||||
Beide gehen jetzt durch `_load_frontmatter()`; ein Test hält sie gegeneinander.
|
||||
|
||||
**Korpus-Cache am Commit-SHA** (`chemenu/corpus_cache.py`), als Objekt, das ein Aufrufer *hält* —
|
||||
kein Modul-Dict, das sich hinter allen einschaltet. Die CLI hält keins und verhält sich unverändert
|
||||
(ein Aufruf pro Prozess, nichts wiederzuverwenden); der residente Prozess aus #19 hält eins.
|
||||
Entscheidend ist nicht die Geschwindigkeit, sondern dass nichts veraltet: **ein schmutziger
|
||||
Arbeitsbaum wird nie gecacht**, sonst bekäme eine Sitzung, die eine Seite schreibt und danach
|
||||
sucht, die Analyse von vor dem Schreiben — bei unverändertem SHA. Kann git nicht antworten, gilt
|
||||
der Baum als schmutzig. Derselbe SHA ist der Antwort-Stempel aus #19, per Konstruktion also die
|
||||
Revision, aus der die Antwort auch wirklich berechnet wurde.
|
||||
|
||||
**Erhalten geblieben** ist die Eigenschaft, die kein Befund war: kein `shell=True`,
|
||||
`--fixed-strings` als Default, `--`-Terminator. Der Modul-Docstring führt sie jetzt als drei
|
||||
tragende Zusagen statt zwei — die dritte ist, dass nutzergesteuerte Muster ausschließlich `rg`
|
||||
sieht.
|
||||
|
||||
**Dateien:** `frontmatter_io.py`, `search/ripgrep.py`, `commands/search.py`, `page.py`,
|
||||
`corpus_cache.py` (neu), `tools/CONTRACT.md`, dazu `tests/test_frontmatter_io.py`,
|
||||
`tests/test_search.py`, `tests/test_corpus_cache.py` (neu).
|
||||
|
||||
---
|
||||
|
||||
## 2.2.3 - 2026-09-02 - Publish-Remote-Gate in diesem Checkout scharf, doctor benennt den Zustand
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
Erster Schritt der Sequenz aus Issue #36 (Weg zum MCP-Leseserver), inhaltlich Issue #34. Das
|
||||
Gate war seit 2.2.0 implementiert und in diesem Checkout **inert**: `.wikitool-remotes.json`
|
||||
fehlte, und eine fehlende Datei heißt unbeschränkt. `ENVIRONMENT.md` beschrieb `origin`
|
||||
gleichzeitig als einziges Publish-Ziel, `AGENTS.md` führt das Gate unter den drei in Code
|
||||
erzwungenen Grenzen. Eine dokumentierte Sicherung, die nicht greift, ist schlechter als eine
|
||||
fehlende — sie erzeugt genau das Vertrauen, das sie nicht verdient.
|
||||
|
||||
**Die Datei ist angelegt** (gitignored, per Checkout, reist nicht mit) und gegengeprüft: ein
|
||||
`publish --remote` auf ein nicht gelistetes Ziel verweigert mit Exit 42, bevor irgendetwas
|
||||
gestaged wird, und der Arbeitsbaum bleibt unberührt. Damit steht die Sicherung **vor** dem Klonen
|
||||
der privaten Instanz (#30) — nachträglich gesetzt ließe sie genau das Fenster offen, das sie
|
||||
schließt.
|
||||
|
||||
**`doctor` sagt jetzt, ob das Gate scharf ist, nicht nur ob die Datei da ist.** Alle drei
|
||||
Zustände beginnen mit `Gate armed:` bzw. `Gate not armed:`; der einzelne Remote ohne Allowlist
|
||||
bleibt `OK` (er hat nichts zu schützen, und ein FAIL machte die Datei durch die Hintertür
|
||||
verpflichtend), sagt aber ausdrücklich, dass jedes Push-Ziel durchkommt. Der Fall, der wirklich
|
||||
beißt — mehrere Remotes ohne Allowlist — bleibt `WARN`. Der Check hatte bislang **keine Tests**;
|
||||
drei sind dazugekommen, einer je Zustand.
|
||||
|
||||
**Dateien:** `.wikitool-remotes.json` (neu, nicht committet), `doctor.check_publish_remotes()`,
|
||||
`tools/chemenu/tests/test_doctor.py`.
|
||||
|
||||
---
|
||||
|
||||
## 2.2.2 - 2026-09-01 - Doku-Verdrahtung: Publish-Remote Gate im Werkzeugvertrag, Projektseite auf oeffentlich
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
+239
@@ -0,0 +1,239 @@
|
||||
# MCP-Leseserver installieren
|
||||
|
||||
Dieses Dokument richtet sich an Menschen. Es beschreibt, wie der MCP-Leseserver eines Chemenu-
|
||||
Wikis lokal läuft, wie ein Client ihn einbindet, und wie er hinter einer Authentifizierung
|
||||
erreichbar wird. Der agent-seitige Betriebsablauf steht in
|
||||
[instructions/mcp-read-server.md](instructions/mcp-read-server.md); die vollständige
|
||||
Kommandoreferenz in [tools/CONTRACT.md](tools/CONTRACT.md).
|
||||
|
||||
**Was der Server ist.** Ein zweiter Konsument desselben Kerns, nicht ein zweites Programm. CLI
|
||||
und Server rufen dieselben Funktionen auf; ein Golden-Test hält ihre Ausgaben gegeneinander.
|
||||
Was `tools/wikitool search --json` liefert, liefert das MCP-Tool `search` auch — plus den
|
||||
Commit, aus dem die Antwort berechnet wurde.
|
||||
|
||||
**Was er nicht ist.** Kein Schreibpfad. Es gibt kein Tool, das eine Seite anlegt, ändert oder
|
||||
publiziert — nicht weil eine Liste gefiltert wird, sondern weil der Server nichts unter
|
||||
`tools/chemenu/commands/` importiert. Die Funktionen sind aus diesem Prozess heraus nicht
|
||||
erreichbar.
|
||||
|
||||
## Voraussetzungen
|
||||
|
||||
- Eine funktionierende Instanz nach [INSTALL.md](INSTALL.md) — inklusive `tools/.venv` und
|
||||
`ripgrep`
|
||||
- Python 3.11 oder neuer (wie die CLI)
|
||||
|
||||
## Schritt 1: Abhängigkeit installieren
|
||||
|
||||
Sie liegt bewusst nicht in `tools/requirements.txt`. Eine Instanz, die nur die CLI benutzt, soll
|
||||
dafür nicht pydantic, starlette, uvicorn und cryptography mitinstallieren müssen.
|
||||
|
||||
```bash
|
||||
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
|
||||
```
|
||||
|
||||
## Schritt 2: Lokal starten (stdio)
|
||||
|
||||
`stdio` ist der Weg zum Ausprobieren und für einen Client auf derselben Maschine: ein Prozess
|
||||
pro Konsument, lokal gestartet, kein Netzwerk.
|
||||
|
||||
```bash
|
||||
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp
|
||||
```
|
||||
|
||||
Der Prozess spricht MCP über stdin/stdout und gibt für sich genommen nichts aus — das ist
|
||||
richtig so. Gestartet wird er normalerweise nicht von Hand, sondern vom Client (Schritt 3).
|
||||
|
||||
**`WIKI_TRACE=0` ist nicht optional.** Telemetrie ist per Default an und schreibt nach
|
||||
`reports/telemetry/` im Repo. Der Server **verweigert den Start**, solange das so ist, statt
|
||||
still umzuleiten:
|
||||
|
||||
```
|
||||
ERROR Telemetry is on and would write into the served checkout (...). Set WIKI_TRACE=0,
|
||||
or point WIKI_TRACE_DIR outside the corpus.
|
||||
```
|
||||
|
||||
Beide Auswege sind gleichwertig: `WIKI_TRACE=0` schaltet ab, `WIKI_TRACE_DIR=/var/log/chemenu`
|
||||
lenkt um. Der Grund steht in Schritt 6 — der Sync darf `reports/` wegräumen.
|
||||
|
||||
## Schritt 3: Einen Client einbinden
|
||||
|
||||
Die Konfiguration folgt der üblichen MCP-Client-Form. Absolute Pfade, weil der Client kein
|
||||
Arbeitsverzeichnis erbt:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"chemenu": {
|
||||
"command": "/pfad/zur/instanz/tools/.venv/bin/python",
|
||||
"args": ["-m", "chemenu.mcp"],
|
||||
"cwd": "/pfad/zur/instanz/tools",
|
||||
"env": {
|
||||
"WIKI_TRACE": "0",
|
||||
"CHEMENU_ROOT": "/pfad/zur/instanz"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`CHEMENU_ROOT` sagt, **welches** Wiki bedient wird. Ohne die Variable nimmt der Server den
|
||||
Checkout, in dem das Paket selbst liegt — für eine einzelne Instanz reicht das, aber wer mehrere
|
||||
Korpora hat, setzt sie besser immer.
|
||||
|
||||
Danach kennt der Client fünf Werkzeuge:
|
||||
|
||||
| Tool | Was es beantwortet |
|
||||
|---|---|
|
||||
| `search` | Seiten in `kb/` nach Text, nach Frontmatter (`confidence<0.6`, `tags~k8s`) oder beidem |
|
||||
| `types` | Welche Seitentypen dieses Wiki kennt |
|
||||
| `describe_type` | Der vollständige Vertrag eines Typs: Felder, Pflichtangaben, Enums |
|
||||
| `lint` | Strukturelle Befunde: kaputte Wikilinks, Waisen, Index-Drift, Schema-Lücken |
|
||||
| `status` | Momentaufnahme: Seitenzahl, Verteilung auf Collections, Befundzahlen |
|
||||
|
||||
## Schritt 4: Ausgeliefert starten (streamable HTTP)
|
||||
|
||||
Der Transport für den Betrieb, und der einzige, vor den sich ein Reverse Proxy setzen kann.
|
||||
|
||||
```bash
|
||||
WIKI_TRACE=0 CHEMENU_ROOT=/srv/chemenu \
|
||||
tools/.venv/bin/python -m chemenu.mcp \
|
||||
--transport streamable-http --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Der Endpunkt ist dann `http://<host>:8000/mcp`.
|
||||
|
||||
`--host 0.0.0.0` ist bewusst nicht der Default. Ohne die Angabe bindet der Server auf Loopback,
|
||||
was lokal richtig und im Container falsch ist — dort muss der Proxy ihn erreichen können. Wer
|
||||
`0.0.0.0` setzt, muss also auch dafür sorgen, dass davor etwas steht (Schritt 5).
|
||||
|
||||
`sse` wird nicht angeboten. Es ist der abgelöste Remote-Transport; jetzt darauf zu bauen
|
||||
verschiebt den Wechsel nur.
|
||||
|
||||
## Schritt 5: Authentifizierung davor
|
||||
|
||||
**Der Server authentifiziert nicht selbst, und das ist Absicht.** Nicht sauber
|
||||
authentifizierte Zugriffe sollen den Python-Prozess gar nicht erst erreichen. Die
|
||||
Authentifizierung ist eine Traefik-ForwardAuth-Middleware:
|
||||
|
||||
> **<https://gitea.nehmer.net/torben/gitea-mcp-forward-auth>**
|
||||
|
||||
Kurz, was sie tut: sie prüft `Authorization: Bearer <token>` gegen SHA-256-Hashes erlaubter
|
||||
Tokens, antwortet `200` bei gültigem und `401` bei fehlendem oder falschem Token, und hält
|
||||
`GET /healthz` immer offen. Klartext-Tokens liegen weder in der Konfiguration noch im Log — nur
|
||||
Hashes und ein kurzer Fingerprint. Konfiguriert wird sie über
|
||||
`AUTH_PROXY_TOKEN_HASHES_DIR` (ein Verzeichnis, eine Datei je Token-Hash — passend für ein
|
||||
Kubernetes-Secret-Volume) oder `AUTH_PROXY_TOKEN_HASHES` (kommagetrennte Liste). Einzelheiten,
|
||||
Referenzmanifeste und ein Testskript stehen im README dort.
|
||||
|
||||
Einen Token-Hash erzeugen:
|
||||
|
||||
```bash
|
||||
echo -n "mein-token" | sha256sum | awk '{print $1}'
|
||||
```
|
||||
|
||||
**Rate Limiting gehört an dieselbe Stelle** — vor den Prozess, neben die Authentifizierung.
|
||||
Nicht in den Iteration Budget Gate: der begrenzt eine *Agenten-Session* am unbemerkten Iterieren
|
||||
über den Wiki-Zustand, weshalb Retrieval von ihm ausgenommen ist. Ihn als Rate Limiter zu
|
||||
benutzen würde ihn dazu verwässern.
|
||||
|
||||
## Schritt 6: Den Korpus aktuell halten
|
||||
|
||||
Der Server liest den Arbeitsbaum. Ein veralteter Checkout antwortet selbstbewusst falsch —
|
||||
deshalb trägt **jede Antwort den Commit**, aus dem sie berechnet wurde:
|
||||
|
||||
```json
|
||||
{ "commit": "<40-stelliger SHA>", "as_of": "<ISO-8601, UTC>", "count": 3, "...": "..." }
|
||||
```
|
||||
|
||||
Aktuell gehalten wird der Baum durch Polling, aus einem Timer neben dem Server:
|
||||
|
||||
```bash
|
||||
git -C "$CHEMENU_ROOT" fetch --quiet origin && \
|
||||
git -C "$CHEMENU_ROOT" reset --hard --quiet origin/main
|
||||
```
|
||||
|
||||
Polling statt Webhook, weil es keinen eingehenden Endpunkt und keine Signaturprüfung braucht —
|
||||
eine kleinere Angriffsfläche als das, was es optimieren würde.
|
||||
|
||||
`reset --hard` ist dabei tragend und keine Bequemlichkeit: der Korpus-Cache verwendet einen
|
||||
Parse wieder, solange der Commit gleich bleibt, und cacht einen **schmutzigen Baum überhaupt
|
||||
nicht**. Ein abgedrifteter Checkout antwortet also zwar richtig, parst aber bei jeder Anfrage
|
||||
neu — und stempelt jede Antwort mit `"commit": null`, weil sie keiner Revision entspricht.
|
||||
|
||||
## Verifikation
|
||||
|
||||
Läuft es? Der schnellste Test ohne Client — startet den Server über stdio, listet die Tools und
|
||||
stellt eine Frage:
|
||||
|
||||
```bash
|
||||
cd tools && WIKI_TRACE=0 .venv/bin/python - <<'EOF'
|
||||
import asyncio, os
|
||||
from mcp import ClientSession, StdioServerParameters
|
||||
from mcp.client.stdio import stdio_client
|
||||
|
||||
async def main():
|
||||
params = StdioServerParameters(
|
||||
command=".venv/bin/python", args=["-m", "chemenu.mcp"],
|
||||
env={"WIKI_TRACE": "0", "PATH": os.environ["PATH"]},
|
||||
)
|
||||
async with stdio_client(params) as (r, w):
|
||||
async with ClientSession(r, w) as s:
|
||||
await s.initialize()
|
||||
print("Tools:", [t.name for t in (await s.list_tools()).tools])
|
||||
out = await s.call_tool("status", {})
|
||||
d = getattr(out, "structuredContent", None) or out.structured_content
|
||||
print("Seiten:", d["pages"], "| Commit:", d["commit"])
|
||||
|
||||
asyncio.run(main())
|
||||
EOF
|
||||
```
|
||||
|
||||
Erwartete Ausgabe, sinngemäß:
|
||||
|
||||
```
|
||||
Tools: ['search', 'types', 'describe_type', 'lint', 'status']
|
||||
Seiten: 176 | Commit: 576df2cdddc96614a7e6641e562022d52112d411
|
||||
```
|
||||
|
||||
Gegen die CLI gegenprüfen — beide müssen dieselbe Antwort geben:
|
||||
|
||||
```bash
|
||||
tools/wikitool search "<begriff>" --json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`ERROR Telemetry is on and would write into the served checkout`** — erwartetes Verhalten,
|
||||
kein Fehler in der Installation. `WIKI_TRACE=0` setzen oder `WIKI_TRACE_DIR` aus dem Korpus
|
||||
heraus zeigen lassen (Schritt 2).
|
||||
|
||||
**`ModuleNotFoundError: No module named 'mcp'`** — Schritt 1 fehlt, oder der Client startet ein
|
||||
anderes Python als das der Instanz. Im Client den absoluten Pfad auf `tools/.venv/bin/python`
|
||||
setzen.
|
||||
|
||||
**`"commit": null` in jeder Antwort** — der bediente Baum hat uncommittete Änderungen. Entweder
|
||||
läuft der Sync nicht, oder etwas schreibt in den Korpus, das dort nichts zu suchen hat. Der
|
||||
Server selbst schreibt nie; ein Test prüft das, indem er alle fünf Tools aufruft und Dateibaum,
|
||||
`HEAD` und `git status --porcelain` vorher/nachher vergleicht.
|
||||
|
||||
**`commit` nennt eine alte Revision** — der Sync aus Schritt 6 läuft nicht.
|
||||
|
||||
**Der Server antwortet anders als `wikitool`** — das ist ein Defekt, keine
|
||||
Konfigurationsdifferenz: beide gehen durch dieselben Funktionen, und ein Golden-Test hält sie
|
||||
zusammen. Zuerst prüfen, ob beide auf denselben Root zeigen; `CHEMENU_ROOT` ist leicht für einen
|
||||
von beiden gesetzt und für den anderen nicht.
|
||||
|
||||
**Von außen nicht erreichbar** — ohne `--host 0.0.0.0` bindet der Server auf Loopback
|
||||
(Schritt 4). Wenn er dann erreichbar ist, aber jeder Aufruf `401` bekommt, arbeitet die
|
||||
Middleware aus Schritt 5 korrekt und das Token stimmt nicht.
|
||||
|
||||
## Was hier bewusst nicht steht
|
||||
|
||||
Deployment — Cluster, Ingress-Hosts, Secret-Store, FluxCD-Quelle. Das ist private Infrastruktur
|
||||
und dieses Repo ist öffentlich.
|
||||
|
||||
Ein **Container-Image** für den Betrieb gibt es noch nicht; es ist als eigenes Vorhaben erfasst,
|
||||
mitsamt den Entscheidungen, die dafür noch offen sind (Korpus im Image oder als Volume, wer den
|
||||
Sync ausführt, Basis-Image, Healthcheck):
|
||||
<https://gitea.nehmer.net/torben/chemenu/issues/37>. Bis dahin ist der Weg oben — venv,
|
||||
`python -m chemenu.mcp`, Proxy davor — der vollständige.
|
||||
+21
-2
@@ -7,6 +7,10 @@ Repo als Upstream** aufsetzen. Der agent-seitige Ablauf steckt in `instructions/
|
||||
nur die menschlichen Teile - für die vollständige Kommandoreferenz siehe
|
||||
[tools/CONTRACT.md](tools/CONTRACT.md).
|
||||
|
||||
Den optionalen **MCP-Leseserver** installiert und betreibt
|
||||
[INSTALL-MCP.md](INSTALL-MCP.md): derselbe Korpus, lesend, für einen Konsumenten, der kein
|
||||
Terminal auf dieser Maschine ist.
|
||||
|
||||
## Voraussetzungen
|
||||
|
||||
- Python 3.11 oder neuer
|
||||
@@ -142,8 +146,21 @@ nicht erreichbarer Feed wird als Fehler gemeldet - **nie** als „aktuell".
|
||||
|
||||
**Was die Versionsnummer aussagt:** kompatibel ist, was in der *linkesten von Null
|
||||
verschiedenen Stelle* übereinstimmt. `0.1.3 → 0.1.4` ist ein sicheres Update, `0.1.3 → 0.2.0`
|
||||
verlangt eine Migration, und ab `1.0.0` liest sich dieselbe Regel als das gewohnte „MAJOR heißt
|
||||
Migration". `version check` sagt das direkt (`state: update` vs. `state: migration`).
|
||||
nicht, und ab `1.0.0` liest sich dieselbe Regel als das gewohnte „MAJOR bricht". `version check`
|
||||
sagt das direkt (`state: update` vs. `state: migration`).
|
||||
|
||||
Was diese Stelle beantwortet, ist **ob die neue Version ein Drop-in-Ersatz ist** - ob sich die
|
||||
Maschinerie einfach darüberkopieren lässt und ob die alte danach noch zurückkann. Ob *Inhalt*
|
||||
migriert werden muss, ist eine **zweite, unabhängige Frage**. Ein MAJOR-Sprung kann eine leere
|
||||
Migrationskette haben und trotzdem Handarbeit verlangen: umbenannter Release-Feed, umbenanntes
|
||||
Artefakt, umbenannter Import- oder Kommandoname, geänderte Envvar - `kb/` bleibt dabei
|
||||
unangetastet, das Update ist trotzdem keins zum Drüberkopieren. Der Abschnitt „Sonderfall:
|
||||
Update von 1.x auf 2.0.0" unten ist genau dieser Fall.
|
||||
|
||||
Deshalb stehen in den Release-Notes eines MAJOR zwei getrennte Zeilen, und beide sind vor dem
|
||||
Update zu lesen: **Breaking Change:** sagt, was aufhört zu funktionieren und was diese Instanz
|
||||
dagegen tun muss; **Migration:** sagt, ob und wie der Korpus umgeschrieben wird (`none required`,
|
||||
wenn nicht). `tools/wikitool version notes` druckt den Eintrag.
|
||||
|
||||
### Eine Instanz aktualisieren
|
||||
|
||||
@@ -223,6 +240,8 @@ behält Schema und Shape.
|
||||
| `WIKITOOL_SESSION_ID` | Scopt das Iteration-Budget-Gate auf eine Aufgabe statt auf ein Terminal | Parent-Process-ID (siehe [instructions/session-setup.md](instructions/session-setup.md)) |
|
||||
| `WIKITOOL_UPDATE_URL` | Release-Feed, den `version check` abfragt | Wert aus `.wikitool-release.json`, sonst der Feed der Ursprungs-Instanz |
|
||||
| `WIKITOOL_UPDATE_TOKEN` | Gitea-Token für den Release-Feed | keiner - gegen `torben/chemenu` nicht nötig, nur für einen privaten Fork (siehe unten) |
|
||||
| `CHEMENU_ROOT` | Auf welchen Korpus das Paket zeigt - für einen Aufrufer, der nicht im Checkout selbst liegt | der Checkout, in dem das Paket liegt (`tools/wikitool` verhält sich ohne die Variable unverändert) |
|
||||
| `WIKI_TRACE` / `WIKI_TRACE_DIR` | Telemetrie abschalten bzw. aus dem Arbeitsbaum heraus umlenken | an, nach `reports/telemetry/` - der MCP-Server verweigert damit den Start, siehe [INSTALL-MCP.md](INSTALL-MCP.md) |
|
||||
|
||||
**Gegen das Ursprungs-Repo braucht es kein Token.** `torben/chemenu` ist öffentlich lesbar;
|
||||
`version check` und der Download in Weg A funktionieren ohne Konfiguration.
|
||||
|
||||
@@ -51,6 +51,7 @@ chemenu/
|
||||
├── CLAUDE.md # Claude Code only: imports AGENTS.md/USER.md/SOUL.md/ENVIRONMENT.md + its Claude-Code-only instructions. No rules of its own
|
||||
├── README.md # This file: human-readable overview of the whole repo
|
||||
├── INSTALL.md # Human-readable setup: new instance vs. cloning this one
|
||||
├── INSTALL-MCP.md # Human-readable setup for the optional MCP read server
|
||||
├── EVALS.md # Human-readable overview of telemetry and evaluation
|
||||
├── CHANGES.md # Changelog for the stack itself
|
||||
├── USER.md # Who operates this instance - context, never instructions
|
||||
@@ -342,6 +343,34 @@ under `instructions/dev/` (never present in a distributed instance - `tools/CONT
|
||||
explains why).
|
||||
<!-- dist:strip-end -->
|
||||
|
||||
### MCP read server (optional)
|
||||
|
||||
The terminal is not the only way in. `tools/chemenu/mcp/` serves the same wiki read-only over
|
||||
MCP - `search`, `types`, `describe_type`, `lint` and `status` - so a consumer that is not a
|
||||
shell on this machine can ask the same questions and get the same answers. Literally the same:
|
||||
the CLI and the server are two adapters over one core, and a golden test holds their output
|
||||
together rather than trusting that it agrees.
|
||||
|
||||
There is no tool that writes, and not because one is filtered out of a list: the server imports
|
||||
nothing under `chemenu/commands/`, so `new`, `publish` and the rest are unreachable from it.
|
||||
Every answer carries the commit it was computed from, so a checkout that has fallen behind
|
||||
produces a visibly stale answer instead of a confident wrong one.
|
||||
|
||||
```bash
|
||||
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
|
||||
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp # stdio
|
||||
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp \
|
||||
--transport streamable-http --host 0.0.0.0 --port 8000 # deployed
|
||||
```
|
||||
|
||||
The dependency is deliberately not in `requirements.txt`: an instance that only uses the CLI
|
||||
should not have to install a web stack to do it.
|
||||
|
||||
Installing it, wiring a client to it, and putting authentication in front of it:
|
||||
[`INSTALL-MCP.md`](INSTALL-MCP.md), for humans. The agent-side operating procedure - keeping the
|
||||
checkout current, and what a stale answer looks like - is
|
||||
[`instructions/mcp-read-server.md`](instructions/mcp-read-server.md).
|
||||
|
||||
### Obsidian
|
||||
|
||||
Open this directory in Obsidian for:
|
||||
|
||||
@@ -48,6 +48,9 @@ stack development happens in the origin repo instead (see AGENTS.md's routing li
|
||||
[testing-conventions.md](../testing-conventions.md) - the suite runs against a deliberately
|
||||
empty machine; what the autouse fixture already neutralizes, and what a test still has to
|
||||
establish itself. Read it before adding or changing a test.
|
||||
[version-parts.md](../version-parts.md) - which part a change bumps: the drop-in test, the
|
||||
catalogue of breaks that cross the compatibility boundary with `kb/` untouched, and what to
|
||||
put in front of the user before a breaking bump. Read it before step 3.
|
||||
More instructions are added here incrementally as stack-development needs come up - this
|
||||
list grows without needing this skill file to change shape.
|
||||
3. **Raise the version, if the change ships.** A change under `tools/`, `types/`,
|
||||
@@ -59,18 +62,27 @@ stack development happens in the origin repo instead (see AGENTS.md's routing li
|
||||
```
|
||||
|
||||
Never edit `VERSION` or the entry's heading by hand - `bump` writes both, and `docs verify`
|
||||
fails a tree where they disagree. Pick the part by what an existing instance would have to do:
|
||||
fails a tree where they disagree. Pick the part by whether the new version is a **drop-in
|
||||
replacement** for the old one - not by whether content has to be migrated:
|
||||
|
||||
| Change | Part |
|
||||
|--------|------|
|
||||
| Fix, no interface change | `--patch` |
|
||||
| New capability, backwards compatible | `--minor` |
|
||||
| **Existing content must be migrated** | `--major` |
|
||||
| New capability, still drop-in in both directions | `--minor` |
|
||||
| **Not a drop-in replacement** - any hand-work by the user or a migration script, or a downgrade that no longer works | `--major` |
|
||||
|
||||
A `--major` bump additionally needs a migration document for the new version - written per
|
||||
[migrate-corpus.md](../../migrate-corpus.md) - or `--no-migration "<reason>"` when no content
|
||||
actually has to change. `bump` refuses otherwise, and so does `docs verify`: an instance
|
||||
learning that it must migrate, with nothing telling it how, is a dead end.
|
||||
Content migration is one way to land in the last row, not the definition of it: a rename of
|
||||
the update path, the artefact, an import name, a flag or an envvar breaks a swap with `kb/`
|
||||
entirely untouched. The full test, the catalogue of such breaks, and what to put in front of
|
||||
the user first are in [version-parts.md](../version-parts.md) - **read it before choosing
|
||||
`--major`.**
|
||||
|
||||
A `--major` bump therefore needs two things recorded. `--breaking "<what stops working>"`
|
||||
is required on every boundary-crossing bump; on top of it, a migration document for the new
|
||||
version - written per [migrate-corpus.md](../../migrate-corpus.md) - or
|
||||
`--no-migration "<reason>"` when no content actually has to change. `bump` refuses without
|
||||
either, and so does `docs verify`: an instance learning that it must migrate, with nothing
|
||||
telling it how, is a dead end.
|
||||
|
||||
Then write the entry's body - `bump` deliberately leaves it empty, the same way `new` leaves
|
||||
the prose.
|
||||
@@ -90,6 +102,12 @@ stack development happens in the origin repo instead (see AGENTS.md's routing li
|
||||
- **Touches both stack code and wiki content in one session?** Apply this skill's rules to the
|
||||
code changes and the normal content skills' rules to the content changes - they are not
|
||||
mutually exclusive within a session, only per change.
|
||||
- **The change turns out not to be a drop-in replacement?** Do not bump across the boundary on
|
||||
your own initiative. Every existing instance pays for a breaking change once, by hand, so the
|
||||
user decides whether it is worth that: show them what breaks, what an instance has to do about
|
||||
it, and the alternatives (avoid the break with a shim, defer and batch it with the next one,
|
||||
or split it behind a deprecation window), then recommend one and wait for a go-ahead.
|
||||
[version-parts.md](../version-parts.md) step 4 has the full shape.
|
||||
|
||||
## Scope
|
||||
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
type: types/instruction.md
|
||||
name: version-parts
|
||||
description: Which part of the stack version a change bumps - the compatibility question (is the new version a drop-in replacement?) separated from the migration question (must existing content change?), plus what to do before a breaking bump.
|
||||
---
|
||||
# Pick the version part for a stack change
|
||||
|
||||
Two questions decide a version bump, and they are **not the same question**:
|
||||
|
||||
1. **Is the new version a drop-in replacement for the old one?** This is what the version
|
||||
number itself says. Compatibility is read off the **leftmost non-zero component** - on this
|
||||
stack (`2.x`) that is MAJOR, on a `0.x` stack it is MINOR. A bump that changes it is called
|
||||
*boundary-crossing* below, because that is the term `version bump` and `docs verify` use in
|
||||
their own messages.
|
||||
2. **Must existing content be migrated?** This is a *consequence* a boundary crossing may or
|
||||
may not have. `kb/` staying untouched does not make a change compatible, and
|
||||
`version bump --no-migration` exists precisely because boundary-crossing bumps with an
|
||||
untouched corpus are a real case.
|
||||
|
||||
Getting these backwards is how a genuinely breaking change ships as a MINOR. It happened once
|
||||
already (see the case study at the end), which is why this file exists.
|
||||
|
||||
## When to run
|
||||
|
||||
Before every `tools/wikitool version bump` - the `stack-dev` skill's step 3 sends you here.
|
||||
Read it in full the first time a change looks like it might be boundary-crossing; afterwards
|
||||
the three-line test below is usually enough.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Apply the drop-in test.** The bump is boundary-crossing if **either** half fails:
|
||||
|
||||
- **Forward:** an existing instance can install the new machinery by copying `tools/`,
|
||||
`types/`, `instructions/`, `AGENTS.md`, `VERSION` and `.wikitool-release.json` over itself,
|
||||
and everything that worked before still works - with **no** hand-work by the user and **no**
|
||||
migration script. Any step beyond the copy, however small, fails this half.
|
||||
- **Backward:** having installed the new version, the user can put the old one back and be
|
||||
where they started. A state file the old version cannot read, a rewritten corpus, a
|
||||
renamed stamp - anything that makes the downgrade fail or leave a broken instance - fails
|
||||
this half.
|
||||
|
||||
Content migration is one way to fail the forward half, not the definition of it.
|
||||
|
||||
2. **Check the catalogue** when the answer still feels like a judgment call. Each of these
|
||||
crosses the boundary with `kb/` entirely untouched:
|
||||
|
||||
| What changed | Why the swap is not drop-in |
|
||||
|---|---|
|
||||
| The update path - `update_url`, the release feed, the repo it points at | The instance cannot repair its own `.wikitool-release.json`: it is machine-written, and invariant 1 forbids the hand-edit. The channel that would have told it to update is the channel that broke |
|
||||
| The release artefact's name | Every download script and every pin against it breaks |
|
||||
| The Python package's import name | `from <old> import ...` outside the shipped tree breaks |
|
||||
| A command, subcommand, or flag that was removed or renamed | Scripts, CI workflows and instruction files calling the old spelling break |
|
||||
| An environment variable's name | An instance configured through it silently loses the configuration |
|
||||
| The shape of a machine-read file - `.wikitool-kb.json`, `.wikitool-release.json`, a generated index | The old version cannot read what the new one wrote, so the downgrade half fails even if the upgrade half passed |
|
||||
| A type-spec's required fields | Existing pages stop validating - this one crosses *and* needs a content migration |
|
||||
|
||||
The catalogue is illustrative, not exhaustive. When something is not on it, go back to step 1.
|
||||
|
||||
3. **Otherwise pick the compatible part:**
|
||||
|
||||
| Change | Part |
|
||||
|---|---|
|
||||
| Fix, no interface change | `--patch` |
|
||||
| New capability, drop-in in both directions | `--minor` |
|
||||
|
||||
4. **Stop and talk to the user before a boundary-crossing bump.** It is expensive in a way the
|
||||
other two parts are not: every existing instance pays for it, once, by hand. Put in front of
|
||||
them, in this order:
|
||||
|
||||
- **What breaks**, concretely - which file, which name, which call site.
|
||||
- **What each existing instance must do**, as the steps they would actually run.
|
||||
- **The alternatives**, so the break is a choice and not a side effect:
|
||||
- *Avoid it* - keep the old name as an alias, read both file shapes, accept both flag
|
||||
spellings. A compatibility shim carried for one release is usually cheaper than a
|
||||
migration everyone runs.
|
||||
- *Defer and batch it* - hold the break until the next boundary crossing, so instances pay
|
||||
once instead of twice.
|
||||
- *Split it* - ship the compatible half now, the breaking half later behind a deprecation
|
||||
window that the changelog announces in advance.
|
||||
- **Your recommendation**, with the trade-off named.
|
||||
|
||||
Then wait for an explicit go-ahead. Do not bump across the boundary on your own initiative.
|
||||
|
||||
5. **Record the break in the bump itself.** A boundary-crossing bump requires
|
||||
`--breaking "<what breaks>"`, which writes a `**Breaking Change:**` line into the entry:
|
||||
|
||||
```bash
|
||||
tools/wikitool version bump --major \
|
||||
--title "<what changed>" \
|
||||
--breaking "<what stops working, and what an instance must do about it>" \
|
||||
--no-migration "<why no page has to change>" # only if that is true
|
||||
```
|
||||
|
||||
`--breaking` is refused on a bump that crosses nothing, and required on one that does;
|
||||
`docs verify` checks the newest boundary-crossing entry still carries the line. Write it for
|
||||
the operator of an instance that has not read this repository: what stops working, and what
|
||||
they do about it.
|
||||
|
||||
6. **Then answer the migration question separately.** Boundary-crossing and
|
||||
content-migrating are independent:
|
||||
|
||||
- Content must change → write the migration document under `instructions/migrations/` per
|
||||
[migrate-corpus.md](../migrate-corpus.md). `bump` finds it by its `migrates_to:` field.
|
||||
- Content need not change → `--no-migration "<reason>"`, which records that in the entry.
|
||||
|
||||
Both are also needed by `docs verify`, for the same reason: an instance that learns it must
|
||||
migrate, with nothing telling it how, is a dead end.
|
||||
|
||||
7. **Write the entry's body.** `bump` leaves it empty on purpose. A boundary-crossing entry
|
||||
earns a paragraph that says *why this is breaking* - it is the one thing a future reader
|
||||
cannot reconstruct from the diff, and it is what the next session in this position will read
|
||||
instead of guessing.
|
||||
|
||||
## Decision points
|
||||
|
||||
- **The change ships no code - only `README.md`, `INSTALL.md`, `EVALS.md`, or `.gitea/`?** No
|
||||
bump at all; CI's version gate is scoped to what changes behaviour.
|
||||
- **A break you can see coming but are not making yet?** File it as an issue and let it
|
||||
accumulate. Boundary crossings are cheaper in batches, and step 4's "defer" alternative is
|
||||
only real if someone wrote the break down.
|
||||
- **Unsure between MINOR and boundary-crossing?** It is boundary-crossing. The cost of an
|
||||
unnecessary MAJOR is one extra release note; the cost of a MINOR that actually breaks is an
|
||||
instance whose update path fails while its version number promised it would not.
|
||||
- **The break only affects this repository, not a distributed instance** - something under
|
||||
`instructions/dev/`, say? Then it is not a stack break at all: `dist export` never shipped it.
|
||||
Judge by what an *exported* instance sees.
|
||||
|
||||
## Scope
|
||||
|
||||
Applies to the stack version in `VERSION` - `tools/`, `types/`, `instructions/`, `AGENTS.md`
|
||||
and the contracts. It says nothing about the content shape in `.wikitool-kb.json`, which is
|
||||
advanced by `wikitool migrate done` and described by [migrate-corpus.md](../migrate-corpus.md),
|
||||
and nothing about wiki content operations, which are logged in `kb/log.md` and carry no version
|
||||
at all.
|
||||
|
||||
Choosing the part remains a judgment call, deliberately: `docs verify` checks that a
|
||||
boundary-crossing entry *documents* its break and its migration, never that the part was chosen
|
||||
correctly. No validator can tell a renamed flag from a new one.
|
||||
|
||||
## Case study: 2.0.0
|
||||
|
||||
The Chemenu rebranding renamed the repo, the release artefact and the Python package. No page
|
||||
in `kb/` changed, so the first attempt was `1.9.0` - the migration question, answered correctly,
|
||||
substituted for the compatibility question, which was never asked. Three things broke: every
|
||||
existing instance's `update_url` pointed at a repo path that no longer existed and could not be
|
||||
hand-repaired; the artefact name changed; the import name changed. The correct bump was
|
||||
`--major --no-migration`, and the `CHANGES.md` entry for `2.0.0` carries the reasoning in full
|
||||
under "Warum das trotzdem MAJOR ist". The error was caught by the user, not by the
|
||||
documentation - which is what step 4 is for.
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
type: types/instruction.md
|
||||
name: mcp-read-server
|
||||
description: Run and keep current the MCP read server that serves this wiki to a second consumer
|
||||
---
|
||||
|
||||
# Running the MCP read server
|
||||
|
||||
Chemenu has a second consumer. `search`, `types`, `describe_type`, `lint` and `status` are
|
||||
served over MCP to callers that are not this terminal - the CLI and the server are two adapters
|
||||
over one core (`chemenu.api.Corpus`), not a CLI with a network interface bolted on.
|
||||
|
||||
This document is about *operating* it: how to start it, what has to be true of the checkout it
|
||||
serves, and how that checkout stays current. What it exposes and why is in
|
||||
[tools/CONTRACT.md](../tools/CONTRACT.md) and in the module's own docstring
|
||||
(`tools/chemenu/mcp/server.py`).
|
||||
|
||||
**Deployment is deliberately not here.** Which cluster, which ingress host, where the credential
|
||||
lives - that is private infrastructure and this is a public repository. What is here is
|
||||
everything an operator needs that is *true of the software* rather than of one installation.
|
||||
|
||||
## When to run
|
||||
|
||||
- Standing something up for a consumer that is not a terminal on this machine.
|
||||
- Diagnosing an answer that looks stale, or one that disagrees with the CLI.
|
||||
- Before pointing a new consumer at an existing server.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Install the server's dependency.** It is deliberately not in `requirements.txt`: an
|
||||
instance that only uses the CLI should not be made to install pydantic, starlette, uvicorn
|
||||
and cryptography to do it.
|
||||
|
||||
```bash
|
||||
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
|
||||
```
|
||||
|
||||
2. **Decide which checkout it serves.** The root resolves by precedence - an explicit `--root`,
|
||||
then `$CHEMENU_ROOT`, then the checkout the package lives in. A deployment points at its
|
||||
corpus with one variable and no code:
|
||||
|
||||
```bash
|
||||
export CHEMENU_ROOT=/srv/chemenu
|
||||
```
|
||||
|
||||
3. **Take tracing out of the served tree.** The server refuses to start otherwise, and the
|
||||
refusal is the point: telemetry defaults to on and writes under `reports/telemetry/` inside
|
||||
the repo, which step 5's sync is entitled to wipe. Either is fine:
|
||||
|
||||
```bash
|
||||
export WIKI_TRACE=0 # off
|
||||
export WIKI_TRACE_DIR=/var/log/chemenu # or elsewhere, outside the corpus
|
||||
```
|
||||
|
||||
4. **Start it on the transport that matches what is in front of it.**
|
||||
|
||||
```bash
|
||||
tools/.venv/bin/python -m chemenu.mcp # stdio
|
||||
tools/.venv/bin/python -m chemenu.mcp --transport streamable-http # deployed
|
||||
```
|
||||
|
||||
`stdio` is for developing and testing without a network - one process per consumer, started
|
||||
locally. `streamable-http` is what a deployed instance speaks, and the only one the
|
||||
authentication middleware can sit in front of, because that middleware is an HTTP reverse
|
||||
proxy. `sse` is reachable through the SDK and deliberately not offered: it is the superseded
|
||||
remote transport, and building on it now only moves the migration later.
|
||||
|
||||
5. **Keep the checkout current by polling, and keep it clean.**
|
||||
|
||||
```bash
|
||||
git -C "$CHEMENU_ROOT" fetch --quiet origin && \
|
||||
git -C "$CHEMENU_ROOT" reset --hard --quiet origin/main
|
||||
```
|
||||
|
||||
Every few minutes, from a timer beside the server. Polling rather than a webhook on purpose:
|
||||
it needs no inbound endpoint and no signature checking, which is a smaller surface than the
|
||||
thing it would optimize. A webhook is a later optimization, not a starting point.
|
||||
|
||||
`reset --hard` is load-bearing, not a convenience. The corpus cache reuses a parse while the
|
||||
commit is unchanged and **refuses to cache a dirty tree at all**, so a checkout that has
|
||||
drifted answers correctly but reparses on every request - and every answer it gives is
|
||||
stamped `"commit": null`, because a dirty tree corresponds to no revision.
|
||||
|
||||
## Decision points
|
||||
|
||||
- **An answer looks stale?** Read `commit` in the response. If it names an old revision, the
|
||||
sync is not running. If it is `null`, the served tree has uncommitted changes - something is
|
||||
writing into the corpus that should not be.
|
||||
- **The server disagrees with `wikitool` on the same query?** That is a defect, not a
|
||||
configuration difference: the two go through the same functions and a golden test holds their
|
||||
output together (`tools/chemenu/tests/test_mcp_server.py`). Check first that both are pointed
|
||||
at the same root - `CHEMENU_ROOT` is easy to set for one and not the other.
|
||||
- **Asked to expose a write tool?** There is none, and the way to add one is not a flag. The
|
||||
server imports nothing under `chemenu.commands`, so `new`, `touch`, `xref`, `cite`, `publish`
|
||||
and `migrate` are unreachable from it rather than filtered out of a list. Submitting documents
|
||||
from outside is a different design with a quarantine in it - Gitea #32 - not a tool added
|
||||
here.
|
||||
- **Asked to rate-limit inside the server?** Rate limiting belongs in the middleware in front of
|
||||
the process, next to authentication. Not the Iteration Budget Gate: that exists to stop an
|
||||
agent *session* from iterating unnoticed over the wiki's state, which is why retrieval is
|
||||
exempt from it, and using it as a rate limiter would dilute it into one.
|
||||
|
||||
## Scope
|
||||
|
||||
Not for setting up an instance ([setup-instance.md](setup-instance.md)) or a fresh clone
|
||||
([bootstrap.md](bootstrap.md)). Not for the authentication or rate-limiting middleware, which is
|
||||
infrastructure configuration rather than part of this repository. Not a write path: see the
|
||||
decision point above.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
type: types/concept.md
|
||||
concept_type: decision
|
||||
tags: []
|
||||
created: 2026-09-01
|
||||
modified: 2026-09-01
|
||||
related: [Chemenu]
|
||||
sources: ['Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]
|
||||
confidence: 0.70
|
||||
confidence_base: 0.70
|
||||
provenance: sourced
|
||||
summary: 'Private Korpusinhalte per Loeschung entfernen statt zu anonymisieren: ein Seitentitel ist der einzige Identifier eines Wikis, Umbenennen ist die volle page-lifecycle-Prozedur je Seite, Loeschen ist ein unterstuetztes Kommando.'
|
||||
---
|
||||
# Delete Rather Than Anonymize
|
||||
|
||||
**Typ:** Decision
|
||||
|
||||
## Definition
|
||||
|
||||
Wenn private oder sensible Inhalte aus einem Wiki entfernt werden müssen, ist Löschen einer
|
||||
zugehörigen Seite in der Regel dem Anonymisieren (Umbenennen, Ersetzen sensibler Details bei
|
||||
sonst unverändertem Inhalt) vorzuziehen - wenn ein unterstütztes Löschkommando existiert.
|
||||
|
||||
## Kernpunkte
|
||||
|
||||
- Ein Seitentitel ist in einem verlinkten Wiki oft der **einzige Identifier** einer Seite: Er
|
||||
lebt in Wikilinks, Zitatmarkern und Frontmatter-Arrays jeder referenzierenden Seite. Ihn zu
|
||||
ändern (Anonymisieren durch Umbenennen) verlangt deshalb eine vollständige Rename-Prozedur pro
|
||||
betroffener Seite - bei mehreren zusammenhängenden Seiten multipliziert sich der Aufwand.
|
||||
- Löschen dagegen ist ein einzelner, unterstützter Vorgang, der eine Seite mechanisch aus dem
|
||||
Rest des Wikis de-linkt (bekannte Referenzarten: Frontmatter-Felder, ganzzeilige
|
||||
Verweis-Aufzählungen). Er ist damit für strukturelle Bereinigung **schneller und weniger
|
||||
fehleranfällig** als Anonymisierung.
|
||||
- Bei Inhalten, die eine reale Topologie beschreiben (z. B. eine Infrastrukturdokumentation),
|
||||
entschärft Anonymisieren einzelner Bezeichner (Hostnamen, IP-Adressen) die eigentliche
|
||||
Preisgabe nicht: Die Struktur - welche Systeme wie zusammenhängen - bleibt erhalten, auch wenn
|
||||
die Namen ausgetauscht sind.
|
||||
- **Grenze der Methode:** Ein mechanisches Löschkommando entfernt typischerweise nur
|
||||
strukturelle Referenzen (Frontmatter, Aufzählungen), nicht zwingend Erwähnungen im Fließtext
|
||||
einer anderen Seite. Nach der Löschung ist eine gezielte Nachkontrolle nötig, ob der entfernte
|
||||
Name noch im Klartext irgendwo im Wiki steht.
|
||||
|
||||
## Wann zu verwenden
|
||||
|
||||
- Der zu entfernende Inhalt ist als eigenständige Seite oder eigenständige Seitengruppe
|
||||
abgrenzbar.
|
||||
- Ein Löschkommando existiert, das Referenzen mechanisch bereinigt (nicht ein bloßes Entfernen
|
||||
der Datei, das tote Links hinterlässt).
|
||||
- Der Inhalt beschreibt eine reale, zusammenhängende Struktur (Infrastruktur, ein Netzwerk, eine
|
||||
Organisation), bei der einzelne Bezeichner austauschen die eigentliche Preisgabe nicht behebt.
|
||||
|
||||
## Wann NICHT zu verwenden
|
||||
|
||||
- Wenn nur ein einzelner sensibler Wert innerhalb einer sonst wertvollen, generischen Seite
|
||||
steht (z. B. ein Firmenname als Beispiel in einer sonst allgemeingültigen Anleitung) - dort ist
|
||||
gezieltes Redigieren der Seite treffender als sie komplett zu verwerfen.
|
||||
- Wenn die Seite Beziehungen trägt, die für sich genommen wertvoll und nicht sensibel sind - dann
|
||||
kann eine Neufassung mit generischem Beispiel sinnvoller sein als Löschung.
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Mass-Update Gate]]
|
||||
|
||||
## Beziehungen
|
||||
|
||||
- **gilt fuer:** [[Chemenu]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
- [[Chemenu]]
|
||||
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
type: types/concept.md
|
||||
concept_type: decision
|
||||
tags: []
|
||||
created: 2026-09-01
|
||||
modified: 2026-09-01
|
||||
related: [Chemenu]
|
||||
sources: ['Source - Public Release, Corpus Purge and History Squash Session 2026-09-01']
|
||||
confidence: 0.70
|
||||
confidence_base: 0.70
|
||||
provenance: sourced
|
||||
summary: Ein Repo mit Code- und Inhaltsanteil erhaelt zwei Lizenzen; die Grenze zwischen ihnen ist kein zweiter, gepflegter Pfadkatalog, sondern der ohnehin vorhandene Dateiplan des Distributionswerkzeugs.
|
||||
---
|
||||
# Dual Licensing by File Plan
|
||||
|
||||
**Typ:** Decision
|
||||
|
||||
## Definition
|
||||
|
||||
Ein Repository, das sowohl Werkzeug-Code als auch inhaltliches Material (Dokumentation, Daten,
|
||||
kompiliertes Wissen) enthält, bekommt zwei Lizenzdateien statt einer - eine für den Code, eine
|
||||
für den Inhalt. Welche Datei zu welcher Lizenz gehört, wird nicht in einer eigenen, zweiten
|
||||
Liste festgehalten, sondern aus dem Dateiplan abgeleitet, den ein vorhandenes
|
||||
Distributions-/Build-Werkzeug ohnehin pflegt.
|
||||
|
||||
## Kernpunkte
|
||||
|
||||
- Der naheliegende Fehler ist, die Grenze zwischen „Code" und „Inhalt" als eigene, gepflegte
|
||||
Aufzählung von Pfaden in der Lizenzdatei selbst festzuschreiben. Das ist eine zweite Kopie
|
||||
einer Regel, die bereits an anderer Stelle existiert (dem Dateiplan des Build-/
|
||||
Distributionswerkzeugs) - und die Kopie, die driftet, wenn sich Verzeichnisse verschieben.
|
||||
- Stattdessen verweist die Lizenz-Notiz auf den bestehenden Plan (z. B. eine Funktion, die
|
||||
berechnet, was in eine Distribution exportiert wird und was nicht) als **einzige** Quelle der
|
||||
Wahrheit für die Grenze.
|
||||
- Welche der beiden Lizenzen den generischen Dateinamen `LICENSE` trägt, ist keine
|
||||
Nebensächlichkeit: Es sollte die Lizenz sein, die ein Code-Hosting-Dienst (Forge) für das
|
||||
Repository insgesamt meldet - typischerweise die restriktivere/Copyleft-Lizenz. Ein Leser, der
|
||||
eine Copyleft-Pflicht übersieht, wird dadurch geschädigt; wer eine Pflicht zu viel annimmt,
|
||||
nicht.
|
||||
- Ein Distributions-Export, der Code unter einer Copyleft-Lizenz ausliefert, muss die
|
||||
zugehörige Lizenzdatei zwingend mitliefern (nicht optional, nicht still übersprungen, wenn sie
|
||||
fehlt) - sonst ist die exportierte Instanz eine Lizenzverletzung, sobald sie veröffentlicht
|
||||
wird.
|
||||
|
||||
## Wann zu verwenden
|
||||
|
||||
- Ein Repository trägt sowohl Software-/Werkzeugcode als auch Inhalt mit eigenem
|
||||
Urheberrechtscharakter (Dokumentation, Wissensbasis, Daten), für die unterschiedliche Lizenzen
|
||||
angemessen sind.
|
||||
- Es existiert bereits ein Werkzeug, das programmatisch entscheidet, welche Dateien zu welcher
|
||||
Kategorie gehören (z. B. für einen Export- oder Build-Schritt).
|
||||
|
||||
## Wann NICHT zu verwenden
|
||||
|
||||
- Bei einem Repository, dessen Inhalt untrennbar mit dem Code verwoben ist und für das keine
|
||||
separate, maschinell nachvollziehbare Grenze existiert - dort wäre die Lizenz-Zuordnung selbst
|
||||
wieder eine unabhängige, drift-anfällige Liste.
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Chemenu]]
|
||||
|
||||
## Beziehungen
|
||||
|
||||
- **gilt fuer:** [[Chemenu]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
- [[Chemenu]]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# kb/concepts/ - Index
|
||||
|
||||
76 page(s). Regenerated by `wikitool index rebuild`.
|
||||
80 page(s). Regenerated by `wikitool index rebuild`.
|
||||
|
||||
## All
|
||||
|
||||
@@ -25,9 +25,11 @@
|
||||
| [[CPPC]] | protocol | Hardwareschnittstelle Collaborative Processor Performance Control für feingranulares CPU-Power-Management zwischen Betriebssystem und AMD-Prozessor. | 2026-08-29 |
|
||||
| [[Cross-platform Agent Skills]] | architecture | Architektur fuer Agent-Skills, die ueber mehrere LLM-Werkzeuge hinweg funktionieren; in Chemenu selbst am 2026-08-04 umgesetzt und ueberprueft | 2026-09-01 |
|
||||
| [[Crystallization]] | workflow | Verdichten abgeschlossener Erkundungen, Debugging-Sitzungen und Recherchen zu strukturierten Wiki-Auszügen als eigenständige Wissensquellen. | 2026-08-29 |
|
||||
| [[Delete Rather Than Anonymize]] | decision | Private Korpusinhalte per Loeschung entfernen statt zu anonymisieren: ein Seitentitel ist der einzige Identifier eines Wikis, Umbenennen ist die volle page-lifecycle-Prozedur je Seite, Loeschen ist ein unterstuetztes Kommando. | 2026-09-01 |
|
||||
| [[Denylist over Allowlist]] | decision | Entscheidung, schreibbare Felder als Schema minus kurzer Sperrliste zu bestimmen statt als gepflegte Positivliste, weil die Positivliste eine zweite Kopie des Schemas waere | 2026-08-31 |
|
||||
| [[Detect-Repair Asymmetry]] | problem | Werkzeugluecke, in der ein Check einen Defekt zuverlaessig meldet, aber kein Befehl ihn behebt - womit die Handeditierung der einzige verbleibende Ausweg ist | 2026-08-31 |
|
||||
| [[Diff-Reviewable Agent Edits]] | decision | Entscheidung, Dateiaenderungen ueber Edit/Write statt ueber Shell-Heredocs zu fahren, weil nur das erste eine pruefbare Diff hinterlaesst | 2026-08-31 |
|
||||
| [[Dual Licensing by File Plan]] | decision | Ein Repo mit Code- und Inhaltsanteil erhaelt zwei Lizenzen; die Grenze zwischen ihnen ist kein zweiter, gepflegter Pfadkatalog, sondern der ohnehin vorhandene Dateiplan des Distributionswerkzeugs. | 2026-09-01 |
|
||||
| [[Entity Extraction]] | pattern | Erkennen und Strukturieren von Entities (Personen, Projekte, Bibliotheken, Concepts, Dateien, Entscheidungen, Systeme, Werkzeuge) samt typspezifischer Attribute aus Rohquellen. | 2026-08-29 |
|
||||
| [[Episodic Memory]] | architecture | Speicherschicht für verdichtete Sitzungszusammenfassungen und Befunde; Brücke zwischen rohem Working Memory und langlebigem Semantic Memory. | 2026-08-29 |
|
||||
| [[Event-Driven Automation]] | workflow | Muster, das automatische Auslöser an Wiki-Lebenszyklusereignisse hängt, um manuellen Pflegeaufwand und das Risiko der Verwahrlosung zu senken. | 2026-08-29 |
|
||||
@@ -40,7 +42,7 @@
|
||||
| [[Implementation Spectrum]] | architecture | Modularer Einführungspfad für die Funktionen von LLM Wiki v2, vom minimal tragfähigen Wiki bis zur vollen Umsetzung mit Automatisierung und Governance. | 2026-08-29 |
|
||||
| [[Index Scaling]] | workflow | Skalierungsregeln für Indexseiten: Tabellenabschnitte ab 50 Einträgen teilen, ab 200 Seiten _meta/topic-map.md anlegen | 2026-08-29 |
|
||||
| [[Issue Label Scheme]] | decision | Zweiachsiges Pflicht-Labelschema fuer das Gitea-Board: prio/1..3 und size/XS..L, bewusst keine dritte Achse; die Regel liegt in instructions/dev/, weil sie keine ausgelieferte Instanz erreichen darf | 2026-08-31 |
|
||||
| [[Iteration and Cost Limits]] | workflow | Im Code durchgesetzte Obergrenze von 60 wikitool-Aufrufen je Session\, Loop-Breaker bei 3 identischen Wiederholungen\, Slot-Erstattung bei abgelehntem Aufruf und ein seit 1.5.0 gemessenes Kalibrierungsband von 5-15 (einfach) bzw. 20-35 (komplex) Aufrufen | 2026-08-31 |
|
||||
| [[Iteration and Cost Limits]] | workflow | Im Code durchgesetzte Obergrenze von 60 wikitool-Aufrufen je Session, Loop-Breaker bei 3 identischen Wiederholungen, Slot-Erstattung, ein gemessenes Kalibrierungsband, und Retrieval sowie der MCP-Leseserver bleiben ausgenommen | 2026-09-02 |
|
||||
| [[KB Migration]] | workflow | Migration des KB-Inhalts entlang einer geordneten Versionskette; abgegrenzt gegen offene Instanz-Aktionen, die in den doctor-Check gehoeren statt in die Kette | 2026-08-31 |
|
||||
| [[KB Stack Versioning]] | decision | Semantische Versionierung des Wiki-Stacks: VERSION beschreibt die Maschinerie, Kompatibilitaet ist die linkeste Nicht-Null-Komponente, und drei getrennte Dateien trennen Maschinerie, Herkunft und Content-Form | 2026-08-30 |
|
||||
| [[Knowledge Compounding]] | workflow | Effekt, bei dem Wissen im Wiki an Wert gewinnt, weil jede neue Quelle an bestehende, untereinander verwiesene Seiten anknüpft und sie ergänzt. | 2026-08-29 |
|
||||
@@ -48,6 +50,7 @@
|
||||
| [[Lint Workflow]] | workflow | Deterministischer Health-Check rund um wikitool lint; seit 1.7.2 maskiert es Code vor dem Notation-Match und zaehlt Zitat-Bloecke statt Zeilen | 2026-09-01 |
|
||||
| [[LLM Wiki Pattern]] | architecture | Methodik für persönliches Wissensmanagement, bei der ein LLM aus Rohquellen ein dauerhaftes Wiki aufbaut - Wissen wird kompiliert statt per RAG neu hergeleitet. | 2026-08-29 |
|
||||
| [[Mass-Update Gate]] | workflow | Mass-Update Gate: publish endet mit 42 (Freigabe durch den Menschen noetig) ab 10 gezaehlten Dateien; generierte Dateien und work/ werden committet\, aber seit 1.5.0 nicht gezaehlt; freigegeben per --confirm <token> | 2026-09-01 |
|
||||
| [[MCP-Leseserver]] | architecture | Zweiter Konsument von chemenu ueber MCP: search/types/describe_type/lint/status auf chemenu.api.Corpus, strukturell ohne Schreibpfad, jede Antwort trage einen Commit-Stempel. | 2026-09-02 |
|
||||
| [[Memory Lifecycle]] | architecture | Architektur des Wissenslebenszyklus mit Confidence Scoring, Supersession, Forgetting und Consolidation Tiers zur Pflege von Fakten über die Zeit. | 2026-08-29 |
|
||||
| [[Mesh Sync]] | pattern | Abgleichsmechanismus, der Beobachtungen paralleler Agenten in ein gemeinsames Wiki überführt; Last-Write-Wins mit Konflikterkennung und manuellem Eingriff. | 2026-08-29 |
|
||||
| [[Modbus]] | protocol | Industrielles Kommunikationsprotokoll von 1979 zur Anbindung speicherprogrammierbarer Steuerungen und Geräte über serielle oder TCP-Netze. | 2026-08-29 |
|
||||
@@ -58,6 +61,7 @@
|
||||
| [[Personalization Plane]] | architecture | Schicht fuer Instanz-Identitaet: USER.md/SOUL.md werden als Template ausgeliefert, im Setup-Interview woertlich befuellt und vom doctor-Check auf fehlend wie unbefuellt geprueft | 2026-08-31 |
|
||||
| [[Privacy and Governance]] | workflow | Rahmenwerk zur Absicherung von Wiki-Inhalten über Datenfilterung beim Ingest, Audit-Trail-Protokollierung und umkehrbare Massenoperationen. | 2026-08-29 |
|
||||
| [[Procedural Memory]] | architecture | Langlebigste Speicherschicht für Abläufe, Muster, bewährte Vorgehensweisen und Rezepte, gewonnen aus wiederholten semantischen Beobachtungen. | 2026-08-29 |
|
||||
| [[Publish-Remote Gate]] | workflow | Drittes, im Code durchgesetztes Gate: publish bricht mit Exit 42 ab, wenn die aufgeloeste Push-URL nicht in einer optionalen, gitignoreten Allowlist steht; doctor benennt seit 2026-09-02 den Gate-Zustand statt nur die Dateiexistenz | 2026-09-02 |
|
||||
| [[Quality and Self-Correction]] | workflow | Automatische Qualitätssicherung für Wikis mit Inhaltsbewertung, Selbstheilung und Widerspruchserkennung. | 2026-08-29 |
|
||||
| [[Quality Scoring]] | pattern | Quantitative Bewertung aller vom LLM geschriebenen Inhalte nach struktureller Qualität, Vollständigkeit der Quellenangaben, Konsistenz mit dem Wiki und Themenabdeckung. | 2026-08-29 |
|
||||
| [[RAG]] | architecture | Architekturmuster, bei dem LLMs die Generierung um Dokumente aus einer Wissensbasis anreichern. | 2026-08-29 |
|
||||
|
||||
@@ -3,13 +3,13 @@ type: types/concept.md
|
||||
concept_type: workflow
|
||||
tags: [gate, safety, iteration-budget, loop-breaker]
|
||||
created: 2026-08-07
|
||||
modified: 2026-08-31
|
||||
related: [Mass-Update Gate, Anti-Cramming Heuristic, Index Scaling, wikitool, Structural Enforcement over Documented Rule]
|
||||
sources: [Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]
|
||||
modified: 2026-09-02
|
||||
related: [Mass-Update Gate, Anti-Cramming Heuristic, Index Scaling, wikitool, Structural Enforcement over Documented Rule, MCP-Leseserver]
|
||||
sources: [Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - MCP Read Server Implementation Session 2026-09-02]
|
||||
confidence: 0.88
|
||||
confidence_base: 0.88
|
||||
provenance: sourced
|
||||
summary: Im Code durchgesetzte Obergrenze von 60 wikitool-Aufrufen je Session\, Loop-Breaker bei 3 identischen Wiederholungen\, Slot-Erstattung bei abgelehntem Aufruf und ein seit 1.5.0 gemessenes Kalibrierungsband von 5-15 (einfach) bzw. 20-35 (komplex) Aufrufen
|
||||
summary: Im Code durchgesetzte Obergrenze von 60 wikitool-Aufrufen je Session, Loop-Breaker bei 3 identischen Wiederholungen, Slot-Erstattung, ein gemessenes Kalibrierungsband, und Retrieval sowie der MCP-Leseserver bleiben ausgenommen
|
||||
---
|
||||
# Iteration and Cost Limits
|
||||
|
||||
@@ -48,6 +48,11 @@ Eine hart in Code durchgesetzte Obergrenze für die Anzahl der Tool-Aufrufe, die
|
||||
|
||||
- Einzelne, begrenzte Einmalvorgänge, bei denen die Aufrufen-Anzahl inhärent festgelegt ist (z. B. ein einzelner `new entity`-Aufruf) - das Gate wird dort immer noch gleichmäßig angewendet, wird aber im Wesentlichen nie ausgelöst.
|
||||
- Als Ersatz für das [[Mass-Update Gate]], das auf den *Schadensradius* eines `publish` (Dateien, die von einem einzelnen Push betroffen sind) begrenzt ist, nicht auf die *Iterationsmenge* über eine Sitzung - die beiden Gates beheben unterschiedliche Ausfallmodi und beide bleiben notwendig.
|
||||
- Im [[MCP-Leseserver]]. Das Gate begrenzt eine Agenten-Session am unbemerkten Iterieren über den
|
||||
Wiki-Zustand - deshalb ist Retrieval bereits generell ausgenommen (`SKIP_COMMANDS`) -, nicht
|
||||
einen Nutzer, der oft sucht. Ein zu häufig suchender Nutzer ist ein Ressourcenproblem, das vor
|
||||
den Serverprozess gehört (Rate Limiting), nicht in dieses Gate - beide zu vermischen würde es
|
||||
zu einem Rate Limiter verwässern.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
@@ -63,6 +68,7 @@ Eine hart in Code durchgesetzte Obergrenze für die Anzahl der Tool-Aufrufe, die
|
||||
- **schützt:** [[Index Scaling]]
|
||||
- **implementiert durch:** [[wikitool]]
|
||||
- **wendet an:** [[Structural Enforcement over Documented Rule]]
|
||||
- **nimmt aus:** [[MCP-Leseserver]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
@@ -73,9 +79,11 @@ Eine hart in Code durchgesetzte Obergrenze für die Anzahl der Tool-Aufrufe, die
|
||||
- [[Source - LLM Improvements Production Agent Gaps 2026]]
|
||||
- [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]]
|
||||
- [[Structural Enforcement over Documented Rule]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
[^s-llm-improvements-production-agent-gaps-2026]: [[Source - LLM Improvements Production Agent Gaps 2026]]
|
||||
[^s-conversation-gate-counting-and-measured-calibration-session-2026-08-31]: [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]]
|
||||
[^s-conversation-comma-bug-budget-refund-and-lint-report-path-session-2026-08-31]: [[Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31]]
|
||||
[^s-mcp-read-server-implementation-session-2026-09-02]: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
type: types/concept.md
|
||||
concept_type: architecture
|
||||
tags: [mcp, library-boundary, search, server]
|
||||
created: 2026-09-02
|
||||
modified: 2026-09-02
|
||||
related: [wikitool, Publish-Remote Gate, Mass-Update Gate, Iteration and Cost Limits, Chemenu]
|
||||
sources: [Source - MCP Read Server Implementation Session 2026-09-02]
|
||||
confidence: 0.50
|
||||
confidence_base: 0.50
|
||||
provenance: sourced
|
||||
summary: 'Zweiter Konsument von chemenu ueber MCP: search/types/describe_type/lint/status auf chemenu.api.Corpus, strukturell ohne Schreibpfad, jede Antwort trage einen Commit-Stempel.'
|
||||
---
|
||||
# MCP-Leseserver
|
||||
|
||||
**Typ:** Architecture
|
||||
|
||||
## Definition
|
||||
|
||||
Ein zweiter Konsument desselben Kerns, nicht ein zweites Programm: `tools/chemenu/mcp/` exponiert
|
||||
`search`, `types`, `describe_type`, `lint` und `status` über MCP, indem es dieselben Funktionen
|
||||
aufruft, die `wikitool` auch aufruft - vermittelt durch `chemenu.api.Corpus`, den In-Process-
|
||||
Einstiegspunkt. Ein Golden-Test hält die Ausgaben beider Wege gegeneinander, statt darauf zu
|
||||
vertrauen, dass sie übereinstimmen.
|
||||
|
||||
## Kernpunkte
|
||||
|
||||
- **Kein Schreibpfad, strukturell.** Weder der Server noch `chemenu.api` importiert etwas unter
|
||||
`chemenu.commands` - `new`, `touch`, `xref`, `publish`, `migrate` sind aus diesem Prozess
|
||||
heraus nicht erreichbar, statt aus einer Liste gefiltert zu werden. Ein Test importiert das
|
||||
Servermodul in einem frischen Interpreter und prüft
|
||||
`sys.modules`.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
- **Zwei Transports.** `stdio` zum Entwickeln und Testen ohne Netz; `streamable-http` für die
|
||||
Auslieferung, der einzige, vor den sich ein HTTP-Reverse-Proxy setzen kann. `sse` ist über das
|
||||
SDK erreichbar und wird bewusst nicht angeboten - der abgelöste Remote-Transport, jetzt darauf
|
||||
zu bauen verschiebt den Wechsel nur.
|
||||
- **Jede Antwort trägt den Commit, aus dem sie berechnet wurde** (`commit`, `as_of`). Ein
|
||||
veralteter Checkout antwortet sonst selbstbewusst falsch. `null` heißt: der bediente Baum hat
|
||||
uncommittete Änderungen, die Antwort entspricht keiner Revision. Der Stempel ist die Revision,
|
||||
aus der die Seiten *tatsächlich* gelesen wurden, nicht die zum Zeitpunkt des Stempelns aktuelle
|
||||
- ein Bug, der genau diesen Unterschied überging, wurde beim Schreiben des Golden-Tests selbst
|
||||
gefunden und behoben.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
- **Telemetrie in den bedienten Baum wird beim Start verweigert**, nicht still umgeleitet. Der
|
||||
Sync, der den Checkout aktuell hält (`git fetch && git reset --hard`), darf `reports/telemetry/`
|
||||
wegräumen; ein Trace, der dort landet, wäre ein Verlust und eine stille Möglichkeit, den Baum
|
||||
zu beschmutzen, dessen Sauberkeit der Korpus-Cache prüft.
|
||||
- **Kein Iteration Budget Gate im Server.** Das Gate begrenzt eine Agenten-Session am unbemerkten
|
||||
Iterieren über den Wiki-Zustand, nicht einen Nutzer, der oft sucht - Retrieval ist deshalb
|
||||
bereits generell davon ausgenommen (siehe [[Iteration and Cost Limits]]). Rate Limiting gehört
|
||||
stattdessen vor den Prozess, neben die Authentifizierung.
|
||||
- **Authentifizierung ist Middleware, nicht Servercode.** Eine Traefik-ForwardAuth-Instanz
|
||||
(Bearer-Token gegen SHA-256-Hashes) sitzt vor dem Prozess; nicht sauber authentifizierte
|
||||
Zugriffe erreichen Python gar nicht erst.
|
||||
- **Gemessen:** Korpus-Parse für 176 Seiten 265 ms → 54 ms (`CSafeLoader`),
|
||||
`wikitool search` end-to-end 593 ms → 347 ms; die verbleibenden ~262 ms sind Modulimport und
|
||||
entfallen im residenten Serverprozess, weil er ihn einmal pro Start statt pro Aufruf
|
||||
zahlt.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
|
||||
## Beispiele
|
||||
|
||||
- `search`/`types`/`describe_type`/`lint`/`status` als die fünf Tools - siehe
|
||||
`tools/chemenu/mcp/server.py`.
|
||||
- Der Korpus-Cache (`chemenu/corpus_cache.py`) hält einen Parse pro Commit und cacht nie einen
|
||||
schmutzigen Arbeitsbaum - dieselbe Eigenschaft, die den Antwort-Stempel korrekt hält.
|
||||
- `chemenu.api.Corpus`: nimmt einen Root, liefert exakt die `--json`-Formen der CLI, raised statt
|
||||
zu exitieren.
|
||||
|
||||
## Wann zu verwenden
|
||||
|
||||
- Ein Konsument, der keine Shell auf der bedienenden Maschine ist, soll dieselben Fragen stellen
|
||||
können wie ein Agent, der `wikitool` direkt aufruft.
|
||||
- Mehrere gleichzeitige Leser eines Korpus, für die ein Prozess pro CLI-Aufruf (Modulimport,
|
||||
Korpus-Parse) unnötigen Overhead bedeutet.
|
||||
|
||||
## Wann NICHT zu verwenden
|
||||
|
||||
- Als Ort für einen Schreibpfad - die Ingest-Queue (geplant, Issue #32) ist ein anderes Design
|
||||
mit einer Quarantäne davor, nicht eine Erweiterung dieses Servers.
|
||||
- Als Ersatz für den Iteration Budget Gate oder das Traefik-Rate-Limiting - beide bleiben
|
||||
notwendig und leben an anderer Stelle.
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Mass-Update Gate]]
|
||||
- [[Iteration and Cost Limits]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
[^s-mcp-read-server-implementation-session-2026-09-02]: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
|
||||
## Beziehungen
|
||||
|
||||
- **implementiert:** [[wikitool]]
|
||||
- **verwandt mit:** [[Publish-Remote Gate]]
|
||||
- **verwandt mit:** [[Mass-Update Gate]]
|
||||
- **ausgenommen von:** [[Iteration and Cost Limits]]
|
||||
- **Teil von:** [[Chemenu]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [[wikitool]]
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Mass-Update Gate]]
|
||||
- [[Iteration and Cost Limits]]
|
||||
- [[Chemenu]]
|
||||
- [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
|
||||
@@ -4,7 +4,7 @@ concept_type: workflow
|
||||
tags: [gate, safety, mass-update, confirmation]
|
||||
created: 2026-08-03
|
||||
modified: 2026-09-01
|
||||
related: [Content Quality Control, wikitool, Iteration and Cost Limits, Structural Enforcement over Documented Rule, Bulk Operations]
|
||||
related: [Content Quality Control, wikitool, Iteration and Cost Limits, Structural Enforcement over Documented Rule, Bulk Operations, Publish-Remote Gate, MCP-Leseserver]
|
||||
sources: [Source - LLM Improvements Sonnet Analysis, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - LLM Improvements Codex Analysis]
|
||||
confidence: 0.88
|
||||
confidence_base: 0.88
|
||||
@@ -77,6 +77,8 @@ Das Mass-Update Gate ist ein Sicherheitsmechanismus, der die Ausführung pausier
|
||||
- **wird gespiegelt durch:** [[Iteration and Cost Limits]]
|
||||
- **wendet an:** [[Structural Enforcement over Documented Rule]]
|
||||
- **grenzt ab gegen:** [[Bulk Operations]]
|
||||
- **verwandtes Gate:** [[Publish-Remote Gate]]
|
||||
- **verwandt mit:** [[MCP-Leseserver]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
@@ -86,6 +88,8 @@ Das Mass-Update Gate ist ein Sicherheitsmechanismus, der die Ausführung pausier
|
||||
- [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]]
|
||||
- [[Structural Enforcement over Documented Rule]]
|
||||
- [[Bulk Operations]]
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
type: types/concept.md
|
||||
concept_type: workflow
|
||||
tags: []
|
||||
created: 2026-09-01
|
||||
modified: 2026-09-02
|
||||
related: [Mass-Update Gate, Chemenu, MCP-Leseserver]
|
||||
sources: [Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01, Source - MCP Read Server Implementation Session 2026-09-02]
|
||||
confidence: 0.70
|
||||
confidence_base: 0.70
|
||||
provenance: sourced
|
||||
summary: 'Drittes, im Code durchgesetztes Gate: publish bricht mit Exit 42 ab, wenn die aufgeloeste Push-URL nicht in einer optionalen, gitignoreten Allowlist steht; doctor benennt seit 2026-09-02 den Gate-Zustand statt nur die Dateiexistenz'
|
||||
---
|
||||
# Publish-Remote Gate
|
||||
|
||||
**Typ:** Workflow
|
||||
|
||||
## Definition
|
||||
|
||||
Ein im Code durchgesetztes Gate, das einen Schreibvorgang (hier: `publish`) auf ein
|
||||
deklariertes, per Datei zugelassenes Ziel beschränkt. Der Vorgang bricht ab, wenn das
|
||||
aufgelöste Push-Ziel nicht in der Allowlist steht - unabhängig davon, unter welchem Namen der
|
||||
Remote lokal konfiguriert ist.
|
||||
|
||||
## Kernpunkte
|
||||
|
||||
- **Geprüft wird die aufgelöste URL, nicht der Remote-Name.** Ein namensbasiertes Gate würde
|
||||
ein `publish` durchlassen, dessen `origin` zwischenzeitlich auf ein anderes Ziel umgebogen
|
||||
wurde - genau der Fall, den das Gate abfangen soll.
|
||||
- **Kein Freigabe-Token, anders als vergleichbare Gates.** Ein Gate, dessen Frage per
|
||||
Änderungssatz beantwortbar ist ("ist diese konkrete Änderung richtig?"), kann sich mit einem
|
||||
Token lösen, den ein Mensch einmalig ausstellt. Ein Gate, dessen Frage eine stehende
|
||||
Eigenschaft des Checkouts ist ("gehört dieser Inhalt grundsätzlich in dieses Ziel?"), sollte
|
||||
keinen Token haben - der einzige Weg daran vorbei ist ein bewusster Edit der Konfigurationsdatei
|
||||
durch den Menschen, nie ein automatisierter Bypass.
|
||||
- **Die Allowlist-Datei ist per Checkout, nicht Teil des versionierten Inhalts.** Sie
|
||||
beschreibt, wohin *dieser* Checkout schreiben darf - eine committete Kopie würde jedem Klon
|
||||
dieselbe Erlaubnis unterschieben, unabhängig davon, ob sie für ihn zutrifft.
|
||||
- **Fehlende Datei bedeutet unbeschränkt, kaputte Datei bedeutet Fehler.** Diese Unterscheidung
|
||||
ist wichtig: Ein Checkout ohne Beschränkungsbedarf soll nicht gezwungen sein, eine leere
|
||||
Konfigurationsdatei zu pflegen; eine beschädigte Datei darf aber nicht wie eine abwesende
|
||||
behandelt werden, sonst wird eine defekte Sicherung zu einer stillschweigend abgeschalteten.
|
||||
- **`doctor` benennt seit 2026-09-02 den Gate-*Zustand*, nicht nur, ob die Datei existiert.**
|
||||
Vorher meldete der Check nur die Anwesenheit von `.wikitool-remotes.json`; ob das
|
||||
gleichbedeutend mit "scharf" ist, musste der Leser selbst schließen. Alle drei Ausgaben
|
||||
beginnen jetzt mit `Gate armed:` bzw. `Gate not armed:` - der Ein-Remote-Fall ohne Allowlist
|
||||
bleibt `OK` (er hat nichts zu schützen), sagt aber ausdrücklich, dass jedes Push-Ziel
|
||||
durchkommt.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
|
||||
## Wann zu verwenden
|
||||
|
||||
- Ein Checkout kann an mehr als ein Remote-Ziel schreiben, und ein Schreibvorgang an das
|
||||
falsche Ziel ist teuer oder nicht rückgängig zu machen (z. B. weil das Ziel öffentlich ist).
|
||||
- Die Menge der zulässigen Ziele ist eine stabile Eigenschaft des Checkouts, keine
|
||||
Einzelfallentscheidung pro Vorgang.
|
||||
|
||||
## Wann NICHT zu verwenden
|
||||
|
||||
- Wenn nur ein Remote existiert und kein Risiko einer Zielverwechslung besteht - dort ist die
|
||||
Allowlist reine Formalität ohne Schutzwirkung.
|
||||
- Für Entscheidungen, die tatsächlich pro Änderungssatz getroffen werden sollen (dafür ist ein
|
||||
Token-basiertes Gate wie das Mass-Update-Gate das richtige Muster).
|
||||
|
||||
## Was das Gate nicht abdeckt: Inhalt, der über einen Merge hereinkommt
|
||||
|
||||
Das Gate schützt den **Push**, nicht den **Merge**. Ein Setup, bei dem eine private Instanz
|
||||
Maschinerie von einem öffentlichen Upstream per `git merge upstream/main` zieht, hat ein
|
||||
eigenes, empirisch geprüftes Problem: Ein einfacher Merge übernimmt Upstream-Änderungen an
|
||||
bereits gelöschten Inhaltsseiten nicht sauber.
|
||||
|
||||
Gemessen an einem Wegwerf-Repo-Paar, bei dem der Upstream nach der einmaligen Löschung des
|
||||
Demo-Korpus eine Seite ändert, eine neue anlegt und eine dritte löscht:
|
||||
|
||||
- Eine **geänderte** Seite erzeugt einen `modify/delete`-Konflikt und lässt die
|
||||
Upstream-Fassung im Arbeitsbaum liegen - ein naives Auflösen mit `git add -A` holt sie zurück.
|
||||
- Eine **neu angelegte** Seite wird **stillschweigend** übernommen, ohne Konflikt und ohne
|
||||
Meldung.
|
||||
- Eine beidseitig gelöschte Seite verursacht nichts - der einzige Fall, der ohne Weiteres
|
||||
funktioniert.
|
||||
|
||||
Ein naheliegender Fix (`.gitattributes` mit `merge=ours` für die betroffenen Verzeichnisse)
|
||||
wurde ebenfalls gemessen und verworfen: Der Treiber wirkt nur bei Inhaltskonflikten auf
|
||||
beidseitig vorhandenen Dateien, nicht bei modify/delete-Paaren oder Neuanlagen.
|
||||
|
||||
Die funktionierende Prozedur hält den Merge mit `--no-commit` offen, erzwingt die
|
||||
Inhaltsverzeichnisse zurück auf den Stand vor dem Merge, solange `HEAD` noch dorthin zeigt, und
|
||||
prüft danach explizit (`git diff --name-only $BEFORE HEAD -- kb raw` muss leer sein) - eine
|
||||
Kontrolle, die nicht stillschweigend übersprungen werden kann, anders als eine bloße Behauptung
|
||||
im Text[^s-private-instance-merge-correction-and-issue-30-session-2026-09-01]. Details, ein
|
||||
getestetes Skript und zwei Architekturvorschläge (das Verfahren als `wikitool`-Kommando bauen,
|
||||
oder den Demo-Korpus grundsätzlich von dem Branch fernhalten, von dem private Instanzen ihre
|
||||
Maschinerie ziehen) stehen in Gitea-Issue #30.
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Mass-Update Gate]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Beziehungen
|
||||
|
||||
- **gilt fuer:** [[Chemenu]]
|
||||
- **verwandtes Gate:** [[Mass-Update Gate]]
|
||||
- **verwandt mit:** [[MCP-Leseserver]]
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
|
||||
- [[Chemenu]]
|
||||
- [[Mass-Update Gate]]
|
||||
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
[^s-private-instance-merge-correction-and-issue-30-session-2026-09-01]: [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
[^s-mcp-read-server-implementation-session-2026-09-02]: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
@@ -19,7 +19,7 @@
|
||||
|------|------|---------|----------------|
|
||||
| [[andybalholm-edl]] | project | Go-basierte EDL-Bibliothek für die Kommunikation mit eingebetteten Geräten. | 2026-08-29 |
|
||||
| [[BCDModule]] | project | Go-Modul, das die Entscheidungslogik für die Batterieladung umsetzt. | 2026-08-29 |
|
||||
| [[Chemenu]] | project | Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich als Testbett und Demo unter AGPL-3.0/CC-BY-4.0 | 2026-09-01 |
|
||||
| [[Chemenu]] | project | Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich, seit 2.4.0 mit einem MCP-Leseserver als zweitem Konsumenten | 2026-09-02 |
|
||||
| [[goresponsiveness]] | project | Go-Werkzeug zur Messung von Anwendungsleistung und Responsiveness. | 2026-08-29 |
|
||||
| [[ha-core]] | project | Kern-Integrationsbibliothek für Home-Assistant-E3DC-Systeme; stellt die E3DC-Kommunikationsprotokolle und den Home-Assistant-Integrationscode bereit. | 2026-08-29 |
|
||||
| [[hacs-e3dc]] | project | Home Assistant Custom Component zur Überwachung von E3DC-Energiesystemen. | 2026-08-29 |
|
||||
@@ -96,8 +96,8 @@
|
||||
| [[pascalandy schema]] | tool | Von der Community beigesteuertes Wiki Schema (Global) aus pascalandys Kommentar in Farzas Gist, mit alternativer Tag-Taxonomie (area/kind/topic/status/pty) | 2026-08-29 |
|
||||
| [[Pi]] | tool | Assistenz-Agent von Inflection AI; kann das LLM-Wiki-Muster wie andere LLM-Agenten umsetzen. | 2026-08-29 |
|
||||
| [[Proton]] | tool | Wine-basierte Kompatibilitätsschicht von Valve; lässt Windows-Spiele über Steam unter Linux laufen, mit optimierter DirectX-Übersetzung. | 2026-08-29 |
|
||||
| [[qmd]] | tool | Lokale Suchmaschine für Markdown-Dateien mit hybrider BM25-Vektor-Suche und LLM-Reranking. | 2026-08-29 |
|
||||
| [[qmd]] | tool | Lokale Suchmaschine fuer Markdown-Dateien: TypeScript/Node.js/Bun, SQLite-FTS5-BM25 plus sqlite-vec-Vektorsuche plus node-llama-cpp-LLM-Reranking. | 2026-09-02 |
|
||||
| [[Steam]] | tool | Valves Plattform für digitalen Spielevertrieb und Spielebibliothek auf dem PC. | 2026-08-29 |
|
||||
| [[wikitool]] | tool | Deterministisches CLI fuer alle mechanischen Wiki-Operationen; seit 2.0.0 liegt es im Python-Paket chemenu, das Kommando heisst weiterhin wikitool | 2026-09-01 |
|
||||
| [[wikitool]] | tool | Deterministisches CLI fuer alle mechanischen Wiki-Operationen; seit 2.0.0 im Paket chemenu, seit 2.4.0 zusaetzlich als MCP-Leseserver erreichbar | 2026-09-02 |
|
||||
| [[Wine]] | tool | Kompatibilitätsschicht, die Windows-API-Aufrufe nach POSIX übersetzt und Windows-Anwendungen unter Linux, BSD und macOS ohne Virtualisierung oder Emulation ausführt. | 2026-08-29 |
|
||||
|
||||
|
||||
@@ -3,13 +3,13 @@ type: types/entity.md
|
||||
entity_type: project
|
||||
tags: [wiki, llm, knowledge-base]
|
||||
created: 2026-08-04
|
||||
modified: 2026-09-01
|
||||
related: [Personalization Plane, Issue Label Scheme, Optional Instance Context File]
|
||||
sources: [Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]
|
||||
modified: 2026-09-02
|
||||
related: [Personalization Plane, Issue Label Scheme, Optional Instance Context File, Delete Rather Than Anonymize, Dual Licensing by File Plan, Publish-Remote Gate, MCP-Leseserver]
|
||||
sources: [Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, 'Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01, Source - MCP Read Server Implementation Session 2026-09-02]
|
||||
confidence: 0.90
|
||||
confidence_base: 0.90
|
||||
provenance: mixed
|
||||
summary: Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich als Testbett und Demo unter AGPL-3.0/CC-BY-4.0
|
||||
summary: Deterministischer Wissenskompiler (raw/ -> kb/); seit 2.0.0 unter dem Namen Chemenu, seit 2026-09-01 oeffentlich, seit 2.4.0 mit einem MCP-Leseserver als zweitem Konsumenten
|
||||
---
|
||||
# Chemenu
|
||||
|
||||
@@ -60,6 +60,10 @@ Das Repository hat bereits ein deterministisches CLI, `tools/wikitool` (Python,
|
||||
- **verwendet:** [[Personalization Plane]]
|
||||
- **verwendet:** [[Issue Label Scheme]]
|
||||
- **verwendet:** [[Optional Instance Context File]]
|
||||
- **wendet an:** [[Delete Rather Than Anonymize]]
|
||||
- **wendet an:** [[Dual Licensing by File Plan]]
|
||||
- **setzt um:** [[Publish-Remote Gate]]
|
||||
- **enthaelt:** [[MCP-Leseserver]]
|
||||
|
||||
## Details
|
||||
|
||||
@@ -139,6 +143,13 @@ Repositorys selbst und keine Aussagen aus einer Rohdatenquelle; der Änderungsda
|
||||
|
||||
## Historie
|
||||
|
||||
- 2026-09-02 - `2.2.3`-`2.4.1`: Issue #36s vierstufige Sequenz zum [[MCP-Leseserver]] umgesetzt
|
||||
([[Publish-Remote Gate]] scharf gestellt, Lesepfad gehärtet, Root-Auflösung von der Importzeit
|
||||
gelöst, der Server selbst), dazu `INSTALL-MCP.md` und Issue #37 für das noch fehlende
|
||||
Container-Image. Alle vier Sequenz-Issues geschlossen; #36 blieb offen, weil sein eigenes
|
||||
Abschlusskriterium - ein Konsument, der nachweislich über die
|
||||
Authentifizierungs-Middleware antwortet - erst mit #37 erfüllbar
|
||||
ist.[^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
- 2026-09-01 - `2.0.0` (Commit `9a7abe6`, 121 Dateien, 730 Tests grün): Rebranding von
|
||||
`llm-wiki-test1` auf **Chemenu** nach Gitea-Issue #3 - Repo-Rename, Produktname,
|
||||
Release-Artefakt (`chemenu-stack-<version>.tar.gz`), Release-Feed, und das Python-Paket
|
||||
@@ -210,6 +221,13 @@ Repositorys selbst und keine Aussagen aus einer Rohdatenquelle; der Änderungsda
|
||||
- [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]]
|
||||
- [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]]
|
||||
- [[Optional Instance Context File]]
|
||||
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
- [[Delete Rather Than Anonymize]]
|
||||
- [[Dual Licensing by File Plan]]
|
||||
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
@@ -221,3 +239,4 @@ Repositorys selbst und keine Aussagen aus einer Rohdatenquelle; der Änderungsda
|
||||
[^s-conversation-gate-counting-and-measured-calibration-session-2026-08-31]: [[Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31]]
|
||||
[^s-conversation-write-once-frontmatter-fields-and-touch-set-session-2026-08-31]: [[Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31]]
|
||||
[^s-conversation-comma-bug-budget-refund-and-lint-report-path-session-2026-08-31]: [[Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31]]
|
||||
[^s-mcp-read-server-implementation-session-2026-09-02]: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
|
||||
+22
-11
@@ -3,13 +3,13 @@ type: types/entity.md
|
||||
entity_type: tool
|
||||
tags: [search, markdown, cli, local]
|
||||
created: 2026-07-26
|
||||
modified: 2026-08-29
|
||||
modified: 2026-09-02
|
||||
related: [Obsidian, LLM Wiki Pattern]
|
||||
sources: [Source - LLM Wiki Pattern]
|
||||
confidence: 0.85
|
||||
confidence_base: 0.85
|
||||
sources: [Source - LLM Wiki Pattern, Source - qmd - GitHub Repository]
|
||||
confidence: 0.80
|
||||
confidence_base: 0.80
|
||||
provenance: sourced
|
||||
summary: Lokale Suchmaschine für Markdown-Dateien mit hybrider BM25-Vektor-Suche und LLM-Reranking.
|
||||
summary: 'Lokale Suchmaschine fuer Markdown-Dateien: TypeScript/Node.js/Bun, SQLite-FTS5-BM25 plus sqlite-vec-Vektorsuche plus node-llama-cpp-LLM-Reranking.'
|
||||
---
|
||||
# qmd
|
||||
|
||||
@@ -22,11 +22,12 @@ qmd ist eine lokale Suchmaschine, die speziell für Markdown-Dateien ausgelegt i
|
||||
## Kerndaten
|
||||
|
||||
- **Typ:** CLI-Tool
|
||||
- **Sprache:** Nicht angegeben (wahrscheinlich Go oder Rust)
|
||||
- **Such-Typen:** Hybrid (BM25 + Vector)
|
||||
- **Neu-Ranking:** LLM-basiert
|
||||
- **Sprache:** TypeScript, Laufzeit Node.js oder Bun
|
||||
- **Such-Typen:** Hybrid (SQLite-FTS5-BM25 + `sqlite-vec`-Vektorsuche)
|
||||
- **Neu-Ranking:** LLM-basiert über `node-llama-cpp` (lokale GGUF-Modelle)
|
||||
- **Bereitstellung:** On-device/lokal
|
||||
- **Repository:** https://github.com/tobi/qmd
|
||||
- **Installation:** `npm install -g @tobilu/qmd` oder `bun install -g @tobilu/qmd`
|
||||
|
||||
## Beziehungen
|
||||
|
||||
@@ -50,11 +51,13 @@ qmd ist eine lokale Suchmaschine, die speziell für Markdown-Dateien ausgelegt i
|
||||
## Installation und Verwendung
|
||||
|
||||
```bash
|
||||
# Installation (hypothetisch, siehe aktuelles Repo für Details)
|
||||
go install github.com/tobi/qmd@latest
|
||||
# Installation (Node oder Bun)
|
||||
npm install -g @tobilu/qmd
|
||||
# oder
|
||||
bun install -g @tobilu/qmd
|
||||
|
||||
# Suche von CLI
|
||||
qmd search "knowledge management"
|
||||
qmd query "knowledge management"
|
||||
|
||||
# Als MCP-Server für LLM-Integration verwenden
|
||||
qmd server
|
||||
@@ -74,9 +77,17 @@ qmd server
|
||||
|
||||
## Historie
|
||||
|
||||
- [2026-09-02] - Sprachangabe korrigiert. Die Seite trug seit ihrer Erstellung "Nicht angegeben
|
||||
(wahrscheinlich Go oder Rust)" bei `confidence: 0.85` - eine Vermutung mit einer Konfidenz, die
|
||||
für eine Vermutung zu hoch war. Direkt gegen das Repository geprüft: TypeScript, Laufzeit
|
||||
Node.js/Bun, SQLite-FTS5-BM25 + `sqlite-vec` + `node-llama-cpp`.[^s-qmd-github-repository]
|
||||
- [2026-07-26] - Entity-Seite während der Aufnahme des LLM-Wiki-Pattern-Artikels erstellt
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [[LLM Wiki Pattern]]
|
||||
- [[Obsidian]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
[^s-qmd-github-repository]: [[Source - qmd - GitHub Repository]]
|
||||
|
||||
@@ -3,13 +3,13 @@ type: types/entity.md
|
||||
entity_type: tool
|
||||
tags: [cli, automation, deterministic, wiki-management]
|
||||
created: 2026-08-03
|
||||
modified: 2026-09-01
|
||||
related: [Semantic Lint Automation, Session Orientation, Iteration and Cost Limits, KB Stack Versioning, KB Migration, Personalization Plane, Detect-Repair Asymmetry, Write-Once Frontmatter Fields, Denylist over Allowlist, Command Round-Trip Integrity, Green Suite Blind Spot, Ambient Environment Dependency, Structural Enforcement over Documented Rule, Optional Instance Context File]
|
||||
sources: [Source - LLM Improvements Codex Analysis, Source - LLM Improvements Sonnet Analysis, Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Auto Mode and Tool Choice Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31]
|
||||
modified: 2026-09-02
|
||||
related: [Semantic Lint Automation, Session Orientation, Iteration and Cost Limits, KB Stack Versioning, KB Migration, Personalization Plane, Detect-Repair Asymmetry, Write-Once Frontmatter Fields, Denylist over Allowlist, Command Round-Trip Integrity, Green Suite Blind Spot, Ambient Environment Dependency, Structural Enforcement over Documented Rule, Optional Instance Context File, MCP-Leseserver]
|
||||
sources: [Source - LLM Improvements Codex Analysis, Source - LLM Improvements Sonnet Analysis, Source - Copilot Skill Restructure Instructions, Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04, Source - LLM Improvements Production Agent Gaps 2026, Source - Conversation - Versioning CI-CD and Content Migration Session 2026-08-30, Source - Conversation - Comma Bug Budget Refund and Lint Report Path Session 2026-08-31, Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31, Source - Conversation - Auto Mode and Tool Choice Session 2026-08-31, Source - Conversation - Write-Once Frontmatter Fields and touch --set Session 2026-08-31, Source - Conversation - Gate Counting and Measured Calibration Session 2026-08-31, Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31, Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31, Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31, 'Source - Public Release, Corpus Purge and History Squash Session 2026-09-01', Source - Publish-Remote Gate and Issue Triage Session 2026-09-01, Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01, Source - MCP Read Server Implementation Session 2026-09-02]
|
||||
confidence: 0.90
|
||||
confidence_base: 0.90
|
||||
provenance: sourced
|
||||
summary: Deterministisches CLI fuer alle mechanischen Wiki-Operationen; seit 2.0.0 liegt es im Python-Paket chemenu, das Kommando heisst weiterhin wikitool
|
||||
summary: Deterministisches CLI fuer alle mechanischen Wiki-Operationen; seit 2.0.0 im Paket chemenu, seit 2.4.0 zusaetzlich als MCP-Leseserver erreichbar
|
||||
---
|
||||
# wikitool
|
||||
|
||||
@@ -51,6 +51,7 @@ Wie in der Codex-Analyse vermerkt, bietet wikitool die deterministische Grundlag
|
||||
- **zeigte:** [[Ambient Environment Dependency]]
|
||||
- **setzt um:** [[Structural Enforcement over Documented Rule]]
|
||||
- **setzt um:** [[Optional Instance Context File]]
|
||||
- **implementiert:** [[MCP-Leseserver]]
|
||||
|
||||
## Befehle
|
||||
|
||||
@@ -73,6 +74,9 @@ wikitool bietet die folgenden Befehlskategorien:
|
||||
Seitenkopf zurückgefaltet statt abgelehnt, und weil der Block immer zuletzt gerendert wird,
|
||||
richtet die erste Zitatoperation eine verrutschte Seite von selbst wieder ein[^s-conversation-two-round-trip-defects-found-by-an-ingest-session-2026-08-31]
|
||||
- **Abfrage:** `search` - Textsuche über `kb/` durch ein austauschbares Backend (`rg` heute), plus `--field`-Prädikate, die auf Frontmatter evaluiert werden (`entity_type=system`, `confidence>=0.8`, `tags=k8s`, `!source_url`). Ohne Text ist es eine reine strukturierte Abfrage. Schreibgeschützt und ausgenommen von der Iteration-Budget-Gate, da Abfrage das Lesen statt das Iterieren ist
|
||||
- **MCP-Server:** `python -m chemenu.mcp` (`--transport stdio|streamable-http`) - kein
|
||||
`wikitool`-Unterbefehl, sondern ein zweiter Einstiegspunkt über `chemenu.api.Corpus`, derselbe
|
||||
Kern wie `search`/`types`/`lint`. Siehe [[MCP-Leseserver]]
|
||||
- **Indizierung:** `index rebuild` - regeneriert die `kb/index.md`-Map plus eine pro-Sammlung `INDEX.md`, wobei ein Bereich bei 50 Zeilen in seine eigene Shard aufgeteilt wird
|
||||
- **Herkunft:** `sources coverage`, `sources trace`, `sources rebuild-index`
|
||||
- **Protokollierung:** `log append`, `log status`
|
||||
@@ -147,6 +151,37 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
|
||||
|
||||
## Historie
|
||||
|
||||
- 2026-09-02 - `2.2.3`-`2.4.1`: die vierstufige Sequenz aus Issue #36 (Publish-Remote-Gate
|
||||
scharf, Lesepfad gehärtet, Root-Auflösung und Bibliotheksgrenze, [[MCP-Leseserver]]) plus
|
||||
Menschendoku. `.wikitool-remotes.json` war trotz gegenteiliger Dokumentation nicht gesetzt -
|
||||
angelegt und gegen ein erfundenes Ziel gegengeprüft (Exit 42). Sechs Lesepfad-Härtungen:
|
||||
gelöschter ReDoS-Zweig in der Ranking-Funktion, ein 30-Sekunden-`rg`-Timeout, ein
|
||||
YAML-Alias-Refusal (267 B → 672.603 Knoten reproduziert), `CSafeLoader` (265 ms → 54 ms),
|
||||
gemeldetes statt geschlucktes kaputtes Frontmatter, ein Korpus-Cache, der nie einen
|
||||
schmutzigen Arbeitsbaum cacht. `config.ROOT` und alle abgeleiteten Pfade waren zur Importzeit
|
||||
gebunden - jetzt lazy aufgelöst (`$CHEMENU_ROOT` → Walk-up), mit einem eigenen
|
||||
`config.reset()` gegen eine Falle, in die `monkeypatch`s eigener Teardown sonst zurückläuft.
|
||||
Neuer typerloser Lesekern (`search/service.py`, `lint_core.py`, `types_core.py`) hinter
|
||||
`chemenu.api.Corpus`, darüber der MCP-Server: `search`/`types`/`describe_type`/`lint`/`status`,
|
||||
strukturell ohne Schreibpfad, jede Antwort mit Commit-Stempel. `wikitool search` end-to-end
|
||||
593 ms → 347 ms. Menschendoku `INSTALL-MCP.md`, ausgeliefert über `dist_cmd.ROOT_FILES`.
|
||||
Siehe [[Publish-Remote Gate]], [[Mass-Update Gate]],
|
||||
[[MCP-Leseserver]][^s-mcp-read-server-implementation-session-2026-09-02]
|
||||
- 2026-09-01 - `2.1.0`-`2.2.2`: Vorbereitung und Durchführung der Veröffentlichung. `dist export`
|
||||
bekommt `REQUIRED_ROOT_FILES` (fehlende Lizenzdateien lassen den Export scheitern statt still
|
||||
weiterzulaufen) und `find_leaks()` (strukturelle Prüfung des fertigen Export-Plans gegen
|
||||
Personalisierungsdateien, `instructions/dev/`, `kb/`- und `raw/`-Inhalte - bewusst kein
|
||||
Text-Muster-Scan, weil der eigene Hostname legitim in `INSTALL.md`/`version.py` steht). `publish`
|
||||
bekommt ein drittes Gate: das **Publish-Remote-Gate** prüft die aufgelöste Push-URL (nicht den
|
||||
Remote-Namen) gegen eine optionale, gitignorete `.wikitool-remotes.json` und hat als einziges
|
||||
der drei Gates keinen `--confirm`-Token - der Weg daran vorbei ist ein bewusster Edit der
|
||||
Datei durch den Nutzer, nie durch einen Agenten. `doctor` bekommt den `publish-remotes`-Check.
|
||||
Ein `raw_dir`-Testfixture löste `config.ROOT` gegen das echte Repo-Root statt die Fixture auf
|
||||
und bestand nur, weil zufällig ein Verzeichnis existierte, das der Korpus-Schnitt entfernte -
|
||||
in der Fixture geschlossen, nicht im einzelnen Test, aus demselben Grund wie Gitea #8.
|
||||
Siehe [[Chemenu]], [[Mass-Update Gate]], [[Publish-Remote Gate]],
|
||||
[[Delete Rather Than Anonymize]],
|
||||
[[Dual Licensing by File Plan]][^s-public-release-corpus-purge-and-history-squash-session-2026-09-01][^s-publish-remote-gate-and-issue-triage-session-2026-09-01]
|
||||
- 2026-09-01 - `2.0.0` (Commit `9a7abe6`, 730 Tests grün): das Python-Paket heißt `chemenu`
|
||||
statt `wiki_tools`, **das Kommando bleibt `wikitool`**. Der Import-Name eines Pakets ist ein
|
||||
flacher globaler Namensraum ohne Kollisionsschutz, `wiki_tools` war dafür zu generisch;
|
||||
@@ -249,6 +284,10 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
|
||||
- [[Structural Enforcement over Documented Rule]]
|
||||
- [[Optional Instance Context File]]
|
||||
- [[Source - Conversation - ENVIRONMENT.md as an Optional Third Session-Level File Session 2026-08-31]]
|
||||
- [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
- [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
|
||||
- [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
- [[MCP-Leseserver]]
|
||||
|
||||
## Fußnoten
|
||||
|
||||
@@ -261,3 +300,6 @@ ist[^s-conversation-versioning-ci-cd-and-content-migration-session-2026-08-30].
|
||||
[^s-conversation-agents-md-skill-restructuring-session-2026-08-04]: [[Source - Conversation - AGENTS.md Skill Restructuring Session 2026-08-04]]
|
||||
[^s-conversation-hardening-the-test-suite-against-silent-environment-dependencies-session-2026-08-31]: [[Source - Conversation - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31]]
|
||||
[^s-llm-improvements-codex-analysis]: [[Source - LLM Improvements Codex Analysis]]
|
||||
[^s-public-release-corpus-purge-and-history-squash-session-2026-09-01]: [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
[^s-publish-remote-gate-and-issue-triage-session-2026-09-01]: [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
|
||||
[^s-mcp-read-server-implementation-session-2026-09-02]: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
|
||||
+6
-6
@@ -13,12 +13,12 @@ The page tables live in a generated `INDEX.md` inside each collection, linked be
|
||||
|
||||
## Statistics
|
||||
|
||||
- **Total Pages:** 170
|
||||
- **Total Pages:** 179
|
||||
- **Comparisons:** 1
|
||||
- **Concepts:** 76
|
||||
- **Concepts:** 80
|
||||
- **Entities:** 72
|
||||
- **Sources:** 21
|
||||
- **Last Updated:** 2026-09-01
|
||||
- **Sources:** 26
|
||||
- **Last Updated:** 2026-09-02
|
||||
|
||||
---
|
||||
|
||||
@@ -27,9 +27,9 @@ The page tables live in a generated `INDEX.md` inside each collection, linked be
|
||||
| Collection | Pages | Index |
|
||||
|------------|------:|-------|
|
||||
| `comparisons/` | 1 | [comparisons/INDEX.md](comparisons/INDEX.md) |
|
||||
| `concepts/` | 76 | [concepts/INDEX.md](concepts/INDEX.md) |
|
||||
| `concepts/` | 80 | [concepts/INDEX.md](concepts/INDEX.md) |
|
||||
| `entities/` | 72 | [entities/INDEX.md](entities/INDEX.md) |
|
||||
| `sources/` | 21 | [sources/INDEX.md](sources/INDEX.md) |
|
||||
| `sources/` | 26 | [sources/INDEX.md](sources/INDEX.md) |
|
||||
|
||||
### entities/
|
||||
|
||||
|
||||
@@ -55,3 +55,39 @@ und `doctor` grün.
|
||||
Die Seite beschrieb sich als persoenliches IT-Wissenswiki; seit der Veroeffentlichung ist diese Instanz Testbett und oeffentliche Demo. Beschreibung, Zweck und Lizenz nachgezogen, die historische Aussage ueber die monolithische AGENTS.md als Ausgangspunkt datiert statt geloescht.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
|
||||
|
||||
Source-Seite angelegt, Chemenu und wikitool aktualisiert, zwei neue Concept-Seiten (Delete Rather Than Anonymize, Dual Licensing by File Plan), quer verlinkt.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
|
||||
|
||||
Source-Seite angelegt, neue Concept-Seite Publish-Remote Gate, wikitool-Historie ergaenzt, quer verlinkt mit Mass-Update Gate und Chemenu.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-01] ingest | raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
|
||||
|
||||
Source-Seite angelegt, Publish-Remote-Gate-Seite um die gemessene Merge-Semantik erweitert, quer verlinkt.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-02] ingest | MCP Read Server Implementation Session 2026-09-02
|
||||
|
||||
Source-Seite und neues Concept MCP-Leseserver angelegt; wikitool, Chemenu, Publish-Remote Gate und Iteration and Cost Limits aktualisiert. Deckt Issue #36s Sequenz (#34/#33/#31/#19), INSTALL-MCP.md und Issue #37 ab.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-02] update | qmd - Sprachangabe korrigiert
|
||||
|
||||
Direkt gegen tobi/qmd auf GitHub geprueft: TypeScript statt der geratenen 'Go oder Rust'-Angabe. Neue Quelle Source - qmd - GitHub Repository (raw/documents/), confidence_base 0.85 -> 0.80 neu begruendet, Installationsbeispiel korrigiert.
|
||||
|
||||
---
|
||||
|
||||
## [2026-09-02] update | MCP Read Server Implementation Session 2026-09-02 - Korrektur
|
||||
|
||||
Aufgaben-Checkbox fuer die qmd.md-Korrektur nachgezogen (war [ ], ist erledigt) und ein Korrektur-Hinweis ergaenzt: der Fidelity-Block des Rohtranskripts kuendigte ein zweites Transkript fuer die qmd-Korrektur an, das nie geschrieben wurde - die Korrektur lief stattdessen als eigene Quellen-Verifikation. raw/ ist unveraenderlich, die Korrektur steht deshalb auf der Source-Seite.
|
||||
|
||||
---
|
||||
|
||||
+27
-2
@@ -8,8 +8,8 @@ inline `[^cite-id]` footnote).
|
||||
|
||||
## Coverage Summary
|
||||
|
||||
- **Total raw files:** 21
|
||||
- **Covered:** 21
|
||||
- **Total raw files:** 26
|
||||
- **Covered:** 26
|
||||
- **Uncovered:** 0
|
||||
|
||||
---
|
||||
@@ -31,6 +31,11 @@ inline `[^cite-id]` footnote).
|
||||
- Covered by: [[Source - LLM Wiki Pattern]]
|
||||
- Cited by: [[Andrej Karpathy]], [[ChatGPT]], [[Claude Code]], [[Dataview]], [[Knowledge Compounding]], [[LLM Wiki Pattern]], [[Marp]], [[Memex]], [[NotebookLM]], [[Obsidian]], [[Obsidian Web Clipper]], [[OpenAI Codex]], [[OpenCode]], [[Pi]], [[RAG]], [[Three-Layer Architecture]], [[Tolkien Gateway]], [[Vannevar Bush]], [[qmd]]
|
||||
|
||||
### `raw/documents/qmd - GitHub Repository.md`
|
||||
|
||||
- Covered by: [[Source - qmd - GitHub Repository]]
|
||||
- Cited by: [[qmd]]
|
||||
|
||||
### `raw/notes/Arch Linux Cheat Sheet.md`
|
||||
|
||||
- Covered by: [[Source - Arch Linux Cheat Sheet]]
|
||||
@@ -71,11 +76,31 @@ inline `[^cite-id]` footnote).
|
||||
- Covered by: [[Source - Conversation - Issue Triage Labels and TODO Retirement Session 2026-08-31]]
|
||||
- Cited by: [[Chemenu]], [[Detect-Repair Asymmetry]], [[Gitea]], [[Gitea Actions]], [[Gitea MCP Server]], [[Issue Label Scheme]], [[KB Stack Versioning]], [[wikitool]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - MCP Read Server Implementation Session 2026-09-02.md`
|
||||
|
||||
- Covered by: [[Source - MCP Read Server Implementation Session 2026-09-02]]
|
||||
- Cited by: [[Chemenu]], [[Iteration and Cost Limits]], [[MCP-Leseserver]], [[Publish-Remote Gate]], [[wikitool]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - Nightly Drift-Check Workflow and doctor's Bootstrap Gap Session 2026-08-31.md`
|
||||
|
||||
- Covered by: [[Source - Conversation - Nightly Drift-Check Workflow and doctor's Bootstrap Gap Session 2026-08-31]]
|
||||
- Cited by: [[CI Integration]], [[Gitea Actions]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md`
|
||||
|
||||
- Covered by: [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]]
|
||||
- Cited by: [[Chemenu]], [[Delete Rather Than Anonymize]], [[Publish-Remote Gate]], [[wikitool]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md`
|
||||
|
||||
- Covered by: [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]]
|
||||
- Cited by: [[Chemenu]], [[Delete Rather Than Anonymize]], [[Dual Licensing by File Plan]], [[wikitool]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md`
|
||||
|
||||
- Covered by: [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]]
|
||||
- Cited by: [[Chemenu]], [[Publish-Remote Gate]], [[wikitool]]
|
||||
|
||||
### `raw/notes/Conversation Transcript - Two Round-Trip Defects Found by an Ingest Session 2026-08-31.md`
|
||||
|
||||
- Covered by: [[Source - Conversation - Two Round-Trip Defects Found by an Ingest Session 2026-08-31]]
|
||||
|
||||
+6
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
# kb/sources/ - Index
|
||||
|
||||
21 page(s). Regenerated by `wikitool index rebuild`.
|
||||
26 page(s). Regenerated by `wikitool index rebuild`.
|
||||
|
||||
## All
|
||||
|
||||
@@ -28,5 +28,10 @@
|
||||
| [[Source - LLM Improvements Sonnet Analysis]] | notes | Sonnet-Analyse, die AGENTS.md und wikitool mit Farzas Gist und awesome-llm-wiki vergleicht und die Codex-Analyse um konkrete Empfehlungen zu Qualitätsschwellen, Stilrichtlinie, Auditrhythmus und Skalierung ergänzt | 2026-08-03 |
|
||||
| [[Source - LLM Wiki Pattern]] | article | Grundlegendes Muster für persönliche Wissensbasen mit LLMs: ein dauerhaftes Wiki schrittweise pflegen, statt es aus den Quellen neu herzuleiten. | 2026-07-26 |
|
||||
| [[Source - LLM Wiki v2]] | article | Erweitertes LLM-Wiki-Muster mit Praxiserfahrungen aus agentmemory zu Memory Lifecycle, Confidence Scoring, Wissensgraph und Automatisierung. | 2026-07-26 |
|
||||
| [[Source - MCP Read Server Implementation Session 2026-09-02]] | notes | Sitzung, die die Sequenz aus Issue #36 umsetzt: Publish-Remote Gate scharf, Lesepfad gehaertet, Root-Aufloesung und Bibliotheksgrenze gezogen, MCP-Leseserver gebaut - vier Versionsstufen 2.2.3 bis 2.4.0, dazu INSTALL-MCP.md und Issue #37. | 2026-09-02 |
|
||||
| [[Source - Private-Instance Merge Correction and Issue 30 Session 2026-09-01]] | notes | Sitzung, die eine ungeprueft niedergeschriebene Merge-Behauptung in private-instance.md durch einen empirischen Test widerlegt, die Prozedur korrigiert (2.2.1) und Issue #30 mit einem getesteten Skript sowie zwei Architekturvorschlaegen anlegt. | 2026-09-01 |
|
||||
| [[Source - Public Release, Corpus Purge and History Squash Session 2026-09-01]] | notes | Sitzung, die den Chemenu-Stack von einer privaten Testinstanz in ein oeffentliches Repo ueberfuehrt: Korpus geloescht statt anonymisiert, Git-History auf einen Commit gesquashed, AGPL-3.0/CC-BY-4.0-Dual-Lizenz gewaehlt, dist export um einen Leak-Canary gehaertet. | 2026-09-01 |
|
||||
| [[Source - Publish-Remote Gate and Issue Triage Session 2026-09-01]] | notes | Sitzung, die ein drittes, Token-loses Gate fuer publish baut, instructions/private-instance.md schreibt, sechs Gitea-Issues auf den Rename und die neue Architektur nachzieht und die Actions-Run-Historie entfernen laesst. | 2026-09-01 |
|
||||
| [[Source - qmd - GitHub Repository]] | document | GitHub-API-Metadaten, package.json und README-Auszuege von tobi/qmd: TypeScript/Node/Bun statt Go oder Rust, BM25 (SQLite FTS5) plus Vektor-Suche (sqlite-vec) plus LLM-Reranking ueber node-llama-cpp. | 2026-09-02 |
|
||||
| [[Source - Wine]] | notes | Wine-Konfiguration für Arch Linux: pacman-NoExtract-Einstellungen und Bottles-Runtime-Optionen einschließlich Proton- und Lutris-Varianten. | 2026-08-01 |
|
||||
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
type: types/source.md
|
||||
source_type: notes
|
||||
author: Torben
|
||||
raw_files: [raw/notes/Conversation Transcript - MCP Read Server Implementation Session 2026-09-02.md]
|
||||
source_language: de
|
||||
date: 2026-09-02
|
||||
tags: []
|
||||
entities: [wikitool, Chemenu]
|
||||
concepts: [Publish-Remote Gate, Mass-Update Gate, Iteration and Cost Limits, MCP-Leseserver]
|
||||
summary: 'Sitzung, die die Sequenz aus Issue #36 umsetzt: Publish-Remote Gate scharf, Lesepfad gehaertet, Root-Aufloesung und Bibliotheksgrenze gezogen, MCP-Leseserver gebaut - vier Versionsstufen 2.2.3 bis 2.4.0, dazu INSTALL-MCP.md und Issue #37.'
|
||||
---
|
||||
# Source: MCP Read Server Implementation Session 2026-09-02
|
||||
|
||||
**Autor:** Torben
|
||||
**Datum:** 2026-09-02
|
||||
**Raw-Dateien:** raw/notes/Conversation Transcript - MCP Read Server Implementation Session 2026-09-02.md
|
||||
**Typ:** Notes
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Diese Sitzung arbeitet die vierstufige Sequenz aus Issue #36 ab, dem Sammel-Issue für den Weg
|
||||
zum MCP-Leseserver: #34 (Publish-Remote-Gate scharf stellen), #33 (Lesepfad vor der Exposition
|
||||
härten), #31 (Root-Auflösung von der Importzeit lösen und eine Bibliotheksgrenze ziehen), #19
|
||||
(der Leseserver selbst). Jeder Schritt endet mit einem Versions-Bump und einem Testlauf; die
|
||||
Reihenfolge folgt dem Master-Issue, weil #33 dieselben Dateien anfasst, die #31 strukturell
|
||||
umbaut, und #31 die Grenze liefert, auf der #19 aufsetzt.
|
||||
|
||||
`.wikitool-remotes.json` fehlte in diesem Checkout trotz Dokumentation, die das Gegenteil
|
||||
behauptete - angelegt und gegen ein erfundenes Ziel gegengeprüft (Exit 42). Der Lesepfad bekam
|
||||
sechs Fixes gegen einen 267-Byte-YAML-Alias, der zu 672.603 Knoten expandiert, gegen einen
|
||||
ReDoS-Zweig in der Ranking-Funktion, einen fehlenden Subprozess-Timeout, und einen Korpus-Cache,
|
||||
der nie einen schmutzigen Arbeitsbaum cacht. `config.ROOT` und alle abgeleiteten Pfade waren zur
|
||||
Importzeit gebunden; die Auflösung ist jetzt lazy (`CHEMENU_ROOT` → Walk-up), und der reine
|
||||
Lesekern (`search/service.py`, `lint_core.py`, `types_core.py`) importiert kein `typer` mehr.
|
||||
Der MCP-Server (`tools/chemenu/mcp/`) exponiert `search`/`types`/`describe_type`/`lint`/`status`
|
||||
über `chemenu.api.Corpus` - strukturell ohne Schreibpfad, mit Commit-Stempel auf jeder Antwort
|
||||
und einer Startverweigerung, falls Telemetrie in den bedienten Baum schreiben würde.
|
||||
|
||||
Nach Freigabe des 37-Datei-Changesets (Mass-Update-Gate, Token `46442f4419c1`) folgten
|
||||
`INSTALL-MCP.md` für Menschen, ein Verweis auf die separate Traefik-ForwardAuth-Middleware
|
||||
(`gitea-mcp-forward-auth`), und Issue #37 für das noch fehlende Container-Image - mit den
|
||||
konkreten CI-Vorlagen aus `gitea-mcp-forward-auth` (Registry-Push) und `gitea-mcp`
|
||||
(Dockerfile-Form, aber DockerHub statt der eigenen Registry). Alle vier Sequenz-Issues wurden
|
||||
geschlossen, #36 blieb offen, weil sein eigenes Abschlusskriterium - ein Konsument, der
|
||||
nachweislich über die Middleware antwortet - erst mit #37 erfüllbar ist.
|
||||
|
||||
## Kernaussagen
|
||||
|
||||
- Gemessen: Korpus-Parse 265 ms → 54 ms (`CSafeLoader`), `wikitool search` end-to-end
|
||||
593 ms → 347 ms; die verbleibenden ~262 ms sind Modulimport und entfallen erst im residenten
|
||||
MCP-Prozess.
|
||||
- Der ReDoS-Zweig (`_contains` mit `re.search` gegen nutzergesteuerten Regex) wurde gelöscht,
|
||||
nicht begrenzt - `rg` wendet das Muster ohnehin mit einer linearen Engine an, bevor die
|
||||
Funktion je läuft.
|
||||
- `monkeypatch.setattr(config, "ROOT", ...)` baute nach der lazy-Auflösung die stale Bindung
|
||||
beim Teardown wieder auf, weil es den *aufgelösten* alten Wert zurückschreibt - `config.reset()`
|
||||
musste dazukommen, in derselben autouse-Fixture, die das Problem eine Ebene höher (Umgebungsvariablen)
|
||||
bereits kannte.
|
||||
- Der MCP-Server hat keinen Schreibpfad, weil `chemenu.api` nichts unter `chemenu.commands`
|
||||
importiert - nicht, weil eine Liste gefiltert wird. Ein Test importiert das Servermodul in
|
||||
einem frischen Interpreter und prüft `sys.modules`.
|
||||
- Ein Stempel-Bug wurde beim Schreiben des Golden-Tests selbst gefunden: `_stamp()` fragte nach
|
||||
der *aktuellen* statt der beim Laden tatsächlich gelesenen Revision und hätte bei einem
|
||||
minimal verzögerten zweiten Zugriff `commit: null` auf einem sauberen Baum liefern können.
|
||||
- `gitea-mcp` ist als Registry-Vorlage ungeeignet - sein Release-Workflow pusht nach DockerHub
|
||||
(Fork des Upstream), nicht in die eigene Gitea-Registry.
|
||||
|
||||
## Aufgaben
|
||||
|
||||
- [x] #34, #33, #31, #19 umgesetzt und mit Abschlusskommentar geschlossen
|
||||
- [x] `INSTALL-MCP.md` geschrieben, in `INSTALL.md`/`README.md` verlinkt, in `dist export` aufgenommen
|
||||
- [x] Issue #37 (Container-Image) angelegt, mit neun offenen Entscheidungen benannt
|
||||
- [ ] #37 selbst umsetzen
|
||||
- [ ] #36 schließen, sobald #37 den Middleware-Nachweis liefert
|
||||
- [ ] #23 (Env-Var-Erzwingung) - `CHEMENU_ROOT` wurde von Hand in `_WIKITOOL_ENV` eingetragen
|
||||
- [x] `kb/entities/tools/qmd.md` - falsche Sprachangabe korrigiert (siehe Korrektur unten)
|
||||
|
||||
## Korrektur zum Transkript-Kopf
|
||||
|
||||
Der Fidelity-Block des Rohtranskripts sagt: "One of two transcripts cut from this session; the
|
||||
other covers fixing `kb/entities/tools/qmd.md`". Dieses zweite Transkript wurde nie geschrieben
|
||||
- `raw/` ist unveränderlich, die Korrektur gehört hierher, nicht in die Datei selbst. Tatsächlich
|
||||
lief die Korrektur ohne eigenes Transkript: direkt gegen `tobi/qmd` auf GitHub geprüft und als
|
||||
eigene Quelle mit eigenem Raw-Beleg abgelegt (`Source - qmd - GitHub Repository`,
|
||||
`raw/documents/qmd - GitHub Repository.md`) - eine Quellen-Verifikation statt eines
|
||||
Gesprächsprotokolls, was für eine Sprachangaben-Korrektur die passendere Belegform ist.
|
||||
|
||||
## Nicht übernommen
|
||||
|
||||
- Der vollständige Wortlaut der geprüften Docstrings, Kommentare und Testfälle - das Transkript
|
||||
benennt Dateien und die tragenden Eigenschaften, der Code selbst ist die Quelle.
|
||||
- Die exakten neun offenen Entscheidungspunkte aus Issue #37 (Korpus im Image vs. Volume,
|
||||
Basis-Image, Healthcheck etc.) - dort bereits vollständig dokumentiert, hier nicht dupliziert.
|
||||
- Der Wortlaut der abgerufenen READMEs von `gitea-mcp-forward-auth` und `gitea-mcp` - nur die
|
||||
für die Entscheidung relevanten Fakten (Config-Variablen, Workflow-Form, Registry-Ziel)
|
||||
wurden übernommen.
|
||||
|
||||
## Verwandte Entities
|
||||
|
||||
- [[wikitool]]
|
||||
- [[Chemenu]]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Mass-Update Gate]]
|
||||
- [[Iteration and Cost Limits]]
|
||||
- [[MCP-Leseserver]]
|
||||
+83
@@ -0,0 +1,83 @@
|
||||
---
|
||||
type: types/source.md
|
||||
source_type: notes
|
||||
author: Torben
|
||||
raw_files: [raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md]
|
||||
source_language: de
|
||||
date: 2026-09-01
|
||||
tags: []
|
||||
entities: [Chemenu, wikitool]
|
||||
concepts: [Publish-Remote Gate, Delete Rather Than Anonymize]
|
||||
summary: 'Sitzung, die eine ungeprueft niedergeschriebene Merge-Behauptung in private-instance.md durch einen empirischen Test widerlegt, die Prozedur korrigiert (2.2.1) und Issue #30 mit einem getesteten Skript sowie zwei Architekturvorschlaegen anlegt.'
|
||||
---
|
||||
# Source: Private-Instance Merge Correction and Issue 30 Session 2026-09-01
|
||||
|
||||
**Autor:** Torben
|
||||
**Datum:** 2026-09-01
|
||||
**Raw-Dateien:** raw/notes/Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session 2026-09-01.md
|
||||
**Typ:** Notes
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Torben fragte, was in der privaten Instanz nach dem beschriebenen Schema passiert, wenn Upstream
|
||||
den Demo-Korpus ändert - eine Frage, die eine am selben Tag geschriebene, aber nie getestete
|
||||
Behauptung in `instructions/private-instance.md` traf. Statt die bestehende Textstelle zu
|
||||
verteidigen, wurde sie an einem Wegwerf-Repo-Paar empirisch geprüft: Ein `git merge
|
||||
upstream/main` löst eine geänderte, gelöschte Demo-Seite **nicht** still auf, sondern erzeugt
|
||||
einen `modify/delete`-Konflikt und lässt die Upstream-Fassung im Arbeitsbaum liegen; eine neu
|
||||
angelegte Demo-Seite wird dagegen **stillschweigend** gestaged, ohne Konflikt und ohne Meldung.
|
||||
Ein zweiter, naheliegender Fix (`.gitattributes` mit `merge=ours` für die Inhaltsverzeichnisse)
|
||||
wurde ebenfalls getestet und ebenfalls widerlegt - der Treiber wirkt nur bei Inhaltskonflikten
|
||||
auf beidseitig vorhandenen Dateien, nicht bei modify/delete oder Neuanlage.
|
||||
|
||||
Die Instruktion wurde korrigiert (Version 2.2.1): Der Merge wird mit `--no-commit` offengehalten,
|
||||
die Inhaltsverzeichnisse werden auf den Stand vor dem Merge zurückgezwungen, solange `HEAD` noch
|
||||
dorthin zeigt, erst dann wird committet - gefolgt von einer Kontrolle
|
||||
(`git diff --name-only $BEFORE HEAD -- kb raw` muss leer sein), die nicht stillschweigend
|
||||
übersprungen werden kann. Ein eigenständiges, getestetes Skript wurde daraus abgeleitet und in
|
||||
Issue #30 hinterlegt, zusammen mit zwei Architekturvorschlägen: das Verfahren als `wikitool`-
|
||||
Kommando statt als Shell-Rezept, oder - als eigentliche Ursachenbehebung - den Demo-Korpus
|
||||
grundsätzlich von dem Branch fernzuhalten, von dem private Instanzen ihre Maschinerie ziehen.
|
||||
|
||||
Anschließend wurde Issue #3 (Chemenu-Rebranding) gegen seine eigenen Abnahmekriterien geprüft
|
||||
und für sauber befunden, und eine Verdrahtungslücke geschlossen (`tools/CONTRACT.md` kannte das
|
||||
Publish-Remote-Gate nicht, `gates.md` verlinkte nicht auf die Prozedur, die Chemenu-Projektseite
|
||||
beschrieb sich noch als privates Wiki) - veröffentlicht als 2.2.2.
|
||||
|
||||
## Kernaussagen
|
||||
|
||||
- Eine Behauptung über Git-Merge-Verhalten ist erst nach einem Test eine Tatsache. Am selben Tag
|
||||
geschrieben zu haben ist kein Beleg für Richtigkeit.
|
||||
- `git merge` behandelt "eine Datei geändert" und "eine Datei neu angelegt" unterschiedlich: Nur
|
||||
Ersteres erzeugt einen sichtbaren Konflikt. Eine Prozedur, die nur den Konfliktfall bedenkt,
|
||||
übersieht die stille Neuanlage.
|
||||
- `.gitattributes`-Merge-Treiber wie `merge=ours` wirken nur auf Inhaltskonflikte zwischen
|
||||
beidseitig vorhandenen Dateiversionen, nicht auf modify/delete-Paare oder Neuanlagen.
|
||||
- Eine Kontrollprüfung, die ein Mensch lesen und verstehen muss, um einen Fehler zu bemerken, ist
|
||||
schwächer als eine, die bei einem Fehler selbst nicht-null zurückgibt.
|
||||
- Wiederkehrende Symptome bei jeder Downstream-Instanz sind oft ein Zeichen, dass die eigentliche
|
||||
Ursache stromaufwärts liegt und dort einmalig behoben werden sollte.
|
||||
|
||||
## Aufgaben
|
||||
|
||||
- [x] `private-instance.md` korrigiert und als 2.2.1 veröffentlicht
|
||||
- [x] Issue #30 angelegt (Skript + zwei Architekturvorschläge, hängt an #28)
|
||||
- [x] Issue #3 gegen Abnahmekriterien geprüft, sauber
|
||||
- [ ] Issue #30 selbst ist offen - siehe dort für den Entscheidungsstand
|
||||
|
||||
## Nicht übernommen
|
||||
|
||||
- Die im Transkript vollständig gezeigten Testskript-Läufe (Shell-Ausgabe der Wegwerf-Repos)
|
||||
sind hier nicht wiederholt - die Kernaussage (welches Verhalten gemessen wurde) ist auf die
|
||||
Concept-Seite [[Publish-Remote Gate]] und in Issue #30 übernommen, der Beleg bleibt im
|
||||
Transkript.
|
||||
|
||||
## Verwandte Entities
|
||||
|
||||
- [[Chemenu]]
|
||||
- [[wikitool]]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Delete Rather Than Anonymize]]
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
---
|
||||
type: types/source.md
|
||||
source_type: notes
|
||||
author: Torben
|
||||
raw_files: ['raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md']
|
||||
source_language: de
|
||||
date: 2026-09-01
|
||||
tags: []
|
||||
entities: [Chemenu, wikitool]
|
||||
concepts: [Mass-Update Gate, Delete Rather Than Anonymize, Dual Licensing by File Plan]
|
||||
summary: 'Sitzung, die den Chemenu-Stack von einer privaten Testinstanz in ein oeffentliches Repo ueberfuehrt: Korpus geloescht statt anonymisiert, Git-History auf einen Commit gesquashed, AGPL-3.0/CC-BY-4.0-Dual-Lizenz gewaehlt, dist export um einen Leak-Canary gehaertet.'
|
||||
---
|
||||
# Source: Public Release, Corpus Purge and History Squash Session 2026-09-01
|
||||
|
||||
**Autor:** Torben
|
||||
**Datum:** 2026-09-01
|
||||
**Raw-Dateien:** raw/notes/Conversation Transcript - Public Release, Corpus Purge and History Squash Session 2026-09-01.md
|
||||
**Typ:** Notes
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Torben bat darum, den Chemenu-Korpus zu bereinigen und das Repo zu veröffentlichen, mit
|
||||
ausdrücklichem Wunsch nach Teufels-Advokat-Modus und einer kleinen Multi-Agent-Debatte. Drei
|
||||
parallele Explore-Agenten inventarisierten private Daten, den Distributionsmechanismus und die
|
||||
Git-History; die History-Prüfung erklärte den Baum fälschlich für „safe to publish", ohne
|
||||
`USER.md` je zu öffnen — ein Befund, der später den Ausschlag für den vollständigen History-Schnitt
|
||||
gab. Drei geforkte Debattierer (harte Trennung, geteiltes Upstream, Teufels-Advokat) argumentierten
|
||||
gegeneinander; die Synthese übernahm „löschen statt anonymisieren" und „History squashen" von der
|
||||
harten Position, das Clone-mit-Upstream-Modell von der geteilten Position, aber unter der
|
||||
Bedingung, dass ein Remote-Gate zuerst existiert — eine Bedingung, die der Teufels-Advokat mit
|
||||
seinem Leck-Argument erzwang.
|
||||
|
||||
Nutzer traf vier Entscheidungen über `AskUserQuestion`: Gitea öffentlich schalten (kleinste
|
||||
Änderung), Korpus chirurgisch löschen, private Instanz als Clone mit Upstream und Gate zuerst,
|
||||
und als Lizenz AGPL-3.0 (Stack) + CC-BY-4.0 (Inhalte) — die Affero-Variante bewusst wegen Issue
|
||||
#19 (MCP-Frontend als Netzdienst).
|
||||
|
||||
Ausgeführt wurde: Lizenzdateien (AGPL-Text von gnu.org geholt, nicht aus dem Gedächtnis
|
||||
rekonstruiert), ein Leak-Canary in `dist export` (`find_leaks()`, strukturell statt textbasiert,
|
||||
weil ein Muster-Scan den eigenen legitimen Host mit ausschließen müsste), die Korpus-Löschung
|
||||
(108 Seiten statt der im Plan geschätzten ~35, weil 40 Seiten mit generischen Titeln tatsächlich
|
||||
um die private Infrastruktur herum geschrieben waren), der History-Squash auf einen Commit, und
|
||||
die Veröffentlichung selbst.
|
||||
|
||||
Zwei Annahmen wurden durch Messung widerlegt und korrigiert: Ein Force-Push allein reicht nicht —
|
||||
der alte HEAD blieb per SHA abrufbar, bis Reflogs auf dem Server verfielen und `gc --prune=now`
|
||||
lief. Und `rg`-Scans ohne `--hidden` übersehen `.gitea/`, `.github/`, `.vibe/` — ein zweiter
|
||||
Fund (private Referenzen in `ci.yml`) kam erst über `git grep` zum Vorschein.
|
||||
|
||||
## Kernaussagen
|
||||
|
||||
- Ein Seitentitel ist der einzige Identifier des Wikis; Löschen (`wikitool rm`) ist dafür billiger
|
||||
und sicherer als Anonymisieren, das die volle `page-lifecycle`-Prozedur pro Seite verlangt.
|
||||
- Der eigentliche Preisgeber bei einem Infrastruktur-Handbuch ist die Topologie, nicht der
|
||||
Hostname — gefälschte IPs entschärfen keine Angriffskarte.
|
||||
- Ein Force-Push macht alte Commits unreferenziert, aber nicht unerreichbar: Sie bleiben per SHA
|
||||
fetchbar, bis Server-Reflogs verfallen sind und `git gc --prune=now` gelaufen ist.
|
||||
- `dist export`s Lizenz-Dateien mussten zur Pflicht werden (`REQUIRED_ROOT_FILES`), weil das
|
||||
übliche `if source.is_file()`-Muster eine fehlende Lizenz still überspringen würde — bei AGPL
|
||||
eine Verletzung, sobald eine Instanz öffentlich landet.
|
||||
- Ein Text-Muster-Scan für Leaks scheitert an legitimen Vorkommen des eigenen Hostnamens; ein
|
||||
struktureller Scan (welche Pfade/Dateien dürfen nie im Plan stehen) umgeht das.
|
||||
|
||||
## Aufgaben
|
||||
|
||||
- [x] Korpus bereinigt, History gesquasht, Lizenzen gesetzt, Repo veröffentlicht (in dieser
|
||||
Sitzung erledigt)
|
||||
- [ ] Siehe Issue #27 (Decay-/Lint-Ausschluss für mitgelieferte Seiten) und #28 (Demo-Korpus als
|
||||
eigene Fixture) für Folgearbeit aus dieser Sitzung
|
||||
|
||||
## Nicht übernommen
|
||||
|
||||
- Die konkreten Namen und Details der gelöschten privaten Infrastruktur (Hostnamen, IP-Bereiche,
|
||||
persönliche Angaben) sind bewusst nicht in diese Source-Seite übernommen — sie zu wiederholen
|
||||
widerspräche dem Zweck der Sitzung. Wo sie als Beispiel dienen mussten, steht hier nur die Art
|
||||
der Information (z. B. "eine Homelab-Cluster-Dokumentation"), nie der Wortlaut.
|
||||
- Der volle Wortlaut der drei Debattenpositionen ist nicht übernommen - nur ihre tragenden
|
||||
Argumente und was aus ihnen in die Synthese einging. Der vollständige Text steht im Transkript.
|
||||
|
||||
## Verwandte Entities
|
||||
|
||||
- [[Chemenu]]
|
||||
- [[wikitool]]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Mass-Update Gate]]
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
type: types/source.md
|
||||
source_type: notes
|
||||
author: Torben
|
||||
raw_files: [raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md]
|
||||
source_language: de
|
||||
date: 2026-09-01
|
||||
tags: []
|
||||
entities: [Chemenu, wikitool]
|
||||
concepts: [Publish-Remote Gate, Mass-Update Gate, Issue Label Scheme]
|
||||
summary: Sitzung, die ein drittes, Token-loses Gate fuer publish baut, instructions/private-instance.md schreibt, sechs Gitea-Issues auf den Rename und die neue Architektur nachzieht und die Actions-Run-Historie entfernen laesst.
|
||||
---
|
||||
# Source: Publish-Remote Gate and Issue Triage Session 2026-09-01
|
||||
|
||||
**Autor:** Torben
|
||||
**Datum:** 2026-09-01
|
||||
**Raw-Dateien:** raw/notes/Conversation Transcript - Publish-Remote Gate and Issue Triage Session 2026-09-01.md
|
||||
**Typ:** Notes
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Direkte Fortsetzung der Veröffentlichungssitzung: Torben bat um eine Remote-Allowlist für
|
||||
`publish`, eine Bitte, vor dem Öffentlich-Schalten von Gitea zu warten und einen Spickzettel
|
||||
dafür, und ein Kurz-Howto für die lokale Dev-Umgebung - dazu, nach eigenem Ermessen, Issues zu
|
||||
pflegen. Gebaut wurde das **Publish-Remote-Gate**: Es prüft die aufgelöste Push-URL (nicht den
|
||||
Remote-Namen, weil ein umgebogener `origin` sonst durchrutschen würde) gegen eine optionale,
|
||||
gitignorete Allowlist-Datei und hat als einziges der drei Gates keinen Freigabe-Token - der Weg
|
||||
daran vorbei ist ein bewusster Edit der Datei durch den Menschen. `instructions/private-instance.md`
|
||||
beschreibt seither das Clone-mit-Upstream-Setup, mit dem Gate als Schritt vor dem ersten `publish`.
|
||||
|
||||
Sechs offene Issues wurden auf den Chemenu-Rename hin durchgesehen; mehrere trugen noch den
|
||||
alten Paketpfad. Drei neue Issues entstanden aus Punkten, die im Tagesverlauf entschieden und
|
||||
dann zurückgestellt worden waren (Handbuch-Vorbedingung, Demo-vs-Testbett-Konflikt, veraltete
|
||||
Issue-Texte). Nach dem Öffentlich-Schalten wurde anonym end-to-end geprüft (Klon, Release-Feed,
|
||||
Distributionsweg), `INSTALL.md` von "Repo ist privat" auf den öffentlichen Zustand umgestellt,
|
||||
und auf Bitte des Nutzers die Gitea-Actions-Run-Historie über die REST-API entfernt, nachdem
|
||||
sich herausstellte, dass weder das MCP-Werkzeug noch eine sichtbare UI-Schaltfläche das können.
|
||||
|
||||
## Kernaussagen
|
||||
|
||||
- Ein Allowlist-Gate muss die **aufgelöste Push-URL** prüfen, nicht den Remote-Namen - sonst
|
||||
schützt es nicht vor einem umbenannten oder umgebogenen Remote.
|
||||
- Ein Gate, dessen Frage eine stehende Eigenschaft des Checkouts ist (nicht ein einzelnes
|
||||
Changeset), braucht keinen Freigabe-Token - der Mensch löst es durch einen bewussten
|
||||
Datei-Edit, nie ein Agent durch einen Bypass.
|
||||
- Ein Werkzeugvertrag (`tools/CONTRACT.md`) muss jeden Fehlerfall eines Kommandos nennen; ein
|
||||
neuer Exit-42-Pfad, der dort fehlt, ist eine Lücke, die kein automatischer Check findet.
|
||||
- Ein MCP-Server kann weniger können als die zugrunde liegende API - hier: Actions-Runs
|
||||
anzeigen/erneut starten, aber nicht löschen, obwohl die REST-API die Route hat.
|
||||
|
||||
## Aufgaben
|
||||
|
||||
- [x] Publish-Remote-Gate, `private-instance.md`, Issue-Pflege in dieser Sitzung erledigt
|
||||
- [ ] #4, #5 tragen laut #29 noch veraltete Pfade und sind noch nicht nachgezogen
|
||||
|
||||
## Nicht übernommen
|
||||
|
||||
- Der vollständige Wortlaut des Gitea-Spickzettels ist nicht auf diese Seite übernommen - er
|
||||
steht im Transkript und in der Chat-Antwort an den Nutzer, ist aber keine dauerhafte
|
||||
Wiki-Aussage, sondern eine einmalige Handlungsanweisung.
|
||||
|
||||
## Verwandte Entities
|
||||
|
||||
- [[Chemenu]]
|
||||
- [[wikitool]]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- [[Publish-Remote Gate]]
|
||||
- [[Mass-Update Gate]]
|
||||
- [[Issue Label Scheme]]
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
type: types/source.md
|
||||
source_type: document
|
||||
author: Torben
|
||||
raw_files: [raw/documents/qmd - GitHub Repository.md]
|
||||
source_url: https://github.com/tobi/qmd
|
||||
source_language: en
|
||||
date: 2026-09-02
|
||||
tags: []
|
||||
entities: [qmd]
|
||||
concepts: []
|
||||
summary: 'GitHub-API-Metadaten, package.json und README-Auszuege von tobi/qmd: TypeScript/Node/Bun statt Go oder Rust, BM25 (SQLite FTS5) plus Vektor-Suche (sqlite-vec) plus LLM-Reranking ueber node-llama-cpp.'
|
||||
---
|
||||
# Source: qmd - GitHub Repository
|
||||
|
||||
**Autor:** Torben
|
||||
**Datum:** 2026-09-02
|
||||
**Raw-Dateien:** raw/documents/qmd - GitHub Repository.md
|
||||
**Typ:** Document
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Direkter Abruf von `tobi/qmd` auf GitHub (Repo-Metadaten über die GitHub-API, `package.json`,
|
||||
README-Auszüge), um eine bestehende, unsichere Behauptung auf der Entity-Seite [[qmd]] zu
|
||||
prüfen. Die Seite trug seit ihrer Erstellung `Sprache: Nicht angegeben (wahrscheinlich Go oder
|
||||
Rust)` bei `confidence: 0.85` - eine geratene Angabe mit einer Konfidenz, die für eine Vermutung
|
||||
zu hoch war. Die GitHub-API meldet `"language": "TypeScript"`, mit einem byte-gewichteten
|
||||
Sprach-Split von 1.597.168 B TypeScript gegenüber 200.880 B Python, 26.136 B Shell,
|
||||
23.132 B JavaScript und weniger als 6 KB restlicher Sprachen - kein Go, kein Rust im Repository.
|
||||
|
||||
Das npm-Paket heißt `@tobilu/qmd`, installiert per `npm install -g` oder `bun install -g`,
|
||||
ausgeführt per `npx`/`bunx` - keine Go-typische `go install`-Zeile, wie die alte Seite
|
||||
behauptete. `package.json` listet `node-llama-cpp` (LLM-Inferenz), `sqlite-vec` (Vektorindex)
|
||||
und `better-sqlite3` (SQLite-Bindung) als direkte Abhängigkeiten; das README bestätigt das
|
||||
Zusammenspiel explizit: „QMD combines BM25 full-text search, vector semantic search, and LLM
|
||||
re-ranking—all running locally via node-llama-cpp with GGUF models." Die Scoring-Tabelle des
|
||||
Architektur-Abschnitts benennt zusätzlich `SQLite FTS5 BM25` als konkreten Volltext-Mechanismus,
|
||||
nicht nur „BM25" allgemein.
|
||||
|
||||
## Kernaussagen
|
||||
|
||||
- Sprache: TypeScript (GitHub-API `language`-Feld, bestätigt durch die Byte-Gewichtung).
|
||||
- Runtime: Node.js oder Bun, nicht Go/Rust - Installationsbefehle im README belegen das direkt.
|
||||
- Such-Stack: SQLite FTS5 (BM25) + `sqlite-vec` (Vektor-Ähnlichkeit) + `node-llama-cpp`
|
||||
(LLM-Reranking mit GGUF-Modellen) - deckungsgleich mit dem, was die alte Seite unter
|
||||
„Such-Capabilities" bereits beschrieb, nur ohne konkrete Technologienamen.
|
||||
- Enthält einen `@modelcontextprotocol/server`-Abhängigkeit, deckungsgleich mit der bereits auf
|
||||
der Seite genannten MCP-Server-Funktion.
|
||||
|
||||
## Aufgaben
|
||||
|
||||
- [x] `kb/entities/tools/qmd.md`s Sprachangabe korrigiert und mit dieser Quelle belegt
|
||||
|
||||
## Nicht übernommen
|
||||
|
||||
- Der vollständige Funktionsumfang (Library-API, MCP-Server-Details, Tree-Sitter-basiertes
|
||||
Code-Chunking, Konfigurationsdatei-Referenz, CLI-Unterbefehle) - nur das zur Korrektur der
|
||||
Sprach-/Stack-Angabe Nötige wurde übernommen. Eine künftige Erweiterung der `qmd`-Seite um
|
||||
diese Themen bräuchte einen neuen Abruf, nicht diese Quelle.
|
||||
|
||||
## Verwandte Entities
|
||||
|
||||
- [[qmd]]
|
||||
|
||||
## Verwandte Concepts
|
||||
|
||||
- None identified
|
||||
@@ -0,0 +1,101 @@
|
||||
# qmd - GitHub Repository (tobi/qmd)
|
||||
|
||||
> Source: https://github.com/tobi/qmd
|
||||
> Retrieved: 2026-09-02, via `https://api.github.com/repos/tobi/qmd`,
|
||||
> `https://api.github.com/repos/tobi/qmd/languages`, and the raw README/`package.json` at the
|
||||
> `main` branch HEAD (`pushed_at: 2026-08-18T14:26:31Z` at retrieval time).
|
||||
> Fidelity: the API JSON fields are quoted verbatim; the README/package.json excerpts below are
|
||||
> quoted verbatim from the fetched files, trimmed to what backs the corrected claims on
|
||||
> `kb/entities/tools/qmd.md` - not a full copy of either file.
|
||||
|
||||
## Repository metadata (GitHub API)
|
||||
|
||||
```json
|
||||
{
|
||||
"full_name": "tobi/qmd",
|
||||
"description": "mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local",
|
||||
"language": "TypeScript",
|
||||
"html_url": "https://github.com/tobi/qmd",
|
||||
"license": {"key": "mit", "name": "MIT License", "spdx_id": "MIT"}
|
||||
}
|
||||
```
|
||||
|
||||
Byte-weighted language breakdown (`/languages` endpoint):
|
||||
|
||||
```json
|
||||
{
|
||||
"TypeScript": 1597168,
|
||||
"Python": 200880,
|
||||
"Shell": 26136,
|
||||
"JavaScript": 23132,
|
||||
"Nix": 5984,
|
||||
"Just": 1081,
|
||||
"Dockerfile": 1074
|
||||
}
|
||||
```
|
||||
|
||||
## `package.json` dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@tobilu/qmd",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/server": "2.0.0",
|
||||
"better-sqlite3": "^13.0.3",
|
||||
"fast-glob": "3.3.3",
|
||||
"node-llama-cpp": "3.20.0",
|
||||
"picomatch": "4.0.5",
|
||||
"sqlite-vec": "0.1.9",
|
||||
"tree-sitter-go": "0.25.0",
|
||||
"tree-sitter-python": "0.25.0",
|
||||
"tree-sitter-rust": "0.24.0",
|
||||
"tree-sitter-typescript": "0.23.2",
|
||||
"web-tree-sitter": "0.26.12",
|
||||
"yaml": "2.9.0",
|
||||
"zod": "4.2.1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## README excerpts
|
||||
|
||||
Opening description (README.md, line 5):
|
||||
|
||||
> QMD combines BM25 full-text search, vector semantic search, and LLM re-ranking—all running
|
||||
> locally via node-llama-cpp with GGUF models.
|
||||
|
||||
Install ("Quick Start" section):
|
||||
|
||||
```sh
|
||||
# Install globally (Node or Bun)
|
||||
npm install -g @tobilu/qmd
|
||||
# or
|
||||
bun install -g @tobilu/qmd
|
||||
|
||||
# Or run directly
|
||||
npx @tobilu/qmd ...
|
||||
bunx @tobilu/qmd ...
|
||||
```
|
||||
|
||||
Scoring table (architecture section):
|
||||
|
||||
| Stage | Method | Formula | Range |
|
||||
|---|---|---|---|
|
||||
| FTS (BM25) | SQLite FTS5 BM25 | `Math.abs(score)` | 0 to ~25+ |
|
||||
| Vector | Cosine distance | `1 / (1 + distance)` | 0.0 to 1.0 |
|
||||
| Reranker | LLM 0-10 rating | `score / 10` | 0.0 to 1.0 |
|
||||
|
||||
Diagnostic command comment, confirming `sqlite-vec` is the vector index used at runtime, not
|
||||
only a listed dependency:
|
||||
|
||||
```
|
||||
# Diagnose the install (runtime, sqlite-vec, embedding fingerprints, GPU probe)
|
||||
```
|
||||
|
||||
## Not extracted
|
||||
|
||||
The full README (1301 lines: library API, MCP server usage, tree-sitter code-chunking options,
|
||||
config file reference, CLI subcommand reference) - only the excerpts backing the corrected
|
||||
`qmd.md` claims (language, runtime, install method, search/rerank stack) were pulled. A future
|
||||
qmd page expansion covering its library API or MCP server would need a fresh fetch, not a reuse
|
||||
of this file's `## Not extracted` judgment.
|
||||
+245
@@ -0,0 +1,245 @@
|
||||
# Conversation Transcript - MCP Read Server Implementation Session
|
||||
|
||||
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
|
||||
> Collected: 2026-09-02
|
||||
> Participant: Torben
|
||||
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instructions are
|
||||
> quoted verbatim; the assistant's design reasoning, code, and file contents are condensed to
|
||||
> what a later reader needs to find the artifact - the artifacts themselves are the code, not
|
||||
> this transcript. Command outputs shown (test counts, timings, the Mass-Update Gate's printed
|
||||
> file list, gate refusal messages) are real, copied from the session. No credentials appeared.
|
||||
> No subagents were used - all work in this transcript is first-hand.
|
||||
> One of two transcripts cut from this session; the other covers fixing `kb/entities/tools/qmd.md`,
|
||||
> a documentation defect noted but not fixed during this one.
|
||||
|
||||
Covers implementing the four-issue sequence held by #36 (the master issue for "path to the MCP
|
||||
read server"): #34 (arm the Publish-Remote Gate), #33 (harden the read path), #31 (root
|
||||
resolution + library boundary), #19 (the MCP server itself), plus the human-facing
|
||||
`INSTALL-MCP.md` and filing #37 (the container-image follow-up) that #19's closure surfaced as
|
||||
still missing. Four version bumps in sequence: 2.2.3, 2.2.4, 2.3.0, 2.4.0, then 2.4.1 for the
|
||||
documentation follow-up. All five touched issues (#34, #33, #31, #19, #36) got closing comments;
|
||||
#36 was left open pending #37.
|
||||
|
||||
---
|
||||
|
||||
## Turn 1 - `/stack-dev implementiere #36`
|
||||
|
||||
The `stack-dev` skill loaded, switching the session into tool-development rules (ordinary
|
||||
software-engineering judgment on `tools/`, `types/`, `instructions/` - no source-binding, no
|
||||
`raw/` citation needed for code).
|
||||
|
||||
Read issue #36 (the master issue) via `gitea-mcp`. It holds no implementation itself - a
|
||||
sequence table (#34 → #33 → #31 → #19), a table of platform/auth/transport decisions already
|
||||
made and closed to debate, a measured baseline (593 ms per `wikitool search`, broken down:
|
||||
262 ms module import, 274 ms corpus parse, 7 ms ripgrep), ten verified findings backing the
|
||||
design, and an explicit "not in scope" list (Go, deployment details, a write path, #15's
|
||||
research capability). Read #34, #33, #31, #19 in full including comments to get the acceptance
|
||||
criteria and the reasoning behind each.
|
||||
|
||||
Worked the sequence in order, starting a session-scoped `WIKITOOL_SESSION_ID`.
|
||||
|
||||
**#34 - Publish-Remote Gate.** `.wikitool-remotes.json` did not exist in this checkout; absence
|
||||
means unrestricted per `git_publish.read_allowed_push_urls()`. Created it with `origin`'s exact
|
||||
push URL as the sole allowed target. Counter-checked by adding a second git remote to a made-up
|
||||
URL and running `publish --remote gatecheck`: refused with **exit 42**, working tree untouched,
|
||||
before any fetch. Remote removed afterward.
|
||||
|
||||
`doctor`'s `check_publish_remotes()` reported only whether the *file* existed, not whether the
|
||||
gate was actually armed. Rewrote its three outcome messages to lead with `Gate armed:` /
|
||||
`Gate not armed:` explicitly - the single-remote-no-file case stays `OK` (nothing to protect,
|
||||
and a FAIL would make the file mandatory through the back door) but now says outright that every
|
||||
push target passes. The check had **no tests**; added three, one per state.
|
||||
|
||||
Bumped `--patch` → **2.2.3**.
|
||||
|
||||
**#33 - Harden the read path.** Six findings, each addressed with a regression:
|
||||
|
||||
1. *ReDoS.* `search/ripgrep.py:_contains` passed `query.regex` to `re.search`. Deleted the
|
||||
branch entirely rather than bounding it - `rg` already applies the pattern with a linear
|
||||
engine before this function runs, so nothing is lost except the extra title/summary ranking
|
||||
boost for a non-literal pattern (and summary/H1 lines are themselves counted by `rg`).
|
||||
Regression test asserts both the timing (`< 0.5s` against `(\w+\s?)+$`) and the return value,
|
||||
because a bound alone would pass with a merely-faster engine.
|
||||
2. *No subprocess timeout.* Added `RIPGREP_TIMEOUT_SECONDS = 30.0`; `TimeoutExpired` translates
|
||||
to the existing `RipgrepFailed` path.
|
||||
3. *YAML aliases.* Frontmatter has no legitimate use for anchors/aliases, so they are refused
|
||||
outright rather than budgeted, checked on the streaming *event* parse (`yaml.parse`) so the
|
||||
check itself never triggers the expansion it is checking for - `*` is a necessary character
|
||||
in any alias node, so its absence proves absence with zero parse cost. Constructed a
|
||||
reproduction: 267 bytes of nested aliases compose into 672,603 nodes on traversal at
|
||||
constant (0.2 ms) parse time, growth 9^n with nesting depth. Added a 64 KiB frontmatter size
|
||||
limit and a `RecursionError` catch (PyYAML composes recursively; deep nesting is not a
|
||||
`YAMLError`).
|
||||
4. *`CSafeLoader` unused.* Switched to it with a `SafeLoader` fallback. Measured over this
|
||||
corpus (176 pages, best of 5): **265 ms → 54 ms**.
|
||||
5. *Silent frontmatter loss.* Decided: the read path now *reports* rather than swallows.
|
||||
Unparseable YAML still degrades to `{}` for bulk operations, but the reason travels with it
|
||||
(`Page.frontmatter_error`, set by a single parser shared between `read_page()` and
|
||||
`frontmatter_error()` - previously two separate `safe_load` call sites that could describe
|
||||
the same broken file differently). `search --json` now always carries an `unreadable` list
|
||||
of `{path, reason}`; the table form writes the same lines to stderr. Added `read_page_strict()`
|
||||
for the future ingest-quarantine path (#32), which must stop on unparseable input rather than
|
||||
empty it.
|
||||
6. *Corpus reparse per call.* New `chemenu/corpus_cache.py`: a `CorpusCache` object a caller
|
||||
*holds* (not a module-global switch). The CLI holds none and is unaffected. Correctness
|
||||
property, not just speed: **a dirty working tree is never cached** - if git cannot answer
|
||||
whether the tree is clean, it is treated as dirty. `wikitool search` end-to-end (best of 5):
|
||||
**593 ms → 347 ms**.
|
||||
|
||||
Bumped `--patch` → **2.2.4**.
|
||||
|
||||
**#31 - Root resolution + library boundary.** `config.ROOT` and everything derived from it were
|
||||
Python module constants, bound at import time from `Path(__file__).resolve().parents[2]` - so
|
||||
`monkeypatch.setattr(config, "ROOT", tmp_path)` repointed `ROOT` but left `KB_DIR`/`RAW_DIR`
|
||||
aimed at the real checkout, the exact failure class `raw_dir`'s own fixture docstring already
|
||||
warned about one layer up.
|
||||
|
||||
Rewrote `config.py`: `resolve_root()` by precedence (explicit argument → `$CHEMENU_ROOT` →
|
||||
walk-up, the walk-up staying default so `tools/wikitool` is unaffected), and every derived path
|
||||
resolved on *attribute access* via module `__getattr__` (PEP 562) rather than stored - so an
|
||||
assignment onto `ROOT` (test or otherwise) is honored by every path under it, live. This
|
||||
surfaced a subtler bug while fixing the first one: `monkeypatch`'s own undo mechanism reads the
|
||||
old value (which resolves it) and writes it back as a real attribute on teardown, recreating the
|
||||
stale binding the rewrite was meant to eliminate. Added `config.reset()`, called by the autouse
|
||||
hermetic-environment fixture on both setup and teardown.
|
||||
|
||||
Split the CLI-coupled command modules into a pure core + thin adapter, three times over:
|
||||
`search/service.py`, `lint_core.py`, `types_core.py` - none of them import `typer` or `rich`.
|
||||
`commands/search.py`, `commands/lint.py`, `commands/types_cmd.py` became the terminal adapters,
|
||||
re-exporting the same names so no existing import breaks.
|
||||
|
||||
New `chemenu/errors.py`: `ChemenuError` → `ValidationError` (also inherits `ValueError`, since
|
||||
`PredicateError` already was one and callers catch it that way) / `BackendError`.
|
||||
`PredicateError`, `FrontmatterError`, `UnknownBackend`, `RipgrepMissing`, `RipgrepFailed` moved
|
||||
under this hierarchy.
|
||||
|
||||
New `chemenu/api.py`: `Corpus` class, the in-process entry point - takes a root, returns exactly
|
||||
the CLI's `--json` shapes, raises instead of exiting, stamps every response with `commit`/
|
||||
`as_of`. Wrote the acceptance test by monkeypatching `Path.read_text`/`Path.rglob` to fail on any
|
||||
access under the real checkout root while a foreign root is set - this test could not have
|
||||
passed before the lazy-resolution rewrite.
|
||||
|
||||
Two more accidental dependencies surfaced and were fixed: `TypeResolver.repo_root` was also
|
||||
import-bound (fixed the same way, with `config.rooted()` as a process-wide context manager for
|
||||
callers that reach `config` directly rather than taking a root argument - and `use_shipped_type_specs()`
|
||||
added to the test fixtures that had been relying on it silently); and `search/registry.resolve()`
|
||||
did not pass `kb_dir`/`root` through to the backend, so a caller pointing `run_search` at a
|
||||
foreign corpus could still have `RipgrepBackend` read `config.KB_DIR` underneath it.
|
||||
|
||||
Bumped `--minor` → **2.3.0** (new capability, backward-compatible).
|
||||
|
||||
**#19 - MCP read server.** New package `tools/chemenu/mcp/` (`server.py`, `__main__.py`).
|
||||
Installed the `mcp` SDK (`mcp>=2.0`, discovered mid-session that this pulled in the v2 API -
|
||||
`FastMCP` renamed to `MCPServer`, imported from `mcp.server.mcpserver`). Five tools over
|
||||
`chemenu.api.Corpus`: `search`, `types`, `describe_type`, `lint`, `status` (`status` is
|
||||
server-composed, not a wrapper - there is no `wikitool status` command to wrap). No write tool,
|
||||
structurally: neither the server module nor `chemenu.api` imports anything under
|
||||
`chemenu.commands`.
|
||||
|
||||
Both transports built and smoke-tested end-to-end against the real 176-page corpus: `stdio` via
|
||||
a hand-written MCP client script, and `streamable-http` (host/port bound explicitly - the
|
||||
default binds loopback, wrong for a container behind a proxy) via a subprocess + HTTP client
|
||||
round-trip. `sse` deliberately not offered (superseded transport).
|
||||
|
||||
`ChemenuError` translated to the SDK's `ToolError` at the tool-call boundary (a deliberate
|
||||
refusal whose message reaches the caller) rather than left to become an `UnexpectedToolError`
|
||||
(a crash whose message stays server-side).
|
||||
|
||||
Found and fixed a stamping bug while writing the golden test: `_stamp()` was asking the cache for
|
||||
the *current* revision after the load had already happened, so a caller with a perfectly clean
|
||||
tree could see `"commit": null` if the cache's cached revision lagged. Fixed by threading the
|
||||
revision the load actually returned through to the stamp.
|
||||
|
||||
Wrote `tools/chemenu/tests/test_mcp_server.py`: the golden test runs `wikitool ... --json` as a
|
||||
subprocess against the same fixture tree (via `$CHEMENU_ROOT`) and asserts the server's
|
||||
structured output matches field-for-field; a before/after test captures file size+contents,
|
||||
`git rev-parse HEAD`, and `git status --porcelain` around all five tool calls to prove nothing
|
||||
writes; a structural test imports the server module in a fresh interpreter and checks
|
||||
`sys.modules` for absence of `chemenu.commands.*`.
|
||||
|
||||
Added `check_trace_destination()`: the server **refuses to start** if `WIKI_TRACE` is on and
|
||||
would write into the served checkout's `reports/telemetry/` - the directory the sync's
|
||||
`git reset --hard` is entitled to wipe. `WIKI_TRACE=0` or `WIKI_TRACE_DIR` outside the corpus
|
||||
both satisfy it.
|
||||
|
||||
New `instructions/mcp-read-server.md` (agent-facing operating procedure), `tools/requirements-mcp.txt`
|
||||
(the SDK's dependency, kept out of `requirements.txt` so a CLI-only instance need not install
|
||||
pydantic/starlette/uvicorn/cryptography), a CI step to install it (so the golden test cannot
|
||||
silently be skipped), and doc updates (`tools/CONTRACT.md`, `tools/README.md`, `README.md`).
|
||||
|
||||
Bumped `--minor` → **2.4.0**.
|
||||
|
||||
Ran `docs verify`, `instructions verify`, and the full suite (**799 passed**) plus an
|
||||
empty-machine run (`env -i ... pytest`) with the same result, before presenting the changeset.
|
||||
|
||||
`tools/wikitool publish` hit the **Mass-Update Gate**: 37 files, +3,037/-625 lines, exit 42.
|
||||
Reproduced the full file-by-area breakdown verbatim in the reply (as the gate's message
|
||||
requires - the output does not reach the user's screen on its own) and stopped without running
|
||||
further commands, per the gate's instruction and AGENTS.md invariant 6.
|
||||
|
||||
## Turn 2 - "Freigegeben. Erstelle eine install-MCP.md für Menschen. Für traefik middleware verweise auf https://gitea.nehmer.net/torben/gitea-mcp-forward-auth ... Wir brauchen ein issue, der uns dicker Container für ein deployment erzeugt. Fasse dort schon mal zusammen, was du bisher weißt... Die ci scripte in https://gitea.nehmer.net/torben/gitea-mcp dürfte ein guter startpunkt sein."
|
||||
|
||||
Published with the token from the gate's own printed line
|
||||
(`--confirm 46442f4419c1`) - commit `576df2c`.
|
||||
|
||||
Fetched both referenced repos via `gitea-mcp` (`get_repository_tree`, `get_file_contents`) to
|
||||
ground the two follow-up documents in what actually exists rather than in assumption:
|
||||
|
||||
- `gitea-mcp-forward-auth`: a small Go Traefik ForwardAuth service - Bearer-token-vs-SHA-256-hash
|
||||
checking, `AUTH_PROXY_TOKEN_HASHES_DIR`/`AUTH_PROXY_TOKEN_HASHES` config, `/healthz` always
|
||||
open, demo token hashes checked into the repo, a registry-image test script with ephemeral
|
||||
Docker/Podman login.
|
||||
- `gitea-mcp`: its Dockerfile (multi-stage, `CGO_ENABLED=0`, distroless non-root, OCI labels) is
|
||||
a usable pattern, but its release workflow pushes to **DockerHub** (`DOCKER_ORG: gitea`) - it
|
||||
is a fork of the upstream project, so it is *not* a usable registry-push template for this
|
||||
repo. Noted explicitly rather than silently copied.
|
||||
|
||||
Wrote `install-MCP.md` (later renamed, see Turn 3): six numbered steps (install dependency, run
|
||||
stdio, wire a client, run streamable-http, put authentication in front, keep the corpus current
|
||||
via polling), a runnable stdio verification script (executed for real before being written into
|
||||
the doc, output: `Tools: [...]`, `Seiten: 176 | Commit: <sha>`), and a troubleshooting section
|
||||
keyed to the server's actual error messages. Linked it from `INSTALL.md` and `README.md`, and
|
||||
added `CHEMENU_ROOT`/`WIKI_TRACE`/`WIKI_TRACE_DIR` to `INSTALL.md`'s configuration table (both
|
||||
had been in effect since earlier work but undocumented there). Added the file to `dist_cmd.py`'s
|
||||
`ROOT_FILES` allowlist and verified with a real `dist export` that it ships.
|
||||
|
||||
Filed **issue #37** ("Container-Image für den MCP-Leseserver"): summarized what is already fixed
|
||||
(startup command, required env vars, the ripgrep-in-the-image trap a naive `pip install`
|
||||
Dockerfile would miss, where auth/rate-limiting belong) and the concrete templates found above,
|
||||
then nine explicit open decisions (corpus baked into the image vs. mounted as a volume and
|
||||
synced by a sidecar; who runs the sync; base image, since Go-style `distroless` does not carry a
|
||||
Python interpreter or `rg`; how the version reaches the image; whether to build `arm64`;
|
||||
**a healthcheck endpoint does not exist yet - the one item that implies new code in this repo**;
|
||||
registry path; OCI labels; whether a smoke test against the built image is added). Labeled
|
||||
`prio/2 size/M`.
|
||||
|
||||
Closed **#34, #33, #31, #19** with detailed comments each restating what was implemented against
|
||||
the issue's own acceptance criteria, including the two accidental-dependency findings from #31
|
||||
and the corrected alias-bomb numbers. Commented on **#36** with a summary table across all four
|
||||
versions, noted its closing criterion (a consumer provably reaching the server through the
|
||||
Traefik middleware) is not yet met - that needs #37's deployment - and proposed leaving #36 open
|
||||
until then; also carried forward its two remaining loose ends (the wrong `qmd.md` language claim,
|
||||
and the still-missing place for architecture decisions).
|
||||
|
||||
## Turn 3 - "Freigegeben. Nenne die Datei INSTALL-MCP.md all Caps sonst fahre mit dem Auftrag fort" (interrupting a `dist export` verification call)
|
||||
|
||||
Renamed `install-MCP.md` → `INSTALL-MCP.md`, fixed every reference (`INSTALL.md`, `README.md`,
|
||||
`tools/chemenu/commands/dist_cmd.py`). Re-ran the full suite (**799 passed**) and `docs verify`,
|
||||
bumped `--patch` → **2.4.1**, wrote the changelog entry, published (commit `83018fc`). Saved a
|
||||
feedback memory (`root-docs-are-all-caps.md`): root-level human docs in this repo are named in
|
||||
ALL CAPS, and a new one has to be added to `dist_cmd.ROOT_FILES` or it silently does not ship.
|
||||
|
||||
## Outcome
|
||||
|
||||
- **Version:** 2.2.2 -> 2.2.3 -> 2.2.4 -> 2.3.0 -> 2.4.0 -> 2.4.1
|
||||
- **Commits:** `576df2c` (2.4.0, the 37-file changeset cleared through the Mass-Update Gate),
|
||||
`83018fc` (2.4.1, `INSTALL-MCP.md` and its wiring)
|
||||
- **Tests:** 776 -> 786 -> 799 passed, green throughout, including on an `env -i` empty machine
|
||||
- **Measured:** corpus parse 265ms -> 54ms; `wikitool search` end-to-end 593ms -> 347ms
|
||||
- **Issues:** #34, #33, #31, #19 closed with detailed comments; #36 commented, left open pending
|
||||
#37; #37 opened (`prio/2 size/M`)
|
||||
- **CI:** `.gitea/workflows/ci.yml` updated to install `tools/requirements-mcp.txt`, otherwise
|
||||
unchanged; not separately re-run in this session (publish triggers it)
|
||||
- **Not done in this session, carried forward:** #37 itself (container image); #23 (env var
|
||||
registration enforcement - `CHEMENU_ROOT` was added to `_WIKITOOL_ENV` by hand); the `qmd.md`
|
||||
language-claim fix (separate transcript); an ADR-style home for architecture decisions
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
# Conversation Transcript - Private-Instance Merge Correction and Issue 30 Session
|
||||
|
||||
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
|
||||
> Collected: 2026-09-01
|
||||
> Participant: Torben
|
||||
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's questions and
|
||||
> instructions are quoted verbatim; the assistant's reasoning is condensed. All shell output
|
||||
> shown was produced by real, disposable test repositories built in a scratch directory during
|
||||
> this session (not the chemenu repository itself) and is reproduced as run.
|
||||
> No credentials appeared in the session.
|
||||
> Third of three transcripts cut from this session; the first two cover the corpus/history work
|
||||
> and the Publish-Remote Gate this one corrects a claim about.
|
||||
|
||||
Covers Torben asking what actually happens to the demo corpus in a private instance when the
|
||||
upstream changes it - which exposed an unverified, incorrect claim the assistant had just
|
||||
written into `instructions/private-instance.md` the same day. The claim was tested empirically,
|
||||
found wrong, corrected in a released patch (2.2.1), and turned into a properly scoped, tested
|
||||
merge procedure plus a new issue (#30) proposing the fix belongs in the tool itself, or that the
|
||||
upstream should stop shipping content on the branch private instances pull machinery from.
|
||||
|
||||
---
|
||||
|
||||
## Turn 1 - "was passiert in der privaten instanz nach diesen schema wenn sich upstream der demo-korpus ändert?"
|
||||
|
||||
The assistant treated this as a request to verify, not answer from the existing text - the
|
||||
previous transcript's `private-instance.md` had said a `git merge upstream/main` would see
|
||||
*deleted-in-ours, unmodified-in-theirs* and resolve silently, which was written as an assumption
|
||||
and never tested.
|
||||
|
||||
Built a throwaway upstream/private pair of git repos in the scratch directory to test it: private
|
||||
instance deletes the demo corpus once, upstream then modifies one demo page, adds a new one, and
|
||||
deletes another. Ran a plain `git merge upstream/main`. Result, measured rather than assumed:
|
||||
|
||||
- The upstream's **modified** page produced `CONFLICT (modify/delete)` and left the upstream's
|
||||
version sitting in the working tree - resolving it the obvious way (`git add -A`) would commit
|
||||
that version, bringing the deleted demo page back.
|
||||
- The upstream's **new** page was staged silently. No conflict, no message.
|
||||
- The upstream's **deleted** page (matching a page already deleted on the private side) produced
|
||||
nothing - the one case that actually worked as claimed.
|
||||
|
||||
The middle case was the one the original claim missed entirely, and it is the dangerous one
|
||||
because nothing announces it - a demo page the upstream adds simply arrives and starts showing
|
||||
up in the private instance's own `lint`, `index`, `search` and `confidence decay`.
|
||||
|
||||
A second candidate fix was tested and also failed: a `.gitattributes` `merge=ours` driver for
|
||||
`kb/**`/`raw/**`. Same three-scenario test showed the modify/delete conflict still occurred and
|
||||
the new page still landed - the driver only resolves content conflicts on files both sides
|
||||
touch, not modify/delete pairs or additions.
|
||||
|
||||
## Turn 2 - Fixing the instruction, in a released patch
|
||||
|
||||
`instructions/private-instance.md` was rewritten: the "Why a clone rather than a tarball"
|
||||
section now states the measured three-case table instead of the wrong assumption. The update
|
||||
procedure was rewritten to hold the merge open (`git merge --no-commit --no-ff`), force the
|
||||
content stages (`kb/`, `raw/`) back to the pre-merge state while `HEAD` still points at the
|
||||
commit before the merge, commit, and then run an explicit check
|
||||
(`git diff --name-only $BEFORE HEAD -- kb raw` must be empty) - a check the reader cannot silently
|
||||
skip past, unlike a claim they might trust. The bad advice in "Decision points" ("resolve a
|
||||
conflict under kb/ as keep-deleted") was replaced, since that advice is exactly what leads
|
||||
someone to `git add -A` the leaked content.
|
||||
|
||||
The corrected procedure was itself run against the same three-scenario test plus a fourth
|
||||
(`raw/` alongside `kb/`), and against error paths (dirty working tree rejected, a second run
|
||||
with nothing new to pull is a no-op) - all passing before publishing.
|
||||
|
||||
Version bumped `2.2.0 -> 2.2.1`. The changelog entry states plainly that the earlier text was
|
||||
"nicht gemessen, sondern angenommen" (not measured, assumed) rather than framing it as a minor
|
||||
wording fix. Published; tests green (752, unchanged in count - this was a documentation fix).
|
||||
|
||||
## Turn 3 - "stelle mal ein sauberes skripting hier exemplarisch dar. gibt es alternative setups? könnten wir die demo/test-seiten aus dem repo main irgendwie heraushalten? erstelle aus dem ganzen thema ein issue... aktualisiere/prüfe, ob #3 im gitea sauber ist. prüfe, ob wir sonst alle offenen enden verdrahtet haben."
|
||||
|
||||
Four asks in one message. Handled in order:
|
||||
|
||||
**A standalone script**, written and tested (not just described) against the same scenario:
|
||||
rejects a dirty working tree, rejects a concurrent merge/rebase, holds the merge open the same
|
||||
way the instruction's procedure now does, is a no-op on a second run with nothing new, and exits
|
||||
non-zero with a rollback command if the post-merge diff check ever finds leaked content - the
|
||||
check the manual procedure relies on a human to run, made unskippable.
|
||||
|
||||
**Alternative setups**, and the negative result from Turn 1 (`merge=ours` does not work) was kept
|
||||
rather than omitted, since a rejected alternative is exactly the kind of finding this capture
|
||||
procedure is meant to preserve.
|
||||
|
||||
**Whether the demo corpus can be kept out of `main` entirely** - assessed as the better fix in
|
||||
principle: the private-instance script (and the instruction's manual procedure) treats a symptom
|
||||
on every downstream instance, repeatedly; keeping content off the branch machinery is pulled from
|
||||
would remove the need for either, by construction, for any new instance. Named as two variants
|
||||
(a separate `demo` branch, or a separate demo repository) with the real cost stated rather than
|
||||
glossed over - `main` becomes a content-free shell, which was the reason the corpus was kept in
|
||||
the first place (as a walkable example), and `nightly.yml`'s corpus lint would need to move.
|
||||
|
||||
**Issue #30 opened**, carrying the measured table, the working script, both alternatives with
|
||||
their tradeoffs, and the explicit note that it is not a duplicate of #28 but depends on it - #28
|
||||
is why the corpus moves at all (demo and testbed sharing one `kb/`), #30 is what that movement
|
||||
does to a downstream clone.
|
||||
|
||||
**Issue #3 checked against its own acceptance criteria** (all five, from the closed issue's own
|
||||
text) rather than assumed closed-and-fine: a grep for the pre-rename name outside history/
|
||||
changelog/transcripts, presence of a project entity page, the "Thoth" persona note in both
|
||||
`SOUL.md` and its template, and no active technical identifier still using the old name. Found
|
||||
clean - the handful of remaining mentions (`INSTALL.md`, two `kb/` pages) were all phrased as
|
||||
history ("bis 2026-09-01", "damals"), which the criteria explicitly exempt.
|
||||
|
||||
**The wiring audit** turned up the two gaps described in the second transcript's closing turn
|
||||
(`tools/CONTRACT.md` missing the new gate, `gates.md` not linking to `private-instance.md`) and
|
||||
the stale project-page description - all three fixed in the same pass and published as 2.2.2,
|
||||
verified with `docs verify`, `instructions verify`, `doctor` (16 checks OK), `lint` (clean but for
|
||||
one pre-existing orphan page unrelated to this work), and the full test suite.
|
||||
|
||||
## Turn 4 - This capture
|
||||
|
||||
Torben asked for this session to be captured via `instructions/capture-session.md`, with an
|
||||
explicit instruction to keep private and confidential material out - consistent with, not
|
||||
separate from, everything the day's work had been about. Handled by generalizing every
|
||||
infrastructure and personal-data reference that appeared during the day's audits rather than
|
||||
reproducing the original identifiers, cutting the session into three topic transcripts (this
|
||||
being the third), and filing this transcript's own existence as evidence rather than as new
|
||||
open work - nothing in this turn required a new issue.
|
||||
|
||||
## Outcome
|
||||
|
||||
- **Version:** 2.2.0 -> 2.2.1 -> 2.2.2
|
||||
- **Commits:** `private-instance.md` correction (2.2.1), documentation wiring + project-page
|
||||
rewrite (2.2.2, shared with the second transcript's closing turn)
|
||||
- **Tests:** 752, green throughout
|
||||
- **Issues:** #30 opened; #3 verified against its own acceptance criteria and confirmed clean,
|
||||
no action needed
|
||||
- **CI:** green after each publish in this transcript's scope
|
||||
+208
@@ -0,0 +1,208 @@
|
||||
# Conversation Transcript - Public Release, Corpus Purge and History Squash Session
|
||||
|
||||
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
|
||||
> Collected: 2026-09-01
|
||||
> Participant: Torben
|
||||
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instructions and
|
||||
> decisions are quoted verbatim; the assistant's reasoning and the exploration agents' findings
|
||||
> are condensed. Command outputs shown are real, but every private hostname, IP address and
|
||||
> personal detail that appeared during the audit has been **generalized rather than repeated** -
|
||||
> reproducing them here would undo the point of the session. No secret values (tokens, keys)
|
||||
> ever appeared.
|
||||
> **Several passages are second-hand.** Three `Explore` subagents (private-data inventory,
|
||||
> distribution mechanism, git-history audit) and three forked debate agents (hard-separation,
|
||||
> shared-upstream, devil's-advocate positions) ran in the background; their findings are
|
||||
> summarized here, not reproduced verbatim, and are named as such at each point they inform a
|
||||
> decision.
|
||||
> One of three transcripts cut from the same session; the other two cover building the
|
||||
> Publish-Remote Gate and the private-instance procedure, and correcting a bug found in that
|
||||
> procedure afterward (issue #30).
|
||||
|
||||
Covers turning the chemenu wiki stack from a private testbed into a publishable repository:
|
||||
deciding what to delete versus anonymize, squashing the git history, choosing a license, and
|
||||
verifying the result anonymously end to end. Resulted in the corpus purge (108 pages, ~90 raw
|
||||
files removed), a single-commit public history, `LICENSE`/`LICENSE-CONTENT`/`NOTICE`, the
|
||||
`dist export` leak canary (2.1.0), a CI fixture fix (2.1.1), and the repository going public on
|
||||
Gitea.
|
||||
|
||||
---
|
||||
|
||||
## Turn 1 - "ich möchte die kb hier bereinigen, es soll nur noch test content sein, ich möchte das repo veröffentlichen..."
|
||||
|
||||
Torben's instruction, in substance: clean the KB down to test content, publish the repository,
|
||||
falsify or remove IP addresses, keep the repo usable as a demo and a testbed going forward; he
|
||||
would rebuild his real private instance separately. He asked for tips on three specific
|
||||
questions - how a private fork could pull upstream fixes back, how to set up that private copy
|
||||
and where to develop the stack, and whether a `kb/manual/` should ship with the distribution so
|
||||
it documents itself - explicitly requested "mindestens ... teufels anwalt modus" and a small
|
||||
multi-agent debate, and flagged himself as new to packaging/distributing an LLM application.
|
||||
|
||||
The session started in plan mode. Three `Explore` subagents ran in parallel:
|
||||
|
||||
- **Private-data inventory** - catalogued what in `raw/` and `kb/` was hard-private (a real
|
||||
home-lab cluster's bootstrap manual, several personal notes, a 2.46 MB conversation transcript
|
||||
full of infrastructure references), what was anonymizable (internal hostnames, private IP
|
||||
ranges), and what was legitimately generic (the stack's own concepts, technologies, tools).
|
||||
- **Distribution mechanism** - read `dist_cmd.py`, `INSTALL.md` and the version-check machinery;
|
||||
confirmed `dist export` already ships no `kb/`/`raw/` content and that the update feed's
|
||||
default URL pointed at the (then-private) origin repo.
|
||||
- **Git-history audit** - measured `.git` size, largest blobs, deleted-file history, and
|
||||
concluded the tree was "safe to publish" - a conclusion the session later disproved, see Turn 4.
|
||||
|
||||
## Turn 2 - The debate
|
||||
|
||||
Three forked debaters argued from the same context:
|
||||
|
||||
- **A (hard separation):** delete and rebuild the corpus rather than anonymize, because a page
|
||||
title is the wiki's only identifier and lives in wikilinks/citations/frontmatter across the
|
||||
corpus - renaming is the full page-lifecycle procedure, deleting is a supported command.
|
||||
Squash history rather than selective `filter-repo`, because the history audit itself had just
|
||||
been shown to under-report risk. No shared git ancestry between the public repo and the
|
||||
private instance, ever.
|
||||
- **B (shared upstream):** clone-with-`upstream`-remote instead, because the tarball update path
|
||||
in `INSTALL.md` is `cp -r` with no three-way merge; argued the leak risk from shared history is
|
||||
a mechanism problem (a remote allowlist), not a discipline problem, and that most of the
|
||||
corpus (the ~190 generic pages) is worth keeping as real test data rather than deleting it all.
|
||||
- **C (devil's advocate):** attacked both positions' shared premises - "just a testbed" was
|
||||
false (`USER.md` is tracked and carries real personal data), "publish the repo" was
|
||||
underspecified (which host, and what breaks), there was no `LICENSE` at all, a shipped manual
|
||||
would decay under `confidence_decay.py` and mislead a new instance, and demo/testbed are
|
||||
conflicting goals for one `kb/`.
|
||||
|
||||
The assistant's synthesis kept A's "delete, don't anonymize" and "squash the history" (C's
|
||||
finding that the audit under-reported risk made this decisive), adopted B's clone-with-`upstream`
|
||||
model but **conditioned it on a remote-allowlist gate existing first** (addressing C's leak
|
||||
concern with code, not discipline), rejected the shipped manual per C's decay argument, and
|
||||
decided the surviving corpus should lean toward the stack's own self-documentation rather than
|
||||
Torben's real infrastructure - which the actual corpus math later validated (Turn 5).
|
||||
|
||||
## Turn 3 - Clarifying questions and plan approval
|
||||
|
||||
The assistant asked Torben to choose among concrete options (`AskUserQuestion`) rather than
|
||||
picking on his behalf:
|
||||
|
||||
- **Publish target:** "Gitea öffentlich schalten" (chosen) over mirroring to GitHub or keeping
|
||||
Gitea private with a GitHub mirror - smallest change, CI/release feed stay correct as-is.
|
||||
- **Corpus:** "Chirurgisch löschen (Empfehlung)" (chosen) over full synthetic rebuild or
|
||||
anonymization.
|
||||
- **Private instance:** "Clone mit upstream, Gate zuerst (Empfehlung)" (chosen) over hard
|
||||
separation or a later migration.
|
||||
- **License:** first pass "Was wäre CC-BY-4 für content und eine GPL oder LGPL fürs Tooling?
|
||||
Berate mich hier noch mal" - the assistant advised against LGPL (no library-linking use case
|
||||
here to justify the fetch-only variant), and named the real choice as GPL vs. **AGPL**, because
|
||||
issue #19 (an MCP frontend for `wikitool`) points toward running the stack as a network
|
||||
service, which is exactly the gap GPL leaves and AGPL closes. Torben chose **AGPL-3.0 +
|
||||
CC-BY-4.0**.
|
||||
|
||||
The written plan covered: publish blockers (license, `dist export` allowlist, `INSTALL.md`
|
||||
token wording), the corpus purge, the history squash, the private-instance model with the gate
|
||||
as a precondition, why the manual idea was declined, and a leak-canary for `dist export`. Torben
|
||||
approved it via `ExitPlanMode` without further changes.
|
||||
|
||||
## Turn 4 - Executing the license and leak-canary work (2.1.0)
|
||||
|
||||
Before touching content: a full repo bundle backup outside the tree, verified by `git bundle
|
||||
verify` and a commit-count comparison (150 commits, all 15 tags present).
|
||||
|
||||
- Fetched the AGPL-3.0 text from `gnu.org` (not reconstructed from memory - a license must be
|
||||
the authoritative text) and copied the CC-BY-4.0 text from the vendored `commonplace`
|
||||
submodule, per the license each already carries.
|
||||
- Added `LICENSE` (AGPL - deliberately the AGPL rather than a separate `LICENSE-CODE`, because
|
||||
that is the file a forge reports for the repository, and under-noticing a copyleft obligation
|
||||
harms a reader in a way over-noticing does not), `LICENSE-CONTENT` (CC-BY), `NOTICE` (the
|
||||
license boundary and the `commonplace` attribution CC-BY requires).
|
||||
- `dist_cmd.py`: added the three license files to `ROOT_FILES`, and a separate
|
||||
`REQUIRED_ROOT_FILES` check that **fails the export** if they are missing - every other
|
||||
`ROOT_FILES` entry is copied `if source.is_file()` and silently skipped otherwise, which is
|
||||
wrong for a license (a distribution shipping AGPL code with no license text is a violation the
|
||||
moment it is published).
|
||||
- Added `find_leaks()`: a structural (not text-pattern) check of the finished export plan against
|
||||
personalization files, `instructions/dev/`, `kb/` pages and `raw/` sources - rejected a
|
||||
hostname/IP text-scan approach, because the project's own host legitimately appears in
|
||||
`INSTALL.md` and `version.py`, so such a scan would either whitelist the string it's looking
|
||||
for or false-positive on every export.
|
||||
- Six new tests in `test_dist_cmd.py`; full suite green; version bumped `2.0.0 -> 2.1.0`.
|
||||
|
||||
## Turn 5 - The corpus purge
|
||||
|
||||
A workshop (`work/publish-cleanup`) was opened per `work/CONTRACT.md`, because ~103 planned
|
||||
`wikitool rm` calls exceed a single iteration-budget unit.
|
||||
|
||||
Analysis found the plan's estimate wrong in a way worth recording: not ~35 pages would become
|
||||
sourceless after deleting the obviously-private ones, but **75**. Splitting them: 28 were the
|
||||
user's own infrastructure (unambiguous), and **40 carried generic technology titles but were
|
||||
written entirely around the user's own cluster** (a storage page explained itself via one
|
||||
specific storage class name, a Kubernetes page via the user's own network ranges) - not
|
||||
reusable pages with a private example, but private documentation with a generic heading. The
|
||||
remaining 7 were genuinely clean in body text but sourced from the user's personal
|
||||
document-processing pipeline.
|
||||
|
||||
Presented to Torben as a three-way choice; he chose **"Alle 75 löschen"** over keeping the
|
||||
clean 7 (via provenance reclassification) or rewriting the 40 as vendor-neutral pages. The
|
||||
result: 152 pages plus 25 sources remained by design - the stack's own self-documentation
|
||||
(gates, lint, versioning, search, the wiki pattern itself), which the earlier debate had argued
|
||||
for without knowing this would be the actual outcome.
|
||||
|
||||
Execution: `wikitool rm --page <title> --yes` per page across three units (27 sources, 28
|
||||
infra, 48 entangled), 90 raw files removed including the full cluster bootstrap tree and the
|
||||
2.46 MB transcript. `wikitool rm` was found to de-link only mechanically (frontmatter refs and
|
||||
whole-line link bullets) and to deliberately leave inline prose wikilinks and plain-text
|
||||
mentions standing - roughly 20 pages needed manual follow-up to actually remove the remaining
|
||||
references, which `lint` and a targeted `rg` scan surfaced.
|
||||
|
||||
Also reset: `kb/log.md` (116 private references, no regenerator - reset to the
|
||||
`dist_templates/log.md` starting state, a deliberate one-time stack-dev operation, not a
|
||||
hand-edit of a generated file), `USER.md`/`SOUL.md` rewritten as an explicit demo-operator
|
||||
persona rather than Torben's real profile.
|
||||
|
||||
## Turn 6 - Squashing the history
|
||||
|
||||
Backup re-verified before the destructive step. An orphan root commit was built from the purged
|
||||
working tree, `main` reset onto it, all 15 tags deleted (locally, then individually on the
|
||||
remote - a batch refspec push failed silently), `git reflog expire --expire=now --all` and
|
||||
`git gc --prune=now --aggressive`, then a `--force` push.
|
||||
|
||||
**Verified rather than trusted:** a fresh clone attempted `git fetch --depth=1 origin
|
||||
<old-head-sha>` immediately afterward and succeeded - the objects were still reachable through
|
||||
Gitea's own reflog on the bare repository, contradicting the earlier belief that force-pushing
|
||||
was sufficient. The rejected shortcut here was accepting Gitea's own scheduled cleanup jobs as
|
||||
proof; Torben ran them once, and the same fetch test still succeeded afterward. Only a direct
|
||||
`reflog expire` + `gc --prune=now` run on the bare repository itself (which Torben ran, having
|
||||
shell access the assistant did not) closed it - reverified by the same fetch test returning
|
||||
`not our ref`.
|
||||
|
||||
A second leak was found by scanning **hidden directories** (`.gitea/`, `.github/`, `.vibe/`),
|
||||
which the earlier `rg` sweeps had silently skipped without `--hidden` - two references to an
|
||||
internal CI branch name in `.gitea/workflows/ci.yml`, fixed and amended into the squashed
|
||||
commit before the first push.
|
||||
|
||||
## Turn 7 - Public switch, CI fixture bug, and verification
|
||||
|
||||
Torben switched the Gitea repository to public. Verified anonymously (no token, no SSH key):
|
||||
repository API reports `private: false`, the release feed serves the latest tag, and a clean
|
||||
`git clone` over HTTPS from an empty directory succeeds. A further leak scan of that anonymous
|
||||
clone found one remaining hit: the 2.1.1 changelog entry (see below) had *listed* the private
|
||||
fixture names it replaced, which put them back into the public history it was announcing the
|
||||
removal from. Corrected in the same commit that introduced it.
|
||||
|
||||
Separately, the first CI run on the squashed history failed on a test unrelated to the corpus
|
||||
purge in content but caused by it: `test_legacy_source_pages_flags_url_and_directory` checks
|
||||
`(config.ROOT / legacy).is_dir()` against the **real** repository root rather than the test
|
||||
fixture's own tree, and had only ever passed because this checkout happened to have a
|
||||
`raw/documents/` directory - which the corpus purge had just emptied. Git does not track empty
|
||||
directories, so the directory vanished from CI's checkout and stayed in the local one: green
|
||||
here, red there. Reproduced locally by removing the directory and rerunning; fixed in the
|
||||
`raw_dir` fixture (`conftest.py`) rather than the one test, matching the reasoning already
|
||||
recorded for a prior, similar case (Gitea #8). Several other test fixtures still used the
|
||||
user's real system names and were renamed to generic, unrelated placeholders
|
||||
alongside. Published as 2.1.1 after Mass-Update Gate clearance from Torben.
|
||||
|
||||
## Outcome
|
||||
|
||||
- **Version:** 2.0.0 -> 2.1.1
|
||||
- **Commits:** license/leak-canary work, corpus purge (multiple), history squashed to one root
|
||||
commit, CI fixture fix
|
||||
- **Tests:** 630 -> 752, all green
|
||||
- **Issues:** none opened in this transcript's scope (see the other two transcripts for #27-#30)
|
||||
- **CI:** green on the squashed history after the fixture fix; release `v2.1.1` built and
|
||||
anonymously downloadable
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
# Conversation Transcript - Publish-Remote Gate and Issue Triage Session
|
||||
|
||||
> Source: Claude Code session (`claude-opus-5`), chemenu workspace
|
||||
> Collected: 2026-09-01
|
||||
> Participant: Torben
|
||||
> Fidelity: **faithful summary transcript, not a verbatim log.** Torben's instructions are
|
||||
> quoted verbatim; the assistant's design reasoning is condensed. Command outputs and test
|
||||
> results shown are real. No credentials appeared in the session; where a private hostname
|
||||
> could have been used as a code example, a generic placeholder is used instead, matching what
|
||||
> was actually committed.
|
||||
> Continues directly from "Public Release, Corpus Purge and History Squash Session" (same day).
|
||||
> Two of three transcripts cut from this session; the third covers a bug found afterward in the
|
||||
> procedure this one builds (issue #30).
|
||||
|
||||
Covers building the Publish-Remote Gate (a third code-enforced gate, alongside the Mass-Update
|
||||
and rebase-review gates), writing `instructions/private-instance.md`, updating six open Gitea
|
||||
issues for the rename and the new architecture, opening three new issues (#27, #28, #29), and
|
||||
deleting the Gitea Actions run history at Torben's request. Resulted in version 2.2.0, then a
|
||||
documentation-only 2.2.2.
|
||||
|
||||
---
|
||||
|
||||
## Turn 1 - "1. remote allowlist. 2. unterbreche und warte auf mich, wenn ich gitea scharf schalten soll... 3. setup a) meiner lokalen dev-umgebung"
|
||||
|
||||
Torben asked for three things in one message, plus: "parallel issuen anlgen und aktualisieren
|
||||
nach eigenem ermessen, prüfe auch vorhandene, evtl. müssen wir hier auch noch tätig werden?"
|
||||
|
||||
**The gate.** Designed to pin the **resolved push URL**, not the remote name - a name-based
|
||||
allowlist would pass a `publish` whose `origin` had been repointed, which is the exact failure
|
||||
it exists to catch. `git remote get-url --push <remote>` is read at check time so a `pushurl`
|
||||
override is respected. The file (`.wikitool-remotes.json`) is gitignored and per-checkout, for
|
||||
the same reason `ENVIRONMENT.md` is: two clones push to two different places, and a committed
|
||||
copy would tell a private clone the public upstream is a legitimate target for its own content.
|
||||
Absence means unrestricted, matching the pattern of the other optional per-checkout files;
|
||||
`doctor` reports the state and WARNs only when a checkout has more than one remote and no
|
||||
allowlist.
|
||||
|
||||
**Deliberately no `--confirm` token**, unlike the other two gates. Their question ("is this
|
||||
change right?") is answerable per changeset; this gate's question ("does this content belong in
|
||||
that repository?") is a standing property of the checkout, so the only way past it is the user
|
||||
editing the file themselves - an agent editing it to clear a refusal would be opening a gate on
|
||||
its own initiative, which the repository's own rules forbid.
|
||||
|
||||
Implementation: `config.PUBLISH_REMOTES_FILENAME`, `read_allowed_push_urls()` /
|
||||
`push_url_for()` / `publish_remote_refusal()` in `git_publish.py`, checked before the reconcile
|
||||
step in `publish_command` (before any network contact, so a refused publish never even fetches
|
||||
from the wrong place), `doctor.check_publish_remotes()`, twelve new tests covering the URL-vs-name
|
||||
distinction, `pushurl` precedence, a broken/missing/empty allowlist, and that no flag exists to
|
||||
bypass it. `instructions/gates.md` and `AGENTS.md` updated to describe a third gate. Version
|
||||
bumped `2.1.1 -> 2.2.0`; the Mass-Update Gate itself fired at 10 files and Torben cleared it with
|
||||
the printed token.
|
||||
|
||||
**`instructions/private-instance.md`** (new): the procedure for cloning with the public repo as
|
||||
`upstream`, arming the gate *before* the first `publish` (not after - the assistant stressed this
|
||||
ordering explicitly, since a gate added later leaves the earlier window open), taking the write
|
||||
credential away from the private clone as a second, independent control, and where stack
|
||||
development happens ("in the public repo, not here" - not a preference but a structural fact,
|
||||
since `instructions/dev/` does not survive `dist export`). **This file's description of what a
|
||||
`git merge upstream/main` actually does to the content stages was wrong as first written** - see
|
||||
the third transcript for the correction.
|
||||
|
||||
**The Gitea admin cheat-sheet** (delivered as a reply, not committed): what to check before
|
||||
flipping the repo public (`DISABLE_REGISTRATION`, `REQUIRE_SIGNIN_VIEW`, rate limits, the
|
||||
Actions runner's network exposure), and the follow-up steps (`INSTALL.md`, anonymous release
|
||||
check, a clone test). Also flagged that the open issues would go public with the repo, which is
|
||||
what motivated the issue-triage pass below.
|
||||
|
||||
## Turn 2 - Issue triage
|
||||
|
||||
Read the labels (`prio/1..3`, `size/XS..L`) and the open issues. Several pre-dated the
|
||||
`llm-wiki-test1` -> `chemenu` rename (issue #3) and still named the old package path
|
||||
(`tools/wiki_tools/...`) or the old repository name in code examples:
|
||||
|
||||
- **#6** title corrected in place (`wiki_tools` -> `chemenu` path).
|
||||
- **#7** (`dist upgrade`) commented: the "origin repo is private" fallback it describes no
|
||||
longer applies once the repo is public, and its urgency for *this* instance specifically
|
||||
dropped, because the private instance now takes updates via `git merge upstream/main` (real
|
||||
three-way merge) rather than the tarball-copy path the issue was written against - it remains
|
||||
the right design for any instance without shared git history.
|
||||
- **#10** (coverage reporting) commented: step 1 is done (1.8.1 shipped `pytest-cov` without a
|
||||
failure threshold), the test count referenced is stale (630 -> 752, and the gap is itself
|
||||
evidence for the issue's own argument - the raw_dir fixture bug from the first transcript), and
|
||||
one code example named a private CI branch and needs neutralizing before being read publicly.
|
||||
- **#4, #5** flagged as needing the same path correction but not rewritten in this pass.
|
||||
|
||||
**Three new issues opened**, each traced to a decision made and then set aside earlier in the
|
||||
day rather than invented fresh:
|
||||
|
||||
- **#27** - the shipped-manual idea the debate rejected (first transcript, Turn 2) needs a
|
||||
decay/lint exemption for distributed pages before it becomes buildable at all; names the exact
|
||||
blocker (`confidence_decay.py` has no exemption path) and the two things still undecided (how
|
||||
a shipped page is recognized, what exactly is exempted).
|
||||
- **#28** - the demo-vs-testbed tension the debate argued through (assistant recommended
|
||||
testbed, C objected) is not resolved, only decided for now; proposes a curated fixture corpus
|
||||
as the eventual answer and names why (`kb_dir`/`raw_dir` fixtures already cover the small case,
|
||||
this is about the dense, interconnected one).
|
||||
- **#29** - catalogs which open issues still carry pre-rename paths and a private hostname
|
||||
(#10's CI example), separating "pointer, fix it" references from "citation of what was true
|
||||
then, date it" references rather than blanket-editing everything.
|
||||
|
||||
## Turn 3 - Public verification and the actions-history request
|
||||
|
||||
After Torben confirmed the repository was public, verification ran anonymously: repo API
|
||||
(`private: false`), release feed (latest tag served without auth), a clean HTTPS clone from an
|
||||
empty directory (three commits, two tags, all three license files present, 170 pages, no leak
|
||||
hits including in hidden files), and the full `INSTALL.md` release-tarball path end to end -
|
||||
checksum verified, license files present in the unpacked tree, `kb/`/`raw/` empty as designed,
|
||||
`instructions/dev/` absent.
|
||||
|
||||
Torben: "an der Stelle: wenn du per mcp die action runner historie löschen kannst, leere sie
|
||||
bitte komplett, da finde ich keinen knopf für und dort dürften noch alte tarballs herumgeistern."
|
||||
The assistant checked: the Gitea MCP server exposes `dispatch`/`cancel`/`rerun` for Actions runs
|
||||
but no delete, and the delete route Gitea's own API defines (`DELETE
|
||||
/api/v1/repos/{owner}/{repo}/actions/runs/{run}`) requires a token the assistant did not hold.
|
||||
Findings reported instead of silently working around the missing capability: **zero artifacts**
|
||||
existed already (confirmed via the anonymous artifacts API), but an anonymous SPA data endpoint
|
||||
the run pages call was readable and returned old commit titles from the squashed-away history -
|
||||
no hostnames or personal data among the six reachable runs, but development history the squash
|
||||
was meant to remove. A `curl` loop using a write-scoped token was handed to Torben to run
|
||||
himself, rather than asking for a token in the session.
|
||||
|
||||
Torben ran it and reported back ("die action-historie habe ich jetzt per api gelöscht") in the
|
||||
same message as "korrigiere die install.md wie vorgeschlagen." Both handled: the anonymous SPA
|
||||
endpoint was re-tested across all previously-reachable run numbers and returned nothing for any
|
||||
of them, closing the loop; `INSTALL.md`'s "repo is currently private, you need a token" language
|
||||
was removed from three places (the download command, the configuration table, a dedicated
|
||||
paragraph - the paragraph rewritten to explain the token is still needed **for a private fork**,
|
||||
not for this repository), "Weg C" was reworded from "Torben's personal wiki" to a description of
|
||||
the testbed/demo nature of the corpus, and a new "Weg D" documented the private-instance path
|
||||
with a pointer to `private-instance.md`. No version bump (prose only).
|
||||
|
||||
## Turn 4 - Wiring audit (from the third transcript's closing turn, included here for the gate/issue thread)
|
||||
|
||||
A later pass (opened by a different question, documented fully in the third transcript) checked
|
||||
whether the gate work here was actually cross-referenced: `tools/CONTRACT.md` did not mention
|
||||
the Publish-Remote Gate at all despite it being `publish`'s third exit-42 path - fixed in both
|
||||
the command-table entry and the error-contract entry, since a tool's error contract is exactly
|
||||
where a caller learns what a given exit code means and whether retrying is safe. `gates.md` did
|
||||
not link to `private-instance.md`, the procedure it exists for - fixed. The project's own KB
|
||||
page (`kb/entities/projects/Chemenu.md`) still described the instance as a personal wiki with no
|
||||
mention of the license or its public, testbed-and-demo status - rewritten, keeping the
|
||||
historical note about the pre-restructuring `AGENTS.md` but dating it explicitly rather than
|
||||
stating it as a current fact. Published as 2.2.2.
|
||||
|
||||
## Outcome
|
||||
|
||||
- **Version:** 2.1.1 -> 2.2.0 -> 2.2.2
|
||||
- **Commits:** Publish-Remote Gate + `private-instance.md` (2.2.0, Mass-Update Gate cleared),
|
||||
`INSTALL.md` correction (unversioned prose), documentation wiring + project-page rewrite
|
||||
(2.2.2)
|
||||
- **Tests:** 752, all green throughout (12 new for the gate)
|
||||
- **Issues:** #6 title fixed; #7, #10 commented; #27, #28, #29 opened
|
||||
- **CI:** green on every push in this transcript's scope; Gitea Actions run history removed by
|
||||
Torben via the API, verified anonymously empty afterward
|
||||
+32
-6
@@ -50,7 +50,7 @@ tools/wikitool <command> --help
|
||||
| `log append --op ingest\|query\|lint\|create\|update\|delete\|rename --title "..." [--body "..."\|--body-file path]` | Append a formatted entry to `kb/log.md` |
|
||||
| `log status` | Read-only: count `ingest` entries logged since the last `lint` entry - the deterministic trigger behind the Maintenance Schedule's "every 10 sources" full-lint cadence |
|
||||
| `lint [--json] [--markdown out.md] [--full] [--fail-on-error]` | Structural + provenance checks: broken wikilinks, dangling frontmatter references, orphan pages, index drift, schema gaps, duplicate titles, title mismatches, uncovered raw files, broken `raw_files:` refs, raw files claimed by more than one source page, unmarked provenance, citation/frontmatter drift, quote-limit overages (>2 blockquoted lines/page, advisory only). Prints only the sections that found something and always writes the full report to `reports/Lint Report <date>.md` (or `--markdown`), naming the path - `--full` prints everything, `--json` prints the findings and writes nothing |
|
||||
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary/confidence so a hit can be judged without opening the page. Read-only, and **exempt from the Iteration Budget Gate** |
|
||||
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary/confidence so a hit can be judged without opening the page. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
|
||||
| `confidence decay [--apply]` | Recompute every page's derived `confidence` as `confidence_base * (1 - 0.01/month)`, floored at 0.2; dry-run by default |
|
||||
| `confidence init-base [--apply]` | One-time backfill: set `confidence_base` from the current `confidence` on pages that predate the derived-confidence model |
|
||||
| `sources coverage [--json]` | List raw files with no source page, broken `raw_files:` references, and legacy directory/URL-only source pages |
|
||||
@@ -74,7 +74,7 @@ tools/wikitool <command> --help
|
||||
| `version show [--json]` | Print this instance's stack version and where it came from (development tree, or a distribution with its export date and origin). Bare `wikitool version` is an alias for this. Read-only, offline, and **exempt from the Iteration Budget Gate** |
|
||||
| `version check [--url U] [--timeout S] [--json]` | Ask the origin's release feed whether a newer stack exists, and whether the step crosses a compatibility boundary (`state: current\|update\|migration\|ahead`). **The only command in `wikitool` that makes a network call** - never reached implicitly from another command, needs no key, times out, and reports an unreachable feed as an error rather than as "up to date". The feed is `$WIKITOOL_UPDATE_URL`, else the release stamp's, else the built-in origin; `$WIKITOOL_UPDATE_TOKEN` is only needed if that feed is not readable anonymously. Read-only and exempt from the budget gate |
|
||||
| `version notes [--version X.Y.Z]` | Print one version's `CHANGES.md` entry, for use as release notes (default: this tree's `VERSION`). Read-only and exempt from the budget gate |
|
||||
| `version bump --major\|--minor\|--patch --title "<...>" [--no-migration "<reason>"] [--dry-run]` | Raise `VERSION` and open the matching `CHANGES.md` entry - heading, date and author only; the body stays the author's to write, the way `new` writes frontmatter and leaves the prose. Refuses more or fewer than one part, an empty title, and a changelog already documenting a version that is not older than the new one. Compatibility follows the **leftmost non-zero component**, which for this stack (at `1.0.0` and up, no pre-release suffixes anywhere) means MAJOR: PATCH is a fix, MINOR a compatible capability, MAJOR requires instances to migrate. A MAJOR bump additionally requires a migration document targeting the new version, or `--no-migration "<reason>"`, which records the reason in the entry |
|
||||
| `version bump --major\|--minor\|--patch --title "<...>" [--breaking "<what breaks>"] [--no-migration "<reason>"] [--dry-run]` | Raise `VERSION` and open the matching `CHANGES.md` entry - heading, date and author only; the body stays the author's to write, the way `new` writes frontmatter and leaves the prose. Refuses more or fewer than one part, an empty title, and a changelog already documenting a version that is not older than the new one. Compatibility follows the **leftmost non-zero component**, which for this stack (at `1.0.0` and up, no pre-release suffixes anywhere) means MAJOR: PATCH is a fix, MINOR a compatible capability, MAJOR a version that is **not a drop-in replacement** - any hand-work on update, or a downgrade that no longer works. Whether content must be migrated is a second, independent question. A MAJOR bump therefore requires `--breaking "<what stops working>"`, which is refused on any other part, and on top of it a migration document targeting the new version or `--no-migration "<reason>"`; both are recorded in the entry. Which part a change earns stays a judgment call: the command enforces that a crossing documents itself, never that the part was chosen correctly |
|
||||
| `migrate list [--json]` | List every migration document under `instructions/migrations/`, oldest target first, with its kind. Read-only and **exempt from the Iteration Budget Gate** |
|
||||
| `migrate status [--json]` | Show the migrations this instance still owes, in the order they must run: every document whose `migrates_to` lies in `(kb_version, VERSION]`. Exits 1 only when `.wikitool-kb.json` is missing - the content's shape is a question the tool refuses to answer by guessing. Read-only and exempt from the budget gate |
|
||||
| `migrate verify --from <rev> [--path P ...] [--expect-body-change] [--json] [--fail-on-error]` | Compare `kb/` against a git revision on the invariants a content migration must not change: wikilink and citation **counts** (not sets), footnote definitions, H1, and structural frontmatter. Reports added/removed pages without failing on them. `--expect-body-change` additionally flags a page whose body did not change at all. Not migration-specific - worth running after any bulk rewrite, and the one question `lint` cannot answer, since it reads a single revision and so cannot see that something went missing. Read-only and exempt from the budget gate |
|
||||
@@ -84,8 +84,34 @@ tools/wikitool <command> --help
|
||||
|
||||
## Design notes
|
||||
|
||||
- All commands operate on the real repo (paths resolved relative to this
|
||||
file's location), so they can be run from any working directory.
|
||||
- All commands operate on the real repo, so they can be run from any working
|
||||
directory. The root is resolved by precedence - an explicit argument, then
|
||||
`$CHEMENU_ROOT`, then a walk up from the package's own location - and the
|
||||
walk-up is the default, so `tools/wikitool` with no variable set behaves
|
||||
exactly as it always has. Nothing under the root is bound at import time:
|
||||
`KB_DIR`, `RAW_DIR` and the rest follow whatever `ROOT` currently is, which
|
||||
is what makes the half-repointed state (a moved `ROOT` with a stale `KB_DIR`)
|
||||
unconstructible rather than merely discouraged.
|
||||
- **The library boundary.** `chemenu.api.Corpus` is the in-process entry point:
|
||||
it takes a corpus root, returns the same structures the `--json` forms print,
|
||||
and raises `ChemenuError` where the CLI prints `ERROR` and exits 1. It is
|
||||
read-only *structurally* - nothing under `chemenu.commands` is imported from
|
||||
it, so `new`, `publish` and the rest are not reachable, rather than filtered.
|
||||
The cores it calls (`search/service.py`, `lint_core.py`, `types_core.py`)
|
||||
import no `typer` and no `rich`; the modules under `commands/` are the
|
||||
terminal adapters over them. A second consumer is therefore a second adapter,
|
||||
not a second implementation.
|
||||
- **The MCP read server** (`chemenu/mcp/`) is that second adapter: `search`,
|
||||
`types`, `describe_type`, `lint` and `status` over `chemenu.api`, on `stdio`
|
||||
or `streamable-http`. Its dependency is optional and lives in
|
||||
`requirements-mcp.txt`, so a CLI-only instance does not install it. There is
|
||||
no write tool - nothing under `commands/` is importable from it, so the write
|
||||
functions are unreachable rather than filtered - and every response carries
|
||||
the commit it was computed from. Running it, and keeping its checkout
|
||||
current, is [instructions/mcp-read-server.md](../instructions/mcp-read-server.md).
|
||||
Authentication and rate limiting are middleware in front of the process, not
|
||||
code here; the Iteration Budget Gate is deliberately not applied, because it
|
||||
bounds an agent session rather than a user.
|
||||
- `new`/`xref`/`log append` only produce structurally-correct frontmatter and
|
||||
body skeletons/edits - the prose (Description, Summary, judgment calls
|
||||
about relationships) is still written by the LLM afterwards.
|
||||
@@ -151,7 +177,7 @@ is atomic, and whether a retry is safe.
|
||||
| `index rebuild` / `sources rebuild-index` | Rare I/O error only | Yes - the file is regenerated from scratch | Safe to retry freely || `log append` | Invalid `--op` or unreadable `--body-file` | Yes - single append | **Not idempotent.** If the previous run's outcome is uncertain, check the tail of `kb/log.md` before retrying |
|
||||
| `log status` | Never fails (reports 0 if `kb/log.md` is missing or empty) | Read-only | Safe to retry freely |
|
||||
| `lint` | Only with `--fail-on-error`: hard findings exist | Writes one report file (single atomic write) unless `--json` | Safe to retry freely, but re-run it to re-*measure*, never to re-read: the printed path holds the full report. Exit 1 means "act on the findings", not "the tool is broken" |
|
||||
| `search` | `rg` is not installed, a malformed `--field` predicate, an unknown field name, or an unknown `--backend` | Read-only | Fix the argument and retry. An unknown field name is reported with the list of fields that do exist - it is never answered with an empty result, because that would read as "no such pages" |
|
||||
| `search` | `rg` is not installed or did not finish within 30 s, a malformed `--field` predicate, an unknown field name, or an unknown `--backend` | Read-only | Fix the argument and retry. A timeout is a pathological pattern or an unresponsive corpus directory, not a slow answer - narrow the query or drop `--regex` rather than retrying it unchanged. An unknown field name is reported with the list of fields that do exist - it is never answered with an empty result, because that would read as "no such pages" |
|
||||
| `confidence decay --apply` / `init-base --apply` | Rare I/O error mid-loop | No - one write per page | Safe to retry freely; both recompute from `confidence_base` and never compound |
|
||||
| `sync` | The automatic rebase hit a real conflict (git failed) | No - fetch, then at most one merge/rebase attempt, aborted cleanly on failure | For a conflict: **do not retry, do not force** - resolve manually and re-run. **Exit 42, not 1**, when the rebase-review gate needs clearance: show the user the command's full output verbatim (upstream commits, the overlapping files, their diff) and stop; re-running with `--confirm-rebase <token>` clears it, and a wrong, invented, or superseded token exits 42 again with the current state. No remote configured, or one that cannot be reached, is not a failure - reported and skipped |
|
||||
| `publish` | git failed, **or** `--yes`/`-y` was passed. **Exit 42, not 1**, when the Mass-Update Gate, the rebase-review gate (raised by the same reconcile `sync` performs), or the Publish-Remote Gate refuses | No - sequential git operations, but both gates run before staging | For git failures: **do not retry, do not force** - report and ask the user (the reconcile step already retried the push once on its own, if a rebase resolved the rejection). For exit 42: show the user the command's full output verbatim and stop; it names the evidence and the `--confirm <token>` or `--confirm-rebase <token>` line to re-run, and re-running without it exits 42 again. The Publish-Remote Gate is the exception with no such line: it names the push URL that would have been written to and the ones this checkout allows, and only the user resolves it |
|
||||
@@ -164,7 +190,7 @@ is atomic, and whether a retry is safe.
|
||||
| `dist export` | Target exists and is not empty, is not a directory, or the tree has no readable `VERSION` | Yes - nothing is written until every file is planned | Point `<target>` at an empty (or new) directory and retry. Never merge into a non-empty one by hand |
|
||||
| `version show` / `version notes` | `VERSION` is missing or unparseable; for `notes`, no `CHANGES.md` entry names the version asked for | Read-only | Fix `VERSION`, or write the changelog entry (`version bump` writes its heading). Safe to retry |
|
||||
| `version check` | The feed could not be reached, answered non-JSON, or carried no `tag_name`. **Never** answers "up to date" for a question it could not ask | Read-only, no local writes | A network failure is transient - retry once, then report it. HTTP 401/403 names `$WIKITOOL_UPDATE_TOKEN`; 404 means no release exists yet or the URL points at the wrong repo |
|
||||
| `version bump` | More or fewer than one of `--major/--minor/--patch`, an empty `--title`, a missing `VERSION`/`CHANGES.md`, a changelog already documenting a version not older than the new one, a boundary-crossing bump with neither a migration document nor `--no-migration`, or `--no-migration` on a bump that crosses nothing | No - `VERSION` then `CHANGES.md` | **Not idempotent**: a second run bumps again. If the outcome is uncertain, read `VERSION` and the top of `CHANGES.md` before retrying |
|
||||
| `version bump` | More or fewer than one of `--major/--minor/--patch`, an empty `--title`, a missing `VERSION`/`CHANGES.md`, a changelog already documenting a version not older than the new one, a boundary-crossing bump without `--breaking` or with neither a migration document nor `--no-migration`, or `--breaking`/`--no-migration` on a bump that crosses nothing | No - `VERSION` then `CHANGES.md` | **Not idempotent**: a second run bumps again. If the outcome is uncertain, read `VERSION` and the top of `CHANGES.md` before retrying |
|
||||
| `migrate list` / `migrate status` | `list` never fails; `status` exits 1 when `.wikitool-kb.json` is missing or unreadable, or `VERSION` is | Read-only | For a missing declaration: run `migrate baseline <version>` once, then retry. Safe to retry freely otherwise |
|
||||
| `migrate verify` | Only with `--fail-on-error`: an invariant changed. Also exits 1 if `--from` is not a revision in this repository | Read-only | Exit 1 from `--fail-on-error` means "act on the findings", not "the tool is broken". A finding is never fixed by re-running - it names a page and what changed on it |
|
||||
| `migrate done` | Unknown version, no `.wikitool-kb.json`, nothing outstanding, or a version that is not the next link in the chain | Yes - single file write | **Not idempotent**: it advances the chain. For "not the next link", run `migrate status` and apply them in the order it prints - never force the order |
|
||||
|
||||
+23
-3
@@ -34,20 +34,40 @@ tools/
|
||||
wikitool entry point
|
||||
chemenu/
|
||||
cli.py Typer app: registers every command, runs the budget gate
|
||||
config.py repo layout constants (ROOT, RAW_DIR, KB_DIR, WORK_DIR, ...)
|
||||
config.py repo layout: root resolution and every path under it
|
||||
api.py the in-process entry point - point Chemenu at a corpus and read it
|
||||
errors.py ChemenuError / ValidationError / BackendError
|
||||
corpus_cache.py one parsed corpus per commit, never cached while the tree is dirty
|
||||
kb_scan.py page iteration/loading over kb/
|
||||
kb_collections.py collection discovery (a directory with COLLECTION.md)
|
||||
type_resolver.py type-spec loading and schema resolution
|
||||
lint_core.py the lint checks and the report, with no CLI attached
|
||||
types_core.py type-spec listing/description, with no CLI attached
|
||||
sections.py the section headings the tool reads and writes in a page body
|
||||
markdown_code.py masks code spans/fences so a page may show wiki notation, not only use it
|
||||
version.py the stack version: VERSION, the release stamp, the compatibility rule
|
||||
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
|
||||
corpus_diff.py invariant comparison of kb/ between two revisions
|
||||
search/ pluggable search backends (base protocol, ripgrep, filters, fuse)
|
||||
commands/ one module per command or command group
|
||||
search/ pluggable search backends, plus service.py - the search core
|
||||
commands/ one module per command or command group: the terminal adapters
|
||||
tests/ pytest suite
|
||||
```
|
||||
|
||||
**Two consumers, one core.** The CLI is not the only caller any more. The cores
|
||||
(`search/service.py`, `lint_core.py`, `types_core.py`) hold what decides an
|
||||
answer and import no `typer` and no `rich`; the modules under `commands/` turn
|
||||
those values into terminal output and those exceptions into exit codes.
|
||||
`api.Corpus` is the in-process entry point over the same functions - it takes a
|
||||
corpus root, returns exactly the structures the `--json` forms print, and
|
||||
raises instead of exiting. A second consumer is therefore a second adapter
|
||||
rather than a second implementation, and the write commands are unreachable
|
||||
from `api` because nothing under `commands/` is imported there.
|
||||
|
||||
**Which corpus.** The root resolves by precedence: an explicit argument, then
|
||||
`$CHEMENU_ROOT`, then a walk up from the package's own location. The walk-up is
|
||||
the default, so the CLI is unaffected by any of this. Nothing under the root is
|
||||
bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.
|
||||
|
||||
## Adding a command
|
||||
|
||||
1. Write the module under `chemenu/commands/`. A group is a `typer.Typer()`
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
"""The in-process entry point: point Chemenu at a corpus and read from it.
|
||||
|
||||
This is the seam the MCP server (Gitea #19) is built on, and the reason it is
|
||||
worth having as its own module rather than as "call the command functions
|
||||
yourself": it fixes the two things that made an in-process caller a second-class
|
||||
one.
|
||||
|
||||
**A corpus you name, not the one this package happens to sit in.** `Corpus`
|
||||
holds the root, threads it into the loader and the search backend, and points
|
||||
`config` at it for the duration of each call so the parts that reach for
|
||||
`config` directly - the `TypeResolver` singleton, which has to find `types/` -
|
||||
follow too. A test proves no path of the developer's checkout is read while a
|
||||
foreign root is set.
|
||||
|
||||
**Values and exceptions, not exit codes.** The functions return the same
|
||||
structures the CLI's `--json` forms print - one wire contract, with the CLI as
|
||||
its executable specification - and raise `ChemenuError` where the CLI would
|
||||
print `ERROR` and leave through `typer.Exit(1)`.
|
||||
|
||||
Read-only, structurally: nothing under `chemenu.commands` is imported, so `new`,
|
||||
`touch`, `xref`, `cite`, `publish`, `migrate` and `version bump` are not
|
||||
reachable from here at all. That is the property #19 asks for - the write
|
||||
functions do not exist in this surface rather than being filtered out of it.
|
||||
|
||||
Import cost is the whole read core and nothing else: `yaml` and `jsonschema`,
|
||||
plus the standard library. No `typer`, no `rich`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Optional
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.corpus_cache import CorpusCache
|
||||
from chemenu.errors import BackendError, ChemenuError, ValidationError
|
||||
from chemenu.lint_core import run_lint
|
||||
from chemenu.search import filters
|
||||
from chemenu.search.registry import resolve
|
||||
from chemenu.search.service import run_search, unreadable_pages
|
||||
from chemenu.search.types import Predicate, SearchQuery
|
||||
from chemenu.types_core import describe_type, list_types
|
||||
|
||||
# Distinguishes "the caller did not pass a revision" from "the caller passed
|
||||
# None", which is itself a meaningful answer: no commit, because the tree is
|
||||
# dirty or is not a checkout.
|
||||
_UNREAD = object()
|
||||
|
||||
__all__ = [
|
||||
"Corpus",
|
||||
"ChemenuError",
|
||||
"ValidationError",
|
||||
"BackendError",
|
||||
]
|
||||
|
||||
|
||||
class Corpus:
|
||||
"""One corpus tree, read repeatedly.
|
||||
|
||||
`root` follows `config.resolve_root()`: an explicit path, else
|
||||
`$CHEMENU_ROOT`, else the checkout this package lives in. `kb_dir` defaults
|
||||
to `<root>/kb` and is separate only because the search backend already
|
||||
distinguishes the two.
|
||||
|
||||
Holds a `CorpusCache`, so a long-lived caller parses the corpus once per
|
||||
commit instead of once per request - and never answers from a superseded
|
||||
parse, because a dirty tree is not cached. Not thread-safe: a server serving
|
||||
concurrent requests holds the lock, for the reason given in
|
||||
`chemenu/corpus_cache.py`.
|
||||
"""
|
||||
|
||||
def __init__(self, root: Optional[Path | str] = None, kb_dir: Optional[Path | str] = None):
|
||||
self.root = config.resolve_root(root)
|
||||
self.kb_dir = Path(kb_dir) if kb_dir is not None else self.root / "kb"
|
||||
self._cache = CorpusCache(self.kb_dir, self.root)
|
||||
|
||||
@property
|
||||
def revision(self) -> Optional[str]:
|
||||
"""The commit every answer from this corpus is stamped with, or None
|
||||
when the tree is dirty or is not a git checkout - in which case the
|
||||
answer corresponds to no commit, and says so."""
|
||||
return self._cache.current_revision()
|
||||
|
||||
def _rooted(self):
|
||||
"""Point `config` at this corpus for the duration of one call.
|
||||
|
||||
Threading a root through every argument gets the search backend and the
|
||||
corpus loader, and misses the module-level `TypeResolver` singleton -
|
||||
which resolves `types/` and is what `Page.kind` goes through. Without
|
||||
this, a foreign corpus is read with *this* checkout's type specs, and
|
||||
the page's `kind` is an answer about the wrong instance.
|
||||
|
||||
Process-wide while open, so `Corpus` inherits `config.rooted()`'s
|
||||
thread-safety constraint: one lock per process, held by the caller.
|
||||
"""
|
||||
return config.rooted(self.root)
|
||||
|
||||
def _stamp(
|
||||
self, payload: dict[str, Any], revision: Optional[str] = _UNREAD
|
||||
) -> dict[str, Any]:
|
||||
"""Every response carries the revision it was computed from.
|
||||
|
||||
A stale checkout otherwise answers confidently and wrongly, which is the
|
||||
failure `SOUL.md` names as the cardinal one. The stamp turns a silent
|
||||
stale answer into a visible one.
|
||||
|
||||
A caller that loaded the corpus passes the revision it got back, rather
|
||||
than letting this ask again: between the load and the stamp the tree can
|
||||
move, and the honest answer is the revision the pages actually came
|
||||
from. Callers that read no pages (`types`) ask for the current one.
|
||||
"""
|
||||
if revision is _UNREAD:
|
||||
revision = self._cache.current_revision()
|
||||
payload["commit"] = revision
|
||||
payload["as_of"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
return payload
|
||||
|
||||
def search(
|
||||
self,
|
||||
text: Optional[str] = None,
|
||||
predicates: Iterable[str] = (),
|
||||
regex: bool = False,
|
||||
limit: int = 20,
|
||||
sort: Optional[str] = None,
|
||||
backend: Optional[str] = None,
|
||||
) -> dict[str, Any]:
|
||||
"""`wikitool search --json`, as a value.
|
||||
|
||||
`predicates` takes the raw `--field` strings, so the CLI and this share
|
||||
one parser and cannot drift on what `confidence<0.6` means.
|
||||
"""
|
||||
raw = list(predicates)
|
||||
if not text and not raw:
|
||||
raise ValidationError(
|
||||
"Nothing to search for: give a query, or at least one predicate."
|
||||
)
|
||||
parsed: tuple[Predicate, ...] = tuple(filters.parse_predicate(p) for p in raw)
|
||||
backends = resolve(backend, self.kb_dir, self.root)
|
||||
query = SearchQuery(
|
||||
text=text, predicates=parsed, regex=regex, limit=limit, sort=sort
|
||||
)
|
||||
|
||||
with self._rooted():
|
||||
pages, revision = self._cache.load()
|
||||
hits = run_search(query, pages, backends, self.kb_dir)
|
||||
return self._stamp({
|
||||
"query": text,
|
||||
"predicates": [p.render() for p in parsed],
|
||||
"backend": ",".join(b.name for b in backends),
|
||||
"count": len(hits),
|
||||
"results": [hit.as_dict() for hit in hits],
|
||||
"unreadable": unreadable_pages(pages),
|
||||
}, revision)
|
||||
|
||||
def lint(self) -> dict[str, Any]:
|
||||
"""`wikitool lint --json`, as a value.
|
||||
|
||||
The JSON form only - `lint` without a flag writes a report into
|
||||
`reports/`, and a read surface does not write into the tree it is
|
||||
reading.
|
||||
"""
|
||||
with self._rooted():
|
||||
return self._stamp(run_lint(self.kb_dir))
|
||||
|
||||
def types(self) -> dict[str, Any]:
|
||||
"""`wikitool types list --json`, as a value."""
|
||||
with self._rooted():
|
||||
return self._stamp({"types": list_types()})
|
||||
|
||||
def describe_type(self, name: str) -> dict[str, Any]:
|
||||
"""`wikitool types describe <name> --json`, as a value. Raises
|
||||
`UnknownType` (a `ValidationError`) for a name that does not exist."""
|
||||
with self._rooted():
|
||||
return self._stamp(describe_type(name))
|
||||
|
||||
def status(self) -> dict[str, Any]:
|
||||
"""A composed snapshot: how big the corpus is and what lint says about
|
||||
it, without the full report.
|
||||
|
||||
Composed here on purpose. There is no `wikitool status` to wrap -
|
||||
`wiki-status` is a *skill* that assembles `kb/index.md`, `lint` and
|
||||
`kb/log.md` - so this is a new surface, and saying so is what keeps
|
||||
anyone from looking for the CLI command it mirrors.
|
||||
"""
|
||||
with self._rooted():
|
||||
pages, revision = self._cache.load()
|
||||
report = run_lint(self.kb_dir)
|
||||
collections: dict[str, int] = {}
|
||||
for page in pages.values():
|
||||
name = filters.collection_of(page.path, self.kb_dir)
|
||||
if name:
|
||||
collections[name] = collections.get(name, 0) + 1
|
||||
return self._stamp({
|
||||
"pages": len(pages),
|
||||
"collections": dict(sorted(collections.items())),
|
||||
"findings": {
|
||||
key: len(value)
|
||||
for key, value in sorted(report.items())
|
||||
if isinstance(value, list)
|
||||
},
|
||||
"unreadable": unreadable_pages(pages),
|
||||
}, revision)
|
||||
@@ -64,8 +64,14 @@ DIST_TEMPLATES_DIR = Path(__file__).resolve().parent.parent / "dist_templates"
|
||||
# and does not load `AGENTS.md`, so it ships for the same reason
|
||||
# `.claude/settings.json` does - a distributed instance running that harness
|
||||
# would otherwise start every session without the control plane.
|
||||
#
|
||||
# `INSTALL-MCP.md` ships beside `INSTALL.md` and for the same reason: the MCP
|
||||
# read server is part of what an instance *has*, even though its dependency is
|
||||
# optional. A distribution whose server is present but undocumented is one
|
||||
# whose operator finds the module by reading the source.
|
||||
ROOT_FILES = (
|
||||
"AGENTS.md", "CLAUDE.md", "README.md", "EVALS.md", "INSTALL.md", ".gitignore", "VERSION",
|
||||
"AGENTS.md", "CLAUDE.md", "README.md", "EVALS.md", "INSTALL.md", "INSTALL-MCP.md",
|
||||
".gitignore", "VERSION",
|
||||
*config.LICENSE_FILES,
|
||||
*config.PERSONALIZATION_TEMPLATES,
|
||||
config.ENVIRONMENT_TEMPLATE,
|
||||
|
||||
@@ -473,6 +473,43 @@ def check_migration_for_boundary() -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def check_breaking_change_for_boundary() -> list[str]:
|
||||
"""A version that crosses the compatibility boundary must say what breaks.
|
||||
|
||||
Separate from `check_migration_for_boundary`, because the two answer
|
||||
different questions: that one asks whether the *content* has to move, this
|
||||
one whether the operator was told the swap is not drop-in at all. A
|
||||
boundary crossing with an untouched corpus - a renamed feed, artefact,
|
||||
import name or flag - satisfies that check and still leaves every existing
|
||||
instance with something to do by hand.
|
||||
|
||||
Only the newest entry is checked, for the same reason: older crossings are
|
||||
history, and re-reporting them forever would make the check noise.
|
||||
"""
|
||||
changes_path = config.ROOT / version_mod.CHANGES_FILENAME
|
||||
version_path = config.ROOT / version_mod.VERSION_FILENAME
|
||||
if not changes_path.is_file() or not version_path.is_file():
|
||||
return [] # already reported by check_version_changelog
|
||||
|
||||
text = changes_path.read_text(encoding="utf-8")
|
||||
current = version_mod.top_changes_version(text)
|
||||
previous = _second_changes_version(text)
|
||||
if current is None or previous is None:
|
||||
return [] # the first versioned entry has no predecessor to cross from
|
||||
if current.compat_key == previous.compat_key:
|
||||
return []
|
||||
|
||||
if version_mod.BREAKING_CHANGE_MARKER in (version_mod.changes_section(text, current) or ""):
|
||||
return []
|
||||
|
||||
return [
|
||||
f"{current} crosses the compatibility boundary from {previous}, so it is not a drop-in "
|
||||
f"replacement - but its {version_mod.CHANGES_FILENAME} entry carries no "
|
||||
f"`{version_mod.BREAKING_CHANGE_MARKER}` line saying what stops working. Add it "
|
||||
"(`version bump --breaking` writes it; see instructions/dev/version-parts.md)"
|
||||
]
|
||||
|
||||
|
||||
@app.command("verify")
|
||||
def verify():
|
||||
"""Check the CLI/README command tables, contract presence, type-form drift, ignore rules, and version/changelog agreement."""
|
||||
@@ -484,6 +521,7 @@ def verify():
|
||||
+ check_ignored_content()
|
||||
+ check_version_changelog()
|
||||
+ check_migration_for_boundary()
|
||||
+ check_breaking_change_for_boundary()
|
||||
)
|
||||
|
||||
if issues:
|
||||
|
||||
@@ -254,12 +254,19 @@ def check_publish_remotes() -> Check:
|
||||
configured and no allowlist. That is the shape a private instance has after
|
||||
it adds the public upstream, and it is exactly when a wrong `--remote`
|
||||
stops being a typo and starts being a disclosure.
|
||||
|
||||
Both absent states say **armed** or **not armed** rather than only naming
|
||||
the file. AGENTS.md lists this among the three limits enforced in code, so a
|
||||
line that reports the file's absence and leaves the reader to infer what
|
||||
that means about the gate is how a checkout ends up trusting a safeguard
|
||||
that is not running - which is worse than having none.
|
||||
"""
|
||||
urls = git_publish.read_allowed_push_urls()
|
||||
if urls is not None:
|
||||
return Check(
|
||||
"publish-remotes", "OK",
|
||||
f"{len(urls)} allowed push target(s) in {config.PUBLISH_REMOTES_FILENAME}",
|
||||
f"Gate armed: {len(urls)} allowed push target(s) in "
|
||||
f"{config.PUBLISH_REMOTES_FILENAME}",
|
||||
)
|
||||
result = subprocess.run(
|
||||
["git", "remote"], cwd=config.ROOT, capture_output=True, text=True
|
||||
@@ -268,13 +275,15 @@ def check_publish_remotes() -> Check:
|
||||
if len(remotes) > 1:
|
||||
return Check(
|
||||
"publish-remotes", "WARN",
|
||||
f"{len(remotes)} remotes ({', '.join(remotes)}) and no publish allowlist",
|
||||
f"Gate not armed: {len(remotes)} remotes ({', '.join(remotes)}) and no "
|
||||
f"{config.PUBLISH_REMOTES_FILENAME} - every one of them is a legal publish target",
|
||||
f"Create {config.PUBLISH_REMOTES_FILENAME} naming the push URL this checkout "
|
||||
"may publish to - see instructions/gates.md",
|
||||
)
|
||||
return Check(
|
||||
"publish-remotes", "OK",
|
||||
f"No {config.PUBLISH_REMOTES_FILENAME} (unrestricted; one remote configured)",
|
||||
f"Gate not armed: no {config.PUBLISH_REMOTES_FILENAME} - any push target passes "
|
||||
"(1 remote configured, nothing to confuse it with)",
|
||||
)
|
||||
|
||||
|
||||
|
||||
+28
-416
@@ -1,16 +1,12 @@
|
||||
"""Deterministic structural health checks for the wiki.
|
||||
"""`wikitool lint` - the terminal adapter over `chemenu.lint_core`.
|
||||
|
||||
This intentionally covers only what can be computed mechanically: broken
|
||||
wikilinks, orphan pages, index/page drift, frontmatter schema gaps, and
|
||||
filename/title mismatches. Semantic judgment (contradictions, staleness,
|
||||
what's worth writing about next) stays with the LLM - this report gives it a
|
||||
verified factual foundation instead of requiring it to re-derive these facts
|
||||
by reading every page.
|
||||
The checks, the report and the hard-error rule live in `chemenu/lint_core.py`,
|
||||
which imports no CLI machinery. This module owns only what a terminal needs:
|
||||
the flags, where the report file lands, and the exit code.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
@@ -18,416 +14,32 @@ import typer
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.commands._util import rel_path, success
|
||||
from chemenu.frontmatter_io import frontmatter_error
|
||||
from chemenu.markdown_code import strip_code_spans
|
||||
from chemenu.provenance import broken_raw_refs as find_broken_raw_refs
|
||||
from chemenu.provenance import duplicate_raw_file_owners as find_duplicate_raw_file_owners
|
||||
from chemenu.provenance import extract_inline_cites
|
||||
from chemenu.provenance import legacy_citation_markers as find_legacy_citation_markers
|
||||
from chemenu.provenance import legacy_source_pages as find_legacy_source_pages
|
||||
from chemenu.provenance import orphan_footnote_defs as find_orphan_footnote_defs
|
||||
from chemenu.provenance import uncovered_raw_files as find_uncovered_raw_files
|
||||
from chemenu.provenance import undefined_footnote_refs as find_undefined_footnote_refs
|
||||
from chemenu.kb_scan import (
|
||||
GENERATED_INDEX,
|
||||
WIKILINK_RE,
|
||||
build_link_graph,
|
||||
find_duplicate_title_paths,
|
||||
inbound_links,
|
||||
load_kb_pages,
|
||||
)
|
||||
from chemenu.type_resolver import resolver
|
||||
|
||||
# Style guide's one mechanically-checkable rule (hard oracle: a plain count).
|
||||
# The rest of the style guide (tone, AI-phrase avoidance) is a soft/proxy judgment
|
||||
# and stays with the LLM - see wiki-manage/wiki-ingest skill guidance, not lint.
|
||||
#
|
||||
# The unit is a quote, not a `>` line. It used to be the line, which measured
|
||||
# the wrap width the rule has no opinion about: one quotation written long
|
||||
# counted 1 and the same quotation wrapped at 100 columns counted 4. An author
|
||||
# who took the finding seriously made the page harder to read to quiet it.
|
||||
QUOTE_LIMIT = 2
|
||||
|
||||
# How many hub pages `most_linked` reports. Purely informational (wiki-status
|
||||
# surfaces it); not a finding, so the cutoff only bounds report size.
|
||||
MOST_LINKED_COUNT = 10
|
||||
|
||||
|
||||
def count_quote_blocks(body: str) -> int:
|
||||
"""How many distinct blockquotes `body` carries.
|
||||
|
||||
A run of consecutive `>` lines is one quote; a blank line or any
|
||||
non-quoted line ends it. Code is masked out first, so a `>` inside a
|
||||
fenced shell transcript is a prompt, not a quotation.
|
||||
|
||||
Lazy continuation - a quote whose wrapped lines drop the `>` - reads here
|
||||
as two quotes rather than one. That over-counts in the direction the limit
|
||||
already errs on, and the corpus prefixes every line, so the alternative
|
||||
(tracking paragraph state) buys nothing.
|
||||
"""
|
||||
count, in_quote = 0, False
|
||||
for line in strip_code_spans(body).splitlines():
|
||||
is_quote = line.lstrip().startswith(">")
|
||||
if is_quote and not in_quote:
|
||||
count += 1
|
||||
in_quote = is_quote
|
||||
return count
|
||||
|
||||
|
||||
def run_lint(kb_dir: Path) -> dict:
|
||||
pages = load_kb_pages(kb_dir)
|
||||
duplicate_titles = find_duplicate_title_paths(kb_dir, config.ROOT)
|
||||
|
||||
# Pages whose frontmatter can't be parsed read back as `{}` everywhere
|
||||
# else, which would let them slip past every frontmatter-driven check
|
||||
# below with no finding at all - so they are detected explicitly.
|
||||
frontmatter_errors = []
|
||||
for title, page in sorted(pages.items()):
|
||||
reason = frontmatter_error(page.path)
|
||||
if reason is None and not page.frontmatter.get("type"):
|
||||
reason = "missing `type:` field"
|
||||
if reason is not None:
|
||||
frontmatter_errors.append({"page": title, "error": reason})
|
||||
|
||||
graph = build_link_graph(pages)
|
||||
broken_links = [
|
||||
{"page": title, "target": target}
|
||||
for title, targets in graph.items()
|
||||
for target in sorted(targets)
|
||||
if target not in pages
|
||||
]
|
||||
|
||||
inbound = inbound_links({t: v for t, v in graph.items() if t != "index"})
|
||||
orphan_pages = sorted(
|
||||
title
|
||||
for title, sources in inbound.items()
|
||||
if not sources
|
||||
and title not in ("index", "log")
|
||||
# comparison pages are not linked to by design; index.md is sufficient coverage
|
||||
and pages[title].kind != "comparison"
|
||||
)
|
||||
|
||||
# Same link graph, opposite end: the most-linked-to pages are the wiki's
|
||||
# hubs. Reported (not judged) so `wiki-status` can show them without
|
||||
# re-deriving the graph.
|
||||
inbound_counts = {title: len(sources) for title, sources in inbound.items()}
|
||||
most_linked = [
|
||||
{"page": title, "inbound": count}
|
||||
for title, count in sorted(inbound_counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
if count > 0
|
||||
][:MOST_LINKED_COUNT]
|
||||
|
||||
# The catalog is sharded: `kb/index.md` is a map carrying counts and links,
|
||||
# and the page rows live in a generated INDEX.md per collection/area. Both
|
||||
# halves have to be read, or every page reads as missing from the index.
|
||||
index_text = "".join(
|
||||
path.read_text(encoding="utf-8")
|
||||
for path in [kb_dir / "index.md", *sorted(kb_dir.rglob(GENERATED_INDEX))]
|
||||
if path.exists()
|
||||
)
|
||||
index_links = {m.group(1).strip() for m in WIKILINK_RE.finditer(index_text)}
|
||||
missing_from_index = sorted(set(pages) - index_links - {"index", "log"})
|
||||
dangling_index_entries = sorted(index_links - set(pages))
|
||||
|
||||
title_mismatches = []
|
||||
for title, page in sorted(pages.items()):
|
||||
if page.kind not in ("entity", "concept"):
|
||||
continue
|
||||
h1 = page.h1_title
|
||||
if h1 is not None and h1 != title:
|
||||
title_mismatches.append({"page": title, "h1": h1})
|
||||
|
||||
unmarked_provenance = []
|
||||
for title, page in sorted(pages.items()):
|
||||
if page.kind not in ("entity", "concept"):
|
||||
continue
|
||||
sources_list = page.frontmatter.get("sources") or []
|
||||
if not sources_list and page.frontmatter.get("provenance") != "general":
|
||||
unmarked_provenance.append(title)
|
||||
|
||||
citation_frontmatter_drift = []
|
||||
for title, page in sorted(pages.items()):
|
||||
sources_list = set(page.frontmatter.get("sources") or [])
|
||||
cited = {cited_title for cited_title, _file in extract_inline_cites(page.body)}
|
||||
cited.discard(title) # a source page citing itself for a specific file within it is not drift
|
||||
for missing_source in sorted(cited - sources_list):
|
||||
citation_frontmatter_drift.append({"page": title, "cited_but_not_in_sources": missing_source})
|
||||
|
||||
legacy_citation_markers = find_legacy_citation_markers(pages)
|
||||
undefined_footnote_refs = find_undefined_footnote_refs(pages)
|
||||
orphan_footnote_defs = find_orphan_footnote_defs(pages)
|
||||
|
||||
# The frontmatter half of the link graph. `broken_links` above only walks
|
||||
# `[[wikilinks]]` in page *bodies*, so a `related:`/`sources:`/`entities:`
|
||||
# entry naming a page that does not exist - a rename that was not
|
||||
# propagated, a deleted page, or a URL pasted where a title belongs - used
|
||||
# to pass every check. Which fields hold page titles is declared by each
|
||||
# type-spec's `page_ref_fields:`, not hardcoded here.
|
||||
dangling_frontmatter_refs = []
|
||||
for title, page in sorted(pages.items()):
|
||||
type_path = page.frontmatter.get("type")
|
||||
if not type_path:
|
||||
continue
|
||||
try:
|
||||
ref_fields = resolver.get_page_ref_fields(type_path, page.path)
|
||||
except ValueError:
|
||||
continue # unresolvable type is already reported as type_resolution_errors
|
||||
for field in ref_fields:
|
||||
for target in page.frontmatter.get(field) or []:
|
||||
if target not in pages:
|
||||
dangling_frontmatter_refs.append(
|
||||
{"page": title, "field": field, "target": target}
|
||||
)
|
||||
|
||||
quote_limit_violations = []
|
||||
for title, page in sorted(pages.items()):
|
||||
quote_count = count_quote_blocks(page.body)
|
||||
if quote_count > QUOTE_LIMIT:
|
||||
quote_limit_violations.append({"page": title, "quote_count": quote_count})
|
||||
|
||||
# Type system validation. Lint reports are not validated here: they are
|
||||
# written to `reports/` outside kb/ and are never pages, so nothing this
|
||||
# loop scans can be one.
|
||||
invalid_type_paths = []
|
||||
type_resolution_errors = []
|
||||
schema_validation_errors = []
|
||||
|
||||
for title, page in sorted(pages.items()):
|
||||
type_path = page.frontmatter.get("type")
|
||||
if not type_path:
|
||||
continue
|
||||
|
||||
# Check if type path is valid
|
||||
if not type_path.endswith('.md'):
|
||||
invalid_type_paths.append({"page": title, "type": type_path, "error": "Type path must end with .md"})
|
||||
continue
|
||||
|
||||
# Try to resolve and validate the type
|
||||
try:
|
||||
resolver.load_type_spec(type_path, page.path)
|
||||
|
||||
# Try schema validation
|
||||
try:
|
||||
resolver.validate_frontmatter(page.frontmatter, type_path, page.path)
|
||||
except ValueError as schema_error:
|
||||
schema_validation_errors.append({"page": title, "type": type_path, "error": str(schema_error)})
|
||||
|
||||
except ValueError as resolution_error:
|
||||
type_resolution_errors.append({"page": title, "type": type_path, "error": str(resolution_error)})
|
||||
|
||||
return {
|
||||
"generated": date.today().isoformat(),
|
||||
"page_count": len(pages),
|
||||
"frontmatter_errors": frontmatter_errors,
|
||||
"broken_links": broken_links,
|
||||
"orphan_pages": orphan_pages,
|
||||
"most_linked": most_linked,
|
||||
"inbound_counts": inbound_counts,
|
||||
"missing_from_index": missing_from_index,
|
||||
"dangling_index_entries": dangling_index_entries,
|
||||
"title_mismatches": title_mismatches,
|
||||
"duplicate_titles": duplicate_titles,
|
||||
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
|
||||
"broken_raw_refs": find_broken_raw_refs(pages),
|
||||
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
|
||||
"legacy_source_pages": find_legacy_source_pages(pages),
|
||||
"unmarked_provenance": unmarked_provenance,
|
||||
"citation_frontmatter_drift": citation_frontmatter_drift,
|
||||
"legacy_citation_markers": legacy_citation_markers,
|
||||
"undefined_footnote_refs": undefined_footnote_refs,
|
||||
"orphan_footnote_defs": orphan_footnote_defs,
|
||||
"dangling_frontmatter_refs": dangling_frontmatter_refs,
|
||||
"quote_limit_violations": quote_limit_violations,
|
||||
"invalid_type_paths": invalid_type_paths,
|
||||
"type_resolution_errors": type_resolution_errors,
|
||||
"schema_validation_errors": schema_validation_errors,
|
||||
}
|
||||
|
||||
|
||||
def _section(lines: list[str], title: str, items: list, formatter) -> None:
|
||||
lines.append(f"## {title}")
|
||||
lines.append("")
|
||||
if not items:
|
||||
lines.append("None found.")
|
||||
else:
|
||||
for item in items:
|
||||
lines.append(f"- {formatter(item)}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
def render_markdown(report: dict) -> str:
|
||||
lines = [f"# Structural Lint Report ({report['generated']})", ""]
|
||||
lines.append(f"Scanned {report['page_count']} pages under `wiki/`. This report covers only")
|
||||
lines.append("mechanically-verifiable structural issues; see the Semantic Review section")
|
||||
lines.append("below for judgment calls the LLM should complete.")
|
||||
lines.append("")
|
||||
|
||||
_section(
|
||||
lines, "Unreadable Frontmatter", report["frontmatter_errors"],
|
||||
lambda i: f"[[{i['page']}]] - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Broken Wikilinks", report["broken_links"],
|
||||
lambda i: f"[[{i['page']}]] links to missing [[{i['target']}]]",
|
||||
)
|
||||
_section(lines, "Orphan Pages (no inbound links)", report["orphan_pages"], lambda i: f"[[{i}]]")
|
||||
_section(
|
||||
lines, f"Most-Linked Pages (top {MOST_LINKED_COUNT} hubs)", report["most_linked"],
|
||||
lambda i: f"[[{i['page']}]] - {i['inbound']} inbound link(s)",
|
||||
)
|
||||
_section(lines, "Pages Missing from index.md", report["missing_from_index"], lambda i: f"[[{i}]]")
|
||||
_section(lines, "Dangling index.md Entries", report["dangling_index_entries"], lambda i: f"[[{i}]]")
|
||||
_section(
|
||||
lines, "Duplicate Titles (naming collisions)", report["duplicate_titles"],
|
||||
lambda i: f"`{i['stem']}` -> {', '.join(f'`{p}`' for p in i['paths'])}",
|
||||
)
|
||||
_section(
|
||||
lines, "Filename / H1 Title Mismatches", report["title_mismatches"],
|
||||
lambda i: f"[[{i['page']}]] H1 is '{i['h1']}'",
|
||||
)
|
||||
_section(
|
||||
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
|
||||
lambda i: f"`{i}`",
|
||||
)
|
||||
_section(
|
||||
lines, "Broken raw_files References", report["broken_raw_refs"],
|
||||
lambda i: f"[[{i['page']}]] -> `{i['raw_path']}` (does not exist)",
|
||||
)
|
||||
_section(
|
||||
lines, "Raw Files With More Than One Owner", report["duplicate_raw_file_owners"],
|
||||
lambda i: f"`{i['raw_file']}` is claimed by " + ", ".join(f"[[{t}]]" for t in i["owners"]),
|
||||
)
|
||||
_section(
|
||||
lines, "Legacy source: Field (not yet migrated to raw_files:)", report["legacy_source_pages"],
|
||||
lambda i: f"[[{i['page']}]] source: `{i['source']}` ({i['reason']})",
|
||||
)
|
||||
_section(
|
||||
lines, "Pages Missing provenance: general Marker", report["unmarked_provenance"],
|
||||
lambda i: f"[[{i}]] has no sources and is not marked `provenance: general`",
|
||||
)
|
||||
_section(
|
||||
lines, "Citation / Frontmatter Drift", report["citation_frontmatter_drift"],
|
||||
lambda i: f"[[{i['page']}]] cites [[{i['cited_but_not_in_sources']}]] inline but it is missing from frontmatter `sources:`",
|
||||
)
|
||||
_section(
|
||||
lines, "Legacy Citation Markers (pre-migration `^[[...]]`)", report["legacy_citation_markers"],
|
||||
lambda i: f"[[{i['page']}]] still has `{i['marker']}` - run `wikitool cite add` and replace it with the `[^cite-id]` it prints",
|
||||
)
|
||||
_section(
|
||||
lines, "Undefined Footnote References", report["undefined_footnote_refs"],
|
||||
lambda i: f"[[{i['page']}]] references `[^{i['ref']}]`, which has no `[^{i['ref']}]: [[...]]` definition",
|
||||
)
|
||||
_section(
|
||||
lines, "Orphan Footnote Definitions", report["orphan_footnote_defs"],
|
||||
lambda i: f"[[{i['page']}]] defines `[^{i['id']}]` (-> [[{i['source']}]]) but nothing references it - run `wikitool cite sync`",
|
||||
)
|
||||
_section(
|
||||
lines, "Dangling Frontmatter References", report["dangling_frontmatter_refs"],
|
||||
lambda i: f"[[{i['page']}]] `{i['field']}:` names `{i['target']}`, which is not a page",
|
||||
)
|
||||
_section(
|
||||
lines, "Invalid Type Paths", report["invalid_type_paths"],
|
||||
lambda i: f"[[{i['page']}]] has type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Type Resolution Errors", report["type_resolution_errors"],
|
||||
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Schema Validation Errors", report["schema_validation_errors"],
|
||||
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, f"Pages Exceeding Quote Limit (>{QUOTE_LIMIT}/page)", report["quote_limit_violations"],
|
||||
lambda i: f"[[{i['page']}]] has {i['quote_count']} quotes - trim or confirm they're load-bearing",
|
||||
)
|
||||
|
||||
lines.append("## Semantic Review (LLM to complete)")
|
||||
lines.append("")
|
||||
lines.append("- Contradictions across pages: TODO")
|
||||
lines.append("- Stale claims (unconfirmed >6 months): TODO")
|
||||
lines.append("- Suggested new pages / missing cross-references: TODO")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# Sections that always carry content but are not findings, so the summary
|
||||
# handles them separately: a hub list is a statistic, and the semantic review
|
||||
# is the checklist that follows the report rather than part of it.
|
||||
INFORMATIONAL_SECTIONS = ("Most-Linked Pages",)
|
||||
SEMANTIC_REVIEW_SECTION = "Semantic Review"
|
||||
|
||||
|
||||
def _split_sections(markdown: str) -> tuple[str, list[tuple[str, str]]]:
|
||||
"""Cut a rendered report into its preamble and (title, body) sections."""
|
||||
preamble, *rest = markdown.split("\n## ")
|
||||
sections = []
|
||||
for part in rest:
|
||||
title, _, body = part.partition("\n")
|
||||
sections.append((title.strip(), body.strip()))
|
||||
return preamble.rstrip(), sections
|
||||
|
||||
|
||||
def render_summary(report: dict) -> str:
|
||||
"""The same report with the empty sections removed.
|
||||
|
||||
On a healthy corpus the full report is better than 90% "None found.", so
|
||||
reading it in the terminal means paging past the answer. The file on disk
|
||||
stays complete - this is what gets printed, and the written path underneath
|
||||
it is how the rest is reached without running lint a second time.
|
||||
"""
|
||||
preamble, sections = _split_sections(render_markdown(report))
|
||||
findings, trailing = [], []
|
||||
for title, body in sections:
|
||||
if title.startswith(SEMANTIC_REVIEW_SECTION):
|
||||
trailing.append((title, body))
|
||||
elif body != "None found." and not title.startswith(INFORMATIONAL_SECTIONS):
|
||||
findings.append((title, body))
|
||||
lines = [preamble, ""]
|
||||
if not findings:
|
||||
lines += ["No structural findings.", ""]
|
||||
for title, body in findings + trailing:
|
||||
lines += [f"## {title}", "", body, ""]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def default_report_path(report: dict) -> Path:
|
||||
"""Where a report goes when the caller names no path.
|
||||
|
||||
`reports/` is derived and gitignored ([reports/CONTRACT.md]), so writing
|
||||
here by default costs the tree nothing.
|
||||
"""
|
||||
return config.ROOT / "reports" / f"Lint Report {report['generated']}.md"
|
||||
|
||||
|
||||
# Findings that make a tree structurally wrong rather than merely untidy.
|
||||
# `orphan_pages` is deliberately absent: many pages are validly reachable
|
||||
# through the index or navigation only. `quote_limit_violations` is advisory
|
||||
# too - it flags a habit, not a broken tree.
|
||||
#
|
||||
# One definition, used by `lint --fail-on-error` and by the eval scorecard: if
|
||||
# the two disagreed, a run could pass its score while lint refused it.
|
||||
HARD_ERROR_KEYS = (
|
||||
"frontmatter_errors",
|
||||
"broken_links",
|
||||
"dangling_index_entries",
|
||||
"duplicate_titles",
|
||||
"broken_raw_refs",
|
||||
"duplicate_raw_file_owners",
|
||||
"legacy_source_pages",
|
||||
"citation_frontmatter_drift",
|
||||
"legacy_citation_markers",
|
||||
"undefined_footnote_refs",
|
||||
"orphan_footnote_defs",
|
||||
"dangling_frontmatter_refs",
|
||||
"invalid_type_paths",
|
||||
"type_resolution_errors",
|
||||
"schema_validation_errors",
|
||||
from chemenu.lint_core import (
|
||||
HARD_ERROR_KEYS,
|
||||
MOST_LINKED_COUNT,
|
||||
QUOTE_LIMIT,
|
||||
count_quote_blocks,
|
||||
default_report_path,
|
||||
has_hard_errors,
|
||||
render_markdown,
|
||||
render_summary,
|
||||
run_lint,
|
||||
)
|
||||
|
||||
|
||||
def has_hard_errors(report: dict) -> bool:
|
||||
return any(report.get(key) for key in HARD_ERROR_KEYS)
|
||||
# Re-exported: `from chemenu.commands.lint import run_lint` still resolves, and
|
||||
# so does every other name the tests and sibling commands already import.
|
||||
__all__ = [
|
||||
"HARD_ERROR_KEYS",
|
||||
"MOST_LINKED_COUNT",
|
||||
"QUOTE_LIMIT",
|
||||
"count_quote_blocks",
|
||||
"default_report_path",
|
||||
"has_hard_errors",
|
||||
"render_markdown",
|
||||
"render_summary",
|
||||
"run_lint",
|
||||
"lint_command",
|
||||
]
|
||||
|
||||
|
||||
def lint_command(
|
||||
|
||||
@@ -21,90 +21,38 @@ Scope is `kb/` only. `instructions/` is discovered through
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import typer
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.commands._util import fail, today_iso
|
||||
from chemenu.frontmatter_io import read_page
|
||||
from chemenu.kb_scan import iter_kb_pages
|
||||
from chemenu.page import Page
|
||||
from chemenu.search import filters
|
||||
from chemenu.search.base import page_key
|
||||
from chemenu.search.filters import PredicateError
|
||||
from chemenu.search.fuse import reciprocal_rank_fusion
|
||||
from chemenu.search.registry import UnknownBackend, resolve
|
||||
from chemenu.search.ripgrep import RipgrepFailed, RipgrepMissing, build_hit
|
||||
from chemenu.search.ripgrep import RipgrepFailed, RipgrepMissing
|
||||
from chemenu.search.service import (
|
||||
load_pages_by_path,
|
||||
run_search,
|
||||
sort_hits,
|
||||
unreadable_pages,
|
||||
)
|
||||
from chemenu.search.types import Predicate, SearchHit, SearchQuery
|
||||
|
||||
# Re-exported so `from chemenu.commands.search import run_search` keeps
|
||||
# resolving. The core lives in `chemenu/search/service.py`, which imports no
|
||||
# CLI machinery; this module is the terminal adapter over it.
|
||||
__all__ = [
|
||||
"load_pages_by_path",
|
||||
"run_search",
|
||||
"sort_hits",
|
||||
"unreadable_pages",
|
||||
"render_table",
|
||||
"search_command",
|
||||
]
|
||||
|
||||
TITLE_WIDTH = 34
|
||||
SUMMARY_WIDTH = 84
|
||||
|
||||
|
||||
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
|
||||
"""Every page under `kb/`, keyed by repo-relative path.
|
||||
|
||||
Path-keyed rather than title-keyed on purpose: `load_kb_pages()` drops one
|
||||
of two pages sharing a stem, and search should still find both - a
|
||||
duplicate title is a lint finding, not a reason to hide a page.
|
||||
"""
|
||||
kb_dir = kb_dir or config.KB_DIR
|
||||
root = root or config.ROOT
|
||||
pages: dict[str, Page] = {}
|
||||
for path in iter_kb_pages(kb_dir):
|
||||
frontmatter, body = read_page(path)
|
||||
pages[page_key(path, root)] = Page(path=path, frontmatter=frontmatter, body=body)
|
||||
return pages
|
||||
|
||||
|
||||
def _sort_key(hit: SearchHit, field: str):
|
||||
value = hit.as_dict().get(field)
|
||||
if value is None:
|
||||
# Missing values sort last in either direction rather than crashing on
|
||||
# a None comparison.
|
||||
return (1, "")
|
||||
if isinstance(value, (int, float)):
|
||||
return (0, value)
|
||||
return (0, str(value).lower())
|
||||
|
||||
|
||||
def sort_hits(hits: list[SearchHit], sort: str | None) -> list[SearchHit]:
|
||||
"""Sort by a hit field. A leading `-` reverses, e.g. `--sort -confidence`."""
|
||||
if not sort:
|
||||
return hits
|
||||
descending = sort.startswith("-")
|
||||
field = sort.lstrip("-")
|
||||
ordered = sorted(hits, key=lambda h: _sort_key(h, field), reverse=descending)
|
||||
return ordered
|
||||
|
||||
|
||||
def run_search(
|
||||
query: SearchQuery,
|
||||
pages: dict[str, Page],
|
||||
backends: list,
|
||||
kb_dir: Path | None = None,
|
||||
) -> list[SearchHit]:
|
||||
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
|
||||
filters.validate_fields(query.predicates, pages)
|
||||
|
||||
if query.text:
|
||||
rankings = [backend.search(query, pages) for backend in backends]
|
||||
hits = rankings[0] if len(rankings) == 1 else reciprocal_rank_fusion(rankings)
|
||||
allowed = filters.apply_predicates(pages, query.predicates, kb_dir)
|
||||
hits = [hit for hit in hits if hit.path in allowed]
|
||||
else:
|
||||
selected = filters.apply_predicates(pages, query.predicates, kb_dir)
|
||||
hits = [
|
||||
build_hit(page, key, [], query, backend="frontmatter", kb_dir=kb_dir)
|
||||
for key, page in selected.items()
|
||||
]
|
||||
hits.sort(key=lambda h: h.title.lower())
|
||||
|
||||
hits = sort_hits(hits, query.sort)
|
||||
return hits[: query.limit] if query.limit else hits
|
||||
|
||||
|
||||
def _truncate(text: str, width: int) -> str:
|
||||
text = " ".join(text.split())
|
||||
return text if len(text) <= width else text[: width - 1] + "\u2026"
|
||||
@@ -206,6 +154,8 @@ def search_command(
|
||||
except RipgrepFailed as exc:
|
||||
fail(str(exc))
|
||||
|
||||
unreadable = unreadable_pages(pages)
|
||||
|
||||
if json_out:
|
||||
payload = {
|
||||
"generated": today_iso(),
|
||||
@@ -214,8 +164,17 @@ def search_command(
|
||||
"backend": ",".join(b.name for b in backends),
|
||||
"count": len(hits),
|
||||
"results": [hit.as_dict() for hit in hits],
|
||||
# Always present, usually empty. A caller that has to look for the
|
||||
# key to learn whether it should worry will not look.
|
||||
"unreadable": unreadable,
|
||||
}
|
||||
typer.echo(json.dumps(payload, indent=2))
|
||||
return
|
||||
|
||||
typer.echo(render_table(hits, show_matches))
|
||||
for entry in unreadable:
|
||||
typer.echo(
|
||||
f"WARN unreadable frontmatter: {entry['path']} ({entry['reason']}) - "
|
||||
"this page cannot match any --field predicate",
|
||||
err=True,
|
||||
)
|
||||
|
||||
@@ -7,35 +7,29 @@ into context on every skill invocation. A type-spec's own frontmatter
|
||||
declared `.schema.yaml` are the single source of truth; this command only
|
||||
formats what `TypeResolver` already resolves - it does not duplicate or
|
||||
re-derive any type knowledge.
|
||||
|
||||
The resolving half lives in `chemenu/types_core.py`, which imports no CLI
|
||||
machinery. This module is the terminal adapter over it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, Dict
|
||||
|
||||
import typer
|
||||
|
||||
from chemenu.commands._util import fail
|
||||
from chemenu.type_resolver import resolver
|
||||
from chemenu.types_core import UnknownType, describe_type, list_types
|
||||
|
||||
app = typer.Typer(help="Discover and describe Chemenu type-spec contracts.")
|
||||
|
||||
|
||||
@app.command("list")
|
||||
def list_types(json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON")):
|
||||
def list_types_command(
|
||||
json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON")
|
||||
):
|
||||
"""List every type-spec under types/, with its name, schema, subtype
|
||||
field (if any), base directory, and description."""
|
||||
rows: list[Dict[str, Any]] = []
|
||||
for type_path, frontmatter in resolver.list_type_specs():
|
||||
rows.append({
|
||||
"name": frontmatter.get("name"),
|
||||
"type_path": type_path,
|
||||
"schema": frontmatter.get("schema"),
|
||||
"subtype_field": frontmatter.get("subtype_field"),
|
||||
"root": frontmatter.get("root") or "kb",
|
||||
"base_dir": frontmatter.get("base_dir"),
|
||||
"description": frontmatter.get("description"),
|
||||
})
|
||||
rows = list_types()
|
||||
|
||||
if json_out:
|
||||
typer.echo(json.dumps(rows, indent=2))
|
||||
@@ -53,7 +47,7 @@ def list_types(json_out: bool = typer.Option(False, "--json", help="Print raw fi
|
||||
|
||||
|
||||
@app.command("describe")
|
||||
def describe_type(
|
||||
def describe_type_command(
|
||||
name: str = typer.Argument(..., help="Type name, e.g. 'entity' (see `types list`)"),
|
||||
json_out: bool = typer.Option(False, "--json", help="Print raw findings as JSON"),
|
||||
):
|
||||
@@ -61,51 +55,26 @@ def describe_type(
|
||||
with enums where declared), its subtype field if any, and its authoring
|
||||
body - the same information an LLM would otherwise gather by reading the
|
||||
raw type-spec and `.schema.yaml` files directly."""
|
||||
type_path = resolver.find_type_by_name(name)
|
||||
if type_path is None:
|
||||
available = sorted(fm.get("name") for _, fm in resolver.list_type_specs())
|
||||
fail(f"No type-spec named '{name}'. Available: {', '.join(available)}")
|
||||
return # unreachable; keeps type-checkers happy about `type_path` below
|
||||
|
||||
type_spec = resolver.load_type_spec(type_path)
|
||||
frontmatter = type_spec["frontmatter"]
|
||||
body = type_spec["body"]
|
||||
schema = resolver.get_schema(type_path)
|
||||
|
||||
fields: list[Dict[str, Any]] = []
|
||||
if schema is not None:
|
||||
required = set(schema.get("required", []))
|
||||
for field_name, field_schema in schema.get("properties", {}).items():
|
||||
fields.append({
|
||||
"field": field_name,
|
||||
"required": field_name in required,
|
||||
"type": field_schema.get("type"),
|
||||
"enum": field_schema.get("enum"),
|
||||
})
|
||||
try:
|
||||
described = describe_type(name)
|
||||
except UnknownType as exc:
|
||||
fail(str(exc))
|
||||
return # unreachable; keeps type-checkers happy about `described` below
|
||||
|
||||
if json_out:
|
||||
typer.echo(json.dumps({
|
||||
"name": frontmatter.get("name"),
|
||||
"type_path": type_path,
|
||||
"description": frontmatter.get("description"),
|
||||
"schema": frontmatter.get("schema"),
|
||||
"subtype_field": frontmatter.get("subtype_field"),
|
||||
"base_dir": frontmatter.get("base_dir"),
|
||||
"title_prefix": frontmatter.get("title_prefix"),
|
||||
"fields": fields,
|
||||
"body": body.strip(),
|
||||
}, indent=2))
|
||||
typer.echo(json.dumps(described, indent=2))
|
||||
return
|
||||
|
||||
typer.echo(f"# {frontmatter.get('name')} ({type_path})")
|
||||
typer.echo(frontmatter.get("description", ""))
|
||||
fields = described["fields"]
|
||||
typer.echo(f"# {described['name']} ({described['type_path']})")
|
||||
typer.echo(described["description"] or "")
|
||||
typer.echo("")
|
||||
if frontmatter.get("subtype_field"):
|
||||
typer.echo(f"subtype_field: {frontmatter['subtype_field']}")
|
||||
if frontmatter.get("base_dir"):
|
||||
typer.echo(f"base_dir: {frontmatter.get('root') or 'kb'}/{frontmatter['base_dir']}")
|
||||
if frontmatter.get("title_prefix"):
|
||||
typer.echo(f"title_prefix: {frontmatter['title_prefix']!r}")
|
||||
if described["subtype_field"]:
|
||||
typer.echo(f"subtype_field: {described['subtype_field']}")
|
||||
if described["base_dir"]:
|
||||
typer.echo(f"base_dir: {described['root']}/{described['base_dir']}")
|
||||
if described["title_prefix"]:
|
||||
typer.echo(f"title_prefix: {described['title_prefix']!r}")
|
||||
typer.echo("")
|
||||
|
||||
if not fields:
|
||||
@@ -119,4 +88,4 @@ def describe_type(
|
||||
typer.echo("")
|
||||
|
||||
typer.echo("## Authoring guidance")
|
||||
typer.echo(body.strip())
|
||||
typer.echo(described["body"])
|
||||
|
||||
@@ -189,6 +189,11 @@ def bump_command(
|
||||
minor: bool = typer.Option(False, "--minor", help="Bump MINOR (resets PATCH)"),
|
||||
patch: bool = typer.Option(False, "--patch", help="Bump PATCH"),
|
||||
title: str = typer.Option(..., "--title", help="One-line title for the new CHANGES.md entry"),
|
||||
breaking: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--breaking",
|
||||
help="What stops working, for a boundary-crossing bump (recorded in CHANGES.md). Required on one, refused on any other",
|
||||
),
|
||||
no_migration: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--no-migration",
|
||||
@@ -203,10 +208,14 @@ def bump_command(
|
||||
frontmatter and leaves the prose. `docs verify` afterwards enforces that
|
||||
the two agree, so a bump with no entry cannot reach a release.
|
||||
|
||||
A bump that crosses the compatibility boundary additionally requires a
|
||||
migration document for the new version, or `--no-migration "<reason>"`.
|
||||
An instance learning that it must migrate, with nothing telling it how, is
|
||||
the gap this closes."""
|
||||
A bump that crosses the compatibility boundary - one whose new version is
|
||||
not a drop-in replacement, whether or not any content moves - requires
|
||||
`--breaking "<what stops working>"`, and on top of that either a migration
|
||||
document for the new version or `--no-migration "<reason>"`. An instance
|
||||
learning that it must migrate, with nothing telling it what broke or how to
|
||||
cross, is the gap these close. Which part to pass stays a judgment call
|
||||
this command does not make - it enforces only that a crossing says what it
|
||||
costs."""
|
||||
selected = [name for name, chosen in (("major", major), ("minor", minor), ("patch", patch)) if chosen]
|
||||
if len(selected) != 1:
|
||||
fail("Pass exactly one of --major / --minor / --patch")
|
||||
@@ -239,6 +248,23 @@ def bump_command(
|
||||
crossing = new_version.compat_key != current.compat_key
|
||||
boundary = " (crosses a compatibility boundary - instances must migrate)" if crossing else ""
|
||||
|
||||
if crossing and not breaking:
|
||||
fail(
|
||||
f"{current} -> {new_version} crosses the compatibility boundary, so it is not a "
|
||||
f"drop-in replacement - re-run with --breaking \"<what stops working, and what an "
|
||||
f"instance must do about it>\".\n"
|
||||
f"If that sentence is hard to write because nothing actually breaks - no hand-work "
|
||||
f"on update, and the old version can still be put back - then the bump is probably "
|
||||
f"not --{selected[0]}."
|
||||
)
|
||||
return
|
||||
if breaking and not crossing:
|
||||
fail(
|
||||
f"--breaking only applies to a bump that crosses the compatibility boundary; "
|
||||
f"{current} -> {new_version} does not."
|
||||
)
|
||||
return
|
||||
|
||||
if crossing and not no_migration:
|
||||
from chemenu import kb_state
|
||||
|
||||
@@ -267,6 +293,7 @@ def bump_command(
|
||||
version_mod.insert_changes_entry(
|
||||
text, new_version, today_iso(), title.strip(), author,
|
||||
no_migration_reason=no_migration.strip() if no_migration else None,
|
||||
breaking_reason=breaking.strip() if breaking else None,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
+151
-19
@@ -4,32 +4,164 @@ The repo is a pipeline: `raw/` (untrusted input) -> `types/` + `tools/` (schema
|
||||
compiler) -> `kb/` (compiled knowledge) -> `reports/` (derived output). Only `kb/` is
|
||||
divided into collections; the other three stages are single-purpose directories.
|
||||
|
||||
Repo root is resolved by walking up from this file's location (tools/chemenu/config.py
|
||||
-> tools/ -> repo root), which is stable regardless of the caller's current working
|
||||
directory.
|
||||
Repo root is resolved in three steps - an explicit argument to `resolve_root()`, then
|
||||
`$CHEMENU_ROOT`, then a walk up from this file's location (tools/chemenu/config.py ->
|
||||
tools/ -> repo root). The walk-up stays the default, so `tools/wikitool` behaves exactly
|
||||
as it always has; the two steps in front of it are what lets an in-process caller point
|
||||
this package at a corpus it does not itself live inside.
|
||||
|
||||
**Nothing below is bound at import time.** `ROOT` and every path derived from it are
|
||||
resolved on each attribute access, through the module `__getattr__` at the bottom. They
|
||||
used to be module constants, which had a failure mode worse than the limitation itself:
|
||||
`monkeypatch.setattr(config, "ROOT", other)` repointed `ROOT` and left `KB_DIR` and
|
||||
`RAW_DIR` aimed at wherever this file happens to sit, so a caller that believed it was
|
||||
working on a target tree was in fact answering out of the developer's checkout. Resolving
|
||||
on access makes the derived paths follow whatever `ROOT` currently is - including a
|
||||
monkeypatched one - so the half-repointed state cannot be constructed.
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
# The last resort, and the default every existing caller gets: the checkout this
|
||||
# file is part of.
|
||||
_PACKAGE_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
RAW_DIR = ROOT / "raw"
|
||||
KB_DIR = ROOT / "kb"
|
||||
TYPES_DIR = ROOT / "types"
|
||||
REPORTS_DIR = ROOT / "reports"
|
||||
WORK_DIR = ROOT / "work"
|
||||
INSTRUCTIONS_DIR = ROOT / "instructions"
|
||||
# Points this package at a corpus other than its own checkout. Registered in
|
||||
# `_WIKITOOL_ENV` (tools/chemenu/tests/conftest.py), so the suite runs with it
|
||||
# cleared and a test that wants it sets it itself.
|
||||
ENV_ROOT = "CHEMENU_ROOT"
|
||||
|
||||
# Generated copies of the skill directories under `instructions/`. Both are
|
||||
# gitignored: they are build output, and a fresh clone publishes them with
|
||||
# `wikitool instructions sync` (see instructions/bootstrap.md).
|
||||
AGENTS_SKILLS_DIR = ROOT / ".agents" / "skills"
|
||||
CLAUDE_SKILLS_DIR = ROOT / ".claude" / "skills"
|
||||
|
||||
INDEX_FILE = KB_DIR / "index.md"
|
||||
LOG_FILE = KB_DIR / "log.md"
|
||||
PROVENANCE_FILE = KB_DIR / "provenance.md"
|
||||
def resolve_root(explicit: "Path | str | None" = None) -> Path:
|
||||
"""The repo root, by the documented precedence: argument, then
|
||||
`$CHEMENU_ROOT`, then the checkout this package lives in.
|
||||
|
||||
An explicit argument wins because a caller serving two corpora cannot use a
|
||||
process-wide variable to tell them apart; the variable exists for the case
|
||||
where the caller is a whole process (a server, a CI job) and there is
|
||||
nothing to pass it through.
|
||||
"""
|
||||
if explicit is not None:
|
||||
return Path(explicit).expanduser().resolve()
|
||||
from_env = os.environ.get(ENV_ROOT, "").strip()
|
||||
if from_env:
|
||||
return Path(from_env).expanduser().resolve()
|
||||
return _PACKAGE_ROOT
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
"""`ROOT` as it stands right now, honouring an assignment onto this module.
|
||||
|
||||
Reads the module dict directly rather than `resolve_root()` so that a test
|
||||
(or any caller) setting `config.ROOT` is what the derived paths follow.
|
||||
That assignment is why the derived paths are computed here at all.
|
||||
"""
|
||||
assigned = globals().get("ROOT")
|
||||
return Path(assigned) if assigned is not None else resolve_root()
|
||||
|
||||
|
||||
# Everything under the root, as a name -> relative-path table rather than as
|
||||
# assignments. One place to read, and the only place that has to know a derived
|
||||
# path exists at all.
|
||||
_DERIVED = {
|
||||
"RAW_DIR": ("raw",),
|
||||
"KB_DIR": ("kb",),
|
||||
"TYPES_DIR": ("types",),
|
||||
"REPORTS_DIR": ("reports",),
|
||||
"WORK_DIR": ("work",),
|
||||
"INSTRUCTIONS_DIR": ("instructions",),
|
||||
# Generated copies of the skill directories under `instructions/`. Both are
|
||||
# gitignored: they are build output, and a fresh clone publishes them with
|
||||
# `wikitool instructions sync` (see instructions/bootstrap.md).
|
||||
"AGENTS_SKILLS_DIR": (".agents", "skills"),
|
||||
"CLAUDE_SKILLS_DIR": (".claude", "skills"),
|
||||
}
|
||||
|
||||
# The generated files, derived from `KB_DIR` rather than from the root: a
|
||||
# caller that repoints only the corpus directory must not be left with a log
|
||||
# and a catalog belonging to a different tree.
|
||||
_KB_DERIVED = {
|
||||
"INDEX_FILE": "index.md",
|
||||
"LOG_FILE": "log.md",
|
||||
"PROVENANCE_FILE": "provenance.md",
|
||||
}
|
||||
|
||||
|
||||
# Every name this module resolves rather than stores. Assigning one is
|
||||
# supported - that is what makes the paths repointable at all - but the
|
||||
# assignment has to be taken back afterwards, or it outlives the caller that
|
||||
# made it. See `reset()`.
|
||||
MANAGED_PATHS = ("ROOT", *_DERIVED, *_KB_DERIVED)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def rooted(root: "Path | str"):
|
||||
"""Resolve every managed path under `root` for the duration of the block.
|
||||
|
||||
Some things below the read core reach for `config` directly rather than
|
||||
taking a root - the module-level `TypeResolver` singleton, which has to
|
||||
find `types/`, is the one that matters - so pointing this package at
|
||||
another corpus means pointing `config` at it, not only the functions that
|
||||
accept an argument.
|
||||
|
||||
**Process-wide while it is open, and therefore not thread-safe.** A caller
|
||||
serving several corpora at once holds a lock around it, the same discipline
|
||||
`CorpusCache` documents. That is a real constraint and not a hidden one:
|
||||
`$CHEMENU_ROOT` is process-wide for the same reason, and the server this
|
||||
exists for (Gitea #19) serves one checkout that a `git reset --hard` keeps
|
||||
clean.
|
||||
|
||||
Restores exactly what was there, including "nothing was assigned" - it must
|
||||
not leave `ROOT` bound behind it, or it recreates the stale-binding bug in
|
||||
the shape `reset()` describes.
|
||||
"""
|
||||
previous = {name: globals()[name] for name in MANAGED_PATHS if name in globals()}
|
||||
reset()
|
||||
globals()["ROOT"] = Path(root)
|
||||
try:
|
||||
yield Path(root)
|
||||
finally:
|
||||
reset()
|
||||
globals().update(previous)
|
||||
|
||||
|
||||
def reset() -> None:
|
||||
"""Drop every assignment onto a managed path name, back to resolution.
|
||||
|
||||
The test suite calls this between tests, and it is not optional there.
|
||||
`monkeypatch.setattr(config, "KB_DIR", tmp)` records the old value by
|
||||
*reading* it - which resolves it - and its undo then writes that resolved
|
||||
path back as a real attribute. The name is bound from then on, so the next
|
||||
caller to repoint only `ROOT` gets a `KB_DIR` still aimed at the previous
|
||||
tree: exactly the half-repointed state this module was rewritten to make
|
||||
unconstructible, rebuilt by the cleanup rather than by the test.
|
||||
"""
|
||||
for name in MANAGED_PATHS:
|
||||
globals().pop(name, None)
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Resolve `ROOT` and the paths under it on access (PEP 562).
|
||||
|
||||
Only reached for names *not* in the module dict, so an explicit assignment
|
||||
- `monkeypatch.setattr(config, "ROOT", tmp)` - keeps working and now also
|
||||
carries the derived paths with it, which is the bug this replaces.
|
||||
"""
|
||||
if name == "ROOT":
|
||||
return resolve_root()
|
||||
if name in _DERIVED:
|
||||
return _root().joinpath(*_DERIVED[name])
|
||||
if name in _KB_DERIVED:
|
||||
kb_dir = globals().get("KB_DIR")
|
||||
base = Path(kb_dir) if kb_dir is not None else _root() / "kb"
|
||||
return base / _KB_DERIVED[name]
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
|
||||
def __dir__() -> list[str]:
|
||||
return sorted([*globals(), "ROOT", *_DERIVED, *_KB_DERIVED])
|
||||
|
||||
# Files/patterns to ignore when scanning raw/ for ingest coverage.
|
||||
# CONTRACT.md is the layer's source contract, not source material.
|
||||
@@ -102,7 +234,7 @@ def default_author() -> str | None:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "config", "user.name"],
|
||||
cwd=ROOT,
|
||||
cwd=_root(),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Load the corpus once per revision instead of once per query.
|
||||
|
||||
`load_pages_by_path()` reads and parses every page under `kb/` on every call.
|
||||
For a CLI that is the right shape - one call per process, nothing to reuse, and
|
||||
a cache would only add a way to answer from a tree that has since changed. For
|
||||
a long-lived reader (the MCP server) it is the opposite: the same corpus is
|
||||
reparsed for every request, and the cost grows linearly with the corpus.
|
||||
|
||||
So the cache is an object a caller holds, not a module-level dict that switches
|
||||
itself on behind everyone's back. The CLI holds none and behaves exactly as
|
||||
before; a resident process holds one.
|
||||
|
||||
**The key is the commit SHA, and a dirty tree is never cached.** The SHA alone
|
||||
would be a correctness bug in any checkout someone edits: a session that writes
|
||||
a page and searches for it would be answered from the parse taken before the
|
||||
write, with nothing about the SHA having changed. A clean tree is the state the
|
||||
server actually runs in - it is kept that way by `git fetch && git reset
|
||||
--hard` - so the fast path is the one that holds there, and every other tree
|
||||
falls back to reloading.
|
||||
|
||||
That same SHA is what a response is stamped with, which is deliberate: the
|
||||
revision a caller is told about is by construction the revision its answer was
|
||||
computed from.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.page import Page
|
||||
|
||||
|
||||
def head_commit(root: Optional[Path] = None) -> Optional[str]:
|
||||
"""The full SHA of `HEAD`, or None outside a git checkout."""
|
||||
result = _git(["rev-parse", "HEAD"], root)
|
||||
if result is None or result.returncode != 0:
|
||||
return None
|
||||
return result.stdout.strip() or None
|
||||
|
||||
|
||||
def is_dirty(root: Optional[Path] = None, path: Optional[Path] = None) -> bool:
|
||||
"""Whether the working tree has uncommitted changes under `path`.
|
||||
|
||||
Errs toward dirty: if git cannot answer, the answer is "assume it changed".
|
||||
A cache that treats "unknown" as clean serves stale pages, which is the one
|
||||
outcome this module exists to prevent.
|
||||
"""
|
||||
root = root or config.ROOT
|
||||
target = path or config.KB_DIR
|
||||
try:
|
||||
relative = Path(target).resolve().relative_to(Path(root).resolve()).as_posix()
|
||||
except ValueError:
|
||||
return True
|
||||
result = _git(["status", "--porcelain", "--", relative], root)
|
||||
if result is None or result.returncode != 0:
|
||||
return True
|
||||
return bool(result.stdout.strip())
|
||||
|
||||
|
||||
def _git(args: list[str], root: Optional[Path] = None):
|
||||
try:
|
||||
return subprocess.run(
|
||||
["git", *args],
|
||||
cwd=root or config.ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
check=False,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
|
||||
|
||||
class CorpusCache:
|
||||
"""One parsed corpus, reused while the checkout stays on the same commit.
|
||||
|
||||
Not thread-safe by itself: a caller serving concurrent requests holds the
|
||||
lock. Kept out of here because the locking discipline belongs to whoever
|
||||
owns the request loop, and a lock hidden in a cache is one nobody can see
|
||||
when they need to reason about it.
|
||||
"""
|
||||
|
||||
def __init__(self, kb_dir: Optional[Path] = None, root: Optional[Path] = None):
|
||||
self.kb_dir = kb_dir
|
||||
self.root = root
|
||||
self._pages: Optional[dict[str, Page]] = None
|
||||
self._revision: Optional[str] = None
|
||||
|
||||
@property
|
||||
def revision(self) -> Optional[str]:
|
||||
"""The commit the cached corpus was read at, or None if nothing is
|
||||
cached (including because the tree was dirty)."""
|
||||
return self._revision
|
||||
|
||||
def current_revision(self) -> Optional[str]:
|
||||
"""The commit this corpus would be cached under right now: `HEAD` on a
|
||||
clean tree, None on a dirty one or outside git. None means uncacheable,
|
||||
which is why it is also what a caller should report as "no revision" -
|
||||
an answer read out of a dirty tree does not correspond to any commit."""
|
||||
root = self.root or config.ROOT
|
||||
if is_dirty(root, self.kb_dir or config.KB_DIR):
|
||||
return None
|
||||
return head_commit(root)
|
||||
|
||||
def load(self) -> tuple[dict[str, Page], Optional[str]]:
|
||||
"""(pages, revision). Reparses whenever the revision is not the cached
|
||||
one, and on every call while the tree is dirty."""
|
||||
# Imported here rather than at module level: `commands.search` pulls in
|
||||
# typer, and this module is meant to be importable by a library caller
|
||||
# that has no CLI. Removing that edge properly is the library-boundary
|
||||
# work, not this file's job.
|
||||
from chemenu.commands.search import load_pages_by_path
|
||||
|
||||
revision = self.current_revision()
|
||||
if revision is None or revision != self._revision or self._pages is None:
|
||||
pages = load_pages_by_path(self.kb_dir, self.root)
|
||||
if revision is None:
|
||||
self._pages, self._revision = None, None
|
||||
return pages, None
|
||||
self._pages, self._revision = pages, revision
|
||||
return self._pages, self._revision
|
||||
@@ -0,0 +1,31 @@
|
||||
"""The exception contract at the library boundary.
|
||||
|
||||
The CLI reports a bad argument by printing an `ERROR` line and leaving through
|
||||
`typer.Exit(1)`. That is the right answer for a terminal and the wrong one for
|
||||
an in-process caller, which gets an exit code where it expected a value, plus
|
||||
module-global state (`_util._declined`) surviving into its next call.
|
||||
|
||||
So the core raises, and the CLI adapter translates. `ChemenuError` is the one
|
||||
class a library caller has to know; the two below it separate "your input was
|
||||
wrong, a different argument would work" from "the machinery underneath failed",
|
||||
which is the same distinction the CLI's exit codes draw.
|
||||
|
||||
`ValidationError` also inherits `ValueError`. Not for elegance: `PredicateError`
|
||||
was a `ValueError` before this existed, and callers catch it that way.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
class ChemenuError(Exception):
|
||||
"""Base for every error this package raises deliberately."""
|
||||
|
||||
|
||||
class ValidationError(ChemenuError, ValueError):
|
||||
"""The caller's input was rejected. Re-running unchanged fails identically;
|
||||
the CLI renders this as its exit-1 `ERROR` line."""
|
||||
|
||||
|
||||
class BackendError(ChemenuError, RuntimeError):
|
||||
"""A dependency the core relies on was missing or failed - `rg` absent, a
|
||||
search that had to be killed. Not the caller's argument, and not
|
||||
necessarily permanent."""
|
||||
+124
-29
@@ -16,29 +16,137 @@ from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from chemenu.errors import ValidationError
|
||||
|
||||
try: # pragma: no cover - which branch runs depends on the host's libyaml
|
||||
from yaml import CSafeLoader as _Loader
|
||||
except ImportError: # pragma: no cover
|
||||
from yaml import SafeLoader as _Loader
|
||||
|
||||
FRONTMATTER_RE = re.compile(r"\A---\n(.*?)\n---\n?(.*)\Z", re.DOTALL)
|
||||
|
||||
# Frontmatter is a flat block of scalars and one-line lists. Real pages sit
|
||||
# well under a kilobyte, so this is not a budget anyone writes against - it is
|
||||
# the bound that keeps parse cost proportional to the corpus rather than to
|
||||
# whatever a single file claims to be. It matters because the parser is on the
|
||||
# read path, and the read path is what gets exposed.
|
||||
MAX_FRONTMATTER_BYTES = 64 * 1024
|
||||
|
||||
# YAML anchors and aliases are refused outright rather than budgeted. A page's
|
||||
# frontmatter has no use for them, and alias expansion is where a small
|
||||
# document becomes an enormous object graph: 267 bytes of nested aliases
|
||||
# compose in 0.2 ms into 672,603 nodes, growing 9**n with nesting depth at
|
||||
# constant parse time. A size limit alone does not touch that, because the
|
||||
# input stays small - see `ALIAS_BOMB` in the tests.
|
||||
#
|
||||
# The check runs on the *event* stream (`yaml.parse`), which is streaming and
|
||||
# resolves nothing - so asking the question costs O(text) and never triggers
|
||||
# the expansion it is asking about. `"*"` is a necessary character in any alias
|
||||
# node, so its absence proves absence without parsing at all, which is the case
|
||||
# every real page takes.
|
||||
_ALIAS_HINT = "*"
|
||||
|
||||
|
||||
class FrontmatterError(ValidationError):
|
||||
"""Frontmatter that cannot be used: missing, malformed, oversized, or
|
||||
refused by a limit. Raised only by the strict entry points - the permissive
|
||||
ones report it as a string instead."""
|
||||
|
||||
|
||||
def _load_frontmatter(fm_text: str) -> tuple[dict[str, Any] | None, str | None]:
|
||||
"""Parse one frontmatter block into (mapping, error). Exactly one is None.
|
||||
|
||||
The single parser behind both `read_page()` and `frontmatter_error()`. They
|
||||
used to have one each, which is how the permissive path could degrade to
|
||||
`{}` for a reason the strict path described differently - and how every
|
||||
caller wanting both answers read and parsed the file twice.
|
||||
"""
|
||||
encoded = len(fm_text.encode("utf-8"))
|
||||
if encoded > MAX_FRONTMATTER_BYTES:
|
||||
return None, (
|
||||
f"frontmatter is {encoded} bytes, over the {MAX_FRONTMATTER_BYTES}-byte limit"
|
||||
)
|
||||
if _ALIAS_HINT in fm_text:
|
||||
try:
|
||||
for event in yaml.parse(fm_text, Loader=_Loader):
|
||||
if isinstance(event, yaml.AliasEvent):
|
||||
return None, (
|
||||
"frontmatter uses a YAML alias (`*"
|
||||
f"{event.anchor}`); anchors and aliases are not allowed here"
|
||||
)
|
||||
except yaml.YAMLError as exc:
|
||||
return None, f"invalid YAML frontmatter: {_first_line(exc)}"
|
||||
except RecursionError:
|
||||
return None, "frontmatter is nested too deeply to parse"
|
||||
try:
|
||||
parsed = yaml.load(fm_text, Loader=_Loader)
|
||||
except yaml.YAMLError as exc:
|
||||
return None, f"invalid YAML frontmatter: {_first_line(exc)}"
|
||||
except RecursionError:
|
||||
# PyYAML composes recursively, so deep nesting exhausts the interpreter
|
||||
# stack rather than raising a YAMLError. Unbounded nesting is bounded by
|
||||
# MAX_FRONTMATTER_BYTES; this catches what fits under it.
|
||||
return None, "frontmatter is nested too deeply to parse"
|
||||
if parsed is None:
|
||||
return None, "empty frontmatter block"
|
||||
if not isinstance(parsed, dict):
|
||||
return None, f"frontmatter is {type(parsed).__name__}, expected a mapping"
|
||||
return parsed, None
|
||||
|
||||
|
||||
def _first_line(exc: Exception) -> str:
|
||||
text = str(exc)
|
||||
return text.splitlines()[0] if text else exc.__class__.__name__
|
||||
|
||||
|
||||
def read_page_with_error(path: Path) -> tuple[dict[str, Any], str, str | None]:
|
||||
"""(frontmatter, body, error) - the permissive read, with the reason it was
|
||||
permissive handed back instead of dropped.
|
||||
|
||||
A page whose YAML is broken reads as `{}`, and a `{}` page then has no
|
||||
`confidence` and no `kind`: it drops out of `--field confidence<0.6` -
|
||||
precisely the query meant to find pages in bad shape - while looking to the
|
||||
caller like a page that simply did not match. Returning the reason is what
|
||||
lets a caller say so instead of losing the page quietly.
|
||||
"""
|
||||
text = path.read_text(encoding="utf-8")
|
||||
match = FRONTMATTER_RE.match(text)
|
||||
if not match:
|
||||
return {}, text, "no `---` frontmatter block"
|
||||
fm_text, body = match.group(1), match.group(2)
|
||||
parsed, error = _load_frontmatter(fm_text)
|
||||
if error == "empty frontmatter block":
|
||||
# An empty block is a legitimate shape for the permissive read - it
|
||||
# carries no fields, and there is nothing to lose. Only `lint` treats
|
||||
# it as a finding.
|
||||
return {}, body, error
|
||||
return (parsed or {}), body, error
|
||||
|
||||
|
||||
def read_page(path: Path) -> tuple[dict[str, Any], str]:
|
||||
"""Return (frontmatter_dict, body) for a markdown file. If the file has no
|
||||
frontmatter block, returns ({}, full_text).
|
||||
|
||||
Deliberately permissive: malformed YAML degrades to an empty dict so bulk
|
||||
operations never crash on one bad page. Use `frontmatter_error()` (which
|
||||
`wikitool lint` does) to surface those pages instead of losing them
|
||||
silently.
|
||||
operations never crash on one bad page. Use `read_page_with_error()` (search
|
||||
does) or `frontmatter_error()` (`wikitool lint` does) to surface those pages
|
||||
instead of losing them silently.
|
||||
"""
|
||||
text = path.read_text(encoding="utf-8")
|
||||
match = FRONTMATTER_RE.match(text)
|
||||
if not match:
|
||||
return {}, text
|
||||
fm_text, body = match.group(1), match.group(2)
|
||||
try:
|
||||
frontmatter = yaml.safe_load(fm_text) or {}
|
||||
except yaml.YAMLError:
|
||||
frontmatter = {}
|
||||
if not isinstance(frontmatter, dict):
|
||||
frontmatter = {}
|
||||
frontmatter, body, _ = read_page_with_error(path)
|
||||
return frontmatter, body
|
||||
|
||||
|
||||
def read_page_strict(path: Path) -> tuple[dict[str, Any], str]:
|
||||
"""`read_page()` that raises `FrontmatterError` instead of degrading.
|
||||
|
||||
For any path that ingests frontmatter this instance did not write itself.
|
||||
The permissive read is right for bulk operations over a corpus the operator
|
||||
committed; it is wrong the moment the document arrives from outside, where
|
||||
"unparseable" must stop the document rather than empty it.
|
||||
"""
|
||||
frontmatter, body, error = read_page_with_error(path)
|
||||
if error is not None:
|
||||
raise FrontmatterError(f"{path}: {error}")
|
||||
return frontmatter, body
|
||||
|
||||
|
||||
@@ -50,20 +158,7 @@ def frontmatter_error(path: Path) -> str | None:
|
||||
YAML is malformed (or whose frontmatter block is missing entirely) reads
|
||||
back as `{}` and then quietly slips past every frontmatter-driven check.
|
||||
"""
|
||||
text = path.read_text(encoding="utf-8")
|
||||
match = FRONTMATTER_RE.match(text)
|
||||
if not match:
|
||||
return "no `---` frontmatter block"
|
||||
try:
|
||||
parsed = yaml.safe_load(match.group(1))
|
||||
except yaml.YAMLError as exc:
|
||||
reason = str(exc).splitlines()[0] if str(exc) else exc.__class__.__name__
|
||||
return f"invalid YAML frontmatter: {reason}"
|
||||
if parsed is None:
|
||||
return "empty frontmatter block"
|
||||
if not isinstance(parsed, dict):
|
||||
return f"frontmatter is {type(parsed).__name__}, expected a mapping"
|
||||
return None
|
||||
return read_page_with_error(path)[2]
|
||||
|
||||
|
||||
def _format_scalar(value: Any, flow: bool = False) -> str:
|
||||
@@ -132,7 +227,7 @@ def _round_trips_as_string(text: str, flow: bool = False) -> bool:
|
||||
"""
|
||||
probe, expected = (f"[{text}]", [text]) if flow else (text, text)
|
||||
try:
|
||||
return yaml.safe_load(probe) == expected
|
||||
return yaml.load(probe, Loader=_Loader) == expected
|
||||
except yaml.YAMLError:
|
||||
# Unparseable bare - quoting is exactly the fix.
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,431 @@
|
||||
"""The lint core, with no CLI attached.
|
||||
|
||||
Split out of `commands/lint.py` for the reason given in
|
||||
`chemenu/search/service.py`: `run_lint()` is a pure function over a corpus
|
||||
directory, and it was sitting in a module that imports `typer` and `rich`, so
|
||||
no in-process caller could reach it without the CLI head.
|
||||
|
||||
Everything that decides *findings* lives here. Everything that decides *how a
|
||||
terminal sees them* - the report file, the exit code, the flags - stays in
|
||||
`commands/lint.py`. `render_markdown()` and `render_summary()` are on this side
|
||||
of the line because the markdown report is a data product (it is what
|
||||
`reports/` holds and what `kb/log.md` refers to), not terminal formatting.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.frontmatter_io import frontmatter_error
|
||||
from chemenu.markdown_code import strip_code_spans
|
||||
from chemenu.provenance import broken_raw_refs as find_broken_raw_refs
|
||||
from chemenu.provenance import duplicate_raw_file_owners as find_duplicate_raw_file_owners
|
||||
from chemenu.provenance import extract_inline_cites
|
||||
from chemenu.provenance import legacy_citation_markers as find_legacy_citation_markers
|
||||
from chemenu.provenance import legacy_source_pages as find_legacy_source_pages
|
||||
from chemenu.provenance import orphan_footnote_defs as find_orphan_footnote_defs
|
||||
from chemenu.provenance import uncovered_raw_files as find_uncovered_raw_files
|
||||
from chemenu.provenance import undefined_footnote_refs as find_undefined_footnote_refs
|
||||
from chemenu.kb_scan import (
|
||||
GENERATED_INDEX,
|
||||
WIKILINK_RE,
|
||||
build_link_graph,
|
||||
find_duplicate_title_paths,
|
||||
inbound_links,
|
||||
load_kb_pages,
|
||||
)
|
||||
from chemenu.type_resolver import resolver
|
||||
|
||||
# Style guide's one mechanically-checkable rule (hard oracle: a plain count).
|
||||
# The rest of the style guide (tone, AI-phrase avoidance) is a soft/proxy judgment
|
||||
# and stays with the LLM - see wiki-manage/wiki-ingest skill guidance, not lint.
|
||||
#
|
||||
# The unit is a quote, not a `>` line. It used to be the line, which measured
|
||||
# the wrap width the rule has no opinion about: one quotation written long
|
||||
# counted 1 and the same quotation wrapped at 100 columns counted 4. An author
|
||||
# who took the finding seriously made the page harder to read to quiet it.
|
||||
|
||||
|
||||
QUOTE_LIMIT = 2
|
||||
|
||||
# How many hub pages `most_linked` reports. Purely informational (wiki-status
|
||||
# surfaces it); not a finding, so the cutoff only bounds report size.
|
||||
MOST_LINKED_COUNT = 10
|
||||
|
||||
|
||||
def count_quote_blocks(body: str) -> int:
|
||||
"""How many distinct blockquotes `body` carries.
|
||||
|
||||
A run of consecutive `>` lines is one quote; a blank line or any
|
||||
non-quoted line ends it. Code is masked out first, so a `>` inside a
|
||||
fenced shell transcript is a prompt, not a quotation.
|
||||
|
||||
Lazy continuation - a quote whose wrapped lines drop the `>` - reads here
|
||||
as two quotes rather than one. That over-counts in the direction the limit
|
||||
already errs on, and the corpus prefixes every line, so the alternative
|
||||
(tracking paragraph state) buys nothing.
|
||||
"""
|
||||
count, in_quote = 0, False
|
||||
for line in strip_code_spans(body).splitlines():
|
||||
is_quote = line.lstrip().startswith(">")
|
||||
if is_quote and not in_quote:
|
||||
count += 1
|
||||
in_quote = is_quote
|
||||
return count
|
||||
|
||||
|
||||
def run_lint(kb_dir: Path) -> dict:
|
||||
pages = load_kb_pages(kb_dir)
|
||||
duplicate_titles = find_duplicate_title_paths(kb_dir, config.ROOT)
|
||||
|
||||
# Pages whose frontmatter can't be parsed read back as `{}` everywhere
|
||||
# else, which would let them slip past every frontmatter-driven check
|
||||
# below with no finding at all - so they are detected explicitly.
|
||||
frontmatter_errors = []
|
||||
for title, page in sorted(pages.items()):
|
||||
reason = frontmatter_error(page.path)
|
||||
if reason is None and not page.frontmatter.get("type"):
|
||||
reason = "missing `type:` field"
|
||||
if reason is not None:
|
||||
frontmatter_errors.append({"page": title, "error": reason})
|
||||
|
||||
graph = build_link_graph(pages)
|
||||
broken_links = [
|
||||
{"page": title, "target": target}
|
||||
for title, targets in graph.items()
|
||||
for target in sorted(targets)
|
||||
if target not in pages
|
||||
]
|
||||
|
||||
inbound = inbound_links({t: v for t, v in graph.items() if t != "index"})
|
||||
orphan_pages = sorted(
|
||||
title
|
||||
for title, sources in inbound.items()
|
||||
if not sources
|
||||
and title not in ("index", "log")
|
||||
# comparison pages are not linked to by design; index.md is sufficient coverage
|
||||
and pages[title].kind != "comparison"
|
||||
)
|
||||
|
||||
# Same link graph, opposite end: the most-linked-to pages are the wiki's
|
||||
# hubs. Reported (not judged) so `wiki-status` can show them without
|
||||
# re-deriving the graph.
|
||||
inbound_counts = {title: len(sources) for title, sources in inbound.items()}
|
||||
most_linked = [
|
||||
{"page": title, "inbound": count}
|
||||
for title, count in sorted(inbound_counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
if count > 0
|
||||
][:MOST_LINKED_COUNT]
|
||||
|
||||
# The catalog is sharded: `kb/index.md` is a map carrying counts and links,
|
||||
# and the page rows live in a generated INDEX.md per collection/area. Both
|
||||
# halves have to be read, or every page reads as missing from the index.
|
||||
index_text = "".join(
|
||||
path.read_text(encoding="utf-8")
|
||||
for path in [kb_dir / "index.md", *sorted(kb_dir.rglob(GENERATED_INDEX))]
|
||||
if path.exists()
|
||||
)
|
||||
index_links = {m.group(1).strip() for m in WIKILINK_RE.finditer(index_text)}
|
||||
missing_from_index = sorted(set(pages) - index_links - {"index", "log"})
|
||||
dangling_index_entries = sorted(index_links - set(pages))
|
||||
|
||||
title_mismatches = []
|
||||
for title, page in sorted(pages.items()):
|
||||
if page.kind not in ("entity", "concept"):
|
||||
continue
|
||||
h1 = page.h1_title
|
||||
if h1 is not None and h1 != title:
|
||||
title_mismatches.append({"page": title, "h1": h1})
|
||||
|
||||
unmarked_provenance = []
|
||||
for title, page in sorted(pages.items()):
|
||||
if page.kind not in ("entity", "concept"):
|
||||
continue
|
||||
sources_list = page.frontmatter.get("sources") or []
|
||||
if not sources_list and page.frontmatter.get("provenance") != "general":
|
||||
unmarked_provenance.append(title)
|
||||
|
||||
citation_frontmatter_drift = []
|
||||
for title, page in sorted(pages.items()):
|
||||
sources_list = set(page.frontmatter.get("sources") or [])
|
||||
cited = {cited_title for cited_title, _file in extract_inline_cites(page.body)}
|
||||
cited.discard(title) # a source page citing itself for a specific file within it is not drift
|
||||
for missing_source in sorted(cited - sources_list):
|
||||
citation_frontmatter_drift.append({"page": title, "cited_but_not_in_sources": missing_source})
|
||||
|
||||
legacy_citation_markers = find_legacy_citation_markers(pages)
|
||||
undefined_footnote_refs = find_undefined_footnote_refs(pages)
|
||||
orphan_footnote_defs = find_orphan_footnote_defs(pages)
|
||||
|
||||
# The frontmatter half of the link graph. `broken_links` above only walks
|
||||
# `[[wikilinks]]` in page *bodies*, so a `related:`/`sources:`/`entities:`
|
||||
# entry naming a page that does not exist - a rename that was not
|
||||
# propagated, a deleted page, or a URL pasted where a title belongs - used
|
||||
# to pass every check. Which fields hold page titles is declared by each
|
||||
# type-spec's `page_ref_fields:`, not hardcoded here.
|
||||
dangling_frontmatter_refs = []
|
||||
for title, page in sorted(pages.items()):
|
||||
type_path = page.frontmatter.get("type")
|
||||
if not type_path:
|
||||
continue
|
||||
try:
|
||||
ref_fields = resolver.get_page_ref_fields(type_path, page.path)
|
||||
except ValueError:
|
||||
continue # unresolvable type is already reported as type_resolution_errors
|
||||
for field in ref_fields:
|
||||
for target in page.frontmatter.get(field) or []:
|
||||
if target not in pages:
|
||||
dangling_frontmatter_refs.append(
|
||||
{"page": title, "field": field, "target": target}
|
||||
)
|
||||
|
||||
quote_limit_violations = []
|
||||
for title, page in sorted(pages.items()):
|
||||
quote_count = count_quote_blocks(page.body)
|
||||
if quote_count > QUOTE_LIMIT:
|
||||
quote_limit_violations.append({"page": title, "quote_count": quote_count})
|
||||
|
||||
# Type system validation. Lint reports are not validated here: they are
|
||||
# written to `reports/` outside kb/ and are never pages, so nothing this
|
||||
# loop scans can be one.
|
||||
invalid_type_paths = []
|
||||
type_resolution_errors = []
|
||||
schema_validation_errors = []
|
||||
|
||||
for title, page in sorted(pages.items()):
|
||||
type_path = page.frontmatter.get("type")
|
||||
if not type_path:
|
||||
continue
|
||||
|
||||
# Check if type path is valid
|
||||
if not type_path.endswith('.md'):
|
||||
invalid_type_paths.append({"page": title, "type": type_path, "error": "Type path must end with .md"})
|
||||
continue
|
||||
|
||||
# Try to resolve and validate the type
|
||||
try:
|
||||
resolver.load_type_spec(type_path, page.path)
|
||||
|
||||
# Try schema validation
|
||||
try:
|
||||
resolver.validate_frontmatter(page.frontmatter, type_path, page.path)
|
||||
except ValueError as schema_error:
|
||||
schema_validation_errors.append({"page": title, "type": type_path, "error": str(schema_error)})
|
||||
|
||||
except ValueError as resolution_error:
|
||||
type_resolution_errors.append({"page": title, "type": type_path, "error": str(resolution_error)})
|
||||
|
||||
return {
|
||||
"generated": date.today().isoformat(),
|
||||
"page_count": len(pages),
|
||||
"frontmatter_errors": frontmatter_errors,
|
||||
"broken_links": broken_links,
|
||||
"orphan_pages": orphan_pages,
|
||||
"most_linked": most_linked,
|
||||
"inbound_counts": inbound_counts,
|
||||
"missing_from_index": missing_from_index,
|
||||
"dangling_index_entries": dangling_index_entries,
|
||||
"title_mismatches": title_mismatches,
|
||||
"duplicate_titles": duplicate_titles,
|
||||
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
|
||||
"broken_raw_refs": find_broken_raw_refs(pages),
|
||||
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
|
||||
"legacy_source_pages": find_legacy_source_pages(pages),
|
||||
"unmarked_provenance": unmarked_provenance,
|
||||
"citation_frontmatter_drift": citation_frontmatter_drift,
|
||||
"legacy_citation_markers": legacy_citation_markers,
|
||||
"undefined_footnote_refs": undefined_footnote_refs,
|
||||
"orphan_footnote_defs": orphan_footnote_defs,
|
||||
"dangling_frontmatter_refs": dangling_frontmatter_refs,
|
||||
"quote_limit_violations": quote_limit_violations,
|
||||
"invalid_type_paths": invalid_type_paths,
|
||||
"type_resolution_errors": type_resolution_errors,
|
||||
"schema_validation_errors": schema_validation_errors,
|
||||
}
|
||||
|
||||
|
||||
def _section(lines: list[str], title: str, items: list, formatter) -> None:
|
||||
lines.append(f"## {title}")
|
||||
lines.append("")
|
||||
if not items:
|
||||
lines.append("None found.")
|
||||
else:
|
||||
for item in items:
|
||||
lines.append(f"- {formatter(item)}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
def render_markdown(report: dict) -> str:
|
||||
lines = [f"# Structural Lint Report ({report['generated']})", ""]
|
||||
lines.append(f"Scanned {report['page_count']} pages under `wiki/`. This report covers only")
|
||||
lines.append("mechanically-verifiable structural issues; see the Semantic Review section")
|
||||
lines.append("below for judgment calls the LLM should complete.")
|
||||
lines.append("")
|
||||
|
||||
_section(
|
||||
lines, "Unreadable Frontmatter", report["frontmatter_errors"],
|
||||
lambda i: f"[[{i['page']}]] - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Broken Wikilinks", report["broken_links"],
|
||||
lambda i: f"[[{i['page']}]] links to missing [[{i['target']}]]",
|
||||
)
|
||||
_section(lines, "Orphan Pages (no inbound links)", report["orphan_pages"], lambda i: f"[[{i}]]")
|
||||
_section(
|
||||
lines, f"Most-Linked Pages (top {MOST_LINKED_COUNT} hubs)", report["most_linked"],
|
||||
lambda i: f"[[{i['page']}]] - {i['inbound']} inbound link(s)",
|
||||
)
|
||||
_section(lines, "Pages Missing from index.md", report["missing_from_index"], lambda i: f"[[{i}]]")
|
||||
_section(lines, "Dangling index.md Entries", report["dangling_index_entries"], lambda i: f"[[{i}]]")
|
||||
_section(
|
||||
lines, "Duplicate Titles (naming collisions)", report["duplicate_titles"],
|
||||
lambda i: f"`{i['stem']}` -> {', '.join(f'`{p}`' for p in i['paths'])}",
|
||||
)
|
||||
_section(
|
||||
lines, "Filename / H1 Title Mismatches", report["title_mismatches"],
|
||||
lambda i: f"[[{i['page']}]] H1 is '{i['h1']}'",
|
||||
)
|
||||
_section(
|
||||
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
|
||||
lambda i: f"`{i}`",
|
||||
)
|
||||
_section(
|
||||
lines, "Broken raw_files References", report["broken_raw_refs"],
|
||||
lambda i: f"[[{i['page']}]] -> `{i['raw_path']}` (does not exist)",
|
||||
)
|
||||
_section(
|
||||
lines, "Raw Files With More Than One Owner", report["duplicate_raw_file_owners"],
|
||||
lambda i: f"`{i['raw_file']}` is claimed by " + ", ".join(f"[[{t}]]" for t in i["owners"]),
|
||||
)
|
||||
_section(
|
||||
lines, "Legacy source: Field (not yet migrated to raw_files:)", report["legacy_source_pages"],
|
||||
lambda i: f"[[{i['page']}]] source: `{i['source']}` ({i['reason']})",
|
||||
)
|
||||
_section(
|
||||
lines, "Pages Missing provenance: general Marker", report["unmarked_provenance"],
|
||||
lambda i: f"[[{i}]] has no sources and is not marked `provenance: general`",
|
||||
)
|
||||
_section(
|
||||
lines, "Citation / Frontmatter Drift", report["citation_frontmatter_drift"],
|
||||
lambda i: f"[[{i['page']}]] cites [[{i['cited_but_not_in_sources']}]] inline but it is missing from frontmatter `sources:`",
|
||||
)
|
||||
_section(
|
||||
lines, "Legacy Citation Markers (pre-migration `^[[...]]`)", report["legacy_citation_markers"],
|
||||
lambda i: f"[[{i['page']}]] still has `{i['marker']}` - run `wikitool cite add` and replace it with the `[^cite-id]` it prints",
|
||||
)
|
||||
_section(
|
||||
lines, "Undefined Footnote References", report["undefined_footnote_refs"],
|
||||
lambda i: f"[[{i['page']}]] references `[^{i['ref']}]`, which has no `[^{i['ref']}]: [[...]]` definition",
|
||||
)
|
||||
_section(
|
||||
lines, "Orphan Footnote Definitions", report["orphan_footnote_defs"],
|
||||
lambda i: f"[[{i['page']}]] defines `[^{i['id']}]` (-> [[{i['source']}]]) but nothing references it - run `wikitool cite sync`",
|
||||
)
|
||||
_section(
|
||||
lines, "Dangling Frontmatter References", report["dangling_frontmatter_refs"],
|
||||
lambda i: f"[[{i['page']}]] `{i['field']}:` names `{i['target']}`, which is not a page",
|
||||
)
|
||||
_section(
|
||||
lines, "Invalid Type Paths", report["invalid_type_paths"],
|
||||
lambda i: f"[[{i['page']}]] has type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Type Resolution Errors", report["type_resolution_errors"],
|
||||
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, "Schema Validation Errors", report["schema_validation_errors"],
|
||||
lambda i: f"[[{i['page']}]] type: `{i['type']}` - {i['error']}",
|
||||
)
|
||||
_section(
|
||||
lines, f"Pages Exceeding Quote Limit (>{QUOTE_LIMIT}/page)", report["quote_limit_violations"],
|
||||
lambda i: f"[[{i['page']}]] has {i['quote_count']} quotes - trim or confirm they're load-bearing",
|
||||
)
|
||||
|
||||
lines.append("## Semantic Review (LLM to complete)")
|
||||
lines.append("")
|
||||
lines.append("- Contradictions across pages: TODO")
|
||||
lines.append("- Stale claims (unconfirmed >6 months): TODO")
|
||||
lines.append("- Suggested new pages / missing cross-references: TODO")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# Sections that always carry content but are not findings, so the summary
|
||||
# handles them separately: a hub list is a statistic, and the semantic review
|
||||
# is the checklist that follows the report rather than part of it.
|
||||
INFORMATIONAL_SECTIONS = ("Most-Linked Pages",)
|
||||
SEMANTIC_REVIEW_SECTION = "Semantic Review"
|
||||
|
||||
|
||||
def _split_sections(markdown: str) -> tuple[str, list[tuple[str, str]]]:
|
||||
"""Cut a rendered report into its preamble and (title, body) sections."""
|
||||
preamble, *rest = markdown.split("\n## ")
|
||||
sections = []
|
||||
for part in rest:
|
||||
title, _, body = part.partition("\n")
|
||||
sections.append((title.strip(), body.strip()))
|
||||
return preamble.rstrip(), sections
|
||||
|
||||
|
||||
def render_summary(report: dict) -> str:
|
||||
"""The same report with the empty sections removed.
|
||||
|
||||
On a healthy corpus the full report is better than 90% "None found.", so
|
||||
reading it in the terminal means paging past the answer. The file on disk
|
||||
stays complete - this is what gets printed, and the written path underneath
|
||||
it is how the rest is reached without running lint a second time.
|
||||
"""
|
||||
preamble, sections = _split_sections(render_markdown(report))
|
||||
findings, trailing = [], []
|
||||
for title, body in sections:
|
||||
if title.startswith(SEMANTIC_REVIEW_SECTION):
|
||||
trailing.append((title, body))
|
||||
elif body != "None found." and not title.startswith(INFORMATIONAL_SECTIONS):
|
||||
findings.append((title, body))
|
||||
lines = [preamble, ""]
|
||||
if not findings:
|
||||
lines += ["No structural findings.", ""]
|
||||
for title, body in findings + trailing:
|
||||
lines += [f"## {title}", "", body, ""]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def default_report_path(report: dict) -> Path:
|
||||
"""Where a report goes when the caller names no path.
|
||||
|
||||
`reports/` is derived and gitignored ([reports/CONTRACT.md]), so writing
|
||||
here by default costs the tree nothing.
|
||||
"""
|
||||
return config.ROOT / "reports" / f"Lint Report {report['generated']}.md"
|
||||
|
||||
|
||||
# Findings that make a tree structurally wrong rather than merely untidy.
|
||||
# `orphan_pages` is deliberately absent: many pages are validly reachable
|
||||
# through the index or navigation only. `quote_limit_violations` is advisory
|
||||
# too - it flags a habit, not a broken tree.
|
||||
#
|
||||
# One definition, used by `lint --fail-on-error` and by the eval scorecard: if
|
||||
# the two disagreed, a run could pass its score while lint refused it.
|
||||
HARD_ERROR_KEYS = (
|
||||
"frontmatter_errors",
|
||||
"broken_links",
|
||||
"dangling_index_entries",
|
||||
"duplicate_titles",
|
||||
"broken_raw_refs",
|
||||
"duplicate_raw_file_owners",
|
||||
"legacy_source_pages",
|
||||
"citation_frontmatter_drift",
|
||||
"legacy_citation_markers",
|
||||
"undefined_footnote_refs",
|
||||
"orphan_footnote_defs",
|
||||
"dangling_frontmatter_refs",
|
||||
"invalid_type_paths",
|
||||
"type_resolution_errors",
|
||||
"schema_validation_errors",
|
||||
)
|
||||
|
||||
|
||||
def has_hard_errors(report: dict) -> bool:
|
||||
return any(report.get(key) for key in HARD_ERROR_KEYS)
|
||||
@@ -0,0 +1,8 @@
|
||||
"""The MCP read server: Chemenu's second consumer.
|
||||
|
||||
The CLI and this server are two adapters over one core (`chemenu.api`), not a
|
||||
CLI with a network interface bolted on. See `server.py`.
|
||||
"""
|
||||
from chemenu.mcp.server import build_server
|
||||
|
||||
__all__ = ["build_server"]
|
||||
@@ -0,0 +1,79 @@
|
||||
"""`python -m chemenu.mcp` - start the read server.
|
||||
|
||||
Deliberately argparse and not typer. This process is the one place that must
|
||||
not pull the CLI head in: the whole point of the library boundary is that a
|
||||
second consumer costs `yaml`, `jsonschema` and the MCP SDK, and nothing else.
|
||||
|
||||
python -m chemenu.mcp # stdio
|
||||
python -m chemenu.mcp --transport streamable-http # behind the proxy
|
||||
CHEMENU_ROOT=/srv/wiki python -m chemenu.mcp --transport streamable-http \
|
||||
--host 0.0.0.0 --port 8000
|
||||
|
||||
`--host`/`--port` apply to `streamable-http` only. They are here because the
|
||||
default binds loopback, and a server in a container with a reverse proxy in
|
||||
front of it has to bind an interface the proxy can reach - that is a property
|
||||
of the software, not of one installation. *Which* host and port a given
|
||||
deployment picks is infrastructure and stays out of this repository.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
from chemenu.mcp.server import TRANSPORTS, TraceWouldWriteIntoCorpus, serve
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python -m chemenu.mcp",
|
||||
description="Serve a Chemenu wiki read-only over MCP.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--transport",
|
||||
choices=TRANSPORTS,
|
||||
default="stdio",
|
||||
help="stdio for local use and testing; streamable-http for a deployed "
|
||||
"instance behind the Traefik middleware (default: stdio)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--root",
|
||||
default=None,
|
||||
help="The corpus to serve. Defaults to $CHEMENU_ROOT, then the checkout "
|
||||
"this package lives in.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
default="127.0.0.1",
|
||||
help="Interface to bind, streamable-http only. The loopback default is "
|
||||
"deliberate; a container behind a reverse proxy needs 0.0.0.0.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=8000,
|
||||
help="Port to bind, streamable-http only (default: 8000)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
# Passed only for the transport they mean anything to: `run_stdio_async`
|
||||
# takes no host or port, and handing it one is a TypeError rather than a
|
||||
# harmless no-op.
|
||||
bind = (
|
||||
{"host": args.host, "port": args.port}
|
||||
if args.transport == "streamable-http"
|
||||
else {}
|
||||
)
|
||||
|
||||
try:
|
||||
serve(transport=args.transport, root=args.root, **bind)
|
||||
except TraceWouldWriteIntoCorpus as exc:
|
||||
# Refused before binding anything: the message names both fixes, and a
|
||||
# server that silently relocated the operator's telemetry instead would
|
||||
# be a surprise buried in a log.
|
||||
print(f"ERROR {exc}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover - process entry point
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,234 @@
|
||||
"""The MCP read server: `search`, `types`, `lint` and `status` over `kb/`.
|
||||
|
||||
Chemenu's second consumer. The CLI and this are two adapters over one core -
|
||||
`chemenu.api.Corpus` - so a question answered here and the same question asked
|
||||
at a terminal go through the same code, and a golden test holds the two
|
||||
outputs against each other rather than trusting that they agree.
|
||||
|
||||
**There is no write path, structurally.** Nothing under `chemenu.commands` is
|
||||
imported here or in `chemenu.api`, so `new`, `touch`, `xref`, `cite`,
|
||||
`publish`, `migrate` and `version bump` are not reachable - the functions do
|
||||
not exist in this process's reach, rather than being filtered out of a list. A
|
||||
test asserts it by importing this module in a clean interpreter and looking at
|
||||
`sys.modules`.
|
||||
|
||||
**Authentication and rate limiting are not here.** Both are Traefik middleware
|
||||
in front of the process, per the operator's decision of 2026-09-01: a request
|
||||
that is not cleanly authenticated does not reach Python at all. What *is* here
|
||||
is the resource protection that middleware cannot give - the search timeout and
|
||||
the frontmatter limits - because those exist against an authenticated consumer
|
||||
damaging itself, which is a different problem from an unauthenticated one.
|
||||
|
||||
**The Iteration Budget Gate is deliberately absent.** It exists to stop an
|
||||
agent *session* from iterating unnoticed over the state of the wiki, which is
|
||||
why retrieval is exempt from it in the first place. A user who searches too
|
||||
often is a resource problem - different instrument, different purpose - and
|
||||
using the gate as a rate limiter would dilute it into one.
|
||||
|
||||
**Every response carries the commit it was computed from.** `chemenu.api`
|
||||
stamps `commit` and `as_of`; a stale checkout otherwise answers confidently and
|
||||
wrongly. Keeping the checkout current is a `git fetch && git reset --hard`
|
||||
poll outside this process - see `instructions/mcp-read-server.md` - which needs
|
||||
no inbound endpoint and no signature checking.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
from mcp.server.mcpserver.exceptions import ToolError
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.api import Corpus
|
||||
from chemenu.errors import ChemenuError
|
||||
|
||||
SERVER_NAME = "chemenu"
|
||||
|
||||
# Transports this server will start on. `stdio` is for developing and testing
|
||||
# it without a network; `streamable-http` is what a deployed instance speaks,
|
||||
# and the only one the Traefik middleware can sit in front of, because Traefik
|
||||
# is an HTTP reverse proxy. `sse` is reachable through the SDK but not offered:
|
||||
# it is the superseded remote transport, and building on it now only moves the
|
||||
# migration later.
|
||||
TRANSPORTS = ("stdio", "streamable-http")
|
||||
|
||||
|
||||
class TraceWouldWriteIntoCorpus(RuntimeError):
|
||||
"""Raised at startup when telemetry would land inside the served tree."""
|
||||
|
||||
|
||||
def check_trace_destination(root: Path) -> None:
|
||||
"""Refuse to start if a trace would be written into the corpus.
|
||||
|
||||
Telemetry defaults to *on* and writes under `reports/telemetry/` in the
|
||||
repo. Today nothing on this path emits - the writer is wired into
|
||||
`cli.main()` and the two gates, none of which run here - so this is a guard
|
||||
against the future rather than a fix for the present. It is worth having
|
||||
anyway: the sync that keeps this checkout current is `git reset --hard`, so
|
||||
a trace written into the tree is both a per-request write into a directory
|
||||
something else is entitled to wipe, and a silent way for the server to
|
||||
dirty the tree its own cache keys on.
|
||||
|
||||
Turn tracing off (`WIKI_TRACE=0`) or point it somewhere else
|
||||
(`WIKI_TRACE_DIR`). Refusing rather than correcting it: a server that
|
||||
quietly relocates the operator's telemetry is a surprise waiting in a log
|
||||
nobody reads.
|
||||
"""
|
||||
if os.environ.get("WIKI_TRACE", "1") == "0":
|
||||
return
|
||||
destination = os.environ.get("WIKI_TRACE_DIR")
|
||||
if destination is None:
|
||||
raise TraceWouldWriteIntoCorpus(
|
||||
"Telemetry is on and would write into the served checkout "
|
||||
f"({root / 'reports' / 'telemetry'}). The sync that keeps this checkout "
|
||||
"current is `git reset --hard`, which is entitled to wipe that directory. "
|
||||
"Set WIKI_TRACE=0, or point WIKI_TRACE_DIR outside the corpus."
|
||||
)
|
||||
resolved = Path(destination).expanduser().resolve()
|
||||
try:
|
||||
resolved.relative_to(Path(root).resolve())
|
||||
except ValueError:
|
||||
return
|
||||
raise TraceWouldWriteIntoCorpus(
|
||||
f"WIKI_TRACE_DIR ({resolved}) is inside the served checkout ({root}). "
|
||||
"Point it outside, or set WIKI_TRACE=0."
|
||||
)
|
||||
|
||||
|
||||
def build_server(
|
||||
root: Optional[Path | str] = None, check_trace: bool = True
|
||||
) -> MCPServer:
|
||||
"""Assemble the server over one corpus.
|
||||
|
||||
`root` follows `config.resolve_root()` - argument, then `$CHEMENU_ROOT`,
|
||||
then the checkout the package lives in - so a deployment points at its
|
||||
corpus with one environment variable and no code.
|
||||
"""
|
||||
corpus = Corpus(root)
|
||||
if check_trace:
|
||||
check_trace_destination(corpus.root)
|
||||
|
||||
server = MCPServer(
|
||||
name=SERVER_NAME,
|
||||
instructions=(
|
||||
"Read access to a Chemenu wiki: compiled, sourced knowledge under kb/. "
|
||||
"Every answer carries the commit it was computed from ('commit') and "
|
||||
"when it was produced ('as_of'); a null commit means the served tree "
|
||||
"has uncommitted changes and the answer corresponds to no revision. "
|
||||
"This server is read-only - there is no tool that writes."
|
||||
),
|
||||
)
|
||||
|
||||
@server.tool(
|
||||
name="search",
|
||||
description=(
|
||||
"Find pages in kb/ by text, by frontmatter, or by both. Returns "
|
||||
"title, path, kind, summary and confidence per hit, so a result can "
|
||||
"be judged without fetching the page. Prefer this over listing "
|
||||
"files: the answer is a few hundred tokens instead of a whole index."
|
||||
),
|
||||
)
|
||||
def search(
|
||||
query: str | None = None,
|
||||
predicates: list[str] | None = None,
|
||||
regex: bool = False,
|
||||
limit: int = 20,
|
||||
sort: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Search the wiki.
|
||||
|
||||
`predicates` are frontmatter filters in the CLI's own `--field` syntax,
|
||||
ANDed: `confidence<0.6`, `entity_type=system`, `tags~k8s`, `source_url:*`
|
||||
(present), `!source_url` (absent). With no `query` this is a pure
|
||||
structured query over frontmatter.
|
||||
|
||||
`regex` applies the pattern with ripgrep's linear engine. It is off by
|
||||
default, so an accidental `.*` is a literal.
|
||||
"""
|
||||
return _guard(
|
||||
lambda: corpus.search(
|
||||
text=query,
|
||||
predicates=predicates or (),
|
||||
regex=regex,
|
||||
limit=limit,
|
||||
sort=sort,
|
||||
)
|
||||
)
|
||||
|
||||
@server.tool(
|
||||
name="types",
|
||||
description=(
|
||||
"List the page types this wiki declares - what kinds of page exist, "
|
||||
"where each lives, and what its schema is. Read this before "
|
||||
"interpreting a page's `kind`."
|
||||
),
|
||||
)
|
||||
def types() -> dict[str, Any]:
|
||||
return _guard(corpus.types)
|
||||
|
||||
@server.tool(
|
||||
name="describe_type",
|
||||
description=(
|
||||
"One page type's full contract: its frontmatter fields with "
|
||||
"required/optional and any enums, its subtype field, and its "
|
||||
"authoring guidance."
|
||||
),
|
||||
)
|
||||
def describe_type(name: str) -> dict[str, Any]:
|
||||
"""`name` is a short type name as listed by `types`, e.g. 'entity'."""
|
||||
return _guard(lambda: corpus.describe_type(name))
|
||||
|
||||
@server.tool(
|
||||
name="lint",
|
||||
description=(
|
||||
"The wiki's structural health: broken wikilinks, orphan pages, "
|
||||
"index drift, schema gaps, provenance gaps. Findings only - the "
|
||||
"JSON form writes no report file."
|
||||
),
|
||||
)
|
||||
def lint() -> dict[str, Any]:
|
||||
return _guard(corpus.lint)
|
||||
|
||||
@server.tool(
|
||||
name="status",
|
||||
description=(
|
||||
"A snapshot: how many pages the wiki holds, how they split across "
|
||||
"collections, and how many findings of each kind lint reports. "
|
||||
"Cheaper to read than the full lint output."
|
||||
),
|
||||
)
|
||||
def status() -> dict[str, Any]:
|
||||
return _guard(corpus.status)
|
||||
|
||||
return server
|
||||
|
||||
|
||||
def _guard(call):
|
||||
"""Turn a `ChemenuError` into a plain message for the protocol layer.
|
||||
|
||||
A bad predicate is the caller's argument, not a server fault, and it should
|
||||
arrive as a tool error the model can act on, carrying the message that says
|
||||
what to do differently. The SDK draws exactly this line: a `ToolError` is a
|
||||
deliberate refusal and its text reaches the caller, while anything else is a
|
||||
crash whose text stays on the server. Only `ChemenuError` is caught -
|
||||
everything else is a genuine fault and belongs in the log, unswallowed.
|
||||
"""
|
||||
try:
|
||||
return call()
|
||||
except ChemenuError as exc:
|
||||
raise ToolError(str(exc)) from exc
|
||||
|
||||
|
||||
def serve(
|
||||
transport: str = "stdio",
|
||||
root: Optional[Path | str] = None,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
"""Start the server. `transport` is one of `TRANSPORTS`."""
|
||||
if transport not in TRANSPORTS:
|
||||
raise ValueError(
|
||||
f"unknown transport {transport!r}. Available: {', '.join(TRANSPORTS)}"
|
||||
)
|
||||
build_server(root).run(transport=transport, **kwargs)
|
||||
@@ -17,6 +17,14 @@ class Page:
|
||||
frontmatter: dict[str, Any] = field(default_factory=dict)
|
||||
body: str = ""
|
||||
|
||||
# Why the frontmatter above is empty, when it is empty for a reason. A page
|
||||
# whose YAML does not parse reads back as `{}`, and a `{}` page has no
|
||||
# `confidence` and no `kind`: it then drops out of `--field confidence<0.6`
|
||||
# - the query whose whole purpose is to find pages in bad shape - looking
|
||||
# exactly like a page that did not match. Loaders that know the reason put
|
||||
# it here so a caller can report the page instead of losing it.
|
||||
frontmatter_error: Optional[str] = None
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
"""The page's canonical title: the filename without extension.
|
||||
|
||||
@@ -14,6 +14,7 @@ from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.page import Page
|
||||
from chemenu.search.types import Predicate
|
||||
|
||||
@@ -26,7 +27,7 @@ _COMPARISON_OPS = (">=", "<=", ">", "<", "~", "=")
|
||||
VIRTUAL_FIELDS = ("title", "kind", "subtype", "collection")
|
||||
|
||||
|
||||
class PredicateError(ValueError):
|
||||
class PredicateError(ValidationError):
|
||||
"""Raised for a malformed `--field` argument or an unknown field name."""
|
||||
|
||||
|
||||
|
||||
@@ -7,28 +7,39 @@ the output shape. Selecting several at once fuses them through RRF.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Callable
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.search.base import SearchBackend
|
||||
from chemenu.search.ripgrep import RipgrepBackend
|
||||
|
||||
DEFAULT_BACKEND = "rg"
|
||||
ENV_VAR = "WIKITOOL_SEARCH_BACKEND"
|
||||
|
||||
BACKENDS: dict[str, Callable[[], SearchBackend]] = {
|
||||
BACKENDS: dict[str, Callable[..., SearchBackend]] = {
|
||||
"rg": RipgrepBackend,
|
||||
}
|
||||
|
||||
|
||||
class UnknownBackend(ValueError):
|
||||
class UnknownBackend(ValidationError):
|
||||
pass
|
||||
|
||||
|
||||
def resolve(spec: str | None = None) -> list[SearchBackend]:
|
||||
def resolve(
|
||||
spec: str | None = None,
|
||||
kb_dir: Optional[Path] = None,
|
||||
root: Optional[Path] = None,
|
||||
) -> list[SearchBackend]:
|
||||
"""Resolve a backend spec into instances.
|
||||
|
||||
Precedence: explicit argument, then `WIKITOOL_SEARCH_BACKEND`, then the
|
||||
default. A comma-separated spec selects several and fuses their rankings.
|
||||
|
||||
`kb_dir`/`root` are handed to the backend rather than left to its own
|
||||
defaults. Without them a caller could pass a corpus to `run_search` and
|
||||
still have the backend walk `config.KB_DIR` - the query answered from one
|
||||
tree and the pages read from another, with nothing saying so.
|
||||
"""
|
||||
raw = spec or os.environ.get(ENV_VAR) or DEFAULT_BACKEND
|
||||
names = [n.strip() for n in raw.split(",") if n.strip()]
|
||||
@@ -38,4 +49,4 @@ def resolve(spec: str | None = None) -> list[SearchBackend]:
|
||||
f"unknown search backend(s): {', '.join(unknown)}. "
|
||||
f"Available: {', '.join(sorted(BACKENDS))}"
|
||||
)
|
||||
return [BACKENDS[name]() for name in names]
|
||||
return [BACKENDS[name](kb_dir, root) for name in names]
|
||||
|
||||
@@ -4,7 +4,7 @@ Why shell out instead of scanning in Python: `rg` is already the retrieval
|
||||
layer the agent instructions point at, it handles large trees fast, and its
|
||||
`--json` mode gives line numbers and matched text without reparsing files.
|
||||
|
||||
Two safety properties are load-bearing and must survive any edit here:
|
||||
Three safety properties are load-bearing and must survive any edit here:
|
||||
|
||||
1. The query is passed as an *argv element*, never through a shell. There is
|
||||
no `shell=True` anywhere in this module, so a query containing `;`, `$(...)`
|
||||
@@ -12,16 +12,19 @@ Two safety properties are load-bearing and must survive any edit here:
|
||||
2. `--fixed-strings` is the default. A user-supplied regex is opt-in via
|
||||
`--regex`, so an accidental `.*` in a search term is a literal, and a
|
||||
pathological pattern cannot be introduced without asking for one.
|
||||
3. A user-supplied pattern is evaluated **only** by `rg`, whose engine is
|
||||
linear in the input. Nothing here hands it to Python's `re`, which
|
||||
backtracks - see `_contains`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.errors import BackendError
|
||||
from chemenu.page import Page
|
||||
from chemenu.search.base import page_key
|
||||
from chemenu.search.filters import collection_of
|
||||
@@ -41,13 +44,22 @@ TITLE_WEIGHT = 5.0
|
||||
SUMMARY_WEIGHT = 3.0
|
||||
LINE_WEIGHT = 1.0
|
||||
|
||||
# A hang-breaker, not a performance budget. A fixed-string search over this
|
||||
# corpus costs 6 ms and a deliberately broad regex 1.4 s, so nothing legitimate
|
||||
# comes near this; it exists so that a pathological pattern, a corpus on a
|
||||
# stalled network mount, or an `rg` that never returns fails as an error
|
||||
# instead of holding the caller open forever while its output buffers into the
|
||||
# heap. The caller sees the ordinary `RipgrepFailed` path.
|
||||
RIPGREP_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
class RipgrepMissing(RuntimeError):
|
||||
|
||||
class RipgrepMissing(BackendError):
|
||||
"""Raised when the `rg` executable is not on PATH."""
|
||||
|
||||
|
||||
class RipgrepFailed(RuntimeError):
|
||||
"""Raised when `rg` exits with an error (exit code 2 or above)."""
|
||||
class RipgrepFailed(BackendError):
|
||||
"""Raised when `rg` exits with an error (exit code 2 or above), or had to
|
||||
be killed for running past `RIPGREP_TIMEOUT_SECONDS`."""
|
||||
|
||||
|
||||
def build_argv(query: SearchQuery, root: Path) -> list[str]:
|
||||
@@ -99,7 +111,20 @@ class RipgrepBackend:
|
||||
|
||||
argv = build_argv(query, self.search_root)
|
||||
try:
|
||||
proc = subprocess.run(argv, capture_output=True, text=True, check=False)
|
||||
proc = subprocess.run(
|
||||
argv,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
timeout=RIPGREP_TIMEOUT_SECONDS,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise RipgrepFailed(
|
||||
f"rg did not finish within {RIPGREP_TIMEOUT_SECONDS:g}s and was killed. "
|
||||
"A search that takes this long is a pathological pattern or an "
|
||||
"unresponsive corpus directory, not a slow answer - narrow the query "
|
||||
"or use a fixed string instead of --regex."
|
||||
) from exc
|
||||
except FileNotFoundError as exc: # pragma: no cover - depends on host
|
||||
raise RipgrepMissing(
|
||||
"ripgrep (rg) is not installed or not on PATH. It is the search "
|
||||
@@ -141,13 +166,26 @@ class RipgrepBackend:
|
||||
|
||||
|
||||
def _contains(haystack: str, query: SearchQuery) -> bool:
|
||||
"""Literal containment, used only for the title and summary ranking boosts.
|
||||
|
||||
It never evaluates `query.text` as a regex, even when `query.regex` is set.
|
||||
It used to, via `re.search`, and Python's engine backtracks: `(\\w+\\s?)+$`
|
||||
against 114 characters of ordinary page text does not terminate in eight
|
||||
seconds, while a pattern that fails deterministically takes 0.2 ms - the
|
||||
difference is the pattern, not the haystack. `build_hit` calls this twice
|
||||
per hit, and a pattern as cheap as `\\w` matches every page, so one request
|
||||
bought two unbounded searches per page in the corpus.
|
||||
|
||||
Deleting the branch rather than bounding it is the right trade: `rg` has
|
||||
already applied the pattern with a linear engine by the time we get here,
|
||||
and the page is a hit *because* of that. What is lost is only the extra
|
||||
weight a regex hit in the title would have scored - and since a summary and
|
||||
an H1 are themselves lines in the file, `rg` still counts them. A pattern
|
||||
that is mostly literal (`longhorn`) still earns its boost through the test
|
||||
below; one that is not gets ranked by match count alone.
|
||||
"""
|
||||
if not query.text:
|
||||
return False
|
||||
if query.regex:
|
||||
try:
|
||||
return re.search(query.text, haystack, re.IGNORECASE) is not None
|
||||
except re.error:
|
||||
return False
|
||||
return query.text.lower() in haystack.lower()
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""The search core, with no CLI attached.
|
||||
|
||||
`run_search()` and the corpus loader used to live in `commands/search.py`,
|
||||
which imports `typer` at module level and `rich` through `_util`. Any
|
||||
in-process caller therefore dragged the whole CLI head in behind it - which
|
||||
made the "two third-party packages" the read core actually needs (`yaml`,
|
||||
`jsonschema`) an accounting fiction rather than a fact about the import graph.
|
||||
|
||||
Nothing here imports `typer`, `rich`, or anything under `chemenu.commands`.
|
||||
That is the boundary, and it is worth keeping: `commands/search.py` is now the
|
||||
adapter that turns these values into terminal output and these exceptions into
|
||||
exit codes, and the MCP server (Gitea #19) is a second adapter over the same
|
||||
functions rather than a second implementation of them.
|
||||
|
||||
Errors are raised, never printed: `PredicateError` for a bad `--field`,
|
||||
`RipgrepMissing`/`RipgrepFailed` for the backend. All of them are
|
||||
`ChemenuError` - see `chemenu/errors.py`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.frontmatter_io import read_page_with_error
|
||||
from chemenu.kb_scan import iter_kb_pages
|
||||
from chemenu.page import Page
|
||||
from chemenu.search import filters
|
||||
from chemenu.search.base import page_key
|
||||
from chemenu.search.fuse import reciprocal_rank_fusion
|
||||
from chemenu.search.ripgrep import build_hit
|
||||
from chemenu.search.types import SearchHit, SearchQuery
|
||||
|
||||
|
||||
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
|
||||
"""Every page under `kb/`, keyed by repo-relative path.
|
||||
|
||||
Path-keyed rather than title-keyed on purpose: `load_kb_pages()` drops one
|
||||
of two pages sharing a stem, and search should still find both - a
|
||||
duplicate title is a lint finding, not a reason to hide a page.
|
||||
"""
|
||||
kb_dir = kb_dir or config.KB_DIR
|
||||
root = root or config.ROOT
|
||||
pages: dict[str, Page] = {}
|
||||
for path in iter_kb_pages(kb_dir):
|
||||
frontmatter, body, error = read_page_with_error(path)
|
||||
pages[page_key(path, root)] = Page(
|
||||
path=path, frontmatter=frontmatter, body=body, frontmatter_error=error
|
||||
)
|
||||
return pages
|
||||
|
||||
|
||||
def unreadable_pages(pages: dict[str, Page]) -> list[dict[str, str]]:
|
||||
"""The pages whose frontmatter could not be used, as `{path, reason}`.
|
||||
|
||||
Reported rather than swallowed. Such a page has no `confidence` and no
|
||||
`kind`, so it silently drops out of every positive `--field` predicate -
|
||||
including the low-confidence sweep that exists to find pages in exactly
|
||||
that state. Saying nothing makes it look like a page that did not match;
|
||||
an empty block is excluded, because a page can legitimately carry one.
|
||||
"""
|
||||
return [
|
||||
{"path": key, "reason": page.frontmatter_error}
|
||||
for key, page in sorted(pages.items())
|
||||
if page.frontmatter_error and page.frontmatter_error != "empty frontmatter block"
|
||||
]
|
||||
|
||||
|
||||
def _sort_key(hit: SearchHit, field: str):
|
||||
value = hit.as_dict().get(field)
|
||||
if value is None:
|
||||
# Missing values sort last in either direction rather than crashing on
|
||||
# a None comparison.
|
||||
return (1, "")
|
||||
if isinstance(value, (int, float)):
|
||||
return (0, value)
|
||||
return (0, str(value).lower())
|
||||
|
||||
|
||||
def sort_hits(hits: list[SearchHit], sort: str | None) -> list[SearchHit]:
|
||||
"""Sort by a hit field. A leading `-` reverses, e.g. `--sort -confidence`."""
|
||||
if not sort:
|
||||
return hits
|
||||
descending = sort.startswith("-")
|
||||
field = sort.lstrip("-")
|
||||
ordered = sorted(hits, key=lambda h: _sort_key(h, field), reverse=descending)
|
||||
return ordered
|
||||
|
||||
|
||||
def run_search(
|
||||
query: SearchQuery,
|
||||
pages: dict[str, Page],
|
||||
backends: list,
|
||||
kb_dir: Path | None = None,
|
||||
) -> list[SearchHit]:
|
||||
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
|
||||
filters.validate_fields(query.predicates, pages)
|
||||
|
||||
if query.text:
|
||||
rankings = [backend.search(query, pages) for backend in backends]
|
||||
hits = rankings[0] if len(rankings) == 1 else reciprocal_rank_fusion(rankings)
|
||||
allowed = filters.apply_predicates(pages, query.predicates, kb_dir)
|
||||
hits = [hit for hit in hits if hit.path in allowed]
|
||||
else:
|
||||
selected = filters.apply_predicates(pages, query.predicates, kb_dir)
|
||||
hits = [
|
||||
build_hit(page, key, [], query, backend="frontmatter", kb_dir=kb_dir)
|
||||
for key, page in selected.items()
|
||||
]
|
||||
hits.sort(key=lambda h: h.title.lower())
|
||||
|
||||
hits = sort_hits(hits, query.sort)
|
||||
return hits[: query.limit] if query.limit else hits
|
||||
@@ -5,6 +5,7 @@ import pytest
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.frontmatter_io import write_page
|
||||
from chemenu.type_resolver import resolver
|
||||
|
||||
# Environment the tool reads for its own behaviour. Cleared for every test, so
|
||||
# that a test which needs one sets it itself and the rest run against the
|
||||
@@ -18,6 +19,7 @@ _WIKITOOL_ENV = (
|
||||
"WIKITOOL_SESSION_ID",
|
||||
"WIKITOOL_UPDATE_URL",
|
||||
"WIKITOOL_UPDATE_TOKEN",
|
||||
"CHEMENU_ROOT",
|
||||
)
|
||||
|
||||
# Environment git reads for identity or for where its repo lives. A stray
|
||||
@@ -36,7 +38,7 @@ _GIT_ENV = (
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
||||
"""Cut every test off from the machine it runs on.
|
||||
|
||||
The suite was green for months while silently depending on whoever ran it:
|
||||
@@ -68,7 +70,34 @@ def hermetic_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Pat
|
||||
|
||||
for name in (*_WIKITOOL_ENV, *_GIT_ENV):
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
return home
|
||||
|
||||
# The same hole as the environment above, one layer in: `config` resolves
|
||||
# its paths on access, and `monkeypatch.setattr(config, "KB_DIR", ...)`
|
||||
# undoes itself by writing the *resolved* old path back as a real
|
||||
# attribute. That binding outlives the test and hands the next one a
|
||||
# corpus directory belonging to the previous tree. Cleared on both sides,
|
||||
# so neither a leak from before nor one from this test can be inherited.
|
||||
config.reset()
|
||||
yield home
|
||||
config.reset()
|
||||
|
||||
|
||||
def use_shipped_type_specs(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Keep the shipped `types/` reachable for a test that repoints `ROOT`.
|
||||
|
||||
`TYPES_DIR` and the `TypeResolver`'s own root both follow `ROOT` now, which
|
||||
is the whole point of making the resolution lazy - but it means a fixture
|
||||
tree without a `types/` resolves no page kind at all, and a check that
|
||||
depends on a page being a `source` silently stops finding one. These tests
|
||||
do want the real schema: a synthetic type-spec would prove the command
|
||||
works against a fixture rather than against what it ships with.
|
||||
|
||||
So the dependency is declared instead of inherited. While both were bound at
|
||||
import time it held by accident, which is the same shape as the hole
|
||||
`raw_dir` was written to close, one layer down.
|
||||
"""
|
||||
monkeypatch.setattr(config, "TYPES_DIR", config._PACKAGE_ROOT / "types")
|
||||
monkeypatch.setattr(resolver, "_repo_root", config._PACKAGE_ROOT)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -113,6 +142,7 @@ def raw_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
same way - in the fixture, not in the one test that happened to trip.
|
||||
"""
|
||||
monkeypatch.setattr(config, "ROOT", tmp_path)
|
||||
use_shipped_type_specs(monkeypatch)
|
||||
raw = tmp_path / "raw"
|
||||
(raw / "notes").mkdir(parents=True)
|
||||
(raw / "notes" / "Aurora.md").write_text("# Aurora raw notes\n", encoding="utf-8")
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Tests for the in-process library boundary (Gitea #31).
|
||||
|
||||
Two properties, and neither is about the values coming back:
|
||||
|
||||
1. A caller can point Chemenu at a corpus tree and **no path of the checkout
|
||||
this package lives in is read**. That is what "library" means here, and it
|
||||
is what could not be asserted before: `config.KB_DIR` was bound at import
|
||||
time, so a caller repointing `ROOT` was still answered out of the developer's
|
||||
own `kb/`.
|
||||
2. The surface is read-only **structurally**. `chemenu.api` imports nothing
|
||||
under `chemenu.commands`, so `new`, `publish` and the rest are not reachable
|
||||
from it - not filtered out of it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.api import Corpus
|
||||
from chemenu.errors import ChemenuError, ValidationError
|
||||
from chemenu.types_core import UnknownType
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def foreign_corpus(tmp_path: Path) -> Path:
|
||||
"""A corpus tree that is not this checkout, with one findable page."""
|
||||
root = tmp_path / "elsewhere"
|
||||
kb = root / "kb" / "entities"
|
||||
kb.mkdir(parents=True)
|
||||
(root / "kb" / "entities" / "COLLECTION.md").write_text("# entities\n", encoding="utf-8")
|
||||
(kb / "Peregrine.md").write_text(
|
||||
"---\ntype: types/entity.md\nentity_type: system\nconfidence: 0.42\n"
|
||||
"summary: A system that exists only in this fixture.\n---\n\n"
|
||||
"# Peregrine\n\nPeregrine is the fixture's own system.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
def test_a_corpus_can_be_named_and_is_the_one_that_answers(foreign_corpus):
|
||||
result = Corpus(foreign_corpus).search("Peregrine")
|
||||
assert [hit["title"] for hit in result["results"]] == ["Peregrine"]
|
||||
assert result["results"][0]["path"] == "kb/entities/Peregrine.md"
|
||||
|
||||
|
||||
def test_no_path_of_this_checkout_is_read_while_a_foreign_root_is_set(foreign_corpus):
|
||||
"""The acceptance criterion, asserted rather than argued.
|
||||
|
||||
`Path.read_text` and `Path.rglob` are the two ways a page reaches the
|
||||
reader; both are watched, and any access under the real repository root
|
||||
fails the test. Before the root resolution was made lazy this test could not
|
||||
pass: `config.KB_DIR` was already bound to this checkout's `kb/`.
|
||||
"""
|
||||
package_root = config._PACKAGE_ROOT
|
||||
trespasses: list[str] = []
|
||||
|
||||
real_read_text = Path.read_text
|
||||
real_rglob = Path.rglob
|
||||
|
||||
def watched_read_text(self, *args, **kwargs):
|
||||
_note(self)
|
||||
return real_read_text(self, *args, **kwargs)
|
||||
|
||||
def watched_rglob(self, *args, **kwargs):
|
||||
_note(self)
|
||||
return real_rglob(self, *args, **kwargs)
|
||||
|
||||
def _note(path: Path) -> None:
|
||||
try:
|
||||
path.resolve().relative_to(package_root)
|
||||
except ValueError:
|
||||
return
|
||||
trespasses.append(str(path))
|
||||
|
||||
monkey = pytest.MonkeyPatch()
|
||||
monkey.setattr(Path, "read_text", watched_read_text)
|
||||
monkey.setattr(Path, "rglob", watched_rglob)
|
||||
try:
|
||||
Corpus(foreign_corpus).search("Peregrine")
|
||||
Corpus(foreign_corpus).search(predicates=["confidence<0.6"])
|
||||
finally:
|
||||
monkey.undo()
|
||||
|
||||
assert trespasses == []
|
||||
|
||||
|
||||
def test_the_env_var_points_the_default_corpus(foreign_corpus, monkeypatch):
|
||||
"""`CHEMENU_ROOT` exists for a caller that *is* a whole process and has
|
||||
nothing to pass an argument through."""
|
||||
monkeypatch.setenv(config.ENV_ROOT, str(foreign_corpus))
|
||||
assert Corpus().root == foreign_corpus.resolve()
|
||||
assert Corpus().kb_dir == foreign_corpus.resolve() / "kb"
|
||||
|
||||
|
||||
def test_without_the_env_var_the_root_is_this_checkout(monkeypatch):
|
||||
"""The default has to be unchanged, or `tools/wikitool` moves under
|
||||
everyone's feet."""
|
||||
monkeypatch.delenv(config.ENV_ROOT, raising=False)
|
||||
assert config.resolve_root() == config._PACKAGE_ROOT
|
||||
|
||||
|
||||
def test_an_explicit_argument_beats_the_env_var(foreign_corpus, tmp_path, monkeypatch):
|
||||
"""A caller serving two corpora cannot tell them apart with a process-wide
|
||||
variable, so the argument has to win."""
|
||||
monkeypatch.setenv(config.ENV_ROOT, str(tmp_path / "somewhere-else"))
|
||||
assert Corpus(foreign_corpus).root == foreign_corpus.resolve()
|
||||
|
||||
|
||||
def test_derived_paths_follow_the_root_instead_of_lagging_behind(foreign_corpus, monkeypatch):
|
||||
"""The failure that made the old shape worse than the limitation: `ROOT`
|
||||
moved and `KB_DIR` did not, so a caller believed it was working on the
|
||||
target tree while reading this one."""
|
||||
monkeypatch.setattr(config, "ROOT", foreign_corpus)
|
||||
assert config.KB_DIR == foreign_corpus / "kb"
|
||||
assert config.RAW_DIR == foreign_corpus / "raw"
|
||||
assert config.INDEX_FILE == foreign_corpus / "kb" / "index.md"
|
||||
|
||||
|
||||
def test_validation_errors_are_raised_not_exited(foreign_corpus):
|
||||
corpus = Corpus(foreign_corpus)
|
||||
with pytest.raises(ValidationError):
|
||||
corpus.search()
|
||||
with pytest.raises(ValidationError):
|
||||
corpus.search("x", predicates=["not a predicate"])
|
||||
with pytest.raises(UnknownType):
|
||||
corpus.describe_type("no-such-type")
|
||||
# One class to catch, whatever went wrong.
|
||||
with pytest.raises(ChemenuError):
|
||||
corpus.describe_type("no-such-type")
|
||||
|
||||
|
||||
def test_the_read_surface_cannot_reach_a_write_command():
|
||||
"""Structural, not filtered: import `chemenu.api` in a clean interpreter and
|
||||
nothing under `chemenu.commands` is loaded, so there is no `publish` to
|
||||
call. Run out-of-process because this suite has already imported the CLI."""
|
||||
code = (
|
||||
"import sys, chemenu.api;"
|
||||
"print([m for m in sys.modules if m.startswith('chemenu.commands')]);"
|
||||
"print([m for m in sys.modules if m in ('typer', 'rich', 'click')])"
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", code],
|
||||
cwd=config._PACKAGE_ROOT / "tools",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
commands_loaded, cli_loaded = result.stdout.strip().splitlines()
|
||||
assert commands_loaded == "[]", f"api pulled in command modules: {commands_loaded}"
|
||||
assert cli_loaded == "[]", f"api pulled in the CLI head: {cli_loaded}"
|
||||
|
||||
|
||||
def test_every_answer_carries_the_revision_it_was_computed_from(foreign_corpus):
|
||||
"""A stale checkout answers confidently and wrongly otherwise. Outside git
|
||||
there is no commit, and the stamp says so rather than inventing one."""
|
||||
corpus = Corpus(foreign_corpus)
|
||||
result = corpus.search("Peregrine")
|
||||
assert result["commit"] is None and result["as_of"]
|
||||
|
||||
subprocess.run(["git", "init", "-b", "main"], cwd=foreign_corpus, check=True,
|
||||
capture_output=True)
|
||||
for key, value in (("user.name", "Fixture"), ("user.email", "f@example.com")):
|
||||
subprocess.run(["git", "config", key, value], cwd=foreign_corpus, check=True,
|
||||
capture_output=True)
|
||||
subprocess.run(["git", "add", "-A"], cwd=foreign_corpus, check=True, capture_output=True)
|
||||
subprocess.run(["git", "commit", "-m", "corpus"], cwd=foreign_corpus, check=True,
|
||||
capture_output=True)
|
||||
|
||||
stamped = Corpus(foreign_corpus).search("Peregrine")
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=foreign_corpus,
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
assert stamped["commit"] == head
|
||||
|
||||
|
||||
def test_lint_and_status_answer_from_the_named_corpus(foreign_corpus):
|
||||
corpus = Corpus(foreign_corpus)
|
||||
assert corpus.lint()["page_count"] == 1
|
||||
status = corpus.status()
|
||||
assert status["pages"] == 1
|
||||
assert status["collections"] == {"entities": 1}
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Tests for the revision-keyed corpus cache (Gitea #33, feeding #19).
|
||||
|
||||
The property under test is not "it is fast" but "it is never stale". A cache on
|
||||
this path that answers from a superseded parse is worse than the reparse it
|
||||
replaces: the caller gets a confident answer about a corpus that no longer
|
||||
exists, which is the failure `SOUL.md` names as the cardinal one.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from chemenu import config, corpus_cache
|
||||
from chemenu.corpus_cache import CorpusCache
|
||||
|
||||
|
||||
def _git(root: Path, *args: str) -> None:
|
||||
subprocess.run(["git", *args], cwd=root, check=True, capture_output=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(kb_dir: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
"""The `kb_dir` fixture tree, committed - so there is a revision to key on."""
|
||||
_git(tmp_path, "init", "-b", "main")
|
||||
_git(tmp_path, "config", "user.name", "Fixture Author")
|
||||
_git(tmp_path, "config", "user.email", "fixture@example.com")
|
||||
_git(tmp_path, "add", "-A")
|
||||
_git(tmp_path, "commit", "-m", "fixture corpus")
|
||||
monkeypatch.setattr(config, "ROOT", tmp_path)
|
||||
monkeypatch.setattr(config, "KB_DIR", kb_dir)
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_a_clean_tree_is_cached_and_reused(repo, kb_dir):
|
||||
cache = CorpusCache(kb_dir, repo)
|
||||
first, revision = cache.load()
|
||||
second, again = cache.load()
|
||||
assert revision is not None and again == revision
|
||||
# Identity, not equality: an equal-but-rebuilt dict would mean the reparse
|
||||
# still happened, which is the whole cost this exists to remove.
|
||||
assert first is second
|
||||
|
||||
|
||||
def test_a_new_commit_invalidates_the_cache(repo, kb_dir):
|
||||
cache = CorpusCache(kb_dir, repo)
|
||||
first, first_revision = cache.load()
|
||||
|
||||
(kb_dir / "entities" / "Latecomer.md").write_text(
|
||||
"---\ntype: types/entity.md\n---\n\n# Latecomer\n", encoding="utf-8"
|
||||
)
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-m", "one more page")
|
||||
|
||||
second, second_revision = cache.load()
|
||||
assert second_revision != first_revision
|
||||
assert len(second) == len(first) + 1
|
||||
|
||||
|
||||
def test_a_dirty_tree_is_never_cached(repo, kb_dir):
|
||||
"""The correctness case. A session that writes a page and then searches for
|
||||
it must not be answered from the parse taken before the write - and nothing
|
||||
about the commit SHA changed in between."""
|
||||
cache = CorpusCache(kb_dir, repo)
|
||||
first, revision = cache.load()
|
||||
assert revision is not None
|
||||
|
||||
(kb_dir / "entities" / "Uncommitted.md").write_text(
|
||||
"---\ntype: types/entity.md\n---\n\n# Uncommitted\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
second, dirty_revision = cache.load()
|
||||
assert dirty_revision is None
|
||||
assert len(second) == len(first) + 1
|
||||
assert cache.revision is None
|
||||
|
||||
|
||||
def test_a_tree_outside_git_reloads_every_time(kb_dir, tmp_path):
|
||||
cache = CorpusCache(kb_dir, tmp_path)
|
||||
pages, revision = cache.load()
|
||||
assert revision is None and pages
|
||||
assert cache.load()[0] is not pages
|
||||
|
||||
|
||||
def test_an_unanswerable_git_counts_as_dirty(repo, kb_dir, monkeypatch):
|
||||
"""Errs toward reloading: a cache that reads "unknown" as "clean" serves
|
||||
stale pages, which is the one outcome this module exists to prevent."""
|
||||
monkeypatch.setattr(corpus_cache, "_git", lambda *a, **k: None)
|
||||
assert corpus_cache.is_dirty(repo, kb_dir) is True
|
||||
assert CorpusCache(kb_dir, repo).load()[1] is None
|
||||
|
||||
|
||||
def test_head_commit_is_the_revision_a_response_gets_stamped_with(repo):
|
||||
expected = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"], cwd=repo, capture_output=True, text=True, check=True
|
||||
).stdout.strip()
|
||||
assert corpus_cache.head_commit(repo) == expected
|
||||
@@ -237,6 +237,43 @@ def test_a_migration_document_satisfies_the_check(tmp_path, monkeypatch):
|
||||
assert docs_verify.check_migration_for_boundary() == []
|
||||
|
||||
|
||||
def test_a_breaking_release_without_a_breaking_note_is_reported(tmp_path, monkeypatch):
|
||||
"""A crossing that migrates nothing still leaves hand-work behind, so the
|
||||
migration check passing is not evidence that anyone was told."""
|
||||
from chemenu import version as version_mod
|
||||
|
||||
_boundary_tree(
|
||||
tmp_path, monkeypatch, "2.0.0", "1.4.0",
|
||||
marker=f"{version_mod.MIGRATION_NONE_MARKER} - nothing to change.\n\n",
|
||||
)
|
||||
assert docs_verify.check_migration_for_boundary() == []
|
||||
issues = docs_verify.check_breaking_change_for_boundary()
|
||||
assert any("2.0.0" in issue and "drop-in" in issue for issue in issues)
|
||||
|
||||
|
||||
def test_a_compatible_release_needs_no_breaking_note(tmp_path, monkeypatch):
|
||||
_boundary_tree(tmp_path, monkeypatch, "1.5.0", "1.4.0")
|
||||
assert docs_verify.check_breaking_change_for_boundary() == []
|
||||
|
||||
|
||||
def test_a_breaking_change_marker_satisfies_the_check(tmp_path, monkeypatch):
|
||||
from chemenu import version as version_mod
|
||||
|
||||
_boundary_tree(
|
||||
tmp_path, monkeypatch, "2.0.0", "1.4.0",
|
||||
marker=f"{version_mod.BREAKING_CHANGE_MARKER} the feed moved.\n\n",
|
||||
)
|
||||
assert docs_verify.check_breaking_change_for_boundary() == []
|
||||
|
||||
|
||||
def test_verify_raises_when_a_boundary_has_no_breaking_note(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
docs_verify, "check_breaking_change_for_boundary", lambda: ["unannounced"]
|
||||
)
|
||||
with pytest.raises(typer.Exit):
|
||||
docs_verify.verify()
|
||||
|
||||
|
||||
def test_verify_raises_when_a_boundary_has_no_migration(monkeypatch):
|
||||
monkeypatch.setattr(docs_verify, "check_migration_for_boundary", lambda: ["unbridged"])
|
||||
with pytest.raises(typer.Exit):
|
||||
|
||||
@@ -220,6 +220,40 @@ def test_a_renamed_but_unfilled_environment_template_warns(instance):
|
||||
assert "template" in next(c.detail for c in checks if c.name == "environment")
|
||||
|
||||
|
||||
def _remotes_detail(checks) -> str:
|
||||
return next(c.detail for c in checks if c.name == "publish-remotes")
|
||||
|
||||
|
||||
def test_publish_remotes_says_not_armed_when_the_file_is_absent(instance):
|
||||
"""Absence stays OK - a single-remote checkout has nothing to protect - but
|
||||
the line has to say the gate is off. AGENTS.md lists it among the limits
|
||||
enforced in code, so "no .wikitool-remotes.json" alone leaves a reader
|
||||
trusting a safeguard that is not running."""
|
||||
checks = doctor.run_doctor()
|
||||
assert _status(checks, "publish-remotes") == "OK"
|
||||
assert "not armed" in _remotes_detail(checks)
|
||||
|
||||
|
||||
def test_publish_remotes_says_armed_when_the_file_lists_a_target(instance):
|
||||
(config.ROOT / config.PUBLISH_REMOTES_FILENAME).write_text(
|
||||
'{ "schema": 1, "allowed_push_urls": ["ssh://git@example.net/one.git"] }',
|
||||
encoding="utf-8",
|
||||
)
|
||||
checks = doctor.run_doctor()
|
||||
assert _status(checks, "publish-remotes") == "OK"
|
||||
detail = _remotes_detail(checks)
|
||||
assert "armed" in detail and "not armed" not in detail
|
||||
|
||||
|
||||
def test_publish_remotes_warns_on_several_remotes_without_an_allowlist(instance):
|
||||
"""The shape a private instance has once it adds the public upstream."""
|
||||
_git(config.ROOT, "remote", "add", "origin", "ssh://git@example.net/mine.git")
|
||||
_git(config.ROOT, "remote", "add", "upstream", "ssh://git@example.net/theirs.git")
|
||||
checks = doctor.run_doctor()
|
||||
assert _status(checks, "publish-remotes") == "WARN"
|
||||
assert "not armed" in _remotes_detail(checks)
|
||||
|
||||
|
||||
def test_missing_generated_file_fails(instance):
|
||||
config.LOG_FILE.unlink()
|
||||
checks = doctor.run_doctor()
|
||||
|
||||
@@ -1,4 +1,17 @@
|
||||
from chemenu.frontmatter_io import dump_frontmatter, read_page, write_page
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from chemenu.frontmatter_io import (
|
||||
MAX_FRONTMATTER_BYTES,
|
||||
FrontmatterError,
|
||||
_Loader,
|
||||
dump_frontmatter,
|
||||
frontmatter_error,
|
||||
read_page,
|
||||
read_page_strict,
|
||||
read_page_with_error,
|
||||
write_page,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -82,3 +95,91 @@ def test_scalar_quoting_is_unchanged_by_the_flow_fix(tmp_path):
|
||||
assert dump_frontmatter({"year": "1945"}) == "year: '1945'"
|
||||
assert dump_frontmatter({"summary": "He said hi"}) == "summary: He said hi"
|
||||
assert dump_frontmatter({"confidence": 0.85}) == "confidence: 0.85"
|
||||
|
||||
|
||||
# --- Read-path limits (Gitea #33) -------------------------------------------
|
||||
#
|
||||
# These are regressions against parser properties, not against a caller. Each
|
||||
# one is reproducible on an unpatched tree, and each one becomes reachable the
|
||||
# moment frontmatter this instance did not write itself reaches the parser.
|
||||
|
||||
# 267 bytes that compose into 672,603 nodes on traversal, parsed in 0.2 ms:
|
||||
# every level references the one below it nine times, so the object graph grows
|
||||
# 9**n while the input stays tiny and the parse time stays constant. A size
|
||||
# limit does not touch this, which is why there is a second check.
|
||||
ALIAS_BOMB = """\
|
||||
a: &a ["x", "x", "x", "x", "x", "x", "x", "x", "x"]
|
||||
b: &b [*a, *a, *a, *a, *a, *a, *a, *a, *a]
|
||||
c: &c [*b, *b, *b, *b, *b, *b, *b, *b, *b]
|
||||
d: &d [*c, *c, *c, *c, *c, *c, *c, *c, *c]
|
||||
e: &e [*d, *d, *d, *d, *d, *d, *d, *d, *d]
|
||||
f: &f [*e, *e, *e, *e, *e, *e, *e, *e, *e]
|
||||
"""
|
||||
|
||||
|
||||
def _page(tmp_path, frontmatter_text):
|
||||
path = tmp_path / "Bomb.md"
|
||||
path.write_text(f"---\n{frontmatter_text}---\n\n# Bomb\n", encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def test_an_alias_bomb_is_refused_rather_than_expanded(tmp_path):
|
||||
path = _page(tmp_path, ALIAS_BOMB)
|
||||
frontmatter, _, error = read_page_with_error(path)
|
||||
assert frontmatter == {}
|
||||
assert "alias" in error
|
||||
|
||||
|
||||
def test_the_alias_check_costs_nothing_on_a_page_that_has_none(tmp_path):
|
||||
"""`*` is a necessary character in an alias node, so its absence proves
|
||||
absence - which is the path every real page takes."""
|
||||
path = _page(tmp_path, "type: types/entity.md\ntags: [k8s, storage]\n")
|
||||
frontmatter, _, error = read_page_with_error(path)
|
||||
assert error is None
|
||||
assert frontmatter["tags"] == ["k8s", "storage"]
|
||||
|
||||
|
||||
def test_oversized_frontmatter_is_refused(tmp_path):
|
||||
path = _page(tmp_path, "summary: " + ("x" * (MAX_FRONTMATTER_BYTES + 1)) + "\n")
|
||||
_, _, error = read_page_with_error(path)
|
||||
assert "over the" in error and "limit" in error
|
||||
|
||||
|
||||
def test_a_page_at_the_size_limit_still_parses(tmp_path):
|
||||
filler = "x" * (MAX_FRONTMATTER_BYTES - len("summary: \n"))
|
||||
path = _page(tmp_path, f"summary: {filler}\n")
|
||||
frontmatter, _, error = read_page_with_error(path)
|
||||
assert error is None and frontmatter["summary"] == filler
|
||||
|
||||
|
||||
def test_the_permissive_read_hands_back_the_reason_it_was_permissive(tmp_path):
|
||||
path = _page(tmp_path, "type: [unclosed\n")
|
||||
frontmatter, _, error = read_page_with_error(path)
|
||||
assert frontmatter == {}
|
||||
assert error and "invalid YAML" in error
|
||||
# The historical behaviour - the reason dropped on the floor - is what let
|
||||
# a broken page slip past every --field predicate looking like a miss.
|
||||
assert read_page(path) == ({}, "\n# Bomb\n")
|
||||
|
||||
|
||||
def test_the_strict_read_raises_instead_of_emptying(tmp_path):
|
||||
path = _page(tmp_path, "type: [unclosed\n")
|
||||
with pytest.raises(FrontmatterError):
|
||||
read_page_strict(path)
|
||||
|
||||
|
||||
def test_frontmatter_error_and_the_permissive_read_agree(tmp_path):
|
||||
"""One parser behind both, so they cannot describe the same file
|
||||
differently - and a caller wanting both answers reads the file once."""
|
||||
for text in (ALIAS_BOMB, "type: [unclosed\n", "type: types/entity.md\n", "\n"):
|
||||
path = _page(tmp_path, text)
|
||||
assert frontmatter_error(path) == read_page_with_error(path)[2]
|
||||
|
||||
|
||||
def test_the_fast_loader_is_used_when_libyaml_is_available():
|
||||
"""The corpus parse is the largest single cost of a search and scales with
|
||||
the corpus, so the factor-3 loader is not micro-tuning."""
|
||||
if hasattr(yaml, "CSafeLoader"):
|
||||
assert _Loader is yaml.CSafeLoader
|
||||
else: # pragma: no cover - depends on the host
|
||||
assert _Loader is yaml.SafeLoader
|
||||
|
||||
@@ -11,6 +11,7 @@ import pytest
|
||||
import typer
|
||||
|
||||
from chemenu import config
|
||||
from chemenu.tests.conftest import use_shipped_type_specs
|
||||
from chemenu.commands import instructions_cmd
|
||||
|
||||
|
||||
@@ -37,6 +38,10 @@ def layer(tmp_path: Path, monkeypatch):
|
||||
monkeypatch.setattr(config, "INSTRUCTIONS_DIR", instructions)
|
||||
monkeypatch.setattr(config, "AGENTS_SKILLS_DIR", root / ".agents" / "skills")
|
||||
monkeypatch.setattr(config, "CLAUDE_SKILLS_DIR", root / ".claude" / "skills")
|
||||
# `verify` resolves each instruction's `type: types/instruction.md`, and the
|
||||
# resolver's root follows `ROOT` now - so the shipped specs have to be named
|
||||
# rather than inherited. See `use_shipped_type_specs`.
|
||||
use_shipped_type_specs(monkeypatch)
|
||||
return root
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
"""Tests for the MCP read server (Gitea #19).
|
||||
|
||||
The acceptance criteria that are not about a value coming back:
|
||||
|
||||
- the wire format **is** the CLI's `--json` form, held together by a golden
|
||||
test rather than by intention;
|
||||
- no path of the server writes into `kb/`, `reports/` or git;
|
||||
- there is no write tool, because nothing under `chemenu.commands` is
|
||||
importable from it - structural, not filtered;
|
||||
- every response carries the commit it was computed from;
|
||||
- telemetry cannot land inside the served checkout.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from chemenu import config
|
||||
|
||||
pytest.importorskip("mcp", reason="the MCP server's dependency is optional; see "
|
||||
"tools/requirements-mcp.txt")
|
||||
|
||||
from chemenu.mcp.server import ( # noqa: E402 - after the skip guard
|
||||
TRANSPORTS,
|
||||
TraceWouldWriteIntoCorpus,
|
||||
build_server,
|
||||
check_trace_destination,
|
||||
serve,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def corpus(tmp_path: Path) -> Path:
|
||||
"""A committed instance that is not this checkout.
|
||||
|
||||
Carries its own `types/`, because a served corpus is a whole instance: the
|
||||
page kinds and `describe_type` are answered from the instance's own schema,
|
||||
not from whichever checkout the package happens to sit in. Copied rather
|
||||
than linked - a link would let a test that writes to `<root>/types/...`
|
||||
write into this repository, which is not hypothetical.
|
||||
"""
|
||||
root = tmp_path / "served"
|
||||
entities = root / "kb" / "entities"
|
||||
entities.mkdir(parents=True)
|
||||
shutil.copytree(config._PACKAGE_ROOT / "types", root / "types")
|
||||
(root / "kb" / "entities" / "COLLECTION.md").write_text("# entities\n", encoding="utf-8")
|
||||
(entities / "Kingfisher.md").write_text(
|
||||
"---\ntype: types/entity.md\nentity_type: system\nconfidence: 0.55\n"
|
||||
"summary: The fixture's own system.\n---\n\n"
|
||||
"# Kingfisher\n\nKingfisher is the system this fixture is about.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
subprocess.run(["git", "init", "-b", "main"], cwd=root, check=True, capture_output=True)
|
||||
for key, value in (("user.name", "Fixture"), ("user.email", "f@example.com")):
|
||||
subprocess.run(["git", "config", key, value], cwd=root, check=True, capture_output=True)
|
||||
subprocess.run(["git", "add", "-A"], cwd=root, check=True, capture_output=True)
|
||||
subprocess.run(["git", "commit", "-m", "corpus"], cwd=root, check=True, capture_output=True)
|
||||
return root
|
||||
|
||||
|
||||
def _call(server, name: str, arguments: dict | None = None) -> dict:
|
||||
result = asyncio.run(server.call_tool(name, arguments or {}))
|
||||
if isinstance(result, tuple):
|
||||
result = result[1]
|
||||
assert not result.is_error, result.content
|
||||
return result.structured_content
|
||||
|
||||
|
||||
def _tree(root: Path) -> dict[str, tuple[int, bytes]]:
|
||||
"""Every file under `root` with its size and contents, for a before/after
|
||||
comparison that catches a rewrite with the same length."""
|
||||
return {
|
||||
str(path.relative_to(root)): (path.stat().st_size, path.read_bytes())
|
||||
for path in sorted(root.rglob("*"))
|
||||
if path.is_file()
|
||||
}
|
||||
|
||||
|
||||
def test_the_four_tools_are_there_and_nothing_that_writes(corpus):
|
||||
server = build_server(corpus, check_trace=False)
|
||||
names = {tool.name for tool in asyncio.run(server.list_tools())}
|
||||
assert names == {"search", "types", "describe_type", "lint", "status"}
|
||||
# Named rather than pattern-matched: the point is that these are absent
|
||||
# because the functions are unreachable, and a test that only looked for
|
||||
# "no tool called publish" would pass on a filtered list too.
|
||||
assert names.isdisjoint({"new", "touch", "xref", "cite", "publish", "migrate", "rm"})
|
||||
|
||||
|
||||
def test_no_tool_writes_anything_into_the_corpus_or_git(corpus):
|
||||
server = build_server(corpus, check_trace=False)
|
||||
before = _tree(corpus)
|
||||
head_before = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
|
||||
).stdout
|
||||
|
||||
_call(server, "search", {"query": "Kingfisher"})
|
||||
_call(server, "search", {"predicates": ["confidence<0.6"]})
|
||||
_call(server, "types")
|
||||
_call(server, "describe_type", {"name": "entity"})
|
||||
_call(server, "lint")
|
||||
_call(server, "status")
|
||||
|
||||
assert _tree(corpus) == before
|
||||
head_after = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
|
||||
).stdout
|
||||
assert head_after == head_before
|
||||
# `lint` without `--json` writes a report into `reports/`; the server must
|
||||
# only ever reach the JSON form.
|
||||
assert not (corpus / "reports").exists()
|
||||
assert subprocess.run(
|
||||
["git", "status", "--porcelain"], cwd=corpus, capture_output=True, text=True, check=True
|
||||
).stdout == ""
|
||||
|
||||
|
||||
def test_the_wire_format_is_the_clis_json_form(corpus):
|
||||
"""The golden test. One contract, with the CLI as its executable
|
||||
specification - so the two cannot drift while both look correct.
|
||||
|
||||
The CLI is run as a subprocess against the same tree, pointed at it with
|
||||
`CHEMENU_ROOT` - which is also an end-to-end check that the root resolution
|
||||
works from outside the process.
|
||||
"""
|
||||
server = build_server(corpus, check_trace=False)
|
||||
served = _call(server, "search", {"query": "Kingfisher", "limit": 5})
|
||||
|
||||
import os
|
||||
|
||||
environment = {**os.environ, config.ENV_ROOT: str(corpus), "WIKI_TRACE": "0"}
|
||||
completed = subprocess.run(
|
||||
[str(config._PACKAGE_ROOT / "tools" / "wikitool"),
|
||||
"search", "Kingfisher", "--limit", "5", "--json"],
|
||||
capture_output=True, text=True, check=True, env=environment,
|
||||
)
|
||||
from_cli = json.loads(completed.stdout)
|
||||
|
||||
# `generated` is the CLI's date stamp and `commit`/`as_of` are the server's
|
||||
# revision stamp - two answers to "when", neither of them a finding. What
|
||||
# has to match is everything that describes the *corpus*.
|
||||
shared = ("query", "predicates", "backend", "count", "results", "unreadable")
|
||||
assert {key: served[key] for key in shared} == {key: from_cli[key] for key in shared}
|
||||
|
||||
|
||||
def test_types_and_describe_match_the_cli_too(corpus):
|
||||
import os
|
||||
|
||||
server = build_server(corpus, check_trace=False)
|
||||
environment = {**os.environ, config.ENV_ROOT: str(corpus), "WIKI_TRACE": "0"}
|
||||
wikitool = str(config._PACKAGE_ROOT / "tools" / "wikitool")
|
||||
|
||||
from_cli = json.loads(subprocess.run(
|
||||
[wikitool, "types", "list", "--json"],
|
||||
capture_output=True, text=True, check=True, env=environment,
|
||||
).stdout)
|
||||
assert _call(server, "types")["types"] == from_cli
|
||||
|
||||
from_cli = json.loads(subprocess.run(
|
||||
[wikitool, "types", "describe", "entity", "--json"],
|
||||
capture_output=True, text=True, check=True, env=environment,
|
||||
).stdout)
|
||||
served = _call(server, "describe_type", {"name": "entity"})
|
||||
assert {k: v for k, v in served.items() if k not in ("commit", "as_of")} == from_cli
|
||||
|
||||
|
||||
def test_every_response_carries_the_commit_it_was_computed_from(corpus):
|
||||
server = build_server(corpus, check_trace=False)
|
||||
head = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"], cwd=corpus, capture_output=True, text=True, check=True
|
||||
).stdout.strip()
|
||||
for name in ("types", "lint", "status"):
|
||||
answer = _call(server, name)
|
||||
assert answer["commit"] == head
|
||||
assert answer["as_of"]
|
||||
|
||||
|
||||
def test_a_dirty_tree_is_stamped_with_no_commit_rather_than_a_wrong_one(corpus):
|
||||
"""The answer no longer corresponds to any revision, and says so instead of
|
||||
naming the commit it is no longer about."""
|
||||
(corpus / "kb" / "entities" / "Latecomer.md").write_text(
|
||||
"---\ntype: types/entity.md\n---\n\n# Latecomer\n", encoding="utf-8"
|
||||
)
|
||||
assert _call(build_server(corpus, check_trace=False), "status")["commit"] is None
|
||||
|
||||
|
||||
def test_a_bad_predicate_comes_back_as_a_tool_error_not_a_traceback(corpus):
|
||||
"""The SDK draws the line this depends on: a `ToolError` is a deliberate
|
||||
refusal whose text reaches the caller, while any other exception is a crash
|
||||
whose text stays on the server. A bad predicate is the caller's argument, so
|
||||
the message that says what to write instead has to travel."""
|
||||
from mcp.server.mcpserver.exceptions import ToolError, UnexpectedToolError
|
||||
|
||||
server = build_server(corpus, check_trace=False)
|
||||
with pytest.raises(ToolError) as excinfo:
|
||||
asyncio.run(server.call_tool("search", {"predicates": ["not a predicate"]}))
|
||||
assert not isinstance(excinfo.value, UnexpectedToolError)
|
||||
assert "field=value" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_the_server_module_cannot_reach_a_write_command():
|
||||
"""Structural: import the server in a clean interpreter and nothing under
|
||||
`chemenu.commands` is loaded, so there is no `publish` to expose."""
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c",
|
||||
"import sys, chemenu.mcp.server;"
|
||||
"print([m for m in sys.modules if m.startswith('chemenu.commands')])"],
|
||||
cwd=config._PACKAGE_ROOT / "tools", capture_output=True, text=True, check=True,
|
||||
)
|
||||
assert result.stdout.strip() == "[]"
|
||||
|
||||
|
||||
def test_tracing_into_the_served_checkout_is_refused(corpus, monkeypatch):
|
||||
"""The sync that keeps this checkout current is `git reset --hard`, which is
|
||||
entitled to wipe `reports/`. A per-request trace written there is both lost
|
||||
work and a silent way to dirty the tree the cache keys on."""
|
||||
monkeypatch.setenv("WIKI_TRACE", "1")
|
||||
monkeypatch.delenv("WIKI_TRACE_DIR", raising=False)
|
||||
with pytest.raises(TraceWouldWriteIntoCorpus):
|
||||
check_trace_destination(corpus)
|
||||
|
||||
monkeypatch.setenv("WIKI_TRACE_DIR", str(corpus / "reports" / "telemetry"))
|
||||
with pytest.raises(TraceWouldWriteIntoCorpus):
|
||||
check_trace_destination(corpus)
|
||||
|
||||
|
||||
def test_tracing_outside_the_corpus_or_switched_off_is_accepted(corpus, tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("WIKI_TRACE", "0")
|
||||
check_trace_destination(corpus)
|
||||
|
||||
monkeypatch.setenv("WIKI_TRACE", "1")
|
||||
monkeypatch.setenv("WIKI_TRACE_DIR", str(tmp_path / "traces"))
|
||||
check_trace_destination(corpus)
|
||||
|
||||
|
||||
def test_only_the_two_chosen_transports_are_offered():
|
||||
"""`sse` is reachable through the SDK and deliberately not offered: it is
|
||||
the superseded remote transport, and building on it now only moves the
|
||||
migration later."""
|
||||
assert TRANSPORTS == ("stdio", "streamable-http")
|
||||
with pytest.raises(ValueError, match="unknown transport"):
|
||||
serve(transport="sse")
|
||||
|
||||
|
||||
def test_bind_arguments_reach_only_the_transport_that_takes_them(monkeypatch):
|
||||
"""`run_stdio_async` accepts no host or port; handing it one is a TypeError,
|
||||
not a harmless no-op. And the loopback default has to be overridable, or a
|
||||
server in a container behind a reverse proxy binds an interface the proxy
|
||||
cannot reach."""
|
||||
from chemenu.mcp import __main__ as entry
|
||||
|
||||
seen: dict = {}
|
||||
monkeypatch.setattr(entry, "serve", lambda **kwargs: seen.update(kwargs))
|
||||
|
||||
entry.main([])
|
||||
assert seen == {"transport": "stdio", "root": None}
|
||||
|
||||
seen.clear()
|
||||
entry.main(["--transport", "streamable-http", "--host", "0.0.0.0", "--port", "9001"])
|
||||
assert seen["transport"] == "streamable-http"
|
||||
assert seen["host"] == "0.0.0.0" and seen["port"] == 9001
|
||||
|
||||
|
||||
def test_a_refused_trace_destination_stops_the_process_with_a_message(monkeypatch, capsys):
|
||||
from chemenu.mcp import __main__ as entry
|
||||
|
||||
monkeypatch.setenv("WIKI_TRACE", "1")
|
||||
monkeypatch.delenv("WIKI_TRACE_DIR", raising=False)
|
||||
assert entry.main([]) == 1
|
||||
assert "WIKI_TRACE" in capsys.readouterr().err
|
||||
@@ -160,7 +160,10 @@ def _fixture_raw_file(monkeypatch, kb_dir, relative: str) -> None:
|
||||
distribution does not have."""
|
||||
import chemenu.config as config
|
||||
|
||||
from chemenu.tests.conftest import use_shipped_type_specs
|
||||
|
||||
monkeypatch.setattr(config, "ROOT", kb_dir.parent)
|
||||
use_shipped_type_specs(monkeypatch)
|
||||
path = kb_dir.parent / relative
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("raw fixture content\n", encoding="utf-8")
|
||||
|
||||
@@ -1,9 +1,18 @@
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from chemenu.commands.search import load_pages_by_path, render_table, run_search, sort_hits
|
||||
from chemenu.search import filters
|
||||
from chemenu.commands.search import (
|
||||
load_pages_by_path,
|
||||
render_table,
|
||||
run_search,
|
||||
sort_hits,
|
||||
unreadable_pages,
|
||||
)
|
||||
from chemenu.search import filters, ripgrep
|
||||
from chemenu.search.base import page_key
|
||||
from chemenu.search.filters import PredicateError, parse_predicate
|
||||
from chemenu.search.fuse import reciprocal_rank_fusion
|
||||
from chemenu.search.registry import UnknownBackend, resolve
|
||||
@@ -238,3 +247,85 @@ def test_known_fields_includes_virtual_and_real(pages):
|
||||
fields = filters.known_fields(pages)
|
||||
assert {"title", "kind", "subtype", "collection"} <= fields
|
||||
assert {"entity_type", "confidence", "tags"} <= fields
|
||||
|
||||
|
||||
# --- Read-path limits (Gitea #33) -------------------------------------------
|
||||
|
||||
|
||||
def test_a_user_regex_never_reaches_pythons_backtracking_engine():
|
||||
"""Regression for the ReDoS. `(\\w+\\s?)+$` against 114 characters of
|
||||
ordinary page text does not terminate in eight seconds under `re`; the
|
||||
ranking helper must not evaluate it as a pattern at all.
|
||||
|
||||
Asserted by time *and* by outcome: a bound alone would pass if the branch
|
||||
came back with a cheaper engine, and the outcome alone would pass while the
|
||||
call still hung on a different pattern."""
|
||||
haystack = (
|
||||
"Longhorn is the distributed block storage layer for Kubernetes that "
|
||||
"this cluster runs, replicated across three nodes and backed up nightly"
|
||||
)
|
||||
query = SearchQuery(text=r"(\w+\s?)+$", regex=True)
|
||||
started = time.perf_counter()
|
||||
assert ripgrep._contains(haystack, query) is False
|
||||
assert time.perf_counter() - started < 0.5
|
||||
|
||||
|
||||
def test_a_mostly_literal_regex_still_earns_its_title_boost():
|
||||
"""What the deleted branch cost, and what it did not: the common case of a
|
||||
pattern that happens to be plain text keeps ranking as before."""
|
||||
assert ripgrep._contains("Longhorn", SearchQuery(text="longhorn", regex=True)) is True
|
||||
|
||||
|
||||
def test_ripgrep_is_called_with_a_timeout(monkeypatch, kb_dir, tmp_path):
|
||||
seen = {}
|
||||
|
||||
def fake_run(argv, **kwargs):
|
||||
seen.update(kwargs)
|
||||
return subprocess.CompletedProcess(argv, 1, "", "")
|
||||
|
||||
monkeypatch.setattr(ripgrep.subprocess, "run", fake_run)
|
||||
RipgrepBackend(kb_dir, tmp_path).search(SearchQuery(text="x"), {})
|
||||
assert seen["timeout"] == ripgrep.RIPGREP_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def test_a_hanging_ripgrep_is_reported_as_a_failure_not_a_hang(monkeypatch, kb_dir, tmp_path):
|
||||
def fake_run(argv, **kwargs):
|
||||
raise subprocess.TimeoutExpired(argv, kwargs["timeout"])
|
||||
|
||||
monkeypatch.setattr(ripgrep.subprocess, "run", fake_run)
|
||||
with pytest.raises(ripgrep.RipgrepFailed) as excinfo:
|
||||
RipgrepBackend(kb_dir, tmp_path).search(SearchQuery(text="x"), {})
|
||||
assert "did not finish" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_a_page_with_broken_frontmatter_is_reported_not_lost(kb_dir, tmp_path):
|
||||
"""It matches no positive predicate - including the low-confidence sweep
|
||||
meant to find pages in exactly that state - so silence reads as 'did not
|
||||
match'. The page has to be nameable."""
|
||||
broken = kb_dir / "entities" / "Broken.md"
|
||||
broken.write_text("---\ntype: [unclosed\n---\n\n# Broken\n", encoding="utf-8")
|
||||
pages = load_pages_by_path(kb_dir, tmp_path)
|
||||
key = page_key(broken, tmp_path)
|
||||
|
||||
assert pages[key].frontmatter == {}
|
||||
assert filters.apply_predicates(pages, (parse_predicate("confidence<0.6"),)) .get(key) is None
|
||||
|
||||
reported = unreadable_pages(pages)
|
||||
assert [entry["path"] for entry in reported] == [key]
|
||||
assert "invalid YAML" in reported[0]["reason"]
|
||||
|
||||
|
||||
def test_an_empty_frontmatter_block_is_not_reported_as_unreadable(kb_dir, tmp_path):
|
||||
"""A page may legitimately carry an empty block - there are no fields to
|
||||
lose, so there is nothing the caller was not told about. `lint` still has
|
||||
an opinion about it; search does not."""
|
||||
(kb_dir / "entities" / "Bare.md").write_text("---\n\n---\n\n# Bare\n", encoding="utf-8")
|
||||
assert unreadable_pages(load_pages_by_path(kb_dir, tmp_path)) == []
|
||||
|
||||
|
||||
def test_a_page_with_no_frontmatter_at_all_is_reported(kb_dir, tmp_path):
|
||||
"""Unlike an empty block, this page has no `type:` either - it cannot match
|
||||
a predicate, and nothing else would say so."""
|
||||
(kb_dir / "entities" / "Naked.md").write_text("# Naked\n\nProse only.\n", encoding="utf-8")
|
||||
reported = unreadable_pages(load_pages_by_path(kb_dir, tmp_path))
|
||||
assert [entry["path"] for entry in reported] == ["kb/entities/Naked.md"]
|
||||
|
||||
@@ -183,7 +183,7 @@ def test_insert_changes_entry_lands_above_the_newest_entry():
|
||||
def test_bump_writes_both_the_version_and_the_changelog_heading(tree):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=True, patch=False, title="Something happened",
|
||||
no_migration=None, dry_run=False,
|
||||
breaking=None, no_migration=None, dry_run=False,
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.1.0"
|
||||
changes = (tree / "CHANGES.md").read_text(encoding="utf-8")
|
||||
@@ -194,7 +194,7 @@ def test_bump_writes_both_the_version_and_the_changelog_heading(tree):
|
||||
|
||||
def test_bump_dry_run_writes_nothing(tree):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=False, patch=True, title="Nope", no_migration=None, dry_run=True
|
||||
major=False, minor=False, patch=True, title="Nope", breaking=None, no_migration=None, dry_run=True
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
assert "1.0.1" not in (tree / "CHANGES.md").read_text(encoding="utf-8")
|
||||
@@ -207,7 +207,7 @@ def test_bump_demands_exactly_one_part(tree, flags):
|
||||
major, minor, patch = flags
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=major, minor=minor, patch=patch, title="x", no_migration=None, dry_run=False
|
||||
major=major, minor=minor, patch=patch, title="x", breaking=None, no_migration=None, dry_run=False
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
|
||||
@@ -215,7 +215,7 @@ def test_bump_demands_exactly_one_part(tree, flags):
|
||||
def test_bump_refuses_an_empty_title(tree):
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=False, patch=True, title=" ", no_migration=None, dry_run=False
|
||||
major=False, minor=False, patch=True, title=" ", breaking=None, no_migration=None, dry_run=False
|
||||
)
|
||||
|
||||
|
||||
@@ -227,7 +227,7 @@ def test_bump_refuses_when_the_changelog_is_already_ahead(tree):
|
||||
)
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=False, patch=True, title="x", no_migration=None, dry_run=False
|
||||
major=False, minor=False, patch=True, title="x", breaking=None, no_migration=None, dry_run=False
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
|
||||
@@ -241,7 +241,7 @@ def test_a_boundary_crossing_bump_without_a_migration_is_refused(tree):
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=True, minor=False, patch=False, title="Breaking",
|
||||
no_migration=None, dry_run=False,
|
||||
breaking="the feed moved", no_migration=None, dry_run=False,
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
|
||||
@@ -255,7 +255,8 @@ def test_a_boundary_crossing_bump_passes_with_a_migration_document(tree):
|
||||
encoding="utf-8",
|
||||
)
|
||||
version_cmd.bump_command(
|
||||
major=True, minor=False, patch=False, title="Breaking", no_migration=None, dry_run=False
|
||||
major=True, minor=False, patch=False, title="Breaking",
|
||||
breaking="every page is retyped", no_migration=None, dry_run=False,
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "2.0.0"
|
||||
|
||||
@@ -263,6 +264,7 @@ def test_a_boundary_crossing_bump_passes_with_a_migration_document(tree):
|
||||
def test_no_migration_records_the_reason_in_the_changelog(tree):
|
||||
version_cmd.bump_command(
|
||||
major=True, minor=False, patch=False, title="Breaking",
|
||||
breaking="the release feed moved",
|
||||
no_migration="no distributed instance exists yet", dry_run=False,
|
||||
)
|
||||
changes = (tree / "CHANGES.md").read_text(encoding="utf-8")
|
||||
@@ -275,10 +277,50 @@ def test_no_migration_is_refused_on_a_compatible_bump(tree):
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=False, patch=True, title="Fix",
|
||||
no_migration="not needed", dry_run=False,
|
||||
breaking=None, no_migration="not needed", dry_run=False,
|
||||
)
|
||||
|
||||
|
||||
# --- version bump: the breaking-change note --------------------------------
|
||||
|
||||
|
||||
def test_a_boundary_crossing_bump_without_breaking_is_refused(tree):
|
||||
"""The corpus question and the drop-in question are independent: a bump
|
||||
can migrate nothing and still leave every instance with hand-work."""
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=True, minor=False, patch=False, title="Renamed the feed",
|
||||
breaking=None, no_migration="kb/ keeps its shape", dry_run=False,
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
|
||||
|
||||
def test_breaking_records_what_stops_working_in_the_changelog(tree):
|
||||
version_cmd.bump_command(
|
||||
major=True, minor=False, patch=False, title="Renamed the feed",
|
||||
breaking="update_url points at a repo path that no longer exists",
|
||||
no_migration="kb/ keeps its shape", dry_run=False,
|
||||
)
|
||||
changes = (tree / "CHANGES.md").read_text(encoding="utf-8")
|
||||
assert version_mod.BREAKING_CHANGE_MARKER in changes
|
||||
assert "update_url points at a repo path that no longer exists" in changes
|
||||
# The break comes before the migration note: it is what an operator acts on.
|
||||
assert changes.index(version_mod.BREAKING_CHANGE_MARKER) < changes.index(
|
||||
version_mod.MIGRATION_NONE_MARKER
|
||||
)
|
||||
|
||||
|
||||
def test_breaking_is_refused_on_a_compatible_bump(tree):
|
||||
"""A compatible bump that claims a break is describing itself wrongly -
|
||||
one of the two is a mistake, and the command will not guess which."""
|
||||
with pytest.raises(typer.Exit):
|
||||
version_cmd.bump_command(
|
||||
major=False, minor=True, patch=False, title="New command",
|
||||
breaking="nothing, really", no_migration=None, dry_run=False,
|
||||
)
|
||||
assert (tree / "VERSION").read_text(encoding="utf-8").strip() == "1.0.0"
|
||||
|
||||
|
||||
# --- version notes ---------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -26,11 +26,19 @@ class TypeResolver:
|
||||
"""Resolves and validates type paths against type-spec files."""
|
||||
|
||||
def __init__(self, repo_root: Path = None):
|
||||
self.repo_root = repo_root or config.ROOT
|
||||
# None means "whatever the root currently resolves to" - see the
|
||||
# `repo_root` property. Binding it here is what made the module-level
|
||||
# `resolver` singleton answer about this checkout even when the caller
|
||||
# had pointed everything else at another tree.
|
||||
self._repo_root = Path(repo_root) if repo_root is not None else None
|
||||
self.type_cache: Dict[str, Dict[str, Any]] = {}
|
||||
self.schema_cache: Dict[str, Dict[str, Any]] = {}
|
||||
self.validator_cache: Dict[str, Any] = {}
|
||||
|
||||
@property
|
||||
def repo_root(self) -> Path:
|
||||
return self._repo_root if self._repo_root is not None else config.ROOT
|
||||
|
||||
def resolve_type_path(self, type_path: str, source_file: Path = None) -> Path:
|
||||
"""Resolve a type path to an absolute, validated path.
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""The type-spec surface, with no CLI attached.
|
||||
|
||||
Same split as `chemenu/search/service.py` and `chemenu/lint_core.py`: these two
|
||||
functions produce the values, `commands/types_cmd.py` renders them and turns
|
||||
`UnknownType` into an `ERROR` line and exit 1.
|
||||
|
||||
They return exactly the structures the CLI's `--json` forms print, because that
|
||||
is the wire contract the MCP server (Gitea #19) is specified against - one
|
||||
contract, not two, with the CLI as its executable specification.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.type_resolver import resolver
|
||||
|
||||
|
||||
class UnknownType(ValidationError):
|
||||
"""No type-spec by that name. Carries the available names, because an
|
||||
unknown type is almost always a near-miss and the list is the answer."""
|
||||
|
||||
def __init__(self, name: str, available: list[str]):
|
||||
self.name = name
|
||||
self.available = available
|
||||
super().__init__(f"No type-spec named '{name}'. Available: {', '.join(available)}")
|
||||
|
||||
|
||||
def available_type_names() -> list[str]:
|
||||
return sorted(fm.get("name") for _, fm in resolver.list_type_specs())
|
||||
|
||||
|
||||
def list_types() -> list[Dict[str, Any]]:
|
||||
"""Every type-spec under `types/`, in the shape `types list --json` prints."""
|
||||
return [
|
||||
{
|
||||
"name": frontmatter.get("name"),
|
||||
"type_path": type_path,
|
||||
"schema": frontmatter.get("schema"),
|
||||
"subtype_field": frontmatter.get("subtype_field"),
|
||||
"root": frontmatter.get("root") or "kb",
|
||||
"base_dir": frontmatter.get("base_dir"),
|
||||
"description": frontmatter.get("description"),
|
||||
}
|
||||
for type_path, frontmatter in resolver.list_type_specs()
|
||||
]
|
||||
|
||||
|
||||
def describe_type(name: str) -> Dict[str, Any]:
|
||||
"""One type's full contract, in the shape `types describe --json` prints.
|
||||
|
||||
Raises `UnknownType` rather than exiting: the caller may be a server, for
|
||||
which an exit code is not an answer.
|
||||
"""
|
||||
type_path = resolver.find_type_by_name(name)
|
||||
if type_path is None:
|
||||
raise UnknownType(name, available_type_names())
|
||||
|
||||
type_spec = resolver.load_type_spec(type_path)
|
||||
frontmatter = type_spec["frontmatter"]
|
||||
schema = resolver.get_schema(type_path)
|
||||
|
||||
fields: list[Dict[str, Any]] = []
|
||||
if schema is not None:
|
||||
required = set(schema.get("required", []))
|
||||
for field_name, field_schema in schema.get("properties", {}).items():
|
||||
fields.append({
|
||||
"field": field_name,
|
||||
"required": field_name in required,
|
||||
"type": field_schema.get("type"),
|
||||
"enum": field_schema.get("enum"),
|
||||
})
|
||||
|
||||
return {
|
||||
"name": frontmatter.get("name"),
|
||||
"type_path": type_path,
|
||||
"description": frontmatter.get("description"),
|
||||
"schema": frontmatter.get("schema"),
|
||||
"subtype_field": frontmatter.get("subtype_field"),
|
||||
# `root` was previously read straight from the frontmatter by the
|
||||
# renderer and left out of the JSON payload, so `describe --json` could
|
||||
# not tell you where a type's pages go while `list --json` could -
|
||||
# `types/instruction.md` declares `root: repo`. Carried here, in the
|
||||
# same shape `list` uses.
|
||||
"root": frontmatter.get("root") or "kb",
|
||||
"base_dir": frontmatter.get("base_dir"),
|
||||
"title_prefix": frontmatter.get("title_prefix"),
|
||||
"fields": fields,
|
||||
"body": type_spec["body"].strip(),
|
||||
}
|
||||
@@ -10,11 +10,23 @@ ingest into a release.
|
||||
|
||||
**Compatibility is read off the leftmost non-zero component**, the rule Cargo's
|
||||
caret ranges use: `0.1.3 -> 0.1.4` is safe, `0.1.3 -> 0.2.0` is not, and from
|
||||
`1.0.0` on the same rule reads as the familiar "MAJOR means migration". Stating
|
||||
it that way is what lets the 0.x era carry the migration signal at all - under
|
||||
plain "MAJOR breaks" semantics every 0.x release would be indistinguishable
|
||||
from every other, which is exactly the signal update detection needs. Nothing
|
||||
about the mechanism changes at 1.0.0.
|
||||
`1.0.0` on the same rule reads as the familiar "MAJOR breaks". Stating it that
|
||||
way is what lets the 0.x era carry the signal at all - under a rule keyed to
|
||||
the MAJOR component alone, every 0.x release would be indistinguishable from
|
||||
every other, which is exactly the signal update detection needs. Nothing about
|
||||
the mechanism changes at 1.0.0.
|
||||
|
||||
What that component answers is **whether the new version is a drop-in
|
||||
replacement**: whether an instance can copy the new machinery over itself with
|
||||
no hand-work and still put the old version back afterwards. Whether *content*
|
||||
must be migrated is a **second, independent question**. It is one way to fail
|
||||
the first - but a renamed release feed, artefact, import name, flag or envvar
|
||||
fails it too, with `kb/` untouched, which is why `--no-migration` exists at all:
|
||||
boundary-crossing bumps that migrate nothing are a real case, not an escape
|
||||
hatch. Hence two markers below rather than one - `BREAKING_CHANGE_MARKER`
|
||||
records the break, `MIGRATION_NONE_MARKER` records the absence of the
|
||||
migration. Which part a change earns stays a judgment call made before the
|
||||
bump; this module only enforces that a crossing says what it costs.
|
||||
|
||||
Paths are resolved through `config.ROOT` at call time rather than at import,
|
||||
because the tests (and `dist export`'s own fixtures) relocate the root.
|
||||
@@ -64,6 +76,11 @@ _SEMVER_RE = re.compile(r"^\s*v?(\d+)\.(\d+)\.(\d+)\s*$")
|
||||
# boundary that needs no content migration. `docs verify` accepts it in place
|
||||
# of a migration document, so the exact string is a contract between the two.
|
||||
MIGRATION_NONE_MARKER = "**Migration:** none required"
|
||||
# Written into every CHANGES.md entry whose version crosses a compatibility
|
||||
# boundary, migration or not: the swap is not drop-in, and the operator of an
|
||||
# existing instance has to be told what stops working. `docs verify` checks the
|
||||
# newest crossing carries it, so this string too is a contract between the two.
|
||||
BREAKING_CHANGE_MARKER = "**Breaking Change:**"
|
||||
# A changelog entry that names a version. Entries predating versioning start
|
||||
# with a date instead and are deliberately not matched - they are history, not
|
||||
# a claim about which version the tree is.
|
||||
@@ -332,15 +349,22 @@ def insert_changes_entry(
|
||||
title: str,
|
||||
author: str,
|
||||
no_migration_reason: Optional[str] = None,
|
||||
breaking_reason: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Add a heading for `version` above the newest existing entry.
|
||||
|
||||
Only the skeleton: heading, date, author, and - when a compatibility
|
||||
boundary is crossed without a migration - the line that says so. The
|
||||
entry's actual content is written afterwards by whoever made the change,
|
||||
which is also why `bump` refuses to invent a title.
|
||||
boundary is crossed - the line saying what breaks, plus the line saying no
|
||||
content has to change where that applies. The entry's actual content is
|
||||
written afterwards by whoever made the change, which is also why `bump`
|
||||
refuses to invent a title.
|
||||
|
||||
The break comes first: it is what an operator reading the release notes has
|
||||
to act on, and the migration line only qualifies it.
|
||||
"""
|
||||
lines = [f"## {version} - {date} - {title}", "", f"**Author:** {author}", ""]
|
||||
if breaking_reason:
|
||||
lines += [f"{BREAKING_CHANGE_MARKER} {breaking_reason}", ""]
|
||||
if no_migration_reason:
|
||||
lines += [f"{MIGRATION_NONE_MARKER} - {no_migration_reason}", ""]
|
||||
entry = "\n".join(lines) + "\n---\n\n"
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
# The MCP read server's dependency, deliberately not in requirements.txt.
|
||||
#
|
||||
# `requirements.txt` describes what an *instance* needs to run `wikitool`, and
|
||||
# it is what `dist export` ships. The server is an optional second consumer:
|
||||
# an instance that only ever uses the CLI should not be made to install
|
||||
# pydantic, starlette, uvicorn and cryptography to do it.
|
||||
#
|
||||
# tools/.venv/bin/pip install -r tools/requirements-mcp.txt
|
||||
# WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp --transport stdio
|
||||
#
|
||||
# The tests under `chemenu/tests/test_mcp_server.py` skip without it; CI
|
||||
# installs it, so they do run.
|
||||
mcp>=2.0
|
||||
Reference in New Issue
Block a user