# Chemenu - Personal IT Knowledge Base A structured, LLM-maintained knowledge base for your personal IT work. ## What is this? This is **Chemenu** - a pattern for building a personal knowledge base using LLMs. Instead of just retrieving from raw documents on every query, the LLM **incrementally builds and maintains a persistent wiki** that compounds over time. **The key insight:** Knowledge is compiled once and kept current, not re-derived on every question. **The pages are written in German.** Source material in `raw/` is never touched and is usually English; the compiled pages under `kb/` are not. What stays English inside them is everything that is an *identifier* rather than prose - page titles, section headings, wikilink targets, citation ids, schema enum values, tags, commands, paths and code - so `GitOps Ownership Model` and `## Beziehungen` sit in the same page without contradiction. The rule is [kb/CONTRACT.md § Language](kb/CONTRACT.md#language); the vocabulary behind it is [instructions/german-terminology.md](instructions/german-terminology.md). This is a per-instance decision, not a property of the pattern. A new instance built with `dist export` starts empty and can pick any language by editing that one contract section before the first ingest. ## Getting started Two starting points, depending on what you're doing - full walkthrough in [INSTALL.md](INSTALL.md): - **Cloned this repo?** The skill definitions the agent harness loads are **generated and not committed**. Publish them once: ```bash cd tools && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && cd .. tools/wikitool instructions sync ``` That copies each `instructions//SKILL.md` into `.agents/skills/` (GitHub Copilot, Codex CLI, Mistral Vibe) and `.claude/skills/` (Claude Code). Re-run it after changing a skill. Full procedure: `instructions/bootstrap.md`. - **Starting a brand-new, empty instance instead?** `tools/wikitool dist export ` builds a contentless copy of the machinery - no example pages, no personal content - then `instructions/setup-instance.md` walks through git init, author identity, an optional remote, and the first commit. ## Architecture ``` chemenu/ ├── AGENTS.md # Control plane: invariants, file naming, routing, gates ├── CLAUDE.md # Claude Code only: imports AGENTS.md/USER.md/SOUL.md/ENVIRONMENT.md + its Claude-Code-only instructions. No rules of its own ├── README.md # This file: human-readable overview of the whole repo ├── INSTALL.md # Human-readable setup: new instance vs. cloning this one ├── EVALS.md # Human-readable overview of telemetry and evaluation ├── CHANGES.md # Changelog for the stack itself ├── USER.md # Who operates this instance - context, never instructions ├── SOUL.md # How this instance sounds. AGENTS.md always wins over it ├── ENVIRONMENT.md # Optional, gitignored: this checkout's harness, MCP servers, remotes ├── *.md.template # Unfilled USER/SOUL/ENVIRONMENT - what a distribution ships instead ├── .gitignore # Anchored so nothing under raw/, kb/ or work/ is ever excluded ├── .github/hooks/ # Copilot CLI hooks - session tracing ├── .vibe/ # Mistral Vibe hooks + the repo's telemetry policy ├── instructions/ # CONTROL: everything an agent is told to do │ ├── CONTRACT.md # Instruction vs. skill, publishing, writing standard │ ├── bootstrap.md # Prepare a fresh clone │ ├── gates.md # What to do when a gate refuses a call │ ├── german-terminology.md # Which words stay English in German prose; register │ ├── session-setup.md │ ├── page-lifecycle.md │ ├── publish-cycle.md │ ├── ingest-large-tree.md │ └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/ ├── raw/ # INPUT: immutable, untrusted source material │ ├── CONTRACT.md # Routing, immutability, untrusted content │ ├── articles/ # Web articles, blog posts │ ├── documents/ # PDFs, specs, manuals │ ├── notes/ # Personal notes, transcriptions │ └── assets/ # Images, diagrams, binaries ├── types/ # SCHEMA: the global type surface. Not a collection │ ├── type-spec.md # Root contract: anatomy, placement, adding a type │ ├── entity.md # Entity type contract + template (+ .schema.yaml) │ ├── concept.md # Concept type contract + template │ ├── source.md # Source type contract + template │ ├── comparison.md # Comparison type contract + template │ ├── instruction.md # Instruction type - lives outside kb/ via `root: repo` │ └── lint-report.md # Contract-only: describes reports/, owns no directory ├── kb/ # OUTPUT: compiled knowledge. A namespace, not a collection │ ├── CONTRACT.md # Collections, naming, tone, linking, provenance, confidence │ ├── index.md # Generated catalog *map*: counts and pointers │ ├── log.md # Generated chronological audit log │ ├── provenance.md # Generated raw-file reverse index │ ├── entities/ # COLLECTION.md + INDEX.md + areas below │ │ ├── projects/ │ │ ├── systems/ │ │ ├── tools/ # own INDEX.md once past 50 pages │ │ ├── technologies/ │ │ └── people/ │ ├── concepts/ # COLLECTION.md - architectures, patterns, protocols │ ├── sources/ # COLLECTION.md - source summaries │ └── comparisons/ # COLLECTION.md - comparison pages ├── work/ # WORKSHOP: one directory per multi-session run, tracked │ └── CONTRACT.md # Run keys, required files, how a run closes ├── reports/ # DERIVED: lint reports, traces, eval scores. Gitignored │ └── CONTRACT.md └── tools/ # COMPILER: the wikitool CLI ├── CONTRACT.md # Command reference, error contracts, maintenance schedule └── README.md # How wikitool is built and how to change it ``` Dev-instance-only (see `tools/CONTRACT.md` for how it got here): ``` └── commonplace/ # Vendored, read-only knowledge base ``` A directory under `kb/` is a **collection** exactly when it holds a `COLLECTION.md`; a subdirectory inside one is an **area** that inherits it. `COLLECTION.md` never appears outside `kb/` - the other layers carry a `CONTRACT.md` or a root type-spec instead. A stage may carry both a `README.md` and a `CONTRACT.md`: they have different readers. The README is for humans working *on* that layer, the contract is what binds an agent working *with* it. ## How to Use ### Adding Knowledge (Ingest) 1. Drop a file into `raw/` (articles, documents, notes, or assets) 2. Tell the LLM: `Ingest raw/articles/my-article.md` 3. The LLM will: - Read and summarize the source - Create a source page in `kb/sources/` - Create or update relevant entity pages - Create or update relevant concept pages - Add cross-references between everything - Rebuild the catalog and append to `kb/log.md` ### Querying Knowledge Ask questions naturally: - "What projects use MQTT?" - "Show me the architecture of ha-core" - "Compare gdeploy and plugnburn-edl" - "What decisions were made about E3DC integration?" The LLM will search the wiki, synthesize an answer, and cite sources. ### Maintaining Knowledge (Lint) Periodically run: `Lint the wiki` The LLM will: - Run `tools/wikitool lint` for a deterministic structural + provenance scan (broken wikilinks, orphan pages, index drift, schema gaps, uncovered raw files, citation/frontmatter drift) - Check for contradictions (semantic judgment) - Find stale claims - Identify orphan pages and missing cross-references - Apply confidence decay (`tools/wikitool confidence decay --apply`) - Rebuild `kb/index.md` and `kb/provenance.md`, append to `kb/log.md` - Generate a report See the [Maintenance](#maintenance) section below for the full schedule and command reference. ## Entity Types Entities are subtyped as project, system, tool, technology, or person, and each subtype has its own directory under `kb/entities/`. The authoritative list - and where each one is written - is declared by the type-spec, so ask the tool rather than a table here: ```bash tools/wikitool types list tools/wikitool types describe entity ``` ## Workflows ### For You (Human) 1. **Curate sources** - Add files to `raw/` that you want processed 2. **Ask questions** - Query the wiki naturally 3. **Review changes** - Check `kb/log.md` and `kb/index.md` 4. **Direct the LLM** - Guide it on what to emphasize or investigate 5. **Browse in Obsidian** - Open the wiki directory in Obsidian for visualization ### For the LLM `AGENTS.md` is the cross-cutting schema/policy; the step-by-step procedures themselves live as independently-discoverable skills under `.agents/skills/` (mirrored to `.claude/skills/` for Claude Code via `tools/wikitool instructions sync`): | Skill | Purpose | |-------|---------| | `wiki-ingest` | Process a new `raw/` source into the wiki: source summary, entity/concept pages, cross-references, index/log, publish | | `wiki-query` | Answer a question from the compiled wiki; read-only, can optionally file a valuable answer back as a new page | | `wiki-lint` | Health-check the wiki: structural scan, raw coverage, semantic review, confidence decay | | `wiki-manage` | Create a new entity/concept/source/comparison page, or update an existing page with new information | | `wiki-status` | Read-only snapshot: page counts, orphans, uncovered raw files, most-connected pages | Each skill's underlying mechanical work (frontmatter, cross-references, index/log, decay math, publishing) is delegated to `tools/wikitool` - never hand-edited. ## Your first ingestion ### First Steps 1. Read `AGENTS.md` - the control plane (invariants, routing, gates) - then the stage contract for whichever of `raw/`, `types/` or `kb/` you are working in, and, inside `kb/`, the `COLLECTION.md` of the collection you are writing to 2. Add your first source to `raw/` 3. Run: `Ingest ` 4. Review the created pages 5. Ask your first query ### Example First Ingestion ```bash # Add a source cp ~/Downloads/my-notes.md raw/notes/my-notes.md # Tell the LLM to process it # (in your LLM agent) Ingest raw/notes/my-notes.md ``` ## Tips ### Naming - Use human-readable titles with spaces for files: `Hybrid Search.md`, not kebab-case - Use singular for entities: `ha-core.md` (not `ha-cores.md`) - Use wikilinks matching the file name exactly: `[[Entity Name]]` - **Titles follow the subject's own established name, not the wiki's language.** `Act Runner` and `GitOps Ownership Model` keep theirs. A title is the only identifier a page has - it also lives in every wikilink and citation id pointing at it - so translating one is a rename, never an edit: `tools/wikitool rename`, per `instructions/page-lifecycle.md` ### Organization - Start with a few broad categories, refine as needed - Don't over-organize early - let structure emerge - Use tags for cross-cutting concerns ### Quality - The LLM will maintain quality standards from `AGENTS.md` - Review changes periodically - Flag issues to the LLM ## Maintenance The wiki is kept healthy by deterministic `tools/wikitool` commands, run by the LLM (via the skills above). The schedule - which task runs how often, and with which command - lives in [`tools/CONTRACT.md`](tools/CONTRACT.md#maintenance-schedule), next to the command reference it depends on, so the two cannot drift apart. The notes below explain the three parts of it that need more than one line. **Confidence decay.** Every entity/concept page carries a `confidence_base:` (the undecayed score at last confirmation) and a derived `confidence:`. `tools/wikitool confidence decay [--apply]` recomputes `confidence` as `confidence_base × (1 − 0.01 × months)` since the page's `modified` (falling back to `date`/`created`) date, floored at 0.2. It's dry-run by default and only writes with `--apply`. Because it always recomputes from the untouched base, repeated runs are idempotent - never edit `confidence:` directly; use `tools/wikitool touch --page "" --confidence-base <value>` to re-assess a page. **Provenance.** Every fact should trace back to a raw file. Source pages declare their backing `raw_files:`; entity/concept pages declare `provenance:` (`sourced`/`general`/`mixed`) and cite specific claims inline with a `[^cite-id]` footnote (`tools/wikitool cite add` mints the id and definition; placing the marker in the prose is still manual). `tools/wikitool sources coverage` finds raw files with no source page yet, `sources trace` answers "where did this come from?" in either direction, and `sources rebuild-index` regenerates the reverse index at `kb/provenance.md`. `lint` cross-checks that citations and frontmatter `sources:` lists agree, and hard-errors on any leftover pre-migration `^[[...]]` marker. **Git automation.** `tools/wikitool publish` stages everything, commits with an auto-generated changed-file list, and pushes to `origin/main` in one step - never run raw `git commit`/`git push` for wiki changes. Publishes touching ≥10 files exit **42** (the **Mass-Update Gate**) - a distinct "a human must see this" code, not an error - printing the full file list and the `--confirm <token>` line that publishes it. The token digests that file list, so a clearance never carries to a changeset the user did not see. **Iteration/cost limits.** Every `tools/wikitool` call is checked against a hard, code-enforced per-session budget before it runs (default: 60 calls, or 3 identical calls in a row) - not just a prompt instruction to stop. Past the limit, the command refuses to run until a human approves continuing with `--override-budget`. `budget status` stays readable at all times; `budget reset` clears the counter and therefore needs `--yes` of its own. See AGENTS.md's "Gates" section. ## Telemetry and evaluation Every `wikitool` call appends an event to `reports/telemetry/<session>/trace.jsonl`, and the hook files under `.github/hooks/` and `.vibe/` add what the agent did between those calls. Nothing leaves the machine: `reports/` is gitignored and no exporter is configured. That record is what makes it possible to ask how a session *worked*, not just what it left behind: ```bash tools/wikitool eval sessions # which sessions have a trace tools/wikitool eval score # score this session ``` A score has two halves - the structural state of the tree, from lint's own checks, and trajectory rules over the trace, which catch things no unit test can: a refused call repeated unchanged, a gate flag passed without that gate having refused anything. [`EVALS.md`](EVALS.md) is the full picture: the event contract, what each agent harness can and cannot report, what is redacted, and why there is deliberately no LLM judge yet. ## Tools Integration ### wikitool (deterministic CLI) Mechanical wiki operations - never hand-edited by the LLM - are handled by `tools/wikitool`: scaffolding pages, renaming and deleting them, cross-references, index/log/provenance regeneration, confidence decay, structural linting, and publishing. The full command reference - every option, the per-command error contracts, and the maintenance schedule - is in [`tools/CONTRACT.md`](tools/CONTRACT.md). It is the single place that list lives, and `tools/wikitool docs verify` checks it against the CLI in both directions. [`tools/README.md`](tools/README.md) is the other half: how the CLI is built and how to add a command. `AGENTS.md` holds the invariants that say when each command is mandatory. ```bash tools/wikitool --help tools/wikitool <command> --help ``` <!-- dist:strip-start --> Dev-instance-only: extending `tools/wikitool`, the type schema, or the instruction/skill layer itself is a separate session type with its own rules, covered by the `stack-dev` skill nested under `instructions/dev/` (never present in a distributed instance - `tools/CONTRACT.md` explains why). <!-- dist:strip-end --> ### MCP read server (optional) The terminal is not the only way in. `tools/chemenu/mcp/` serves the same wiki read-only over MCP - `search`, `types`, `describe_type`, `lint` and `status` - so a consumer that is not a shell on this machine can ask the same questions and get the same answers. Literally the same: the CLI and the server are two adapters over one core, and a golden test holds their output together rather than trusting that it agrees. There is no tool that writes, and not because one is filtered out of a list: the server imports nothing under `chemenu/commands/`, so `new`, `publish` and the rest are unreachable from it. Every answer carries the commit it was computed from, so a checkout that has fallen behind produces a visibly stale answer instead of a confident wrong one. ```bash tools/.venv/bin/pip install -r tools/requirements-mcp.txt WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp # stdio WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp \ --transport streamable-http --host 0.0.0.0 --port 8000 # deployed ``` The dependency is deliberately not in `requirements.txt`: an instance that only uses the CLI should not have to install a web stack to do it. Running it, keeping its checkout current, and where authentication belongs (in front of the process, not in it) are in [`instructions/mcp-read-server.md`](instructions/mcp-read-server.md). ### Obsidian Open this directory in Obsidian for: - Graph view of connections - Easy navigation with wikilinks - Plugins: Dataview, Marp, etc. ### Git This is a git repo. Use it for: - Version history - Branching for experiments - Collaboration ### Search `tools/wikitool search "<text>"` searches `kb/` directly - by text, or by frontmatter with `--field entity_type=system` or `--field 'confidence<0.6'`. It is read-only and is the one command not counted against the session budget, because looking before acting is the habit worth encouraging. For browsing rather than searching, `kb/index.md` is the catalog map and each collection carries its own `INDEX.md`. ## IT-Specific Features This wiki is tailored for IT work with: - **Entity types** specific to software development and systems - **Relationship types** like `hängt ab von`, `verwendet`, `implementiert` - the vocabulary is in [kb/CONTRACT.md § Linking](kb/CONTRACT.md#linking) - **Templates** for projects, systems, tools, technologies, ADRs - **Guidelines** for documenting technical decisions - **Cross-reference patterns** for code and architecture ## Files Created Automatically The LLM will create and maintain: - `kb/index.md` - Always up-to-date catalog - `kb/log.md` - Complete audit trail - `kb/provenance.md` - Raw-file reverse index - Source pages in `kb/sources/` - Entity pages in `kb/entities/` - Concept pages in `kb/concepts/` - Comparison pages in `kb/comparisons/` - Lint reports, session traces and eval scores in `reports/` (gitignored) ## Changelog Changes to the wiki stack (schema, skills, `wikitool`, READMEs) are tracked in [`CHANGES.md`](CHANGES.md), not in an inline version history here. ## Resources - Original idea: [Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) - Extended with: [agentmemory](https://github.com/rohitg00/agentmemory) patterns - Search tool: [qmd](https://github.com/tobi/qmd) (for scaling) ## License Chemenu is dual-licensed, because it is two things in one repository. | Half | Licence | File | |------|---------|------| | The stack — `tools/`, `types/` | GNU AGPL-3.0-or-later | [LICENSE](LICENSE) | | The content — `kb/`, `raw/`, `instructions/`, the `CONTRACT.md` layer, and the prose documents at the root | CC-BY-4.0 | [LICENSE-CONTENT](LICENSE-CONTENT) | The boundary between the two is not a list someone maintains by hand: it is the file plan `tools/wikitool dist export` already computes, so it cannot drift out of agreement with what actually ships. [NOTICE](NOTICE) states that, and carries the attribution for the vendored [Commonplace](https://github.com/zby/commonplace) research base. **What this means for your own wiki.** The pages you write in your own instance are yours; the AGPL covers the compiler, not the knowledge it compiles. What the copyleft asks is that changes *to the machinery* stay available — including when you run it as a service rather than shipping it, which is why the licence is the Affero variant. If you set up an instance and later publish it, keep `LICENSE`, `LICENSE-CONTENT` and `NOTICE` in the tree. `dist export` puts them there and refuses to build a distribution without them, so under normal use this takes no effort.