chemenu/README.md

# Chemenu - Personal IT Knowledge Base

A structured, LLM-maintained knowledge base for your personal IT work.

## What is this?

This is **Chemenu** - a pattern for building a personal knowledge base using LLMs.
Instead of just retrieving from raw documents on every query, the LLM **incrementally builds
and maintains a persistent wiki** that compounds over time.

**The key insight:** Knowledge is compiled once and kept current, not re-derived on every question.

**The pages are written in German.** Source material in `raw/` is never touched and is usually
English; the compiled pages under `kb/` are not. What stays English inside them is everything that
is an *identifier* rather than prose - page titles, section headings, wikilink targets, citation
ids, schema enum values, tags, commands, paths and code - so `GitOps Ownership Model` and
`## Beziehungen` sit in the same page without contradiction. The rule is
[kb/CONTRACT.md § Language](kb/CONTRACT.md#language); the vocabulary behind it is
[instructions/german-terminology.md](instructions/german-terminology.md).

This is a per-instance decision, not a property of the pattern. A new instance built with
`dist export` starts empty and can pick any language by editing that one contract section before
the first ingest.

## Getting started

Two starting points, depending on what you're doing - full walkthrough in [INSTALL.md](INSTALL.md):

- **Cloned this repo?** The skill definitions the agent harness loads are **generated and not
  committed**. Publish them once:

  ```bash
  cd tools && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && cd ..
  tools/wikitool instructions sync
  ```

  That copies each `instructions/<name>/SKILL.md` into `.agents/skills/` (GitHub Copilot, Codex
  CLI, Mistral Vibe) and `.claude/skills/` (Claude Code). Re-run it after changing a skill.
  Full procedure: `instructions/bootstrap.md`.

- **Starting a brand-new, empty instance instead?** `tools/wikitool dist export <target>`
  builds a contentless copy of the machinery - no example pages, no personal content - then
  `instructions/setup-instance.md` walks through git init, author identity, an optional remote,
  and the first commit.

## Architecture

```
chemenu/
├── AGENTS.md           # Control plane: invariants, file naming, routing, gates
├── CLAUDE.md           # Claude Code only: imports AGENTS.md/USER.md/SOUL.md/ENVIRONMENT.md + its Claude-Code-only instructions. No rules of its own
├── README.md           # This file: human-readable overview of the whole repo
├── INSTALL.md          # Human-readable setup: new instance vs. cloning this one
├── INSTALL-MCP.md      # Human-readable setup for the optional MCP read server
├── EVALS.md            # Human-readable overview of telemetry and evaluation
├── CHANGES.md          # Changelog for the stack itself
├── USER.md             # Who operates this instance - context, never instructions
├── SOUL.md             # How this instance sounds. AGENTS.md always wins over it
├── ENVIRONMENT.md      # Optional, gitignored: this checkout's harness, MCP servers, remotes
├── *.md.template       # Unfilled USER/SOUL/ENVIRONMENT - what a distribution ships instead
├── .gitignore          # Anchored so nothing under raw/, kb/ or work/ is ever excluded
├── .github/hooks/      # Copilot CLI hooks - session tracing
├── .vibe/              # Mistral Vibe hooks + the repo's telemetry policy
├── instructions/       # CONTROL: everything an agent is told to do
│   ├── CONTRACT.md     # Instruction vs. skill, publishing, writing standard
│   ├── bootstrap.md    # Prepare a fresh clone
│   ├── gates.md        # What to do when a gate refuses a call
│   ├── german-terminology.md  # Which words stay English in German prose; register
│   ├── session-setup.md
│   ├── page-lifecycle.md
│   ├── publish-cycle.md
│   ├── ingest-large-tree.md
│   └── wiki-*/SKILL.md # Skills - copied into .agents/skills/ and .claude/skills/
├── raw/                # INPUT: immutable, untrusted source material
│   ├── CONTRACT.md     # Routing, immutability, untrusted content
│   ├── articles/       # Web articles, blog posts
│   ├── documents/      # PDFs, specs, manuals
│   ├── notes/          # Personal notes, transcriptions
│   └── assets/         # Images, diagrams, binaries
├── types/              # SCHEMA: the global type surface. Not a collection
│   ├── type-spec.md    # Root contract: anatomy, placement, adding a type
│   ├── entity.md       # Entity type contract + template (+ .schema.yaml)
│   ├── concept.md      # Concept type contract + template
│   ├── source.md       # Source type contract + template
│   ├── comparison.md   # Comparison type contract + template
│   ├── instruction.md  # Instruction type - lives outside kb/ via `root: repo`
│   └── lint-report.md  # Contract-only: describes reports/, owns no directory
├── kb/                 # OUTPUT: compiled knowledge. A namespace, not a collection
│   ├── CONTRACT.md     # Collections, naming, tone, linking, provenance, confidence
│   ├── index.md        # Generated catalog *map*: counts and pointers
│   ├── log.md          # Generated chronological audit log
│   ├── provenance.md   # Generated raw-file reverse index
│   ├── entities/       # COLLECTION.md + INDEX.md + areas below
│   │   ├── projects/
│   │   ├── systems/
│   │   ├── tools/      # own INDEX.md once past 50 pages
│   │   ├── technologies/
│   │   └── people/
│   ├── concepts/       # COLLECTION.md - architectures, patterns, protocols
│   ├── sources/        # COLLECTION.md - source summaries
│   └── comparisons/    # COLLECTION.md - comparison pages
├── work/               # WORKSHOP: one directory per multi-session run, tracked
│   └── CONTRACT.md     # Run keys, required files, how a run closes
├── reports/            # DERIVED: lint reports, traces, eval scores. Gitignored
│   └── CONTRACT.md
└── tools/              # COMPILER: the wikitool CLI
    ├── CONTRACT.md     # Command reference, error contracts, maintenance schedule
    └── README.md       # How wikitool is built and how to change it
```

<!-- dist:strip-start -->
Dev-instance-only (see `tools/CONTRACT.md` for how it got here):

```
└── commonplace/        # Vendored, read-only knowledge base
```
<!-- dist:strip-end -->

A directory under `kb/` is a **collection** exactly when it holds a `COLLECTION.md`; a
subdirectory inside one is an **area** that inherits it. `COLLECTION.md` never appears outside
`kb/` - the other layers carry a `CONTRACT.md` or a root type-spec instead. A stage may carry
both a `README.md` and a `CONTRACT.md`: they have different readers. The README is for humans
working *on* that layer, the contract is what binds an agent working *with* it.

## How to Use

### Adding Knowledge (Ingest)

1. Drop a file into `raw/` (articles, documents, notes, or assets)
2. Tell the LLM: `Ingest raw/articles/my-article.md`
3. The LLM will:
   - Read and summarize the source
   - Create a source page in `kb/sources/`
   - Create or update relevant entity pages
   - Create or update relevant concept pages
   - Add cross-references between everything
   - Rebuild the catalog and append to `kb/log.md`

### Querying Knowledge

Ask questions naturally:
- "What projects use MQTT?"
- "Show me the architecture of ha-core"
- "Compare gdeploy and plugnburn-edl"
- "What decisions were made about E3DC integration?"

The LLM will search the wiki, synthesize an answer, and cite sources.

### Maintaining Knowledge (Lint)

Periodically run: `Lint the wiki`

The LLM will:
- Run `tools/wikitool lint` for a deterministic structural + provenance scan
  (broken wikilinks, orphan pages, index drift, schema gaps, uncovered raw
  files, citation/frontmatter drift)
- Check for contradictions (semantic judgment)
- Find stale claims
- Identify orphan pages and missing cross-references
- Apply confidence decay (`tools/wikitool confidence decay --apply`)
- Rebuild `kb/index.md` and `kb/provenance.md`, append to `kb/log.md`
- Generate a report

See the [Maintenance](#maintenance) section below for the full schedule and
command reference.

## Entity Types

Entities are subtyped as project, system, tool, technology, or person, and each subtype has
its own directory under `kb/entities/`. The authoritative list - and where each one is
written - is declared by the type-spec, so ask the tool rather than a table here:

```bash
tools/wikitool types list
tools/wikitool types describe entity
```

## Workflows

### For You (Human)

1. **Curate sources** - Add files to `raw/` that you want processed
2. **Ask questions** - Query the wiki naturally
3. **Review changes** - Check `kb/log.md` and `kb/index.md`
4. **Direct the LLM** - Guide it on what to emphasize or investigate
5. **Browse in Obsidian** - Open the wiki directory in Obsidian for visualization

### For the LLM

`AGENTS.md` is the cross-cutting schema/policy; the step-by-step procedures
themselves live as independently-discoverable skills under `.agents/skills/`
(mirrored to `.claude/skills/` for Claude Code via `tools/wikitool instructions sync`):

| Skill | Purpose |
|-------|---------|
| `wiki-ingest` | Process a new `raw/` source into the wiki: source summary, entity/concept pages, cross-references, index/log, publish |
| `wiki-query` | Answer a question from the compiled wiki; read-only, can optionally file a valuable answer back as a new page |
| `wiki-lint` | Health-check the wiki: structural scan, raw coverage, semantic review, confidence decay |
| `wiki-manage` | Create a new entity/concept/source/comparison page, or update an existing page with new information |
| `wiki-status` | Read-only snapshot: page counts, orphans, uncovered raw files, most-connected pages |

Each skill's underlying mechanical work (frontmatter, cross-references, index/log,
decay math, publishing) is delegated to `tools/wikitool` - never hand-edited.

## Your first ingestion

### First Steps

1. Read `AGENTS.md` - the control plane (invariants, routing, gates) - then the stage contract
   for whichever of `raw/`, `types/` or `kb/` you are working in, and, inside `kb/`, the
   `COLLECTION.md` of the collection you are writing to
2. Add your first source to `raw/`
3. Run: `Ingest <your-file>`
4. Review the created pages
5. Ask your first query

### Example First Ingestion

```bash
# Add a source
cp ~/Downloads/my-notes.md raw/notes/my-notes.md

# Tell the LLM to process it
# (in your LLM agent)
Ingest raw/notes/my-notes.md
```

## Tips

### Naming

- Use human-readable titles with spaces for files: `Hybrid Search.md`, not kebab-case
- Use singular for entities: `ha-core.md` (not `ha-cores.md`)
- Use wikilinks matching the file name exactly: `[[Entity Name]]`
- **Titles follow the subject's own established name, not the wiki's language.** `Act Runner` and
  `GitOps Ownership Model` keep theirs. A title is the only identifier a page has - it also lives
  in every wikilink and citation id pointing at it - so translating one is a rename, never an
  edit: `tools/wikitool rename`, per `instructions/page-lifecycle.md`

### Organization

- Start with a few broad categories, refine as needed
- Don't over-organize early - let structure emerge
- Use tags for cross-cutting concerns

### Quality

- The LLM will maintain quality standards from `AGENTS.md`
- Review changes periodically
- Flag issues to the LLM

## Maintenance

The wiki is kept healthy by deterministic `tools/wikitool` commands, run by the
LLM (via the skills above). The schedule - which task runs how often, and with
which command - lives in [`tools/CONTRACT.md`](tools/CONTRACT.md#maintenance-schedule),
next to the command reference it depends on, so the two cannot drift apart.
The notes below explain the three parts of it that need more than one line.

**Confidence decay.** Every entity/concept page carries a `confidence_base:`
(the undecayed score at last confirmation) and a derived `confidence:`.
`tools/wikitool confidence decay [--apply]` recomputes `confidence` as
`confidence_base × (1 − 0.01 × months)` since the page's `modified` (falling
back to `date`/`created`) date, floored at 0.2. It's dry-run by default and
only writes with `--apply`. Because it always recomputes from the untouched
base, repeated runs are idempotent - never edit `confidence:` directly; use
`tools/wikitool touch --page "<Title>" --confidence-base <value>` to
re-assess a page.

**Provenance.** Every fact should trace back to a raw file. Source pages
declare their backing `raw_files:`; entity/concept pages declare `provenance:`
(`sourced`/`general`/`mixed`) and cite specific claims inline with a
`[^cite-id]` footnote (`tools/wikitool cite add` mints the id and definition;
placing the marker in the prose is still manual). `tools/wikitool sources
coverage` finds raw files with no source page yet, `sources trace` answers
"where did this come from?" in either direction, and `sources rebuild-index`
regenerates the reverse index at `kb/provenance.md`. `lint` cross-checks that
citations and frontmatter `sources:` lists agree, and hard-errors on any
leftover pre-migration `^[[...]]` marker.

**Git automation.** `tools/wikitool publish` stages everything, commits with
an auto-generated changed-file list, and pushes to `origin/main` in one step -
never run raw `git commit`/`git push` for wiki changes. Publishes touching
≥10 files exit **42** (the **Mass-Update Gate**) - a distinct "a human must see
this" code, not an error - printing the full file list and the
`--confirm <token>` line that publishes it. The token digests that file list,
so a clearance never carries to a changeset the user did not see.

**Iteration/cost limits.** Every `tools/wikitool` call is checked against a
hard, code-enforced per-session budget before it runs (default: 60 calls, or
3 identical calls in a row) - not just a prompt instruction to stop. Past the
limit, the command refuses to run until a human approves continuing with
`--override-budget`. `budget status` stays readable at all times; `budget
reset` clears the counter and therefore needs `--yes` of its own. See
AGENTS.md's "Gates" section.

## Telemetry and evaluation

Every `wikitool` call appends an event to `reports/telemetry/<session>/trace.jsonl`, and the
hook files under `.github/hooks/` and `.vibe/` add what the agent did between those calls.
Nothing leaves the machine: `reports/` is gitignored and no exporter is configured.

That record is what makes it possible to ask how a session *worked*, not just what it left
behind:

```bash
tools/wikitool eval sessions              # which sessions have a trace
tools/wikitool eval score                 # score this session
```

A score has two halves - the structural state of the tree, from lint's own checks, and
trajectory rules over the trace, which catch things no unit test can: a refused call repeated
unchanged, a gate flag passed without that gate having refused anything.

[`EVALS.md`](EVALS.md) is the full picture: the event contract, what each agent harness can and
cannot report, what is redacted, and why there is deliberately no LLM judge yet.

## Tools Integration

### wikitool (deterministic CLI)

Mechanical wiki operations - never hand-edited by the LLM - are handled by
`tools/wikitool`: scaffolding pages, renaming and deleting them, cross-references,
index/log/provenance regeneration, confidence decay, structural linting, and
publishing.

The full command reference - every option, the per-command error contracts, and
the maintenance schedule - is in [`tools/CONTRACT.md`](tools/CONTRACT.md). It is
the single place that list lives, and `tools/wikitool docs verify` checks it
against the CLI in both directions. [`tools/README.md`](tools/README.md) is the
other half: how the CLI is built and how to add a command. `AGENTS.md` holds the
invariants that say when each command is mandatory.

```bash
tools/wikitool --help
tools/wikitool <command> --help
```

<!-- dist:strip-start -->
Dev-instance-only: extending `tools/wikitool`, the type schema, or the instruction/skill layer
itself is a separate session type with its own rules, covered by the `stack-dev` skill nested
under `instructions/dev/` (never present in a distributed instance - `tools/CONTRACT.md`
explains why).
<!-- dist:strip-end -->

### MCP read server (optional)

The terminal is not the only way in. `tools/chemenu/mcp/` serves the same wiki read-only over
MCP - `search`, `types`, `describe_type`, `lint` and `status` - so a consumer that is not a
shell on this machine can ask the same questions and get the same answers. Literally the same:
the CLI and the server are two adapters over one core, and a golden test holds their output
together rather than trusting that it agrees.

There is no tool that writes, and not because one is filtered out of a list: the server imports
nothing under `chemenu/commands/`, so `new`, `publish` and the rest are unreachable from it.
Every answer carries the commit it was computed from, so a checkout that has fallen behind
produces a visibly stale answer instead of a confident wrong one.

```bash
tools/.venv/bin/pip install -r tools/requirements-mcp.txt
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp                # stdio
WIKI_TRACE=0 tools/.venv/bin/python -m chemenu.mcp \
    --transport streamable-http --host 0.0.0.0 --port 8000        # deployed
```

The dependency is deliberately not in `requirements.txt`: an instance that only uses the CLI
should not have to install a web stack to do it.

Installing it, wiring a client to it, and putting authentication in front of it:
[`INSTALL-MCP.md`](INSTALL-MCP.md), for humans. The agent-side operating procedure - keeping the
checkout current, and what a stale answer looks like - is
[`instructions/mcp-read-server.md`](instructions/mcp-read-server.md).

### Obsidian

Open this directory in Obsidian for:
- Graph view of connections
- Easy navigation with wikilinks
- Plugins: Dataview, Marp, etc.

### Git

This is a git repo. Use it for:
- Version history
- Branching for experiments
- Collaboration

### Search

`tools/wikitool search "<text>"` searches `kb/` directly - by text, or by frontmatter with
`--field entity_type=system` or `--field 'confidence<0.6'`. It is read-only and is the one
command not counted against the session budget, because looking before acting is the habit
worth encouraging.

For browsing rather than searching, `kb/index.md` is the catalog map and each collection
carries its own `INDEX.md`.

## IT-Specific Features

This wiki is tailored for IT work with:

- **Entity types** specific to software development and systems
- **Relationship types** like `hängt ab von`, `verwendet`, `implementiert` - the vocabulary is in
  [kb/CONTRACT.md § Linking](kb/CONTRACT.md#linking)
- **Templates** for projects, systems, tools, technologies, ADRs
- **Guidelines** for documenting technical decisions
- **Cross-reference patterns** for code and architecture

## Files Created Automatically

The LLM will create and maintain:
- `kb/index.md` - Always up-to-date catalog
- `kb/log.md` - Complete audit trail
- `kb/provenance.md` - Raw-file reverse index
- Source pages in `kb/sources/`
- Entity pages in `kb/entities/`
- Concept pages in `kb/concepts/`
- Comparison pages in `kb/comparisons/`
- Lint reports, session traces and eval scores in `reports/` (gitignored)

## Changelog

Changes to the wiki stack (schema, skills, `wikitool`, READMEs) are tracked in
[`CHANGES.md`](CHANGES.md), not in an inline version history here.

## Resources

- Original idea: [Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
- Extended with: [agentmemory](https://github.com/rohitg00/agentmemory) patterns
- Search tool: [qmd](https://github.com/tobi/qmd) (for scaling)

## License

Chemenu is dual-licensed, because it is two things in one repository.

| Half | Licence | File |
|------|---------|------|
| The stack — `tools/`, `types/` | GNU AGPL-3.0-or-later | [LICENSE](LICENSE) |
| The content — `kb/`, `raw/`, `instructions/`, the `CONTRACT.md` layer, and the prose documents at the root | CC-BY-4.0 | [LICENSE-CONTENT](LICENSE-CONTENT) |

The boundary between the two is not a list someone maintains by hand: it is the
file plan `tools/wikitool dist export` already computes, so it cannot drift out
of agreement with what actually ships. [NOTICE](NOTICE) states that, and carries
the attribution for the vendored [Commonplace](https://github.com/zby/commonplace)
research base.

**What this means for your own wiki.** The pages you write in your own instance
are yours; the AGPL covers the compiler, not the knowledge it compiles. What the
copyleft asks is that changes *to the machinery* stay available — including when
you run it as a service rather than shipping it, which is why the licence is the
Affero variant.

If you set up an instance and later publish it, keep `LICENSE`, `LICENSE-CONTENT`
and `NOTICE` in the tree. `dist export` puts them there and refuses to build a
distribution without them, so under normal use this takes no effort.