feat: incoming/ as a queue - raw pending picks the next entry, raw accept takes a whole folder, a file in a subdirectory of incoming/ is refused (#112)
CI / verify (push) Successful in 5m24s
CI / pwsh (push) Successful in 1m58s
Release / release (push) Successful in 35s

Files changed:
- CHANGES.md
- README.md
- VERSION
- instructions/ingest-large-tree.md
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_raw_cmd.py
- tools/chemenu/tests/test_raw_fetch.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-03 09:49:20 +02:00
1 parent c4dcff76e6
commit 59c06e5ddc
11 files changed
+1009 -182

No files matched your search

+37 -8
View File
@@ -1,15 +1,16 @@
---
name: wiki-ingest
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki".
description: Processes a new source file into the LLM wiki - extracts entities and concepts, creates a source summary page, files a tracker item for any commitment the source also carries, cross-references, rebuilds indexes, and publishes. Use when the user drops a file or folder into incoming/ or raw/, names a URL to ingest, or says "ingest <file>", "ingest <url>", "process this source", "add this to the wiki" - or just "ingest" with nothing named, which takes the oldest entry waiting in incoming/.
---
# Wiki Ingest
**Purpose:** Process a new source file and integrate its knowledge into the wiki.
**Trigger:** User drops a file into `incoming/` (the normal path - see step 5) or directly into
`raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or explicitly requests
ingestion.
**Trigger:** User drops a file or a folder into `incoming/` (the normal path - see step 5) or
directly into `raw/`, names a URL to ingest (step 1 fetches it into `incoming/` first), or
explicitly requests ingestion - with or without naming what (step 1 picks the entry when nothing
is named). **One run is one source:** one file, one bundle or one folder.
**Before the first `wikitool` call:** `instructions/session-setup.md`.
@@ -45,6 +46,24 @@ validator complains - and the ticked list is the only record that they happened.
just wrote, for instance, which skips step 5 entirely). If it is binary or an image, note its
presence and what it shows.
**The user named nothing** ("ingest", "process the inbox")? Pick the entry from the queue:
```bash
tools/wikitool raw pending
```
It lists what waits in `incoming/`, oldest first, and marks the default - the oldest entry
`raw accept` would take as it stands. **Announce it and carry on with it:** which entry, why
this one (the oldest that can be accepted), and how many wait after it. Ask nothing here -
step 4 is the halt before anything is written. One run takes exactly that one entry. If
nothing acceptable is waiting, the run ends here: say so, and name each entry the listing
marked as not acceptable, with its reason - those need the user, not a guess.
**The source is a folder** (`incoming/<folder>/`, named or picked)? It is one source - read
every file in it. Whether it is ingested here or by the large-tree procedure is decided by
the size check below, by its thresholds, not by its being a folder: three notes in a folder
do not earn a workshop.
**The user named a URL instead of a file?** Fetch it into `incoming/` first - never with
`curl` or the harness's own web fetch, which returns a model's summary rather than the page:
@@ -202,8 +221,14 @@ validator complains - and the ticked list is the only record that they happened.
List every file this one source produced (e.g. an uploaded PDF plus its converted Markdown)
in the same call, so they land bundled together rather than as two independent promotions. A
file already in `raw/` skips this step entirely. A subdirectory under `incoming/` (an old
`incoming/<type>/` habit) is tolerated and ignored - it carries no meaning any more.
file already in `raw/` skips this step entirely. A folder is accepted as a whole, on its own:
```bash
tools/wikitool raw accept --fidelity <value> --authority <value> incoming/<folder>
```
A file inside a subdirectory of `incoming/` is refused on its own - the subdirectory is the
source.
**A file that arrived through the MCP `submit` tool is not yet in `incoming/`** - it sits in
`mcp-upload/<id>/`, a quarantine no command in this step reads. A reviewer promotes it first
@@ -343,6 +368,9 @@ validator complains - and the ticked list is the only record that they happened.
deterministic count behind the "every 10 sources" cadence. If the threshold is reached,
tell the user a full lint is due and offer to run `wiki-lint` next.
Then say how many entries still wait in `incoming/` (`tools/wikitool raw pending`), so the
user knows whether another run is due.
## Decision points
- **Subject already has a page?** Update it (step 7, `touch`) instead of creating a second one.
@@ -374,7 +402,7 @@ validator complains - and the ticked list is the only record that they happened.
## wikitool commands used
`raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
`raw pending`, `raw fetch`, `raw accept`, `search`, `types describe`, `task new`, `task list`, `task close`,
`new project`, `new source`, `new entity`, `new concept`, `touch`, `cite add`, `xref add`,
`xref link-source`, `sources coverage`, `sources rebuild-index`, `index rebuild`, `log append`,
`log status`, `publish`
@@ -383,4 +411,5 @@ validator complains - and the ticked list is the only record that they happened.
Updated wiki with the source's knowledge integrated, published to `origin/main`.
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post"
**Example triggers:** "Ingest raw/articles/my-article.md", "Ingest https://example.org/post",
"Ingest incoming/projekt-x", "Ingest" (the oldest entry waiting in `incoming/`)