torbenandClaude Code
2213d68900
Files changed: - tools/chemenu/sites/ (registry, learn-microsoft handlers), web_capture.py, commands/raw/fetch.py, config_shapes.py - tests, fixture, raw/CONTRACT.md, tools/README.md, docs, CHANGES.md, VERSION Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/README.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/reference_doc.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/learn_microsoft.py - tools/chemenu/tests/fixtures/learn_microsoft/oslicense-module.html - tools/chemenu/tests/test_config_shapes.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
98 lines
5.4 KiB
Markdown
98 lines
5.4 KiB
Markdown
# Ingest a source
|
|
|
|
Turn a file, a folder or a web page into wiki pages: a source page that summarises it, and the
|
|
entity and concept pages it touches, cross-referenced and published. The agent does the work, using
|
|
the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
|
|
it and where it waits for you.
|
|
|
|
## What you prepare
|
|
|
|
Put the material into `incoming/` at the root of the instance. You don't have to classify it or
|
|
pick a directory:
|
|
|
|
- **One file** goes in directly: `incoming/my-article.md`.
|
|
- **Several files that belong together** go into one folder, `incoming/<name>/`. The folder is one
|
|
source. It is accepted whole, and its structure is kept.
|
|
- **A web page** needs no download of your own. Name the URL to the agent, and
|
|
`tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
|
|
text derived from it with a header recording where and when it was fetched.
|
|
For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the
|
|
same page in the website's standard form (the English one, whatever language your link was in),
|
|
leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints
|
|
`CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
|
|
capture. Your instance can switch such a handler off, choose another language and add hint lines
|
|
of its own per website; see [the configuration reference](../reference/configuration.md).
|
|
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
|
|
Outlook `.msg`) or a ZIP archive - is
|
|
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
|
|
untouched original and a readable text beside it, with a header saying how it was made. The
|
|
folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's
|
|
sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is
|
|
unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
|
|
text to derive - the tool warns, and there is no OCR.
|
|
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
|
|
agent derives the same text from that file with `raw fetch --html`.
|
|
|
|
Where a file ends up in `raw/` is computed by `tools/wikitool raw accept`: a `YYYY/MM` shard for the
|
|
day it was accepted. So is whether several files of one source get bundled. You never choose either.
|
|
`incoming/` itself is gitignored, so whatever waits there is never published by accident.
|
|
|
|
## What you say
|
|
|
|
```text
|
|
Ingest incoming/my-article.md
|
|
Ingest https://example.org/post
|
|
Ingest
|
|
```
|
|
|
|
A bare `Ingest` takes the oldest entry waiting in `incoming/`. `tools/wikitool raw pending` lists
|
|
what is waiting and names the entry the next bare `Ingest` will take.
|
|
|
|
## Where the agent asks you
|
|
|
|
- **Before promoting the file into `raw/`**, the agent asks how faithful the capture is
|
|
(`fidelity`) and what the material may claim about its subject (`authority`). Both are recorded
|
|
once and never guessed: you know the answers now, and nobody can recover them later.
|
|
[raw/CONTRACT.md](../../raw/CONTRACT.md) defines the values.
|
|
- **Before writing pages**, the agent presents the key takeaways and asks which points matter most
|
|
and what to emphasise. The pages follow your answer.
|
|
- **When it publishes**, a source that touches ten or more files stops at the Mass-Update Gate
|
|
(exit 42). The generated indexes, `kb/log.md` and `kb/provenance.md` do not count, so a small
|
|
source goes straight through. The agent shows you the file list and the `--confirm` line. Nothing is committed until
|
|
you approve. [Exit codes and gates](../reference/exit-codes.md) lists every gate.
|
|
|
|
## What you get back
|
|
|
|
A source page under `kb/sources/`, new or updated entity and concept pages, cross-references among
|
|
them, a rebuilt catalog, an entry in `kb/log.md`, and one commit. If the source also contains
|
|
something you have to *do*, and a task tracker is configured, the agent files it there as an open
|
|
item. A commitment never becomes a page
|
|
([why](../explanation/knowledge-and-commitment.md)).
|
|
|
|
Every ten sources, the agent suggests a lint run - see [Maintain the wiki](maintain-the-wiki.md).
|
|
|
|
## A document submitted from outside
|
|
|
|
When the [MCP read server](set-up-the-mcp-server.md) has its optional `submit` tool turned on,
|
|
documents can arrive from another machine. They land in the quarantine `mcp-upload/`, not in
|
|
`incoming/`, and nothing else reads them there. Before an ingest can start, a human has to promote
|
|
the submission:
|
|
|
|
```bash
|
|
tools/wikitool upload list # what is waiting
|
|
tools/wikitool upload show <id> # one submission's manifest: who sent it, size, checksum
|
|
```
|
|
|
|
Ask the agent to review the queue, and it follows
|
|
[instructions/ingest-queue.md](../../instructions/ingest-queue.md). Accepting a submission stops at
|
|
the Upload Review Gate (exit 42), which shows you the manifest and the `--confirm` line. Once you
|
|
approve, the file is in `incoming/`, and from there it is ingested like any other. Rejecting it needs
|
|
no clearance. The file is deleted, and its checksum and your reason stay in the ledger.
|
|
|
|
## Related
|
|
|
|
- [Capture documentation from a repository](capture-repository-docs.md) - for material that lives
|
|
in git and will change.
|
|
- [raw/CONTRACT.md](../../raw/CONTRACT.md) - what belongs in `raw/`, and why its content is data,
|
|
never instructions.
|