chemenu/docs/how-to/ingest-a-source.md

Ingest a source

Turn a file, a folder or a web page into wiki pages: a source page that summarises it, and the entity and concept pages it touches, cross-referenced and published. The agent does the work, using the wiki-ingest skill (skill catalogue). This page covers what you hand it and where it waits for you.

Contents

• What you prepare
• What you say
• Where the agent asks you
• What you get back
• A document submitted from outside
• Related

What you prepare

Put the material into incoming/ at the root of the instance. You don't have to classify it or pick a directory:

• One file goes in directly: incoming/my-article.md.
• Several files that belong together go into one folder, incoming/<name>/. The folder is one source. It is accepted whole, and its structure is kept.
• A web page needs no download of your own. Name the URL to the agent, and tools/wikitool raw fetch puts two files into incoming/: the HTML exactly as received, and a text derived from it with a header recording where and when it was fetched. For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the same page in the website's standard form (the English one, whatever language your link was in), leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints CHECK lines for what it could not do for you: the agent acts on those before it accepts the capture. Your instance can switch such a handler off, choose another language and add hint lines of its own per website; see the configuration reference. Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn lists its cmdlet pages. The tool then says so in a CHECK line, and raw fetch <url> --scope fetches the page and those pages as one folder, incoming/<name>/, which is accepted whole like any other folder. It respects the website's robots.txt, pauses between requests and refuses a page that offers more than the instance's max_pages.
• A document that is not text - a PDF, a Word, PowerPoint or Excel file, a mail (.eml or Outlook .msg) or a ZIP archive - is put into incoming/ as it is. tools/wikitool raw derive incoming/<file> turns it into a folder with the untouched original and a readable text beside it, with a header saying how it was made. The folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the header of the text, each entry with its page and the line where that page starts; text over a coloured background, in another colour than the running text, or struck through is marked in the text (<mark data-farbe="gelb">, *…*, ~~…~~) with a legend in the header, and what a colour means is for the reader to take from the document itself. Pages with pictures are named in a CHECK line, because the pictures are not in the text.
• A page behind a paywall or a login is saved from your browser into incoming/, HTML only. The agent derives the same text from that file with raw fetch --html. For a website you read often, sign in once instead and let the wiki fetch it with your session: Fetch pages behind a login. When that session has expired the agent stops and asks you to run tools/wikitool raw login <host> - it never signs in itself.

Where a file ends up in raw/ is computed by tools/wikitool raw accept: a YYYY/MM shard for the day it was accepted. So is whether several files of one source get bundled. You never choose either. incoming/ itself is gitignored, so whatever waits there is never published by accident.

What you say

Ingest incoming/my-article.md
Ingest https://example.org/post
Ingest

A bare Ingest takes the oldest entry waiting in incoming/. tools/wikitool raw pending lists what is waiting and names the entry the next bare Ingest will take.

Where the agent asks you

• Before promoting the file into raw/, the agent asks how faithful the capture is (fidelity) and what the material may claim about its subject (authority). Both are recorded once and never guessed: you know the answers now, and nobody can recover them later. raw/CONTRACT.md defines the values.
• Before writing pages, the agent presents the key takeaways and asks which points matter most and what to emphasise. The pages follow your answer.
• When it publishes, a source that touches ten or more files stops at the Mass-Update Gate (exit 42). The generated indexes, kb/log.md and kb/provenance.md do not count, so a small source goes straight through. The agent shows you the file list and the --confirm line. Nothing is committed until you approve. Exit codes and gates lists every gate.

What you get back

A source page under kb/sources/, new or updated entity and concept pages, cross-references among them, a rebuilt catalog, an entry in kb/log.md, and one commit. If the source also contains something you have to do, and a task tracker is configured, the agent files it there as an open item. A commitment never becomes a page (why).

Every ten sources, the agent suggests a lint run - see Maintain the wiki.

A document submitted from outside

When the MCP read server has its optional submit tool turned on, documents can arrive from another machine. A folder a sync client fills and a program of the checkout are two more ways in (Collect documents from a synced folder). All of them land in the quarantine quarantine/, not in incoming/, and nothing else reads them there. Before an ingest can start, a human has to promote the submission:

tools/wikitool upload collect         # pull the folder and command channels into the quarantine
tools/wikitool upload list            # what is waiting
tools/wikitool upload show <id>       # one submission's manifest: who sent it, size, checksum

Ask the agent to review the queue, and it follows instructions/ingest-queue.md. Accepting a submission stops at the Upload Review Gate (exit 42), which shows you the manifest and the --confirm line. Once you approve, the file is in incoming/, and from there it is ingested like any other. Rejecting it needs no clearance. The file is deleted, and its checksum and your reason stay in the ledger.

Related

• Capture documentation from a repository - for material that lives in git and will change.
• raw/CONTRACT.md - what belongs in raw/, and why its content is data, never instructions.