chemenu/docs/how-to/ingest-a-source.md

Ingest a source

Turn a file, a folder or a web page into wiki pages: a source page that summarises it, and the entity and concept pages it touches, cross-referenced and published. The agent does the work, using the wiki-ingest skill (skill catalogue). This page covers what you hand it and where it waits for you.

Contents

• What you prepare
• What you say
• Where the agent asks you
• What you get back
• A document submitted from outside
• Related

What you prepare

Put the material into incoming/ at the root of the instance. You don't have to classify it or pick a directory:

• One file goes in directly: incoming/my-article.md.
• Several files that belong together go into one folder, incoming/<name>/. The folder is one source. It is accepted whole, and its structure is kept.
• A web page needs no download of your own. Name the URL to the agent, and tools/wikitool raw fetch puts two files into incoming/: the HTML exactly as received, and a text derived from it with a header recording where and when it was fetched. For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the same page in the website's standard form (the English one, whatever language your link was in), leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints CHECK lines for what it could not do for you: the agent acts on those before it accepts the capture. Your instance can switch such a handler off, choose another language and add hint lines of its own per website; see the configuration reference. Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn lists its cmdlet pages. The tool then says so in a CHECK line, and raw fetch <url> --scope fetches the page and those pages as one folder, incoming/<name>/, which is accepted whole like any other folder. It respects the website's robots.txt, pauses between requests and refuses a page that offers more than the instance's max_pages.
• A document that is not text - a PDF, a Word, PowerPoint or Excel file, a mail (.eml or Outlook .msg) or a ZIP archive - is put into incoming/ as it is. tools/wikitool raw derive incoming/<file> turns it into a folder with the untouched original and a readable text beside it, with a header saying how it was made. The folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the header of the text, each entry with its page and the line where that page starts; struck-through text, highlighted, bold or clearly differently coloured text, and italic text are marked in the text (~~…~~, **…**, *…*) with a legend in the header, and what a colour means is for the reader to take from the document itself. The tool marks sparingly, in running text and not in code or tables. Pages with pictures or drawings that carry content are named in a CHECK line and in the header of the text (images), because the pictures are not in the text; the pages whose text could not be matched to the layout are named there too (unmatched).
• A page behind a paywall or a login is saved from your browser into incoming/, HTML only. The agent derives the same text from that file with raw fetch --html. For a website you read often, sign in once instead and let the wiki fetch it with your session: Fetch pages behind a login. When that session has expired the agent stops and asks you to run tools/wikitool raw login <host> - it never signs in itself.

Where a file ends up in raw/ is computed by tools/wikitool raw accept: a YYYY/MM shard for the day it was accepted. So is whether several files of one source get bundled. You never choose either. incoming/ itself is gitignored, so whatever waits there is never published by accident.

What you say

Ingest incoming/my-article.md
Ingest https://example.org/post
Ingest

A bare Ingest takes the oldest entry waiting in incoming/. tools/wikitool raw pending lists what is waiting and names the entry the next bare Ingest will take.

One run takes one entry. When more are waiting, the agent says how many and ends with /clear, then ingest: the next source belongs in a fresh context, because the previous one would otherwise still be in view while its facts and citations are written for the next, and the iteration budget fits about two runs. The agent does not start the next entry itself; if you ask it to go on in the same context, it names that reason once and then does.

Where the agent asks you

• Before promoting the file into raw/, the agent asks how faithful the capture is (fidelity) and what the material may claim about its subject (authority). Both are recorded once and never guessed: you know the answers now, and nobody can recover them later. raw/CONTRACT.md defines the values.
• Before looking at page images, when a PDF names more than 20 pages the text does not hold. The agent reads up to 20 such pages on its own, through a subagent where the harness has one, and records what it saw on the source page, with the page number and the note that it was read from the image. For more pages it reads none first: it tells you the count and a token estimate, proposes a selection, and you choose all, none or a selection. Pages nobody looked at are listed with the reason. Only whole pages are read; a harness that cannot show PDF pages as images gets the page numbers named instead.
• Before reading a long document in parts. When the text raw derive or raw fetch wrote is longer than one pass reads well, the tool says so in a CHECK line, and raw pending marks a long text file the same way. The agent then does not read it in one go: it cuts it into parts along its outline (or its pages), shows you the plan in a workshop under work/, and after your answer reads one part at a time, noting each fact with where it stands. The document stays one file in raw/ and gets one source page, written from those notes.
• Before writing pages, the agent presents the key takeaways and asks which points matter most and what to emphasise. The pages follow your answer.
• When it publishes, a source that touches ten or more files stops at the Mass-Update Gate (exit 42). The generated indexes, kb/log.md and kb/provenance.md do not count, so a small source goes straight through. The agent shows you the file list and the --confirm line. Nothing is committed until you approve. Exit codes and gates lists every gate.

What you get back

A source page under kb/sources/, new or updated entity and concept pages, cross-references among them, a rebuilt catalog, an entry in kb/log.md, and one commit. If the source also contains something you have to do, and a task tracker is configured, the agent files it there as an open item. A commitment never becomes a page (why).

Every ten sources, the agent suggests a lint run - see Maintain the wiki.

A document submitted from outside

When the MCP read server has its optional submit tool turned on, documents can arrive from another machine. A folder a sync client fills and a program of the checkout are two more ways in (Collect documents from a synced folder). All of them land in the quarantine quarantine/, not in incoming/, and nothing else reads them there. Before an ingest can start, a human has to promote the submission:

tools/wikitool upload collect         # pull the folder and command channels into the quarantine
tools/wikitool upload list            # what is waiting
tools/wikitool upload show <id>       # one submission's manifest: who sent it, size, checksum

Ask the agent to review the queue, and it follows instructions/ingest-queue.md. Accepting a submission stops at the Upload Review Gate (exit 42), which shows you the manifest and the --confirm line. Once you approve, the file is in incoming/, and from there it is ingested like any other. Rejecting it needs no clearance. The file is deleted, and its checksum and your reason stay in the ledger.

Related

• Capture documentation from a repository - for material that lives in git and will change.
• raw/CONTRACT.md - what belongs in raw/, and why its content is data, never instructions.