chemenu/docs/how-to/ingest-a-source.md

# Ingest a source

Turn a file, a folder or a web page into wiki pages: a source page that summarises it, and the
entity and concept pages it touches, cross-referenced and published. The agent does the work, using
the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
it and where it waits for you.

<!-- wikitool:toc -->
## Contents

- [What you prepare](#what-you-prepare)
- [What you say](#what-you-say)
- [Where the agent asks you](#where-the-agent-asks-you)
- [What you get back](#what-you-get-back)
- [A document submitted from outside](#a-document-submitted-from-outside)
- [Related](#related)
<!-- /wikitool:toc -->

## What you prepare

Put the material into `incoming/` at the root of the instance. You don't have to classify it or
pick a directory:

- **One file** goes in directly: `incoming/my-article.md`.
- **Several files that belong together** go into one folder, `incoming/<name>/`. The folder is one
  source. It is accepted whole, and its structure is kept.
- **A web page** needs no download of your own. Name the URL to the agent, and
  `tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a
  text derived from it with a header recording where and when it was fetched.
  For the pages of a website the stack knows - Microsoft Learn, for one - the tool fetches the
  same page in the website's standard form (the English one, whatever language your link was in),
  leaves the page's menus, sign-in notice and feedback footer out of the derived text, and prints
  `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
  capture. Your instance can switch such a handler off, choose another language and add hint lines
  of its own per website; see [the configuration reference](../reference/configuration.md).
  Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
  lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
  fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
  any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
  page that offers more than the instance's `max_pages`.
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
  Outlook `.msg`) or a ZIP archive - is
  put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
  untouched original and a readable text beside it, with a header saying how it was made. The
  folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's
  sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is
  unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no
  text to derive - the tool warns, and there is no OCR. A PDF with an outline gets it in the
  header of the text, each entry with its page and the line where that page starts; text over a
  coloured background, in another colour than the running text, or struck through is marked in
  the text (`<mark data-farbe="gelb">`, `*…*`, `~~…~~`) with a legend in the header, and what a
  colour means is for the reader to take from the document itself. Pages with pictures are
  named in a `CHECK` line, because the pictures are not in the text.
- **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The
  agent derives the same text from that file with `raw fetch --html`. For a website you read often,
  sign in once instead and let the wiki fetch it with your session:
  [Fetch pages behind a login](fetch-pages-behind-a-login.md). When that session has expired the
  agent stops and asks you to run `tools/wikitool raw login <host>` - it never signs in itself.

Where a file ends up in `raw/` is computed by `tools/wikitool raw accept`: a `YYYY/MM` shard for the
day it was accepted. So is whether several files of one source get bundled. You never choose either.
`incoming/` itself is gitignored, so whatever waits there is never published by accident.

## What you say

```text
Ingest incoming/my-article.md
Ingest https://example.org/post
Ingest
```

A bare `Ingest` takes the oldest entry waiting in `incoming/`. `tools/wikitool raw pending` lists
what is waiting and names the entry the next bare `Ingest` will take.

## Where the agent asks you

- **Before promoting the file into `raw/`**, the agent asks how faithful the capture is
  (`fidelity`) and what the material may claim about its subject (`authority`). Both are recorded
  once and never guessed: you know the answers now, and nobody can recover them later.
  [raw/CONTRACT.md](../../raw/CONTRACT.md) defines the values.
- **Before writing pages**, the agent presents the key takeaways and asks which points matter most
  and what to emphasise. The pages follow your answer.
- **When it publishes**, a source that touches ten or more files stops at the Mass-Update Gate
  (exit 42). The generated indexes, `kb/log.md` and `kb/provenance.md` do not count, so a small
  source goes straight through. The agent shows you the file list and the `--confirm` line. Nothing is committed until
  you approve. [Exit codes and gates](../reference/exit-codes.md) lists every gate.

## What you get back

A source page under `kb/sources/`, new or updated entity and concept pages, cross-references among
them, a rebuilt catalog, an entry in `kb/log.md`, and one commit. If the source also contains
something you have to *do*, and a task tracker is configured, the agent files it there as an open
item. A commitment never becomes a page
([why](../explanation/knowledge-and-commitment.md)).

Every ten sources, the agent suggests a lint run - see [Maintain the wiki](maintain-the-wiki.md).

## A document submitted from outside

When the [MCP read server](set-up-the-mcp-server.md) has its optional `submit` tool turned on,
documents can arrive from another machine. A folder a sync client fills and a program of the
checkout are two more ways in ([Collect documents from a synced
folder](collect-from-a-synced-folder.md)). All of them land in the quarantine `quarantine/`, not in
`incoming/`, and nothing else reads them there. Before an ingest can start, a human has to promote
the submission:

```bash
tools/wikitool upload collect         # pull the folder and command channels into the quarantine
tools/wikitool upload list            # what is waiting
tools/wikitool upload show <id>       # one submission's manifest: who sent it, size, checksum
```

Ask the agent to review the queue, and it follows
[instructions/ingest-queue.md](../../instructions/ingest-queue.md). Accepting a submission stops at
the Upload Review Gate (exit 42), which shows you the manifest and the `--confirm` line. Once you
approve, the file is in `incoming/`, and from there it is ingested like any other. Rejecting it needs
no clearance. The file is deleted, and its checksum and your reason stay in the ledger.

## Related

- [Capture documentation from a repository](capture-repository-docs.md) - for material that lives
  in git and will change.
- [raw/CONTRACT.md](../../raw/CONTRACT.md) - what belongs in `raw/`, and why its content is data,
  never instructions.