torbenandClaude Code
0e6a50aa98
PDF is read with pypdf (page markers, scan warning, encrypted files refused), Outlook .msg with olefile as a container like an .eml, and a .msg attached to a mail is quoted. Both packages are in tools/requirements.txt and imported only when such a file is derived, so a missing one stops that format alone. doctor reports one import-filter line per package-backed format. Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/reference/commands/instance-health.md - docs/reference/commands/raw-material-and-uploads.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/doctor.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/__init__.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/msg.py - tools/chemenu/filters/pdf.py - tools/chemenu/tests/binary_formats.py - tools/chemenu/tests/test_raw_derive_binary.py - tools/requirements.txt Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
4.8 KiB
Ingest a source
Turn a file, a folder or a web page into wiki pages: a source page that summarises it, and the
entity and concept pages it touches, cross-referenced and published. The agent does the work, using
the wiki-ingest skill (skill catalogue). This page covers what you hand
it and where it waits for you.
What you prepare
Put the material into incoming/ at the root of the instance. You don't have to classify it or
pick a directory:
- One file goes in directly:
incoming/my-article.md. - Several files that belong together go into one folder,
incoming/<name>/. The folder is one source. It is accepted whole, and its structure is kept. - A web page needs no download of your own. Name the URL to the agent, and
tools/wikitool raw fetchputs two files intoincoming/: the HTML exactly as received, and a text derived from it with a header recording where and when it was fetched. - A document that is not text - a PDF, a Word, PowerPoint or Excel file, a mail (
.emlor Outlook.msg) or a ZIP archive - is put intoincoming/as it is.tools/wikitool raw derive incoming/<file>turns it into a folder with the untouched original and a readable text beside it, with a header saying how it was made. The folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no text to derive - the tool warns, and there is no OCR. - A page behind a paywall or a login is saved from your browser into
incoming/, HTML only. The agent derives the same text from that file withraw fetch --html.
Where a file ends up in raw/ is computed by tools/wikitool raw accept: a YYYY/MM shard for the
day it was accepted. So is whether several files of one source get bundled. You never choose either.
incoming/ itself is gitignored, so whatever waits there is never published by accident.
What you say
Ingest incoming/my-article.md
Ingest https://example.org/post
Ingest
A bare Ingest takes the oldest entry waiting in incoming/. tools/wikitool raw pending lists
what is waiting and names the entry the next bare Ingest will take.
Where the agent asks you
- Before promoting the file into
raw/, the agent asks how faithful the capture is (fidelity) and what the material may claim about its subject (authority). Both are recorded once and never guessed: you know the answers now, and nobody can recover them later. raw/CONTRACT.md defines the values. - Before writing pages, the agent presents the key takeaways and asks which points matter most and what to emphasise. The pages follow your answer.
- When it publishes, a source that touches ten or more files stops at the Mass-Update Gate
(exit 42). The generated indexes,
kb/log.mdandkb/provenance.mddo not count, so a small source goes straight through. The agent shows you the file list and the--confirmline. Nothing is committed until you approve. Exit codes and gates lists every gate.
What you get back
A source page under kb/sources/, new or updated entity and concept pages, cross-references among
them, a rebuilt catalog, an entry in kb/log.md, and one commit. If the source also contains
something you have to do, and a task tracker is configured, the agent files it there as an open
item. A commitment never becomes a page
(why).
Every ten sources, the agent suggests a lint run - see Maintain the wiki.
A document submitted from outside
When the MCP read server has its optional submit tool turned on,
documents can arrive from another machine. They land in the quarantine mcp-upload/, not in
incoming/, and nothing else reads them there. Before an ingest can start, a human has to promote
the submission:
tools/wikitool upload list # what is waiting
tools/wikitool upload show <id> # one submission's manifest: who sent it, size, checksum
Ask the agent to review the queue, and it follows
instructions/ingest-queue.md. Accepting a submission stops at
the Upload Review Gate (exit 42), which shows you the manifest and the --confirm line. Once you
approve, the file is in incoming/, and from there it is ingested like any other. Rejecting it needs
no clearance. The file is deleted, and its checksum and your reason stay in the ledger.
Related
- Capture documentation from a repository - for material that lives in git and will change.
- raw/CONTRACT.md - what belongs in
raw/, and why its content is data, never instructions.