chemenu/docs/reference/commands/raw-material-and-uploads.md

Raw material and uploads

Bringing new material into raw/: fetching and capturing it, seeing what is waiting, accepting it, and reviewing what an MCP client uploaded.

Contents

• raw fetch
• raw login
• raw derive
• raw capture
• raw status
• raw pending
• raw accept
• upload collect
• upload list
• upload show
• upload accept
• upload reject

raw fetch

Capture a web page the user names into incoming/: the HTML as received plus a derived text, for raw accept to promote. Pages of a known website are fetched in the website's standard form and cleaned of its page furniture; a host the instance routes so is fetched with the user's own session or through an external fetcher.

Synopsis

• wikitool raw fetch <url> [--name <stem>] - Fetch the page and write incoming/<stem>.html and incoming/<stem>.md
• wikitool raw fetch <url> --scope [--name <stem>] - Fetch the page and the subpages its site handler offers, and write the folder incoming/<stem>/ - a .html + .md pair per page and the manifest _scope.json
• wikitool raw fetch --html incoming/<file>.html --url <url> - Derive the .md from a page a human saved from their browser - no network access

Options

Arguments:
  url  The http(s) URL to fetch

Options:
  --html <path>  Derive the text from this HTML file under incoming/ (saved from
                 a browser) instead of fetching - no network access
  --url <str>    With --html: the page's URL, for the header and relative links
  --name <str>   File stem in incoming/ instead of the one computed from the URL
  --scope        Also fetch the subpages the site handler offers for this page,
                 into the folder incoming/<stem>/
  -h, --help     Show this message and exit.

Examples

tools/wikitool raw fetch https://example.org/blog/post
tools/wikitool raw fetch https://example.org/ --name example-start
tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post

Behaviour

• Writes into incoming/ only, never into raw/: raw accept promotes both files afterwards, in one call, as one bundle under raw/<YYYY>/<MM>/<stem>/.
• The .html holds the response body byte for byte. The .md starts with a fixed header block (fetched_by, url, final_url, retrieved, http_status, content_type, charset, title, derived_from) followed by the derived text. A page of a website with a site handler adds site, and requested_url when the URL fetched is not the one typed.
• Character set, in this order: byte-order mark, the HTTP Content-Type charset, a <meta> declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
• Derived text: the content root is <main>, else the single <article>, else <body>; script, style, noscript, nav, header, footer, aside, form, template and svg are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
• Site handlers: for the pages of a website the stack or the instance knows (the stack's today: learn.microsoft.com; the instance's are JSON files in sites/), raw fetch <url> fetches the standard form of the URL (the locale segment becomes en-us, the view= query stays) and <url> as typed goes into the header as requested_url; url is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the .html - and prints hints that apply to this page as CHECK lines: act on one before accepting. --html --url <url> removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as <name> (instance). A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The sites section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
• --scope is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on learn.microsoft.com. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without --scope the one page is fetched as before and a CHECK line says how many subpages it offers. With it, incoming/<stem>/ holds the page and every subpage as a <name>.html + <name>.md pair - the .html the response byte for byte, the .md derived as for a single page, <name> from the subpage's URL - and _scope.json: the URL, the retrieval time and the sha256 of the .html of each page. raw accept incoming/<stem> takes the folder as one bundle.
• --scope is bounded. The robots.txt of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than sites.max_pages (200) refuses it before the first subpage is requested; sites.pause_seconds (1) is the pause before each subpage request. Both are settings of the instance tier. Everything is checked before the first write: the path budget under raw/<YYYY>/<MM>/, a folder of that name already in incoming/ or raw/. A failing request leaves incoming/ as it was.
• The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, --separated, at most 60 characters; --name overrides it.
• A text/plain or text/markdown response is stored as received as .txt/.md; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
• Only http/https, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting.
• Fetch routes: the fetch section of the settings sends a host's pages another way. session fetches them with the cookies of the session the user stored with raw login, still with the built-in fetch and the bytes as received; the header adds session: <host> and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the .html byte for byte, and the header says fetched_by: wikitool raw fetch (fetcher: <name>) and carries no http_status. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before.
• Rendered pages: a site handler that declares render has its pages fetched through a headless browser - with the session, when the host has one - and the .html is the page as the browser drew it, not the bytes the server sent; the header says rendered: true. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. --scope fetches every page of the bundle the same way, and _scope.json names the session, fetcher or rendered it used.
• --html derives the .md beside a page saved to incoming/ from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries derived (when the text was derived) instead of retrieved, final_url, http_status and content_type, and the .html is left untouched.
• Length: a CHECK line names every .md this run wrote - and a text/plain or text/markdown response stored as received - that is longer than 50,000 characters, the threshold of raw derive: more than one ingest reads in one pass, so it is read in parts (instructions/ingest-large-tree.md).
• Success prints the raw accept line for the written files and the source_url for the source page.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: Yes for what it leaves behind - one or two new files in incoming/, created exclusively; a failure on the second removes the first. With --scope: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
• Counts toward the session call budget: yes
• Can reach the network: yes

Exit codes

• 0 - success
• 1 - The URL is not http/https (also after a redirect), or neither or both of <url> and --html were given
• 1 - The sites section of the settings is malformed, disabled names a handler that does not exist, or a file in sites/ cannot be used and names this URL's host (or none can be read)
• 1 - A target file already exists in incoming/
• 1 - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
• 1 - The fetch section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define
• 1 - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB
• 1 - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than sites.max_pages, robots.txt forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder incoming/<stem>/ exists, or a path would be over the budget
• 1 - raw fetch --html: --url is missing, the file is not directly in incoming/ or does not exist, or incoming/<stem>.md already exists
• 42 - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired)
• 42 - The site handler declares render and the optional browser component is not installed

See also

• raw/CONTRACT.md "Getting a URL in: raw fetch" - the rules and why
• sites/CONTRACT.md - the instance's own handler files
• docs/reference/configuration.md - the sites section, which tunes the site handlers, and the fetch section, which routes a host
• wikitool raw login - the user signs in once for a host routed through the session
• wikitool raw accept - promotes the written files into raw/
• instructions/wiki-ingest/SKILL.md - where a URL to ingest starts

raw login

Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for raw fetch to fetch that website's pages with. Only the user runs it.

Synopsis

• wikitool raw login <host>

Options

Arguments:
  host  The website's host, such as www.example.org  [required]

Options:
  -h, --help  Show this message and exit.

Examples

tools/wikitool raw login www.heise.de

Behaviour

• Opens the login page the host's site handler names (its session block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes.
• Stores only the browser state the window leaves (cookies and storage), never a password, under session_<host> (every run of other characters as one _) in the secrets file of the instance, outside the working tree and readable by its owner only. WIKITOOL_SECRET_SESSION_<HOST> answers before the file, as for any named secret. The session is never written to 1Password.
• Needs the host routed through the session first (fetch.routes, set with wikitool config set), which also gives the instance the id its secrets file is found by.
• Needs the optional browser component, which the user installs with the preflight's --with browser option.
• Prints no part of the session. Signing in again replaces the stored session.

Properties

• Changes files: yes
• Idempotent: yes
• Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept
• Counts toward the session call budget: yes
• Can reach the network: yes

Exit codes

• 0 - success
• 1 - The host is not routed through the session in fetch.routes, no enabled site handler of the host names a login page, or the fetch settings are malformed
• 1 - The browser could not be started, or the window closed before any state could be read
• 42 - The optional browser component is not installed

See also

• wikitool raw fetch - fetches the pages of a routed host with the stored session
• raw/CONTRACT.md "Getting a URL in: raw fetch" - fetching behind a login
• sites/CONTRACT.md - a handler's session block
• docs/reference/configuration.md - the fetch section

raw derive

Derive reading text from a PDF, e-mail, Office file or web page in incoming/, or unpack a ZIP, and leave the original beside it, for raw accept to promote.

Synopsis

• wikitool raw derive incoming/<file> [--name <stem>] - Move the file into the new folder incoming/<stem>/ as original.<ext> and write incoming/<stem>/derived.md
• wikitool raw derive incoming/<folder> - Write a <stem>.md beside every file in the folder that has a filter

Options

Arguments:
  path  A file or a folder directly in incoming/  [required]

Options:
  --name <str>  Folder stem for a single file instead of the file's own stem
  -h, --help    Show this message and exit.

Examples

tools/wikitool raw derive incoming/report.docx
tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz
tools/wikitool raw derive incoming/meeting-files
tools/wikitool raw derive incoming/scans.zip

Behaviour

• Writes into incoming/ only, never into raw/, and reads no network: raw accept promotes the folder afterwards as one source under raw/<YYYY>/<MM>/<stem>/.
• A single file directly in incoming/ becomes a folder named after its stem (or --name) holding original.<ext> - the original bytes, moved - and derived.md. The fixed names keep the path inside the path budget; the header of derived.md records the file's own name as original_name.
• A folder is derived in place: each file with a filter gets <stem>.md beside it, nothing is renamed, and the files without a filter are named in the output.
• derived.md starts with a fixed header (derived_by, filter, engine, derived, derived_from, original_name for a renamed file, title, then the fields of the format) followed by the derived text. The same bytes always give the same text; only derived differs between runs. engine names the tool version and, where a package reads the format, its version.
• Formats: web pages (.html), Word (.docx), PowerPoint (.pptx), Excel (.xlsx, .xlsm), e-mail (.eml) and ZIP archives (.zip), read with the standard library only; PDF (.pdf, pypdf and pdfminer.six) and Outlook messages (.msg, olefile), each read by packages in tools/requirements.txt that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; wikitool doctor shows one import-filter-<name> line per package-backed format. A PDF gets <!-- page N --> markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A .msg is a container like an .eml; an embedded message inside it is named, not read. The text of a mail (.eml and .msg alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a multipart/alternative is not an attachment.
• A PDF also gets, where the document has them: an outline header field (one entry per outline item: title, level, page and the line of derived.md where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: ~~…~~ for strike-through, **…** for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), *…* for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A markings header field is the legend (at most three lines: form, places, pages; the ** line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document - and an images header field names the same pages, so that they outlast the output. An unmatched header field names the pages a WARN reports as left without markings because their text did not match the page layout. Both are written only when not empty, as page ranges (6-8, 10). A PDF without outline, markings, images and unmatched pages is derived as before.
• A workbook (.xlsx, .xlsm) gets its cells as files: beside its .md a folder <md stem>.sheets/ (derived.sheets/ for a single file) holds one NN-<sheet>.tsv per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, NN-<sheet>.formulas.tsv listing cell, formula and last stored value. The .md is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named shared:<master cell> with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
• A ZIP and a mail are containers. A ZIP at the top of incoming/ becomes the folder incoming/<stem>/ holding its files under their own names, each with a <stem>.md where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with original.eml, derived.md and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in derived.md, not unpacked. A ZIP inside a ZIP, an absolute or .. path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (__MACOSX/, .DS_Store, anything starting with a dot) are named, not written. A name taken twice, or reserved (original.* and derived.md at the top of a mail's folder, _capture.json anywhere), gets -2, -3 before its extension, and the output lists each rename.
• A name already taken under raw/ is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with raw accept --replaces; there is no --replaces-bundle for it.
• After a success the output prints the warnings the filters raised, at most three CHECK lines per format that was derived in this run - what to look at before the text is trusted - and the raw accept line for the result.
• Length: one more CHECK line names every .md this run wrote that is longer than 50,000 characters, counted in the file as written, header included - more than one ingest reads in one pass, so it is read in parts (instructions/ingest-large-tree.md). Characters, not tokens: they do not depend on a model's tokenizer. raw fetch and raw pending measure against the same value.
• Never edit derived.md by hand: a better derivation is a new raw derive of the original.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: Yes for what it leaves behind - everything is read, derived and checked before the first write; the files are created exclusively, and a failure part-way puts back what this call changed
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success
• 1 - The path is not directly in incoming/, does not exist, or no filter reads the file's extension
• 1 - The target folder, a .md or a workbook's .sheets folder already exists, or the path would be over the path budget
• 1 - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed

See also

• raw/CONTRACT.md "Getting a document in: raw derive" - the rules and why
• wikitool raw accept - promotes the folder into raw/
• instructions/wiki-ingest/SKILL.md - where a PDF, mail or Office file to ingest starts

raw capture

Capture documentation from a git repository into incoming/<bundle>/, with a manifest naming the repository, ref rule and commit, for raw accept to promote.

Synopsis

• wikitool raw capture <repo-url> --ref <branch|tag-pattern> --path <glob> [--path <glob> ...] --name <bundle> --fidelity <v> --authority <v> - First capture of a repository
• wikitool raw capture --update <raw-bundle> [--fidelity <v>] [--authority <v>] - Capture the current state of an already-accepted bundle, for raw accept --replaces-bundle

Options

Arguments:
  repo_url  The repository URL (ssh://, https:// or user@host:path)

Options:
  --ref <str>        A branch name (main) or a tag pattern (v*)
  --path <str>       A glob over the repository's paths; repeatable
  --name <str>       The bundle name - the folder in incoming/ and under raw/
  --fidelity <str>   How faithful the capture is (see `wikitool types describe
                     source`)
  --authority <str>  What the material may claim about its subject (see
                     `wikitool types describe source`)
  --update <path>    Capture the current state of this captured bundle under
                     raw/, from its own manifest
  -h, --help         Show this message and exit.

Examples

tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative
tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting
tools/wikitool raw capture --update raw/2026/10/service-docs

Behaviour

• Writes into incoming/<bundle>/ only, never into raw/: raw accept incoming/<bundle> promotes it, as one folder, to raw/<YYYY>/<MM>/<bundle>/.
• --ref is a branch name (main) or a tag pattern (v*, any of *?[), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at.
• The commit is fetched by ref name, shallowly, into a bare cache repository per URL under tools/.wikitool_capture/ (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter.
• --path globs (repeatable) select files by their path in the repository, as git's :(glob) pathspec does: *, ? and [...] do not cross /; ** as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle.
• Never captured, each named in the output with its reason: a file whose first line starts with <!-- wikitool:export or a leading frontmatter block with a wikitool_export: key, a symlink, a submodule, a file over 25 MiB, a path with a segment starting with ., a file named _capture.json, and a Git LFS pointer.
• incoming/<bundle>/_capture.json records schema, repo, ref, commit, paths, captured (UTC), fidelity, authority and files. It is the only declaration that and how this instance follows the repository.
• Only ssh://, https:// and the scp form user@host:path; a URL with a password or token in it is refused, since the manifest is committed. Git runs with the host's own credentials and configuration, restricted to those protocols, and never prompts: a repository that asks for credentials is reported as unreachable. Each git call times out after 120 s.
• --update <raw-bundle> takes URL, ref rule and globs from that bundle's manifest and writes the current state to incoming/<same bundle name>/; --fidelity/--authority default to the manifest's values, and passing one is the way to correct it.
• Refused before anything is written: incoming/<bundle> already exists, the name is already taken under raw/ (a first capture), nothing matches the globs, or a target path - measured at its later place under raw/ - is over the path budget.
• Success prints the commit and the raw accept line that comes next.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: Yes for what it leaves in incoming/ - the bundle is written into a hidden staging directory and renamed into place in one step; a failure removes the staging directory. The git cache under tools/.wikitool_capture/ is updated either way
• Counts toward the session call budget: yes
• Can reach the network: yes

Exit codes

• 0 - success
• 1 - The URL is not ssh://, https:// or user@host:path, or carries a password; the --ref rule, a --path glob or --name is unusable; or an argument of the other variant was mixed in
• 1 - --fidelity/--authority is missing on a first capture, or names unknown or a value outside the schema's enum
• 1 - The repository could not be reached, asked for credentials, or timed out; or no branch or tag matches --ref
• 1 - incoming/<bundle> already exists, the name is already taken under raw/, nothing matches the globs, or a path would be over the budget
• 1 - raw capture --update: The path is not a bundle under raw/ with a readable _capture.json, or its manifest names a URL that is refused

See also

• raw/CONTRACT.md "Getting a repository in: raw capture" - the rules and why
• wikitool raw accept - promotes the bundle; --replaces-bundle for a new edition
• wikitool raw status - which captured bundles have fallen behind their repository

raw status

Report which captured bundles under raw/ have fallen behind their repository, with each changed file as A/M/D.

Synopsis

• wikitool raw status [--json]

Options

  --json      Print one JSON object per captured bundle
  -h, --help  Show this message and exit.

Examples

tools/wikitool raw status
tools/wikitool raw status --json

Behaviour

• Reads every _capture.json under raw/ and resolves its ref rule with git ls-remote. Only where the commit moved is the new commit fetched, and compared - with the same globs and exclusions raw capture applies - against the bundle's files under raw/.
• A bundle whose commit moved without a change inside its globs is reported as unchanged, not listed. A tag pattern follows new matching tags only, never new commits on a branch.
• Each changed bundle prints old -> new and its files as A/M/D, grouped by owning source page with its citing pages, followed by the raw capture --update and raw accept --replaces-bundle lines that take the new edition in.
• An unreachable repository, a refused URL or an unreadable manifest is one line beside the others, never an abort; the exit status is 0 as long as the command itself ran.
• --json prints one object per bundle: bundle, repo, ref, old, new, changed, files ([{path, status}]), error (or null).
• Git runs exactly as for raw capture: the host's credentials, only ssh/https, never a prompt, 120 s per call.

Properties

• Changes files: no
• Idempotent: yes
• Atomic: Read-only for the tree - writes nothing but the git cache under tools/.wikitool_capture/
• Counts toward the session call budget: yes
• Can reach the network: yes

Exit codes

• 0 - success
• 0 - A repository could not be reached, its manifest or URL was refused, or no ref matches its rule

See also

• wikitool raw capture - --update captures the new edition
• wikitool raw accept - --replaces-bundle takes it in
• raw/CONTRACT.md "Getting a repository in: raw capture"

raw pending

List what waits in incoming/, oldest first, and name the entry an ingest without an argument takes next.

Synopsis

• wikitool raw pending [--json]

Options

  --json      Print the candidates as JSON
  -h, --help  Show this message and exit.

Examples

tools/wikitool raw pending
tools/wikitool raw pending --json

Behaviour

• A candidate is a top-level entry of incoming/: a single file, a bundle of top-level files sharing a stem (a raw fetch pair), or a folder with every file below it. Dotfiles, empty directories and their contents are none; quarantine/ is outside incoming/ and never listed.
• Order: oldest first by modification time. A bundle or folder counts as new as its newest file; a tie goes by name. The mtime is when a document last changed only if it was copied with its timestamps kept (cp -p, rsync -a, an unpacked archive) - for a download or a raw fetch it is merely when it was dropped.
• Each candidate shows its path(s), kind, file count and mtime, and whether raw accept would take it as it stands - the same checks, minus --fidelity/--authority. One it would refuse is listed with the reason and skipped: it needs a human.
• The default is the first candidate raw accept would take, and the output names it.
• --json prints the same candidates in the same order: kind, paths, files, mtime, acceptable, reason, default, hint, length.
• A lone file that a raw derive filter reads (a PDF, mail, Office file or web page) is listed with the raw derive line for it, in text output and as hint in JSON (null for every other candidate). The hint changes nothing about the order or the default.
• A lone .md or .txt file longer than the length threshold of raw derive and raw fetch is listed with its length in characters, in text output and as length in JSON (null for every other candidate) - that text reaches a session with no derivation in front of it to print the length CHECK. Like the hint, it changes nothing about the order or the default.
• Reads directory listings and lstat, and a file's content only to measure a lone .md/.txt whose size in bytes is over the threshold - a UTF-8 text has no more characters than bytes; an empty incoming/ is exit 0 with nothing to do.

Properties

• Changes files: no
• Idempotent: yes
• Atomic: Read-only
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success

See also

• wikitool raw accept - promotes the chosen candidate
• instructions/wiki-ingest/SKILL.md - ingest without an argument starts here
• raw/CONTRACT.md "Getting a file in: incoming/" - candidates and order, and why

raw accept

Promote one or more files, or one folder, from incoming/ into raw/.

Synopsis

• wikitool raw accept <file> [<file> ...] --fidelity <v> --authority <v> [--page "<Title>"] [--dry-run] - Promote one or more files from incoming/ into today's raw/<YYYY>/<MM>/ shard
• wikitool raw accept incoming/<folder> --fidelity <v> --authority <v> [--dry-run] - Promote a whole folder as one source, its structure kept, into raw/<YYYY>/<MM>/<folder>/
• wikitool raw accept <file> --replaces <raw-path> [--fidelity <v>] [--authority <v>] [--dry-run] - Overwrite one existing raw file in place with a new edition
• wikitool raw accept incoming/<bundle> --replaces-bundle <raw-bundle> [--dry-run] - Replace a captured bundle as a whole with the new edition raw capture --update wrote

Options

Arguments:
  files...  One or more files directly in incoming/, all belonging to the same
            source - or one folder in incoming/, accepted whole as one source
            [required]

Options:
  --fidelity <str>          How faithful the capture is (see `wikitool types
                            describe source`). Required unless --replaces is
                            given, where it is the optional, sanctioned way to
                            correct an already-set value
  --authority <str>         What the material may claim about its subject (see
                            `wikitool types describe source`). Required unless
                            --replaces is given, where it is the optional,
                            sanctioned way to correct an already-set value
  --page <str>              Extend this existing source page's raw_files: with
                            the promoted file(s), folding in its already-
                            promoted file if this raises it past one
  --replaces <path>         Overwrite this existing raw/ file in place with the
                            single incoming/ file passed alongside it - the only
                            sanctioned way past the stem-uniqueness rule below
  --dry-run                 List what would move without writing
  --replaces-bundle <path>  Replace this captured bundle under raw/ as a whole
                            with the new edition in the incoming/ folder passed
                            alongside it (written by raw capture --update)
  -h, --help                Show this message and exit.

Examples

tools/wikitool raw accept incoming/docker-cheatsheet.md --fidelity verbatim --authority reporting
tools/wikitool raw accept incoming/part-2.md --fidelity verbatim --authority reporting --page "Source - Docker Cheatsheet"
tools/wikitool raw accept incoming/projekt-x --fidelity verbatim --authority reporting
tools/wikitool raw accept incoming/cluster.md --replaces raw/documents/cluster.md
tools/wikitool raw accept incoming/chemenu --replaces-bundle raw/2026/10/chemenu

Behaviour

• Promotes files from incoming/ into raw/<YYYY>/<MM>/, computed from the accept date rather than chosen by hand. A file argument must sit directly in incoming/; a file inside a subdirectory is refused, since a subdirectory is a source of its own.
• A folder argument (incoming/<folder>) is one source: every file below it moves to raw/<YYYY>/<MM>/<folder>/ at the same relative path, and the directories left empty are removed - incoming/<folder> no longer exists afterwards. The folder name is the bundle name; two README.md in different subfolders are no conflict.
• A folder is accepted alone - no other argument, no --page, no --replaces - with one --fidelity/--authority pair for all of it. It is refused, before anything moves, if it is empty, or if a hidden entry (name starting with .), a symlink or a special file sits anywhere below it; the refusal names each one.
• One file promoted alone lands with no directory of its own; several files in one call nest under raw/<YYYY>/<MM>/<stem>/, named after the first file's stem.
• --fidelity/--authority are required on a plain accept (types describe source lists the values); unknown is refused - it is backfill-only.
• --page "<Title>" additionally extends that existing source page's raw_files: in the same call and writes both capture fields onto it - refused if it already carries a different value, since a capture field is fixed once.
• If --page raises the page past one file, its already-promoted file is folded into a bundle at its own parent directory, not today's shard, so a bundle never mixes an old and a new capture date - after checking that file has no other owner.
• Every name occupied anywhere under raw/ - file stems and bundle directory names alike, old type directories and date shards together - stays unique: a promote whose target name already belongs to something this call does not itself own is refused, naming --replaces and renaming in incoming/ as the two routes, without recommending either.
• --replaces <raw-path> overwrites that file in place with the single incoming file (same filename required), leaves every page's raw_files: untouched and writes no kb/ page; the previous edition survives only in git log --follow <raw-path>.
• With --replaces, --fidelity/--authority are optional, and passing one overwrites the owning page's already-set value - the one path the fixed-once rule does not block.
• --replaces refuses a target with more than one owning source page; with none, it replaces anyway and says so. It cannot be combined with --page or with more than one incoming file.
• --replaces prints the source page (if any) and its citing pages, so their update lands in the same commit as the replacement.
• A file already at its computed destination is what "already exists" reports, not a partial prior run to resume - safe to retry as-is once a cause is fixed.
• A folder carrying _capture.json at its top is a captured bundle (raw capture): --fidelity/--authority come from that manifest only and are refused on the command line; the folder must hold exactly the files the manifest lists. _capture.json goes in no raw_files:.
• --replaces-bundle <raw-bundle> replaces a captured bundle under raw/ with the new edition in incoming/<bundle> at the same address: files gone from the new edition are removed, emptied directories rmdired, and afterwards the bundle holds exactly the new manifest's files plus _capture.json. raw_files: is left untouched; an owning source page's fidelity/authority is overwritten where the new manifest differs.
• --replaces-bundle prints each changed file as A/M/D, grouped by owning source page with its citing pages, and the touch --page ... --add/--remove raw_files= lines that follow; git diff on the bundle shows the M and D files, the A files appear only in that list because they are still untracked.
• --replaces and --page refuse a target inside a captured bundle; the refusal names raw capture --update and --replaces-bundle. A file named _capture.json anywhere but at a captured folder's top is refused - the name is reserved.
• --dry-run reports the moves without making them.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: raw accept: No - one filesystem move per file, then (with --page) one page write. With a folder: No - one move per file, then one rmdir per emptied directory; a half-accepted folder is not resumed. raw accept --replaces: No - one unlink() + one rename(), plus (if --fidelity/--authority was given) one page write. raw accept --replaces-bundle: No - one unlink() per removed file, one rename() per added or modified file, then the manifest, then one rmdir per emptied directory, plus one page write per owning page whose capture fields change; a half-replaced bundle is not resumed
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success
• 1 - raw accept: A file does not exist or is not directly in incoming/; two files in one call share a filename; a target path already exists; or a target path would be over the path budget (160 UTF-16 code units below the instance root)
• 1 - raw accept incoming/: The folder is not directly in incoming/, is combined with another argument, --page or --replaces, holds no file, or holds a hidden entry, a symlink or a special file; a target path is over the budget; or the folder name is already occupied under raw/
• 1 - A file - given directly, inside a folder, or as --replaces/--replaces-bundle material - is an export of this wiki: it carries the export marker (a first line starting with <!-- wikitool:export, or a wikitool_export: key in a leading frontmatter block), or it is byte for byte a file in the manifest of a configured export target
• 1 - raw accept: --fidelity/--authority is missing, or names unknown or a value outside the schema's enum
• 1 - raw accept: The target name is already occupied anywhere under raw/ by something the call does not own
• 1 - raw accept: --page names an unknown page or one with no raw_files: yet, an existing raw_files: entry is missing on disk, a file to be moved has more than one owning page, or --page would overwrite an already-set fidelity/authority with a different value
• 1 - raw accept --replaces: More than one incoming file, or --page also given
• 1 - raw accept --replaces: The incoming file does not exist or is not directly in incoming/, its filename differs from the target's, or the target does not lie under raw/ or does not exist
• 1 - raw accept --replaces: --fidelity/--authority names unknown or a value outside the schema's enum, or the target has more than one owning source page
• 1 - raw accept --replaces / --page: The target, or a file the page already has, lies in a captured bundle
• 1 - raw accept incoming/: A captured folder is given --fidelity/--authority, its _capture.json cannot be read or names an invalid value, or its files differ from the ones the manifest lists
• 1 - raw accept --replaces-bundle: Either side has no _capture.json, the two manifests name different repositories, the folder's name differs from the bundle's, --page, --replaces, --fidelity or --authority was also given, the folder does not match its manifest, a new path is over the budget, or a file of the bundle has more than one owning source page

See also

• raw/CONTRACT.md "Getting a file in: incoming/" - the rules and why
• wikitool raw pending - what is waiting in incoming/, and which entry is next
• wikitool raw capture - writes a captured bundle, or its new edition, into incoming/
• wikitool types describe source - the capture field values
• wikitool new source - the source page for a promoted file

upload collect

Pull material from the configured folder and command intake channels into the quarantine - never past it.

Synopsis

• wikitool upload collect [<channel>...]

Options

Arguments:
  names...  Channels to collect from; all pulled channels when none is named

Options:
  -h, --help  Show this message and exit.

Examples

tools/wikitool upload collect
tools/wikitool upload collect onedrive

Behaviour

• Without arguments, collects from every channel of the intake section whose handler is folder or command; with names, from those channels only. A name no channel has, or the name of the mcp channel (which is pushed, not pulled), exits 1 before anything happens.
• Everything a channel delivers lands in quarantine/<id>/ unchanged, with a manifest that names channel, handler and origin; the submitter is unknown. Nothing reaches incoming/ or raw/ - that is upload accept, behind its gate, and an agent may run collect because collecting gives no trust.
• folder: only the top level of path. A file is taken once its modification time and, where the platform has it, its creation time are older than settle_seconds, its extension and size fit, and its name passes the filename rules. It is removed from the folder only after its copy is fully written and the source still has the size and sha256 that were read; if it changed meanwhile, nothing is kept and it stays. A file whose sha256 already waits in the quarantine is reported as a duplicate and removed from the folder, because its content is kept. Subdirectories, hidden files (names starting with . or ~$, or the hidden or system attribute on Windows), symbolic links, other extensions and oversized files are left untouched and reported.
• command: the program of commands.<name> runs without a shell, with stdin closed, a 300 s timeout and an empty staging directory under quarantine/.staging/ as its last argument. On exit 0 each regular file at the top of that directory with an allowed extension and size becomes a submission; on anything else nothing is taken and the end of stderr is printed. The staging directory is always removed.
• folder never empties backup or export data: a path that holds a .chemenu-backup.json (a machine's folder of the backup, any machine's) or a .wikitool-export.json (an export), is a bare git repository, or lies in one of these, is that channel's error - nothing is read and nothing removed. A path above such folders is fine, since only its top level is read.
• Channels are independent: when one fails the others still run, and the exit status is 1 if any channel or file failed. With no pulled channel it prints nothing to collect and exits 0.
• Exits 1, before it does anything, while an mcp-upload/ of an earlier version still holds submissions, and when the intake section is malformed.
• Not idempotent: a second run finds the folder emptied.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: No - per file: one copy into quarantine/<id>/ (manifest last), then one removal of the source; a channel's failure does not stop the others
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success
• 1 - Unknown channel name, an mcp channel as argument, a malformed intake section, or an mcp-upload/ that still holds submissions
• 1 - A channel or a file failed (folder missing, program exited non-zero, timed out or could not be started, a source that could not be removed)

See also

• wikitool upload list - what waits for review afterwards
• wikitool upload accept - the gate out of the quarantine
• docs/how-to/collect-from-a-synced-folder.md

upload list

List every submission currently waiting in the quarantine (quarantine/), whatever channel it came through.

Synopsis

• wikitool upload list [--json]

Options

  --json      Print every waiting submission as JSON
  -h, --help  Show this message and exit.

Examples

tools/wikitool upload list
tools/wikitool upload list --json

Behaviour

• Lists every submission waiting in quarantine/, oldest id first: id, filename, size, channel, submitter.
• Only ever non-empty when the intake section of .wikitool-local.json declares a channel.
• Exits 1, before it does anything, while an mcp-upload/ of an earlier version still holds submissions - the message names the command that moves it.
• A submission directory with a corrupt manifest is silently skipped.
• Never fails; read-only and safe to retry freely.

Properties

• Changes files: no
• Idempotent: yes
• Atomic: Read-only
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success

See also

• wikitool upload show - one submission's manifest
• docs/how-to/set-up-the-mcp-server.md § "Optional: the submit path"

upload show

Print one submission's manifest in full.

Synopsis

• wikitool upload show <id> [--json]

Options

Arguments:
  submission_id  A submission id from `upload list`  [required]

Options:
  --json
  -h, --help  Show this message and exit.

Examples

tools/wikitool upload show <id>

Behaviour

• Prints one submission's manifest in full: filename, size, sha256, submitter, submitter source (the header name or the handler, not a claim the header was honest), submission time, and where it came from: channel, handler, origin (the source file's path, command:<name> or mcp:submit).
• A submitter of unknown means the channel has no sender to trust - read channel and origin instead.
• What a reviewer reads before upload accept.
• Read-only.

Properties

• Changes files: no
• Idempotent: yes
• Atomic: Read-only
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success
• 1 - Unknown or malformed submission id

See also

• wikitool upload accept - promotes it after review
• wikitool upload reject - declines it

upload accept

Upload Review Gate: promote a submission's file from quarantine into incoming/.

Synopsis

• wikitool upload accept <id> [--confirm TOKEN]

Options

Arguments:
  submission_id  A submission id from `upload list`  [required]

Options:
  --confirm <str>  The token from a prior refusal, once a human has reviewed it
  -h, --help       Show this message and exit.

Examples

tools/wikitool upload accept <id>
tools/wikitool upload accept <id> --confirm <token>    # re-run after exit 42, once the user approved the manifest

Behaviour

• Promotes a submission's file from quarantine/<id>/ into incoming/, deletes the quarantine directory, and appends an accepted event to quarantine/ledger.jsonl.
• Upload Review Gate: without a matching --confirm, exits 42 and prints the manifest in full plus the exact re-run line - one submission at a time. The gate check runs before anything is moved.
• The token digests id, filename, size, sha256 and submitter, so an edited or superseded manifest invalidates it.
• An occupied incoming/<filename> is an ordinary validation error, not the gate.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: No - one filesystem move, one directory delete, one ledger append; the gate check runs first, before any of them
• Counts toward the session call budget: yes
• Can reach the network: no
• Gates: upload-review - see exit codes and gates

Exit codes

• 0 - success
• 1 - Unknown or malformed submission id, or the submission's file is missing from quarantine/<id>/
• 1 - incoming/<filename> already exists
• 42 - Upload Review Gate: --confirm is absent or does not match the manifest's current token

See also

• wikitool upload show - the manifest to review
• wikitool raw accept - the next step for the file in incoming/
• instructions/gates.md - the gate procedure

upload reject

Delete a submission's material, keeping only its ledger trail.

Synopsis

• wikitool upload reject <id> --reason "<why>"

Options

Arguments:
  submission_id  A submission id from `upload list`  [required]

Options:
  --reason <str>  Why this submission was declined  [required]
  -h, --help      Show this message and exit.

Examples

tools/wikitool upload reject <id> --reason "duplicate of raw/articles/llm-wiki.md"

Behaviour

• Appends an append-only rejected event naming the reason and the sha256 of what was declined, then deletes the submission's material.
• The ledger write happens first, so an interruption still leaves the reason on record.
• No gate - rejecting needs no clearance, only accepting does.
• Not idempotent: a second call with the same id reports "unknown id", which confirms the first call worked.

Properties

• Changes files: yes
• Idempotent: no
• Atomic: No - one ledger append, then one recursive delete; the ledger write happens first, so an interruption still leaves the reason on record
• Counts toward the session call budget: yes
• Can reach the network: no

Exit codes

• 0 - success
• 1 - Unknown or malformed submission id, or an empty --reason

See also

• wikitool upload show - the manifest to review