# Raw material and uploads Bringing new material into `raw/`: fetching and capturing it, seeing what is waiting, accepting it, and reviewing what an MCP client uploaded. ## Contents - [`raw fetch`](#raw-fetch) - [`raw capture`](#raw-capture) - [`raw status`](#raw-status) - [`raw pending`](#raw-pending) - [`raw accept`](#raw-accept) - [`upload list`](#upload-list) - [`upload show`](#upload-show) - [`upload accept`](#upload-accept) - [`upload reject`](#upload-reject) ## `raw fetch` Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. **Synopsis** - `wikitool raw fetch [--name ]` - Fetch the page and write `incoming/.html` and `incoming/.md` - `wikitool raw fetch --html incoming/.html --url ` - Derive the `.md` from a page a human saved from their browser - no network access **Options** ```text Arguments: url The http(s) URL to fetch Options: --html Derive the text from this HTML file under incoming/ (saved from a browser) instead of fetching - no network access --url With --html: the page's URL, for the header and relative links --name File stem in incoming/ instead of the one computed from the URL -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw fetch https://example.org/blog/post tools/wikitool raw fetch https://example.org/ --name example-start tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post ``` **Behaviour** - Writes into `incoming/` only, never into `raw/`: `raw accept` promotes both files afterwards, in one call, as one bundle under `raw////`. - The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from. - Derived text: the content root is `
`, else the single `
`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched. - Success prints the `raw accept` line for the written files and the `source_url` for the source page. **Properties** - Changes files: yes - Idempotent: no - Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first - Counts toward the session call budget: yes - Can reach the network: yes **Exit codes** - `0` - success - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given - `1` - A target file already exists in `incoming/` - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists **See also** - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why - `wikitool raw accept` - promotes the written files into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts ## `raw capture` Capture documentation from a git repository into `incoming//`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote. **Synopsis** - `wikitool raw capture --ref --path [--path ...] --name --fidelity --authority ` - First capture of a repository - `wikitool raw capture --update [--fidelity ] [--authority ]` - Capture the current state of an already-accepted bundle, for `raw accept --replaces-bundle` **Options** ```text Arguments: repo_url The repository URL (ssh://, https:// or user@host:path) Options: --ref A branch name (main) or a tag pattern (v*) --path A glob over the repository's paths; repeatable --name The bundle name - the folder in incoming/ and under raw/ --fidelity How faithful the capture is (see `wikitool types describe source`) --authority What the material may claim about its subject (see `wikitool types describe source`) --update Capture the current state of this captured bundle under raw/, from its own manifest -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting tools/wikitool raw capture --update raw/2026/10/service-docs ``` **Behaviour** - Writes into `incoming//` only, never into `raw/`: `raw accept incoming/` promotes it, as one folder, to `raw////`. - `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at. - The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter. - `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle. - Never captured, each named in the output with its reason: a file whose first line starts with `