`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. - `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming//` holds the page and every subpage as a `.html` + `.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/` takes the folder as one bundle. - `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw///`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched. - Success prints the `raw accept` line for the written files and the `source_url` for the source page. **Properties** - Changes files: yes - Idempotent: no - Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole - Counts toward the session call budget: yes - Can reach the network: yes **Exit codes** - `0` - success - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - `1` - A target file already exists in `incoming/` - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB - `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming//` exists, or a path would be over the budget - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists **See also** - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why - `sites/CONTRACT.md` - the instance's own handler files - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers - `wikitool raw accept` - promotes the written files into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts ## `raw derive` Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. **Synopsis** - `wikitool raw derive incoming/ [--name ]` - Move the file into the new folder `incoming//` as `original.` and write `incoming//derived.md` - `wikitool raw derive incoming/` - Write a `.md` beside every file in the folder that has a filter **Options** ```text Arguments: path A file or a folder directly in incoming/ [required] Options: --name Folder stem for a single file instead of the file's own stem -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw derive incoming/report.docx tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz tools/wikitool raw derive incoming/meeting-files tools/wikitool raw derive incoming/scans.zip ``` **Behaviour** - Writes into `incoming/` only, never into `raw/`, and reads no network: `raw accept` promotes the folder afterwards as one source under `raw////`. - A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`. - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment. - A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original. **Properties** - Changes files: yes - Idempotent: no - Atomic: Yes for what it leaves behind - everything is read, derived and checked before the first write; the files are created exclusively, and a failure part-way puts back what this call changed - Counts toward the session call budget: yes - Can reach the network: no **Exit codes** - `0` - success - `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension - `1` - The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget - `1` - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **See also** - `raw/CONTRACT.md` "Getting a document in: `raw derive`" - the rules and why - `wikitool raw accept` - promotes the folder into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a PDF, mail or Office file to ingest starts ## `raw capture` Capture documentation from a git repository into `incoming//`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote. **Synopsis** - `wikitool raw capture --ref --path [--path ...] --name --fidelity --authority ` - First capture of a repository - `wikitool raw capture --update [--fidelity ] [--authority ]` - Capture the current state of an already-accepted bundle, for `raw accept --replaces-bundle` **Options** ```text Arguments: repo_url The repository URL (ssh://, https:// or user@host:path) Options: --ref A branch name (main) or a tag pattern (v*) --path A glob over the repository's paths; repeatable --name The bundle name - the folder in incoming/ and under raw/ --fidelity How faithful the capture is (see `wikitool types describe source`) --authority What the material may claim about its subject (see `wikitool types describe source`) --update Capture the current state of this captured bundle under raw/, from its own manifest -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting tools/wikitool raw capture --update raw/2026/10/service-docs ``` **Behaviour** - Writes into `incoming//` only, never into `raw/`: `raw accept incoming/` promotes it, as one folder, to `raw////`. - `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at. - The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter. - `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle. - Never captured, each named in the output with its reason: a file whose first line starts with `