`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. - `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming//` holds the page and every subpage as a `.html` + `.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/` takes the folder as one bundle. - `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw///`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies and no JavaScript unless the host is routed or its handler says so (below): a derived text under 200 characters is written, with a warning to read it before accepting. - Fetch routes: the `fetch` section of the settings sends a host's pages another way. `session` fetches them with the cookies of the session the user stored with `raw login`, still with the built-in fetch and the bytes as received; the header adds `session: ` and never the session's content. The host's site handler names how a page shows it arrived signed in: no stored session, or a page without that sign (an expired session, a teaser), stops with exit 42 and writes nothing. Any other route names an external fetcher of this checkout: run without a shell with the URL as its last argument (120 s, 25 MiB), its stdout is the `.html` byte for byte, and the header says `fetched_by: wikitool raw fetch (fetcher: )` and carries no `http_status`. A fetcher that fails is never replaced by the built-in fetch. A host no route names is fetched as before. - Rendered pages: a site handler that declares `render` has its pages fetched through a headless browser - with the session, when the host has one - and the `.html` is the page as the browser drew it, not the bytes the server sent; the header says `rendered: true`. No page is rendered without that declaration, and an external fetcher's page never is. It needs the optional browser component; without it the fetch stops with exit 42. `--scope` fetches every page of the bundle the same way, and `_scope.json` names the `session`, `fetcher` or `rendered` it used. - `--html` derives the `.md` beside a page saved to `incoming/` from a logged-in browser - the way past a paywall or a script-rendered page. Its header carries `derived` (when the text was derived) instead of `retrieved`, `final_url`, `http_status` and `content_type`, and the `.html` is left untouched. - Length: a `CHECK` line names every `.md` this run wrote - and a `text/plain` or `text/markdown` response stored as received - that is longer than 50,000 characters, the threshold of `raw derive`: more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). - Success prints the `raw accept` line for the written files and the `source_url` for the source page. **Properties** - Changes files: yes - Idempotent: no - Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole - Counts toward the session call budget: yes - Can reach the network: yes **Exit codes** - `0` - success - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - `1` - A target file already exists in `incoming/` - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB - `1` - The `fetch` section is malformed, routes the host through the session though no site handler of the host names a login page, or names a fetcher this checkout does not define - `1` - The external fetcher could not be started, timed out, ended with an exit other than 0, printed nothing or more than 25 MiB - `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming//` exists, or a path would be over the budget - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists - `42` - The host is fetched with the user's session and none is stored, or the page arrived without the sign of being signed in that its site handler names (the session expired) - `42` - The site handler declares `render` and the optional browser component is not installed **See also** - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why - `sites/CONTRACT.md` - the instance's own handler files - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers, and the `fetch` section, which routes a host - `wikitool raw login` - the user signs in once for a host routed through the session - `wikitool raw accept` - promotes the written files into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts ## `raw login` Let the user sign in to a website once, in a visible browser window, and store the session it leaves as a secret outside the working tree, for `raw fetch` to fetch that website's pages with. Only the user runs it. **Synopsis** - `wikitool raw login ` **Options** ```text Arguments: host The website's host, such as www.example.org [required] Options: -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw login www.heise.de ``` **Behaviour** - Opens the login page the host's site handler names (its `session` block) in a visible browser window and waits until the user has signed in - captcha and second factor included - and closed the window, at most 15 minutes. - Stores only the browser state the window leaves (cookies and storage), never a password, under `session_` (every run of other characters as one `_`) in the secrets file of the instance, outside the working tree and readable by its owner only. `WIKITOOL_SECRET_SESSION_` answers before the file, as for any named secret. The session is never written to 1Password. - Needs the host routed through the session first (`fetch.routes`, set with `wikitool config set`), which also gives the instance the id its secrets file is found by. - Needs the optional browser component, which the user installs with the preflight's `--with browser` option. - Prints no part of the session. Signing in again replaces the stored session. **Properties** - Changes files: yes - Idempotent: yes - Atomic: Yes - the secrets file is replaced in one rename, with every other entry kept - Counts toward the session call budget: yes - Can reach the network: yes **Exit codes** - `0` - success - `1` - The host is not routed through the session in `fetch.routes`, no enabled site handler of the host names a login page, or the `fetch` settings are malformed - `1` - The browser could not be started, or the window closed before any state could be read - `42` - The optional browser component is not installed **See also** - `wikitool raw fetch` - fetches the pages of a routed host with the stored session - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - fetching behind a login - `sites/CONTRACT.md` - a handler's `session` block - `docs/reference/configuration.md` - the `fetch` section ## `raw derive` Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. **Synopsis** - `wikitool raw derive incoming/ [--name ]` - Move the file into the new folder `incoming//` as `original.` and write `incoming//derived.md` - `wikitool raw derive incoming/` - Write a `.md` beside every file in the folder that has a filter **Options** ```text Arguments: path A file or a folder directly in incoming/ [required] Options: --name Folder stem for a single file instead of the file's own stem -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw derive incoming/report.docx tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz tools/wikitool raw derive incoming/meeting-files tools/wikitool raw derive incoming/scans.zip ``` **Behaviour** - Writes into `incoming/` only, never into `raw/`, and reads no network: `raw accept` promotes the folder afterwards as one source under `raw////`. - A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`. - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf` and `pdfminer.six`) and Outlook messages (`.msg`, `olefile`), each read by packages in `tools/requirements.txt` that are imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment. - A PDF also gets, where the document has them: an `outline` header field (one entry per outline item: title, level, page and the line of `derived.md` where that page's marker is), and markings in the text that follow the visible form only, in plain Markdown: `~~…~~` for strike-through, `**…**` for a highlighter stroke, bold type or a colour clearly unlike the running text (not under a link), `*…*` for italic. Bold, italic and colour count in running text only, never in program text, a table's grid or a shape the document repeats on its pages. A `markings` header field is the legend (at most three lines: form, places, pages; the `**` line names the highlighter colours); removing the markings gives the text the PDF has without them, and a tag never starts or ends on whitespace. A CHECK names the pages with a picture or drawing the text does not contain - not a background, a filler or a picture repeated across the document - and an `images` header field names the same pages, so that they outlast the output. An `unmatched` header field names the pages a WARN reports as left without markings because their text did not match the page layout. Both are written only when not empty, as page ranges (`6-8, 10`). A PDF without outline, markings, images and unmatched pages is derived as before. - A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. - Length: one more `CHECK` line names every `.md` this run wrote that is longer than 50,000 characters, counted in the file as written, header included - more than one ingest reads in one pass, so it is read in parts (`instructions/ingest-large-tree.md`). Characters, not tokens: they do not depend on a model's tokenizer. `raw fetch` and `raw pending` measure against the same value. - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original. **Properties** - Changes files: yes - Idempotent: no - Atomic: Yes for what it leaves behind - everything is read, derived and checked before the first write; the files are created exclusively, and a failure part-way puts back what this call changed - Counts toward the session call budget: yes - Can reach the network: no **Exit codes** - `0` - success - `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension - `1` - The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget - `1` - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **See also** - `raw/CONTRACT.md` "Getting a document in: `raw derive`" - the rules and why - `wikitool raw accept` - promotes the folder into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a PDF, mail or Office file to ingest starts ## `raw capture` Capture documentation from a git repository into `incoming//`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote. **Synopsis** - `wikitool raw capture --ref --path [--path ...] --name --fidelity --authority ` - First capture of a repository - `wikitool raw capture --update [--fidelity ] [--authority ]` - Capture the current state of an already-accepted bundle, for `raw accept --replaces-bundle` **Options** ```text Arguments: repo_url The repository URL (ssh://, https:// or user@host:path) Options: --ref A branch name (main) or a tag pattern (v*) --path A glob over the repository's paths; repeatable --name The bundle name - the folder in incoming/ and under raw/ --fidelity How faithful the capture is (see `wikitool types describe source`) --authority What the material may claim about its subject (see `wikitool types describe source`) --update Capture the current state of this captured bundle under raw/, from its own manifest -h, --help Show this message and exit. ``` **Examples** ```bash tools/wikitool raw capture ssh://git@example.org/team/service.git --ref main --path 'docs/**/*.md' --path README.md --name service-docs --fidelity verbatim --authority normative tools/wikitool raw capture https://example.org/team/lib.git --ref 'v*' --path docs --name lib-docs --fidelity verbatim --authority reporting tools/wikitool raw capture --update raw/2026/10/service-docs ``` **Behaviour** - Writes into `incoming//` only, never into `raw/`: `raw accept incoming/` promotes it, as one folder, to `raw////`. - `--ref` is a branch name (`main`) or a tag pattern (`v*`, any of `*?[`), which names the newest matching tag by version order. It resolves to one commit - for an annotated tag, the commit it points at. - The commit is fetched by ref name, shallowly, into a bare cache repository per URL under `tools/.wikitool_capture/` (gitignored). Files are read from it as blobs, never through a checkout, so they are byte-identical to the repository's - no line-ending conversion, no filter. - `--path` globs (repeatable) select files by their path in the repository, as git's `:(glob)` pathspec does: `*`, `?` and `[...]` do not cross `/`; `**` as a whole segment spans any number of directories; a glob without wildcards also takes everything below it as a directory. Files keep their repository paths inside the bundle. - Never captured, each named in the output with its reason: a file whose first line starts with `