diff --git a/CHANGES.md b/CHANGES.md index b28aa49..4945f3f 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.0.0-beta.5 - 2026-10-11 - doctor: die Beschreibung der Strukturprüfung nennt sites/CONTRACT.md, die doctor seit den Site-Handlern der Instanz verlangt +## 10.0.0-beta.6 - 2026-10-11 - raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/ **Author:** Torben Nehmer @@ -30,6 +30,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. - Einstellungen in drei Stufen: wikitool.json (Instanz), .wikitool-local.json (Checkout), Secrets nur per Verweis - auch über die 1Password-CLI; neues Kommando config set - Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen - Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind +- raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/ **Low impact** - Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied @@ -46,6 +47,10 @@ Neue Einstellungen liegen in `wikitool.json` (eingecheckt, gleich in jedem Klon) Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt Hinweise als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind JSON-Dateien nach `tools/chemenu/sites/site.schema.json`: die des Stacks unter `tools/chemenu/sites/handlers/` (zuerst `learn-microsoft` und `learn-microsoft-powershell-module`), die der Instanz unter `sites/`; ein Handler der Instanz mit demselben Host und Pfad ersetzt den des Stacks. `doctor` meldet eine unbrauchbare Datei als FAIL, und `raw fetch` bricht für deren Host mit Exit 1 ab, bevor etwas geholt wird. Der Header nennt `site:` (` (instance)` bei der Instanz) und `requested_url`; das `.html` bleibt byte-genau. Der Abschnitt `sites` der Einstellungen schaltet Handler ab, wählt das Gebietsschema und nimmt Hinweiszeilen je Host-Muster auf. Regeln: `sites/CONTRACT.md`. +### `raw fetch --scope`: ein Bündel aus Index und Unterseiten + +Bietet der Site-Handler einer Seite Unterseiten an (Baustein `subpages` in der Handler-Datei; der Modulindex von learn.microsoft.com bietet seine Cmdlet-Seiten an), nennt `raw fetch` das als eine `CHECK`-Zeile, und `raw fetch --scope` holt Index und Unterseiten als einen Ordner `incoming//` mit einem Paar je Seite und dem Manifest `_scope.json`; `raw accept incoming/` nimmt ihn unverändert an. Geholt wird nur, was der Handler benennt, nie von einem anderen Host. Vorher prüft der Befehl `robots.txt` und die Obergrenze `sites.max_pages` (Standard 200); jeder Fehler bricht ab, bevor etwas in `incoming/` liegt. Zwischen den Unterseiten wartet er `sites.pause_seconds` (Standard 1). + ### Altlasten der Konfiguration bereinigt (#208) Die vier Einzeldateien `.wikitool-remotes.json`, `.wikitool-telemetry.json`, `.wikitool-upload.json` und `.wikitool-tasks.json` gibt es nicht mehr; ihre Felder sind die Abschnitte `remotes`, `telemetry`, `upload`, `tasks`, `superproductivity` und `caldav` der drei Stufen. Zugangsdaten sind dort nur noch Verweise. Der Code liest die alten Dateien nicht mehr: ein Kommando, das die Einstellung braucht, endet mit Exit 42 und nennt `tools/wikitool config migrate`, `doctor` meldet FAIL, der Telemetrie-Hook bleibt still. `config migrate` läuft einmal je Checkout, ist wiederholbar und lässt bei jedem Abbruch die alten Dateien unverändert. `WIKITOOL_TASKS_CONFIG` zeigt auf ein Profil im Format von `.wikitool-local.json`. Nur `.wikitool-tools.json` bleibt eine eigene Datei. diff --git a/VERSION b/VERSION index f24aff4..b594e8f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.0.0-beta.5 +10.0.0-beta.6 diff --git a/docs/how-to/ingest-a-source.md b/docs/how-to/ingest-a-source.md index 59059ea..92f5d29 100644 --- a/docs/how-to/ingest-a-source.md +++ b/docs/how-to/ingest-a-source.md @@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand it and where it waits for you. + +## Contents + +- [What you prepare](#what-you-prepare) +- [What you say](#what-you-say) +- [Where the agent asks you](#where-the-agent-asks-you) +- [What you get back](#what-you-get-back) +- [A document submitted from outside](#a-document-submitted-from-outside) +- [Related](#related) + + ## What you prepare Put the material into `incoming/` at the root of the instance. You don't have to classify it or @@ -22,6 +33,11 @@ pick a directory: `CHECK` lines for what it could not do for you: the agent acts on those before it accepts the capture. Your instance can switch such a handler off, choose another language and add hint lines of its own per website; see [the configuration reference](../reference/configuration.md). + Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn + lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch --scope` + fetches the page and those pages as one folder, `incoming//`, which is accepted whole like + any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a + page that offers more than the instance's `max_pages`. - **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or Outlook `.msg`) or a ZIP archive - is put into `incoming/` as it is. `tools/wikitool raw derive incoming/` turns it into a folder with the diff --git a/docs/how-to/write-a-site-handler.md b/docs/how-to/write-a-site-handler.md index 1d1111c..3acb76c 100644 --- a/docs/how-to/write-a-site-handler.md +++ b/docs/how-to/write-a-site-handler.md @@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host: website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`, `checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md). +A page that is an index - a module overview, a table of contents - can name its subpages, so that +`raw fetch --scope` fetches it and them as one bundle in `incoming//`: + +```json +"subpages": {"links": {"tag": "a", "inside": "td"}} +``` + +The links are the `` elements inside a table cell; only those on the same host and under the +page's own path count. A page with such a handler prints a `CHECK` line with their number when it +is fetched without `--scope`. + ## 3. Try it Fetch a page again, or derive a saved one without a fetch: diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index bc03fe5..7e8cd18 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -25,6 +25,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a **Synopsis** - `wikitool raw fetch [--name ]` - Fetch the page and write `incoming/.html` and `incoming/.md` +- `wikitool raw fetch --scope [--name ]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming//` - a `.html` + `.md` pair per page and the manifest `_scope.json` - `wikitool raw fetch --html incoming/.html --url ` - Derive the `.md` from a page a human saved from their browser - no network access **Options** @@ -38,6 +39,8 @@ Options: a browser) instead of fetching - no network access --url With --html: the page's URL, for the header and relative links --name File stem in incoming/ instead of the one computed from the URL + --scope Also fetch the subpages the site handler offers for this page, + into the folder incoming// -h, --help Show this message and exit. ``` @@ -46,6 +49,7 @@ Options: ```bash tools/wikitool raw fetch https://example.org/blog/post tools/wikitool raw fetch https://example.org/ --name example-start +tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post ``` @@ -56,6 +60,8 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from. - Derived text: the content root is `
`, else the single `
`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. +- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming//` holds the page and every subpage as a `.html` + `.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/` takes the folder as one bundle. +- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw///`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. @@ -66,7 +72,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - Changes files: yes - Idempotent: no -- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first +- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole - Counts toward the session call budget: yes - Can reach the network: yes @@ -77,6 +83,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - `1` - A target file already exists in `incoming/` - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB +- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming//` exists, or a path would be over the budget - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists **See also** diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 905eaaf..d06cd9c 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -413,8 +413,8 @@ Size and quota are your deployment's decision, not defaults worth copying blindl *site handler* fetches the page in the website's standard form, leaves the page furniture out of the derived text and prints hints about the page as `CHECK` lines; today there are two, both for `learn.microsoft.com`: `learn-microsoft` for every page of the host and -`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint -naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/` +`learn-microsoft-powershell-module` for a PowerShell module's overview page, which offers the cmdlet +pages it lists as subpages for `raw fetch --scope`. Handlers are JSON files: the stack's, and the instance's own under `sites/` (see [How to write a site handler](../how-to/write-a-site-handler.md)). The `sites` section is what the instance decides about the handlers, and it is optional: without it every handler applies, in the locale `en-us`. @@ -432,6 +432,9 @@ handler applies, in the locale `en-us`. once; an entry whose `path` is left out applies to the whole host. A line is plain text and is output as it is, so it is for a reminder to the reader of the capture, never for an instruction the tool should carry out. +- `max_pages` and `pause_seconds` bound `raw fetch --scope`: the most subpages one page may offer + before the fetch is refused, and the pause before each subpage request, which keeps the load on + the website low. | Field | Tier | Type | Required | Default | Description | @@ -439,6 +442,8 @@ handler applies, in the locale `en-us`. | `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. | | `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. | | `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. | +| `max_pages` | instance | positive integer | no | `200` | The most subpages `raw fetch --scope` fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted. | +| `pause_seconds` | instance | number | no | `1` | The pause, in seconds, before each subpage request of `raw fetch --scope`. `0` means none. A number written as a string is accepted. | - **Absent:** Every handler applies, with the locale `en-us` and no further hint lines. - **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers". @@ -452,6 +457,8 @@ Example: "learn-microsoft-powershell-module" ], "locale": "en-us", + "max_pages": 200, + "pause_seconds": 1, "notes": [ { "host": "learn.microsoft.com", diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index ef2f4f4..12495d3 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -270,6 +270,20 @@ resolution and what a broken file does are in [sites/CONTRACT.md](../sites/CONTR switches a handler off, sets the locale, and adds hint lines of its own per host and path pattern, printed as `CHECK` lines too - at most three per pattern. +**A page that is an index can be fetched together with its subpages.** A handler may name which +links of its page are subpages (the `subpages` field of [sites/CONTRACT.md](../sites/CONTRACT.md)); +for the module overview of `learn.microsoft.com` those are the cmdlet pages it lists. `raw fetch` +then prints one `CHECK` line with their number, and nothing more is fetched. `raw fetch +--scope` fetches the page and its subpages as one bundle - a folder `incoming//` holding a +`.html`/`.md` pair per page, written as for a single page, and `_scope.json`, a manifest of the +pages with their URL, retrieval time and the SHA-256 of the received HTML. `raw accept +incoming/` takes the folder as one source. Only the URLs the handler names are fetched, on +the host of the page and under the path the handler allows, never a link followed further. The +`robots.txt` of the host is honoured for the subpages: one it forbids, or one that cannot be read, +refuses the whole fetch. So do more subpages than `sites.max_pages`, a subpage that fails or is not +HTML, and a folder of that name in `incoming/` or `raw/`; everything is checked before the first +write, and a failure leaves `incoming/` as it was. + Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser can be longer than the 200 characters under which `raw fetch` warns. The session that reads the `.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a diff --git a/sites/CONTRACT.md b/sites/CONTRACT.md index f292249..2a7a33f 100644 --- a/sites/CONTRACT.md +++ b/sites/CONTRACT.md @@ -23,6 +23,7 @@ as `site: (instance)`. | `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` | | `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received | | `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page | +| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own | A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class` (one token of the element's classes), `attr` (an attribute that must be present, equal to `value` @@ -35,6 +36,14 @@ it applies only when at least one element matches the selector, and `{count}` is neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have the field that fills it. +`subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `` elements whose +`href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the +path of the page itself, with a trailing slash, and a value starting with `/` is a literal path. +A link is resolved against the page's URL, loses its fragment and counts only on the page's own +host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched +because of it unless the user runs `raw fetch --scope`; without it the page prints one +`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched. + `tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is refused. diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 0794080..df2d431 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -1425,13 +1425,14 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a **SYNOPSIS** - `wikitool raw fetch [--name ]` - Fetch the page and write `incoming/.html` and `incoming/.md` +- `wikitool raw fetch --scope [--name ]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming//` - a `.html` + `.md` pair per page and the manifest `_scope.json` - `wikitool raw fetch --html incoming/.html --url ` - Derive the `.md` from a page a human saved from their browser - no network access **PROPERTIES** - effect: write - idempotent: no -- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first +- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole - budget: counted - network: yes @@ -1439,6 +1440,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - `tools/wikitool raw fetch https://example.org/blog/post` - `tools/wikitool raw fetch https://example.org/ --name example-start` +- `tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope` - `tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post` **EXIT STATUS** @@ -1448,6 +1450,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - 1 The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - 1 A target file already exists in `incoming/` - 1 The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB +- 1 raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming//` exists, or a path would be over the budget - 1 raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists **ON FAILURE** @@ -1456,6 +1459,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name `, then retry once - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once +- raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming//` exists, or a path would be over the budget -> Nothing was written. Read the message: fetch the page without `--scope`, raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first **NEVER** @@ -1470,6 +1474,8 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from. - Derived text: the content root is `
`, else the single `
`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. - Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. +- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming//` holds the page and every subpage as a `.html` + `.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/` takes the folder as one bundle. +- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw///`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. diff --git a/tools/README.md b/tools/README.md index 7617e61..008f90c 100644 --- a/tools/README.md +++ b/tools/README.md @@ -118,7 +118,7 @@ tools/ kb_state.py the KB version (.wikitool-kb.json) and the migration chain corpus_diff.py invariant comparison of kb/ between two revisions web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic - sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root + sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines` guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached diff --git a/tools/chemenu/commands/raw/fetch.py b/tools/chemenu/commands/raw/fetch.py index 26d6980..7a9ff66 100644 --- a/tools/chemenu/commands/raw/fetch.py +++ b/tools/chemenu/commands/raw/fetch.py @@ -7,23 +7,38 @@ the caller is told. """ from __future__ import annotations +import hashlib +import json +import os +import shutil +import tempfile +import time +import urllib.parse from pathlib import Path -from typing import Optional +from typing import Annotated, Optional import typer from rich.markup import escape -from chemenu import cli_contract, sites, web_capture -from chemenu.commands._util import fail, rel_path, success +from chemenu import cli_contract, config, sites, web_capture +from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success from chemenu.errors import ChemenuError from chemenu.version import VersionError, read_version from chemenu.commands.raw._common import ( + _folder_collision_message, _incoming_dir, + _occupied_stems, _resolve, + _shard_dir, + _unusable_name, _validate_under_incoming, _write_exclusive, ) +SCOPE_MANIFEST = "_scope.json" +SCOPE_MANIFEST_VERSION = 1 +_LISTED_REFUSALS = 10 + _TEASER_HINT = ( " If the text stops at a teaser (paywall, login, script-rendered page): save the page from a\n" @@ -73,15 +88,22 @@ def _warn(derivation: web_capture.Derivation, md_path: Path) -> None: ) -def _plan(url: str, *, canonical: bool) -> sites.Plan: +def _preferences() -> sites.Preferences: try: - return sites.plan_for(url, sites.read_preferences(), canonical=canonical) + return sites.read_preferences() except ChemenuError as exc: fail(escape(str(exc))) -def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation) -> None: - for line in plan.checks(derivation.decoded.text): +def _plan(url: str, *, canonical: bool, preferences: Optional[sites.Preferences] = None) -> sites.Plan: + try: + return sites.plan_for(url, preferences or _preferences(), canonical=canonical) + except ChemenuError as exc: + fail(escape(str(exc))) + + +def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation, subpages: int = 0) -> None: + for line in plan.checks(derivation.decoded.text, subpages): typer.echo(f"CHECK {line}") @@ -121,6 +143,176 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: )) +def _subpage_names(stem: str, urls: list[str]) -> list[str]: + """One file stem per subpage, from its URL, unique within the bundle and + never the overview's.""" + taken = {stem} + names = [] + for url in urls: + base = web_capture.stem_for(url) + name, number = base, 1 + while name in taken: + number += 1 + name = f"{base}-{number}" + taken.add(name) + names.append(name) + return names + + +def _listed(urls: list[str]) -> str: + shown = "\n".join(f" - {url}" for url in urls[:_LISTED_REFUSALS]) + more = len(urls) - _LISTED_REFUSALS + return shown + (f"\n ... and {more} more" if more > 0 else "") + + +def _page_entry(name: str, role: str, response: web_capture.Response, retrieved: str) -> dict: + return { + "name": name, + "role": role, + "url": response.url, + "final_url": response.final_url, + "retrieved": retrieved, + "sha256": hashlib.sha256(response.body).hexdigest(), + "files": [f"{name}.html", f"{name}.md"], + } + + +def _fetch_scope(url: str, name: Optional[str]) -> None: + preferences = _preferences() + plan = _plan(url, canonical=True, preferences=preferences) + stem = name or web_capture.stem_for(plan.url) + if _unusable_name(stem): + fail(escape(f"{stem!r} is not a usable folder name - choose another with --name.")) + target = _incoming_dir() / stem + + def refuse_taken() -> None: + if os.path.lexists(target): + fail(escape( + f"{rel_path(target)} already exists in incoming/ - raw fetch never overwrites. " + "Accept or remove what is there, or pass --name ." + )) + holder = _occupied_stems(config.RAW_DIR).get(stem) + if holder is not None: + fail(escape(_folder_collision_message(stem, holder))) + + if plan.site is None or plan.site.subpages is None: + fail(escape( + f"--scope needs a site handler that offers subpages, and none applies to {plan.url}. " + "Fetch the page without --scope." + )) + refuse_taken() + + agent = _user_agent() + try: + index = web_capture.fetch( + plan.url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES + ) + except ChemenuError as exc: + fail(escape(str(exc))) + index_retrieved = web_capture.utc_now() + if index.media_type not in web_capture.HTML_TYPES: + fail(escape( + f"{plan.url} answered {index.media_type or 'no content type'}, not an HTML page - " + "--scope bundles HTML pages. Nothing was written." + )) + index_derivation = web_capture.derive_document( + index.body, index.final_url, f"{stem}.html", response=index, + drop=plan.drop, site=plan.label, requested_url=plan.requested_url, derived_at=index_retrieved, + ) + urls = plan.subpages(index_derivation.decoded.text, index.final_url) + if not urls: + fail(escape(f"{index.final_url} offers no subpages - nothing was written. Fetch it without --scope.")) + if len(urls) > preferences.max_pages: + fail(escape( + f"{index.final_url} offers {len(urls)} subpages, more than the {preferences.max_pages} " + "that `sites.max_pages` allows - nothing was fetched or written. Raise the setting " + "(`tools/wikitool config set sites.max_pages `) if the whole family is wanted." + )) + try: + forbidden = web_capture.robots_refusals(urls, agent) + except ChemenuError as exc: + fail(escape(f"{exc} Nothing was written.")) + if forbidden: + fail(escape( + f"robots.txt of {urllib.parse.urlsplit(index.final_url).netloc} forbids {len(forbidden)} " + f"of the {len(urls)} subpages - none was fetched, nothing was written:\n" + f"{_listed(forbidden)}" + )) + + names = _subpage_names(stem, urls) + files = [f"{stem}.html", f"{stem}.md", SCOPE_MANIFEST] + files += [f"{n}{ext}" for n in names for ext in (".html", ".md")] + over = [p for rel in files if (p := path_budget_problem_for(_shard_dir() / stem / rel))] + if over: + fail(escape( + f"Cannot fetch - these files would be over the path budget at their place under raw/:\n" + f"{_listed(over)}\n Choose a shorter --name. Nothing was written." + )) + + work = Path(tempfile.mkdtemp(prefix="wikitool-scope-")) + try: + bundle = work / stem + bundle.mkdir() + host = urllib.parse.urlsplit(plan.url).netloc.lower() + (bundle / f"{stem}.html").write_bytes(index.body) + (bundle / f"{stem}.md").write_bytes(index_derivation.markdown) + entries = [_page_entry(stem, "index", index, index_retrieved)] + derivations = [(stem, index_derivation)] + for position, (sub_url, sub_name) in enumerate(zip(urls, names), start=1): + if preferences.pause_seconds: + time.sleep(preferences.pause_seconds) + try: + page = web_capture.fetch( + sub_url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES + ) + except ChemenuError as exc: + fail(escape(f"{exc} (subpage {position} of {len(urls)}) Nothing was written.")) + retrieved = web_capture.utc_now() + if urllib.parse.urlsplit(page.final_url).netloc.lower() != host: + fail(escape( + f"{sub_url} was redirected to {page.final_url}, another host - --scope stays on " + f"{host}. Nothing was written." + )) + if page.media_type not in web_capture.HTML_TYPES: + fail(escape( + f"{sub_url} answered {page.media_type or 'no content type'}, not an HTML page - " + "--scope bundles HTML pages. Nothing was written." + )) + sub_plan = _plan(sub_url, canonical=False, preferences=preferences) + derivation = web_capture.derive_document( + page.body, page.final_url, f"{sub_name}.html", response=page, + drop=sub_plan.drop, site=sub_plan.label, derived_at=retrieved, + ) + (bundle / f"{sub_name}.html").write_bytes(page.body) + (bundle / f"{sub_name}.md").write_bytes(derivation.markdown) + entries.append(_page_entry(sub_name, "subpage", page, retrieved)) + derivations.append((sub_name, derivation)) + manifest = { + "scope": SCOPE_MANIFEST_VERSION, + "fetched_by": "wikitool raw fetch --scope", + "url": plan.url, + **({"requested_url": plan.requested_url} if plan.requested_url else {}), + "retrieved": index_retrieved, + "pages": entries, + } + (bundle / SCOPE_MANIFEST).write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n") + refuse_taken() + shutil.move(str(bundle), str(target)) + except OSError as exc: + fail(escape(f"Could not write {rel_path(target)}: {exc}. Nothing was left in incoming/.")) + finally: + shutil.rmtree(work, ignore_errors=True) + + for page_name, derivation in derivations: + _warn(derivation, target / f"{page_name}.md") + _echo_checks(plan, index_derivation) + success(escape( + f"Fetched {_fetched(plan)} with {len(urls)} subpage(s) to {rel_path(target)}/ " + f"({len(entries)} page pairs and {SCOPE_MANIFEST}).\n" + + _next_lines([target], plan.url) + )) + + @cli_contract.record(cli_contract.CommandRecord( path="raw fetch", summary="Capture a web page the user names into `incoming/`: the HTML as received plus a " @@ -131,6 +323,11 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: usage="raw fetch [--name ]", notes="Fetch the page and write `incoming/.html` and `incoming/.md`", ), + cli_contract.Variant( + usage="raw fetch --scope [--name ]", + notes="Fetch the page and the subpages its site handler offers, and write the folder " + "`incoming//` - a `.html` + `.md` pair per page and the manifest `_scope.json`", + ), cli_contract.Variant( usage="raw fetch --html incoming/.html --url ", notes="Derive the `.md` from a page a human saved from their browser - no network " @@ -141,7 +338,8 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: effect=cli_contract.Effect.WRITE, idempotent=cli_contract.Idempotent.NO, atomic="Yes for what it leaves behind - one or two new files in `incoming/`, created " - "exclusively; a failure on the second removes the first", + "exclusively; a failure on the second removes the first. With `--scope`: all of the folder " + "or none of it - the pages are fetched into a temporary place and moved in as a whole", budget=cli_contract.Budget.COUNTED, network=cli_contract.Network.YES, ), @@ -173,6 +371,23 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: "URL's host or no readable host, stops the fetch before anything is fetched or written. " "The `sites` section of the settings switches handlers " "off, sets the locale and adds hint lines of the instance's own per host.", + "`--scope` is for a page whose site handler offers subpages - the stack's today: a " + "PowerShell module's overview page on `learn.microsoft.com`. The handler names them from " + "the page's own HTML: the links it selects that lead to the same host, under the path of " + "the page itself (or the prefix the handler sets); no other link is followed, and a " + "subpage's own links never are. Without `--scope` the one page is fetched as before and " + "a `CHECK` line says how many subpages it offers. With it, `incoming//` holds the " + "page and every subpage as a `.html` + `.md` pair - the `.html` the response " + "byte for byte, the `.md` derived as for a single page, `` from the subpage's URL - " + "and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each " + "page. `raw accept incoming/` takes the folder as one bundle.", + "`--scope` is bounded. The `robots.txt` of the host is honoured for the subpages " + "(not for the page itself): one it forbids refuses the whole fetch and names the URL. " + "More subpages than `sites.max_pages` (200) refuses it before the first subpage is " + "requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both " + "are settings of the `instance` tier. Everything is checked before the first write: " + "the path budget under `raw///`, a folder of that name already in `incoming/` " + "or `raw/`. A failing request leaves `incoming/` as it was.", "The stem is the URL's last path segment without its extension, else its host name - " "ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.", "A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any " @@ -215,6 +430,16 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: "with the user; for a paywall or login, save the page in a browser and use " "`--html`. An unreachable host or a timeout may be retried once", ), + cli_contract.Failure( + label="raw fetch --scope", + cause="No site handler offers subpages for the URL, the page offers none or more than " + "`sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot " + "be fetched, answers something other than HTML or leaves the host, the folder " + "`incoming//` exists, or a path would be over the budget", + reaction="Nothing was written. Read the message: fetch the page without `--scope`, " + "raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then " + "retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user", + ), cli_contract.Failure( label="raw fetch --html", cause="`--url` is missing, the file is not directly in `incoming/` or does not " @@ -226,6 +451,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: examples=( "tools/wikitool raw fetch https://example.org/blog/post", "tools/wikitool raw fetch https://example.org/ --name example-start", + "tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope", "tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post", ), never=( @@ -255,6 +481,13 @@ def raw_fetch_command( name: Optional[str] = typer.Option( None, "--name", help="File stem in incoming/ instead of the one computed from the URL" ), + # `Annotated`, so that a direct call - the way the test suite calls this + # function - gets a real `False` here rather than a truthy `OptionInfo`. + scope: Annotated[bool, typer.Option( + "--scope", + help="Also fetch the subpages the site handler offers for this page, into the folder " + "incoming//", + )] = False, ): """Capture a web page into incoming/ as received HTML plus derived text. See raw/CONTRACT.md "Getting a URL in: `raw fetch`".""" @@ -263,6 +496,8 @@ def raw_fetch_command( if html is not None: if name is not None: fail("--name does not apply to --html - the stem is the saved file's own.") + if scope: + fail("--scope does not apply to --html - it fetches subpages, and --html makes no request.") _fetch_html_file(html, source_url) return if source_url is not None: @@ -270,6 +505,9 @@ def raw_fetch_command( if name is not None and (not name.strip() or name in (".", "..") or any(c in name for c in "/\\")): fail(f"--name {name!r} is not a usable file stem: no path separators, not empty.") + if scope: + _fetch_scope(url, name) + return plan = _plan(url, canonical=True) stem = name or web_capture.stem_for(plan.url) @@ -291,7 +529,7 @@ def raw_fetch_command( ) _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)]) _warn(derivation, md_path) - _echo_checks(plan, derivation) + _echo_checks(plan, derivation, len(plan.subpages(derivation.decoded.text, response.final_url))) success(escape( f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n" + _next_lines([html_path, md_path], plan.url) + "\n" diff --git a/tools/chemenu/config_shapes.py b/tools/chemenu/config_shapes.py index 4fe6bd1..823ec82 100644 --- a/tools/chemenu/config_shapes.py +++ b/tools/chemenu/config_shapes.py @@ -248,6 +248,10 @@ def _positive(value: int) -> Optional[str]: return None if value > 0 else "must be positive" +def non_negative(value: float) -> Optional[str]: + return None if value >= 0 else "must not be negative" + + def dotted_extensions(value: tuple[str, ...]) -> Optional[str]: if not value: return "must not be empty" @@ -453,8 +457,8 @@ SITES = Shape( key="sites", title="Site handlers", purpose="Tunes the site handlers `raw fetch` applies to the pages of well-known websites: " - "switches one off, picks the locale a handler fetches, and adds hint lines of the " - "instance's own per host.", + "switches one off, picks the locale a handler fetches, adds hint lines of the " + "instance's own per host, and bounds `--scope`, the fetch of a page with its subpages.", fields=( Field( "disabled", "array of strings", "The names of the site handlers that do not apply. " @@ -479,10 +483,23 @@ SITES = Shape( default=[], json_type="array", coerce=site_notes, check=site_notes_problem, tier=INSTANCE, ), + positive_integer( + "max_pages", "The most subpages `raw fetch --scope` fetches for one page. A page that " + "offers more is refused before the first subpage is requested.", + default=200, tier=INSTANCE, + ), + Field( + "pause_seconds", "number", "The pause, in seconds, before each subpage request of " + "`raw fetch --scope`. `0` means none.", + default=1, json_type="number", coerce=float, check=non_negative, + lenient="A number written as a string is accepted.", tier=INSTANCE, + ), ), examples=({ "disabled": ["learn-microsoft-powershell-module"], "locale": "en-us", + "max_pages": 200, + "pause_seconds": 1, "notes": [{ "host": "learn.microsoft.com", "path": "^/[a-z-]+/azure/", "lines": ["Check the Azure retirement notice at the top of the page."], @@ -515,6 +532,7 @@ _JSON_TYPES: dict[str, Callable[[Any], bool]] = { "string": lambda v: isinstance(v, str), "array": lambda v: isinstance(v, list), "object": lambda v: isinstance(v, dict), + "number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool), } _TYPE_NAMES = {bool: "a boolean", int: "an integer", float: "a number", str: "a string", diff --git a/tools/chemenu/sites/__init__.py b/tools/chemenu/sites/__init__.py index e639c87..1549ea8 100644 --- a/tools/chemenu/sites/__init__.py +++ b/tools/chemenu/sites/__init__.py @@ -14,7 +14,9 @@ with an optional condition. So a handler is a JSON file (`data.py` reads it, in `sites/` at its root, by the same building blocks. An instance handler for a host and path the stack has a handler for replaces it; the `sites` section of the settings still switches handlers off, sets the `locale` and adds hint lines -(`notes`). A handler may still be registered from a Python module in this +(`notes`). A handler may also offer the subpages of a page +(`Subpages`: the links to take, under a path prefix), which `raw fetch --scope` +fetches as one bundle. A handler may still be registered from a Python module in this package. The core knows neither `typer` nor `rich` and raises `ValidationError`. """ from __future__ import annotations @@ -33,6 +35,7 @@ from chemenu.web_capture import Rule MAX_CHECKS = 3 LOCALE_PLACEHOLDER = "{locale}" +PAGE_PLACEHOLDER = "{page}" INSTANCE_DIRECTORY = "sites" @@ -79,6 +82,47 @@ class Check: return text +@dataclass(frozen=True) +class Subpages: + """The subpages a page offers: the links `links` selects, resolved against + the page's URL, that stay on its host and lie under `prefix`. `{page}` in + `prefix` is the path of the page itself with a trailing slash, which is + the default; the page is never its own subpage. Document order, each URL + once, the fragment dropped - and nothing else is followed.""" + + links: Rule + prefix: str = PAGE_PLACEHOLDER + + def __post_init__(self) -> None: + if not (self.prefix.startswith(PAGE_PLACEHOLDER) or self.prefix.startswith("/")): + raise ValueError(f"a subpages prefix starts with {PAGE_PLACEHOLDER} or with /, not {self.prefix!r}") + + def urls(self, html: str, page_url: str) -> list[str]: + page = urllib.parse.urlsplit(page_url) + own = urllib.parse.unquote(page.path) or "/" + own = own if own.endswith("/") else own + "/" + prefix = self.prefix.replace(PAGE_PLACEHOLDER, own) + found: list[str] = [] + for element in web_capture.matching(web_capture.parse(html), self.links): + href = element.attrs.get("href", "").strip() + if not href: + continue + url = urllib.parse.urldefrag(urllib.parse.urljoin(page_url, href))[0] + parts = urllib.parse.urlsplit(url) + path = urllib.parse.unquote(parts.path) + if ( + parts.scheme.lower() not in web_capture.ALLOWED_SCHEMES + or parts.netloc.lower() != page.netloc.lower() + or not path.startswith(prefix) + or path.rstrip("/") == own.rstrip("/") + or path == prefix + or url in found + ): + continue + found.append(url) + return found + + @dataclass(frozen=True) class Site: """One handler. `host` is matched exactly (lower case); `path` is a regular @@ -94,6 +138,7 @@ class Site: rewrites: tuple[Rewrite, ...] = () drop: tuple[Rule, ...] = () checks: tuple[Check, ...] = () + subpages: Optional[Subpages] = None origin: str = "stack" def __post_init__(self) -> None: @@ -131,6 +176,7 @@ def inherit(candidate: Site, parent: Site) -> Site: rewrites=parent.rewrites + candidate.rewrites, drop=parent.drop + candidate.drop, checks=parent.checks + candidate.checks, + subpages=candidate.subpages or parent.subpages, ) if len(flat.checks) > MAX_CHECKS: raise ValueError( @@ -220,6 +266,8 @@ class Preferences: notes: tuple[Note, ...] = () instance: tuple[Site, ...] = () broken: tuple[Broken, ...] = () + max_pages: int = 200 + pause_seconds: float = 1.0 def read_instance(root: "Path | str | None" = None) -> tuple[tuple[Site, ...], tuple[Broken, ...]]: @@ -250,7 +298,9 @@ def read_preferences(root: "Path | str | None" = None) -> Preferences: notes = tuple( Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"] ) - return Preferences(disabled, section["locale"], notes, instance, broken) + return Preferences( + disabled, section["locale"], notes, instance, broken, section["max_pages"], section["pause_seconds"] + ) @dataclass(frozen=True) @@ -278,11 +328,23 @@ class Plan: def drop(self) -> tuple[Rule, ...]: return self.site.drop if self.site else () - def checks(self, html: str) -> list[str]: - """The `CHECK` lines for the page: the handler's, then the instance's notes.""" + def checks(self, html: str, subpages: int = 0) -> list[str]: + """The `CHECK` lines for the page: the handler's, the offer of `subpages` + subpages when there are any, then the instance's notes.""" lines = self.site.check_lines(html) if self.site else [] + if subpages: + lines.append( + f"this page is an index with {subpages} subpage(s) - rerun with `--scope` to " + "fetch them as one bundle" + ) return lines + list(self.notes) + def subpages(self, html: str, page_url: str) -> list[str]: + """The subpage URLs the handler offers for this page, else none.""" + if self.site is None or self.site.subpages is None: + return [] + return self.site.subpages.urls(html, page_url) + def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan: """The plan for `url`. `canonical=False` keeps the URL as given - for a page diff --git a/tools/chemenu/sites/data.py b/tools/chemenu/sites/data.py index 55e811d..593d349 100644 --- a/tools/chemenu/sites/data.py +++ b/tools/chemenu/sites/data.py @@ -15,7 +15,7 @@ from typing import Any, Optional from jsonschema import Draft202012Validator -from chemenu.sites import Broken, Check, Rewrite, Rule, Site, inherit +from chemenu.sites import Broken, Check, Rewrite, Rule, Site, Subpages, inherit SCHEMA_PATH = Path(__file__).with_name("site.schema.json") STACK_DIRECTORY = Path(__file__).with_name("handlers") @@ -70,6 +70,13 @@ def build(name: str, document: Any, origin: str) -> Site: if "{count}" in text and "count" not in entry: raise ValueError("checks: a text with {count} needs a `count` rule") checks.append(Check(text, entry.get("meta", ""), _rule(entry["count"]) if "count" in entry else None)) + subpages = None + if "subpages" in document: + entry = document["subpages"] + try: + subpages = Subpages(_rule(entry["links"]), entry.get("prefix", "{page}")) + except ValueError as exc: + raise ValueError(f"subpages: {exc}") from None try: return Site( name=name, @@ -80,6 +87,7 @@ def build(name: str, document: Any, origin: str) -> Site: rewrites=tuple(rewrites), drop=tuple(_rule(entry) for entry in document.get("drop", [])), checks=tuple(checks), + subpages=subpages, origin=origin, ) except (ValueError, re.error) as exc: diff --git a/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json b/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json index 0cfb4d7..472eecc 100644 --- a/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json +++ b/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json @@ -1,12 +1,9 @@ { "host": "learn.microsoft.com", - "summary": "A PowerShell module's overview page: names the cmdlet pages it lists, which are not fetched with it.", + "summary": "A PowerShell module's overview page: offers the cmdlet pages it lists as its subpages, which `--scope` fetches with it as one bundle.", "path": "/[a-z]{2,3}(?:-[a-z0-9]{2,8})+/powershell/module/[^/]+/?$", "parent": "learn-microsoft", - "checks": [ - { - "text": "This module page lists {count} cmdlet page(s) as links; they are not fetched - ingest the ones the topic needs, each with its own raw fetch.", - "count": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"} - } - ] + "subpages": { + "links": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"} + } } diff --git a/tools/chemenu/sites/site.schema.json b/tools/chemenu/sites/site.schema.json index ab7e70b..4082441 100644 --- a/tools/chemenu/sites/site.schema.json +++ b/tools/chemenu/sites/site.schema.json @@ -37,6 +37,15 @@ "count": {"$ref": "#/$defs/rule"} } } + }, + "subpages": { + "type": "object", + "additionalProperties": false, + "required": ["links"], + "properties": { + "links": {"$ref": "#/$defs/rule"}, + "prefix": {"type": "string", "pattern": "^(\\{page\\}|/)", "maxLength": 200} + } } }, "$defs": { diff --git a/tools/chemenu/tests/test_raw_fetch.py b/tools/chemenu/tests/test_raw_fetch.py index 3038caa..e2c5e84 100644 --- a/tools/chemenu/tests/test_raw_fetch.py +++ b/tools/chemenu/tests/test_raw_fetch.py @@ -100,8 +100,8 @@ def tree(kb_dir): return root -def _fetch(url=None, html=None, source_url=None, name=None): - return raw_fetch_command(url=url, html=html, source_url=source_url, name=name) +def _fetch(url=None, html=None, source_url=None, name=None, scope=False): + return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope) def _refused(**kwargs) -> None: diff --git a/tools/chemenu/tests/test_raw_fetch_scope.py b/tools/chemenu/tests/test_raw_fetch_scope.py new file mode 100644 index 0000000..1091cf8 --- /dev/null +++ b/tools/chemenu/tests/test_raw_fetch_scope.py @@ -0,0 +1,346 @@ +"""`wikitool raw fetch --scope` - a page and the subpages its site handler offers, +fetched as one bundle. + +Like `test_raw_fetch.py`, every request goes to a real `http.server` on 127.0.0.1. +The handler is an instance handler in `sites/`, because that is the form in which +the subpages building block is data an instance can write. +""" +from __future__ import annotations + +import datetime +import hashlib +import json +from pathlib import Path + +import pytest +import typer + +from chemenu import sites, web_capture +from chemenu.commands.raw.accept import raw_accept_command +from chemenu.commands.raw.fetch import raw_fetch_command +from chemenu.config_shapes import SITES, check +from chemenu.tests.conftest import write_settings +from chemenu.tests.test_raw_fetch import Site + +HANDLER = { + "host": "127.0.0.1", + "summary": "A test module: the links in its table are its subpages.", + "path": "/mod/", + "subpages": {"links": {"tag": "a", "inside": "td"}}, +} + + +def _index(port: int) -> bytes: + return ( + "Module

Module

" + '' + '' + '' + '' + '' + f'' + '' + '' + "
A
B
C
A again
outside the prefix
another host
mail
the page itself
" + '

a link outside the table

' + "
" + ).encode() + + +def _sub(name: str) -> bytes: + return ( + f"{name}

{name}

" + f"

{name} text

" + ).encode() + + +@pytest.fixture +def site(monkeypatch): + for var in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY", "all_proxy", "ALL_PROXY"): + monkeypatch.delenv(var, raising=False) + server = Site() + yield server + server.stop() + + +@pytest.fixture +def tree(kb_dir, monkeypatch, site): + root = kb_dir.parent + (root / "raw").mkdir() + (root / "incoming").mkdir(exist_ok=True) + (root / "sites").mkdir() + (root / "sites/test-module.json").write_text(json.dumps(HANDLER), encoding="utf-8") + write_settings(root, monkeypatch, sites={"pause_seconds": 0}) + return root + + +@pytest.fixture +def module(site): + """The index and the pages around it, served.""" + site.html("/mod/", _index(site.httpd.server_address[1])) + for name in ("a?v=1", "b?v=1", "c", "hidden", "foreign"): + site.html(f"/mod/{name}", _sub(name.split("?")[0])) + return site + + +def _fetch(url, name=None, scope=True): + return raw_fetch_command(url=url, html=None, source_url=None, name=name, scope=scope) + + +def _refused(url, **kwargs) -> None: + with pytest.raises(typer.Exit) as excinfo: + _fetch(url, **kwargs) + assert excinfo.value.exit_code == 1 + + +def _snapshot(root: Path) -> list[str]: + return sorted(str(p.relative_to(root)) for p in root.rglob("*")) + + +def _requested(site: Site) -> list[str]: + return [path for path, _ in site.requests] + + +def _checks(capsys) -> list[str]: + return [line[len("CHECK "):] for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")] + + +# --- without --scope -------------------------------------------------------- + + +def test_without_scope_the_page_is_fetched_alone_and_the_offer_is_one_check(tree, module, capsys): + _fetch(module.url("/mod/"), name="mod", scope=False) + + assert sorted(p.name for p in (tree / "incoming").iterdir()) == ["mod.html", "mod.md"] + assert (tree / "incoming/mod.html").read_bytes() == module.routes["/mod/"][2] + assert _requested(module) == ["/mod/"] + lines = _checks(capsys) + assert len(lines) == 1 + assert "3 subpage(s)" in lines[0] and "--scope" in lines[0] + + +# --- with --scope ----------------------------------------------------------- + + +def test_scope_writes_a_folder_with_a_pair_per_page_and_the_manifest(tree, module): + before = datetime.datetime.now(datetime.timezone.utc) + _fetch(module.url("/mod/"), name="mod") + + folder = tree / "incoming/mod" + assert sorted(p.name for p in folder.iterdir()) == [ + "_scope.json", "a.html", "a.md", "b.html", "b.md", "c.html", "c.md", "mod.html", "mod.md", + ] + assert [p.name for p in (tree / "incoming").iterdir()] == ["mod"] + for name, path in (("mod", "/mod/"), ("a", "/mod/a?v=1"), ("b", "/mod/b?v=1"), ("c", "/mod/c")): + assert (folder / f"{name}.html").read_bytes() == module.routes[path][2] + text = (folder / "a.md").read_text(encoding="utf-8") + assert "a text" in text and "derived_from: a.html" in text + + manifest = json.loads((folder / "_scope.json").read_text(encoding="utf-8")) + assert manifest["scope"] == 1 and manifest["url"] == module.url("/mod/") + assert [(p["name"], p["role"]) for p in manifest["pages"]] == [ + ("mod", "index"), ("a", "subpage"), ("b", "subpage"), ("c", "subpage"), + ] + for page in manifest["pages"]: + html = (folder / f"{page['name']}.html").read_bytes() + assert page["sha256"] == hashlib.sha256(html).hexdigest() + assert page["files"] == [f"{page['name']}.html", f"{page['name']}.md"] + stamp = datetime.datetime.strptime(page["retrieved"], "%Y-%m-%dT%H:%M:%SZ").replace( + tzinfo=datetime.timezone.utc) + assert before - datetime.timedelta(seconds=2) <= stamp <= datetime.datetime.now(datetime.timezone.utc) + assert [p["url"] for p in manifest["pages"][1:]] == [ + module.url("/mod/a?v=1"), module.url("/mod/b?v=1"), module.url("/mod/c"), + ] + + +def test_only_the_urls_the_handler_names_are_requested(tree, module): + _fetch(module.url("/mod/"), name="mod") + + paths = _requested(module) + assert sorted(paths) == sorted(["/mod/", "/robots.txt", "/mod/a?v=1", "/mod/b?v=1", "/mod/c"]) + assert not {"/mod/hidden", "/mod/foreign", "/elsewhere/x"} & set(paths) + + +def test_the_requests_identify_the_tool(tree, module): + _fetch(module.url("/mod/"), name="mod") + assert module.requests and all(agent.startswith("chemenu-wikitool/") for _, agent in module.requests) + + +def test_the_folder_is_accepted_by_raw_accept_unchanged(tree, module): + _fetch(module.url("/mod/"), name="mod") + snapshot = {p.name: p.read_bytes() for p in (tree / "incoming/mod").iterdir()} + + raw_accept_command( + files=[tree / "incoming/mod"], fidelity="published", authority="reporting", + page=None, replaces=None, dry_run=False, + ) + + accepted = [p for p in (tree / "raw").rglob("*") if p.is_dir() and p.name == "mod"] + assert len(accepted) == 1 + assert {p.name: p.read_bytes() for p in accepted[0].iterdir()} == snapshot + assert not (tree / "incoming/mod").exists() + + +def test_the_pause_is_taken_before_each_subpage(tree, module, monkeypatch): + write_settings(tree, monkeypatch, sites={"pause_seconds": 0.25}) + slept: list[float] = [] + monkeypatch.setattr("chemenu.commands.raw.fetch.time.sleep", slept.append) + _fetch(module.url("/mod/"), name="mod") + assert slept == [0.25, 0.25, 0.25] + + +# --- refusals: nothing is written -------------------------------------------- + + +def test_robots_forbidding_a_subpage_refuses_the_fetch_and_names_it(tree, module, capsys): + module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /mod/b\n") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + + assert _snapshot(tree) == before + assert module.url("/mod/b?v=1") in capsys.readouterr().out.replace("\n", "") + assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module)) + + +def test_a_robots_txt_for_another_agent_does_not_apply(tree, module): + module.routes["/robots.txt"] = ( + 200, {"Content-Type": "text/plain"}, b"User-agent: somebodyelse\nDisallow: /\n", + ) + _fetch(module.url("/mod/"), name="mod") + assert (tree / "incoming/mod/c.html").is_file() + + +def test_a_missing_robots_txt_allows_everything(tree, module): + assert "/robots.txt" not in module.routes + _fetch(module.url("/mod/"), name="mod") + assert (tree / "incoming/mod/c.html").is_file() + + +def test_a_robots_txt_that_cannot_be_read_refuses_the_fetch(tree, module, capsys): + module.routes["/robots.txt"] = (503, {}, b"later") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + assert _snapshot(tree) == before + assert "robots.txt" in capsys.readouterr().out + + +def test_robots_does_not_apply_to_the_page_without_scope(tree, module): + module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /\n") + _fetch(module.url("/mod/"), name="mod", scope=False) + assert (tree / "incoming/mod.html").is_file() + assert "/robots.txt" not in _requested(module) + + +def test_a_failing_third_fetch_leaves_incoming_as_it_was(tree, module): + module.routes["/mod/b?v=1"] = (500, {}, b"broken") + (tree / "incoming/other.md").write_text("other", encoding="utf-8") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + + assert _snapshot(tree) == before + assert "/mod/b?v=1" in _requested(module) + assert "/mod/c" not in _requested(module) + + +def test_a_subpage_that_is_not_html_refuses_the_fetch(tree, module): + module.routes["/mod/c"] = (200, {"Content-Type": "application/pdf"}, b"%PDF-1.7") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + assert _snapshot(tree) == before + + +def test_more_subpages_than_the_maximum_refuse_before_the_first_subpage(tree, module, monkeypatch, capsys): + write_settings(tree, monkeypatch, sites={"max_pages": 2}) + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + + assert _snapshot(tree) == before + assert _requested(module) == ["/mod/"] + assert "max_pages" in " ".join(capsys.readouterr().out.split()) + + +def test_a_raised_maximum_takes_effect(tree, module, monkeypatch): + write_settings(tree, monkeypatch, sites={"max_pages": 4}) + _fetch(module.url("/mod/"), name="mod") + assert (tree / "incoming/mod/c.md").is_file() + + +def test_an_existing_folder_is_never_overwritten(tree, module): + (tree / "incoming/mod").mkdir() + (tree / "incoming/mod/mine.md").write_text("mine", encoding="utf-8") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + assert _snapshot(tree) == before + assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module)) + + +def test_a_path_over_the_budget_is_refused_before_a_subpage_is_requested(tree, module): + before = _snapshot(tree) + _refused(module.url("/mod/"), name="m" * 140) + assert _snapshot(tree) == before + assert "/mod/a?v=1" not in _requested(module) + + +def test_a_page_whose_handler_offers_no_subpages_is_refused(tree, module): + module.html("/plain", _sub("plain")) + before = _snapshot(tree) + _refused(module.url("/plain"), name="plain") + assert _snapshot(tree) == before + + +def test_an_index_without_subpages_is_refused(tree, module): + module.html("/mod/", b"

No table at all, only text.

") + before = _snapshot(tree) + _refused(module.url("/mod/"), name="mod") + assert _snapshot(tree) == before + + +def test_scope_does_not_go_with_html(tree): + saved = tree / "incoming/mod.html" + saved.write_bytes(b"

x

") + with pytest.raises(typer.Exit) as excinfo: + raw_fetch_command(url=None, html=saved, source_url="http://127.0.0.1/mod/", name=None, scope=True) + assert excinfo.value.exit_code == 1 + + +# --- the building block as data --------------------------------------------- + + +def test_the_stack_handler_for_a_module_offers_its_cmdlet_pages_as_subpages(): + special = next(s for s in sites.all_sites() if s.name == "learn-microsoft-powershell-module") + assert special.subpages is not None and special.subpages.prefix == "{page}" + html = ( + '' + '
x
y
' + 'outside the table' + ) + assert special.subpages.urls(html, "https://learn.microsoft.com/en-us/powershell/module/m/?view=v") == [ + "https://learn.microsoft.com/en-us/powershell/module/m/get-x?view=v", + ] + + +@pytest.mark.parametrize("subpages", [ + {}, + {"links": {}}, + {"links": {"tag": "a"}, "prefix": "relative/"}, + {"links": {"tag": "a"}, "extra": 1}, +]) +def test_a_malformed_subpages_block_is_a_rejected_file(subpages): + from chemenu.sites import data + + with pytest.raises(ValueError): + data.build("x", {"host": "example.org", "summary": "s", "subpages": subpages}, "instance") + + +def test_a_literal_prefix_replaces_the_page_path(): + block = sites.Subpages(web_capture.Rule(tag="a"), "/docs/ref/") + html = '12self' + assert block.urls(html, "https://example.org/docs/index") == ["https://example.org/docs/ref/one"] + + +def test_the_new_settings_have_their_defaults_and_refuse_nonsense(): + assert check(SITES, {}).values["max_pages"] == 200 + assert check(SITES, {}).values["pause_seconds"] == 1 + assert check(SITES, {"max_pages": 0}).problems + assert check(SITES, {"pause_seconds": -1}).problems + assert not check(SITES, {"pause_seconds": 0}).problems diff --git a/tools/chemenu/tests/test_site_files.py b/tools/chemenu/tests/test_site_files.py index 893db1e..bbe5480 100644 --- a/tools/chemenu/tests/test_site_files.py +++ b/tools/chemenu/tests/test_site_files.py @@ -60,7 +60,7 @@ def _handler(root: Path, name: str, content) -> Path: def _fetch(url, name=None, html=None): - return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name) + return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name, scope=False) def _refused(url, **kwargs) -> None: diff --git a/tools/chemenu/tests/test_sites.py b/tools/chemenu/tests/test_sites.py index 5eb1919..c09c724 100644 --- a/tools/chemenu/tests/test_sites.py +++ b/tools/chemenu/tests/test_sites.py @@ -57,8 +57,8 @@ def served(monkeypatch): return asked -def _fetch(url=None, html=None, source_url=None, name=None): - return raw_fetch_command(url=url, html=html, source_url=source_url, name=name) +def _fetch(url=None, html=None, source_url=None, name=None, scope=False): + return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope) def _refused(**kwargs) -> None: @@ -130,13 +130,13 @@ def test_the_stem_comes_from_the_standard_form(tree, served): assert (tree / "incoming/get-oslicenseinfo.md").is_file() -def test_the_checks_name_the_source_and_the_cmdlet_pages(tree, served, capsys): +def test_the_checks_name_the_source_and_the_subpages(tree, served, capsys): _fetch(MODULE_URL, name="oslicense") lines = _checks(capsys) assert len(lines) == 2 assert SOURCE_URL in lines[0] - assert "11 cmdlet page(s)" in lines[1] + assert "index with 11 subpage(s)" in lines[1] and "--scope" in lines[1] def test_a_page_of_the_general_handler_has_only_the_source_check(tree, served, capsys): @@ -188,7 +188,7 @@ def test_html_applies_the_rules_but_keeps_the_url_as_typed(tree, monkeypatch, ca assert "requested_url" not in header assert header["site"] == "learn-microsoft-powershell-module" assert "Exit editor mode" not in text and "# OSLicense Module" in text - assert len(_checks(capsys)) == 2 + assert len(_checks(capsys)) == 1 # --- the instance's part ----------------------------------------------------- @@ -286,7 +286,8 @@ def test_a_special_handler_inherits_from_the_general_one(): assert special.drop == general.drop assert special.rewrites == general.rewrites assert special.checks[: len(general.checks)] == general.checks - assert len(special.checks) == len(general.checks) + 1 + assert special.checks == general.checks + assert special.subpages is not None and general.subpages is None def test_a_duplicate_host_and_path_is_a_registration_error(registry): diff --git a/tools/chemenu/web_capture.py b/tools/chemenu/web_capture.py index 677d29e..19e42d3 100644 --- a/tools/chemenu/web_capture.py +++ b/tools/chemenu/web_capture.py @@ -25,6 +25,7 @@ import mimetypes import re import time import unicodedata +import urllib.robotparser import urllib.error import urllib.parse import urllib.request @@ -66,6 +67,14 @@ class Response: return media_type(self.content_type) +class HttpError(BackendError): + """A server answered with an error status; `status` is the number.""" + + def __init__(self, message: str, status: int): + super().__init__(message) + self.status = status + + def media_type(content_type: str) -> str: if not content_type.strip(): return "" @@ -140,7 +149,7 @@ def fetch( body=b"".join(chunks), ) except urllib.error.HTTPError as exc: - raise BackendError(f"{url} answered HTTP {exc.code} {exc.reason}.") from exc + raise HttpError(f"{url} answered HTTP {exc.code} {exc.reason}.", exc.code) from exc except urllib.error.URLError as exc: if isinstance(exc.reason, TimeoutError): raise BackendError(f"{url} did not answer within {timeout:g} s.") from exc @@ -598,8 +607,12 @@ def prune(root: _Element, rules: "tuple[Rule, ...]") -> None: root.children = kept +def matching(document: _Element, rule: Rule) -> "list[_Element]": + return [element for element in document.iter() if rule.matches(element)] + + def count_matches(document: _Element, rule: Rule) -> int: - return sum(1 for element in document.iter() if rule.matches(element)) + return len(matching(document, rule)) def meta_content(document: _Element, name: str) -> str: @@ -622,6 +635,33 @@ def derive_text(html: str, base_url: str, drop: "tuple[Rule, ...]" = ()) -> str: return _join(blocks) +# --- robots.txt ------------------------------------------------------------- + + +def robots_refusals( + urls: "list[str]", user_agent: str, timeout: float = TIMEOUT_SECONDS, max_bytes: int = MAX_BYTES +) -> "list[str]": + """The URLs among `urls` (all of one host) that the host's `robots.txt` + forbids this tool. A missing `robots.txt` (any 4xx) forbids nothing; one + that cannot be read at all (a 5xx, no answer) is a `BackendError`, because + "unknown" is not "allowed".""" + if not urls: + return [] + parts = urllib.parse.urlsplit(urls[0]) + robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt" + try: + body = fetch(robots_url, user_agent, timeout=timeout, max_bytes=max_bytes).body + except HttpError as exc: + if 400 <= exc.status < 500: + return [] + raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc + except BackendError as exc: + raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc + parser = urllib.robotparser.RobotFileParser() + parser.parse(body.decode("utf-8", errors="replace").splitlines()) + return [url for url in urls if not parser.can_fetch(user_agent, url)] + + # --- the capture file -------------------------------------------------------