sites: raw fetch --scope fetches a page and the subpages its site handler offers as one bundle in incoming/<stem>/, with robots.txt, max_pages and pause_seconds; 10.0.0-beta.6
Files changed: - CHANGES.md - VERSION - docs/how-to/ingest-a-source.md - docs/how-to/write-a-site-handler.md - docs/reference/commands/raw-material-and-uploads.md - docs/reference/configuration.md - raw/CONTRACT.md - sites/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/commands/raw/fetch.py - tools/chemenu/config_shapes.py - tools/chemenu/sites/__init__.py - tools/chemenu/sites/data.py - tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json - tools/chemenu/sites/site.schema.json - tools/chemenu/tests/test_raw_fetch.py - tools/chemenu/tests/test_raw_fetch_scope.py - tools/chemenu/tests/test_site_files.py - tools/chemenu/tests/test_sites.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed
1 parent
f25ee6a1ce
commit
21cdd787b7
21 files changed
+835
-41
No files matched your search
+6
-1
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
||||
|
||||
---
|
||||
|
||||
## 10.0.0-beta.5 - 2026-10-11 - doctor: die Beschreibung der Strukturprüfung nennt sites/CONTRACT.md, die doctor seit den Site-Handlern der Instanz verlangt
|
||||
## 10.0.0-beta.6 - 2026-10-11 - raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -30,6 +30,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
||||
- Einstellungen in drei Stufen: wikitool.json (Instanz), .wikitool-local.json (Checkout), Secrets nur per Verweis - auch über die 1Password-CLI; neues Kommando config set
|
||||
- Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen
|
||||
- Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind
|
||||
- raw fetch --scope: eine Seite und die Unterseiten, die ihr Site-Handler anbietet, als ein Bündel in incoming/
|
||||
|
||||
**Low impact**
|
||||
- Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied
|
||||
@@ -46,6 +47,10 @@ Neue Einstellungen liegen in `wikitool.json` (eingecheckt, gleich in jedem Klon)
|
||||
|
||||
Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt Hinweise als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind JSON-Dateien nach `tools/chemenu/sites/site.schema.json`: die des Stacks unter `tools/chemenu/sites/handlers/` (zuerst `learn-microsoft` und `learn-microsoft-powershell-module`), die der Instanz unter `sites/`; ein Handler der Instanz mit demselben Host und Pfad ersetzt den des Stacks. `doctor` meldet eine unbrauchbare Datei als FAIL, und `raw fetch` bricht für deren Host mit Exit 1 ab, bevor etwas geholt wird. Der Header nennt `site:` (`<name> (instance)` bei der Instanz) und `requested_url`; das `.html` bleibt byte-genau. Der Abschnitt `sites` der Einstellungen schaltet Handler ab, wählt das Gebietsschema und nimmt Hinweiszeilen je Host-Muster auf. Regeln: `sites/CONTRACT.md`.
|
||||
|
||||
### `raw fetch --scope`: ein Bündel aus Index und Unterseiten
|
||||
|
||||
Bietet der Site-Handler einer Seite Unterseiten an (Baustein `subpages` in der Handler-Datei; der Modulindex von learn.microsoft.com bietet seine Cmdlet-Seiten an), nennt `raw fetch` das als eine `CHECK`-Zeile, und `raw fetch <url> --scope` holt Index und Unterseiten als einen Ordner `incoming/<name>/` mit einem Paar je Seite und dem Manifest `_scope.json`; `raw accept incoming/<name>` nimmt ihn unverändert an. Geholt wird nur, was der Handler benennt, nie von einem anderen Host. Vorher prüft der Befehl `robots.txt` und die Obergrenze `sites.max_pages` (Standard 200); jeder Fehler bricht ab, bevor etwas in `incoming/` liegt. Zwischen den Unterseiten wartet er `sites.pause_seconds` (Standard 1).
|
||||
|
||||
### Altlasten der Konfiguration bereinigt (#208)
|
||||
|
||||
Die vier Einzeldateien `.wikitool-remotes.json`, `.wikitool-telemetry.json`, `.wikitool-upload.json` und `.wikitool-tasks.json` gibt es nicht mehr; ihre Felder sind die Abschnitte `remotes`, `telemetry`, `upload`, `tasks`, `superproductivity` und `caldav` der drei Stufen. Zugangsdaten sind dort nur noch Verweise. Der Code liest die alten Dateien nicht mehr: ein Kommando, das die Einstellung braucht, endet mit Exit 42 und nennt `tools/wikitool config migrate`, `doctor` meldet FAIL, der Telemetrie-Hook bleibt still. `config migrate` läuft einmal je Checkout, ist wiederholbar und lässt bei jedem Abbruch die alten Dateien unverändert. `WIKITOOL_TASKS_CONFIG` zeigt auf ein Profil im Format von `.wikitool-local.json`. Nur `.wikitool-tools.json` bleibt eine eigene Datei.
|
||||
|
||||
@@ -5,6 +5,17 @@ entity and concept pages it touches, cross-referenced and published. The agent d
|
||||
the `wiki-ingest` skill ([skill catalogue](../reference/skills.md)). This page covers what you hand
|
||||
it and where it waits for you.
|
||||
|
||||
<!-- wikitool:toc -->
|
||||
## Contents
|
||||
|
||||
- [What you prepare](#what-you-prepare)
|
||||
- [What you say](#what-you-say)
|
||||
- [Where the agent asks you](#where-the-agent-asks-you)
|
||||
- [What you get back](#what-you-get-back)
|
||||
- [A document submitted from outside](#a-document-submitted-from-outside)
|
||||
- [Related](#related)
|
||||
<!-- /wikitool:toc -->
|
||||
|
||||
## What you prepare
|
||||
|
||||
Put the material into `incoming/` at the root of the instance. You don't have to classify it or
|
||||
@@ -22,6 +33,11 @@ pick a directory:
|
||||
`CHECK` lines for what it could not do for you: the agent acts on those before it accepts the
|
||||
capture. Your instance can switch such a handler off, choose another language and add hint lines
|
||||
of its own per website; see [the configuration reference](../reference/configuration.md).
|
||||
Some pages are an index of further pages - the overview of a PowerShell module on Microsoft Learn
|
||||
lists its cmdlet pages. The tool then says so in a `CHECK` line, and `raw fetch <url> --scope`
|
||||
fetches the page and those pages as one folder, `incoming/<name>/`, which is accepted whole like
|
||||
any other folder. It respects the website's `robots.txt`, pauses between requests and refuses a
|
||||
page that offers more than the instance's `max_pages`.
|
||||
- **A document that is not text** - a PDF, a Word, PowerPoint or Excel file, a mail (`.eml` or
|
||||
Outlook `.msg`) or a ZIP archive - is
|
||||
put into `incoming/` as it is. `tools/wikitool raw derive incoming/<file>` turns it into a folder with the
|
||||
|
||||
@@ -48,6 +48,17 @@ A small handler that drops a cookie banner and a footer on one host:
|
||||
website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`,
|
||||
`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md).
|
||||
|
||||
A page that is an index - a module overview, a table of contents - can name its subpages, so that
|
||||
`raw fetch <url> --scope` fetches it and them as one bundle in `incoming/<name>/`:
|
||||
|
||||
```json
|
||||
"subpages": {"links": {"tag": "a", "inside": "td"}}
|
||||
```
|
||||
|
||||
The links are the `<a>` elements inside a table cell; only those on the same host and under the
|
||||
page's own path count. A page with such a handler prints a `CHECK` line with their number when it
|
||||
is fetched without `--scope`.
|
||||
|
||||
## 3. Try it
|
||||
|
||||
Fetch a page again, or derive a saved one without a fetch:
|
||||
|
||||
@@ -25,6 +25,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
**Synopsis**
|
||||
|
||||
- `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
|
||||
- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
|
||||
- `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
|
||||
|
||||
**Options**
|
||||
@@ -38,6 +39,8 @@ Options:
|
||||
a browser) instead of fetching - no network access
|
||||
--url <str> With --html: the page's URL, for the header and relative links
|
||||
--name <str> File stem in incoming/ instead of the one computed from the URL
|
||||
--scope Also fetch the subpages the site handler offers for this page,
|
||||
into the folder incoming/<stem>/
|
||||
-h, --help Show this message and exit.
|
||||
```
|
||||
|
||||
@@ -46,6 +49,7 @@ Options:
|
||||
```bash
|
||||
tools/wikitool raw fetch https://example.org/blog/post
|
||||
tools/wikitool raw fetch https://example.org/ --name example-start
|
||||
tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope
|
||||
tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post
|
||||
```
|
||||
|
||||
@@ -56,6 +60,8 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
- Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
|
||||
- Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
|
||||
- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
|
||||
- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
|
||||
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
|
||||
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
|
||||
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
|
||||
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
|
||||
@@ -66,7 +72,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
|
||||
- Changes files: yes
|
||||
- Idempotent: no
|
||||
- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
|
||||
- Atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
|
||||
- Counts toward the session call budget: yes
|
||||
- Can reach the network: yes
|
||||
|
||||
@@ -77,6 +83,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo
|
||||
- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
|
||||
- `1` - A target file already exists in `incoming/`
|
||||
- `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
|
||||
- `1` - raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
|
||||
- `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
|
||||
|
||||
**See also**
|
||||
|
||||
@@ -413,8 +413,8 @@ Size and quota are your deployment's decision, not defaults worth copying blindl
|
||||
*site handler* fetches the page in the website's standard form, leaves the page furniture out of the
|
||||
derived text and prints hints about the page as `CHECK` lines; today there are two, both for
|
||||
`learn.microsoft.com`: `learn-microsoft` for every page of the host and
|
||||
`learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint
|
||||
naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/`
|
||||
`learn-microsoft-powershell-module` for a PowerShell module's overview page, which offers the cmdlet
|
||||
pages it lists as subpages for `raw fetch --scope`. Handlers are JSON files: the stack's, and the instance's own under `sites/`
|
||||
(see [How to write a site handler](../how-to/write-a-site-handler.md)).
|
||||
The `sites` section is what the instance decides about the handlers, and it is optional: without it every
|
||||
handler applies, in the locale `en-us`.
|
||||
@@ -432,6 +432,9 @@ handler applies, in the locale `en-us`.
|
||||
once; an entry whose `path` is left out applies to the whole host. A line is plain text and is
|
||||
output as it is, so it is for a reminder to the reader of the capture, never for an instruction
|
||||
the tool should carry out.
|
||||
- `max_pages` and `pause_seconds` bound `raw fetch --scope`: the most subpages one page may offer
|
||||
before the fetch is refused, and the pause before each subpage request, which keeps the load on
|
||||
the website low.
|
||||
|
||||
<!-- wikitool:reference-config-sites -->
|
||||
| Field | Tier | Type | Required | Default | Description |
|
||||
@@ -439,6 +442,8 @@ handler applies, in the locale `en-us`.
|
||||
| `disabled` | instance | array of strings | no | `[]` | The names of the site handlers that do not apply. `raw fetch` then treats a page of that website as it does any other. A name no handler has is an error when `raw fetch` reads the section, so a typo cannot leave a handler on. |
|
||||
| `locale` | instance | string | no | `"en-us"` | The locale a handler for a multi-language website fetches, in the website's own spelling and lower case. An empty value means the default. |
|
||||
| `notes` | instance | array of objects | no | `[]` | Hint lines of the instance's own, printed as `CHECK` lines after a fetch or an `--html` derivation of a page of that host. Each entry has `host` (exact, lower case), an optional `path` (a regular expression matched against the start of the URL path) and `lines` (one to 3 single-line strings of at most 300 characters). A host and path pattern appears once. |
|
||||
| `max_pages` | instance | positive integer | no | `200` | The most subpages `raw fetch --scope` fetches for one page. A page that offers more is refused before the first subpage is requested. A number written as a string is accepted. |
|
||||
| `pause_seconds` | instance | number | no | `1` | The pause, in seconds, before each subpage request of `raw fetch --scope`. `0` means none. A number written as a string is accepted. |
|
||||
|
||||
- **Absent:** Every handler applies, with the locale `en-us` and no further hint lines.
|
||||
- **Malformed:** An error when `raw fetch` reads it: it stops with exit 1 before fetching anything. A broken section is never read as "no handlers".
|
||||
@@ -452,6 +457,8 @@ Example:
|
||||
"learn-microsoft-powershell-module"
|
||||
],
|
||||
"locale": "en-us",
|
||||
"max_pages": 200,
|
||||
"pause_seconds": 1,
|
||||
"notes": [
|
||||
{
|
||||
"host": "learn.microsoft.com",
|
||||
|
||||
@@ -270,6 +270,20 @@ resolution and what a broken file does are in [sites/CONTRACT.md](../sites/CONTR
|
||||
switches a handler off, sets the locale, and adds hint lines of its own per host and path pattern,
|
||||
printed as `CHECK` lines too - at most three per pattern.
|
||||
|
||||
**A page that is an index can be fetched together with its subpages.** A handler may name which
|
||||
links of its page are subpages (the `subpages` field of [sites/CONTRACT.md](../sites/CONTRACT.md));
|
||||
for the module overview of `learn.microsoft.com` those are the cmdlet pages it lists. `raw fetch`
|
||||
then prints one `CHECK` line with their number, and nothing more is fetched. `raw fetch <url>
|
||||
--scope` fetches the page and its subpages as one bundle - a folder `incoming/<stem>/` holding a
|
||||
`.html`/`.md` pair per page, written as for a single page, and `_scope.json`, a manifest of the
|
||||
pages with their URL, retrieval time and the SHA-256 of the received HTML. `raw accept
|
||||
incoming/<stem>` takes the folder as one source. Only the URLs the handler names are fetched, on
|
||||
the host of the page and under the path the handler allows, never a link followed further. The
|
||||
`robots.txt` of the host is honoured for the subpages: one it forbids, or one that cannot be read,
|
||||
refuses the whole fetch. So do more subpages than `sites.max_pages`, a subpage that fails or is not
|
||||
HTML, and a folder of that name in `incoming/` or `raw/`; everything is checked before the first
|
||||
write, and a failure leaves `incoming/` as it was.
|
||||
|
||||
Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser
|
||||
can be longer than the 200 characters under which `raw fetch` warns. The session that reads the
|
||||
`.md` in full is what judges it: text that visibly breaks off ("continue reading with...", a
|
||||
|
||||
@@ -23,6 +23,7 @@ as `site: <name> (instance)`.
|
||||
| `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` |
|
||||
| `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received |
|
||||
| `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page |
|
||||
| `subpages` | Optional. Which links of a page are its subpages, so `raw fetch --scope` can fetch the page and them as one bundle. A child inherits the parent's block unless it has its own |
|
||||
|
||||
A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class`
|
||||
(one token of the element's classes), `attr` (an attribute that must be present, equal to `value`
|
||||
@@ -35,6 +36,14 @@ it applies only when at least one element matches the selector, and `{count}` is
|
||||
neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have
|
||||
the field that fills it.
|
||||
|
||||
`subpages` is `{"links", "prefix"?}`. `links` is a selector, as above, for the `<a>` elements whose
|
||||
`href` names a subpage; `prefix` limits the path a subpage may have: `{page}` (the default) is the
|
||||
path of the page itself, with a trailing slash, and a value starting with `/` is a literal path.
|
||||
A link is resolved against the page's URL, loses its fragment and counts only on the page's own
|
||||
host, under the prefix, and once; the page itself is never its own subpage. Nothing is fetched
|
||||
because of it unless the user runs `raw fetch <url> --scope`; without it the page prints one
|
||||
`CHECK` line with the number of subpages. At most `max_pages` of the `sites` section are fetched.
|
||||
|
||||
`tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is
|
||||
refused.
|
||||
|
||||
|
||||
+7
-1
@@ -1425,13 +1425,14 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
**SYNOPSIS**
|
||||
|
||||
- `wikitool raw fetch <url> [--name <stem>]` - Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`
|
||||
- `wikitool raw fetch <url> --scope [--name <stem>]` - Fetch the page and the subpages its site handler offers, and write the folder `incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`
|
||||
- `wikitool raw fetch --html incoming/<file>.html --url <url>` - Derive the `.md` from a page a human saved from their browser - no network access
|
||||
|
||||
**PROPERTIES**
|
||||
|
||||
- effect: write
|
||||
- idempotent: no
|
||||
- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first
|
||||
- atomic: Yes for what it leaves behind - one or two new files in `incoming/`, created exclusively; a failure on the second removes the first. With `--scope`: all of the folder or none of it - the pages are fetched into a temporary place and moved in as a whole
|
||||
- budget: counted
|
||||
- network: yes
|
||||
|
||||
@@ -1439,6 +1440,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
|
||||
- `tools/wikitool raw fetch https://example.org/blog/post`
|
||||
- `tools/wikitool raw fetch https://example.org/ --name example-start`
|
||||
- `tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope`
|
||||
- `tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post`
|
||||
|
||||
**EXIT STATUS**
|
||||
@@ -1448,6 +1450,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- 1 The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
|
||||
- 1 A target file already exists in `incoming/`
|
||||
- 1 The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
|
||||
- 1 raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget
|
||||
- 1 raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
|
||||
|
||||
**ON FAILURE**
|
||||
@@ -1456,6 +1459,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once
|
||||
- A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
|
||||
- The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
|
||||
- raw fetch --scope: No site handler offers subpages for the URL, the page offers none or more than `sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot be fetched, answers something other than HTML or leaves the host, the folder `incoming/<stem>/` exists, or a path would be over the budget -> Nothing was written. Read the message: fetch the page without `--scope`, raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user
|
||||
- raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
|
||||
|
||||
**NEVER**
|
||||
@@ -1470,6 +1474,8 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
|
||||
- Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
|
||||
- Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
|
||||
- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
|
||||
- `--scope` is for a page whose site handler offers subpages - the stack's today: a PowerShell module's overview page on `learn.microsoft.com`. The handler names them from the page's own HTML: the links it selects that lead to the same host, under the path of the page itself (or the prefix the handler sets); no other link is followed, and a subpage's own links never are. Without `--scope` the one page is fetched as before and a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each page. `raw accept incoming/<stem>` takes the folder as one bundle.
|
||||
- `--scope` is bounded. The `robots.txt` of the host is honoured for the subpages (not for the page itself): one it forbids refuses the whole fetch and names the URL. More subpages than `sites.max_pages` (200) refuses it before the first subpage is requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both are settings of the `instance` tier. Everything is checked before the first write: the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` or `raw/`. A failing request leaves `incoming/` as it was.
|
||||
- The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
|
||||
- A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
|
||||
- Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
|
||||
|
||||
+1
-1
@@ -118,7 +118,7 @@ tools/
|
||||
kb_state.py the KB version (.wikitool-kb.json) and the migration chain
|
||||
corpus_diff.py invariant comparison of kb/ between two revisions
|
||||
web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
|
||||
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
|
||||
sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, hint lines, and which links are subpages for `raw fetch --scope` - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
|
||||
filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
|
||||
repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
|
||||
guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached
|
||||
|
||||
@@ -7,23 +7,38 @@ the caller is told.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from typing import Annotated, Optional
|
||||
|
||||
import typer
|
||||
from rich.markup import escape
|
||||
|
||||
from chemenu import cli_contract, sites, web_capture
|
||||
from chemenu.commands._util import fail, rel_path, success
|
||||
from chemenu import cli_contract, config, sites, web_capture
|
||||
from chemenu.commands._util import fail, path_budget_problem_for, rel_path, success
|
||||
from chemenu.errors import ChemenuError
|
||||
from chemenu.version import VersionError, read_version
|
||||
from chemenu.commands.raw._common import (
|
||||
_folder_collision_message,
|
||||
_incoming_dir,
|
||||
_occupied_stems,
|
||||
_resolve,
|
||||
_shard_dir,
|
||||
_unusable_name,
|
||||
_validate_under_incoming,
|
||||
_write_exclusive,
|
||||
)
|
||||
|
||||
SCOPE_MANIFEST = "_scope.json"
|
||||
SCOPE_MANIFEST_VERSION = 1
|
||||
_LISTED_REFUSALS = 10
|
||||
|
||||
|
||||
_TEASER_HINT = (
|
||||
" If the text stops at a teaser (paywall, login, script-rendered page): save the page from a\n"
|
||||
@@ -73,15 +88,22 @@ def _warn(derivation: web_capture.Derivation, md_path: Path) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _plan(url: str, *, canonical: bool) -> sites.Plan:
|
||||
def _preferences() -> sites.Preferences:
|
||||
try:
|
||||
return sites.plan_for(url, sites.read_preferences(), canonical=canonical)
|
||||
return sites.read_preferences()
|
||||
except ChemenuError as exc:
|
||||
fail(escape(str(exc)))
|
||||
|
||||
|
||||
def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation) -> None:
|
||||
for line in plan.checks(derivation.decoded.text):
|
||||
def _plan(url: str, *, canonical: bool, preferences: Optional[sites.Preferences] = None) -> sites.Plan:
|
||||
try:
|
||||
return sites.plan_for(url, preferences or _preferences(), canonical=canonical)
|
||||
except ChemenuError as exc:
|
||||
fail(escape(str(exc)))
|
||||
|
||||
|
||||
def _echo_checks(plan: sites.Plan, derivation: web_capture.Derivation, subpages: int = 0) -> None:
|
||||
for line in plan.checks(derivation.decoded.text, subpages):
|
||||
typer.echo(f"CHECK {line}")
|
||||
|
||||
|
||||
@@ -121,6 +143,176 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
))
|
||||
|
||||
|
||||
def _subpage_names(stem: str, urls: list[str]) -> list[str]:
|
||||
"""One file stem per subpage, from its URL, unique within the bundle and
|
||||
never the overview's."""
|
||||
taken = {stem}
|
||||
names = []
|
||||
for url in urls:
|
||||
base = web_capture.stem_for(url)
|
||||
name, number = base, 1
|
||||
while name in taken:
|
||||
number += 1
|
||||
name = f"{base}-{number}"
|
||||
taken.add(name)
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
|
||||
def _listed(urls: list[str]) -> str:
|
||||
shown = "\n".join(f" - {url}" for url in urls[:_LISTED_REFUSALS])
|
||||
more = len(urls) - _LISTED_REFUSALS
|
||||
return shown + (f"\n ... and {more} more" if more > 0 else "")
|
||||
|
||||
|
||||
def _page_entry(name: str, role: str, response: web_capture.Response, retrieved: str) -> dict:
|
||||
return {
|
||||
"name": name,
|
||||
"role": role,
|
||||
"url": response.url,
|
||||
"final_url": response.final_url,
|
||||
"retrieved": retrieved,
|
||||
"sha256": hashlib.sha256(response.body).hexdigest(),
|
||||
"files": [f"{name}.html", f"{name}.md"],
|
||||
}
|
||||
|
||||
|
||||
def _fetch_scope(url: str, name: Optional[str]) -> None:
|
||||
preferences = _preferences()
|
||||
plan = _plan(url, canonical=True, preferences=preferences)
|
||||
stem = name or web_capture.stem_for(plan.url)
|
||||
if _unusable_name(stem):
|
||||
fail(escape(f"{stem!r} is not a usable folder name - choose another with --name."))
|
||||
target = _incoming_dir() / stem
|
||||
|
||||
def refuse_taken() -> None:
|
||||
if os.path.lexists(target):
|
||||
fail(escape(
|
||||
f"{rel_path(target)} already exists in incoming/ - raw fetch never overwrites. "
|
||||
"Accept or remove what is there, or pass --name <stem>."
|
||||
))
|
||||
holder = _occupied_stems(config.RAW_DIR).get(stem)
|
||||
if holder is not None:
|
||||
fail(escape(_folder_collision_message(stem, holder)))
|
||||
|
||||
if plan.site is None or plan.site.subpages is None:
|
||||
fail(escape(
|
||||
f"--scope needs a site handler that offers subpages, and none applies to {plan.url}. "
|
||||
"Fetch the page without --scope."
|
||||
))
|
||||
refuse_taken()
|
||||
|
||||
agent = _user_agent()
|
||||
try:
|
||||
index = web_capture.fetch(
|
||||
plan.url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
|
||||
)
|
||||
except ChemenuError as exc:
|
||||
fail(escape(str(exc)))
|
||||
index_retrieved = web_capture.utc_now()
|
||||
if index.media_type not in web_capture.HTML_TYPES:
|
||||
fail(escape(
|
||||
f"{plan.url} answered {index.media_type or 'no content type'}, not an HTML page - "
|
||||
"--scope bundles HTML pages. Nothing was written."
|
||||
))
|
||||
index_derivation = web_capture.derive_document(
|
||||
index.body, index.final_url, f"{stem}.html", response=index,
|
||||
drop=plan.drop, site=plan.label, requested_url=plan.requested_url, derived_at=index_retrieved,
|
||||
)
|
||||
urls = plan.subpages(index_derivation.decoded.text, index.final_url)
|
||||
if not urls:
|
||||
fail(escape(f"{index.final_url} offers no subpages - nothing was written. Fetch it without --scope."))
|
||||
if len(urls) > preferences.max_pages:
|
||||
fail(escape(
|
||||
f"{index.final_url} offers {len(urls)} subpages, more than the {preferences.max_pages} "
|
||||
"that `sites.max_pages` allows - nothing was fetched or written. Raise the setting "
|
||||
"(`tools/wikitool config set sites.max_pages <n>`) if the whole family is wanted."
|
||||
))
|
||||
try:
|
||||
forbidden = web_capture.robots_refusals(urls, agent)
|
||||
except ChemenuError as exc:
|
||||
fail(escape(f"{exc} Nothing was written."))
|
||||
if forbidden:
|
||||
fail(escape(
|
||||
f"robots.txt of {urllib.parse.urlsplit(index.final_url).netloc} forbids {len(forbidden)} "
|
||||
f"of the {len(urls)} subpages - none was fetched, nothing was written:\n"
|
||||
f"{_listed(forbidden)}"
|
||||
))
|
||||
|
||||
names = _subpage_names(stem, urls)
|
||||
files = [f"{stem}.html", f"{stem}.md", SCOPE_MANIFEST]
|
||||
files += [f"{n}{ext}" for n in names for ext in (".html", ".md")]
|
||||
over = [p for rel in files if (p := path_budget_problem_for(_shard_dir() / stem / rel))]
|
||||
if over:
|
||||
fail(escape(
|
||||
f"Cannot fetch - these files would be over the path budget at their place under raw/:\n"
|
||||
f"{_listed(over)}\n Choose a shorter --name. Nothing was written."
|
||||
))
|
||||
|
||||
work = Path(tempfile.mkdtemp(prefix="wikitool-scope-"))
|
||||
try:
|
||||
bundle = work / stem
|
||||
bundle.mkdir()
|
||||
host = urllib.parse.urlsplit(plan.url).netloc.lower()
|
||||
(bundle / f"{stem}.html").write_bytes(index.body)
|
||||
(bundle / f"{stem}.md").write_bytes(index_derivation.markdown)
|
||||
entries = [_page_entry(stem, "index", index, index_retrieved)]
|
||||
derivations = [(stem, index_derivation)]
|
||||
for position, (sub_url, sub_name) in enumerate(zip(urls, names), start=1):
|
||||
if preferences.pause_seconds:
|
||||
time.sleep(preferences.pause_seconds)
|
||||
try:
|
||||
page = web_capture.fetch(
|
||||
sub_url, agent, timeout=web_capture.TIMEOUT_SECONDS, max_bytes=web_capture.MAX_BYTES
|
||||
)
|
||||
except ChemenuError as exc:
|
||||
fail(escape(f"{exc} (subpage {position} of {len(urls)}) Nothing was written."))
|
||||
retrieved = web_capture.utc_now()
|
||||
if urllib.parse.urlsplit(page.final_url).netloc.lower() != host:
|
||||
fail(escape(
|
||||
f"{sub_url} was redirected to {page.final_url}, another host - --scope stays on "
|
||||
f"{host}. Nothing was written."
|
||||
))
|
||||
if page.media_type not in web_capture.HTML_TYPES:
|
||||
fail(escape(
|
||||
f"{sub_url} answered {page.media_type or 'no content type'}, not an HTML page - "
|
||||
"--scope bundles HTML pages. Nothing was written."
|
||||
))
|
||||
sub_plan = _plan(sub_url, canonical=False, preferences=preferences)
|
||||
derivation = web_capture.derive_document(
|
||||
page.body, page.final_url, f"{sub_name}.html", response=page,
|
||||
drop=sub_plan.drop, site=sub_plan.label, derived_at=retrieved,
|
||||
)
|
||||
(bundle / f"{sub_name}.html").write_bytes(page.body)
|
||||
(bundle / f"{sub_name}.md").write_bytes(derivation.markdown)
|
||||
entries.append(_page_entry(sub_name, "subpage", page, retrieved))
|
||||
derivations.append((sub_name, derivation))
|
||||
manifest = {
|
||||
"scope": SCOPE_MANIFEST_VERSION,
|
||||
"fetched_by": "wikitool raw fetch --scope",
|
||||
"url": plan.url,
|
||||
**({"requested_url": plan.requested_url} if plan.requested_url else {}),
|
||||
"retrieved": index_retrieved,
|
||||
"pages": entries,
|
||||
}
|
||||
(bundle / SCOPE_MANIFEST).write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n")
|
||||
refuse_taken()
|
||||
shutil.move(str(bundle), str(target))
|
||||
except OSError as exc:
|
||||
fail(escape(f"Could not write {rel_path(target)}: {exc}. Nothing was left in incoming/."))
|
||||
finally:
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
|
||||
for page_name, derivation in derivations:
|
||||
_warn(derivation, target / f"{page_name}.md")
|
||||
_echo_checks(plan, index_derivation)
|
||||
success(escape(
|
||||
f"Fetched {_fetched(plan)} with {len(urls)} subpage(s) to {rel_path(target)}/ "
|
||||
f"({len(entries)} page pairs and {SCOPE_MANIFEST}).\n"
|
||||
+ _next_lines([target], plan.url)
|
||||
))
|
||||
|
||||
|
||||
@cli_contract.record(cli_contract.CommandRecord(
|
||||
path="raw fetch",
|
||||
summary="Capture a web page the user names into `incoming/`: the HTML as received plus a "
|
||||
@@ -131,6 +323,11 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
usage="raw fetch <url> [--name <stem>]",
|
||||
notes="Fetch the page and write `incoming/<stem>.html` and `incoming/<stem>.md`",
|
||||
),
|
||||
cli_contract.Variant(
|
||||
usage="raw fetch <url> --scope [--name <stem>]",
|
||||
notes="Fetch the page and the subpages its site handler offers, and write the folder "
|
||||
"`incoming/<stem>/` - a `.html` + `.md` pair per page and the manifest `_scope.json`",
|
||||
),
|
||||
cli_contract.Variant(
|
||||
usage="raw fetch --html incoming/<file>.html --url <url>",
|
||||
notes="Derive the `.md` from a page a human saved from their browser - no network "
|
||||
@@ -141,7 +338,8 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
effect=cli_contract.Effect.WRITE,
|
||||
idempotent=cli_contract.Idempotent.NO,
|
||||
atomic="Yes for what it leaves behind - one or two new files in `incoming/`, created "
|
||||
"exclusively; a failure on the second removes the first",
|
||||
"exclusively; a failure on the second removes the first. With `--scope`: all of the folder "
|
||||
"or none of it - the pages are fetched into a temporary place and moved in as a whole",
|
||||
budget=cli_contract.Budget.COUNTED,
|
||||
network=cli_contract.Network.YES,
|
||||
),
|
||||
@@ -173,6 +371,23 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
"URL's host or no readable host, stops the fetch before anything is fetched or written. "
|
||||
"The `sites` section of the settings switches handlers "
|
||||
"off, sets the locale and adds hint lines of the instance's own per host.",
|
||||
"`--scope` is for a page whose site handler offers subpages - the stack's today: a "
|
||||
"PowerShell module's overview page on `learn.microsoft.com`. The handler names them from "
|
||||
"the page's own HTML: the links it selects that lead to the same host, under the path of "
|
||||
"the page itself (or the prefix the handler sets); no other link is followed, and a "
|
||||
"subpage's own links never are. Without `--scope` the one page is fetched as before and "
|
||||
"a `CHECK` line says how many subpages it offers. With it, `incoming/<stem>/` holds the "
|
||||
"page and every subpage as a `<name>.html` + `<name>.md` pair - the `.html` the response "
|
||||
"byte for byte, the `.md` derived as for a single page, `<name>` from the subpage's URL - "
|
||||
"and `_scope.json`: the URL, the retrieval time and the `sha256` of the `.html` of each "
|
||||
"page. `raw accept incoming/<stem>` takes the folder as one bundle.",
|
||||
"`--scope` is bounded. The `robots.txt` of the host is honoured for the subpages "
|
||||
"(not for the page itself): one it forbids refuses the whole fetch and names the URL. "
|
||||
"More subpages than `sites.max_pages` (200) refuses it before the first subpage is "
|
||||
"requested; `sites.pause_seconds` (1) is the pause before each subpage request. Both "
|
||||
"are settings of the `instance` tier. Everything is checked before the first write: "
|
||||
"the path budget under `raw/<YYYY>/<MM>/`, a folder of that name already in `incoming/` "
|
||||
"or `raw/`. A failing request leaves `incoming/` as it was.",
|
||||
"The stem is the URL's last path segment without its extension, else its host name - "
|
||||
"ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.",
|
||||
"A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any "
|
||||
@@ -215,6 +430,16 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
"with the user; for a paywall or login, save the page in a browser and use "
|
||||
"`--html`. An unreachable host or a timeout may be retried once",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
label="raw fetch --scope",
|
||||
cause="No site handler offers subpages for the URL, the page offers none or more than "
|
||||
"`sites.max_pages`, `robots.txt` forbids a subpage or cannot be read, a subpage cannot "
|
||||
"be fetched, answers something other than HTML or leaves the host, the folder "
|
||||
"`incoming/<stem>/` exists, or a path would be over the budget",
|
||||
reaction="Nothing was written. Read the message: fetch the page without `--scope`, "
|
||||
"raise the limit with `tools/wikitool config set`, or pass a shorter `--name`, then "
|
||||
"retry once. A subpage `robots.txt` forbids is not fetched around it - tell the user",
|
||||
),
|
||||
cli_contract.Failure(
|
||||
label="raw fetch --html",
|
||||
cause="`--url` is missing, the file is not directly in `incoming/` or does not "
|
||||
@@ -226,6 +451,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
|
||||
examples=(
|
||||
"tools/wikitool raw fetch https://example.org/blog/post",
|
||||
"tools/wikitool raw fetch https://example.org/ --name example-start",
|
||||
"tools/wikitool raw fetch https://learn.microsoft.com/en-us/powershell/module/oslicense/ --scope",
|
||||
"tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blog/post",
|
||||
),
|
||||
never=(
|
||||
@@ -255,6 +481,13 @@ def raw_fetch_command(
|
||||
name: Optional[str] = typer.Option(
|
||||
None, "--name", help="File stem in incoming/ instead of the one computed from the URL"
|
||||
),
|
||||
# `Annotated`, so that a direct call - the way the test suite calls this
|
||||
# function - gets a real `False` here rather than a truthy `OptionInfo`.
|
||||
scope: Annotated[bool, typer.Option(
|
||||
"--scope",
|
||||
help="Also fetch the subpages the site handler offers for this page, into the folder "
|
||||
"incoming/<stem>/",
|
||||
)] = False,
|
||||
):
|
||||
"""Capture a web page into incoming/ as received HTML plus derived text.
|
||||
See raw/CONTRACT.md "Getting a URL in: `raw fetch`"."""
|
||||
@@ -263,6 +496,8 @@ def raw_fetch_command(
|
||||
if html is not None:
|
||||
if name is not None:
|
||||
fail("--name does not apply to --html - the stem is the saved file's own.")
|
||||
if scope:
|
||||
fail("--scope does not apply to --html - it fetches subpages, and --html makes no request.")
|
||||
_fetch_html_file(html, source_url)
|
||||
return
|
||||
if source_url is not None:
|
||||
@@ -270,6 +505,9 @@ def raw_fetch_command(
|
||||
|
||||
if name is not None and (not name.strip() or name in (".", "..") or any(c in name for c in "/\\")):
|
||||
fail(f"--name {name!r} is not a usable file stem: no path separators, not empty.")
|
||||
if scope:
|
||||
_fetch_scope(url, name)
|
||||
return
|
||||
plan = _plan(url, canonical=True)
|
||||
stem = name or web_capture.stem_for(plan.url)
|
||||
|
||||
@@ -291,7 +529,7 @@ def raw_fetch_command(
|
||||
)
|
||||
_write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
|
||||
_warn(derivation, md_path)
|
||||
_echo_checks(plan, derivation)
|
||||
_echo_checks(plan, derivation, len(plan.subpages(derivation.decoded.text, response.final_url)))
|
||||
success(escape(
|
||||
f"Fetched {_fetched(plan)} to {rel_path(html_path)}, {rel_path(md_path)}\n"
|
||||
+ _next_lines([html_path, md_path], plan.url) + "\n"
|
||||
|
||||
@@ -248,6 +248,10 @@ def _positive(value: int) -> Optional[str]:
|
||||
return None if value > 0 else "must be positive"
|
||||
|
||||
|
||||
def non_negative(value: float) -> Optional[str]:
|
||||
return None if value >= 0 else "must not be negative"
|
||||
|
||||
|
||||
def dotted_extensions(value: tuple[str, ...]) -> Optional[str]:
|
||||
if not value:
|
||||
return "must not be empty"
|
||||
@@ -453,8 +457,8 @@ SITES = Shape(
|
||||
key="sites",
|
||||
title="Site handlers",
|
||||
purpose="Tunes the site handlers `raw fetch` applies to the pages of well-known websites: "
|
||||
"switches one off, picks the locale a handler fetches, and adds hint lines of the "
|
||||
"instance's own per host.",
|
||||
"switches one off, picks the locale a handler fetches, adds hint lines of the "
|
||||
"instance's own per host, and bounds `--scope`, the fetch of a page with its subpages.",
|
||||
fields=(
|
||||
Field(
|
||||
"disabled", "array of strings", "The names of the site handlers that do not apply. "
|
||||
@@ -479,10 +483,23 @@ SITES = Shape(
|
||||
default=[], json_type="array", coerce=site_notes, check=site_notes_problem,
|
||||
tier=INSTANCE,
|
||||
),
|
||||
positive_integer(
|
||||
"max_pages", "The most subpages `raw fetch --scope` fetches for one page. A page that "
|
||||
"offers more is refused before the first subpage is requested.",
|
||||
default=200, tier=INSTANCE,
|
||||
),
|
||||
Field(
|
||||
"pause_seconds", "number", "The pause, in seconds, before each subpage request of "
|
||||
"`raw fetch --scope`. `0` means none.",
|
||||
default=1, json_type="number", coerce=float, check=non_negative,
|
||||
lenient="A number written as a string is accepted.", tier=INSTANCE,
|
||||
),
|
||||
),
|
||||
examples=({
|
||||
"disabled": ["learn-microsoft-powershell-module"],
|
||||
"locale": "en-us",
|
||||
"max_pages": 200,
|
||||
"pause_seconds": 1,
|
||||
"notes": [{
|
||||
"host": "learn.microsoft.com", "path": "^/[a-z-]+/azure/",
|
||||
"lines": ["Check the Azure retirement notice at the top of the page."],
|
||||
@@ -515,6 +532,7 @@ _JSON_TYPES: dict[str, Callable[[Any], bool]] = {
|
||||
"string": lambda v: isinstance(v, str),
|
||||
"array": lambda v: isinstance(v, list),
|
||||
"object": lambda v: isinstance(v, dict),
|
||||
"number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
|
||||
}
|
||||
|
||||
_TYPE_NAMES = {bool: "a boolean", int: "an integer", float: "a number", str: "a string",
|
||||
|
||||
@@ -14,7 +14,9 @@ with an optional condition. So a handler is a JSON file (`data.py` reads it,
|
||||
in `sites/` at its root, by the same building blocks. An instance handler for a
|
||||
host and path the stack has a handler for replaces it; the `sites` section of the
|
||||
settings still switches handlers off, sets the `locale` and adds hint lines
|
||||
(`notes`). A handler may still be registered from a Python module in this
|
||||
(`notes`). A handler may also offer the subpages of a page
|
||||
(`Subpages`: the links to take, under a path prefix), which `raw fetch --scope`
|
||||
fetches as one bundle. A handler may still be registered from a Python module in this
|
||||
package. The core knows neither `typer` nor `rich` and raises `ValidationError`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -33,6 +35,7 @@ from chemenu.web_capture import Rule
|
||||
|
||||
MAX_CHECKS = 3
|
||||
LOCALE_PLACEHOLDER = "{locale}"
|
||||
PAGE_PLACEHOLDER = "{page}"
|
||||
INSTANCE_DIRECTORY = "sites"
|
||||
|
||||
|
||||
@@ -79,6 +82,47 @@ class Check:
|
||||
return text
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Subpages:
|
||||
"""The subpages a page offers: the links `links` selects, resolved against
|
||||
the page's URL, that stay on its host and lie under `prefix`. `{page}` in
|
||||
`prefix` is the path of the page itself with a trailing slash, which is
|
||||
the default; the page is never its own subpage. Document order, each URL
|
||||
once, the fragment dropped - and nothing else is followed."""
|
||||
|
||||
links: Rule
|
||||
prefix: str = PAGE_PLACEHOLDER
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not (self.prefix.startswith(PAGE_PLACEHOLDER) or self.prefix.startswith("/")):
|
||||
raise ValueError(f"a subpages prefix starts with {PAGE_PLACEHOLDER} or with /, not {self.prefix!r}")
|
||||
|
||||
def urls(self, html: str, page_url: str) -> list[str]:
|
||||
page = urllib.parse.urlsplit(page_url)
|
||||
own = urllib.parse.unquote(page.path) or "/"
|
||||
own = own if own.endswith("/") else own + "/"
|
||||
prefix = self.prefix.replace(PAGE_PLACEHOLDER, own)
|
||||
found: list[str] = []
|
||||
for element in web_capture.matching(web_capture.parse(html), self.links):
|
||||
href = element.attrs.get("href", "").strip()
|
||||
if not href:
|
||||
continue
|
||||
url = urllib.parse.urldefrag(urllib.parse.urljoin(page_url, href))[0]
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
path = urllib.parse.unquote(parts.path)
|
||||
if (
|
||||
parts.scheme.lower() not in web_capture.ALLOWED_SCHEMES
|
||||
or parts.netloc.lower() != page.netloc.lower()
|
||||
or not path.startswith(prefix)
|
||||
or path.rstrip("/") == own.rstrip("/")
|
||||
or path == prefix
|
||||
or url in found
|
||||
):
|
||||
continue
|
||||
found.append(url)
|
||||
return found
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Site:
|
||||
"""One handler. `host` is matched exactly (lower case); `path` is a regular
|
||||
@@ -94,6 +138,7 @@ class Site:
|
||||
rewrites: tuple[Rewrite, ...] = ()
|
||||
drop: tuple[Rule, ...] = ()
|
||||
checks: tuple[Check, ...] = ()
|
||||
subpages: Optional[Subpages] = None
|
||||
origin: str = "stack"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
@@ -131,6 +176,7 @@ def inherit(candidate: Site, parent: Site) -> Site:
|
||||
rewrites=parent.rewrites + candidate.rewrites,
|
||||
drop=parent.drop + candidate.drop,
|
||||
checks=parent.checks + candidate.checks,
|
||||
subpages=candidate.subpages or parent.subpages,
|
||||
)
|
||||
if len(flat.checks) > MAX_CHECKS:
|
||||
raise ValueError(
|
||||
@@ -220,6 +266,8 @@ class Preferences:
|
||||
notes: tuple[Note, ...] = ()
|
||||
instance: tuple[Site, ...] = ()
|
||||
broken: tuple[Broken, ...] = ()
|
||||
max_pages: int = 200
|
||||
pause_seconds: float = 1.0
|
||||
|
||||
|
||||
def read_instance(root: "Path | str | None" = None) -> tuple[tuple[Site, ...], tuple[Broken, ...]]:
|
||||
@@ -250,7 +298,9 @@ def read_preferences(root: "Path | str | None" = None) -> Preferences:
|
||||
notes = tuple(
|
||||
Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"]
|
||||
)
|
||||
return Preferences(disabled, section["locale"], notes, instance, broken)
|
||||
return Preferences(
|
||||
disabled, section["locale"], notes, instance, broken, section["max_pages"], section["pause_seconds"]
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -278,11 +328,23 @@ class Plan:
|
||||
def drop(self) -> tuple[Rule, ...]:
|
||||
return self.site.drop if self.site else ()
|
||||
|
||||
def checks(self, html: str) -> list[str]:
|
||||
"""The `CHECK` lines for the page: the handler's, then the instance's notes."""
|
||||
def checks(self, html: str, subpages: int = 0) -> list[str]:
|
||||
"""The `CHECK` lines for the page: the handler's, the offer of `subpages`
|
||||
subpages when there are any, then the instance's notes."""
|
||||
lines = self.site.check_lines(html) if self.site else []
|
||||
if subpages:
|
||||
lines.append(
|
||||
f"this page is an index with {subpages} subpage(s) - rerun with `--scope` to "
|
||||
"fetch them as one bundle"
|
||||
)
|
||||
return lines + list(self.notes)
|
||||
|
||||
def subpages(self, html: str, page_url: str) -> list[str]:
|
||||
"""The subpage URLs the handler offers for this page, else none."""
|
||||
if self.site is None or self.site.subpages is None:
|
||||
return []
|
||||
return self.site.subpages.urls(html, page_url)
|
||||
|
||||
|
||||
def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan:
|
||||
"""The plan for `url`. `canonical=False` keeps the URL as given - for a page
|
||||
|
||||
@@ -15,7 +15,7 @@ from typing import Any, Optional
|
||||
|
||||
from jsonschema import Draft202012Validator
|
||||
|
||||
from chemenu.sites import Broken, Check, Rewrite, Rule, Site, inherit
|
||||
from chemenu.sites import Broken, Check, Rewrite, Rule, Site, Subpages, inherit
|
||||
|
||||
SCHEMA_PATH = Path(__file__).with_name("site.schema.json")
|
||||
STACK_DIRECTORY = Path(__file__).with_name("handlers")
|
||||
@@ -70,6 +70,13 @@ def build(name: str, document: Any, origin: str) -> Site:
|
||||
if "{count}" in text and "count" not in entry:
|
||||
raise ValueError("checks: a text with {count} needs a `count` rule")
|
||||
checks.append(Check(text, entry.get("meta", ""), _rule(entry["count"]) if "count" in entry else None))
|
||||
subpages = None
|
||||
if "subpages" in document:
|
||||
entry = document["subpages"]
|
||||
try:
|
||||
subpages = Subpages(_rule(entry["links"]), entry.get("prefix", "{page}"))
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"subpages: {exc}") from None
|
||||
try:
|
||||
return Site(
|
||||
name=name,
|
||||
@@ -80,6 +87,7 @@ def build(name: str, document: Any, origin: str) -> Site:
|
||||
rewrites=tuple(rewrites),
|
||||
drop=tuple(_rule(entry) for entry in document.get("drop", [])),
|
||||
checks=tuple(checks),
|
||||
subpages=subpages,
|
||||
origin=origin,
|
||||
)
|
||||
except (ValueError, re.error) as exc:
|
||||
|
||||
@@ -1,12 +1,9 @@
|
||||
{
|
||||
"host": "learn.microsoft.com",
|
||||
"summary": "A PowerShell module's overview page: names the cmdlet pages it lists, which are not fetched with it.",
|
||||
"summary": "A PowerShell module's overview page: offers the cmdlet pages it lists as its subpages, which `--scope` fetches with it as one bundle.",
|
||||
"path": "/[a-z]{2,3}(?:-[a-z0-9]{2,8})+/powershell/module/[^/]+/?$",
|
||||
"parent": "learn-microsoft",
|
||||
"checks": [
|
||||
{
|
||||
"text": "This module page lists {count} cmdlet page(s) as links; they are not fetched - ingest the ones the topic needs, each with its own raw fetch.",
|
||||
"count": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"}
|
||||
}
|
||||
]
|
||||
"subpages": {
|
||||
"links": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"}
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,15 @@
|
||||
"count": {"$ref": "#/$defs/rule"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"subpages": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["links"],
|
||||
"properties": {
|
||||
"links": {"$ref": "#/$defs/rule"},
|
||||
"prefix": {"type": "string", "pattern": "^(\\{page\\}|/)", "maxLength": 200}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
|
||||
@@ -100,8 +100,8 @@ def tree(kb_dir):
|
||||
return root
|
||||
|
||||
|
||||
def _fetch(url=None, html=None, source_url=None, name=None):
|
||||
return raw_fetch_command(url=url, html=html, source_url=source_url, name=name)
|
||||
def _fetch(url=None, html=None, source_url=None, name=None, scope=False):
|
||||
return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope)
|
||||
|
||||
|
||||
def _refused(**kwargs) -> None:
|
||||
|
||||
@@ -0,0 +1,346 @@
|
||||
"""`wikitool raw fetch --scope` - a page and the subpages its site handler offers,
|
||||
fetched as one bundle.
|
||||
|
||||
Like `test_raw_fetch.py`, every request goes to a real `http.server` on 127.0.0.1.
|
||||
The handler is an instance handler in `sites/`, because that is the form in which
|
||||
the subpages building block is data an instance can write.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import typer
|
||||
|
||||
from chemenu import sites, web_capture
|
||||
from chemenu.commands.raw.accept import raw_accept_command
|
||||
from chemenu.commands.raw.fetch import raw_fetch_command
|
||||
from chemenu.config_shapes import SITES, check
|
||||
from chemenu.tests.conftest import write_settings
|
||||
from chemenu.tests.test_raw_fetch import Site
|
||||
|
||||
HANDLER = {
|
||||
"host": "127.0.0.1",
|
||||
"summary": "A test module: the links in its table are its subpages.",
|
||||
"path": "/mod/",
|
||||
"subpages": {"links": {"tag": "a", "inside": "td"}},
|
||||
}
|
||||
|
||||
|
||||
def _index(port: int) -> bytes:
|
||||
return (
|
||||
"<html><head><title>Module</title></head><body><main><h1>Module</h1><table>"
|
||||
'<tr><td><a href="a?v=1#top">A</a></td></tr>'
|
||||
'<tr><td><a href="b?v=1">B</a></td></tr>'
|
||||
'<tr><td><a href="/mod/c">C</a></td></tr>'
|
||||
'<tr><td><a href="a?v=1">A again</a></td></tr>'
|
||||
'<tr><td><a href="/elsewhere/x">outside the prefix</a></td></tr>'
|
||||
f'<tr><td><a href="http://localhost:{port}/mod/foreign">another host</a></td></tr>'
|
||||
'<tr><td><a href="mailto:x@example.org">mail</a></td></tr>'
|
||||
'<tr><td><a href="./">the page itself</a></td></tr>'
|
||||
"</table>"
|
||||
'<p><a href="/mod/hidden">a link outside the table</a></p>'
|
||||
"</main></body></html>"
|
||||
).encode()
|
||||
|
||||
|
||||
def _sub(name: str) -> bytes:
|
||||
return (
|
||||
f"<html><head><title>{name}</title></head><body><main><h1>{name}</h1>"
|
||||
f"<p>{name} text</p></main></body></html>"
|
||||
).encode()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def site(monkeypatch):
|
||||
for var in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY", "all_proxy", "ALL_PROXY"):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
server = Site()
|
||||
yield server
|
||||
server.stop()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tree(kb_dir, monkeypatch, site):
|
||||
root = kb_dir.parent
|
||||
(root / "raw").mkdir()
|
||||
(root / "incoming").mkdir(exist_ok=True)
|
||||
(root / "sites").mkdir()
|
||||
(root / "sites/test-module.json").write_text(json.dumps(HANDLER), encoding="utf-8")
|
||||
write_settings(root, monkeypatch, sites={"pause_seconds": 0})
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def module(site):
|
||||
"""The index and the pages around it, served."""
|
||||
site.html("/mod/", _index(site.httpd.server_address[1]))
|
||||
for name in ("a?v=1", "b?v=1", "c", "hidden", "foreign"):
|
||||
site.html(f"/mod/{name}", _sub(name.split("?")[0]))
|
||||
return site
|
||||
|
||||
|
||||
def _fetch(url, name=None, scope=True):
|
||||
return raw_fetch_command(url=url, html=None, source_url=None, name=name, scope=scope)
|
||||
|
||||
|
||||
def _refused(url, **kwargs) -> None:
|
||||
with pytest.raises(typer.Exit) as excinfo:
|
||||
_fetch(url, **kwargs)
|
||||
assert excinfo.value.exit_code == 1
|
||||
|
||||
|
||||
def _snapshot(root: Path) -> list[str]:
|
||||
return sorted(str(p.relative_to(root)) for p in root.rglob("*"))
|
||||
|
||||
|
||||
def _requested(site: Site) -> list[str]:
|
||||
return [path for path, _ in site.requests]
|
||||
|
||||
|
||||
def _checks(capsys) -> list[str]:
|
||||
return [line[len("CHECK "):] for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")]
|
||||
|
||||
|
||||
# --- without --scope --------------------------------------------------------
|
||||
|
||||
|
||||
def test_without_scope_the_page_is_fetched_alone_and_the_offer_is_one_check(tree, module, capsys):
|
||||
_fetch(module.url("/mod/"), name="mod", scope=False)
|
||||
|
||||
assert sorted(p.name for p in (tree / "incoming").iterdir()) == ["mod.html", "mod.md"]
|
||||
assert (tree / "incoming/mod.html").read_bytes() == module.routes["/mod/"][2]
|
||||
assert _requested(module) == ["/mod/"]
|
||||
lines = _checks(capsys)
|
||||
assert len(lines) == 1
|
||||
assert "3 subpage(s)" in lines[0] and "--scope" in lines[0]
|
||||
|
||||
|
||||
# --- with --scope -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_scope_writes_a_folder_with_a_pair_per_page_and_the_manifest(tree, module):
|
||||
before = datetime.datetime.now(datetime.timezone.utc)
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
|
||||
folder = tree / "incoming/mod"
|
||||
assert sorted(p.name for p in folder.iterdir()) == [
|
||||
"_scope.json", "a.html", "a.md", "b.html", "b.md", "c.html", "c.md", "mod.html", "mod.md",
|
||||
]
|
||||
assert [p.name for p in (tree / "incoming").iterdir()] == ["mod"]
|
||||
for name, path in (("mod", "/mod/"), ("a", "/mod/a?v=1"), ("b", "/mod/b?v=1"), ("c", "/mod/c")):
|
||||
assert (folder / f"{name}.html").read_bytes() == module.routes[path][2]
|
||||
text = (folder / "a.md").read_text(encoding="utf-8")
|
||||
assert "a text" in text and "derived_from: a.html" in text
|
||||
|
||||
manifest = json.loads((folder / "_scope.json").read_text(encoding="utf-8"))
|
||||
assert manifest["scope"] == 1 and manifest["url"] == module.url("/mod/")
|
||||
assert [(p["name"], p["role"]) for p in manifest["pages"]] == [
|
||||
("mod", "index"), ("a", "subpage"), ("b", "subpage"), ("c", "subpage"),
|
||||
]
|
||||
for page in manifest["pages"]:
|
||||
html = (folder / f"{page['name']}.html").read_bytes()
|
||||
assert page["sha256"] == hashlib.sha256(html).hexdigest()
|
||||
assert page["files"] == [f"{page['name']}.html", f"{page['name']}.md"]
|
||||
stamp = datetime.datetime.strptime(page["retrieved"], "%Y-%m-%dT%H:%M:%SZ").replace(
|
||||
tzinfo=datetime.timezone.utc)
|
||||
assert before - datetime.timedelta(seconds=2) <= stamp <= datetime.datetime.now(datetime.timezone.utc)
|
||||
assert [p["url"] for p in manifest["pages"][1:]] == [
|
||||
module.url("/mod/a?v=1"), module.url("/mod/b?v=1"), module.url("/mod/c"),
|
||||
]
|
||||
|
||||
|
||||
def test_only_the_urls_the_handler_names_are_requested(tree, module):
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
|
||||
paths = _requested(module)
|
||||
assert sorted(paths) == sorted(["/mod/", "/robots.txt", "/mod/a?v=1", "/mod/b?v=1", "/mod/c"])
|
||||
assert not {"/mod/hidden", "/mod/foreign", "/elsewhere/x"} & set(paths)
|
||||
|
||||
|
||||
def test_the_requests_identify_the_tool(tree, module):
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
assert module.requests and all(agent.startswith("chemenu-wikitool/") for _, agent in module.requests)
|
||||
|
||||
|
||||
def test_the_folder_is_accepted_by_raw_accept_unchanged(tree, module):
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
snapshot = {p.name: p.read_bytes() for p in (tree / "incoming/mod").iterdir()}
|
||||
|
||||
raw_accept_command(
|
||||
files=[tree / "incoming/mod"], fidelity="published", authority="reporting",
|
||||
page=None, replaces=None, dry_run=False,
|
||||
)
|
||||
|
||||
accepted = [p for p in (tree / "raw").rglob("*") if p.is_dir() and p.name == "mod"]
|
||||
assert len(accepted) == 1
|
||||
assert {p.name: p.read_bytes() for p in accepted[0].iterdir()} == snapshot
|
||||
assert not (tree / "incoming/mod").exists()
|
||||
|
||||
|
||||
def test_the_pause_is_taken_before_each_subpage(tree, module, monkeypatch):
|
||||
write_settings(tree, monkeypatch, sites={"pause_seconds": 0.25})
|
||||
slept: list[float] = []
|
||||
monkeypatch.setattr("chemenu.commands.raw.fetch.time.sleep", slept.append)
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
assert slept == [0.25, 0.25, 0.25]
|
||||
|
||||
|
||||
# --- refusals: nothing is written --------------------------------------------
|
||||
|
||||
|
||||
def test_robots_forbidding_a_subpage_refuses_the_fetch_and_names_it(tree, module, capsys):
|
||||
module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /mod/b\n")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
|
||||
assert _snapshot(tree) == before
|
||||
assert module.url("/mod/b?v=1") in capsys.readouterr().out.replace("\n", "")
|
||||
assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module))
|
||||
|
||||
|
||||
def test_a_robots_txt_for_another_agent_does_not_apply(tree, module):
|
||||
module.routes["/robots.txt"] = (
|
||||
200, {"Content-Type": "text/plain"}, b"User-agent: somebodyelse\nDisallow: /\n",
|
||||
)
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
assert (tree / "incoming/mod/c.html").is_file()
|
||||
|
||||
|
||||
def test_a_missing_robots_txt_allows_everything(tree, module):
|
||||
assert "/robots.txt" not in module.routes
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
assert (tree / "incoming/mod/c.html").is_file()
|
||||
|
||||
|
||||
def test_a_robots_txt_that_cannot_be_read_refuses_the_fetch(tree, module, capsys):
|
||||
module.routes["/robots.txt"] = (503, {}, b"later")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
assert _snapshot(tree) == before
|
||||
assert "robots.txt" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_robots_does_not_apply_to_the_page_without_scope(tree, module):
|
||||
module.routes["/robots.txt"] = (200, {"Content-Type": "text/plain"}, b"User-agent: *\nDisallow: /\n")
|
||||
_fetch(module.url("/mod/"), name="mod", scope=False)
|
||||
assert (tree / "incoming/mod.html").is_file()
|
||||
assert "/robots.txt" not in _requested(module)
|
||||
|
||||
|
||||
def test_a_failing_third_fetch_leaves_incoming_as_it_was(tree, module):
|
||||
module.routes["/mod/b?v=1"] = (500, {}, b"broken")
|
||||
(tree / "incoming/other.md").write_text("other", encoding="utf-8")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
|
||||
assert _snapshot(tree) == before
|
||||
assert "/mod/b?v=1" in _requested(module)
|
||||
assert "/mod/c" not in _requested(module)
|
||||
|
||||
|
||||
def test_a_subpage_that_is_not_html_refuses_the_fetch(tree, module):
|
||||
module.routes["/mod/c"] = (200, {"Content-Type": "application/pdf"}, b"%PDF-1.7")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
assert _snapshot(tree) == before
|
||||
|
||||
|
||||
def test_more_subpages_than_the_maximum_refuse_before_the_first_subpage(tree, module, monkeypatch, capsys):
|
||||
write_settings(tree, monkeypatch, sites={"max_pages": 2})
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
|
||||
assert _snapshot(tree) == before
|
||||
assert _requested(module) == ["/mod/"]
|
||||
assert "max_pages" in " ".join(capsys.readouterr().out.split())
|
||||
|
||||
|
||||
def test_a_raised_maximum_takes_effect(tree, module, monkeypatch):
|
||||
write_settings(tree, monkeypatch, sites={"max_pages": 4})
|
||||
_fetch(module.url("/mod/"), name="mod")
|
||||
assert (tree / "incoming/mod/c.md").is_file()
|
||||
|
||||
|
||||
def test_an_existing_folder_is_never_overwritten(tree, module):
|
||||
(tree / "incoming/mod").mkdir()
|
||||
(tree / "incoming/mod/mine.md").write_text("mine", encoding="utf-8")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
assert _snapshot(tree) == before
|
||||
assert not {"/mod/a?v=1", "/mod/b?v=1", "/mod/c"} & set(_requested(module))
|
||||
|
||||
|
||||
def test_a_path_over_the_budget_is_refused_before_a_subpage_is_requested(tree, module):
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="m" * 140)
|
||||
assert _snapshot(tree) == before
|
||||
assert "/mod/a?v=1" not in _requested(module)
|
||||
|
||||
|
||||
def test_a_page_whose_handler_offers_no_subpages_is_refused(tree, module):
|
||||
module.html("/plain", _sub("plain"))
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/plain"), name="plain")
|
||||
assert _snapshot(tree) == before
|
||||
|
||||
|
||||
def test_an_index_without_subpages_is_refused(tree, module):
|
||||
module.html("/mod/", b"<html><body><main><p>No table at all, only text.</p></main></body></html>")
|
||||
before = _snapshot(tree)
|
||||
_refused(module.url("/mod/"), name="mod")
|
||||
assert _snapshot(tree) == before
|
||||
|
||||
|
||||
def test_scope_does_not_go_with_html(tree):
|
||||
saved = tree / "incoming/mod.html"
|
||||
saved.write_bytes(b"<p>x</p>")
|
||||
with pytest.raises(typer.Exit) as excinfo:
|
||||
raw_fetch_command(url=None, html=saved, source_url="http://127.0.0.1/mod/", name=None, scope=True)
|
||||
assert excinfo.value.exit_code == 1
|
||||
|
||||
|
||||
# --- the building block as data ---------------------------------------------
|
||||
|
||||
|
||||
def test_the_stack_handler_for_a_module_offers_its_cmdlet_pages_as_subpages():
|
||||
special = next(s for s in sites.all_sites() if s.name == "learn-microsoft-powershell-module")
|
||||
assert special.subpages is not None and special.subpages.prefix == "{page}"
|
||||
html = (
|
||||
'<table><tr><td><a data-linktype="relative-path" href="get-x?view=v">x</a></td></tr>'
|
||||
'<tr><td><a data-linktype="absolute-path" href="/en-us/other">y</a></td></tr></table>'
|
||||
'<a data-linktype="relative-path" href="get-y">outside the table</a>'
|
||||
)
|
||||
assert special.subpages.urls(html, "https://learn.microsoft.com/en-us/powershell/module/m/?view=v") == [
|
||||
"https://learn.microsoft.com/en-us/powershell/module/m/get-x?view=v",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("subpages", [
|
||||
{},
|
||||
{"links": {}},
|
||||
{"links": {"tag": "a"}, "prefix": "relative/"},
|
||||
{"links": {"tag": "a"}, "extra": 1},
|
||||
])
|
||||
def test_a_malformed_subpages_block_is_a_rejected_file(subpages):
|
||||
from chemenu.sites import data
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
data.build("x", {"host": "example.org", "summary": "s", "subpages": subpages}, "instance")
|
||||
|
||||
|
||||
def test_a_literal_prefix_replaces_the_page_path():
|
||||
block = sites.Subpages(web_capture.Rule(tag="a"), "/docs/ref/")
|
||||
html = '<a href="/docs/ref/one">1</a><a href="/docs/guide/two">2</a><a href="/docs/ref/">self</a>'
|
||||
assert block.urls(html, "https://example.org/docs/index") == ["https://example.org/docs/ref/one"]
|
||||
|
||||
|
||||
def test_the_new_settings_have_their_defaults_and_refuse_nonsense():
|
||||
assert check(SITES, {}).values["max_pages"] == 200
|
||||
assert check(SITES, {}).values["pause_seconds"] == 1
|
||||
assert check(SITES, {"max_pages": 0}).problems
|
||||
assert check(SITES, {"pause_seconds": -1}).problems
|
||||
assert not check(SITES, {"pause_seconds": 0}).problems
|
||||
@@ -60,7 +60,7 @@ def _handler(root: Path, name: str, content) -> Path:
|
||||
|
||||
|
||||
def _fetch(url, name=None, html=None):
|
||||
return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name)
|
||||
return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name, scope=False)
|
||||
|
||||
|
||||
def _refused(url, **kwargs) -> None:
|
||||
|
||||
@@ -57,8 +57,8 @@ def served(monkeypatch):
|
||||
return asked
|
||||
|
||||
|
||||
def _fetch(url=None, html=None, source_url=None, name=None):
|
||||
return raw_fetch_command(url=url, html=html, source_url=source_url, name=name)
|
||||
def _fetch(url=None, html=None, source_url=None, name=None, scope=False):
|
||||
return raw_fetch_command(url=url, html=html, source_url=source_url, name=name, scope=scope)
|
||||
|
||||
|
||||
def _refused(**kwargs) -> None:
|
||||
@@ -130,13 +130,13 @@ def test_the_stem_comes_from_the_standard_form(tree, served):
|
||||
assert (tree / "incoming/get-oslicenseinfo.md").is_file()
|
||||
|
||||
|
||||
def test_the_checks_name_the_source_and_the_cmdlet_pages(tree, served, capsys):
|
||||
def test_the_checks_name_the_source_and_the_subpages(tree, served, capsys):
|
||||
_fetch(MODULE_URL, name="oslicense")
|
||||
lines = _checks(capsys)
|
||||
|
||||
assert len(lines) == 2
|
||||
assert SOURCE_URL in lines[0]
|
||||
assert "11 cmdlet page(s)" in lines[1]
|
||||
assert "index with 11 subpage(s)" in lines[1] and "--scope" in lines[1]
|
||||
|
||||
|
||||
def test_a_page_of_the_general_handler_has_only_the_source_check(tree, served, capsys):
|
||||
@@ -188,7 +188,7 @@ def test_html_applies_the_rules_but_keeps_the_url_as_typed(tree, monkeypatch, ca
|
||||
assert "requested_url" not in header
|
||||
assert header["site"] == "learn-microsoft-powershell-module"
|
||||
assert "Exit editor mode" not in text and "# OSLicense Module" in text
|
||||
assert len(_checks(capsys)) == 2
|
||||
assert len(_checks(capsys)) == 1
|
||||
|
||||
|
||||
# --- the instance's part -----------------------------------------------------
|
||||
@@ -286,7 +286,8 @@ def test_a_special_handler_inherits_from_the_general_one():
|
||||
assert special.drop == general.drop
|
||||
assert special.rewrites == general.rewrites
|
||||
assert special.checks[: len(general.checks)] == general.checks
|
||||
assert len(special.checks) == len(general.checks) + 1
|
||||
assert special.checks == general.checks
|
||||
assert special.subpages is not None and general.subpages is None
|
||||
|
||||
|
||||
def test_a_duplicate_host_and_path_is_a_registration_error(registry):
|
||||
|
||||
@@ -25,6 +25,7 @@ import mimetypes
|
||||
import re
|
||||
import time
|
||||
import unicodedata
|
||||
import urllib.robotparser
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
@@ -66,6 +67,14 @@ class Response:
|
||||
return media_type(self.content_type)
|
||||
|
||||
|
||||
class HttpError(BackendError):
|
||||
"""A server answered with an error status; `status` is the number."""
|
||||
|
||||
def __init__(self, message: str, status: int):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
|
||||
|
||||
def media_type(content_type: str) -> str:
|
||||
if not content_type.strip():
|
||||
return ""
|
||||
@@ -140,7 +149,7 @@ def fetch(
|
||||
body=b"".join(chunks),
|
||||
)
|
||||
except urllib.error.HTTPError as exc:
|
||||
raise BackendError(f"{url} answered HTTP {exc.code} {exc.reason}.") from exc
|
||||
raise HttpError(f"{url} answered HTTP {exc.code} {exc.reason}.", exc.code) from exc
|
||||
except urllib.error.URLError as exc:
|
||||
if isinstance(exc.reason, TimeoutError):
|
||||
raise BackendError(f"{url} did not answer within {timeout:g} s.") from exc
|
||||
@@ -598,8 +607,12 @@ def prune(root: _Element, rules: "tuple[Rule, ...]") -> None:
|
||||
root.children = kept
|
||||
|
||||
|
||||
def matching(document: _Element, rule: Rule) -> "list[_Element]":
|
||||
return [element for element in document.iter() if rule.matches(element)]
|
||||
|
||||
|
||||
def count_matches(document: _Element, rule: Rule) -> int:
|
||||
return sum(1 for element in document.iter() if rule.matches(element))
|
||||
return len(matching(document, rule))
|
||||
|
||||
|
||||
def meta_content(document: _Element, name: str) -> str:
|
||||
@@ -622,6 +635,33 @@ def derive_text(html: str, base_url: str, drop: "tuple[Rule, ...]" = ()) -> str:
|
||||
return _join(blocks)
|
||||
|
||||
|
||||
# --- robots.txt -------------------------------------------------------------
|
||||
|
||||
|
||||
def robots_refusals(
|
||||
urls: "list[str]", user_agent: str, timeout: float = TIMEOUT_SECONDS, max_bytes: int = MAX_BYTES
|
||||
) -> "list[str]":
|
||||
"""The URLs among `urls` (all of one host) that the host's `robots.txt`
|
||||
forbids this tool. A missing `robots.txt` (any 4xx) forbids nothing; one
|
||||
that cannot be read at all (a 5xx, no answer) is a `BackendError`, because
|
||||
"unknown" is not "allowed"."""
|
||||
if not urls:
|
||||
return []
|
||||
parts = urllib.parse.urlsplit(urls[0])
|
||||
robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
|
||||
try:
|
||||
body = fetch(robots_url, user_agent, timeout=timeout, max_bytes=max_bytes).body
|
||||
except HttpError as exc:
|
||||
if 400 <= exc.status < 500:
|
||||
return []
|
||||
raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc
|
||||
except BackendError as exc:
|
||||
raise BackendError(f"{robots_url} could not be read ({exc}) - the subpages are not fetched without it.") from exc
|
||||
parser = urllib.robotparser.RobotFileParser()
|
||||
parser.parse(body.decode("utf-8", errors="replace").splitlines())
|
||||
return [url for url in urls if not parser.can_fetch(user_agent, url)]
|
||||
|
||||
|
||||
# --- the capture file -------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user