diff --git a/.gitignore b/.gitignore index caf4ea4..7555c3a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,6 @@ # Chemenu - .gitignore # -# Rule for this file: `raw/`, `kb/` and `work/` are the repository's content, and +# Rule for this file: `raw/`, `kb/`, `work/` and `sites/` are the repository's content, and # a pattern that silently excludes one of their files is a data-loss bug - the # wiki keeps reporting the file as covered while `publish` never commits it. # So: @@ -193,9 +193,10 @@ npm-debug.log* # `incoming/`. /mcp-upload/ -# Content backstop - keep this block last. Nothing under raw/, kb/ or work/ may +# Content backstop - keep this block last. Nothing under raw/, kb/, work/ or sites/ may # be excluded by a pattern above; see the header note for why directory patterns # still have to be anchored rather than relying on these negations. !raw/** !kb/** !work/** +!sites/** diff --git a/AGENTS.md b/AGENTS.md index b5bdb3a..d9b93d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -262,6 +262,7 @@ and this file. | `kb/` | [kb/CONTRACT.md](kb/CONTRACT.md) + `kb/CONVENTIONS.md` | What the stack enforces about a page (collections, linking, provenance), and beside it what this instance decided (language, naming, tone, labels, hedging) | | `reports/` | [reports/CONTRACT.md](reports/CONTRACT.md) | Why reports and traces are generated, gitignored, and carried into `kb/log.md` | | `work/` | [work/CONTRACT.md](work/CONTRACT.md) | Workshop runs: run keys, required files, why they are tracked, how a run closes | +| `sites/` | [sites/CONTRACT.md](sites/CONTRACT.md) | The instance's own site handlers for `raw fetch`: file format, which handler applies, what a broken file does | | `tools/` | [tools/CONTRACT.md](tools/CONTRACT.md) | Command reference: one generated data record per command (name, synopsis, properties, exit status, retry policy), plus an index and the maintenance schedule - a command to look up (`wikitool -h`, or a `grep` here) rather than a file to read through, as its own opening paragraph says | | `instructions/` | [instructions/CONTRACT.md](instructions/CONTRACT.md) | Instruction vs. skill, publishing, writing standard | diff --git a/CHANGES.md b/CHANGES.md index cff981f..807a074 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 10.0.0-beta.3 - 2026-10-11 - Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen +## 10.0.0-beta.4 - 2026-10-11 - Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind **Author:** Torben Nehmer @@ -29,6 +29,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. **Medium impact** - Einstellungen in drei Stufen: wikitool.json (Instanz), .wikitool-local.json (Checkout), Secrets nur per Verweis - auch über die 1Password-CLI; neues Kommando config set - Site-Handler für raw fetch: Standardform der URL, Bereinigung und Hinweise pro Website als Code-Registry (learn.microsoft.com zuerst); Abschnitt sites in den Einstellungen +- Site-Handler der Instanz als JSON-Dateien in sites/, nach denselben Bausteinen wie die des Stacks - die nun ebenfalls Datendateien sind **Low impact** - Ketten von Paketen: stack-dev schreibt ## Kette, stack-close nennt das nächste Glied @@ -40,9 +41,9 @@ place only: `instructions/dev/version-parts.md` § The candidate model. Neue Einstellungen liegen in `wikitool.json` (eingecheckt, gleich in jedem Klon) oder `.wikitool-local.json` (gitignoriert, nur dieser Checkout); jedes Feld deklariert seine Stufe, ein Wert in der falschen Datei ist ein Fehler, keine Überschreibung. Ein Feld, das ein Programm benennt oder ein Gate betrifft, ist nie `instance`. Secrets stehen nur als Verweis in den Dateien: `{"ref": ""}` über `WIKITOOL_SECRET_` oder eine Secret-Datei im Nutzerverzeichnis, `{"ref": "op://..."}` über die 1Password-CLI, die der Preflight als optionales Werkzeug erfasst; gesperrtes 1Password ergibt exit 42. `config set` schreibt dorthin, wo die Stufe es verlangt, und verweigert Secret-Werte; `doctor` zeigt eine Zeile pro Stufe, `docs verify` prüft `wikitool.json` auf Klartext-Secrets. Noch liest kein Feature seine Einstellungen so. -### Site-Handler für `raw fetch` (#200) +### Site-Handler für `raw fetch` (#200, #210) -Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt die Hinweise der Website als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind Code in `tools/chemenu/sites/` (ein Modul je Website, der längste Pfad gewinnt, ein spezieller erbt vom allgemeinen, eine doppelte Registrierung scheitert beim Import), ihre Bausteine sind auch als Daten ausdrückbar. Der Header nennt `site:` und, wenn die Standardform abweicht, die eingegebene URL als `requested_url`; das `.html` bleibt byte-genau. Erste Handler: `learn-microsoft` und `learn-microsoft-powershell-module`. Der Abschnitt `sites` der Einstellungen (Stufe `instance`) schaltet Handler ab, wählt das Gebietsschema und nimmt bis zu drei Hinweiszeilen je Host-Muster auf. URLs ohne Handler und `raw derive` bleiben unverändert. +Für URLs einer bekannten Website holt `raw fetch` die Seite in Standardform, leitet den Text ohne Seitenbedienung ab und gibt Hinweise als `CHECK`-Zeilen aus, nur wenn sie zutreffen. Handler sind JSON-Dateien nach `tools/chemenu/sites/site.schema.json`: die des Stacks unter `tools/chemenu/sites/handlers/` (zuerst `learn-microsoft` und `learn-microsoft-powershell-module`), die der Instanz unter `sites/`; ein Handler der Instanz mit demselben Host und Pfad ersetzt den des Stacks. `doctor` meldet eine unbrauchbare Datei als FAIL, und `raw fetch` bricht für deren Host mit Exit 1 ab, bevor etwas geholt wird. Der Header nennt `site:` (` (instance)` bei der Instanz) und `requested_url`; das `.html` bleibt byte-genau. Der Abschnitt `sites` der Einstellungen schaltet Handler ab, wählt das Gebietsschema und nimmt Hinweiszeilen je Host-Muster auf. Regeln: `sites/CONTRACT.md`. ### Altlasten der Konfiguration bereinigt (#208) diff --git a/VERSION b/VERSION index 092be9a..fd553b4 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -10.0.0-beta.3 +10.0.0-beta.4 diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index 7d62b94..2ee6090 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md @@ -140,6 +140,8 @@ chemenu/ │ └── CONTRACT.md # Run keys, required files, how a run closes ├── reports/ # DERIVED: lint reports, traces, eval scores. Gitignored │ └── CONTRACT.md +├── sites/ # The instance's own site handlers for `raw fetch`, one JSON file per website +│ └── CONTRACT.md # File format, which handler applies, what a broken file does └── tools/ # COMPILER: the wikitool CLI ├── CONTRACT.md # Command records (generated) and index, maintenance schedule └── README.md # How wikitool is built and how to change it diff --git a/docs/how-to/README.md b/docs/how-to/README.md index e1ab38d..b3a3ada 100644 --- a/docs/how-to/README.md +++ b/docs/how-to/README.md @@ -10,6 +10,7 @@ Working with the wiki: - [Ingest a source](ingest-a-source.md) - a file, a folder, a web page, or a document submitted through the MCP server +- [Write a site handler](write-a-site-handler.md) - teach `raw fetch` how a website serves its pages - [Capture documentation from a git repository](capture-repository-docs.md) - and keep it current - [Export guidelines to captured repositories](export-guidelines.md) - [Query the wiki](query-the-wiki.md) - asking, and searching yourself diff --git a/docs/how-to/write-a-site-handler.md b/docs/how-to/write-a-site-handler.md new file mode 100644 index 0000000..1d1111c --- /dev/null +++ b/docs/how-to/write-a-site-handler.md @@ -0,0 +1,74 @@ +# Write a site handler + +Teach `tools/wikitool raw fetch` how one website serves its pages: the standard form of its URLs, +the page furniture to leave out of the derived text, and a hint or two for whoever reads the +capture. A handler is a JSON file in `sites/` at the root of the instance - data, not a program - +so every clone fetches the website the same way. + +The stack already ships handlers, for `learn.microsoft.com` among others. They are the same kind of +file, so the quickest start is to copy one: + +```bash +cp tools/chemenu/sites/handlers/learn-microsoft.json sites/my-website.json +``` + +The file name without `.json` is the handler's name: lower-case letters, digits and hyphens. + +## 1. Find out what the website does + +Fetch a page of the website and read the derived `.md`: + +```bash +tools/wikitool raw fetch +``` + +Two things are worth a handler: + +- **The same page under several URLs** - a language segment, a tracking parameter. A rewrite + turns every spelling into one standard form, so two people ingesting the page get the same source. +- **Page furniture in the derived text** - a sign-in notice, an action bar, a feedback footer. Open + the saved `.html`, find the element that holds it, and note its `id`, `class` or an attribute. + +## 2. Write the file + +A small handler that drops a cookie banner and a footer on one host: + +```json +{ + "host": "docs.example.org", + "summary": "Drops the cookie banner and the feedback footer.", + "drop": [ + {"id": "cookie-banner"}, + {"class": "feedback", "inside": "footer"} + ] +} +``` + +`host` is required and matched exactly, in lower case. `path` narrows the handler to a part of the +website; among the handlers of a host the longest matching path wins. The other fields - `rewrites`, +`checks`, `parent` - and what each selector may say are in [sites/CONTRACT.md](../../sites/CONTRACT.md). + +## 3. Try it + +Fetch a page again, or derive a saved one without a fetch: + +```bash +tools/wikitool raw fetch +tools/wikitool raw fetch --html --url +``` + +The header of the derived text then says `site: (instance)`, and the furniture is gone. A +check shows up as a `CHECK` line when it applies to the page. + +`tools/wikitool doctor` lists the handlers of the instance and fails on a file that cannot be used, +naming the file and the reason. Such a file also stops a `raw fetch` of its own host with exit 1, +before anything is fetched or written, so a handler that did not apply never leaves furniture in a +source unnoticed. + +## Replace or extend a stack handler + +A handler of the instance for the same host and path as a stack handler **replaces** it. To keep +what the stack knows and add to it, name it as `parent` and give the new file a `path` of its own. + +To switch a handler off without deleting it, list its name under `disabled` in the `sites` section +of the settings - see [Configuration](../reference/configuration.md#site-handlers). diff --git a/docs/reference/commands/instance-health.md b/docs/reference/commands/instance-health.md index e53afb5..5314116 100644 --- a/docs/reference/commands/instance-health.md +++ b/docs/reference/commands/instance-health.md @@ -43,6 +43,7 @@ tools/wikitool doctor --json - Settings (`settings-instance`, `settings-checkout`, `settings-secrets`): one line per tier. `wikitool.json` and `.wikitool-local.json` each present or absent - absent is `OK` - and a `FAIL` naming file, key and the right tier when a value sits in the wrong file, a secret or a committed private field holds a plain value, or a value fails its field; a section this version does not read is a `WARN`. For the references: how many resolve, never a value - an unresolvable one is a `FAIL`, one 1Password will not hand out until a human unlocks it a `WARN`. - Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback. - Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows. +- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such. - Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault. - Read-only and exempt from the Iteration Budget Gate. diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index f7e7599..bc03fe5 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -55,7 +55,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed. - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from. - Derived text: the content root is `
`, else the single `
`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. -- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. +- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. @@ -74,7 +74,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo - `0` - success - `1` - The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given -- `1` - The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist +- `1` - The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - `1` - A target file already exists in `incoming/` - `1` - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB - `1` - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists @@ -82,6 +82,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo **See also** - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why +- `sites/CONTRACT.md` - the instance's own handler files - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers - `wikitool raw accept` - promotes the written files into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 976acd9..905eaaf 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -414,8 +414,9 @@ Size and quota are your deployment's decision, not defaults worth copying blindl derived text and prints hints about the page as `CHECK` lines; today there are two, both for `learn.microsoft.com`: `learn-microsoft` for every page of the host and `learn-microsoft-powershell-module` for a PowerShell module's overview page, which adds a hint -naming the cmdlet pages it lists. Handlers are code in the stack - the instance does not write one. -The `sites` section is what the instance decides about them, and it is optional: without it every +naming the cmdlet pages it lists. Handlers are JSON files: the stack's, and the instance's own under `sites/` +(see [How to write a site handler](../how-to/write-a-site-handler.md)). +The `sites` section is what the instance decides about the handlers, and it is optional: without it every handler applies, in the locale `en-us`. - `disabled` switches handlers off by name. A page of that website is then handled as any other: diff --git a/instructions/dev/doc-pull-through.md b/instructions/dev/doc-pull-through.md index 08166eb..2ea824d 100644 --- a/instructions/dev/doc-pull-through.md +++ b/instructions/dev/doc-pull-through.md @@ -34,7 +34,7 @@ touched; a row that does not apply needs no action. | Touched surface | Document(s) that make a claim about it | |---|---| | A `wikitool` command's behaviour, flags, or interface | Its `cli_contract.CommandRecord` (name, synopsis, properties, exit status, retry policy - `tools/chemenu/cli_contract.py`), then `wikitool docs contract --apply` to regenerate its copy in [tools/CONTRACT.md](../../tools/CONTRACT.md) and `wikitool docs reference --apply` for the human reference under `docs/reference/` (a new group in `cli_contract.GROUPS` first needs its page there, with a title, a one-sentence head and the marker pair) | - | A stage's authoring rules (`raw/`, `kb/`, `types/`, `reports/`, `work/`, `tools/`, `instructions/`) | The touched `/CONTRACT.md` | + | A stage's authoring rules (`raw/`, `kb/`, `types/`, `reports/`, `work/`, `sites/`, `tools/`, `instructions/`) | The touched `/CONTRACT.md` | | A rule, gate, or invariant `AGENTS.md` itself states | The relevant `AGENTS.md` section (Invariants, Gates, File naming, Routing, ...) | | A workflow, stage, or command a human operates by hand | Whichever of `README.md`, `tools/README.md`, `INSTALL.md` or the pages under `docs/tutorials/`, `docs/how-to/` and `docs/reference/` names it - AGENTS.md § File naming says which document is for which reader. A new page goes into the area its kind of documentation belongs to; that area's `README.md` says which | | The reasoning behind a gate, boundary, or design decision | The `docs/explanation/` page that carries it, if one exists (AGENTS.md § File naming lists all seven reached from AGENTS.md itself, plus an eighth reached only from CLAUDE.md). **A decision with no page yet is the gap worth closing**: reasoning that lives only in a Gitea issue never ships - `dist export` carries `docs/explanation/` and no issue tracker, so a distributed instance gets the mechanism without the why | diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index 50c436a..ef2f4f4 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -248,8 +248,9 @@ not claim. website often exists under many URLs - in every language, with tracking parameters - and carries page furniture the browser hides but the HTML contains: an action bar, a sign-in notice, a feedback footer. Left in, it ends up in the derived text, and with it in every claim made from the -page. A *site handler* - code in the stack, one module per website under `tools/chemenu/sites/`, -chosen by host name and the longest matching path pattern - states what is known about that +page. A *site handler* - a JSON file per website, the stack's under `tools/chemenu/sites/handlers/` +and the instance's under `sites/`, chosen by host name and the longest matching path pattern - +states what is known about that website: the standard form of its URLs, the page furniture to leave out of the derived text, and hints about what the derivation cannot do for the reader. For `learn.microsoft.com` that is the locale segment (`en-us`, so two people ingesting the German and the English page get the same @@ -258,12 +259,15 @@ from, which is not fetched. A handler's hints are printed as `CHECK` lines, only to the page; act on one before accepting the capture. The standard form is what is fetched and what `url:` and the stem come from; the URL as typed is -kept in the header as `requested_url:`, and `site:` names the handler. The `.html` is still the +kept in the header as `requested_url:`, and `site:` names the handler (` (instance)` for +one of the instance's own). The `.html` is still the response body byte for byte - only the derived text is cleaned. `--html --url ` removes the same furniture from a saved page but keeps the URL as typed, since no fetch happened. A URL no -handler claims behaves exactly as without handlers. The instance does not write handlers: its +handler claims behaves exactly as without handlers. The instance writes its own handlers as files +under `sites/` and may replace a stack handler for the same host and path; the format, the +resolution and what a broken file does are in [sites/CONTRACT.md](../sites/CONTRACT.md). Its `sites` section in the settings ([docs/reference/configuration.md](../docs/reference/configuration.md)) -switches one off, sets the locale, and adds hint lines of its own per host and path pattern, +switches a handler off, sets the locale, and adds hint lines of its own per host and path pattern, printed as `CHECK` lines too - at most three per pattern. Whether a capture is a whole article or only its teaser cannot be told mechanically - a teaser diff --git a/sites/CONTRACT.md b/sites/CONTRACT.md new file mode 100644 index 0000000..f292249 --- /dev/null +++ b/sites/CONTRACT.md @@ -0,0 +1,64 @@ +# sites/ - Handlers of this Instance + +One JSON file per website, read by `tools/wikitool raw fetch`: what this instance knows about how +a website serves its pages, beyond what the stack's own handlers know. This file is the stack's; +the handler files beside it are the instance's own, and no release touches them. They are +tracked like `kb/` and `raw/`, so every clone fetches a website the same way. + +The stack's handlers are the same kind of file, under `tools/chemenu/sites/handlers/` - open one +to see a working example, copy it to `sites/.json` to start from it. + +## A handler file + +The file name without `.json` is the handler's name: lower-case letters, digits and hyphens. It is +what `disabled` in the `sites` section of the settings names, and what the capture header shows +as `site: (instance)`. + +| Field | Meaning | +|-------|---------| +| `host` | Required. The website's host name, lower case, matched exactly | +| `summary` | Required. One sentence on what the handler does - what a reader of the file needs | +| `path` | Optional. A regular expression matched against the start of the URL path; among the handlers of a host, the longest matching `path` wins. Without it the handler takes every path of the host | +| `parent` | Optional. The name of a handler for the same host whose `rewrites`, `drop` rules and `checks` this one starts with | +| `rewrites` | Optional, at most 5. Each `{"pattern", "replacement"}` is applied to the URL once, in order, as `re.sub(pattern, replacement, url, count=1)`; `{locale}` in a replacement is the `locale` of the `sites` section. The result is the standard form that is fetched; the URL as typed goes into the header as `requested_url` | +| `drop` | Optional, at most 40. Selectors for the page furniture to leave out of the derived text; the `.html` stays as received | +| `checks` | Optional, at most 3 - counting the parent's. Hint lines printed as `CHECK` when they apply to the page | + +A selector in `drop`, and the `count` of a check, is an object with any of `tag`, `id`, `class` +(one token of the element's classes), `attr` (an attribute that must be present, equal to `value` +when that is given) and `inside` (the name of an ancestor element). At least one of `tag`, `id`, +`class`, `attr` is required, and `value` needs `attr`; every field given must match. + +A check is `{"text", "meta"?, "count"?}`. With `meta`, the line applies only when the page has a +`` with that name and content, which replaces `{meta}` in the text. With `count`, +it applies only when at least one element matches the selector, and `{count}` is how many. With +neither, it applies to every page of the handler. A text that uses `{meta}` or `{count}` must have +the field that fills it. + +`tools/chemenu/sites/site.schema.json` is the exact shape; every field outside the table above is +refused. + +## Which handler applies + +A handler of this instance for the same `host` and `path` as a stack handler **replaces** it - the +stack's is not applied at all for that host and path, and `wikitool doctor` names the replacement. +Handlers of the stack that inherited from the replaced one keep what they inherited. To keep what +the stack knows and add to it, give the file a `parent` instead and a `path` of its own. + +A file named like a stack handler but for another host or path is refused, and so are two files +with the same host and path. A `parent` is a handler of the stack or of this instance; a missing +one, or a cycle, refuses the file. + +## A file that cannot be used + +A file that is not valid JSON, breaks the schema, has a pattern that does not compile, or breaks +a rule above is not applied. `wikitool doctor` fails and names the file and the reason. A +`raw fetch` whose URL has the host such a file names - or any URL, when no host can be read from +it - exits 1 before it fetches or writes anything: a handler that silently did not apply would +leave furniture in the text and a locale-specific page in the source. A broken file for another +host does not stop the fetch. + +## Safety + +A pattern runs only against the URL and its path, never against the page. A handler cannot run +code, read a file or reach another host: it is data, not a program. diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 852e1aa..a8d8f25 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -1445,7 +1445,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - 0 success - 1 The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given -- 1 The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist +- 1 The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) - 1 A target file already exists in `incoming/` - 1 The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB - 1 raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists @@ -1453,7 +1453,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a **ON FAILURE** - The URL is not `http`/`https` (also after a redirect), or neither or both of `` and `--html` were given -> Fix the call and retry once. A local file is dropped into `incoming/` by hand, not fetched -- The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist -> Nothing was fetched or written. Fix the section named in the message - `tools/wikitool config set` or the settings file - then retry once +- The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name `, then retry once - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first @@ -1469,7 +1469,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a - The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed. - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from. - Derived text: the content root is `
`, else the single `
`, else ``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text. -- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. +- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch ` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url ` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as ` (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host. - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it. - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header. - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting. @@ -1479,6 +1479,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a **SEE ALSO** - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why +- `sites/CONTRACT.md` - the instance's own handler files - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers - `wikitool raw accept` - promotes the written files into `raw/` - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts @@ -3841,6 +3842,7 @@ Check that this instance is correctly configured. - Settings (`settings-instance`, `settings-checkout`, `settings-secrets`): one line per tier. `wikitool.json` and `.wikitool-local.json` each present or absent - absent is `OK` - and a `FAIL` naming file, key and the right tier when a value sits in the wrong file, a secret or a committed private field holds a plain value, or a value fails its field; a section this version does not read is a `WARN`. For the references: how many resolve, never a value - an unresolvable one is a `FAIL`, one 1Password will not hand out until a human unlocks it a `WARN`. - Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback. - Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows. +- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such. - Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault. - Read-only and exempt from the Iteration Budget Gate. diff --git a/tools/README.md b/tools/README.md index 43b422c..7617e61 100644 --- a/tools/README.md +++ b/tools/README.md @@ -118,7 +118,7 @@ tools/ kb_state.py the KB version (.wikitool-kb.json) and the migration chain corpus_diff.py invariant comparison of kb/ between two revisions web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic - sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`) and one module per website, each registering the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text), standard library only + sites/ the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines` guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached diff --git a/tools/chemenu/commands/dist_cmd.py b/tools/chemenu/commands/dist_cmd.py index 39748fc..adfbdf5 100644 --- a/tools/chemenu/commands/dist_cmd.py +++ b/tools/chemenu/commands/dist_cmd.py @@ -567,7 +567,7 @@ def build_plan(origin: Optional[Origin] = None) -> dict[str, PlannedFile]: # it is `ownership.is_stack_owned`. Only the export-only stubs # (`ownership.EXPORT_STUB_NAMES`) are allowed here without also being # stack-owned - export writes a fresh one rather than shipping this repo's. -_CONTENT_PREFIXES = ("kb/", "raw/") +_CONTENT_PREFIXES = ("kb/", "raw/", "sites/") _INSTANCE_OWNED_KB_FILES = (kb_collections.CONTRACT_NAME, conventions.CONVENTIONS_FILENAME) @@ -598,7 +598,10 @@ def find_leaks(plan: dict[str, PlannedFile]) -> list[str]: and not ownership.is_stack_owned(relative) and not ownership.is_export_stub(name) ): - leaks.append(f"{relative} (wiki content, not machinery)") + leaks.append( + f"{relative} ({'an instance site handler' if relative.startswith('sites/') else 'wiki content'}" + ", not machinery)" + ) return leaks diff --git a/tools/chemenu/commands/docs_verify.py b/tools/chemenu/commands/docs_verify.py index 18c6006..2a54d5a 100644 --- a/tools/chemenu/commands/docs_verify.py +++ b/tools/chemenu/commands/docs_verify.py @@ -95,6 +95,7 @@ STAGE_CONTRACTS = ( "types/type-spec.md", "reports/CONTRACT.md", "work/CONTRACT.md", + "sites/CONTRACT.md", "tools/CONTRACT.md", "instructions/CONTRACT.md", ) @@ -103,7 +104,7 @@ STAGE_CONTRACTS = ( # therefore may never be excluded by an ignore rule. `work/` is here because a # workshop is the only record of a multi-session run: unlike `reports/`, losing # it loses judgment that nothing can recompute. -CONTENT_DIRS = ("raw", "kb", "work") +CONTENT_DIRS = ("raw", "kb", "work", "sites") # Paths that must never be ignored. They deliberately do not have to exist: # `git check-ignore --no-index` answers about the *pattern set*, not the @@ -125,6 +126,9 @@ IGNORE_CANARIES = ( # judgment in progress that nothing can recompute, so an ignore rule # reaching it would silently discard a multi-session run's only record. "work/ingest-documents-example/extract-00-architecture.md", + # `sites/` is the instance's own handler files: an ignore rule over them + # would leave a fetch behaving differently in each clone. + "sites/example.json", ) # The mirror image of IGNORE_CANARIES. `reports/` holds derived output that must diff --git a/tools/chemenu/commands/doctor.py b/tools/chemenu/commands/doctor.py index bc92d97..446427c 100644 --- a/tools/chemenu/commands/doctor.py +++ b/tools/chemenu/commands/doctor.py @@ -298,7 +298,7 @@ def check_structure() -> Check: missing = [] for relative_path in ( "kb/CONTRACT.md", "raw/CONTRACT.md", "reports/CONTRACT.md", - "work/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md", + "work/CONTRACT.md", "sites/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md", ): if not (config.ROOT / relative_path).exists(): missing.append(relative_path) @@ -783,6 +783,27 @@ def check_settings() -> list[Check]: return [instance_check, local_check, Check("settings-secrets", "OK", summary)] +def check_site_handlers() -> Check: + from chemenu import sites + + instance, broken = sites.read_instance() + if broken: + return Check( + "site-handlers", "FAIL", + "; ".join(f"{b.file}: {b.error}" for b in broken) + + " - `raw fetch` refuses a URL of the host such a file names (or any URL, if no host can be read)", + "Fix or remove the file named; `sites/CONTRACT.md` has the format", + ) + if not instance: + return Check("site-handlers", "OK", "no handler files in sites/") + replaced = {mine.name: theirs.name for mine, theirs in sites.replacements(instance)} + detail = ", ".join( + f"{site.name} (replaces the stack's {replaced[site.name]})" if site.name in replaced else site.name + for site in instance + ) + return Check("site-handlers", "OK", f"{len(instance)} handler(s) of this instance: {detail}") + + def check_session_id() -> Check: """Three-valued, not two: an explicit `WIKITOOL_SESSION_ID` and a recognised harness variable (see `chemenu.session.HARNESS_ENV_VARS`) both @@ -905,6 +926,7 @@ def run_doctor() -> list[Check]: check_upload_intake(), check_tasks_provider(), *check_settings(), + check_site_handlers(), check_generated_files(), check_session_id(), check_telemetry(), @@ -981,6 +1003,10 @@ def run_doctor() -> list[Check]: "unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive " "integer, or an unknown key - naming the field; telemetry itself drops or reinterprets " "such a value without a word, so this is where it shows.", + "Site handlers (`site-handlers`): every file in `sites/` is read against the handler " + "schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, " + "naming the file and the reason, and `OK` listing the instance's handlers, each one that " + "replaces a stack handler (the same host and path) named as such.", "Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a " "fault.", "Read-only and exempt from the Iteration Budget Gate.", diff --git a/tools/chemenu/commands/git_publish.py b/tools/chemenu/commands/git_publish.py index bcb61e2..5d22024 100644 --- a/tools/chemenu/commands/git_publish.py +++ b/tools/chemenu/commands/git_publish.py @@ -468,6 +468,7 @@ FILE_GROUPS: tuple[tuple[str, tuple[str, ...], str], ...] = ( ("Harness config", (".claude/", ".github/", ".vibe/", ".agents/"), "hooks and permissions - affects what the harness lets an agent do"), ("Human docs", ("README.md", "INSTALL.md", "CHANGES.md", "docs/"), "prose, no runtime effect"), + ("Site handlers", ("sites/",), "how `raw fetch` treats a website: what it drops and which hints it prints"), ("Workshop", ("work/",), "scratch state, deleted when the run closes"), ) OTHER_GROUP = ("Other", (), "") diff --git a/tools/chemenu/commands/raw/fetch.py b/tools/chemenu/commands/raw/fetch.py index bb31db6..26d6980 100644 --- a/tools/chemenu/commands/raw/fetch.py +++ b/tools/chemenu/commands/raw/fetch.py @@ -110,7 +110,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: _refuse_existing([md_path]) derivation = web_capture.derive_document( - path.read_bytes(), source_url, path.name, url=source_url, drop=plan.drop, site=plan.name + path.read_bytes(), source_url, path.name, url=source_url, drop=plan.drop, site=plan.label ) _write_exclusive([(md_path, derivation.markdown)]) _warn(derivation, md_path) @@ -160,14 +160,18 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: "``; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, " "`template` and `svg` are dropped below it. Headings, lists, code, tables and links " "(made absolute) become Markdown. The same bytes always give the same text.", - "Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), " + "Site handlers: for the pages of a website the stack or the instance knows (the stack's " + "today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), " "`raw fetch ` fetches the standard form of the URL (the locale segment becomes " "`en-us`, the `view=` query stays) and `` as typed goes into the header as " "`requested_url`; `url` is the form fetched. The handler also removes the page furniture " "the website renders around the article from the derived text - never from the `.html` - " "and prints hints that apply to this page as `CHECK` lines: act on one before accepting. " "`--html --url ` removes the same furniture but keeps the URL as typed. A URL no " - "handler claims is handled as before. The `sites` section of the settings switches handlers " + "handler claims is handled as before. The header names an instance's handler as " + "` (instance)`. A handler file of the instance that cannot be used, and names the " + "URL's host or no readable host, stops the fetch before anything is fetched or written. " + "The `sites` section of the settings switches handlers " "off, sets the locale and adds hint lines of the instance's own per host.", "The stem is the URL's last path segment without its extension, else its host name - " "ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.", @@ -192,10 +196,12 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: "hand, not fetched", ), cli_contract.Failure( - cause="The `sites` section of the settings is malformed, or `disabled` names a handler " - "that does not exist", - reaction="Nothing was fetched or written. Fix the section named in the message - " - "`tools/wikitool config set` or the settings file - then retry once", + cause="The `sites` section of the settings is malformed, `disabled` names a handler " + "that does not exist, or a file in `sites/` cannot be used and names this URL's host " + "(or none can be read)", + reaction="Nothing was fetched or written. Fix the section or the handler file named in " + "the message - `tools/wikitool config set`, the settings file or the file in `sites/` " + "(`wikitool doctor` lists the files) - then retry once", ), cli_contract.Failure( cause="A target file already exists in `incoming/`", @@ -229,6 +235,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None: ), see_also=( "`raw/CONTRACT.md` \"Getting a URL in: `raw fetch`\" - the rules and why", + "`sites/CONTRACT.md` - the instance's own handler files", "`docs/reference/configuration.md` - the `sites` section, which tunes the site handlers", "`wikitool raw accept` - promotes the written files into `raw/`", "`instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts", @@ -280,7 +287,7 @@ def raw_fetch_command( _refuse_existing([html_path, md_path]) derivation = web_capture.derive_document( response.body, response.final_url, html_path.name, response=response, - drop=plan.drop, site=plan.name, requested_url=plan.requested_url, + drop=plan.drop, site=plan.label, requested_url=plan.requested_url, ) _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)]) _warn(derivation, md_path) diff --git a/tools/chemenu/ownership.py b/tools/chemenu/ownership.py index 7f963dc..019f542 100644 --- a/tools/chemenu/ownership.py +++ b/tools/chemenu/ownership.py @@ -14,10 +14,11 @@ from __future__ import annotations # The stages whose content belongs to *this instance*, not the stack. Mirrors # the sentence .gitignore already makes about raw/, kb/ and work/ being the # repo's content, plus reports/ - only reports/CONTRACT.md is tracked there, -# the rest is gitignored, so restoring it is a no-op today. It stays in the -# set anyway: a set that is "almost" this one is the beginning of the same +# the rest is gitignored, so restoring it is a no-op today, plus sites/ - the +# instance's own site-handler files, of which only sites/CONTRACT.md is the +# stack's. Reports stays in the set anyway: a set that is "almost" this one is the beginning of the same # drift this module exists to end. -CONTENT_STAGES = ("kb", "raw", "work", "reports") +CONTENT_STAGES = ("kb", "raw", "work", "reports", "sites") # Bare filenames `dist export` overwrites with a fresh stub rather than # shipping the stack's own copy. Not stack-owned: they are the instance's own diff --git a/tools/chemenu/sites/__init__.py b/tools/chemenu/sites/__init__.py index e976580..e639c87 100644 --- a/tools/chemenu/sites/__init__.py +++ b/tools/chemenu/sites/__init__.py @@ -1,20 +1,21 @@ -"""The site handlers behind `wikitool raw fetch`: one module per website, each -registering what is known about how that website serves its pages. +"""The site handlers behind `wikitool raw fetch`: what is known about how one +website serves its pages, as data. A handler turns the URL a user typed into the website's standard form (`canonical`), names the page furniture that is not part of the text (`drop`), and says what the derivation cannot do for the reader (`checks`, printed as `CHECK` lines). The registry is code in this stack - a website is a module here, a registration and tests; the command, the header and the skill stay as they are. -The instance does not write handlers: it switches them off, parametrises them -(`locale`) and adds hint lines of its own (`notes`), all in the `sites` section -of the settings. Every building block is data, not a function: a canonical form is a list of regular-expression replacements, `drop` a list of selectors, a check a text line -with an optional condition. That is what lets a handler of an instance's own be -declared as data later without a second mechanism. The core knows neither -`typer` nor `rich` and raises `ValidationError`. +with an optional condition. So a handler is a JSON file (`data.py` reads it, +`site.schema.json` is its shape): the stack's under `handlers/`, an instance's +in `sites/` at its root, by the same building blocks. An instance handler for a +host and path the stack has a handler for replaces it; the `sites` section of the +settings still switches handlers off, sets the `locale` and adds hint lines +(`notes`). A handler may still be registered from a Python module in this +package. The core knows neither `typer` nor `rich` and raises `ValidationError`. """ from __future__ import annotations @@ -32,6 +33,7 @@ from chemenu.web_capture import Rule MAX_CHECKS = 3 LOCALE_PLACEHOLDER = "{locale}" +INSTANCE_DIRECTORY = "sites" @dataclass(frozen=True) @@ -92,6 +94,7 @@ class Site: rewrites: tuple[Rewrite, ...] = () drop: tuple[Rule, ...] = () checks: tuple[Check, ...] = () + origin: str = "stack" def __post_init__(self) -> None: if not self.name or not self.host or self.host != self.host.lower(): @@ -118,9 +121,27 @@ class Site: _REGISTRY: list[Site] = [] +def inherit(candidate: Site, parent: Site) -> Site: + """`candidate` with its parent's rewrites, rules and checks first, so a + lookup never walks a chain.""" + if parent.host != candidate.host: + raise ValueError(f"site {candidate.name!r}: parent {parent.name!r} is for another host") + flat = replace( + candidate, + rewrites=parent.rewrites + candidate.rewrites, + drop=parent.drop + candidate.drop, + checks=parent.checks + candidate.checks, + ) + if len(flat.checks) > MAX_CHECKS: + raise ValueError( + f"site {candidate.name!r}: with its parent's checks it has {len(flat.checks)}, " + f"at most {MAX_CHECKS} are allowed" + ) + return flat + + def register(candidate: Site) -> Site: - """Add a handler. One that names a `parent` is stored flattened - the - parent's rewrites, rules and checks first - so a lookup never walks a chain.""" + """Add a handler. One that names a `parent` is stored flattened.""" for known in _REGISTRY: if known.name == candidate.name: raise ValueError(f"site {candidate.name!r} is already registered") @@ -133,19 +154,7 @@ def register(candidate: Site) -> Site: parent = next((s for s in _REGISTRY if s.name == candidate.parent), None) if parent is None: raise ValueError(f"site {candidate.name!r}: parent {candidate.parent!r} is not registered (yet)") - if parent.host != candidate.host: - raise ValueError(f"site {candidate.name!r}: parent {parent.name!r} is for another host") - candidate = replace( - candidate, - rewrites=parent.rewrites + candidate.rewrites, - drop=parent.drop + candidate.drop, - checks=parent.checks + candidate.checks, - ) - if len(candidate.checks) > MAX_CHECKS: - raise ValueError( - f"site {candidate.name!r}: with its parent's checks it has {len(candidate.checks)}, " - f"at most {MAX_CHECKS} are allowed" - ) + candidate = inherit(candidate, parent) _REGISTRY.append(candidate) return candidate @@ -159,11 +168,28 @@ def _host_and_path(url: str) -> tuple[str, str]: return (parts.hostname or "").lower(), urllib.parse.unquote(parts.path) or "/" -def site_for(url: str, disabled: frozenset[str] = frozenset()) -> Optional[Site]: +def catalogue(instance: tuple[Site, ...] = ()) -> list[Site]: + """The handlers in effect: the stack's, except those an instance handler + replaces (the same host and path), then the instance's.""" + taken = {(s.host, s.path) for s in instance} + return [s for s in _REGISTRY if (s.host, s.path) not in taken] + list(instance) + + +def replacements(instance: tuple[Site, ...]) -> list[tuple[Site, Site]]: + """Each instance handler paired with the stack handler it replaces.""" + return [ + (mine, theirs) + for mine in instance + for theirs in _REGISTRY + if (mine.host, mine.path) == (theirs.host, theirs.path) + ] + + +def site_for(url: str, disabled: frozenset[str] = frozenset(), instance: tuple[Site, ...] = ()) -> Optional[Site]: """The handler for `url`: the enabled one of its host whose path pattern is the longest among those that match, else None.""" host, path = _host_and_path(url) - found = [s for s in _REGISTRY if s.name not in disabled and s.matches(host, path)] + found = [s for s in catalogue(instance) if s.name not in disabled and s.matches(host, path)] return max(found, key=lambda s: len(s.path), default=None) @@ -177,11 +203,34 @@ class Note: lines: tuple[str, ...] +@dataclass(frozen=True) +class Broken: + """A file of the instance's `sites/` that cannot be used, and why. `host` is + the host it names, when that could be read.""" + + file: str + error: str + host: Optional[str] + + @dataclass(frozen=True) class Preferences: disabled: frozenset[str] = frozenset() locale: str = "en-us" notes: tuple[Note, ...] = () + instance: tuple[Site, ...] = () + broken: tuple[Broken, ...] = () + + +def read_instance(root: "Path | str | None" = None) -> tuple[tuple[Site, ...], tuple[Broken, ...]]: + """The handlers in `sites/` under `root` (default: the instance root), and + the files among them that cannot be used.""" + from chemenu import config + from chemenu.sites import data + + directory = Path(root if root is not None else config.ROOT) / INSTANCE_DIRECTORY + found, broken = data.instance_handlers(directory, list(_REGISTRY)) + return tuple(found), tuple(broken) def read_preferences(root: "Path | str | None" = None) -> Preferences: @@ -189,17 +238,19 @@ def read_preferences(root: "Path | str | None" = None) -> Preferences: A malformed section, or a name in `disabled` that no handler has, is a `ValidationError` - a typo must not leave a handler on.""" section = settings.load(root, only=(config_shapes.SITES.key,)).section(config_shapes.SITES.key) + instance, broken = read_instance(root) + names = {s.name for s in catalogue(instance)} disabled = frozenset(section["disabled"]) - unknown = sorted(disabled - {s.name for s in _REGISTRY}) + unknown = sorted(disabled - names) if unknown: raise ValidationError( f"sites.disabled names {', '.join(unknown)}, which no handler has. " - f"The handlers are: {', '.join(sorted(s.name for s in _REGISTRY))}." + f"The handlers are: {', '.join(sorted(names))}." ) notes = tuple( Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"] ) - return Preferences(disabled, section["locale"], notes) + return Preferences(disabled, section["locale"], notes, instance, broken) @dataclass(frozen=True) @@ -216,6 +267,13 @@ class Plan: def name(self) -> str: return self.site.name if self.site else "" + @property + def label(self) -> str: + """The name as the capture header states it: an instance's own handler says so.""" + if not self.site: + return "" + return f"{self.site.name} (instance)" if self.site.origin == "instance" else self.site.name + @property def drop(self) -> tuple[Rule, ...]: return self.site.drop if self.site else () @@ -229,7 +287,15 @@ class Plan: def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan: """The plan for `url`. `canonical=False` keeps the URL as given - for a page that is already on disk, where the standard form would be a fiction.""" - site = site_for(url, preferences.disabled) + host = _host_and_path(url)[0] + blocking = [b for b in preferences.broken if b.host is None or b.host == host] + if blocking: + raise ValidationError( + f"{host or url}: a handler file of this instance cannot be used, so the fetch is not made - " + + "; ".join(f"{b.file}: {b.error}" for b in blocking) + + ". Fix or remove the file (`wikitool doctor` lists them)." + ) + site = site_for(url, preferences.disabled, preferences.instance) target = url if site is not None and canonical: target = site.canonical(url, preferences.locale) @@ -244,8 +310,13 @@ def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> P def _load_modules() -> None: + from chemenu.sites import data + + for site in data.stack_handlers(): + register(site) for module in pkgutil.iter_modules(__path__): - importlib.import_module(f"{__name__}.{module.name}") + if module.name != "data": + importlib.import_module(f"{__name__}.{module.name}") _load_modules() diff --git a/tools/chemenu/sites/data.py b/tools/chemenu/sites/data.py new file mode 100644 index 0000000..55e811d --- /dev/null +++ b/tools/chemenu/sites/data.py @@ -0,0 +1,181 @@ +"""Site handlers as data: one JSON file per website, read, checked and resolved here. + +The stack ships its handlers as files under `handlers/`; an instance puts its own +under `sites/` at the root, in the same format. The file's name without `.json` is +the handler's name. `site.schema.json` is the shape; what a schema cannot say - a +pattern that compiles, a placeholder with the field it needs, a parent that exists - +is checked here, so a file is rejected whole, with its reason, before it can act. +""" +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import Any, Optional + +from jsonschema import Draft202012Validator + +from chemenu.sites import Broken, Check, Rewrite, Rule, Site, inherit + +SCHEMA_PATH = Path(__file__).with_name("site.schema.json") +STACK_DIRECTORY = Path(__file__).with_name("handlers") +NAME = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*") +MAX_FILE_BYTES = 64 * 1024 + +_validator: Optional[Draft202012Validator] = None + + +def _schema_errors(document: Any) -> list[str]: + global _validator + if _validator is None: + _validator = Draft202012Validator(json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))) + return [ + f"{'/'.join(str(part) for part in error.absolute_path) or '(file)'}: {error.message}" + for error in sorted(_validator.iter_errors(document), key=lambda e: list(map(str, e.absolute_path))) + ] + + +def _rule(entry: dict) -> Rule: + return Rule( + tag=entry.get("tag", ""), + id=entry.get("id", ""), + cls=entry.get("class", ""), + attr=entry.get("attr", ""), + value=entry.get("value"), + inside=entry.get("inside", ""), + ) + + +def build(name: str, document: Any, origin: str) -> Site: + """The handler a file describes, not yet resolved against its parent. + Raises `ValueError` with the reason.""" + if not NAME.fullmatch(name): + raise ValueError(f"the file name {name!r} is not a handler name: lower-case letters, digits and hyphens") + problems = _schema_errors(document) + if problems: + raise ValueError("; ".join(problems)) + rewrites = [] + for step in document.get("rewrites", []): + try: + compiled = re.compile(step["pattern"]) + compiled.sub(step["replacement"].replace("{locale}", "en-us"), "") + except re.error as exc: + raise ValueError(f"rewrites: {exc}") from None + rewrites.append(Rewrite(step["pattern"], step["replacement"])) + checks = [] + for entry in document.get("checks", []): + text = entry["text"] + if "{meta}" in text and "meta" not in entry: + raise ValueError("checks: a text with {meta} needs a `meta` name") + if "{count}" in text and "count" not in entry: + raise ValueError("checks: a text with {count} needs a `count` rule") + checks.append(Check(text, entry.get("meta", ""), _rule(entry["count"]) if "count" in entry else None)) + try: + return Site( + name=name, + host=document["host"], + summary=document["summary"], + path=document.get("path", ""), + parent=document.get("parent", ""), + rewrites=tuple(rewrites), + drop=tuple(_rule(entry) for entry in document.get("drop", [])), + checks=tuple(checks), + origin=origin, + ) + except (ValueError, re.error) as exc: + raise ValueError(str(exc)) from None + + +def read_file(path: Path, origin: str) -> Site: + if path.is_symlink() or not path.is_file(): + raise ValueError("not a regular file") + if path.stat().st_size > MAX_FILE_BYTES: + raise ValueError(f"larger than {MAX_FILE_BYTES // 1024} KiB") + try: + document = json.loads(path.read_text(encoding="utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ValueError(f"not readable JSON: {exc}") from None + return build(path.stem, document, origin) + + +def _host_of(path: Path) -> Optional[str]: + try: + host = json.loads(path.read_text(encoding="utf-8")).get("host") + except (OSError, ValueError, AttributeError): + return None + return host.lower() if isinstance(host, str) and host else None + + +def stack_handlers() -> list[Site]: + """The stack's own handlers as written, a parent before the handlers that + name it - `register` flattens them in that order.""" + parsed = {path.stem: read_file(path, "stack") for path in sorted(STACK_DIRECTORY.glob("*.json"))} + ordered: list[Site] = [] + while parsed: + ready = [n for n, s in parsed.items() if not s.parent or s.parent not in parsed] + if not ready: + raise ValueError(f"stack handlers whose parents form a cycle: {', '.join(sorted(parsed))}") + ordered += [parsed.pop(name) for name in ready] + return ordered + + +def instance_handlers(directory: Path, stack: list[Site]) -> tuple[list[Site], list[Broken]]: + """The handlers in an instance's `sites/` directory, resolved against each + other and the stack's, and the files that could not be used with their reason.""" + broken: list[Broken] = [] + if not directory.is_dir(): + return [], broken + + def fail(path: Path, error: str) -> None: + broken.append(Broken(f"{directory.name}/{path.name}", error, _host_of(path))) + + parsed: dict[str, tuple[Site, Path]] = {} + for path in sorted(directory.glob("*.json")): + try: + parsed[path.stem] = (read_file(path, "instance"), path) + except (ValueError, OSError) as exc: + fail(path, str(exc)) + + stack_by_name = {s.name: s for s in stack} + seen: dict[tuple[str, str], str] = {} + for name, (site, path) in list(parsed.items()): + known = stack_by_name.get(name) + if known is not None and (known.host, known.path) != (site.host, site.path): + fail(path, f"the stack has a handler {name!r} for another host or path; " + "replace it under its own host and path, or pick another name") + del parsed[name] + for name, (site, path) in list(parsed.items()): + key = (site.host, site.path) + if key in seen: + fail(path, f"has the same host and path as {seen[key]}.json") + del parsed[name] + else: + seen[key] = name + + resolved: dict[str, Site] = {} + failed: set[str] = {path.stem for path in directory.glob("*.json")} - set(parsed) + progress = True + while parsed and progress: + progress = False + for name in list(parsed): + site, path = parsed[name] + if site.parent and site.parent in parsed: + continue + progress = True + del parsed[name] + if site.parent: + parent = resolved.get(site.parent) or (None if site.parent in failed else stack_by_name.get(site.parent)) + if parent is None: + fail(path, f"the parent {site.parent!r} is not a usable handler") + failed.add(name) + continue + try: + site = inherit(site, parent) + except ValueError as exc: + fail(path, str(exc)) + failed.add(name) + continue + resolved[name] = site + for name, (_, path) in parsed.items(): + fail(path, "its parents form a cycle") + return list(resolved.values()), broken diff --git a/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json b/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json new file mode 100644 index 0000000..0cfb4d7 --- /dev/null +++ b/tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json @@ -0,0 +1,12 @@ +{ + "host": "learn.microsoft.com", + "summary": "A PowerShell module's overview page: names the cmdlet pages it lists, which are not fetched with it.", + "path": "/[a-z]{2,3}(?:-[a-z0-9]{2,8})+/powershell/module/[^/]+/?$", + "parent": "learn-microsoft", + "checks": [ + { + "text": "This module page lists {count} cmdlet page(s) as links; they are not fetched - ingest the ones the topic needs, each with its own raw fetch.", + "count": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"} + } + ] +} diff --git a/tools/chemenu/sites/handlers/learn-microsoft.json b/tools/chemenu/sites/handlers/learn-microsoft.json new file mode 100644 index 0000000..01003f0 --- /dev/null +++ b/tools/chemenu/sites/handlers/learn-microsoft.json @@ -0,0 +1,27 @@ +{ + "host": "learn.microsoft.com", + "summary": "Fetches the page in the instance's locale (`en-us` unless set otherwise), keeps the `view=` of the URL, drops the page furniture, and names the page's Markdown source.", + "rewrites": [ + { + "pattern": "^(https?://learn\\.microsoft\\.com/)[a-z]{2,3}(?:-[a-z0-9]{2,8})+(?=/)", + "replacement": "\\1{locale}" + } + ], + "drop": [ + {"id": "ms--content-header"}, + {"id": "action-panel"}, + {"id": "article-header"}, + {"attr": "unauthorized-private-section"}, + {"class": "authentication-determined"}, + {"id": "site-user-feedback-footer"}, + {"id": "ms--inline-notifications"}, + {"id": "assertive-live-region"}, + {"id": "polite-live-region"} + ], + "checks": [ + { + "text": "Microsoft keeps the Markdown source of this page at {meta} - not fetched; fetch it separately if the source wording is needed.", + "meta": "original_content_git_url" + } + ] +} diff --git a/tools/chemenu/sites/learn_microsoft.py b/tools/chemenu/sites/learn_microsoft.py deleted file mode 100644 index 5a52736..0000000 --- a/tools/chemenu/sites/learn_microsoft.py +++ /dev/null @@ -1,56 +0,0 @@ -"""learn.microsoft.com: Microsoft Learn, the home of the Microsoft product documentation. - -The same page exists in many locales and in one version per `view=`; the page -furniture around the article (the action bar, the sign-in notice, the feedback -footer) is rendered into the HTML for the script that hides it. -""" -from __future__ import annotations - -import re - -from chemenu.sites import Check, Rewrite, Rule, Site, register - -HOST = "learn.microsoft.com" -LOCALE = r"[a-z]{2,3}(?:-[a-z0-9]{2,8})+" - -register(Site( - name="learn-microsoft", - host=HOST, - summary="Fetches the page in the instance's locale (`en-us` unless set otherwise), keeps the " - "`view=` of the URL, drops the page furniture, and names the page's Markdown source.", - rewrites=(Rewrite(rf"^(https?://{re.escape(HOST)}/){LOCALE}(?=/)", r"\1{locale}"),), - drop=( - Rule(id="ms--content-header"), - Rule(id="action-panel"), - Rule(id="article-header"), - Rule(attr="unauthorized-private-section"), - Rule(cls="authentication-determined"), - Rule(id="site-user-feedback-footer"), - Rule(id="ms--inline-notifications"), - Rule(id="assertive-live-region"), - Rule(id="polite-live-region"), - ), - checks=( - Check( - "Microsoft keeps the Markdown source of this page at {meta} - not fetched; fetch it " - "separately if the source wording is needed.", - meta="original_content_git_url", - ), - ), -)) - -register(Site( - name="learn-microsoft-powershell-module", - host=HOST, - summary="A PowerShell module's overview page: names the cmdlet pages it lists, which are not " - "fetched with it.", - path=rf"/{LOCALE}/powershell/module/[^/]+/?$", - parent="learn-microsoft", - checks=( - Check( - "This module page lists {count} cmdlet page(s) as links; they are not fetched - " - "ingest the ones the topic needs, each with its own raw fetch.", - count=Rule(tag="a", attr="data-linktype", value="relative-path", inside="td"), - ), - ), -)) diff --git a/tools/chemenu/sites/site.schema.json b/tools/chemenu/sites/site.schema.json new file mode 100644 index 0000000..ab7e70b --- /dev/null +++ b/tools/chemenu/sites/site.schema.json @@ -0,0 +1,63 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "A site handler", + "description": "One website's handler for `wikitool raw fetch`: the file's name (without .json) is the handler's name.", + "type": "object", + "additionalProperties": false, + "required": ["host", "summary"], + "properties": { + "host": {"type": "string", "pattern": "^[^\\s/:]+$", "maxLength": 253}, + "summary": {"type": "string", "minLength": 1, "maxLength": 400}, + "path": {"type": "string", "maxLength": 200}, + "parent": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", "maxLength": 80}, + "rewrites": { + "type": "array", + "maxItems": 5, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["pattern", "replacement"], + "properties": { + "pattern": {"type": "string", "minLength": 1, "maxLength": 400}, + "replacement": {"type": "string", "maxLength": 200} + } + } + }, + "drop": {"type": "array", "maxItems": 40, "items": {"$ref": "#/$defs/rule"}}, + "checks": { + "type": "array", + "maxItems": 3, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["text"], + "properties": { + "text": {"type": "string", "minLength": 1, "maxLength": 600}, + "meta": {"type": "string", "minLength": 1, "maxLength": 100}, + "count": {"$ref": "#/$defs/rule"} + } + } + } + }, + "$defs": { + "rule": { + "type": "object", + "additionalProperties": false, + "properties": { + "tag": {"type": "string", "minLength": 1, "maxLength": 50}, + "id": {"type": "string", "minLength": 1, "maxLength": 200}, + "class": {"type": "string", "minLength": 1, "maxLength": 200}, + "attr": {"type": "string", "minLength": 1, "maxLength": 100}, + "value": {"type": "string", "maxLength": 400}, + "inside": {"type": "string", "minLength": 1, "maxLength": 50} + }, + "anyOf": [ + {"required": ["tag"]}, + {"required": ["id"]}, + {"required": ["class"]}, + {"required": ["attr"]} + ], + "dependentRequired": {"value": ["attr"]} + } + } +} diff --git a/tools/chemenu/tests/test_dist_cmd.py b/tools/chemenu/tests/test_dist_cmd.py index 99a9596..f04a13a 100644 --- a/tools/chemenu/tests/test_dist_cmd.py +++ b/tools/chemenu/tests/test_dist_cmd.py @@ -155,7 +155,7 @@ def repo(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: (docs_dir / "dev").mkdir() (docs_dir / "dev" / "development.md").write_text("# Developing this stack\n", encoding="utf-8") - for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md"): + for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md", "sites/CONTRACT.md"): path = root / relative path.parent.mkdir(parents=True, exist_ok=True) path.write_text(f"# {relative}\n", encoding="utf-8") @@ -405,6 +405,7 @@ def test_find_leaks_is_silent_on_a_clean_plan(repo): "instructions/dev/commonplace-kb.md", "kb/entities/aurora.md", "raw/notes/personal-note.md", + "sites/example.json", # Both bind every page and both are the instance's to write, so the # filled name must never cross - only the `.template` beside it does. "kb/CONVENTIONS.md", @@ -453,6 +454,14 @@ def test_plan_ships_collection_contracts_as_templates_not_pages(repo): assert plan["kb/entities/COLLECTION.md.template"].content == "# entities collection\n" +def test_plan_ships_the_sites_contract_and_no_handler_file_of_the_instance(repo): + (repo / "sites").mkdir(exist_ok=True) + (repo / "sites" / "example.json").write_text("{}", encoding="utf-8") + plan = dist_cmd.build_plan() + assert "sites/CONTRACT.md" in plan + assert [relative for relative in plan if relative.startswith("sites/")] == ["sites/CONTRACT.md"] + + def test_plan_ships_the_conventions_template_and_not_the_filled_file(repo): plan = dist_cmd.build_plan() assert "kb/CONVENTIONS.md.template" in plan diff --git a/tools/chemenu/tests/test_doctor.py b/tools/chemenu/tests/test_doctor.py index 6a08b93..372cde1 100644 --- a/tools/chemenu/tests/test_doctor.py +++ b/tools/chemenu/tests/test_doctor.py @@ -61,7 +61,7 @@ def instance(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: (root / "SOUL.md").write_text("# SOUL.md - Fixture\n", encoding="utf-8") for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md", - "instructions/CONTRACT.md", "types/type-spec.md"): + "sites/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md"): path = root / relative path.parent.mkdir(parents=True, exist_ok=True) path.write_text("# contract\n", encoding="utf-8") diff --git a/tools/chemenu/tests/test_site_files.py b/tools/chemenu/tests/test_site_files.py new file mode 100644 index 0000000..893db1e --- /dev/null +++ b/tools/chemenu/tests/test_site_files.py @@ -0,0 +1,289 @@ +"""The instance's own site handlers: JSON files in `sites/` (`chemenu.sites.data`). + +The stack's handlers are the same kind of file, so the checks of `test_sites.py` +over the Learn handlers cover the format itself; this module is about what is +particular to a file the instance wrote - that it takes effect, that it can replace +a stack handler, and that one that cannot be used stops the fetch instead of being +skipped. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import typer +import yaml + +from chemenu import sites, web_capture +from chemenu.commands import doctor +from chemenu.commands.raw.fetch import raw_fetch_command +from chemenu.errors import ValidationError +from chemenu.tests.conftest import write_settings + +PAGE = (Path(__file__).parent / "fixtures/learn_microsoft/oslicense-module.html").read_bytes() + +EXAMPLE = { + "host": "docs.example.org", + "summary": "Example docs: drops the action bar and says the source is elsewhere.", + "drop": [{"id": "action-panel"}, {"attr": "data-list-type", "value": "plan"}], + "checks": [{"text": "The page keeps its source at {meta}.", "meta": "original_content_git_url"}], +} + + +@pytest.fixture +def tree(kb_dir): + root = kb_dir.parent + (root / "raw").mkdir() + (root / "incoming").mkdir(exist_ok=True) + return root + + +@pytest.fixture +def served(monkeypatch): + asked: list[str] = [] + + def fake(url, user_agent, timeout=0, max_bytes=0): + asked.append(url) + return web_capture.Response(url, url, 200, "text/html; charset=utf-8", PAGE) + + monkeypatch.setattr(web_capture, "fetch", fake) + return asked + + +def _handler(root: Path, name: str, content) -> Path: + directory = root / "sites" + directory.mkdir(exist_ok=True) + path = directory / f"{name}.json" + path.write_text(content if isinstance(content, str) else json.dumps(content), encoding="utf-8") + return path + + +def _fetch(url, name=None, html=None): + return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name) + + +def _refused(url, **kwargs) -> None: + with pytest.raises(typer.Exit) as excinfo: + _fetch(url, **kwargs) + assert excinfo.value.exit_code == 1 + + +def _md(path: Path) -> tuple[dict, str]: + head, _, text = path.read_text(encoding="utf-8")[4:].partition("\n---\n") + return yaml.safe_load(head), text + + +# --- a handler of the instance takes effect ---------------------------------- + + +def test_an_instance_handler_cleans_the_page_and_says_so_in_the_header(tree, served, capsys): + _handler(tree, "example", EXAMPLE) + _fetch("https://docs.example.org/guide", name="guide") + header, text = _md(tree / "incoming/guide.md") + + assert header["site"] == "example (instance)" + assert "Add to Plans" not in text and "Exit editor mode" in text + checks = [line for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")] + assert len(checks) == 1 and "docs-source/blob/live" in checks[0] + + +def test_an_instance_handler_rewrites_the_url_like_a_stack_handler(tree, served): + _handler(tree, "example", { + **EXAMPLE, + "rewrites": [{"pattern": r"^(https://docs\.example\.org/)[a-z]{2}/", "replacement": r"\1{locale}/"}], + }) + _fetch("https://docs.example.org/de/guide", name="guide") + header, _ = _md(tree / "incoming/guide.md") + + assert served == ["https://docs.example.org/en-us/guide"] + assert header["requested_url"] == "https://docs.example.org/de/guide" + + +def test_a_handler_for_another_host_does_not_touch_the_page(tree, served): + _handler(tree, "example", EXAMPLE) + _fetch("https://example.org/guide", name="guide") + header, text = _md(tree / "incoming/guide.md") + + assert "site" not in header and "Add to Plans" in text + + +def test_the_handler_applies_to_a_saved_page_too(tree, monkeypatch): + monkeypatch.setattr(web_capture, "fetch", lambda *a, **k: pytest.fail("no network")) + _handler(tree, "example", EXAMPLE) + saved = tree / "incoming/guide.html" + saved.write_bytes(PAGE) + _fetch("https://docs.example.org/guide", html=saved) + + header, text = _md(tree / "incoming/guide.md") + assert header["site"] == "example (instance)" and "Add to Plans" not in text + + +def test_an_instance_handler_may_inherit_from_a_stack_handler(tree, served): + _handler(tree, "learn-extra", { + "host": "learn.microsoft.com", + "summary": "Also drops the plan button.", + "parent": "learn-microsoft", + "path": "/en-us/azure/", + "drop": [{"attr": "data-list-type", "value": "plan"}], + }) + _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure") + header, text = _md(tree / "incoming/azure.md") + + assert header["site"] == "learn-extra (instance)" + assert "Add to Plans" not in text and "Exit editor mode" not in text + + +def test_without_a_sites_directory_nothing_changes(tree): + preferences = sites.read_preferences(tree) + assert preferences.instance == () and preferences.broken == () + + +# --- replacing a stack handler ----------------------------------------------- + + +def test_a_handler_for_the_same_host_and_path_replaces_the_stack_handler(tree, served): + _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Keeps everything.", "drop": [{"id": "action-panel"}]}) + _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure") + header, text = _md(tree / "incoming/azure.md") + + assert header["site"] == "mine (instance)" + assert "Exit editor mode" in text + assert "learn-microsoft" not in [s.name for s in sites.catalogue(sites.read_preferences(tree).instance)] + + +def test_the_stack_handlers_that_inherited_from_a_replaced_one_keep_what_they_inherited(tree, served): + _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."}) + _fetch("https://learn.microsoft.com/de-de/powershell/module/oslicense/", name="module") + header, text = _md(tree / "incoming/module.md") + + assert header["site"] == "learn-microsoft-powershell-module" + assert "Exit editor mode" not in text + + +def test_doctor_lists_the_replacement(tree): + _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."}) + check = doctor.check_site_handlers() + + assert check.status == "OK" + assert "mine (replaces the stack's learn-microsoft)" in check.detail + + +def test_a_replaced_stack_handler_can_no_longer_be_disabled_by_its_name(tree, monkeypatch): + _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."}) + write_settings(tree, monkeypatch, sites={"disabled": ["learn-microsoft"]}) + with pytest.raises(ValidationError, match="no handler has"): + sites.read_preferences(tree) + + +# --- a file that cannot be used ---------------------------------------------- + +BROKEN = { + "unknown field": {**EXAMPLE, "colour": "red"}, + "missing summary": {"host": "docs.example.org"}, + "a rule that matches everything": {**EXAMPLE, "drop": [{"inside": "td"}]}, + "a value without an attr": {**EXAMPLE, "drop": [{"tag": "a", "value": "x"}]}, + "more than three checks": {**EXAMPLE, "checks": [{"text": "a"}] * 4}, + "a pattern that does not compile": {**EXAMPLE, "rewrites": [{"pattern": "(", "replacement": "x"}]}, + "a replacement naming a group that is not there": {**EXAMPLE, "rewrites": [{"pattern": "a", "replacement": r"\1"}]}, + "an upper-case host": {**EXAMPLE, "host": "Docs.Example.org"}, + "a placeholder without its field": {**EXAMPLE, "checks": [{"text": "see {count}"}]}, + "a parent that does not exist": {**EXAMPLE, "parent": "nobody"}, + "a parent for another host": {**EXAMPLE, "parent": "learn-microsoft"}, +} + + +@pytest.mark.parametrize("case", sorted(BROKEN)) +def test_a_file_that_breaks_a_rule_fails_doctor_and_stops_a_fetch_of_its_host(tree, served, case): + _handler(tree, "example", BROKEN[case]) + host = BROKEN[case]["host"].lower() + + check = doctor.check_site_handlers() + assert check.status == "FAIL" and "sites/example.json" in check.detail + + _refused(f"https://{host}/guide", name="guide") + assert served == [] and list((tree / "incoming").iterdir()) == [] + + +def test_a_stack_name_for_another_host_is_refused(tree, served): + _handler(tree, "learn-microsoft", {"host": "other.example.org", "summary": "x"}) + check = doctor.check_site_handlers() + + assert check.status == "FAIL" and "another host or path" in check.detail + _refused("https://other.example.org/guide", name="guide") + assert served == [] + + +def test_a_file_that_is_not_json_stops_every_fetch(tree, served): + _handler(tree, "example", "{not json") + assert doctor.check_site_handlers().status == "FAIL" + _refused("https://example.org/guide", name="guide") + assert served == [] and list((tree / "incoming").iterdir()) == [] + + +def test_a_broken_file_for_another_host_does_not_stop_a_fetch(tree, served): + _handler(tree, "example", {**EXAMPLE, "colour": "red"}) + _fetch("https://example.org/guide", name="guide") + assert served == ["https://example.org/guide"] + + +def test_a_saved_page_is_refused_the_same_way(tree): + _handler(tree, "example", {**EXAMPLE, "colour": "red"}) + saved = tree / "incoming/guide.html" + saved.write_bytes(PAGE) + _refused("https://docs.example.org/guide", html=saved) + assert not (tree / "incoming/guide.md").exists() + + +def test_a_file_whose_name_is_not_a_handler_name_is_broken(tree): + _handler(tree, "My Handler", EXAMPLE) + assert doctor.check_site_handlers().status == "FAIL" + + +def test_two_files_for_the_same_host_and_path_are_an_error(tree): + _handler(tree, "a", EXAMPLE) + _handler(tree, "b", EXAMPLE) + check = doctor.check_site_handlers() + + assert check.status == "FAIL" and "sites/b.json" in check.detail and "same host and path" in check.detail + assert [s.name for s in sites.read_instance(tree)[0]] == ["a"] + + +def test_parents_that_form_a_cycle_are_broken(tree): + _handler(tree, "a", {**EXAMPLE, "path": "/a", "parent": "b"}) + _handler(tree, "b", {**EXAMPLE, "path": "/b", "parent": "a"}) + instance, broken = sites.read_instance(tree) + + assert instance == () and {b.file for b in broken} == {"sites/a.json", "sites/b.json"} + + +def test_a_child_of_a_broken_parent_is_broken_too(tree): + _handler(tree, "base", {**EXAMPLE, "colour": "red"}) + _handler(tree, "child", {**EXAMPLE, "path": "/x", "parent": "base"}) + instance, broken = sites.read_instance(tree) + + assert instance == () and {b.file for b in broken} == {"sites/base.json", "sites/child.json"} + + +def test_a_symlink_is_not_followed(tree): + target = tree / "elsewhere.json" + target.write_text(json.dumps(EXAMPLE), encoding="utf-8") + (tree / "sites").mkdir() + (tree / "sites" / "example.json").symlink_to(target) + assert doctor.check_site_handlers().status == "FAIL" + + +def test_doctor_is_quiet_without_a_sites_directory(tree): + assert doctor.check_site_handlers().status == "OK" + + +# --- the stack's handlers are files, too ------------------------------------- + + +def test_the_stack_handlers_are_data_files_that_pass_their_own_schema(): + from chemenu.sites import data + + names = {path.stem for path in data.STACK_DIRECTORY.glob("*.json")} + assert {"learn-microsoft", "learn-microsoft-powershell-module"} <= names + assert {s.name for s in sites.all_sites()} >= names + assert all(s.origin == "stack" for s in sites.all_sites()) diff --git a/tools/chemenu/toc.py b/tools/chemenu/toc.py index d740714..162d438 100644 --- a/tools/chemenu/toc.py +++ b/tools/chemenu/toc.py @@ -101,7 +101,7 @@ THRESHOLD = 100 # Mirrors docs_verify.STAGE_CONTRACTS - kept as a separate constant rather than # imported, because `docs_verify` importing `toc` (for the new check) would -# make a `toc` -> `docs_verify` import a cycle. Both lists are the seven stage +# make a `toc` -> `docs_verify` import a cycle. Both lists are the stage # contracts; `docs_verify.STAGE_CONTRACTS`'s own docstring is the place that # explains what they are. _STAGE_CONTRACTS = ( @@ -110,6 +110,7 @@ _STAGE_CONTRACTS = ( "types/type-spec.md", "reports/CONTRACT.md", "work/CONTRACT.md", + "sites/CONTRACT.md", "tools/CONTRACT.md", "instructions/CONTRACT.md", )