sites: the instance writes its own site handlers as JSON files under sites/, the stack's learn.microsoft.com handlers become data files in the same format (#210); 10.0.0-beta.4

Files changed:
- .gitignore
- AGENTS.md
- CHANGES.md
- VERSION
- docs/explanation/architecture.md
- docs/how-to/README.md
- docs/how-to/write-a-site-handler.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/configuration.md
- instructions/dev/doc-pull-through.md
- raw/CONTRACT.md
- sites/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/commands/dist_cmd.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/doctor.py
- tools/chemenu/commands/git_publish.py
- tools/chemenu/commands/raw/fetch.py
- tools/chemenu/ownership.py
- tools/chemenu/sites/__init__.py
- tools/chemenu/sites/data.py
- tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json
- tools/chemenu/sites/handlers/learn-microsoft.json
- tools/chemenu/sites/learn_microsoft.py
- tools/chemenu/sites/site.schema.json
- tools/chemenu/tests/test_dist_cmd.py
- tools/chemenu/tests/test_doctor.py
- tools/chemenu/tests/test_site_files.py
- tools/chemenu/toc.py

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

tools/CONTRACT.md

@@ -1445,7 +1445,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 
 - 0    success
 - 1    The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given
-- 1    The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist
+- 1    The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read)
 - 1    A target file already exists in `incoming/`
 - 1    The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB
 - 1    raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists
@@ -1453,7 +1453,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **ON FAILURE**
 
 - The URL is not `http`/`https` (also after a redirect), or neither or both of `<url>` and `--html` were given -> Fix the call and retry once. A local file is dropped into `incoming/` by hand, not fetched
-- The `sites` section of the settings is malformed, or `disabled` names a handler that does not exist -> Nothing was fetched or written. Fix the section named in the message - `tools/wikitool config set` or the settings file - then retry once
+- The `sites` section of the settings is malformed, `disabled` names a handler that does not exist, or a file in `sites/` cannot be used and names this URL's host (or none can be read) -> Nothing was fetched or written. Fix the section or the handler file named in the message - `tools/wikitool config set`, the settings file or the file in `sites/` (`wikitool doctor` lists the files) - then retry once
 - A target file already exists in `incoming/` -> Nothing was written or overwritten. Accept or remove what is there, or pass `--name <stem>`, then retry once
 - The server answered with an HTTP error, could not be reached, took longer than 30 s, or sent more than 25 MiB -> Nothing was written. An HTTP 4xx is not fixed by retrying - check the URL with the user; for a paywall or login, save the page in a browser and use `--html`. An unreachable host or a timeout may be retried once
 - raw fetch --html: `--url` is missing, the file is not directly in `incoming/` or does not exist, or `incoming/<stem>.md` already exists -> Fix the named argument and retry once - nothing was written. A saved page inside a subdirectory of `incoming/` is moved up into `incoming/` first
@@ -1469,7 +1469,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 - The `.html` holds the response body byte for byte. The `.md` starts with a fixed header block (`fetched_by`, `url`, `final_url`, `retrieved`, `http_status`, `content_type`, `charset`, `title`, `derived_from`) followed by the derived text. A page of a website with a site handler adds `site`, and `requested_url` when the URL fetched is not the one typed.
 - Character set, in this order: byte-order mark, the HTTP `Content-Type` charset, a `<meta>` declaration in the first 4 KiB, UTF-8. Undecodable bytes are replaced with U+FFFD and reported; the header names the charset and where it came from.
 - Derived text: the content root is `<main>`, else the single `<article>`, else `<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, `template` and `svg` are dropped below it. Headings, lists, code, tables and links (made absolute) become Markdown. The same bytes always give the same text.
-- Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
+- Site handlers: for the pages of a website the stack or the instance knows (the stack's today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), `raw fetch <url>` fetches the standard form of the URL (the locale segment becomes `en-us`, the `view=` query stays) and `<url>` as typed goes into the header as `requested_url`; `url` is the form fetched. The handler also removes the page furniture the website renders around the article from the derived text - never from the `.html` - and prints hints that apply to this page as `CHECK` lines: act on one before accepting. `--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no handler claims is handled as before. The header names an instance's handler as `<name> (instance)`. A handler file of the instance that cannot be used, and names the URL's host or no readable host, stops the fetch before anything is fetched or written. The `sites` section of the settings switches handlers off, sets the locale and adds hint lines of the instance's own per host.
 - The stem is the URL's last path segment without its extension, else its host name - ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.
 - A `text/plain` or `text/markdown` response is stored as received as `.txt`/`.md`; any other non-HTML response (PDF, image, ...) as received under the extension of its content type. Neither gets a derived file or a header.
 - Only `http`/`https`, on every redirect hop too. 30 s for the whole transfer, 25 MiB at most. No cookies, no JavaScript: a derived text under 200 characters is written, with a warning to read it before accepting.
@@ -1479,6 +1479,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a
 **SEE ALSO**
 
 - `raw/CONTRACT.md` "Getting a URL in: `raw fetch`" - the rules and why
+- `sites/CONTRACT.md` - the instance's own handler files
 - `docs/reference/configuration.md` - the `sites` section, which tunes the site handlers
 - `wikitool raw accept` - promotes the written files into `raw/`
 - `instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts
@@ -3841,6 +3842,7 @@ Check that this instance is correctly configured.
 - Settings (`settings-instance`, `settings-checkout`, `settings-secrets`): one line per tier. `wikitool.json` and `.wikitool-local.json` each present or absent - absent is `OK` - and a `FAIL` naming file, key and the right tier when a value sits in the wrong file, a secret or a committed private field holds a plain value, or a value fails its field; a section this version does not read is a `WARN`. For the references: how many resolve, never a value - an unresolvable one is a `FAIL`, one 1Password will not hand out until a human unlocks it a `WARN`.
 - Session id source: `OK` for `WIKITOOL_SESSION_ID` or a registered harness variable, `WARN` only for the bare parent-pid fallback.
 - Telemetry: on or off and why - installation-form default, the `telemetry` section of `.wikitool-local.json`, or `WIKI_TRACE` - and the current session's count and byte total against both caps; never a `FAIL`. A `WARN` when that section is not read as written - unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive integer, or an unknown key - naming the field; telemetry itself drops or reinterprets such a value without a word, so this is where it shows.
+- Site handlers (`site-handlers`): every file in `sites/` is read against the handler schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, naming the file and the reason, and `OK` listing the instance's handlers, each one that replaces a stack handler (the same host and path) named as such.
 - Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a fault.
 - Read-only and exempt from the Iteration Budget Gate.
 

tools/README.md

@@ -118,7 +118,7 @@ tools/
     kb_state.py            the KB version (.wikitool-kb.json) and the migration chain
     corpus_diff.py         invariant comparison of kb/ between two revisions
     web_capture.py         `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic
-    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`) and one module per website, each registering the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text), standard library only
+    sites/                 the site handlers behind `raw fetch`: a registry (`__init__.py`), the JSON file format (`site.schema.json`, `data.py`) and the stack's handlers as one file per website under `handlers/`, each stating the standard form of its URLs, the page furniture to leave out of the derived text, and hint lines - all of it data (rewrites, selectors, text); the instance's own sit in `sites/` at the root
     filters/               the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Outlook MSG, Office, PDF, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format
     repo_capture.py        `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines`
     guideline_export.py    `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached

tools/chemenu/commands/dist_cmd.py

@@ -567,7 +567,7 @@ def build_plan(origin: Optional[Origin] = None) -> dict[str, PlannedFile]:
 # it is `ownership.is_stack_owned`. Only the export-only stubs
 # (`ownership.EXPORT_STUB_NAMES`) are allowed here without also being
 # stack-owned - export writes a fresh one rather than shipping this repo's.
-_CONTENT_PREFIXES = ("kb/", "raw/")
+_CONTENT_PREFIXES = ("kb/", "raw/", "sites/")
 _INSTANCE_OWNED_KB_FILES = (kb_collections.CONTRACT_NAME, conventions.CONVENTIONS_FILENAME)
 
 
@@ -598,7 +598,10 @@ def find_leaks(plan: dict[str, PlannedFile]) -> list[str]:
             and not ownership.is_stack_owned(relative)
             and not ownership.is_export_stub(name)
         ):
-            leaks.append(f"{relative} (wiki content, not machinery)")
+            leaks.append(
+                f"{relative} ({'an instance site handler' if relative.startswith('sites/') else 'wiki content'}"
+                ", not machinery)"
+            )
     return leaks
 
 

tools/chemenu/commands/docs_verify.py

@@ -95,6 +95,7 @@ STAGE_CONTRACTS = (
     "types/type-spec.md",
     "reports/CONTRACT.md",
     "work/CONTRACT.md",
+    "sites/CONTRACT.md",
     "tools/CONTRACT.md",
     "instructions/CONTRACT.md",
 )
@@ -103,7 +104,7 @@ STAGE_CONTRACTS = (
 # therefore may never be excluded by an ignore rule. `work/` is here because a
 # workshop is the only record of a multi-session run: unlike `reports/`, losing
 # it loses judgment that nothing can recompute.
-CONTENT_DIRS = ("raw", "kb", "work")
+CONTENT_DIRS = ("raw", "kb", "work", "sites")
 
 # Paths that must never be ignored. They deliberately do not have to exist:
 # `git check-ignore --no-index` answers about the *pattern set*, not the
@@ -125,6 +126,9 @@ IGNORE_CANARIES = (
     # judgment in progress that nothing can recompute, so an ignore rule
     # reaching it would silently discard a multi-session run's only record.
     "work/ingest-documents-example/extract-00-architecture.md",
+    # `sites/` is the instance's own handler files: an ignore rule over them
+    # would leave a fetch behaving differently in each clone.
+    "sites/example.json",
 )
 
 # The mirror image of IGNORE_CANARIES. `reports/` holds derived output that must

tools/chemenu/commands/doctor.py

@@ -298,7 +298,7 @@ def check_structure() -> Check:
     missing = []
     for relative_path in (
         "kb/CONTRACT.md", "raw/CONTRACT.md", "reports/CONTRACT.md",
-        "work/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md",
+        "work/CONTRACT.md", "sites/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md",
     ):
         if not (config.ROOT / relative_path).exists():
             missing.append(relative_path)
@@ -783,6 +783,27 @@ def check_settings() -> list[Check]:
     return [instance_check, local_check, Check("settings-secrets", "OK", summary)]
 
 
+def check_site_handlers() -> Check:
+    from chemenu import sites
+
+    instance, broken = sites.read_instance()
+    if broken:
+        return Check(
+            "site-handlers", "FAIL",
+            "; ".join(f"{b.file}: {b.error}" for b in broken)
+            + " - `raw fetch` refuses a URL of the host such a file names (or any URL, if no host can be read)",
+            "Fix or remove the file named; `sites/CONTRACT.md` has the format",
+        )
+    if not instance:
+        return Check("site-handlers", "OK", "no handler files in sites/")
+    replaced = {mine.name: theirs.name for mine, theirs in sites.replacements(instance)}
+    detail = ", ".join(
+        f"{site.name} (replaces the stack's {replaced[site.name]})" if site.name in replaced else site.name
+        for site in instance
+    )
+    return Check("site-handlers", "OK", f"{len(instance)} handler(s) of this instance: {detail}")
+
+
 def check_session_id() -> Check:
     """Three-valued, not two: an explicit `WIKITOOL_SESSION_ID` and a
     recognised harness variable (see `chemenu.session.HARNESS_ENV_VARS`) both
@@ -905,6 +926,7 @@ def run_doctor() -> list[Check]:
         check_upload_intake(),
         check_tasks_provider(),
         *check_settings(),
+        check_site_handlers(),
         check_generated_files(),
         check_session_id(),
         check_telemetry(),
@@ -981,6 +1003,10 @@ def run_doctor() -> list[Check]:
         "unreadable JSON, not an object, `enabled` not a JSON boolean, a cap not a positive "
         "integer, or an unknown key - naming the field; telemetry itself drops or reinterprets "
         "such a value without a word, so this is where it shows.",
+        "Site handlers (`site-handlers`): every file in `sites/` is read against the handler "
+        "schema and the rules the schema cannot state; `FAIL` for a file that cannot be used, "
+        "naming the file and the reason, and `OK` listing the instance's handlers, each one that "
+        "replaces a stack handler (the same host and path) named as such.",
         "Exits 1 only on a `FAIL`; a missing remote, session id or `VERSION` is a `WARN`, not a "
         "fault.",
         "Read-only and exempt from the Iteration Budget Gate.",

tools/chemenu/commands/git_publish.py

@@ -468,6 +468,7 @@ FILE_GROUPS: tuple[tuple[str, tuple[str, ...], str], ...] = (
     ("Harness config", (".claude/", ".github/", ".vibe/", ".agents/"),
      "hooks and permissions - affects what the harness lets an agent do"),
     ("Human docs", ("README.md", "INSTALL.md", "CHANGES.md", "docs/"), "prose, no runtime effect"),
+    ("Site handlers", ("sites/",), "how `raw fetch` treats a website: what it drops and which hints it prints"),
     ("Workshop", ("work/",), "scratch state, deleted when the run closes"),
 )
 OTHER_GROUP = ("Other", (), "")

tools/chemenu/commands/raw/fetch.py

@@ -110,7 +110,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     _refuse_existing([md_path])
 
     derivation = web_capture.derive_document(
-        path.read_bytes(), source_url, path.name, url=source_url, drop=plan.drop, site=plan.name
+        path.read_bytes(), source_url, path.name, url=source_url, drop=plan.drop, site=plan.label
     )
     _write_exclusive([(md_path, derivation.markdown)])
     _warn(derivation, md_path)
@@ -160,14 +160,18 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
         "`<body>`; `script`, `style`, `noscript`, `nav`, `header`, `footer`, `aside`, `form`, "
         "`template` and `svg` are dropped below it. Headings, lists, code, tables and links "
         "(made absolute) become Markdown. The same bytes always give the same text.",
-        "Site handlers: for the pages of a website the stack knows (today `learn.microsoft.com`), "
+        "Site handlers: for the pages of a website the stack or the instance knows (the stack's "
+        "today: `learn.microsoft.com`; the instance's are JSON files in `sites/`), "
         "`raw fetch <url>` fetches the standard form of the URL (the locale segment becomes "
         "`en-us`, the `view=` query stays) and `<url>` as typed goes into the header as "
         "`requested_url`; `url` is the form fetched. The handler also removes the page furniture "
         "the website renders around the article from the derived text - never from the `.html` - "
         "and prints hints that apply to this page as `CHECK` lines: act on one before accepting. "
         "`--html --url <url>` removes the same furniture but keeps the URL as typed. A URL no "
-        "handler claims is handled as before. The `sites` section of the settings switches handlers "
+        "handler claims is handled as before. The header names an instance's handler as "
+        "`<name> (instance)`. A handler file of the instance that cannot be used, and names the "
+        "URL's host or no readable host, stops the fetch before anything is fetched or written. "
+        "The `sites` section of the settings switches handlers "
         "off, sets the locale and adds hint lines of the instance's own per host.",
         "The stem is the URL's last path segment without its extension, else its host name - "
         "ASCII, lowercase, `-`-separated, at most 60 characters; `--name` overrides it.",
@@ -192,10 +196,12 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
             "hand, not fetched",
         ),
         cli_contract.Failure(
-            cause="The `sites` section of the settings is malformed, or `disabled` names a handler "
-            "that does not exist",
-            reaction="Nothing was fetched or written. Fix the section named in the message - "
-            "`tools/wikitool config set` or the settings file - then retry once",
+            cause="The `sites` section of the settings is malformed, `disabled` names a handler "
+            "that does not exist, or a file in `sites/` cannot be used and names this URL's host "
+            "(or none can be read)",
+            reaction="Nothing was fetched or written. Fix the section or the handler file named in "
+            "the message - `tools/wikitool config set`, the settings file or the file in `sites/` "
+            "(`wikitool doctor` lists the files) - then retry once",
         ),
         cli_contract.Failure(
             cause="A target file already exists in `incoming/`",
@@ -229,6 +235,7 @@ def _fetch_html_file(html: Path, source_url: Optional[str]) -> None:
     ),
     see_also=(
         "`raw/CONTRACT.md` \"Getting a URL in: `raw fetch`\" - the rules and why",
+        "`sites/CONTRACT.md` - the instance's own handler files",
         "`docs/reference/configuration.md` - the `sites` section, which tunes the site handlers",
         "`wikitool raw accept` - promotes the written files into `raw/`",
         "`instructions/wiki-ingest/SKILL.md` - where a URL to ingest starts",
@@ -280,7 +287,7 @@ def raw_fetch_command(
         _refuse_existing([html_path, md_path])
         derivation = web_capture.derive_document(
             response.body, response.final_url, html_path.name, response=response,
-            drop=plan.drop, site=plan.name, requested_url=plan.requested_url,
+            drop=plan.drop, site=plan.label, requested_url=plan.requested_url,
         )
         _write_exclusive([(html_path, response.body), (md_path, derivation.markdown)])
         _warn(derivation, md_path)

tools/chemenu/ownership.py

@@ -14,10 +14,11 @@ from __future__ import annotations
 # The stages whose content belongs to *this instance*, not the stack. Mirrors
 # the sentence .gitignore already makes about raw/, kb/ and work/ being the
 # repo's content, plus reports/ - only reports/CONTRACT.md is tracked there,
-# the rest is gitignored, so restoring it is a no-op today. It stays in the
-# set anyway: a set that is "almost" this one is the beginning of the same
+# the rest is gitignored, so restoring it is a no-op today, plus sites/ - the
+# instance's own site-handler files, of which only sites/CONTRACT.md is the
+# stack's. Reports stays in the set anyway: a set that is "almost" this one is the beginning of the same
 # drift this module exists to end.
-CONTENT_STAGES = ("kb", "raw", "work", "reports")
+CONTENT_STAGES = ("kb", "raw", "work", "reports", "sites")
 
 # Bare filenames `dist export` overwrites with a fresh stub rather than
 # shipping the stack's own copy. Not stack-owned: they are the instance's own

tools/chemenu/sites/__init__.py

@@ -1,20 +1,21 @@
-"""The site handlers behind `wikitool raw fetch`: one module per website, each
-registering what is known about how that website serves its pages.
+"""The site handlers behind `wikitool raw fetch`: what is known about how one
+website serves its pages, as data.
 
 A handler turns the URL a user typed into the website's standard form
 (`canonical`), names the page furniture that is not part of the text (`drop`),
 and says what the derivation cannot do for the reader (`checks`, printed as
 `CHECK` lines). The registry is code in this stack - a website is a module here,
 a registration and tests; the command, the header and the skill stay as they are.
-The instance does not write handlers: it switches them off, parametrises them
-(`locale`) and adds hint lines of its own (`notes`), all in the `sites` section
-of the settings.
 
 Every building block is data, not a function: a canonical form is a list of
 regular-expression replacements, `drop` a list of selectors, a check a text line
-with an optional condition. That is what lets a handler of an instance's own be
-declared as data later without a second mechanism. The core knows neither
-`typer` nor `rich` and raises `ValidationError`.
+with an optional condition. So a handler is a JSON file (`data.py` reads it,
+`site.schema.json` is its shape): the stack's under `handlers/`, an instance's
+in `sites/` at its root, by the same building blocks. An instance handler for a
+host and path the stack has a handler for replaces it; the `sites` section of the
+settings still switches handlers off, sets the `locale` and adds hint lines
+(`notes`). A handler may still be registered from a Python module in this
+package. The core knows neither `typer` nor `rich` and raises `ValidationError`.
 """
 from __future__ import annotations
 
@@ -32,6 +33,7 @@ from chemenu.web_capture import Rule
 
 MAX_CHECKS = 3
 LOCALE_PLACEHOLDER = "{locale}"
+INSTANCE_DIRECTORY = "sites"
 
 
 @dataclass(frozen=True)
@@ -92,6 +94,7 @@ class Site:
     rewrites: tuple[Rewrite, ...] = ()
     drop: tuple[Rule, ...] = ()
     checks: tuple[Check, ...] = ()
+    origin: str = "stack"
 
     def __post_init__(self) -> None:
         if not self.name or not self.host or self.host != self.host.lower():
@@ -118,9 +121,27 @@ class Site:
 _REGISTRY: list[Site] = []
 
 
+def inherit(candidate: Site, parent: Site) -> Site:
+    """`candidate` with its parent's rewrites, rules and checks first, so a
+    lookup never walks a chain."""
+    if parent.host != candidate.host:
+        raise ValueError(f"site {candidate.name!r}: parent {parent.name!r} is for another host")
+    flat = replace(
+        candidate,
+        rewrites=parent.rewrites + candidate.rewrites,
+        drop=parent.drop + candidate.drop,
+        checks=parent.checks + candidate.checks,
+    )
+    if len(flat.checks) > MAX_CHECKS:
+        raise ValueError(
+            f"site {candidate.name!r}: with its parent's checks it has {len(flat.checks)}, "
+            f"at most {MAX_CHECKS} are allowed"
+        )
+    return flat
+
+
 def register(candidate: Site) -> Site:
-    """Add a handler. One that names a `parent` is stored flattened - the
-    parent's rewrites, rules and checks first - so a lookup never walks a chain."""
+    """Add a handler. One that names a `parent` is stored flattened."""
     for known in _REGISTRY:
         if known.name == candidate.name:
             raise ValueError(f"site {candidate.name!r} is already registered")
@@ -133,19 +154,7 @@ def register(candidate: Site) -> Site:
         parent = next((s for s in _REGISTRY if s.name == candidate.parent), None)
         if parent is None:
             raise ValueError(f"site {candidate.name!r}: parent {candidate.parent!r} is not registered (yet)")
-        if parent.host != candidate.host:
-            raise ValueError(f"site {candidate.name!r}: parent {parent.name!r} is for another host")
-        candidate = replace(
-            candidate,
-            rewrites=parent.rewrites + candidate.rewrites,
-            drop=parent.drop + candidate.drop,
-            checks=parent.checks + candidate.checks,
-        )
-        if len(candidate.checks) > MAX_CHECKS:
-            raise ValueError(
-                f"site {candidate.name!r}: with its parent's checks it has {len(candidate.checks)}, "
-                f"at most {MAX_CHECKS} are allowed"
-            )
+        candidate = inherit(candidate, parent)
     _REGISTRY.append(candidate)
     return candidate
 
@@ -159,11 +168,28 @@ def _host_and_path(url: str) -> tuple[str, str]:
     return (parts.hostname or "").lower(), urllib.parse.unquote(parts.path) or "/"
 
 
-def site_for(url: str, disabled: frozenset[str] = frozenset()) -> Optional[Site]:
+def catalogue(instance: tuple[Site, ...] = ()) -> list[Site]:
+    """The handlers in effect: the stack's, except those an instance handler
+    replaces (the same host and path), then the instance's."""
+    taken = {(s.host, s.path) for s in instance}
+    return [s for s in _REGISTRY if (s.host, s.path) not in taken] + list(instance)
+
+
+def replacements(instance: tuple[Site, ...]) -> list[tuple[Site, Site]]:
+    """Each instance handler paired with the stack handler it replaces."""
+    return [
+        (mine, theirs)
+        for mine in instance
+        for theirs in _REGISTRY
+        if (mine.host, mine.path) == (theirs.host, theirs.path)
+    ]
+
+
+def site_for(url: str, disabled: frozenset[str] = frozenset(), instance: tuple[Site, ...] = ()) -> Optional[Site]:
     """The handler for `url`: the enabled one of its host whose path pattern is
     the longest among those that match, else None."""
     host, path = _host_and_path(url)
-    found = [s for s in _REGISTRY if s.name not in disabled and s.matches(host, path)]
+    found = [s for s in catalogue(instance) if s.name not in disabled and s.matches(host, path)]
     return max(found, key=lambda s: len(s.path), default=None)
 
 
@@ -177,11 +203,34 @@ class Note:
     lines: tuple[str, ...]
 
 
+@dataclass(frozen=True)
+class Broken:
+    """A file of the instance's `sites/` that cannot be used, and why. `host` is
+    the host it names, when that could be read."""
+
+    file: str
+    error: str
+    host: Optional[str]
+
+
 @dataclass(frozen=True)
 class Preferences:
     disabled: frozenset[str] = frozenset()
     locale: str = "en-us"
     notes: tuple[Note, ...] = ()
+    instance: tuple[Site, ...] = ()
+    broken: tuple[Broken, ...] = ()
+
+
+def read_instance(root: "Path | str | None" = None) -> tuple[tuple[Site, ...], tuple[Broken, ...]]:
+    """The handlers in `sites/` under `root` (default: the instance root), and
+    the files among them that cannot be used."""
+    from chemenu import config
+    from chemenu.sites import data
+
+    directory = Path(root if root is not None else config.ROOT) / INSTANCE_DIRECTORY
+    found, broken = data.instance_handlers(directory, list(_REGISTRY))
+    return tuple(found), tuple(broken)
 
 
 def read_preferences(root: "Path | str | None" = None) -> Preferences:
@@ -189,17 +238,19 @@ def read_preferences(root: "Path | str | None" = None) -> Preferences:
     A malformed section, or a name in `disabled` that no handler has, is a
     `ValidationError` - a typo must not leave a handler on."""
     section = settings.load(root, only=(config_shapes.SITES.key,)).section(config_shapes.SITES.key)
+    instance, broken = read_instance(root)
+    names = {s.name for s in catalogue(instance)}
     disabled = frozenset(section["disabled"])
-    unknown = sorted(disabled - {s.name for s in _REGISTRY})
+    unknown = sorted(disabled - names)
     if unknown:
         raise ValidationError(
             f"sites.disabled names {', '.join(unknown)}, which no handler has. "
-            f"The handlers are: {', '.join(sorted(s.name for s in _REGISTRY))}."
+            f"The handlers are: {', '.join(sorted(names))}."
         )
     notes = tuple(
         Note(entry["host"], entry.get("path", ""), tuple(entry["lines"])) for entry in section["notes"]
     )
-    return Preferences(disabled, section["locale"], notes)
+    return Preferences(disabled, section["locale"], notes, instance, broken)
 
 
 @dataclass(frozen=True)
@@ -216,6 +267,13 @@ class Plan:
     def name(self) -> str:
         return self.site.name if self.site else ""
 
+    @property
+    def label(self) -> str:
+        """The name as the capture header states it: an instance's own handler says so."""
+        if not self.site:
+            return ""
+        return f"{self.site.name} (instance)" if self.site.origin == "instance" else self.site.name
+
     @property
     def drop(self) -> tuple[Rule, ...]:
         return self.site.drop if self.site else ()
@@ -229,7 +287,15 @@ class Plan:
 def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> Plan:
     """The plan for `url`. `canonical=False` keeps the URL as given - for a page
     that is already on disk, where the standard form would be a fiction."""
-    site = site_for(url, preferences.disabled)
+    host = _host_and_path(url)[0]
+    blocking = [b for b in preferences.broken if b.host is None or b.host == host]
+    if blocking:
+        raise ValidationError(
+            f"{host or url}: a handler file of this instance cannot be used, so the fetch is not made - "
+            + "; ".join(f"{b.file}: {b.error}" for b in blocking)
+            + ". Fix or remove the file (`wikitool doctor` lists them)."
+        )
+    site = site_for(url, preferences.disabled, preferences.instance)
     target = url
     if site is not None and canonical:
         target = site.canonical(url, preferences.locale)
@@ -244,8 +310,13 @@ def plan_for(url: str, preferences: Preferences, *, canonical: bool = True) -> P
 
 
 def _load_modules() -> None:
+    from chemenu.sites import data
+
+    for site in data.stack_handlers():
+        register(site)
     for module in pkgutil.iter_modules(__path__):
-        importlib.import_module(f"{__name__}.{module.name}")
+        if module.name != "data":
+            importlib.import_module(f"{__name__}.{module.name}")
 
 
 _load_modules()

tools/chemenu/sites/data.py

@@ -0,0 +1,181 @@
+"""Site handlers as data: one JSON file per website, read, checked and resolved here.
+
+The stack ships its handlers as files under `handlers/`; an instance puts its own
+under `sites/` at the root, in the same format. The file's name without `.json` is
+the handler's name. `site.schema.json` is the shape; what a schema cannot say - a
+pattern that compiles, a placeholder with the field it needs, a parent that exists -
+is checked here, so a file is rejected whole, with its reason, before it can act.
+"""
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+from typing import Any, Optional
+
+from jsonschema import Draft202012Validator
+
+from chemenu.sites import Broken, Check, Rewrite, Rule, Site, inherit
+
+SCHEMA_PATH = Path(__file__).with_name("site.schema.json")
+STACK_DIRECTORY = Path(__file__).with_name("handlers")
+NAME = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")
+MAX_FILE_BYTES = 64 * 1024
+
+_validator: Optional[Draft202012Validator] = None
+
+
+def _schema_errors(document: Any) -> list[str]:
+    global _validator
+    if _validator is None:
+        _validator = Draft202012Validator(json.loads(SCHEMA_PATH.read_text(encoding="utf-8")))
+    return [
+        f"{'/'.join(str(part) for part in error.absolute_path) or '(file)'}: {error.message}"
+        for error in sorted(_validator.iter_errors(document), key=lambda e: list(map(str, e.absolute_path)))
+    ]
+
+
+def _rule(entry: dict) -> Rule:
+    return Rule(
+        tag=entry.get("tag", ""),
+        id=entry.get("id", ""),
+        cls=entry.get("class", ""),
+        attr=entry.get("attr", ""),
+        value=entry.get("value"),
+        inside=entry.get("inside", ""),
+    )
+
+
+def build(name: str, document: Any, origin: str) -> Site:
+    """The handler a file describes, not yet resolved against its parent.
+    Raises `ValueError` with the reason."""
+    if not NAME.fullmatch(name):
+        raise ValueError(f"the file name {name!r} is not a handler name: lower-case letters, digits and hyphens")
+    problems = _schema_errors(document)
+    if problems:
+        raise ValueError("; ".join(problems))
+    rewrites = []
+    for step in document.get("rewrites", []):
+        try:
+            compiled = re.compile(step["pattern"])
+            compiled.sub(step["replacement"].replace("{locale}", "en-us"), "")
+        except re.error as exc:
+            raise ValueError(f"rewrites: {exc}") from None
+        rewrites.append(Rewrite(step["pattern"], step["replacement"]))
+    checks = []
+    for entry in document.get("checks", []):
+        text = entry["text"]
+        if "{meta}" in text and "meta" not in entry:
+            raise ValueError("checks: a text with {meta} needs a `meta` name")
+        if "{count}" in text and "count" not in entry:
+            raise ValueError("checks: a text with {count} needs a `count` rule")
+        checks.append(Check(text, entry.get("meta", ""), _rule(entry["count"]) if "count" in entry else None))
+    try:
+        return Site(
+            name=name,
+            host=document["host"],
+            summary=document["summary"],
+            path=document.get("path", ""),
+            parent=document.get("parent", ""),
+            rewrites=tuple(rewrites),
+            drop=tuple(_rule(entry) for entry in document.get("drop", [])),
+            checks=tuple(checks),
+            origin=origin,
+        )
+    except (ValueError, re.error) as exc:
+        raise ValueError(str(exc)) from None
+
+
+def read_file(path: Path, origin: str) -> Site:
+    if path.is_symlink() or not path.is_file():
+        raise ValueError("not a regular file")
+    if path.stat().st_size > MAX_FILE_BYTES:
+        raise ValueError(f"larger than {MAX_FILE_BYTES // 1024} KiB")
+    try:
+        document = json.loads(path.read_text(encoding="utf-8"))
+    except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+        raise ValueError(f"not readable JSON: {exc}") from None
+    return build(path.stem, document, origin)
+
+
+def _host_of(path: Path) -> Optional[str]:
+    try:
+        host = json.loads(path.read_text(encoding="utf-8")).get("host")
+    except (OSError, ValueError, AttributeError):
+        return None
+    return host.lower() if isinstance(host, str) and host else None
+
+
+def stack_handlers() -> list[Site]:
+    """The stack's own handlers as written, a parent before the handlers that
+    name it - `register` flattens them in that order."""
+    parsed = {path.stem: read_file(path, "stack") for path in sorted(STACK_DIRECTORY.glob("*.json"))}
+    ordered: list[Site] = []
+    while parsed:
+        ready = [n for n, s in parsed.items() if not s.parent or s.parent not in parsed]
+        if not ready:
+            raise ValueError(f"stack handlers whose parents form a cycle: {', '.join(sorted(parsed))}")
+        ordered += [parsed.pop(name) for name in ready]
+    return ordered
+
+
+def instance_handlers(directory: Path, stack: list[Site]) -> tuple[list[Site], list[Broken]]:
+    """The handlers in an instance's `sites/` directory, resolved against each
+    other and the stack's, and the files that could not be used with their reason."""
+    broken: list[Broken] = []
+    if not directory.is_dir():
+        return [], broken
+
+    def fail(path: Path, error: str) -> None:
+        broken.append(Broken(f"{directory.name}/{path.name}", error, _host_of(path)))
+
+    parsed: dict[str, tuple[Site, Path]] = {}
+    for path in sorted(directory.glob("*.json")):
+        try:
+            parsed[path.stem] = (read_file(path, "instance"), path)
+        except (ValueError, OSError) as exc:
+            fail(path, str(exc))
+
+    stack_by_name = {s.name: s for s in stack}
+    seen: dict[tuple[str, str], str] = {}
+    for name, (site, path) in list(parsed.items()):
+        known = stack_by_name.get(name)
+        if known is not None and (known.host, known.path) != (site.host, site.path):
+            fail(path, f"the stack has a handler {name!r} for another host or path; "
+                       "replace it under its own host and path, or pick another name")
+            del parsed[name]
+    for name, (site, path) in list(parsed.items()):
+        key = (site.host, site.path)
+        if key in seen:
+            fail(path, f"has the same host and path as {seen[key]}.json")
+            del parsed[name]
+        else:
+            seen[key] = name
+
+    resolved: dict[str, Site] = {}
+    failed: set[str] = {path.stem for path in directory.glob("*.json")} - set(parsed)
+    progress = True
+    while parsed and progress:
+        progress = False
+        for name in list(parsed):
+            site, path = parsed[name]
+            if site.parent and site.parent in parsed:
+                continue
+            progress = True
+            del parsed[name]
+            if site.parent:
+                parent = resolved.get(site.parent) or (None if site.parent in failed else stack_by_name.get(site.parent))
+                if parent is None:
+                    fail(path, f"the parent {site.parent!r} is not a usable handler")
+                    failed.add(name)
+                    continue
+                try:
+                    site = inherit(site, parent)
+                except ValueError as exc:
+                    fail(path, str(exc))
+                    failed.add(name)
+                    continue
+            resolved[name] = site
+    for name, (_, path) in parsed.items():
+        fail(path, "its parents form a cycle")
+    return list(resolved.values()), broken

tools/chemenu/sites/handlers/learn-microsoft-powershell-module.json

@@ -0,0 +1,12 @@
+{
+  "host": "learn.microsoft.com",
+  "summary": "A PowerShell module's overview page: names the cmdlet pages it lists, which are not fetched with it.",
+  "path": "/[a-z]{2,3}(?:-[a-z0-9]{2,8})+/powershell/module/[^/]+/?$",
+  "parent": "learn-microsoft",
+  "checks": [
+    {
+      "text": "This module page lists {count} cmdlet page(s) as links; they are not fetched - ingest the ones the topic needs, each with its own raw fetch.",
+      "count": {"tag": "a", "attr": "data-linktype", "value": "relative-path", "inside": "td"}
+    }
+  ]
+}

tools/chemenu/sites/handlers/learn-microsoft.json

@@ -0,0 +1,27 @@
+{
+  "host": "learn.microsoft.com",
+  "summary": "Fetches the page in the instance's locale (`en-us` unless set otherwise), keeps the `view=` of the URL, drops the page furniture, and names the page's Markdown source.",
+  "rewrites": [
+    {
+      "pattern": "^(https?://learn\\.microsoft\\.com/)[a-z]{2,3}(?:-[a-z0-9]{2,8})+(?=/)",
+      "replacement": "\\1{locale}"
+    }
+  ],
+  "drop": [
+    {"id": "ms--content-header"},
+    {"id": "action-panel"},
+    {"id": "article-header"},
+    {"attr": "unauthorized-private-section"},
+    {"class": "authentication-determined"},
+    {"id": "site-user-feedback-footer"},
+    {"id": "ms--inline-notifications"},
+    {"id": "assertive-live-region"},
+    {"id": "polite-live-region"}
+  ],
+  "checks": [
+    {
+      "text": "Microsoft keeps the Markdown source of this page at {meta} - not fetched; fetch it separately if the source wording is needed.",
+      "meta": "original_content_git_url"
+    }
+  ]
+}

tools/chemenu/sites/learn_microsoft.py

@@ -1,56 +0,0 @@
-"""learn.microsoft.com: Microsoft Learn, the home of the Microsoft product documentation.
-
-The same page exists in many locales and in one version per `view=`; the page
-furniture around the article (the action bar, the sign-in notice, the feedback
-footer) is rendered into the HTML for the script that hides it.
-"""
-from __future__ import annotations
-
-import re
-
-from chemenu.sites import Check, Rewrite, Rule, Site, register
-
-HOST = "learn.microsoft.com"
-LOCALE = r"[a-z]{2,3}(?:-[a-z0-9]{2,8})+"
-
-register(Site(
-    name="learn-microsoft",
-    host=HOST,
-    summary="Fetches the page in the instance's locale (`en-us` unless set otherwise), keeps the "
-    "`view=` of the URL, drops the page furniture, and names the page's Markdown source.",
-    rewrites=(Rewrite(rf"^(https?://{re.escape(HOST)}/){LOCALE}(?=/)", r"\1{locale}"),),
-    drop=(
-        Rule(id="ms--content-header"),
-        Rule(id="action-panel"),
-        Rule(id="article-header"),
-        Rule(attr="unauthorized-private-section"),
-        Rule(cls="authentication-determined"),
-        Rule(id="site-user-feedback-footer"),
-        Rule(id="ms--inline-notifications"),
-        Rule(id="assertive-live-region"),
-        Rule(id="polite-live-region"),
-    ),
-    checks=(
-        Check(
-            "Microsoft keeps the Markdown source of this page at {meta} - not fetched; fetch it "
-            "separately if the source wording is needed.",
-            meta="original_content_git_url",
-        ),
-    ),
-))
-
-register(Site(
-    name="learn-microsoft-powershell-module",
-    host=HOST,
-    summary="A PowerShell module's overview page: names the cmdlet pages it lists, which are not "
-    "fetched with it.",
-    path=rf"/{LOCALE}/powershell/module/[^/]+/?$",
-    parent="learn-microsoft",
-    checks=(
-        Check(
-            "This module page lists {count} cmdlet page(s) as links; they are not fetched - "
-            "ingest the ones the topic needs, each with its own raw fetch.",
-            count=Rule(tag="a", attr="data-linktype", value="relative-path", inside="td"),
-        ),
-    ),
-))

tools/chemenu/sites/site.schema.json

@@ -0,0 +1,63 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "A site handler",
+  "description": "One website's handler for `wikitool raw fetch`: the file's name (without .json) is the handler's name.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["host", "summary"],
+  "properties": {
+    "host": {"type": "string", "pattern": "^[^\\s/:]+$", "maxLength": 253},
+    "summary": {"type": "string", "minLength": 1, "maxLength": 400},
+    "path": {"type": "string", "maxLength": 200},
+    "parent": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", "maxLength": 80},
+    "rewrites": {
+      "type": "array",
+      "maxItems": 5,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["pattern", "replacement"],
+        "properties": {
+          "pattern": {"type": "string", "minLength": 1, "maxLength": 400},
+          "replacement": {"type": "string", "maxLength": 200}
+        }
+      }
+    },
+    "drop": {"type": "array", "maxItems": 40, "items": {"$ref": "#/$defs/rule"}},
+    "checks": {
+      "type": "array",
+      "maxItems": 3,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["text"],
+        "properties": {
+          "text": {"type": "string", "minLength": 1, "maxLength": 600},
+          "meta": {"type": "string", "minLength": 1, "maxLength": 100},
+          "count": {"$ref": "#/$defs/rule"}
+        }
+      }
+    }
+  },
+  "$defs": {
+    "rule": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "tag": {"type": "string", "minLength": 1, "maxLength": 50},
+        "id": {"type": "string", "minLength": 1, "maxLength": 200},
+        "class": {"type": "string", "minLength": 1, "maxLength": 200},
+        "attr": {"type": "string", "minLength": 1, "maxLength": 100},
+        "value": {"type": "string", "maxLength": 400},
+        "inside": {"type": "string", "minLength": 1, "maxLength": 50}
+      },
+      "anyOf": [
+        {"required": ["tag"]},
+        {"required": ["id"]},
+        {"required": ["class"]},
+        {"required": ["attr"]}
+      ],
+      "dependentRequired": {"value": ["attr"]}
+    }
+  }
+}

tools/chemenu/tests/test_dist_cmd.py

@@ -155,7 +155,7 @@ def repo(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
     (docs_dir / "dev").mkdir()
     (docs_dir / "dev" / "development.md").write_text("# Developing this stack\n", encoding="utf-8")
 
-    for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md"):
+    for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md", "sites/CONTRACT.md"):
         path = root / relative
         path.parent.mkdir(parents=True, exist_ok=True)
         path.write_text(f"# {relative}\n", encoding="utf-8")
@@ -405,6 +405,7 @@ def test_find_leaks_is_silent_on_a_clean_plan(repo):
         "instructions/dev/commonplace-kb.md",
         "kb/entities/aurora.md",
         "raw/notes/personal-note.md",
+        "sites/example.json",
         # Both bind every page and both are the instance's to write, so the
         # filled name must never cross - only the `.template` beside it does.
         "kb/CONVENTIONS.md",
@@ -453,6 +454,14 @@ def test_plan_ships_collection_contracts_as_templates_not_pages(repo):
     assert plan["kb/entities/COLLECTION.md.template"].content == "# entities collection\n"
 
 
+def test_plan_ships_the_sites_contract_and_no_handler_file_of_the_instance(repo):
+    (repo / "sites").mkdir(exist_ok=True)
+    (repo / "sites" / "example.json").write_text("{}", encoding="utf-8")
+    plan = dist_cmd.build_plan()
+    assert "sites/CONTRACT.md" in plan
+    assert [relative for relative in plan if relative.startswith("sites/")] == ["sites/CONTRACT.md"]
+
+
 def test_plan_ships_the_conventions_template_and_not_the_filled_file(repo):
     plan = dist_cmd.build_plan()
     assert "kb/CONVENTIONS.md.template" in plan

tools/chemenu/tests/test_doctor.py

@@ -61,7 +61,7 @@ def instance(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
     (root / "SOUL.md").write_text("# SOUL.md - Fixture\n", encoding="utf-8")
 
     for relative in ("raw/CONTRACT.md", "reports/CONTRACT.md", "work/CONTRACT.md",
-                      "instructions/CONTRACT.md", "types/type-spec.md"):
+                      "sites/CONTRACT.md", "instructions/CONTRACT.md", "types/type-spec.md"):
         path = root / relative
         path.parent.mkdir(parents=True, exist_ok=True)
         path.write_text("# contract\n", encoding="utf-8")

tools/chemenu/tests/test_site_files.py

@@ -0,0 +1,289 @@
+"""The instance's own site handlers: JSON files in `sites/` (`chemenu.sites.data`).
+
+The stack's handlers are the same kind of file, so the checks of `test_sites.py`
+over the Learn handlers cover the format itself; this module is about what is
+particular to a file the instance wrote - that it takes effect, that it can replace
+a stack handler, and that one that cannot be used stops the fetch instead of being
+skipped.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+import typer
+import yaml
+
+from chemenu import sites, web_capture
+from chemenu.commands import doctor
+from chemenu.commands.raw.fetch import raw_fetch_command
+from chemenu.errors import ValidationError
+from chemenu.tests.conftest import write_settings
+
+PAGE = (Path(__file__).parent / "fixtures/learn_microsoft/oslicense-module.html").read_bytes()
+
+EXAMPLE = {
+    "host": "docs.example.org",
+    "summary": "Example docs: drops the action bar and says the source is elsewhere.",
+    "drop": [{"id": "action-panel"}, {"attr": "data-list-type", "value": "plan"}],
+    "checks": [{"text": "The page keeps its source at {meta}.", "meta": "original_content_git_url"}],
+}
+
+
+@pytest.fixture
+def tree(kb_dir):
+    root = kb_dir.parent
+    (root / "raw").mkdir()
+    (root / "incoming").mkdir(exist_ok=True)
+    return root
+
+
+@pytest.fixture
+def served(monkeypatch):
+    asked: list[str] = []
+
+    def fake(url, user_agent, timeout=0, max_bytes=0):
+        asked.append(url)
+        return web_capture.Response(url, url, 200, "text/html; charset=utf-8", PAGE)
+
+    monkeypatch.setattr(web_capture, "fetch", fake)
+    return asked
+
+
+def _handler(root: Path, name: str, content) -> Path:
+    directory = root / "sites"
+    directory.mkdir(exist_ok=True)
+    path = directory / f"{name}.json"
+    path.write_text(content if isinstance(content, str) else json.dumps(content), encoding="utf-8")
+    return path
+
+
+def _fetch(url, name=None, html=None):
+    return raw_fetch_command(url=None if html else url, html=html, source_url=url if html else None, name=name)
+
+
+def _refused(url, **kwargs) -> None:
+    with pytest.raises(typer.Exit) as excinfo:
+        _fetch(url, **kwargs)
+    assert excinfo.value.exit_code == 1
+
+
+def _md(path: Path) -> tuple[dict, str]:
+    head, _, text = path.read_text(encoding="utf-8")[4:].partition("\n---\n")
+    return yaml.safe_load(head), text
+
+
+# --- a handler of the instance takes effect ----------------------------------
+
+
+def test_an_instance_handler_cleans_the_page_and_says_so_in_the_header(tree, served, capsys):
+    _handler(tree, "example", EXAMPLE)
+    _fetch("https://docs.example.org/guide", name="guide")
+    header, text = _md(tree / "incoming/guide.md")
+
+    assert header["site"] == "example (instance)"
+    assert "Add to Plans" not in text and "Exit editor mode" in text
+    checks = [line for line in capsys.readouterr().out.splitlines() if line.startswith("CHECK ")]
+    assert len(checks) == 1 and "docs-source/blob/live" in checks[0]
+
+
+def test_an_instance_handler_rewrites_the_url_like_a_stack_handler(tree, served):
+    _handler(tree, "example", {
+        **EXAMPLE,
+        "rewrites": [{"pattern": r"^(https://docs\.example\.org/)[a-z]{2}/", "replacement": r"\1{locale}/"}],
+    })
+    _fetch("https://docs.example.org/de/guide", name="guide")
+    header, _ = _md(tree / "incoming/guide.md")
+
+    assert served == ["https://docs.example.org/en-us/guide"]
+    assert header["requested_url"] == "https://docs.example.org/de/guide"
+
+
+def test_a_handler_for_another_host_does_not_touch_the_page(tree, served):
+    _handler(tree, "example", EXAMPLE)
+    _fetch("https://example.org/guide", name="guide")
+    header, text = _md(tree / "incoming/guide.md")
+
+    assert "site" not in header and "Add to Plans" in text
+
+
+def test_the_handler_applies_to_a_saved_page_too(tree, monkeypatch):
+    monkeypatch.setattr(web_capture, "fetch", lambda *a, **k: pytest.fail("no network"))
+    _handler(tree, "example", EXAMPLE)
+    saved = tree / "incoming/guide.html"
+    saved.write_bytes(PAGE)
+    _fetch("https://docs.example.org/guide", html=saved)
+
+    header, text = _md(tree / "incoming/guide.md")
+    assert header["site"] == "example (instance)" and "Add to Plans" not in text
+
+
+def test_an_instance_handler_may_inherit_from_a_stack_handler(tree, served):
+    _handler(tree, "learn-extra", {
+        "host": "learn.microsoft.com",
+        "summary": "Also drops the plan button.",
+        "parent": "learn-microsoft",
+        "path": "/en-us/azure/",
+        "drop": [{"attr": "data-list-type", "value": "plan"}],
+    })
+    _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure")
+    header, text = _md(tree / "incoming/azure.md")
+
+    assert header["site"] == "learn-extra (instance)"
+    assert "Add to Plans" not in text and "Exit editor mode" not in text
+
+
+def test_without_a_sites_directory_nothing_changes(tree):
+    preferences = sites.read_preferences(tree)
+    assert preferences.instance == () and preferences.broken == ()
+
+
+# --- replacing a stack handler -----------------------------------------------
+
+
+def test_a_handler_for_the_same_host_and_path_replaces_the_stack_handler(tree, served):
+    _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Keeps everything.", "drop": [{"id": "action-panel"}]})
+    _fetch("https://learn.microsoft.com/en-us/azure/overview", name="azure")
+    header, text = _md(tree / "incoming/azure.md")
+
+    assert header["site"] == "mine (instance)"
+    assert "Exit editor mode" in text
+    assert "learn-microsoft" not in [s.name for s in sites.catalogue(sites.read_preferences(tree).instance)]
+
+
+def test_the_stack_handlers_that_inherited_from_a_replaced_one_keep_what_they_inherited(tree, served):
+    _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."})
+    _fetch("https://learn.microsoft.com/de-de/powershell/module/oslicense/", name="module")
+    header, text = _md(tree / "incoming/module.md")
+
+    assert header["site"] == "learn-microsoft-powershell-module"
+    assert "Exit editor mode" not in text
+
+
+def test_doctor_lists_the_replacement(tree):
+    _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."})
+    check = doctor.check_site_handlers()
+
+    assert check.status == "OK"
+    assert "mine (replaces the stack's learn-microsoft)" in check.detail
+
+
+def test_a_replaced_stack_handler_can_no_longer_be_disabled_by_its_name(tree, monkeypatch):
+    _handler(tree, "mine", {"host": "learn.microsoft.com", "summary": "Replaces the general one."})
+    write_settings(tree, monkeypatch, sites={"disabled": ["learn-microsoft"]})
+    with pytest.raises(ValidationError, match="no handler has"):
+        sites.read_preferences(tree)
+
+
+# --- a file that cannot be used ----------------------------------------------
+
+BROKEN = {
+    "unknown field": {**EXAMPLE, "colour": "red"},
+    "missing summary": {"host": "docs.example.org"},
+    "a rule that matches everything": {**EXAMPLE, "drop": [{"inside": "td"}]},
+    "a value without an attr": {**EXAMPLE, "drop": [{"tag": "a", "value": "x"}]},
+    "more than three checks": {**EXAMPLE, "checks": [{"text": "a"}] * 4},
+    "a pattern that does not compile": {**EXAMPLE, "rewrites": [{"pattern": "(", "replacement": "x"}]},
+    "a replacement naming a group that is not there": {**EXAMPLE, "rewrites": [{"pattern": "a", "replacement": r"\1"}]},
+    "an upper-case host": {**EXAMPLE, "host": "Docs.Example.org"},
+    "a placeholder without its field": {**EXAMPLE, "checks": [{"text": "see {count}"}]},
+    "a parent that does not exist": {**EXAMPLE, "parent": "nobody"},
+    "a parent for another host": {**EXAMPLE, "parent": "learn-microsoft"},
+}
+
+
+@pytest.mark.parametrize("case", sorted(BROKEN))
+def test_a_file_that_breaks_a_rule_fails_doctor_and_stops_a_fetch_of_its_host(tree, served, case):
+    _handler(tree, "example", BROKEN[case])
+    host = BROKEN[case]["host"].lower()
+
+    check = doctor.check_site_handlers()
+    assert check.status == "FAIL" and "sites/example.json" in check.detail
+
+    _refused(f"https://{host}/guide", name="guide")
+    assert served == [] and list((tree / "incoming").iterdir()) == []
+
+
+def test_a_stack_name_for_another_host_is_refused(tree, served):
+    _handler(tree, "learn-microsoft", {"host": "other.example.org", "summary": "x"})
+    check = doctor.check_site_handlers()
+
+    assert check.status == "FAIL" and "another host or path" in check.detail
+    _refused("https://other.example.org/guide", name="guide")
+    assert served == []
+
+
+def test_a_file_that_is_not_json_stops_every_fetch(tree, served):
+    _handler(tree, "example", "{not json")
+    assert doctor.check_site_handlers().status == "FAIL"
+    _refused("https://example.org/guide", name="guide")
+    assert served == [] and list((tree / "incoming").iterdir()) == []
+
+
+def test_a_broken_file_for_another_host_does_not_stop_a_fetch(tree, served):
+    _handler(tree, "example", {**EXAMPLE, "colour": "red"})
+    _fetch("https://example.org/guide", name="guide")
+    assert served == ["https://example.org/guide"]
+
+
+def test_a_saved_page_is_refused_the_same_way(tree):
+    _handler(tree, "example", {**EXAMPLE, "colour": "red"})
+    saved = tree / "incoming/guide.html"
+    saved.write_bytes(PAGE)
+    _refused("https://docs.example.org/guide", html=saved)
+    assert not (tree / "incoming/guide.md").exists()
+
+
+def test_a_file_whose_name_is_not_a_handler_name_is_broken(tree):
+    _handler(tree, "My Handler", EXAMPLE)
+    assert doctor.check_site_handlers().status == "FAIL"
+
+
+def test_two_files_for_the_same_host_and_path_are_an_error(tree):
+    _handler(tree, "a", EXAMPLE)
+    _handler(tree, "b", EXAMPLE)
+    check = doctor.check_site_handlers()
+
+    assert check.status == "FAIL" and "sites/b.json" in check.detail and "same host and path" in check.detail
+    assert [s.name for s in sites.read_instance(tree)[0]] == ["a"]
+
+
+def test_parents_that_form_a_cycle_are_broken(tree):
+    _handler(tree, "a", {**EXAMPLE, "path": "/a", "parent": "b"})
+    _handler(tree, "b", {**EXAMPLE, "path": "/b", "parent": "a"})
+    instance, broken = sites.read_instance(tree)
+
+    assert instance == () and {b.file for b in broken} == {"sites/a.json", "sites/b.json"}
+
+
+def test_a_child_of_a_broken_parent_is_broken_too(tree):
+    _handler(tree, "base", {**EXAMPLE, "colour": "red"})
+    _handler(tree, "child", {**EXAMPLE, "path": "/x", "parent": "base"})
+    instance, broken = sites.read_instance(tree)
+
+    assert instance == () and {b.file for b in broken} == {"sites/base.json", "sites/child.json"}
+
+
+def test_a_symlink_is_not_followed(tree):
+    target = tree / "elsewhere.json"
+    target.write_text(json.dumps(EXAMPLE), encoding="utf-8")
+    (tree / "sites").mkdir()
+    (tree / "sites" / "example.json").symlink_to(target)
+    assert doctor.check_site_handlers().status == "FAIL"
+
+
+def test_doctor_is_quiet_without_a_sites_directory(tree):
+    assert doctor.check_site_handlers().status == "OK"
+
+
+# --- the stack's handlers are files, too -------------------------------------
+
+
+def test_the_stack_handlers_are_data_files_that_pass_their_own_schema():
+    from chemenu.sites import data
+
+    names = {path.stem for path in data.STACK_DIRECTORY.glob("*.json")}
+    assert {"learn-microsoft", "learn-microsoft-powershell-module"} <= names
+    assert {s.name for s in sites.all_sites()} >= names
+    assert all(s.origin == "stack" for s in sites.all_sites())

tools/chemenu/toc.py

@@ -101,7 +101,7 @@ THRESHOLD = 100
 
 # Mirrors docs_verify.STAGE_CONTRACTS - kept as a separate constant rather than
 # imported, because `docs_verify` importing `toc` (for the new check) would
-# make a `toc` -> `docs_verify` import a cycle. Both lists are the seven stage
+# make a `toc` -> `docs_verify` import a cycle. Both lists are the stage
 # contracts; `docs_verify.STAGE_CONTRACTS`'s own docstring is the place that
 # explains what they are.
 _STAGE_CONTRACTS = (
@@ -110,6 +110,7 @@ _STAGE_CONTRACTS = (
     "types/type-spec.md",
     "reports/CONTRACT.md",
     "work/CONTRACT.md",
+    "sites/CONTRACT.md",
     "tools/CONTRACT.md",
     "instructions/CONTRACT.md",
 )