feat: bug-report collector tools/bugreport.py and instructions/bug-report.md (#157)

Files changed: - .gitea/workflows/ci.yml - CHANGES.md - INSTALL.md - VERSION - instructions/bug-report.md - instructions/gates.md - instructions/setup-instance.md - instructions/upgrade-instance.md - reports/CONTRACT.md - tools/README.md - tools/bugreport.py - tools/chemenu/tests/test_bugreport.py
2026-09-30 17:29:39 +02:00
1 parent 40413f966d
commit f9c047bd2e
@@ -279,3 +279,18 @@ jobs:
            echo "reports/telemetry/ exists in a fresh distributed instance - telemetry should default off"
            exit 1
          fi
      - name: The bug-report collector works in the distribution
        # tools/bugreport.py is the one part of the stack that must run when
        # nothing else does, so it is exercised as shipped: from the export,
        # on the runner's plain python3, in the instance the step above set up.
        # The unit tests cover what it collects; this covers that it arrives.
        run: |
          set -eu
          cd "$DIST_DIR"
          python3 tools/bugreport.py --no-trace
          bundle=$(ls -d reports/bugreport-*/ | head -n 1)
          test -f "$bundle/MANIFEST.md"
          test -f "$bundle/environment.json"
          grep -q '`wikitool` started' "$bundle/MANIFEST.md"
          ls reports/bugreport-*.zip
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
 ---
-## 8.0.0-beta.4 - 2026-09-30 - Demo corpus follows the decided project pages: three states, seed with their items (#156)
+## 8.0.0-beta.5 - 2026-09-30 - Bug-report collector: tools/bugreport.py and instructions/bug-report.md
 **Author:** Torben Nehmer
@@ -80,6 +80,7 @@ concern - readable here, never shipped as something to parse.
 - Budget gate and loop-breaker refusals exit without a traceback
 - Super Productivity API path: unwrap the {ok, data} envelope, exclude the inbox project, ready-aware health (#162)
 - Live tracker suite: WIKITOOL_TASKS_CONFIG override, real-tracker tests for Super Productivity and CalDAV, nightly workflow and test image
 - Bug-report collector: tools/bugreport.py and instructions/bug-report.md
 **Low impact**
 - version bump no longer points at version release in its output
@@ -111,6 +112,34 @@ concern - readable here, never shipped as something to parse.
 - Demo corpus follows the decided project pages: three states, seed with their items (#156)
 <!-- /wikitool:bumps -->
 ### Bug-report collector: tools/bugreport.py and instructions/bug-report.md (Gitea #157)
 A failure on one machine used to reach the maintainer as a description. `tools/bugreport.py` now
 collects the first round of answers into one bundle, and it does so when `wikitool` itself does
 not start: it uses the standard library only, imports nothing from `chemenu`, and keeps to
 Python 3.8 syntax.
 - **Four layers.** The environment (OS, every Python and shell found, harness, `PATH`, git
  configuration, venv, line endings, on Windows also long paths, execution policy and
  mark-of-the-web), the stack (`VERSION`, the `.wikitool-*.json` files, git status and log, the
  shape of `kb/` and `raw/` with path lengths and names that break on Windows), verbatim output of
  `version show`, `doctor`, `budget status`, `instructions verify` and `docs verify` plus the
  caller's session trace, and the agent's chronology with any transcripts. The result is
  `reports/bugreport-<UTC stamp>/` and a zip beside it.
 - **Secrets are always removed; page titles are kept out of everything the script generates**
  unless `--titles` is given. Environment variable names are all recorded, values only for a
  fixed list. The trace, the chronology and the transcripts are marked in the manifest as
  possibly containing page content and titles. Nothing is uploaded.
 - **Its own session id.** The two counting calls run under `WIKITOOL_SESSION_ID=bugreport-<stamp>`,
  so collecting a report neither spends nor is refused by the caller's budget. `gates.md` §
  "Taking a new session id" names this as the second permitted case, for that script alone.
 - **`instructions/bug-report.md`** (`manual: true`) is the agent's side: what to tell the user,
  a fact-only chronology template, and the rule to stop before sending anything. `setup-instance.md`
  and `upgrade-instance.md` offer it at a failure that has no obvious cause.
 - `reports/CONTRACT.md` names the bundle as a third kind of output, `tools/README.md` explains why the
  script sits beside the package, `INSTALL.md` has a troubleshooting entry, and CI runs the script
  from the exported distribution.
 ### Demo corpus follows the decided project pages: three states, seed with their items (Gitea #156)
 The first build of the live tracker suite shipped two demo project pages whose names came from a
@@ -473,6 +473,13 @@ tools/wikitool instructions verify
  als beim Mass-Update-Gate gibt es hier **keinen Token und keine Flagge** - stimmt das Ziel
  wirklich, trägt der Mensch dessen URL selbst in die Datei ein. Ein Agent, der die Datei
  anfasst, um an der Verweigerung vorbeizukommen, öffnet ein Gate aus eigenem Antrieb.
 - **Setup oder Update schlägt fehl und ich will es melden** - den Agenten
  [instructions/bug-report.md](instructions/bug-report.md) ausführen lassen, oder direkt
  `python3 tools/bugreport.py`. Das Skript braucht nur Python 3.8 oder neuer, läuft auch ohne
  funktionierendes `wikitool` und schreibt ein Bündel nach `reports/bugreport-<Zeitstempel>/`
  samt Zip. Geheimnisse werden entfernt, Seitentitel bleiben draußen (`--titles` nimmt sie mit);
  das Bündel enthält trotzdem Maschinen-, Benutzer- und Pfadnamen. Es wird nirgends hochgeladen -
  den Kanal wählst du selbst.
 - **Ich will am Tool-Stack selbst weiterarbeiten (nicht nur Wiki-Inhalt betreiben)** - eine neue
  Instanz hat dafür keinen Weg: `dist export` lässt `instructions/dev/` (Stack-Entwicklung,
  inkl. der vendorten `commonplace/`-Wissensbasis) bewusst und dauerhaft weg, ohne
@@ -1 +1 @@
-8.0.0-beta.4
+8.0.0-beta.5
@@ -0,0 +1,135 @@
 ---
 type: types/instruction.md
 name: bug-report
 description: How to collect a bug-report bundle when the stack misbehaves on this machine - run tools/bugreport.py, write a fact-only chronology, tell the user what the bundle contains, and stop short of sending it anywhere.
 manual: true
 ---
 # Collect a bug report
 When setup, an upgrade or a command fails on one machine and works on another, the person who has to
 fix it sees nothing of what happened here. This procedure produces one bundle that answers the first
 round of their questions - which machine, which Python, which shell, which harness, what the stack
 looked like, what `wikitool` printed - so that the report is not a guessing game.
 **Run this only when asked, by name, or when the user agrees to it after a failure.** It is
 `manual: true` on purpose: the bundle contains private data, and whether to produce one is the
 user's decision, not the agent's. Nothing links to this file from `AGENTS.md` or a skill, apart from
 the pointers at the failure decision points of [setup-instance.md](setup-instance.md) and
 [upgrade-instance.md](upgrade-instance.md), which only offer it.
 <!-- wikitool:toc -->
 ## Contents
 - [What the bundle holds](#what-the-bundle-holds)
 - [When to run](#when-to-run)
 - [Steps](#steps)
 - [Chronology template](#chronology-template)
 - [Decision points](#decision-points)
 - [Scope](#scope)
 <!-- /wikitool:toc -->
 ## What the bundle holds
 `tools/bugreport.py` is a standalone script. It uses the standard library only and imports nothing
 from the stack, so it still runs when `wikitool` does not start - no venv, a broken package, a Python
 that is too old for the stack. It writes `reports/bugreport-<UTC stamp>/` and a zip beside it, in
 four layers:
 | Layer | Files | Holds |
 |-------|-------|-------|
 | 1 Environment | `environment.json` | OS, every Python and shell found, harness, `PATH`, environment variable names (values only for a fixed list), git configuration, venv, line endings, on Windows also long paths, execution policy and mark-of-the-web |
 | 2 Stack | `stack.json`, `tree-structure.json` | `VERSION`, the `.wikitool-*.json` files with secrets removed, git status and the last commits, and the shape of `kb/` and `raw/` (counts, depths, path lengths, names that break on Windows) |
 | 3 wikitool | `wikitool/*.txt`, `trace.jsonl` | Verbatim output of `version show`, `doctor`, `budget status`, `instructions verify` and `docs verify`, and the caller's session trace. If `version show` fails, `wikitool` counts as not started and nothing else runs |
 | 4 Chronology | `CHRONOLOGY.md`, `transcripts/` | What the agent did and saw, and harness transcripts if the user asked for them |
 `MANIFEST.md` lists every file and marks the ones that may contain page content and titles: the trace,
 the chronology and the transcripts.
 Two rules hold for everything the script generates itself. **Secrets are always removed**: values of
 keys that look like a token, password, secret, key or auth entry, credentials in URLs, and every value
 of that kind found while collecting is also replaced wherever else it turns up. **Page titles are kept
 out** unless `--titles` is given: paths under `kb/` and `raw/` are replaced by their shape
 (depth, length, whether they hold a space or a non-ASCII character) and `[[wikilinks]]` by the same
 flags. A title that stands as bare prose is not found - which is why the trace, the chronology and the
 transcripts are marked instead.
 ## When to run
 - Setup or an upgrade failed and the user wants to report it.
 - A command fails in a way that looks tied to the machine, and the user asks for a report.
 - The user asks for a bug report by name.
 ## Steps
 1. **Tell the user what is about to happen**, in the instance's KB language: a bundle will be written
   under `reports/`, it is not pseudonymised, it holds machine, user and path names and the git
   remotes, and it is not sent anywhere. Ask whether the session trace, page titles and transcripts
   may go in. The defaults are: trace in, titles out, no transcripts.
 2. **Write the chronology** to a file under `reports/` (which is gitignored) from the
   [template](#chronology-template) below. Facts only - no diagnosis. If the session's own history is
   too long to reconstruct, say what is missing instead of filling the gap.
 3. **Run the collector**, with whichever Python 3.8 or later the machine has:
   ```bash
   python3 tools/bugreport.py --chronology reports/chronology.md
   ```
   Add `--no-trace` if the user declined the trace, `--titles` if titles may stay,
   `--transcript <file>` (repeatable) for a transcript the user pointed at, `--session <id>` to take a
   trace other than the caller's. `python3` may be `python` or `py -3` on Windows. If no Python starts
   at all, that is the report: give the user the exact error text, verbatim.
 4. **Do not imitate the collector's session id.** It runs its counting `wikitool` calls under
   `WIKITOOL_SESSION_ID=bugreport-<stamp>` itself; do not export that variable, or any `bugreport-*`
   one, in the session. The exception it gets in [gates.md](gates.md) § "Taking a new session id"
   belongs to the script alone.
 5. **Report the result.** Quote the bundle path, the archive path and the privacy notice the script
   prints, name the gaps in `MANIFEST.md`, and tell the user to read the bundle before sharing it.
   Then stop: the channel - a tracker issue, an email, a chat - is the user's choice, and the agent
   never uploads the bundle.
 ## Chronology template
 ```markdown
 # Chronology
 ## Goal
 One sentence: what the user asked for, e.g. "Set up a new instance from a fresh clone."
 ## Environment as the agent saw it
 Harness, shell, OS, anything the user said about the machine that the collector cannot know.
 ## Steps
 1. `<exact command>` - exit code, and the first line of the error or output that mattered, verbatim.
 2. ...
 ## Expected and observed
 - Expected: what the instruction said would happen.
 - Observed: what happened instead, verbatim where it is short.
 ## Changes made by hand
 Every file edited or created outside a `wikitool` command, and every setting changed, in order.
 ## Not known
 What the agent could not find out, or did not check.
 ```
 Facts only: no page content, no guessed cause, no advice.
 ## Decision points
 - **The user declines the trace?** Run with `--no-trace`. The manifest records the exclusion.
 - **The user wants the report to name pages?** Run with `--titles`; the bundle then also carries
  `tree-paths.txt`, which lists every path under `kb/` and `raw/`.
 - **The collector exits 1?** It could not write the bundle (a missing input file, a full disk). Read
  the message, fix the cause, retry once, then report the exact error.
 - **A `wikitool` output in the bundle looks wrong or refuses to run?** Do not re-run it to see more.
  The bundle records what happened; that is the report.
 ## Scope
 Covers producing the bundle. It does not cover reading a bundle someone else sent, triaging the
 report, or filing it - all of that is the maintainer's side and the user's choice of channel.
@@ -215,6 +215,14 @@ never in response to a gate refusal.** The plan is the human approval the gate w
 have to ask for; a refusal means that approval has not been given yet. If you are tempted to
 re-export the variable after an `ERROR` line, that is the gate working.
 **One tool takes a new id itself, for a fixed set of read commands.** `tools/bugreport.py`
 ([bug-report.md](bug-report.md)) runs `instructions verify` and `docs verify` under
 `WIKITOOL_SESSION_ID=bugreport-<stamp>`, so that collecting a report neither spends the
 caller's budget nor is refused by it at exactly the moment something has gone wrong. That is
 permitted because the script sets the id for those two commands only, never in the caller's
 shell, and the set is a constant in the script. It is not a precedent for an agent: an agent that
 exports a `bugreport-*` id, or any other, outside the two cases above is opening the gate.
 Background: the [[Iteration and Cost Limits]] concept page in `kb/`.
 ## Scope
@@ -305,6 +305,10 @@ and ready for its first ingest.
    afterwards are `wiki-ingest`, `wiki-query`, `wiki-manage`, `wiki-lint`, `wiki-status` and
    `gtd-weekly-review` available.
 **A step fails and the cause is not obvious?** Do not improvise around it (invariant 7). Offer the
 user a bug report - [bug-report.md](bug-report.md) - and run it only if they agree; the collector
 works even when `wikitool` does not start.
 ## Scope
 Applies only to an empty distribution produced by `dist export`. For an existing clone of this
@@ -247,6 +247,9 @@ fresh clone ([bootstrap.md](bootstrap.md)), and not for the clone-with-upstream
  intermediate state is legitimate and `migrate status` resumes it; what is not legitimate is
  recording a migration with `migrate done` that was not carried out - the version then describes
  a shape the corpus is not in.
 - **A step fails and the cause is not obvious?** Stop rather than improvise, and offer the user a
  bug report - [bug-report.md](bug-report.md). It runs even when `wikitool` does not start, and
  only when the user agrees.
 - **`doctor` reports `kb-version` behind `VERSION` after everything is done?** Correct when the
  release's chain was empty or carried only `offered` entries: an offer changes a file the
  instance owns, not the shape of its content, so the content version stays where it was.
@@ -1,12 +1,14 @@
 # reports/ - Generated Output
-Derived output that must stay out of git. Two kinds live here:
+Derived output that must stay out of git. Three kinds live here:
 - **Lint reports**, written by `tools/wikitool lint --markdown "reports/Lint Report
  <YYYY-MM-DD>.md"`.
 - **Traces**, under `reports/telemetry/<session>/trace.jsonl` - the append-only record of
  what a session did, written by `wikitool` itself and by the harness hooks. See
  [../EVALS.md](../EVALS.md) for the event contract and what is redacted.
 - **Bug-report bundles**, under `reports/bugreport-<UTC stamp>/` and a zip beside each, written by
  `tools/bugreport.py` on request - see [../instructions/bug-report.md](../instructions/bug-report.md).
 **Everything in this directory except this file is gitignored.** A lint report is a derived
 copy of recomputable truth: its structural sections can be regenerated from the tree at any
@@ -39,12 +41,27 @@ installation-form default (on for a dev checkout, off for a distributed instance
 keeps lengths and digests instead of text. See [../EVALS.md](../EVALS.md) § "Whether it runs at
 all" for the full precedence and both quantity caps below.
 ## Bug-report bundles
 A bundle is a snapshot of one machine at one moment, made so that someone else can read what
 happened here. It is not recomputable and not durable, and it **contains private data**: machine,
 user and path names, `PATH` entries, git remotes and commit subjects, and - when included - the
 session trace, the chronology and transcripts, which may hold page content and titles. Secrets are
 removed by the collector; the rest is the reader's to check before a bundle leaves the machine.
 Nothing uploads it: the channel is the user's choice.
 The collector's own `wikitool` calls run under the session id `bugreport-<stamp>`, so a
 `reports/telemetry/bugreport-*` directory is that run's trace and belongs to no session of yours.
 ## Retention
 **Lint reports: none.** Old ones are local scratch; delete them freely. There is nothing to
 retire with `wikitool rm`, because no report is ever a wiki page - `lint-report` is a
 contract-only type-spec with no `base_dir:` and cannot be instantiated under `kb/`.
 **Bug-report bundles: none.** Delete them freely once they have been read or sent; nothing refers
 to one afterwards.
 **Traces: two enforced caps, applied by the writer itself, never by a separate cleanup pass.**
 A byte cap per session trace (default 5 MiB, `WIKI_TRACE_MAX_SESSION_BYTES`) and a retention
 limit on the number of session directories under `reports/telemetry/` (default 250,
@@ -60,6 +60,13 @@ tools/
    tests/                 pytest suite
 ```
 **A script beside the package.** `bugreport.py` is not part of `chemenu` and imports nothing
 from it: it is the bug-report collector ([../instructions/bug-report.md](../instructions/bug-report.md)),
 and the case it exists for is a checkout where `wikitool` does not start. It therefore uses the
 standard library only, keeps to Python 3.8 syntax so that an old interpreter can still run it, and
 exits 0 or 1 - never 42, since it opens no gate. `dist export` ships it with the rest of `tools/`;
 its tests (`tests/test_bugreport.py`) run it on a bare interpreter (`-I -S`) to keep that promise.
 **Two consumers, one core.** The CLI is not the only caller any more. The cores
 (`search/service.py`, `lint_core.py`, `types_core.py`, `catalog.py`) hold what
 decides an answer and import no `typer` and no `rich`; the modules under `commands/` turn
@@ -0,0 +1,942 @@
 #!/usr/bin/env python3
 """Collect a bug-report bundle from this checkout.
 Runs on the base Python with the standard library only, and imports nothing from
 `chemenu`: the case it exists for is a checkout where `wikitool` does not start
 - no venv, a broken package, a Python that is too old. Syntax stays at Python
 3.8 so that even an old interpreter can still produce a report.
    python tools/bugreport.py [--chronology FILE] [--transcript FILE]...
                              [--session ID | --no-trace] [--titles]
                              [--root DIR] [--out DIR]
 The bundle is `<out>/bugreport-<UTC stamp>/` plus a zip beside it, in four
 layers: the environment, the stack, what `wikitool` prints (if it starts), and
 the agent's chronology. Secrets are always removed. Page titles are kept out of
 everything this script generates unless `--titles` is given. Nothing is uploaded.
 Exit 0 = bundle written, 1 = it could not be written. No exit 42: this script
 opens no gate. See instructions/bug-report.md.
 """
 from __future__ import annotations
 import argparse
 import datetime
 import json
 import os
 import platform
 import re
 import shutil
 import subprocess
 import sys
 import unicodedata
 import zipfile
 from pathlib import Path
 TIMEOUT_SECONDS = 300
 PROBE_TIMEOUT_SECONDS = 20
 REMOVED = "<removed>"
 NOT_COLLECTED = "<not collected>"
 SESSION_ENV = "WIKITOOL_SESSION_ID"
 HARNESS_SESSION_ENV = "CLAUDE_CODE_SESSION_ID"
 SECRET_NAME = re.compile(r"token|password|secret|key|auth", re.IGNORECASE)
 HARNESS_MARKERS = (
    (HARNESS_SESSION_ENV, "claude-code"),
 )
 ENV_ALLOW_EXACT = frozenset(
    {
        "PATH", "PATHEXT", "SHELL", "TERM", "TERM_PROGRAM", "TERM_PROGRAM_VERSION",
        "COLORTERM", "LANG", "LANGUAGE", "VIRTUAL_ENV", "MSYSTEM", "COMSPEC",
        "PSMODULEPATH",
    }
    | {name for name, _ in HARNESS_MARKERS}
 )
 ENV_ALLOW_PREFIX = ("LC_", "PYTHON", "WIKI_", "WIKITOOL_", "CHEMENU_")
 # Names under kb/ that the stack generates or owns: not titles, and unreadable
 # in a bundle if they are masked.
 STACK_NAMES = frozenset(
    {
        "CONTRACT.md", "CONVENTIONS.md", "CONVENTIONS.md.template", "COLLECTION.md",
        "COLLECTION.md.template", "INDEX.md", "index.md", "log.md", "provenance.md",
        ".gitkeep",
    }
 )
 CONTENT_DIRS = ("kb", "raw")
 CONTENT_COMMIT_DIRS = ("kb", "raw", "work")
 MAY_CONTAIN_CONTENT = "may contain page content and titles"
 PRIVACY_NOTICE = (
    "This bundle is not pseudonymised. It contains private data: machine, user and path "
    "names, PATH entries, git remotes and commit subjects and - if included - the session "
    "trace, the chronology and transcripts, which may contain page content and titles. "
    "Secrets are removed, but read the bundle before you share it. Choose the channel "
    "yourself; neither this tool nor the agent uploads anything."
 )
 WINDOWS_RESERVED = re.compile(r"^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(\..*)?$", re.IGNORECASE)
 WINDOWS_FORBIDDEN = re.compile(r'[<>:"|?*\x00-\x1f]')
 WIKILINK = re.compile(r"\[\[([^\]\n]+)\]\]")
 URL_USERINFO = re.compile(r"(?P<scheme>[A-Za-z][A-Za-z0-9+.\-]*)://(?P<userinfo>[^/\s@\"'<>]+)@")
 AUTH_HEADER = re.compile(r"(?im)(authorization\s*[:=]\s*)(?:(?:bearer|basic|token)\s+)?\S+")
 BEARER = re.compile(r"(?i)\b(bearer)\s+[A-Za-z0-9._~+/=\-]{8,}")
 # --------------------------------------------------------------------------- secrets
 class Redactor:
    """Every secret value found while collecting, replaced again in a final pass."""
    def __init__(self) -> None:
        self.values = set()
    def add(self, value) -> None:
        if isinstance(value, str) and len(value) >= 4:
            self.values.add(value)
    def add_any(self, node) -> None:
        if isinstance(node, dict):
            for item in node.values():
                self.add_any(item)
        elif isinstance(node, (list, tuple)):
            for item in node:
                self.add_any(item)
        else:
            self.add(node)
    def scrub_urls(self, text: str) -> str:
        def repl(match):
            scheme, userinfo = match.group("scheme"), match.group("userinfo")
            if scheme.lower() in ("http", "https"):
                self.add(userinfo)
                for part in userinfo.split(":"):
                    self.add(part)
                return scheme + "://"
            if ":" in userinfo:
                user, password = userinfo.split(":", 1)
                self.add(password)
                return scheme + "://" + user + "@"
            return match.group(0)
        return URL_USERINFO.sub(repl, text)
    def scrub(self, text: str) -> str:
        text = self.scrub_urls(text)
        text = AUTH_HEADER.sub(lambda m: m.group(1) + REMOVED, text)
        text = BEARER.sub(lambda m: m.group(1) + " " + REMOVED, text)
        forms = set()
        for value in self.values:
            forms.add(value)
            escaped = json.dumps(value, ensure_ascii=False)[1:-1]
            if escaped != value:
                forms.add(escaped)
        for form in sorted(forms, key=len, reverse=True):
            text = text.replace(form, REMOVED)
        return text
    def clean_json(self, node):
        if isinstance(node, dict):
            out = {}
            for key, value in node.items():
                if SECRET_NAME.search(str(key)):
                    self.add_any(value)
                    out[key] = REMOVED
                else:
                    out[key] = self.clean_json(value)
            return out
        if isinstance(node, list):
            return [self.clean_json(item) for item in node]
        if isinstance(node, str):
            return self.scrub_urls(node)
        return node
 # ------------------------------------------------------------------------ title shield
 def _shape(rel: str) -> str:
    parts = rel.split("/")
    name = parts[-1]
    stem, dot, suffix = name.rpartition(".")
    flags = ["depth=%d" % len(parts), "len=%d" % len(rel)]
    if " " in rel:
        flags.append("space")
    if any(ord(ch) > 127 for ch in rel):
        flags.append("non-ascii")
    return "%s/<%s>%s" % (parts[0], " ".join(flags), (dot + suffix) if dot and stem else "")
 class TitleShield:
    """Replaces the relative path of a file under kb/ or raw/ with its shape.
    A title that stands as bare prose is not found, on purpose: replacing titles
    word by word would hit a title like "Git" everywhere in the bundle.
    """
    def __init__(self, rel_paths) -> None:
        self.forms = {}
        for rel in rel_paths:
            shape = _shape(rel)
            for form in (rel, rel.replace("/", "\\"), rel.replace("/", "\\\\")):
                self.forms[form] = shape
        self.lengths = sorted({len(form) for form in self.forms}, reverse=True)
        self.start = re.compile(r"(?:%s)(?:/|\\)" % "|".join(CONTENT_DIRS))
    def apply(self, text: str) -> str:
        if self.forms:
            out, last = [], 0
            for match in self.start.finditer(text):
                i = match.start()
                if i < last:
                    continue
                for length in self.lengths:
                    shape = self.forms.get(text[i:i + length])
                    if shape:
                        out.append(text[last:i])
                        out.append(shape)
                        last = i + length
                        break
            out.append(text[last:])
            text = "".join(out)
        return WIKILINK.sub(lambda m: "[[<title %s>]]" % _title_flags(m.group(1)), text)
 def _title_flags(title: str) -> str:
    flags = ["len=%d" % len(title)]
    if " " in title:
        flags.append("space")
    if any(ord(ch) > 127 for ch in title):
        flags.append("non-ascii")
    return " ".join(flags)
 # ---------------------------------------------------------------------------- helpers
 class Result:
    def __init__(self, returncode, stdout, stderr, note=None) -> None:
        self.returncode = returncode
        self.stdout = stdout
        self.stderr = stderr
        self.note = note
    @property
    def ok(self) -> bool:
        return self.returncode == 0
    def first_line(self) -> str:
        for text in (self.stderr, self.stdout, self.note or ""):
            for line in text.splitlines():
                if line.strip():
                    return line.strip()
        return ""
 def _decode(data) -> str:
    if data is None:
        return ""
    if isinstance(data, str):
        return data
    return data.decode("utf-8", errors="replace")
 def run(cmd, cwd, env=None, timeout=PROBE_TIMEOUT_SECONDS) -> Result:
    try:
        proc = subprocess.run(
            cmd, cwd=str(cwd), env=env, stdin=subprocess.DEVNULL,
            stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout,
        )
    except subprocess.TimeoutExpired as exc:
        return Result(None, _decode(exc.stdout), _decode(exc.stderr),
                      "timed out after %d s" % timeout)
    except OSError as exc:
        return Result(None, "", "", "could not start: %s" % exc)
    return Result(proc.returncode, _decode(proc.stdout), _decode(proc.stderr))
 def read_text(path: Path) -> str:
    with open(str(path), "r", encoding="utf-8", errors="replace", newline="") as handle:
        return handle.read()
 def write_text(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with open(str(path), "w", encoding="utf-8", newline="\n") as handle:
        handle.write(text)
 def dump_json(data) -> str:
    return json.dumps(data, indent=2, ensure_ascii=False) + "\n"
 def guarded(fn, *args):
    """A broken layer must not take the whole report down."""
    try:
        return fn(*args)
    except Exception as exc:  # noqa: BLE001 - the report is the point
        return {"collector_error": "%s: %s" % (type(exc).__name__, exc)}
 def session_slug(value: str) -> str:
    return re.sub(r"[^A-Za-z0-9._-]+", "__", value).strip("_") or "unknown"
 # ------------------------------------------------------------------- layer 1: environment
 def _version_line(cmd, cwd) -> str:
    result = run(cmd, cwd)
    return result.first_line() if result.ok or result.first_line() else (result.note or "")
 def collect_environment(root: Path, red: Redactor) -> dict:
    data = {
        "os": {
            "platform": platform.platform(),
            "system": platform.system(),
            "release": platform.release(),
            "version": platform.version(),
            "machine": platform.machine(),
        },
        "python": _collect_python(root),
        "shells": _collect_shells(root),
        "harness": _collect_harness(),
        "path_entries": [
            {"entry": entry, "exists": os.path.isdir(entry)}
            for entry in os.environ.get("PATH", "").split(os.pathsep) if entry
        ],
        "env": _collect_env(red),
        "tools_config": _collect_tools_config(root, red),
        "git": _collect_git_config(root, red),
        "venv": _collect_venv(root, red),
        "line_endings": _collect_line_endings(root),
        "gitattributes": _read_optional(root / ".gitattributes", red),
        "windows": _collect_windows(root),
    }
    return data
 def _collect_python(root: Path) -> dict:
    running = {
        "executable": sys.executable,
        "version": sys.version.replace("\n", " "),
        "version_info": list(sys.version_info[:3]),
    }
    candidates = []
    for name, extra in (("python3", []), ("python", []), ("py", ["-3"])):
        path = shutil.which(name)
        entry = {"name": " ".join([name] + extra), "path": path}
        if path:
            entry["version"] = _version_line([path] + extra + ["--version"], root)
            entry["windows_store_alias"] = "windowsapps" in path.lower()
        candidates.append(entry)
    return {"running": running, "candidates": candidates}
 def _collect_shells(root: Path) -> dict:
    shells = {"SHELL": os.environ.get("SHELL"), "COMSPEC": os.environ.get("COMSPEC")}
    for name in ("bash", "zsh", "fish"):
        path = shutil.which(name)
        shells[name] = {"path": path,
                        "version": _version_line([path, "--version"], root) if path else None}
    for name in ("pwsh", "powershell"):
        path = shutil.which(name)
        shells[name] = {
            "path": path,
            "version": _version_line(
                [path, "-NoProfile", "-Command", "$PSVersionTable.PSVersion.ToString()"], root
            ) if path else None,
        }
    return shells
 def _collect_harness() -> dict:
    detected = [
        {"marker": name, "harness": harness}
        for name, harness in HARNESS_MARKERS if os.environ.get(name)
    ]
    if os.environ.get("TERM_PROGRAM") == "vscode":
        detected.append({"marker": "TERM_PROGRAM=vscode", "harness": "vscode terminal"})
    return {"detected": detected or "unrecognised"}
 def _env_allowed(name: str) -> bool:
    upper = name.upper()
    return upper in ENV_ALLOW_EXACT or upper.startswith(ENV_ALLOW_PREFIX)
 def _collect_env(red: Redactor) -> dict:
    env = {}
    for name in sorted(os.environ):
        value = os.environ[name]
        if SECRET_NAME.search(name):
            red.add(value)
            env[name] = REMOVED if _env_allowed(name) else NOT_COLLECTED
        elif _env_allowed(name):
            env[name] = value
        else:
            env[name] = NOT_COLLECTED
    return env
 def _collect_tools_config(root: Path, red: Redactor) -> dict:
    path = root / ".wikitool-tools.json"
    if not path.is_file():
        return {"status": "absent"}
    try:
        parsed = json.loads(read_text(path))
    except ValueError as exc:
        return {"status": "unparseable", "error": str(exc)}
    checks = []
    def walk(node, where):
        if isinstance(node, dict):
            for key, value in node.items():
                walk(value, where + [str(key)])
        elif isinstance(node, list):
            for index, value in enumerate(node):
                walk(value, where + [str(index)])
        elif isinstance(node, str) and os.path.isabs(node):
            checks.append({"at": ".".join(where), "path": node, "exists": os.path.exists(node)})
    walk(parsed, [])
    return {"status": "present", "content": red.clean_json(parsed), "path_checks": checks}
 def _collect_git_config(root: Path, red: Redactor) -> dict:
    version = run(["git", "--version"], root)
    if not version.ok:
        return {"available": False, "error": version.note or version.first_line()}
    config = {}
    for key in ("core.autocrlf", "core.longpaths", "core.filemode"):
        result = run(["git", "config", "--get", key], root)
        config[key] = result.stdout.strip() if result.ok else "(unset)"
    name = run(["git", "config", "--get", "user.name"], root)
    config["user.name is set"] = bool(name.ok and name.stdout.strip())
    remotes = run(["git", "remote", "-v"], root)
    return {
        "available": True,
        "version": version.stdout.strip(),
        "config": config,
        "remotes": red.scrub_urls(remotes.stdout).splitlines() if remotes.ok
        else ["(not a repository, or git failed: %s)" % remotes.first_line()],
    }
 def _collect_venv(root: Path, red: Redactor) -> dict:
    venv = root / "tools" / ".venv"
    if not venv.is_dir():
        return {"present": False}
    layout = ("bin" if (venv / "bin").is_dir() else None) or (
        "Scripts" if (venv / "Scripts").is_dir() else None)
    cfg = venv / "pyvenv.cfg"
    return {
        "present": True,
        "layout": layout or "unknown",
        "python_found": any(
            (venv / sub / exe).exists()
            for sub, exe in (("bin", "python"), ("Scripts", "python.exe"))
        ),
        "pyvenv.cfg": read_text(cfg) if cfg.is_file() else None,
    }
 def _collect_line_endings(root: Path) -> dict:
    tools = root / "tools"
    files = [tools / "wikitool"] + (sorted(tools.glob("*.ps1")) if tools.is_dir() else [])
    result = {}
    for path in files:
        if not path.is_file():
            continue
        raw = path.read_bytes()
        crlf = raw.count(b"\r\n")
        result[path.name] = {"crlf": crlf, "lf_only": raw.count(b"\n") - crlf}
    return result
 def _read_optional(path: Path, red: Redactor):
    return red.scrub_urls(read_text(path)) if path.is_file() else None
 def _collect_windows(root: Path) -> dict:
    if sys.platform != "win32":
        return {"applicable": False}
    data = {"applicable": True}
    try:
        import winreg  # noqa: PLC0415 - only exists on Windows
        with winreg.OpenKey(
            winreg.HKEY_LOCAL_MACHINE, r"SYSTEM\CurrentControlSet\Control\FileSystem"
        ) as key:
            data["LongPathsEnabled"] = winreg.QueryValueEx(key, "LongPathsEnabled")[0]
    except Exception as exc:  # noqa: BLE001
        data["LongPathsEnabled"] = "unreadable: %s" % exc
    pwsh = shutil.which("pwsh")
    if pwsh:
        policy = run([pwsh, "-NoProfile", "-Command", "Get-ExecutionPolicy -List | Out-String"], root)
        data["execution_policy"] = policy.stdout.strip() or policy.first_line()
    else:
        data["execution_policy"] = "pwsh not found"
    data["powershell_5_present"] = bool(shutil.which("powershell"))
    marked = []
    tools = root / "tools"
    if tools.is_dir():
        for path in sorted(p for p in tools.iterdir() if p.is_file()):
            try:
                with open(str(path) + ":Zone.Identifier", "r", encoding="utf-8",
                          errors="replace") as stream:
                    marked.append({"file": path.name, "zone": stream.read().strip()})
            except OSError:
                pass
    data["mark_of_the_web"] = marked
    return data
 # ------------------------------------------------------------------------ layer 2: stack
 def scan_tree(root: Path, name: str):
    """Structure of kb/ or raw/, and the relative paths it holds. No content."""
    top = root / name
    if not top.is_dir():
        return {"present": False}, []
    files, dirs = [], 0
    depth_histogram, longest_rel, longest_abs = {}, 0, 0
    counts = {
        "abs_path_over_240": 0, "abs_path_over_260": 0, "names_with_space": 0,
        "names_non_ascii": 0, "names_windows_forbidden_chars": 0,
        "names_trailing_dot_or_space": 0, "names_reserved_device": 0,
        "names_not_nfc": 0, "case_collisions": 0,
    }
    def judge(entry: str) -> None:
        if " " in entry:
            counts["names_with_space"] += 1
        if any(ord(ch) > 127 for ch in entry):
            counts["names_non_ascii"] += 1
        if WINDOWS_FORBIDDEN.search(entry):
            counts["names_windows_forbidden_chars"] += 1
        if entry.endswith((" ", ".")):
            counts["names_trailing_dot_or_space"] += 1
        if WINDOWS_RESERVED.match(entry):
            counts["names_reserved_device"] += 1
        if unicodedata.normalize("NFC", entry) != entry:
            counts["names_not_nfc"] += 1
    for current, subdirs, names in os.walk(str(top)):
        entries = sorted(subdirs) + sorted(names)
        seen = {}
        for entry in entries:
            seen.setdefault(entry.casefold(), []).append(entry)
            judge(entry)
        counts["case_collisions"] += sum(1 for group in seen.values() if len(group) > 1)
        dirs += len(subdirs)
        for entry in names:
            absolute = os.path.join(current, entry)
            rel = os.path.relpath(absolute, str(root)).replace(os.sep, "/")
            files.append(rel)
            depth = rel.count("/") + 1
            depth_histogram[str(depth)] = depth_histogram.get(str(depth), 0) + 1
            longest_rel = max(longest_rel, len(rel))
            longest_abs = max(longest_abs, len(os.path.abspath(absolute)))
            if len(os.path.abspath(absolute)) > 240:
                counts["abs_path_over_240"] += 1
            if len(os.path.abspath(absolute)) > 260:
                counts["abs_path_over_260"] += 1
    summary = {
        "present": True, "files": len(files), "directories": dirs,
        "depth_histogram": depth_histogram, "longest_relative_path": longest_rel,
        "longest_absolute_path": longest_abs,
    }
    summary.update(counts)
    return summary, files
 def collect_stack(root: Path, red: Redactor) -> dict:
    version = root / "VERSION"
    configs = {}
    for path in sorted(root.glob(".wikitool-*.json")):
        if path.name == ".wikitool-tools.json":
            continue
        try:
            configs[path.name] = red.clean_json(json.loads(read_text(path)))
        except ValueError:
            configs[path.name] = {"unparseable": True, "bytes": path.stat().st_size}
    return {
        "VERSION": read_text(version).strip() if version.is_file() else "absent",
        "config_files": configs,
        "git": _collect_git_state(root),
    }
 def _collect_git_state(root: Path) -> dict:
    quiet = ["git", "-c", "core.quotepath=off"]
    inside = run(quiet + ["rev-parse", "--is-inside-work-tree"], root)
    if not inside.ok:
        return {"repository": False, "error": inside.first_line()}
    branch = run(quiet + ["rev-parse", "--abbrev-ref", "HEAD"], root)
    status = run(quiet + ["status", "--porcelain", "-z"], root)
    log = run(quiet + ["log", "-n", "20", "--name-only", "--format=%x1e%h%x1f%s"], root)
    return {
        "repository": True,
        "branch": branch.stdout.strip() if branch.ok else branch.first_line(),
        "status": _parse_status(status.stdout) if status.ok else [status.first_line()],
        "log": _parse_log(log.stdout) if log.ok else [log.first_line()],
    }
 def _parse_status(raw: str) -> list:
    entries = raw.split("\0")
    lines, i = [], 0
    while i < len(entries):
        entry = entries[i]
        i += 1
        if len(entry) < 4:
            continue
        code, path = entry[:2], entry[3:]
        if code[0] in "RC" and i < len(entries):
            lines.append("%s %s (from %s)" % (code, path, entries[i]))
            i += 1
        else:
            lines.append("%s %s" % (code, path))
    return lines
 def _parse_log(raw: str) -> list:
    lines = []
    for block in raw.split("\x1e"):
        if not block.strip():
            continue
        head, _, rest = block.partition("\n")
        sha, _, subject = head.partition("\x1f")
        files = [line for line in rest.splitlines() if line.strip()]
        if files and all(f.startswith(tuple(d + "/" for d in CONTENT_COMMIT_DIRS)) for f in files):
            parts = []
            for directory in CONTENT_COMMIT_DIRS:
                count = sum(1 for f in files if f.startswith(directory + "/"))
                if count:
                    parts.append("%s %d" % (directory, count))
            subject = "<content commit: %s files>" % ", ".join(parts)
        lines.append("%s %s" % (sha.strip(), subject.strip()))
    return lines
 # ---------------------------------------------------------------------- layer 3: wikitool
 def launcher_command(root: Path):
    tools = root / "tools"
    if sys.platform == "win32":
        ps1, sh = tools / "wikitool.ps1", tools / "wikitool"
        pwsh = shutil.which("pwsh")
        if ps1.is_file() and pwsh:
            return [pwsh, "-NoProfile", "-File", str(ps1)]
        bash = shutil.which("bash")
        if sh.is_file() and bash:
            return [bash, str(sh)]
        return None
    launcher = tools / "wikitool"
    return [str(launcher)] if launcher.is_file() else None
 def resolve_trace_session(explicit):
    if explicit:
        return explicit, "--session"
    for name in (SESSION_ENV, HARNESS_SESSION_ENV):
        if os.environ.get(name):
            return os.environ[name], name
    return None, None
 def trace_root(root: Path) -> Path:
    override = os.environ.get("WIKI_TRACE_DIR")
    return Path(override) if override else root / "reports" / "telemetry"
 def collect_trace(root: Path, explicit):
    """The caller's session trace, or a listing of what exists instead."""
    session, source = resolve_trace_session(explicit)
    base = trace_root(root)
    if session:
        path = base / session_slug(session) / "trace.jsonl"
        if path.is_file():
            return {"found": True, "session": session, "source": source,
                    "text": read_text(path)}
    listing = []
    if base.is_dir():
        for child in sorted(base.iterdir()):
            if child.is_dir() and not child.name.startswith("bugreport-"):
                trace = child / "trace.jsonl"
                listing.append({"session_directory": child.name,
                                "bytes": trace.stat().st_size if trace.is_file() else 0})
    return {"found": False, "session": session, "source": source, "listing": listing}
 WIKITOOL_CALLS = (
    ("version show", ("version", "show"), "inherited"),
    ("doctor", ("doctor",), "inherited"),
    ("budget status", ("budget", "status"), "inherited"),
    ("instructions verify", ("instructions", "verify"), "own"),
    ("docs verify", ("docs", "verify"), "own"),
 )
 def run_wikitool(root: Path, launcher, own_session: str) -> dict:
    """Five fixed read commands. `version show` is the startup probe: if it fails,
    `wikitool` counts as not started and nothing else is run."""
    outputs = []
    started = False
    reason = None
    for label, args, session in WIKITOOL_CALLS:
        env = dict(os.environ)
        if session == "own":
            env[SESSION_ENV] = own_session
        result = run(launcher + list(args), root, env=env, timeout=TIMEOUT_SECONDS)
        prefix = "%s=%s " % (SESSION_ENV, own_session) if session == "own" else ""
        outputs.append({
            "label": label,
            "file": "wikitool/%s.txt" % label.replace(" ", "-"),
            "session": ("%s (set by the collector)" % own_session) if session == "own"
            else "inherited from the caller",
            "text": (
                "command: %s%s\nsession: %s\nexit: %s\n%s\n--- stdout ---\n%s\n--- stderr ---\n%s\n"
                % (prefix, " ".join(launcher + list(args)),
                   "own" if session == "own" else "inherited",
                   result.returncode if result.returncode is not None else result.note,
                   ("note: " + result.note) if result.note and result.returncode is not None
                   else "", result.stdout, result.stderr)
            ),
        })
        if label == "version show":
            if not result.ok:
                reason = "version show %s: %s" % (
                    ("exit %s" % result.returncode) if result.returncode is not None
                    else result.note, result.first_line())
                break
            started = True
    return {"started": started, "reason": reason, "outputs": outputs}
 # ------------------------------------------------------------------------------ bundle
 class Bundle:
    def __init__(self, directory: Path) -> None:
        self.directory = directory
        self.files = []
    def add(self, rel: str, text: str, layer: str, description: str, note: str = "") -> None:
        write_text(self.directory / rel, text)
        self.files.append((rel, layer, description, note))
 def stamp_now() -> str:
    return datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
 def unique_stamp(out: Path) -> str:
    stamp = stamp_now()
    candidate, n = stamp, 1
    while (out / ("bugreport-" + candidate)).exists() or (out / ("bugreport-%s.zip" % candidate)).exists():
        n += 1
        candidate = "%s-%d" % (stamp, n)
    return candidate
 def build_manifest(args, root, stamp, bundle, wikitool, trace, gaps) -> str:
    lines = ["# Bug report bundle", "",
             "Collected: %s UTC  " % datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M:%S"),
             "Bundle: `bugreport-%s`" % stamp, "", "## Privacy", "", PRIVACY_NOTICE, "",
             "## Options", "",
             "- chronology: %s" % ("supplied" if args.chronology else "not supplied"),
             "- transcripts: %d" % len(args.transcript),
             "- page titles (`--titles`): %s" % ("included" if args.titles else "kept out"),
             "- session trace: %s" % (
                 "excluded (`--no-trace`)" if args.no_trace else
                 ("included" if trace and trace.get("found") else "not found")),
             "", "## wikitool", ""]
    if wikitool is None:
        lines.append("`wikitool` did not start: no launcher found under `tools/`.")
    elif wikitool["started"]:
        lines.append("`wikitool` started. Five read commands ran; outputs are verbatim.")
    else:
        lines.append("`wikitool` did not start (%s). The four other commands were not run."
                     % wikitool["reason"])
    if wikitool:
        lines += ["", "| Command | Session id |", "|---|---|"]
        for out in wikitool["outputs"]:
            lines.append("| `%s` | %s |" % (out["label"], out["session"]))
    caller, source = resolve_trace_session(args.session)
    lines += ["", "Caller session id: %s" % (
        "%s (from %s)" % (caller, source) if caller else
        "not set; `wikitool` falls back to its parent process id"), ""]
    if trace and not trace.get("found") and not args.no_trace:
        lines += ["No trace found for the caller's session%s. Existing session directories "
                  "(name and size only):" % (
                      " `%s`" % trace["session"] if trace.get("session") else ""), ""]
        for item in trace.get("listing", [])[:100]:
            lines.append("- `%s` (%d bytes)" % (item["session_directory"], item["bytes"]))
        if not trace.get("listing"):
            lines.append("- none")
        lines.append("")
    lines += ["## Files", "", "| File | Layer | Contents | Note |", "|---|---|---|---|"]
    for rel, layer, description, note in sorted(bundle.files, key=lambda f: f[0]):
        lines.append("| `%s` | %s | %s | %s |" % (rel, layer, description, note))
    lines.append("| `MANIFEST.md` | - | this file | |")
    if gaps:
        lines += ["", "## Gaps", ""] + ["- %s" % gap for gap in gaps]
    return "\n".join(lines) + "\n"
 def final_pass(bundle_dir: Path, red: Redactor, shield, skip) -> None:
    for path in sorted(bundle_dir.rglob("*")):
        if not path.is_file():
            continue
        rel = path.relative_to(bundle_dir).as_posix()
        text = read_text(path)
        cleaned = red.scrub(text)
        if shield is not None and rel not in skip:
            cleaned = shield.apply(cleaned)
        if cleaned != text:
            write_text(path, cleaned)
 def make_zip(bundle_dir: Path, archive: Path) -> None:
    with zipfile.ZipFile(str(archive), "w", zipfile.ZIP_DEFLATED) as zf:
        for path in sorted(bundle_dir.rglob("*")):
            if path.is_file():
                zf.write(str(path), bundle_dir.name + "/" + path.relative_to(bundle_dir).as_posix())
 def parse_args(argv):
    parser = argparse.ArgumentParser(
        prog="bugreport.py",
        description="Collect a bug-report bundle. Removes secrets, keeps page titles out "
        "unless --titles is given, uploads nothing.",
    )
    parser.add_argument("--chronology", help="the agent's fact-only chronology (Markdown)")
    parser.add_argument("--transcript", action="append", default=[],
                        help="a harness transcript to include (repeatable; only on request)")
    group = parser.add_mutually_exclusive_group()
    group.add_argument("--session", help="session id whose trace to include")
    group.add_argument("--no-trace", action="store_true", help="leave the session trace out")
    parser.add_argument("--titles", action="store_true",
                        help="keep page titles and paths (writes tree-paths.txt)")
    parser.add_argument("--root", help="checkout root (default: parent of tools/)")
    parser.add_argument("--out", help="output directory (default: <root>/reports)")
    return parser.parse_args(argv)
 def main(argv=None) -> int:
    args = parse_args(argv)
    root = Path(args.root).resolve() if args.root else Path(__file__).resolve().parent.parent
    out = Path(args.out).resolve() if args.out else root / "reports"
    inputs = [("chronology", args.chronology)] + [("transcript", t) for t in args.transcript]
    for label, value in inputs:
        if value and not Path(value).is_file():
            sys.stderr.write("bugreport: %s file not found: %s\n" % (label, value))
            return 1
    try:
        out.mkdir(parents=True, exist_ok=True)
        stamp = unique_stamp(out)
        bundle_dir = out / ("bugreport-" + stamp)
        bundle_dir.mkdir()
        return _collect(args, root, out, stamp, bundle_dir)
    except OSError as exc:
        sys.stderr.write("bugreport: cannot write the bundle: %s\n" % exc)
        return 1
 def _collect(args, root: Path, out: Path, stamp: str, bundle_dir: Path) -> int:
    red = Redactor()
    bundle = Bundle(bundle_dir)
    gaps = []
    trace = None
    if not args.no_trace:
        trace = guarded(collect_trace, root, args.session)
        if trace.get("found"):
            bundle.add("trace.jsonl", trace["text"], "3", "session trace of the caller",
                       MAY_CONTAIN_CONTENT)
    bundle.add("environment.json", dump_json(guarded(collect_environment, root, red)), "1",
               "OS, Python, shells, harness, PATH, environment variables (names; values for a "
               "fixed list), git configuration, venv, line endings, Windows details")
    stack = guarded(collect_stack, root, red)
    bundle.add("stack.json", dump_json(stack), "2",
               "VERSION, `.wikitool-*.json` (secrets removed), git status and log")
    content_paths = []
    tree = {}
    for name in CONTENT_DIRS:
        summary, files = _safe_scan(root, name)
        tree[name] = summary
        content_paths.extend(files)
    bundle.add("tree-structure.json", dump_json(tree), "2",
               "kb/ and raw/: counts, depth, path lengths, name problems - no content")
    if args.titles:
        bundle.add("tree-paths.txt", "\n".join(sorted(content_paths)) + "\n", "2",
                   "all relative paths under kb/ and raw/", "contains page titles")
    wikitool = None
    launcher = launcher_command(root)
    if launcher:
        wikitool = run_wikitool(root, launcher, "bugreport-" + stamp)
        for item in wikitool["outputs"]:
            bundle.add(item["file"], item["text"], "3", "verbatim output of `%s`" % item["label"])
    else:
        gaps.append("no launcher found under tools/, so no wikitool command ran")
    if args.chronology:
        bundle.add("CHRONOLOGY.md", read_text(Path(args.chronology)), "4",
                   "the agent's chronology", MAY_CONTAIN_CONTENT)
    else:
        gaps.append("no chronology was supplied")
    used = set()
    for source in args.transcript:
        name = Path(source).name
        target, n = name, 1
        while target in used:
            n += 1
            target = "%d-%s" % (n, name)
        used.add(target)
        bundle.add("transcripts/" + target, read_text(Path(source)), "4",
                   "harness transcript", MAY_CONTAIN_CONTENT)
    write_text(bundle_dir / "MANIFEST.md",
               build_manifest(args, root, stamp, bundle, wikitool, trace, gaps))
    shield = None
    if not args.titles:
        shield = TitleShield(
            rel for rel in content_paths if rel.rsplit("/", 1)[-1] not in STACK_NAMES
        )
    final_pass(bundle_dir, red, shield, {"tree-paths.txt"})
    archive = out / ("bugreport-%s.zip" % stamp)
    make_zip(bundle_dir, archive)
    print("Bundle:  %s" % bundle_dir)
    print("Archive: %s" % archive)
    print()
    print(PRIVACY_NOTICE)
    return 0
 def _safe_scan(root: Path, name: str):
    try:
        return scan_tree(root, name)
    except Exception as exc:  # noqa: BLE001
        return {"collector_error": "%s: %s" % (type(exc).__name__, exc)}, []
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,327 @@
 """`tools/bugreport.py`, the bug-report collector.
 The collector is a standalone script - standard library only, no `chemenu` import -
 so these tests import it the way `test_trace_ingest.py` imports its script, and run
 it against a small fake checkout with a fake `tools/wikitool` launcher.
 """
 import ast
 import json
 import os
 import subprocess
 import sys
 import zipfile
 from pathlib import Path
 import pytest
 import bugreport
 SCRIPT = Path(bugreport.__file__)
 TOKEN = "s3cr3t-tok3n-value-9F2"
 PASSWORD = "hunter2-hunter2"
 TITLE = "Vertraulicher Titel Alpha"
 OTHER_TITLE = "Zweite Seite Beta"
 LAUNCHER = """#!{python}
 import os, sys
 args = sys.argv[1:]
 print("args:", " ".join(args))
 print("session:", os.environ.get("WIKITOOL_SESSION_ID", "<unset>"))
 print("token-in-env:", os.environ.get("MY_API_TOKEN", "<unset>"))
 print("see [[{title}]] and kb/Concepts/{title}.md and kb/Concepts/CONTRACT.md")
 if args[:2] == ["version", "show"] and os.environ.get("FAKE_NOT_STARTING"):
    sys.stderr.write("ModuleNotFoundError: No module named 'typer'\\n")
    sys.exit(1)
 if os.environ.get("FAKE_SLEEP") and args[0] == "doctor":
    import time
    time.sleep(30)
 """
 def git(root: Path, *args: str) -> None:
    subprocess.run(
        ["git", "-c", "user.name=Fixture", "-c", "user.email=f@example.invalid", *args],
        cwd=root, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
    )
@pytest.fixture
 def checkout(tmp_path: Path, monkeypatch) -> Path:
    root = tmp_path / "checkout"
    (root / "kb" / "Concepts").mkdir(parents=True)
    (root / "raw" / "articles").mkdir(parents=True)
    (root / "tools").mkdir()
    (root / "kb" / "Concepts" / f"{TITLE}.md").write_text("body of the page\n")
    (root / "kb" / "Concepts" / f"{OTHER_TITLE}.md").write_text("more\n")
    (root / "kb" / "Concepts" / "CONTRACT.md").write_text("stack file\n")
    (root / "raw" / "articles" / "Quelle Gamma.md").write_text("source\n")
    (root / "VERSION").write_text("1.2.3\n")
    (root / ".wikitool-upload.json").write_text(
        json.dumps({"schema": 1, "api_token": TOKEN, "enabled": True})
    )
    (root / ".wikitool-tools.json").write_text(
        json.dumps({"schema": 1, "tools": {"python": {"path": "/definitely/not/here/python"}}})
    )
    launcher = root / "tools" / "wikitool"
    launcher.write_text(LAUNCHER.format(python=sys.executable, title=TITLE))
    launcher.chmod(0o755)
    git(root, "init", "-q", "-b", "main")
    git(root, "remote", "add", "origin", f"https://oauth:{PASSWORD}@git.example.invalid/x/y.git")
    git(root, "add", "-A")
    git(root, "commit", "-q", "-m", "seed")
    (root / "kb" / "Concepts" / f"{TITLE}.md").write_text("changed\n")
    monkeypatch.delenv("WIKI_TRACE_DIR", raising=False)
    monkeypatch.setenv("MY_API_TOKEN", TOKEN)
    monkeypatch.setenv("PATH", os.environ["PATH"])
    return root
 def collect(root: Path, tmp_path: Path, *extra: str) -> Path:
    out = tmp_path / "out"
    code = bugreport.main(["--root", str(root), "--out", str(out), *extra])
    assert code == 0
    bundles = [p for p in out.iterdir() if p.is_dir()]
    assert len(bundles) == 1
    return bundles[0]
 def all_text(bundle: Path) -> str:
    return "\n".join(
        p.read_text(encoding="utf-8", errors="replace") for p in bundle.rglob("*") if p.is_file()
    )
 def test_the_script_is_standard_library_only_and_python38_syntax():
    source = SCRIPT.read_text(encoding="utf-8")
    tree = ast.parse(source, feature_version=(3, 8))
    imported = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            imported.update(alias.name.split(".")[0] for alias in node.names)
        elif isinstance(node, ast.ImportFrom) and node.module:
            imported.add(node.module.split(".")[0])
    assert "chemenu" not in imported
    assert imported <= set(sys.stdlib_module_names) if hasattr(sys, "stdlib_module_names") else True
 def test_the_script_starts_on_a_bare_interpreter():
    proc = subprocess.run(
        [sys.executable, "-I", "-S", str(SCRIPT), "--help"], capture_output=True, text=True
    )
    assert proc.returncode == 0
    assert "--no-trace" in proc.stdout
 def test_a_bundle_and_an_archive_are_written_with_a_manifest(checkout, tmp_path):
    bundle = collect(checkout, tmp_path, "--no-trace")
    for name in ("MANIFEST.md", "environment.json", "stack.json", "tree-structure.json"):
        assert (bundle / name).is_file(), name
    assert (bundle / "wikitool" / "doctor.txt").is_file()
    manifest = (bundle / "MANIFEST.md").read_text()
    assert "not pseudonymised" in manifest
    assert "no chronology was supplied" in manifest
    archive = bundle.with_name(bundle.name + ".zip")
    with zipfile.ZipFile(archive) as zf:
        names = zf.namelist()
    assert f"{bundle.name}/MANIFEST.md" in names
 def test_secrets_are_removed_everywhere_including_the_archive(checkout, tmp_path):
    bundle = collect(checkout, tmp_path, "--no-trace")
    text = all_text(bundle)
    assert TOKEN not in text
    assert PASSWORD not in text
    stack = json.loads((bundle / "stack.json").read_text())
    assert stack["config_files"][".wikitool-upload.json"]["api_token"] == "<removed>"
    archive = bundle.with_name(bundle.name + ".zip")
    with zipfile.ZipFile(archive) as zf:
        for name in zf.namelist():
            data = zf.read(name).decode("utf-8", errors="replace")
            assert TOKEN not in data and PASSWORD not in data, name
 def test_a_secret_from_a_config_file_is_also_scrubbed_from_a_chronology(checkout, tmp_path):
    chron = tmp_path / "chron.md"
    chron.write_text(f"1. Setup failed with {TOKEN} in the header\n")
    bundle = collect(checkout, tmp_path, "--no-trace", "--chronology", str(chron))
    assert TOKEN not in (bundle / "CHRONOLOGY.md").read_text()
    assert "1. Setup failed" in (bundle / "CHRONOLOGY.md").read_text()
 def test_environment_records_every_name_but_values_only_for_the_allowlist(
    checkout, tmp_path, monkeypatch
 ):
    monkeypatch.setenv("SOMETHING_PRIVATE", "very-private-value")
    monkeypatch.setenv("WIKI_TRACE", "1")
    monkeypatch.setenv("WIKITOOL_UPDATE_TOKEN", "allowlisted-but-secret")
    bundle = collect(checkout, tmp_path, "--no-trace")
    env = json.loads((bundle / "environment.json").read_text())["env"]
    assert env["SOMETHING_PRIVATE"] == "<not collected>"
    assert env["MY_API_TOKEN"] == "<not collected>"
    assert env["WIKI_TRACE"] == "1"
    assert env["WIKITOOL_UPDATE_TOKEN"] == "<removed>"
    assert "PATH" in env and env["PATH"] == os.environ["PATH"]
    text = all_text(bundle)
    assert "very-private-value" not in text
    assert "allowlisted-but-secret" not in text
 def test_page_titles_are_kept_out_by_default_and_kept_with_titles(checkout, tmp_path):
    bundle = collect(checkout, tmp_path, "--no-trace")
    text = all_text(bundle)
    assert TITLE not in text
    assert OTHER_TITLE not in text
    assert "Quelle Gamma" not in text
    assert not (bundle / "tree-paths.txt").exists()
    assert "kb/Concepts/CONTRACT.md" in text  # stack-generated names are not titles
    kept = collect(checkout, tmp_path / "second", "--no-trace", "--titles")
    assert TITLE in all_text(kept)
    assert TITLE in (kept / "tree-paths.txt").read_text()
 def test_wikilinks_in_captured_output_lose_their_title(checkout, tmp_path):
    bundle = collect(checkout, tmp_path, "--no-trace")
    doctor = (bundle / "wikitool" / "doctor.txt").read_text()
    assert f"[[{TITLE}]]" not in doctor
    assert "[[<title" in doctor
 def test_counting_calls_run_under_the_collectors_own_session_inherited_ones_do_not(
    checkout, tmp_path, monkeypatch
 ):
    monkeypatch.setenv("WIKITOOL_SESSION_ID", "callers-session")
    bundle = collect(checkout, tmp_path, "--no-trace")
    own = f"bugreport-{bundle.name.split('bugreport-', 1)[1]}"
    for name in ("instructions-verify", "docs-verify"):
        assert f"session: {own}" in (bundle / "wikitool" / f"{name}.txt").read_text()
    for name in ("version-show", "doctor", "budget-status"):
        assert "session: callers-session" in (bundle / "wikitool" / f"{name}.txt").read_text()
    manifest = (bundle / "MANIFEST.md").read_text()
    assert "callers-session (from WIKITOOL_SESSION_ID)" in manifest
 def test_the_callers_trace_is_copied_and_no_trace_leaves_it_out(
    checkout, tmp_path, monkeypatch
 ):
    monkeypatch.setenv("WIKITOOL_SESSION_ID", "callers-session")
    trace_dir = checkout / "reports" / "telemetry" / "callers-session"
    trace_dir.mkdir(parents=True)
    (trace_dir / "trace.jsonl").write_text('{"event": "x"}\n')
    other = checkout / "reports" / "telemetry" / "other-session"
    other.mkdir()
    (other / "trace.jsonl").write_text('{"event": "y"}\n')
    with_trace = collect(checkout, tmp_path / "a")
    assert (with_trace / "trace.jsonl").read_text() == '{"event": "x"}\n'
    assert "may contain page content and titles" in (with_trace / "MANIFEST.md").read_text()
    without = collect(checkout, tmp_path / "b", "--no-trace")
    assert not (without / "trace.jsonl").exists()
 def test_a_missing_trace_lists_the_sessions_that_exist(checkout, tmp_path, monkeypatch):
    monkeypatch.setenv("WIKITOOL_SESSION_ID", "nobody-wrote-this")
    other = checkout / "reports" / "telemetry" / "some-session"
    other.mkdir(parents=True)
    (other / "trace.jsonl").write_text("12345")
    bundle = collect(checkout, tmp_path)
    assert not (bundle / "trace.jsonl").exists()
    manifest = (bundle / "MANIFEST.md").read_text()
    assert "some-session" in manifest and "5 bytes" in manifest
 def test_a_bugreport_bucket_is_never_offered_as_a_trace(checkout, monkeypatch):
    monkeypatch.setenv("WIKITOOL_SESSION_ID", "missing")
    base = checkout / "reports" / "telemetry"
    (base / "bugreport-20260101T000000Z").mkdir(parents=True)
    (base / "real-session").mkdir()
    trace = bugreport.collect_trace(checkout, None)
    assert [item["session_directory"] for item in trace["listing"]] == ["real-session"]
 def test_a_wikitool_that_does_not_start_still_yields_a_report(checkout, tmp_path, monkeypatch):
    monkeypatch.setenv("FAKE_NOT_STARTING", "1")
    bundle = collect(checkout, tmp_path, "--no-trace")
    assert (bundle / "wikitool" / "version-show.txt").is_file()
    assert not (bundle / "wikitool" / "doctor.txt").exists()
    assert "ModuleNotFoundError" in (bundle / "wikitool" / "version-show.txt").read_text()
    manifest = (bundle / "MANIFEST.md").read_text()
    assert "did not start" in manifest
    assert (bundle / "environment.json").is_file()
 def test_no_launcher_at_all_still_yields_a_report(checkout, tmp_path):
    (checkout / "tools" / "wikitool").unlink()
    bundle = collect(checkout, tmp_path, "--no-trace")
    assert not (bundle / "wikitool").exists()
    assert "no launcher" in (bundle / "MANIFEST.md").read_text()
 def test_a_hanging_wikitool_call_is_cut_off(checkout, tmp_path, monkeypatch):
    monkeypatch.setattr(bugreport, "TIMEOUT_SECONDS", 1)
    monkeypatch.setenv("FAKE_SLEEP", "1")
    bundle = collect(checkout, tmp_path, "--no-trace")
    assert "timed out after 1 s" in (bundle / "wikitool" / "doctor.txt").read_text()
    assert (bundle / "wikitool" / "docs-verify.txt").is_file()
 def test_the_tools_config_path_check_reports_what_is_missing(checkout, tmp_path):
    bundle = collect(checkout, tmp_path, "--no-trace")
    config = json.loads((bundle / "environment.json").read_text())["tools_config"]
    assert config["status"] == "present"
    assert config["path_checks"] == [
        {"at": "tools.python.path", "path": "/definitely/not/here/python", "exists": False}
    ]
 def test_a_missing_chronology_file_is_an_error_and_writes_nothing(checkout, tmp_path):
    out = tmp_path / "out"
    code = bugreport.main(
        ["--root", str(checkout), "--out", str(out), "--chronology", str(tmp_path / "nope.md")]
    )
    assert code == 1
    assert not out.exists() or not list(out.iterdir())
 def test_session_and_no_trace_exclude_each_other(checkout, tmp_path):
    with pytest.raises(SystemExit):
        bugreport.main(["--root", str(checkout), "--session", "x", "--no-trace"])
 def test_transcripts_are_added_and_scrubbed(checkout, tmp_path):
    transcript = tmp_path / "t.jsonl"
    transcript.write_text(f'{{"msg": "token {TOKEN}"}}\n')
    bundle = collect(checkout, tmp_path, "--no-trace", "--transcript", str(transcript))
    copied = (bundle / "transcripts" / "t.jsonl").read_text()
    assert TOKEN not in copied
 def test_two_bundles_in_the_same_second_do_not_collide(checkout, tmp_path, monkeypatch):
    monkeypatch.setattr(bugreport, "stamp_now", lambda: "20260101T000000Z")
    out = tmp_path / "out"
    assert bugreport.main(["--root", str(checkout), "--out", str(out), "--no-trace"]) == 0
    assert bugreport.main(["--root", str(checkout), "--out", str(out), "--no-trace"]) == 0
    assert sorted(p.name for p in out.iterdir() if p.is_dir()) == [
        "bugreport-20260101T000000Z", "bugreport-20260101T000000Z-2",
    ]
 def test_the_collector_leaves_the_checkout_untouched(checkout, tmp_path):
    def snapshot():
        return sorted(
            (p.relative_to(checkout).as_posix(), p.stat().st_mtime_ns)
            for p in checkout.rglob("*")
            if p.is_file() and ".git/" not in p.as_posix()
        )
    before = snapshot()
    collect(checkout, tmp_path, "--no-trace")
    assert snapshot() == before
 def test_the_collector_ships_with_a_distribution():
    from chemenu.commands import dist_cmd
    assert "tools/bugreport.py" in dist_cmd.build_plan()