tools/bugreport: --bundle/--chronology tauscht die Chronologie eines vorhandenen Bündels - Geheimnisse, Titelschutz und Pseudonyme wie beim Sammeln, kein wikitool-Aufruf, kein zweites Bündel; bug-report.md prüft die Chronologie vor dem Sammeln und nennt den Austausch als Entscheidungspunkt (#220); 10.1.0-beta.15
Files changed: - CHANGES.md - VERSION - docs/how-to/troubleshoot.md - instructions/bug-report.md - tools/README.md - tools/bugreport.py - tools/chemenu/tests/test_bugreport.py Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
b219adde91
commit
b48a35b224
7 files changed
+371
-27
No files matched your search
+2
-1
@@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10.1.0-beta.14 - 2026-10-16 - gates.md/session-setup.md: die Regel für eine neue Session-Id steht nur noch in gates.md, mit dem ausdrücklich gewünschten weiteren Ingest-Lauf als zweiter Grenze neben plan.md (#219)
|
## 10.1.0-beta.15 - 2026-10-16 - tools/bugreport: eine nach dem Sammeln korrigierte Chronologie wird mit --bundle/--chronology in das vorhandene Bündel getauscht (gleiche Verarbeitung, kein zweiter Lauf, kein zweites Bündel); bug-report.md prüft die Chronologie vor dem Sammeln (#220)
|
||||||
|
|
||||||
**Author:** Torben Nehmer
|
**Author:** Torben Nehmer
|
||||||
|
|
||||||
@@ -28,6 +28,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model.
|
|||||||
- wiki-query liest den Wortlaut geführt in der Rohdatei nach: über Seite, Source-Seite und Gliederung der Ableitung bis zum Zeilenbereich, nie freies Suchen in raw/; kb/CONTRACT.md präzisiert das Qualitätsziel auf die Substanz (#227)
|
- wiki-query liest den Wortlaut geführt in der Rohdatei nach: über Seite, Source-Seite und Gliederung der Ableitung bis zum Zeilenbereich, nie freies Suchen in raw/; kb/CONTRACT.md präzisiert das Qualitätsziel auf die Substanz (#227)
|
||||||
- export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221)
|
- export run: commit= im Marker je Seite statt je Ziel - ein Commit schreibt nur die Dateien der Seiten neu, die er berührt hat; der erste Lauf nach dem Update schreibt jede Datei einmal neu (#221)
|
||||||
- wiki-ingest: ein Lauf endet nach einer Quelle mit Stopp und /clear-Empfehlung - keine Schleife über mehrere Einträge im selben Kontext; capture-session delegiert wiki-ingest nicht mehr an Subagents (#219)
|
- wiki-ingest: ein Lauf endet nach einer Quelle mit Stopp und /clear-Empfehlung - keine Schleife über mehrere Einträge im selben Kontext; capture-session delegiert wiki-ingest nicht mehr an Subagents (#219)
|
||||||
|
- tools/bugreport: eine nach dem Sammeln korrigierte Chronologie wird mit --bundle/--chronology in das vorhandene Bündel getauscht (gleiche Verarbeitung, kein zweiter Lauf, kein zweites Bündel); bug-report.md prüft die Chronologie vor dem Sammeln (#220)
|
||||||
|
|
||||||
**Low impact**
|
**Low impact**
|
||||||
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
- instructions/dev: Konverter-Vergleich – eigene Ableitung gegen Referenzwerkzeuge am privaten Korpus, mit Grundlinie
|
||||||
|
|||||||
@@ -7,6 +7,20 @@ names its own stops - what is missing, why, and the command that fixes it - and
|
|||||||
For the MCP read server, see the troubleshooting section of
|
For the MCP read server, see the troubleshooting section of
|
||||||
[Set up the MCP read server](set-up-the-mcp-server.md#troubleshooting).
|
[Set up the MCP read server](set-up-the-mcp-server.md#troubleshooting).
|
||||||
|
|
||||||
|
<!-- wikitool:toc -->
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- [`tools/wikitool` ends with `STOP - this checkout is not set up yet` (exit 42)](#toolswikitool-ends-with-stop---this-checkout-is-not-set-up-yet-exit-42)
|
||||||
|
- [The agent offers no skills (`wiki-ingest`, `wiki-query`, ...)](#the-agent-offers-no-skills-wiki-ingest-wiki-query)
|
||||||
|
- [`doctor` reports `personalization: FAIL`](#doctor-reports-personalization-fail)
|
||||||
|
- [`doctor` reports `environment: WARN`](#doctor-reports-environment-warn)
|
||||||
|
- [`doctor` reports `telemetry: WARN`](#doctor-reports-telemetry-warn)
|
||||||
|
- [`new` stops with "No author configured"](#new-stops-with-no-author-configured)
|
||||||
|
- [`publish` ends with exit 42 (Mass-Update Gate)](#publish-ends-with-exit-42-mass-update-gate)
|
||||||
|
- [`publish` ends with exit 42 (Publish-Remote Gate)](#publish-ends-with-exit-42-publish-remote-gate)
|
||||||
|
- [Setup or an update fails, and you want to report it](#setup-or-an-update-fails-and-you-want-to-report-it)
|
||||||
|
<!-- /wikitool:toc -->
|
||||||
|
|
||||||
## `tools/wikitool` ends with `STOP - this checkout is not set up yet` (exit 42)
|
## `tools/wikitool` ends with `STOP - this checkout is not set up yet` (exit 42)
|
||||||
|
|
||||||
The preflight has not run in this checkout yet, or not since the last update. Run it:
|
The preflight has not run in this checkout yet, or not since the last update. Run it:
|
||||||
@@ -94,4 +108,11 @@ The script writes a bundle to `reports/bugreport-<timestamp>/`, with a zip besid
|
|||||||
the checklist and the candidates file contain the originals, and they sit beside the bundle,
|
the checklist and the candidates file contain the originals, and they sit beside the bundle,
|
||||||
never in it.
|
never in it.
|
||||||
|
|
||||||
|
- **Correcting the chronology afterwards:** if the chronology turns out wrong or missing once the
|
||||||
|
bundle exists, the agent swaps it with `tools/bugreport --bundle … --chronology …` instead of
|
||||||
|
collecting again. That changes only `CHRONOLOGY.md` and the zip - no `wikitool` command runs
|
||||||
|
again and no second bundle appears - and the new text is scrubbed, kept clear of titles and
|
||||||
|
pseudonymised like the rest. After a stage 2 run the agent reads the new chronology and runs
|
||||||
|
stage 2 once more for names it finds there.
|
||||||
|
|
||||||
Nothing is uploaded anywhere. You choose the channel.
|
Nothing is uploaded anywhere. You choose the channel.
|
||||||
@@ -47,6 +47,14 @@ four layers:
|
|||||||
`MANIFEST.md` lists every file and marks the ones that may contain page content and titles: the trace,
|
`MANIFEST.md` lists every file and marks the ones that may contain page content and titles: the trace,
|
||||||
the chronology and the transcripts.
|
the chronology and the transcripts.
|
||||||
|
|
||||||
|
**A bundle is the output of one run.** Its `wikitool` outputs come from one moment. A chronology
|
||||||
|
that is wrong or missing afterwards is swapped into the bundle that exists
|
||||||
|
(`--bundle <dir> --chronology <file>`, see [Decision points](#decision-points)), never collected again: that changes
|
||||||
|
`CHRONOLOGY.md` and the zip and nothing else, runs no `wikitool` command, and leaves no second bundle
|
||||||
|
under `reports/` for the user to choose between. The new text goes through the same processing as at
|
||||||
|
collection - secrets removed, page titles kept out unless the bundle was collected with `--titles`,
|
||||||
|
and in a pseudonymised bundle every identity replaced from its mapping.
|
||||||
|
|
||||||
Two rules hold for everything the script generates itself. **Secrets are always removed**: values of
|
Two rules hold for everything the script generates itself. **Secrets are always removed**: values of
|
||||||
keys that look like a token, password, secret, key or auth entry, credentials in URLs, and every value
|
keys that look like a token, password, secret, key or auth entry, credentials in URLs, and every value
|
||||||
of that kind found while collecting is also replaced wherever else it turns up. **Page titles are kept
|
of that kind found while collecting is also replaced wherever else it turns up. **Page titles are kept
|
||||||
@@ -96,6 +104,12 @@ session. The bundle does not leave that place because of it.
|
|||||||
[template](#chronology-template) below. Facts only - no diagnosis. If the session's own history is
|
[template](#chronology-template) below. Facts only - no diagnosis. If the session's own history is
|
||||||
too long to reconstruct, say what is missing instead of filling the gap.
|
too long to reconstruct, say what is missing instead of filling the gap.
|
||||||
|
|
||||||
|
**Check the chronology against the template before you collect.** Each step carries its command
|
||||||
|
literally, as it ran - no placeholder and no abbreviation such as `types describe <entity|concept|...>` -
|
||||||
|
with its exit code and the first relevant line of output, verbatim. Fix it now: it is the
|
||||||
|
cheap place to do so. Once the bundle exists, a correction is the swap in
|
||||||
|
[Decision points](#decision-points).
|
||||||
|
|
||||||
3. **Run the collector** through its launcher, which finds a Python 3.8 or later itself:
|
3. **Run the collector** through its launcher, which finds a Python 3.8 or later itself:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -131,8 +145,8 @@ session. The bundle does not leave that place because of it.
|
|||||||
one, in the session. The exception it gets in [gates.md](gates.md) § "Taking a new session id"
|
one, in the session. The exception it gets in [gates.md](gates.md) § "Taking a new session id"
|
||||||
belongs to the script alone.
|
belongs to the script alone.
|
||||||
|
|
||||||
6. **Report the result.** Quote the bundle path, the archive path and the privacy notice the script
|
6. **Report the result.** Name exactly one bundle: quote its path, the archive path and the privacy
|
||||||
prints, name the gaps in `MANIFEST.md`, and tell the user to read the bundle before sharing it.
|
notice the script prints (after a swap, the one the swap printed), name the gaps in `MANIFEST.md`, and tell the user to read the bundle before sharing it.
|
||||||
After stage 2, name the residual uncertainty in words of your own: stage 2 is a model's judgement
|
After stage 2, name the residual uncertainty in words of your own: stage 2 is a model's judgement
|
||||||
and can have missed names, above all in the free text of a trace or transcript over 100 KB.
|
and can have missed names, above all in the free text of a trace or transcript over 100 KB.
|
||||||
Say that the mapping, the review list and the candidate file hold originals and stay on this
|
Say that the mapping, the review list and the candidate file hold originals and stay on this
|
||||||
@@ -172,6 +186,15 @@ Facts only: no page content, no guessed cause, no advice.
|
|||||||
- **The mapping is gone before stage 2?** Stage 2 refuses: it needs the salt stage 1 used, or it would
|
- **The mapping is gone before stage 2?** Stage 2 refuses: it needs the salt stage 1 used, or it would
|
||||||
replace a word differently from the path it already replaced. Collect the report again with
|
replace a word differently from the path it already replaced. Collect the report again with
|
||||||
`--pseudonymise`.
|
`--pseudonymise`.
|
||||||
|
- **The chronology is wrong, incomplete or missing after collecting?** Write the corrected file and
|
||||||
|
run `tools/bugreport --bundle reports/bugreport-<stamp> --chronology <file>`. **Never collect a
|
||||||
|
second time**: that makes a second bundle with `wikitool` outputs from another moment, and nothing
|
||||||
|
says which one counts. The swap works on a bundle collected without a chronology too (it adds
|
||||||
|
one). Add `--root <dir>` if the checkout is not the one holding `tools/`. If stage 2 had already
|
||||||
|
run, read the new `CHRONOLOGY.md` completely and run stage 2 again with the names you now find
|
||||||
|
there. If the swap exits 1 because the mapping is gone, handle it like "The mapping is gone"
|
||||||
|
above: collect again with `--pseudonymise`, and then say in the report which earlier bundle is
|
||||||
|
superseded. Delete nothing under `reports/` yourself; which bundles to keep is the user's call.
|
||||||
- **The user wants a name added after stage 2?** Run stage 2 again with a candidate file holding only
|
- **The user wants a name added after stage 2?** Run stage 2 again with a candidate file holding only
|
||||||
that line. The same run also works when the human spots a name while reading the bundle.
|
that line. The same run also works when the human spots a name while reading the bundle.
|
||||||
- **The user declines the trace?** Run with `--no-trace`. The manifest records the exclusion.
|
- **The user declines the trace?** Run with `--no-trace`. The manifest records the exclusion.
|
||||||
|
|||||||
@@ -148,6 +148,12 @@ exits 0 or 1 - never 42, since it opens no gate. It has a second mode: `--pseudo
|
|||||||
identities it can read from the machine (stage 1), and `--bundle`/`--candidates` applies names a
|
identities it can read from the machine (stage 1), and `--bundle`/`--candidates` applies names a
|
||||||
model found in the result (stage 2), both word by word so that length, separators and depth
|
model found in the result (stage 2), both word by word so that length, separators and depth
|
||||||
survive. The mapping, the review list and the candidate file stay beside the bundle directory.
|
survive. The mapping, the review list and the candidate file stay beside the bundle directory.
|
||||||
|
A third mode, `--bundle`/`--chronology` (with `--root` if the checkout is not the one holding
|
||||||
|
`tools/`), swaps the chronology of a bundle that exists already: it runs no `wikitool` call and
|
||||||
|
writes no second bundle, changes `CHRONOLOGY.md` and the zip (and the review list of a
|
||||||
|
pseudonymised bundle), and puts the new text through the same secret scrub, title shield and
|
||||||
|
pseudonyms the collection would have. A bundle collected without a chronology gets one, and its
|
||||||
|
manifest changes in three fixed places.
|
||||||
`dist export` ships it with the rest of `tools/`;
|
`dist export` ships it with the rest of `tools/`;
|
||||||
its tests (`tests/test_bugreport.py`, in the origin repository) run it on a bare interpreter (`-I -S`) to keep that promise.
|
its tests (`tests/test_bugreport.py`, in the origin repository) run it on a bare interpreter (`-I -S`) to keep that promise.
|
||||||
|
|
||||||
|
|||||||
+148
-23
@@ -10,6 +10,7 @@ Runs on the base Python with the standard library only, and imports nothing from
|
|||||||
[--session ID | --no-trace] [--titles]
|
[--session ID | --no-trace] [--titles]
|
||||||
[--pseudonymise] [--root DIR] [--out DIR]
|
[--pseudonymise] [--root DIR] [--out DIR]
|
||||||
tools/bugreport --bundle DIR --candidates FILE
|
tools/bugreport --bundle DIR --candidates FILE
|
||||||
|
tools/bugreport --bundle DIR --chronology FILE [--root DIR]
|
||||||
|
|
||||||
`tools/bugreport` (and `tools/bugreport.ps1` under PowerShell) finds a Python 3.8
|
`tools/bugreport` (and `tools/bugreport.ps1` under PowerShell) finds a Python 3.8
|
||||||
or newer and runs this file with it, skipping the Microsoft Store's aliases that
|
or newer and runs this file with it, skipping the Microsoft Store's aliases that
|
||||||
@@ -21,7 +22,9 @@ layers: the environment, the stack, what `wikitool` prints (if it starts), and
|
|||||||
the agent's chronology. Secrets are always removed. Page titles are kept out of
|
the agent's chronology. Secrets are always removed. Page titles are kept out of
|
||||||
everything this script generates unless `--titles` is given. With `--pseudonymise`
|
everything this script generates unless `--titles` is given. With `--pseudonymise`
|
||||||
known identities are replaced by consistent, shape-preserving placeholders (stage 1);
|
known identities are replaced by consistent, shape-preserving placeholders (stage 1);
|
||||||
`--bundle`/`--candidates` applies the names a model found beyond them (stage 2). The
|
`--bundle`/`--candidates` applies the names a model found beyond them (stage 2).
|
||||||
|
`--bundle`/`--chronology` swaps the chronology of a bundle that exists already, with the
|
||||||
|
same processing and without a second run: no `wikitool` call, no second bundle. The
|
||||||
mapping stays beside the bundle, never in it. Nothing is uploaded.
|
mapping stays beside the bundle, never in it. Nothing is uploaded.
|
||||||
|
|
||||||
Exit 0 = bundle written, 1 = it could not be written. No exit 42: this script
|
Exit 0 = bundle written, 1 = it could not be written. No exit 42: this script
|
||||||
@@ -1171,15 +1174,25 @@ def build_manifest(args, root, stamp, bundle, wikitool, trace, gaps) -> str:
|
|||||||
return "\n".join(lines) + "\n"
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def clean_text(text: str, red: Redactor, shield) -> str:
|
||||||
|
"""What every generated or supplied text goes through: secrets, then page titles."""
|
||||||
|
cleaned = red.scrub(text)
|
||||||
|
if shield is not None:
|
||||||
|
cleaned = shield.apply(cleaned)
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def title_shield(content_paths):
|
||||||
|
return TitleShield(rel for rel in content_paths if rel.rsplit("/", 1)[-1] not in STACK_NAMES)
|
||||||
|
|
||||||
|
|
||||||
def final_pass(bundle_dir: Path, red: Redactor, shield, skip) -> None:
|
def final_pass(bundle_dir: Path, red: Redactor, shield, skip) -> None:
|
||||||
for path in sorted(bundle_dir.rglob("*")):
|
for path in sorted(bundle_dir.rglob("*")):
|
||||||
if not path.is_file():
|
if not path.is_file():
|
||||||
continue
|
continue
|
||||||
rel = path.relative_to(bundle_dir).as_posix()
|
rel = path.relative_to(bundle_dir).as_posix()
|
||||||
text = read_text(path)
|
text = read_text(path)
|
||||||
cleaned = red.scrub(text)
|
cleaned = clean_text(text, red, None if rel in skip else shield)
|
||||||
if shield is not None and rel not in skip:
|
|
||||||
cleaned = shield.apply(cleaned)
|
|
||||||
if cleaned != text:
|
if cleaned != text:
|
||||||
write_text(path, cleaned)
|
write_text(path, cleaned)
|
||||||
|
|
||||||
@@ -1210,22 +1223,32 @@ def parse_args(argv):
|
|||||||
"placeholders (stage 1); the mapping stays beside the bundle")
|
"placeholders (stage 1); the mapping stays beside the bundle")
|
||||||
parser.add_argument("--root", help="checkout root (default: parent of tools/)")
|
parser.add_argument("--root", help="checkout root (default: parent of tools/)")
|
||||||
parser.add_argument("--out", help="output directory (default: <root>/reports)")
|
parser.add_argument("--out", help="output directory (default: <root>/reports)")
|
||||||
parser.add_argument("--bundle", help="stage 2: an existing pseudonymised bundle directory")
|
parser.add_argument("--bundle", help="an existing bundle directory: stage 2 with --candidates, "
|
||||||
|
"a new chronology with --chronology")
|
||||||
parser.add_argument("--candidates", help="stage 2: file with one further name per line "
|
parser.add_argument("--candidates", help="stage 2: file with one further name per line "
|
||||||
"('#' starts a comment)")
|
"('#' starts a comment)")
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
if bool(args.bundle) != bool(args.candidates):
|
if args.candidates and not args.bundle:
|
||||||
parser.error("--bundle and --candidates belong together")
|
parser.error("--candidates belongs with --bundle")
|
||||||
if args.bundle and (args.chronology or args.transcript or args.session or args.no_trace
|
if args.bundle:
|
||||||
or args.titles or args.pseudonymise or args.root or args.out):
|
if args.candidates and args.chronology:
|
||||||
parser.error("--bundle/--candidates excludes every collection option")
|
parser.error("--bundle takes --candidates or --chronology, not both")
|
||||||
|
if not (args.candidates or args.chronology):
|
||||||
|
parser.error("--bundle needs --candidates (stage 2) or --chronology (swap)")
|
||||||
|
if (args.transcript or args.session or args.no_trace or args.titles
|
||||||
|
or args.pseudonymise or args.out or (args.candidates and args.root)):
|
||||||
|
parser.error("--bundle excludes every collection option "
|
||||||
|
"(--root is allowed with --chronology only)")
|
||||||
return args
|
return args
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
def main(argv=None) -> int:
|
||||||
args = parse_args(argv)
|
args = parse_args(argv)
|
||||||
if args.bundle:
|
if args.bundle and args.candidates:
|
||||||
return run_stage2(Path(args.bundle), Path(args.candidates))
|
return run_stage2(Path(args.bundle), Path(args.candidates))
|
||||||
|
if args.bundle:
|
||||||
|
root = Path(args.root).resolve() if args.root else Path(__file__).resolve().parent.parent
|
||||||
|
return run_exchange(Path(args.bundle), Path(args.chronology), root)
|
||||||
root = Path(args.root).resolve() if args.root else Path(__file__).resolve().parent.parent
|
root = Path(args.root).resolve() if args.root else Path(__file__).resolve().parent.parent
|
||||||
out = Path(args.out).resolve() if args.out else root / "reports"
|
out = Path(args.out).resolve() if args.out else root / "reports"
|
||||||
|
|
||||||
@@ -1308,9 +1331,7 @@ def _collect(args, root: Path, out: Path, stamp: str, bundle_dir: Path) -> int:
|
|||||||
|
|
||||||
shield = None
|
shield = None
|
||||||
if not args.titles:
|
if not args.titles:
|
||||||
shield = TitleShield(
|
shield = title_shield(content_paths)
|
||||||
rel for rel in content_paths if rel.rsplit("/", 1)[-1] not in STACK_NAMES
|
|
||||||
)
|
|
||||||
final_pass(bundle_dir, red, shield, {"tree-paths.txt"})
|
final_pass(bundle_dir, red, shield, {"tree-paths.txt"})
|
||||||
|
|
||||||
pz = None
|
pz = None
|
||||||
@@ -1354,25 +1375,30 @@ def parse_candidates(text: str):
|
|||||||
return names
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def load_pseudonymiser(mapping: Path):
|
||||||
|
"""The pseudonymiser stage 1 left beside a bundle, or the reason it cannot be had."""
|
||||||
|
if not mapping.is_file():
|
||||||
|
return None, ("no mapping beside the bundle (%s): stage 2 needs the salt stage 1 used, so "
|
||||||
|
"collect the report again with --pseudonymise" % mapping.name)
|
||||||
|
try:
|
||||||
|
return Pseudonymiser.from_json(json.loads(read_text(mapping))), None
|
||||||
|
except (ValueError, KeyError) as exc:
|
||||||
|
return None, "the mapping cannot be read: %s" % exc
|
||||||
|
|
||||||
|
|
||||||
def run_stage2(bundle_dir: Path, candidates: Path) -> int:
|
def run_stage2(bundle_dir: Path, candidates: Path) -> int:
|
||||||
bundle_dir = bundle_dir.resolve()
|
bundle_dir = bundle_dir.resolve()
|
||||||
mapping = _sibling(bundle_dir, ".pseudonyms.json")
|
mapping = _sibling(bundle_dir, ".pseudonyms.json")
|
||||||
problem = None
|
pz, problem = None, None
|
||||||
if not bundle_dir.is_dir():
|
if not bundle_dir.is_dir():
|
||||||
problem = "bundle directory not found: %s" % bundle_dir
|
problem = "bundle directory not found: %s" % bundle_dir
|
||||||
elif not candidates.is_file():
|
elif not candidates.is_file():
|
||||||
problem = "candidates file not found: %s" % candidates
|
problem = "candidates file not found: %s" % candidates
|
||||||
elif not mapping.is_file():
|
else:
|
||||||
problem = ("no mapping beside the bundle (%s): stage 2 needs the salt stage 1 used, so "
|
pz, problem = load_pseudonymiser(mapping)
|
||||||
"collect the report again with --pseudonymise" % mapping.name)
|
|
||||||
if problem:
|
if problem:
|
||||||
sys.stderr.write("bugreport: %s\n" % problem)
|
sys.stderr.write("bugreport: %s\n" % problem)
|
||||||
return 1
|
return 1
|
||||||
try:
|
|
||||||
pz = Pseudonymiser.from_json(json.loads(read_text(mapping)))
|
|
||||||
except (ValueError, KeyError) as exc:
|
|
||||||
sys.stderr.write("bugreport: the mapping cannot be read: %s\n" % exc)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
files = text_files(bundle_dir)
|
files = text_files(bundle_dir)
|
||||||
corpus = "\n".join(read_text(path) for path in files)
|
corpus = "\n".join(read_text(path) for path in files)
|
||||||
@@ -1424,6 +1450,105 @@ def run_stage2(bundle_dir: Path, candidates: Path) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
TITLES_LINE = re.compile(r"(?m)^- page titles \(`--titles`\): (included|kept out)$")
|
||||||
|
PSEUDONYMISATION_LINE = re.compile(r"(?m)^- pseudonymisation \(`--pseudonymise`\): (.*)$")
|
||||||
|
CHRONOLOGY_ROW = ("| `CHRONOLOGY.md` | 4 | the agent's chronology | %s |" % MAY_CONTAIN_CONTENT)
|
||||||
|
NO_CHRONOLOGY_GAP = "- no chronology was supplied"
|
||||||
|
|
||||||
|
|
||||||
|
def manifest_with_chronology(manifest: str) -> str:
|
||||||
|
"""The three fixed places a bundle collected without a chronology changes in."""
|
||||||
|
manifest = manifest.replace("- chronology: not supplied", "- chronology: supplied", 1)
|
||||||
|
lines = manifest.split("\n")
|
||||||
|
if CHRONOLOGY_ROW not in lines:
|
||||||
|
end = lines.index("| `MANIFEST.md` | - | this file | |")
|
||||||
|
at = end
|
||||||
|
for index in range(end - 1, -1, -1):
|
||||||
|
if not lines[index].startswith("| `"):
|
||||||
|
break
|
||||||
|
if lines[index].split("`")[1] > "CHRONOLOGY.md":
|
||||||
|
at = index
|
||||||
|
lines.insert(at, CHRONOLOGY_ROW)
|
||||||
|
sections = re.split(r"(?m)^(?=## )", "\n".join(lines))
|
||||||
|
kept = []
|
||||||
|
for section in sections:
|
||||||
|
if section.startswith("## Gaps\n"):
|
||||||
|
bullets = [ln for ln in section.splitlines()[1:] if ln.strip() and ln != NO_CHRONOLOGY_GAP]
|
||||||
|
if not bullets:
|
||||||
|
continue
|
||||||
|
section = "\n".join(["## Gaps", ""] + bullets) + "\n"
|
||||||
|
kept.append(section)
|
||||||
|
return "\n\n".join(part.strip("\n") for part in kept if part.strip()) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def run_exchange(bundle_dir: Path, chronology: Path, root: Path) -> int:
|
||||||
|
"""Swap (or add) CHRONOLOGY.md of an existing bundle: no wikitool call, no second bundle."""
|
||||||
|
bundle_dir = bundle_dir.resolve()
|
||||||
|
manifest_path = bundle_dir / "MANIFEST.md"
|
||||||
|
problem, manifest, titles = None, "", None
|
||||||
|
if not bundle_dir.is_dir():
|
||||||
|
problem = "bundle directory not found: %s" % bundle_dir
|
||||||
|
elif not chronology.is_file():
|
||||||
|
problem = "chronology file not found: %s" % chronology
|
||||||
|
elif not manifest_path.is_file():
|
||||||
|
problem = "no MANIFEST.md in %s: not a bundle of this script" % bundle_dir
|
||||||
|
else:
|
||||||
|
manifest = read_text(manifest_path)
|
||||||
|
titles = TITLES_LINE.search(manifest)
|
||||||
|
if titles is None:
|
||||||
|
problem = "MANIFEST.md in %s has no page titles line: not a bundle of this script" % bundle_dir
|
||||||
|
pz, pseudonymisation = None, PSEUDONYMISATION_LINE.search(manifest)
|
||||||
|
if problem is None and pseudonymisation is not None:
|
||||||
|
pz, problem = load_pseudonymiser(_sibling(bundle_dir, ".pseudonyms.json"))
|
||||||
|
if problem:
|
||||||
|
sys.stderr.write("bugreport: %s\n" % problem)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
# Everything is worked out in memory first; nothing is written before it all succeeded.
|
||||||
|
red = Redactor()
|
||||||
|
guarded(collect_environment, root, red)
|
||||||
|
guarded(collect_stack, root, red)
|
||||||
|
shield = None
|
||||||
|
if titles.group(1) == "kept out":
|
||||||
|
shield = title_shield(f for name in CONTENT_DIRS for f in _safe_scan(root, name)[1])
|
||||||
|
text = clean_text(read_text(chronology), red, shield)
|
||||||
|
if pz is not None:
|
||||||
|
text = pz.apply(text)
|
||||||
|
target = bundle_dir / "CHRONOLOGY.md"
|
||||||
|
added = not target.is_file()
|
||||||
|
new_manifest = manifest_with_chronology(manifest) if added else None
|
||||||
|
|
||||||
|
archive = bundle_dir.with_name(bundle_dir.name + ".zip")
|
||||||
|
review = _sibling(bundle_dir, ".review.txt")
|
||||||
|
try:
|
||||||
|
write_text(target, text)
|
||||||
|
if new_manifest is not None:
|
||||||
|
write_text(manifest_path, new_manifest)
|
||||||
|
if pz is not None:
|
||||||
|
write_text(review, build_review(bundle_dir, pz))
|
||||||
|
temporary = archive.with_name(archive.name + ".tmp")
|
||||||
|
make_zip(bundle_dir, temporary)
|
||||||
|
os.replace(str(temporary), str(archive))
|
||||||
|
except OSError as exc:
|
||||||
|
sys.stderr.write("bugreport: cannot write the bundle: %s\n" % exc)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print("Bundle: %s" % bundle_dir)
|
||||||
|
print("Archive: %s" % archive)
|
||||||
|
if pz is not None:
|
||||||
|
print("Review: %s" % review)
|
||||||
|
print("Read CHRONOLOGY.md and the review list again, then run stage 2: "
|
||||||
|
"tools/bugreport --bundle %s --candidates <file>" % bundle_dir)
|
||||||
|
print()
|
||||||
|
if pz is None:
|
||||||
|
print(PRIVACY_NOTICE)
|
||||||
|
elif "stage 1 and stage 2 applied" in pseudonymisation.group(1):
|
||||||
|
print(RESIDUAL_NOTICE)
|
||||||
|
else:
|
||||||
|
print(STAGE1_NOTICE)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def _safe_scan(root: Path, name: str):
|
def _safe_scan(root: Path, name: str):
|
||||||
try:
|
try:
|
||||||
return scan_tree(root, name)
|
return scan_tree(root, name)
|
||||||
|
|||||||
@@ -557,3 +557,171 @@ def test_bundle_and_candidates_exclude_the_collection_options(checkout, tmp_path
|
|||||||
def test_without_the_flag_nothing_but_the_bundle_and_its_archive_is_written(checkout, tmp_path):
|
def test_without_the_flag_nothing_but_the_bundle_and_its_archive_is_written(checkout, tmp_path):
|
||||||
collect(checkout, tmp_path, "--no-trace")
|
collect(checkout, tmp_path, "--no-trace")
|
||||||
assert sorted(p.suffix for p in (tmp_path / "out").iterdir()) == ["", ".zip"]
|
assert sorted(p.suffix for p in (tmp_path / "out").iterdir()) == ["", ".zip"]
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ swapping the chronology
|
||||||
|
|
||||||
|
|
||||||
|
def swap(bundle: Path, chronology: Path, *extra: str) -> int:
|
||||||
|
return bugreport.main(["--bundle", str(bundle), "--chronology", str(chronology), *extra])
|
||||||
|
|
||||||
|
|
||||||
|
def entries(out: Path) -> list:
|
||||||
|
return sorted(p.name for p in out.iterdir())
|
||||||
|
|
||||||
|
|
||||||
|
def archive_bytes(bundle: Path) -> bytes:
|
||||||
|
return bundle.with_name(bundle.name + ".zip").read_bytes()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swapped_chronology_changes_nothing_else_and_makes_no_second_bundle(checkout, tmp_path):
|
||||||
|
old = tmp_path / "old.md"
|
||||||
|
old.write_text("1. `types describe <entity|concept|...>`\n")
|
||||||
|
bundle = collect(checkout, tmp_path, "--no-trace", "--chronology", str(old))
|
||||||
|
before, listing = snapshot(bundle), entries(tmp_path / "out")
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("1. `types describe entity` - exit 0\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 0
|
||||||
|
after = snapshot(bundle)
|
||||||
|
assert after.pop("CHRONOLOGY.md") == b"1. `types describe entity` - exit 0\n"
|
||||||
|
assert before.pop("CHRONOLOGY.md") != b"1. `types describe entity` - exit 0\n"
|
||||||
|
assert after == before # wikitool/*.txt and MANIFEST.md included
|
||||||
|
assert entries(tmp_path / "out") == listing
|
||||||
|
assert "types describe entity` - exit 0" in zip_text(bundle)
|
||||||
|
assert "<entity|concept" not in zip_text(bundle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swapped_chronology_is_scrubbed_of_a_secret_from_a_config_file(checkout, tmp_path):
|
||||||
|
bundle = collect(checkout, tmp_path, "--no-trace")
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text(f"1. Setup failed with {TOKEN} in the header\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 0
|
||||||
|
text = (bundle / "CHRONOLOGY.md").read_text()
|
||||||
|
assert TOKEN not in text and "1. Setup failed" in text
|
||||||
|
assert TOKEN not in zip_text(bundle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swapped_chronology_keeps_titles_out_unless_the_bundle_was_collected_with_them(
|
||||||
|
checkout, tmp_path
|
||||||
|
):
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text(f"see kb/Concepts/{TITLE}.md and [[{OTHER_TITLE}]]\n")
|
||||||
|
hidden = collect(checkout, tmp_path / "a", "--no-trace")
|
||||||
|
assert swap(hidden, new, "--root", str(checkout)) == 0
|
||||||
|
text = (hidden / "CHRONOLOGY.md").read_text()
|
||||||
|
assert TITLE not in text and OTHER_TITLE not in text and "[[<title" in text
|
||||||
|
|
||||||
|
shown = collect(checkout, tmp_path / "b", "--no-trace", "--titles")
|
||||||
|
assert swap(shown, new, "--root", str(checkout)) == 0
|
||||||
|
assert TITLE in (shown / "CHRONOLOGY.md").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swapped_chronology_in_a_pseudonymised_bundle_uses_both_stages_and_the_same_mapping(
|
||||||
|
checkout, tmp_path, monkeypatch, capsys
|
||||||
|
):
|
||||||
|
first = tmp_path / "first.md"
|
||||||
|
first.write_text("Kunde Acme Corp zahlt.\n")
|
||||||
|
bundle = pseudonymised(checkout, tmp_path, monkeypatch, "--chronology", str(first))
|
||||||
|
candidates = tmp_path / "candidates.txt"
|
||||||
|
candidates.write_text("Acme Corp\n")
|
||||||
|
assert bugreport.main(["--bundle", str(bundle), "--candidates", str(candidates)]) == 0
|
||||||
|
stage_two_word = (bundle / "CHRONOLOGY.md").read_text().split()[1]
|
||||||
|
mapping, review = mapping_of(bundle).read_bytes(), bundle.with_name(bundle.name + ".review.txt")
|
||||||
|
review.write_text("stale\n")
|
||||||
|
manifest_before = (bundle / "MANIFEST.md").read_bytes()
|
||||||
|
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("Acme Corp hat Erika Mustermann angerufen\n")
|
||||||
|
capsys.readouterr()
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 0
|
||||||
|
text = (bundle / "CHRONOLOGY.md").read_text()
|
||||||
|
for original in ("Acme", "Erika", "Mustermann"):
|
||||||
|
assert original not in text and original not in zip_text(bundle)
|
||||||
|
assert text.startswith(stage_two_word + " ") # same placeholder as the stage 2 run gave
|
||||||
|
assert mapping_of(bundle).read_bytes() == mapping
|
||||||
|
assert (bundle / "MANIFEST.md").read_bytes() == manifest_before
|
||||||
|
assert review.read_text() != "stale\n"
|
||||||
|
printed = capsys.readouterr().out
|
||||||
|
assert "Review:" in printed and "stage 1 and stage 2 applied" in printed
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_pseudonymised_bundle_without_its_mapping_is_left_as_it_was(
|
||||||
|
checkout, tmp_path, monkeypatch, capsys
|
||||||
|
):
|
||||||
|
bundle = pseudonymised(checkout, tmp_path, monkeypatch)
|
||||||
|
mapping_of(bundle).unlink()
|
||||||
|
before, archive = snapshot(bundle), archive_bytes(bundle)
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("Erika\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 1
|
||||||
|
assert "collect the report again" in capsys.readouterr().err
|
||||||
|
assert snapshot(bundle) == before and archive_bytes(bundle) == archive
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swap_adds_a_missing_chronology_and_touches_the_manifest_in_three_places(
|
||||||
|
checkout, tmp_path
|
||||||
|
):
|
||||||
|
bundle = collect(checkout, tmp_path, "--no-trace")
|
||||||
|
assert not (bundle / "CHRONOLOGY.md").exists()
|
||||||
|
before = snapshot(bundle)
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("1. `doctor` - exit 1\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 0
|
||||||
|
after = snapshot(bundle)
|
||||||
|
assert after.pop("CHRONOLOGY.md") == b"1. `doctor` - exit 1\n"
|
||||||
|
manifest_before, manifest_after = before.pop("MANIFEST.md"), after.pop("MANIFEST.md")
|
||||||
|
assert after == before
|
||||||
|
old_lines, new_lines = manifest_before.decode().splitlines(), manifest_after.decode().splitlines()
|
||||||
|
removed = [ln for ln in old_lines if ln not in new_lines]
|
||||||
|
added = [ln for ln in new_lines if ln not in old_lines]
|
||||||
|
assert sorted(removed) == ["## Gaps", "- chronology: not supplied", "- no chronology was supplied"]
|
||||||
|
assert sorted(added) == [
|
||||||
|
"- chronology: supplied",
|
||||||
|
"| `CHRONOLOGY.md` | 4 | the agent's chronology | may contain page content and titles |",
|
||||||
|
]
|
||||||
|
files = new_lines[new_lines.index("## Files"):]
|
||||||
|
assert files[4].startswith("| `CHRONOLOGY.md`") # sorted like build_manifest sorts
|
||||||
|
assert new_lines[-1] == "| `MANIFEST.md` | - | this file | |"
|
||||||
|
assert "`doctor` - exit 1" in zip_text(bundle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swap_keeps_the_other_gaps(checkout, tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(bugreport, "launcher_command", lambda root: None)
|
||||||
|
bundle = collect(checkout, tmp_path, "--no-trace")
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("text\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 0
|
||||||
|
manifest = (bundle / "MANIFEST.md").read_text()
|
||||||
|
assert "## Gaps" in manifest and "no launcher found" in manifest
|
||||||
|
assert "no chronology was supplied" not in manifest
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_swap_with_a_missing_file_or_no_manifest_changes_nothing(checkout, tmp_path, capsys):
|
||||||
|
bundle = collect(checkout, tmp_path, "--no-trace")
|
||||||
|
before, archive = snapshot(bundle), archive_bytes(bundle)
|
||||||
|
assert swap(bundle, tmp_path / "nowhere.md", "--root", str(checkout)) == 1
|
||||||
|
assert "chronology file not found" in capsys.readouterr().err
|
||||||
|
new = tmp_path / "new.md"
|
||||||
|
new.write_text("text\n")
|
||||||
|
(bundle / "MANIFEST.md").unlink()
|
||||||
|
before.pop("MANIFEST.md")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 1
|
||||||
|
assert "MANIFEST.md" in capsys.readouterr().err
|
||||||
|
(bundle / "MANIFEST.md").write_text("# Bug report bundle\n")
|
||||||
|
assert swap(bundle, new, "--root", str(checkout)) == 1
|
||||||
|
assert "not a bundle of this script" in capsys.readouterr().err
|
||||||
|
assert not (bundle / "CHRONOLOGY.md").exists()
|
||||||
|
assert archive_bytes(bundle) == archive
|
||||||
|
|
||||||
|
|
||||||
|
def test_bundle_takes_exactly_one_of_candidates_and_chronology(checkout, tmp_path):
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
bugreport.main(["--bundle", "b", "--candidates", "c", "--chronology", "x"])
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
bugreport.main(["--bundle", "b"])
|
||||||
|
for extra in (["--no-trace"], ["--titles"], ["--pseudonymise"], ["--session", "s"],
|
||||||
|
["--transcript", "t"], ["--out", str(tmp_path)]):
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
bugreport.main(["--bundle", "b", "--chronology", "x", *extra])
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
bugreport.main(["--bundle", "b", "--candidates", "c", "--root", str(checkout)])
|
||||||
Reference in new issue
Block a user