chemenu/raw/notes/Conversation Transcript - Hardening the Test Suite Against Silent Environment Dependencies Session 2026-08-31.md

Conversation Transcript - Hardening the Test Suite Against Silent Environment Dependencies Session

Source: Claude Code session (claude-opus-5), llm-wiki-test1 workspace Collected: 2026-08-31 Participant: Torben Fidelity: faithful summary transcript, not a verbatim log. The user's instructions, command output, test counts, commit hashes and CI log excerpts are quoted verbatim. The assistant's reasoning and the order in which files were read are condensed. All numbers and paths below were observed in the session, not reconstructed afterwards. No second-hand material: no subagent was spawned, and every claim was verified in-session by running the command that shows it. No credentials appeared in the session.

Covers the implementation of Gitea issue #8 - an autouse pytest fixture that cuts every test off from the machine it runs on - shipped as 1.7.1 (31c9b81, tag v1.7.1), plus the two follow-up issues the work exposed (#22, #23). The session ran under the stack-dev skill throughout.

---

Turn 1 - the instruction

implementiere issue #8

Issue #8, prio/1 size/M, titled "Testsuite gegen stille Umgebungsabhängigkeiten härten". Read in full along with its one comment, before any code was opened.

What the issue said

The first CI run that ever reached pytest (run 52, 2026-08-30) failed two tests that had been green on every developer machine for months:

FAILED wiki_tools/tests/test_new_page.py::test_new_source_author_falls_back_to_git_config
FAILED wiki_tools/tests/test_provenance.py::test_new_source_with_multiple_raw_files
AssertionError: ERROR No author configured for this instance.
2 failed, 628 passed

Cause: config.default_author() runs git config user.name with cwd=config.ROOT. The fixture root is not a repository, so the answer came from the global git configuration of whoever started pytest. As root in the job container there is none.

Both were repaired in 1.0.1. The issue's own framing of what remained: the suite "hat nicht gewarnt, sie war einfach grün, weil die Umgebung zufällig passte."

The comment, added 2026-08-31 during the 1.2.0 work on #12, is the load-bearing part:

Das ist der dritte und vierte Fall derselben Abhängigkeit, geschrieben von jemandem, der das Issue vorher gelesen hatte. Die Frage aus dem Issue-Text — „wie viele unbekannte gibt es" — ist damit weniger interessant als die andere: die Suite lädt neue Fälle schneller ein, als jemand sie findet.

Two options were on the table, and the issue had already ranked them: an autouse fixture in conftest.py (preferred), or a second hardened pytest step in CI (fallback, "schwächer, weil es die Abhängigkeit erst nach dem Push meldet").

The decision, and why the fallback was rejected

The fixture. The comment settles it: a guard that reports after the push loses to a suite that acquires new cases faster than anyone finds them. A CI-only guard also never protects the developer's own run, which is where the cases are written.

This was not a close call and was not re-litigated. What the session did add was a reason the issue could not have known - see Turn 6.

Turn 2 - measuring before changing

Rather than start from the issue's list, the environment surface was measured directly:

grep -rn "environ\|getenv" wiki_tools/*.py | grep -v tests/

Which yields the tool's own reads: WIKI_AUTHOR (config.py:59), WIKITOOL_SESSION_ID (session.py:17), WIKITOOL_UPDATE_URL / WIKITOOL_UPDATE_TOKEN (version.py:50,53), and the telemetry set WIKI_TRACE, WIKI_TRACE_DIR, WIKI_TRACE_CONTENT, WIKI_TRACE_MAX_CONTENT (telemetry/writer.py, telemetry/scrub.py).

Then the baseline, which turned out to matter:

env -u WIKI_AUTHOR GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null HOME=<empty> \
  .venv/bin/python -m pytest -q
→ 695 passed in 10.30s

The suite was already green under the hardened environment. The four known cases had all been patched in 1.0.1 and 1.2.0, and no fifth existed at that moment. This reframed the change: it is not a repair, it is a guard - and a guard whose value has to be demonstrated separately, because "everything still passes" proves nothing about it.

Turn 3 - what was built

The fixture

hermetic_environment, autouse, in tools/wiki_tools/tests/conftest.py, next to the existing isolated_trace_dir. Per test: HOME and XDG_CONFIG_HOME into that test's own tmp_path, GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM to /dev/null, and two lists cleared.

Beyond the issue's list, git's own identity and location variables were added: GIT_DIR, GIT_WORK_TREE, GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, EMAIL. The reasoning, recorded in the code comment: GIT_AUTHOR_NAME outranks git config user.name, so it is literally the same failure the issue describes through a different door; and a stray GIT_DIR would point every fixture repo at the developer's own checkout.

The ordering problem between two autouse fixtures

hermetic_environment clears WIKI_TRACE. isolated_trace_dir sets WIKI_TRACE_DIR. If the clearing ran second it would not touch WIKI_TRACE_DIR, so the suite would still work - but the correctness would rest on pytest's declaration order rather than on anything stated.

isolated_trace_dir now takes hermetic_environment as a parameter. Not for its value, for the ordering. Both docstrings say so.

WIKI_TRACE_DIR is deliberately the one variable left set. Acceptance criterion 4 of the issue: tracing must never be disabled suite-wide, because two telemetry tests assert that a trace gets written.

What was deliberately not done

No WIKI_AUTHOR in the fixture base. This was the cheaper fix and the wrong one, and the issue had pre-empted it in acceptance criterion 2: a shared default would make default_author()'s None branch untestable, because that branch only exists on a machine that knows nobody. The suite would look greener and prove less.

test_new_source_fails_hard_without_any_author keeps patching default_author directly. Under the fixture the environment would now resolve to None on its own, so the patch could have been deleted as redundant. It was kept: the patch pins the value regardless of what the environment does, which keeps the test about the CLI's error path rather than about the environment. Criterion 3 asked exactly this and the reasoning is recorded in testing-conventions.md's decision points.

Turn 4 - proving the guard actually guards

A fixture nothing asserts against can be weakened or lose a variable, and everything stays green until the next run on a foreign machine - the same failure one level up. So tools/wiki_tools/tests/test_hermetic_env.py was written: the cleared variables, the empty HOME inside tmp_path, that git config user.name answers nothing outside a repo with a local identity, that tracing stays on and redirected.

Plus all three branches of config.default_author(), including the None branch that the hardening makes writable for the first time.

The counter-proof that this means something, run in-session:

default_author() with the ambient environment: 'Torben Nehmer'

Without the fixture, default_author() in a non-repository returns the developer's global git identity. With it, None. The new test would have failed before the change and passes after - which is the demonstration the Turn 2 baseline could not provide.

Turn 5 - verification across four environments

Three locally, all producing the identical count:

Environment	Before	After
Developer shell	green	702 passed
Deliberately poisoned: WIKI_AUTHOR, WIKI_TRACE=0, WIKITOOL_*, GIT_* set to junk	untested	702 passed
env -i, empty HOME, no git configuration	red (historically)	702 passed

Plus docs verify, instructions verify, lint --fail-on-error, instructions sync - all OK.

The poisoned run is the one the issue did not ask for. It tests the opposite direction from the empty-machine run: not "does the suite survive having nothing", but "does it survive having the wrong thing". A fixture that only unsets on an already-clean machine would pass the third row and fail the second.

Turn 6 - publish, and what CI revealed

publish ist freigegeben aktualisiere den issue gleich mit.

tools/wikitool publish first refused with Missing option '--message' (exit 2), then:

[main 31c9b81] test: autouse-Fixture härtet die Suite gegen stille Umgebungsabhängigkeiten (1.7.1, #8)
 8 files changed, 327 insertions(+), 3 deletions(-)
OK Published changes to origin/main.

Per SOUL.md, the tool's success line was not taken as proof. git ls-remote confirmed 31c9b81 on origin/main and tag v1.7.1 at the same commit.

CI run 79: green, all eight steps, 702 passed in 15.84s - the fourth environment, and the same number as the three local ones. Run 80 tagged the release. The dist export step reported 11 instructions and 5 skills against 14 and 6 in the dev tree, confirming that instructions/dev/testing-conventions.md stays out of the distribution.

The finding that retroactively justified the choice

Reading the run-79 log turned up something the issue could not have known:

Copying '/root/.gitconfig' to '/tmp/b93ea7a3-.../.gitconfig'
Temporarily overriding HOME='/tmp/b93ea7a3-...' before making global git config changes
[command]/usr/bin/git config --global --add safe.directory /workspace/torben/llm-wiki-test1

actions/checkout@v7 now creates a global git configuration inside the container itself, and the Tool-environment step writes safe.directory globally on top.

The job container is therefore no longer reliably "the machine without a global configuration". It had that property in run 52 by accident. The rejected fallback - a second hardened CI step - would have depended on that accident and would eventually have stopped catching anything, silently, with no red run to say so. The fixture depends on none of it.

This was recorded in the CI comment on .gitea/workflows/ci.yml's Tests step, so nobody adds the second job back for the reason it was once needed.

Turn 7 - the two open threads, filed rather than written down

Following capture-session.md step 4, both went to the tracker before the transcript:

• #22 (prio/3 size/XS) - lint.py:157 counts > lines, not quote blocks. The --fail-on-error run during verification flagged Source - Conversation - Auto Mode and Tool Choice Session 2026-08-31 for "4 quoted lines" when the page holds exactly one quote wrapped over four lines. The rule as written rewards overlong lines and punishes the repo's own wrap width. Not fixed in this session: different file, different rule, different change.
• #23 (prio/2 size/S) - nothing enforces that a newly introduced tool environment variable reaches _WIKITOOL_ENV. testing-conventions.md step 4 says to add it, but the premise of #8 was that a prose rule does not prevent this class of mistake. Noticed while writing that very step: the rule could be written down but not enforced.

Outcome

Version	1.7.1 (PATCH - no command changes behaviour)
Commit	31c9b81, tag v1.7.1
Files	8 changed, 327 insertions, 3 deletions
New	tools/wiki_tools/tests/test_hermetic_env.py, instructions/dev/testing-conventions.md
Changed	conftest.py, test_new_page.py (git init -q -b main), .gitea/workflows/ci.yml, stack-dev/SKILL.md, CHANGES.md, VERSION
Tests	702 passed in four environments: developer shell, poisoned, env -i, CI container
Issues	#8 closed with two comments; #22 and #23 opened
CI	Run 79 green (all 8 steps), run 80 tagged the release