torben/chemenu
1
1
Fork 0
Code Issues 26 Pull Requests Actions Packages Releases 32 Activity

stack: TOC-Pflicht fuer Referenzdateien ueber 100 Zeilen (docs toc); session-setup.md/gates.md nennen die tatsaechliche Budget-Ausnahmeliste (schliesst #73, #76)
CI / verify (push) Failing after 57s
Details
Release / release (push) Successful in 37s
Details

Browse Source 
Files changed:
- AGENTS.md
- CHANGES.md
- VERSION
- instructions/CONTRACT.md
- instructions/capture-session.md
- instructions/claude-code-model-selection.md
- instructions/dev/issue-tracking.md
- instructions/dev/testing-conventions.md
- instructions/dev/version-parts.md
- instructions/evolve-subtypes.md
- instructions/gates.md
- instructions/german-terminology.md
- instructions/ingest-large-tree.md
- instructions/kb-profiles.md
- instructions/link-taxonomy.md
- instructions/mcp-read-server.md
- instructions/migrate-corpus.md
- instructions/migrations/3.0.0-authoring-conventions.md
- instructions/migrations/4.0.0-link-taxonomy.md
- instructions/private-instance.md
- instructions/session-setup.md
- instructions/setup-instance.md
- kb/CONTRACT.md
- kb/CONVENTIONS.md
- kb/concepts/COLLECTION.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/commands/instructions_cmd.py
- tools/chemenu/tests/test_docs_verify.py
- tools/chemenu/tests/test_instructions_cmd.py
- tools/chemenu/tests/test_toc.py
- tools/chemenu/toc.py
- types/type-spec.md
This commit is contained in:
Torben Nehmer
2026-09-09 20:38:42 +02:00
parent a51d7a322f
commit 2c4c2b1c7c
34 changed files with 929 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/CONTRACT.md
+15 -1
View File
@@ -8,6 +8,19 @@ using these commands is mandatory; this file is the full reference. Changes to
`wikitool` itself are tracked in the repo root [`CHANGES.md`](../CHANGES.md),
not here.
<!-- wikitool:toc -->
## Contents
- [Setup (one time)](#setup-one-time)
- [Usage](#usage)
- [Commands](#commands)
- [Design notes](#design-notes)
- [Tests](#tests)
- [Error contracts](#error-contracts)
- [Maintenance schedule](#maintenance-schedule)
- [Future considerations (not implemented)](#future-considerations-not-implemented)
<!-- /wikitool:toc -->
## Setup (one time)
Full bootstrap for a fresh clone - including publishing the skills, which are not
@@ -72,6 +85,7 @@ tools/wikitool <command> --help
| `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout |
| `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only |
| `docs verify` | Check the docs that mirror the code: every CLI command documented here (and vice versa), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories), and no `.md`/`.template` file `dist export` would ship citing an issue number - the tracker exists only in the origin repo, so such a number in a distributed instance is a reference its reader can neither resolve nor recognise as unresolvable (a `<!-- dist:strip-start/end -->` region is exempt: it is already gone from the text the check reads, which is the export plan's, not the working tree's). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks |
| `docs toc [--apply]` | Create, refresh or remove the generated table-of-contents region (`<!-- wikitool:toc -->` ... `<!-- /wikitool:toc -->`, placed after the title and before the first `##`) on every reference file over 100 lines that `AGENTS.md`, a stage/collection contract, or the flat `instructions/**.md` form covers - the scope Anthropic's skill-authoring guidance names for a file previewed rather than read in full. Dry-run by default (prints which files would change); `--apply` writes. `docs verify` checks the result stays current the same way it checks every other generated-from-code copy |
| `eval sessions [--json]` | List the sessions that have a trace under `reports/telemetry/`, most recent first. Read-only and exempt from the Iteration Budget Gate |
| `eval score [--session <id>] [--json] [--markdown out.md] [--save] [--fail-on-error]` | Score one traced session: structural state from `lint`'s own checks (L1) plus trajectory rules over the trace (L2) - was a refused call repeated unchanged, was a gate flag passed without that gate having refused anything, did a publish of `kb/` pages go unlogged. Defaults to the current session. `--save` writes `reports/evals/<date>/<session>.{json,md}`. Read-only over `kb/` and exempt from the budget; see [../EVALS.md](../EVALS.md) |
| `dist export <target> [--dry-run] [--source-repo U] [--source-commit SHA] [--release-url U] [--update-url U]` | Write a contentless, distributable copy of this repo's machinery into an empty `<target>` directory: `AGENTS.md`/`README.md`/`EVALS.md` with any `<!-- dist:strip-start -->...<!-- dist:strip-end -->` region removed, `instructions/` (minus `instructions/dev/`), `types/` (the `root: kb` page type-specs and their schemas re-keyed as `.template`, the stack's own verbatim), `docs/` verbatim, `tools/` (no venv/caches), the `.github/hooks/`+`.vibe/` session-tracing config plus `.claude/settings.json`, `kb/CONTRACT.md` (no pages, no areas), empty `raw/{articles,documents,notes,assets}/` and the
@@ -251,4 +265,4 @@ Run by the LLM through the skills, on this cadence:
[INSTALL.md](../INSTALL.md) § "Eine Instanz aktualisieren". The `files` block
of `.wikitool-release.json` is the groundwork - it records what the machinery
looked like at install time, which is the only way to tell a file the
instance edited from one it merely received.
instance edited from one it merely received.
tools/chemenu/commands/docs_verify.py
+58 -4
View File
@@ -41,7 +41,7 @@ from typing import Optional
import typer
from chemenu import config, conventions, kb_collections, version as version_mod
from chemenu import config, conventions, kb_collections, toc, version as version_mod
from chemenu.commands import dist_cmd
from chemenu.commands._util import fail, rel_path, success
@@ -351,6 +351,26 @@ def check_legacy_type_blocks() -> list[str]:
return issues
def check_toc_regions() -> list[str]:
"""Every reference file over the line threshold carries a current TOC.
`toc.upsert` is idempotent (`toc.py`'s own docstring), so comparing its
output against the file on disk catches both a missing region and a
stale one - a heading added, renamed or reordered without re-running
`wikitool docs toc --apply` - in one check, the same way `docs verify`
checks every other generated-from-code copy.
"""
issues = []
for path in toc.target_files():
text = path.read_text(encoding="utf-8")
if toc.upsert(text) != text:
issues.append(
f"{rel_path(path)} needs a table-of-contents region refreshed - "
"run `wikitool docs toc --apply`"
)
return issues
def command_table_free_readmes() -> list[Path]:
"""Every README that must not carry a copy of the command table.
@@ -400,9 +420,12 @@ def check_readmes_have_no_command_table() -> list[str]:
# The pattern knows nothing about Gitea - no client, no URL, no issue state -
# which is what keeps `instructions/dev/issue-tracking.md` § "What no tool
# checks" intact. It is a character pattern over shipped text, and `wikitool`
# stays as ignorant of the board as it was. Markdown anchors are word
# characters (`](#gates)`), so a link never matches.
ISSUE_REFERENCE_RE = re.compile(r"#\d+")
# stays as ignorant of the board as it was. Markdown anchors are usually word
# characters (`](#gates)`), but a numbered step's TOC entry is not
# (`](#2-fix-the-fidelity-before-writing-a-word)`) - the lookbehind excludes
# exactly the `](#...` link-fragment shape, not `#\d+` generally, so a real
# citation immediately after other punctuation still matches.
ISSUE_REFERENCE_RE = re.compile(r"(?<!\]\()#\d+")
# What counts as shipped prose: Markdown, plus the `.template` files an instance
# renames into place during setup. `tools/**/*.py` is deliberately outside it.
@@ -675,6 +698,7 @@ def verify():
+ check_migration_for_boundary()
+ check_breaking_change_for_boundary()
+ check_no_issue_references()
+ check_toc_regions()
)
if issues:
@@ -686,6 +710,36 @@ def verify():
f"{len(STAGE_CONTRACTS)} stage contract(s) present, no legacy type blocks, "
f"{len(IGNORE_CANARIES)} ignore canaries clear, "
f"no issue references in {len(shipped_prose())} shipped document(s), "
f"tables of contents current on {len(toc.target_files())} reference file(s), "
f"{version_mod.CHANGES_FILENAME} documents version "
f"{(config.ROOT / version_mod.VERSION_FILENAME).read_text(encoding='utf-8').strip()}."
)
@app.command("toc")
def toc_command(
apply: bool = typer.Option(False, "--apply", help="Write changes; default is dry-run (preview only)"),
):
"""Create, refresh or remove the generated table-of-contents region on
every reference file `toc.target_files()` covers - AGENTS.md, the stage
and collection contracts, and every flat `instructions/**.md` file."""
changed = []
for path in toc.target_files():
before = path.read_text(encoding="utf-8")
after = toc.upsert(before)
if after != before:
changed.append((path, after))
if not changed:
success("Every table of contents is already current.")
return
for path, after in changed:
typer.echo(rel_path(path))
if apply:
path.write_text(after, encoding="utf-8")
if apply:
success(f"Refreshed the table of contents on {len(changed)} file(s).")
else:
typer.echo(f"\n{len(changed)} file(s) would change. Re-run with --apply to write.")
tools/chemenu/commands/instructions_cmd.py
+9 -1
View File
@@ -24,6 +24,7 @@ expected state of a clean checkout rather than a fault.
from __future__ import annotations
import filecmp
import re
import shutil
from pathlib import Path
@@ -300,7 +301,14 @@ def dev_only_forbidden_references(instructions_dir: Path | None = None) -> set[s
except OSError: # pragma: no cover - unreadable file
continue
for name in dev_names:
if name in text:
# Word-bounded, not a bare substring test: a generated TOC anchor
# like `#where-stack-development-happens` contains "stack-dev"
# as a raw substring without mentioning the skill at all. `\b`
# does not fire between "v" and "e" (both word characters), so
# "stack-development" is correctly not a match while a real
# mention (`` `stack-dev` ``, `instructions/dev/stack-dev/`) -
# bounded by punctuation on both sides - still is.
if re.search(rf"\b{re.escape(name)}\b", text):
referenced.add(name)
return referenced
tools/chemenu/tests/test_docs_verify.py
+43 -2
View File
@@ -353,6 +353,14 @@ def test_verify_raises_when_content_is_ignored(monkeypatch):
# --- issue references in shipped documents -----------------------------------
def test_every_reference_files_toc_is_current():
"""Forward direction, against the real tree: every file `toc.target_files()`
covers must already carry the region `wikitool docs toc --apply` would
write - this is what a session forgetting to re-run it after adding a
heading is caught by."""
assert docs_verify.check_toc_regions() == []
def test_no_shipped_document_cites_an_issue():
"""Forward direction, against the real tree: a `#42` in a file `dist export`
ships points at a board only the origin repo has, and the reader of a
@@ -374,8 +382,9 @@ def test_a_cited_issue_number_is_reported(monkeypatch):
def test_a_markdown_anchor_is_not_an_issue_reference(monkeypatch):
"""The reason this check can be a bare pattern at all: a link anchor and a
heading are word characters, so neither collides with `#<digits>`."""
"""An ordinary heading's slug is word characters, so it never collides
with `#<digits>` in the first place - this pins that the lookbehind
exclusion does not accidentally start matching it either."""
monkeypatch.setattr(
docs_verify,
"shipped_prose",
@@ -390,6 +399,38 @@ def test_a_markdown_anchor_is_not_an_issue_reference(monkeypatch):
assert docs_verify.check_no_issue_references() == []
def test_a_numbered_heading_anchor_is_not_an_issue_reference(monkeypatch):
"""A table-of-contents entry for a numbered step (`toc.py`) anchors on
the number itself - `#2-fix-the-fidelity-before-writing-a-word` - unlike
an ordinary heading's slug, which starts with a letter. The bare
`#\\d+` pattern would flag that as citing issue #2; the lookbehind
excludes exactly the `](#...` link-fragment shape it appears in."""
monkeypatch.setattr(
docs_verify,
"shipped_prose",
lambda: {
"instructions/example.md": (
"# Heading\n"
"- [2. Fix the fidelity before writing a word](#2-fix-the-fidelity-before-writing-a-word)\n"
)
},
)
assert docs_verify.check_no_issue_references() == []
def test_a_parenthesized_issue_number_is_still_reported(monkeypatch):
"""The lookbehind excludes `](#...`, not bare `(#...` - a real citation
written as a plain parenthetical must still be caught."""
monkeypatch.setattr(
docs_verify,
"shipped_prose",
lambda: {"instructions/example.md": "A rule (#66).\n"},
)
issues = docs_verify.check_no_issue_references()
assert len(issues) == 1
assert "#66" in issues[0]
def test_python_source_is_out_of_scope():
"""The scope decision, pinned: `tools/` ships as runtime machinery, and a
code comment addresses whoever edits that line - which only ever happens in
tools/chemenu/tests/test_instructions_cmd.py
+22
View File
@@ -384,6 +384,28 @@ def test_dev_only_instruction_referenced_from_claude_md_is_reported(layer):
instructions_cmd.verify()
def test_a_superstring_mention_is_not_a_reference(layer):
"""`stack-dev` is a real skill name, but a heading "Where stack
development happens" renders a TOC anchor `#where-stack-development-happens`
that contains "stack-dev" as a raw substring without mentioning the skill
at all. The check has to be word-bounded, not `name in text`, or every
coincidental superstring becomes a false boundary violation."""
dev_dir = layer / "instructions" / "dev"
dev_dir.mkdir()
dev_skill = dev_dir / "stack-dev"
dev_skill.mkdir()
(dev_skill / "SKILL.md").write_text(
"---\nname: stack-dev\ndescription: x\n---\n\n# Stack Dev\n", encoding="utf-8"
)
(layer / "AGENTS.md").write_text(
"# AGENTS\n\nSee gates.md.\n"
"- [Where stack development happens](#where-stack-development-happens)\n",
encoding="utf-8",
)
instructions_cmd.sync(force=False)
assert "stack-dev" not in instructions_cmd.dev_only_forbidden_references()
def test_a_readme_mention_alone_does_not_keep_an_instruction_alive(layer):
"""README.md is 'never by an agent as instruction' (AGENTS.md's file-naming
table), so a mention there documents an instruction without deploying it.
tools/chemenu/tests/test_toc.py
+168
View File
@@ -0,0 +1,168 @@
from chemenu import blocks, toc
def _body(n_lines: int, sections: int) -> str:
"""A synthetic `# Title` + intro + `sections` `##` headings, `n_lines` long."""
lines = ["# Title", "", "Intro paragraph."]
per_section = max(1, (n_lines - len(lines) - sections) // sections) if sections else 0
for i in range(sections):
lines += ["", f"## Section {i}", ""]
lines += [f"Body line {j}." for j in range(per_section)]
while len(lines) < n_lines:
lines.append("Padding.")
return "\n".join(lines[:n_lines]) + "\n"
def test_needs_toc_is_false_at_the_threshold():
body = _body(toc.THRESHOLD, 3)
assert len(body.splitlines()) == toc.THRESHOLD
assert not toc.needs_toc(body)
def test_needs_toc_is_true_just_over_the_threshold():
body = _body(toc.THRESHOLD + 1, 3)
assert toc.needs_toc(body)
def test_needs_toc_measures_without_an_existing_region():
"""A file that shrank back under the threshold but still carries a stale
TOC region from when it was much longer must not count the region's own
lines toward keeping itself around - `needs_toc` strips it before
measuring. A 150-entry region is long enough on its own to push the
file over the threshold if it were (wrongly) counted."""
huge_region = toc.render([toc.Heading(level=2, text=f"Section {i}") for i in range(150)])
shrunk_with_stale_region = (
"# Title\n\nIntro paragraph.\n\n" + huge_region + "\n\n## Only Section\n\nOne line.\n"
)
assert len(shrunk_with_stale_region.splitlines()) > toc.THRESHOLD
assert not toc.needs_toc(shrunk_with_stale_region)
def test_iter_headings_skips_fenced_code():
body = "\n".join(
[
"# Title",
"",
"## Real Section",
"",
"```bash",
"# Not a heading",
"## Also not a heading",
"```",
"",
"### Real Subsection",
]
)
headings = toc.iter_headings(body)
assert [h.text for h in headings] == ["Real Section", "Real Subsection"]
assert [h.level for h in headings] == [2, 3]
def test_iter_headings_preserves_inline_code_in_heading_text():
"""Masking blanks inline code spans for *detection*, but the text has to
come back from the unmasked line or `` `instructions/dev/` `` would
render as blank space instead of its real content."""
body = "# Title\n\n## `instructions/dev/`\n"
headings = toc.iter_headings(body)
assert headings == [toc.Heading(level=2, text="`instructions/dev/`")]
def test_render_produces_nested_marker_region():
headings = [
toc.Heading(level=2, text="First"),
toc.Heading(level=3, text="Nested"),
toc.Heading(level=2, text="Second"),
]
region = toc.render(headings)
assert region.startswith(blocks.open_marker(toc.REGION_NAME))
assert region.endswith(blocks.close_marker(toc.REGION_NAME))
assert f"## {toc.HEADING_TEXT}" in region
assert "- [First](#first)" in region
assert " - [Nested](#nested)" in region
assert "- [Second](#second)" in region
def test_render_deduplicates_repeated_slugs():
headings = [toc.Heading(level=2, text="Steps"), toc.Heading(level=2, text="Steps")]
region = toc.render(headings)
assert "(#steps)" in region
assert "(#steps-1)" in region
def test_render_is_empty_for_no_headings():
assert toc.render([]) == ""
def test_upsert_inserts_before_the_first_level2_heading():
body = _body(toc.THRESHOLD + 20, 4)
result = toc.upsert(body)
assert result != body
marker_pos = result.index(blocks.open_marker(toc.REGION_NAME))
first_heading_pos = result.index("## Section 0")
assert marker_pos < first_heading_pos
# Title and intro paragraph still precede the region.
assert result.index("# Title") < marker_pos
assert result.index("Intro paragraph.") < marker_pos
def test_upsert_is_idempotent():
body = _body(toc.THRESHOLD + 30, 5)
once = toc.upsert(body)
twice = toc.upsert(once)
assert once == twice
def test_upsert_is_idempotent_without_a_trailing_newline():
"""A file missing its final newline (tools/CONTRACT.md and
types/type-spec.md both did, in the real repo) must not make the
"no region yet" and "region already exists" paths disagree on
round two."""
body = _body(toc.THRESHOLD + 20, 4).rstrip("\n")
once = toc.upsert(body)
twice = toc.upsert(once)
assert once == twice
def test_upsert_refreshes_a_stale_region_in_place():
body = _body(toc.THRESHOLD + 20, 3)
stale = toc.upsert(body)
# Rename a heading without touching the (now stale) TOC region.
changed = stale.replace("## Section 1", "## Renamed Section", 1)
refreshed = toc.upsert(changed)
assert "[Renamed Section](#renamed-section)" in refreshed
assert "[Section 1]" not in refreshed
def test_upsert_removes_the_region_once_the_file_shrinks():
body = _body(toc.THRESHOLD + 20, 3)
with_region = toc.upsert(body)
assert blocks.find(with_region, toc.REGION_NAME) is not None
shrunk = "# Title\n\nIntro paragraph.\n\n## Only Section\n\nOne line.\n"
result = toc.upsert(shrunk)
assert blocks.find(result, toc.REGION_NAME) is None
def test_stale_regions_is_false_right_after_upsert():
body = _body(toc.THRESHOLD + 20, 3)
once = toc.upsert(body)
assert toc.stale_regions(once) is False
def test_target_files_matches_the_documented_scope():
"""Integration check against the real repo: the scope is AGENTS.md, every
stage contract, kb/CONVENTIONS.md, every COLLECTION.md, and the flat
`instructions/**.md` form - never a `SKILL.md`."""
from chemenu import config
files = toc.target_files()
relatives = {f.relative_to(config.ROOT).as_posix() for f in files}
assert "AGENTS.md" in relatives
assert "kb/CONTRACT.md" in relatives
assert "kb/CONVENTIONS.md" in relatives
assert "types/type-spec.md" in relatives
assert "instructions/CONTRACT.md" in relatives
assert "instructions/dev/version-parts.md" in relatives # flat, still instructions/**.md
assert not any(rel.endswith("SKILL.md") for rel in relatives)
assert not any(rel.startswith("types/") and rel != "types/type-spec.md" for rel in relatives)
tools/chemenu/toc.py
+223
View File
@@ -0,0 +1,223 @@
"""Generate the table-of-contents region for reference files over 100 lines.
Anthropic's skill-authoring guidance: "For reference files longer than 100
lines, include a table of contents at the top. This ensures Claude can see
the full scope of available information even when previewing with partial
reads." (`codex-skill-creator/SKILL.md:221-222` names the same threshold as
the mitigation for the exact preview mechanic `instructions/CONTRACT.md`
§ "Reference depth" already treats as real for a repo-wide contract reached
at a second hop.)
A hand-maintained TOC is the next drift source the moment a heading changes -
AGENTS.md invariant 1 ("never hand-edit generated files") applies here the
same way it applies to a page's links/footnotes region. So this is a third
generated region beside `xref`'s and `cite`'s, reusing `blocks`' marker
convention (`<!-- wikitool:toc -->` ... `<!-- /wikitool:toc -->`) without
joining `blocks.BLOCKS`: that tuple feeds `xref`, `cite` and the KB-page
`unbalanced_markers` lint check, and none of the target files here (AGENTS.md,
the stage/collection contracts, the flat `instructions/**.md` files) is a
`kb/` page. `version.py`'s `bumps` region inside `CHANGES.md` set this same
precedent first.
**Scope is computed, never a hand-picked list** - the same principle that
governs `wikitool` itself. `target_files()` walks the file-naming categories
AGENTS.md's own table calls agent-loaded reference material: `AGENTS.md`,
every stage contract, `kb/CONVENTIONS.md`, every `kb/*/COLLECTION.md`, and
every flat `instructions/**.md` file (a `SKILL.md` is excluded - it is loaded
whole by the harness, not previewed at a second hop; `instructions/CONTRACT.md`
§ "How much reasoning a step may carry" already treats a checklist read once
as the table of contents it replaced). Human docs (`README.md`, `CHANGES.md`,
`EVALS.md`, `INSTALL.md`, `tools/README.md`) are excluded too: the file-naming
table says they are "Never loaded by an agent as instruction," so the preview
mechanic this exists to mitigate does not apply to them.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import NamedTuple
from chemenu import blocks, config, kb_collections, markdown_code
REGION_NAME = "toc"
HEADING_TEXT = "Contents"
# The line threshold Anthropic's own guidance names. Measured on the body
# with any existing TOC region stripped out first, so inserting or updating
# the region can never be what pushes a file over the line.
THRESHOLD = 100
# Mirrors docs_verify.STAGE_CONTRACTS - kept as a separate constant rather than
# imported, because `docs_verify` importing `toc` (for the new check) would
# make a `toc` -> `docs_verify` import a cycle. Both lists are the seven stage
# contracts; `docs_verify.STAGE_CONTRACTS`'s own docstring is the place that
# explains what they are.
_STAGE_CONTRACTS = (
"raw/CONTRACT.md",
"kb/CONTRACT.md",
"types/type-spec.md",
"reports/CONTRACT.md",
"work/CONTRACT.md",
"tools/CONTRACT.md",
"instructions/CONTRACT.md",
)
_HEADING_RE = re.compile(r"^(#{2,3})[ \t]+(.+?)[ \t]*$")
_SLUG_DROP_RE = re.compile(r"[^\w\s-]", re.UNICODE)
_SLUG_SPACE_RE = re.compile(r"\s+")
class Heading(NamedTuple):
level: int # 2 or 3
text: str
def target_files() -> list[Path]:
"""Every file the TOC region applies to, root-relative, sorted.
Walking `instructions/` picks up `instructions/dev/` and
`instructions/migrations/` along with the rest: both are still the flat
`instructions/<name>.md` form (`instructions/CONTRACT.md` says the `dev/`
split is orthogonal to Linked/Manual, not a different file shape), so
there is no separate rule for them to fall out of - and no hand-picking
for a future file under either to be missed.
"""
files: list[Path] = [config.ROOT / "AGENTS.md"]
files += [config.ROOT / rel for rel in _STAGE_CONTRACTS]
files.append(config.ROOT / "kb" / "CONVENTIONS.md")
files += sorted(
collection / kb_collections.CONTRACT_NAME
for collection in kb_collections.iter_kb_collections()
)
instructions_dir = config.ROOT / "instructions"
if instructions_dir.is_dir():
files += sorted(
path
for path in instructions_dir.rglob("*.md")
if path.name != "SKILL.md"
)
return sorted({f for f in files if f.is_file()})
def body_without_region(text: str) -> str:
"""`text` with any existing TOC region removed - what the 100-line
threshold is measured against, so a stale TOC never counts toward keeping
itself around.
Trailing newlines are canonicalized to exactly one (`blocks.strip`'s own
removal path already does this when a region existed; a file with no
region yet is normalized here too), which is what keeps `upsert`
idempotent - without it, a file missing its final newline would insert
its first TOC one way and every later re-run one byte shorter, since the
"remove an existing region" and "there was never one" paths would
otherwise start from differently-terminated strings.
"""
stripped = blocks.strip(text, REGION_NAME)
return stripped.rstrip("\n") + "\n" if stripped else stripped
def needs_toc(text: str) -> bool:
return len(body_without_region(text).splitlines()) > THRESHOLD
def iter_headings(body: str) -> list[Heading]:
"""Every `##`/`###` heading in document order, code-aware.
Detection runs on `markdown_code.strip_code_spans(body)`, which blanks
fenced blocks and inline code spans but preserves line structure - a
template line inside a fence (`# {Type name}`, a `raw/CONTRACT.md`
example path prefixed `#`) no longer starts with a real `#` marker once
masked, so it is not mistaken for a section. The heading *text* is read
back from the corresponding *unmasked* line, not the masked one: masking
also blanks inline code spans, which would otherwise turn a heading like
`` ## `instructions/dev/` `` into blank spaces instead of its real text.
"""
original = body.split("\n")
masked = markdown_code.strip_code_spans(body).split("\n")
headings: list[Heading] = []
for masked_line, original_line in zip(masked, original):
if not masked_line.startswith("#"):
continue
match = _HEADING_RE.match(masked_line)
if not match:
continue
# The masked line only still starts with `#...` for a genuine heading
# (fenced lines are blanked to spaces); recover the text from the
# unmasked line at the same position.
original_match = _HEADING_RE.match(original_line)
if not original_match:
continue
headings.append(Heading(level=len(match.group(1)), text=original_match.group(2).strip()))
return headings
def _slugify(text: str, seen: dict[str, int]) -> str:
"""A GitHub/Gitea-style anchor slug, de-duplicated like both renderers do
for a repeated heading (`foo`, `foo-1`, `foo-2`, ...)."""
base = _SLUG_DROP_RE.sub("", text.strip().lower())
base = _SLUG_SPACE_RE.sub("-", base).strip("-")
count = seen.get(base, 0)
seen[base] = count + 1
return base if count == 0 else f"{base}-{count}"
def render(headings: list[Heading]) -> str:
"""The TOC region ready to place into a body, or `""` for no headings."""
if not headings:
return ""
seen: dict[str, int] = {}
lines = []
for heading in headings:
indent = " " if heading.level == 3 else ""
slug = _slugify(heading.text, seen)
lines.append(f"{indent}- [{heading.text}](#{slug})")
return blocks.render(REGION_NAME, HEADING_TEXT, lines)
def _first_level2_offset(body: str) -> int | None:
"""Character offset of the first real (non-fenced) `## ` heading, or None."""
masked = markdown_code.strip_code_spans(body)
pos = 0
for masked_line in masked.split("\n"):
if masked_line.startswith("## "):
return pos
pos += len(masked_line) + 1
return None
def upsert(text: str) -> str:
"""`text` with its TOC region created or refreshed, or removed if the
file (measured without any existing region) is at or under the threshold.
Always strips first, then always reinserts fresh - never patches an
existing region in place. `blocks.replace`'s in-place path is right for
the links/footnotes regions it was built for (trailing material, appended
at the end when absent) but wrong here twice over: a TOC has to sit "at
the top" per the guidance this exists to satisfy, not at the end: and
`blocks.strip`'s removal collapses the blank lines around a matched
region to a single newline, which would make patching in place drift
from a fresh insertion's spacing on every second run. Always rebuilding
from the stripped body sidesteps both: the same deterministic
`rstrip("\\n") + "\\n\\n"` join runs whether or not a region existed
before, so a second `upsert` on its own output reproduces that output
exactly.
"""
body = body_without_region(text)
if not needs_toc(text):
return body
region = render(iter_headings(body))
if not region:
return body
offset = _first_level2_offset(body)
if offset is None:
return body.rstrip("\n") + "\n\n" + region + "\n"
head, tail = body[:offset], body[offset:]
return head.rstrip("\n") + "\n\n" + region + "\n\n" + tail
def stale_regions(text: str) -> bool:
"""Whether `text`'s current TOC region (if any) no longer matches what
`upsert` would write - drift `docs verify` can catch mechanically."""
return upsert(text) != text