torben/chemenu
1
1 Fork 0
Code Issues 40 Pull Requests Actions Packages Releases 41 Wiki Activity

feat: docs reference generates a human command reference under docs/reference/ from the command records, one page per group plus exit codes and gates; toc never lands inside another generated region (#188)
CI / verify (push) Successful in 6m11s
Details
CI / pwsh (push) Successful in 2m0s
Details
Release / release (push) Successful in 35s
Details

Browse Source 
Files changed:
- CHANGES.md
- README.md
- VERSION
- docs/reference/README.md
- docs/reference/commands/README.md
- docs/reference/commands/catalog-and-log.md
- docs/reference/commands/content-migrations.md
- docs/reference/commands/distribution-and-versioning.md
- docs/reference/commands/finding-and-checking.md
- docs/reference/commands/git.md
- docs/reference/commands/guideline-export.md
- docs/reference/commands/instance-health.md
- docs/reference/commands/links-and-citations.md
- docs/reference/commands/pages.md
- docs/reference/commands/provenance.md
- docs/reference/commands/raw-material-and-uploads.md
- docs/reference/commands/telemetry.md
- docs/reference/commands/types-instructions-and-docs.md
- docs/reference/commands/workshop-runs-and-session-budget.md
- docs/reference/exit-codes.md
- instructions/dev/doc-pull-through.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli.py
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/reference_doc.py
- tools/chemenu/tests/test_reference_doc.py
- tools/chemenu/toc.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-06 21:29:01 +02:00
1 parent 01c888d3cd
commit 53383e7e9b
29 files changed
+5079 -29
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/CONTRACT.md
+65 -3
View File
@@ -11,7 +11,7 @@ not here.
**Look up the command you need; do not read this file end to end.** Every
command carries one data record - name, synopsis, properties, exit status, and
so on - attached to its function in code (`tools/chemenu/cli_contract.py`).
Three views render from that one source:
Three views for an agent render from that one source:
```bash
tools/wikitool raw accept -h # the full record, for one command
@@ -24,6 +24,11 @@ The § Commands section below is `wikitool docs contract`'s output: an index
as `#### <path>` records, in the same order `wikitool -h` prints. Reading the
file end to end is for changing the CLI itself.
A fourth view is for a person: `wikitool docs reference` renders the same
records into one page per group under
[`docs/reference/commands/`](../docs/reference/commands/README.md), leaving out
ON FAILURE and NEVER, which address an agent.
<!-- wikitool:toc -->
## Contents
@@ -140,6 +145,7 @@ docs verify read idempotent budget:counted exit:0,1
docs toc write idempotent budget:counted exit:0 Create, refresh or remove the generated table-of-contents region.
docs prerequisites write idempotent budget:counted exit:0,1 Regenerate `INSTALL.md`'s prerequisites lists from `tools/prerequisites.txt`.
docs contract write idempotent budget:counted exit:0,1 Regenerate `tools/CONTRACT.md`'s `<!-- wikitool:commands -->` region.
docs reference write idempotent budget:counted exit:0,1 Regenerate the human command reference under `docs/reference/`.
eval sessions read idempotent budget:exempt exit:0 List the sessions that have a trace under `reports/telemetry/`.
eval score read idempotent budget:exempt exit:0,1 Score one traced session.
dist export write idempotent budget:counted exit:0,1 Write a contentless, distributable copy of this repo's machinery.
@@ -2510,6 +2516,8 @@ Check the docs that mirror the code.
- 1 A type-spec's own frontmatter fails its schema
- 1 A subtype template `types/<type>.<value>.md` has no type-spec with a `subtype_field:` beside it, names a value outside that field's enum, or carries frontmatter
- 1 A shipped `.md`/`.template` cites an issue number
- 1 A region of the human command reference under `docs/reference/` is stale
- 1 A page of the human command reference is missing or lacks its marker pair, or a page under `docs/reference/commands/` belongs to no group
- 1 A reference file's table-of-contents region is missing or stale
- 1 A file sits directly in `docs/`, or in a directory there that is not one of its five areas
- 1 A reference file's relative markdown link does not resolve to an existing file
@@ -2526,6 +2534,8 @@ Check the docs that mirror the code.
- A type-spec's own frontmatter fails its schema -> Fix the field, or add a matching line to `types/type-spec.schema.yaml` if the field is legitimately new
- A subtype template `types/<type>.<value>.md` has no type-spec with a `subtype_field:` beside it, names a value outside that field's enum, or carries frontmatter -> Rename the file to the type and value it was meant for, delete it, or remove its frontmatter block
- A shipped `.md`/`.template` cites an issue number -> Say what was decided instead of pointing at where, or move the pointer behind a `<!-- dist:strip-start/end -->` block
- A region of the human command reference under `docs/reference/` is stale -> Run `docs reference --apply`, then re-run
- A page of the human command reference is missing or lacks its marker pair, or a page under `docs/reference/commands/` belongs to no group -> Create the page with a title, a one-sentence head and the marker pair the message names (or add the pair), then run `docs reference --apply`; `git mv` or `git rm` a page left over from a renamed or removed group
- A reference file's table-of-contents region is missing or stale -> Run `docs toc --apply`, then re-run
- A file sits directly in `docs/`, or in a directory there that is not one of its five areas -> Move it (`git mv`) into the area its kind of documentation belongs to, and fix the links that pointed at the old path
- A reference file's relative markdown link does not resolve to an existing file -> Fix the `../` count or the target's name
@@ -2535,11 +2545,11 @@ Check the docs that mirror the code.
**NEVER**
- Never hand-write a table-of-contents region or the commands region - regenerate it.
- Never hand-write a table-of-contents region, the commands region or a reference region - regenerate it.
**NOTES**
- Checks the docs that mirror the code. The name is about documentation parity, not the `docs/` directory - it reads that only for its layout, and never requires one.
- Checks the docs that mirror the code. The name is about documentation parity, not the `docs/` directory - it reads that only for its layout and its generated reference pages, and never requires one.
- Commands: every command has a `cli_contract` record and is listed in `cli_contract.GROUPS`, in both directions; every command's non-hidden flags appear in its record's SYNOPSIS and vice versa; `tools/CONTRACT.md`'s generated `<!-- wikitool:commands -->` region matches what `docs contract` would write; no command's rendered `--help`/`-h` text cites an issue number.
- Collections: every directory under `kb/` has a `COLLECTION.md` and no directory outside it does; every collection declares `profile:` and a `required_by_stack:` that agrees with the stack's own list.
- Types: every type the stack lists (currently `source` and `project`) has a type-spec of that name whose schema requires the field the stack list names (`raw_files:`/`state:`); every file under `types/` declaring `type: types/type-spec.md` validates against `types/type-spec.schema.yaml`; every subtype template `types/<type>.<value>.md` (any `<value>` but `guidance`) sits beside a type-spec declaring `subtype_field:`, names a value that field's schema enum allows, and carries no frontmatter; no pre-migration `type: entity` blocks are left in the contracts.
@@ -2549,6 +2559,7 @@ Check the docs that mirror the code.
- Every reference file `docs toc` covers carries the current table-of-contents region for its own headings - missing and stale are one check.
- Every relative markdown link in one of those reference files resolves to an existing file. A target's `#anchor` suffix is stripped first, and code fences and inline code spans are masked before scanning, so link syntax shown as an example is not mistaken for a real reference.
- Every file under `docs/` sits in one of its areas - `tutorials/`, `how-to/`, `reference/`, `explanation/` or `dev/` - never directly in `docs/` or in another directory.
- If `docs/` exists: every generated region of the human command reference under `docs/reference/` matches what `docs reference` would write, every `cli_contract.GROUPS` entry has its page under `docs/reference/commands/`, and no page there belongs to no group.
- `INSTALL.md` carries one generated `<!-- wikitool:prerequisites -->` region per platform value of `tools/prerequisites.txt` (`prerequisites-<platform>` for a platform-specific one), each current; and the `<!-- setup-question: <key> -->` markers in `instructions/setup-instance.md` and `INSTALL.md` name the same set of keys, so a question the agent asks is never one the human guide leaves out, nor the reverse.
- `VERSION` parses and the newest versioned `CHANGES.md` entry names it. An entry that crosses the compatibility boundary from the last release carries `**Breaking Change:**`, has a migration document or `**Migration:** none required`, and its `**Migration:**` line names every required migration document targeting it.
- The newest versioned `CHANGES.md` entry is at most 32 000 bytes (UTF-8) as `version notes` prints it, and each of its `###` sections is exactly one paragraph of at most 1 000 bytes, heading line excluded. Older entries and a changelog with no versioned entry are not checked.
@@ -2558,6 +2569,7 @@ Check the docs that mirror the code.
- `wikitool docs toc` - regenerates tables of contents
- `wikitool docs contract` - regenerates the commands region
- `wikitool docs reference` - regenerates the human command reference
- `wikitool docs prerequisites` - regenerates `INSTALL.md`'s prerequisites lists
- `wikitool instructions verify` - the same kind of check for `instructions/`
@@ -2691,8 +2703,58 @@ Regenerate `tools/CONTRACT.md`'s `<!-- wikitool:commands -->` region.
**SEE ALSO**
- `tools/README.md` § Adding a command - how a command gets its record
- `wikitool docs reference` - the same records, rendered for a person under `docs/reference/`
- `wikitool docs verify` - checks the region is current
#### `docs reference`
Regenerate the human command reference under `docs/reference/`.
**SYNOPSIS**
- `wikitool docs reference [--apply]`
**PROPERTIES**
- effect: write
- idempotent: yes
- atomic: `--apply` rewrites each page in place, one at a time and idempotently, so a re-run after an interruption converges; the dry-run form is read-only
- budget: counted
- network: no
**EXAMPLES**
- `tools/wikitool docs reference`
- `tools/wikitool docs reference --apply`
**EXIT STATUS**
- 0 success
- 1 A reference page is missing, or lacks its region's marker pair
**ON FAILURE**
- A reference page is missing, or lacks its region's marker pair -> Not transient - create the page the message names with a title, a one-sentence head and the marker pair, or add the marker pair below its head, then retry
**NEVER**
- Never hand-edit a reference region - change the record in code and re-run this.
**NOTES**
- Rewrites the generated region of every page of the human command reference from the `cli_contract` records: one `docs/reference/commands/<group>.md` per `cli_contract.GROUPS` entry (`<!-- wikitool:reference-commands -->`), the overview in `docs/reference/commands/README.md` (`<!-- wikitool:reference-commands-index -->`), and the gates and exit-42 causes in `docs/reference/exit-codes.md` (`<!-- wikitool:reference-exit-codes -->`).
- Renders each command's summary, synopsis, options, examples, notes, properties in words, exit codes with their causes and see-also - but not the ON FAILURE reactions or NEVER, which address an agent; `tools/CONTRACT.md` keeps those.
- Refreshes the table of contents of each page it writes in the same step.
- Never creates a page or places a region: a page's title and one-sentence head are written by hand. A group's page is named after the group - lower case, every run of other characters one hyphen.
- Dry-run by default (lists the pages that would change); `--apply` writes.
- `docs verify` checks the result stays current.
**SEE ALSO**
- `wikitool docs contract` - the same records, rendered for an agent into `tools/CONTRACT.md`
- `wikitool docs verify` - checks every reference region is current
- `tools/README.md` § Adding a command - how a command gets its record
### Telemetry
#### `eval sessions`
tools/README.md
+6 -1
View File
@@ -164,7 +164,12 @@ bound at import time - `KB_DIR` and friends follow whatever `ROOT` currently is.
a comment next to the code). `docs verify` fails in both directions - a command with
no record and a `GROUPS` entry naming no real command are equally reported -
and also checks that every non-hidden flag appears in the record's SYNOPSIS.
Then regenerate the copy: `wikitool docs contract --apply`. Write the
Then regenerate both copies: `wikitool docs contract --apply` for
CONTRACT.md and `wikitool docs reference --apply` for the human reference
under `docs/reference/commands/`. A new group needs its page there first -
`docs/reference/commands/<group>.md` with a title, a one-sentence head and
the `<!-- wikitool:reference-commands -->` marker pair; the command never
creates one. Write the
record's prose without citing an issue number: `docs verify` also refuses
one in a command's rendered `--help` text (a docstring above its `\f`
marker, or an option's `help=`) and in any `.md`/`.template` `dist export`
tools/chemenu/cli.py
+24 -2
View File
@@ -193,7 +193,7 @@ app.command("sync")(git_publish.sync_command)
app.command("doctor")(doctor.doctor_command)
def _render_options_text(command, ctx) -> str:
def render_options_text(command, ctx) -> str:
"""Click's own Arguments/Options sections, plain-formatted, for splicing
into a `cli_contract` record's OPTIONS section. A fixed width (not the
real terminal's) is what keeps `wikitool <cmd> -h` byte-identical with
@@ -210,6 +210,28 @@ def _render_options_text(command, ctx) -> str:
return text
def options_texts(paths) -> dict[str, str]:
"""`render_options_text` for each command path in `paths`, outside a
help request - for the generated reference under `docs/reference/`.
Each context is built down the real chain from the root with
`resilient_parsing`, as Click does for `--help`, so the result carries the
root's `-h` alias and matches the OPTIONS section of `wikitool <path> -h`
byte for byte."""
import typer.main
root = typer.main.get_command(app)
texts: dict[str, str] = {}
for path in paths:
ctx = root.make_context("wikitool", [], resilient_parsing=True)
command = root
for name in path.split():
command = command.commands[name]
ctx = command.make_context(name, [], parent=ctx, resilient_parsing=True)
texts[path] = render_options_text(command, ctx)
return texts
def _render_root_help() -> str:
"""`wikitool -h`/`--help`/no-args: usage, the full index, and where the
per-command record lives - never Click's default subcommand listing,
@@ -258,7 +280,7 @@ try:
_original_format_help(self, ctx, formatter)
return
formatter.write(
cli_contract.render_text(record, options_text=_render_options_text(self, ctx))
cli_contract.render_text(record, options_text=render_options_text(self, ctx))
)
_typer_click_core.Command.format_help = _contract_format_help
tools/chemenu/cli_contract.py
+19 -12
View File
@@ -1,7 +1,9 @@
"""One data record per `wikitool` command - the single source three views are
"""One data record per `wikitool` command - the single source four views are
rendered from: `wikitool <cmd> -h` (the full record, plain text), the index
line (`wikitool -h` and the top of `tools/CONTRACT.md`), and the generated
`<!-- wikitool:commands -->` region of `tools/CONTRACT.md` itself.
line (`wikitool -h` and the top of `tools/CONTRACT.md`), the generated
`<!-- wikitool:commands -->` region of `tools/CONTRACT.md` itself, and the
human command reference under `docs/reference/` (`render_reference_*`), which
leaves out what is addressed to an agent - the ON FAILURE reactions and NEVER.
A record is attached to its command function, in that function's own module,
via the `@record(...)` decorator - never centralised, so the contract sits
@@ -274,7 +276,7 @@ GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = (
("Types, instructions and docs", (
"types list", "types describe",
"instructions sync", "instructions verify", "instructions list",
"docs verify", "docs toc", "docs prerequisites", "docs contract",
"docs verify", "docs toc", "docs prerequisites", "docs contract", "docs reference",
)),
("Telemetry", (
"eval sessions", "eval score",
@@ -359,17 +361,22 @@ def render_properties_lines(props: Properties) -> list[str]:
return lines
def render_exit_status_lines(rec: CommandRecord) -> list[str]:
"""`0 success` first, then one line per `Failure` in code order (stable
within a code). A command with gates but no explicit exit-42 cause gets
one generic 42 line naming them."""
lines = ["0 success"]
def exit_status_entries(rec: CommandRecord) -> list[tuple[int, str]]:
"""`(0, "success")` first, then one `(code, cause)` per `Failure` in code
order (stable within a code). A command with gates but no explicit exit-42
cause gets one generic 42 entry naming them."""
entries = [(0, "success")]
for failure in sorted(rec.failures, key=lambda f: f.code):
lines.append(f"{str(failure.code).ljust(4)} {_labelled(failure, failure.cause)}")
entries.append((failure.code, _labelled(failure, failure.cause)))
if rec.properties.gates and not any(f.code == 42 for f in rec.failures):
gate_list = ", ".join(rec.properties.gates)
lines.append(f"42 needs clearance - {gate_list} (see AGENTS.md § Gates)")
return lines
entries.append((42, f"needs clearance - {gate_list} (see AGENTS.md § Gates)"))
return entries
def render_exit_status_lines(rec: CommandRecord) -> list[str]:
"""EXIT STATUS as `<code> <cause>` lines - see `exit_status_entries`."""
return [f"{str(code).ljust(4)} {cause}" for code, cause in exit_status_entries(rec)]
def render_on_failure_lines(rec: CommandRecord) -> list[str]:
tools/chemenu/commands/docs_verify.py
+158 -3
View File
@@ -76,7 +76,7 @@ from typing import Optional
import typer
from chemenu import blocks, cli_contract, config, conventions, install_doc, kb_collections, markdown_code, toc, toolpaths, version as version_mod
from chemenu import blocks, cli_contract, config, conventions, install_doc, kb_collections, markdown_code, reference_doc, toc, toolpaths, version as version_mod
from chemenu.commands import dist_cmd
from chemenu.commands._util import fail, rel_path, success
@@ -336,6 +336,47 @@ def check_commands_region() -> list[str]:
return []
def check_reference_regions() -> list[str]:
"""Every generated region of the human command reference under
`docs/reference/` matches what `docs reference` would write, every
`GROUPS` entry has its page, and no page is left over from a group that
is gone.
A tree without `docs/` passes, as for `check_docs_layout`: `docs verify`
says what a present `docs/` must hold, not that one must exist. With
`docs/` present, every page is required - a deleted one is drift too.
"""
if not (config.ROOT / "docs").is_dir():
return []
issues = []
for region in reference_doc.regions():
path = config.ROOT / region.path
if not path.is_file():
issues.append(
f"{region.path} is missing - create it with a title, a one-sentence head and the "
f"marker pair `{blocks.open_marker(region.name)}` / "
f"`{blocks.close_marker(region.name)}`, then run `wikitool docs reference --apply`"
)
continue
existing = blocks.find(path.read_text(encoding="utf-8"), region.name)
if existing is None:
issues.append(
f"{region.path} has no `{blocks.open_marker(region.name)}` region - add the marker "
"pair below the page's head, then run `wikitool docs reference --apply`"
)
elif existing.strip("\n") != region.render().strip("\n"):
issues.append(
f"{region.path}'s `{blocks.open_marker(region.name)}` region is stale - run "
"`wikitool docs reference --apply`"
)
issues += [
f"{rel_path(page)} belongs to no group in cli_contract.GROUPS - `git mv` it to the page "
"of the group it was renamed to, or `git rm` it"
for page in reference_doc.orphan_pages()
]
return issues
# A `--flag` or `-f` token in a SYNOPSIS usage string. Excludes anything that
# is not immediately preceded/followed by a word character or hyphen, so
# `--set field=value` matches only `--set`, never `field` or `value`.
@@ -831,6 +872,10 @@ def command_table_free_readmes() -> list[Path]:
Built at call time rather than at import, so a test can point ROOT_README at
a fixture.
`docs/reference/commands/README.md` is deliberately not one of them: its
overview of every command is a generated region `check_reference_regions`
holds to the records - the checked kind of copy, not the drifting kind.
"""
return [ROOT_README, *(config.ROOT / relative for relative in STAGE_READMES)]
@@ -1208,7 +1253,8 @@ def check_breaking_change_for_boundary() -> list[str]:
),
notes=(
"Checks the docs that mirror the code. The name is about documentation parity, not the "
"`docs/` directory - it reads that only for its layout, and never requires one.",
"`docs/` directory - it reads that only for its layout and its generated reference "
"pages, and never requires one.",
"Commands: every command has a `cli_contract` record and is listed in "
"`cli_contract.GROUPS`, in both directions; every command's non-hidden flags appear in "
"its record's SYNOPSIS and vice versa; `tools/CONTRACT.md`'s generated "
@@ -1242,6 +1288,10 @@ def check_breaking_change_for_boundary() -> list[str]:
"Every file under `docs/` sits in one of its areas - `tutorials/`, `how-to/`, "
"`reference/`, `explanation/` or `dev/` - never directly in `docs/` or in another "
"directory.",
"If `docs/` exists: every generated region of the human command reference under "
"`docs/reference/` matches what `docs reference` would write, every `cli_contract.GROUPS` "
"entry has its page under `docs/reference/commands/`, and no page there belongs to no "
"group.",
"`INSTALL.md` carries one generated `<!-- wikitool:prerequisites -->` region per "
"platform value of `tools/prerequisites.txt` (`prerequisites-<platform>` for a "
"platform-specific one), each current; and the `<!-- setup-question: <key> -->` markers "
@@ -1294,6 +1344,17 @@ def check_breaking_change_for_boundary() -> list[str]:
reaction="Say what was decided instead of pointing at where, or move the pointer "
"behind a `<!-- dist:strip-start/end -->` block",
),
cli_contract.Failure(
cause="A region of the human command reference under `docs/reference/` is stale",
reaction="Run `docs reference --apply`, then re-run",
),
cli_contract.Failure(
cause="A page of the human command reference is missing or lacks its marker pair, or "
"a page under `docs/reference/commands/` belongs to no group",
reaction="Create the page with a title, a one-sentence head and the marker pair the "
"message names (or add the pair), then run `docs reference --apply`; `git mv` or "
"`git rm` a page left over from a renamed or removed group",
),
cli_contract.Failure(
cause="A reference file's table-of-contents region is missing or stale",
reaction="Run `docs toc --apply`, then re-run",
@@ -1330,11 +1391,13 @@ def check_breaking_change_for_boundary() -> list[str]:
"tools/wikitool docs verify",
),
never=(
"Never hand-write a table-of-contents region or the commands region - regenerate it.",
"Never hand-write a table-of-contents region, the commands region or a reference region "
"- regenerate it.",
),
see_also=(
"`wikitool docs toc` - regenerates tables of contents",
"`wikitool docs contract` - regenerates the commands region",
"`wikitool docs reference` - regenerates the human command reference",
"`wikitool docs prerequisites` - regenerates `INSTALL.md`'s prerequisites lists",
"`wikitool instructions verify` - the same kind of check for `instructions/`",
),
@@ -1359,6 +1422,7 @@ def verify():
+ check_changelog_budget()
+ check_no_issue_references()
+ check_docs_layout()
+ check_reference_regions()
+ check_toc_regions()
+ check_reference_targets()
+ install_doc.check_prerequisite_regions()
@@ -1378,6 +1442,7 @@ def verify():
f"{len(IGNORE_CANARIES)} ignore canaries clear, "
f"no issue references in {len(shipped_prose())} shipped document(s) or command help, "
f"every docs/ file in one of its {len(DOCS_AREAS)} areas, "
f"the command reference under {reference_doc.REFERENCE_DIR}/ current, "
f"tables of contents current and every link resolving on "
f"{len(toc.target_files())} reference file(s), "
f"{install_doc.INSTALL_DOC} in step with tools/prerequisites.txt and "
@@ -1569,6 +1634,8 @@ def prerequisites_command(
),
see_also=(
"`tools/README.md` § Adding a command - how a command gets its record",
"`wikitool docs reference` - the same records, rendered for a person under "
"`docs/reference/`",
"`wikitool docs verify` - checks the region is current",
),
))
@@ -1597,3 +1664,91 @@ def contract_command(
CLI_README.write_text(after, encoding="utf-8", newline="\n")
success(f"Regenerated the command region in {rel_path(CLI_README)}.")
@app.command("reference")
@cli_contract.record(cli_contract.CommandRecord(
path="docs reference",
summary="Regenerate the human command reference under `docs/reference/`.",
synopsis=(cli_contract.Variant(usage="docs reference [--apply]"),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.YES,
atomic="`--apply` rewrites each page in place, one at a time and idempotently, so a "
"re-run after an interruption converges; the dry-run form is read-only",
budget=cli_contract.Budget.COUNTED,
),
notes=(
"Rewrites the generated region of every page of the human command reference from the "
"`cli_contract` records: one `docs/reference/commands/<group>.md` per `cli_contract.GROUPS` "
"entry (`<!-- wikitool:reference-commands -->`), the overview in "
"`docs/reference/commands/README.md` (`<!-- wikitool:reference-commands-index -->`), and "
"the gates and exit-42 causes in `docs/reference/exit-codes.md` "
"(`<!-- wikitool:reference-exit-codes -->`).",
"Renders each command's summary, synopsis, options, examples, notes, properties in words, "
"exit codes with their causes and see-also - but not the ON FAILURE reactions or NEVER, "
"which address an agent; `tools/CONTRACT.md` keeps those.",
"Refreshes the table of contents of each page it writes in the same step.",
"Never creates a page or places a region: a page's title and one-sentence head are "
"written by hand. A group's page is named after the group - lower case, every run of "
"other characters one hyphen.",
"Dry-run by default (lists the pages that would change); `--apply` writes.",
"`docs verify` checks the result stays current.",
),
failures=(cli_contract.Failure(
cause="A reference page is missing, or lacks its region's marker pair",
reaction="Not transient - create the page the message names with a title, a one-sentence "
"head and the marker pair, or add the marker pair below its head, then retry",
),),
examples=(
"tools/wikitool docs reference",
"tools/wikitool docs reference --apply",
),
never=(
"Never hand-edit a reference region - change the record in code and re-run this.",
),
see_also=(
"`wikitool docs contract` - the same records, rendered for an agent into `tools/CONTRACT.md`",
"`wikitool docs verify` - checks every reference region is current",
"`tools/README.md` § Adding a command - how a command gets its record",
),
))
def reference_command(
apply: bool = typer.Option(False, "--apply", help="Write changes; default is dry-run (preview only)"),
):
"""Regenerate the human command reference under `docs/reference/`."""
missing = []
changed = []
for region in reference_doc.regions():
path = config.ROOT / region.path
if not path.is_file():
missing.append(f"{region.path} (create it with `{blocks.open_marker(region.name)}`)")
continue
before = path.read_text(encoding="utf-8")
after = reference_doc.refresh(before, region)
if after is None:
missing.append(f"{region.path} (add `{blocks.open_marker(region.name)}`)")
elif after != before:
changed.append((path, after))
if missing:
fail(
"These reference pages cannot be filled:\n"
+ "\n".join(f"- {entry}" for entry in missing)
+ "\nGive each a title, a one-sentence head and the marker pair "
"(opening and closing), then re-run."
)
if not changed:
success("Every reference page is already current.")
return
for path, after in changed:
typer.echo(rel_path(path))
if apply:
path.write_text(after, encoding="utf-8", newline="\n")
if apply:
success(f"Regenerated {len(changed)} reference page(s).")
else:
typer.echo(f"\n{len(changed)} page(s) would change. Re-run with --apply to write.")
tools/chemenu/reference_doc.py
+253
View File
@@ -0,0 +1,253 @@
"""The human command reference under `docs/reference/`, generated from the records.
`cli_contract` is the one source of what a command does; `tools/CONTRACT.md`
renders it for an agent, and this module renders it for a person. The two
differ on purpose: a record's ON FAILURE reactions and its NEVER lines tell an
agent what to do ("show the output verbatim and stop"), not a person what
happens, so they stay out here - only each failure's cause is kept, under the
exit code it produces. Properties are spelled out in words rather than as the
index's `budget:counted` tokens.
Every file here is half human, half generated, like `INSTALL.md`'s
prerequisites lists: a title and a one-sentence head someone wrote, then one
`<!-- wikitool:reference-... -->` region `docs reference --apply` rewrites. The
tool never creates a page or places a region - a missing one is reported - so
a group added to `cli_contract.GROUPS` gets its page, and its head, from a
person.
`regions()` is the one list both `docs reference` and `docs verify` walk; a
further generated reference page is one more entry in it.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Callable, NamedTuple, Optional
from chemenu import blocks, cli_contract, config, toc
REFERENCE_DIR = "docs/reference"
COMMANDS_DIR = f"{REFERENCE_DIR}/commands"
COMMANDS_INDEX = f"{COMMANDS_DIR}/README.md"
EXIT_CODES = f"{REFERENCE_DIR}/exit-codes.md"
COMMANDS_REGION = "reference-commands"
INDEX_REGION = "reference-commands-index"
EXIT_CODES_REGION = "reference-exit-codes"
_SLUG_RE = re.compile(r"[^a-z0-9]+")
class Region(NamedTuple):
"""One generated region: the file it lives in (root-relative), the region's
name, and what renders its content."""
path: str
name: str
render: Callable[[], str]
def group_slug(title: str) -> str:
"""The page name of a `GROUPS` entry: lower case, every run of anything
else than a letter or digit one hyphen (`Types, instructions and docs` ->
`types-instructions-and-docs`)."""
return _SLUG_RE.sub("-", title.lower()).strip("-")
def group_page(title: str) -> str:
return f"{COMMANDS_DIR}/{group_slug(title)}.md"
def command_anchor(path: str) -> str:
"""The anchor of a command's `## `<path>`` heading on its group page."""
return toc.anchor(f"`{path}`")
# ---------------------------------------------------------------------------
# Rendering
# ---------------------------------------------------------------------------
_BUDGET_WORDS = {
cli_contract.Budget.COUNTED: "yes",
cli_contract.Budget.EXEMPT: "no",
cli_contract.Budget.EXEMPT_WITHOUT_ARGS: "only when called with arguments",
}
def _yes_no(value: bool) -> str:
return "yes" if value else "no"
def properties_lines(props: cli_contract.Properties) -> list[str]:
"""A record's properties in words, one list item each."""
lines = [
f"- Changes files: {_yes_no(props.effect == cli_contract.Effect.WRITE)}",
f"- Idempotent: {_yes_no(props.idempotent == cli_contract.Idempotent.YES)}",
f"- Atomic: {props.atomic}",
f"- Counts toward the session call budget: {_BUDGET_WORDS[props.budget]}",
f"- Can reach the network: {_yes_no(props.network == cli_contract.Network.YES)}",
]
if props.gates:
lines.append(
f"- Gates: {', '.join(props.gates)} - see [exit codes and gates](../exit-codes.md)"
)
return lines
def render_command(rec: cli_contract.CommandRecord, options_text: str = "") -> str:
"""One command's section on its group page."""
lines = [f"## `{rec.path}`", "", rec.summary, ""]
lines += ["**Synopsis**", ""]
for variant in rec.synopsis:
note = f" - {variant.notes}" if variant.notes else ""
lines.append(f"- `wikitool {variant.usage}`{note}")
lines.append("")
if options_text.strip():
lines += ["**Options**", "", "```text", options_text.rstrip(), "```", ""]
if rec.examples:
lines += ["**Examples**", "", "```bash", *rec.examples, "```", ""]
lines += ["**Behaviour**", ""]
lines += [f"- {note}" for note in rec.notes]
lines.append("")
lines += ["**Properties**", "", *properties_lines(rec.properties), ""]
lines += ["**Exit codes**", ""]
lines += [f"- `{code}` - {cause}" for code, cause in cli_contract.exit_status_entries(rec)]
lines.append("")
if rec.see_also:
lines += ["**See also**", ""]
lines += [f"- {item}" for item in rec.see_also]
lines.append("")
return "\n".join(lines).rstrip() + "\n"
def render_group(
paths: tuple[str, ...],
records: dict[str, cli_contract.CommandRecord],
options: dict[str, str],
) -> str:
"""The region body of one group page: its commands, in `GROUPS` order."""
return "\n".join(
render_command(records[path], options.get(path, "")) for path in paths if path in records
).rstrip() + "\n"
def render_index(
records: dict[str, cli_contract.CommandRecord],
groups: cli_contract.GroupsType,
) -> str:
"""The region body of the commands README: every group, every command,
its one-sentence summary, linked to its section."""
lines: list[str] = []
for title, paths in groups:
page = f"{group_slug(title)}.md"
lines += [f"## {title}", "", f"Full records: [{page}]({page})", ""]
for path in paths:
if path in records:
lines.append(
f"- [`{path}`]({page}#{command_anchor(path)}) - {records[path].summary}"
)
lines.append("")
return "\n".join(lines).rstrip() + "\n"
def _command_link(path: str, groups: cli_contract.GroupsType) -> str:
title = cli_contract.group_of(path, groups)
if title is None:
return f"`{path}`"
return f"[`{path}`](commands/{group_slug(title)}.md#{command_anchor(path)})"
def render_exit_codes(
records: dict[str, cli_contract.CommandRecord],
groups: cli_contract.GroupsType,
) -> str:
"""The region body of `exit-codes.md`: which command declares which gate,
and every cause of an exit 42. What an exit code means is linked, not
repeated - `AGENTS.md` is where that rule lives."""
ordered = [path for path in cli_contract.grouped_paths(groups) if path in records]
lines = [
"What each exit code means, and how a caller reacts to it, is "
"[AGENTS.md § Tool error contract](../../AGENTS.md#tool-error-contract). What a gate is "
"and why it is enforced in code is [AGENTS.md § Gates](../../AGENTS.md#gates). This page "
"lists which commands declare a gate, and every cause for which a command exits 42.",
"",
"## Gates by command",
"",
]
gates: dict[str, list[str]] = {}
for path in ordered:
for gate in records[path].properties.gates:
gates.setdefault(gate, []).append(path)
for gate in sorted(gates):
lines += [f"### {gate}", ""]
lines += [f"- {_command_link(path, groups)}" for path in gates[gate]]
lines.append("")
lines += ["## Exit 42 causes", ""]
for path in ordered:
causes = [cause for code, cause in cli_contract.exit_status_entries(records[path]) if code == 42]
lines += [f"- {_command_link(path, groups)}: {cause}" for cause in causes]
return "\n".join(lines).rstrip() + "\n"
# ---------------------------------------------------------------------------
# The generated regions, and writing one
# ---------------------------------------------------------------------------
def regions(
records: Optional[dict[str, cli_contract.CommandRecord]] = None,
groups: Optional[cli_contract.GroupsType] = None,
options: Optional[dict[str, str]] = None,
) -> list[Region]:
"""Every generated region under `docs/reference/`, in a stable order.
`options` defaults to the real CLI's rendered OPTIONS blocks; a test
passing its own records passes its own options too (or `{}`)."""
records = records if records is not None else cli_contract.all_records()
groups = groups if groups is not None else cli_contract.GROUPS
if options is None:
from chemenu import cli # deferred: `cli` imports every command module, this one's caller included
options = cli.options_texts([p for p in cli_contract.grouped_paths(groups) if p in records])
found = [
Region(group_page(title), COMMANDS_REGION,
lambda paths=paths: render_group(paths, records, options))
for title, paths in groups
]
found.append(Region(COMMANDS_INDEX, INDEX_REGION, lambda: render_index(records, groups)))
found.append(Region(EXIT_CODES, EXIT_CODES_REGION, lambda: render_exit_codes(records, groups)))
return found
def refresh(text: str, region: Region) -> Optional[str]:
"""`text` with `region` regenerated and its table of contents brought in
line, or None when the file carries no such region to fill.
The TOC is refreshed in the same step because the generated `##` headings
are what it lists - without it, every regeneration that added a command
would leave `docs verify` reporting a stale TOC until `docs toc --apply`
ran as well."""
if blocks.find(text, region.name) is None:
return None
content = region.render().strip("\n")
wrapped = blocks.open_marker(region.name) + "\n" + content + "\n" + blocks.close_marker(region.name)
return toc.upsert(blocks.replace(text, region.name, wrapped))
def orphan_pages(groups: Optional[cli_contract.GroupsType] = None) -> list[Path]:
"""Pages under `docs/reference/commands/` no group in `groups` names."""
groups = groups if groups is not None else cli_contract.GROUPS
directory = config.ROOT / COMMANDS_DIR
if not directory.is_dir():
return []
expected = {Path(group_page(title)).name for title, _ in groups} | {Path(COMMANDS_INDEX).name}
return sorted(p for p in directory.glob("*.md") if p.name not in expected)
tools/chemenu/tests/test_reference_doc.py
+288
View File
@@ -0,0 +1,288 @@
import re
import pytest
from typer.testing import CliRunner
from chemenu import blocks, cli, cli_contract, config, reference_doc, toc
from chemenu.cli import app
from chemenu.commands import dist_cmd, docs_verify
runner = CliRunner()
# --- the real tree --------------------------------------------------------
def _group_pages() -> dict[str, str]:
return {
title: (config.ROOT / reference_doc.group_page(title)).read_text(encoding="utf-8")
for title, _ in cli_contract.GROUPS
}
def test_the_real_reference_is_current():
"""What a session that changed a record and forgot `docs reference --apply`
is caught by."""
assert docs_verify.check_reference_regions() == []
def test_every_group_has_one_page_and_every_command_one_section():
pages = _group_pages()
for path in cli_contract.all_records():
heading = f"## `{path}`\n"
hits = [title for title, text in pages.items() if text.count(heading) == 1]
assert len(hits) == 1, path
assert sum(text.count(heading) for text in pages.values()) == 1, path
def test_group_slugs_are_unique():
slugs = [reference_doc.group_slug(title) for title, _ in cli_contract.GROUPS]
assert len(slugs) == len(set(slugs))
def test_no_page_carries_an_agent_addressed_field():
"""ON FAILURE reactions and NEVER tell an agent what to do; the human
reference keeps only the cause of each failure."""
text = "\n".join(_group_pages().values())
text += (config.ROOT / reference_doc.EXIT_CODES).read_text(encoding="utf-8")
for rec in cli_contract.all_records().values():
for failure in rec.failures:
if failure.reaction:
assert failure.reaction not in text, (rec.path, failure.reaction)
for line in rec.never:
assert line not in text, (rec.path, line)
def test_options_match_the_options_section_of_h():
texts = cli.options_texts(sorted(cli_contract.all_records()))
for path, options in texts.items():
result = runner.invoke(app, [*path.split(" "), "-h"])
assert result.exit_code == 0, (path, result.output)
if options.strip():
assert f"OPTIONS\n{options.rstrip()}\n\n" in result.output, path
else:
assert "\nOPTIONS\n" not in result.output, path
def _section(markdown: str, heading: str) -> str:
start = markdown.index(f"## {heading}\n")
end = markdown.find("\n## ", start + 1)
return markdown[start:end if end != -1 else None]
def test_exit_codes_names_every_gate_and_links_rather_than_repeats_agents_md():
page = (config.ROOT / reference_doc.EXIT_CODES).read_text(encoding="utf-8")
for rec in cli_contract.all_records().values():
for gate in rec.properties.gates:
assert f"### {gate}\n" in page, gate
assert "(../../AGENTS.md#tool-error-contract)" in page
assert "(../../AGENTS.md#gates)" in page
agents = (config.ROOT / "AGENTS.md").read_text(encoding="utf-8")
for heading in ("Tool error contract", "Gates"):
for sentence in re.split(r"(?<=[.:])\s+", _section(agents, heading)):
sentence = sentence.strip()
if len(sentence) >= 30:
assert sentence not in page, sentence
def test_the_reference_pages_survive_dist_export_unchanged():
"""`dist export` strips `dist:strip` spans from every shipped text file; a
record quoting the marker pair verbatim would be cut in the exported page
and the fresh instance's `docs verify` would call it stale."""
plan = dist_cmd.build_plan()
for region in reference_doc.regions():
assert plan[region.path].content == (config.ROOT / region.path).read_text(encoding="utf-8")
# --- a fixture tree -------------------------------------------------------
GROUPS = (("Fixture things", ("frobnicate",)), ("Other", ("gate keep",)))
def _record(path: str, gates: tuple[str, ...] = (), summary: str = "Does a thing.") -> cli_contract.CommandRecord:
failures = [cli_contract.Failure(cause="It broke", reaction="Show the user and stop")]
if gates:
failures.append(cli_contract.Failure(cause="Needs a person", reaction="Wait", code=42))
return cli_contract.CommandRecord(
path=path,
summary=summary,
synopsis=(cli_contract.Variant(usage=f"{path} [--flag]"),),
properties=cli_contract.Properties(
effect=cli_contract.Effect.WRITE,
idempotent=cli_contract.Idempotent.NO,
atomic="Yes",
budget=cli_contract.Budget.EXEMPT_WITHOUT_ARGS,
gates=gates,
),
notes=("Does a thing, mechanically.",),
failures=tuple(failures),
examples=("tools/wikitool frobnicate --flag",),
never=("Never frobnicate twice.",),
see_also=("`wikitool other` - a neighbour",),
)
HEADS = {
"docs/reference/commands/fixture-things.md": "# Fixture things\n\nThings, by hand.\n",
"docs/reference/commands/other.md": "# Other\n\nThe other one.\n",
"docs/reference/commands/README.md": "# Command reference\n\nAll of it.\n",
"docs/reference/exit-codes.md": "# Exit codes and gates\n\nWhere it stops.\n",
}
REGION_OF = {
"docs/reference/commands/fixture-things.md": reference_doc.COMMANDS_REGION,
"docs/reference/commands/other.md": reference_doc.COMMANDS_REGION,
"docs/reference/commands/README.md": reference_doc.INDEX_REGION,
"docs/reference/exit-codes.md": reference_doc.EXIT_CODES_REGION,
}
@pytest.fixture
def tree(tmp_path, monkeypatch):
monkeypatch.setattr(config, "ROOT", tmp_path)
records = {"frobnicate": _record("frobnicate"), "gate keep": _record("gate keep", ("keep",))}
monkeypatch.setattr(cli_contract, "all_records", lambda: dict(records))
monkeypatch.setattr(cli_contract, "GROUPS", GROUPS)
monkeypatch.setattr(cli, "options_texts", lambda paths: {p: " --flag A flag." for p in paths})
for relative, head in HEADS.items():
path = tmp_path / relative
path.parent.mkdir(parents=True, exist_ok=True)
name = REGION_OF[relative]
path.write_text(
f"{head}\n{blocks.open_marker(name)}\n{blocks.close_marker(name)}\n", encoding="utf-8"
)
return tmp_path, records
def _snapshot(root) -> dict[str, str]:
return {
p.relative_to(root).as_posix(): p.read_text(encoding="utf-8")
for p in sorted(root.rglob("*.md"))
}
def test_a_fresh_tree_is_reported_stale_until_applied(tree):
root, _ = tree
issues = docs_verify.check_reference_regions()
assert len(issues) == len(HEADS)
assert all("stale" in issue and "docs reference --apply" in issue for issue in issues)
dry = runner.invoke(app, ["docs", "reference"])
assert dry.exit_code == 0, dry.output
assert "would change" in dry.output
assert docs_verify.check_reference_regions() != []
result = runner.invoke(app, ["docs", "reference", "--apply"])
assert result.exit_code == 0, result.output
assert docs_verify.check_reference_regions() == []
def test_apply_twice_changes_nothing_the_second_time(tree):
root, _ = tree
runner.invoke(app, ["docs", "reference", "--apply"])
first = _snapshot(root)
second_run = runner.invoke(app, ["docs", "reference", "--apply"])
assert second_run.exit_code == 0
assert "already current" in second_run.output
assert _snapshot(root) == first
def test_the_head_survives_apply_byte_for_byte(tree):
root, _ = tree
runner.invoke(app, ["docs", "reference", "--apply"])
for relative, head in HEADS.items():
text = (root / relative).read_text(encoding="utf-8")
assert text.startswith(head), relative
def test_a_changed_record_makes_verify_name_the_page_and_the_command(tree, monkeypatch):
root, records = tree
runner.invoke(app, ["docs", "reference", "--apply"])
records["frobnicate"] = _record("frobnicate", summary="Does a different thing.")
monkeypatch.setattr(cli_contract, "all_records", lambda: dict(records))
issues = docs_verify.check_reference_regions()
assert any(
"docs/reference/commands/fixture-things.md" in issue and "docs reference --apply" in issue
for issue in issues
)
def test_a_generated_page_leaves_out_reactions_and_never(tree):
root, _ = tree
runner.invoke(app, ["docs", "reference", "--apply"])
text = (root / "docs/reference/commands/fixture-things.md").read_text(encoding="utf-8")
assert "It broke" in text
assert "Show the user and stop" not in text
assert "Never frobnicate twice." not in text
assert "- Counts toward the session call budget: only when called with arguments" in text
assert "```text\n --flag A flag.\n```" in text
def test_a_long_page_gets_its_toc_outside_the_generated_region(tree, monkeypatch):
"""The first `##` of a group page is generated; a TOC inserted before it
would sit inside the region and be wiped by the next regeneration."""
root, records = tree
many = tuple(f"cmd{i}" for i in range(12))
for path in many:
records[path] = _record(path)
monkeypatch.setattr(cli_contract, "all_records", lambda: dict(records))
monkeypatch.setattr(cli_contract, "GROUPS", (("Fixture things", many), GROUPS[1]))
assert runner.invoke(app, ["docs", "reference", "--apply"]).exit_code == 0
page = root / "docs/reference/commands/fixture-things.md"
text = page.read_text(encoding="utf-8")
assert toc.needs_toc(text)
assert text.index(blocks.close_marker(toc.REGION_NAME)) < text.index(
blocks.open_marker(reference_doc.COMMANDS_REGION)
)
assert toc.upsert(text) == text
assert docs_verify.check_reference_regions() == []
def test_a_group_without_a_page_fails_and_nothing_is_created(tree, monkeypatch):
root, _ = tree
monkeypatch.setattr(cli_contract, "GROUPS", GROUPS + (("New group", ()),))
issues = docs_verify.check_reference_regions()
assert any("docs/reference/commands/new-group.md is missing" in issue for issue in issues)
result = runner.invoke(app, ["docs", "reference", "--apply"])
assert result.exit_code == 1
assert "docs/reference/commands/new-group.md" in result.output
assert not (root / "docs/reference/commands/new-group.md").exists()
def test_a_page_without_its_marker_pair_fails_and_is_left_alone(tree):
root, _ = tree
page = root / "docs/reference/exit-codes.md"
page.write_text("# Exit codes and gates\n\nNo region here.\n", encoding="utf-8")
issues = docs_verify.check_reference_regions()
assert any("exit-codes.md has no" in issue for issue in issues)
result = runner.invoke(app, ["docs", "reference", "--apply"])
assert result.exit_code == 1
assert page.read_text(encoding="utf-8") == "# Exit codes and gates\n\nNo region here.\n"
def test_a_page_without_a_group_fails(tree):
root, _ = tree
(root / "docs/reference/commands/gone.md").write_text("# Gone\n", encoding="utf-8")
issues = docs_verify.check_reference_regions()
assert any("gone.md belongs to no group" in issue for issue in issues)
def test_a_tree_without_docs_passes(tmp_path, monkeypatch):
monkeypatch.setattr(config, "ROOT", tmp_path)
assert docs_verify.check_reference_regions() == []
def test_exit_codes_lists_each_gate_and_its_exit_42_causes(tree):
root, _ = tree
runner.invoke(app, ["docs", "reference", "--apply"])
page = (root / "docs/reference/exit-codes.md").read_text(encoding="utf-8")
assert "### keep\n\n- [`gate keep`](commands/other.md#gate-keep)" in page
assert "- [`gate keep`](commands/other.md#gate-keep): Needs a person" in page
assert "frobnicate" not in page
def test_the_index_links_every_command_to_its_section(tree):
root, _ = tree
runner.invoke(app, ["docs", "reference", "--apply"])
index = (root / "docs/reference/commands/README.md").read_text(encoding="utf-8")
assert "- [`frobnicate`](fixture-things.md#frobnicate) - Does a thing." in index
assert "- [`gate keep`](other.md#gate-keep) - Does a thing." in index
tools/chemenu/toc.py
+22 -1
View File
@@ -276,6 +276,12 @@ def _slugify(text: str, seen: dict[str, int]) -> str:
return base if count == 0 else f"{base}-{count}"
def anchor(text: str) -> str:
"""The anchor a heading with this text gets, the first time it occurs on
a page - for a link written from outside that page."""
return _slugify(text, {})
def render(headings: list[Heading]) -> str:
"""The TOC region ready to place into a body, or `""` for no headings."""
if not headings:
@@ -289,12 +295,27 @@ def render(headings: list[Heading]) -> str:
return blocks.render(REGION_NAME, HEADING_TEXT, lines)
# Any generated region other than the TOC itself, markers included.
_OTHER_REGION_RE = re.compile(
r"<!-- wikitool:([a-z][a-z0-9-]*) -->.*?<!-- /wikitool:\1 -->", re.DOTALL
)
def _first_level2_offset(body: str) -> int | None:
"""Character offset of the first real (non-fenced) `## ` heading, or None."""
"""Character offset of the first real (non-fenced) `## ` heading, or None.
A heading inside another generated region moves the offset to that
region's opening marker: a page under `docs/reference/` can open with a
generated region whose first line is a `##` heading, and a TOC inserted
there would sit inside a region the next regeneration replaces wholesale.
"""
masked = markdown_code.strip_code_spans(body)
pos = 0
for masked_line in masked.split("\n"):
if masked_line.startswith("## "):
for region in _OTHER_REGION_RE.finditer(masked):
if region.start() < pos < region.end():
return region.start()
return pos
pos += len(masked_line) + 1
return None