fix: types/type-spec.schema.yaml enforced against real type-spec frontmatter, doc-pull-through.md docs/-page count corrected (closes #105)
CI / verify (push) Failing after 45s
Release / release (push) Successful in 38s

Files changed:
- CHANGES.md
- VERSION
- instructions/dev/doc-pull-through.md
- tools/CONTRACT.md
- tools/chemenu/commands/docs_verify.py
- tools/chemenu/tests/test_docs_verify.py
- types/type-spec.md
- types/type-spec.schema.yaml
This commit is contained in:
2026-09-15 19:45:33 +02:00
parent 6eb3f84256
commit 55f65c1ab1
8 changed files with 161 additions and 7 deletions
+2 -2
View File
@@ -170,7 +170,7 @@ tools/wikitool <command> --help
| `instructions sync [--force]` | Publish every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and delete published skills whose source is gone. Both targets are gitignored, so a fresh clone runs this once - see `instructions/bootstrap.md`. Re-running is also how a drifted copy is repaired: the source always wins. `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it) |
| `instructions verify` | Check the instruction layer: flat instructions validate against `types/instruction.schema.yaml`, each `SKILL.md` carries the frontmatter its harness reads, no `SKILL.md` carries a relative markdown link (`sync` copies it to a different depth than the source, so a `SKILL.md` references a target as a repo-root-relative plain path instead - see [instructions/CONTRACT.md](../instructions/CONTRACT.md) § "A skill's outbound reference is a plain path, not a link"), every published copy is byte-identical to its source, no instruction is left that nothing references, and nothing under `instructions/dev/` is referenced from outside it (a `<!-- dist:strip-start/end -->` block is exempt - see [instructions/CONTRACT.md](../instructions/CONTRACT.md)). Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout |
| `instructions list [--json]` | List the flat instructions with their descriptions. This is how the layer is discovered; `search` deliberately covers `kb/` only |
| `docs verify` | Check the docs that mirror the code: every CLI command documented in this file's own § Commands table and, separately, in its § Error contracts table (both directions, checked per table, so a row dropped from one is not hidden by the same name surviving in the other, and only a name's presence in a row is checked, never the rest of that row's text), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, no pre-migration `type: entity` blocks left in the contracts, the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories), and no `.md`/`.template` file `dist export` would ship citing an issue number - the tracker exists only in the origin repo, so such a number in a distributed instance is a reference its reader can neither resolve nor recognise as unresolvable (a `<!-- dist:strip-start/end -->` region is exempt: it is already gone from the text the check reads, which is the export plan's, not the working tree's), every reference file `docs toc` covers carrying the current table-of-contents region for its own headings - missing and stale are one check, because the generator is idempotent - and every relative markdown link in one of those same reference files resolving to a file that actually exists (a target's `#anchor` suffix is stripped first; code fences and inline code spans are masked before scanning, so a passage showing link syntax as an example is not mistaken for a real reference). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks |
| `docs verify` | Check the docs that mirror the code: every CLI command documented in this file's own § Commands table and, separately, in its § Error contracts table (both directions, checked per table, so a row dropped from one is not hidden by the same name surviving in the other, and only a name's presence in a row is checked, never the rest of that row's text), every directory under `kb/` has a `COLLECTION.md` and no directory outside it does, every collection declaring `profile:` and a `required_by_stack:` that agrees with the stack's own list, `kb/CONVENTIONS.md` naming all three tool-owned section headings if it exists at all, every stage contract present, every file under `types/` declaring `type: types/type-spec.md` validating against `types/type-spec.schema.yaml`, no pre-migration `type: entity` blocks left in the contracts, the `.gitignore` canaries clear in both directions (nothing ignored under `raw/`/`kb/`, `incoming/` ignored, everything ignored under `reports/` and the published skill directories), and no `.md`/`.template` file `dist export` would ship citing an issue number - the tracker exists only in the origin repo, so such a number in a distributed instance is a reference its reader can neither resolve nor recognise as unresolvable (a `<!-- dist:strip-start/end -->` region is exempt: it is already gone from the text the check reads, which is the export plan's, not the working tree's), every reference file `docs toc` covers carrying the current table-of-contents region for its own headings - missing and stale are one check, because the generator is idempotent - and every relative markdown link in one of those same reference files resolving to a file that actually exists (a target's `#anchor` suffix is stripped first; code fences and inline code spans are masked before scanning, so a passage showing link syntax as an example is not mistaken for a real reference). The name is about documentation parity, not about the `docs/` directory - it neither reads nor requires one, the same way `kb/` predates the collection it now checks |
| `docs toc [--apply]` | Create, refresh or remove the generated table-of-contents region (`<!-- wikitool:toc -->` ... `<!-- /wikitool:toc -->`, placed after the title and before the first `##`) on every reference file over 100 lines, in the scope Anthropic's skill-authoring guidance names for a file previewed rather than read in full: `AGENTS.md`, every stage contract, `kb/CONVENTIONS.md`, every `kb/*/COLLECTION.md`, every flat `instructions/**.md` file, every `types/*.md` type-spec, and every `docs/` page. Computed from those categories rather than listed, so a file added later is in scope without a code change. `SKILL.md` is the one exception, and the same guidance is why: it places a skill body on the loading level that is read whole when the skill triggers, and aims its own TOC advice at the bundled reference files a skill points *at*. Human docs (`README.md`, `CHANGES.md`, `EVALS.md`, `INSTALL.md`, `tools/README.md`) are out of scope because AGENTS.md § File naming says no agent loads them as instruction. Dry-run by default (prints which files would change); `--apply` writes. `docs verify` checks the result stays current the same way it checks every other generated-from-code copy |
### Telemetry
@@ -381,7 +381,7 @@ is atomic, and whether a retry is safe.
| `instructions sync` | No skills found under `instructions/`, or a target directory is not a published skill (no `SKILL.md`) and `--force` was not passed | No - one directory copy per skill per target (`.agents/skills/`, `.claude/skills/`); each copy is idempotent, so a re-run converges even after a partial failure | Check whether the flagged target holds anything worth keeping, then re-run with `--force` if not; otherwise fix the named cause and retry |
| `instructions verify` | Nothing found under `instructions/` at all, a malformed instruction or `SKILL.md`, a `SKILL.md` carrying a relative markdown link, a published copy that drifted from its source, an instruction nothing references (or, for `manual: true`, one that IS linked from AGENTS.md, CLAUDE.md, or a skill and so risks running implicitly), or something under `instructions/dev/` referenced from outside it and outside a `dist:strip` block | Read-only | Fix the flagged file, then re-run. For a relative link in a `SKILL.md`, rewrite it as a repo-root-relative plain path instead. For drift, re-run `sync` instead of hand-editing the published copy - the source under `instructions/` always wins |
| `instructions list` | Never fails - an empty `instructions/` prints "No instructions found." | Read-only | Safe to retry freely |
| `docs verify` | A command, contract, or type-form mismatch was found, a shipped `.md`/`.template` cites an issue number, a reference file's table-of-contents region is missing or stale, or a reference file's relative markdown link does not resolve to an existing file | Read-only | Fix the documentation it names, then re-run. For an issue reference: say what was decided instead of pointing at where, or move the pointer behind a `<!-- dist:strip-start/end -->` block. For a table of contents: run `docs toc --apply` - never hand-write the region. For a dead link: fix the `../` count or the target's name |
| `docs verify` | A command, contract, or type-form mismatch was found, a type-spec's own frontmatter fails its schema, a shipped `.md`/`.template` cites an issue number, a reference file's table-of-contents region is missing or stale, or a reference file's relative markdown link does not resolve to an existing file | Read-only | Fix the documentation it names, then re-run. For a type-spec's own frontmatter: fix the field, or add a matching line to `types/type-spec.schema.yaml` if the field is legitimately new. For an issue reference: say what was decided instead of pointing at where, or move the pointer behind a `<!-- dist:strip-start/end -->` block. For a table of contents: run `docs toc --apply` - never hand-write the region. For a dead link: fix the `../` count or the target's name |
| `docs toc` | Never fails on content: a file with no `##` heading, or one at or under the threshold, is simply left without a region | `--apply` rewrites each named file in place, one at a time and idempotently, so a re-run after an interruption converges rather than doubling a region; the dry-run form is read-only | Nothing to fix - re-run with `--apply` to write what the dry run listed. If `docs verify` still reports a stale region afterwards, the file's `##` headings changed in between; run it again |
### Telemetry
+45 -1
View File
@@ -44,6 +44,15 @@ assume - is `instructions verify`'s job, not this one, since that module
already owns the Skill/Instruction split (`skill_dirs()` vs
`instruction_files()`).
An eighth checks the type layer against its own schema: every file under
`types/` declaring `type: types/type-spec.md` must validate against
`types/type-spec.schema.yaml`. Before this check existed the schema had
already drifted behind two fields real type-specs carry (`root:`,
`capture_fields:`) while `additionalProperties: false` sat there describing a
contract nothing enforced - the exact "checked or absent" failure this file's
opening paragraph names, just one level up, for the schema that describes the
type layer instead of a copy the type layer's code produces (Gitea #105).
Everything here is a hard oracle: a set comparison or a regex, no judgment.
Content quality of the contracts themselves stays with the LLM.
"""
@@ -422,6 +431,37 @@ def check_stack_required_types() -> list[str]:
return issues
def check_type_spec_frontmatter() -> list[str]:
"""Every type-spec's own frontmatter must validate against
`types/type-spec.schema.yaml` - the schema that describes the type layer
gets the same enforcement any other type's schema gets (Gitea #105).
Before this check nothing ever called `validate_frontmatter` against a
type-spec's own frontmatter, so the schema had quietly drifted behind two
fields real type-specs actually carry (`root:`, `capture_fields:`)
without anything failing - `additionalProperties: false` described a
contract that bound nothing. `resolver.list_type_specs()` already reads
every file's frontmatter once for `wikitool types list`; reusing it here
means this check costs no second parse pass.
"""
from chemenu.type_resolver import resolver
issues: list[str] = []
for type_path, frontmatter in resolver.list_type_specs():
try:
resolver.validate_frontmatter(
frontmatter, "types/type-spec.md", source_file=config.ROOT / type_path
)
except ValueError as exc:
# `validate_frontmatter`'s own message names the type path it
# validated *against* (always `types/type-spec.md` here, since
# every type-spec is validated against the same schema) rather
# than the specific file that failed - prefix that file's own
# path so two failures in one run stay distinguishable.
issues.append(f"{type_path}: {exc}")
return issues
def check_legacy_type_blocks() -> list[str]:
issues = []
guarded = [
@@ -857,11 +897,12 @@ def check_breaking_change_for_boundary() -> list[str]:
@app.command("verify")
def verify():
"""Check the CLI/README command tables, contract presence, type-form drift, ignore rules, version/changelog agreement, issue references, and link targets in shipped documents."""
"""Check the CLI/README command tables, contract presence, type-form drift, every type-spec's frontmatter against its own schema, ignore rules, version/changelog agreement, issue references, and link targets in shipped documents."""
issues = (
check_cli_readme()
+ check_readmes_have_no_command_table()
+ check_collection_contracts()
+ check_type_spec_frontmatter()
+ check_legacy_type_blocks()
+ check_ignored_content()
+ check_version_changelog()
@@ -875,10 +916,13 @@ def verify():
if issues:
fail("Documentation issues found:\n" + "\n".join(f"- {i}" for i in issues))
from chemenu.type_resolver import resolver
success(
f"Docs verified: {len(registered_commands())} command(s) documented, "
f"{len(kb_collections.iter_kb_collections())} collection(s) and "
f"{len(STAGE_CONTRACTS)} stage contract(s) present, no legacy type blocks, "
f"{len(resolver.list_type_specs())} type-spec(s) validating against their own schema, "
f"{len(IGNORE_CANARIES)} ignore canaries clear, "
f"no issue references in {len(shipped_prose())} shipped document(s), "
f"tables of contents current and every link resolving on "
+47 -1
View File
@@ -1,8 +1,9 @@
import pytest
import typer
from chemenu import config
from chemenu import config, type_resolver
from chemenu.commands import dist_cmd, docs_verify
from chemenu.type_resolver import TypeResolver
def test_every_registered_command_is_documented():
@@ -289,6 +290,51 @@ def test_an_absent_listed_doc_is_skipped_not_reported(tmp_path, monkeypatch):
assert docs_verify.check_readmes_have_no_command_table() == []
def test_this_repos_type_specs_validate_against_their_own_schema():
"""Regression guard for Gitea #105: types/type-spec.schema.yaml declared
`additionalProperties: false` while real type-specs already carried
`root:` and `capture_fields:`, and nothing ever validated a type-spec's
own frontmatter against it - so the mismatch shipped silently."""
assert docs_verify.check_type_spec_frontmatter() == []
def test_an_unknown_type_spec_field_is_reported(tmp_path, monkeypatch):
"""Once the schema is enforced, a type-spec frontmatter field its own
schema does not know about must fail loudly rather than validating
silently - the other direction of the #105 regression guard above."""
types_dir = tmp_path / "types"
types_dir.mkdir()
(types_dir / "type-spec.md").write_text(
"---\n"
"type: types/type-spec.md\n"
"name: type-spec\n"
"description: Authoring and validation contract for type specs\n"
"schema: types/type-spec.schema.yaml\n"
"---\n",
encoding="utf-8",
)
(types_dir / "type-spec.schema.yaml").write_text(
(config.ROOT / "types" / "type-spec.schema.yaml").read_text(encoding="utf-8"),
encoding="utf-8",
)
(types_dir / "widget.md").write_text(
"---\n"
"type: types/type-spec.md\n"
"name: widget\n"
"description: A type-spec with a field its own schema does not know\n"
"schema: null\n"
"not_a_real_field: true\n"
"---\n",
encoding="utf-8",
)
monkeypatch.setattr(config, "ROOT", tmp_path)
monkeypatch.setattr(config, "TYPES_DIR", types_dir)
monkeypatch.setattr(type_resolver, "resolver", TypeResolver(repo_root=tmp_path))
issues = docs_verify.check_type_spec_frontmatter()
assert any("widget.md" in issue and "not_a_real_field" in issue for issue in issues)
def test_legacy_type_blocks_are_absent():
assert docs_verify.check_legacy_type_blocks() == []