instructions: the agent speaks the KB language in chat - AGENTS.md § Conversation language, and instructions sync compiles the language tag into every published SKILL.md (#116); 10.0.0-beta.9

Files changed: - AGENTS.md - CHANGES.md - VERSION - docs/explanation/language-boundaries.md - docs/reference/commands/types-instructions-and-docs.md - instructions/CONTRACT.md - instructions/dev/stack-build/SKILL.md - instructions/dev/stack-close/SKILL.md - instructions/dev/stack-dev/SKILL.md - kb/CONVENTIONS.md - kb/CONVENTIONS.md.template - tools/CONTRACT.md - tools/chemenu/commands/doctor.py - tools/chemenu/commands/instructions_cmd.py - tools/chemenu/tests/test_doctor.py - tools/chemenu/tests/test_instructions_cmd.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
2026-10-12 06:57:51 +02:00
1 parent c06bde2e95
commit 0f0a5799fd
@@ -2542,6 +2542,7 @@ Publish every `instructions/<name>/SKILL.md` into the harness skill directories.
 **NOTES**

 - Publishes every `instructions/<name>/SKILL.md` into `.agents/skills/` and `.claude/skills/` as **copies**, and deletes published skills whose source is gone.
+- Each published `SKILL.md` is its source plus one generated line directly under the frontmatter: the `language:` of `kb/CONVENTIONS.md` and a pointer to `AGENTS.md` § Conversation language, so the agent has the value in context when the task begins. The frontmatter and every other file are copied unchanged. With no language declared, the `SKILL.md` is copied unchanged too, and the success message says so.
 - Both targets are gitignored, so a fresh clone runs this once.
 - Re-running repairs a drifted copy: the source always wins.
 - `--force` is required only to replace a target directory that is not a published skill at all (no `SKILL.md` in it).
@@ -2577,7 +2578,7 @@ Check the instruction layer.
 - 0    success
 - 1    Nothing found under `instructions/` at all, or a malformed instruction or `SKILL.md`
 - 1    A `SKILL.md` carries a relative markdown link
- 1    A published copy drifted from its source
+- 1    A published copy drifted from its source - including a language line that belongs to an earlier `language:` (the message names it as such)
 - 1    An instruction nothing references, or a `manual: true` one that IS linked from AGENTS.md, CLAUDE.md or a skill and so risks running implicitly
 - 1    Something under `instructions/dev/` is referenced from outside it and outside a `dist:strip` block

@@ -2585,7 +2586,7 @@ Check the instruction layer.

 - Nothing found under `instructions/` at all, or a malformed instruction or `SKILL.md` -> Fix the flagged file, then re-run
 - A `SKILL.md` carries a relative markdown link -> Rewrite it as a repo-root-relative plain path, then re-run
- A published copy drifted from its source -> Re-run `instructions sync` - the source under `instructions/` always wins
+- A published copy drifted from its source - including a language line that belongs to an earlier `language:` (the message names it as such) -> Re-run `instructions sync` - the source under `instructions/` always wins
 - An instruction nothing references, or a `manual: true` one that IS linked from AGENTS.md, CLAUDE.md or a skill and so risks running implicitly -> Link it from where it is used, or drop the link to a manual one, then re-run
 - Something under `instructions/dev/` is referenced from outside it and outside a `dist:strip` block -> Remove the reference or wrap it in a `dist:strip` block, then re-run

@@ -2597,7 +2598,7 @@ Check the instruction layer.

 - Flat instructions validate against `types/instruction.schema.yaml`, and each `SKILL.md` carries the frontmatter its harness reads.
 - No `SKILL.md` carries a relative markdown link: `sync` copies it to a different depth than the source, so a `SKILL.md` references a target as a repo-root-relative plain path instead (`instructions/CONTRACT.md` § "A skill's outbound reference is a plain path, not a link").
- Every published copy is byte-identical to its source. Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout.
+- Every published copy is what `sync` would write from its source: byte-identical, except for the generated language line in `SKILL.md` that the current `language:` of `kb/CONVENTIONS.md` calls for. Missing *every* copy is reported as "run sync", not as drift - that is a clean checkout.
 - No instruction is left that nothing references; one marked `manual: true` must instead not be linked from AGENTS.md, CLAUDE.md or a skill.
 - Nothing under `instructions/dev/` is referenced from outside it; a `<!-- dist:strip-start/end -->` block is exempt (`instructions/CONTRACT.md`).
 - Read-only.
@@ -288,7 +288,8 @@ def check_skills() -> Check:
        )
    if drifted:
        return Check(
-            "skills", "FAIL", f"{len(drifted)} published copy/copies drifted from source",
+            "skills", "FAIL",
+            f"{len(drifted)} published copy/copies drifted from source, first: {drifted[0]}",
            "Run `tools/wikitool instructions sync`",
        )
    return Check("skills", "OK", f"{expected} published copy/copies match their source")
@@ -12,9 +12,17 @@ nothing else). This reverses an earlier design that used relative symlinks. The
 symlink argument was that a link cannot go stale; the counter-arguments that
 won are that symlinks are unreliable on Windows checkouts and do not survive
 being archived or copied, and that this repo is meant to stay reproducible
-elsewhere. The price of a copy is drift, so `verify` checks every copy byte for
-byte against its source - the copy is derived truth, and derived truth is either
-checked or absent.
+elsewhere. The price of a copy is drift, so `verify` checks every copy against
+what `sync` would write from its source - the copy is derived truth, and derived
+truth is either checked or absent.
+
+The copy is byte-identical to its source except for one generated block: when
+`kb/CONVENTIONS.md` declares a `language:`, `sync` inserts a line carrying that
+tag directly under each `SKILL.md`'s frontmatter, so the value that decides what
+language the agent speaks is in context when the task begins instead of one file
+read away (AGENTS.md § Conversation language holds the rule; the block holds
+only the value and a pointer to it). `render_skill_md` is the one place that text
+is made, and both `sync` and `drift` use it.

 Both target directories are gitignored. A fresh clone has no skills until `sync`
 runs; `instructions/bootstrap.md` is the procedure, and `verify` says so rather
@@ -44,7 +52,7 @@ from pathlib import Path
 import typer
 import yaml

-from chemenu import cli_contract, config, markdown_code
+from chemenu import cli_contract, config, conventions, markdown_code
 from chemenu.commands import dist_cmd, docs_verify
 from chemenu.commands._util import fail, rel_path, success
 from chemenu.type_resolver import resolver
@@ -142,7 +150,66 @@ def _read_frontmatter(path: Path) -> tuple[dict | None, str]:
    return frontmatter, ""


-def _copy_skill(source: Path, target: Path, force: bool) -> None:
+# The closing line of the frontmatter, matched line-wise: `_read_frontmatter`'s
+# `find("---")` would stop at a `---` inside a value.
+_FRONTMATTER_RE = re.compile(r"\A---[ \t]*\r?\n.*?\r?\n---[ \t]*(\r?\n|\Z)", re.DOTALL)
+_LANGUAGE_BLOCK_TAG_RE = re.compile(r"^> \*\*Conversation language: `([^`\n]+)`\*\*", re.MULTILINE)
+
+
+def language_block(language: str) -> str:
+    """The generated line `sync` writes under a published skill's frontmatter.
+
+    A tag, never a language name: a name table would be the compiler's opinion
+    about languages, which `conventions.py` deliberately has none of."""
+    return (
+        f"> **Conversation language: `{language}`** - see `AGENTS.md` § Conversation language "
+        "(value from `kb/CONVENTIONS.md` `language:`, written by `wikitool instructions sync`)."
+    )
+
+
+def render_skill_md(source_text: str, language: str | None) -> str:
+    """What the published `SKILL.md` holds for this source and language: the
+    source itself when no language is declared (or it has no closed
+    frontmatter to insert under), else the source with `language_block` as the
+    first paragraph after the frontmatter."""
+    if not language:
+        return source_text
+    match = _FRONTMATTER_RE.match(source_text)
+    if match is None:
+        return source_text
+    end = match.end()
+    eol = "\r\n" if match.group(0).endswith("\r\n") else "\n"
+    head = source_text[:end] if match.group(1) else source_text + eol
+    return f"{head}{eol}{language_block(language)}{eol}{source_text[end:]}"
+
+
+def _write_skill_md(target: Path, language: str | None) -> None:
+    path = target / SKILL_FILE
+    source_text = path.read_bytes().decode("utf-8")
+    rendered = render_skill_md(source_text, language)
+    if rendered != source_text:
+        path.write_bytes(rendered.encode("utf-8"))
+
+
+def _skill_md_drift(source: Path, target: Path, language: str | None) -> str | None:
+    """How a published `SKILL.md` differs from what `sync` would write, or None.
+
+    A difference that is only the language block says so: `language:` changed
+    since the last `sync`, which is not a hand edit and should not read as one."""
+    source_text = (source / SKILL_FILE).read_bytes().decode("utf-8")
+    target_text = (target / SKILL_FILE).read_bytes().decode("utf-8")
+    if target_text == render_skill_md(source_text, language):
+        return None
+    found = _LANGUAGE_BLOCK_TAG_RE.search(target_text)
+    stated = f"conventions say `{language}`" if language else "conventions declare no language"
+    if found and target_text == render_skill_md(source_text, found.group(1)):
+        return f"language block is for `{found.group(1)}`, {stated}"
+    if language and target_text == source_text:
+        return f"language block missing, {stated}"
+    return f"differs in {SKILL_FILE}"
+
+
+def _copy_skill(source: Path, target: Path, force: bool, language: str | None) -> None:
    if target.is_symlink():
        # Left over from the symlink-based mirror this replaced.
        target.unlink()
@@ -155,6 +222,7 @@ def _copy_skill(source: Path, target: Path, force: bool) -> None:
            )
        shutil.rmtree(target)
    shutil.copytree(source, target)
+    _write_skill_md(target, language)


 def _looks_like_a_published_skill(target: Path) -> bool:
@@ -176,13 +244,16 @@ def drift(source: Path, target: Path) -> str | None:
    if target.is_symlink():
        return "is a symlink, not a copy"
    comparison = filecmp.dircmp(str(source), str(target))
-    if comparison.diff_files:
-        return f"differs in {', '.join(sorted(comparison.diff_files))}"
+    # SKILL.md is the one file that is not a byte copy; it is judged against its
+    # rendering below.
+    differing = sorted(set(comparison.diff_files) - {SKILL_FILE})
+    if differing:
+        return f"differs in {', '.join(differing)}"
    if comparison.left_only:
        return f"missing {', '.join(sorted(comparison.left_only))}"
    if comparison.right_only:
        return f"has extra {', '.join(sorted(comparison.right_only))}"
-    return None
+    return _skill_md_drift(source, target, conventions.language())


 def _reference_haystacks(
@@ -381,6 +452,11 @@ def check_skill_reference_paths() -> list[str]:
    notes=(
        "Publishes every `instructions/<name>/SKILL.md` into `.agents/skills/` and "
        "`.claude/skills/` as **copies**, and deletes published skills whose source is gone.",
+        "Each published `SKILL.md` is its source plus one generated line directly under the "
+        "frontmatter: the `language:` of `kb/CONVENTIONS.md` and a pointer to `AGENTS.md` § "
+        "Conversation language, so the agent has the value in context when the task begins. The "
+        "frontmatter and every other file are copied unchanged. With no language declared, the "
+        "`SKILL.md` is copied unchanged too, and the success message says so.",
        "Both targets are gitignored, so a fresh clone runs this once.",
        "Re-running repairs a drifted copy: the source always wins.",
        "`--force` is required only to replace a target directory that is not a published "
@@ -426,12 +502,13 @@ def sync(
        fail(f"No skills found under {rel_path(config.INSTRUCTIONS_DIR)}.")

    names = {source.name for source in sources}
+    language = conventions.language()
    published = []
    removed = []
    for target_root in target_dirs():
        target_root.mkdir(parents=True, exist_ok=True)
        for source in sources:
-            _copy_skill(source, target_root / source.name, force)
+            _copy_skill(source, target_root / source.name, force, language)
        # A skill that no longer exists must not keep being offered.
        for stale in sorted(target_root.iterdir()):
            if stale.name not in names:
@@ -443,7 +520,8 @@ def sync(
        published.append(rel_path(target_root))

    suffix = f", removed {len(removed)} stale" if removed else ""
-    success(f"Published {len(sources)} skill(s) to {' and '.join(published)}{suffix}")
+    stamped = f"conversation language `{language}`" if language else "no language declared"
+    success(f"Published {len(sources)} skill(s) to {' and '.join(published)} ({stamped}){suffix}")


@app.command("verify")
@@ -464,8 +542,10 @@ def sync(
        "than the source, so a `SKILL.md` references a target as a repo-root-relative plain "
        "path instead (`instructions/CONTRACT.md` § \"A skill's outbound reference is a plain "
        "path, not a link\").",
-        "Every published copy is byte-identical to its source. Missing *every* copy is "
-        "reported as \"run sync\", not as drift - that is a clean checkout.",
+        "Every published copy is what `sync` would write from its source: byte-identical, except "
+        "for the generated language line in `SKILL.md` that the current `language:` of "
+        "`kb/CONVENTIONS.md` calls for. Missing *every* copy is reported as \"run sync\", not as "
+        "drift - that is a clean checkout.",
        "No instruction is left that nothing references; one marked `manual: true` must "
        "instead not be linked from AGENTS.md, CLAUDE.md or a skill.",
        "Nothing under `instructions/dev/` is referenced from outside it; a "
@@ -483,7 +563,8 @@ def sync(
            reaction="Rewrite it as a repo-root-relative plain path, then re-run",
        ),
        cli_contract.Failure(
-            cause="A published copy drifted from its source",
+            cause="A published copy drifted from its source - including a language line that "
+            "belongs to an earlier `language:` (the message names it as such)",
            reaction="Re-run `instructions sync` - the source under `instructions/` always wins",
        ),
        cli_contract.Failure(
@@ -510,7 +591,7 @@ def sync(
    ),
 ))
 def verify():
-    """Check instructions/ against its type, that no skill carries a relative markdown link, and every published copy against its source."""
+    """Check instructions/ against its type, that no skill carries a relative markdown link, and every published copy against what sync would write from its source."""
    sources = skill_dirs()
    instructions = instruction_files()
    if not sources and not instructions:
@@ -461,6 +461,22 @@ def test_unpublished_skills_fail(instance):
    assert _status(checks, "skills") == "FAIL"


+def test_a_changed_conversation_language_fails_skills_and_names_the_block(instance):
+    (config.KB_DIR / conventions.CONVENTIONS_FILENAME).write_text(
+        "---\nlanguage: de\nprofile: none\nsections:\n  links: Relationships\n"
+        "  footnotes: Footnotes\n---\n\n# conventions\n",
+        encoding="utf-8",
+    )
+    conventions.reset_cache()
+    checks = doctor.run_doctor()
+    assert _status(checks, "skills") == "FAIL"
+    assert "2 published copy/copies drifted" in _detail(checks, "skills")
+    assert "language block is for `en`, conventions say `de`" in _detail(checks, "skills")
+
+    instructions_cmd.sync(force=False)
+    assert _status(doctor.run_doctor(), "skills") == "OK"
+
+
 def test_no_author_fails(instance, monkeypatch):
    monkeypatch.delenv("WIKI_AUTHOR", raising=False)
    monkeypatch.setattr(config, "default_author", lambda: None)
@@ -10,7 +10,7 @@ from pathlib import Path
 import pytest
 import typer

-from chemenu import config
+from chemenu import config, conventions
 from chemenu.tests.conftest import use_shipped_type_specs
 from chemenu.commands import instructions_cmd

@@ -230,6 +230,184 @@ def test_drift_detects_missing_edited_and_extra(layer):
    assert "extra" in instructions_cmd.drift(source, target)


+# --- conversation language block --------------------------------------------
+
+
+def _declare_language(root: Path, tag: str | None) -> None:
+    path = root / "kb" / "CONVENTIONS.md"
+    if tag is None:
+        path.unlink(missing_ok=True)
+    else:
+        path.write_text(f"---\nlanguage: {tag}\nprofile: none\n---\n\n# conventions\n", encoding="utf-8")
+    conventions.reset_cache()
+
+
+def _rich_skill(layer) -> Path:
+    """A skill whose frontmatter has a flag and a value holding `---`, plus a sibling file."""
+    source = layer / "instructions" / "wiki-demo"
+    (source / "SKILL.md").write_text(
+        "---\nname: wiki-demo\ndescription: Demo --- skill.\ndisable-model-invocation: true\n---\n\n# Demo\n\nSee gates.md.\n",
+        encoding="utf-8",
+    )
+    (source / "reference.md").write_text("# reference\n", encoding="utf-8")
+    return source
+
+
+def _block_lines(text: str) -> list[str]:
+    return [line for line in text.splitlines() if line.startswith("> **Conversation language:")]
+
+
+def test_sync_writes_the_language_block_directly_under_the_frontmatter_in_both_targets(layer):
+    source = _rich_skill(layer)
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+
+    source_text = (source / "SKILL.md").read_text(encoding="utf-8")
+    frontmatter_end = source_text.index("---\n\n# Demo") + len("---\n")
+    for harness in (".agents", ".claude"):
+        copy = _skill_copy(layer, harness)
+        text = (copy / "SKILL.md").read_text(encoding="utf-8")
+        assert len(_block_lines(text)) == 1
+        assert text[:frontmatter_end] == source_text[:frontmatter_end]
+        after = text[frontmatter_end:].split("\n")
+        assert after[0] == "" and after[1].startswith("> **Conversation language: `de`**")
+        assert "`AGENTS.md` § Conversation language" in after[1]
+        assert text.replace(after[1] + "\n\n", "", 1) == source_text
+        assert (copy / "reference.md").read_bytes() == (source / "reference.md").read_bytes()
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".agents")) is None
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".claude")) is None
+
+
+def test_the_block_is_the_tag_and_never_a_language_name(layer):
+    _declare_language(layer, "pt-BR")
+    instructions_cmd.sync(force=False)
+    text = (_skill_copy(layer, ".claude") / "SKILL.md").read_text(encoding="utf-8")
+    assert "`pt-BR`" in _block_lines(text)[0]
+    assert "Portuguese" not in text
+
+
+def test_a_second_sync_is_byte_identical_and_adds_no_second_block(layer):
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    first = (_skill_copy(layer, ".agents") / "SKILL.md").read_bytes()
+    instructions_cmd.sync(force=False)
+    second = (_skill_copy(layer, ".agents") / "SKILL.md").read_bytes()
+    assert first == second
+    assert len(_block_lines(second.decode("utf-8"))) == 1
+
+
+def test_without_a_declared_language_every_copy_is_the_source(layer, capsys):
+    source = _rich_skill(layer)
+    _declare_language(layer, None)
+    instructions_cmd.sync(force=False)
+    assert "no language declared" in " ".join(capsys.readouterr().out.split())
+    for harness in (".agents", ".claude"):
+        copy = _skill_copy(layer, harness)
+        assert (copy / "SKILL.md").read_bytes() == (source / "SKILL.md").read_bytes()
+        assert instructions_cmd.drift(source, copy) is None
+
+
+def test_sync_names_the_language_it_stamped(layer, capsys):
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    assert "conversation language `de`" in " ".join(capsys.readouterr().out.split())
+
+
+def test_a_skill_without_closed_frontmatter_is_copied_unchanged(layer):
+    source = layer / "instructions" / "wiki-demo"
+    (source / "SKILL.md").write_text("---\nname: wiki-demo\n# never closed\n", encoding="utf-8")
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    assert (_skill_copy(layer, ".claude") / "SKILL.md").read_bytes() == (source / "SKILL.md").read_bytes()
+
+
+def test_a_crlf_skill_keeps_its_line_endings(layer):
+    source = layer / "instructions" / "wiki-demo"
+    (source / "SKILL.md").write_bytes(b"---\r\nname: wiki-demo\r\ndescription: Demo.\r\n---\r\n\r\n# Demo\r\n")
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    data = (_skill_copy(layer, ".claude") / "SKILL.md").read_bytes()
+    assert data.count(b"\n") == data.count(b"\r\n")
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".claude")) is None
+
+
+def test_changing_the_language_after_sync_is_drift_that_names_the_block(layer):
+    source = layer / "instructions" / "wiki-demo"
+    _declare_language(layer, "en")
+    instructions_cmd.sync(force=False)
+    _declare_language(layer, "de")
+    for harness in (".agents", ".claude"):
+        assert instructions_cmd.drift(source, _skill_copy(layer, harness)) == (
+            "language block is for `en`, conventions say `de`"
+        )
+    instructions_cmd.sync(force=False)
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".agents")) is None
+
+
+def test_declaring_a_language_after_a_plain_sync_is_drift_that_names_the_block(layer):
+    source = layer / "instructions" / "wiki-demo"
+    instructions_cmd.sync(force=False)
+    _declare_language(layer, "de")
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".agents")) == (
+        "language block missing, conventions say `de`"
+    )
+
+
+def test_dropping_the_language_after_sync_is_drift_that_names_the_block(layer):
+    source = layer / "instructions" / "wiki-demo"
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    _declare_language(layer, None)
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".agents")) == (
+        "language block is for `de`, conventions declare no language"
+    )
+
+
+def test_a_hand_edit_next_to_the_block_is_still_reported_as_a_difference(layer):
+    source = layer / "instructions" / "wiki-demo"
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    target = _skill_copy(layer, ".agents") / "SKILL.md"
+    target.write_text(target.read_text(encoding="utf-8") + "extra\n", encoding="utf-8")
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".agents")) == "differs in SKILL.md"
+
+
+def test_a_hand_edit_of_another_file_is_still_reported(layer):
+    source = _rich_skill(layer)
+    _declare_language(layer, "de")
+    instructions_cmd.sync(force=False)
+    (_skill_copy(layer, ".claude") / "reference.md").write_text("# edited\n", encoding="utf-8")
+    assert instructions_cmd.drift(source, _skill_copy(layer, ".claude")) == "differs in reference.md"
+
+
+def test_verify_reports_a_language_change_per_copy_and_sync_repairs_it(layer, capsys):
+    _declare_language(layer, "en")
+    instructions_cmd.sync(force=False)
+    instructions_cmd.verify()
+    capsys.readouterr()
+
+    _declare_language(layer, "de")
+    with pytest.raises(typer.Exit):
+        instructions_cmd.verify()
+    out = capsys.readouterr().out
+    for harness in (".agents", ".claude"):
+        assert f"{harness}/skills/wiki-demo: language block is for `en`, conventions say `de`" in out
+
+    instructions_cmd.sync(force=False)
+    instructions_cmd.verify()
+    assert "match their source" in " ".join(capsys.readouterr().out.split())
+
+
+def test_every_real_skill_takes_the_block():
+    """A source whose frontmatter the insertion cannot find would be published
+    without the language and nothing would say so."""
+    for source in instructions_cmd.skill_dirs():
+        text = (source / "SKILL.md").read_text(encoding="utf-8")
+        rendered = instructions_cmd.render_skill_md(text, "de")
+        assert rendered != text, source.name
+        assert len(_block_lines(rendered)) == 1, source.name
+
+
 # --- verify -----------------------------------------------------------------