search: Pfad und Titel vollstaendig in der Trefferzeile, Trunkierung wird benannt (schliesst #100)
CI / verify (push) Failing after 40s
Release / release (push) Successful in 36s

Files changed:
- AGENTS.md
- CHANGES.md
- VERSION
- instructions/wiki-query/SKILL.md
- tools/CONTRACT.md
- tools/chemenu/api.py
- tools/chemenu/commands/search.py
- tools/chemenu/mcp/server.py
- tools/chemenu/search/service.py
- tools/chemenu/search/types.py
- tools/chemenu/tests/test_api.py
- tools/chemenu/tests/test_mcp_server.py
- tools/chemenu/tests/test_search.py
This commit is contained in:
2026-09-15 21:26:39 +02:00
parent 55f65c1ab1
commit bb097f614b
13 changed files with 352 additions and 41 deletions
+1 -1
View File
@@ -124,7 +124,7 @@ tools/wikitool <command> --help
| Command | Purpose |
|---------|---------|
| `lint [--json] [--markdown out.md] [--full] [--fail-on-error]` | Structural + provenance checks: broken wikilinks, dangling frontmatter references, orphan pages, index drift, schema gaps, duplicate titles, title mismatches, pages nested more than one directory below their collection (hard - the generated catalog folds these into their area silently rather than merely reading it), uncovered raw files, broken `raw_files:` refs, raw files claimed by more than one source page, unmarked provenance, citation/frontmatter drift, unbalanced generated-region markers, edges whose label is missing or not authorised by the source collection's `outbound:` (both hard once `kb_version` has reached the release that introduced labelled edges - advisory below it, so a corpus mid-migration is not refused by the check measuring it), `see-also` edges whose reverse direction already carries a specific label (advisory only - redundant rather than wrong, and never migration-gated, since no version turns the redundancy into an error), a collection past the catalog's per-area shard threshold that has no areas to shard (advisory only - sharding is automatic but per *area*, so a collection nobody gave areas keeps one table however large it grows; reported with the split its subtype field would produce, and only when that split puts every resulting area at or under the threshold, so a lopsided or small collection stays silent), source pages sitting in the `unclassified` catalog slot (advisory only - `unclassified` is the visible fallback for a genuinely unclear source, not a defect), quote-limit overages (>2 blockquoted lines/page, advisory only). Prints only the sections that found something and always writes the full report to `reports/Lint Report <date>.md` (or `--markdown`), naming the path - `--full` prints everything, `--json` prints the findings and writes nothing |
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary so a hit can be judged without opening the page. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. One hit per line, ` | `-separated as `score \| kind/subtype \| title \| path \| summary`, so a hit can be judged without opening the page and then opened without looking it up: **title and path are never truncated** (the title is the identifier `touch`/`xref`/`cite` take), and the summary - the one lossy field, and the only one that may contain the separator - goes last, so splitting on `" \| "` with `maxsplit=4` is unambiguous. Scope is pages: the backend walks `kb/` but drops anything `kb_scan.iter_kb_pages` excludes (the kb-root meta files, every `COLLECTION.md`, every generated `INDEX.md`), which is why a hand-run grep over `kb/` can add none of them but those. `--limit` defaults to 50 (`0` for no limit) and **a truncated result says so** - `50 of 182 result(s)` in the table, `total`/`truncated`/`limit` beside `count` in `--json`, where `count` stays the number of results in the payload; the same default and the same fields are what `api.search` and the MCP `search` tool carry, from one constant. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
### Provenance
+10 -5
View File
@@ -38,7 +38,7 @@ from chemenu.lint_core import run_lint
from chemenu.search import filters
from chemenu.search.registry import resolve
from chemenu.search.service import run_search, unreadable_pages
from chemenu.search.types import Predicate, SearchQuery
from chemenu.search.types import DEFAULT_LIMIT, Predicate, SearchQuery
from chemenu.types_core import describe_type, list_types
# Distinguishes "the caller did not pass a revision" from "the caller passed
@@ -120,7 +120,7 @@ class Corpus:
text: Optional[str] = None,
predicates: Iterable[str] = (),
regex: bool = False,
limit: int = 20,
limit: int = DEFAULT_LIMIT,
sort: Optional[str] = None,
backend: Optional[str] = None,
) -> dict[str, Any]:
@@ -142,13 +142,18 @@ class Corpus:
with self._rooted():
pages, revision = self._cache.load()
hits = run_search(query, pages, backends, self.kb_dir)
result = run_search(query, pages, backends, self.kb_dir)
return self._stamp({
"query": text,
"predicates": [p.render() for p in parsed],
"backend": ",".join(b.name for b in backends),
"count": len(hits),
"results": [hit.as_dict() for hit in hits],
# Same shape the CLI's `--json` prints: `count` is what came back,
# `total` is how many matched before `limit` cut it.
"count": len(result.hits),
"total": result.total,
"truncated": result.truncated,
"limit": result.limit,
"results": [hit.as_dict() for hit in result.hits],
"unreadable": unreadable_pages(pages),
}, revision)
+69 -14
View File
@@ -35,7 +35,7 @@ from chemenu.search.service import (
sort_hits,
unreadable_pages,
)
from chemenu.search.types import Predicate, SearchHit, SearchQuery
from chemenu.search.types import DEFAULT_LIMIT, Predicate, SearchQuery, SearchResult
# Re-exported so `from chemenu.commands.search import run_search` keeps
# resolving. The core lives in `chemenu/search/service.py`, which imports no
@@ -49,32 +49,76 @@ __all__ = [
"search_command",
]
TITLE_WIDTH = 34
SUMMARY_WIDTH = 84
# One hit per line, ` | `-separated, in the order score, kind, title, path,
# summary. Three properties are load-bearing and should survive any edit here:
#
# 1. **The path is present.** It was not, and the instructions that drive this
# command tell an agent to "read only the pages the search points at" - which
# it could not do, because nothing here pointed anywhere. What a session did
# instead was run `grep -rl` over `kb/` for the filenames, a second search
# that can find no page this one missed (the backend *is* `rg` over `kb/`).
# 2. **Title and path are never truncated.** The title is the wiki's only
# identifier for a page (AGENTS.md invariant 2) and the argument `xref add`,
# `cite add` and `touch` all take; a title clipped to a column width is not
# one. The old fixed 34-char field clipped four of five hits in the report
# that prompted this. Only the summary is lossy, which is why it goes last.
# 3. **The separator is unambiguous.** A `|` cannot occur in a title - the
# wikilink syntax reserves it, so a page carrying one could not be linked at
# all - and a `|` in the summary is harmless, because the summary is the
# final field: split on " | " with maxsplit=4 and prose cannot shift a
# column.
#
# Column padding is gone with the widths: it aligned the table for an eye, and
# the reader here is an agent that pays for the spaces by the token.
SEPARATOR = " | "
def _truncate(text: str, width: int) -> str:
text = " ".join(text.split())
return text if len(text) <= width else text[: width - 1] + "\u2026"
def render_table(hits: list[SearchHit], show_matches: bool) -> str:
if not hits:
def _count_line(result: SearchResult) -> str:
"""The last line: how many hits, and whether that is all of them.
A bare `N result(s).` reads as the whole answer, so it is only used when it
is one. A capped search says what it capped, which is the number the caller
would otherwise have to run a second, unlimited search to learn.
"""
if not result.truncated:
return f"{len(result.hits)} result(s)."
return (
f"{len(result.hits)} of {result.total} result(s) - "
f"raise --limit (0 for all) or narrow the query."
)
def render_table(result: SearchResult, show_matches: bool) -> str:
if not result.hits:
return "No matches."
lines = []
for hit in hits:
for hit in result.hits:
kind = hit.kind or "?"
if hit.subtype:
kind = f"{kind}/{hit.subtype}"
lines.append(
f"{hit.score:6.1f} {_truncate(hit.title, TITLE_WIDTH):<{TITLE_WIDTH}} "
f"{kind:<18} {_truncate(hit.summary, SUMMARY_WIDTH)}"
SEPARATOR.join(
(
f"{hit.score:.1f}",
kind,
hit.title,
hit.path,
_truncate(hit.summary, SUMMARY_WIDTH),
)
)
)
if show_matches:
for match in hit.matches:
lines.append(f" {hit.path}:{match.line}: {_truncate(match.text, 100)}")
lines.append(f" {hit.path}:{match.line}: {_truncate(match.text, 100)}")
lines.append("")
lines.append(f"{len(hits)} result(s).")
lines.append(_count_line(result))
return "\n".join(lines)
@@ -99,7 +143,11 @@ def search_command(
regex: bool = typer.Option(
False, "--regex", help="Treat the query as a regex. Off by default: terms are literal."
),
limit: int = typer.Option(20, "--limit", help="Maximum number of results. 0 for no limit."),
limit: int = typer.Option(
DEFAULT_LIMIT,
"--limit",
help="Maximum number of results. 0 for no limit. A capped result says so.",
),
sort: str = typer.Option(
None, "--sort", help="Sort by a result field; prefix with '-' to reverse, e.g. -modified."
),
@@ -146,7 +194,7 @@ def search_command(
pages = load_pages_by_path()
try:
hits = run_search(query, pages, backends)
result = run_search(query, pages, backends)
except PredicateError as exc:
fail(str(exc))
except RipgrepMissing as exc:
@@ -162,8 +210,15 @@ def search_command(
"query": text,
"predicates": [p.render() for p in predicates],
"backend": ",".join(b.name for b in backends),
"count": len(hits),
"results": [hit.as_dict() for hit in hits],
# `count` keeps its meaning - how many results are in this payload -
# so a consumer written against the old shape reads the same number
# it always did. `total`/`truncated`/`limit` are what it could not
# ask before.
"count": len(result.hits),
"total": result.total,
"truncated": result.truncated,
"limit": result.limit,
"results": [hit.as_dict() for hit in result.hits],
# Always present, usually empty. A caller that has to look for the
# key to learn whether it should worry will not look.
"unreadable": unreadable,
@@ -171,7 +226,7 @@ def search_command(
typer.echo(json.dumps(payload, indent=2))
return
typer.echo(render_table(hits, show_matches))
typer.echo(render_table(result, show_matches))
for entry in unreadable:
typer.echo(
f"WARN unreadable frontmatter: {entry['path']} ({entry['reason']}) - "
+5 -2
View File
@@ -56,6 +56,7 @@ from mcp.server.mcpserver.exceptions import ToolError
from chemenu import config, upload
from chemenu.api import Corpus
from chemenu.errors import ChemenuError
from chemenu.search.types import DEFAULT_LIMIT
from chemenu.telemetry import policy
SERVER_NAME = "chemenu"
@@ -158,14 +159,16 @@ def build_server(
"Find pages in kb/ by text, by frontmatter, or by both. Returns "
"title, path, kind and summary per hit, so a result can be judged "
"without fetching the page. Prefer this over listing files: the "
"answer is a few hundred tokens instead of a whole index."
"answer is a few hundred tokens instead of a whole index. "
"'count' is how many hits came back and 'total' how many matched; "
"when 'truncated' is true, raise 'limit' (0 for all) to see the rest."
),
)
def search(
query: str | None = None,
predicates: list[str] | None = None,
regex: bool = False,
limit: int = 20,
limit: int = DEFAULT_LIMIT,
sort: str | None = None,
) -> dict[str, Any]:
"""Search the wiki.
+14 -4
View File
@@ -28,7 +28,7 @@ from chemenu.search import filters
from chemenu.search.base import page_key
from chemenu.search.fuse import reciprocal_rank_fusion
from chemenu.search.ripgrep import build_hit
from chemenu.search.types import SearchHit, SearchQuery
from chemenu.search.types import SearchHit, SearchQuery, SearchResult
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
@@ -92,8 +92,13 @@ def run_search(
pages: dict[str, Page],
backends: list,
kb_dir: Path | None = None,
) -> list[SearchHit]:
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
) -> SearchResult:
"""Answer a query. Pure: no I/O beyond whatever a backend does.
Returns the truncated hits *and* the number there were before the limit,
because the caller cannot recover the second from the first - see
`SearchResult`.
"""
filters.validate_fields(query.predicates, pages)
if query.text:
@@ -110,4 +115,9 @@ def run_search(
hits.sort(key=lambda h: h.title.lower())
hits = sort_hits(hits, query.sort)
return hits[: query.limit] if query.limit else hits
total = len(hits)
return SearchResult(
hits=hits[: query.limit] if query.limit else hits,
total=total,
limit=query.limit,
)
+44 -1
View File
@@ -4,6 +4,18 @@ from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Optional
# How many hits a caller gets when it asks for no particular number. Defined
# once, here, because three adapters offer the same knob - the CLI's `--limit`,
# `api.search(limit=...)` and the MCP `search` tool - and three literals is how
# they start disagreeing about what "the default search" returns.
#
# 50 rather than a smaller number because the queries that actually hit the cap
# are the *structured* sweeps (`--field '!sources'`), which are ordered
# alphabetically rather than by relevance: truncating those throws away an
# arbitrary slice of the answer rather than its weakest tail. A capped result
# is only safe at all because it now says so - see `SearchResult.truncated`.
DEFAULT_LIMIT = 50
@dataclass(frozen=True)
class Predicate:
@@ -30,7 +42,7 @@ class SearchQuery:
text: Optional[str] = None
predicates: tuple[Predicate, ...] = ()
regex: bool = False
limit: int = 20
limit: int = DEFAULT_LIMIT
sort: Optional[str] = None
@@ -77,3 +89,34 @@ class SearchHit:
"backend": self.backend,
"matches": [m.as_dict() for m in self.matches],
}
@dataclass(frozen=True)
class SearchResult:
"""The hits a caller gets back, plus how many there were before the limit.
`run_search` used to return the truncated list alone, which made the
truncation impossible to report: every adapter counted `len(hits)` and
printed it as the answer, so `20 result(s).` on a query matching 182 pages
was indistinguishable from a query that really matched twenty. That is a
completeness claim none of them were in a position to make, and the only
way to find out was to ask again with `--limit 0` - a second full search to
learn a number the first one already knew.
So the total travels with the hits. Nothing here decides how to say it;
that belongs to each adapter (`render_table`, the `--json` payload,
`api.search`).
"""
hits: list[SearchHit]
total: int
limit: int
@property
def truncated(self) -> bool:
"""Whether the limit actually cut something off.
`limit=0` means "no limit", so it never truncates however large the
corpus is.
"""
return bool(self.limit) and self.total > len(self.hits)
+15
View File
@@ -47,6 +47,21 @@ def test_a_corpus_can_be_named_and_is_the_one_that_answers(foreign_corpus):
assert result["results"][0]["path"] == "kb/entities/Peregrine.md"
def test_a_capped_answer_says_how_much_it_left_out(foreign_corpus):
"""`count` keeps meaning "what is in this payload", so a consumer written
against the old shape reads the number it always did; `total` is what it
could not ask before, and without it a truncated answer is indistinguishable
from a complete one."""
corpus = Corpus(foreign_corpus)
whole = corpus.search("Peregrine")
assert (whole["count"], whole["total"], whole["truncated"]) == (1, 1, False)
assert whole["count"] == len(whole["results"])
capped = corpus.search("Peregrine", limit=0)
assert capped["truncated"] is False, "limit 0 means no limit, so it caps nothing"
def test_no_path_of_this_checkout_is_read_while_a_foreign_root_is_set(foreign_corpus):
"""The acceptance criterion, asserted rather than argued.
+20 -1
View File
@@ -125,6 +125,24 @@ def test_the_four_tools_are_there_and_nothing_that_writes(corpus):
assert names.isdisjoint({"new", "touch", "xref", "cite", "publish", "migrate", "rm"})
def test_all_three_adapters_offer_the_same_default_limit(corpus):
"""The CLI's `--limit`, `api.search(limit=...)` and this tool are three
knobs on one search, and three literals is how they start disagreeing about
what "the default search" returns. They read one constant."""
import inspect
from chemenu.api import Corpus
from chemenu.commands.search import search_command
from chemenu.search.types import DEFAULT_LIMIT
server = build_server(corpus, check_trace=False)
tool = next(t for t in asyncio.run(server.list_tools()) if t.name == "search")
assert tool.input_schema["properties"]["limit"]["default"] == DEFAULT_LIMIT
assert inspect.signature(Corpus.search).parameters["limit"].default == DEFAULT_LIMIT
assert inspect.signature(search_command).parameters["limit"].default.default == DEFAULT_LIMIT
def test_no_tool_writes_anything_into_the_corpus_or_git(corpus):
server = build_server(corpus, check_trace=False)
before = _tree(corpus)
@@ -176,7 +194,8 @@ def test_the_wire_format_is_the_clis_json_form(corpus):
# `generated` is the CLI's date stamp and `commit`/`as_of` are the server's
# revision stamp - two answers to "when", neither of them a finding. What
# has to match is everything that describes the *corpus*.
shared = ("query", "predicates", "backend", "count", "results", "unreadable")
shared = ("query", "predicates", "backend", "count", "total", "truncated", "limit",
"results", "unreadable")
assert {key: served[key] for key in shared} == {key: from_cli[key] for key in shared}
+102 -6
View File
@@ -17,7 +17,14 @@ from chemenu.search.filters import PredicateError, parse_predicate
from chemenu.search.fuse import reciprocal_rank_fusion
from chemenu.search.registry import UnknownBackend, resolve
from chemenu.search.ripgrep import RipgrepBackend, build_argv
from chemenu.search.types import Match, Predicate, SearchHit, SearchQuery
from chemenu.search.types import (
DEFAULT_LIMIT,
Match,
Predicate,
SearchHit,
SearchQuery,
SearchResult,
)
from chemenu.frontmatter_io import write_page
@@ -32,8 +39,12 @@ def backend(kb_dir: Path, tmp_path: Path):
@pytest.fixture
def search(kb_dir: Path, pages):
"""Run a query against the fixture kb rather than the real one."""
def search_result(kb_dir: Path, pages):
"""Run a query against the fixture kb and return the whole `SearchResult`.
For the tests that care about the limit and the total; most only want the
hits, and use `search` below.
"""
def _search(query: SearchQuery, backends=()):
return run_search(query, pages, list(backends), kb_dir)
@@ -41,6 +52,16 @@ def search(kb_dir: Path, pages):
return _search
@pytest.fixture
def search(search_result):
"""The hits alone, for tests whose subject is ranking or filtering."""
def _search(query: SearchQuery, backends=()):
return search_result(query, backends).hits
return _search
def _titles(hits):
return [hit.title for hit in hits]
@@ -211,6 +232,20 @@ def test_limit_and_sort(search):
assert len(search(_q("kind=entity", limit=2))) == 2
def test_the_total_survives_the_limit_so_no_second_search_is_needed(search_result):
"""The caller cannot recover the total from a truncated list, and asking
again with `--limit 0` is a second full search to learn a number the first
one already had."""
capped = search_result(_q("kind=entity", limit=2))
assert len(capped.hits) == 2
assert capped.total == 3
assert capped.truncated is True
whole = search_result(_q("kind=entity"))
assert (whole.total, whole.truncated) == (3, False)
def test_sort_puts_missing_values_last():
hits = [
SearchHit(title="b", path="b", modified=None),
@@ -240,18 +275,79 @@ def test_resolve_defaults_to_rg_and_rejects_unknown():
# --- output -----------------------------------------------------------------
def _result(hits, total=None, limit=DEFAULT_LIMIT):
return SearchResult(hits=hits, total=total if total is not None else len(hits), limit=limit)
def test_render_table_is_compact_and_reports_the_count():
hit = SearchHit(title="aurora", path="kb/x.md", kind="entity", subtype="system",
summary="Server hosting DocStore", score=8.0, matches=[Match(3, "DocStore")])
out = render_table([hit], show_matches=False)
out = render_table(_result([hit]), show_matches=False)
assert "aurora" in out and "entity/system" in out
assert "kb/x.md:3" not in out
assert "1 result(s)." in out
assert "kb/x.md:3" in render_table([hit], show_matches=True)
assert "kb/x.md:3" in render_table(_result([hit]), show_matches=True)
def test_render_table_says_so_when_nothing_matched():
assert render_table([], show_matches=False) == "No matches."
assert render_table(_result([]), show_matches=False) == "No matches."
def test_every_hit_carries_the_path_so_the_page_can_be_opened():
"""The reason this format changed: the instructions tell an agent to read
the pages the search points at, and the table pointed nowhere. A session
that needed filenames ran `grep -rl` over kb/ for them - a second search
that can find no page this one missed."""
hit = SearchHit(title="aurora", path="kb/entities/systems/aurora.md", kind="entity")
assert "kb/entities/systems/aurora.md" in render_table(_result([hit]), show_matches=False)
def test_title_and_path_are_never_truncated_only_the_summary_is():
"""A clipped title is not an identifier. It is what `xref add`, `cite add`
and `touch` take as an argument, and the old fixed 34-char column cut four
of five hits in the report that prompted this."""
title = "Source - Pelletofenkondensator und Verkabelung Recherche"
path = f"kb/sources/llm-sessions/{title}.md"
hit = SearchHit(title=title, path=path, kind="source", summary="x" * 400)
line = render_table(_result([hit]), show_matches=False).splitlines()[0]
assert title in line
assert path in line
assert "" in line, "the summary is still the one lossy field"
def test_a_hit_line_parses_into_its_five_fields_even_with_prose_pipes():
"""The separator has to survive a summary that contains one. It does,
because the summary is last: a `|` there cannot shift a column. A `|` in a
title is impossible - the wikilink syntax reserves it."""
hit = SearchHit(title="aurora", path="kb/x.md", kind="entity", subtype="system",
summary="Runs `a | b` nightly", score=8.0)
line = render_table(_result([hit]), show_matches=False).splitlines()[0]
score, kind, title, path, summary = line.split(" | ", 4)
assert (score, kind, title, path) == ("8.0", "entity/system", "aurora", "kb/x.md")
assert summary == "Runs `a | b` nightly"
def test_a_capped_result_says_what_it_capped():
"""`20 result(s).` on a query matching 182 pages is a completeness claim
the output was in no position to make, and the only way to find the real
number was a second, unlimited search."""
hits = [SearchHit(title=f"p{i}", path=f"kb/p{i}.md") for i in range(20)]
out = render_table(_result(hits, total=182, limit=20), show_matches=False)
assert "20 of 182 result(s)" in out
assert "--limit" in out
def test_an_uncapped_result_claims_nothing_about_a_limit():
hits = [SearchHit(title=f"p{i}", path=f"kb/p{i}.md") for i in range(3)]
assert "3 result(s)." in render_table(_result(hits), show_matches=False)
assert " of " not in render_table(_result(hits), show_matches=False)
def test_limit_zero_never_counts_as_truncated_however_large_the_corpus():
assert _result([SearchHit(title="a", path="a")] * 182, total=182, limit=0).truncated is False
def test_hit_serialises_for_json():