search: Pfad und Titel vollstaendig in der Trefferzeile, Trunkierung wird benannt (schliesst #100)
Files changed: - AGENTS.md - CHANGES.md - VERSION - instructions/wiki-query/SKILL.md - tools/CONTRACT.md - tools/chemenu/api.py - tools/chemenu/commands/search.py - tools/chemenu/mcp/server.py - tools/chemenu/search/service.py - tools/chemenu/search/types.py - tools/chemenu/tests/test_api.py - tools/chemenu/tests/test_mcp_server.py - tools/chemenu/tests/test_search.py
This commit is contained in:
@@ -251,6 +251,13 @@ tools/wikitool search --field entity_type=system --field '!sources'
|
||||
|
||||
`search` is read-only and exempt from the iteration budget.
|
||||
|
||||
**It is also exhaustive, so do not grep `kb/` yourself.** `search` *is* a `rg` run over `kb/`,
|
||||
enriched with each hit's frontmatter and ranked; a grep of your own can therefore surface no
|
||||
page it missed, only the generated files it deliberately excludes - `kb/index.md`, `kb/log.md`,
|
||||
`kb/provenance.md`, every `INDEX.md` - which invariant 1 forbids acting on anyway. Each hit
|
||||
carries the page's full path and full title, so it can be opened and passed to the commands
|
||||
that take a title. A result cut short by `--limit` says so and names the total.
|
||||
|
||||
## Gates
|
||||
|
||||
Four limits are enforced in code rather than by instruction, because a prompt-level limit is
|
||||
|
||||
+56
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
||||
|
||||
---
|
||||
|
||||
## 6.0.0-beta.7 - 2026-09-15 - types/type-spec.schema.yaml enforced against real type-spec frontmatter
|
||||
## 6.0.0-beta.8 - 2026-09-15 - search: Pfad und Titel vollstaendig, Trunkierung sichtbar
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -80,6 +80,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- docs verify: ein nur als .template ausgeliefertes Linkziel gilt als aufgeloest
|
||||
- Control-Plane-Sprache universell: Achse ist das Publikum, kein Instanz-Schalter
|
||||
- types/type-spec.schema.yaml enforced against real type-spec frontmatter
|
||||
- search: Pfad und Titel vollstaendig, Trunkierung sichtbar
|
||||
|
||||
**Low impact**
|
||||
- gates.md/session-setup.md: die Budget-Ausnahme von version regrade haengt an der Aufrufform
|
||||
@@ -342,6 +343,60 @@ Verifiziert: `tools/wikitool docs verify`/`instructions verify` gruen, 1263 Test
|
||||
alle Type-Specs dieses Repos validieren gegen ihr eigenes Schema; ein Type-Spec mit einem dem
|
||||
Schema unbekannten Feld wird gemeldet, mit Dateiname und Feldname in der Meldung).
|
||||
|
||||
### search: Pfad und Titel vollstaendig, Trunkierung sichtbar
|
||||
|
||||
Gemeldet wurde eine Sitzung, die nach `wikitool search` zusaetzlich `grep -rl` ueber `kb/`
|
||||
laufen liess. Der Grep war redundant - `search` *ist* ein `rg`-Lauf ueber `kb/` und kann keine
|
||||
Seite verfehlen, die ein Grep findet -, aber die Ausgabe gab ihr drei Gruende dafuer, und die
|
||||
sind der eigentliche Befund.
|
||||
|
||||
Die Tabelle nannte **keinen Pfad**, obwohl `wiki-query` verlangt, nur die Seiten zu lesen, auf
|
||||
die die Suche zeigt. Sie kappte ausserdem den **Titel** auf 34 Zeichen - im gemeldeten
|
||||
Transkript vier von fuenf Treffern -, und der Titel ist nach Invariante 2 der einzige
|
||||
Identifier einer Seite und das Argument, das `touch`, `xref add` und `cite add` nehmen. Die
|
||||
Sitzung hatte also weder etwas zum Oeffnen noch etwas zum Weiterreichen; `grep -rl` lieferte
|
||||
genau beides.
|
||||
|
||||
Drittens war `N result(s).` die *gekappte* Zahl: `run_search` gab nur die beschnittene Liste
|
||||
zurueck, also konnte kein Adapter die Gesamtzahl melden, und `20 result(s).` auf einer Anfrage
|
||||
mit 182 Treffern war von einem vollstaendigen Ergebnis nicht zu unterscheiden. Eine
|
||||
Vollstaendigkeitsaussage, zu der die Ausgabe nicht berechtigt war - der staerkste denkbare
|
||||
Anlass, ihr zu misstrauen.
|
||||
|
||||
Die Zeile hat jetzt die Form `score | kind/subtype | titel | pfad | summary`, ohne
|
||||
Spaltenauffuellung. Titel und Pfad werden nie gekappt; die Summary ist das einzige verlustige
|
||||
Feld und steht deshalb am Ende, wo ein `|` in Prosa beim Trennen mit `maxsplit=4` folgenlos
|
||||
bleibt (ein `|` im Titel schliesst die Wikilink-Syntax ohnehin aus). JSON als Default-Ausgabe
|
||||
wurde erwogen und verworfen: ein Treffer ist flach, JSON kostet dafuer ein Vielfaches an Tokens,
|
||||
und `search` existiert dafuer, Retrieval billig zu machen - der Fehler war ein fehlendes Feld,
|
||||
kein Parse-Problem. Wer Struktur braucht, hat `--json`, `api.search` und MCP.
|
||||
|
||||
`run_search` gibt jetzt ein `SearchResult` mit Treffern, Gesamtzahl und Limit zurueck. Die
|
||||
Tabelle schreibt `50 of 182 result(s) - raise --limit (0 for all) or narrow the query.`, das
|
||||
JSON traegt `total`/`truncated`/`limit` neben `count`, dessen Bedeutung unveraendert bleibt
|
||||
(`len(results)`), und `api.search` sowie der MCP-`search`-Tool tragen dieselben Felder. Das
|
||||
Default-Limit steigt von 20 auf 50 und liegt als eine Konstante `DEFAULT_LIMIT` statt als drei
|
||||
Literale in drei Adaptern: gekappt wurden bisher vor allem die *strukturellen* Sweeps
|
||||
(`--field '!sources'`), die alphabetisch und nicht nach Relevanz sortiert sind, wo die Kappung
|
||||
also eine beliebige Scheibe der Antwort wegwirft statt ihres schwaechsten Endes. Sichtbar zu
|
||||
sein ist es, was ein endliches Default ueberhaupt erst unbedenklich macht.
|
||||
|
||||
`AGENTS.md` § Routing traegt die Regel an genau einer Stelle - `search` ist erschoepfend, ein
|
||||
eigener Grep ueber `kb/` fuegt nur die generierten Dateien hinzu, die Invariante 1 ohnehin
|
||||
verbietet; `tools/CONTRACT.md` traegt daneben nur den Mechanismus.
|
||||
|
||||
Grenzuebertritt-Frage geprueft und verneint: kein Flag entfernt oder umbenannt, keine
|
||||
Umgebungsvariable, keine maschinengelesene Datei in ihrer Form veraendert, JSON rein additiv.
|
||||
Die Tabelle liest ein Agent, kein Skript, und ihre Aenderung verlangt keiner Instanz Handarbeit
|
||||
ab.
|
||||
|
||||
Verifiziert: `docs verify`/`instructions verify` gruen, 1272 Tests gruen (9 neu: Pfad vorhanden;
|
||||
Titel und Pfad ungekappt bei langem Titel; eine Trefferzeile zerfaellt trotz `|` in der Prosa in
|
||||
ihre fuenf Felder; ein gekapptes Ergebnis nennt die Gesamtzahl, ein ungekapptes behauptet
|
||||
nichts; `--limit 0` gilt nie als gekappt; die Gesamtzahl ueberlebt das Limit in `run_search`;
|
||||
`api.search` meldet dasselbe; alle drei Adapter teilen ein Default-Limit - der MCP-Golden-Test
|
||||
haelt die neuen Felder zwischen CLI und Server zusammen).
|
||||
|
||||
## 5.1.0 - 2026-09-12 - changelog: Kandidaten-Eintrag nach Impact gruppiert, version regrade zur Nachkorrektur, version release verlangt eine Zusammenfassung
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -34,15 +34,18 @@ invariant - rather than synthesizing a plausible-sounding answer from general kn
|
||||
```bash
|
||||
tools/wikitool search "backup" --kind entity --subtype system
|
||||
tools/wikitool search --field entity_type=system --field '!sources' --sort -modified
|
||||
tools/wikitool search --field tags=k8s --limit 30
|
||||
tools/wikitool search --field tags=k8s --limit 0 # a sweep: every match, not the first 50
|
||||
tools/wikitool search "Longhorn" --matches # show the matching lines
|
||||
```
|
||||
|
||||
`search` is read-only and exempt from the iteration budget, so searching again is always
|
||||
cheaper than reading more.
|
||||
cheaper than reading more. A result that hit `--limit` says so and names the total, so read
|
||||
the last line before treating a list as the whole answer - and do not grep `kb/` yourself,
|
||||
per AGENTS.md § Routing.
|
||||
|
||||
3. **Read only the pages the search points at**, then follow their `related:` and `sources:`
|
||||
entries. Check `kb/sources/` when the question is about what a specific source said.
|
||||
3. **Read only the pages the search points at** - each hit carries its full path - then follow
|
||||
their `related:` and `sources:` entries. Check `kb/sources/` when the question is about what
|
||||
a specific source said.
|
||||
|
||||
4. **Answer and cite.** Name the wiki pages the answer came from, and the sources behind them.
|
||||
Hedge to what those sources carry, not to a number - see
|
||||
|
||||
+1
-1
@@ -124,7 +124,7 @@ tools/wikitool <command> --help
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `lint [--json] [--markdown out.md] [--full] [--fail-on-error]` | Structural + provenance checks: broken wikilinks, dangling frontmatter references, orphan pages, index drift, schema gaps, duplicate titles, title mismatches, pages nested more than one directory below their collection (hard - the generated catalog folds these into their area silently rather than merely reading it), uncovered raw files, broken `raw_files:` refs, raw files claimed by more than one source page, unmarked provenance, citation/frontmatter drift, unbalanced generated-region markers, edges whose label is missing or not authorised by the source collection's `outbound:` (both hard once `kb_version` has reached the release that introduced labelled edges - advisory below it, so a corpus mid-migration is not refused by the check measuring it), `see-also` edges whose reverse direction already carries a specific label (advisory only - redundant rather than wrong, and never migration-gated, since no version turns the redundancy into an error), a collection past the catalog's per-area shard threshold that has no areas to shard (advisory only - sharding is automatic but per *area*, so a collection nobody gave areas keeps one table however large it grows; reported with the split its subtype field would produce, and only when that split puts every resulting area at or under the threshold, so a lopsided or small collection stays silent), source pages sitting in the `unclassified` catalog slot (advisory only - `unclassified` is the visible fallback for a genuinely unclear source, not a defect), quote-limit overages (>2 blockquoted lines/page, advisory only). Prints only the sections that found something and always writes the full report to `reports/Lint Report <date>.md` (or `--markdown`), naming the path - `--full` prints everything, `--json` prints the findings and writes nothing |
|
||||
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. Results carry kind/summary so a hit can be judged without opening the page. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
|
||||
| `search ["<text>"] [--field <predicate> ...] [--kind/--subtype/--collection/--tag <v>] [--regex] [--limit N] [--sort [-]<field>] [--backend <name>] [--matches] [--json]` | Find pages in `kb/` without reading the index. Text search runs through a pluggable backend (`rg` today); `--field` predicates are evaluated on frontmatter - `f=v`, `f~substring`, `'f>=v'`, `'f:*'` (present), `'!f'` (absent), repeatable and ANDed. With no text this is a pure structured query. One hit per line, ` | `-separated as `score \| kind/subtype \| title \| path \| summary`, so a hit can be judged without opening the page and then opened without looking it up: **title and path are never truncated** (the title is the identifier `touch`/`xref`/`cite` take), and the summary - the one lossy field, and the only one that may contain the separator - goes last, so splitting on `" \| "` with `maxsplit=4` is unambiguous. Scope is pages: the backend walks `kb/` but drops anything `kb_scan.iter_kb_pages` excludes (the kb-root meta files, every `COLLECTION.md`, every generated `INDEX.md`), which is why a hand-run grep over `kb/` can add none of them but those. `--limit` defaults to 50 (`0` for no limit) and **a truncated result says so** - `50 of 182 result(s)` in the table, `total`/`truncated`/`limit` beside `count` in `--json`, where `count` stays the number of results in the payload; the same default and the same fields are what `api.search` and the MCP `search` tool carry, from one constant. A page whose frontmatter does not parse can match no positive predicate, so it is **named** rather than dropped: `--json` always carries an `unreadable` list of `{path, reason}` (usually empty), and the table form writes the same lines to stderr. `--regex` is applied by `rg` alone, whose engine is linear; the ranking boosts for title and summary are literal-containment only, so a non-literal pattern is ranked by match count. `rg` is killed after 30 s and reported as a failure. Read-only, and **exempt from the Iteration Budget Gate** |
|
||||
|
||||
### Provenance
|
||||
|
||||
|
||||
+10
-5
@@ -38,7 +38,7 @@ from chemenu.lint_core import run_lint
|
||||
from chemenu.search import filters
|
||||
from chemenu.search.registry import resolve
|
||||
from chemenu.search.service import run_search, unreadable_pages
|
||||
from chemenu.search.types import Predicate, SearchQuery
|
||||
from chemenu.search.types import DEFAULT_LIMIT, Predicate, SearchQuery
|
||||
from chemenu.types_core import describe_type, list_types
|
||||
|
||||
# Distinguishes "the caller did not pass a revision" from "the caller passed
|
||||
@@ -120,7 +120,7 @@ class Corpus:
|
||||
text: Optional[str] = None,
|
||||
predicates: Iterable[str] = (),
|
||||
regex: bool = False,
|
||||
limit: int = 20,
|
||||
limit: int = DEFAULT_LIMIT,
|
||||
sort: Optional[str] = None,
|
||||
backend: Optional[str] = None,
|
||||
) -> dict[str, Any]:
|
||||
@@ -142,13 +142,18 @@ class Corpus:
|
||||
|
||||
with self._rooted():
|
||||
pages, revision = self._cache.load()
|
||||
hits = run_search(query, pages, backends, self.kb_dir)
|
||||
result = run_search(query, pages, backends, self.kb_dir)
|
||||
return self._stamp({
|
||||
"query": text,
|
||||
"predicates": [p.render() for p in parsed],
|
||||
"backend": ",".join(b.name for b in backends),
|
||||
"count": len(hits),
|
||||
"results": [hit.as_dict() for hit in hits],
|
||||
# Same shape the CLI's `--json` prints: `count` is what came back,
|
||||
# `total` is how many matched before `limit` cut it.
|
||||
"count": len(result.hits),
|
||||
"total": result.total,
|
||||
"truncated": result.truncated,
|
||||
"limit": result.limit,
|
||||
"results": [hit.as_dict() for hit in result.hits],
|
||||
"unreadable": unreadable_pages(pages),
|
||||
}, revision)
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ from chemenu.search.service import (
|
||||
sort_hits,
|
||||
unreadable_pages,
|
||||
)
|
||||
from chemenu.search.types import Predicate, SearchHit, SearchQuery
|
||||
from chemenu.search.types import DEFAULT_LIMIT, Predicate, SearchQuery, SearchResult
|
||||
|
||||
# Re-exported so `from chemenu.commands.search import run_search` keeps
|
||||
# resolving. The core lives in `chemenu/search/service.py`, which imports no
|
||||
@@ -49,32 +49,76 @@ __all__ = [
|
||||
"search_command",
|
||||
]
|
||||
|
||||
TITLE_WIDTH = 34
|
||||
SUMMARY_WIDTH = 84
|
||||
|
||||
# One hit per line, ` | `-separated, in the order score, kind, title, path,
|
||||
# summary. Three properties are load-bearing and should survive any edit here:
|
||||
#
|
||||
# 1. **The path is present.** It was not, and the instructions that drive this
|
||||
# command tell an agent to "read only the pages the search points at" - which
|
||||
# it could not do, because nothing here pointed anywhere. What a session did
|
||||
# instead was run `grep -rl` over `kb/` for the filenames, a second search
|
||||
# that can find no page this one missed (the backend *is* `rg` over `kb/`).
|
||||
# 2. **Title and path are never truncated.** The title is the wiki's only
|
||||
# identifier for a page (AGENTS.md invariant 2) and the argument `xref add`,
|
||||
# `cite add` and `touch` all take; a title clipped to a column width is not
|
||||
# one. The old fixed 34-char field clipped four of five hits in the report
|
||||
# that prompted this. Only the summary is lossy, which is why it goes last.
|
||||
# 3. **The separator is unambiguous.** A `|` cannot occur in a title - the
|
||||
# wikilink syntax reserves it, so a page carrying one could not be linked at
|
||||
# all - and a `|` in the summary is harmless, because the summary is the
|
||||
# final field: split on " | " with maxsplit=4 and prose cannot shift a
|
||||
# column.
|
||||
#
|
||||
# Column padding is gone with the widths: it aligned the table for an eye, and
|
||||
# the reader here is an agent that pays for the spaces by the token.
|
||||
SEPARATOR = " | "
|
||||
|
||||
|
||||
def _truncate(text: str, width: int) -> str:
|
||||
text = " ".join(text.split())
|
||||
return text if len(text) <= width else text[: width - 1] + "\u2026"
|
||||
|
||||
|
||||
def render_table(hits: list[SearchHit], show_matches: bool) -> str:
|
||||
if not hits:
|
||||
def _count_line(result: SearchResult) -> str:
|
||||
"""The last line: how many hits, and whether that is all of them.
|
||||
|
||||
A bare `N result(s).` reads as the whole answer, so it is only used when it
|
||||
is one. A capped search says what it capped, which is the number the caller
|
||||
would otherwise have to run a second, unlimited search to learn.
|
||||
"""
|
||||
if not result.truncated:
|
||||
return f"{len(result.hits)} result(s)."
|
||||
return (
|
||||
f"{len(result.hits)} of {result.total} result(s) - "
|
||||
f"raise --limit (0 for all) or narrow the query."
|
||||
)
|
||||
|
||||
|
||||
def render_table(result: SearchResult, show_matches: bool) -> str:
|
||||
if not result.hits:
|
||||
return "No matches."
|
||||
lines = []
|
||||
for hit in hits:
|
||||
for hit in result.hits:
|
||||
kind = hit.kind or "?"
|
||||
if hit.subtype:
|
||||
kind = f"{kind}/{hit.subtype}"
|
||||
lines.append(
|
||||
f"{hit.score:6.1f} {_truncate(hit.title, TITLE_WIDTH):<{TITLE_WIDTH}} "
|
||||
f"{kind:<18} {_truncate(hit.summary, SUMMARY_WIDTH)}"
|
||||
SEPARATOR.join(
|
||||
(
|
||||
f"{hit.score:.1f}",
|
||||
kind,
|
||||
hit.title,
|
||||
hit.path,
|
||||
_truncate(hit.summary, SUMMARY_WIDTH),
|
||||
)
|
||||
)
|
||||
)
|
||||
if show_matches:
|
||||
for match in hit.matches:
|
||||
lines.append(f" {hit.path}:{match.line}: {_truncate(match.text, 100)}")
|
||||
lines.append("")
|
||||
lines.append(f"{len(hits)} result(s).")
|
||||
lines.append(_count_line(result))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
@@ -99,7 +143,11 @@ def search_command(
|
||||
regex: bool = typer.Option(
|
||||
False, "--regex", help="Treat the query as a regex. Off by default: terms are literal."
|
||||
),
|
||||
limit: int = typer.Option(20, "--limit", help="Maximum number of results. 0 for no limit."),
|
||||
limit: int = typer.Option(
|
||||
DEFAULT_LIMIT,
|
||||
"--limit",
|
||||
help="Maximum number of results. 0 for no limit. A capped result says so.",
|
||||
),
|
||||
sort: str = typer.Option(
|
||||
None, "--sort", help="Sort by a result field; prefix with '-' to reverse, e.g. -modified."
|
||||
),
|
||||
@@ -146,7 +194,7 @@ def search_command(
|
||||
|
||||
pages = load_pages_by_path()
|
||||
try:
|
||||
hits = run_search(query, pages, backends)
|
||||
result = run_search(query, pages, backends)
|
||||
except PredicateError as exc:
|
||||
fail(str(exc))
|
||||
except RipgrepMissing as exc:
|
||||
@@ -162,8 +210,15 @@ def search_command(
|
||||
"query": text,
|
||||
"predicates": [p.render() for p in predicates],
|
||||
"backend": ",".join(b.name for b in backends),
|
||||
"count": len(hits),
|
||||
"results": [hit.as_dict() for hit in hits],
|
||||
# `count` keeps its meaning - how many results are in this payload -
|
||||
# so a consumer written against the old shape reads the same number
|
||||
# it always did. `total`/`truncated`/`limit` are what it could not
|
||||
# ask before.
|
||||
"count": len(result.hits),
|
||||
"total": result.total,
|
||||
"truncated": result.truncated,
|
||||
"limit": result.limit,
|
||||
"results": [hit.as_dict() for hit in result.hits],
|
||||
# Always present, usually empty. A caller that has to look for the
|
||||
# key to learn whether it should worry will not look.
|
||||
"unreadable": unreadable,
|
||||
@@ -171,7 +226,7 @@ def search_command(
|
||||
typer.echo(json.dumps(payload, indent=2))
|
||||
return
|
||||
|
||||
typer.echo(render_table(hits, show_matches))
|
||||
typer.echo(render_table(result, show_matches))
|
||||
for entry in unreadable:
|
||||
typer.echo(
|
||||
f"WARN unreadable frontmatter: {entry['path']} ({entry['reason']}) - "
|
||||
|
||||
@@ -56,6 +56,7 @@ from mcp.server.mcpserver.exceptions import ToolError
|
||||
from chemenu import config, upload
|
||||
from chemenu.api import Corpus
|
||||
from chemenu.errors import ChemenuError
|
||||
from chemenu.search.types import DEFAULT_LIMIT
|
||||
from chemenu.telemetry import policy
|
||||
|
||||
SERVER_NAME = "chemenu"
|
||||
@@ -159,13 +160,15 @@ def build_server(
|
||||
"title, path, kind and summary per hit, so a result can be judged "
|
||||
"without fetching the page. Prefer this over listing files: the "
|
||||
"answer is a few hundred tokens instead of a whole index. "
|
||||
"'count' is how many hits came back and 'total' how many matched; "
|
||||
"when 'truncated' is true, raise 'limit' (0 for all) to see the rest."
|
||||
),
|
||||
)
|
||||
def search(
|
||||
query: str | None = None,
|
||||
predicates: list[str] | None = None,
|
||||
regex: bool = False,
|
||||
limit: int = 20,
|
||||
limit: int = DEFAULT_LIMIT,
|
||||
sort: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Search the wiki.
|
||||
|
||||
@@ -28,7 +28,7 @@ from chemenu.search import filters
|
||||
from chemenu.search.base import page_key
|
||||
from chemenu.search.fuse import reciprocal_rank_fusion
|
||||
from chemenu.search.ripgrep import build_hit
|
||||
from chemenu.search.types import SearchHit, SearchQuery
|
||||
from chemenu.search.types import SearchHit, SearchQuery, SearchResult
|
||||
|
||||
|
||||
def load_pages_by_path(kb_dir: Path | None = None, root: Path | None = None) -> dict[str, Page]:
|
||||
@@ -92,8 +92,13 @@ def run_search(
|
||||
pages: dict[str, Page],
|
||||
backends: list,
|
||||
kb_dir: Path | None = None,
|
||||
) -> list[SearchHit]:
|
||||
"""Answer a query. Pure: no I/O beyond whatever a backend does."""
|
||||
) -> SearchResult:
|
||||
"""Answer a query. Pure: no I/O beyond whatever a backend does.
|
||||
|
||||
Returns the truncated hits *and* the number there were before the limit,
|
||||
because the caller cannot recover the second from the first - see
|
||||
`SearchResult`.
|
||||
"""
|
||||
filters.validate_fields(query.predicates, pages)
|
||||
|
||||
if query.text:
|
||||
@@ -110,4 +115,9 @@ def run_search(
|
||||
hits.sort(key=lambda h: h.title.lower())
|
||||
|
||||
hits = sort_hits(hits, query.sort)
|
||||
return hits[: query.limit] if query.limit else hits
|
||||
total = len(hits)
|
||||
return SearchResult(
|
||||
hits=hits[: query.limit] if query.limit else hits,
|
||||
total=total,
|
||||
limit=query.limit,
|
||||
)
|
||||
|
||||
@@ -4,6 +4,18 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Optional
|
||||
|
||||
# How many hits a caller gets when it asks for no particular number. Defined
|
||||
# once, here, because three adapters offer the same knob - the CLI's `--limit`,
|
||||
# `api.search(limit=...)` and the MCP `search` tool - and three literals is how
|
||||
# they start disagreeing about what "the default search" returns.
|
||||
#
|
||||
# 50 rather than a smaller number because the queries that actually hit the cap
|
||||
# are the *structured* sweeps (`--field '!sources'`), which are ordered
|
||||
# alphabetically rather than by relevance: truncating those throws away an
|
||||
# arbitrary slice of the answer rather than its weakest tail. A capped result
|
||||
# is only safe at all because it now says so - see `SearchResult.truncated`.
|
||||
DEFAULT_LIMIT = 50
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Predicate:
|
||||
@@ -30,7 +42,7 @@ class SearchQuery:
|
||||
text: Optional[str] = None
|
||||
predicates: tuple[Predicate, ...] = ()
|
||||
regex: bool = False
|
||||
limit: int = 20
|
||||
limit: int = DEFAULT_LIMIT
|
||||
sort: Optional[str] = None
|
||||
|
||||
|
||||
@@ -77,3 +89,34 @@ class SearchHit:
|
||||
"backend": self.backend,
|
||||
"matches": [m.as_dict() for m in self.matches],
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SearchResult:
|
||||
"""The hits a caller gets back, plus how many there were before the limit.
|
||||
|
||||
`run_search` used to return the truncated list alone, which made the
|
||||
truncation impossible to report: every adapter counted `len(hits)` and
|
||||
printed it as the answer, so `20 result(s).` on a query matching 182 pages
|
||||
was indistinguishable from a query that really matched twenty. That is a
|
||||
completeness claim none of them were in a position to make, and the only
|
||||
way to find out was to ask again with `--limit 0` - a second full search to
|
||||
learn a number the first one already knew.
|
||||
|
||||
So the total travels with the hits. Nothing here decides how to say it;
|
||||
that belongs to each adapter (`render_table`, the `--json` payload,
|
||||
`api.search`).
|
||||
"""
|
||||
|
||||
hits: list[SearchHit]
|
||||
total: int
|
||||
limit: int
|
||||
|
||||
@property
|
||||
def truncated(self) -> bool:
|
||||
"""Whether the limit actually cut something off.
|
||||
|
||||
`limit=0` means "no limit", so it never truncates however large the
|
||||
corpus is.
|
||||
"""
|
||||
return bool(self.limit) and self.total > len(self.hits)
|
||||
|
||||
@@ -47,6 +47,21 @@ def test_a_corpus_can_be_named_and_is_the_one_that_answers(foreign_corpus):
|
||||
assert result["results"][0]["path"] == "kb/entities/Peregrine.md"
|
||||
|
||||
|
||||
def test_a_capped_answer_says_how_much_it_left_out(foreign_corpus):
|
||||
"""`count` keeps meaning "what is in this payload", so a consumer written
|
||||
against the old shape reads the number it always did; `total` is what it
|
||||
could not ask before, and without it a truncated answer is indistinguishable
|
||||
from a complete one."""
|
||||
corpus = Corpus(foreign_corpus)
|
||||
|
||||
whole = corpus.search("Peregrine")
|
||||
assert (whole["count"], whole["total"], whole["truncated"]) == (1, 1, False)
|
||||
assert whole["count"] == len(whole["results"])
|
||||
|
||||
capped = corpus.search("Peregrine", limit=0)
|
||||
assert capped["truncated"] is False, "limit 0 means no limit, so it caps nothing"
|
||||
|
||||
|
||||
def test_no_path_of_this_checkout_is_read_while_a_foreign_root_is_set(foreign_corpus):
|
||||
"""The acceptance criterion, asserted rather than argued.
|
||||
|
||||
|
||||
@@ -125,6 +125,24 @@ def test_the_four_tools_are_there_and_nothing_that_writes(corpus):
|
||||
assert names.isdisjoint({"new", "touch", "xref", "cite", "publish", "migrate", "rm"})
|
||||
|
||||
|
||||
def test_all_three_adapters_offer_the_same_default_limit(corpus):
|
||||
"""The CLI's `--limit`, `api.search(limit=...)` and this tool are three
|
||||
knobs on one search, and three literals is how they start disagreeing about
|
||||
what "the default search" returns. They read one constant."""
|
||||
import inspect
|
||||
|
||||
from chemenu.api import Corpus
|
||||
from chemenu.commands.search import search_command
|
||||
from chemenu.search.types import DEFAULT_LIMIT
|
||||
|
||||
server = build_server(corpus, check_trace=False)
|
||||
tool = next(t for t in asyncio.run(server.list_tools()) if t.name == "search")
|
||||
|
||||
assert tool.input_schema["properties"]["limit"]["default"] == DEFAULT_LIMIT
|
||||
assert inspect.signature(Corpus.search).parameters["limit"].default == DEFAULT_LIMIT
|
||||
assert inspect.signature(search_command).parameters["limit"].default.default == DEFAULT_LIMIT
|
||||
|
||||
|
||||
def test_no_tool_writes_anything_into_the_corpus_or_git(corpus):
|
||||
server = build_server(corpus, check_trace=False)
|
||||
before = _tree(corpus)
|
||||
@@ -176,7 +194,8 @@ def test_the_wire_format_is_the_clis_json_form(corpus):
|
||||
# `generated` is the CLI's date stamp and `commit`/`as_of` are the server's
|
||||
# revision stamp - two answers to "when", neither of them a finding. What
|
||||
# has to match is everything that describes the *corpus*.
|
||||
shared = ("query", "predicates", "backend", "count", "results", "unreadable")
|
||||
shared = ("query", "predicates", "backend", "count", "total", "truncated", "limit",
|
||||
"results", "unreadable")
|
||||
assert {key: served[key] for key in shared} == {key: from_cli[key] for key in shared}
|
||||
|
||||
|
||||
|
||||
@@ -17,7 +17,14 @@ from chemenu.search.filters import PredicateError, parse_predicate
|
||||
from chemenu.search.fuse import reciprocal_rank_fusion
|
||||
from chemenu.search.registry import UnknownBackend, resolve
|
||||
from chemenu.search.ripgrep import RipgrepBackend, build_argv
|
||||
from chemenu.search.types import Match, Predicate, SearchHit, SearchQuery
|
||||
from chemenu.search.types import (
|
||||
DEFAULT_LIMIT,
|
||||
Match,
|
||||
Predicate,
|
||||
SearchHit,
|
||||
SearchQuery,
|
||||
SearchResult,
|
||||
)
|
||||
from chemenu.frontmatter_io import write_page
|
||||
|
||||
|
||||
@@ -32,8 +39,12 @@ def backend(kb_dir: Path, tmp_path: Path):
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def search(kb_dir: Path, pages):
|
||||
"""Run a query against the fixture kb rather than the real one."""
|
||||
def search_result(kb_dir: Path, pages):
|
||||
"""Run a query against the fixture kb and return the whole `SearchResult`.
|
||||
|
||||
For the tests that care about the limit and the total; most only want the
|
||||
hits, and use `search` below.
|
||||
"""
|
||||
|
||||
def _search(query: SearchQuery, backends=()):
|
||||
return run_search(query, pages, list(backends), kb_dir)
|
||||
@@ -41,6 +52,16 @@ def search(kb_dir: Path, pages):
|
||||
return _search
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def search(search_result):
|
||||
"""The hits alone, for tests whose subject is ranking or filtering."""
|
||||
|
||||
def _search(query: SearchQuery, backends=()):
|
||||
return search_result(query, backends).hits
|
||||
|
||||
return _search
|
||||
|
||||
|
||||
def _titles(hits):
|
||||
return [hit.title for hit in hits]
|
||||
|
||||
@@ -211,6 +232,20 @@ def test_limit_and_sort(search):
|
||||
assert len(search(_q("kind=entity", limit=2))) == 2
|
||||
|
||||
|
||||
def test_the_total_survives_the_limit_so_no_second_search_is_needed(search_result):
|
||||
"""The caller cannot recover the total from a truncated list, and asking
|
||||
again with `--limit 0` is a second full search to learn a number the first
|
||||
one already had."""
|
||||
capped = search_result(_q("kind=entity", limit=2))
|
||||
|
||||
assert len(capped.hits) == 2
|
||||
assert capped.total == 3
|
||||
assert capped.truncated is True
|
||||
|
||||
whole = search_result(_q("kind=entity"))
|
||||
assert (whole.total, whole.truncated) == (3, False)
|
||||
|
||||
|
||||
def test_sort_puts_missing_values_last():
|
||||
hits = [
|
||||
SearchHit(title="b", path="b", modified=None),
|
||||
@@ -240,18 +275,79 @@ def test_resolve_defaults_to_rg_and_rejects_unknown():
|
||||
# --- output -----------------------------------------------------------------
|
||||
|
||||
|
||||
def _result(hits, total=None, limit=DEFAULT_LIMIT):
|
||||
return SearchResult(hits=hits, total=total if total is not None else len(hits), limit=limit)
|
||||
|
||||
|
||||
def test_render_table_is_compact_and_reports_the_count():
|
||||
hit = SearchHit(title="aurora", path="kb/x.md", kind="entity", subtype="system",
|
||||
summary="Server hosting DocStore", score=8.0, matches=[Match(3, "DocStore")])
|
||||
out = render_table([hit], show_matches=False)
|
||||
out = render_table(_result([hit]), show_matches=False)
|
||||
assert "aurora" in out and "entity/system" in out
|
||||
assert "kb/x.md:3" not in out
|
||||
assert "1 result(s)." in out
|
||||
assert "kb/x.md:3" in render_table([hit], show_matches=True)
|
||||
assert "kb/x.md:3" in render_table(_result([hit]), show_matches=True)
|
||||
|
||||
|
||||
def test_render_table_says_so_when_nothing_matched():
|
||||
assert render_table([], show_matches=False) == "No matches."
|
||||
assert render_table(_result([]), show_matches=False) == "No matches."
|
||||
|
||||
|
||||
def test_every_hit_carries_the_path_so_the_page_can_be_opened():
|
||||
"""The reason this format changed: the instructions tell an agent to read
|
||||
the pages the search points at, and the table pointed nowhere. A session
|
||||
that needed filenames ran `grep -rl` over kb/ for them - a second search
|
||||
that can find no page this one missed."""
|
||||
hit = SearchHit(title="aurora", path="kb/entities/systems/aurora.md", kind="entity")
|
||||
assert "kb/entities/systems/aurora.md" in render_table(_result([hit]), show_matches=False)
|
||||
|
||||
|
||||
def test_title_and_path_are_never_truncated_only_the_summary_is():
|
||||
"""A clipped title is not an identifier. It is what `xref add`, `cite add`
|
||||
and `touch` take as an argument, and the old fixed 34-char column cut four
|
||||
of five hits in the report that prompted this."""
|
||||
title = "Source - Pelletofenkondensator und Verkabelung Recherche"
|
||||
path = f"kb/sources/llm-sessions/{title}.md"
|
||||
hit = SearchHit(title=title, path=path, kind="source", summary="x" * 400)
|
||||
line = render_table(_result([hit]), show_matches=False).splitlines()[0]
|
||||
|
||||
assert title in line
|
||||
assert path in line
|
||||
assert "…" in line, "the summary is still the one lossy field"
|
||||
|
||||
|
||||
def test_a_hit_line_parses_into_its_five_fields_even_with_prose_pipes():
|
||||
"""The separator has to survive a summary that contains one. It does,
|
||||
because the summary is last: a `|` there cannot shift a column. A `|` in a
|
||||
title is impossible - the wikilink syntax reserves it."""
|
||||
hit = SearchHit(title="aurora", path="kb/x.md", kind="entity", subtype="system",
|
||||
summary="Runs `a | b` nightly", score=8.0)
|
||||
line = render_table(_result([hit]), show_matches=False).splitlines()[0]
|
||||
score, kind, title, path, summary = line.split(" | ", 4)
|
||||
|
||||
assert (score, kind, title, path) == ("8.0", "entity/system", "aurora", "kb/x.md")
|
||||
assert summary == "Runs `a | b` nightly"
|
||||
|
||||
|
||||
def test_a_capped_result_says_what_it_capped():
|
||||
"""`20 result(s).` on a query matching 182 pages is a completeness claim
|
||||
the output was in no position to make, and the only way to find the real
|
||||
number was a second, unlimited search."""
|
||||
hits = [SearchHit(title=f"p{i}", path=f"kb/p{i}.md") for i in range(20)]
|
||||
out = render_table(_result(hits, total=182, limit=20), show_matches=False)
|
||||
|
||||
assert "20 of 182 result(s)" in out
|
||||
assert "--limit" in out
|
||||
|
||||
|
||||
def test_an_uncapped_result_claims_nothing_about_a_limit():
|
||||
hits = [SearchHit(title=f"p{i}", path=f"kb/p{i}.md") for i in range(3)]
|
||||
assert "3 result(s)." in render_table(_result(hits), show_matches=False)
|
||||
assert " of " not in render_table(_result(hits), show_matches=False)
|
||||
|
||||
|
||||
def test_limit_zero_never_counts_as_truncated_however_large_the_corpus():
|
||||
assert _result([SearchHit(title="a", path="a")] * 182, total=182, limit=0).truncated is False
|
||||
|
||||
|
||||
def test_hit_serialises_for_json():
|
||||
|
||||
Reference in New Issue
Block a user