kb/concepts/ bekommt Areas: layout: fuer concept, Area-Titel aus jedem Type-Spec, Schwellen-Empfehlung im lint (schliesst #59)
Files changed: - CHANGES.md - README.md - VERSION - kb/concepts/Ambient Environment Dependency.md - kb/concepts/Anti-Cramming Heuristic.md - kb/concepts/Audit Trail.md - kb/concepts/BM25.md - kb/concepts/Bulk Operations.md - kb/concepts/CI Integration.md - kb/concepts/COLLECTION.md - kb/concepts/CPPC.md - kb/concepts/Checkpoint Audit.md - kb/concepts/Claude Code Auto Mode.md - kb/concepts/Command Round-Trip Integrity.md - kb/concepts/Confidence Scoring.md - kb/concepts/Consolidation Tiers.md - kb/concepts/Content Quality Control.md - kb/concepts/Context Isolation.md - kb/concepts/Contradiction Resolution.md - kb/concepts/Cross-platform Agent Skills.md - kb/concepts/Crystallization.md - kb/concepts/Delete Rather Than Anonymize.md - kb/concepts/Denylist over Allowlist.md - kb/concepts/Detect-Repair Asymmetry.md - kb/concepts/Diff-Reviewable Agent Edits.md - kb/concepts/Dual Licensing by File Plan.md - kb/concepts/Entity Extraction.md - kb/concepts/Episodic Memory.md - kb/concepts/Event-Driven Automation.md - kb/concepts/Filter on Ingest.md - kb/concepts/Forgetting.md - kb/concepts/Graph Traversal.md - kb/concepts/Green Suite Blind Spot.md - kb/concepts/Hooks.md - kb/concepts/Hybrid Search.md - kb/concepts/INDEX.md - kb/concepts/Implementation Spectrum.md - kb/concepts/Index Scaling.md - kb/concepts/Issue Label Scheme.md - kb/concepts/Iteration and Cost Limits.md - kb/concepts/KB Migration.md - kb/concepts/KB Stack Versioning.md - kb/concepts/Knowledge Compounding.md - kb/concepts/Knowledge Graph.md - kb/concepts/LLM Wiki Pattern.md - kb/concepts/Lint Workflow.md - kb/concepts/MCP-Leseserver.md - kb/concepts/Mass-Update Gate.md - kb/concepts/Memory Lifecycle.md - kb/concepts/Mesh Sync.md - kb/concepts/Modbus.md - kb/concepts/Multi-Agent Collaboration.md - kb/concepts/Naming Convention Conflict.md - kb/concepts/OKF Compatibility.md - kb/concepts/Optional Instance Context File.md - kb/concepts/Personalization Plane.md - kb/concepts/Privacy and Governance.md - kb/concepts/Procedural Memory.md - kb/concepts/Publish-Remote Gate.md - kb/concepts/Quality Scoring.md - kb/concepts/Quality and Self-Correction.md - kb/concepts/RAG.md - kb/concepts/Reciprocal Rank Fusion.md - kb/concepts/SSD TRIM.md - kb/concepts/Scale Ceiling.md - kb/concepts/Self-Healing.md - kb/concepts/Semantic Lint Automation.md - kb/concepts/Semantic Memory.md - kb/concepts/Session Orientation.md - kb/concepts/Shared vs Private.md - kb/concepts/Split Merge Reclassify.md - kb/concepts/Split Threshold.md - kb/concepts/Structural Enforcement over Documented Rule.md - kb/concepts/Stub Threshold.md - kb/concepts/Supersession.md - kb/concepts/Three-Layer Architecture.md - kb/concepts/Token Economics.md - kb/concepts/Typed Relationships.md - kb/concepts/User Management.md - kb/concepts/Vector Search.md - kb/concepts/Work Coordination.md - kb/concepts/Workflow Extraction.md - kb/concepts/Workflow Orchestration.md - kb/concepts/Working Memory.md - kb/concepts/Write-Once Frontmatter Fields.md - kb/concepts/architectures/Consolidation Tiers.md - kb/concepts/architectures/Context Isolation.md - kb/concepts/architectures/Cross-platform Agent Skills.md - kb/concepts/architectures/Episodic Memory.md - kb/concepts/architectures/Hybrid Search.md - kb/concepts/architectures/Implementation Spectrum.md - kb/concepts/architectures/Knowledge Graph.md - kb/concepts/architectures/LLM Wiki Pattern.md - kb/concepts/architectures/MCP-Leseserver.md - kb/concepts/architectures/Memory Lifecycle.md - kb/concepts/architectures/OKF Compatibility.md - kb/concepts/architectures/Optional Instance Context File.md - kb/concepts/architectures/Personalization Plane.md - kb/concepts/architectures/Procedural Memory.md - kb/concepts/architectures/RAG.md - kb/concepts/architectures/Scale Ceiling.md - kb/concepts/architectures/Semantic Memory.md - kb/concepts/architectures/Three-Layer Architecture.md - kb/concepts/architectures/Token Economics.md - kb/concepts/architectures/Working Memory.md - kb/concepts/decisions/Delete Rather Than Anonymize.md - kb/concepts/decisions/Denylist over Allowlist.md - kb/concepts/decisions/Diff-Reviewable Agent Edits.md - kb/concepts/decisions/Dual Licensing by File Plan.md - kb/concepts/decisions/Issue Label Scheme.md - kb/concepts/decisions/KB Stack Versioning.md - kb/concepts/decisions/Structural Enforcement over Documented Rule.md - kb/concepts/patterns/Audit Trail.md - kb/concepts/patterns/BM25.md - kb/concepts/patterns/Command Round-Trip Integrity.md - kb/concepts/patterns/Confidence Scoring.md - kb/concepts/patterns/Contradiction Resolution.md - kb/concepts/patterns/Entity Extraction.md - kb/concepts/patterns/Filter on Ingest.md - kb/concepts/patterns/Forgetting.md - kb/concepts/patterns/Graph Traversal.md - kb/concepts/patterns/Mesh Sync.md - kb/concepts/patterns/Quality Scoring.md - kb/concepts/patterns/Reciprocal Rank Fusion.md - kb/concepts/patterns/Self-Healing.md - kb/concepts/patterns/Shared vs Private.md - kb/concepts/patterns/Typed Relationships.md - kb/concepts/patterns/Vector Search.md - kb/concepts/patterns/Work Coordination.md - kb/concepts/problems/Ambient Environment Dependency.md - kb/concepts/problems/Detect-Repair Asymmetry.md - kb/concepts/problems/Green Suite Blind Spot.md - kb/concepts/problems/Naming Convention Conflict.md - kb/concepts/problems/Write-Once Frontmatter Fields.md - kb/concepts/protocols/CPPC.md - kb/concepts/protocols/Modbus.md - kb/concepts/protocols/SSD TRIM.md - kb/concepts/workflows/Anti-Cramming Heuristic.md - kb/concepts/workflows/Bulk Operations.md - kb/concepts/workflows/CI Integration.md - kb/concepts/workflows/Checkpoint Audit.md - kb/concepts/workflows/Claude Code Auto Mode.md - kb/concepts/workflows/Content Quality Control.md - kb/concepts/workflows/Crystallization.md - kb/concepts/workflows/Event-Driven Automation.md - kb/concepts/workflows/Hooks.md - kb/concepts/workflows/Index Scaling.md - kb/concepts/workflows/Iteration and Cost Limits.md - kb/concepts/workflows/KB Migration.md - kb/concepts/workflows/Knowledge Compounding.md - kb/concepts/workflows/Lint Workflow.md - kb/concepts/workflows/Mass-Update Gate.md - kb/concepts/workflows/Multi-Agent Collaboration.md - kb/concepts/workflows/Privacy and Governance.md - kb/concepts/workflows/Publish-Remote Gate.md - kb/concepts/workflows/Quality and Self-Correction.md - kb/concepts/workflows/Semantic Lint Automation.md - kb/concepts/workflows/Session Orientation.md - kb/concepts/workflows/Split Merge Reclassify.md - kb/concepts/workflows/Split Threshold.md - kb/concepts/workflows/Stub Threshold.md - kb/concepts/workflows/Supersession.md - kb/concepts/workflows/User Management.md - kb/concepts/workflows/Workflow Extraction.md - kb/concepts/workflows/Workflow Orchestration.md - kb/index.md - kb/log.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/catalog.py - tools/chemenu/commands/index_build.py - tools/chemenu/lint_core.py - tools/chemenu/tests/conftest.py - tools/chemenu/tests/test_cite_cmd.py - tools/chemenu/tests/test_git_publish.py - tools/chemenu/tests/test_index_build.py - tools/chemenu/tests/test_lint.py - tools/chemenu/tests/test_new_page.py - tools/chemenu/tests/test_provenance.py - tools/chemenu/tests/test_type_resolver.py - tools/chemenu/tests/test_xref.py - types/concept.md - types/type-spec.md
This commit is contained in:
@@ -16,7 +16,10 @@ from __future__ import annotations
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from chemenu import blocks, config, kb_collections, links
|
||||
from chemenu.catalog import SHARD_THRESHOLD, group_pages
|
||||
from chemenu.frontmatter_io import frontmatter_error
|
||||
from chemenu.markdown_code import strip_code_spans
|
||||
from chemenu.provenance import broken_raw_refs as find_broken_raw_refs
|
||||
@@ -141,6 +144,88 @@ def nested_pages(kb_dir: Path, pages: dict[str, Page]) -> list[dict]:
|
||||
]
|
||||
|
||||
|
||||
def unsharded_collections(kb_dir: Path, pages: dict[str, Page]) -> list[dict]:
|
||||
"""Collections past the catalog's shard threshold that have no areas to
|
||||
shard, together with the subtype split that would give them some.
|
||||
|
||||
Sharding is already automatic, and it is per *area*: `index rebuild` hands
|
||||
an area over `SHARD_THRESHOLD` rows its own `INDEX.md`. Creating an area is
|
||||
not automatic and nothing ever asked for one - so a collection that never
|
||||
grew any keeps its whole catalog in a single table, past the threshold,
|
||||
forever. The threshold is then not a threshold but a dead value (Gitea
|
||||
#59), and this is the only check that can notice: an ingest sees one
|
||||
source and cannot see a collection's size, while `lint` sees the corpus and
|
||||
runs every 10 sources anyway.
|
||||
|
||||
**A recommendation, not a failure** (it is deliberately absent from
|
||||
`HARD_ERROR_KEYS`), and narrow enough to stay one: it fires only where the
|
||||
split actually helps - every area it would create, the ungrouped remainder
|
||||
included, lands at or under the threshold. That self-limits in both
|
||||
directions. A collection under the threshold never fires, so a small
|
||||
`kb/comparisons/` is not permanently in the report; and a collection whose
|
||||
subtype values are lopsided (25 of 29 `source_type: notes`) does not fire
|
||||
either, because splitting it would produce one area over the threshold and
|
||||
a handful of splinters. What is left is a finding that appears when a
|
||||
collection grows into it and is silent when it does not.
|
||||
"""
|
||||
findings: list[dict] = []
|
||||
for collection in group_pages(kb_dir, pages):
|
||||
if collection.count <= SHARD_THRESHOLD:
|
||||
continue
|
||||
# An area already exists, so the collection has been split once and
|
||||
# `index rebuild` shards whatever outgrows the threshold from here.
|
||||
# A page still sitting in the root is `misplaced_pages`' finding, not
|
||||
# this one.
|
||||
if any(area.name for area in collection.areas):
|
||||
continue
|
||||
|
||||
counts: Counter[str] = Counter()
|
||||
fields: set[str] = set()
|
||||
# Whether every type writing here already declares the `layout:` that
|
||||
# turns the subtype into a directory. It decides which half of the fix
|
||||
# is still owed: without it there is nothing for `move` to compute a
|
||||
# destination from, with it the move is all that is left.
|
||||
layouts: set[bool] = set()
|
||||
for area in collection.areas:
|
||||
for page in area.pages:
|
||||
type_path = page.frontmatter.get("type")
|
||||
if not type_path:
|
||||
continue
|
||||
try:
|
||||
field = resolver.get_subtype_field(type_path, page.path)
|
||||
layout = resolver.get_layout(type_path, page.path)
|
||||
except ValueError:
|
||||
continue
|
||||
value = page.frontmatter.get(field) if field else None
|
||||
if not value:
|
||||
continue
|
||||
counts[str(value)] += 1
|
||||
fields.add(str(field))
|
||||
layouts.add(bool(layout))
|
||||
|
||||
if not counts:
|
||||
continue
|
||||
# The pages the subtype cannot place stay in the collection root, so
|
||||
# they are an area of their own for the purpose of this test.
|
||||
unplaced = collection.count - sum(counts.values())
|
||||
if max([*counts.values(), unplaced]) > SHARD_THRESHOLD:
|
||||
continue
|
||||
|
||||
findings.append(
|
||||
{
|
||||
"collection": collection.name,
|
||||
"count": collection.count,
|
||||
"field": ", ".join(sorted(fields)),
|
||||
"layout_declared": layouts == {True},
|
||||
"distribution": [
|
||||
{"value": value, "count": count}
|
||||
for value, count in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
],
|
||||
}
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
def run_lint(kb_dir: Path) -> dict:
|
||||
pages = load_kb_pages(kb_dir)
|
||||
duplicate_titles = find_duplicate_title_paths(kb_dir, config.ROOT)
|
||||
@@ -388,6 +473,7 @@ def run_lint(kb_dir: Path) -> dict:
|
||||
"duplicate_titles": duplicate_titles,
|
||||
"misplaced_pages": misplaced,
|
||||
"nested_pages": nested,
|
||||
"unsharded_collections": unsharded_collections(kb_dir, pages),
|
||||
"uncovered_raw_files": find_uncovered_raw_files(config.RAW_DIR, pages),
|
||||
"broken_raw_refs": find_broken_raw_refs(pages),
|
||||
"duplicate_raw_file_owners": find_duplicate_raw_file_owners(pages),
|
||||
@@ -464,6 +550,23 @@ def render_markdown(report: dict) -> str:
|
||||
"the catalog folds this into its area silently; `wikitool move --reconcile` fixes it "
|
||||
"when the page's type resolves to a shallower directory, otherwise move it up by hand",
|
||||
)
|
||||
_section(
|
||||
lines, f"Collections Past the Shard Threshold (>{SHARD_THRESHOLD}) With No Areas "
|
||||
"- recommendation, not an error",
|
||||
report.get("unsharded_collections", []),
|
||||
lambda i: f"`kb/{i['collection']}/` holds {i['count']} pages in a single table and has no "
|
||||
f"areas, so the per-area shard threshold never fires. Splitting on `{i['field']}` would "
|
||||
"give: "
|
||||
+ ", ".join(f"{d['value']} {d['count']}" for d in i["distribution"])
|
||||
+ f" - all at or under {SHARD_THRESHOLD}. "
|
||||
+ (
|
||||
"The type-spec already declares the `layout:` for those values, so "
|
||||
"`wikitool move --reconcile` and `wikitool index rebuild` are the whole fix"
|
||||
if i.get("layout_declared")
|
||||
else "Declare a `layout:` for those values in the type-spec, then "
|
||||
"`wikitool move --reconcile` and `wikitool index rebuild`"
|
||||
),
|
||||
)
|
||||
_section(
|
||||
lines, "Uncovered Raw Files (no source page)", report["uncovered_raw_files"],
|
||||
lambda i: f"`{i}`",
|
||||
@@ -617,6 +720,14 @@ def default_report_path(report: dict) -> Path:
|
||||
# and there is no version at which "not under the computed directory" becomes
|
||||
# wrong - only `wikitool move` someone does or does not get to run.
|
||||
#
|
||||
# `unsharded_collections` is advisory by construction rather than by tolerance:
|
||||
# it does not describe anything that is wrong, only a collection that has grown
|
||||
# past the size at which areas start paying for themselves. Whether to split it
|
||||
# is an authoring decision about how the corpus is organised - the tool can see
|
||||
# that the split would work and say so, and that is the whole of its authority.
|
||||
# Failing on it would also make `lint` red on a corpus that is entirely
|
||||
# self-consistent, which is the state the recommendation is asking to improve.
|
||||
#
|
||||
# `malformed_edges` and `unbalanced_markers` are hard from the start: neither
|
||||
# describes an unconverted page, only a broken one.
|
||||
#
|
||||
|
||||
Reference in New Issue
Block a user