tools: Export - include "all" wählt die ganze KB, index: true schreibt index.md und INDEX.md in den Zielordner (#217); 10.1.0-beta.21

Files changed:
- CHANGES.md
- VERSION
- docs/how-to/export-to-a-synced-folder.md
- docs/reference/commands/exports.md
- docs/reference/configuration.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/catalog.py
- tools/chemenu/commands/export_cmd.py
- tools/chemenu/commands/index_build.py
- tools/chemenu/config_shapes.py
- tools/chemenu/export_index.py
- tools/chemenu/guideline_export.py
- tools/chemenu/page_export.py
- tools/chemenu/tests/test_config_shapes.py
- tools/chemenu/tests/test_export_run.py
- tools/chemenu/tests/test_index_build.py

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2

docs/how-to/export-to-a-synced-folder.md

@@ -20,10 +20,17 @@ look like: [configuration reference](../reference/configuration.md#export-to-a-f
 
 ## Pick the pages and the format
 
-- **Which pages.** Only what a target's `include` selects is written; there is no "everything". A
-  selection is a list of `search` predicates that must all hold - `tags=shared`, `kind=system`,
-  `!draft` - and a page that fits any selection of the target is exported. Try a selection with
-  `tools/wikitool search --field tags=shared` first.
+- **Which pages.** Only what a target's `include` selects is written. `"include": "all"` selects
+  every page, also one that lies directly under `kb/`, and no generated file. Otherwise a selection
+  is a list of `search` predicates that must all hold - `tags=shared`, `kind=system`, `!draft` - and
+  a page that fits any selection of the target is exported: `"include": [["tags=shared"]]`. Try a
+  selection with `tools/wikitool search --field tags=shared` first. A missing or empty `include` is
+  an error, not "all". With `"all"`, every page you add later goes into the export too - on a target
+  with `after_publish` at the next `publish`.
+- **An entry point.** The folder holds the pages only. `"index": true` adds `index.md` in the root
+  of the folder and an `INDEX.md` per collection, so someone who opens the folder finds the
+  collections, their areas and the pages with their summaries by following links. The files list what
+  the target exports and nothing else, and the link text is the page's title.
 - **Which format.** Two ways, equally valid:
   - `markdown` writes the pages as `.md` files. It needs nothing but the folder.
   - `command` hands those files to a converter of your own and exports what it produces - Word
@@ -69,6 +76,10 @@ look like: [configuration reference](../reference/configuration.md#export-to-a-f
    (the default) the title, type, summary and modified date, `full` the page's whole frontmatter.
    `links: relative` (the default) turns a link to an exported page into a relative Markdown link;
    `links: text` leaves the page's name. A link to a page that is not exported is always plain text.
+   The index files link relatively whatever `links` says.
+
+   To put the whole of `kb/` into the folder with an entry point, write `"include": "all"` and
+   `"index": true` instead of the `include` selection above.
 3. **Tell the agent:**
 
    > Export the pages.
@@ -84,8 +95,8 @@ every file names the commit its page comes from. Publish first.
 
 ## What a run does to the folder
 
-The folder holds the pages and one file of the stack's own, `.wikitool-export.json`, which lists
-what the last run wrote.
+The folder holds the pages, the index files if the target asks for them, and one file of the stack's
+own, `.wikitool-export.json`, which lists what the last run wrote.
 
 - **Only its own files.** A run removes a file only when it is listed and no longer exported, and a
   folder only when the run created it. Anything else in the folder stays as it is.
@@ -102,6 +113,11 @@ what the last run wrote.
   A file of a page the commit did not touch changes only when its text does: a link to a page that
   has just entered the selection becomes a relative link, one to a page that left it plain text.
   The first run after updating to a version that brought this rewrites every file once.
+- **An index follows the pages.** A collection's `INDEX.md` changes when one of its pages does, and
+  `index.md`, which lists them all, with every commit to an exported page. Setting `index` back to
+  `false` removes the index files at the next run and leaves every other file as it is. A page
+  `index.md` in a collection, next to the generated `INDEX.md`, stops the target: a synced folder on
+  Windows would make them one file.
 - **A growing selection needs no confirmation.** New pages are reported as new; read the list
   before the sync client uploads them.
 
@@ -150,7 +166,8 @@ program's stderr. The staging directory is always removed.
 ## What does not come back in
 
 An export is never a source. Every Markdown file starts with a frontmatter block whose first key is
-`wikitool_export`, and `tools/wikitool raw accept` and `raw capture` refuse a file that carries it.
+`wikitool_export` - the index files too - and `tools/wikitool raw accept` and `raw capture` refuse a
+file that carries it.
 A converter may lose that key, so `raw accept` also refuses any file in `incoming/` that is byte
 for byte one of the files in the manifest of a configured target - and warns, without refusing,
 when a target has no manifest yet or the `export` section is broken.

docs/reference/commands/exports.md

@@ -109,7 +109,8 @@ tools/wikitool export run work-notes
 
 **Behaviour**
 
-- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with an allowlist of `search --field` predicates (`include`: a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). Nothing is exported that no selection names.
+- An export is derived: regenerated from `kb/` on every run, never edited, never read back into the wiki. Each target names its pages with `include`: either the string `"all"` (every page `search` knows, also one directly under `kb/`; never a generated or meta file) or an allowlist of `search --field` predicates (a list of selections, AND within one, OR across them), a handler (`markdown`, or `command` for the operator's converter) and the folder (`path`, outside the instance). A missing or empty `include` is an error, never "all": nothing is exported that the target does not name.
+- With `index: true` a target also gets `index.md` in its root and an `INDEX.md` per collection (and per area the catalog gives a file of its own), built from the exported pages alone: collections and areas with their counts, then tables of page, type, summary and last modified, with relative Markdown links whatever `links` says. No date of the day and no hint for an agent; the marker is `kind=index` with the newest commit of the pages the file lists, so a commit to one collection rewrites that collection's `INDEX.md` and `index.md` only. The files are ordinary files of the run (manifest, atomic write, removed once no longer generated); the `command` handler receives them with the pages. A page `<collection>/index.md` beside the generated `INDEX.md` stops the target.
 - Every Markdown file starts with a frontmatter block whose `wikitool_export` line says `kind=pages instance=<origin URL without userinfo, or local> commit=<newest commit touching this page>`, followed by `title`, `type`, `summary`, `modified` where the page has them (`frontmatter: summary`, the default; `minimal` keeps only the marker, `full` the whole frontmatter), then the page text. A file keeps its path below `kb/`. A wikilink to an exported page becomes a relative Markdown link (`links: relative`, the default) or plain text (`links: text`); a link to a page that is not exported is always plain text. Footnote markers stay; code is never rewritten. LF endings, no timestamp: the same pages give the same bytes, and a commit that touches one page rewrites that page's file alone - plus any file whose links change because a page entered or left the selection. The manifest's `commit` is the newest commit touching any selected page; all of them come from one `git log`, however many pages are selected.
 - Handler `command`: the Markdown is rendered into a staging directory, the program of `export.commands` runs with no shell, stdin closed and a timeout of 300 s, and receives the Markdown directory and an empty output directory as its last two arguments. On exit 0 the regular files in the output directory (recursively) become the target's content; any other end leaves the target unchanged and prints the end of its stderr. The staging directory is always removed.
 - The folder is only ever touched by what it holds in its manifest, `.wikitool-export.json`: the files an earlier run wrote, each with its sha256. A run removes only manifest files that are no longer exported (and folders it created, once empty), rewrites a manifest file that is missing, leaves every other file byte for byte as it was, and writes nothing - not even a modification time - for a file that did not change. A run that changes nothing writes nothing at all.
@@ -132,7 +133,7 @@ tools/wikitool export run work-notes
 - `0` - success
 - `1` - No `export` section, an unknown target name, or a broken section (a predicate that does not parse, a path inside the instance or in the way of another target, an `intake` folder or a folder the backup writes into: `<backup.target>/git` or this machine's folder)
 - `1` - `kb/` or `types/` has uncommitted changes, the instance is not a git checkout, or a page under `kb/` has unreadable frontmatter
-- `1` - A target failed: a foreign folder, a folder in backup data, a file changed in the target, an empty selection, an unknown field, a converter that did not exit 0
+- `1` - A target failed: a foreign folder, a folder in backup data, a file changed in the target, an empty selection, an unknown field, two files that differ only in case (also a page and a generated index file), a converter that did not exit 0
 
 **See also**
 

docs/reference/configuration.md

@@ -644,7 +644,7 @@ them into a staging directory and hands that to a converter of this checkout.
 <!-- wikitool:reference-config-export -->
 | Field | Tier | Type | Required | Default | Description |
 |-------|------|------|----------|---------|-------------|
-| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target or an `intake` folder, and not be, lie in or contain `<backup.target>/git` or `<backup.target>/<machine>`, nor lie in any other machine's folder or a bare repository) and `include` (an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported; there is no "everything"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
+| `targets` | checkout, gate | object | yes | - | Target name (lower-case letters, digits and hyphens) to target; the name is the argument of `export run`. Every target has `handler` (`markdown`: the pages as Markdown files; `command`: the same Markdown run through a converter of this checkout), `path` (an absolute path outside the instance; it must not lie in or contain another target or an `intake` folder, and not be, lie in or contain `<backup.target>/git` or `<backup.target>/<machine>`, nor lie in any other machine's folder or a bare repository) and `include` (the string `"all"`: every page `search` knows, a page directly under `kb/` included and no generated or meta file; or an array of selections, each an array of `search --field` predicates: all predicates of a selection must hold, and a page that fits any selection is exported. A missing or empty `include` is an error, never "all"). Optional: `frontmatter` (`minimal`: only the export marker; `summary`, the default: also `title`, `type`, `summary` and `modified` where the page has them; `full`: the page's whole frontmatter), `links` (`relative`, the default: a wikilink to an exported page becomes a relative Markdown link; `text`: plain text; a link to a page that is not exported is always plain text), `index` (default false; true: also write `index.md` in the target's root and an `INDEX.md` per collection, and per area the catalog gives a file of its own, listing the exported pages only, with relative Markdown links whatever `links` says) and `after_publish` (default false: refresh this target after every `publish` that committed). The handler `command` takes `command` (required: the name of an entry of `commands`). A field the handler does not take is an error. |
 | `commands` | checkout, names a command | object | no | `{}` | Command name to command: an array of the program and its arguments, run by the `command` handler without a shell, with stdin closed and a timeout of 300 s. The last two arguments appended are the directory holding the Markdown and an empty output directory. Exit 0 makes the regular files in the output directory the target's content; anything else leaves the target unchanged and prints the end of the program's stderr. |
 
 - **Absent:** No export: `export run` stops with exit 1 naming the section, `publish` exports nothing and `doctor` reports `OK`.
@@ -672,6 +672,12 @@ Example:
         "links": "relative",
         "after_publish": true
       },
+      "whole-kb": {
+        "handler": "markdown",
+        "path": "C:\\Users\\<user>\\OneDrive - <company>\\Chemenu-KB",
+        "include": "all",
+        "index": true
+      },
       "as-word": {
         "handler": "command",
         "command": "pandoc-docx",
@@ -696,12 +702,24 @@ Example:
 ```
 <!-- /wikitool:reference-config-export -->
 
-Within one target, `include` is a list of selections, each a list of `search` predicates (`tags=docs`,
-`entity_type=system`, `!source_url`): a page is exported when it satisfies every predicate of at least
-one selection. Only what `include` selects is written - there is no way to exclude by default. The
-syntax is checked when the section is read, the field names against the corpus when `export run`
-runs. `path` must not lie in or contain the instance, another target, an `intake` folder or
-`backup.target`.
+Within one target, `include` is either the string `"all"` or a list of selections, each a list of
+`search` predicates (`tags=docs`, `entity_type=system`, `!source_url`): a page is exported when it
+satisfies every predicate of at least one selection. `"all"` selects every page `search` knows without
+a predicate - one directly under `kb/` too, never a generated or meta file (`kb/index.md`, `kb/log.md`,
+`kb/provenance.md`, `kb/CONTRACT.md`, `kb/CONVENTIONS.md`, an `INDEX.md`, a `COLLECTION.md`). Only what
+`include` selects is written - there is no way to exclude by default, and a missing or empty `include`
+is an error, never "all". The syntax is checked when the section is read, the field names against the
+corpus when `export run` runs. `path` must not lie in or contain the instance, another target, an
+`intake` folder or `backup.target`.
+
+With `"index": true` a run also writes `index.md` in the root of the target and an `INDEX.md` per
+collection - and per area the catalog under `kb/` would give a file of its own. They list the exported
+pages only (Page, Type, Summary, Last Modified), link with relative Markdown links whatever `links`
+says, and carry no date of the day: Last Modified is the page's own `modified`, `date` or `created`,
+empty where it has none. The column and heading labels are English. Each starts with a frontmatter
+block whose only key is `wikitool_export` (`kind=index`), so `raw accept` refuses it like a page. A
+page `<collection>/index.md` beside the generated `<collection>/INDEX.md` stops the target, since a
+synced folder on Windows would make them one file.
 
 Every Markdown file starts with a frontmatter block whose first key is `wikitool_export`, carrying
 the kind, the instance and the newest commit that touched that page. `frontmatter` decides what follows: