diff --git a/.gitattributes b/.gitattributes index a1ea335..d9df1e0 100644 --- a/.gitattributes +++ b/.gitattributes @@ -11,3 +11,7 @@ # and normalizing a CRLF source on `git add` would change it. /raw/** -text /incoming/** -text + +# Real documents for the `raw derive` tests are compared byte for byte (an .eml with CRLF +# included), so nothing under testfiles/ is ever normalized. +/testfiles/** -text diff --git a/CHANGES.md b/CHANGES.md index 0fd801d..4a9791f 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 9.0.0-beta.17 - 2026-10-08 - raw pending names only the raw fetch pair as a bundle; a derived document is a folder +## 9.0.0-beta.18 - 2026-10-09 - raw derive writes an Excel file's sheets as tab-separated files with a formula list; derived.md becomes the overview (#202) **Author:** Torben Nehmer @@ -47,6 +47,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. - tools/README.md: Asset-Modus beschreibt Entpacken in den eigenen Ordner, Wiederholung und Zurückrollen - Gitea-Wiki als generierter Spiegel von docs/: Konverter, Push-Skript, Workflows - raw pending names only the raw fetch pair as a bundle; a derived document is a folder +- raw derive writes an Excel file's sheets as tab-separated files with a formula list; derived.md becomes the overview (#202) ### Tutorials: installation in English, and a first source to a first answer (#191) diff --git a/VERSION b/VERSION index fe92d2e..f6c3260 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -9.0.0-beta.17 +9.0.0-beta.18 diff --git a/docs/dev/development.md b/docs/dev/development.md index de2437b..a128437 100644 --- a/docs/dev/development.md +++ b/docs/dev/development.md @@ -40,6 +40,9 @@ Then have the agent run `instructions/bootstrap.md`: the preflight, then and how `dist export` runs as a test tool - is [instructions/dev/dev-setup.md](../../instructions/dev/dev-setup.md). +Real documents for `raw derive` live in [testfiles/](../../testfiles/README.md) and are +committed, so CI derives them too. The repository is public: that README says what may go in. + What a clone has and an instance does not: - **The demo corpus.** Some 170 pages documenting the stack itself: gates, lint, versioning, diff --git a/docs/how-to/ingest-a-source.md b/docs/how-to/ingest-a-source.md index fbbd718..ed90ad7 100644 --- a/docs/how-to/ingest-a-source.md +++ b/docs/how-to/ingest-a-source.md @@ -20,7 +20,8 @@ pick a directory: Outlook `.msg`) or a ZIP archive - is put into `incoming/` as it is. `tools/wikitool raw derive incoming/` turns it into a folder with the untouched original and a readable text beside it, with a header saying how it was made. The - folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is + folder is accepted whole; the agent runs this itself when it meets such a file. An Excel file's + sheets land beside the text as tab-separated files, with their formulas listed apart. A ZIP is unpacked into a folder instead, and a mail's attachments land beside it. A scanned PDF has no text to derive - the tool warns, and there is no OCR. - **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index 82be1e3..2b72a69 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -119,6 +119,7 @@ tools/wikitool raw derive incoming/scans.zip - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. +- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. @@ -136,7 +137,7 @@ tools/wikitool raw derive incoming/scans.zip - `0` - success - `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -- `1` - The target folder or a `.md` already exists, or the path would be over the path budget +- `1` - The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget - `1` - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **See also** diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index b02a43d..e283e33 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -289,6 +289,17 @@ output. A name that is taken or reserved (`original.*` and `derived.md` at the t folder, `_capture.json` anywhere) gets `-2`, `-3` before its extension, and the output lists every rename. +**A workbook's cells are files, not text.** An Excel file becomes an overview in `derived.md` - +one section per sheet with its range, its file names and a short preview - and, beside it, a +folder `.sheets/` (`derived.sheets/` for a single file) with one tab-separated file per +sheet. The format is RFC 4180 with a tab as the delimiter, so a cell holding a tab, a quote or a +line break is quoted and a reader gets every cell back unchanged; row *n* of the file is row *n* +of the sheet, so a cell address in the file is the address in Excel. A sheet with formulas gets +a second file listing each cell, its formula and the value Excel last stored: the tool reads +the file and does not recalculate it. Text that is data is read by a script, not by a model's +eye; the overview is what tells the reader which file to open. The folder is part of the +derivation, accepted with it, and an existing one is never overwritten. + **The original stays.** The derivation is the tool's reading of it, never the source: with the original kept, a claim in `kb/` can still be checked against what the document said when a derivation dropped something, and a later version of the tool can derive it better. Both go into diff --git a/testfiles/README.md b/testfiles/README.md new file mode 100644 index 0000000..d475d37 --- /dev/null +++ b/testfiles/README.md @@ -0,0 +1,47 @@ +# testfiles + +Real documents (PDF, mail, Office, ZIP) that CI feeds to `raw derive`, beyond the synthetic +fixtures in `tools/chemenu/tests/binary_formats.py`. `tools/chemenu/tests/test_raw_derive_testfiles.py` +picks up every file under this directory, so adding a file is adding a test case - no code change. +The directory exists only in a development clone: `dist export` does not ship it. + +## What goes in + +**Everything here is committed to a public repository.** A file belongs here only when it can be +public: + +- a document you wrote for the purpose, or one whose licence allows redistribution; +- no real people, customers, employers, addresses or internal URLs - in the text *and* in the + metadata (PDF author and producer, Office author and company, mail headers, ZIP entry names); +- nothing that was received in confidence. + +Removing a file later does not remove it from the history. + +## Layout + +| Folder | For | +|--------|-----| +| `pdf/` | PDFs, including scanned ones without a text layer | +| `mail/` | `.eml` and `.msg`, with and without attachments, with attached mails | +| `office/` | `.docx`, `.pptx`, `.xlsx` (tracked changes, comments, hidden sheets) | +| `zip/` | archives, including mail-in-ZIP cases | + +The folders are a convention; the test reads every file below this directory. Keep file names +short and free of spaces - a name that is too long for the `raw/` path budget fails the test for +the right reason, but it is a nuisance to find. + +## What the test checks + +For each file, in a throwaway instance: `raw derive` succeeds; the original comes back byte for +byte (a ZIP is unpacked and gets a `.md` beside its documents instead); a second run over the same +bytes gives the same text apart from `derived:`; and `raw accept` takes the folder. + +A file may carry a sidecar `.expect` (`offer.pdf.expect` for `offer.pdf`): one line per +piece of text that must appear in the derived text, empty lines and lines starting with `#` +ignored. Without one the file only has to derive, be stable and be accepted. A document that is +meant to be refused (encrypted, damaged) does not belong here; the synthetic tests cover refusals. + +## Trying a file by hand + +`raw derive` moves what it is given, so work on a copy in a throwaway instance, never on +`incoming/` of this clone. diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index f8f8a97..53d84e2 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -1495,13 +1495,13 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, - 0 success - 1 The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -- 1 The target folder or a `.md` already exists, or the path would be over the path budget +- 1 The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget - 1 The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **ON FAILURE** - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -> Nothing was written. Fix the argument and retry once - a file in a subfolder is moved up, or its folder is derived whole -- The target folder or a `.md` already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name `, then retry once. An existing `.md` is never overwritten +- The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name `, then retry once. An existing `.md` is never overwritten - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed -> Nothing was written. A missing package is reported with the preflight hint - show it to the user and stop, never install it. Any other filter keeps working **NEVER** @@ -1516,6 +1516,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. - Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-` line per package-backed format. A PDF gets `` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. +- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten. - A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. diff --git a/tools/chemenu/commands/raw_cmd.py b/tools/chemenu/commands/raw_cmd.py index ea8615f..aa19094 100644 --- a/tools/chemenu/commands/raw_cmd.py +++ b/tools/chemenu/commands/raw_cmd.py @@ -1616,15 +1616,23 @@ def _write_exclusive(writes: list[tuple[Path, bytes]]) -> None: """Create every file or none: `x` mode refuses a file that appeared since the check above, and a failure part-way removes what this call wrote.""" written: list[Path] = [] + made: list[Path] = [] try: for path, data in writes: + missing = [d for d in reversed(path.parents) if not d.exists()] path.parent.mkdir(parents=True, exist_ok=True) + made.extend(missing) with path.open("xb") as handle: written.append(path) handle.write(data) except OSError as exc: for path in written: path.unlink(missing_ok=True) + for directory in reversed(made): + try: + directory.rmdir() + except OSError: + pass fail(escape(f"Could not write {rel_path(path)}: {exc}. Nothing was left in incoming/.")) @@ -1851,6 +1859,7 @@ class _Derived: source: Path # the file in incoming/ markdown: bytes target: Path # where the .md goes + extras: tuple[tuple[Path, bytes], ...] = () # the derivation's further files, beside the .md def _derive_hint(path: Path) -> Optional[str]: @@ -1897,6 +1906,7 @@ class _FilePlan: markdown: bytes container: Optional[filters.Container] derivations: list[filters.Derivation] + extras: tuple[tuple[str, bytes], ...] = () # (path in the folder, bytes) of the derivation's further files def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan: @@ -1942,15 +1952,19 @@ def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan: markdown = b"" targets = [rel for rel, _data in container.files] derivations = list(container.derivations) + extras = () else: original = folder / f"{ORIGINAL_STEM}{extension}" derived = folder / DERIVED_NAME derivation = container.text if container is not None else _run_filter(candidate, data, path) assert derivation is not None + companion = filters.companion_name(DERIVED_NAME) if derivation.files else None markdown = filters.render( - candidate, derivation, derived_from=original.name, original_name=path.name + candidate, derivation, derived_from=original.name, original_name=path.name, + companion=companion, ) - targets = [original.name, derived.name] + ( + extras = tuple((f"{companion}/{name}", content) for name, content in derivation.files) + targets = [original.name, derived.name] + [rel for rel, _data in extras] + ( [rel for rel, _data in container.files] if container is not None else [] ) derivations = [derivation] + (list(container.derivations) if container is not None else []) @@ -1961,7 +1975,7 @@ def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan: f"Cannot write {problem}. The folder name comes from the file's name: pass " "--name ." ) - return _FilePlan(folder, stem, candidate, original, derived, markdown, container, derivations) + return _FilePlan(folder, stem, candidate, original, derived, markdown, container, derivations, extras) def _write_folder(source: Path, plan: _FilePlan) -> None: @@ -1975,6 +1989,11 @@ def _write_folder(source: Path, plan: _FilePlan) -> None: if plan.derived is not None: with plan.derived.open("xb") as handle: handle.write(plan.markdown) + for rel, data in plan.extras: + target = plan.folder / rel + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("xb") as handle: + handle.write(data) if plan.container is not None: for rel, data in plan.container.files: target = plan.folder / rel @@ -2054,8 +2073,26 @@ def _plan_derive_folder(folder: Path): derived: list[_Derived] = [] for file, candidate, md in plan: derivation = _run_filter(candidate, _read_for_derive(file), file) - markdown = filters.render(candidate, derivation, derived_from=file.name) - derived.append(_Derived(candidate, derivation, file, markdown, md)) + companion = md.with_name(filters.companion_name(md.name)) + extras: tuple[tuple[Path, bytes], ...] = () + if derivation.files: + if os.path.lexists(companion): + raise _Refused( + f"{rel_path(companion)} already exists - raw derive never overwrites. Remove it " + "or rename it, then derive the folder again." + ) + extras = tuple((companion / name, content) for name, content in derivation.files) + for target, _content in extras: + problem = path_budget_problem_for(_shard_dir() / folder.name / target.relative_to(folder)) + if problem: + raise _Refused( + f"Cannot write {problem}. The path comes from the folder in incoming/: " + "shorten the folder's name or the names inside it." + ) + markdown = filters.render( + candidate, derivation, derived_from=file.name, companion=companion.name + ) + derived.append(_Derived(candidate, derivation, file, markdown, md, extras)) return derived, untouched @@ -2141,6 +2178,15 @@ def _derive_next(target: Path) -> str: "`` markers and a WARN when it yields almost no text (a scan; there is " "no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an " "embedded message inside it is named, not read.", + "A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder " + "`.sheets/` (`derived.sheets/` for a single file) holds one `NN-.tsv` " + "per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet " + "- and, for a sheet with formulas, `NN-.formulas.tsv` listing cell, formula and " + "last stored value. The `.md` is the overview: one section per sheet with its range, " + "the file names and a preview. Formulas are not recalculated; a shared formula is " + "written out per cell, and one that cannot be is named `shared:` with a " + "WARN. The folder counts against the path budget and is accepted with the rest; an " + "existing one is never overwritten.", "A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder " "`incoming//` holding its files under their own names, each with a `.md` " "where a filter reads it; the ZIP is removed and the output names it with its sha256. A " @@ -2170,7 +2216,7 @@ def _derive_next(target: Path) -> str: "subfolder is moved up, or its folder is derived whole", ), cli_contract.Failure( - cause="The target folder or a `.md` already exists, or the path would be over the " + cause="The target folder, a `.md` or a workbook's `.sheets` folder already exists, or the path would be over the " "path budget", reaction="Nothing was written. Remove what is there or pass `--name `, " "then retry once. An existing `.md` is never overwritten", @@ -2214,9 +2260,11 @@ def raw_derive_command( if name is not None: fail("--name applies to a single file; a folder keeps its own name.") derived, untouched = _refusals_fail(_plan_derive_folder, resolved) - _write_exclusive([(d.target, d.markdown) for d in derived]) + _write_exclusive([write for d in derived for write in [(d.target, d.markdown), *d.extras]]) lines = [f"Derived {len(derived)} file(s) in {rel_path(resolved)}/:"] - lines += [f" {rel_path(d.source)} -> {rel_path(d.target)} ({d.filter.name})" for d in derived] + for d in derived: + more = f" + {len(d.extras)} file(s) in {rel_path(d.extras[0][0].parent)}/" if d.extras else "" + lines.append(f" {rel_path(d.source)} -> {rel_path(d.target)} ({d.filter.name}){more}") if untouched: lines.append( " No filter for, left as they are: " + ", ".join(rel_path(p) for p in untouched) @@ -2248,6 +2296,9 @@ def raw_derive_command( f"Derived {rel_path(plan.derived)} from {resolved.name} ({plan.candidate.name}); the " f"original is now {rel_path(plan.original)}." )) + if plan.extras: + lines = f" + {len(plan.extras)} file(s) beside it in {rel_path(plan.folder)}/{plan.extras[0][0].rsplit('/', 1)[0]}/" + typer.echo(escape(lines)) if container is not None: report = _container_report(container) if report: diff --git a/tools/chemenu/filters/__init__.py b/tools/chemenu/filters/__init__.py index 078fe5e..738e8a2 100644 --- a/tools/chemenu/filters/__init__.py +++ b/tools/chemenu/filters/__init__.py @@ -23,6 +23,9 @@ from chemenu.version import VersionError, read_version MAX_CHECKS = 3 +COMPANION_SUFFIX = ".sheets" +COMPANION_DIR = "\x00companion\x00" # NUL cannot occur in XML text: a safe placeholder inside `Derivation.text` + PREFLIGHT_HINT = ( "Run the preflight to install it (tools/preflight.sh, or tools/preflight.ps1 on Windows; " "instructions/preflight.md). Every other format keeps working." @@ -36,6 +39,11 @@ class Derivation: `fields` are the format-specific header lines, after `title`; a value is a string or a list of strings. `checks` are the short English lines `raw derive` prints after a success - already narrowed to the case that occurred. + + `files` are further files of the derivation (a spreadsheet's sheets), as + `(name, bytes)`. They are written into a folder beside the `.md`, named + `.sheets`; the text refers to that folder as `COMPANION_DIR`, which + `render` replaces with the folder's real name. """ text: str @@ -43,6 +51,7 @@ class Derivation: fields: tuple[tuple[str, "str | list[str]"], ...] = () warnings: tuple[str, ...] = () checks: tuple[str, ...] = () + files: tuple[tuple[str, bytes], ...] = () def __post_init__(self) -> None: if len(self.checks) > MAX_CHECKS: @@ -176,9 +185,11 @@ def render( derived_from: str, original_name: Optional[str] = None, derived_at: Optional[str] = None, + companion: Optional[str] = None, ) -> bytes: """The `.md` for one document: the fixed header, then the derived text. - `original_name` is left out for a file that was not renamed.""" + `original_name` is left out for a file that was not renamed. `companion` is the + name of the folder holding `derivation.files`.""" fields: list[tuple[str, "str | list[str]"]] = [ ("derived_by", "wikitool raw derive"), ("filter", candidate.name), @@ -191,10 +202,20 @@ def render( fields.append(("title", derivation.title)) fields.extend(derivation.fields) text = derivation.text.strip("\n") + if COMPANION_DIR in text: + if companion is None: + raise ValueError("a derivation with files needs the name of the folder they go into") + text = text.replace(COMPANION_DIR, companion) content = web_capture.header(fields) + "\n" + (text + "\n" if text else "") return content.encode("utf-8") +def companion_name(md_name: str) -> str: + """`derived.md` -> `derived.sheets`: the folder beside a `.md` for its `files`.""" + stem = md_name[:-3] if md_name.lower().endswith(".md") else md_name + return stem + COMPANION_SUFFIX + + def unique(lines: Iterable[str]) -> list[str]: seen: set[str] = set() out = [] diff --git a/tools/chemenu/filters/archive.py b/tools/chemenu/filters/archive.py index a046d8b..14171f9 100644 --- a/tools/chemenu/filters/archive.py +++ b/tools/chemenu/filters/archive.py @@ -19,6 +19,7 @@ from chemenu.filters import ( Container, Derivation, Filter, + companion_name, filter_for, register, render, @@ -211,10 +212,19 @@ class Collector: raise ValidationError(f"{path}: {exc}") from None stem = path[: len(path) - len(os.path.splitext(base)[1])] md = self._tree.claim(stem + ".md") + companion = "" + if derivation.files: + directory, _, md_base = md.rpartition("/") + companion = self._tree.claim_folder((directory + "/" if directory else "") + companion_name(md_base)) files.append(( md, - render(candidate, derivation, derived_from=base, original_name=came_as if came_as != base else None), + render( + candidate, derivation, derived_from=base, + original_name=came_as if came_as != base else None, + companion=companion.rsplit("/", 1)[-1], + ), )) + files.extend((f"{companion}/{name}", content) for name, content in derivation.files) derived.append((path, md, candidate.name)) derivations.append(derivation) checks: list[str] = [] diff --git a/tools/chemenu/filters/ooxml.py b/tools/chemenu/filters/ooxml.py index 94eeacf..d309210 100644 --- a/tools/chemenu/filters/ooxml.py +++ b/tools/chemenu/filters/ooxml.py @@ -12,10 +12,11 @@ import posixpath import re import xml.etree.ElementTree as ET import zipfile -from typing import Iterator, Optional +from dataclasses import dataclass, field +from typing import Iterable, Iterator, Optional from chemenu.errors import ValidationError -from chemenu.filters import Derivation, Filter, register +from chemenu.filters import COMPANION_DIR, Derivation, Filter, register PART_LIMIT = 100 * 1024 * 1024 CELL_LIMIT = 2_000_000 @@ -596,11 +597,20 @@ class _Workbook: self.formulas_without_value += 1 return text - def table(self, part: str) -> str: + def sheet(self, part: str) -> "_Sheet": + """The cells and formulas of one worksheet part, read as stored.""" cells: dict[tuple[int, int], str] = {} + pending: list[_PendingFormula] = [] row_no, last_col = 0, 0 - for _event, el in ET.iterparse(io.BytesIO(self.pkg.read(part)), events=("end",)): + root_seen = False + chart = False + for event, el in ET.iterparse(io.BytesIO(self.pkg.read(part)), events=("start", "end")): tag = _local(el.tag) + if event == "start": + if not root_seen: + root_seen = True + chart = tag == "chartsheet" + continue if tag == "c": ref = re.fullmatch(r"([A-Za-z]+)(\d+)", el.get("r") or "") if ref: @@ -609,25 +619,226 @@ class _Workbook: col, row = last_col + 1, row_no + 1 last_col = col text = self._value(el) - if text.strip(): + if text != "": cells[(row, col)] = text + f = _child(el, "f") + if f is not None: + pending.append(_PendingFormula( + row, col, _attr(f, "t") or "normal", _attr(f, "si") or "", _attr(f, "ref") or "", + (f.text or "").strip(), (_attr(f, "r1") or "", _attr(f, "r2") or ""), text, + )) el.clear() elif tag == "row": row_no = int(el.get("r") or row_no + 1) last_col = 0 el.clear() - if not cells: - return "*(empty sheet)*" - rows = [r for r, _c in cells] - cols = [c for _r, c in cells] - min_r, max_r, min_c, max_c = min(rows), max(rows), min(cols), max(cols) - if (max_r - min_r + 1) * (max_c - min_c + 1) > CELL_LIMIT: + if chart: + return _Sheet(chart=True) + rows = max((r for r, _c in cells), default=0) + cols = max((c for _r, c in cells), default=0) + if rows * cols > CELL_LIMIT: raise ValidationError( - f"A sheet spans {max_r - min_r + 1} rows by {max_c - min_c + 1} columns - too large " - "or too sparse to derive." + f"A sheet spans {rows} rows by {cols} columns from A1 - too large or too sparse " + "to derive." ) - grid = [[cells.get((r, c), "") for c in range(min_c, max_c + 1)] for r in range(min_r, max_r + 1)] - return _table(grid) + formulas, untranslated = _resolve_formulas(pending) + return _Sheet(cells=cells, rows=rows, cols=cols, formulas=formulas, untranslated=untranslated) + + +@dataclass(frozen=True) +class _PendingFormula: + row: int + col: int + kind: str # normal, shared, array, dataTable + si: str + ref: str + text: str + inputs: tuple[str, str] + value: str + + +@dataclass(frozen=True) +class _Formula: + row: int + col: int + text: str + value: str + + +@dataclass(frozen=True) +class _Sheet: + chart: bool = False + cells: dict = field(default_factory=dict) + rows: int = 0 + cols: int = 0 + formulas: tuple[_Formula, ...] = () + untranslated: int = 0 + + +class _Unsafe(Exception): + pass + + +_MAX_COL, _MAX_ROW = 16384, 1048576 +_NOT_AFTER = r"(?\$?[A-Z]{1,3}\$?\d+(?::\$?[A-Z]{1,3}\$?\d+)?)" + _NOT_BEFORE, + _NOT_AFTER + r"(?P\$?[A-Z]{1,3}:\$?[A-Z]{1,3})" + _NOT_BEFORE, + _NOT_AFTER + r"(?P\$?\d+:\$?\d+)" + _NOT_BEFORE, + )), + re.IGNORECASE, +) +_CELL = re.compile(r"(\$?)([A-Z]{1,3})(\$?)(\d+)", re.IGNORECASE) + + +def _col_letters(index: int) -> str: + letters = "" + while index: + index, rest = divmod(index - 1, 26) + letters = chr(65 + rest) + letters + return letters + + +def _shift_formula(text: str, dr: int, dc: int) -> str: + """The formula of a shared-formula cell: the master's text with every relative + reference moved by the cell's offset. Strings, quoted sheet names and structured + references are left alone; a reference that would leave the sheet raises `_Unsafe`.""" + + def moved(index: int, absolute: str, limit: int, offset: int) -> int: + if absolute: + return index + index += offset + if not 1 <= index <= limit: + raise _Unsafe + return index + + def cell(ref: str) -> str: + c_abs, letters, r_abs, digits = _CELL.fullmatch(ref).groups() + column, row = _col_index(letters.upper()), int(digits) + if not (1 <= column <= _MAX_COL and 1 <= row <= _MAX_ROW): + return ref + return (f"{c_abs}{_col_letters(moved(column, c_abs, _MAX_COL, dc))}" + f"{r_abs}{moved(row, r_abs, _MAX_ROW, dr)}") + + def column(ref: str) -> str: + absolute, letters = re.fullmatch(r"(\$?)([A-Za-z]+)", ref).groups() + return f"{absolute}{_col_letters(moved(_col_index(letters.upper()), absolute, _MAX_COL, dc))}" + + def row(ref: str) -> str: + absolute, digits = re.fullmatch(r"(\$?)(\d+)", ref).groups() + return f"{absolute}{moved(int(digits), absolute, _MAX_ROW, dr)}" + + def replace(match: re.Match) -> str: + for group, shift in (("cells", cell), ("cols", column), ("rows", row)): + if match.group(group): + return ":".join(shift(part) for part in match.group(group).split(":")) + return match.group(0) + + return _REFERENCE.sub(replace, text) + + +def _a1(row: int, col: int) -> str: + return f"{_col_letters(col)}{row}" + + +def _resolve_formulas(pending: list[_PendingFormula]) -> tuple[tuple[_Formula, ...], int]: + """The formula text of every formula cell. Excel stores a shared formula once, in + its master cell; the others get the master's text moved to their own position. + Returns the formulas in sheet order and how many could not be written out.""" + masters = {f.si: f for f in pending if f.kind == "shared" and f.text} + out: list[_Formula] = [] + untranslated = 0 + for f in sorted(pending, key=lambda item: (item.row, item.col)): + if f.kind == "shared" and not f.text: + master = masters.get(f.si) + try: + if master is None: + raise _Unsafe + text = "=" + _shift_formula(master.text, f.row - master.row, f.col - master.col) + except _Unsafe: + text = f"shared:{_a1(master.row, master.col)}" if master else f"shared:si={f.si}" + untranslated += 1 + elif f.kind == "array": + text = f"={f.text}" + (f" (array formula, range {f.ref})" if f.ref else "") + elif f.kind == "dataTable": + text = f"=TABLE({f.inputs[0]},{f.inputs[1]})" + else: + text = "=" + f.text + out.append(_Formula(f.row, f.col, text, f.value)) + return tuple(out), untranslated + + +def _tsv(rows: Iterable[list[str]]) -> bytes: + """RFC 4180 with a tab as the delimiter: a field holding a tab, a quote or a line + break is quoted and its quotes doubled. A row of one empty field is written + quoted, or a reader would take the line for no row at all.""" + lines = [] + for row in rows: + fields = [ + '"' + value.replace('"', '""') + '"' if any(ch in value for ch in '\t"\r\n') else value + for value in row + ] + if fields == [""]: + fields = ['""'] + lines.append("\t".join(fields) + "\n") + return "".join(lines).encode("utf-8") + + +PREVIEW_ROWS = 5 +PREVIEW_COLUMNS = 20 +PREVIEW_CELL = 60 + + +def _slug(name: str) -> str: + slug = re.sub(r"\W+", "-", name.lower()).strip("-")[:32].strip("-") + return slug or "sheet" + + +def _fence(text: str) -> str: + longest = max((len(run) for run in re.findall(r"`+", text)), default=0) + return "`" * max(3, longest + 1) + + +def _sheet_section( + number: int, total: int, width: int, name: str, state: str, sheet: _Sheet, files: list[tuple[str, bytes]] +) -> str: + lines = [f"## {name}", "", f"- Sheet {number} of {total}, {state}"] + if sheet.chart: + return "\n".join(lines + ["", "*(chart sheet, no cell data)*"]) + if not sheet.cells: + return "\n".join(lines + ["", "*(empty sheet)*"]) + grid = [[sheet.cells.get((r, c), "") for c in range(1, sheet.cols + 1)] for r in range(1, sheet.rows + 1)] + stem = f"{number:0{width}d}-{_slug(name)}" + files.append((f"{stem}.tsv", _tsv(grid))) + lines.append( + f"- Range A1:{_a1(sheet.rows, sheet.cols)} ({sheet.rows} rows x {sheet.cols} columns), " + + (f"{len(sheet.formulas)} formula cell(s)" if sheet.formulas else "no formulas") + ) + lines.append(f"- Cells: `{COMPANION_DIR}/{stem}.tsv`") + if sheet.formulas: + rows = [["cell", "formula", "value"]] + [[_a1(f.row, f.col), f.text, f.value] for f in sheet.formulas] + files.append((f"{stem}.formulas.tsv", _tsv(rows))) + lines.append(f"- Formulas: `{COMPANION_DIR}/{stem}.formulas.tsv`") + cut = sheet.cols > PREVIEW_COLUMNS or any( + len(value) > PREVIEW_CELL for row in grid[:PREVIEW_ROWS] for value in row[:PREVIEW_COLUMNS] + ) + preview = _tsv( + [[v if len(v) <= PREVIEW_CELL else v[: PREVIEW_CELL - 1] + "…" for v in row[:PREVIEW_COLUMNS]] + for row in grid[:PREVIEW_ROWS]] + ).decode("utf-8") + fence = _fence(preview) + label = ( + f"Preview (all {sheet.rows} rows)" if sheet.rows <= PREVIEW_ROWS + else f"Preview (first {PREVIEW_ROWS} of {sheet.rows} rows)" + ) + if cut: + label += f" (columns past {PREVIEW_COLUMNS} and text past {PREVIEW_CELL} characters cut)" + return "\n".join(lines + ["", f"{label}:", "", fence + "tsv", preview.rstrip("\n"), fence]) class XlsxFilter(Filter): @@ -640,31 +851,49 @@ class XlsxFilter(Filter): sheets = book.sheets() if not sheets: raise ValidationError("The XLSX file lists no sheets.") - sections, hidden, warnings = [], [], [] - for sheet_name, state, part in sheets: + width = max(2, len(str(len(sheets)))) + sections, files, hidden, warnings = [], [], [], [] + formulas = untranslated = 0 + for number, (sheet_name, state, part) in enumerate(sheets, 1): if state != "visible": hidden.append(sheet_name) if not part or not pkg.has(part): warnings.append(f"Sheet '{sheet_name}' has no readable worksheet part and was left out.") continue - if _child(pkg.xml(part), "sheetData") is None and _local(pkg.xml(part).tag) == "chartsheet": - sections.append(f"## {sheet_name}\n\n*(chart sheet, no cell data)*") - continue - sections.append(f"## {sheet_name}\n\n{book.table(part)}") + sheet = book.sheet(part) + formulas += len(sheet.formulas) + untranslated += sheet.untranslated + sections.append(_sheet_section(number, len(sheets), width, sheet_name, state, sheet, files)) if book.formulas_without_value: warnings.append( f"{book.formulas_without_value} formula cell(s) have no stored value (the file was " "never calculated) and are empty." ) + if untranslated: + warnings.append( + f"{untranslated} shared formula cell(s) could not be written out and show " + "'shared:' - the master cell holds the formula." + ) + intro = [] + if files: + intro.append( + "Each sheet's cells are in a TSV file beside this one: RFC 4180 with a tab as the " + "delimiter, so a cell holding a tab, a quote or a line break is quoted. Read the " + 'files with a CSV reader (`csv.reader(f, delimiter="\\t")`, `pandas.read_csv(sep="\\t")`), ' + "not with `cut` or `awk`. Row n of a file is row n of the sheet, column k its k-th column." + ) checks = ["Values are as last saved; formulas were not recalculated."] + if formulas: + checks.append("Each sheet with formulas has a .formulas.tsv beside its cells: cell, formula, stored value.") if hidden: checks.append("Hidden sheets are included and named in hidden_sheets.") return Derivation( - text="\n\n".join(sections), + text="\n\n".join(intro + sections), title=pkg.title(), - fields=(("sheets", str(len(sheets))), ("hidden_sheets", hidden)), + fields=(("sheets", str(len(sheets))), ("hidden_sheets", hidden), ("formulas", str(formulas))), warnings=tuple(warnings), checks=tuple(checks), + files=tuple(files), ) diff --git a/tools/chemenu/tests/test_raw_derive.py b/tools/chemenu/tests/test_raw_derive.py index 2298310..d6c36d5 100644 --- a/tools/chemenu/tests/test_raw_derive.py +++ b/tools/chemenu/tests/test_raw_derive.py @@ -5,9 +5,11 @@ readable in the test that uses it and no binary sits in the repository. """ from __future__ import annotations +import csv import hashlib import io import json +import shutil import zipfile from email import message_from_bytes, policy from email.message import EmailMessage @@ -156,6 +158,30 @@ def make_xlsx() -> bytes: }) +def make_book(sheets: list[tuple[str, str, str]]) -> bytes: + """A workbook of `(name, state, elements)` with inline strings only.""" + rels = "".join( + f'' + for n in range(1, len(sheets) + 1) + ) + listed = "".join( + f'' + for n, (name, state, _rows) in enumerate(sheets, 1) + ) + parts = { + "_rels/.rels": _rels("xl/workbook.xml"), + "xl/workbook.xml": f'{listed}', + "xl/_rels/workbook.xml.rels": f'{rels}', + } + for n, (_name, _state, rows) in enumerate(sheets, 1): + parts[f"xl/worksheets/sheet{n}.xml"] = f'{rows}' + return _zip(parts) + + +def _text_cell(ref: str, text: str) -> str: + return f'{text.replace(chr(13), " ")}' + + def make_eml(body: str = "Hallo Welt\n", *, html: str = "", attach: bool = False, inline: bool = False, subject: str = "Angebot") -> bytes: msg = EmailMessage() @@ -374,7 +400,11 @@ def test_a_folder_gets_a_md_beside_each_file_with_a_filter(tree, capsys): _derive("incoming/mappe") out = capsys.readouterr().out - assert sorted(p.name for p in folder.iterdir()) == ["foto.jpg", "plan.docx", "plan.md", "zahlen.md", "zahlen.xlsx"] + assert sorted(p.name for p in folder.iterdir()) == ["foto.jpg", "plan.docx", "plan.md", "zahlen.md", "zahlen.sheets", "zahlen.xlsx"] + assert sorted(p.name for p in (folder / "zahlen.sheets").iterdir()) == [ + "01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv", + ] + assert "`zahlen.sheets/01-budget.tsv`" in (folder / "zahlen.md").read_text(encoding="utf-8") header, _ = _header_and_text(folder / "plan.md") assert header["derived_from"] == "plan.docx" assert "original_name" not in header @@ -487,20 +517,175 @@ def test_pptx_slides_notes_and_footer_placeholders(tree): # --- XLSX ------------------------------------------------------------------- -def test_xlsx_values_hidden_sheet_and_trimmed_edges(tree, capsys): +def _tsv_rows(path: Path) -> list[list[str]]: + with path.open(newline="", encoding="utf-8") as handle: + return list(csv.reader(handle, delimiter="\t")) + + +def test_xlsx_becomes_an_overview_and_one_tsv_per_sheet(tree, capsys): _put(tree, "zahlen.xlsx", make_xlsx()) _derive("incoming/zahlen.xlsx") out = capsys.readouterr().out - header, text = _header_and_text(tree / "incoming/zahlen/derived.md") + folder = tree / "incoming/zahlen" + header, text = _header_and_text(folder / "derived.md") - assert header["sheets"] == 2 and header["hidden_sheets"] == ["Geheim"] - assert "## Budget" in text and "## Geheim" in text and "VERSTECKT-INHALT" in text - assert "| Posten | Wert |" in text - assert "| Miete | 1200 |" in text - assert "2024-01-01" in text - assert "TRUE" in text - assert text.splitlines()[text.splitlines().index("## Budget") + 2].startswith("| Posten") - assert "Values are as last saved" in out and "Hidden sheets" in out + assert header["sheets"] == 2 and header["hidden_sheets"] == ["Geheim"] and header["formulas"] == 1 + assert "## Budget" in text and "## Geheim" in text + assert "Sheet 2 of 2, hidden" in text + assert "Range A1:C5 (5 rows x 3 columns), 1 formula cell(s)" in text + assert "`derived.sheets/01-budget.tsv`" in text and "`derived.sheets/01-budget.formulas.tsv`" in text + assert "`derived.sheets/02-geheim.tsv`" in text and "02-geheim.formulas" not in text + assert "Preview" in text and "Posten\tWert" in text + + # Row n of the file is row n of the sheet, column k its k-th column: the sheet starts at B2. + assert _tsv_rows(folder / "derived.sheets/01-budget.tsv") == [ + ["", "", ""], + ["", "Posten", "Wert"], + ["", "Miete", "1200"], + ["", "Stichtag", "2024-01-01"], + ["", "Fertig", "TRUE"], + ] + assert _tsv_rows(folder / "derived.sheets/01-budget.formulas.tsv") == [ + ["cell", "formula", "value"], + ["C3", "=1200+0", "1200"], + ] + assert _tsv_rows(folder / "derived.sheets/02-geheim.tsv") == [["VERSTECKT-INHALT"]] + assert "Values are as last saved" in out and "Hidden sheets" in out and ".formulas.tsv" in out + assert "derived.sheets" in out + + +def test_tsv_quotes_per_rfc_4180_and_reads_back_cell_for_cell(tree): + cells = ["a\tb", 'say "hi"', "two\nlines", "cr\rhere", " padded ", "plain", "="] + row = "".join(_text_cell(f"{chr(65 + i)}1", c) for i, c in enumerate(cells)) + single = '' + _text_cell("A2", "x") + "" + _put(tree, "q.xlsx", make_book([("Quoting", "visible", f'{row}'), ("One column", "visible", single)])) + _derive("incoming/q.xlsx") + sheets = tree / "incoming/q/derived.sheets" + + assert _tsv_rows(sheets / "01-quoting.tsv") == [cells] + raw = (sheets / "01-quoting.tsv").read_bytes() + assert raw.endswith(b"\n") and b'"a\tb"' in raw and b'"say ""hi"""' in raw and b'"cr\rhere"' in raw + # A single-column sheet whose first row is empty: the empty row is still a row. + assert (sheets / "02-one-column.tsv").read_bytes() == b'""\nx\n' + only_empty_first = make_book([("S", "visible", '' + _text_cell("A2", "x") + "")]) + _put(tree, "e.xlsx", only_empty_first) + _derive("incoming/e.xlsx") + assert _tsv_rows(tree / "incoming/e/derived.sheets/01-s.tsv") == [[""], ["x"]] + assert (tree / "incoming/e/derived.sheets/01-s.tsv").read_bytes() == b'""\nx\n' + + +def test_shared_array_and_plain_formulas_are_listed_with_their_stored_values(tree, capsys): + master = "A2*$B$1+B2+SUM(A$1:B$1)+LOG10(A2)&\"A2\"&'My Sheet'!A2" + rows = ( + '102' + '203' + f'{master}1' + '302' + '3' + 'SUM(A2:A4*B2:B4)7' + ) + _put(tree, "f.xlsx", make_book([("Calc", "visible", rows)])) + _derive("incoming/f.xlsx") + formulas = _tsv_rows(tree / "incoming/f/derived.sheets/01-calc.formulas.tsv") + + def shifted(n: int) -> str: + return f"=A{n}*$B$1+B{n}+SUM(A$1:B$1)+LOG10(A{n})&\"A2\"&'My Sheet'!A{n}" + + assert formulas == [ + ["cell", "formula", "value"], + ["C2", shifted(2), "1"], + ["C3", shifted(3), "2"], + ["C4", shifted(4), "3"], + ["D4", "=SUM(A2:A4*B2:B4) (array formula, range D4:D5)", "7"], + ] + assert "WARN" not in capsys.readouterr().out + assert _header_and_text(tree / "incoming/f/derived.md")[0]["formulas"] == 4 + + +def test_a_shared_formula_that_would_leave_the_sheet_is_named_by_its_master(tree, capsys): + rows = ( + 'A4+11' + '23' + ) + _put(tree, "u.xlsx", make_book([("S", "visible", rows)])) + _derive("incoming/u.xlsx") + out = capsys.readouterr().out + formulas = _tsv_rows(tree / "incoming/u/derived.sheets/01-s.formulas.tsv") + + assert formulas[1:] == [["B5", "=A4+1", "1"], ["A6", "shared:B5", "2"], ["C6", "shared:si=9", "3"]] + assert "WARN 2 shared formula cell(s) could not be written out" in out + + +def test_xlsx_files_are_the_same_on_a_second_run(tree): + data = make_xlsx() + _put(tree, "a.xlsx", data) + _derive("incoming/a.xlsx") + first = {k: v for k, v in _snapshot(tree / "incoming").items() if k.endswith(".tsv")} + shutil.rmtree(tree / "incoming/a") + _put(tree, "a.xlsx", data) + _derive("incoming/a.xlsx") + assert {k: v for k, v in _snapshot(tree / "incoming").items() if k.endswith(".tsv")} == first + assert first + + +def test_an_empty_and_a_chart_sheet_get_no_file(tree): + data = make_book([("Leer", "visible", ""), ("Daten", "visible", '' + _text_cell("A1", "x") + "")]) + _put(tree, "l.xlsx", data) + _derive("incoming/l.xlsx") + _header, text = _header_and_text(tree / "incoming/l/derived.md") + assert "*(empty sheet)*" in text + assert [p.name for p in (tree / "incoming/l/derived.sheets").iterdir()] == ["02-daten.tsv"] + + +def test_sheet_files_count_against_the_path_budget_and_a_refusal_leaves_incoming_alone(tree, capsys): + stem = "s" * 120 + _put(tree, f"{stem}.xlsx", make_xlsx()) + before = _snapshot(tree / "incoming") + _refused(f"incoming/{stem}.xlsx") + assert "--name" in capsys.readouterr().out + assert _snapshot(tree / "incoming") == before + _derive(f"incoming/{stem}.xlsx", "kurz") + assert (tree / "incoming/kurz/derived.sheets/01-budget.tsv").is_file() + + +def test_the_folder_form_never_overwrites_a_sheets_folder_and_changes_nothing(tree): + folder = tree / "incoming/mappe" + (folder / "zahlen.sheets").mkdir(parents=True) + (folder / "zahlen.xlsx").write_bytes(make_xlsx()) + before = _snapshot(tree / "incoming") + (folder / "zahlen.sheets/x.txt").write_text("mine", encoding="utf-8") + before = _snapshot(tree / "incoming") + _refused("incoming/mappe") + assert _snapshot(tree / "incoming") == before + + +def test_a_workbook_in_a_zip_or_a_mail_keeps_its_sheets_beside_its_md(tree): + _put(tree, "s.zip", make_zip({"sub/Budget 2024.xlsx": make_xlsx(), "Budget 2024.sheets": b"in the way"})) + _derive("incoming/s.zip") + folder = tree / "incoming/s" + assert (folder / "sub/Budget 2024.md").is_file() + assert sorted(p.name for p in (folder / "sub/Budget 2024.sheets").iterdir()) == [ + "01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv", + ] + assert "`Budget 2024.sheets/01-budget.tsv`" in (folder / "sub/Budget 2024.md").read_text(encoding="utf-8") + + _put(tree, "m.eml", make_mail([("derived.xlsx", make_xlsx(), "application/octet-stream")])) + _derive("incoming/m.eml") + mail = tree / "incoming/m" + assert (mail / "derived-2.md").is_file() and (mail / "derived-2.sheets/01-budget.tsv").is_file() + assert "`derived-2.sheets/01-budget.tsv`" in (mail / "derived-2.md").read_text(encoding="utf-8") + + +def test_the_derived_folder_with_its_sheet_files_is_accepted_whole(tree): + _put(tree, "zahlen.xlsx", make_xlsx()) + _derive("incoming/zahlen.xlsx") + raw_accept_command( + files=[Path("incoming/zahlen")], fidelity="verbatim", authority="reporting", + page=None, replaces=None, dry_run=False, + ) + accepted = sorted(p for p in (tree / "raw").rglob("*.tsv")) + assert [p.name for p in accepted] == ["01-budget.formulas.tsv", "01-budget.tsv", "02-geheim.tsv"] + assert not (tree / "incoming/zahlen").exists() def test_xlsx_check_is_printed_only_for_the_formats_derived_in_the_run(tree, capsys): diff --git a/tools/chemenu/tests/test_raw_derive_testfiles.py b/tools/chemenu/tests/test_raw_derive_testfiles.py new file mode 100644 index 0000000..4243ee5 --- /dev/null +++ b/tools/chemenu/tests/test_raw_derive_testfiles.py @@ -0,0 +1,104 @@ +"""`wikitool raw derive` against the real documents in `testfiles/`. + +The synthetic fixtures in `binary_formats.py` and `test_raw_derive.py` prove the filters on +documents built to be easy; the files in `testfiles/` are whatever the world produced. Every file +there is derived, compared with itself on a second run and accepted into `raw/`, and an optional +`.expect` next to it names the text that must come out. A distributed instance has no +`testfiles/`, so the whole module is then skipped. +""" +from __future__ import annotations + +import importlib.util +import shutil +from pathlib import Path + +import pytest + +from chemenu import filters +from chemenu.commands.raw_cmd import raw_accept_command +from chemenu.tests.test_raw_derive import ( # noqa: F401 (tree is a fixture) + _derive, + _header_and_text, + tree, +) + +TESTFILES = Path(__file__).resolve().parents[3] / "testfiles" +SIDECAR = ".expect" + +pytestmark = pytest.mark.skipif( + importlib.util.find_spec("pypdf") is None or importlib.util.find_spec("olefile") is None, + reason="pypdf and olefile are needed for real PDFs, MSGs and archives holding them", +) + + +def _documents() -> list[Path]: + if not TESTFILES.is_dir(): + return [] + return sorted( + path + for path in TESTFILES.rglob("*") + if path.is_file() + and not path.name.startswith(".") + and path.name != "README.md" + and path.suffix != SIDECAR + ) + + +def _id(path: Path) -> str: + return path.relative_to(TESTFILES).as_posix() + + +def _texts(folder: Path) -> dict[str, str]: + texts = {} + for md in sorted(folder.rglob("*.md")): + lines = md.read_text(encoding="utf-8").splitlines() + texts[md.relative_to(folder).as_posix()] = "\n".join( + line for line in lines if not line.startswith("derived:") + ) + return texts + + +def _expected(document: Path) -> list[str]: + sidecar = document.with_name(document.name + SIDECAR) + if not sidecar.is_file(): + return [] + lines = sidecar.read_text(encoding="utf-8").splitlines() + return [line for line in lines if line.strip() and not line.startswith("#")] + + +def _put_and_derive(tree: Path, document: Path) -> Path: + (tree / "incoming" / document.name).write_bytes(document.read_bytes()) + _derive(f"incoming/{document.name}") + return tree / "incoming" / document.stem + + +@pytest.mark.parametrize("document", _documents(), ids=_id) +def test_a_real_document_derives_stably_and_is_accepted(tree, document): + assert filters.filter_for(document.name) is not None, ( + f"no import filter takes {document.name}; a file in testfiles/ must be one `raw derive` reads" + ) + + folder = _put_and_derive(tree, document) + assert not (tree / "incoming" / document.name).exists() + + if document.suffix.lower() == ".zip": + assert any(folder.rglob("*.md")), "an unpacked archive gets a .md beside its documents" + else: + original = folder / f"original{document.suffix.lower()}" + assert original.read_bytes() == document.read_bytes() + header, _text = _header_and_text(folder / "derived.md") + assert header["derived_by"] == "wikitool raw derive" + assert header["original_name"] == document.name + + first = _texts(folder) + for needle in _expected(document): + assert any(needle in text for text in first.values()), f"{needle!r} not in the derived text" + + shutil.rmtree(folder) + assert _texts(_put_and_derive(tree, document)) == first, "the same bytes gave a different text" + + raw_accept_command( + files=[Path(f"incoming/{document.stem}")], fidelity="verbatim", authority="reporting", + page=None, replaces=None, dry_run=False, + ) + assert not (tree / "incoming" / document.stem).exists()