diff --git a/CHANGES.md b/CHANGES.md index 0d59ca2..f2b8524 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. --- -## 9.0.0-beta.14 - 2026-10-08 - raw derive reads PDF-free documents: Office files and mail become a folder with the original and a derived text +## 9.0.0-beta.15 - 2026-10-08 - raw derive unpacks ZIP archives and mails with attachments into a folder **Author:** Torben Nehmer @@ -36,6 +36,7 @@ place only: `instructions/dev/version-parts.md` § The candidate model. - Gate-Liste: AGENTS.md § Gates ist die eine Liste (mit Rebase-Review Gate), cli_contract.GATES hält sie, docs verify vergleicht beide - Asset-Preflight: Verschieben aus dem Entpackordner wird bis ca. 30 s wiederholt, nennt die Systemmeldung und rollt nach endgültigem Scheitern zurück - raw derive reads PDF-free documents: Office files and mail become a folder with the original and a derived text +- raw derive unpacks ZIP archives and mails with attachments into a folder **Low impact** - version bump: Abschlussmeldung nennt Stichpunkt oder Themenabsatz statt pauschal Prosa diff --git a/VERSION b/VERSION index 72cb8d7..74ad31d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -9.0.0-beta.14 +9.0.0-beta.15 diff --git a/docs/how-to/ingest-a-source.md b/docs/how-to/ingest-a-source.md index 7197dfa..13af1a1 100644 --- a/docs/how-to/ingest-a-source.md +++ b/docs/how-to/ingest-a-source.md @@ -16,10 +16,11 @@ pick a directory: - **A web page** needs no download of your own. Name the URL to the agent, and `tools/wikitool raw fetch` puts two files into `incoming/`: the HTML exactly as received, and a text derived from it with a header recording where and when it was fetched. -- **A document that is not text** - a Word, PowerPoint or Excel file, or a mail - is put into - `incoming/` as it is. `tools/wikitool raw derive incoming/` turns it into a folder with the +- **A document that is not text** - a Word, PowerPoint or Excel file, a mail or a ZIP archive - is + put into `incoming/` as it is. `tools/wikitool raw derive incoming/` turns it into a folder with the untouched original and a readable text beside it, with a header saying how it was made. The - folder is accepted whole; the agent runs this itself when it meets such a file. + folder is accepted whole; the agent runs this itself when it meets such a file. A ZIP is + unpacked into a folder instead, and a mail's attachments land beside it. - **A page behind a paywall or a login** is saved from your browser into `incoming/`, HTML only. The agent derives the same text from that file with `raw fetch --html`. diff --git a/docs/reference/commands/README.md b/docs/reference/commands/README.md index 4aa0b82..0d0285c 100644 --- a/docs/reference/commands/README.md +++ b/docs/reference/commands/README.md @@ -79,7 +79,7 @@ Full records: [provenance.md](provenance.md) Full records: [raw-material-and-uploads.md](raw-material-and-uploads.md) - [`raw fetch`](raw-material-and-uploads.md#raw-fetch) - Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. -- [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote. +- [`raw derive`](raw-material-and-uploads.md#raw-derive) - Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. - [`raw capture`](raw-material-and-uploads.md#raw-capture) - Capture documentation from a git repository into `incoming//`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote. - [`raw status`](raw-material-and-uploads.md#raw-status) - Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`. - [`raw pending`](raw-material-and-uploads.md#raw-pending) - List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next. diff --git a/docs/reference/commands/raw-material-and-uploads.md b/docs/reference/commands/raw-material-and-uploads.md index edcff22..f98e6da 100644 --- a/docs/reference/commands/raw-material-and-uploads.md +++ b/docs/reference/commands/raw-material-and-uploads.md @@ -85,7 +85,7 @@ tools/wikitool raw fetch --html incoming/post.html --url https://example.org/blo ## `raw derive` -Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote. +Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. **Synopsis** @@ -109,6 +109,7 @@ Options: tools/wikitool raw derive incoming/report.docx tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz tools/wikitool raw derive incoming/meeting-files +tools/wikitool raw derive incoming/scans.zip ``` **Behaviour** @@ -117,7 +118,8 @@ tools/wikitool raw derive incoming/meeting-files - A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`. - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. -- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`) and e-mail (`.eml`) read with the standard library only. A mail with attachments is refused. +- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only. +- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original. @@ -135,7 +137,7 @@ tools/wikitool raw derive incoming/meeting-files - `0` - success - `1` - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension - `1` - The target folder or a `.md` already exists, or the path would be over the path budget -- `1` - The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed +- `1` - The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **See also** diff --git a/raw/CONTRACT.md b/raw/CONTRACT.md index 2b82a53..b02a43d 100644 --- a/raw/CONTRACT.md +++ b/raw/CONTRACT.md @@ -276,6 +276,19 @@ a stem of well over a hundred characters, where keeping the file's own name twic as `original_name`. A folder already in `incoming/` is derived in place instead: each file with a filter gets a `.md` beside it, and nothing is renamed. +**A container is unpacked into its folder.** A ZIP becomes the folder `incoming//` holding +its files under their own names, each with a filter next to a `.md`, and the ZIP itself is +removed - the tool's output names it and its sha256, and naming it on the source page is a human's +step. A mail with attachments becomes a folder holding `original.eml`, `derived.md` and the +attachments under their own names; a ZIP attached to a mail is unpacked once into a folder of its +stem, and a mail attached to a mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, +and any entry that is an absolute or `..` path, a symbolic link or encrypted, is refused, as is +an archive past the entry, file or total size limit that `raw derive -h` names. Hidden entries +(`__MACOSX/`, `.DS_Store`, anything starting with a dot) are not written but are named in the +output. A name that is taken or reserved (`original.*` and `derived.md` at the top of a mail +folder, `_capture.json` anywhere) gets `-2`, `-3` before its extension, and the output lists +every rename. + **The original stays.** The derivation is the tool's reading of it, never the source: with the original kept, a claim in `kb/` can still be checked against what the document said when a derivation dropped something, and a later version of the tool can derive it better. Both go into diff --git a/tools/CONTRACT.md b/tools/CONTRACT.md index 8e205cc..88f1bfa 100644 --- a/tools/CONTRACT.md +++ b/tools/CONTRACT.md @@ -123,7 +123,7 @@ sources coverage read idempotent budget:counted exit:0 sources trace read idempotent budget:counted exit:0,1 Trace provenance in either direction: raw file, or page. sources rebuild-index write idempotent budget:counted exit:0,1 Regenerate the `kb/provenance.md` reverse index. raw fetch write non-idempotent budget:counted exit:0,1 Capture a web page the user names into `incoming/`: the HTML as received plus a derived text, for `raw accept` to promote. -raw derive write non-idempotent budget:counted exit:0,1 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote. +raw derive write non-idempotent budget:counted exit:0,1 Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. raw capture write non-idempotent budget:counted exit:0,1 Capture documentation from a git repository into `incoming//`, with a manifest naming the repository, ref rule and commit, for `raw accept` to promote. raw status read idempotent budget:counted exit:0 Report which captured bundles under `raw/` have fallen behind their repository, with each changed file as `A`/`M`/`D`. raw pending read idempotent budget:counted exit:0 List what waits in `incoming/`, oldest first, and name the entry an ingest without an argument takes next. @@ -1469,7 +1469,7 @@ Capture a web page the user names into `incoming/`: the HTML as received plus a #### `raw derive` -Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and leave the original beside it, for `raw accept` to promote. +Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or unpack a ZIP, and leave the original beside it, for `raw accept` to promote. **SYNOPSIS** @@ -1489,19 +1489,20 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` a - `tools/wikitool raw derive incoming/report.docx` - `tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz` - `tools/wikitool raw derive incoming/meeting-files` +- `tools/wikitool raw derive incoming/scans.zip` **EXIT STATUS** - 0 success - 1 The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension - 1 The target folder or a `.md` already exists, or the path would be over the path budget -- 1 The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed +- 1 The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed **ON FAILURE** - The path is not directly in `incoming/`, does not exist, or no filter reads the file's extension -> Nothing was written. Fix the argument and retry once - a file in a subfolder is moved up, or its folder is derived whole - The target folder or a `.md` already exists, or the path would be over the path budget -> Nothing was written. Remove what is there or pass `--name `, then retry once. An existing `.md` is never overwritten -- The document cannot be read (damaged, encrypted, a mail with attachments) or the filter's package is not installed -> Nothing was written. A missing package is reported with the preflight hint - show it to the user and stop, never install it. Any other filter keeps working +- The document cannot be read (damaged, encrypted), a container breaks one of its rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not installed -> Nothing was written. A missing package is reported with the preflight hint - show it to the user and stop, never install it. Any other filter keeps working **NEVER** @@ -1514,7 +1515,8 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` a - A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`. - A folder is derived in place: each file with a filter gets `.md` beside it, nothing is renamed, and the files without a filter are named in the output. - `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version. -- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`) and e-mail (`.eml`) read with the standard library only. A mail with attachments is refused. +- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only. +- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming//` holding its files under their own names, each with a `.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename. - A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it. - After a success the output prints the warnings the filters raised, at most three `CHECK` lines per format that was derived in this run - what to look at before the text is trusted - and the `raw accept` line for the result. - Never edit `derived.md` by hand: a better derivation is a new `raw derive` of the original. diff --git a/tools/README.md b/tools/README.md index 446391d..37a1e84 100644 --- a/tools/README.md +++ b/tools/README.md @@ -112,7 +112,7 @@ tools/ kb_state.py the KB version (.wikitool-kb.json) and the migration chain corpus_diff.py invariant comparison of kb/ between two revisions web_capture.py `raw fetch`'s core: fetch a page, decide its charset, derive Markdown-like text from the HTML - standard library only, deterministic - filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Office), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format + filters/ the import filters behind `raw derive`: a registry (`__init__.py`) and one module per format family (HTML, mail, Office, ZIP), each registering the extensions it reads - standard library, or a package imported lazily so a missing one fails only its own format repo_capture.py `raw capture`/`raw status`'s core: resolve a ref rule, fetch it into a bare cache, select files by glob as blobs, the `_capture.json` manifest - git with the host's credentials, never a prompt; plus the one write back, a single-file commit pushed without force for `export guidelines` guideline_export.py `export guidelines`' core: select pages by `search` predicates, render them into one deterministic GUIDELINES.md, and recognise an export again (`EXPORT_MARKER`) - with no CLI attached search/ pluggable search backends, plus service.py - the search core diff --git a/tools/chemenu/commands/raw_cmd.py b/tools/chemenu/commands/raw_cmd.py index c24da8c..8564930 100644 --- a/tools/chemenu/commands/raw_cmd.py +++ b/tools/chemenu/commands/raw_cmd.py @@ -1886,9 +1886,24 @@ def _collision_warning(name: str) -> Optional[str]: return _folder_collision_message(name, holder) -def _plan_derive_file(path: Path, name: Optional[str]): +@dataclass(frozen=True) +class _FilePlan: + """A single file turned into a folder, not yet written.""" + + folder: Path + stem: str + candidate: filters.Filter + original: Optional[Path] # where the source moves to; None when the source is only unpacked + derived: Optional[Path] + markdown: bytes + container: Optional[filters.Container] + derivations: list[filters.Derivation] + + +def _plan_derive_file(path: Path, name: Optional[str]) -> _FilePlan: """The single-file form: `incoming//` with the original moved in as - `original.` and the derived text beside it.""" + `original.` and the derived text beside it - or, for a container (a ZIP, + a mail), the members under their own names.""" incoming = _incoming_dir() if not path.is_file(): raise _Refused(f"{rel_path(path)} does not exist or is not a file.") @@ -1916,46 +1931,67 @@ def _plan_derive_file(path: Path, name: Optional[str]): "or remove what is there, or pass --name ." ) extension = filters.extension_of(path.name) - original = folder / f"{ORIGINAL_STEM}{extension}" - derived = folder / DERIVED_NAME - for target in (original, derived): - problem = path_budget_problem_for(_shard_dir() / stem / target.name) + data = _read_for_derive(path) + container = None + try: + filters.require_available(candidate) + container = candidate.container(data, path.name) + except ValidationError as exc: + raise _Refused(f"{rel_path(path)}: {exc}") from None + if container is not None and container.text is None: + original = derived = None + markdown = b"" + targets = [rel for rel, _data in container.files] + derivations = list(container.derivations) + else: + original = folder / f"{ORIGINAL_STEM}{extension}" + derived = folder / DERIVED_NAME + derivation = container.text if container is not None else _run_filter(candidate, data, path) + assert derivation is not None + markdown = filters.render( + candidate, derivation, derived_from=original.name, original_name=path.name + ) + targets = [original.name, derived.name] + ( + [rel for rel, _data in container.files] if container is not None else [] + ) + derivations = [derivation] + (list(container.derivations) if container is not None else []) + for target in targets: + problem = path_budget_problem_for(_shard_dir() / stem / target) if problem: raise _Refused( f"Cannot write {problem}. The folder name comes from the file's name: pass " "--name ." ) - derivation = _run_filter(candidate, _read_for_derive(path), path) - markdown = filters.render( - candidate, - derivation, - derived_from=original.name, - original_name=path.name, - ) - return folder, original, derived, candidate, derivation, markdown, stem + return _FilePlan(folder, stem, candidate, original, derived, markdown, container, derivations) -def _move_in(source: Path, folder: Path, original: Path, derived: Path, markdown: bytes) -> None: - """Create `folder`, write the derived text, then move the original in - last, - so a failure before it has taken nothing from `incoming/`. A failure part-way - puts back what this call changed.""" +def _write_folder(source: Path, plan: _FilePlan) -> None: + """Create the folder and write everything into it, then move the original in + (or, for an unpacked archive, remove it) - last, so a failure before it has + taken nothing from `incoming/`. A failure part-way removes the folder this + call made and puts the original back.""" moved = False try: - folder.mkdir() - with derived.open("xb") as handle: - handle.write(markdown) - source.rename(original) - moved = True + plan.folder.mkdir() + if plan.derived is not None: + with plan.derived.open("xb") as handle: + handle.write(plan.markdown) + if plan.container is not None: + for rel, data in plan.container.files: + target = plan.folder / rel + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("xb") as handle: + handle.write(data) + if plan.original is not None: + source.rename(plan.original) + moved = True + else: + source.unlink() except OSError as exc: - if moved: - original.rename(source) - derived.unlink(missing_ok=True) - if folder.is_dir(): - try: - folder.rmdir() - except OSError: - pass - fail(escape(f"Could not write {rel_path(folder)}: {exc}. incoming/ was left as it was.")) + if moved and plan.original is not None: + plan.original.rename(source) + shutil.rmtree(plan.folder, ignore_errors=True) + fail(escape(f"Could not write {rel_path(plan.folder)}: {exc}. incoming/ was left as it was.")) def _plan_derive_folder(folder: Path): @@ -1988,7 +2024,7 @@ def _plan_derive_folder(folder: Path): untouched: list[Path] = [] for file in files: candidate = filters.filter_for(file.name) - if candidate is None: + if candidate is None or candidate.container_only: untouched.append(file) else: plan.append((file, candidate, file.with_suffix(".md"))) @@ -2024,14 +2060,34 @@ def _plan_derive_folder(folder: Path): return derived, untouched -def _echo_notes(derivations: list[filters.Derivation]) -> None: - for derivation in derivations: - for warning in derivation.warnings: - typer.echo(f"WARN {warning}") - for line in filters.unique(c for d in derivations for c in d.checks): +def _echo_notes( + derivations: list[filters.Derivation], container: Optional[filters.Container] = None +) -> None: + warnings = [w for d in derivations for w in d.warnings] + checks = [c for d in derivations for c in d.checks] + if container is not None: + warnings += container.warnings + checks += container.checks + for warning in filters.unique(warnings): + typer.echo(f"WARN {warning}") + for line in filters.unique(checks): typer.echo(f"CHECK {line}") +def _container_report(container: filters.Container) -> list[str]: + lines = [f" {src} -> {md} ({name})" for src, md, name in container.derived] + if container.plain: + lines.append(" No filter for, written as they are: " + ", ".join(container.plain)) + if container.renamed: + lines.append( + " Renamed (a name was taken or reserved): " + + ", ".join(f"{old} -> {new}" for old, new in container.renamed) + ) + if container.hidden: + lines.append(" Hidden entries, named but not written: " + ", ".join(container.hidden)) + return lines + + def _derive_next(target: Path) -> str: return ( " Next (after the commitment question in wiki-ingest):\n" @@ -2041,8 +2097,8 @@ def _derive_next(target: Path) -> str: @cli_contract.record(cli_contract.CommandRecord( path="raw derive", - summary="Derive reading text from a PDF, e-mail, Office file or web page in `incoming/` and " - "leave the original beside it, for `raw accept` to promote.", + summary="Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`, or " + "unpack a ZIP, and leave the original beside it, for `raw accept` to promote.", synopsis=( cli_contract.Variant( usage="raw derive incoming/ [--name ]", @@ -2078,8 +2134,20 @@ def _derive_next(target: Path) -> str: "`derived` differs between runs. `engine` names the tool version and, where a package " "reads the format, its version.", "Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, " - "`.xlsm`) and e-mail (`.eml`) read with the standard library only. A mail with " - "attachments is refused.", + "`.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library " + "only.", + "A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder " + "`incoming//` holding its files under their own names, each with a `.md` " + "where a filter reads it; the ZIP is removed and the output names it with its sha256. A " + "mail becomes a folder with `original.eml`, `derived.md` and its attachments under their " + "own names; a ZIP attached to it is unpacked once into a folder of its own, and an " + "attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an " + "absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a " + "file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries " + "(`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name " + "taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, " + "`_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists " + "each rename.", "A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a " "later edition of that source or a separate one is the human's call. A later edition of " "a derived source is replaced file by file with `raw accept --replaces`; there is no " @@ -2103,8 +2171,9 @@ def _derive_next(target: Path) -> str: "then retry once. An existing `.md` is never overwritten", ), cli_contract.Failure( - cause="The document cannot be read (damaged, encrypted, a mail with attachments) or " - "the filter's package is not installed", + cause="The document cannot be read (damaged, encrypted), a container breaks one of its " + "rules (a ZIP inside a ZIP, an unsafe path, a limit) or the filter's package is not " + "installed", reaction="Nothing was written. A missing package is reported with the preflight " "hint - show it to the user and stop, never install it. Any other filter keeps " "working", @@ -2114,6 +2183,7 @@ def _derive_next(target: Path) -> str: "tools/wikitool raw derive incoming/report.docx", "tools/wikitool raw derive incoming/Angebot-XYZ.xlsx --name angebot-xyz", "tools/wikitool raw derive incoming/meeting-files", + "tools/wikitool raw derive incoming/scans.zip", ), never=( "Never edit `derived.md` or its header by hand.", @@ -2154,19 +2224,34 @@ def raw_derive_command( typer.echo(_derive_next(resolved)) return - folder, original, derived_md, candidate, derivation, markdown, stem = _refusals_fail( - _plan_derive_file, resolved, name - ) - _move_in(resolved, folder, original, derived_md, markdown) - success(escape( - f"Derived {rel_path(derived_md)} from {resolved.name} ({candidate.name}); the original is " - f"now {rel_path(original)}." - )) - warning = _collision_warning(stem) + plan = _refusals_fail(_plan_derive_file, resolved, name) + _write_folder(resolved, plan) + container = plan.container + if plan.original is None: + success(escape( + f"Unpacked {resolved.name} into {rel_path(plan.folder)}/ ({plan.candidate.name}); " + f"{resolved.name} is removed." + )) + elif container is not None and container.files: + success(escape( + f"Derived {rel_path(plan.derived)} from {resolved.name} ({plan.candidate.name}) with " + f"{len(container.files)} file(s) from its attachments; the original is now " + f"{rel_path(plan.original)}." + )) + else: + success(escape( + f"Derived {rel_path(plan.derived)} from {resolved.name} ({plan.candidate.name}); the " + f"original is now {rel_path(plan.original)}." + )) + if container is not None: + report = _container_report(container) + if report: + typer.echo("\n".join(report)) + warning = _collision_warning(plan.stem) if warning: typer.echo("WARN " + warning) - _echo_notes([derivation]) - typer.echo(_derive_next(folder)) + _echo_notes(plan.derivations, container) + typer.echo(_derive_next(plan.folder)) # --- raw capture / raw status ------------------------------------------------- diff --git a/tools/chemenu/filters/__init__.py b/tools/chemenu/filters/__init__.py index 9ba6710..9a50d2d 100644 --- a/tools/chemenu/filters/__init__.py +++ b/tools/chemenu/filters/__init__.py @@ -49,12 +49,43 @@ class Derivation: raise ValueError(f"a filter prints at most {MAX_CHECKS} CHECK lines, got {len(self.checks)}") +@dataclass(frozen=True) +class Container: + """What a container filter makes of one document that holds other files. + + `files` is everything `raw derive` writes into the new folder besides the + container's own `derived.md`: the members under their claimed names and the + `.md` beside each member that has a filter. `text` is the container's own + derivation (a mail's body); a ZIP has none, because the archive is removed. + """ + + files: tuple[tuple[str, bytes], ...] = () + derived: tuple[tuple[str, str, str], ...] = () # (member, its .md, filter name) + derivations: tuple[Derivation, ...] = () + plain: tuple[str, ...] = () # members written as they are - no filter reads them + renamed: tuple[tuple[str, str], ...] = () # (name in the container, name written) + hidden: tuple[str, ...] = () # named but not written + warnings: tuple[str, ...] = () + checks: tuple[str, ...] = () + text: Optional[Derivation] = None + + def __post_init__(self) -> None: + if len(self.checks) > MAX_CHECKS: + raise ValueError(f"a filter prints at most {MAX_CHECKS} CHECK lines, got {len(self.checks)}") + + class Filter: - """One format family. Subclasses set the class attributes and implement `derive`.""" + """One format family. Subclasses set the class attributes and implement `derive`. + + A filter that can hold other files also implements `container`; with + `container_only` it is not a document format inside a folder or another + container - a ZIP is unpacked at the top or refused, never derived in place. + """ name: str = "" extensions: tuple[str, ...] = () requires: tuple[str, ...] = () + container_only: bool = False def available(self) -> bool: for package in self.requires: @@ -88,6 +119,11 @@ class Filter: def derive(self, data: bytes, name: str) -> Derivation: raise NotImplementedError + def container(self, data: bytes, name: str) -> Optional[Container]: + """The folder content for a document at the top of `incoming/`, or None + when the document is an ordinary single file.""" + return None + _REGISTRY: list[Filter] = [] @@ -169,4 +205,4 @@ def unique(lines: Iterable[str]) -> list[str]: return out -from chemenu.filters import html_page, mail, ooxml # noqa: E402,F401 (register themselves) +from chemenu.filters import archive, html_page, mail, ooxml # noqa: E402,F401 (register themselves) diff --git a/tools/chemenu/filters/archive.py b/tools/chemenu/filters/archive.py new file mode 100644 index 0000000..a046d8b --- /dev/null +++ b/tools/chemenu/filters/archive.py @@ -0,0 +1,255 @@ +"""ZIP archives, and the machinery every container shares: claiming names in the +new folder, unpacking an archive under limits, and deriving the members. + +A container's members are checked and derived completely before `raw derive` +writes the first file, so a refusal leaves `incoming/` as it was. +""" +from __future__ import annotations + +import hashlib +import io +import os +import zipfile +from typing import Optional + +from chemenu import repo_capture +from chemenu.errors import ValidationError +from chemenu.filters import ( + MAX_CHECKS, + Container, + Derivation, + Filter, + filter_for, + register, + render, + require_available, +) + +MAX_ENTRIES = 1000 +MAX_FILE_BYTES = 25 * 1024 * 1024 +MAX_TOTAL_BYTES = 250 * 1024 * 1024 + +_SYMLINK = 0o120000 +_TYPE_MASK = 0o170000 + + +def clean_name(name: str, fallback: str) -> str: + """One file name out of whatever a mail or an archive calls an attachment: + the part after the last separator, without control characters or leading + dots.""" + base = name.replace("\\", "/").rsplit("/", 1)[-1] + base = "".join(ch for ch in base if ch >= " " and ch != "\x7f").strip().lstrip(".").strip() + return base or fallback + + +def _split(name: str) -> tuple[str, str]: + base, ext = os.path.splitext(name) + return (name, "") if not base else (base, ext) + + +def _is_hidden(parts: list[str]) -> bool: + return any(part.startswith(".") or part == "__MACOSX" for part in parts) + + +class Tree: + """The names claimed in the new folder. A name already taken, or reserved, + gets `-2`, `-3` before its extension; names compare case-insensitively + because the folder may travel to a case-insensitive file system.""" + + def __init__(self, reserve_top: bool) -> None: + self._reserve_top = reserve_top + self._files: set[str] = set() + self._dirs: set[str] = set() + + def _reserved(self, path: str) -> bool: + name = path.rsplit("/", 1)[-1].lower() + if name == repo_capture.MANIFEST_NAME.lower(): + return True + return self._reserve_top and "/" not in path and ( + name == "derived.md" or name == "original" or name.startswith("original.") + ) + + def _taken(self, path: str) -> bool: + key = path.lower() + return key in self._files or key in self._dirs or self._reserved(path) + + def _free(self, path: str, split: bool) -> str: + directory, _, name = path.rpartition("/") + base, ext = _split(name) if split else (name, "") + candidate, number = path, 1 + while self._taken(candidate): + number += 1 + candidate = (directory + "/" if directory else "") + f"{base}-{number}{ext}" + return candidate + + def _check_parents(self, path: str, original: str) -> None: + parts = path.split("/") + for depth in range(1, len(parts)): + parent = "/".join(parts[:depth]) + if parent.lower() in self._files: + raise ValidationError( + f"{original!r} needs a folder named {parent!r}, but a file of that name is " + "already in the container." + ) + self._dirs.add(parent.lower()) + + def claim(self, path: str) -> str: + claimed = self._free(path, split=True) + self._check_parents(claimed, path) + self._files.add(claimed.lower()) + return claimed + + def claim_folder(self, path: str) -> str: + claimed = self._free(path, split=False) + self._check_parents(claimed, path) + self._dirs.add(claimed.lower()) + return claimed + + +class Collector: + """Collects the members of one container, then derives them.""" + + def __init__(self, reserve_top: bool) -> None: + self._tree = Tree(reserve_top) + self._pending: list[tuple[str, str, bytes]] = [] # (claimed path, name it came with, data) + self._hidden: list[str] = [] + self._renamed: list[tuple[str, str]] = [] + self._unpacked: list[tuple[str, str]] = [] + self._bytes = 0 + + def add_file(self, path: str, data: bytes, came_as: Optional[str] = None) -> str: + claimed = self._tree.claim(path) + if claimed != path: + self._renamed.append((came_as or path, claimed)) + self._pending.append((claimed, (came_as or path).rsplit("/", 1)[-1], data)) + return claimed + + def add_zip(self, folder: str, data: bytes, label: str) -> str: + """Unpack an archive under `folder` (empty: at the top). Refuses an archive that + holds another one, an unsafe entry or anything over a limit.""" + prefix = "" + if folder: + claimed = self._tree.claim_folder(folder) + if claimed != folder: + self._renamed.append((folder + "/", claimed + "/")) + prefix = claimed + "/" + try: + archive = zipfile.ZipFile(io.BytesIO(data)) + infos = archive.infolist() + except zipfile.BadZipFile as exc: + raise ValidationError(f"{label} is not a readable ZIP archive: {exc}.") from None + if len(infos) > MAX_ENTRIES: + raise ValidationError( + f"{label} holds {len(infos)} entries; raw derive unpacks at most {MAX_ENTRIES}." + ) + added = hidden_here = 0 + for info in infos: + name = info.filename.replace("\\", "/") + if info.is_dir() or name.endswith("/"): + continue + if name.startswith("/") or (len(name) > 1 and name[1] == ":"): + raise ValidationError(f"{label}: the entry {info.filename!r} has an absolute path.") + parts = [part for part in name.split("/") if part not in ("", ".")] + if ".." in parts: + raise ValidationError(f"{label}: the entry {info.filename!r} leaves the folder (..).") + if (info.external_attr >> 16) & _TYPE_MASK == _SYMLINK: + raise ValidationError(f"{label}: the entry {info.filename!r} is a symbolic link.") + if info.flag_bits & 0x1: + raise ValidationError(f"{label}: the entry {info.filename!r} is encrypted.") + if not parts: + continue + if _is_hidden(parts): + self._hidden.append(prefix + "/".join(parts)) + hidden_here += 1 + continue + if parts[-1].lower().endswith(".zip"): + raise ValidationError( + f"{label}: the entry {info.filename!r} is itself a ZIP archive; nested archives " + "are not unpacked. Unpack it by hand and put its files into incoming/." + ) + if info.file_size > MAX_FILE_BYTES: + raise ValidationError( + f"{label}: the entry {info.filename!r} is {info.file_size // (1024 * 1024)} MiB " + f"unpacked; raw derive unpacks files up to {MAX_FILE_BYTES // (1024 * 1024)} MiB." + ) + try: + with archive.open(info) as handle: + content = handle.read(MAX_FILE_BYTES + 1) + except (zipfile.BadZipFile, NotImplementedError, RuntimeError, OSError) as exc: + raise ValidationError(f"{label}: cannot read {info.filename!r}: {exc}.") from None + if len(content) > MAX_FILE_BYTES: + raise ValidationError(f"{label}: the entry {info.filename!r} is larger than it says.") + self._bytes += len(content) + if self._bytes > MAX_TOTAL_BYTES: + raise ValidationError( + f"{label} unpacks to more than {MAX_TOTAL_BYTES // (1024 * 1024)} MiB in total; " + "raw derive refuses it." + ) + cleaned = [clean_name(part, "file") for part in parts] + self.add_file(prefix + "/".join(cleaned), content, came_as="/".join(parts)) + added += 1 + if added == 0 and hidden_here == 0: + raise ValidationError(f"{label} holds no files.") + self._unpacked.append((label, hashlib.sha256(data).hexdigest())) + return prefix + + def finish(self, text: Optional[Derivation] = None, warnings: tuple[str, ...] = ()) -> Container: + files: list[tuple[str, bytes]] = [(path, data) for path, _name, data in self._pending] + derived: list[tuple[str, str, str]] = [] + derivations: list[Derivation] = [] + plain: list[str] = [] + for path, came_as, data in self._pending: + base = path.rsplit("/", 1)[-1] + candidate = filter_for(base) + if candidate is None or candidate.container_only: + plain.append(path) + continue + try: + require_available(candidate) + derivation = candidate.derive(data, base) + except ValidationError as exc: + raise ValidationError(f"{path}: {exc}") from None + stem = path[: len(path) - len(os.path.splitext(base)[1])] + md = self._tree.claim(stem + ".md") + files.append(( + md, + render(candidate, derivation, derived_from=base, original_name=came_as if came_as != base else None), + )) + derived.append((path, md, candidate.name)) + derivations.append(derivation) + checks: list[str] = [] + if self._unpacked: + named = ", ".join(f"{label} (sha256 {digest})" for label, digest in self._unpacked) + checks.append( + f"Unpacked from {named}; the archive is not kept - name it on the source page." + ) + return Container( + files=tuple(files), + derived=tuple(derived), + derivations=tuple(derivations), + plain=tuple(plain), + renamed=tuple(self._renamed), + hidden=tuple(self._hidden), + warnings=warnings, + checks=tuple(checks[:MAX_CHECKS]), + text=text, + ) + + +class ZipFilter(Filter): + """A ZIP archive at the top of `incoming/` becomes a folder of its files.""" + + name = "zip" + extensions = (".zip",) + container_only = True + + def derive(self, data: bytes, name: str) -> Derivation: + raise ValidationError(f"{name}: a ZIP archive is unpacked at the top of incoming/, not derived in place.") + + def container(self, data: bytes, name: str) -> Optional[Container]: + collector = Collector(reserve_top=False) + collector.add_zip("", data, name) + return collector.finish() + + +register(ZipFilter()) diff --git a/tools/chemenu/filters/mail.py b/tools/chemenu/filters/mail.py index cc11116..046fd8f 100644 --- a/tools/chemenu/filters/mail.py +++ b/tools/chemenu/filters/mail.py @@ -9,7 +9,8 @@ from typing import Optional from chemenu import web_capture from chemenu.errors import ValidationError -from chemenu.filters import Derivation, Filter, register +from chemenu.filters import Container, Derivation, Filter, register +from chemenu.filters.archive import Collector, clean_name _QUOTED = re.compile( r"^(>|-{2,}\s*(Original Message|Ursprüngliche Nachricht)\s*-{2,}|On .{5,200} wrote:\s*$|Am .{5,200} schrieb .{0,200}:\s*$)", @@ -25,7 +26,7 @@ class Attachment: @property def is_message(self) -> bool: - return self.content_type == "message/rfc822" or self.name.lower().endswith((".eml", ".msg")) + return self.content_type == "message/rfc822" or self.name.lower().endswith(".eml") @dataclass @@ -88,6 +89,16 @@ def _filename(part: EmailMessage, number: int) -> str: return name +def _leaves(part: EmailMessage): + """The parts of a mail that are not containers, without descending into an + attached message: its own body and attachments belong to it, not to this mail.""" + if part.get_content_type() == "message/rfc822" or not part.is_multipart(): + yield part + return + for child in part.iter_parts(): + yield from _leaves(child) + + def parse(data: bytes) -> ParsedMail: msg = message_from_bytes(data, policy=policy.default) if not isinstance(msg, EmailMessage) or not (msg.keys() or msg.get_payload()): @@ -108,8 +119,8 @@ def parse(data: bytes) -> ParsedMail: out.body = web_capture.derive_text(_text_of(body_part), "").strip() else: out.body = _text_of(body_part).replace("\r\n", "\n").replace("\r", "\n").strip() - for number, part in enumerate(msg.walk(), 1): - if part.is_multipart() or part is body_part or part is msg: + for number, part in enumerate(_leaves(msg), 1): + if part is body_part or part is msg: continue disposition = (part.get_content_disposition() or "").lower() if ( @@ -152,26 +163,83 @@ def mail_checks(parsed: ParsedMail) -> tuple[str, ...]: return tuple(checks) +def _quote_section(inner: ParsedMail) -> str: + h = inner.headers + lines = [f"## Attached message: {h['subject'] or '(no subject)'}", ""] + for label, key in (("From", "from"), ("To", "to"), ("Cc", "cc"), ("Date", "date")): + value = h[key] + value = ", ".join(value) if isinstance(value, list) else value + if value: + lines.append(f"{label}: {value}") + if inner.attachments: + lines.append("Attachments (not unpacked): " + ", ".join(a.name for a in inner.attachments)) + lines.append("") + lines += [f"> {line}" if line else ">" for line in inner.body.splitlines()] + return "\n".join(lines) + + +def _subject(parsed: ParsedMail) -> str: + subject = parsed.headers["subject"] + return subject if isinstance(subject, str) else "" + + class EmlFilter(Filter): + """A mail. At the top of `incoming/` it is a container: its attachments become files + beside it. Anywhere else - in a ZIP or in a folder - its attachments are only named.""" + name = "eml" extensions = (".eml",) def derive(self, data: bytes, name: str) -> Derivation: parsed = parse(data) + warnings = list(parsed.warnings) if parsed.attachments: - names = ", ".join(a.name for a in parsed.attachments) - raise ValidationError( - f"The mail carries {len(parsed.attachments)} attachment(s) ({names}); deriving a mail " - "with attachments is not supported yet. Save the attachments as files beside the " - "mail and derive each file on its own." + warnings.append( + f"The mail has {len(parsed.attachments)} attachment(s) that are not unpacked here: " + + ", ".join(a.name for a in parsed.attachments) ) return Derivation( text=parsed.body, - title=parsed.headers["subject"] if isinstance(parsed.headers["subject"], str) else "", + title=_subject(parsed), fields=mail_fields(parsed), - warnings=tuple(parsed.warnings), + warnings=tuple(warnings), checks=mail_checks(parsed), ) + def container(self, data: bytes, name: str) -> Optional[Container]: + parsed = parse(data) + collector = Collector(reserve_top=True) + sections: list[str] = [] + warnings = list(parsed.warnings) + for number, attachment in enumerate(parsed.attachments, 1): + file_name = clean_name(attachment.name, f"attachment-{number}") + if attachment.is_message: + try: + inner = parse(attachment.data) + except ValidationError: + collector.add_file(file_name, attachment.data) + continue + sections.append(_quote_section(inner)) + if inner.attachments: + warnings.append( + f"The attached message {attachment.name!r} has attachments that are not " + "unpacked: " + ", ".join(a.name for a in inner.attachments) + ) + elif file_name.lower().endswith(".zip"): + collector.add_zip(file_name[: -len(".zip")] or "archive", attachment.data, file_name) + else: + collector.add_file(file_name, attachment.data) + text = parsed.body + if sections: + text = (text + "\n\n" if text else "") + "\n\n".join(sections) + derivation = Derivation( + text=text, + title=_subject(parsed), + fields=mail_fields(parsed), + warnings=tuple(warnings), + checks=mail_checks(parsed), + ) + return collector.finish(text=derivation) + register(EmlFilter()) diff --git a/tools/chemenu/tests/test_portability.py b/tools/chemenu/tests/test_portability.py index 9935e04..04ac71d 100644 --- a/tools/chemenu/tests/test_portability.py +++ b/tools/chemenu/tests/test_portability.py @@ -27,9 +27,10 @@ TOOLS_DIR = Path(__file__).resolve().parents[2] PACKAGE_DIR = TOOLS_DIR / "chemenu" # Receivers whose `open` is not a text-file open: `os.open` takes flags, the -# archive modules open binary members, and `opener` is a `urllib` -# `OpenerDirector` (`web_capture.fetch`), whose `open` sends an HTTP request. -NOT_TEXT_OPEN = {"os", "tarfile", "zipfile", "gzip", "opener"} +# archive modules open binary members (`archive` is the `zipfile.ZipFile` in +# `filters/archive.py`), and `opener` is a `urllib` `OpenerDirector` +# (`web_capture.fetch`), whose `open` sends an HTTP request. +NOT_TEXT_OPEN = {"os", "tarfile", "zipfile", "gzip", "archive", "opener"} # The one module allowed to import the platform lock modules. LOCK_MODULE = PACKAGE_DIR / "filelock.py" diff --git a/tools/chemenu/tests/test_raw_derive.py b/tools/chemenu/tests/test_raw_derive.py index 49e506a..2298310 100644 --- a/tools/chemenu/tests/test_raw_derive.py +++ b/tools/chemenu/tests/test_raw_derive.py @@ -5,9 +5,11 @@ readable in the test that uses it and no binary sits in the repository. """ from __future__ import annotations +import hashlib import io import json import zipfile +from email import message_from_bytes, policy from email.message import EmailMessage from pathlib import Path @@ -565,14 +567,6 @@ def test_eml_with_inline_image_warns_and_is_not_an_attachment(tree, capsys): assert "inline image" in capsys.readouterr().out -def test_eml_with_an_attachment_is_refused_untouched(tree, capsys): - _put(tree, "mail.eml", make_eml(attach=True)) - before = _snapshot(tree / "incoming") - _refused("incoming/mail.eml") - assert "a.bin" in capsys.readouterr().out - assert _snapshot(tree / "incoming") == before - - # --- the registry ----------------------------------------------------------- @@ -674,3 +668,279 @@ def test_the_web_page_filter_derives_a_saved_html_file(tree): assert header["filter"] == "html" and header["title"] == "T" assert header["charset"].startswith("utf-8") assert "# Kopf" in text + + +# --- containers: ZIP and mail with attachments ------------------------------- + + +def make_zip(entries: dict[str, bytes]) -> bytes: + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as zf: + for name, content in entries.items(): + zf.writestr(name, content) + return buffer.getvalue() + + +def make_zip_with(info: zipfile.ZipInfo, content: bytes = b"x") -> bytes: + encrypted = bool(info.flag_bits & 0x1) + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w") as zf: + zf.writestr(info, content) + data = buffer.getvalue() + if encrypted: + # zipfile resets the flags on write; set the encryption bit in both headers by hand + marker = data.index(b"PK\x01\x02") + flags = marker + 8 + data = data[:flags] + bytes([data[flags] | 1]) + data[flags + 1:] + return data + + +def make_mail(attachments: list[tuple[str, bytes, str]], body: str = "See attached.\n") -> bytes: + msg = EmailMessage() + msg["From"] = "Anna Beispiel " + msg["To"] = "bob@example.org" + msg["Subject"] = "Unterlagen" + msg["Date"] = "Wed, 07 Oct 2026 09:30:00 +0200" + msg.set_content(body) + for name, data, ctype in attachments: + maintype, subtype = ctype.split("/") + if ctype == "message/rfc822": + msg.add_attachment(message_from_bytes(data, policy=policy.default)) + else: + msg.add_attachment(data, maintype=maintype, subtype=subtype, filename=name) + return msg.as_bytes() + + +DOCX = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + + +def _fresh_docx(text: str) -> bytes: + return make_docx(_p(text)) + + +def test_a_zip_becomes_a_folder_of_its_files_and_is_removed(tree, capsys): + one, two = _fresh_docx("One"), _fresh_docx("Two") + entries = {"one.docx": one, "sub/two.docx": two, "notes.txt": b"plain\n"} + archive = make_zip(entries) + _put(tree, "bundle.zip", archive) + _derive("incoming/bundle.zip") + out = capsys.readouterr().out + + folder = tree / "incoming/bundle" + assert (folder / "one.docx").read_bytes() == one + assert (folder / "sub/two.docx").read_bytes() == two + assert (folder / "notes.txt").read_bytes() == b"plain\n" + assert "One" in (folder / "one.md").read_text(encoding="utf-8") + assert "Two" in (folder / "sub/two.md").read_text(encoding="utf-8") + assert not (tree / "incoming/bundle.zip").exists() + assert not (folder / "original.zip").exists() and not (folder / "derived.md").exists() + assert "bundle.zip" in out and hashlib.sha256(archive).hexdigest() in out + assert "notes.txt" in out + + raw_accept_command( + files=[Path("incoming/bundle")], fidelity="verbatim", authority="reporting", + page=None, replaces=None, dry_run=False, + ) + assert list((tree / "raw").rglob("one.docx")) + + +def test_a_zip_takes_name_and_names_the_hidden_entries(tree, capsys): + _put(tree, "bundle.zip", make_zip({ + "a.txt": b"a", "__MACOSX/._a.txt": b"x", ".DS_Store": b"x", "sub/.hidden": b"x", + })) + _derive("incoming/bundle.zip", "kurz") + out = capsys.readouterr().out + assert sorted(p.name for p in (tree / "incoming/kurz").rglob("*") if p.is_file()) == ["a.txt"] + for hidden in ("__MACOSX/._a.txt", ".DS_Store", "sub/.hidden"): + assert hidden in out + + +@pytest.mark.parametrize("builder", [ + lambda: make_zip({"inner.zip": make_zip({"a.txt": b"a"})}), + lambda: make_zip({"../escape.txt": b"x"}), + lambda: make_zip({"/abs.txt": b"x"}), + lambda: make_zip_with(_info("link", symlink=True)), + lambda: make_zip_with(_info("secret.txt", encrypted=True)), + lambda: b"not a zip at all", +]) +def test_a_zip_that_breaks_a_rule_is_refused_and_incoming_is_untouched(tree, builder): + _put(tree, "bad.zip", builder()) + before = _snapshot(tree / "incoming") + _refused("incoming/bad.zip") + assert _snapshot(tree / "incoming") == before + assert not (tree / "incoming/bad").exists() + + +def _info(name: str, *, symlink: bool = False, encrypted: bool = False) -> zipfile.ZipInfo: + info = zipfile.ZipInfo(name) + if symlink: + info.external_attr = 0o120777 << 16 + if encrypted: + info.flag_bits |= 0x1 + return info + + +@pytest.mark.parametrize("limit", ["MAX_ENTRIES", "MAX_FILE_BYTES", "MAX_TOTAL_BYTES"]) +def test_a_zip_over_a_limit_is_refused(tree, monkeypatch, limit): + from chemenu.filters import archive + + monkeypatch.setattr(archive, limit, 2 if limit == "MAX_ENTRIES" else 5) + _put(tree, "big.zip", make_zip({"a.txt": b"123456", "b.txt": b"123456", "c.txt": b"1"})) + before = _snapshot(tree / "incoming") + _refused("incoming/big.zip") + assert _snapshot(tree / "incoming") == before + + +def test_a_zip_entry_whose_path_would_be_over_budget_is_refused_naming_name(tree, capsys): + _put(tree, "z.zip", make_zip({"d" * 100 + "/" + "f" * 80 + ".txt": b"x"})) + before = _snapshot(tree / "incoming") + _refused("incoming/z.zip") + assert "--name" in capsys.readouterr().out + assert _snapshot(tree / "incoming") == before + + +def test_a_utf8_entry_name_survives(tree): + _put(tree, "u.zip", make_zip({"Übersicht Ä.txt": b"x"})) + _derive("incoming/u.zip") + assert (tree / "incoming/u/Übersicht Ä.txt").is_file() + + +def test_duplicate_zip_entries_get_a_number(tree, capsys): + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w") as zf: + zf.writestr("a.txt", b"first") + zf.writestr("A.txt", b"second") + zf.writestr("_capture.json", b"{}") + _put(tree, "d.zip", buffer.getvalue()) + _derive("incoming/d.zip") + out = capsys.readouterr().out + names = sorted(p.name for p in (tree / "incoming/d").iterdir()) + assert names == ["A-2.txt", "_capture-2.json", "a.txt"] + assert "A.txt -> A-2.txt" in out + + +def test_a_zip_inside_a_folder_is_left_alone_and_named(tree, capsys): + folder = tree / "incoming/f" + folder.mkdir() + (folder / "a.docx").write_bytes(_fresh_docx("A")) + (folder / "z.zip").write_bytes(make_zip({"a.txt": b"a"})) + _derive("incoming/f") + assert "z.zip" in capsys.readouterr().out + assert (folder / "z.zip").is_file() and (folder / "a.md").is_file() + + +def test_a_mail_with_attachments_is_a_folder_of_them(tree, capsys): + report = _fresh_docx("Report") + inner_one, inner_two = _fresh_docx("Inner one"), _fresh_docx("Inner two") + archive = make_zip({"i1.docx": inner_one, "i2.docx": inner_two}) + raw_mail = make_mail([ + ("report.docx", report, DOCX), + ("papers.zip", archive, "application/zip"), + ("picture.bin", b"\x00\x01", "application/octet-stream"), + ]) + _put(tree, "mail.eml", raw_mail) + _derive("incoming/mail.eml") + out = capsys.readouterr().out + + folder = tree / "incoming/mail" + assert (folder / "original.eml").read_bytes() == raw_mail + assert (folder / "report.docx").read_bytes() == report + assert "Report" in (folder / "report.md").read_text(encoding="utf-8") + assert (folder / "papers/i1.docx").read_bytes() == inner_one + assert "Inner two" in (folder / "papers/i2.md").read_text(encoding="utf-8") + assert (folder / "picture.bin").is_file() + assert not (folder / "papers.zip").exists() and not (tree / "incoming/mail.eml").exists() + header, text = _header_and_text(folder / "derived.md") + assert header["attachments"] == ["report.docx", "papers.zip", "picture.bin"] + assert "See attached." in text + assert "papers.zip" in out and "sha256" in out + + raw_accept_command( + files=[Path("incoming/mail")], fidelity="verbatim", authority="reporting", + page=None, replaces=None, dry_run=False, + ) + assert list((tree / "raw").rglob("original.eml")) + + +def test_a_mail_whose_zip_attachment_holds_a_zip_is_refused_untouched(tree): + nested = make_zip({"inner.zip": make_zip({"a.txt": b"a"})}) + _put(tree, "mail.eml", make_mail([("n.zip", nested, "application/zip")])) + before = _snapshot(tree / "incoming") + _refused("incoming/mail.eml") + assert _snapshot(tree / "incoming") == before + + +def test_an_attached_mail_is_quoted_not_unpacked(tree, capsys): + inner = make_mail([("deep.bin", b"zz", "application/octet-stream")], body="Inner words\n") + _put(tree, "mail.eml", make_mail([("fwd.eml", inner, "message/rfc822")])) + _derive("incoming/mail.eml") + out = capsys.readouterr().out + folder = tree / "incoming/mail" + assert sorted(p.name for p in folder.iterdir()) == ["derived.md", "original.eml"] + _header, text = _header_and_text(folder / "derived.md") + assert "## Attached message: Unterlagen" in text + assert "> Inner words" in text + assert "deep.bin" in text + assert "WARN" in out and "deep.bin" in out + + +def test_a_mail_inside_a_zip_gets_its_md_and_its_attachments_are_only_named(tree, capsys): + mail = make_mail([("a.bin", b"x", "application/octet-stream")]) + _put(tree, "m.zip", make_zip({"msg.eml": mail})) + _derive("incoming/m.zip") + out = capsys.readouterr().out + folder = tree / "incoming/m" + assert sorted(p.name for p in folder.iterdir()) == ["msg.eml", "msg.md"] + assert "a.bin" in out and "WARN" in out + + +def test_attachment_names_that_collide_get_a_number(tree, capsys): + _put(tree, "mail.eml", make_mail([ + ("a.bin", b"1", "application/octet-stream"), + ("a.bin", b"2", "application/octet-stream"), + ("derived.md", b"3", "text/markdown"), + ("_capture.json", b"4", "application/json"), + ("original.docx", _fresh_docx("O"), DOCX), + ("../evil/b.bin", b"5", "application/octet-stream"), + ])) + _derive("incoming/mail.eml") + out = capsys.readouterr().out + names = sorted(p.name for p in (tree / "incoming/mail").iterdir()) + assert names == [ + "_capture-2.json", "a-2.bin", "a.bin", "b.bin", "derived-2.md", "derived.md", + "original-2.docx", "original-2.md", "original.eml", + ] + for renamed in ("a.bin -> a-2.bin", "derived.md -> derived-2.md", "_capture.json -> _capture-2.json"): + assert renamed in out + + +def test_a_derived_md_never_takes_the_name_of_a_real_file(tree): + _put(tree, "z.zip", make_zip({"a.docx": _fresh_docx("From docx"), "a.md": b"real notes\n"})) + _derive("incoming/z.zip") + folder = tree / "incoming/z" + assert (folder / "a.md").read_bytes() == b"real notes\n" + assert "From docx" in (folder / "a-2.md").read_text(encoding="utf-8") + + +def test_a_container_failing_to_write_leaves_incoming_as_it_was(tree, monkeypatch): + _put(tree, "bundle.zip", make_zip({"a.txt": b"a", "b.txt": b"b"})) + before = _snapshot(tree / "incoming") + + def broken(self): + raise OSError("disk full") + + monkeypatch.setattr(Path, "unlink", broken) + with pytest.raises(typer.Exit): + _derive("incoming/bundle.zip") + monkeypatch.undo() + assert _snapshot(tree / "incoming") == before + assert not (tree / "incoming/bundle").exists() + + +def test_check_lines_for_a_zip_of_xlsx_have_no_mail_line(tree, capsys): + _put(tree, "s.zip", make_zip({"sheet.xlsx": make_xlsx()})) + _derive("incoming/s.zip") + out = capsys.readouterr().out + checks = [line for line in out.splitlines() if line.startswith("CHECK")] + assert any("Values are as last saved" in line for line in checks) + assert not any("commitment" in line for line in checks)