raw derive: a placeholder text/plain no longer displaces the HTML text of a mail (#204)
A mail's text is its plain text unless that is under 10% of the HTML text, then the HTML text with a warning; one rule for .eml and .msg. The other version of the text in a multipart/alternative is no longer an attachment. Files changed: - CHANGES.md - VERSION - docs/reference/commands/raw-material-and-uploads.md - testfiles/mail/gog-roundcube.eml.expect - tools/CONTRACT.md - tools/chemenu/commands/raw_cmd.py - tools/chemenu/filters/mail.py - tools/chemenu/filters/msg.py - tools/chemenu/tests/test_raw_derive.py - tools/chemenu/tests/test_raw_derive_binary.py Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Code committed
1 parent
be049217a9
commit
918157cb41
10 files changed
+193
-21
No files matched your search
+1
-1
@@ -1515,7 +1515,7 @@ Derive reading text from a PDF, e-mail, Office file or web page in `incoming/`,
|
||||
- A single file directly in `incoming/` becomes a folder named after its stem (or `--name`) holding `original.<ext>` - the original bytes, moved - and `derived.md`. The fixed names keep the path inside the path budget; the header of `derived.md` records the file's own name as `original_name`.
|
||||
- A folder is derived in place: each file with a filter gets `<stem>.md` beside it, nothing is renamed, and the files without a filter are named in the output.
|
||||
- `derived.md` starts with a fixed header (`derived_by`, `filter`, `engine`, `derived`, `derived_from`, `original_name` for a renamed file, `title`, then the fields of the format) followed by the derived text. The same bytes always give the same text; only `derived` differs between runs. `engine` names the tool version and, where a package reads the format, its version.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read.
|
||||
- Formats: web pages (`.html`), Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`, `.xlsm`), e-mail (`.eml`) and ZIP archives (`.zip`), read with the standard library only; PDF (`.pdf`, `pypdf`) and Outlook messages (`.msg`, `olefile`), each read by a package in `tools/requirements.txt` that is imported only when such a file is derived. A missing package refuses that format alone, with the preflight hint; `wikitool doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets `<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` alike) is its plain text, unless that is under 10% of the length of the HTML text - a "plain text version not available" placeholder - in which case it is the HTML text and a WARN names both lengths. The other version of the text in a `multipart/alternative` is not an attachment.
|
||||
- A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder `<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet - and, for a sheet with formulas, `NN-<sheet>.formulas.tsv` listing cell, formula and last stored value. The `.md` is the overview: one section per sheet with its range, the file names and a preview. Formulas are not recalculated; a shared formula is written out per cell, and one that cannot be is named `shared:<master cell>` with a WARN. The folder counts against the path budget and is accepted with the rest; an existing one is never overwritten.
|
||||
- A ZIP and a mail are containers. A ZIP at the top of `incoming/` becomes the folder `incoming/<stem>/` holding its files under their own names, each with a `<stem>.md` where a filter reads it; the ZIP is removed and the output names it with its sha256. A mail becomes a folder with `original.eml`, `derived.md` and its attachments under their own names; a ZIP attached to it is unpacked once into a folder of its own, and an attached mail is quoted in `derived.md`, not unpacked. A ZIP inside a ZIP, an absolute or `..` path, a symbolic link, an encrypted entry, more than 1000 entries, a file over 25 MiB or more than 250 MiB in total refuse the whole call. Hidden entries (`__MACOSX/`, `.DS_Store`, anything starting with a dot) are named, not written. A name taken twice, or reserved (`original.*` and `derived.md` at the top of a mail's folder, `_capture.json` anywhere), gets `-2`, `-3` before its extension, and the output lists each rename.
|
||||
- A name already taken under `raw/` is a WARN, not a refusal: whether the folder is a later edition of that source or a separate one is the human's call. A later edition of a derived source is replaced file by file with `raw accept --replaces`; there is no `--replaces-bundle` for it.
|
||||
|
||||
@@ -2177,7 +2177,11 @@ def _derive_next(target: Path) -> str:
|
||||
"doctor` shows one `import-filter-<name>` line per package-backed format. A PDF gets "
|
||||
"`<!-- page N -->` markers and a WARN when it yields almost no text (a scan; there is "
|
||||
"no OCR); an encrypted PDF is refused. A `.msg` is a container like an `.eml`; an "
|
||||
"embedded message inside it is named, not read.",
|
||||
"embedded message inside it is named, not read. The text of a mail (`.eml` and `.msg` "
|
||||
"alike) is its plain text, unless that is under 10% of the length of the HTML text - a "
|
||||
"\"plain text version not available\" placeholder - in which case it is the HTML text "
|
||||
"and a WARN names both lengths. The other version of the text in a "
|
||||
"`multipart/alternative` is not an attachment.",
|
||||
"A workbook (`.xlsx`, `.xlsm`) gets its cells as files: beside its `.md` a folder "
|
||||
"`<md stem>.sheets/` (`derived.sheets/` for a single file) holds one `NN-<sheet>.tsv` "
|
||||
"per sheet - tab-separated per RFC 4180, UTF-8, row 1 of the file is row 1 of the sheet "
|
||||
|
||||
@@ -39,6 +39,24 @@ class ParsedMail:
|
||||
inline_images: int = 0
|
||||
|
||||
|
||||
PLAIN_MIN_SHARE = 0.10
|
||||
|
||||
|
||||
def choose_text(plain: str, html_text: str) -> tuple[str, bool, Optional[str]]:
|
||||
"""The mail text from its two versions, both stripped: the plain text, unless it is
|
||||
shorter than `PLAIN_MIN_SHARE` of the HTML text (a "plain text version not available"
|
||||
placeholder). Returns the text, whether it came from the HTML, and a warning if the
|
||||
plain text was passed over. Shared by `.eml` and `.msg`."""
|
||||
if not plain:
|
||||
return html_text, bool(html_text), None
|
||||
if html_text and len(plain) < PLAIN_MIN_SHARE * len(html_text):
|
||||
return html_text, True, (
|
||||
f"The plain text ({len(plain)} characters) is under {PLAIN_MIN_SHARE:.0%} of the HTML text "
|
||||
f"({len(html_text)} characters); the text is taken from the HTML part."
|
||||
)
|
||||
return plain, False, None
|
||||
|
||||
|
||||
def _header(msg: EmailMessage, name: str) -> str:
|
||||
try:
|
||||
return re.sub(r"\s+", " ", str(msg[name] or "")).strip()
|
||||
@@ -100,6 +118,36 @@ def _leaves(part: EmailMessage):
|
||||
yield from _leaves(child)
|
||||
|
||||
|
||||
def _other_versions(msg: EmailMessage, body_part: Optional[EmailMessage]) -> set[int]:
|
||||
"""The text/plain and text/html leaves under the nearest `multipart/alternative` that holds
|
||||
the chosen body part: other versions of the mail text, not attachments. A part that is
|
||||
explicitly an attachment stays one."""
|
||||
if body_part is None:
|
||||
return set()
|
||||
|
||||
def holder(part: EmailMessage, alternative: Optional[EmailMessage]) -> Optional[EmailMessage]:
|
||||
if part is body_part:
|
||||
return alternative
|
||||
if part.get_content_type() == "message/rfc822" or not part.is_multipart():
|
||||
return None
|
||||
nearest = part if part.get_content_type() == "multipart/alternative" else alternative
|
||||
for child in part.iter_parts():
|
||||
found = holder(child, nearest)
|
||||
if found is not None:
|
||||
return found
|
||||
return None
|
||||
|
||||
alternative = holder(msg, None)
|
||||
if alternative is None:
|
||||
return set()
|
||||
return {
|
||||
id(leaf)
|
||||
for leaf in _leaves(alternative)
|
||||
if leaf.get_content_type() in ("text/plain", "text/html")
|
||||
and (leaf.get_content_disposition() or "").lower() != "attachment"
|
||||
}
|
||||
|
||||
|
||||
def parse(data: bytes) -> ParsedMail:
|
||||
msg = message_from_bytes(data, policy=policy.default)
|
||||
if not isinstance(msg, EmailMessage) or not (msg.keys() or msg.get_payload()):
|
||||
@@ -113,15 +161,21 @@ def parse(data: bytes) -> ParsedMail:
|
||||
"subject": _header(msg, "subject"),
|
||||
"message_id": _header(msg, "message-id"),
|
||||
}
|
||||
body_part: Optional[EmailMessage] = msg.get_body(preferencelist=("plain", "html"))
|
||||
if body_part is None:
|
||||
plain_part: Optional[EmailMessage] = msg.get_body(preferencelist=("plain",))
|
||||
html_part: Optional[EmailMessage] = msg.get_body(preferencelist=("html",))
|
||||
body_part: Optional[EmailMessage] = None
|
||||
if plain_part is None and html_part is None:
|
||||
out.warnings.append("The mail has no text/plain or text/html body (RTF-only or empty); the text is empty.")
|
||||
elif body_part.get_content_type() == "text/html":
|
||||
out.body = web_capture.derive_text(_text_of(body_part), "").strip()
|
||||
else:
|
||||
out.body = _text_of(body_part).replace("\r\n", "\n").replace("\r", "\n").strip()
|
||||
plain = _text_of(plain_part).replace("\r\n", "\n").replace("\r", "\n").strip() if plain_part is not None else ""
|
||||
html_text = web_capture.derive_text(_text_of(html_part), "").strip() if html_part is not None else ""
|
||||
out.body, from_html, warning = choose_text(plain, html_text)
|
||||
if warning:
|
||||
out.warnings.append(warning)
|
||||
body_part = html_part if from_html or plain_part is None else plain_part
|
||||
versions = _other_versions(msg, body_part)
|
||||
for number, part in enumerate(_leaves(msg), 1):
|
||||
if part is body_part or part is msg:
|
||||
if part is body_part or part is msg or id(part) in versions:
|
||||
continue
|
||||
disposition = (part.get_content_disposition() or "").lower()
|
||||
if (
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Outlook `.msg` (a compound file), read with `olefile` (imported when one is derived).
|
||||
|
||||
Only what a reader of the mail needs: the headers, the body (plain text, else HTML)
|
||||
and the attachments by name and bytes. The result is the same `ParsedMail` an `.eml`
|
||||
Only what a reader of the mail needs: the headers, the body (plain text, else HTML;
|
||||
HTML too when the plain text is under 10% of it - `mail.choose_text`) and the
|
||||
attachments by name and bytes. The result is the same `ParsedMail` an `.eml`
|
||||
yields, so everything after the parsing is shared with `mail.py`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -17,7 +18,7 @@ from typing import Optional
|
||||
from chemenu import web_capture
|
||||
from chemenu.errors import ValidationError
|
||||
from chemenu.filters import register
|
||||
from chemenu.filters.mail import Attachment, EmlFilter, ParsedMail
|
||||
from chemenu.filters.mail import Attachment, EmlFilter, ParsedMail, choose_text
|
||||
|
||||
_SUBJECT = "0037"
|
||||
_BODY = "1000"
|
||||
@@ -129,12 +130,12 @@ def _html_text(raw: bytes) -> str:
|
||||
|
||||
|
||||
def _body(message: _Message, out: ParsedMail) -> None:
|
||||
plain = message.text(_BODY)
|
||||
plain = message.text(_BODY).replace("\r\n", "\n").replace("\r", "\n").strip()
|
||||
html = message.raw(_HTML)
|
||||
if plain:
|
||||
out.body = plain.replace("\r\n", "\n").replace("\r", "\n").strip()
|
||||
elif html:
|
||||
out.body = _html_text(html)
|
||||
if plain or html:
|
||||
out.body, _from_html, warning = choose_text(plain, _html_text(html) if html else "")
|
||||
if warning:
|
||||
out.warnings.append(warning)
|
||||
else:
|
||||
what = "has only an RTF body" if message.raw(_RTF) else "has no body"
|
||||
out.warnings.append(f"The message {what}, which raw derive does not read; the text is empty.")
|
||||
|
||||
@@ -182,8 +182,8 @@ def _text_cell(ref: str, text: str) -> str:
|
||||
return f'<c r="{ref}" t="inlineStr"><is><t xml:space="preserve">{text.replace(chr(13), " ")}</t></is></c>'
|
||||
|
||||
|
||||
def make_eml(body: str = "Hallo Welt\n", *, html: str = "", attach: bool = False, inline: bool = False,
|
||||
subject: str = "Angebot") -> bytes:
|
||||
def make_eml(body: str = "Hallo Welt\n", *, html: str = "", alternative: str = "", attach: bool = False,
|
||||
inline: bool = False, subject: str = "Angebot") -> bytes:
|
||||
msg = EmailMessage()
|
||||
msg["From"] = "Anna Beispiel <anna@example.org>"
|
||||
msg["To"] = "bob@example.org, Carla <carla@example.org>"
|
||||
@@ -194,6 +194,8 @@ def make_eml(body: str = "Hallo Welt\n", *, html: str = "", attach: bool = False
|
||||
msg.set_content(html, subtype="html")
|
||||
else:
|
||||
msg.set_content(body)
|
||||
if alternative:
|
||||
msg.add_alternative(alternative, subtype="html")
|
||||
if inline:
|
||||
msg.add_related(b"\x89PNG", maintype="image", subtype="png", cid="<logo@x>")
|
||||
if attach:
|
||||
@@ -752,6 +754,87 @@ def test_eml_with_inline_image_warns_and_is_not_an_attachment(tree, capsys):
|
||||
assert "inline image" in capsys.readouterr().out
|
||||
|
||||
|
||||
_NEWSLETTER = "<html><body><h1>Newsletter</h1><p>" + "Real content of the mail. " * 20 + "</p></body></html>"
|
||||
|
||||
|
||||
def _eml_parsed(data: bytes):
|
||||
from chemenu.filters import mail
|
||||
|
||||
return mail.parse(data)
|
||||
|
||||
|
||||
def test_eml_placeholder_plain_text_gives_way_to_the_html_text(tree, capsys):
|
||||
_put(tree, "mail.eml", make_eml("Plain text version not available", alternative=_NEWSLETTER))
|
||||
_derive("incoming/mail.eml")
|
||||
out = capsys.readouterr().out
|
||||
header, text = _header_and_text(tree / "incoming/mail/derived.md")
|
||||
|
||||
assert "Real content of the mail." in text and "not available" not in text
|
||||
assert "The plain text (32 characters) is under 10% of the HTML text" in out
|
||||
assert header["attachments"] == []
|
||||
assert sorted(p.name for p in (tree / "incoming/mail").iterdir()) == ["derived.md", "original.eml"]
|
||||
|
||||
|
||||
def test_eml_good_plain_text_wins_over_html_without_a_warning(tree, capsys):
|
||||
plain = "Real content of the mail. " * 18
|
||||
_put(tree, "mail.eml", make_eml(plain, alternative=_NEWSLETTER))
|
||||
_derive("incoming/mail.eml")
|
||||
out = capsys.readouterr().out
|
||||
header, text = _header_and_text(tree / "incoming/mail/derived.md")
|
||||
|
||||
assert text.strip() == plain.strip()
|
||||
assert "is under" not in out
|
||||
assert header["attachments"] == []
|
||||
|
||||
|
||||
def test_eml_keeps_a_real_attachment_beside_the_two_text_versions(tree):
|
||||
_put(tree, "mail.eml", make_eml("Plain text version not available", alternative=_NEWSLETTER, attach=True))
|
||||
_derive("incoming/mail.eml")
|
||||
header, _text = _header_and_text(tree / "incoming/mail/derived.md")
|
||||
|
||||
assert header["attachments"] == ["a.bin"]
|
||||
assert (tree / "incoming/mail/a.bin").read_bytes() == b"data"
|
||||
|
||||
|
||||
def test_eml_html_inside_a_related_group_is_still_a_text_version_and_its_image_is_inline():
|
||||
msg = EmailMessage()
|
||||
msg["Subject"] = "S"
|
||||
msg.set_content("Plain text version not available")
|
||||
msg.add_alternative(_NEWSLETTER, subtype="html")
|
||||
msg.get_payload()[1].add_related(b"\x89PNG", maintype="image", subtype="png", cid="<logo@x>")
|
||||
parsed = _eml_parsed(msg.as_bytes())
|
||||
|
||||
assert "Real content of the mail." in parsed.body
|
||||
assert parsed.attachments == []
|
||||
assert parsed.inline_images == 1
|
||||
|
||||
|
||||
def test_eml_text_version_marked_as_an_attachment_stays_one():
|
||||
msg = EmailMessage()
|
||||
msg["Subject"] = "S"
|
||||
msg.set_content("Hello there\n")
|
||||
msg.add_alternative(_NEWSLETTER, subtype="html")
|
||||
msg.get_payload()[1].add_header("Content-Disposition", "attachment", filename="version.html")
|
||||
parsed = _eml_parsed(msg.as_bytes())
|
||||
|
||||
assert parsed.body == "Hello there"
|
||||
assert [a.name for a in parsed.attachments] == ["version.html"]
|
||||
|
||||
|
||||
def test_eml_plain_and_html_in_a_mixed_mail_without_alternative_keep_the_html_as_attachment():
|
||||
msg = EmailMessage()
|
||||
msg["Subject"] = "S"
|
||||
msg.set_content("Hello there. " * 40)
|
||||
msg.make_mixed()
|
||||
html = EmailMessage()
|
||||
html.set_content(_NEWSLETTER, subtype="html")
|
||||
msg.attach(html)
|
||||
parsed = _eml_parsed(msg.as_bytes())
|
||||
|
||||
assert parsed.body == ("Hello there. " * 40).strip()
|
||||
assert len(parsed.attachments) == 1
|
||||
|
||||
|
||||
# --- the registry -----------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -165,6 +165,32 @@ def test_a_msg_with_only_html_takes_the_html_text(tree):
|
||||
assert "Only" in text and "html here" in text and "<p>" not in text
|
||||
|
||||
|
||||
_HTML_BODY = "<html><body><p>" + "Real content of the mail. " * 20 + "</p></body></html>"
|
||||
|
||||
|
||||
@needs_olefile
|
||||
def test_a_msg_with_a_placeholder_plain_text_takes_the_html_text_and_warns(tree, capsys):
|
||||
_put(tree, "p.msg", build_msg(subject="S", body="Plain text version not available", html=_HTML_BODY))
|
||||
_derive("incoming/p.msg")
|
||||
out = capsys.readouterr().out
|
||||
_header, text = _header_and_text(tree / "incoming/p/derived.md")
|
||||
|
||||
assert "Real content of the mail." in text and "not available" not in text
|
||||
assert "The plain text (32 characters) is under 10% of the HTML text" in out
|
||||
|
||||
|
||||
@needs_olefile
|
||||
def test_a_msg_with_a_good_plain_text_keeps_it_without_a_warning(tree, capsys):
|
||||
plain = "Real content of the mail. " * 18
|
||||
_put(tree, "g.msg", build_msg(subject="S", body=plain, html=_HTML_BODY))
|
||||
_derive("incoming/g.msg")
|
||||
out = capsys.readouterr().out
|
||||
_header, text = _header_and_text(tree / "incoming/g/derived.md")
|
||||
|
||||
assert text.strip() == plain.strip()
|
||||
assert "is under" not in out
|
||||
|
||||
|
||||
@needs_olefile
|
||||
def test_a_msg_with_only_rtf_warns_and_has_no_text(tree, capsys):
|
||||
_put(tree, "r.msg", build_msg(subject="S", rtf=b"{\\rtf1 hello}"))
|
||||
|
||||
Reference in new issue
Block a user