feat: raw fetch - a sanctioned intake for a URL into incoming/, HTML as received plus derived text (#120)
CI / verify (push) Successful in 5m21s
CI / pwsh (push) Successful in 2m6s
Release / release (push) Successful in 34s

Files changed:
- CHANGES.md
- README.md
- VERSION
- instructions/wiki-ingest/SKILL.md
- raw/CONTRACT.md
- tools/CONTRACT.md
- tools/README.md
- tools/chemenu/cli_contract.py
- tools/chemenu/commands/raw_cmd.py
- tools/chemenu/tests/test_cli.py
- tools/chemenu/tests/test_portability.py
- tools/chemenu/tests/test_raw_fetch.py
- tools/chemenu/web_capture.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
torbenandClaude Opus 5.5 committed 2026-10-02 22:26:06 +02:00
1 parent c0f324ff96
commit fba263af68
13 files changed
+1571 -16

No files matched your search

+31 -1
View File
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
---
## 8.0.0-beta.24 - 2026-10-02 - Stack-Entwicklung in drei Phasen: stack-dev (Design), stack-build, stack-close - Übergabe über den Tracker, kein Modellwechsel in der Sitzung
## 8.0.0-beta.25 - 2026-10-02 - raw fetch: a sanctioned intake for a URL into incoming/
**Author:** Torben Nehmer
@@ -99,6 +99,7 @@ concern - readable here, never shipped as something to parse.
- tools/bugreport: Starter für den Bugreport-Sammler, überspringt die Store-Aliase (#166)
- publish keeps a closing trailer block of --message last, so git reads Co-Authored-By again (#149)
- Stack-Entwicklung in drei Phasen: stack-dev (Design), stack-build, stack-close - Übergabe über den Tracker, kein Modellwechsel in der Sitzung
- raw fetch: a sanctioned intake for a URL into incoming/
**Low impact**
- version bump no longer points at version release in its output
@@ -136,6 +137,35 @@ concern - readable here, never shipped as something to parse.
- preflight.ps1: the asset-mode error helper is Exit-Asset, so PSScriptAnalyzer passes
<!-- /wikitool:bumps -->
### raw fetch: a sanctioned intake for a URL into incoming/ (#120)
A URL the user wanted ingested had no tool and no procedure: every session built its own chain
of `curl`, a guessed character set, boilerplate cut by line number and a header written from
memory, so two sessions turned the same article into two different, permanent `raw/` files. The
new `tools/wikitool raw fetch <url>` fetches the page and writes two files into `incoming/` - the
HTML exactly as received, and a `.md` with a fixed header (`url`, `final_url`, `retrieved`,
`http_status`, `content_type`, `charset` and where it came from, `title`, `derived_from`) above a
Markdown-like text derived from it. `raw accept` then promotes both in one call as one bundle,
unchanged, so the capture fields are still asked once, at the same point of `wiki-ingest`. The
HTML is kept because it is what was received: a claim stays checkable against it even where the
derivation lost something.
The derivation is deterministic and needs no new dependency (`urllib`, `html.parser`): charset
from the byte-order mark, then the HTTP header, then a `<meta>` in the first 4 KiB, then UTF-8,
with undecodable bytes replaced and reported; content root `<main>`, else a single `<article>`,
else `<body>`, with navigation, header, footer, aside, forms and scripts dropped and links made
absolute. Only `http`/`https` on every redirect hop, 30 s, 25 MiB, no cookies; a non-HTML answer
(plain text, PDF, image) is stored as received with no derivation. For a paywall, a login or a
script-rendered page, `raw fetch --html incoming/<file>.html --url <url>` derives the same `.md`
from a page the human saved from their own browser, without network access - the tool holds no
credentials. The header's values are YAML-quoted where needed (`charset: "utf-8 (from: header)"`),
so the block parses as the frontmatter its `---` fences suggest.
`raw/CONTRACT.md` has a new section "Getting a URL in: `raw fetch`" (the bundle rule, the header,
`--html`, and that only a URL the user named is fetched), and says why the MCP server gets no
fetch tool. `wiki-ingest` starts a URL with `raw fetch` and stops at a teaser instead of
ingesting it.
### Stack-Entwicklung in drei Phasen: `stack-dev` (Design), `stack-build`, `stack-close` (#168)
Bisher hatte die Stack-Entwicklung zwei Skills für drei Phasen und koppelte jeden Phasenwechsel an