feat: raw fetch - a sanctioned intake for a URL into incoming/, HTML as received plus derived text (#120)
Files changed: - CHANGES.md - README.md - VERSION - instructions/wiki-ingest/SKILL.md - raw/CONTRACT.md - tools/CONTRACT.md - tools/README.md - tools/chemenu/cli_contract.py - tools/chemenu/commands/raw_cmd.py - tools/chemenu/tests/test_cli.py - tools/chemenu/tests/test_portability.py - tools/chemenu/tests/test_raw_fetch.py - tools/chemenu/web_capture.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnAJ7Z3CpVD3PRbN73QtU2
This commit is contained in:
1 parent
c0f324ff96
commit
fba263af68
13 files changed
+1571
-16
No files matched your search
+31
-1
@@ -59,7 +59,7 @@ concern - readable here, never shipped as something to parse.
|
||||
|
||||
---
|
||||
|
||||
## 8.0.0-beta.24 - 2026-10-02 - Stack-Entwicklung in drei Phasen: stack-dev (Design), stack-build, stack-close - Übergabe über den Tracker, kein Modellwechsel in der Sitzung
|
||||
## 8.0.0-beta.25 - 2026-10-02 - raw fetch: a sanctioned intake for a URL into incoming/
|
||||
|
||||
**Author:** Torben Nehmer
|
||||
|
||||
@@ -99,6 +99,7 @@ concern - readable here, never shipped as something to parse.
|
||||
- tools/bugreport: Starter für den Bugreport-Sammler, überspringt die Store-Aliase (#166)
|
||||
- publish keeps a closing trailer block of --message last, so git reads Co-Authored-By again (#149)
|
||||
- Stack-Entwicklung in drei Phasen: stack-dev (Design), stack-build, stack-close - Übergabe über den Tracker, kein Modellwechsel in der Sitzung
|
||||
- raw fetch: a sanctioned intake for a URL into incoming/
|
||||
|
||||
**Low impact**
|
||||
- version bump no longer points at version release in its output
|
||||
@@ -136,6 +137,35 @@ concern - readable here, never shipped as something to parse.
|
||||
- preflight.ps1: the asset-mode error helper is Exit-Asset, so PSScriptAnalyzer passes
|
||||
<!-- /wikitool:bumps -->
|
||||
|
||||
### raw fetch: a sanctioned intake for a URL into incoming/ (#120)
|
||||
|
||||
A URL the user wanted ingested had no tool and no procedure: every session built its own chain
|
||||
of `curl`, a guessed character set, boilerplate cut by line number and a header written from
|
||||
memory, so two sessions turned the same article into two different, permanent `raw/` files. The
|
||||
new `tools/wikitool raw fetch <url>` fetches the page and writes two files into `incoming/` - the
|
||||
HTML exactly as received, and a `.md` with a fixed header (`url`, `final_url`, `retrieved`,
|
||||
`http_status`, `content_type`, `charset` and where it came from, `title`, `derived_from`) above a
|
||||
Markdown-like text derived from it. `raw accept` then promotes both in one call as one bundle,
|
||||
unchanged, so the capture fields are still asked once, at the same point of `wiki-ingest`. The
|
||||
HTML is kept because it is what was received: a claim stays checkable against it even where the
|
||||
derivation lost something.
|
||||
|
||||
The derivation is deterministic and needs no new dependency (`urllib`, `html.parser`): charset
|
||||
from the byte-order mark, then the HTTP header, then a `<meta>` in the first 4 KiB, then UTF-8,
|
||||
with undecodable bytes replaced and reported; content root `<main>`, else a single `<article>`,
|
||||
else `<body>`, with navigation, header, footer, aside, forms and scripts dropped and links made
|
||||
absolute. Only `http`/`https` on every redirect hop, 30 s, 25 MiB, no cookies; a non-HTML answer
|
||||
(plain text, PDF, image) is stored as received with no derivation. For a paywall, a login or a
|
||||
script-rendered page, `raw fetch --html incoming/<file>.html --url <url>` derives the same `.md`
|
||||
from a page the human saved from their own browser, without network access - the tool holds no
|
||||
credentials. The header's values are YAML-quoted where needed (`charset: "utf-8 (from: header)"`),
|
||||
so the block parses as the frontmatter its `---` fences suggest.
|
||||
|
||||
`raw/CONTRACT.md` has a new section "Getting a URL in: `raw fetch`" (the bundle rule, the header,
|
||||
`--html`, and that only a URL the user named is fetched), and says why the MCP server gets no
|
||||
fetch tool. `wiki-ingest` starts a URL with `raw fetch` and stops at a teaser instead of
|
||||
ingesting it.
|
||||
|
||||
### Stack-Entwicklung in drei Phasen: `stack-dev` (Design), `stack-build`, `stack-close` (#168)
|
||||
|
||||
Bisher hatte die Stack-Entwicklung zwei Skills für drei Phasen und koppelte jeden Phasenwechsel an
|
||||
|
||||
Reference in new issue
Block a user