Repository navigation
feat(tools): add web_fetch to read public pages and APIs - #176
Merged
Merged
Conversation
The agent answered current-fact questions from memory or stale search snippets, and jev_browser timed out on most official pages. web_fetch reads a public URL from the host and returns its text in 12,000-character windows. It refuses private, loopback, and link-local targets on every redirect hop and again at connect time. See ADR 0023.
…d the MCP result guard Replace the hand-written HTML conversion, address classification, redirect loop, and windowing with @mdream/js, Node's BlockList, undici's redirects and connect lookup, and the existing bound-and-spill guard for MCP results.
Deploying mikan with
|
| Latest commit: |
b265f8c
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8ef45ee3.mikan-1wv.pages.dev |
| Branch Preview URL: | https://feat-web-fetch.mikan-1wv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Asked for current facts (package and language versions, a release, today's weather, this week's news), the agent answered from memory or from stale search snippets.
jev_browsertimed out or failed on most official pages.What
web_fetchreads a public web page or API from the host:@mdream/js(minimal preset, which drops navigation and page chrome); JSON and plain text are returned as they are.guardMcpToolResult, so a long page is bounded and the full text is saved in the workspace forread/grep, like an MCP result.Agentrefuses private, loopback, and link-local addresses (NodeBlockList) before connecting: IP literals in the connector, names in the connect lookup. Redirects and DNS answers go through the same check.@mdream/js(MIT, 0.6 MB installed, 2 dependencies).@mdream/jswas chosen after converting seven real pages withhtml-to-text,turndown,node-html-markdown, and Readability with Turndown: it kept the answer on every page with the smallest output. Readability dropped a version table;node-html-markdowntook 29 s on a 3 MB page.Pi's web tools are coding-agent extensions that pi-durable cannot load, so this is a mikan tool (ADR 0023).
Acceptance eval
Scripted model and faked network,
mainvs this branch:web_fetchtool127.0.0.1:8080,169.254.169.254,localhost,[::1],file://Real model, same 10 questions, run through the conversation runtime without a chat platform, graded against the official sources:
Trade-offs
Agentbypasses the daemon'sEnvHttpProxyAgent, because a proxy would resolve names itself and defeat the address check. A deployment that reaches the web only through a proxy cannot useweb_fetch.Verification
npm run lint && npm run fmt:check && npm run knip && npm run build && npm run typecheck && npm test: 2,101 passed.src/test/web-fetch-tool.test.ts(27 tests, real local HTTP servers on127.0.0.1and[::1]).