[![Actions Status](https://github.com/m-doughty/MCP-Server-Tool-Web/actions/workflows/test.yml/badge.svg)](https://github.com/m-doughty/MCP-Server-Tool-Web/actions)

MCP::Server::Tool::Web
======================

A web tools (search, fetch, crawl, grep) toolkit for [MCP::Server](https://github.com/m-doughty/MCP-Server).

Four tools — `web_search`, `web_fetch`, `web_crawl` and `web_grep` — under one rule that runs beneath all of them: a URL a model supplied is fetched only after the address of the socket that was *actually opened* has been checked, so a hostname that resolves onto the host machine's own network is refused whatever it is spelled like. See [Security](#Security).

Synopsis
--------

Plug it into a server you are already building:

```raku
use MCP::Server;
use MCP::Server::Tool::Web;

my $server = MCP::Server.new(:name<research>, :version<1.0>);

$server.plug: MCP::Server::Tool::Web.new;

$server.run;   # registers web_search, web_fetch, web_crawl, web_grep
```

Or assemble a server out of toolkit packs in one shot, with no glue code — the shape a host's JSON config takes:

```raku
MCP::Server.new(
    :name<research>,
    :tools[
        'Web' => {
            provider        => 'brave',
            'api-key-env'   => 'BRAVE_API_KEY',
            'max-bytes'     => 4 * 1024 * 1024,
            'total-timeout' => 90,
            'crawl-delay'   => 0.5,
            'allow-hosts'   => <wiki.corp docs.corp>,   # exact names only
        },
    ],
).run;
```

Or run it straight off the `raku-mcp` command line that ships with `MCP::Server`:

```bash
raku-mcp --tool=Web='{"crawl-delay":0.5,"max-bytes":4194304}'
```

```bash
raku-mcp --config=mcp.json
```

```json
{
  "name": "research",
  "instructions": "Search and read the web. Prefer web_grep over web_fetch when you know what you are looking for.",
  "tools": {
    "Web": { "crawl-delay": 0.5, "allow-hosts": ["wiki.corp"] }
  }
}
```

Wiring that into Claude Code:

```json
{
  "mcpServers": {
    "research": {
      "command": "raku-mcp",
      "args": ["--config=/home/me/mcp.json"],
      "env": { "BRAVE_API_KEY": "..." }
    }
  }
}
```

Description
-----------

`web_search` answers JSON so the model can pick a URL out of it; `web_fetch`, `web_crawl` and `web_grep` answer plain text, because a page — or a report about pages — is text. Nothing here needs a browser: pages are fetched over plain HTTP(S) and their markup, scripts and navigation are stripped by a hand-rolled extractor ([MCP::Server::Tool::Web::Extract](MCP::Server::Tool::Web::Extract)), not rendered.

`web_search` needs a search provider's API key to answer anything; `web_fetch`, `web_crawl` and `web_grep` need nothing beyond network access, and construct successfully even with no key configured anywhere — see [Search provider](#Search provider).

Tools
-----

Parameter names are a contract, not a matter of taste: a policy layer in front of the server (such as `MCP::Client::Policy`) matches calls by them to decide what a call would reach, so `url` stays `url` and multi-word parameters stay kebab-case.

### web_search

The search engine, behind a seam. Answers JSON so the model can pick a URL out of it. Zero results is an empty list and a notice, not an error.

<table class="pod-table">
<thead><tr>
<th>Argument</th> <th>Type</th> <th>Required</th> <th>Meaning</th>
</tr></thead>
<tbody>
<tr> <td>query</td> <td>string</td> <td>yes</td> <td>What to search for; up to 400 characters</td> </tr> <tr> <td>count</td> <td>integer</td> <td>no</td> <td>How many results, 1 to 20; defaults to 10</td> </tr>
</tbody>
</table>

```text
# web_search — JSON, so the model can pick a URL out of it
{"count":10,"provider":"brave","query":"raku grammars",
 "results":[{"snippet":"…","title":"Grammars","url":"https://docs.raku.org/…"}]}

# a query that matches nothing:
{"count":10,"notice":"No results for 'asdkfjhasdkfjh39847'. The engine matched nothing: try fewer or more common terms, or drop any quoting.",
 "provider":"brave","query":"asdkfjhasdkfjh39847","results":[]}
```

### web_fetch

One page, as text: a `# title` line, the final URL when a redirect moved it, then the page with its markup, scripts and navigation removed. Exactly one parameter, deliberately: a windowing tool would invite the model to page through a document it should have grepped instead. Only http and https URLs can be fetched, redirects are followed (every hop re-checked — see [Security](#Security)), and a page bigger than `max-bytes` comes back cut short and says so on its last line.

<table class="pod-table">
<thead><tr>
<th>Argument</th> <th>Type</th> <th>Required</th> <th>Meaning</th>
</tr></thead>
<tbody>
<tr> <td>url</td> <td>string</td> <td>yes</td> <td>Absolute http or https URL of the page to read</td> </tr>
</tbody>
</table>

```text
# web_fetch — plain text, because a page is text
# Grammars
Retrieved from https://docs.raku.org/language/grammars

Grammars are a powerful tool used to destructure text …

# a page cut short by the byte cap:
[truncated: kept the first 2097152 characters because the server's byte cap of 2097152 bytes was reached; the rest of the page was not fetched]
```

### web_crawl

Follows the links from a page and returns an **index** of what is there — never the pages themselves. Only links on the same origin as the starting page (same scheme, host and effective port) are followed; `depth` counts hops from the starting page; robots.txt is respected by default (see [Security](#Security)). See [MCP::Server::Tool::Web::Crawl](MCP::Server::Tool::Web::Crawl) for the walk itself.

<table class="pod-table">
<thead><tr>
<th>Argument</th> <th>Type</th> <th>Required</th> <th>Meaning</th>
</tr></thead>
<tbody>
<tr> <td>url</td> <td>string</td> <td>yes</td> <td>Absolute http or https URL to start from</td> </tr> <tr> <td>depth</td> <td>integer</td> <td>no</td> <td>Hops to follow, 0 to 3; 0 reports only the starting page; defaults to 1</td> </tr> <tr> <td>max-pages</td> <td>integer</td> <td>no</td> <td>Stop after fetching this many pages, counting the seed, 1 to 50; defaults to 20</td> </tr>
</tbody>
</table>

```text
# web_crawl — an index, not the pages
Crawled 3 pages from https://docs.example.com/ (depth 1, same origin only).

https://docs.example.com/  200  4213 bytes  Documentation
https://docs.example.com/install  200  2288 bytes  Installing

[stopped after 20 pages; raise max-pages to go further]
```

Why an index rather than the pages: concatenating twenty pages into one tool result is a way to spend a context window without answering anything — what the model would actually receive is page one, half of page two, and a truncation marker. So a crawl reports what is there, one line per page, and the corpus stays reachable through `web_grep` (with a `depth`) and `web_fetch`.

### web_grep

`fs_grep` for the web: search one page, or a whole site at a `depth`, and return only the matching lines as `url:line:text`. The pattern is literal text unless `regex` is set, in which case it is a *Raku* regex, not a PCRE one. This is far cheaper than `web_fetch` or `web_crawl` when you know what you are looking for.

<table class="pod-table">
<thead><tr>
<th>Argument</th> <th>Type</th> <th>Required</th> <th>Meaning</th>
</tr></thead>
<tbody>
<tr> <td>pattern</td> <td>string</td> <td>yes</td> <td>Text to search for, or a Raku regex when regex is true</td> </tr> <tr> <td>url</td> <td>string</td> <td>yes</td> <td>Page to search, and to crawl from when depth is above 0</td> </tr> <tr> <td>regex</td> <td>boolean</td> <td>no</td> <td>Treat pattern as a Raku regex; defaults to false</td> </tr> <tr> <td>context</td> <td>integer</td> <td>no</td> <td>Lines of surrounding context, 0 to 20; defaults to 0</td> </tr> <tr> <td>max-results</td> <td>integer</td> <td>no</td> <td>Stop after this many matching lines, 1 to 1000; defaults to 50</td> </tr> <tr> <td>depth</td> <td>integer</td> <td>no</td> <td>Hops to follow from url, 0 to 3; defaults to 0 (that page alone)</td> </tr> <tr> <td>max-pages</td> <td>integer</td> <td>no</td> <td>When depth is above 0, stop after this many pages, 1 to 50; defaults to 20</td> </tr>
</tbody>
</table>

```text
# web_grep — only the lines that matched, exactly as fs_grep reports them
https://docs.example.com/install:42:  zef install Foo::Bar

[stopped at 50 matching lines; raise max-results to see more]
```

With the default `depth` of 0 only the URL given is searched; with a depth above 0 the same-origin links are followed as `web_crawl` does, and no more pages are fetched once `max-results` lines have matched — which is the whole reason `web_grep` is cheaper than crawling and then fetching.

Configuration
-------------

Every attribute below is a JSON config key — `from-config` validates against exactly this set, so a typo names the valid keys instead of doing nothing quietly:

```raku
MCP::Server::Tool::Web.from-config({ providr => 'brave' });
# Unknown config key(s) for MCP::Server::Tool::Web: providr.
# Valid keys: allow-cidrs, allow-hosts, allow-loopback, allow-private,
# api-key, api-key-env, connect-timeout, crawl-delay, headers-timeout,
# max-bytes, proxy, provider, respect-robots, safesearch, search-country,
# search-lang, timeout, total-timeout, user-agent
```

<table class="pod-table">
<thead><tr>
<th>Key</th> <th>Type</th> <th>Default</th> <th>Meaning</th>
</tr></thead>
<tbody>
<tr> <td>provider</td> <td>Str</td> <td>&#39;brave&#39;</td> <td>Which search engine web_search asks; see L&lt;#Search provider&gt;</td> </tr> <tr> <td>api-key-env</td> <td>Str</td> <td>&#39;BRAVE_API_KEY&#39;</td> <td>Environment variable the provider&#39;s API key is read from</td> </tr> <tr> <td>api-key</td> <td>Str</td> <td>none</td> <td>The API key itself; beats api-key-env when both are set</td> </tr> <tr> <td>search-country</td> <td>Str</td> <td>provider&#39;s own default</td> <td>Two-letter country code biasing search results (&#39;US&#39;, &#39;GB&#39;, ...)</td> </tr> <tr> <td>search-lang</td> <td>Str</td> <td>provider&#39;s own default</td> <td>Language code for results (&#39;en&#39;, &#39;de&#39;, ...)</td> </tr> <tr> <td>safesearch</td> <td>Str</td> <td>&#39;moderate&#39;</td> <td>Adult-content filtering: &#39;off&#39;, &#39;moderate&#39; or &#39;strict&#39;</td> </tr> <tr> <td>timeout</td> <td>Real (seconds)</td> <td>15</td> <td>One search request&#39;s budget; fetches use the three below instead</td> </tr> <tr> <td>max-bytes</td> <td>Int (bytes)</td> <td>2097152 (2 MiB)</td> <td>Body bytes B&lt;one tool call&gt; may retain, across every page it fetches</td> </tr> <tr> <td>total-timeout</td> <td>Real (seconds)</td> <td>60</td> <td>Wall-clock budget for a whole tool call, crawls included</td> </tr> <tr> <td>connect-timeout</td> <td>Real (seconds)</td> <td>15</td> <td>One connection attempt&#39;s budget, clamped down by whatever is left of the call&#39;s own budget</td> </tr> <tr> <td>headers-timeout</td> <td>Real (seconds)</td> <td>30</td> <td>One response&#39;s headers budget, likewise clamped</td> </tr> <tr> <td>crawl-delay</td> <td>Real (seconds)</td> <td>0.25</td> <td>Seconds left between requests while crawling; a robots.txt Crawl-delay may raise it, nothing lowers it</td> </tr> <tr> <td>respect-robots</td> <td>Bool</td> <td>True</td> <td>Consult robots.txt when traversing (web_crawl, and web_grep with a depth); never for a single named page</td> </tr> <tr> <td>user-agent</td> <td>Str</td> <td>MCP-Server-Tool-Web/0.1.0 (+…)</td> <td>The User-agent every request carries, and the product token robots.txt groups are matched against</td> </tr> <tr> <td>allow-private</td> <td>Bool</td> <td>False</td> <td>Allow every private range — RFC 1918, CGNAT, link-local, cloud metadata endpoints, the lot</td> </tr> <tr> <td>allow-loopback</td> <td>Bool</td> <td>False</td> <td>Allow loopback only (127.0.0.0/8 and ::1)</td> </tr> <tr> <td>allow-hosts</td> <td>Array of Str</td> <td>[]</td> <td>Exact, lowercase host names that may resolve anywhere — no wildcards or suffix matching</td> </tr> <tr> <td>allow-cidrs</td> <td>Array of Str</td> <td>[]</td> <td>CIDR blocks that may be connected to, IPv4 and IPv6</td> </tr> <tr> <td>proxy</td> <td>Str</td> <td>none</td> <td>An explicit proxy URL; without one, HTTP_PROXY/HTTPS_PROXY in the environment are refused rather than silently obeyed</td> </tr>
</tbody>
</table>

  * **max-bytes** has a floor of 4096: below that a page cannot hold a document, and every fetch would report more truncation notice than page.

  * **timeout**, **total-timeout**, **connect-timeout** and **headers-timeout** must all be greater than zero.

  * **crawl-delay** cannot be negative.

  * **user-agent** must not be empty — robots.txt groups are matched against its product token, and an origin cannot report who visited it if it is blank.

  * **allow-hosts** entries are exact host names, not URLs and not patterns: a `*` or `?` in one is a configuration error, refused at construction, because a pattern here would match nothing and look like it matched everything — see [Security](#Security).

Two more seams exist and are deliberately **not** config keys, because a `Callable` or a live object cannot come out of JSON — they are `is built` private attributes, so `.new` can pass them and `from-config` cannot:

  * **search-provider** — anything doing [MCP::Server::Tool::Web::SearchProvider](MCP::Server::Tool::Web::SearchProvider), used instead of building the named `provider`. This is how a test dispatches `web_search` without a network, and how a host wires up an engine this pack has never heard of. See [Search provider](#Search provider).

  * **fetcher** — a prepared [MCP::Server::Tool::Web::Fetcher](MCP::Server::Tool::Web::Fetcher). Given one, the pack uses it for single-page work and clones it (with the robots policy swapped in) for traversal, so the transport — and its cache of vetted connections — is shared.

Search provider
---------------

`web_search` does not talk to any particular engine directly: it asks whatever object composes [MCP::Server::Tool::Web::SearchProvider](MCP::Server::Tool::Web::SearchProvider), a role of exactly two methods —

```raku
method name(--> Str:D) { ... }                              # 'brave', 'my-provider', ...
method search(Str:D $query, Int:D $count --> List:D) { ... } # [ {title,url,snippet,age?}, ... ]
```

— which `MCP::Server::Tool::Web` composes a concrete implementation of via its `provider` config key. This distribution ships one: [MCP::Server::Tool::Web::Provider::Brave](MCP::Server::Tool::Web::Provider::Brave), talking to the Brave Search API's `web/search` endpoint. A provider that throws is the whole story the model gets: its first line is written to teach, and `MCP::Server::Tool::Web`'s `web_search` tool surfaces it verbatim as the tool's error text.

The API key is resolved **at call time**, inside `.search`, never at construction — so a pack with no `BRAVE_API_KEY` in its environment still constructs and serves `web_fetch`, `web_crawl` and `web_grep` perfectly well, and only `web_search` fails, the moment it is actually called, naming the environment variable and the two config keys that would fix it:

```text
web_search needs a Brave Search API key. Set the environment variable
BRAVE_API_KEY, or point "api-key-env" at the variable that holds it, or put
an "api-key" in the web config.
```

Bring your own engine by passing an object doing `MCP::Server::Tool::Web::SearchProvider` as `search-provider` — the constructor accepts it in place of `provider`, since a live object cannot come out of JSON config.

### The search provider is not guard-subject

**The search provider deliberately does not go through the SSRF guard described in [Security](#Security).** The endpoint `web_search` calls is the operator's, named in configuration — not the model's. A SearXNG on the operator's own LAN, or a self-hosted Brave-compatible proxy, is a legitimate thing to point this at, so the search provider talks to its API directly. The guard exists for URLs an attacker — or a confused model — chose via `web_fetch`, `web_crawl` or `web_grep`; a config key set once, by whoever deploys this pack, is not that threat model. If a provider's endpoint is ever made settable from an untrusted source, guarding it becomes that call site's responsibility — the shipped provider intentionally does not.

Security
--------

Every URL a model supplies — to `web_fetch`, `web_crawl` or `web_grep` — passes through the same floor before a byte is sent anywhere.

### Connect, then verify the peer

A URL is checked for shape first (scheme, credentials, host spelling, port range — [MCP::Server::Tool::Web::Url](MCP::Server::Tool::Web::Url)), then a TCP connection is opened, and only then is the address of the **socket that was actually opened** checked against policy ([MCP::Server::Tool::Web::Guard](MCP::Server::Tool::Web::Guard), enforced by [MCP::Server::Tool::Web::Transport](MCP::Server::Tool::Web::Transport)). Nothing is written to that socket until the check passes — no request line, no `Host` header, not even a TLS ClientHello — so a refused connection is a handshake and an immediate reset, and the peer learns nothing about what was going to be asked for.

This kills DNS rebinding rather than narrowing it: the address that is checked *is* the address of the established connection, so there is no window between "resolve, check" and "connect" for an attacker with a short-TTL DNS zone to win. The usual "resolve, check, then connect" shape has exactly that window; this pack does not use that shape.

### What is refused

Anything that is not classified `public` is refused unless a configuration key says otherwise — loopback, RFC 1918 private ranges, CGNAT, link-local addresses, cloud metadata endpoints, and the handful of other reserved and special-purpose ranges IANA has carved out (documentation ranges, benchmark ranges, multicast, and so on — see [MCP::Server::Tool::Web::Addr](MCP::Server::Tool::Web::Addr) for the full table and the RFC behind each one). An address that does not parse is refused too: the guard never treats "I could not tell" as "it is fine". The `file:`, `javascript:` and similar schemes are refused on shape, before any of this runs, and **no allow-list entry can rescue a URL refused on shape** — allow-lists widen which machines may be reached, and `file:///etc/passwd` is not a machine.

### The escape hatches

  * **allow-private** — allow every non-public class at once. The blunt instrument: one switch for "this machine may reach its own network".

  * **allow-loopback** — allow loopback only. The narrow key for test rigs and local development servers, so enabling it does not open the whole LAN.

  * **allow-hosts** — exact, lowercase host names, matched exactly. `corp` permits the host `corp` and never `wiki.corp`, and emphatically never `evil.com.corp`: suffix matching is where allow-list CVEs come from, so there is no wildcard syntax, and a pattern in the list is refused at construction rather than silently doing nothing.

  * **allow-cidrs** — CIDR blocks, IPv4 and IPv6, matched against the peer's actual address (a v4 address written in its v6-mapped form still matches a v4 block).

Every refusal names the rule, the address class, the RFC behind it, the host, the address it resolved to, and the configuration key that would permit it — the audience is a model deciding what to try next, and "denied" with nothing else just makes it try the same thing spelled differently.

### Redirects are re-checked, every hop

Redirects are followed by [MCP::Server::Tool::Web::Fetcher](MCP::Server::Tool::Web::Fetcher) itself, one hop at a time, rather than by the underlying HTTP client — so a `Location: http://10.0.0.1/` or a `Location: file:///etc/passwd` is refused on the hop that produced it, with the same rule and the same message a first request would have gotten. A redirect that drops from https to http is refused outright, downgrade or not: the bytes that would come back are readable, and rewritable, by anything on the path.

### Connection reuse is a security property here, not a risk

The HTTP client is persistent, and that is deliberate: a cached pipeline is an **already vetted, still established** socket. Rebinding a name cannot retarget an open connection, so reusing one costs nothing in safety and saves a re-check on every subsequent request. The consequence to respect the other way: one client belongs to one guard configuration, and a Transport is never shared between packs configured with different allow-lists.

### Fetch truncation is head-only

When a page runs into `max-bytes` or the deadline, only the **head** is kept — never a head-and-tail splice. The tail of a truncated HTML document is closing markup: the extractor needs a parseable prefix to produce anything at all, and `web_grep`'s line numbers have to mean something stable. (Compare [MCP::Server::Tool::Shell](https://github.com/m-doughty/MCP-Server-Tool-Shell), whose command output keeps head *and* tail, because the interesting part of a build log is usually at the end — a web page is a different shape of document, and gets a different rule.)

### robots.txt

Per the pack's ruling: `web_crawl`, and `web_grep` with a `depth` above zero, respect robots.txt by default (`respect-robots`, default `True`); `web_fetch` and depth-zero `web_grep` never consult it, regardless of that setting. A page a person asked for by name is not crawling, and no robots.txt has ever meant "this URL may not be read once, deliberately, on a human's behalf" — traversal is a different act, and that one is asked for. Setting `respect-robots` to `False` turns it off for the traversing tools too. See [MCP::Server::Tool::Web::Robots](MCP::Server::Tool::Web::Robots) for the RFC 9309 details (group matching, longest-pattern-wins, `Crawl-delay`, and why a robots.txt that cannot be fetched at all means "allowed", not "deny everything").

Known limitations
-----------------

  * **Static fetch only.** These tools issue a plain HTTP(S) GET and parse the bytes that come back; there is no browser anywhere in the code path, so a single-page application whose content is assembled by client-side JavaScript renders as its empty shell. Sites that server-render their content, or that offer a JSON API alongside their UI, are unaffected.

  * **A harmless dependency-internal deprecation notice.** The installed `JSON::Fast` 0.19 prints a `Method perl (from Mu)` deprecation warning to standard error at process exit when `from-json` is handed input that is not valid JSON (the path `web_search` and `web_grep` exercise when a provider or a page returns something unparseable). It is harmless, entirely internal to that dependency, and a candidate to report upstream; it does not affect the result either tool returns.

Examples
--------

  * `examples/web-server.raku` — a research server plugging this pack with sensible traversal limits, with the invocation line in the header.

See Also
--------

  * [MCP::Server](https://github.com/m-doughty/MCP-Server) — the server framework, the `MCP::Server::Toolkit` role this pack implements, and the `raku-mcp` command.

  * [MCP::Server::Tool::Shell](https://github.com/m-doughty/MCP-Server-Tool-Shell) — a command-runner toolkit for the same framework, with its own gate rather than a sandbox.

Author
------

Matt Doughty

License
-------

Artistic-2.0

