=begin pod

=head1 MCP::Server::Tool::Web

A web tools (search, fetch, crawl, grep) toolkit for
L<MCP::Server|https://github.com/m-doughty/MCP-Server>.

Four tools — C<web_search>, C<web_fetch>, C<web_crawl> and C<web_grep> — under
one rule that runs beneath all of them: a URL a model supplied is fetched only
after the address of the socket that was I<actually opened> has been checked,
so a hostname that resolves onto the host machine's own network is refused
whatever it is spelled like. See L<#Security>.

=head2 Synopsis

Plug it into a server you are already building:

=begin code :lang<raku>
use MCP::Server;
use MCP::Server::Tool::Web;

my $server = MCP::Server.new(:name<research>, :version<1.0>);

$server.plug: MCP::Server::Tool::Web.new;

$server.run;   # registers web_search, web_fetch, web_crawl, web_grep
=end code

Or assemble a server out of toolkit packs in one shot, with no glue code — the
shape a host's JSON config takes:

=begin code :lang<raku>
MCP::Server.new(
    :name<research>,
    :tools[
        'Web' => {
            provider        => 'brave',
            'api-key-env'   => 'BRAVE_API_KEY',
            'max-bytes'     => 4 * 1024 * 1024,
            'total-timeout' => 90,
            'crawl-delay'   => 0.5,
            'allow-hosts'   => <wiki.corp docs.corp>,   # exact names only
        },
    ],
).run;
=end code

Or run it straight off the C<raku-mcp> command line that ships with
C<MCP::Server>:

=begin code :lang<bash>
raku-mcp --tool=Web='{"crawl-delay":0.5,"max-bytes":4194304}'
=end code

=begin code :lang<bash>
raku-mcp --config=mcp.json
=end code

=begin code :lang<json>
{
  "name": "research",
  "instructions": "Search and read the web. Prefer web_grep over web_fetch when you know what you are looking for.",
  "tools": {
    "Web": { "crawl-delay": 0.5, "allow-hosts": ["wiki.corp"] }
  }
}
=end code

Wiring that into Claude Code:

=begin code :lang<json>
{
  "mcpServers": {
    "research": {
      "command": "raku-mcp",
      "args": ["--config=/home/me/mcp.json"],
      "env": { "BRAVE_API_KEY": "..." }
    }
  }
}
=end code

=head2 Description

C<web_search> answers JSON so the model can pick a URL out of it;
C<web_fetch>, C<web_crawl> and C<web_grep> answer plain text, because a page
— or a report about pages — is text. Nothing here needs a browser: pages are
fetched over plain HTTP(S) and their markup, scripts and navigation are
stripped by a hand-rolled extractor (L<MCP::Server::Tool::Web::Extract>), not
rendered.

C<web_search> needs a search provider's API key to answer anything;
C<web_fetch>, C<web_crawl> and C<web_grep> need nothing beyond network access,
and construct successfully even with no key configured anywhere — see
L<#Search provider>.

=head2 Tools

Parameter names are a contract, not a matter of taste: a policy layer in
front of the server (such as C<MCP::Client::Policy>) matches calls by them to
decide what a call would reach, so C<url> stays C<url> and multi-word
parameters stay kebab-case.

=head3 web_search

The search engine, behind a seam. Answers JSON so the model can pick a URL
out of it. Zero results is an empty list and a notice, not an error.

=begin table
Argument | Type    | Required | Meaning
=========|=========|==========|=========================================
query    | string  | yes      | What to search for; up to 400 characters
count    | integer | no       | How many results, 1 to 20; defaults to 10
=end table

=begin code :lang<text>
# web_search — JSON, so the model can pick a URL out of it
{"count":10,"provider":"brave","query":"raku grammars",
 "results":[{"snippet":"…","title":"Grammars","url":"https://docs.raku.org/…"}]}

# a query that matches nothing:
{"count":10,"notice":"No results for 'asdkfjhasdkfjh39847'. The engine matched nothing: try fewer or more common terms, or drop any quoting.",
 "provider":"brave","query":"asdkfjhasdkfjh39847","results":[]}
=end code

=head3 web_fetch

One page, as text: a C<# title> line, the final URL when a redirect moved
it, then the page with its markup, scripts and navigation removed. Exactly
one parameter, deliberately: a windowing tool would invite the model to page
through a document it should have grepped instead. Only http and https URLs
can be fetched, redirects are followed (every hop re-checked — see
L<#Security>), and a page bigger than C<max-bytes> comes back cut short and
says so on its last line.

=begin table
Argument | Type   | Required | Meaning
=========|========|==========|==========================================
url      | string | yes      | Absolute http or https URL of the page to read
=end table

=begin code :lang<text>
# web_fetch — plain text, because a page is text
# Grammars
Retrieved from https://docs.raku.org/language/grammars

Grammars are a powerful tool used to destructure text …

# a page cut short by the byte cap:
[truncated: kept the first 2097152 characters because the server's byte cap of 2097152 bytes was reached; the rest of the page was not fetched]
=end code

=head3 web_crawl

Follows the links from a page and returns an B<index> of what is there —
never the pages themselves. Only links on the same origin as the starting
page (same scheme, host and effective port) are followed; C<depth> counts
hops from the starting page; robots.txt is respected by default (see
L<#Security>). See L<MCP::Server::Tool::Web::Crawl> for the walk itself.

=begin table
Argument   | Type    | Required | Meaning
===========|=========|==========|=========================================
url        | string  | yes      | Absolute http or https URL to start from
depth      | integer | no       | Hops to follow, 0 to 3; 0 reports only the starting page; defaults to 1
max-pages  | integer | no       | Stop after fetching this many pages, counting the seed, 1 to 50; defaults to 20
=end table

=begin code :lang<text>
# web_crawl — an index, not the pages
Crawled 3 pages from https://docs.example.com/ (depth 1, same origin only).

https://docs.example.com/  200  4213 bytes  Documentation
https://docs.example.com/install  200  2288 bytes  Installing

[stopped after 20 pages; raise max-pages to go further]
=end code

Why an index rather than the pages: concatenating twenty pages into one tool
result is a way to spend a context window without answering anything — what
the model would actually receive is page one, half of page two, and a
truncation marker. So a crawl reports what is there, one line per page, and
the corpus stays reachable through C<web_grep> (with a C<depth>) and
C<web_fetch>.

=head3 web_grep

C<fs_grep> for the web: search one page, or a whole site at a C<depth>, and
return only the matching lines as C<url:line:text>. The pattern is literal
text unless C<regex> is set, in which case it is a I<Raku> regex, not a
PCRE one. This is far cheaper than C<web_fetch> or C<web_crawl> when you
know what you are looking for.

=begin table
Argument     | Type    | Required | Meaning
=============|=========|==========|=========================================
pattern      | string  | yes      | Text to search for, or a Raku regex when regex is true
url          | string  | yes      | Page to search, and to crawl from when depth is above 0
regex        | boolean | no       | Treat pattern as a Raku regex; defaults to false
context      | integer | no       | Lines of surrounding context, 0 to 20; defaults to 0
max-results  | integer | no       | Stop after this many matching lines, 1 to 1000; defaults to 50
depth        | integer | no       | Hops to follow from url, 0 to 3; defaults to 0 (that page alone)
max-pages    | integer | no       | When depth is above 0, stop after this many pages, 1 to 50; defaults to 20
=end table

=begin code :lang<text>
# web_grep — only the lines that matched, exactly as fs_grep reports them
https://docs.example.com/install:42:  zef install Foo::Bar

[stopped at 50 matching lines; raise max-results to see more]
=end code

With the default C<depth> of 0 only the URL given is searched; with a depth
above 0 the same-origin links are followed as C<web_crawl> does, and no more
pages are fetched once C<max-results> lines have matched — which is the
whole reason C<web_grep> is cheaper than crawling and then fetching.

=head2 Configuration

Every attribute below is a JSON config key — C<from-config> validates
against exactly this set, so a typo names the valid keys instead of doing
nothing quietly:

=begin code :lang<raku>
MCP::Server::Tool::Web.from-config({ providr => 'brave' });
# Unknown config key(s) for MCP::Server::Tool::Web: providr.
# Valid keys: allow-cidrs, allow-hosts, allow-loopback, allow-private,
# api-key, api-key-env, connect-timeout, crawl-delay, headers-timeout,
# max-bytes, proxy, provider, respect-robots, safesearch, search-country,
# search-lang, timeout, total-timeout, user-agent
=end code

=begin table
Key             | Type            | Default                  | Meaning
================|=================|===========================|=========================================
provider        | Str             | 'brave'                  | Which search engine web_search asks; see L<#Search provider>
api-key-env     | Str             | 'BRAVE_API_KEY'          | Environment variable the provider's API key is read from
api-key         | Str             | none                     | The API key itself; beats api-key-env when both are set
search-country  | Str             | provider's own default   | Two-letter country code biasing search results ('US', 'GB', ...)
search-lang     | Str             | provider's own default   | Language code for results ('en', 'de', ...)
safesearch      | Str             | 'moderate'               | Adult-content filtering: 'off', 'moderate' or 'strict'
timeout         | Real (seconds)  | 15                       | One search request's budget; fetches use the three below instead
max-bytes       | Int (bytes)     | 2097152 (2 MiB)          | Body bytes B<one tool call> may retain, across every page it fetches
total-timeout   | Real (seconds)  | 60                       | Wall-clock budget for a whole tool call, crawls included
connect-timeout | Real (seconds)  | 15                       | One connection attempt's budget, clamped down by whatever is left of the call's own budget
headers-timeout | Real (seconds)  | 30                       | One response's headers budget, likewise clamped
crawl-delay     | Real (seconds)  | 0.25                     | Seconds left between requests while crawling; a robots.txt Crawl-delay may raise it, nothing lowers it
respect-robots  | Bool            | True                     | Consult robots.txt when traversing (web_crawl, and web_grep with a depth); never for a single named page
user-agent      | Str             | MCP-Server-Tool-Web/0.1.0 (+…) | The User-agent every request carries, and the product token robots.txt groups are matched against
allow-private   | Bool            | False                    | Allow every private range — RFC 1918, CGNAT, link-local, cloud metadata endpoints, the lot
allow-loopback  | Bool            | False                    | Allow loopback only (127.0.0.0/8 and ::1)
allow-hosts     | Array of Str    | []                       | Exact, lowercase host names that may resolve anywhere — no wildcards or suffix matching
allow-cidrs     | Array of Str    | []                       | CIDR blocks that may be connected to, IPv4 and IPv6
proxy           | Str             | none                     | An explicit proxy URL; without one, HTTP_PROXY/HTTPS_PROXY in the environment are refused rather than silently obeyed
=end table

=item B<max-bytes> has a floor of 4096: below that a page cannot hold a
document, and every fetch would report more truncation notice than page.
=item B<timeout>, B<total-timeout>, B<connect-timeout> and B<headers-timeout>
must all be greater than zero.
=item B<crawl-delay> cannot be negative.
=item B<user-agent> must not be empty — robots.txt groups are matched against
its product token, and an origin cannot report who visited it if it is blank.
=item B<allow-hosts> entries are exact host names, not URLs and not patterns:
a C<*> or C<?> in one is a configuration error, refused at construction,
because a pattern here would match nothing and look like it matched
everything — see L<#Security>.

Two more seams exist and are deliberately B<not> config keys, because a
C<Callable> or a live object cannot come out of JSON — they are C<is built>
private attributes, so C<.new> can pass them and C<from-config> cannot:

=item B<search-provider> — anything doing
L<MCP::Server::Tool::Web::SearchProvider>, used instead of building the named
C<provider>. This is how a test dispatches C<web_search> without a network,
and how a host wires up an engine this pack has never heard of. See
L<#Search provider>.
=item B<fetcher> — a prepared L<MCP::Server::Tool::Web::Fetcher>. Given one,
the pack uses it for single-page work and clones it (with the robots policy
swapped in) for traversal, so the transport — and its cache of vetted
connections — is shared.

=head2 Search provider

C<web_search> does not talk to any particular engine directly: it asks
whatever object composes L<MCP::Server::Tool::Web::SearchProvider>, a role
of exactly two methods —

=begin code :lang<raku>
method name(--> Str:D) { ... }                              # 'brave', 'my-provider', ...
method search(Str:D $query, Int:D $count --> List:D) { ... } # [ {title,url,snippet,age?}, ... ]
=end code

— which C<MCP::Server::Tool::Web> composes a concrete implementation of via
its C<provider> config key. This distribution ships one:
L<MCP::Server::Tool::Web::Provider::Brave>, talking to the Brave Search
API's C<web/search> endpoint. A provider that throws is the whole story the
model gets: its first line is written to teach, and
C<MCP::Server::Tool::Web>'s C<web_search> tool surfaces it verbatim as the
tool's error text.

The API key is resolved B<at call time>, inside C<.search>, never at
construction — so a pack with no C<BRAVE_API_KEY> in its environment still
constructs and serves C<web_fetch>, C<web_crawl> and C<web_grep> perfectly
well, and only C<web_search> fails, the moment it is actually called,
naming the environment variable and the two config keys that would fix it:

=begin code :lang<text>
web_search needs a Brave Search API key. Set the environment variable
BRAVE_API_KEY, or point "api-key-env" at the variable that holds it, or put
an "api-key" in the web config.
=end code

Bring your own engine by passing an object doing
C<MCP::Server::Tool::Web::SearchProvider> as C<search-provider> — the
constructor accepts it in place of C<provider>, since a live object cannot
come out of JSON config.

=head3 The search provider is not guard-subject

B<The search provider deliberately does not go through the SSRF guard
described in L<#Security>.> The endpoint C<web_search> calls is the
operator's, named in configuration — not the model's. A SearXNG on the
operator's own LAN, or a self-hosted Brave-compatible proxy, is a legitimate
thing to point this at, so the search provider talks to its API directly.
The guard exists for URLs an attacker — or a confused model — chose via
C<web_fetch>, C<web_crawl> or C<web_grep>; a config key set once, by whoever
deploys this pack, is not that threat model. If a provider's endpoint is
ever made settable from an untrusted source, guarding it becomes that call
site's responsibility — the shipped provider intentionally does not.

=head2 Security

Every URL a model supplies — to C<web_fetch>, C<web_crawl> or C<web_grep> —
passes through the same floor before a byte is sent anywhere.

=head3 Connect, then verify the peer

A URL is checked for shape first (scheme, credentials, host spelling, port
range — L<MCP::Server::Tool::Web::Url>), then a TCP connection is opened,
and only then is the address of the B<socket that was actually opened>
checked against policy (L<MCP::Server::Tool::Web::Guard>, enforced by
L<MCP::Server::Tool::Web::Transport>). Nothing is written to that socket
until the check passes — no request line, no C<Host> header, not even a TLS
ClientHello — so a refused connection is a handshake and an immediate reset,
and the peer learns nothing about what was going to be asked for.

This kills DNS rebinding rather than narrowing it: the address that is
checked I<is> the address of the established connection, so there is no
window between "resolve, check" and "connect" for an attacker with a
short-TTL DNS zone to win. The usual "resolve, check, then connect" shape
has exactly that window; this pack does not use that shape.

=head3 What is refused

Anything that is not classified C<public> is refused unless a configuration
key says otherwise — loopback, RFC 1918 private ranges, CGNAT, link-local
addresses, cloud metadata endpoints, and the handful of other reserved and
special-purpose ranges IANA has carved out (documentation ranges, benchmark
ranges, multicast, and so on — see L<MCP::Server::Tool::Web::Addr> for the
full table and the RFC behind each one). An address that does not parse is
refused too: the guard never treats "I could not tell" as "it is fine". The
C<file:>, C<javascript:> and similar schemes are refused on shape, before
any of this runs, and B<no allow-list entry can rescue a URL refused on
shape> — allow-lists widen which machines may be reached, and
C<file:///etc/passwd> is not a machine.

=head3 The escape hatches

=item B<allow-private> — allow every non-public class at once. The blunt
instrument: one switch for "this machine may reach its own network".
=item B<allow-loopback> — allow loopback only. The narrow key for test rigs
and local development servers, so enabling it does not open the whole LAN.
=item B<allow-hosts> — exact, lowercase host names, matched exactly.
C<corp> permits the host C<corp> and never C<wiki.corp>, and emphatically
never C<evil.com.corp>: suffix matching is where allow-list CVEs come from,
so there is no wildcard syntax, and a pattern in the list is refused at
construction rather than silently doing nothing.
=item B<allow-cidrs> — CIDR blocks, IPv4 and IPv6, matched against the
peer's actual address (a v4 address written in its v6-mapped form still
matches a v4 block).

Every refusal names the rule, the address class, the RFC behind it, the
host, the address it resolved to, and the configuration key that would
permit it — the audience is a model deciding what to try next, and "denied"
with nothing else just makes it try the same thing spelled differently.

=head3 Redirects are re-checked, every hop

Redirects are followed by L<MCP::Server::Tool::Web::Fetcher> itself, one hop
at a time, rather than by the underlying HTTP client — so a
C<Location: http://10.0.0.1/> or a C<Location: file:///etc/passwd> is
refused on the hop that produced it, with the same rule and the same
message a first request would have gotten. A redirect that drops from https
to http is refused outright, downgrade or not: the bytes that would come
back are readable, and rewritable, by anything on the path.

=head3 Connection reuse is a security property here, not a risk

The HTTP client is persistent, and that is deliberate: a cached pipeline is
an B<already vetted, still established> socket. Rebinding a name cannot
retarget an open connection, so reusing one costs nothing in safety and
saves a re-check on every subsequent request. The consequence to respect
the other way: one client belongs to one guard configuration, and a
Transport is never shared between packs configured with different
allow-lists.

=head3 Fetch truncation is head-only

When a page runs into C<max-bytes> or the deadline, only the B<head> is
kept — never a head-and-tail splice. The tail of a truncated HTML document
is closing markup: the extractor needs a parseable prefix to produce
anything at all, and C<web_grep>'s line numbers have to mean something
stable. (Compare L<MCP::Server::Tool::Shell|https://github.com/m-doughty/MCP-Server-Tool-Shell>,
whose command output keeps head I<and> tail, because the interesting part
of a build log is usually at the end — a web page is a different shape of
document, and gets a different rule.)

=head3 robots.txt

Per the pack's ruling: C<web_crawl>, and C<web_grep> with a C<depth> above
zero, respect robots.txt by default (C<respect-robots>, default C<True>);
C<web_fetch> and depth-zero C<web_grep> never consult it, regardless of that
setting. A page a person asked for by name is not crawling, and no
robots.txt has ever meant "this URL may not be read once, deliberately, on
a human's behalf" — traversal is a different act, and that one is asked
for. Setting C<respect-robots> to C<False> turns it off for the traversing
tools too. See L<MCP::Server::Tool::Web::Robots> for the RFC 9309 details
(group matching, longest-pattern-wins, C<Crawl-delay>, and why a robots.txt
that cannot be fetched at all means "allowed", not "deny everything").

=head2 Known limitations

=item B<Static fetch only.> These tools issue a plain HTTP(S) GET and parse
the bytes that come back; there is no browser anywhere in the code path, so
a single-page application whose content is assembled by client-side
JavaScript renders as its empty shell. Sites that server-render their
content, or that offer a JSON API alongside their UI, are unaffected.
=item B<A harmless dependency-internal deprecation notice.> The installed
C<JSON::Fast> 0.19 prints a C<Method perl (from Mu)> deprecation warning to
standard error at process exit when C<from-json> is handed input that is
not valid JSON (the path C<web_search> and C<web_grep> exercise when a
provider or a page returns something unparseable). It is harmless, entirely
internal to that dependency, and a candidate to report upstream; it does
not affect the result either tool returns.

=head2 Examples

=item C<examples/web-server.raku> — a research server plugging this pack
with sensible traversal limits, with the invocation line in the header.

=head2 See Also

=item L<MCP::Server|https://github.com/m-doughty/MCP-Server> — the server
framework, the C<MCP::Server::Toolkit> role this pack implements, and the
C<raku-mcp> command.
=item L<MCP::Server::Tool::Shell|https://github.com/m-doughty/MCP-Server-Tool-Shell> —
a command-runner toolkit for the same framework, with its own gate rather
than a sandbox.

=head2 Author

Matt Doughty

=head2 License

Artistic-2.0

=end pod
