# NAME

Sitemap - Sitemap generator for Raku

# DESCRIPTION

Sitemap is a Raku module for generating XML sitemaps, crawling websites, and fetching sitemaps from servers.

## Features

- Generate standard XML sitemaps (sitemap.org protocol)
- Image & hreflang link extraction during crawl (configurable via --no-images / --no-hreflang)
- Automatic video & news extraction during crawl (HTML tags + JSON-LD, `--videos` / `--news` flags)
- Separate news sitemap output with 48-hour stale detection
- Lastmod extraction from HTTP headers (configurable via --no-lastmod)
- Priority calculation from crawl depth (configurable via --no-priority)
- Manual API: Video & News sitemaps via `Sitemap::Item::Video` / `Sitemap::Item::News`
- Output formats: XML, HTML, TXT, RSS 2.0, Atom 1.0, Media RSS (mRSS)
- Crawl websites with respect for robots.txt
- Parse existing sitemaps (including recursive parsing of sitemap indexes)
- Fetch sitemaps from remote servers (with recursive support for sitemap indexes)
- Automatic gzip decompression of .gz sitemaps
- Automatic sitemap discovery (robots.txt + fallback to /sitemap.xml)
- Recursive fetch creates descriptive directories (e.g., `example-com-sitemap/`)
- Child sitemaps named with path context (e.g., `sitemap-blog.xml`, `sitemap-news-2024.xml`)
- Gzip compression for output files
- XSL stylesheet support for human-readable XML
- Build hierarchical site trees with depth-inferred priorities
- Serialize/deserialize site trees to/from YAML
- Sitemap index for large sites (auto-split at 50k URLs, or manually from existing sitemaps)

Note: Image, hreflang, lastmod, and priority data only appears in XML format output.

## Dependencies

- [Cro::HTTP](https://raku.land/zef:cro/Cro::HTTP) - HTTP client for crawling and fetching
- [URI](https://raku.land/zef:raku-community-modules/URI) - URL parsing and resolution
- [LibXML](https://raku.land/zef:dwarring/LibXML)- XML parsing
- [LibXML::Writer](https://raku.land/zef:dwarring/LibXML::Writer) - XML generation (sitemaps, RSS, Atom)
- [Compress::Zlib](https://raku.land/github:retupmoca/Compress::Zlib) - Gzip compression and decompression
- [JSON::Fast](https://raku.land/zef:timo/JSON::Fast) - JSON parsing (JSON-LD extraction)
- [YAMLish](https://raku.land/zef:leont/YAMLish) - YAML serialization (SiteTree)
- [Syndicate](https://raku.land/zef:sasha/Syndicate) - Feed format generation (RSS, Atom, MRSS)

## Installation

```bash
zef install Sitemap
```

### Installing from source

```bash
cd /path/to/Sitemap
zef install .
```

Requires Raku 6.d or later.

# SYNOPSIS

## Command Line

```bash
# Crawl a website and generate XML sitemap
sitemap https://example.com

# Crawl with options
sitemap https://example.com -o sitemap.xml -f xml -v

# Generate HTML sitemap
sitemap https://example.com -f html -o sitemap.html

# Generate RSS feed
sitemap https://example.com -f rss -o feed.rss

# Build from URL list file
sitemap build urls.txt -o sitemap.xml

# Parse existing sitemap
sitemap parse existing-sitemap.xml

# Fetch sitemap from server (auto-discover from robots.txt)
sitemap fetch example.com

# Fetch with recursive support (downloads all child sitemaps in index)
sitemap fetch example.com --recursive

# Recursive fetch with descriptive directory naming
# Creates example-com-sitemap/ directory with context-aware filenames
sitemap fetch example.com --recursive -v

# Crawl local directory and generate sitemap
sitemap dir ./my-site

# Crawl directory with custom base URL
sitemap dir ./my-site --base-url https://example.com -v

# Build from JSON or YAML site tree definition (priorities inferred from depth)
sitemap tree site.json -o sitemap.xml
sitemap tree site.yaml -o sitemap.xml

# View the tree structure
sitemap tree site.json --verbose
```

## Library Usage

### Sitemap::Builder (Generate Sitemaps)

```raku
use Sitemap::Builder;

# Create sitemap from URLs
my $builder = Sitemap::Builder.new;
$builder.add-item('https://example.com/page1', :priority(0.8));
$builder.add-item('https://example.com/page2', :lastmod(DateTime.now));

# Write to file (auto-splits at 50k URLs / when add-sitemap was used)
$builder.write: 'sitemap.xml';
# When items exceed $builder.max-entries (default 50000) this auto-splits
# into sitemap-1.xml, sitemap-2.xml, ... plus a sitemap-index.xml pointing
# at the chunks.

# With compression and XSL
my $builder2 = Sitemap::Builder.new(:compress, :xsl-url('/sitemap.xsl'));
$builder2.add-item('https://example.com/page');
$builder2.write: 'sitemap.xml';

# Manually assemble a sitemap index from existing sitemaps
my $index = Sitemap::Builder.new;
$index.add-sitemap('https://example.com/posts.xml', :lastmod(DateTime.now));
$index.add-sitemap('https://example.com/news.xml',  :lastmod('2024-01-01'));
$index.add-sitemap('https://example.com/video.xml');
$index.write: 'sitemap-index.xml';        # <sitemapindex> wrapper file
my $xml = $index.render-index;            # or capture the index XML directly

# Discover sitemap from robots.txt (with /sitemap.xml fallback)
my $discovered = $builder.discover-sitemap('https://example.com');
```

### Sitemap::Crawler (Crawl Websites)

```raku
use Sitemap::Crawler;

# Crawl a website
my $crawler = Sitemap::Crawler.new('https://example.com');

$crawler.on-add: -> $url {
    say "Found: $url";
};

my $builder = $crawler.crawl;
$builder.write: 'sitemap.xml';

# Crawl with automatic video and news extraction
my $crawler2 = Sitemap::Crawler.new(
    'https://example.com',
    :extract-videos,
    :extract-news,
);
my $builder2 = $crawler2.crawl;
$builder2.write: 'sitemap.xml';

# News builder contains extracted NewsArticle objects
# (only populated when extract-news is True)
if $crawler2.news-builder -> $nb {
    $nb.write: 'sitemap-news.xml';
}
```

### Sitemap::DirScanner (Crawl Local Directories)

```raku
use Sitemap::DirScanner;

# scan-dir returns a Hash: builder, news-builder, stale-news-count
my %result = scan-dir('./my-site',
    :base-url<https://example.com>,
    :extract-videos,
    :extract-news,
);
my $builder = %result<builder>;

$builder.write: 'sitemap.xml';

if %result<news-builder> -> $nb {
    $nb.write: 'sitemap-news.xml';
}
```

### Sitemap::SiteTree (Build from Site Hierarchy)

```raku
use Sitemap::SiteTree;

# Build a tree - priorities auto-inferred from depth
my $root = Sitemap::SiteTree.new;
my $blog = $root.add-child('blog', :priority(0.8));
$blog.add-child('first-post');
$blog.add-child('second-post');

my $builder = $root.to-builder('https://example.com');
$builder.write: 'sitemap.xml';
# Priorities: root=1.0, blog=0.8, first-post=0.6, second-post=0.6

# Wire parents from a flat list
my $home  = Sitemap::SiteTree.new(:stub<home>);
my $about = Sitemap::SiteTree.new(:stub<about>, :parent-stub<home>);
Sitemap::SiteTree.wire-parents([$home, $about]);

# Debug tree visualization
say $root.tree;

# Serialize to/from YAML
my $yaml = $root.to-yaml;
"site-tree.yml".IO.spurt: $yaml;

my $reconstructed = Sitemap::SiteTree.from-yaml($yaml);
```

### YAML Serialization

`Sitemap::SiteTree` and `Sitemap::Item` support serialization to/from YAML
via `to-yaml`/`from-yaml` (on SiteTree) and `to-hash`/`from-hash` (on both).

```raku
use Sitemap::SiteTree;

# Build a tree
my $root = Sitemap::SiteTree.new;
my $blog = $root.add-child('blog', :priority(0.8));
$blog.add-child('first-post');

# Roundtrip through YAML
my $yaml = $root.to-yaml;
my $tree = Sitemap::SiteTree.from-yaml($yaml);

# Rebuild the sitemap
my $builder = $tree.to-builder('https://example.com');
$builder.write: 'sitemap.xml';
```

Items and sub-resources (images, videos, links, news) can be serialized
independently:

```raku
use Sitemap::Item;
use YAMLish;

my $item = Sitemap::Item.new(
    url => 'https://example.com/page',
    priority => 0.8,
);
$item.add-image('https://example.com/img.jpg');

my $yaml = save-yaml($item.to-hash);
my $copy = Sitemap::Item.from-hash(load-yaml($yaml));
```

The YAML format maps directly to the object structure - parent-child
relationships are preserved via nested `children` arrays.

### Sitemap::Fetcher (Fetch Sitemaps from Servers)

```raku
use Sitemap::Fetcher;

# Fetch a single sitemap
my $xml = fetch('https://example.com/sitemap.xml', :verbose);

# Fetch sitemap with auto-discovery (robots.txt + /sitemap.xml fallback)
my $xml = fetch('https://example.com', :ssl-verify(False));

# Recursive fetch (downloads all child sitemaps in a sitemap index)
my %result = fetch-recursive(
    'https://example.com/sitemap.xml',
    :format<xml>,
    :verbose
);
# %result<index-file> = saved index file
# %result<dir> = directory with child sitemaps
# %result<children> = list of child sitemap files

# Generate output filename with path context
my $filename = output-filename(
    'https://example.com/blog/sitemap.xml',
    :format<xml>
);
# Returns: sitemap-blog.xml
```

For non-crawl workflows, Video and News data can be added manually via the builder API:
```raku
use Sitemap::Builder;
my $builder = Sitemap::Builder.new;

# Add video sitemap data
$builder.add-item('https://example.com/video-page',
    videos => [
        Sitemap::Item::Video.new(
            thumbnail-loc => 'https://example.com/thumb.jpg',
            title         => 'My Video Title',
            description   => 'A description of the video',
        )
    ]
);

# Add news sitemap data
$builder.add-item('https://example.com/news-article',
    news => [
        Sitemap::Item::News.new(
            publication          => 'Example News',
            publication-language => 'en',
            title                => 'Article Headline',
            publication-date     => DateTime.new('2024-01-01'),
        )
    ]
);
```

### Sitemap::JsonLd (Extract JSON-LD Structured Data)

```raku
use Sitemap::JsonLd;

# Extract all JSON-LD objects from HTML
my @objects = extract-jsonld($html);

# Filter by type (supports schema.org URL prefixes and @graph)
my @videos = find-by-type(@objects.List, 'VideoObject');
my @news = find-by-type(@objects.List, 'NewsArticle');

# Convenience: extract video/news objects with full field mapping
my @video-objects = extract-video-objects($html);
# Returns: url, thumbnail, title, description, duration (seconds), pub-date

my @news-objects = extract-news-objects($html);
# Returns: publication, language, title, publication-date, stale (Bool)
# Matches: NewsArticle, ReportageNewsArticle, OpinionNewsArticle,
#          ReviewNewsArticle, AnalysisNewsArticle, BackgroundNewsArticle
# Does NOT match: Article, BlogPosting, or other non-news types
# Articles older than 48h are marked stale => True

# Parse ISO 8601 durations (e.g., from VideoObject)
my $seconds = parse-iso8601-duration('PT1H2M3S');  # 3723
```

## CLI Options

### Crawl Command (`sitemap <url>`)
- `-o, --output <file>` - Output file (default: `sitemap.<fmt>`)
- `-f, --format <format>` - Output format: xml, html, txt, rss, atom, mrss (default: xml)
- `-v, --verbose` - Verbose output
- `--debug` - Debug output (crawl tracing)
- `-d, --max-depth <n>` - Maximum crawl depth
- `-u, --max-urls <n>` - Maximum number of URLs (default: 0 = unlimited; unbounded crawls buffer every URL in memory — set a limit for large sites to cap memory use)
- `-m, --max-urls-per-file <n>` - Max URLs per sitemap file (default: 50000)
- `-c, --concurrency <n>` - Max concurrent page fetches (default: 20)
- `-a, --user-agent <str>` - User agent string
- `--no-respect-robots` - Don't respect robots.txt (default: respect)
- `--no-images` - Don't extract images during crawl (default: extract)
- `--max-images <n>` - Max images per page (default: 1000)
- `--max-videos <n>` - Max videos per page (default: 100)
- `--no-hreflang` - Don't extract hreflang links during crawl (default: extract)
- `--videos` - Extract video metadata from HTML tags and JSON-LD (default: off)
- `--news` - Extract NewsArticle JSON-LD and write a separate news sitemap (default: off)
- `--no-lastmod` - Don't extract lastmod from HTTP headers (default: extract)
- `--no-priority` - Don't calculate priority from crawl depth (default: calculate)
- `--no-follow` - Crawl only the start URL; don't follow links
- `--no-verify-ssl` - Don't verify SSL certificates (default: verify)
- `--allow-http-fallback` - Allow HTTPS→HTTP downgrade on SSL/TLS errors (default: off)
- `--max-redirects <n>` - Max redirects followed per chain (default: 5)
- `--exclude-dirs <list>` - Skip URL paths starting with any comma-separated prefix (matches whole path segments, so /admin also excludes /admin/x)
- `--exclude-extensions <list>` - Comma-separated extensions to exclude, REPLACING the default list
- `--extra-exclude-extensions <list>` - Comma-separated extensions to exclude, ADDED to the default list
- `-z, --compress` - Gzip compress output file
- `--no-pretty` - Disable pretty printing (XML output only, default: pretty)
- `--xsl <url>` - Add XSL stylesheet reference

### Fetch Command (`sitemap fetch <url>`)
- `-o, --output <file>` - Output file (default: preserve original filename)
- `-f, --format <fmt>` - Output format: xml (default), html, txt, rss, atom, mrss
- `-r, --recursive` - Fetch all sitemaps in a sitemap index
- `--force` - Delete existing output dir before recursive fetch
- `--follow-foreign-children` - Fetch child sitemaps on other hosts (default: same-host only; disables the same-origin guard)
- `--no-verify-ssl` - Don't verify SSL certificates
- `-z, --compress` - Gzip the fetched sitemap (single fetch only; not with --recursive)
- `-v, --verbose` - Verbose output

Note: Recursive fetch creates a descriptive directory (e.g., `example-com-sitemap/`) with path-aware filenames (e.g., `sitemap-blog.xml`, `sitemap-news-2024.xml`).

### Build Command (`sitemap build <file>`)
- `-o, --output <file>` - Output file (default: `sitemap.<fmt>`)
- `-f, --format <fmt>` - Output format: xml, html, txt, rss, atom, mrss (default: xml)
- `-m, --max-urls-per-file <n>` - Max URLs per sitemap file (default: 50000)
- `--base-url <url>` - Base URL for relative URLs
- `-v, --verbose` - Verbose output

Note: indexes are produced automatically when you lower `-m` below the item
count (or exceed 50k): the command writes chunk files (e.g. `sitemap-1.xml`,
`sitemap-2.xml`) plus a `sitemap-index.xml` pointing at them. Assembling an
index from already-existing sitemap files is a library operation
(`Sitemap::Builder.add-sitemap`), not a CLI subcommand.

### Parse Command (`sitemap parse <file|url>`)
- `-r, --recursive` - Enable recursive parsing
- `--recursive-depth=<n>` - Set recursive depth; any value >= 0 enables recursion (0=unlimited)
- `--follow-foreign-children` - Follow child sitemaps on other hosts (default: same-host only; disables the same-origin guard)
- `-c, --concurrency=<n>` - Parallel child fetches for --recursive (default: 4)
- `--no-verify-ssl` - Don't verify SSL certificates
- `-v, --verbose` - Verbose output (show lastmod)
- `-h, --help` - Show this help message

### Convert Command (`sitemap convert <file>`)
- `-o, --output <file>` - Output file (default: derived from input)
- `-f, --format <fmt>` - Output format: xml, html, txt, rss, atom, mrss (default: txt)
- `-v, --verbose` - Verbose output
- `-h, --help` - Show this help message

### Tree Command (`sitemap tree <file>`)
Build a sitemap from a JSON (.json) or YAML (.yml/.yaml) site tree definition. Priorities are auto-inferred from tree depth.

JSON format:
```json
{
  "base_url": "https://example.com",
  "pages": [
    {"stub": "home", "priority": 1.0},
    {"stub": "about",
     "children": [
       {"stub": "team"}
     ]},
    {"stub": "blog",
     "children": [
       {"stub": "first-post"},
       {"stub": "second-post"}
     ]}
  ]
}
```

YAML format:
```yaml
base_url: "https://example.com"
pages:
  - stub: home
    priority: 1.0
  - stub: about
    children:
      - stub: team
  - stub: blog
    children:
      - stub: first-post
      - stub: second-post
```

- `-o, --output <file>` - Output file (default: `sitemap.<fmt>`)
- `-f, --format <fmt>` - Output format: xml, html, txt, rss, atom, mrss (default: xml)
- `-z, --compress` - Gzip compress output file
- `--no-pretty` - Disable pretty printing (XML only)
- `--xsl <url>` - Add XSL stylesheet reference
- `-v, --verbose` - Show tree structure during generation

### Dir Command (`sitemap dir <directory>`)
- `--base-url <url>` - Base URL for sitemap entries (auto-detect from robots.txt, fallback: file:///)
- `-d, --max-depth <n>` - Maximum crawl depth (default: 0 = unlimited)
- `-u, --max-urls <n>` - Maximum number of URLs (default: 0 = unlimited; unbounded scans buffer every URL in memory — set a limit for large sites to cap memory use)
- `-c, --concurrency <n>` - Max concurrent file processors (default: 1)
- `-m, --max-urls-per-file <n>` - Max URLs per sitemap file (default: 50000)
- `--no-images` - Don't extract images (default: extract)
- `--max-images <n>` - Max images per page (default: 1000)
- `--max-videos <n>` - Max videos per page (default: 100)
- `--no-hreflang` - Don't extract hreflang links (default: extract)
- `--videos` - Extract video metadata from HTML tags and JSON-LD (default: off)
- `--news` - Extract NewsArticle JSON-LD and write a separate news sitemap (default: off)
- `--no-priority` - Don't calculate priority from depth (default: calculate)
- Standard output options also available: `-o`, `-f`, `-z`, `--no-pretty`, `--xsl`, `-v`

# DEVELOPMENT

## Running tests

The suite lives in `t/` as one `.rakutest` file per area:

```sh
raku -I lib t/04-grammars.rakutest     # a single file
raku -I lib t/*.rakutest               # the whole suite (fast, CLI skipped)
```

`t/14-cli.rakutest` spawns a fresh Raku VM for every CLI invocation
(41 subprocess starts) and runs local Cro HTTP servers, so it takes ~40s
on its own. For that reason it is skipped by default. Run the full suite
with the CLI coverage enabled via the `SITEMAP_SLOW_TESTS` environment
variable:

```sh
SITEMAP_SLOW_TESTS=1 raku -I lib t/*.rakutest
SITEMAP_SLOW_TESTS=1 prove6 -e 'raku -Ilib' t/*.rakutest   # CI-style
SITEMAP_SLOW_TESTS=1 zef test .   # as zef/mi6 would run them
```

Running the full suite can take a few minutes — give it a generous
timeout instead of assuming it has hung.

# AUTHOR

Sasha Abbott <sashaa@disroot.org>

# LICENSE

This software is dedicated to the public domain under the CC0 1.0 Universal (CC0 1.0) Public Domain Dedication.

To the extent possible under law, the author has waived all copyright and related or neighboring rights to this work.
