=begin pod

=head1 LLM::Chat

Simple framework for LLM inferencing in Raku. Supports multiple backends
(OpenAI-compatible, KoboldCpp), chat templates (ChatML, Llama 3/4, Mistral,
Gemma 2, and any HuggingFace Jinja2 template), conversation management
with context shifting, and token counting.

=head2 Synopsis

=begin code :lang<raku>
use LLM::Chat::Backend::KoboldCpp;
use LLM::Chat::Template::ChatML;
use LLM::Chat::Conversation;

my $backend = LLM::Chat::Backend::KoboldCpp.new(
    api_url  => 'http://localhost:5001/v1',
    template => LLM::Chat::Template::ChatML.new,
);

my $conv = LLM::Chat::Conversation.new;
$conv.add-message('user', 'Hello!');

my $response = $backend.text-completion($conv.messages);
=end code

=head2 Templates

=head3 Built-in Templates

=begin code :lang<raku>
use LLM::Chat::Template::ChatML;
use LLM::Chat::Template::Llama3;
use LLM::Chat::Template::Llama4;
use LLM::Chat::Template::MistralV7;
use LLM::Chat::Template::Gemma2;

my $template = LLM::Chat::Template::ChatML.new;
=end code

=head3 Jinja2 Templates (HuggingFace)

Load any HuggingFace C<chat_template> directly from a C<tokenizer_config.json>:

=begin code :lang<raku>
use LLM::Chat::Template::Jinja2;

# From tokenizer_config.json
my $json = 'tokenizer_config.json'.IO.slurp;
my $template = LLM::Chat::Template::Jinja2.from-tokenizer-config($json);

# Or provide the template string directly
my $template = LLM::Chat::Template::Jinja2.new(
    template  => $jinja2-string,
    bos-token => '<s>',
    eos-token => '</s>',
);
=end code

The Jinja2 template support is powered by
L<Template::Jinja2|https://raku.land/zef:apogee/Template::Jinja2>,
a complete Jinja2 engine for Raku with byte-identical output to Python Jinja2.

=head2 Backends

=head3 KoboldCpp

=begin code :lang<raku>
use LLM::Chat::Backend::KoboldCpp;

my $backend = LLM::Chat::Backend::KoboldCpp.new(
    api_url   => 'http://localhost:5001/v1',
    template  => $template,  # for text completions
    max_tokens => 200,
);
=end code

=head3 OpenAI-compatible

Any OpenAI-compatible API (vLLM, Ollama, etc.):

=begin code :lang<raku>
use LLM::Chat::Backend::OpenAICommon;

my $backend = LLM::Chat::Backend::OpenAICommon.new(
    api_url => 'http://localhost:8000/v1',
    model   => 'my-model',
);
=end code

=head3 Mock (for tests)

Canned-response backend for unit and integration tests. Returns
pre-configured responses in order, records every call for assertions,
and can be scripted to fail on specific calls to exercise retry /
fallback paths in downstream consumers.

=begin code :lang<raku>
use LLM::Chat::Backend::Mock;
use LLM::Chat::Backend::Settings;

my $mock = LLM::Chat::Backend::Mock.new(
    settings  => LLM::Chat::Backend::Settings.new,
    responses => ['first', 'second', 'third'],
    # Optional: script per-call failures by index. Returning a defined
    # hash fails that call; returning Nil proceeds normally.
    error-producer => -> $i {
        when $i == 0 { { class => 'http', status => 503,
                         message => 'bad gateway' } }
        default      { Nil }
    },
    # Optional: script a truncated-but-successful call by index. The
    # call still succeeds with its canned body; the Response just
    # carries finish-reason 'length', the way a real blocking
    # completion reports an exhausted token budget.
    finish-reason-producer => -> $i { $i == 0 ?? 'length' !! Str },
);

my $resp = $mock.chat-completion(@messages);
# $mock.recorded-calls[0]<messages>, <response>, <error>, <call-index>, ...
# $mock.call-index — monotonic counter, bumped on every call
=end code

See L<LLM::Chat::Backend::Mock|../lib/LLM/Chat/Backend/Mock.rakumod>
for the full attribute list and recording contract.

=head2 Response

Every completion method returns an C<LLM::Chat::Backend::Response>
(or C<::Stream> for streaming calls). Callers poll C<.is-done>, read
C<.msg> on success, and inspect C<.err> on failure.

Responses also carry structured error metadata on the failure path
so consumers can classify errors without regex-parsing raw messages:

=begin code :lang<raku>

until $resp.is-done { sleep 0.01 }

if $resp.is-success {
    say $resp.msg;
}
else {
    say "failed: {$resp.err}";
    say "  class:  {$resp.error-class  // '(none)'}";   # 'http' / 'timeout' /
                                                        # 'connection' /
                                                        # 'response' / 'unknown'
    say "  status: {$resp.error-status // '(none)'}";   # HTTP code when
                                                        # error-class eq 'http'
}

=end code

C<error-class> values:

=item C<'http'> — HTTP-level error. C<error-status> is populated with the code.
=item C<'timeout'> — request exceeded the client-side deadline.
=item C<'connection'> — network unreachable / connection reset / DNS failure.
=item C<'response'> — HTTP succeeded but the body was malformed, empty, or finished with a C<'length'> / C<'content_filter'> quit.
=item C<'unknown'> — catch-all for exceptions that don't classify.

L<LLM::Data::Inference::Task|https://raku.land/zef:apogee/LLM::Data::Inference>
reads these fields to decide between abort / retry-same / advance in
its model-fallback policy — consumers that want the same policy
without depending on that module can implement it against the
C<Response.error-class> / C<.error-status> pair directly.

Provider-reported usage is also available on the Response when the
backend emits it:

=begin code :lang<raku>

$resp.prompt-tokens;       # Int, undefined on backends that don't emit usage
$resp.completion-tokens;   # Int
$resp.total-tokens;        # Int
$resp.cost;                # Num (credits)
$resp.model-used;          # Str, provider-reported routed model
$resp.provider-id;         # Str, provider-assigned request id
$resp.finish-reason;       # Str ('stop' / 'length' / 'content_filter' / ...)

=end code

=head2 Tool Loop

C<LLM::Chat::ToolLoop> wraps a streaming backend and runs OpenAI-style
tool-call rounds until the model produces a final answer or the safety
limits are reached. It is backend-agnostic: pass tools in OpenAI format
and an executor callback that returns C<role => "tool"> result hashes.

=begin code :lang<raku>

use LLM::Chat::ToolLoop;

my @tools = $mcp-server.tools-for-llm;

my $loop = LLM::Chat::ToolLoop.new(
    backend => $backend,
    tools => @tools,
    execute-tools => -> @calls {
        $mcp-server.execute-tool-calls(@calls)
    },
    # Defaults: 4 tool rounds, 12 total calls, 2 identical calls.
);

my $resp = $loop.chat-completion-stream(@messages);
until $resp.is-done { sleep 0.01 }
say $resp.msg if $resp.is-success;

=end code

For KoboldCpp, use the OpenAI-compatible chat endpoint
C<http://host:5001/v1>. Streaming tool calls arrive as
C<delta.tool_calls> chunks and are exposed through C<.tool-calls> on
the response before the loop executes them.

=head2 Retry Policy

C<LLM::Chat::Retry> is the shared retry/fallback policy: five pure subs
that classify a failure, decide how long to wait, sleep without
ignoring a cancel, and build the attempt/telemetry records. Pair it
with C<LLM::Chat::Retry::Exceptions> for the typed failures a chain
throws when it gives up.

=begin code :lang<raku>

use LLM::Chat::Retry;
use LLM::Chat::Retry::Exceptions;

my @attempts;
my Int $retries-left = 2;

given classify-error(
    error-class  => $resp.error-class,
    error-status => $resp.error-status,
) {
    when 'abort'      { die "config/account error: {$resp.err}" }
    when 'retry-same' {
        my Num $wait = retry-backoff(3 - $retries-left);   # 1s, 2s, 4s ...
        $retries-left--;
        # Returns False the moment &cancelled flips — a cancelled run
        # never sits out a 16-second backoff.
        last unless sleep-with-cancel($wait, :&cancelled);
    }
    default { }   # 'advance' — try the next backend at once
}

@attempts.push: attempt-record(
    backend-index => 0, model => $backend.model, error => "{$resp.err}",
);

X::LLM::Chat::Retry::Exhausted.new(
    :@attempts, summary => 'all backends exhausted',
).throw;

=end code

Buckets: C<'abort'> (HTTP 400/401/402/403/404 — config, account or
access errors that repeat identically), C<'retry-same'> (5xx,
connection drops, anything unclassifiable — usually one upstream
provider having a bad minute), C<'advance'> (timeout, 429, malformed
response, parser failure — model-specific pathologies).

The exception types are C<Exhausted>, its subclasses C<Truncated>
(cut off by C<max_tokens>) and C<TimedOut> (missed a deadline), and
C<Cancelled> — which is deliberately B<not> an C<Exhausted>, so a user
pressing Ctrl-C never lands in a dead-letter queue. C<Truncated>
declares C<item-retryable> C<False>, the duck-typed advice an
orchestration layer reads as C<< $ex.?item-retryable // True >> before
re-running failed work.

L<LLM::Data::Inference::Task|https://raku.land/zef:apogee/LLM::Data::Inference>
runs on this module, and its own exception types subclass these — so
handlers written against either hierarchy match.

=head2 Conversation Management

=begin code :lang<raku>
use LLM::Chat::Conversation;

my $conv = LLM::Chat::Conversation.new;
$conv.add-message('system', 'You are helpful.');
$conv.add-message('user', 'Hello!');
$conv.add-message('assistant', 'Hi there!');

# Access messages
say $conv.messages;
=end code

=head2 Token Counting

=begin code :lang<raku>
use LLM::Chat::TokenCounter;

my $counter = LLM::Chat::TokenCounter.new(
    tokenizer-path => 'path/to/tokenizer.json',
    template       => $template,
);

my $count = $counter.count-messages(@messages);
=end code

=head2 Dependencies

=item L<Cro::HTTP> — HTTP client for API calls
=item L<Template::Jinja2|https://raku.land/zef:apogee/Template::Jinja2> — Jinja2 template engine
=item L<Tokenizers> — HuggingFace tokenizers via Rust FFI
=item L<JSON::Fast> — JSON parsing

=head2 Author

Matt Doughty

=head2 License

Artistic-2.0

=end pod
