=begin pod

=head1 NAME

LLM::Data::Inference - Structured LLM task layer with model-fallback, JSON parsing, and query-based routing

=head1 SYNOPSIS

=begin code :lang<raku>

use LLM::Data::Inference;

# Simple blocking LLM call (single backend — legacy shape)
my $task = LLM::Data::Inference::Task.new(
    :backend($my-backend),
    :system-prompt('You are a helpful assistant.'),
    :user-prompt('What is 2+2?'),
);
say $task.execute;  # "4"

# Model-fallback chain: try backends in order, advance to the next
# on model-specific failures (timeout, malformed output, 429, etc.),
# retry the head on transient errors (connection drop, 5xx), abort
# immediately on config errors (401 / 402 / 403).
my $task = LLM::Data::Inference::Task.new(
    :backends($primary, $secondary, $cheap-fallback),
    :user-prompt('Write a scene.'),
);
say $task.execute;  # serves from whichever backend produced a good response

# JSON output with schema checks — parser failures advance through
# the chain rather than retrying the same model on malformed JSON.
my $json-task = LLM::Data::Inference::JSONTask.new(
    :backends($primary, $fallback),
    :user-prompt('Return a JSON object with name and age.'),
    :required-keys('name', 'age'),
);
my %result = $json-task.execute;
say %result<name>;  # "Alice"

# Template-based prompts
my $pb = LLM::Data::Inference::PromptBuilder.new(
    :template('Write a {{genre}} story about {{topic}}.')
);
say $pb.render(%(:genre('fantasy'), :topic('dragons')));

# Query-based routing (orthogonal to the fallback chain — routers
# pick a backend, Tasks run fallbacks against that backend).
my $router = LLM::Data::Inference::Router.new(:default-backend($cloud-api));
$router.add-route('confidential OR restricted', $local-model);
$router.add-route('genre:technical', $reasoning-model);

my $backend = $router.select-backend($tags, $doc-id);

=end code

=head1 DESCRIPTION

LLM::Data::Inference provides a structured task layer on top of LLM::Chat for
use in data generation pipelines. It wraps the async LLM::Chat API into
blocking calls with a three-bucket retry + model-fallback policy, JSON
extraction, and content-based model routing.

=head2 LLM::Data::Inference::Task

Blocking LLM call with a model-fallback chain. Accepts either the
legacy single C<:$backend> or an ordered C<:@backends> list; internally
both are stored as a chain, and a single-backend chain behaves exactly
like the pre-fallback Task on retry-same errors.

=begin code :lang<raku>

my $task = LLM::Data::Inference::Task.new(
    :backends($primary, $fallback),   # OR :backend($single)
    :system-prompt('Be helpful.'),    # Optional system prompt
    :user-prompt('Hello'),            # User message (required)
    :max-retries(3),                  # Per-backend same-model retry
                                      # budget for transient errors
                                      # (see "Fallback policy" below)
    :timeout(120e0),                  # Seconds per HTTP round-trip
    :parser(-> $text { ... }),        # Optional parser; die to flag
                                      # malformed output (advances
                                      # to the next backend)
    :on-call-complete(-> %p { ... }), # Optional per-call telemetry hook
);

my $result = $task.execute;  # blocks until a backend returns a good
                             # response, or dies if every backend
                             # in the chain fails

=end code

=head3 Fallback policy

Failures classify into three buckets:

=begin item
B<abort> — HTTP 400 / 401 / 402 / 403 / 404.

Config or account errors (bad API key, payment required, model not
found for your region, etc.). Iterating the chain would just produce
the same error on every backend, so the Task re-raises immediately
with context about which backend failed.
=end item

=begin item
B<retry-same> — connection drops, HTTP 5xx, or unclassifiable errors.

Likely transient — a specific OpenRouter upstream provider failed; a
retry often routes to a different one. The current backend gets up to
C<$.max-retries> total attempts (initial + C<max-retries - 1> retries)
with exponential backoff and jitter capped at 30 s. After the budget
is spent the Task advances to the next backend.
=end item

=begin item
B<advance> — timeout, 429, empty body, parser failure, streamed
finish-reason quits (finish_reason 'length' / 'content_filter'), other
4xx.

Model-specific pathology — a reasoning loop, sanitising rewrite,
malformed JSON, rate-limit on this model specifically. The current
model's retry budget is short-circuited and the Task moves straight to
the next backend. If the chain is exhausted, dies with an "all
backend(s) exhausted" summary that includes per-backend error info.
=end item

=head3 Truncated output (finish_reason 'length')

Only a I<streamed> C<'length'> is a transport-level failure. The
blocking path the Task uses reports budget exhaustion as an HTTP 200
B<success> carrying a partial body, so a truncated payload would sail
into C<&.parser>, fail there, and (with C<:parse-retries> set) be
re-rolled as the same over-budget request until the budget was gone.

C<:truncation-policy> intercepts it:

=item C<'fail'> — record the truncation and advance to the next backend at once, without spending a parse re-roll. C<max_tokens> lives in each backend's C<Settings>, so only a different backend can change it.
=item C<'accept'> — historical behaviour: the partial text is the result.
=item C<''> (default) — derive it: C<'fail'> when a parser is installed, C<'accept'> when not. So C<JSONTask> fails fast while prose Tasks are unchanged.

A chain that truncates throws
C<X::LLM::Data::Inference::Truncated>, a B<subclass> of
C<X::LLM::Data::Inference::Exhausted> — existing handlers and
C<:on-exhausted> DLQ sinks keep working untouched — whose summary
names the levers (raise C<max_tokens>, add a larger-budget fallback
backend, or shrink the request).

=begin code :lang<raku>

my $task = LLM::Data::Inference::JSONTask.new(
    :backends($small-budget, $big-budget),
    :$user-prompt,
    :parse-retries(2),   # never burned on a truncation
);

=end code

=head3 Parser failures advance (behavioural change from pre-fallback)

When C<:&parser> is set, a thrown exception from inside the parser is
classified as an advance-class failure. The Task does NOT retry the
same backend on parser failure — in practice, a model that emits
malformed JSON once rarely recovers on a second attempt against the
same model, and a chain of C<[primary, fallback]> produces cleaner
recovery with lower latency.

Consumers on a single-backend Task that previously relied on parser
recovery via retry should either (a) add a fallback model to the
chain, (b) pass C<:backends($primary, $primary)> to preserve the old
"try the same model twice" shape on advance-class errors, or (c)
build a retry loop at the application layer.

=head3 Retry feedback (C<:retry-feedback>)

Callers who do opt into same-backend re-rolls with C<:parse-retries(N)>
get B<blind> re-rolls by default: identical prompt, identical model,
no word about what was wrong with the answer just rejected. That is
survivable for malformed JSON on a stochastic sampler and hopeless for
a I<semantic> rejection — a validator that reports "quote 2 does not
appear verbatim in the passage" reports exactly that on every re-roll,
so the item spends its whole budget failing identically.

C<:retry-feedback> (default C<False>) makes the re-roll informed: an
attempt that follows a parser or validator failure on the B<same>
backend carries one extra C<user> turn,

=begin code :lang<text>

Your previous response was rejected: <the parser's message>. Respond
again with the same JSON contract, corrected.

=end code

=begin code :lang<raku>

my $task = LLM::Data::Inference::JSONTask.new(
    :backends($primary, $fallback),
    :user-prompt('Extract the beat quotes verbatim.'),
    :validator(&quotes-appear-verbatim),
    :parse-retries(2),
    :retry-feedback,
);

=end code

The rules, each of them a decision:

=item B<Replace, never accumulate.> Every attempt re-derives its
message list from the pristine prompt, so re-roll 2 argues about
re-roll 2's rejection and not re-roll 1's. A growing transcript of
complaints would spend more and more prompt on staler and staler
objections, and models fixate on the first item in such a list.

=item B<Parse and validation failures only.> Truncation is a budget
fact rather than a correctable mistake; a network retry-same produced
no output to critique; and the pending reason is held B<per backend>,
so a backend advance starts the next model on the pristine prompt. A
model is only ever told about output it actually produced.

=item B<No echo of the rejected text.> It is already the model's own
last turn in context; the validator's diagnostic is the part the model
cannot infer. Every attempt record still carries the raw text for the
caller (see C<X::LLM::Data::Inference::Exhausted>).

The canned wording names a B<JSON contract> — correct for C<JSONTask>
and hand-rolled JSON parsers, wrong for prose Tasks, which is exactly
why the default is off. Telemetry is unchanged: the extra turn is
visible only as a larger C<prompt-tokens> figure (roughly a hundred
tokens) on the retry row.

=head3 Telemetry

C<:&on-call-complete> fires once per HTTP round-trip with a hash:

=begin code :lang<raku>

%(
    attempt       => 1,         # monotonic across the execute call
    backend-index => 0,         # 0-based position within @.backends
    model-name    => 'z-ai/glm-5.1',
    latency-ms    => 1234,
    success       => True,
    stage         => 'network',
    error         => Str,       # present on failure
    error-class   => Str,       # 'http' / 'timeout' / 'connection' /
                                # 'response' / 'unknown' (on failure)
    error-status  => Int,       # HTTP code (when error-class eq 'http')
    # OAI-spec usage — presence-gated, lifted off any
    # LLM::Chat::Backend::Response when the provider supplies them:
    prompt-tokens, completion-tokens, total-tokens,
    model-used, finish-reason,
    # Provider-specific extras — presence-gated, lifted off Response
    # subclasses that expose them (currently
    # LLM::Chat::Backend::Response::OpenRouter):
    cost, generation-id, provider-name, is-byok,
)

=end code

=head3 classify-error — inspect the policy

The public C<classify-error> method maps an error shape to the bucket
name for consumers that want to implement the same policy outside
the Task:

=begin code :lang<raku>

my $bucket = $task.classify-error(
    error-class  => 'http',
    error-status => 401,
);
# returns 'abort'

$task.classify-error(error-class => 'timeout');       # 'advance'
$task.classify-error(error-class => 'connection');    # 'retry-same'
$task.classify-error(:parser-failed);                 # 'advance'

=end code

Since 0.9.0 the rule table itself lives in
L<LLM::Chat::Retry|https://raku.land/zef:apogee/LLM::Chat>, together
with the backoff, the cancel-aware sleep and the attempt/telemetry
record shapes — the policy is shared with other executors rather than
copied. This method is unchanged and delegates to it, so callers that
already hold a Task need not reach for another distribution.

=head3 Exception types

The four exceptions in C<LLM::Data::Inference::Exceptions> are
subclasses of the shared C<X::LLM::Chat::Retry::*> types as of 0.9.0,
so B<both> hierarchies match the same throw:

=begin code :lang<raku>

# Existing handler — unchanged, still catches everything it did.
CATCH { when X::LLM::Data::Inference::Exhausted { ... } }

# New generic handler — catches Task throws AND any other executor's.
CATCH { when X::LLM::Chat::Retry::Exhausted { ... } }

=end code

Attributes, C<.message> strings, the attempt-record shape and the
C<item-retryable> advice are all unchanged.

=head2 LLM::Data::Inference::JSONTask

JSON extraction from LLM responses with key validation and optional
custom validator. Handles LLMs that wrap JSON in prose by extracting
the outermost C<{ }> or C<[ ]>. Accepts C<:$backend> or C<:@backends>
and threads either through to the inner C<Task> unchanged — all
fallback semantics come from the Task layer.

=begin code :lang<raku>

my $task = LLM::Data::Inference::JSONTask.new(
    :backends($primary, $fallback),
    :user-prompt('Give me a character card as JSON.'),
    :required-keys('name', 'description'),    # Missing key → advance
    :validator(-> %h { %h<name>.chars > 0 }), # False return → advance
    :max-retries(3),                          # Same-model retry budget
                                              # for transient errors
);

my %character = $task.execute;

=end code

=head2 LLM::Data::Inference::Router

Query-based routing using Roaring::Tags. Each route is a tag query
string paired with a backend. Routes are evaluated in order — first
match wins. Orthogonal to the Task-level fallback chain: a Router
selects which backend (or chain) to hand to the Task, and the Task
handles retries / fallbacks on top.

=begin code :lang<raku>

my $router = LLM::Data::Inference::Router.new(
    :default-backend($cloud-api),
);

$router.add-route('confidential', $local-model);
$router.add-route('confidential, sensitive', $air-gapped-model);
$router.add-route('genre:technical', $reasoning-model);

my $backend = $router.select-backend($tags, $doc-id);

=end code

=head2 LLM::Data::Inference::PromptBuilder

Mustache-style template rendering with C<{{variable}}> substitution.

=begin code :lang<raku>

my $pb = LLM::Data::Inference::PromptBuilder.new(
    :template('Write a {{length}} word {{genre}} story about {{topic}}.')
);
my $prompt = $pb.render(%(:length('500'), :genre('sci-fi'), :topic('AI')));

=end code

Dies if a C<{{variable}}> has no matching key in the vars hash.

=head1 AUTHOR

Matt Doughty <matt@apogee.guru>

=head1 COPYRIGHT AND LICENSE

Copyright 2026 Matt Doughty

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.

=end pod
