Revision history for LLM::Data::Inference

0.9.0  2026-08-09T20:28:57+01:00
    - [2026-08-09] The retry/fallback policy this distribution
      pioneered now lives in C<LLM::Chat::Retry> (LLM::Chat 0.8.0) so a
      second executor can share it instead of copying it, and Task runs
      on the shared version. Nothing about its behaviour changed: the
      bucket table, the backoff curve (C<2 ** (n - 1)> plus 0..0.5 s of
      jitter, capped at 30 s), the chunked cancel-aware sleep, the
      attempt-record shape (C<raw-text> key ABSENT rather than
      undefined when there is no body) and the telemetry payload's
      presence-gating are the same code, moved. The public
      C<Task.classify-error> method is B<kept> and delegates — a caller
      holding a Task should not have to reach for another distribution
      to ask "what would you do with this error?". Every one of this
      distribution's thirteen pre-existing test files passes unedited,
      which is the migration's acceptance criterion.
    - [2026-08-09] C<X::LLM::Data::Inference::Exhausted>, C<Truncated>,
      C<TimedOut> and C<Cancelled> are now B<subclasses> of the shared
      C<X::LLM::Chat::Retry::*> types, so BOTH hierarchies match the
      same throw: existing C<when X::LLM::Data::Inference::Exhausted>
      handlers keep firing byte-for-byte identically, while an app that
      runs a Task alongside another executor can write one generic
      C<when X::LLM::Chat::Retry::Exhausted> arm instead of two.
      C<Truncated> still declares C<item-retryable> C<False> (inherited
      from its shared twin), C<TimedOut> and C<Exhausted> still
      abstain, C<Cancelled> is still deliberately NOT an exhaustion in
      either hierarchy, and every C<.message> string — including
      "LLM::Data::Inference::Task: cancelled by caller after N recorded
      attempt(s)" — is unchanged. The C<item-retryable> advice
      contract's canonical documentation moved to
      C<LLM::Chat::Retry::Exceptions> along with the types; the section
      here now points at it. New C<t/14-shared-exceptions.rakutest>
      pins the whole compatibility promise, against constructed
      exceptions and against exceptions a real Task throws.
    - [2026-07-29] Typed deadline exhaustion. A chain that ran out of
      time died as a bare C<X::LLM::Data::Inference::Exhausted> whose
      summary named no lever at all: a review step that hit its 300 s
      deadline three times in a row reported only "all 1 backend(s)
      exhausted" plus the raw per-attempt error, so nothing told the
      operator that the fix was a bigger deadline (or a faster
      backend) rather than a better model. Task now throws the new
      C<X::LLM::Data::Inference::TimedOut> when at least one attempt
      in the exhausted chain died on a response deadline — its own
      C<$.timeout> firing mid-poll, or a backend reporting error
      class 'timeout'. Like C<Truncated> it is a SUBCLASS of
      C<Exhausted>, so DLQ routing, C<:on-exhausted> hooks and every
      existing C<when Exhausted> handler keep working unchanged,
      while a caller that wants to distinguish "raise the deadline"
      from "this model cannot do the task" can match it first. The
      summary gains a lever line quoting the deadline that was in
      force ("At least one attempt hit the 300-second response
      deadline before completing: raise the task timeout, use a
      faster backend, or shrink the response."), and a chain that
      truncated AND timed out carries BOTH lever lines — they name
      different fixes. Precedence for the thrown TYPE is
      C<Truncated> > C<TimedOut> > C<Exhausted>: truncation is the
      more specific diagnosis and the deterministic one. Summaries
      of chains that hit neither pathology are byte-identical to
      before.
    - [2026-07-29] C<X::LLM::Data::Inference::Truncated> now declares
      C<method item-retryable(--> Bool:D) { False }>. A truncation is
      deterministic — C<max_tokens> is per-backend C<Settings> state,
      so an orchestration layer that blindly re-runs the failed item
      re-issues the identical request and gets cut off in exactly the
      same place, burning the whole item budget on the way to the same
      dead-letter record. The method is duck-typed ADVICE in both
      directions: nothing here reads it, consumers probe with C<.?>
      and default to True when it is absent, and neither side imports
      the other's types — so every other exception (including
      C<TimedOut> and plain C<Exhausted>, which deliberately abstain)
      keeps its full attempt budget. See the C<item-retryable> section
      of the L<LLM::Data::Inference::Exceptions> Pod.

0.7.0  2026-07-27T15:25:50+01:00
    - [2026-07-25] Cancellation and deadline aborts now route through
      C<Backend.cancel> instead of C<Response.cancel>. Closing the
      Response only quits the local supplier; the upstream stop signal
      lives in the backend override, so against KoboldCpp a cancelled
      Task now fires C<POST /api/extra/abort> and the server actually
      stops generating. Previously every Task-mediated job (reconcile,
      narrative-state, downstream-projection, image prompt-gen,
      scene-shakeup, relationship-mode in App::Cantina) looked
      cancelled to the caller while the backend drained the full
      completion. Backends with no upstream abort endpoint degrade to
      exactly the previous local close, and a backend whose C<cancel>
      throws falls back to the local close so the poll loop can never
      spin on a live response. Both the C<is-cancelled> branch and the
      C<$.timeout> branch are covered.
    - [2026-07-27] Typed truncation. A blocking completion cut off by
      C<max_tokens> comes back as an HTTP 200 SUCCESS with a partial
      body and C<finish_reason> 'length' — the "advance on
      finish-reason quit" rule in the docs was only ever true of the
      STREAMING path. Task therefore handed half-finished output to
      the parser, watched it fail, and (with C<:parse-retries> set)
      re-rolled the identical over-budget request until the budget
      was gone. Task and JSONTask now take
      C<:truncation-policy('fail' | 'accept' | '')>: 'fail' records
      the truncation and advances to the next backend at once WITHOUT
      consuming a parse re-roll (C<max_tokens> is per-backend
      C<Settings>, so only a different backend can change it);
      'accept' is the historical behaviour; the default C<''> derives
      'fail' when a parser is installed and 'accept' when one is not,
      so JSON tasks fail fast while prose tasks (continuations, style
      passes) keep their partial output bit-for-bit as before. The
      check runs BEFORE the empty-body rule, so a reasoning model that
      burns its whole budget inside C<< <think> >> is diagnosed as
      truncation rather than "empty response body". A chain that
      truncates throws the new
      C<X::LLM::Data::Inference::Truncated> — a SUBCLASS of
      C<Exhausted>, so DLQ routing, C<:on-exhausted> hooks and every
      existing C<when Exhausted> handler keep working unchanged — and
      its summary names the levers (raise C<max_tokens>, add a
      larger-budget fallback, or shrink the request). Failed responses
      also keep any text they emitted before quitting, so partial
      streamed output is capturable instead of dropped. Requires
      LLM::Chat 0.7.1 for the Mock C<finish-reason-producer> knob the
      tests drive this with.
    - [2026-07-27] Add C<:retry-feedback> to Task and JSONTask (default
      False). A parse re-roll used to be blind — identical prompt,
      identical model, no word about what was wrong with the answer
      just rejected. That is survivable for malformed JSON on a
      stochastic sampler, but hopeless for a SEMANTIC rejection: a
      verbatim-quote validator that says "quote 2 does not appear in
      the passage" says exactly the same thing on every re-roll, so
      the item burned its whole budget failing identically and then
      dead-lettered something the model could have fixed on the second
      try. With the flag set, an attempt that follows a parser or
      validator failure on the SAME backend carries one extra C<user>
      turn: "Your previous response was rejected: <the parser's
      message>. Respond again with the same JSON contract,
      corrected." Each re-roll REPLACES the previous feedback instead
      of accumulating a transcript of complaints (the message list is
      re-derived from the pristine prompt every attempt), and the
      pending reason lives per backend, so truncation advances,
      backend advances and network retry-sames all send the prompt
      they always did — a model is only ever told about output it
      actually produced. The rejected text itself is deliberately not
      echoed back (it is already the model's own last turn, and this
      ecosystem's validator diagnostics are the part it cannot infer);
      the canned wording names a JSON contract, which is why prose
      Tasks are expected to leave the flag alone. Telemetry is
      unchanged — the extra turn shows up only as a larger
      C<prompt-tokens> figure (~100 tokens) on the retry row.

0.5.1  2026-07-21T03:15:12+01:00
    - [2026-07-19] Add C<:on-exhausted> to Task: an optional hook that
      fires exactly once, on the execute thread, immediately before
      C<X::LLM::Data::Inference::Exhausted> is thrown (never on
      success, never on the Cancelled path). The payload carries the
      same C<attempts> records and C<summary> string the thrown
      exception carries, plus C<stage => 'exhausted'> and a
      C<backends> count — letting callers (e.g. a pipeline DLQ) get
      the rich attempts/raw-text payload without re-parsing the
      caught exception. Shielded exactly like C<on-call-complete>: a
      throwing hook is caught and noted, never suppressing or
      altering the Exhausted throw. JSONTask now forwards
      C<:on-exhausted> to its inner Task, same as its other hooks.
    - [2026-07-04] Cooperative cancellation: Task and JSONTask accept
      C<:is-cancelled>, polled before every round-trip, on each
      in-flight completion poll (~10 ms cadence — the pending response
      is aborted via Response.cancel, same as the timeout path), and
      between chunked backoff sleeps. When the hook reports True the
      Task throws the new typed X::LLM::Data::Inference::Cancelled,
      which carries the same C<.attempts> records as Exhausted but is
      deliberately NOT its subclass — a user cancel must never be
      swallowed by exhaustion handlers. A cancelled Task no longer
      burns the remaining retry chain (previously up to
      max-retries x backends round-trips plus full backoff sleeps
      after the caller had already walked away).
    - [2026-07-03] Add C<:parse-retries> to Task and JSONTask: an
      opt-in same-backend re-roll budget consumed only by parser
      failures (default 0 keeps the historical advance-immediately
      behaviour). Interactive callers on stochastic samplers re-roll
      malformed JSON instead of exhausting the chain on one bad shot.
    - [2026-07-03] Chain exhaustion now throws the typed
      X::LLM::Data::Inference::Exhausted (new
      LLM::Data::Inference::Exceptions module) whose C<.attempts>
      records every failure — including the RAW model output for each
      failed parse — while C<.message> keeps the exact historical
      summary string for back-compat.
    - [2026-07-03] Harden JSONTask extraction: strip <think> blocks and
      markdown code-fence markers, then pick the longest complete
      top-level JSON structure via a string-aware balanced-bracket
      scan that actually parses (reasoning preambles with stray
      braces, plan-out-loud draft objects, and fenced answers all
      extract correctly). Hopeless responses fall back to the legacy
      slice so established error messages are preserved.

0.3.0  2026-04-29T23:49:54+01:00
    - Bump Github Actions to use node 24+
    - BREAKING (telemetry payload): the on-call-complete hook payload
      key C<:provider-id> is removed and replaced with C<:generation-id>
      (clearer name — the value was always OpenRouter's gen-XXXX id,
      not a provider name). New keys C<:provider-name> and C<:is-byok>
      are added alongside, lifted off
      C<LLM::Chat::Backend::Response::OpenRouter> when present. Reads
      now use C<.?> so non-OR Response subclasses (which have no such
      accessors) don't blow up — they simply omit the keys.
    - Sink contract is otherwise unchanged: every key remains
      presence-gated (absent when the provider didn't supply it).
      Migrating sinks: rename C<%payload<provider-id>> →
      C<%payload<generation-id>>; optionally consume the new
      C<provider-name> / C<is-byok> keys for richer routing visibility.

0.2.0  2026-04-23T15:57:28+01:00
    - Added auth specifiers to META6.json dependencies
    - Task + JSONTask accept an ordered :@backends fallback chain in
      addition to the legacy single :$backend. The retry loop
      classifies failures into three buckets: abort (HTTP 400 / 401 /
      402 / 403 / 404 — config/account errors where retrying any
      model in the chain won't help), retry-same (connection errors /
      5xx / unclassifiable — likely transient), and advance (timeout /
      429 / empty body / parser failure / other 4xx / content-filter-
      style finish-reason quits — model-specific pathology). Each
      backend gets up to $.max-retries HTTP attempts (initial +
      retries-same, exponential backoff with jitter capped at 30 s);
      an advance-class error on any attempt short-circuits the budget
      and moves on. Abort-class errors re-raise immediately without
      trying the rest of the chain. See classify-error and the module
      Pod for the full rule table.
    - Task.classify-error(:$error-class, :$error-status,
      :$parser-failed) exposed as a public method for testability and
      for consumers that want to implement the same policy outside
      the retry loop.
    - Telemetry hook payload adds :backend-index, :model-name,
      :error-class, :error-status, so sinks can identify which model
      in the chain served each call and what failed. Existing keys
      (:attempt, :success, :error, :latency-ms, :prompt-tokens,
      :completion-tokens, :total-tokens, :cost, :model-used,
      :provider-id, :finish-reason, :stage) are unchanged.
    - Single-:backend constructor shape is preserved; a one-element
      chain behaves exactly like the pre-fallback Task on the
      retry-same path (connection / 5xx still get $.max-retries
      same-model attempts with exponential backoff).

    BACKWARD COMPATIBILITY — one behavioural break:

    Advance-class errors (timeout / 429 / parser failure / empty body
    / content-filter-style finish quits / other 4xx) used to retry
    the same model up to $.max-retries times. They now advance to
    the next backend in the chain, or die immediately with
    "all backend(s) exhausted" if the Task only has one. Retry-same
    errors (connection drop / 5xx) are unchanged — still retried on
    the same backend up to $.max-retries times before advancing.

    Consequence for single-backend callers: a Task that previously
    survived a stochastic parser failure via retry now dies on the
    first parse error. Three mitigations:
      - Preferred: pass :@backends with a fallback model. A chain
        of [primary, primary] is also legal and preserves the exact
        old "try the same model twice" behaviour on advance-class
        failures while keeping retry-same semantics intact.
      - Build the retry loop at the application layer if the same
        model genuinely recovers for your workload.
      - Accept the fail — in practice the old behaviour rarely
        recovered on parser failures (malformed JSON tended to
        repeat), which is the motivation for the change.

0.1.3  2026-04-07T20:03:42+01:00
    - Updated readme examples to use generic routing terms

0.1.2  2026-04-07T20:00:40+01:00
    - Removed Windows from CI (upstream Digest::SHA256::Native does not build on Windows)

0.1.1  2026-04-07T19:52:19+01:00
    - Added Windows CI support via MSVC

0.1.0  2026-04-07T18:52:19+01:00
    - Initial release
    - Task: blocking LLM calls with configurable parser and retry
    - JSONTask: JSON extraction from LLM responses with key validation
    - Router: query-based routing using Roaring::Tags
    - PromptBuilder: mustache-style template rendering
