Revision history for LLM::Data::Inference

0.7.0  2026-07-27T15:25:50+01:00
    - [2026-07-25] Cancellation and deadline aborts now route through
      C<Backend.cancel> instead of C<Response.cancel>. Closing the
      Response only quits the local supplier; the upstream stop signal
      lives in the backend override, so against KoboldCpp a cancelled
      Task now fires C<POST /api/extra/abort> and the server actually
      stops generating. Previously every Task-mediated job (reconcile,
      narrative-state, downstream-projection, image prompt-gen,
      scene-shakeup, relationship-mode in App::Cantina) looked
      cancelled to the caller while the backend drained the full
      completion. Backends with no upstream abort endpoint degrade to
      exactly the previous local close, and a backend whose C<cancel>
      throws falls back to the local close so the poll loop can never
      spin on a live response. Both the C<is-cancelled> branch and the
      C<$.timeout> branch are covered.
    - [2026-07-27] Typed truncation. A blocking completion cut off by
      C<max_tokens> comes back as an HTTP 200 SUCCESS with a partial
      body and C<finish_reason> 'length' — the "advance on
      finish-reason quit" rule in the docs was only ever true of the
      STREAMING path. Task therefore handed half-finished output to
      the parser, watched it fail, and (with C<:parse-retries> set)
      re-rolled the identical over-budget request until the budget
      was gone. Task and JSONTask now take
      C<:truncation-policy('fail' | 'accept' | '')>: 'fail' records
      the truncation and advances to the next backend at once WITHOUT
      consuming a parse re-roll (C<max_tokens> is per-backend
      C<Settings>, so only a different backend can change it);
      'accept' is the historical behaviour; the default C<''> derives
      'fail' when a parser is installed and 'accept' when one is not,
      so JSON tasks fail fast while prose tasks (continuations, style
      passes) keep their partial output bit-for-bit as before. The
      check runs BEFORE the empty-body rule, so a reasoning model that
      burns its whole budget inside C<< <think> >> is diagnosed as
      truncation rather than "empty response body". A chain that
      truncates throws the new
      C<X::LLM::Data::Inference::Truncated> — a SUBCLASS of
      C<Exhausted>, so DLQ routing, C<:on-exhausted> hooks and every
      existing C<when Exhausted> handler keep working unchanged — and
      its summary names the levers (raise C<max_tokens>, add a
      larger-budget fallback, or shrink the request). Failed responses
      also keep any text they emitted before quitting, so partial
      streamed output is capturable instead of dropped. Requires
      LLM::Chat 0.7.1 for the Mock C<finish-reason-producer> knob the
      tests drive this with.
    - [2026-07-27] Add C<:retry-feedback> to Task and JSONTask (default
      False). A parse re-roll used to be blind — identical prompt,
      identical model, no word about what was wrong with the answer
      just rejected. That is survivable for malformed JSON on a
      stochastic sampler, but hopeless for a SEMANTIC rejection: a
      verbatim-quote validator that says "quote 2 does not appear in
      the passage" says exactly the same thing on every re-roll, so
      the item burned its whole budget failing identically and then
      dead-lettered something the model could have fixed on the second
      try. With the flag set, an attempt that follows a parser or
      validator failure on the SAME backend carries one extra C<user>
      turn: "Your previous response was rejected: <the parser's
      message>. Respond again with the same JSON contract,
      corrected." Each re-roll REPLACES the previous feedback instead
      of accumulating a transcript of complaints (the message list is
      re-derived from the pristine prompt every attempt), and the
      pending reason lives per backend, so truncation advances,
      backend advances and network retry-sames all send the prompt
      they always did — a model is only ever told about output it
      actually produced. The rejected text itself is deliberately not
      echoed back (it is already the model's own last turn, and this
      ecosystem's validator diagnostics are the part it cannot infer);
      the canned wording names a JSON contract, which is why prose
      Tasks are expected to leave the flag alone. Telemetry is
      unchanged — the extra turn shows up only as a larger
      C<prompt-tokens> figure (~100 tokens) on the retry row.

0.5.1  2026-07-21T03:15:12+01:00
    - [2026-07-19] Add C<:on-exhausted> to Task: an optional hook that
      fires exactly once, on the execute thread, immediately before
      C<X::LLM::Data::Inference::Exhausted> is thrown (never on
      success, never on the Cancelled path). The payload carries the
      same C<attempts> records and C<summary> string the thrown
      exception carries, plus C<stage => 'exhausted'> and a
      C<backends> count — letting callers (e.g. a pipeline DLQ) get
      the rich attempts/raw-text payload without re-parsing the
      caught exception. Shielded exactly like C<on-call-complete>: a
      throwing hook is caught and noted, never suppressing or
      altering the Exhausted throw. JSONTask now forwards
      C<:on-exhausted> to its inner Task, same as its other hooks.
    - [2026-07-04] Cooperative cancellation: Task and JSONTask accept
      C<:is-cancelled>, polled before every round-trip, on each
      in-flight completion poll (~10 ms cadence — the pending response
      is aborted via Response.cancel, same as the timeout path), and
      between chunked backoff sleeps. When the hook reports True the
      Task throws the new typed X::LLM::Data::Inference::Cancelled,
      which carries the same C<.attempts> records as Exhausted but is
      deliberately NOT its subclass — a user cancel must never be
      swallowed by exhaustion handlers. A cancelled Task no longer
      burns the remaining retry chain (previously up to
      max-retries x backends round-trips plus full backoff sleeps
      after the caller had already walked away).
    - [2026-07-03] Add C<:parse-retries> to Task and JSONTask: an
      opt-in same-backend re-roll budget consumed only by parser
      failures (default 0 keeps the historical advance-immediately
      behaviour). Interactive callers on stochastic samplers re-roll
      malformed JSON instead of exhausting the chain on one bad shot.
    - [2026-07-03] Chain exhaustion now throws the typed
      X::LLM::Data::Inference::Exhausted (new
      LLM::Data::Inference::Exceptions module) whose C<.attempts>
      records every failure — including the RAW model output for each
      failed parse — while C<.message> keeps the exact historical
      summary string for back-compat.
    - [2026-07-03] Harden JSONTask extraction: strip <think> blocks and
      markdown code-fence markers, then pick the longest complete
      top-level JSON structure via a string-aware balanced-bracket
      scan that actually parses (reasoning preambles with stray
      braces, plan-out-loud draft objects, and fenced answers all
      extract correctly). Hopeless responses fall back to the legacy
      slice so established error messages are preserved.

0.3.0  2026-04-29T23:49:54+01:00
    - Bump Github Actions to use node 24+
    - BREAKING (telemetry payload): the on-call-complete hook payload
      key C<:provider-id> is removed and replaced with C<:generation-id>
      (clearer name — the value was always OpenRouter's gen-XXXX id,
      not a provider name). New keys C<:provider-name> and C<:is-byok>
      are added alongside, lifted off
      C<LLM::Chat::Backend::Response::OpenRouter> when present. Reads
      now use C<.?> so non-OR Response subclasses (which have no such
      accessors) don't blow up — they simply omit the keys.
    - Sink contract is otherwise unchanged: every key remains
      presence-gated (absent when the provider didn't supply it).
      Migrating sinks: rename C<%payload<provider-id>> →
      C<%payload<generation-id>>; optionally consume the new
      C<provider-name> / C<is-byok> keys for richer routing visibility.

0.2.0  2026-04-23T15:57:28+01:00
    - Added auth specifiers to META6.json dependencies
    - Task + JSONTask accept an ordered :@backends fallback chain in
      addition to the legacy single :$backend. The retry loop
      classifies failures into three buckets: abort (HTTP 400 / 401 /
      402 / 403 / 404 — config/account errors where retrying any
      model in the chain won't help), retry-same (connection errors /
      5xx / unclassifiable — likely transient), and advance (timeout /
      429 / empty body / parser failure / other 4xx / content-filter-
      style finish-reason quits — model-specific pathology). Each
      backend gets up to $.max-retries HTTP attempts (initial +
      retries-same, exponential backoff with jitter capped at 30 s);
      an advance-class error on any attempt short-circuits the budget
      and moves on. Abort-class errors re-raise immediately without
      trying the rest of the chain. See classify-error and the module
      Pod for the full rule table.
    - Task.classify-error(:$error-class, :$error-status,
      :$parser-failed) exposed as a public method for testability and
      for consumers that want to implement the same policy outside
      the retry loop.
    - Telemetry hook payload adds :backend-index, :model-name,
      :error-class, :error-status, so sinks can identify which model
      in the chain served each call and what failed. Existing keys
      (:attempt, :success, :error, :latency-ms, :prompt-tokens,
      :completion-tokens, :total-tokens, :cost, :model-used,
      :provider-id, :finish-reason, :stage) are unchanged.
    - Single-:backend constructor shape is preserved; a one-element
      chain behaves exactly like the pre-fallback Task on the
      retry-same path (connection / 5xx still get $.max-retries
      same-model attempts with exponential backoff).

    BACKWARD COMPATIBILITY — one behavioural break:

    Advance-class errors (timeout / 429 / parser failure / empty body
    / content-filter-style finish quits / other 4xx) used to retry
    the same model up to $.max-retries times. They now advance to
    the next backend in the chain, or die immediately with
    "all backend(s) exhausted" if the Task only has one. Retry-same
    errors (connection drop / 5xx) are unchanged — still retried on
    the same backend up to $.max-retries times before advancing.

    Consequence for single-backend callers: a Task that previously
    survived a stochastic parser failure via retry now dies on the
    first parse error. Three mitigations:
      - Preferred: pass :@backends with a fallback model. A chain
        of [primary, primary] is also legal and preserves the exact
        old "try the same model twice" behaviour on advance-class
        failures while keeping retry-same semantics intact.
      - Build the retry loop at the application layer if the same
        model genuinely recovers for your workload.
      - Accept the fail — in practice the old behaviour rarely
        recovered on parser failures (malformed JSON tended to
        repeat), which is the motivation for the change.

0.1.3  2026-04-07T20:03:42+01:00
    - Updated readme examples to use generic routing terms

0.1.2  2026-04-07T20:00:40+01:00
    - Removed Windows from CI (upstream Digest::SHA256::Native does not build on Windows)

0.1.1  2026-04-07T19:52:19+01:00
    - Added Windows CI support via MSVC

0.1.0  2026-04-07T18:52:19+01:00
    - Initial release
    - Task: blocking LLM calls with configurable parser and retry
    - JSONTask: JSON extraction from LLM responses with key validation
    - Router: query-based routing using Roaring::Tags
    - PromptBuilder: mustache-style template rendering
