Revision history for LLM::Chat

0.8.1  2026-08-09T20:57:18+01:00
    - META fix: Response::OpenRouter and Response::OpenRouter::Stream were
      missing from provides and listed under DEPENDS instead. On a machine
      without install history, zef resolved those module names to the fossil
      LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
      pin only builds with nmake — breaking every fresh Windows install
      (caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
      published carries the broken META and should not be depended on.

0.8.0  2026-08-09T19:41:06+01:00
    - [2026-08-09] New C<LLM::Chat::Retry> and
      C<LLM::Chat::Retry::Exceptions>: the retry/fallback policy that
      has run inside C<LLM::Data::Inference::Task> since 0.5, factored
      out so a second executor can share it instead of copying it.
      C<LLM::Chat::Retry> exports five pure subs — C<classify-error>
      (the abort / retry-same / advance bucket table),
      C<retry-backoff> (C<min(2 ** (n - 1) + jitter, cap)>, with the
      anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
      deterministic tests), C<sleep-with-cancel> (chunked sleep that
      consults a cancel hook before EVERY chunk including the first,
      and returns a Bool rather than throwing, because only the caller
      knows which exception type its layer promises),
      C<attempt-record> and C<telemetry-payload> (which duck-types the
      response with C<.?> and therefore does not import
      C<LLM::Chat::Backend::Response>).
      C<LLM::Chat::Retry::Exceptions> provides
      C<X::LLM::Chat::Retry::Exhausted> and its C<Truncated> /
      C<TimedOut> subclasses plus C<Cancelled> (deliberately NOT an
      C<Exhausted>, so a user cancel can never be swallowed by a
      dead-letter handler), and is now the canonical home of the
      C<item-retryable> advice contract. Those four are plain global
      classes rather than C<is export>ed ones on purpose: C<is export>
      on a nested-name class also exports its LEAF name, so two
      modules declaring an C<X::Something::Exhausted> cannot be
      imported into the same scope at all — which would have made the
      shared hierarchy unusable next to a subclassing one.
      Deliberately NOT shared: the
      retry loop itself, the abort bucket's C<die> message, hook
      payloads and hook shielding — those are shaped by the layer that
      owns them. B<No existing module changed>: this release is purely
      additive, every previous test file is untouched, and
      C<LLM::Data::Inference> 0.9.0 migrates onto it (its exception
      types become subclasses of these, so both hierarchies keep
      matching).
    - [2026-08-08] Fix: a streamed completion could be KILLED MID-
      GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
      (C<chat-completion-stream> and C<text-completion-stream>) did
      C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
      but TCP splits a body wherever it likes — including through the
      middle of a multi-byte UTF-8 sequence. C<.decode> on the half
      that ends mid-sequence throws "Malformed termination of UTF-8
      string", which landed in the streaming CATCH and quit the
      Response with error class 'unknown', discarding everything
      generated so far. Buffering text could not fix it: the throw
      happened before the text reached the buffer. The bytes now go
      through an incremental C<Encoding::Decoder> (new shared internal
      C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
      sees the line endings that are actually on the wire), which
      keeps an unfinished sequence to itself until the bytes
      completing it arrive. Framing, heartbeat skipping, C<[DONE]>
      handling, malformed-chunk tolerance and reasoning/content
      accumulation are untouched. Emoji, CJK and curly quotes are
      ordinary in model output, so this was an intermittent
      production crash for anything but plain ASCII.
    - [2026-08-08] Fix: non-ASCII in a response body could come back as
      mojibake (café -> cafÃ©). Cro's C<body-text-encoding> returns the
      LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
      charset, and C<Cro::MessageWithBody.body-text> loops that list
      with no C<last> — so the latin-1 attempt, which cannot fail
      whatever the bytes are, always overwrote the correct utf-8
      decode. Many OpenAI-compatible servers answer
      C<Content-Type: application/json> with no charset parameter, and
      model output is full of non-ASCII (names, curly quotes, emoji,
      CJK). OpenAICommon now decodes response bytes itself via new
      internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
      utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
      latin-1 kept only as a fallback for when the utf-8 decode
      actually throws — a proxy's non-JSON error page, say. Affected
      the 4xx error-body read that feeds LLM::Chat::Debug, which is
      where an operator reads a rejection reason.
    - [2026-08-08] The blocking C<chat-completion> / C<text-completion>
      paths and OpenRouter's C</generation> metadata lookup now parse
      their JSON from the raw bytes instead of C<await $res.body>.
      Cro's response body-parser selector reaches the correct parser
      only while the server labels the body C<application/json>; a
      server that answers JSON as C<text/plain> fell through to
      TextFallback and produced a mojibaked Str, which the completion
      path then indexed as a Hash and SILENTLY emitted as empty
      content. Well-behaved servers see no change. Transport failures
      still propagate as transport exceptions, so error classification
      is unchanged.

0.7.1  2026-07-27T15:27:39+01:00
    - [2026-07-27] LLM::Chat::Backend::Mock gains
      C<&.finish-reason-producer>, an optional
      C<(Int $call-index --> Str)> callback mirroring
      C<&.error-producer>'s per-call indexing. A defined Str return is
      stamped on the Response via C<_set-finish-reason> before the
      emission task starts, while the call still SUCCEEDS — the exact
      shape of a real blocking completion that ran out of completion
      budget (HTTP 200, well-formed but partial body, finish_reason
      'length'). Undefined returns leave C<finish-reason> unset, so
      existing mocks are byte-for-byte unchanged. A scripted error
      wins over a scripted finish reason (a failed call never produced
      a body). Wired into the non-streaming paths only —
      C<chat-completion>, and C<text-completion> by delegation —
      because the streaming paths already model finish reasons the way
      the real transports do. Lets downstream consumers
      (LLM::Data::Inference's truncation policy) test truncation
      handling without a live model.
    - [2026-07-27] Fix: OpenAICommon's C<text-completion-stream> acted
      on a terminal C<finish_reason> without recording it, so a stream
      cut off by the token budget quit with error class 'response' but
      an UNDEFINED C<.finish-reason> — consumers could not tell budget
      exhaustion apart from a content filter or an unknown reason.
      It now calls C<_set-finish-reason> before dispatching on the
      reason, matching C<chat-completion-stream>.

0.7.0  2026-07-21T03:09:53+01:00
    - [2026-07-03] Structured outputs: Backend::Settings gains
      C<json_schema> (a plain-Hash JSON Schema, default undefined) and
      C<json_schema_name>. OpenAI-compatible backends (incl.
      OpenRouter) send it as C<response_format: { type: 'json_schema',
      json_schema: { name, strict: false, schema } }> — best-effort,
      provider-dependent enforcement; KoboldCpp sends the native
      C<json_schema> generate field, which it compiles to a grammar
      for sampler-level enforcement. Unset schema changes no request
      bytes. Callers should keep parse validation as the backstop for
      providers that ignore response_format.

0.6.0  2026-06-22T23:33:11+01:00
    - Add LLM::Chat::ToolLoop, a generic streaming tool-call
      round loop. It sends OpenAI-format tools to any chat backend,
      executes returned tool calls through a caller-supplied
      callback, appends assistant/tool turns, and continues until a
      final answer or guardrail limit is reached.
    - OpenAI-compatible streaming now assembles `delta.tool_calls`
      chunks and treats `finish_reason: tool_calls` as successful.
      This enables KoboldCpp streamed tool calling.
    - Conversation messages can now serialize assistant `tool_calls`
      and tool-result `tool_call_id` fields for follow-up chat
      completion requests.
    - LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
      failure-proof: the local response stream closes FIRST,
      synchronously, then the upstream POST /api/extra/abort fires on
      a worker thread, best-effort, with network failures swallowed.
      The old ordering awaited the abort before closing locally, so a
      busy KoboldCpp froze the caller for the full HTTP round-trip
      (interactive cancels hung the UI) and an unreachable one made
      cancel throw without ever closing the stream. cancel now
      returns the background Promise so callers/tests can await the
      abort attempt. Also fixes the abort URL: stripping the /v1
      suffix used to leave its leading slash behind, producing
      //api/extra/abort.

0.5.1  2026-04-29T23:47:12+01:00
    - Bump Github Actions to use node 24+
    - LLM::Chat::Backend::OpenAICommon gains a symmetric
      `_on-blocking-complete` hook on the non-streaming path,
      mirroring the existing `_on-stream-complete` contract: fires
      after the response body has been parsed and `_lift-usage`
      has lifted OAI/provider usage fields, before
      `$response.done`. Default implementation is a no-op.
      Subclasses use it to attach post-call metadata that isn't
      in the body itself — symmetric with what was already
      possible on streams.
    - LLM::Chat::Backend::OpenRouter wires the new hook through
      to the same `/generation?id=...` lookup the streaming path
      already uses, so blocking callers (e.g. App::Storygen, which
      calls `chat-completion` rather than `chat-completion-stream`)
      now see `.cost` populated by the time `$response.done` fires.
      Pre-fix this was the silent regression from dropping
      `usage: { include: true }` — `Response::OpenRouter.cost`
      stayed Nil on every blocking call.
    - LLM::Chat::Backend::OpenRouter refactor: the lookup logic
      is now in a private `!fetch-generation-metadata` helper that
      both `_on-stream-complete` and `_on-blocking-complete`
      delegate to. No behaviour change on the streaming path.
    - Tests — t/12-openrouter-backend.rakutest gains a subtest
      covering the defensive guards on both completion hooks
      (no-op when generation-id is undefined; no crash when the
      response isn't OR-augmented). Plan goes 11 → 12; total
      LLM::Chat tests 130.

0.5.0  2026-04-27T22:52:25+01:00
    - LLM::Chat::Backend::OpenRouter request shape now mirrors
      SillyTavern's wire bytes verbatim. Removed two body fields
      that were causing OpenRouter's upstream router to hold 200 OK
      headers indefinitely against some providers (~80% header-phase
      timeouts in App::Cantina vs ~0% in SillyTavern on the same
      models / keys / network):
        * `usage: { include: true }` — no longer sent.
        * `stream_options: { include_usage: true }` — no longer sent
          (also removed from OpenAICommon.chat-completion-stream so
          all OAI-compatible streams now match this shape).
        * `reasoning: { effort, enabled }` — `enabled` key dropped;
          we now send only `{ effort }` when reasoning_effort is
          configured, matching ST.
      Added on every request, also matching ST:
        * `include_reasoning: Bool` — Boolean parity with ST's flag.
        * `top_k` — plumbed from Settings into the OAI body.
      `repetition_penalty` is now omitted when at the default 1.0
      (was previously sent unconditionally).
    - Cost telemetry that the inline `usage: { include: true }`
      block used to carry now arrives via a one-shot post-stream
      GET against `/generation?id=...` after `[DONE]`. Lookup is
      async and best-effort; on failure $resp.cost stays Nil rather
      than escalating. Lifts cost, provider-name, and (when not
      already populated from the stream) prompt/completion tokens.
      Latency: ~50–200ms after .is-done becomes True before .cost
      is readable. New hook _on-stream-complete on OpenAICommon so
      future provider subclasses can do the same kind of
      post-stream metadata fetch.
    - LLM::Chat::Backend::OpenAICommon stream parser now buffers
      bytes across body-byte-stream emissions and splits on the
      SSE `\n\n` event delimiter before parsing, instead of
      decoding+parsing each TCP chunk independently. Pre-fix, a
      `data: {...}` JSON object split across two TCP packets would
      crash from-json on the truncated half and terminate the
      stream as 'unknown' error class. Both chat-completion-stream
      and text-completion-stream got the fix. Heartbeat / SSE
      comment lines (`: OPENROUTER PROCESSING`) are dropped per
      spec — never produce a chunk.
    - LLM::Chat::Backend::OpenAICommon.!classify-exception no
      longer string-matches "timeout" / "timed out" in the default
      arm; only X::Cro::HTTP::Client::Timeout maps to error-class
      'timeout' now. Substring matching was masking unrelated
      errors (JSON parse failures, stream-cancel messages) as
      header timeouts. Connection-error pattern (refused / reset /
      DNS / unreachable) stayed — those don't have a typed
      exception class to discriminate on.
    - LLM::Chat::Debug log format gains elapsed-ms timestamps for
      streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
      EXCEPTION lines all carry "+Nms" relative to the call start
      so latency can be diagnosed without external instrumentation.
      Existing log labels unchanged.

0.3.0  2026-04-23T15:56:28+01:00
    - LLM::Chat::Backend::Response gains structured error metadata:
      $.error-status (Int HTTP code) and $.error-class (Str —
      'http' / 'timeout' / 'connection' / 'response' / 'unknown').
      Populated via _set-error-info(:$class, :$status) alongside
      the existing .quit path. Lets consumers branch on error kind
      without regex-parsing raw messages — used by the
      LLM::Data::Inference::Task model-fallback policy.
    - LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
      exceptions (X::Cro::HTTP::Error — picks up status off
      .response.status; X::Cro::HTTP::Client::Timeout) plus
      heuristic socket-error detection into the Response's error
      fields before quitting. Finish-reason quits (length /
      content_filter / unknown) are tagged error-class => 'response'
      so the fallback layer advances on them.
    - LLM::Chat::Backend::Mock gains &.error-producer — an optional
      (Int $call-index --> Hash) callback that scripts per-call
      failures for fallback / retry tests. Returning a hash like
      { class => 'http', status => 500, message => 'x' } fails
      that call without consuming a slot from @.responses. Also
      exposes $.call-index (monotonic per-backend call count,
      bumped on every completion regardless of outcome).

0.2.6  2026-04-13T17:03:42+01:00
    - LLM::Chat::Backend::Mock: new test-only backend that returns
      canned responses in order. Supports streaming (default splits
      on whitespace, configurable via :token-splitter), optional delay
      between tokens via :stream-delay, and an :initial-delay (10ms
      default) so consumers can attach taps before tokens flow.
      :fail-on-empty makes the backend die when the response queue
      is exhausted instead of repeating the last entry. Useful for
      exercising error paths in downstream consumers.
      Recording: every completion call is logged to @.recorded-calls
      as a hash with kind / messages / tools / response / at. Tests
      can assert on what reached the backend, not just what came
      back — catches prompt-assembly and template-substitution
      regressions. clear-recorded-calls resets the log between
      test phases.

0.2.5  2026-04-09T12:30:48+01:00
    - Optional :@tools parameter on chat-completion and chat-completion-stream
    - Response.tool-calls and has-tool-calls for detecting LLM tool call requests
    - Response.finish-reason field

0.2.4  2026-04-09T05:15:41+01:00
    - CI: exclude Windows (Tokenizers Rust FFI build not yet supported)

0.2.3  2026-04-09T05:09:48+01:00
    - Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
    - Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
    - Add docs/Readme.rakudoc

0.2.2  2026-04-09T04:59:41+01:00
    - Add stub LLM::Chat to make mi6 stop renaming the module.

0.2.1  2026-04-09T04:41:36+01:00
    - Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
    - from-tokenizer-config class method loads templates from tokenizer_config.json
    - Supports bos_token, eos_token passthrough
    - Continuation mode maps to add_generation_prompt=false

0.2.0
    - Previous releases
