Revision history for LLM::Chat

0.10.0  2026-08-21T22:05:38+01:00
    - [2026-08-21] Require Template::Jinja2 0.3.0. The byte-identical
      rendering of upstream GLM and Kimi chat templates leans on its Python
      method-call dispatch and Jinja2-faithful filters; an older renderer
      would produce silently different prompts.
    - [2026-08-20] The OpenRouter backend marks `cache_control`
      breakpoints on outgoing chat bodies, so providers that only
      cache a prompt prefix when the request says where it ends
      (Anthropic, Qwen, Gemini) can actually cache one. Markers go on
      the head message and on the last two `user` / `assistant` turns
      — the trailing pair is what makes the marker written one round
      ago still cover what the next request shares with it — with
      tool results and tool-calling assistants stepped over, and
      never more than OpenRouter's ceiling of four. Implicit-caching
      routes ignore the annotation, so the new `cache-breakpoints`
      attribute defaults to On across a mixed fleet; set it to False
      to send the plain string shape. Conversation state is untouched
      either way: the rewrite happens on the serialized body, so
      `Message.to-hash` and `Message.get-checksum` read the same
      after a request as before it.
      Placing them needed a seam, so `OpenAICommon` gained one:
      `_finalize-request-body` is called on all four completion paths
      with the assembled body, after messages / prompt / tools /
      stream are attached and before the POST. The default returns
      its argument, so nothing changes for backends that don't
      override it.
    - [2026-08-20] New `Response.cached-prompt-tokens`: the slice of
      the prompt a provider served out of its own cache rather than
      processing fresh, which is what tells a caller whether the
      input it is paying full price for is actually being reused.
      Lifted from the OAI-spec `usage.prompt_tokens_details.
      cached_tokens` on any OpenAI-compatible backend, and on
      OpenRouter also from the `/generation` poll — the only place
      the streaming path can pick it up, since that backend
      deliberately asks for no inline usage frame. A value the
      response body reported always wins over the poll's. Undefined
      means the provider said nothing about caching, which is not the
      same as nothing having been cached, so a reported zero stays a
      reported zero.
    - [2026-08-19] A streamed response now always reaches a terminal
      state. Both streaming paths route every ending through one
      place, so `[DONE]`, a `stop` / `tool_calls` finish reason
      followed by the body closing, and a body that closes with
      neither all settle the Response — the last of those as a
      failure with error-class `response`, since a reply cut off in
      transit cannot be told from a reply that ended. Previously that
      third case left the Response neither done nor quit: a consumer
      polling `.is-done` waited out its own timeout, minutes at a
      time, for a stream that had been over from the start. Providers
      that close on the finish_reason chunk without ever sending
      `[DONE]` are common, and they now complete promptly and
      successfully instead. The `length` / `content_filter` /
      unknown-reason failures are unchanged.
      Fixes a hook-ordering bug on the way: text-completion-stream's
      `[DONE]` arm completed the supply BEFORE `_on-stream-complete`,
      so a subclass's post-stream metadata landed after consumers had
      been told the response was finished. Both paths now fire the
      hook first, as the hook has always documented.
    - [2026-08-19] New `Response.dropped-frames`: streamed `data:`
      lines that fail to parse are still skipped — providers
      interleave junk into healthy streams — but they are now counted,
      and a stream that dropped frames while assembling tool calls
      FAILS instead of succeeding. A hole in prose costs a few words
      and is visible to whoever reads the reply; a hole in a tool
      call's `arguments` is spliced out of a JSON string that is then
      concatenated blind, and what comes out can be valid JSON saying
      something the model never asked for. Nothing downstream can
      detect that, so it no longer reaches `is-success`. Prose-only
      streams keep their success and expose the counter. SSE
      heartbeats and empty `data:` lines are not counted.
    - [2026-08-19] Streaming tool-call deltas that carry no `index`
      are assembled correctly. The OAI spec puts an `index` on every
      fragment and it is still authoritative when present, but some
      providers omit it, and the old fallback used the number of calls
      seen so far — which sent every argument fragment to a fresh
      call. One tool call arrived shattered into N, each holding an
      unparseable slice of the JSON. The fallback now reads `id`,
      which the spec sends only on a call's first fragment: an `id`
      starts a new call, an arguments-only fragment continues the one
      in flight.
    - [2026-08-19] `LLM::Chat::Template::Jinja2.render` presents
      tool-call `function.arguments` to the template as a decoded
      object when they are a JSON object, matching what HuggingFace
      chat templates are written against (transformers passes a dict).
      GLM-family templates iterate `arguments.items()` and rendered
      nothing at all against the JSON string the OpenAI wire format
      uses — a silently truncated prompt, and for a token counter an
      undercount of the largest part of it. The messages themselves
      are untouched: the render works on a copy, so `to-hash` and
      every request body built from a conversation still carry the
      string. Arguments that are not a JSON object — empty, a fragment
      cut short mid-stream, an array, a bare scalar, or a Hash a
      caller supplied directly — are passed through exactly as they
      are.
    - [2026-08-13] `LLM::Chat::TokenCounter.get-request-count` counts a
      complete selected-model request: optional runtime context head, stored
      conversation, optional runtime context tail, and tool catalogue rendered
      through the model's actual template, then tokenized without a second
      special-token pass. This is the exact-counting seam for models such as
      Kimi K2.7 whose tool declarations live in the chat template. Existing
      `get-conversation-count` remains conversation-only and unchanged.
      Kimi's published Jinja fallback serializes ordinary OpenAI tool
      declarations with compact `tojson`; `tools_ts_str` is an optional
      preformatted override, so no model-specific duplicate renderer is used.
      Runtime context may be supplied as its natural string halves or as
      prebuilt system messages. Rendered prompts opt into declared special
      tokens explicitly, which is required by deny-by-default tiktoken
      backends and harmless for Hugging Face JSON tokenizers.
    - [2026-08-13] Add `LLM::Chat::Template::DeepSeekV4`, a byte-compatible
      Raku port of the full prompt-encoding half of DeepSeek's pinned DSV4
      reference implementation. It supports chat/thinking modes, reasoning
      effort, context and thinking removal, developer/latest-reminder turns,
      quick tasks, response formats, OpenAI-format tools/calls, DSML argument
      encoding, and merged/call-ordered tool results. `encode-json` preserves
      JSON object order for differential goldens; `encode-messages` accepts
      ordinary associative message structures; `render` adapts existing
      `Conversation::Message` objects. Completion parsing is intentionally not
      included. The four official Python outputs are pinned as offline golden
      tests with DeepSeek's MIT attribution.
    - [2026-08-12] A Cro timeout is now classified BY THE PHASE it
      expired in. `X::Cro::HTTP::Client::Timeout` carries a `phase` of
      `connection`, `headers` or `body`, and OpenAICommon's classifier
      used to file all three as error-class `timeout` — the ADVANCE
      bucket. A connection-phase timeout is Cro's 30-second connect
      budget running out: nothing was sent, the endpoint said nothing,
      and the only evidence is a network that was unwell. That is now
      error-class `connection`, the RETRY-SAME bucket. Headers- and
      body-phase timeouts keep `timeout`: a backend that accepted the
      connection and then would not answer has said something about
      itself, and the next backend answers sooner than a backoff ends.
      The lumping was fatal on a ONE-BACKEND chain, where `advance`
      means the chain is exhausted: a single flaky connect ended the
      run having made exactly one call. `classify-error`'s own bucket
      table is unchanged — the split is at the source, which is where
      the information is.
      The classification test no longer re-implements the classifier
      it is testing; it calls the real private method through
      `^find_private_method`, so the two cannot drift.
    - [2026-08-12] `LLM::Chat::Template::Jinja2.render` takes an
      optional `:@tools` — the tool declarations the request will
      carry, in the provider's own shape — and passes them to the
      template as its `tools` variable. Real HuggingFace chat
      templates render a tools block when they are given one, so a
      render meant to match what the server sees needs them.
      Passed ONLY when non-empty: `tools => []` and no tools at all
      both render exactly as they did before the parameter existed
      (pinned by a test, because a template asking
      `{% if tools is defined %}` would otherwise take a different
      branch for a caller who named nothing). Deliberately NOT wired
      into `LLM::Chat::TokenCounter`: the declarations are a
      per-request constant that provider-reported usage has already
      been charged for, and counting them per message would bill them
      twice.

0.8.1  2026-08-09T20:57:18+01:00
    - META fix: Response::OpenRouter and Response::OpenRouter::Stream were
      missing from provides and listed under DEPENDS instead. On a machine
      without install history, zef resolved those module names to the fossil
      LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
      pin only builds with nmake — breaking every fresh Windows install
      (caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
      published carries the broken META and should not be depended on.

0.8.0  2026-08-09T19:41:06+01:00
    - [2026-08-09] New C<LLM::Chat::Retry> and
      C<LLM::Chat::Retry::Exceptions>: the retry/fallback policy that
      has run inside C<LLM::Data::Inference::Task> since 0.5, factored
      out so a second executor can share it instead of copying it.
      C<LLM::Chat::Retry> exports five pure subs — C<classify-error>
      (the abort / retry-same / advance bucket table),
      C<retry-backoff> (C<min(2 ** (n - 1) + jitter, cap)>, with the
      anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
      deterministic tests), C<sleep-with-cancel> (chunked sleep that
      consults a cancel hook before EVERY chunk including the first,
      and returns a Bool rather than throwing, because only the caller
      knows which exception type its layer promises),
      C<attempt-record> and C<telemetry-payload> (which duck-types the
      response with C<.?> and therefore does not import
      C<LLM::Chat::Backend::Response>).
      C<LLM::Chat::Retry::Exceptions> provides
      C<X::LLM::Chat::Retry::Exhausted> and its C<Truncated> /
      C<TimedOut> subclasses plus C<Cancelled> (deliberately NOT an
      C<Exhausted>, so a user cancel can never be swallowed by a
      dead-letter handler), and is now the canonical home of the
      C<item-retryable> advice contract. Those four are plain global
      classes rather than C<is export>ed ones on purpose: C<is export>
      on a nested-name class also exports its LEAF name, so two
      modules declaring an C<X::Something::Exhausted> cannot be
      imported into the same scope at all — which would have made the
      shared hierarchy unusable next to a subclassing one.
      Deliberately NOT shared: the
      retry loop itself, the abort bucket's C<die> message, hook
      payloads and hook shielding — those are shaped by the layer that
      owns them. B<No existing module changed>: this release is purely
      additive, every previous test file is untouched, and
      C<LLM::Data::Inference> 0.9.0 migrates onto it (its exception
      types become subclasses of these, so both hierarchies keep
      matching).
    - [2026-08-08] Fix: a streamed completion could be KILLED MID-
      GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
      (C<chat-completion-stream> and C<text-completion-stream>) did
      C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
      but TCP splits a body wherever it likes — including through the
      middle of a multi-byte UTF-8 sequence. C<.decode> on the half
      that ends mid-sequence throws "Malformed termination of UTF-8
      string", which landed in the streaming CATCH and quit the
      Response with error class 'unknown', discarding everything
      generated so far. Buffering text could not fix it: the throw
      happened before the text reached the buffer. The bytes now go
      through an incremental C<Encoding::Decoder> (new shared internal
      C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
      sees the line endings that are actually on the wire), which
      keeps an unfinished sequence to itself until the bytes
      completing it arrive. Framing, heartbeat skipping, C<[DONE]>
      handling, malformed-chunk tolerance and reasoning/content
      accumulation are untouched. Emoji, CJK and curly quotes are
      ordinary in model output, so this was an intermittent
      production crash for anything but plain ASCII.
    - [2026-08-08] Fix: non-ASCII in a response body could come back as
      mojibake (café -> cafÃ©). Cro's C<body-text-encoding> returns the
      LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
      charset, and C<Cro::MessageWithBody.body-text> loops that list
      with no C<last> — so the latin-1 attempt, which cannot fail
      whatever the bytes are, always overwrote the correct utf-8
      decode. Many OpenAI-compatible servers answer
      C<Content-Type: application/json> with no charset parameter, and
      model output is full of non-ASCII (names, curly quotes, emoji,
      CJK). OpenAICommon now decodes response bytes itself via new
      internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
      utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
      latin-1 kept only as a fallback for when the utf-8 decode
      actually throws — a proxy's non-JSON error page, say. Affected
      the 4xx error-body read that feeds LLM::Chat::Debug, which is
      where an operator reads a rejection reason.
    - [2026-08-08] The blocking C<chat-completion> / C<text-completion>
      paths and OpenRouter's C</generation> metadata lookup now parse
      their JSON from the raw bytes instead of C<await $res.body>.
      Cro's response body-parser selector reaches the correct parser
      only while the server labels the body C<application/json>; a
      server that answers JSON as C<text/plain> fell through to
      TextFallback and produced a mojibaked Str, which the completion
      path then indexed as a Hash and SILENTLY emitted as empty
      content. Well-behaved servers see no change. Transport failures
      still propagate as transport exceptions, so error classification
      is unchanged.

0.7.1  2026-07-27T15:27:39+01:00
    - [2026-07-27] LLM::Chat::Backend::Mock gains
      C<&.finish-reason-producer>, an optional
      C<(Int $call-index --> Str)> callback mirroring
      C<&.error-producer>'s per-call indexing. A defined Str return is
      stamped on the Response via C<_set-finish-reason> before the
      emission task starts, while the call still SUCCEEDS — the exact
      shape of a real blocking completion that ran out of completion
      budget (HTTP 200, well-formed but partial body, finish_reason
      'length'). Undefined returns leave C<finish-reason> unset, so
      existing mocks are byte-for-byte unchanged. A scripted error
      wins over a scripted finish reason (a failed call never produced
      a body). Wired into the non-streaming paths only —
      C<chat-completion>, and C<text-completion> by delegation —
      because the streaming paths already model finish reasons the way
      the real transports do. Lets downstream consumers
      (LLM::Data::Inference's truncation policy) test truncation
      handling without a live model.
    - [2026-07-27] Fix: OpenAICommon's C<text-completion-stream> acted
      on a terminal C<finish_reason> without recording it, so a stream
      cut off by the token budget quit with error class 'response' but
      an UNDEFINED C<.finish-reason> — consumers could not tell budget
      exhaustion apart from a content filter or an unknown reason.
      It now calls C<_set-finish-reason> before dispatching on the
      reason, matching C<chat-completion-stream>.

0.7.0  2026-07-21T03:09:53+01:00
    - [2026-07-03] Structured outputs: Backend::Settings gains
      C<json_schema> (a plain-Hash JSON Schema, default undefined) and
      C<json_schema_name>. OpenAI-compatible backends (incl.
      OpenRouter) send it as C<response_format: { type: 'json_schema',
      json_schema: { name, strict: false, schema } }> — best-effort,
      provider-dependent enforcement; KoboldCpp sends the native
      C<json_schema> generate field, which it compiles to a grammar
      for sampler-level enforcement. Unset schema changes no request
      bytes. Callers should keep parse validation as the backstop for
      providers that ignore response_format.

0.6.0  2026-06-22T23:33:11+01:00
    - Add LLM::Chat::ToolLoop, a generic streaming tool-call
      round loop. It sends OpenAI-format tools to any chat backend,
      executes returned tool calls through a caller-supplied
      callback, appends assistant/tool turns, and continues until a
      final answer or guardrail limit is reached.
    - OpenAI-compatible streaming now assembles `delta.tool_calls`
      chunks and treats `finish_reason: tool_calls` as successful.
      This enables KoboldCpp streamed tool calling.
    - Conversation messages can now serialize assistant `tool_calls`
      and tool-result `tool_call_id` fields for follow-up chat
      completion requests.
    - LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
      failure-proof: the local response stream closes FIRST,
      synchronously, then the upstream POST /api/extra/abort fires on
      a worker thread, best-effort, with network failures swallowed.
      The old ordering awaited the abort before closing locally, so a
      busy KoboldCpp froze the caller for the full HTTP round-trip
      (interactive cancels hung the UI) and an unreachable one made
      cancel throw without ever closing the stream. cancel now
      returns the background Promise so callers/tests can await the
      abort attempt. Also fixes the abort URL: stripping the /v1
      suffix used to leave its leading slash behind, producing
      //api/extra/abort.

0.5.1  2026-04-29T23:47:12+01:00
    - Bump Github Actions to use node 24+
    - LLM::Chat::Backend::OpenAICommon gains a symmetric
      `_on-blocking-complete` hook on the non-streaming path,
      mirroring the existing `_on-stream-complete` contract: fires
      after the response body has been parsed and `_lift-usage`
      has lifted OAI/provider usage fields, before
      `$response.done`. Default implementation is a no-op.
      Subclasses use it to attach post-call metadata that isn't
      in the body itself — symmetric with what was already
      possible on streams.
    - LLM::Chat::Backend::OpenRouter wires the new hook through
      to the same `/generation?id=...` lookup the streaming path
      already uses, so blocking callers (e.g. App::Storygen, which
      calls `chat-completion` rather than `chat-completion-stream`)
      now see `.cost` populated by the time `$response.done` fires.
      Pre-fix this was the silent regression from dropping
      `usage: { include: true }` — `Response::OpenRouter.cost`
      stayed Nil on every blocking call.
    - LLM::Chat::Backend::OpenRouter refactor: the lookup logic
      is now in a private `!fetch-generation-metadata` helper that
      both `_on-stream-complete` and `_on-blocking-complete`
      delegate to. No behaviour change on the streaming path.
    - Tests — t/12-openrouter-backend.rakutest gains a subtest
      covering the defensive guards on both completion hooks
      (no-op when generation-id is undefined; no crash when the
      response isn't OR-augmented). Plan goes 11 → 12; total
      LLM::Chat tests 130.

0.5.0  2026-04-27T22:52:25+01:00
    - LLM::Chat::Backend::OpenRouter request shape now mirrors
      SillyTavern's wire bytes verbatim. Removed two body fields
      that were causing OpenRouter's upstream router to hold 200 OK
      headers indefinitely against some providers (~80% header-phase
      timeouts in App::Cantina vs ~0% in SillyTavern on the same
      models / keys / network):
        * `usage: { include: true }` — no longer sent.
        * `stream_options: { include_usage: true }` — no longer sent
          (also removed from OpenAICommon.chat-completion-stream so
          all OAI-compatible streams now match this shape).
        * `reasoning: { effort, enabled }` — `enabled` key dropped;
          we now send only `{ effort }` when reasoning_effort is
          configured, matching ST.
      Added on every request, also matching ST:
        * `include_reasoning: Bool` — Boolean parity with ST's flag.
        * `top_k` — plumbed from Settings into the OAI body.
      `repetition_penalty` is now omitted when at the default 1.0
      (was previously sent unconditionally).
    - Cost telemetry that the inline `usage: { include: true }`
      block used to carry now arrives via a one-shot post-stream
      GET against `/generation?id=...` after `[DONE]`. Lookup is
      async and best-effort; on failure $resp.cost stays Nil rather
      than escalating. Lifts cost, provider-name, and (when not
      already populated from the stream) prompt/completion tokens.
      Latency: ~50–200ms after .is-done becomes True before .cost
      is readable. New hook _on-stream-complete on OpenAICommon so
      future provider subclasses can do the same kind of
      post-stream metadata fetch.
    - LLM::Chat::Backend::OpenAICommon stream parser now buffers
      bytes across body-byte-stream emissions and splits on the
      SSE `\n\n` event delimiter before parsing, instead of
      decoding+parsing each TCP chunk independently. Pre-fix, a
      `data: {...}` JSON object split across two TCP packets would
      crash from-json on the truncated half and terminate the
      stream as 'unknown' error class. Both chat-completion-stream
      and text-completion-stream got the fix. Heartbeat / SSE
      comment lines (`: OPENROUTER PROCESSING`) are dropped per
      spec — never produce a chunk.
    - LLM::Chat::Backend::OpenAICommon.!classify-exception no
      longer string-matches "timeout" / "timed out" in the default
      arm; only X::Cro::HTTP::Client::Timeout maps to error-class
      'timeout' now. Substring matching was masking unrelated
      errors (JSON parse failures, stream-cancel messages) as
      header timeouts. Connection-error pattern (refused / reset /
      DNS / unreachable) stayed — those don't have a typed
      exception class to discriminate on.
    - LLM::Chat::Debug log format gains elapsed-ms timestamps for
      streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
      EXCEPTION lines all carry "+Nms" relative to the call start
      so latency can be diagnosed without external instrumentation.
      Existing log labels unchanged.

0.3.0  2026-04-23T15:56:28+01:00
    - LLM::Chat::Backend::Response gains structured error metadata:
      $.error-status (Int HTTP code) and $.error-class (Str —
      'http' / 'timeout' / 'connection' / 'response' / 'unknown').
      Populated via _set-error-info(:$class, :$status) alongside
      the existing .quit path. Lets consumers branch on error kind
      without regex-parsing raw messages — used by the
      LLM::Data::Inference::Task model-fallback policy.
    - LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
      exceptions (X::Cro::HTTP::Error — picks up status off
      .response.status; X::Cro::HTTP::Client::Timeout) plus
      heuristic socket-error detection into the Response's error
      fields before quitting. Finish-reason quits (length /
      content_filter / unknown) are tagged error-class => 'response'
      so the fallback layer advances on them.
    - LLM::Chat::Backend::Mock gains &.error-producer — an optional
      (Int $call-index --> Hash) callback that scripts per-call
      failures for fallback / retry tests. Returning a hash like
      { class => 'http', status => 500, message => 'x' } fails
      that call without consuming a slot from @.responses. Also
      exposes $.call-index (monotonic per-backend call count,
      bumped on every completion regardless of outcome).

0.2.6  2026-04-13T17:03:42+01:00
    - LLM::Chat::Backend::Mock: new test-only backend that returns
      canned responses in order. Supports streaming (default splits
      on whitespace, configurable via :token-splitter), optional delay
      between tokens via :stream-delay, and an :initial-delay (10ms
      default) so consumers can attach taps before tokens flow.
      :fail-on-empty makes the backend die when the response queue
      is exhausted instead of repeating the last entry. Useful for
      exercising error paths in downstream consumers.
      Recording: every completion call is logged to @.recorded-calls
      as a hash with kind / messages / tools / response / at. Tests
      can assert on what reached the backend, not just what came
      back — catches prompt-assembly and template-substitution
      regressions. clear-recorded-calls resets the log between
      test phases.

0.2.5  2026-04-09T12:30:48+01:00
    - Optional :@tools parameter on chat-completion and chat-completion-stream
    - Response.tool-calls and has-tool-calls for detecting LLM tool call requests
    - Response.finish-reason field

0.2.4  2026-04-09T05:15:41+01:00
    - CI: exclude Windows (Tokenizers Rust FFI build not yet supported)

0.2.3  2026-04-09T05:09:48+01:00
    - Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
    - Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
    - Add docs/Readme.rakudoc

0.2.2  2026-04-09T04:59:41+01:00
    - Add stub LLM::Chat to make mi6 stop renaming the module.

0.2.1  2026-04-09T04:41:36+01:00
    - Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
    - from-tokenizer-config class method loads templates from tokenizer_config.json
    - Supports bos_token, eos_token passthrough
    - Continuation mode maps to add_generation_prompt=false

0.2.0
    - Previous releases
