Revision history for LLM::Chat

0.10.3  2026-10-25T13:25:17Z
    - [2026-10-25] A fresh `zef install` failed in t/30 because the
      child process it starts could not see the dependencies zef had
      staged but not yet installed: it died with "Could not find
      Digest::SHA256::Native". zef tests a distribution from a staging
      repository and installs nothing until every test has passed, and
      the test handed its child only the FileSystem module
      repositories. It passed wherever the dependencies were already
      installed, so installing LLM::Chat failed only on a machine that
      did not have them yet. t/29, t/30 and t/35 now hand each child
      the test's whole module search path, every repository by its
      path-spec, the staging repository included. The library itself
      was unaffected.

0.10.2  2026-10-24T18:26:08+01:00
    - [2026-10-24] Installs on Rakudo 2026.07 work again. The test suite
      failed there, so installing 0.10.1 on 2026.07 stopped at the test
      step: one test pinned the exception class behind Rakudo's message
      stub, which is X::AdHoc before 2026.08 and X::StubCode since. It
      now accepts either; the library itself was unaffected.
    - [2026-10-24] A finished Response could briefly look failed.
      `.is-done` turned True one step before `.is-success` (and, on a
      failure, before `.is-cancelled` and `.err`), so a consumer polling
      `.is-done` from another thread could see a completed generation
      as one that had not succeeded. The terminal state is now published
      all at once: when `.is-done` is True, `.is-success`,
      `.is-cancelled`, `.err` and `.msg` are already final.

0.10.1  2026-10-24T17:46:46+01:00
    - [2026-10-24] The HTTP-level tests run against `raw-server`, the
      fixture server in LLM::Chat::Test::Kit: an HTTP/1.1 server on
      IO::Socket::Async that answers each request with what a responder
      returns (a body, chunks, a Supply of chunks, a delay) and
      finishes writing every response before it closes anything. Cro's
      own server can lose the end of a response on its Connection:
      close path when the machine is busy, which would fail the suite
      intermittently on a loaded machine. Installing no longer pulls
      in Cro::HTTP::Server or Cro::HTTP::Router for the tests.
    - [2026-10-24] Requests no longer leave their connection open. Each
      one used a fresh Cro client, persistent by default, so its
      connection stayed cached in a client that was then thrown away:
      a socket left open per request for the life of the process,
      which on a 256-descriptor limit (the macOS default) exhausted
      file descriptors over a long session or a busy batch. Every
      client the backends make (completions, streams, server probes,
      KoboldCpp's abort and length check, OpenRouter's generation
      lookup) is now non-persistent: it asks the server to close the
      connection once the response is done. The test suite also
      passes, with nothing but TAP in its output, under that limit.
    - [2026-10-24] A transport, hook or tool error whose exception has
      no message no longer kills the process. Cro raises one when a
      server closes the connection before sending response headers
      (Cro::HTTP::Client::PipelineClosedBeforeHeaders). Such an
      exception's message is the base Exception's stub, which throws
      as soon as it is used, and the error handlers interpolated it
      into their log lines — so each threw again from inside its
      CATCH, on the request's worker thread, where that is "Unhandled
      exception in code scheduled on thread N" and the program exits.
      Such a request now fails like any other, with the exception's
      class name as its error text; error-class 'unknown'. The side
      calls are covered too: a `_reconcile-finish-reason` hook or
      OpenRouter's generation lookup that dies that way is logged and
      shrugged off like any other failure of theirs, where it used to
      fail the completion with "Stub code executed"; so is a
      Response's aborter, where `cancel` used to throw at its caller;
      the server probes (`probe-server-limits`, KoboldCpp's length
      check) end their `note` with "(no message)" rather than nothing;
      and ToolLoop quits with the class name when a backend call dies
      that way, and hands it back as the tool result's content when
      `execute-tools` does. The new LLM::Chat::ExceptionText exports
      the two subs behind the fix, `exception-message` and
      `exception-text`, which read any exception's message without
      ever throwing.
    - [2026-09-28] Requires JSON::Fast 0.21.0 or later (zef:timo). 0.19
      (cpan:TIMOTIMO) fails its own tests under the RakuAST compiler
      frontend that Rakudo 2026.09 makes the default, so installing this
      distribution stopped at that dependency.
    - [2026-09-25] Thinking, per request. `reasoning_effort` now takes
      'none' and 'minimal' beside 'low', 'medium' and 'high', and
      OpenAICommon sends it as the top-level OpenAI field (it used to
      reach OpenRouter only): llama.cpp's server hands it to the chat
      template, where 'none' turns thinking off and a Mistral Small 4
      template takes 'high'. A new `reasoning_budget_tokens` setting
      caps the thinking per request (llama.cpp's
      `reasoning_budget_tokens`, which overrides `--reasoning-budget`
      and forces the end-of-thinking tag once spent; KoboldCpp gets it
      as `thinking_budget_tokens`). Both go out only when set.
      OpenRouter keeps its `reasoning: {effort}` object and drops the
      top-level fields.
    - [2026-09-25] Thinking is captured, never mixed in. A blocking
      reply's `reasoning_content` (llama.cpp, KoboldCpp) now lands on
      `reasoning-text` as `reasoning` (OpenRouter) and the streamed
      deltas already did; the content stays the answer alone.
      `usage.completion_tokens_details.reasoning_tokens` is lifted onto
      a new `reasoning-tokens` (a subset of `completion-tokens`,
      undefined when unreported).
    - [2026-09-25] `id_slot`: a new setting that pins a request to one
      llama.cpp server slot, so requests sharing a long prefix reuse
      that slot's KV cache dependably. OpenAICommon sends it only when
      set; KoboldCpp and OpenRouter never send it.
    - [2026-09-25] `probe-server-limits`: ask the server for its context
      size. OpenAICommon reads llama.cpp's `GET /props`
      (`default_generation_settings.n_ctx`, plus `total_slots`, or the
      length of `GET /slots` when `/props` has no slot count);
      KoboldCpp reads `GET /api/extra/true_max_context_length`;
      OpenRouter and the base class ask nothing. The answer always has
      `source` and either `context_length` (and `slots` with its
      `slots_source`, or a `slots_note`) or a `note`; it never throws
      and each request is bounded by the new `server-probe-timeout`
      (3 s). New shared internals for subclasses: `_server-root`,
      `_probe-json` and `_probe-json-array`.
    - [2026-09-25] Mock: `fake-reasoning` (a thinking trace on every
      response), `server-limits` (what `probe-server-limits` answers)
      and a `reasoning` usage key.
    - [2026-09-23] Streams keep their usage when they fail. Both
      streaming paths quit the Response the moment they read a
      `length`, `content_filter` or unknown finish reason — and the
      text path settled on `stop` the same way — but the usage frame
      that stream_options.include_usage asks for comes AFTER the
      finish chunk, so a failed stream, and every text stream, lost
      its token counts. A finish reason is now recorded where it
      arrives and acted on where the stream settles (`[DONE]`, the
      body closing, or a new `stream-finish-grace` timer); later
      chunks are read for usage and model only, never for content,
      reasoning or tool-call deltas. The outcomes are unchanged — the
      same success/failure split, error class and error text ("Hit
      max tokens", ...) — except that a failure now carries its usage,
      and `[DONE]` after a failure reason still fails. A server that
      sends its finish chunk and then neither `[DONE]` nor a close is
      settled after `stream-finish-grace` seconds (default 10; Inf
      waits for the body) and its connection dropped, rather than
      holding the Response for the body's own, unbounded, timeout. A
      cancel in that gap stays a plain cancel. The blocking text path
      now lifts usage and records the finish reason too; it read the
      text and nothing else.
    - [2026-09-23] New `_reconcile-finish-reason` hook on OpenAICommon:
      it sees every finish reason once, on all four paths (on a stream
      after the usage frame), and returns the reason the Response acts
      on. The Response now has `provider-finish-reason` (the wire
      value, verbatim) beside `finish-reason` (the decided one), and
      `finish-reason-evidence` (a detached Hash of how a backend
      decided, when it checked).
    - [2026-09-23] KoboldCpp: truthful stop reasons. KoboldCpp reports
      every stop that is not an EOS token as `length` (koboldcpp.py:
      `"length" if (sr!=1) else "stop"`), so a generation that ended
      on a custom stop sequence was indistinguishable from a truncated
      one and failed the stream — commonly after a tool result, with
      output "" or "\n". On a `length` finish the backend now asks
      `GET /api/extra/perf` and finishes it as `stop` when
      `stop_reason` is 2 (CUSTOM_STOPPER in KoboldCpp's expose.h) and
      `last_token_count` equals the usage `completion_tokens` — or,
      when there is no usage (KoboldCpp's tool-call streams never send
      it), is at most 90% of max_tokens. Anything else, including a
      perf request that fails or outlasts
      `verify-length-finish-timeout` (default 2s), keeps `length`. The
      wire value stays on `provider-finish-reason` and the numbers on
      `finish-reason-evidence`. Off with `verify-length-finish =>
      False`.
    - [2026-09-23] Opt-in message normalization for strict chat
      templates: `message-normalization => 'alternate-roles'` on every
      OpenAI-compatible backend (default 'none'). Mistral-family
      templates, which llama.cpp and KoboldCpp --jinja run as-is,
      reject a conversation whose roles do not alternate ("After the
      optional system message, conversation roles must alternate user
      and assistant roles except for tool calls and results").
      Adjacent plain messages of one role merge, joined by "\n\n";
      the leading system messages become one; a later system message
      becomes user text under a `[System message]` marker line
      (`folded-system-marker`) and merges into the user turn beside
      it, or stands as its own user turn. Tool calls and tool results
      are never merged or crossed. Those templates skip tool turns
      before checking, so a user message directly after tool results
      reads as the user speaking twice: it gets one assistant turn in
      front of it, whose text is `tool-result-acknowledgement`
      (default 'Understood.'; '' inserts none). Nothing is reordered,
      only empty system messages are dropped. The result is a pure
      function of the message list and prefix-stable — appending
      messages only ever grows the last normalized message, and a
      user message after tool results adds the acknowledgement and
      itself after them — so repeated prefixes stay byte-identical
      for prompt caches. Applies to both chat paths (inside
      `wire-message-hashes`, after `system-message-mode`) and to the
      messages both text paths hand their template (new
      `normalized-messages`).
    - [2026-09-23] The text paths no longer hand their template the
      caller's own message array. ChatML, Gemma2, Llama3, Llama4 and
      MistralV7 `pop` the last message when rendering a continuation,
      so `text-completion(@messages, True)` removed the caller's last
      message — from a worker thread, while the request was in flight.
    - [2026-09-23] Request bodies are now serialized deterministically,
      and a JSON Schema's property order is the caller's to choose.
      Every completion path handed Cro the body as a Hash, and Cro
      wrote its members in whatever order the Hash iterated — which
      differs between two hashes holding the same keys and between
      two runs of the same program. That included the schema of a
      structured-output request. llama.cpp and KoboldCpp compile a
      `response_format` / `json_schema` schema into a grammar that
      makes the model write an object's fields in the order the
      schema lists them, so the model wrote its fields in a random
      order per request, and an application could not ask for, say,
      the evidence before the conclusion it supports. Random bytes
      also defeated provider prompt-prefix caching and made a logged
      request impossible to replay exactly.
      All four paths (chat and text, blocking and streaming) on
      OpenAICommon, OpenRouter and KoboldCpp now post bytes from the
      new overridable `_encode-request-body`, with
      `Content-Type: application/json` as before. The rule: arrays
      keep their order; every object's members are sorted by code
      point; a schema's `properties` map follows the sibling
      `required` array (in array order), then the optional
      properties sorted — at every depth, including nested objects,
      `items`, `$defs`, `anyOf` / `oneOf` / `allOf` branches, tool
      `parameters` and KoboldCpp's native `json_schema`. So the order
      of `required` is the order the model writes the fields. Names
      inside property and `$defs`-style maps are never read as
      keywords, and data is never reordered as schema: the values of
      `const` / `default` / `enum` / `examples`, and the whole
      `messages` member, are plain sorted data. The same Hash is now
      the same bytes on every call and in every process, and parses
      to the same data as before.
      The serializer is exported as `LLM::Chat::WireJSON`
      (`to-wire-json`, with `:data-members` and `:!schema-order`) for
      applications that build prompt text or schemas themselves.
      Numbers, booleans, enums and other non-string scalars are still
      written by JSON::Fast; strings are escaped in a single linear
      pass. Two wire defects of the old path go with it: JSON::Fast
      escapes by splicing, which is quadratic in the number of
      characters needing an escape (a 1 MB transcript full of
      newlines took seconds to serialize), and a combining mark right
      after an escape sequence normalized into the escape's letter —
      a newline followed by a combining acute went out as `\ń`, an
      invalid escape the server rejects. Such a mark is now written
      as a `\u` escape.
      With LLM_CHAT_DEBUG set, the chat paths now log REQUEST BODY as
      the exact compact bytes posted rather than a pretty-printed
      re-rendering (new `LLM::Chat::Debug.log-bytes`, which decodes
      only when logging is enabled).
      Two adjacent Hash-order defects: a streamed tool call whose
      `arguments` arrived as an object is now stored as sorted JSON,
      so the string echoed back into the next request is stable; and
      `ToolLoop`'s identical-call limit now recognises two calls with
      equal object arguments built in different orders as identical.
    - [2026-09-15] Add opt-in OpenAICommon request-stream-usage for
      stream_options.include_usage, preserving other stream options and
      blocking request bodies. Both chat and text streaming now retain
      usage-only terminal chunks in Response token counters.
    - [2026-09-14] OpenAI-compatible HTTP failures now preserve bounded
      provider error evidence on the Response before the generation is
      quit. Both 4xx and 5xx Cro HTTP errors read at most 16 KiB of the
      response body, never headers or request content, and expose
      `Response.provider-error` as detached data with status, retained
      byte counts, truncation state, content type, bounded text, and
      OpenAI-style `error.message` / `type` / `code` when present. The
      Response error string is augmented only with those safe structured
      error fields, so existing attempt-failure diagnostics can show a
      provider reason such as "System message must be at the beginning"
      without enabling plaintext debug logs.
      `OpenAICommon` also gains `system-message-mode`, default
      `preserve`. Set it to `leading` for OpenAI-compatible servers that
      reject later system messages: `wire-message-hashes` merges all
      system-message content, in original relative order, into one
      leading system message separated by a documented stable delimiter
      while preserving non-system message order and tool-call fields.
    - [2026-09-02] Cancelling a generation on an OpenAI-compatible
      backend now actually stops it. `cancel` only ever quit the local
      Supplier: the consumer stopped seeing tokens while the backend's
      react loop went on draining the body to the end, so the server —
      llama.cpp's, OpenRouter, any openai-compat gateway — generated
      the whole reply for a request that had been abandoned, and billed
      for it. A caller that cancels freely (a chat UI where every
      re-roll supersedes the last request) accumulated abandoned
      generations on exactly the slow, expensive models where it hurt.
      The cancel was cosmetic; only the consumer stopped looking.
      There is no abort message in the OAI protocol — a client that no
      longer wants a completion hangs up — so all four completion paths
      now hand the Response the means to hang up: `Response.cancel`
      quits the local supply first, synchronously and unconditionally,
      and then drops the HTTP connection via `Cro::HTTP::Response.cancel`,
      which quits the body stream and closes the socket rather than
      returning a quietly-drained connection to Cro's pool. That is the
      same mechanism Cro uses for its own body / total request
      timeouts, so a deadline and a cancel now terminate the upstream
      generation the same way. Transport teardown is best-effort and
      swallows its own failures — a dead endpoint must never make a
      cancel throw on the thread that asked for it.
      Two edges came with it. A cancel raised while the server is still
      prefilling arrives BEFORE there is a response to close, and Cro's
      client exposes no handle on a request in flight; the Response
      records the request and aborts the moment the transport exists,
      so the abort lands on the first response byte instead of being
      lost. That window is the common case, not the corner one. And a
      cancelled Response is no longer classified: Cro reports the
      torn-down body as a `body`-phase timeout, indistinguishable from
      a genuinely stalled one, and filing it as `error-class 'timeout'`
      would have told a fallback policy to retry the request the user
      just abandoned. `.error-class` now stays undefined on a cancel.
      New on `Response`: `_attach-aborter` (backends register transport
      teardown) and `cancel-requested`. The aborter runs at most once
      and is dropped when the Response reaches any terminal state, so a
      cancel that loses the race to `[DONE]` is a genuine no-op and a
      completed Response keeps `.is-success`.
      `KoboldCpp.cancel` is unchanged and still POSTs
      `/api/extra/abort`; it now rides along on the connection close
      too. The two cover different windows — the POST stops a
      generation still in prefill, the close is instant once a body
      exists — so both stay.
    - MAINT: [2026-08-27] moved the shared HTTP-fixture test support from
      C<t/lib/ChatTestKit.rakumod> into C<lib/LLM/Chat/Test/Kit.rakumod>
      (C<LLM::Chat::Test::Kit>). A C<t/lib> directory is its own
      CompUnit repo that starts cold on a fresh C<zef install>, and
      parallel test files pulling the fixture's dependency chain into
      that cold store can race and fail the install outright. C<lib/>
      is staged and precompiled before the test phase runs, so
      shipping test support inside the dist's own namespace dissolves
      the race. This is the fleet-wide convention going forward.

0.10.0  2026-08-21T22:05:38+01:00
    - [2026-08-21] Require Template::Jinja2 0.3.0. The byte-identical
      rendering of upstream GLM and Kimi chat templates leans on its Python
      method-call dispatch and Jinja2-faithful filters; an older renderer
      would produce silently different prompts.
    - [2026-08-20] The OpenRouter backend marks `cache_control`
      breakpoints on outgoing chat bodies, so providers that only
      cache a prompt prefix when the request says where it ends
      (Anthropic, Qwen, Gemini) can actually cache one. Markers go on
      the head message and on the last two `user` / `assistant` turns
      — the trailing pair is what makes the marker written one round
      ago still cover what the next request shares with it — with
      tool results and tool-calling assistants stepped over, and
      never more than OpenRouter's ceiling of four. Implicit-caching
      routes ignore the annotation, so the new `cache-breakpoints`
      attribute defaults to On across a mixed fleet; set it to False
      to send the plain string shape. Conversation state is untouched
      either way: the rewrite happens on the serialized body, so
      `Message.to-hash` and `Message.get-checksum` read the same
      after a request as before it.
      Placing them needed a seam, so `OpenAICommon` gained one:
      `_finalize-request-body` is called on all four completion paths
      with the assembled body, after messages / prompt / tools /
      stream are attached and before the POST. The default returns
      its argument, so nothing changes for backends that don't
      override it.
    - [2026-08-20] New `Response.cached-prompt-tokens`: the slice of
      the prompt a provider served out of its own cache rather than
      processing fresh, which is what tells a caller whether the
      input it is paying full price for is actually being reused.
      Lifted from the OAI-spec `usage.prompt_tokens_details.
      cached_tokens` on any OpenAI-compatible backend, and on
      OpenRouter also from the `/generation` poll — the only place
      the streaming path can pick it up, since that backend
      deliberately asks for no inline usage frame. A value the
      response body reported always wins over the poll's. Undefined
      means the provider said nothing about caching, which is not the
      same as nothing having been cached, so a reported zero stays a
      reported zero.
    - [2026-08-19] A streamed response now always reaches a terminal
      state. Both streaming paths route every ending through one
      place, so `[DONE]`, a `stop` / `tool_calls` finish reason
      followed by the body closing, and a body that closes with
      neither all settle the Response — the last of those as a
      failure with error-class `response`, since a reply cut off in
      transit cannot be told from a reply that ended. Previously that
      third case left the Response neither done nor quit: a consumer
      polling `.is-done` waited out its own timeout, minutes at a
      time, for a stream that had been over from the start. Providers
      that close on the finish_reason chunk without ever sending
      `[DONE]` are common, and they now complete promptly and
      successfully instead. The `length` / `content_filter` /
      unknown-reason failures are unchanged.
      Fixes a hook-ordering bug on the way: text-completion-stream's
      `[DONE]` arm completed the supply BEFORE `_on-stream-complete`,
      so a subclass's post-stream metadata landed after consumers had
      been told the response was finished. Both paths now fire the
      hook first, as the hook has always documented.
    - [2026-08-19] New `Response.dropped-frames`: streamed `data:`
      lines that fail to parse are still skipped — providers
      interleave junk into healthy streams — but they are now counted,
      and a stream that dropped frames while assembling tool calls
      FAILS instead of succeeding. A hole in prose costs a few words
      and is visible to whoever reads the reply; a hole in a tool
      call's `arguments` is spliced out of a JSON string that is then
      concatenated blind, and what comes out can be valid JSON saying
      something the model never asked for. Nothing downstream can
      detect that, so it no longer reaches `is-success`. Prose-only
      streams keep their success and expose the counter. SSE
      heartbeats and empty `data:` lines are not counted.
    - [2026-08-19] Streaming tool-call deltas that carry no `index`
      are assembled correctly. The OAI spec puts an `index` on every
      fragment and it is still authoritative when present, but some
      providers omit it, and the old fallback used the number of calls
      seen so far — which sent every argument fragment to a fresh
      call. One tool call arrived shattered into N, each holding an
      unparseable slice of the JSON. The fallback now reads `id`,
      which the spec sends only on a call's first fragment: an `id`
      starts a new call, an arguments-only fragment continues the one
      in flight.
    - [2026-08-19] `LLM::Chat::Template::Jinja2.render` presents
      tool-call `function.arguments` to the template as a decoded
      object when they are a JSON object, matching what HuggingFace
      chat templates are written against (transformers passes a dict).
      GLM-family templates iterate `arguments.items()` and rendered
      nothing at all against the JSON string the OpenAI wire format
      uses — a silently truncated prompt, and for a token counter an
      undercount of the largest part of it. The messages themselves
      are untouched: the render works on a copy, so `to-hash` and
      every request body built from a conversation still carry the
      string. Arguments that are not a JSON object — empty, a fragment
      cut short mid-stream, an array, a bare scalar, or a Hash a
      caller supplied directly — are passed through exactly as they
      are.
    - [2026-08-13] `LLM::Chat::TokenCounter.get-request-count` counts a
      complete selected-model request: optional runtime context head, stored
      conversation, optional runtime context tail, and tool catalogue rendered
      through the model's actual template, then tokenized without a second
      special-token pass. This is the exact-counting seam for models such as
      Kimi K2.7 whose tool declarations live in the chat template. Existing
      `get-conversation-count` remains conversation-only and unchanged.
      Kimi's published Jinja fallback serializes ordinary OpenAI tool
      declarations with compact `tojson`; `tools_ts_str` is an optional
      preformatted override, so no model-specific duplicate renderer is used.
      Runtime context may be supplied as its natural string halves or as
      prebuilt system messages. Rendered prompts opt into declared special
      tokens explicitly, which is required by deny-by-default tiktoken
      backends and harmless for Hugging Face JSON tokenizers.
    - [2026-08-13] Add `LLM::Chat::Template::DeepSeekV4`, a byte-compatible
      Raku port of the full prompt-encoding half of DeepSeek's pinned DSV4
      reference implementation. It supports chat/thinking modes, reasoning
      effort, context and thinking removal, developer/latest-reminder turns,
      quick tasks, response formats, OpenAI-format tools/calls, DSML argument
      encoding, and merged/call-ordered tool results. `encode-json` preserves
      JSON object order for differential goldens; `encode-messages` accepts
      ordinary associative message structures; `render` adapts existing
      `Conversation::Message` objects. Completion parsing is intentionally not
      included. The four official Python outputs are pinned as offline golden
      tests with DeepSeek's MIT attribution.
    - [2026-08-12] A Cro timeout is now classified BY THE PHASE it
      expired in. `X::Cro::HTTP::Client::Timeout` carries a `phase` of
      `connection`, `headers` or `body`, and OpenAICommon's classifier
      used to file all three as error-class `timeout` — the ADVANCE
      bucket. A connection-phase timeout is Cro's 30-second connect
      budget running out: nothing was sent, the endpoint said nothing,
      and the only evidence is a network that was unwell. That is now
      error-class `connection`, the RETRY-SAME bucket. Headers- and
      body-phase timeouts keep `timeout`: a backend that accepted the
      connection and then would not answer has said something about
      itself, and the next backend answers sooner than a backoff ends.
      The lumping was fatal on a ONE-BACKEND chain, where `advance`
      means the chain is exhausted: a single flaky connect ended the
      run having made exactly one call. `classify-error`'s own bucket
      table is unchanged — the split is at the source, which is where
      the information is.
      The classification test no longer re-implements the classifier
      it is testing; it calls the real private method through
      `^find_private_method`, so the two cannot drift.
    - [2026-08-12] `LLM::Chat::Template::Jinja2.render` takes an
      optional `:@tools` — the tool declarations the request will
      carry, in the provider's own shape — and passes them to the
      template as its `tools` variable. Real HuggingFace chat
      templates render a tools block when they are given one, so a
      render meant to match what the server sees needs them.
      Passed ONLY when non-empty: `tools => []` and no tools at all
      both render exactly as they did before the parameter existed
      (pinned by a test, because a template asking
      `{% if tools is defined %}` would otherwise take a different
      branch for a caller who named nothing). Deliberately NOT wired
      into `LLM::Chat::TokenCounter`: the declarations are a
      per-request constant that provider-reported usage has already
      been charged for, and counting them per message would bill them
      twice.

0.8.1  2026-08-09T20:57:18+01:00
    - META fix: Response::OpenRouter and Response::OpenRouter::Stream were
      missing from provides and listed under DEPENDS instead. On a machine
      without install history, zef resolved those module names to the fossil
      LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
      pin only builds with nmake — breaking every fresh Windows install
      (caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
      published carries the broken META and should not be depended on.

0.8.0  2026-08-09T19:41:06+01:00
    - [2026-08-09] New C<LLM::Chat::Retry> and
      C<LLM::Chat::Retry::Exceptions>: the retry/fallback policy that
      has run inside C<LLM::Data::Inference::Task> since 0.5, factored
      out so a second executor can share it instead of copying it.
      C<LLM::Chat::Retry> exports five pure subs — C<classify-error>
      (the abort / retry-same / advance bucket table),
      C<retry-backoff> (C<min(2 ** (n - 1) + jitter, cap)>, with the
      anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
      deterministic tests), C<sleep-with-cancel> (chunked sleep that
      consults a cancel hook before EVERY chunk including the first,
      and returns a Bool rather than throwing, because only the caller
      knows which exception type its layer promises),
      C<attempt-record> and C<telemetry-payload> (which duck-types the
      response with C<.?> and therefore does not import
      C<LLM::Chat::Backend::Response>).
      C<LLM::Chat::Retry::Exceptions> provides
      C<X::LLM::Chat::Retry::Exhausted> and its C<Truncated> /
      C<TimedOut> subclasses plus C<Cancelled> (deliberately NOT an
      C<Exhausted>, so a user cancel can never be swallowed by a
      dead-letter handler), and is now the canonical home of the
      C<item-retryable> advice contract. Those four are plain global
      classes rather than C<is export>ed ones on purpose: C<is export>
      on a nested-name class also exports its LEAF name, so two
      modules declaring an C<X::Something::Exhausted> cannot be
      imported into the same scope at all — which would have made the
      shared hierarchy unusable next to a subclassing one.
      Deliberately NOT shared: the
      retry loop itself, the abort bucket's C<die> message, hook
      payloads and hook shielding — those are shaped by the layer that
      owns them. B<No existing module changed>: this release is purely
      additive, every previous test file is untouched, and
      C<LLM::Data::Inference> 0.9.0 migrates onto it (its exception
      types become subclasses of these, so both hierarchies keep
      matching).
    - [2026-08-08] Fix: a streamed completion could be KILLED MID-
      GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
      (C<chat-completion-stream> and C<text-completion-stream>) did
      C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
      but TCP splits a body wherever it likes — including through the
      middle of a multi-byte UTF-8 sequence. C<.decode> on the half
      that ends mid-sequence throws "Malformed termination of UTF-8
      string", which landed in the streaming CATCH and quit the
      Response with error class 'unknown', discarding everything
      generated so far. Buffering text could not fix it: the throw
      happened before the text reached the buffer. The bytes now go
      through an incremental C<Encoding::Decoder> (new shared internal
      C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
      sees the line endings that are actually on the wire), which
      keeps an unfinished sequence to itself until the bytes
      completing it arrive. Framing, heartbeat skipping, C<[DONE]>
      handling, malformed-chunk tolerance and reasoning/content
      accumulation are untouched. Emoji, CJK and curly quotes are
      ordinary in model output, so this was an intermittent
      production crash for anything but plain ASCII.
    - [2026-08-08] Fix: non-ASCII in a response body could come back as
      mojibake (café -> cafÃ©). Cro's C<body-text-encoding> returns the
      LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
      charset, and C<Cro::MessageWithBody.body-text> loops that list
      with no C<last> — so the latin-1 attempt, which cannot fail
      whatever the bytes are, always overwrote the correct utf-8
      decode. Many OpenAI-compatible servers answer
      C<Content-Type: application/json> with no charset parameter, and
      model output is full of non-ASCII (names, curly quotes, emoji,
      CJK). OpenAICommon now decodes response bytes itself via new
      internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
      utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
      latin-1 kept only as a fallback for when the utf-8 decode
      actually throws — a proxy's non-JSON error page, say. Affected
      the 4xx error-body read that feeds LLM::Chat::Debug, which is
      where an operator reads a rejection reason.
    - [2026-08-08] The blocking C<chat-completion> / C<text-completion>
      paths and OpenRouter's C</generation> metadata lookup now parse
      their JSON from the raw bytes instead of C<await $res.body>.
      Cro's response body-parser selector reaches the correct parser
      only while the server labels the body C<application/json>; a
      server that answers JSON as C<text/plain> fell through to
      TextFallback and produced a mojibaked Str, which the completion
      path then indexed as a Hash and SILENTLY emitted as empty
      content. Well-behaved servers see no change. Transport failures
      still propagate as transport exceptions, so error classification
      is unchanged.

0.7.1  2026-07-27T15:27:39+01:00
    - [2026-07-27] LLM::Chat::Backend::Mock gains
      C<&.finish-reason-producer>, an optional
      C<(Int $call-index --> Str)> callback mirroring
      C<&.error-producer>'s per-call indexing. A defined Str return is
      stamped on the Response via C<_set-finish-reason> before the
      emission task starts, while the call still SUCCEEDS — the exact
      shape of a real blocking completion that ran out of completion
      budget (HTTP 200, well-formed but partial body, finish_reason
      'length'). Undefined returns leave C<finish-reason> unset, so
      existing mocks are byte-for-byte unchanged. A scripted error
      wins over a scripted finish reason (a failed call never produced
      a body). Wired into the non-streaming paths only —
      C<chat-completion>, and C<text-completion> by delegation —
      because the streaming paths already model finish reasons the way
      the real transports do. Lets downstream consumers
      (LLM::Data::Inference's truncation policy) test truncation
      handling without a live model.
    - [2026-07-27] Fix: OpenAICommon's C<text-completion-stream> acted
      on a terminal C<finish_reason> without recording it, so a stream
      cut off by the token budget quit with error class 'response' but
      an UNDEFINED C<.finish-reason> — consumers could not tell budget
      exhaustion apart from a content filter or an unknown reason.
      It now calls C<_set-finish-reason> before dispatching on the
      reason, matching C<chat-completion-stream>.

0.7.0  2026-07-21T03:09:53+01:00
    - [2026-07-03] Structured outputs: Backend::Settings gains
      C<json_schema> (a plain-Hash JSON Schema, default undefined) and
      C<json_schema_name>. OpenAI-compatible backends (incl.
      OpenRouter) send it as C<response_format: { type: 'json_schema',
      json_schema: { name, strict: false, schema } }> — best-effort,
      provider-dependent enforcement; KoboldCpp sends the native
      C<json_schema> generate field, which it compiles to a grammar
      for sampler-level enforcement. Unset schema changes no request
      bytes. Callers should keep parse validation as the backstop for
      providers that ignore response_format.

0.6.0  2026-06-22T23:33:11+01:00
    - Add LLM::Chat::ToolLoop, a generic streaming tool-call
      round loop. It sends OpenAI-format tools to any chat backend,
      executes returned tool calls through a caller-supplied
      callback, appends assistant/tool turns, and continues until a
      final answer or guardrail limit is reached.
    - OpenAI-compatible streaming now assembles `delta.tool_calls`
      chunks and treats `finish_reason: tool_calls` as successful.
      This enables KoboldCpp streamed tool calling.
    - Conversation messages can now serialize assistant `tool_calls`
      and tool-result `tool_call_id` fields for follow-up chat
      completion requests.
    - LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
      failure-proof: the local response stream closes FIRST,
      synchronously, then the upstream POST /api/extra/abort fires on
      a worker thread, best-effort, with network failures swallowed.
      The old ordering awaited the abort before closing locally, so a
      busy KoboldCpp froze the caller for the full HTTP round-trip
      (interactive cancels hung the UI) and an unreachable one made
      cancel throw without ever closing the stream. cancel now
      returns the background Promise so callers/tests can await the
      abort attempt. Also fixes the abort URL: stripping the /v1
      suffix used to leave its leading slash behind, producing
      //api/extra/abort.

0.5.1  2026-04-29T23:47:12+01:00
    - Bump Github Actions to use node 24+
    - LLM::Chat::Backend::OpenAICommon gains a symmetric
      `_on-blocking-complete` hook on the non-streaming path,
      mirroring the existing `_on-stream-complete` contract: fires
      after the response body has been parsed and `_lift-usage`
      has lifted OAI/provider usage fields, before
      `$response.done`. Default implementation is a no-op.
      Subclasses use it to attach post-call metadata that isn't
      in the body itself — symmetric with what was already
      possible on streams.
    - LLM::Chat::Backend::OpenRouter wires the new hook through
      to the same `/generation?id=...` lookup the streaming path
      already uses, so blocking callers (e.g. App::Storygen, which
      calls `chat-completion` rather than `chat-completion-stream`)
      now see `.cost` populated by the time `$response.done` fires.
      Pre-fix this was the silent regression from dropping
      `usage: { include: true }` — `Response::OpenRouter.cost`
      stayed Nil on every blocking call.
    - LLM::Chat::Backend::OpenRouter refactor: the lookup logic
      is now in a private `!fetch-generation-metadata` helper that
      both `_on-stream-complete` and `_on-blocking-complete`
      delegate to. No behaviour change on the streaming path.
    - Tests — t/12-openrouter-backend.rakutest gains a subtest
      covering the defensive guards on both completion hooks
      (no-op when generation-id is undefined; no crash when the
      response isn't OR-augmented). Plan goes 11 → 12; total
      LLM::Chat tests 130.

0.5.0  2026-04-27T22:52:25+01:00
    - LLM::Chat::Backend::OpenRouter request shape now mirrors
      SillyTavern's wire bytes verbatim. Removed two body fields
      that were causing OpenRouter's upstream router to hold 200 OK
      headers indefinitely against some providers (~80% header-phase
      timeouts in App::Cantina vs ~0% in SillyTavern on the same
      models / keys / network):
        * `usage: { include: true }` — no longer sent.
        * `stream_options: { include_usage: true }` — no longer sent
          (also removed from OpenAICommon.chat-completion-stream so
          all OAI-compatible streams now match this shape).
        * `reasoning: { effort, enabled }` — `enabled` key dropped;
          we now send only `{ effort }` when reasoning_effort is
          configured, matching ST.
      Added on every request, also matching ST:
        * `include_reasoning: Bool` — Boolean parity with ST's flag.
        * `top_k` — plumbed from Settings into the OAI body.
      `repetition_penalty` is now omitted when at the default 1.0
      (was previously sent unconditionally).
    - Cost telemetry that the inline `usage: { include: true }`
      block used to carry now arrives via a one-shot post-stream
      GET against `/generation?id=...` after `[DONE]`. Lookup is
      async and best-effort; on failure $resp.cost stays Nil rather
      than escalating. Lifts cost, provider-name, and (when not
      already populated from the stream) prompt/completion tokens.
      Latency: ~50–200ms after .is-done becomes True before .cost
      is readable. New hook _on-stream-complete on OpenAICommon so
      future provider subclasses can do the same kind of
      post-stream metadata fetch.
    - LLM::Chat::Backend::OpenAICommon stream parser now buffers
      bytes across body-byte-stream emissions and splits on the
      SSE `\n\n` event delimiter before parsing, instead of
      decoding+parsing each TCP chunk independently. Pre-fix, a
      `data: {...}` JSON object split across two TCP packets would
      crash from-json on the truncated half and terminate the
      stream as 'unknown' error class. Both chat-completion-stream
      and text-completion-stream got the fix. Heartbeat / SSE
      comment lines (`: OPENROUTER PROCESSING`) are dropped per
      spec — never produce a chunk.
    - LLM::Chat::Backend::OpenAICommon.!classify-exception no
      longer string-matches "timeout" / "timed out" in the default
      arm; only X::Cro::HTTP::Client::Timeout maps to error-class
      'timeout' now. Substring matching was masking unrelated
      errors (JSON parse failures, stream-cancel messages) as
      header timeouts. Connection-error pattern (refused / reset /
      DNS / unreachable) stayed — those don't have a typed
      exception class to discriminate on.
    - LLM::Chat::Debug log format gains elapsed-ms timestamps for
      streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
      EXCEPTION lines all carry "+Nms" relative to the call start
      so latency can be diagnosed without external instrumentation.
      Existing log labels unchanged.

0.3.0  2026-04-23T15:56:28+01:00
    - LLM::Chat::Backend::Response gains structured error metadata:
      $.error-status (Int HTTP code) and $.error-class (Str —
      'http' / 'timeout' / 'connection' / 'response' / 'unknown').
      Populated via _set-error-info(:$class, :$status) alongside
      the existing .quit path. Lets consumers branch on error kind
      without regex-parsing raw messages — used by the
      LLM::Data::Inference::Task model-fallback policy.
    - LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
      exceptions (X::Cro::HTTP::Error — picks up status off
      .response.status; X::Cro::HTTP::Client::Timeout) plus
      heuristic socket-error detection into the Response's error
      fields before quitting. Finish-reason quits (length /
      content_filter / unknown) are tagged error-class => 'response'
      so the fallback layer advances on them.
    - LLM::Chat::Backend::Mock gains &.error-producer — an optional
      (Int $call-index --> Hash) callback that scripts per-call
      failures for fallback / retry tests. Returning a hash like
      { class => 'http', status => 500, message => 'x' } fails
      that call without consuming a slot from @.responses. Also
      exposes $.call-index (monotonic per-backend call count,
      bumped on every completion regardless of outcome).

0.2.6  2026-04-13T17:03:42+01:00
    - LLM::Chat::Backend::Mock: new test-only backend that returns
      canned responses in order. Supports streaming (default splits
      on whitespace, configurable via :token-splitter), optional delay
      between tokens via :stream-delay, and an :initial-delay (10ms
      default) so consumers can attach taps before tokens flow.
      :fail-on-empty makes the backend die when the response queue
      is exhausted instead of repeating the last entry. Useful for
      exercising error paths in downstream consumers.
      Recording: every completion call is logged to @.recorded-calls
      as a hash with kind / messages / tools / response / at. Tests
      can assert on what reached the backend, not just what came
      back — catches prompt-assembly and template-substitution
      regressions. clear-recorded-calls resets the log between
      test phases.

0.2.5  2026-04-09T12:30:48+01:00
    - Optional :@tools parameter on chat-completion and chat-completion-stream
    - Response.tool-calls and has-tool-calls for detecting LLM tool call requests
    - Response.finish-reason field

0.2.4  2026-04-09T05:15:41+01:00
    - CI: exclude Windows (Tokenizers Rust FFI build not yet supported)

0.2.3  2026-04-09T05:09:48+01:00
    - Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
    - Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
    - Add docs/Readme.rakudoc

0.2.2  2026-04-09T04:59:41+01:00
    - Add stub LLM::Chat to make mi6 stop renaming the module.

0.2.1  2026-04-09T04:41:36+01:00
    - Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
    - from-tokenizer-config class method loads templates from tokenizer_config.json
    - Supports bos_token, eos_token passthrough
    - Continuation mode maps to add_generation_prompt=false

0.2.0
    - Previous releases
