Revision history for LLM::Data::Pipeline

0.7.1  2026-08-21T22:46:34+01:00

    - DOCS: Operational recipes (2026-08-21): the Step::Items and
      run-until-done sections made two promises — the "coordinator model,
      checkpoint v2, DLQ schema, and failure-mode table" and the
      "Operational recipes section" — that pointed nowhere. Both now land
      on real sections; the references link in-page (#runner-internals,
      #operational-recipes).
      - New C<Runner internals> under LLM::Data::Pipeline::Runner: step
        drive loop (skip / checkpoint-before-step-completed / fail-on-dead
        ordering, Cancelled is not run-failed), plain-step retry contract,
        the single-writer coordinator model (one inbox loop, worker-side
        C<item-retryable> probing, degree-throttled dispatch), the
        checkpoint v2 schema including fingerprint and the
        results/partials pruning, the DLQ journal schema (dead vs
        requeued records, DLQ-first write order, deliberately fatal
        append failure), and a nine-row failure-mode reference table.
      - New C<Operational recipes> h2: C<run>/C<resume>/C<run-until-done>
        as three failure lifetimes; in-process retries vs an external
        supervisor (complementary, not competing); DLQ triage
        (transient / poison / systematic); the retry-dead decision table —
        spelling out that the digest check replays items exactly as they
        were dead-lettered, so edited content CheckpointDrifts by design
        — and checkpoint sizing (coalescing, pruning, partial shape).
      - README.md hand-mirrored to match; run C<mi6 build> to regenerate
        authoritatively.

0.7.0  2026-08-10T20:07:42+01:00
    - ITEM-RETRY ADVICE (2026-07-29): an item failure can now tell the
      engine that retrying it is pointless. The RetryPolicy answers "how
      many attempts may this item have?" and nothing could answer "would
      another attempt DO anything?" — so a deterministic death (the
      motivating one: an LLM completion cut off by max_tokens, which
      re-issues identically at the same max_tokens) burned its whole
      budget, three times the latency and spend, on the way to the same
      dead-letter record.
      - The worker probes the caught exception for C<item-retryable> and
        ships a plain Bool to the coordinator. An advised-False failure
        dead-letters IMMEDIATELY, on that attempt, whatever budget the
        policy had left.
      - Duck-typed in both directions: the probe is C<.?>-based and
        defaults to True, so every exception that has never heard of the
        contract behaves exactly as it did in 0.6.0; Pipeline imports no
        exception type to ask, and the advising library imports nothing
        of Pipeline's. C<X::LLM::Data::Inference::Truncated> (0.8.0) is
        the first opt-in.
      - The probe happens WORKER-side because the exception object never
        crosses the inbox Channel — the coordinator has only the
        stringified type name and message, and now this Bool. Nothing
        thread-hostile is shared and the DLQ record shape is unchanged.
      - Advice cannot malfunction into a dead letter: an C<item-retryable>
        that throws is read as retryable, and a non-Bool return is coerced
        (C<.so>) on the worker rather than reaching the single-writer
        coordinator.
      - Attempt accounting stays true — a first-attempt non-retryable
        death records attempts 1 and one attempt-history entry, not the
        budget it declined to spend. Event train is the usual
        C<item-failed> then C<item-dead-lettered>, with no second
        C<item-started>.
      - Whole-step retries and run-until-done are untouched: this is
        strictly an item-engine contract.

0.6.0  2026-07-28T01:44:53+01:00
    - RESUMABLE ITEMS (2026-07-28): an item that is a run of sequential
      sub-units can now persist its progress DURING the item, and is handed it
      back on the next attempt. The motivating failure: one item was an 89-turn
      chapter rewrite, so a turn failing mid-item made the retry re-run all 89
      turns from the first — the sub-unit counter visibly restarted.
      - Runner.partial-sink(:$step!, :$key!) mints a thread-safe save closure
        for one item (a Channel send, safe from the worker thread), binding the
        active inbox AT MINT TIME exactly as telemetry-sink does — so a late
        save from an abandoned attempt can never land in another step's store.
      - process-item receives the last saved Hash as :%partial (%() = start
        fresh). It is a NAMED argument, so every pre-0.6.0 implementation keeps
        working unchanged via its implicit *%_.
      - Partials are deliberately NOT events: one can be an item's entire
        generated output, which has no business on the on-event stream or in
        the dead-letter journal. No new event kind; seq stays contiguous.
      - Persistence rides the existing coalescing gate with a fifth trigger
        string, 'partial', UNFORCED — so checkpoint-every / checkpoint-interval
        throttle it like item terminals. This is sound because an in-process
        retry is handed the IN-MEMORY partial and never reads the checkpoint:
        coalescing can only widen the kill-resume redo window, never the retry
        one.
      - Every partial is normalized through JSON at receipt, so what a retry
        gets from memory is identical to what a resume gets off disk; a partial
        that cannot round-trip fails loudly at its first save.
      - Clearing: on item success, on dead-letter, on a retry-dead requeue
        (belt and braces — the dead arm already dropped it), and when
        retry-dead reopens a completed step. A completed step's step-state
        carries "partials": {}. Saves naming another step, or a key that is
        already terminal, are dropped with a note.
      - Cancellation KEEPS accepting partials: a run cancelled 60 turns into a
        chunk resumes at turn 61. That is the payoff, not a leak.
    - Checkpoint format stays v2 — partials are purely additive. A pre-0.6.0
      checkpoint loads with every item starting fresh, and a 0.6.0 checkpoint
      loads into an older Runner, which ignores the key.
    - at-least-once now narrows to the SUB-UNIT boundary for steps that save:
      sub-units covered by a persisted partial are not re-run, the ones after
      the last save are, and external side effects must still be idempotent.
    - Pod6: a "Resumable items" section in the Runner (contract, cadence,
      clearing rules, normalization, checkpoint-size trade-off), the same
      contract from the implementor's side in Step::Items (including "treat an
      unrecognised partial as absent, never die"), the checkpoint v2 example
      and the event-taxonomy trigger list.
    - Tests: t/08-items-partial.rakutest (10 subtests) — retry resumes with no
      sub-unit re-run, fresh items get %(), cancel-then-resume across two
      Runners, clearing on success/dead, pre-0.6.0 checkpoints, coalescing plus
      a frozen mid-run checkpoint proving the resume redoes only the
      unpersisted tail, wrong-step and terminal-key drops, retry-dead over an
      injected stale partial, event-kind/seq invariance, and the no-active-step
      drop. PipelineTestKit's ScriptedItemStep grows a 'sub-fail' script kind
      plus received-partial-for / sub-exec-count-for.

0.5.1  2026-07-27T15:27:05+01:00
    - Item steps now emit one progress event at ACTIVATION (2026-07-26): the
      Runner emits it as soon as the step's item set is materialized and the
      ready queue is computed, before the first dispatch, so a consumer sees
      0/N for a fresh stage and done/N for a resumed one instead of nothing
      until the stage's first item terminal. in-flight is 0 and pending covers
      everything not already recorded in the checkpoint; payload semantics are
      otherwise identical to the terminal-driven emissions (same helper).
    - Ordering: an item step's activation progress precedes its first
      item-started. A step skipped as already-complete on resume never
      activates and emits no progress.

0.5.0  2026-07-21T03:12:48+01:00
    - run-until-done: whole-run supervision that resumes a pipeline to completion
      across process-local retries (attempt 1 resumes an existing checkpoint,
      else runs; later attempts resume). Transient failures back off via
      &.schedule-after and retry; Cancelled / CheckpointDrift / plan-validation
      failures rethrow immediately, as does ItemsDead unless :retry-dead (which
      makes it retryable). Budget exhaustion rethrows the last error unchanged.
    - run-retry event promoted from reserved to emitted (0.5.0): a supervision
      event delivered via on-event, carrying seq 0 (outside any single run's
      sequence) and the lineage run-id; payload attempt/max-attempts/delay/
      error/exception.
    - Documented that DLQ dedupe by (run-id, step, key) is sound only because
      run-id is a stable lineage id (minted by run, adopted by every resume).
    - Pod6: run-until-done reference + an Operational Recipes section
      (in-process retries vs external supervisor, DLQ triage with JSONL::Reader /
      jq, the retry-dead decision guide).

    - Step::Items: parallel, per-item pipeline steps. Implement items /
      process-item / finalize instead of execute; the Runner drives them through
      a single-coordinator engine (N workers pull from a work Channel, message
      results back through one inbox; the coordinator owns all state, so event
      seq, checkpoint, and DLQ writes are single-writer with no locks). Per-step
      degree + item-retry overrides; :degree(1) is the sequential path.
    - Context freeze/thaw: the Context is frozen during item processing so a
      worker-thread mutation dies loudly instead of racing.
    - Checkpoint v2: version/run-id/step-state/updated-at, with per-item and
      dead-letter checkpoints (coalesced via checkpoint-every / checkpoint-
      interval; dead-letter/step/cancel writes are never coalesced away). v1
      checkpoints are still accepted on resume. run-id is now a stable lineage id
      adopted from the checkpoint on resume.
    - Dead-letter queue: a crash-safe JSONL journal (foo.checkpoint.json →
      foo.dlq.jsonl) written DLQ-first-then-checkpoint on exhaustion; append
      failure is fatal. Records carry item-digest, attempt-history, error, and an
      optional inference block sourced from telemetry-sink.
    - resume(:retry-dead): requeue dead items, reopen a completed step and
      invalidate every later step, verifying item digests (CheckpointDrift on
      mismatch).
    - Cooperative cancellation via &.is-cancelled: drain in-flight, checkpoint,
      emit run-cancelled, throw X::LLM::Data::Pipeline::Cancelled.
    - fail-on-dead item steps throw X::LLM::Data::Pipeline::ItemsDead after the
      step is finalized and checkpointed.
    - Item/Progress/Telemetry/RunCancelled/ItemRequeued events promoted from
      reserved to emitted; step-completed carries items-done/items-dead for item
      steps.
    - New depends: JSONL, Digest::SHA256::Native (Pipeline is now tier 1).

    - Step retry infrastructure. New LLM::Data::Pipeline::RetryPolicy
      (exponential backoff with cap + jitter; deliberately slower than the
      inference layer's per-call backoff). New LLM::Data::Pipeline::Exceptions:
      X::LLM::Data::Pipeline::StepExhausted thrown on exhausted retries (carries
      step, attempts, last-error); Cancelled/ItemsDead/CheckpointDrift declared
      for later stages.
    - Runner: &.step-retry policy (default max-attempts 1 = unchanged
      abort-on-first-failure, rethrowing the step's ORIGINAL exception; retry is
      opt-in). With retries enabled, failed attempts emit step-failed
      (will-retry, retry-delay) and step-retry, backing off via the injectable
      &.schedule-after; exhaustion throws StepExhausted. Injectable &.now / &.
      schedule-after for deterministic tests.
    - step-failed / step-retry events promoted from reserved to emitted;
      step-failed payload gained will-retry and retry-delay.
    - NOTE: a failed attempt's Context mutations are not rolled back — step
      retry implies the step is idempotent or tolerant of its own partial
      writes.

    - Unified typed event stream (LLM::Data::Pipeline::Event): immutable event
      classes over a shared role carrying seq/at/run-id + JSON-safe to-hash.
      Full taxonomy declared now; run-started, step-skipped, step-started,
      step-completed, checkpoint-written, run-completed and run-failed emitted.
    - Runner: new &.on-event synchronous callback (strict per-run seq order,
      handler exceptions shielded) and events() Supply adapter (done fires on
      success and failure). run-failed wraps the step loop and rethrows the
      original exception unchanged.
    - Per-run run-id minted for every run()/resume() call.
    - &.on-step retained as a deprecated shim over the event stream (removed at
      1.0); behavior is byte-for-byte identical to prior releases.

0.1.1  2026-04-29T23:51:00+01:00
    - Bump Github actions to use node 24+
    - Added auth specifiers to META6.json dependencies

0.1.0  2026-04-07T18:51:48+01:00
    - Initial release
    - Step role with requires/optional/provides declarations
    - Context data bag with JSON snapshot/restore
    - Plan with dependency validation
    - Runner with sequential execution, checkpointing, and resume
    - Progress callbacks (start/complete/skip events)
