=begin pod

=head1 NAME

LLM::Data::Pipeline - Generic pipeline framework for sequential data processing with checkpointing

=head1 SYNOPSIS

=begin code :lang<raku>

use LLM::Data::Pipeline;
use LLM::Data::Pipeline::Context;
use LLM::Data::Pipeline::Step;
use LLM::Data::Pipeline::Plan;
use LLM::Data::Pipeline::Runner;

class MyStep does LLM::Data::Pipeline::Step {
    method name(--> Str:D) { 'my-step' }
    method description(--> Str:D) { 'Does something useful' }
    method requires(--> List) { ('input',) }
    method optional(--> List) { ('config',) }
    method provides(--> List) { ('output',) }
    method execute(LLM::Data::Pipeline::Context:D $ctx --> Nil) {
        my $input = $ctx.get('input');
        $ctx.set('output', "processed: $input");
    }
}

my LLM::Data::Pipeline::Plan $plan .= new;
$plan.add-step(MyStep.new);

my LLM::Data::Pipeline::Context $ctx .= new;
$ctx.set('input', 'hello');

my LLM::Data::Pipeline::Runner $runner .= new(
    on-event => -> LLM::Data::Pipeline::Event $e {
        say "{$e.seq}\t{$e.kind}";
    }
);
$runner.run($plan, $ctx, :checkpoint-path('checkpoint.json'.IO));

say $ctx.get('output');  # "processed: hello"

# Resume from checkpoint (completed steps arrive as step-skipped events)
my $ctx2 = $runner.resume($plan, 'checkpoint.json'.IO);

=end code

=head1 DESCRIPTION

LLM::Data::Pipeline provides a domain-agnostic framework for building sequential
data processing pipelines with automatic checkpointing and resume capability.

=head2 LLM::Data::Pipeline::Step (Role)

Any class can be a pipeline step by doing this role.

=begin code :lang<raku>

method name(--> Str:D) { ... }           # Unique step identifier
method description(--> Str:D) { ... }    # Human-readable description
method requires(--> List) { () }         # Context keys this step must have
method optional(--> List) { () }         # Context keys this step can use
method provides(--> List) { ... }        # Context keys this step writes
method execute(Context:D $ctx --> Nil) { ... }  # Do the work

=end code

=head2 LLM::Data::Pipeline::Context

String-keyed data bag flowing through the pipeline. Values must be JSON-serializable.

=begin code :lang<raku>

my LLM::Data::Pipeline::Context $ctx .= new;
$ctx.set('key', 'value');
$ctx.get('key');              # 'value'
$ctx.has('key');              # True
$ctx.keys;                   # Sorted list of all keys

# Serialization (for checkpointing)
my %snap = $ctx.snapshot;    # Deep-copy Hash via JSON round-trip
my $restored = LLM::Data::Pipeline::Context.from-snapshot(%snap);

=end code

=head2 LLM::Data::Pipeline::Plan

Ordered list of steps with dependency validation.

=begin code :lang<raku>

my LLM::Data::Pipeline::Plan $plan .= new;
$plan.add-step(StepA.new);
$plan.add-step(StepB.new);
$plan.steps;                          # List of steps in order
$plan.validate($ctx);                 # Dies with diagnostic if deps unsatisfied

=end code

Validation walks steps in order, tracking available keys from each step's C<provides>.
Reports all missing keys and duplicate step names in one error.

=head2 LLM::Data::Pipeline::Runner

Executes a Plan against a Context with checkpointing, surfacing everything
observable as a typed L<event stream|#LLM::Data::Pipeline::Event>.

=begin code :lang<raku>

my LLM::Data::Pipeline::Runner $runner .= new(
    on-event => -> LLM::Data::Pipeline::Event $e {
        given $e {
            when LLM::Data::Pipeline::Event::StepStarted {
                say "→ {$e.step} ({$e.ordinal}/{$e.total})";
            }
            when LLM::Data::Pipeline::Event::StepCompleted {
                say "✓ {$e.step} in {$e.duration.round(0.01)}s";
            }
            when LLM::Data::Pipeline::Event::RunCompleted {
                say "done: {$e.steps-run} run, {$e.steps-skipped} skipped";
            }
        }
    }
);

# Run from beginning (validates first)
my $ctx = $runner.run($plan, $ctx, :checkpoint-path($path));

# Resume from checkpoint (completed steps arrive as step-skipped events)
my $ctx = $runner.resume($plan, $checkpoint-path);

=end code

Checkpoints are JSON files written after each step, containing completed step names
and the full context snapshot. Writes use temp file + rename for atomicity.

=head3 Observing a run

C<&.on-event> is the primary hook: a synchronous callback invoked once per event
on the run thread, in strict C<seq> order. Handler exceptions are shielded (logged
via C<note>) so a broken observer can neither abort the run nor cause later events
to be dropped.

A secondary C<method events(--> Supply)> mirrors the same stream for reactive
consumers. Taps run B<synchronously on the run thread>, so tap before calling
C<run>/C<resume>; the Supply receives C<done> when the run finishes, on normal
completion B<and> on failure (C<run-failed> is emitted first, then the exception
is rethrown, then C<done> fires). Bridge with C<.Channel> for thread isolation.

=begin code :lang<raku>

$runner.events.tap(
    -> LLM::Data::Pipeline::Event $e { say to-json($e.to-hash) },
    done => { say 'stream closed' },
);
$runner.run($plan, $ctx);   # tap BEFORE running

=end code

The legacy C<&.on-step> string callback (C<'skip'>/C<'start'>/C<'complete'>) is
retained as a thin shim over the event stream, firing identically to prior
releases. It is B<deprecated> and will be removed at C<1.0> — migrate to
C<&.on-event>.

=head3 Ordering guarantees

=item Events form a B<per-run total order> by C<seq> (contiguous, starting at
      C<1>, no gaps).
=item C<step-started> precedes its item events, which precede that step's
      C<step-completed>/C<step-failed> (item events arrive from C<0.4.0>).
=item Cross-item ordering is defined B<only by C<seq>>.

C<run-id> is stable within a single C<run>/C<resume> call and differs between
calls. A validation failure throws before C<run-started> is emitted.

=head3 Step retries

Each plain step runs under the C<&.step-retry>
L<RetryPolicy|#LLM::Data::Pipeline::RetryPolicy>. The default is a single
attempt, which preserves the pre-0.3.0 contract exactly — a failing step emits
C<run-failed> and rethrows its B<original> exception (so domain types like
C<X::LLM::Data::Inference::Exhausted> keep flowing out).

=begin code :lang<raku>

# Opt into retries for every plain step:
my $runner = LLM::Data::Pipeline::Runner.new(
    step-retry => LLM::Data::Pipeline::RetryPolicy.new(:max-attempts(3)),
);

=end code

With C<max-attempts> above 1, each failed attempt emits C<step-failed>
(C<will-retry> True, C<retry-delay> = the backoff), the Runner waits (via the
injectable C<&.schedule-after>), then emits C<step-retry>. When the budget is
spent it emits a final C<step-failed> (C<will-retry> False), then C<run-failed>,
then throws C<X::LLM::Data::Pipeline::StepExhausted> carrying the step name,
attempt count, and the last error.

B<Retry implies idempotency>: a failed attempt's Context mutations are not rolled
back. A step that opts into retries must tolerate its own partial writes; one
that cannot must keep C<max-attempts> at 1.

C<&.now> and C<&.schedule-after> are injectable so tests can drive a virtual
clock with zero real sleeps.

=head3 Runner internals

You do not need this section to use the Runner, but if you are debugging a
failure, supervising a long pipeline, or implementing C<Step::Items>, these are
the rules the engine actually enforces.

=head4 Step drive loop

C<run> validates the Plan, then walks the steps. C<resume> checks the
checkpoint file exists, loads it (v2, or legacy v1 with no C<version> key; a v2
checkpoint's C<run-id> is adopted so DLQ/checkpoint lineage stays stable across
resumes), validates the plan against the rehydrated Context, optionally applies
C<retry-dead> to reopen dead items, and walks.

For each step: steps already recorded in C<completed-steps> emit C<step-skipped>
and are not re-run. Every other step emits C<step-started>, executes, is pushed
onto the completed list, and — if a checkpoint path is configured — writes a
step-boundary checkpoint (trigger C<'step'>) B<before> C<step-completed> is
emitted. Only after a C<Step::Items> step completes does the C<fail-on-dead>
check run: a step with dead items and C<fail-on-dead> set throws
C<X::LLM::Data::Pipeline::ItemsDead> (after C<step-completed>, after the
checkpoint). Any exception propagates out of the loop as C<run-failed> —
except C<Cancelled>, which rethrows without a second terminal event because it
is the caller's intent, not a failure.

=head4 Plain-step retry contract

With the default budget of one attempt a failing step's original exception
propagates unchanged. With more, each failed attempt emits C<step-failed>
(C<will-retry> True), the Runner backs off via the injectable
C<&.schedule-after> (blocking the run thread — correct, plain steps have no
worker threads), emits C<step-retry>, and re-runs. The spent budget emits a
final C<step-failed> (C<will-retry> False) and throws
C<X::LLM::Data::Pipeline::StepExhausted>. A failed attempt's Context mutations
are never rolled back, so retries require idempotency or partial-write
tolerance.

=head4 Item coordinator model

=item C<items($ctx)> is called B<once per activation> and materialized. Each
      item gets a key via C<item-key>; duplicates abort the run. A fingerprint
      (count, SHA-256 of the sorted keys' JSON, SHA-256 of the items' JSON) is
      stored in the checkpoint; on resume it is recomputed and any mismatch
      throws C<X::LLM::Data::Pipeline::CheckpointDrift>.

=item The coordinator (the run thread) owns B<all> mutable state: done results,
      attempts, dead keys, attempt history, telemetry, in-flight set, partials.
      Workers are C<start> blocks pulling from a work Channel; they run
      C<process-item> under C<try> and message results back through an inbox
      Channel, touching no state, files, or events. Every state transition —
      and therefore event C<seq>, checkpoint, and DLQ writes — happens in one
      place: the inbox loop. Single-writer by construction, no locks.

=item The Context is frozen and snapshotted once before the first worker
      starts; every per-item, dead-letter, and cancel checkpoint reuses that
      snapshot.

=item Dispatch is throttled to at most C<degree> items in flight, so the work
      Channel never buffers more than C<degree> jobs and cancellation only has
      to drain the in-flight ones.

A worker can never strand its key: the whole job body is wrapped so a result is
always sent, even if the engine itself throws. A failure's C<item-retryable>
advice is probed
B<worker-side> (the exception object never crosses the Channel; a plain Bool is
shipped), duck-typed so an exception that never heard of the contract reads as
retryable, and an accessor that throws reads as retryable — advice must never
dead-letter an item by malfunctioning.

=head4 Checkpoint v2 schema

Written as v2 always, atomically (temp file + rename), with sorted keys:

=begin code :lang<json>

{
  "version": 2,
  "run-id": "…stable across resumes…",
  "completed-steps": ["step-a", "step-b"],
  "context": { "…full Context snapshot…": null },
  "step-state": {
    "step-name": {
      "fingerprint": { "count": 100, "keys-sha256": "…", "items-sha256": "…" },
      "started-at": "2026-08-21T12:00:00Z",
      "done":     { "item-key": true },
      "results":  { "item-key": "…item result value…" },
      "attempts": { "item-key": 3 },
      "dead":     { "item-key": true },
      "partials": { "item-key": { "…saved sub-unit state…": null } }
    }
  },
  "updated-at": "2026-08-21T12:34:56Z"
}

=end code

Once a step completes, C<results> (and C<partials>) are pruned from its state —
a completed step's checkpoint record shrinks to keys and counters, so a finished
stage never bloats the checkpoint.

=head4 DLQ schema

The dead-letter journal sits beside the checkpoint as JSONL:
C<foo.checkpoint.json> → C<foo.dlq.jsonl>; any other checkpoint name keeps its
basename minus the extension plus C<.dlq.jsonl>. No checkpoint path ⇒ no DLQ.
Each record:

=begin table :caption<DLQ record schema>
Field | Present | Meaning
=================================================================
schema | always | 1
kind | always | C<dead> or C<requeued>
run-id | always | lineage id (stable across resumes)
step, key | always | identity of the item
item-digest | always | SHA-256 of the item's JSON
attempts | always | attempts made when it died (C<0> on C<requeued>)
attempt-history | C<dead> | per-attempt number, POSIX time, error, duration
error | C<dead> | C<{ exception, message }>
inference | optional | C<exhausted>-stage telemetry (C<attempts>, C<summary>)
first-attempt-at, dead-at | always | POSIX seconds bookends (written on both C<dead> and C<requeued>)
=end table

The write order on death is B<DLQ first, then checkpoint> (forced — dead-letter
writes are never coalesced away). A DLQ append failure is deliberately fatal:
the journal is the one artifact this feature exists to produce.

C<resume(:retry-dead)> requeues every dead item: the digest is re-verified
against the journal's C<dead> record (C<CheckpointDrift> on mismatch; the
checkpoint, not the journal, is authoritative for control flow), the key is
cleared from C<dead>/C<attempts>/C<partials>, a C<requeued> record is appended,
and C<item-requeued> is emitted. Reopening a completed step also invalidates
every later completed step (they consumed its C<provides>), while rehydrating
the reopened step's successes from its reserved Context key — only the
formerly-dead items re-run.

=head4 Failure modes

=begin table :caption<Failure-mode reference>
Failure | Caught | Events emitted | Automatic action | Recovery
=================================================================
Step transient error | step retry loop | C<step-failed>, C<step-retry> | backoff and retry within C<step-retry> | raise the budget or fix the step
Step budget spent | step retry loop | C<step-failed> (C<will-retry> False), C<run-failed> | throws C<StepExhausted>, aborts | fix, then C<run>/C<resume>
Plan validation error | before C<run-started> | none | throws immediately | fix C<requires>/C<provides>/seed Context
Item transient error | coordinator inbox | C<item-failed> | retry within C<item-retry> (unless advised non-retryable) | C<resume(:retry-dead)> once fixed
Item exhausted / poison | coordinator inbox | C<item-failed>, C<item-dead-lettered> | DLQ and continue (or C<ItemsDead> under C<fail-on-dead>) | DLQ triage, then C<resume(:retry-dead)>
Cancellation | C<&.is-cancelled> poll | C<run-cancelled> | drain, checkpoint (trigger C<'cancel'>), throw C<Cancelled> | C<resume> — partials kept, resumes mid-item
Checkpoint drift | resume / retry-dead | none | throws C<CheckpointDrift> | restore the original inputs
DLQ append failure | DLQ writer | none | fatal, unshielded | inspect the filesystem
Worker/engine fault | worker last-resort catch | C<item-failed> | consumes the next attempt within C<item-retry>; full budget retained only if C<item-retryable> says retryable | as item transient
=end table

=head2 LLM::Data::Pipeline::RetryPolicy

Exponential-backoff retry budget:
C<delay-for(attempt) = min(max-delay, base-delay * 2 ** (attempt - 1)) + rand * jitter>
(1-based C<attempt>). C<exhausted($attempts)> reports when the budget is spent.
Defaults: C<max-attempts> 3, C<base-delay> 5s, C<max-delay> 120s, C<jitter> 0.5s
— deliberately slower than the inference layer's per-call backoff, since it is
the outer retry ring above C<Task>'s own fast fallback chain.

=head2 LLM::Data::Pipeline::Exceptions

C<X::LLM::Data::Pipeline::StepExhausted> (C<step>, C<attempts>, C<last-error>) is
thrown when a step's retry budget is spent. C<X::LLM::Data::Pipeline::Cancelled>,
C<ItemsDead>, and C<CheckpointDrift> are declared now and thrown from C<0.4.0>.

=head2 LLM::Data::Pipeline::Event

Every observable moment in a run is an immutable object composing the
C<LLM::Data::Pipeline::Event> role, which supplies three envelope fields —
C<UInt $.seq> (per-run, from 1), C<Instant $.at>, C<Str $.run-id> — plus
C<kind> (a kebab-case discriminator) and C<to-hash> (a JSON-safe Hash; C<at>
serializes to a POSIX epoch-seconds number).

=begin code :lang<raku>

use LLM::Data::Pipeline::Event;
use JSON::Fast;

# Pattern-match on class, or switch on the kind string:
given $e {
    when LLM::Data::Pipeline::Event::RunStarted   { ... }
    when LLM::Data::Pipeline::Event::StepCompleted { ... }
}

say to-json($e.to-hash);   # JSON-safe log line, round-trips via from-json

=end code

The full taxonomy is declared now; the Stage-2 subset is emitted by this release.
Reserved kinds let consumers pattern-match today and start receiving events in the
noted release without a breaking change.

=begin table :caption<Pipeline event taxonomy>
Kind                 | Class             | Payload fields                                             | Status
====================================================================================================================
run-started          | RunStarted        | plan-size, resumed, checkpoint-path                        | emitted (0.2.0)
step-skipped         | StepSkipped       | step, ordinal, total, description                          | emitted (0.2.0)
step-started         | StepStarted       | step, ordinal, total, description                          | emitted (0.2.0)
step-completed       | StepCompleted     | step, ordinal, total, description, duration, items-done?, items-dead? | emitted (0.2.0)
checkpoint-written   | CheckpointWritten | path, trigger                                              | emitted (0.2.0)
run-completed        | RunCompleted      | duration, steps-run, steps-skipped, items-done, items-dead | emitted (0.2.0)
run-failed           | RunFailed         | step, error, exception                                     | emitted (0.2.0)
step-failed          | StepFailed        | step, attempt, error, exception, will-retry, retry-delay   | emitted (0.3.0)
step-retry           | StepRetry         | step, attempt, delay, error                                | emitted (0.3.0)
item-started         | ItemStarted       | step, key, attempt                                         | emitted (0.4.0)
item-completed       | ItemCompleted     | step, key, attempt, duration                               | emitted (0.4.0)
item-failed          | ItemFailed        | step, key, attempt, error, exception                       | emitted (0.4.0)
item-dead-lettered   | ItemDeadLettered  | step, key, attempts, error                                 | emitted (0.4.0)
item-requeued        | ItemRequeued      | step, key                                                  | emitted (0.4.0)
progress             | Progress          | step, done, dead, total, in-flight, pending                | emitted (0.4.0; also at activation from 0.5.1)
run-cancelled        | RunCancelled      | step, items-done, items-dead                               | emitted (0.4.0)
telemetry            | Telemetry         | step, key, stage, data                                     | emitted (0.4.0)
run-retry            | RunRetry          | attempt, max-attempts, delay, error, exception             | emitted (0.5.0)
=end table

An item step emits one C<progress> at B<activation> (from C<0.5.1>), before its
first C<item-started>, so a progress bar can show C<0/N> for a fresh stage and
C<done/N> for a resumed one immediately instead of staying blank until the
stage's first item finishes.

=head2 LLM::Data::Pipeline::Step::Items

A step whose body is a set of items processed in parallel by the Runner. Instead
of C<execute>, implement C<items> (deterministic, JSON-safe), C<process-item>
(worker thread; no Context mutation; at-least-once), and C<finalize> (coordinator
thread; assembles C<provides> from the reserved C<"{name}/items"> key). Per-item
retries dead-letter to a crash-safe JSONL journal beside the checkpoint; a run
can be resumed, cancelled, or C<resume(:retry-dead)>'d. See
L<the Runner internals section|#runner-internals> for the coordinator model,
checkpoint v2, DLQ schema, and failure-mode table.

=begin code :lang<raku>

use LLM::Data::Pipeline::Step::Items;

class TagChunks does LLM::Data::Pipeline::Step::Items {
    method name(--> Str:D) { 'tag-chunks' }
    method description(--> Str:D) { 'Tag every chunk' }
    method provides(--> List) { ('tag-chunks/items', 'tags') }
    method items($ctx --> Iterable) { $ctx.get('chunks').list }
    method item-key(Int:D $i, $chunk --> Str:D) { $chunk<id> }
    method process-item($ctx, $chunk, Str:D $key --> Any) { tag-one($chunk) }
    method finalize($ctx --> Nil) {
        $ctx.set('tags', $ctx.get('tag-chunks/items').values.flat.unique.sort.list);
    }
}

my $runner = LLM::Data::Pipeline::Runner.new(:degree(8));
$runner.run($plan, $ctx, :checkpoint-path('run.checkpoint.json'.IO));
# Dead items are journaled to run.dlq.jsonl; retry them once fixed:
$runner.resume($plan, 'run.checkpoint.json'.IO, :retry-dead);

=end code

=head3 Resumable items (0.6.0)

An item that is a B<run of sequential sub-units> — twenty model calls to rewrite
twenty turns of one chunk — can persist its progress as it goes and be handed it
back on the next attempt, so a failure on sub-unit 19 no longer throws eighteen
finished ones away. Opt in with two lines: mint
C<$runner.partial-sink(:$step, :$key)> and call it with a JSON-safe Hash after
each completed sub-unit; read C<:%partial> (C<%()> = start fresh) in
C<process-item>.

=begin code :lang<raku>

method process-item($ctx, $chunk, Str:D $key, :%partial --> Any) {
    my &save   = $runner.partial-sink(:step(self.name), :$key);
    my @turns  = (%partial<turns> // []).list.Array;
    my Int $at = (%partial<next>  // 0).Int;
    for @($chunk<beats>)[$at ..^ *] -> $beat {
        @turns.push(write-turn($beat));
        save({ turns => @turns, next => ++$at });
    }
    @turns;
}

=end code

Partials are state, not events: they ride the checkpoint (C<step-state.partials>,
trigger C<'partial'>, coalesced by C<checkpoint-every>) and never the event
stream or the DLQ. They are cleared when the item goes terminal and B<kept>
across a cancellation, which is what makes a cancelled run resume mid-item.

=head2 Operational recipes

Three entry points, three failure lifetimes:

=item B<C<run>> — one shot. Use it when the pipeline is short and you want
      failures in your face immediately. Exceptions propagate unchanged.

=item B<C<resume>> — restart from a checkpoint. Use it after an interrupt, a
      process death, or a deliberate shutdown. Completed steps replay as
      C<step-skipped>; an in-progress item step picks up from the checkpoint,
      mid-item if partials were saved.

=item B<C<run-until-done>> — stay alive across transient failures. Use it for
      long or unattended pipelines. Attempt 1 resumes an existing checkpoint
      (so it composes with an external re-invoker after a crash), else runs;
      transient failures back off (C<run-retry> event) and resume, while
      validation / cancellation / drift rethrow immediately. Dead items with
      C<:retry-dead> are requeued on each attempt; without it C<ItemsDead> is
      unretryable — a rethrow would recur identically, since nothing about the
      dead items changed.

B<In-process retries do not replace an external supervisor; they complement
one.> C<run-until-done> retries failures inside a live process — a transient
502 from the API, an exhausted step budget. The supervisor (cron, systemd, CI,
or just re-running the command) handles the process itself dying: because a
checkpoint is written after every step and every item terminal, the next
invocation resumes from the latest state with no special-mode flag. If the
failure mode is "the machine restarted", rely on the supervisor; if it is "the
wire glitched for thirty seconds", rely on C<run-until-done>.

=head3 DLQ triage

Dead items are B<journaled, not thrown away>: each one lands in the
C<.dlq.jsonl> sibling as a C<dead> record with its full attempt history and
(when the step provided it) an inference summary. Read it with
C<JSONL::Reader>, group by C<error.exception> / C<error.message>, and the
decision tree is short:

=item B<Transient> (rate limits, DNS, timeouts, spot kills): fix the
      environment, then C<resume(:retry-dead)>. Only the dead items re-run.

=item B<Poison item> (one malformed chunk, a schema rejection): fix the item
      at its source B<before> retrying — but note the digest check compares
      the item re-materialized from C<items()> against the journal's record,
      so B<edited> content will C<CheckpointDrift> (deliberately: a re-run
      must be against the item that died, or nothing was verified).
      C<retry-dead> re-runs items exactly as they were dead-lettered; fix the
      code that handles them or accept the dead keys (they are listed in
      C<"{name}/dead">) and handle them downstream.

=item B<Systematic> (the whole step failed one way): do not C<retry-dead> —
      the same items die again. Fix the step code; the next C<run> or
      C<resume> re-executes it (a completed step re-runs after C<retry-dead>
      invalidation, or start a fresh checkpoint).

B<:retry-dead is an explicit, deliberate act> — nudge it per attempt via
C<run-until-done(:retry-dead)> if the deaths are infra-shaped, or withhold it
while triaging. The digest verification turns "the inputs changed upstream"
into a loud C<CheckpointDrift> instead of a silent replay against new data.

=head3 The retry-dead decision guide

=begin table :caption<When to pass :retry-dead>
Situation | Pass it? | What happens
=================================================================
Transient infra fixed (limits, DNS, spot kill) | Yes | Dead items re-run with fresh budgets; likely to succeed now
Poison item fixed at source, same key | No — expect C<CheckpointDrift> | The digest records the item that died; edited content fails verification on purpose
Inputs changed upstream since the DLQ write | No — expect C<CheckpointDrift> | Fail loudly rather than replay stale items against new data
Step systematically failing | No | Every dead item dies again, re-dead-lettered; fix the step instead
You want every C<run-until-done> attempt to requeue | Yes — C<:retry-dead> | Each run retry re-admits dead items once
Dead items should fail the run | No; set C<fail-on-dead> on the step | C<ItemsDead> after the checkpoint; triage, then C<:retry-dead>
=end table

Note the invalidation cascade: re-opening a completed step removes it B<and
every later step> from C<completed-steps>, because they consumed its
C<provides>. Downstream work re-runs too — that is the honest cost of
re-admitting a finished stage's dead items; budget for it.

=head3 Checkpoint sizing

=item C<checkpoint-every> / C<checkpoint-interval> coalesce per-item
      checkpoints (defaults: every item, no time floor). Dead-letter,
      step-boundary, and cancel writes are never coalesced away.

=item C<results> are pruned once a step completes, so finished stages shrink to
      counters; the long pole is always the B<current> step's accumulated
      results and partials.

=item Partials are whole Hashes per item. Saving a running transcript of a
      20-turn chunk 20 times grows the checkpoint quadratically-ish; save the
      prefix structure that lets you B<re-derive> the rest, not a duplicate of
      everything already produced.

=head2 run-until-done

Supervise a pipeline to completion across whole-run retries — the outer ring
above per-item and per-step retries. Attempt 1 resumes an existing checkpoint
(so it composes with an external re-invoker after a crash), else runs; transient
failures back off (C<run-retry> event) and resume, while validation /
cancellation / drift rethrow immediately. See
L<the Operational recipes section|#operational-recipes> for the full contract
and the operational recipes (in-process retries vs external supervisor, DLQ
triage, the retry-dead decision guide).

=begin code :lang<raku>

my $ctx = $runner.run-until-done(
    $plan, $seed-ctx,
    checkpoint-path => 'run.checkpoint.json'.IO,
    run-retry       => LLM::Data::Pipeline::RetryPolicy.new(:max-attempts(5), :base-delay(30)),
    retry-dead      => True,   # requeue transiently-dead items on each attempt
);

=end code

=head1 AUTHOR

Matt Doughty <matt@apogee.guru>

=head1 COPYRIGHT AND LICENSE

Copyright 2026 Matt Doughty

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.

=end pod
