=begin pod

=head1 NAME

ONNX::Native - Minimal Raku bindings for ONNX Runtime (inference only)

=head1 SYNOPSIS

=begin code :lang<raku>

use ONNX::Native;
use ONNX::Native::Types;

# Load a model, introspect its I/O surface.
my $session = ONNX::Native::Session.new(
	:path<model.onnx>,
	:providers<coreml cpu>,   # macOS: CoreML first, fall back to CPU
);

say $session.input-names;       # <input_ids attention_mask>
say $session.output-names;      # (logits)
say $session.input-info('input_ids').shape;   # (-1, -1)  (dynamic)

# Run inference.
my $ids  = ONNX::Native::Tensor.from-ints(
	@token-ids, :shape([1, @token-ids.elems]), :dtype(INT64));
my $mask = ONNX::Native::Tensor.from-ints(
	@attention,  :shape([1, @attention.elems]),  :dtype(INT64));

my %out = $session.run(
	inputs  => { input_ids => $ids, attention_mask => $mask },
	outputs => ['logits',],
);

my @logits = %out<logits>.to-num-array;
my @shape  = %out<logits>.shape;

=end code

=head1 STATUS

This is v0.1 — the minimum surface needed to run classification,
embedding, and small-model inference on top of HuggingFace-exported
ONNX files. Scoped deliberately tight so the full surface can land
on CI without stretching review.

=head2 What works

=item Load models from disk path or in-memory Blob.
=item Introspect input/output counts, names, tensor shapes and
      dtypes.
=item Create input tensors from Raku lists (Int / Num) or from
      existing Blobs (zero-copy).
=item Run inference with CPU or — on macOS — CoreML.
=item Read output tensors as Raku lists (Num / Int) or raw Blobs.
=item Explicit C<.dispose> and automatic C<DESTROY> cleanup on
      sessions and tensors.

=head2 What's deferred to later versions

=item GPU providers beyond the "CPU + platform accelerator" default
      (CUDA on Linux, DirectML on Windows, ROCm / TensorRT / CoreML
      on non-macOS, QNN, OpenVINO).
=item Training API, IO Binding, sparse / sequence / map tensor types.
=item String / fp16 / bf16 / int8 / int4 tensor I/O.
=item Fine-grained threading configuration (intra/inter-op pools).
=item Async C<RunAsync>.
=item Model metadata API and profiling.

If you need something from the deferred list, file an issue or
PR — none of them are deep changes, they're just more of the same
shim-and-wrap pattern and I'd rather ship a tight v0.1 first.

=head1 INSTALLATION

=head2 macOS / Linux — prebuilt

=begin code :lang<shell>

zef install ONNX::Native

=end code

This downloads Microsoft's official ONNX Runtime prebuilt for your
platform from their GitHub Releases (verified via SHA256 against
bundled checksums), stages it under
C<$XDG_DATA_HOME/ONNX-Native/>, and compiles a tiny C shim against
it. Total install size: ~25 MB for the ORT dylib plus a ~35 KB
shim.

No system-wide changes are made. The staged libs live under your
user's data dir; C<zef uninstall ONNX::Native> leaves them in place
(harmless, safe to C<rm -rf>).

=head2 macOS / Linux — system ORT

If your system already has libonnxruntime and its headers (e.g.
C<brew install onnxruntime>, or an apt package providing
C<libonnxruntime-dev>), set:

=begin code :lang<shell>

ONNX_NATIVE_PREFER_SYSTEM=1 zef install ONNX::Native

=end code

Build.rakumod probes Homebrew (C</opt/homebrew> / C</usr/local>)
and common Linux lib paths for the runtime and header set. It
compiles the shim against what it finds and skips staging.

=head2 Linux / Windows — CUDA GPU

=begin code :lang<shell>

ONNX_NATIVE_WITH_CUDA=1 zef install ONNX::Native

=end code

Picks Microsoft's GPU prebuilt (`onnxruntime-linux-x64-gpu-<ver>.tgz`
or `onnxruntime-win-x64-gpu-<ver>.zip` — both bundle libcudart /
libcudnn / libcublas, so the end user only needs the NVIDIA
driver, not a full CUDA toolkit install) and compiles the shim
with the CUDA dispatch path enabled. The bundle is staged under
a `-gpu` suffixed dir so CPU and GPU variants don't collide.

Use at the call site via:

=begin code :lang<raku>

my $session = ONNX::Native::Session.new(
    :path<my-model.onnx>,
    :providers<cuda cpu>,           # CUDA first, CPU for unsupported ops
    :cuda-device-id(0),             # which GPU (default 0)
);

=end code

Falls back to CPU automatically for any op CUDA can't handle —
common case for transformers' embedding lookups.

Supported platforms: x86_64 Linux + x86_64 Windows. Microsoft
doesn't publish CUDA prebuilts for ARM64 (Linux aarch64 is Jetson
territory; Windows-on-ARM has no NVIDIA driver), and macOS uses
CoreML / Metal — set `ONNX_NATIVE_WITH_CUDA=1` on those platforms
and the install will refuse cleanly.

CI builds the GPU bundles but doesn't smoke-test them (no free
GPU runners). See `RELEASING.md` for the manual pre-tag smoke
checklist that runs before each binary release.

=head2 Windows — DirectML

DirectML support is deferred. The DML provider isn't included in
Microsoft's standard `onnxruntime-win-x64.zip` (DML is shipped
exclusively as a NuGet package, separate URL + layout), and
NVIDIA-on-Windows users — the dominant Windows-local-ML case —
get a better experience via CUDA. AMD / Intel-iGPU Windows users
who'd benefit from DML are a smaller minority; revisit if there's
demand.

=head2 Environment variables

=begin table
ONNX_NATIVE_PREFER_SYSTEM | Skip prebuilt download, compile shim against system libonnxruntime
ONNX_NATIVE_BINARY_ONLY   | Refuse system fallback; fail if prebuilt unavailable
ONNX_NATIVE_BINARY_URL    | Alternate GitHub release base URL (default: Microsoft upstream)
ONNX_NATIVE_CACHE_DIR     | Override download cache dir (default: $XDG_CACHE_HOME)
ONNX_NATIVE_DATA_DIR      | Override staged-libs base dir (default: $XDG_DATA_HOME)
ONNX_NATIVE_LIB_DIR       | Runtime: load shim from this dir instead of the staged dir
ONNX_NATIVE_WITH_CUDA     | Opt in to the CUDA GPU prebuilt variant (linux-x86_64 + windows-x86_64; install-time)
=end table

=head1 API

=head2 ONNX::Native::Session

=head3 new(:$path!, :@providers = [CPU], :$log-id, :$cuda-device-id, :$coreml-flags)

Load a model from disk. C<:@providers> can be a list of C<Provider>
enum values or lowercase strings; they're tried in order at
inference time (per-node), falling back to CPU for any op the
chosen provider can't handle.

Per-provider tuning args (only consulted when the matching
provider is in C<:@providers>):

=item C<:cuda-device-id> — integer GPU index, default 0. With a
      multi-GPU box this picks which CUDA device ORT runs on.
=item C<:coreml-flags> — bitfield matching CoreML's
      C<COREMLFlags> enum (e.g. C<COREML_FLAG_USE_CPU_ONLY = 1>,
      C<COREML_FLAG_ONLY_ENABLE_DEVICE_WITH_ANE = 4>). Default 0
      keeps CoreML's full ANE-eligible op set.

DirectML and CUDA-V2 provider options aren't surfaced yet — the
shim's C<int64_t flags> channel takes a single int per provider,
which is the right shape for CUDA C<device_id> and CoreML
C<COREMLFlags> but not for the richer
C<OrtCUDAProviderOptionsV2> struct (cudnn algo search, memory
limit, copy-on-default-stream). Planned for a future
C<:%provider-options> hash.

Throws C<X::ONNX::Native::Error> if the model can't be opened.
Throws C<X::ONNX::Native::ProviderUnavailable> if a requested
provider isn't compiled into this build of the shim.

=head3 new(:$bytes!, :@providers, :$log-id, :$cuda-device-id, :$coreml-flags)

Load a model from an in-memory Blob. Same provider-tuning args
as the path-based ctor.

=head3 input-count(), output-count() → Int

Number of inputs / outputs in the model graph.

=head3 input-names(), output-names() → List[Str]

Input / output names in index order. Cached after first call.

=head3 input-info($name), output-info($name) → TensorInfo

Tensor spec (element type + shape) for the named I/O. Dimensions
of C<-1> denote symbolic / dynamic shapes.

=head3 run(:%inputs!, :@outputs!) → Hash[Str, Tensor]

Run inference. C<%inputs> maps each input name to a Tensor.
C<@outputs> lists output names to fetch. Returns a hash of output
tensors.

=head3 dispose() / DESTROY

Release the underlying native handles. Idempotent; safe to call
manually or rely on GC.

=head2 ONNX::Native::Tensor

=head3 from-blob($data, :@shape!, :$dtype = FLOAT32) → Tensor

Construct a tensor that borrows the given Blob's bytes. The Blob
is kept alive for the tensor's lifetime — ORT holds a pointer into
its backing storage and doesn't copy on the way in.

=head3 from-nums(@values, :@shape!, :$dtype = FLOAT32) → Tensor

FLOAT32 or DOUBLE tensor from a flat Raku list of numbers.

=head3 from-ints(@values, :@shape!, :$dtype = INT64) → Tensor

INT32 or INT64 tensor from a flat Raku list of integers. Default
dtype is INT64 because HuggingFace tokenizers emit int64 token IDs
by convention.

=head3 shape() → List[Int]

Concrete dimension list.

=head3 dtype() → DType

Tensor element type.

=head3 byte-length() → Int

Total bytes in the tensor's data (C<elem-count × dtype-byte-size>).

=head3 to-num-array() → List[Num]

Decode the tensor's data as a flat list of Num. Requires FLOAT32 or
DOUBLE dtype.

=head3 to-int-array() → List[Int]

Decode as a flat list of Int. Accepts any integer dtype (INT8 ..
INT64, UINT8 .. UINT32, BOOL).

=head3 to-blob() → Blob

Copy the tensor's data into a fresh Raku Blob. Independent of the
tensor; survives disposal.

=head3 dispose() / DESTROY

Release the underlying OrtValue. Input tensors drop their Blob
anchor; output tensors free ORT-owned storage.

=head2 ONNX::Native::Types

=head3 DType enum

C<FLOAT32 INT32 INT64 UINT8 INT8 BOOL DOUBLE UINT16 INT16 UINT32
UINT64 FLOAT16 BFLOAT16 STRING>. Only the first three support
Raku ↔ ORT round-trips in v0.1. The rest decode for introspection
but can't yet be constructed from Raku data.

=head3 Provider enum

C<CPU COREML CUDA DML>. CPU is always available. COREML is
available on macOS. CUDA / DML require opt-in prebuilt variants.

=head4 Provider flags

The C<onnx_shim_enable_provider> shim entry takes a 64-bit
C<flags> argument that is B<not> a generic option dictionary —
each provider interprets the bits differently:

=item C<COREML> — Bitfield matching CoreML's
      C<COREMLFlags> enum (C<COREML_FLAG_USE_CPU_ONLY = 1>,
      C<COREML_FLAG_ENABLE_ON_SUBGRAPH = 2>,
      C<COREML_FLAG_ONLY_ENABLE_DEVICE_WITH_ANE = 4>, etc.). OR
      together to tune behaviour.
=item C<CUDA> — Integer C<device_id> selecting which GPU to
      bind to (0 = first CUDA device).
=item C<DML> — Integer C<device_id> selecting the DirectX
      adapter (0 = primary). Same shape as CUDA but a different
      enumeration (D3D adapter index, not CUDA ordinal).
=item C<CPU> — Ignored. CPU is always registered and takes no
      flags.

C<Session.new> surfaces the CoreML and CUDA channels via the
C<:coreml-flags> and C<:cuda-device-id> named args (see API §
C<new>). Both default to 0, which preserves the original
behaviour: CoreML's full ANE-eligible op set and CUDA device 0.
DML's C<device_id> isn't surfaced yet (DML support itself is
deferred — see Installation § DirectML).

The ENUM values themselves are stable across ORT versions but
the set of recognised flags grows release-to-release — consult
the matching ORT C header (C<coreml_provider_factory.h> for
CoreML, C<cuda_provider_factory.h> for CUDA) for the exact bit
meanings tied to the BINARY_TAG you're running against.

=head3 TensorInfo

Static description of a tensor's shape + dtype. Returned by
C<Session.input-info> / C<Session.output-info>.

=head3 Exceptions

=item C<X::ONNX::Native::Error> — base class; C<$.code> is an
      OrtErrorCode value, C<$.reason> is the raw message.
=item C<X::ONNX::Native::Unsupported> — a feature isn't implemented
      in this version of ONNX::Native.
=item C<X::ONNX::Native::ProviderUnavailable> — requested execution
      provider isn't compiled into this shim build.

=head1 MEMORY MANAGEMENT

Session and Tensor both manage native resources via C<DESTROY>.
Normal Raku GC releases them eventually, but if you're running
many inferences in a tight loop, explicit C<.dispose> on
intermediate tensors reclaims memory faster:

=begin code :lang<raku>

for @batches -> $batch {
	my $input = ONNX::Native::Tensor.from-ints($batch, :shape([1, $batch.elems]));
	my %out = $session.run(:inputs({input_ids => $input}), :outputs(['logits',]));
	process(%out<logits>.to-num-array);
	$input.dispose;
	.dispose for %out.values;
}

=end code

Input tensors built with C<from-blob> keep the backing Blob alive
via an internal anchor — don't worry about the Blob being GC'd out
from under ORT mid-inference.

=head1 BINARY_TAG

The C<BINARY_TAG> file at the dist root pins the ONNX Runtime
prebuilt version. Format:
C<binaries-onnxruntime-E<lt>verE<gt>-rE<lt>revE<gt>>, e.g.
C<binaries-onnxruntime-1.20.0-r1>. The Raku dist version (e.g.
0.1.3) bumps independently for Raku-only bugfix releases that
reuse the same ORT binary tag.

At install time Build.rakumod reads BINARY_TAG, fetches the matching
tarball from C<microsoft/onnxruntime>'s GitHub Releases, verifies
SHA256 against C<resources/checksums.txt>, and stages the libs
under C<$XDG_DATA_HOME/ONNX-Native/E<lt>tagE<gt>/>.

=head1 PLATFORM NOTES

=head2 macOS

CoreML EP is baked into Microsoft's macOS prebuilts since ORT 1.16 —
pass C<:providers<coreml cpu>> to C<Session.new> to use it. On
Apple Silicon the Neural Engine handles most common ops; falls back
to CPU for the rest.

Gatekeeper may quarantine the downloaded dylib the first time it's
loaded. If you hit a signing error, C<xattr -dr com.apple.quarantine
~/.local/share/ONNX-Native/> clears it.

=head2 Linux

Microsoft's prebuilts target glibc 2.31+ (Ubuntu 20.04). Older
systems (CentOS 7, Ubuntu 18.04) trigger the system-libonnxruntime
fallback — install C<libonnxruntime-dev> from your package manager
or build from source.

=head2 Apple Silicon note

If you hit a C<dyld[...]: Library not loaded: @rpath/libonnxruntime.*>
error at runtime, the shim didn't pick up the C<@loader_path>
rpath — most likely because you moved the shim out of its staged
directory. Either move it back or rebuild with
C<ONNX_NATIVE_LIB_DIR=/path/to/dir>.

=head1 LIMITATIONS

See the "What's deferred" list under STATUS. Major limitations worth
calling out explicitly:

=item No Windows support in v0.1.
=item CUDA path on Linux is untested in CI.
=item Input / output tensors are CPU-backed only. GPU tensors
      (CUDA-device or CoreML-device) require IO Bindings which are
      deferred.
=item Model shape introspection returns up to 16 dims; higher-rank
      tensors error. 16 is plenty for anything outside niche
      research use.

=head1 AUTHOR

=item Matt Doughty

=head1 COPYRIGHT AND LICENSE

Copyright 2026 Matt Doughty

This library is free software; you can redistribute it and/or modify
it under the Artistic License 2.0.

=end pod
