Latency#
anira computes the latency of every session in closed form, together with the number of inference slots the session allocates and the sizes of its ring buffers. The calculation lives in anira::LatencyCalculator; anira::SessionElement applies its result in prepare() (both private classes of the scheduler, src/scheduler/).
The model behind the formulas is the scheduler’s actual worst case:
every inference takes exactly the Hard contract’s budget,
at most \(n\) inferences run at once (the session’s instances), in submission order,
an inference is submitted by the host callback that completes its hop of input,
results are collected once per callback; a
_waitentry collects after its wait ofwait_ratioblock periods (ANIRA_WAIT_CONTRACT), theANIRA_NONBLOCKINGentries collect what has completed,the host pushes and pops a sample only once a whole one has accumulated in its block, so that a block may be a fractional number of samples on a stream (see the note below).
Quantities#
For a session with reference stream size \(R\) (the hop of the model’s anchor, Flexible windows and the anchor hop), host block \(B\) (the contract’s host_block_max) and host rate \(f_s\) (the contract’s host_rate), both stated in samples of the reference stream, the calculator derives:
Symbol |
Value |
Meaning |
|---|---|---|
\(\rho = p / q\) |
\(B / R\) |
the host block measured in hops, as a reduced fraction |
\(\kappa\) |
\(T f_s / (1000 R)\) |
the inference time \(T\) (ms: the contract’s explicit budget, or under a measured budget the slowest plan’s warm-up figure) measured in hop periods |
\(\tau\) |
\(\kappa / \rho\) |
the inference time measured in host blocks |
\(\beta\) |
|
the fraction of a host block a |
\(n\) |
the session’s instances |
the parallelism of the session: |
\(H\) |
largest streamable hop |
the stream on which the host block is finest; the unit of the smaller-block grid ( |
\(d_m\) |
\(\max(0, \lceil (m+1)\tau - \beta \rceil)\) |
callbacks after submission at which the \((m+1)\)-th batch of \(n\) inferences submitted together is collected |
Buffer adaptation#
Repackaging a stream from host blocks of \(b\) samples into model blocks of \(P\) samples requires a delay. Rath and Geier [RG26] prove that the minimum delay for constant block sizes is
In hop units the adaptation is \(1 - 1/q\) for every stream at once, because \(\gcd(\rho P, P) = P / q\) whenever the block \(\rho P\) is a whole number of samples. If the host block may vary in size (host_block_min below host_block_max) or is a fractional number of samples on a stream, the worst case is \(P - 1\) samples [RG26], section 5.
Inference-caused latency#
The receive ring of an output never runs dry if and only if its zero priming \(L\) (in hops) covers, at every callback \(k\), the samples the host has popped minus the results that have been collected:
where \(C(k)\) is the number of inferences collected by callback \(k\). Inference \(j\) is submitted at callback \(a_j = \lceil (j+1)/\rho \rceil - 1\) (the last host block overlapping model block \(j\), the observation at the heart of [RG26]), and with \(n\) processors in submission order it finishes at \(F_j = \max(a_j, F_{j-n}) + \tau\). Unrolling the recursion gives
and since \((k+1)\rho - \lfloor \rho (k+1-d_m) \rfloor = \rho\, d_m + \mathrm{frac}(\rho (k + 1 - d_m))\), whose largest value over \(k\) is \(\rho\, d_m + (q-1)/q\), the latency in hops is
The first term is the buffer adaptation, the second the inference queue. The maximum exists if and only if \(\kappa < n\): every hop of audio brings \(\kappa\) hop periods of inference work for \(n\) processors, and beyond that the queue grows without bound (LatencyCalculator::is_feasible is false, the handler’s prepare logs a Warning that the pool cannot keep up, and the values describe one host block processed by an idle pool). \(\kappa\) is also the real-time factor a plan row reports (anira_plan_info.rt_factor, The plan report), from the plan’s own inference time, and \(\kappa / n\) its load: below 0.75 comfortable, below 1 marginal, at 1 or more not real-time (the infeasible case here). Only batches with \(m < \rho / (n - \kappa)\) can contribute, so the search is finite; in the common case \(\rho \lceil \tau \rceil \le n\) the maximum is the first term, \(\rho \lceil \tau - \beta \rceil\).
The latency of output tensor \(i\) in its own samples is \(P_i \Lambda\). It is the same number of hops for every output, and it is rounded down because the host pops whole samples only (LatencyCalculator::get_output_latencies returns the unrounded value).
Note
The Hard contract’s host blocks may be fractional (host_block_min and host_block_max are doubles, e.g. contract.hard_host_blocks(samples_per_block / 2048.0, samples_per_block / 2048.0, sample_rate / 2048.0) for a model whose anchor stream runs at one sample per 2048 host samples; the rate is in anchor samples per second as well). The host and that stream then exchange samples at a non-integer ratio, and anira assumes the worst case: after its \(k\)-th call the host has moved (pushed, or popped) \(\lfloor k b \rfloor\) samples of every stream whose block is \(b\), so one call carries \(\lfloor b \rfloor\) or \(\lceil b \rceil\) of them. For example, with a block of 0.25 samples of a control-rate stream, the handler receives one sample every four host calls, and the latency is calculated as if the sample is delivered during the fourth call. If your system always sends the sample at the first call, a lower latency is possible; declare it with anira_contract_hard_set_latency (the latencies of anira::Hard). \(B\) is the contract’s host_block_max, and \(\rho\) the rational LatencyCalculator::rationalize recovers from it (a relative tolerance of \(10^{-6}\) and a denominator of at most \(2^{20}\): 1/3 and 1/4 exactly). \(\lceil b \rceil\) is also the most one call may carry: a Hard entry handed more on a stream, pushed or popped, is refused with ANIRA_ERROR_INVALID_ARGUMENT before anything moves, and a pop within it that the stream cannot cover is a miss (ANIRA_MISSED).
Inference slots#
The same recursion bounds the number of inferences that are submitted but not yet collected right after the submissions of a callback, which is the number of thread-safe structures a session allocates:
With a wait ratio that covers the inference time this is 1: a _wait entry collects the result in the callback that submitted it.
The session allocates twice this many structures. anira::InferenceHandler::reset() is wait-free: it does not wait for in-flight inferences but marks them stale, and each keeps its structure until its worker publishes completion. Meanwhile the fresh schedule needs \(S\) structures of its own, so one pool drains while the other serves and a reset never drops a hop.
When the inference does not keep up#
The latency holds while the inferences take at most the budget the contract states. When one runs far over it (a stalled machine, a model slower than declared), two things happen, and neither moves the stream:
A block whose inference has not been collected when the host pops it is a miss: the entry returns
ANIRA_MISSEDwith a count of0and the miss policy fills the block (Miss policy). The late result is discarded when it arrives, and so is every other late sample: a delivered block is the stream at its latency, never late.A hop that finds every inference slot busy is dropped: it gets no inference and costs exactly one hop of the miss policy’s fill at its own stream position, behind the inferences submitted before it; every other hop of the call is dropped the same way, so the input stays on its hop grid.
rt_errorreportsANIRA_ERROR_CAPACITY.
Every hop of input therefore yields exactly one hop of output, in order: the model’s result, or a fill. Once the late inferences are collected, the output is the model’s at exactly the reported latency again.
Adaptive buffer handling#
For hosts that may use smaller blocks than the contract’s maximum (host_block_min below host_block_max), the host block may be any whole number \(j\) of samples of the finest stream, \(1 \le j \le \lfloor \rho H \rfloor\), i.e. \(\rho' = j / H\) hops with \(\tau' = \kappa / \rho'\). The calculator takes the worst case over that grid: the flexible-host adaptation \((H-1)/H\) plus the maximum over \(j\) of the inference term and of the slot count. Both are increasing in \(j\) on every interval where \(\lceil (m+1)\tau' - \beta \rceil\) is a constant \(c\), so the maximum is attained at the last grid point of each interval, \(j_c = \lceil (m+1)\kappa H / (c - 1 + \beta) \rceil - 1\), and the remaining intervals are bounded by \((m+1)\kappa\, c / (c - 1 + \beta)\). Only these breakpoints are evaluated.
Note that a smaller host block can require more latency and more slots than the maximum block: a block slightly shorter than one inference time means that results arrive two callbacks after their submission instead of one.
Internal model latency#
Additional latency inherent to the model itself, such as look-ahead requirements or internal buffering (the output spec’s latency, TensorSpec::latency / anira_tensor_spec_set_latency, in samples of that output), is added to every Streamed output after the calculation above.
Latency synchronization#
When more than one Streamed output is present, their integer latencies \(L_i = \lfloor P_i \Lambda \rfloor\) are raised so that the outputs stay coherent: every Streamed output is delayed by the same time. A delay of \(x\) hops is a whole number of samples of every output, \(P_i x\), exactly when \(x\) is a multiple of \(1/g\), with \(g = \gcd_i P_i\) (Bézout). The outputs therefore meet at the least such delay at or above every \(L_i\), \(k/g\) hops with
which is \((P_i / g) \cdot k\) samples of output \(i\), in exact integer arithmetic. Outputs of equal hops are coherent at their own figure already (\(g = P\)) and keep \(\lfloor P \Lambda \rfloor\); coprime hops (\(g = 1\)) meet at a whole number of hops. For example, outputs of hops 2048 and 512 (\(g = 512\)) with figures 2662 and 665 meet at \(k = 666\), i.e. 2664 and 666 samples, \(666/512\) hops. Only Streamed outputs count: a Static or a State output beside a single Streamed one aligns nothing, and that Streamed output keeps its figure \(\lfloor P \Lambda \rfloor\). A stateful model with one audio output and its recurrent state as a State output (hop 2048, host block 512, 44.1 kHz, a 20 ms budget, an internal latency of 2224) reports \(1.25 \cdot 2048 + 2224 = 4784\) samples, the inference queue’s 1.25 hops unrounded.
The latency vector returned by anira::InferenceHandler::get_latencies() is index-aligned with the output tensor list. Outputs without a stream (a Static or a State output) carry no stream latency and always report 0. anira_handler_get_latencies is the same vector as a C array indexed by slot, one entry per output tensor of the model config’s list (0 for a Static or a State output), and anira_handler_get_latency(h, slot) one entry of it.
Declaring the figure#
A host that needs another figure than the computed one declares it on the Hard contract, per streamed output and in samples of that output: anira_contract_hard_set_latency(contract, "audio_out", 4096), or "latencies": {"audio_out": 4096} in the contract file. The declared figure replaces everything above for that output (the buffer adaptation, the inference queue, the wait credit and the internal model latency): anira_handler_get_latency reports it, and the receive ring is primed to it: with the figure less the internal model latency in zeros, the model’s own delay supplying the rest. Every output never named keeps its computed figure, and the synchronization above does not raise a declared one. A figure below the model’s internal latency cannot be delivered and is ANIRA_ERROR_CONFIG at prepare, as is a name that is no streamed output. Declaring the same figure for every block size, or the figure of another stream, is how a host keeps its reported latency constant or aligns two streams.
Ring buffer sizes#
A send ring holds one host block plus the largest leftover the adaptation can leave behind, \(\lceil b \rceil + P - \gcd(\lceil b \rceil, P)\) for a constant whole-sample block and \(\lceil b \rceil + P - 1\) for a fractional or variable one, plus the history a receptive-field model peeks at. A receive ring holds one result per inference slot plus the latency priming, \(S P_i + L_i\).
These calculations count buffer sizes in samples of the anchor tensor of the model configuration — a streamable input for effects and analysers, the streamable output for a generator model with no streamable input (State, instances, anchor).
The figure is the delay, in samples of each output, between a sample entering and its result leaving: for the first \(L_i\) samples (the zero priming) process and pop_data deliver zeros.
A measured latency#
Under a measured budget, the default, \(T\) is the slowest plan’s warm-up figure, so the latency depends on the machine and on its load at the prepare that measured it (a later prepare of the model reuses that figure, Budget and warm-up): a DAW’s delay compensation sees it differ per machine, as it already does per block size. An explicit budget gives a reproducible figure. Below the first \(T\) at which a \(d_m\) or \(S\) steps, the latency and the entry count (the \(2S\) structures, anira_handler_num_entries) do not move with \(T\); a probe of the bundled models at 48 kHz, bisected, put those edges at:
Configuration |
latency flat below \(T\) |
entries flat below \(T\) |
|---|---|---|
blocks 1..512, the CNN family, the RNN, RAVE |
5.354 ms |
25.625 ms |
blocks 1..512, the gain model, hybridnn |
5.354 ms |
7.125 ms |
the web, render quantum 128, burst 1024 |
10.688 ms |
14.229 ms |
The browser’s clock resolution (5 µs in Chromium, 20 µs in Firefox) never moves a latency by a step: the warm-up’s clock-step floor covers models that run in microseconds.
Matthias Rath and Matthias Geier, “Minimum required delay for realtime block size adaptation in digital audio signal processing”, Proceedings of the 20th Linux Audio Conference (LAC-26), Maynooth, 2026. hal-05697688