Class anira::InferenceHandler

class InferenceHandler

Collaboration diagram for anira::InferenceHandler:

Main handler class for neural network inference operations.

The InferenceHandler provides a high-level interface for performing neural network inference in real-time audio processing contexts. It manages the inference backend, data buffering, and processing pipeline while ensuring real-time safety.

This class supports multiple processing modes:

• Single tensor processing for simple models

• Multi-tensor processing for complex models with multiple inputs/outputs

• Push/pop data patterns for decoupled processing

Note

This class is designed for real-time audio processing and uses appropriate memory allocation strategies to avoid audio dropouts.

Public Functions

InferenceHandler() = delete

Default constructor is deleted to prevent uninitialized instances.

InferenceHandler(const InferenceHandler&) = delete

Copy constructor is deleted to prevent copying.

InferenceHandler &operator=(const InferenceHandler&) = delete

Copy assignment is deleted to prevent copying.

InferenceHandler(InferenceHandler&&) = delete

Move constructor is deleted to prevent moving.

InferenceHandler &operator=(InferenceHandler&&) = delete

Move assignment is deleted to prevent moving.

InferenceHandler(PrePostProcessor &pp_processor, InferenceConfig &inference_config, const ContextConfig &context_config = ContextConfig())

Constructs an InferenceHandler with pre/post processor and inference configuration.

Parameters:

• pp_processor – Reference to the pre/post processor for data transformation

• inference_config – Reference to the inference configuration containing model settings

• context_config – Optional context configuration for advanced settings (default: ContextConfig())

InferenceHandler(PrePostProcessor &pp_processor, InferenceConfig &inference_config, BackendBase &custom_processor, const ContextConfig &context_config = ContextConfig())

Constructs an InferenceHandler with custom backend processor.

Parameters:

• pp_processor – Reference to the pre/post processor for data transformation

• inference_config – Reference to the inference configuration containing model settings

• custom_processor – Reference to a custom backend processor implementation

• context_config – Optional context configuration for advanced settings (default: ContextConfig())

~InferenceHandler()

Destructor that properly cleans up inference resources.

void set_inference_backend(InferenceBackend inference_backend)

Sets the inference backend to use for neural network processing.

Calling this is optional: the active backend defaults to the first model in the InferenceConfig whose backend is available in this build, or to CUSTOM when a custom processor was passed to the constructor (or when no configured backend is available).

Parameters:

inference_backend – The backend type to use (e.g., ONNX, LibTorch, TensorFlow Lite or custom)

InferenceBackend get_inference_backend()

Gets the currently active inference backend.

Unless set_inference_backend() was called, this is the default described there: the first available configured backend, or CUSTOM.

Returns:

The currently configured inference backend type

void prepare(HostConfig new_audio_config)

Prepares the inference handler for processing with new audio configuration.

This method must be called before processing begins or when audio settings change. It initializes internal buffers and prepares the inference pipeline.

Note

Blocking quiescence point: waits until no inference thread is executing any of this handler’s work before rebuilding the internal buffers, and invalidates everything dispatched before the call — so once prepare() returns, no inference thread will run user code (a custom backend or the PrePostProcessor::before_inference()/ after_inference() hooks) for pre-prepare work. This is the guarantee reset() deliberately does not provide. Never call from the audio thread.

Parameters:

new_audio_config – The new audio configuration containing sample rate, buffer size, etc.

void prepare(HostConfig new_audio_config, unsigned int custom_latency, size_t tensor_index = 0)

Prepares the inference handler for processing with new audio configuration and a custom latency.

This method must be called before processing begins or when audio settings change. It initializes internal buffers and prepares the inference pipeline.

Parameters:

• new_audio_config – The new audio configuration containing sample rate, buffer size, etc.

• custom_latency – Custom latency value in samples to override the calculated latency

• tensor_index – Index of the tensor to apply the custom latency (default: 0)

void prepare(HostConfig new_audio_config, std::vector<unsigned int> custom_latency)

Prepares the inference handler for processing with new audio configuration and custom latencies for each tensor.

This method must be called before processing begins or when audio settings change. It initializes internal buffers and prepares the inference pipeline.

Parameters:

• new_audio_config – The new audio configuration containing sample rate, buffer size, etc.

• custom_latency – Vector of custom latency values in samples for each tensor

size_t process(float *const *data, size_t num_samples, size_t tensor_index = 0)

Processes audio data in-place for models with identical input/output shapes.

This is the most simple processing method when input and output have the same data shape and only one tensor index is streamable (e.g., audio effects with non-streamable parameters).

Note

This method is real-time safe and does not allocate memory. If the blocking_ratio in the inference configuration is > 0 (not default), this method introduces a controlled blocking operation to wait for processed data (semaphore.try_acquire_until()) in order to further reduce latency.

Parameters:

• data – Audio data buffer organized as data[channel][sample]

• num_samples – Number of samples to process

• tensor_index – Index of the tensor to process (default: 0)

Returns:

Number of samples actually processed

size_t process(const float *const *input_data, size_t num_input_samples, float *const *output_data, size_t num_output_samples, size_t tensor_index = 0)

Processes audio data with separate input and output buffers.

This method allows for different input and output buffer sizes and is suitable for models that have different input and output shapes.

Note

This method is real-time safe and does not allocate memory. If the blocking_ratio in the inference configuration is > 0 (not default), this method introduces a controlled blocking operation to wait for processed data (semaphore.try_acquire_until()) in order to further reduce latency.

Parameters:

• input_data – Input audio data organized as data[channel][sample]

• num_input_samples – Number of input samples

• output_data – Output audio data buffer organized as data[channel][sample]

• num_output_samples – Maximum number of output samples the buffer can hold

• tensor_index – Index of the tensor to process (default: 0)

Returns:

Number of output samples actually written

size_t *process(const float *const *const *input_data, size_t *num_input_samples, float *const *const *output_data, size_t *num_output_samples)

Processes multiple tensors simultaneously.

This method handles complex models with multiple input and output tensors, processing all tensors in a single call.

Note

This method is real-time safe and does not allocate memory. If the blocking_ratio in the inference configuration is > 0 (not default), this method introduces a controlled blocking operation to wait for processed data (semaphore.try_acquire_until()) in order to further reduce latency.

Parameters:

• input_data – Input data organized as data[tensor_index][channel][sample]

• num_input_samples – Array of input sample counts for each tensor

• output_data – Output data buffers organized as data[tensor_index][channel][sample]

• num_output_samples – Array of maximum output sample counts for each tensor

Returns:

Array of actual output sample counts for each tensor

void push_data(const float *const *input_data, size_t num_input_samples, size_t tensor_index = 0)

Pushes input data to the processing pipeline for a specific tensor.

This method enables decoupled input/output processing where data can be pushed and popped independently. Useful for buffered processing scenarios.

Note

This method is real-time safe and does not allocate memory.

Parameters:

• input_data – Input audio data organized as data[channel][sample]

• num_input_samples – Number of input samples to push

• tensor_index – Index of the tensor to receive the data (default: 0)

void push_data(const float *const *const *input_data, size_t *num_input_samples)

Pushes input data for multiple tensors simultaneously.

Note

This method is real-time safe and does not allocate memory.

Parameters:

• input_data – Input data organized as data[tensor_index][channel][sample]

• num_input_samples – Array of input sample counts for each tensor

size_t pop_data(float *const *output_data, size_t num_output_samples, size_t tensor_index = 0)

Pops processed output data from the pipeline for a specific tensor (non-blocking)

Retrieves processed data from the inference pipeline for a specific tensor. Should be used in conjunction with push_data for decoupled processing. This method is non-blocking and returns immediately with available samples.

Note

This method is real-time safe and does not allocate memory.

Parameters:

• output_data – Output buffer organized as data[channel][sample]

• num_output_samples – Maximum number of samples the output buffer can hold

• tensor_index – Index of the tensor to retrieve data from (default: 0)

Returns:

Number of samples actually written to the output buffer

size_t pop_data(float *const *output_data, size_t num_output_samples, std::chrono::steady_clock::time_point wait_until, size_t tensor_index = 0)

Pops processed output data from the pipeline for a specific tensor (blocking with timeout)

Retrieves processed data from the inference pipeline for a specific tensor. This method blocks until data is available or until the specified timeout is reached. Should be used in conjunction with push_data for decoupled processing.

Note

This method is not 100% real-time safe due to potential blocking to wait for data.

Parameters:

• output_data – Output buffer organized as data[channel][sample]

• num_output_samples – Maximum number of samples the output buffer can hold

• wait_until – Time point until which to wait for available data

• tensor_index – Index of the tensor to retrieve data from (default: 0)

Returns:

Number of samples actually written to the output buffer

size_t *pop_data(float *const *const *output_data, size_t *num_output_samples)

Pops processed output data for multiple tensors simultaneously (non-blocking)

Retrieves processed data for all tensors from the inference pipeline. This method is non-blocking and returns immediately with available samples for each tensor.

Note

This method is real-time safe and does not allocate memory.

Parameters:

• output_data – Output buffers organized as data[tensor_index][channel][sample]

• num_output_samples – Array of maximum output sample counts for each tensor

Returns:

Array of actual output sample counts for each tensor

size_t *pop_data(float *const *const *output_data, size_t *num_output_samples, std::chrono::steady_clock::time_point wait_until)

Pops processed output data for multiple tensors simultaneously (blocking with timeout)

Retrieves processed data for all tensors from the inference pipeline. This method blocks until data is available for each tensor or until the specified timeout is reached.

Note

This method is not 100% real-time safe due to potential blocking to wait for data.

Parameters:

• output_data – Output buffers organized as data[tensor_index][channel][sample]

• num_output_samples – Array of maximum output sample counts for each tensor

• wait_until – Time point until which to wait for available data

Returns:

Array of actual output sample counts for each tensor

unsigned int get_latency(size_t tensor_index = 0) const

Gets the processing latency for a specific tensor.

Returns the latency introduced by the inference processing in samples for a specific tensor. This includes buffering delays and model-specific processing latency.

Parameters:

tensor_index – Index of the tensor to query (default: 0)

Returns:

Latency in samples for the specified tensor

std::vector<unsigned int> get_latency_vector() const

Gets the processing latency for all tensors.

Returns:

Vector containing latency values in samples for each tensor index

size_t get_available_samples(size_t tensor_index, size_t channel = 0) const

Gets the number of samples received for a specific tensor and channel.

This method is useful for monitoring the data flow, benchmarking and debugging purposes.

Parameters:

• tensor_index – Index of the tensor to query

• channel – Channel index to query (default: 0)

Returns:

Number of samples received for the specified tensor and channel

void set_non_realtime(bool is_non_realtime)

Configures the handler for non-real-time (offline) operation.

When enabled, process()/pop_data() block the calling thread until every pending inference for this session completes, instead of returning early or giving up at a deadline. Output is therefore always complete &#8212; never a dropped/zero-filled chunk &#8212; at the cost of an unbounded wait, so this is intended for offline rendering (e.g. bounce-to-disk), not the live audio thread.

Warning

Not real-time safe while enabled. Refused with a warning when no inference threads exist to satisfy the waits (see InferenceManager::set_non_realtime()); on WebAssembly spin up at least one inference worker first, and prefer running offline processing in a Worker or under an OfflineAudioContext &#8212; the waits there spin.

Parameters:

is_non_realtime – True to block for complete output (non-real-time mode), false to restore the bounded/non-blocking real-time behavior

void reset()

Resets the inference handler to its initial state (wait-free, real-time safe).

Clears the internal audio ring buffers, re-seeds the latency zero-padding, and invalidates every inference dispatched so far: results still in flight are discarded and their internal structures reclaimed lazily. The handler is ready for new data immediately — intended for stream re-anchoring (e.g. transport jumps or onset/transient re-sync), for stateless and stateful (session-exclusive) configurations alike. Never waits, sleeps, locks, allocates, or performs any syscall, and is annotated [[clang::nonblocking]] under RealtimeSanitizer builds.

Note

Call from the thread that drives process()/push_data()/pop_data() (or ensure no such call is concurrent), and never concurrently with prepare() or destruction.

Note

Does NOT wait for in-flight inferences to finish: a worker thread may still be executing a — discarded — inference after this returns, including user code in a custom backend or the PrePostProcessor::before_inference()/after_inference() hooks. If you need that quiescence (e.g. before mutating state such code reads), call prepare() instead, or synchronize within your own backend.

Note

Until in-flight work finishes (bounded by one inference duration), its internal structures stay captive; if fresh data submitted in that window exhausts the remaining pool — likely on session-exclusive configurations, whose pools are small — the affected chunks complete as silence at their correct stream positions. The stream stays time-aligned and recovers by itself.

Note

Model-internal state (e.g. a recurrent hidden state inside the backend) is not touched — no reset variant has ever reset it; splice such state via the before_inference()/after_inference() hooks.

Public Static Functions

static unsigned int get_num_inference_threads()

Number of inference threads currently active in the process.

Process-wide, not per-session: all sessions share one thread pool. Native: threads currently executing their processing loop — the auto-managed pool once started plus any user-created threads. WebAssembly: the inference workers currently spun up (started and not yet stopped). Useful e.g. to verify threads exist before enabling non-real-time mode. See Context::get_num_inference_threads().

Returns:

Number of active inference threads.