Class anira::InferenceThread

class InferenceThread : public anira::HighPriorityThread

Inheritance diagram for anira::InferenceThread:

Collaboration diagram for anira::InferenceThread:

Thread class for executing neural network inference operations.

The InferenceThread class provides a dedicated thread for executing neural network inference operations in real-time audio processing contexts. It manages a concurrent queue of inference requests and processes them with minimal latency while maintaining thread safety and real-time performance guarantees.

On native builds, this inherits from HighPriorityThread and owns its own OS thread. Under Emscripten there is no owned OS thread — a JS Worker drives the loop externally by calling run_loop(), and start()/stop() simply flip an atomic flag. This is required because each WASM worker instance shares memory with the main instance; spawning OS threads from C++ inside a worker would interact badly with the shared allocator.

Dequeueing is deliberately done without a moodycamel::ConsumerToken: the non-tokenized try_dequeue scans all producer sub-queues, so any enqueued task is reliably picked up even by a single consumer, and it never allocates — execute() and run_loop() stay fully allocation-free. A ConsumerToken’s sticky sub-queue rotation is a many-consumer throughput optimization that can intermittently miss items enqueued via producer tokens (lost inference tasks, see issue #77).

Public Functions

InferenceThread(InferenceQueue &next_inference, WaitStrategy wait_strategy = WaitStrategy::SpinBackoff)

Constructor that initializes the inference thread with a task queue.

Creates an inference thread that will process inference requests from the provided concurrent queue. The thread is not started automatically and must be explicitly started using the start() method.

Parameters:

• next_inference – Reference to a thread-safe concurrent queue containing inference data structures to process

• wait_strategy – How run_loop() waits for new work when the queue is empty (see WaitStrategy). Ignored on WebAssembly builds, where JS Workers drive the loop cooperatively.

~InferenceThread() override

bool execute()

Executes a single iteration of inference processing.

Attempts to dequeue and process one inference request from the queue. This method is designed to be called repeatedly in a loop and provides efficient processing with automatic backoff when no work is available.

The method handles:

• Dequeuing inference data from the concurrent queue

• Processing the inference request through the appropriate session

• Managing CPU usage through exponential backoff strategies

• Thread-safe access to shared data structures

Note

This method is real-time safe and designed for repeated calls in a high-frequency processing loop.

Returns:

True if an inference operation was executed, false if no work was available

void run_loop()

Run the main processing loop.

Natively, this is invoked by the inherited HighPriorityThread via the run() override, and waits for work according to the configured WaitStrategy: either the exponential-backoff polling loop or a blocking wait on the queue’s semaphore. Under Emscripten, JS Workers call this directly and the loop always polls (blocking is not possible there). Returns when should_exit() becomes true.

Public Static Functions

static unsigned int get_num_active_threads()

Number of inference threads currently active in the process.

Native: threads currently executing run_loop() — the auto-managed pool once started plus any user-created threads. WebAssembly: externally driven threads between start() and stop(); counted there (start() runs synchronously on the main instance) rather than at run_loop() entry, so the count is already visible when AniraWeb.spinUpInferenceWorker() returns, before the worker asynchronously enters its loop. The counter has static storage duration — on WebAssembly that is shared memory, so every WASM instance sees the same value.