Class anira::Context

class Context

Collaboration diagram for anira::Context:

Singleton context class managing global inference resources and session coordination.

The Context class serves as a singleton manager for all neural network inference resources, including thread pools, backend processors, and session management. It provides centralized coordination for multiple inference sessions while maintaining efficient resource sharing and thread safety across the entire inference system.

Key responsibilities:

• Managing singleton instance lifecycle and configuration

• Coordinating inference thread pool with configurable size

• Managing backend processor instances (LibTorch, ONNX, TensorFlow Lite)

• Session creation, management, and cleanup

• Thread-safe concurrent queue management for inference requests

• Resource pooling and efficient allocation/deallocation

The Context uses a singleton pattern to ensure:

• Global resource coordination across multiple inference instances

• Efficient sharing of expensive resources (thread pools)

• Centralized configuration and lifecycle management

• Thread-safe access to shared components

See also

ContextConfig, SessionElement, InferenceThread, PrePostProcessor, BackendBase

Note

This class is thread-safe and manages its own lifecycle. All access should be through the static interface methods rather than direct instantiation.

Public Functions

Context(const ContextConfig &context_config)

Constructor that initializes the context with specified configuration.

Creates a new context instance with the provided configuration settings. This constructor is should not be called directly. Use get_instance() to obtain a context instance.

Parameters:

context_config – Configuration settings for thread pool size, backend preferences, etc.

~Context() = default

Destructor that cleans up all context resources.

Properly shuts down the thread pool, releases all backend processors, and cleans up any remaining sessions or inference data.

void prepare_session(const std::shared_ptr<SessionElement> &session, HostConfig new_config, std::vector<long> custom_latency = {})

Prepares a session for processing with new audio configuration.

Configures the specified session with new audio host settings and optional custom latency values. This method handles buffer allocation, latency calculation, and session state updates.

Note

Thread-safe with respect to other sessions’ lifecycle calls. Not safe against concurrent processing calls on the same session — the host must not process a session it is currently preparing.

Parameters:

• session – Shared pointer to the session to prepare

• new_config – New host configuration with audio settings

• custom_latency – Optional vector of custom latency values for each tensor

void new_data_submitted(const std::shared_ptr<SessionElement> &session)

Notifies the context that new data has been submitted for a session.

Signals to the inference system that new audio data is available for processing by the specified session. This triggers the inference pipeline to begin processing the submitted data.

Parameters:

session – Shared pointer to the session that has new data available

void new_data_request(const std::shared_ptr<SessionElement> &session)

Requests new data processing for a session.

Requests that the inference system process data for the specified session. This is used for scheduling and managing inference operations. The request is processed immediately.

Note

If the session is in non-real-time mode (see InferenceManager::set_non_realtime()), this blocks until the pending inference completes instead of returning immediately.

Parameters:

session – Shared pointer to the session requesting data processing

void new_data_request(const std::shared_ptr<SessionElement> &session, std::chrono::steady_clock::time_point wait_until)

Requests new data processing for a session at a specific time.

Requests that the inference system process data for the specified session, but waits for the data until the given time point before processing.

Note

If the session is in non-real-time mode (see InferenceManager::set_non_realtime()), this blocks until the pending inference completes instead of honoring wait_until.

Parameters:

• session – Shared pointer to the session requesting data processing

• wait_until – Time point at which to begin processing the data request

void reset_session(const std::shared_ptr<SessionElement> &session)

Wait-free reset of a session, safe on the session’s driving (audio) thread.

NEVER blocks the caller on in-flight inferences: it bumps the session generation (invalidating every already-dispatched inference) and then calls SessionElement::clear(). Stale inferences complete on their worker threads, have their results discarded (new_data_request() generation guard), and their structs reclaimed lazily by reclaim_stale_structs() from new_data_submitted(). The observable output is identical to the former blocking reset, which also discarded the in-flight result — it merely waited first so it could safely wipe the struct memory.

Supported for all session types. For a session-exclusive (stateful) session, the pending-dispatch chain is reconciled without waiting: pending entries are returned to the free pool by the gate-holder (this call when the gate is free, otherwise the worker at its next task boundary — see SessionElement::try_acquire_next_dispatch). Nothing is ever enqueued from this call, so it performs no queue, semaphore, or logging syscalls.

Must be called from the session’s single driving thread (the thread that runs process()/push_data()/pop_data()), or with no such call concurrent.

Parameters:

session – Shared pointer to the session to reset

Public Static Functions

static std::shared_ptr<Context> get_instance(const ContextConfig &context_config)

Gets or creates the singleton context instance.

Returns the existing context instance or creates a new one with the specified configuration if none exists. This is the primary method for accessing the global inference cntext.

Note

If a context already exists, the provided configuration is ignored. The configuration is only used when creating a new instance.

Note

Thread-safe: may be called from any non-realtime thread, including concurrently with other sessions’ lifecycle calls.

Parameters:

context_config – Configuration settings for the context (used only on first creation)

Returns:

Shared pointer to the singleton context instance

static std::shared_ptr<SessionElement> create_session(PrePostProcessor &pp_processor, InferenceConfig &inference_config, BackendBase *custom_processor)

Creates a new inference session with specified components.

Creates and registers a new inference session with the provided preprocessing/ postprocessing pipeline, inference configuration, and optional custom backend. The session is automatically assigned a unique ID and integrated into the global resource management system.

Note

Thread-safe: may be called from any non-realtime thread, including concurrently with other sessions’ lifecycle calls.

Parameters:

• pp_processor – Reference to the preprocessing/postprocessing pipeline

• inference_config – Reference to the inference configuration

• custom_processor – Pointer to custom backend processor (nullptr for default backends)

Returns:

Shared pointer to the newly created session

static void release_session(const std::shared_ptr<SessionElement> &session)

Releases an inference session and its resources.

Properly shuts down and releases the specified session, including cleanup of associated backend processors, buffers, and other resources.

Note

Thread-safe: may be called from any non-realtime thread, including concurrently with other sessions’ lifecycle calls. Exactly one releaser tears down the shared thread pool when the last session goes away.

Parameters:

session – Shared pointer to the session to release

static void release_instance()

Releases the singleton context instance.

Shuts down and releases the global context instance, including all sessions, thread pools, and backend processors. This should be called during application shutdown to ensure proper cleanup.

static void release_thread_pool()

Releases the inference thread pool.

Shuts down all inference threads and releases thread pool resources. This is typically called as part of context cleanup or reconfiguration.

static int get_num_sessions()

Gets the number of active inference sessions.

Returns the current count of active inference sessions managed by the context. This is useful for monitoring and debugging purposes.

Returns:

Number of currently active sessions

static std::vector<std::shared_ptr<SessionElement>> &get_sessions()

Gets a reference to all active sessions.

Returns a reference to the vector containing all currently active inference sessions. This method is primarily used for internal management and debugging.

Note

This method provides direct access to internal data structures and should be used carefully to avoid disrupting session management.

Returns:

Reference to the vector of active session shared pointers

static InferenceQueue &get_static_inference_queue()

Get a reference to the static inference queue Returns a reference to the global concurrent queue used for inference requests. This is used to construct InferenceThreads (user-managed or WASM worker-driven) that consume from the global queue; dequeueing is non-tokenized and allocation-free.

Returns:

Reference to the static inference queue

static std::unique_ptr<InferenceThread> make_inference_thread()

Factory for a user-owned InferenceThread bound to the static inference queue.

Returns a new InferenceThread whose lifecycle is fully managed by the caller. The thread is not started automatically — call start() on the returned object to begin processing. The caller must also call stop() (or simply destroy the object) before program exit.

This is purely additive: the auto-managed thread pool sized via ContextConfig::m_num_threads continues to work unchanged. Users who want full control over threading typically construct Context with ContextConfig(0) so that no auto-pool threads exist, then create and manage threads themselves via this factory.

The returned thread references the static inference queue, which has static storage duration — so the thread remains valid even after all sessions and the Context singleton itself are released.

Returns:

Unique pointer to a new user-owned InferenceThread.

static unsigned int get_num_inference_threads()

Number of inference threads currently active in the process.

Native: threads currently executing their processing loop — the auto-managed pool once started plus any user-created threads. WebAssembly: externally driven threads that have been started and not yet stopped (i.e. the inference workers currently spun up; exposed to JavaScript as AniraWeb.getNumInferenceThreads()). See InferenceThread::get_num_active_threads() for the exact semantics.

Returns:

Number of active inference threads.

static bool has_inference_threads()

Whether any inference threads exist that could satisfy blocking (non-real-time) waits.

True when the auto-managed pool is non-empty (native; its threads are started in prepare_session()) or at least one externally driven thread is active (user-created on native, JS-driven on WebAssembly, where the pool is always empty). Used to gate InferenceManager::set_non_realtime(true), whose unbounded waits would otherwise never complete.

Returns:

True if at least one inference thread is configured or active.