Class anira::Engine#

class Engine#

Inheritance diagram for anira::Engine:

digraph {
    graph [bgcolor="#00000000"]
    node [shape=rectangle style=filled fillcolor="#FFFFFF" font=Helvetica padding=2]
    edge [color="#1414CE"]
    "1" [label="anira::Engine" tooltip="anira::Engine" fillcolor="#BFBFBF"]
    "2" [label="anira::v2::PassthroughEngine" tooltip="anira::v2::PassthroughEngine"]
    "2" -> "1" [dir=forward tooltip="public-inheritance"]
}

A custom engine as a class: the engine object. Subclass it, construct it with its reverse-URI id (Engine(std::string id)), add it to a Pipeline (Pipeline::add_engine), and name the id on a model entry (ModelConfig::add_model_path(id, path)): that entry is then a plan like a built-in engine’s, ANIRA_ENGINE_CUSTOM with the id wherever the engine-provider pair travels. It is anira_engine_desc with virtual functions in place of the function pointers, and the semantics are those of anira/abi/engine.h, whose lifecycle it follows, the processor’s lifecycle with one level more and the same words (Processor, Processor::Prepared). The object is the engine’s identity and its id the engine’s name, as an anira_custom_engine and its id are in C: the first Pipeline::add_engine of an object creates its C engine under the id (flags(), consumed_kinds() and providers() are read then, once), and every later addition of the same object, on another Pipeline, adds that same C engine for as long as anything holds it (a Pipeline, a handler created from one, a loaded model in the pool: the object keeps a detached handle of it, anira_custom_engine_detach). The levels, from the outermost in: init() is called once per C engine, by the first anira_handler_prepare that reaches it, with the facts of the core in effect (InitInfo); load() is called once per loaded model anira pools and returns that model’s Engine::Loaded, the C lifecycle’s loaded pointer as a class: the weights live there, with the shared executors of its instances; Loaded::prepare() is called once per handler that runs the model and returns that handler’s Engine::Prepared, the C prepared pointer as a class: process and reset run on it, so what an exclusive handler keeps per stream (its own executor, since its calls claim no shared instance) is a member of the Prepared, and what every handler shares a member of the Loaded; nothing an inference needs is a member of the Engine. Two handlers share one Loaded and its instances when they run the same Engine object on an equal model configuration resolved to equal tensors, whichever Pipelines they were created from (two Engine objects never share), a stateful model included: it is loaded once, with no shared instance, and every handler of it is exclusive (PrepareInfo::exclusive). So instances of one plugin that share a model load it once when they register one Engine object, kept by the plugin for the whole binary. anira owns the Prepared from prepare on and deletes it at the C unprepare (the handler’s next prepare or its destroy, on the thread of that entry, after its in-flight inferences have drained), the Loaded from load on and deletes it at the C unload (when the last handler holding it is re-prepared or destroyed, after every Prepared over it); release() runs once per C engine, when the last reference to it dies, after every Loaded was deleted, whether or not init ever ran.

anira never opens a registered row’s path or reads its bytes: load does, through the record (EngineLoadInfo::model_path(row()), model_bytes(row()), the config getters on the variant), and binds the slots by the names the record carries where its side has names, so every slot of the plan reports ANIRA_BINDING_ENGINE. process runs on an inference thread, in ANIRA_PHASE_INFERENCE, over an EngineContext, on the shared instance the context names or, for an exclusive call (EngineContext::exclusive), on what the Prepared built; it may block and must not allocate per call. reset runs for an exclusive handler at the first inference of a new stream (after prepare, after anira_handler_reset), with that inference’s context, right before its process; a handler whose model is stateless is never reset. A status other than ANIRA_OK from process fails the chunk: it lands in anira_handler_rt_error with one latched record, anira zeroes the outputs, the chunk delivers zeros at its stream position, a State pair keeps its last good value (an aliasing engine writes in place: what a failed call left in the buffer is its own).

flags() is the engine’s promise (anira_engine_desc::flags): an OR of ANIRA_ENGINE_FLAG_NEEDS_NO_MODEL, ANIRA_ENGINE_FLAG_REALTIME_SAFE, ANIRA_ENGINE_FLAG_DYNAMIC_TIME and ANIRA_ENGINE_FLAG_STATE_ALIAS, declared and reported in anira_plan_info::engine_flags in this pre-release; the default promises nothing. process and reset are noexcept (an override that throws does not compile without it, and terminates with it) and carry no real-time attribute of their own, like the C typedefs: whether process is real-time is the flag, and everything EngineContext and Tensor offer is nonblocking, so a body that keeps the promise is composed of those calls and the engine’s own real-time code.

Why the ownership spelling: the Loaded and the Prepared are polymorphic and live on the heap; load and prepare return a std::unique_ptr because anira is their one owner from then on (the trampoline releases it into the C out pointer on success; a refused call destroys nothing that was not returned); the Engine is a std::shared_ptr because it IS shared, by every pipeline it is registered on and every handler created from them: its C engine holds a copy until release.

Subclassed by anira::v2::PassthroughEngine

Public Functions

inline explicit Engine(std::string id)#

The engine object under its id: the reverse-URI name model entries name it by (ModelConfig::add_model_path(id, path)), which the plan report and every message about the engine carry, fixed for the object’s life as anira_custom_engine_create fixes a C engine’s. It is checked when the object’s C engine is created, at its first registration: an id without a ‘.’ or with the prefix “anira.” is ANIRA_ERROR_INVALID_ARGUMENT there (anira.v2.custom excepted: the id of a 2.x custom backend, anira/compat/v2.hpp). The engines of one Pipeline have distinct ids; two objects may carry one id on two Pipelines (two instances of a plugin, each with its own engine) and never share a loaded model. They share the model’s figure, which a prepare’s warm-up pass measures, so that identical plugins report the same latency: an id names one implementation (anira.v2.custom excepted, whose figures stay per object).

virtual ~Engine() = default#
Engine(const Engine&) = delete#
Engine &operator=(const Engine&) = delete#
Engine(Engine&&) = delete#
Engine &operator=(Engine&&) = delete#
inline const std::string &id() const noexcept#

The id the object was constructed with.

inline virtual uint32_t flags() const noexcept#

The engine’s promises, written into anira_engine_desc::flags at registration: an OR of ANIRA_ENGINE_FLAG_NEEDS_NO_MODEL, ANIRA_ENGINE_FLAG_REALTIME_SAFE, ANIRA_ENGINE_FLAG_DYNAMIC_TIME and ANIRA_ENGINE_FLAG_STATE_ALIAS; 0 (the default) promises nothing. A bit anira/abi/enums.h does not define is refused at registration with ANIRA_ERROR_INVALID_ARGUMENT. Read once, when the engine is registered.

inline virtual std::span<const char*const> consumed_kinds() const noexcept#

The extensions the engine reads at load, as “<host>:<kind>” strings (hosts: tensor_spec, model, model_config, context, contract); they join the consumed-or-fail walk for the entries of this engine, keyed by its id. Read once, when the engine is registered; the strings are copied.

inline virtual std::span<const char*const> providers() const noexcept#

The providers the engine serves, written into anira_engine_desc::providers at registration: exactly the listed ones (ANIRA_PROVIDER_CPU only if the list has “cpu”). The JSON spellings of anira_provider (“cpu”, “cuda”, “webgpu”, “vulkan”, “directml”, “coreml”, “xnnpack”) name a provider of the enum, any other string a custom provider in the engine’s own vocabulary (the name a candidate’s provider_id and a model entry’s pin spell); “none” and “default” are refused. Empty (the default) serves ANIRA_PROVIDER_CPU alone; a candidate naming a provider the engine does not serve is ANIRA_ERROR_NOT_SUPPORTED at anira_handler_create. Read once, when the engine is registered; the strings are copied. A provider is part of the loaded model: two providers of one model are two loads (EngineLoadInfo::provider, provider_id).

inline virtual std::uint64_t query(const InitInfo&) const#

Which of providers() are usable here, now, as a bitmask over the list (bit i set: providers()

[i]; the mask covers the list alone, and an empty list serves the CPU path without a bit): the engine’s own GetAvailableProviders, a device present, a library loaded. Called before init and any number of times, on the main thread ([main-thread]; it may log, must not call an entry that takes the core’s lifecycle lock): at anira_handler_create, where every candidate’s provider of the engine is checked against the answer (a declared provider the answer clears is ANIRA_ERROR_NOT_SUPPORTED, “declares provider ‘x’ but its query

reports it unavailable here”), and at

Pipeline::capabilities, which reports the engine’s rows beside the context’s. A handler’s prepare uses the answer its create got and never calls query, whatever thread it runs on. The base answers every bit: every declared provider is usable. It may throw, as init may: the status fails the calling entry.

inline virtual void init(const InitInfo&)#

Called once per C engine of this object, by the first anira_handler_prepare that reaches it (a row of the engine survives validation), under the core’s lifecycle lock ([main-thread]; it may allocate and log, and may query the context’s capabilities, but must not call an entry that takes that lock), before the engine’s first load, with the facts of the core in effect: the level, the thread count, the handler’s context. What the engine keeps for its whole life (a device context, a thread pool, a weights cache) is built here, as a member of the object. It may throw, as load may: the status fails that anira_handler_prepare, the C engine counts as uninitialised, and the next prepare calls init again. The base class does nothing.

virtual std::unique_ptr<Loaded> load(const EngineLoadInfo &info) = 0#

Called by anira_handler_prepare on its caller’s thread, once per loaded model anira pools ([main-thread], under the core’s lifecycle lock, so it may allocate, read files and log, but must not call an entry that takes that lock: context create and destroy, handler create, destroy and prepare, anira_inference_thread_create, anira_shutdown, anira_release_core_if_idle): loads the row’s model out of the record, binds the slots by the record’s names and sizes the shared instances the record counts (none for a model whose handlers are exclusive), and returns the Loaded, which anira owns from then on. It may throw: an anira::Error fails the prepare with its status, std::bad_alloc with ANIRA_ERROR_OUT_OF_MEMORY, anything else with ANIRA_ERROR_INTERNAL, and what() goes to the log (group “anira.hpp”), since the C callback has no error record; a null return fails it with ANIRA_ERROR_INTERNAL, since nothing could run the inference. The handler’s prepare then fails with the status, its message naming the engine, and unload is not called.

inline virtual void release() noexcept#

Called once per C engine of this object, when the last reference to it dies (every Pipeline that registered it, every handler created from them, every loaded model), on the thread of that destroy ([main-thread]), after every Loaded of it was deleted, whether or not init ever ran; no callback of that C engine runs afterwards. A registration after that creates a new C engine, answered by a release of its own. The object itself lives as long as a shared_ptr to it does.

Friends

friend struct detail::EngineTrampolines
class Loaded#

Inheritance diagram for anira::Engine::Loaded:

digraph {
    graph [bgcolor="#00000000"]
    node [shape=rectangle style=filled fillcolor="#FFFFFF" font=Helvetica padding=2]
    edge [color="#1414CE"]
    "1" [label="anira::Engine::Loaded" tooltip="anira::Engine::Loaded" fillcolor="#BFBFBF"]
    "2" [label="anira::v2::PassthroughEngine::Loaded" tooltip="anira::v2::PassthroughEngine::Loaded"]
    "2" -> "1" [dir=forward tooltip="public-inheritance"]
}

One loaded model of the engine: what Engine::load returns, the C loaded pointer as a class. The weights and the executors of the shared instances (EngineLoadInfo::instances of them, never two calls at once on one) live here; prepare hands every handler that runs the model its Prepared. Created by load on the main thread under the core’s lifecycle lock (it may allocate and read files there), deleted by anira at the C unload (when the last handler holding the model is re-prepared or destroyed), after every Prepared over it; never copied, never moved.

Subclassed by anira::v2::PassthroughEngine::Loaded

Public Functions

Loaded() = default#
virtual ~Loaded() = default#
Loaded(const Loaded&) = delete#
Loaded &operator=(const Loaded&) = delete#
Loaded(Loaded&&) = delete#
Loaded &operator=(Loaded&&) = delete#
virtual std::unique_ptr<Prepared> prepare(const PrepareInfo &info) = 0#

Called by anira_handler_prepare on its caller’s thread, once per handler that runs this model ([main-thread]; it may allocate), after the plan report is built, with the record the processor’s prepare receives: returns the Prepared that runs the handler’s inferences, which anira owns from then on. Under PrepareInfo::exclusive the handler’s calls claim no shared instance, so the Prepared builds the executor they run on (and what it keeps per stream); a stateless handler’s Prepared routes its calls to the shared instances. Of anira it may call the [callback-safe] entries, the handler’s getters and the two Static entries, never prepare, destroy or a Hard entry. The prepares of one Loaded never overlap, nor a prepare and the destruction of a Prepared of it (anira serialises them per loaded model; those of different loaded models may run at once). It may throw, as Engine::load may; a null return is ANIRA_ERROR_INTERNAL.

inline virtual EngineEdge edge(bool, uint32_t, const EngineEdge &provider_rule) const#

How this model’s engine passes one slot to its runtime (the input slot slot under is_input, else the output slot), for the plan report’s slot rows and the contract’s edge cost (anira_engine_edge_fn): called right after Engine::load returned this Loaded, once per slot, inputs first, on the same thread and under the same lock. provider_rule is what anira reports without a statement, the provider rule for the slot’s direction, and the base returns it: zero-copy on ANIRA_PROVIDER_CPU and XNNPACK; on any other provider ANIRA_EDGE_UNIFIED_TRANSFER where its device computes in unified memory, ANIRA_EDGE_DEDICATED_TRANSFER where it has memory of its own, ANIRA_EDGE_UNKNOWN where no probe established which (the plan row’s memory_kind, anira_memory_kind), the same class both ways, the reason saying which, what the probe saw and the direction’s move (an input’s upload, an output’s readback). An engine that copies the slot into host memory of its own returns ANIRA_EDGE_COPY and why. It may throw, as load may: the prepare fails with the status, and the Loaded is deleted.

class Prepared#

Inheritance diagram for anira::Engine::Prepared:

digraph {
    graph [bgcolor="#00000000"]
    node [shape=rectangle style=filled fillcolor="#FFFFFF" font=Helvetica padding=2]
    edge [color="#1414CE"]
    "1" [label="anira::Engine::Prepared" tooltip="anira::Engine::Prepared" fillcolor="#BFBFBF"]
    "2" [label="anira::v2::PassthroughEngine::Prepared" tooltip="anira::v2::PassthroughEngine::Prepared"]
    "2" -> "1" [dir=forward tooltip="public-inheritance"]
}

One handler’s prepared handle over a loaded model: what Engine::Loaded::prepare returns, the C prepared pointer as a class. process and reset run on it: a shared call on the instance the context names (the Loaded’s), an exclusive call on what this object built at prepare (its own executor). Created by prepare on the main thread (it may allocate there), deleted by anira at the C unprepare (the handler’s next prepare or its destroy), after the last process and before the Loaded; never copied, never moved.

Subclassed by anira::v2::PassthroughEngine::Prepared

Public Functions

Prepared() = default#
virtual ~Prepared() = default#
Prepared(const Prepared&) = delete#
Prepared &operator=(const Prepared&) = delete#
Prepared(Prepared&&) = delete#
Prepared &operator=(Prepared&&) = delete#
virtual anira_status process(EngineContext &ctx) noexcept = 0#

ANIRA_PHASE_INFERENCE, on an inference thread: one inference over the context’s tensors, on ctx.instance() of the Loaded, or on this object’s own executor when ctx.exclusive(). May block, must not allocate per call; every extent and every memory handle from the context’s tensors, never from load or prepare. Any status but ANIRA_OK fails the chunk (zeros at its stream position, the status latched in anira_handler_rt_error).

inline virtual void reset(EngineContext&) noexcept#

This plan’s first inference of a new stream of an exclusive handler (after prepare, after anira_handler_reset; every plan is reset at its own first inference, a plan switch alone is no boundary), with that inference’s context, right before its process; never for a handler whose model is stateless: the state this object keeps per stream starts over. The base class does nothing.