State, sharing, capabilities and edges#
What a registered engine needs beyond its lifecycle (Custom Engines): the declared state it is fed, when anira shares one load between handlers, what it reports it can serve, and how it states its edges.
Declared state and the two buffers#
A model with a declared State pair (Declared state) is fed its state by anira without a copy:
the handler owns two buffers per pair, binds the model’s State input to the one the last inference
wrote and its State output to the other, and flips the pair behind every successful inference. For
the engine this is the adapter rule and nothing more: the two descriptors name the pair’s buffers in
turn from one call to the next, so an engine that read the data pointer at load or prepare would read
the wrong half every other inference. An engine that binds caller memory (an Ort::Value over the
descriptor, a torch::from_blob view) runs the pair without a copy; one that copies its inputs
copies the pair as it copies every other slot.
An engine whose runtime wants the addresses to stay put, a captured graph above all (CUDA graphs, WebGPU replay), sets
ANIRA_ENGINE_FLAG_STATE_ALIAS: for every call of a plan of that engine anira binds one buffer as both halves and never flips, and the engine reads the state before it writes it (in place) or copies for itself. A failed call leaves the buffer as the engine left it (the last-good rule is the engine’s to keep here), and a chunk of a plan without the bit in between flips the pair, so the addresses stay put across the chunks of aliasing plans alone (re-capture after such a switch).The pair is the model’s, not a plan’s: every plan of the variant runs it, in host memory in this pre-release (a
host_domainon the State input overrides it), so a plan switch keeps the state whether the plans flip or alias, and the first inference of a new stream reads zeros either way.A model with a State pair is stateful: it is loaded once with no shared slot, and every handler of it is exclusive, so an engine with state of its own beside the pair builds it at
prepareand re-initialises it atreset.The two buffers take the spec’s dtype, whatever the ring dtype is. A registered engine binds what its
loadaccepts; a built-in one refuses at load, withANIRA_ERROR_NOT_SUPPORTEDnaming the engine and the tensor, what its runtime cannot carry.
What a custom engine can serve here#
A built-in engine’s capabilities are the context’s (Capabilities). A custom engine has
the same three parts, on the pipeline it belongs to: its declared vocabulary (providers), a
measurement (the query slot) and a visible report, the pipeline’s capabilities.
query (anira_engine_query_fn, a tail field) is the engine’s own GetAvailableProviders:
which of providers are usable here, now, as a bitmask over the list (bit i: providers[i];
a descriptor without a list serves the CPU path without a bit). It runs before init and any number
of times, on the main thread, at anira_handler_create and at the pipeline’s capabilities entries,
with an anira_init_info; it may log and never calls an entry that takes the lifecycle lock.
anira_handler_prepare never calls it: the handler keeps its create’s answer and checks the plans
against it again. A status other than ANIRA_OK fails the calling entry, naming the engine; a
declared provider the query cleared is refused for a candidate as “declares provider ‘x’ but its
query reports it unavailable here”. In C++: Engine::query(const InitInfo&).
/* The rows a handler of this pipeline sees on this context: the context's, then one per
added engine and provider usable here (engine ANIRA_ENGINE_CUSTOM, engine_id the
engine's id): the listed providers the query keeps, in list order, or the CPU path
alone for an engine without a list. Every call runs the engines' queries. */
uint32_t count = 0;
anira_pipeline_capabilities_backends(pipe, context, sizeof(anira_backend_id), &count, NULL);
anira_backend_id* rows = malloc(count * sizeof *rows);
anira_pipeline_capabilities_backends(pipe, context, sizeof(anira_backend_id), &count, rows);
/* One edge, by its two ends: host memory to the engine on a provider (zero-copy to
ANIRA_PROVIDER_CPU and XNNPACK; to every other a move the engine makes itself,
ANIRA_MOVER_ENGINE, whose class is the memory kind's: ANIRA_EDGE_UNIFIED_TRANSFER on
unified memory, ANIRA_EDGE_DEDICATED_TRANSFER on dedicated memory, ANIRA_EDGE_UNKNOWN on
unknown memory, edge.memory_kind; Core ML's device is the Metal default device). */
anira_edge_info edge = ANIRA_EDGE_INFO_INIT;
edge.from_domain = ANIRA_DOMAIN_HOST;
edge.to_engine = ANIRA_ENGINE_CUSTOM;
edge.to_engine_id = "com.example.myengine";
edge.to_provider = ANIRA_PROVIDER_COREML;
anira_pipeline_capabilities_edge(pipe, context, &edge);
/* The way back, from the engine to host memory: the same class and mover for the
provider rule, the reason naming the readback where the other names the upload. */
anira_edge_info back = ANIRA_EDGE_INFO_INIT;
back.from_engine = ANIRA_ENGINE_CUSTOM;
back.from_engine_id = "com.example.myengine";
back.from_provider = ANIRA_PROVIDER_COREML;
back.to_domain = ANIRA_DOMAIN_HOST;
anira_pipeline_capabilities_edge(pipe, context, &back);
The rows’ strings point into the pipeline’s engines and the context’s store: valid until the pipeline
is destroyed and until the context’s next probe, a reason while the context lives. The backends list
asks no device’s memory kind; the edge entry asks it for the row it reads, once per device API, and
the context keeps the answer. anira_handler_create asks the same question for every candidate’s
provider of the engine, so what the report lists is what a handler runs on. In C++:
pipe.capabilities(context).backends() / .edge(from, to)
(anira::PipelineCapabilities).
How the engine passes the slots#
The plan report says per slot where the edge takes the bytes and who moves them
(anira_plan_slot.edge_class and moved_by, Edges, movers and memory), and the contract’s edge cost
drops a plan above its limit (Wait ratio, ring dtypes, latencies, host domains, edge cost). anira hands a custom engine the chunk’s
buffer itself, so by default its rows follow the provider rule: zero-copy on
ANIRA_PROVIDER_CPU and XNNPACK; on any other provider the engine moves an input to its device and
an output back itself, ANIRA_EDGE_UNIFIED_TRANSFER where the device computes in unified memory,
ANIRA_EDGE_DEDICATED_TRANSFER where it has its own and ANIRA_EDGE_UNKNOWN where no probe
established which (the plan row’s memory_kind); every edge-cost limit refuses an unknown edge. On
ANIRA_PROVIDER_CUDA the device asked is CUDA device 0 (a custom engine’s provider options are its
own vocabulary); a custom provider’s name maps to no device, so its rows are ANIRA_EDGE_UNKNOWN.
An engine that does something else says so in the descriptor’s edge slot
(anira_engine_edge_fn): anira asks it once per slot right after each load, inputs first, on the
thread and under the lock of the load, with an anira_engine_edge already filled with the provider
rule’s edge for the slot’s direction (the same class both ways, the reason naming an input’s upload
or an output’s readback), and the engine overwrites what differs. The mover follows from the class:
whatever moves, the engine’s code moves it (ANIRA_MOVER_ENGINE; ANIRA_MOVER_NONE on a
zero-copy edge). anira copies the reason before the call returns.
/* The engine copies its outputs out of its runtime's own tensors; its inputs it binds. */
static anira_status ANIRA_CALL my_edge(void* loaded, uint32_t is_input, uint32_t slot,
anira_engine_edge* out, void* user_data) {
(void)loaded; (void)slot; (void)user_data;
if (!is_input) {
out->edge_class = ANIRA_EDGE_COPY;
out->reason = "com.example.myengine: the runtime returns tensors of its own";
}
return ANIRA_OK;
}
desc.edge = my_edge;
A status other than ANIRA_OK fails the prepare with it, a class anira_edge_class does not
name with ANIRA_ERROR_ENGINE; either way the load’s unload follows. In C++ the statement is a
member of the loaded model, Engine::Loaded::edge(is_input, slot, provider_rule), whose base
returns the rule:
anira::EngineEdge edge(bool is_input, uint32_t, const anira::EngineEdge& rule) const override {
if (is_input) { return rule; }
return {.edge_class = ANIRA_EDGE_COPY,
.reason = "com.example.myengine: the runtime returns tensors of its own"};
}
On the web a JavaScript engine runs in the inference Workers, so the main thread states its edges
per direction, once for every input slot and once for every output slot, in Pipeline.addEngine’s
EngineOptions.edge ({ inputs, outputs }; per direction since the engine is added before the
model is known), and a direction left out keeps the provider rule’s
(Custom Inference Engines).