MI50 inference.
Three tiers of cache.
Our llama.cpp fork brings ROCm / gfx906 tuning and reusable prompt checkpoints to Atreyu’s primary inference provider.
The engine behind machinazero:primary
Atreyu’s machinazero:primary model/provider is powered by our independent llama.cpp fork, optimized for ROCm and AMD Instinct MI50 GPUs using the gfx906 architecture. The fork develops the inference and prompt-checkpoint layer; Atreyu supplies the surrounding agent, workflow, and conversation environment.
The public reference profile was developed around GLM-5.3-Flash on six MI50 cards. The provider identity belongs to Atreyu’s configuration; it is not a promise that a particular model or build will remain fixed.
Three tiers, one continuity of work
Active state
GPU-resident state serves live inference and immediate reuse.
Range checkpoints
System memory retains reusable prompt state beyond active GPU slots.
Cold backing
A bounded background writer persists immutable payloads for later recovery.
The aim is to avoid recomputing useful context when requests branch, a conversation resumes, or a model is unloaded. This is inference-state caching: it complements Atreyu’s transcript and memory systems without replacing them.
Checkpoints that understand branches
Range checkpoints preserve shared prefixes and branch tails, including attention, indexer, and recurrent state with exact token and media boundaries. Matching content can reuse compatible state instead of requiring every request to retain a separate whole-prompt snapshot.
Clients can identify branches, distinguish interactive from background work, and choose retained, ephemeral, or one-shot lifecycles. Replacement policy considers age, bounded reuse frequency, and recomputation cost while protecting shared ancestors and active leases. A branch name never authorizes reuse of different prompt content.
Tuned for MI50 / gfx906
The fork includes HIP peer-transfer controls, scheduler alias reuse, model-specific attention work, and quantized matrix kernels. Transport instrumentation makes data movement and cache behavior observable. Together, these target the practical costs of serving large models across older AMD GPUs.
The repository also contains experimental paths. Their presence is separate from their use in the reference deployment: that profile disables speculative decoding and Turbo3. Performance depends on the model, workload, topology, and configuration; the project does not claim a universal speedup.
Persistence with explicit boundaries
With persistence configured, Linux tmpfs mappings can outlive a server process, but not a machine reboot. SSD recovery covers completed writes; an in-flight capture is not yet durable. Recovery validates descriptors and loads cold payloads when needed.
Cache compatibility is tied to a verified identity covering the model, projector, chat template, runtime libraries, and state-affecting settings. A familiar filename is not enough to make saved state safe to reuse.
Open source at the inference layer
This fork is publicly available under the existing MIT license, with upstream attribution retained. Atreyu itself remains closed source. You can inspect, build, or adapt the inference engine independently; doing so does not provide access to Atreyu accounts or hosted provider credentials.
The repository’s cache-rocm branch contains build instructions, the reference configuration, storage semantics, and reproducible CPU cache checks. Those tests establish checkpoint behavior on their fixtures; they are not production throughput benchmarks. The modernized public source also needs validation separately from the older deployment build.