Inferenza, distribuzione e runtime

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

Apri il repository ufficiale ↗ Pagina iniziale del progetto ↗
★ 312Stelle
⑂ 35Fork
0Problemi aperti
RustLingua
MITLicenza
Q@project.QualityScorePunteggio editoriale

Screenshot del progetto

Screenshot di memra Screenshot di memra Screenshot di memra Screenshot di memra Screenshot di memra Screenshot di memra

Panoramica

memra is a high-performance LLM inference engine written from scratch in Rust and CUDA. It targets two specific GPU architectures—NVIDIA RTX 50-series Blackwell (sm_120a) and H100 Hopper (sm_90a)—with auto-detection at build time. The engine differentiates itself through a strict exactness discipline: speculative and graph-replay outputs are guaranteed token-identical to plain decode, and greedy serving is isolated-identical under concurrent load, ensuring that speed and batching never alter model outputs. It features hand-written kernels tuned against measured hardware limits and includes comprehensive per-model performance benchmarking against llama.cpp and vLLM.

Funzionalità principali

  • From-scratch Rust + CUDA engine with no ggml dependency
  • Auto-detected dual-architecture support for RTX 5090 (sm_120a) and H100 (sm_90a)
  • Strict exactness-gated inference ensuring token-identical outputs
  • MTP speculative decoding with adaptive draft depth and confidence cuts
  • MoE on 24GB via resident-if-fits expert residency and AUTO-KQUANT
  • Hopper wgmma/TMA kernels and FA3-class prefill attention
  • CUDA-graph decode with minimal host traffic
  • OpenAI-compatible server with batched decode and cross-request prefill batching
  • m-invariant prefill router ensuring isolated-identical concurrent serving

Requisiti, installazione e avvio rapido

Prebuilt Linux x86_64 binaries (sm_120a) are available on the GitHub releases page. To build from source, use `cargo build --release`. The build probes the GPU's compute capability and selects the arch automatically; `MEMRA_CUDA_ARCH` can be used to override. At startup, the engine verifies the binary matches the device and fails early with a rebuild hint if there is a mismatch (`MEMRA_ARCH_CHECK=0` bypasses).

Utilizzo

Run `./target/release/kernel-check` to verify every kernel against the CPU reference (must end with `ALL GREEN`). For generation, use `MEMRA_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."`. For speculative decoding, use `MEMRA_SPEC_K=3 ./target/release/run-spec /path/to/model.gguf`. To auto-download and run from Hugging Face, use `./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"`. To start the OpenAI-compatible server, run `./target/release/memra-server`.

Compatibilità del modello e casi d'uso

Supported models include Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma-4 12B, Gemma-4 26B-A4B MoE, Gemma-4 31B, Gemma-4 E4B, Ornith-1.0-9B, and Ornith-1.0-35B. Hy3 Layer103.5 is supported but under tuning. KAT-Coder-V2.5 and Qwen-AgentWorld-35B-A3B are in bring-up. MiniMax-M3 REAP50 is in progress.

Note su licenza e rischi

Repository metadata declares MIT. Before publishing or commercial use, review the repository LICENSE file, copyright notices and dependency terms.

Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: MIT. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

Rilascio e manutenzione

Not stated in the repository metadata

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython