Inférence, déploiement et exécution

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

Ouvrir le dépôt officiel ↗ Page d'accueil du projet ↗
★ 312Étoiles
⑂ 35Forks
0Problèmes ouverts
RustLangue
MITLicence
Q@project.QualityScoreScore éditorial

Captures d'écran du projet

Capture d’écran de memra Capture d’écran de memra Capture d’écran de memra Capture d’écran de memra Capture d’écran de memra Capture d’écran de memra

Vue d’ensemble

memra is a high-performance LLM inference engine written from scratch in Rust and CUDA. It targets two specific GPU architectures—NVIDIA RTX 50-series Blackwell (sm_120a) and H100 Hopper (sm_90a)—with auto-detection at build time. The engine differentiates itself through a strict exactness discipline: speculative and graph-replay outputs are guaranteed token-identical to plain decode, and greedy serving is isolated-identical under concurrent load, ensuring that speed and batching never alter model outputs. It features hand-written kernels tuned against measured hardware limits and includes comprehensive per-model performance benchmarking against llama.cpp and vLLM.

Fonctionnalités clés

  • From-scratch Rust + CUDA engine with no ggml dependency
  • Auto-detected dual-architecture support for RTX 5090 (sm_120a) and H100 (sm_90a)
  • Strict exactness-gated inference ensuring token-identical outputs
  • MTP speculative decoding with adaptive draft depth and confidence cuts
  • MoE on 24GB via resident-if-fits expert residency and AUTO-KQUANT
  • Hopper wgmma/TMA kernels and FA3-class prefill attention
  • CUDA-graph decode with minimal host traffic
  • OpenAI-compatible server with batched decode and cross-request prefill batching
  • m-invariant prefill router ensuring isolated-identical concurrent serving

Prérequis, installation et démarrage rapide

Prebuilt Linux x86_64 binaries (sm_120a) are available on the GitHub releases page. To build from source, use `cargo build --release`. The build probes the GPU's compute capability and selects the arch automatically; `MEMRA_CUDA_ARCH` can be used to override. At startup, the engine verifies the binary matches the device and fails early with a rebuild hint if there is a mismatch (`MEMRA_ARCH_CHECK=0` bypasses).

Utilisation

Run `./target/release/kernel-check` to verify every kernel against the CPU reference (must end with `ALL GREEN`). For generation, use `MEMRA_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."`. For speculative decoding, use `MEMRA_SPEC_K=3 ./target/release/run-spec /path/to/model.gguf`. To auto-download and run from Hugging Face, use `./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"`. To start the OpenAI-compatible server, run `./target/release/memra-server`.

Compatibilité des modèles et cas d’usage

Supported models include Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma-4 12B, Gemma-4 26B-A4B MoE, Gemma-4 31B, Gemma-4 E4B, Ornith-1.0-9B, and Ornith-1.0-35B. Hy3 Layer103.5 is supported but under tuning. KAT-Coder-V2.5 and Qwen-AgentWorld-35B-A3B are in bring-up. MiniMax-M3 REAP50 is in progress.

Licence et notes sur les risques

Repository metadata declares MIT. Before publishing or commercial use, review the repository LICENSE file, copyright notices and dependency terms.

Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: MIT. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

Publication et maintenance

Not stated in the repository metadata

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython