Inferência, implantação e execução

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

Abrir repositório oficial ↗ Página inicial do projeto ↗
★ 312Estrelas
⑂ 35Forks
0Problemas em aberto
RustIdioma
MITLicença
Q@project.QualityScorePontuação editorial

Capturas de tela do projeto

Captura de tela de memra Captura de tela de memra Captura de tela de memra Captura de tela de memra Captura de tela de memra Captura de tela de memra

Visão geral

memra is a high-performance LLM inference engine written from scratch in Rust and CUDA. It targets two specific GPU architectures—NVIDIA RTX 50-series Blackwell (sm_120a) and H100 Hopper (sm_90a)—with auto-detection at build time. The engine differentiates itself through a strict exactness discipline: speculative and graph-replay outputs are guaranteed token-identical to plain decode, and greedy serving is isolated-identical under concurrent load, ensuring that speed and batching never alter model outputs. It features hand-written kernels tuned against measured hardware limits and includes comprehensive per-model performance benchmarking against llama.cpp and vLLM.

Principais recursos

  • From-scratch Rust + CUDA engine with no ggml dependency
  • Auto-detected dual-architecture support for RTX 5090 (sm_120a) and H100 (sm_90a)
  • Strict exactness-gated inference ensuring token-identical outputs
  • MTP speculative decoding with adaptive draft depth and confidence cuts
  • MoE on 24GB via resident-if-fits expert residency and AUTO-KQUANT
  • Hopper wgmma/TMA kernels and FA3-class prefill attention
  • CUDA-graph decode with minimal host traffic
  • OpenAI-compatible server with batched decode and cross-request prefill batching
  • m-invariant prefill router ensuring isolated-identical concurrent serving

Requisitos, instalação e início rápido

Prebuilt Linux x86_64 binaries (sm_120a) are available on the GitHub releases page. To build from source, use `cargo build --release`. The build probes the GPU's compute capability and selects the arch automatically; `MEMRA_CUDA_ARCH` can be used to override. At startup, the engine verifies the binary matches the device and fails early with a rebuild hint if there is a mismatch (`MEMRA_ARCH_CHECK=0` bypasses).

Uso

Run `./target/release/kernel-check` to verify every kernel against the CPU reference (must end with `ALL GREEN`). For generation, use `MEMRA_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."`. For speculative decoding, use `MEMRA_SPEC_K=3 ./target/release/run-spec /path/to/model.gguf`. To auto-download and run from Hugging Face, use `./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"`. To start the OpenAI-compatible server, run `./target/release/memra-server`.

Compatibilidade do modelo e casos de uso

Supported models include Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma-4 12B, Gemma-4 26B-A4B MoE, Gemma-4 31B, Gemma-4 E4B, Ornith-1.0-9B, and Ornith-1.0-35B. Hy3 Layer103.5 is supported but under tuning. KAT-Coder-V2.5 and Qwen-AgentWorld-35B-A3B are in bring-up. MiniMax-M3 REAP50 is in progress.

Observações sobre licença e riscos

Repository metadata declares MIT. Before publishing or commercial use, review the repository LICENSE file, copyright notices and dependency terms.

Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: MIT. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

Lançamento e manutenção

Not stated in the repository metadata

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython