추론, 배포 및 런타임

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

공식 저장소 열기 ↗ 프로젝트 홈페이지 ↗
★ 312별점
⑂ 35포크 수
0미해결 이슈
Rust언어
MIT라이선스
Q@project.QualityScore편집 점수

프로젝트 스크린샷

memra 스크린샷 memra 스크린샷 memra 스크린샷 memra 스크린샷 memra 스크린샷 memra 스크린샷

개요

memra is a high-performance LLM inference engine written from scratch in Rust and CUDA. It targets two specific GPU architectures—NVIDIA RTX 50-series Blackwell (sm_120a) and H100 Hopper (sm_90a)—with auto-detection at build time. The engine differentiates itself through a strict exactness discipline: speculative and graph-replay outputs are guaranteed token-identical to plain decode, and greedy serving is isolated-identical under concurrent load, ensuring that speed and batching never alter model outputs. It features hand-written kernels tuned against measured hardware limits and includes comprehensive per-model performance benchmarking against llama.cpp and vLLM.

주요 기능

  • From-scratch Rust + CUDA engine with no ggml dependency
  • Auto-detected dual-architecture support for RTX 5090 (sm_120a) and H100 (sm_90a)
  • Strict exactness-gated inference ensuring token-identical outputs
  • MTP speculative decoding with adaptive draft depth and confidence cuts
  • MoE on 24GB via resident-if-fits expert residency and AUTO-KQUANT
  • Hopper wgmma/TMA kernels and FA3-class prefill attention
  • CUDA-graph decode with minimal host traffic
  • OpenAI-compatible server with batched decode and cross-request prefill batching
  • m-invariant prefill router ensuring isolated-identical concurrent serving

요구 사항, 설치 및 빠른 시작

Prebuilt Linux x86_64 binaries (sm_120a) are available on the GitHub releases page. To build from source, use `cargo build --release`. The build probes the GPU's compute capability and selects the arch automatically; `MEMRA_CUDA_ARCH` can be used to override. At startup, the engine verifies the binary matches the device and fails early with a rebuild hint if there is a mismatch (`MEMRA_ARCH_CHECK=0` bypasses).

사용 정보

Run `./target/release/kernel-check` to verify every kernel against the CPU reference (must end with `ALL GREEN`). For generation, use `MEMRA_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."`. For speculative decoding, use `MEMRA_SPEC_K=3 ./target/release/run-spec /path/to/model.gguf`. To auto-download and run from Hugging Face, use `./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"`. To start the OpenAI-compatible server, run `./target/release/memra-server`.

모델 호환성 및 사용 사례

Supported models include Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma-4 12B, Gemma-4 26B-A4B MoE, Gemma-4 31B, Gemma-4 E4B, Ornith-1.0-9B, and Ornith-1.0-35B. Hy3 Layer103.5 is supported but under tuning. KAT-Coder-V2.5 and Qwen-AgentWorld-35B-A3B are in bring-up. MiniMax-M3 REAP50 is in progress.

라이선스 및 위험 참고 사항

Repository metadata declares MIT. Before publishing or commercial use, review the repository LICENSE file, copyright notices and dependency terms.

Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: MIT. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

릴리스 및 유지 관리

Not stated in the repository metadata

OpenVINO

openvinotoolkit/openvino

★ 10.6KC++

GPUStack

gpustack/gpustack

★ 5.4KPython