Suy luận, Triển khai & Thời gian chạy

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

Mở kho lưu trữ chính thức ↗ Trang chủ dự án ↗
★ 312Số sao
⑂ 35Fork
0Vấn đề đang mở
RustNgôn ngữ
MITGiấy phép
Q@project.QualityScoreĐiểm biên tập

Ảnh chụp màn hình dự án

Ảnh chụp màn hình của memra Ảnh chụp màn hình của memra Ảnh chụp màn hình của memra Ảnh chụp màn hình của memra Ảnh chụp màn hình của memra Ảnh chụp màn hình của memra

Tổng quan

memra is a high-performance LLM inference engine written from scratch in Rust and CUDA. It targets two specific GPU architectures—NVIDIA RTX 50-series Blackwell (sm_120a) and H100 Hopper (sm_90a)—with auto-detection at build time. The engine differentiates itself through a strict exactness discipline: speculative and graph-replay outputs are guaranteed token-identical to plain decode, and greedy serving is isolated-identical under concurrent load, ensuring that speed and batching never alter model outputs. It features hand-written kernels tuned against measured hardware limits and includes comprehensive per-model performance benchmarking against llama.cpp and vLLM.

Tính năng chính

  • From-scratch Rust + CUDA engine with no ggml dependency
  • Auto-detected dual-architecture support for RTX 5090 (sm_120a) and H100 (sm_90a)
  • Strict exactness-gated inference ensuring token-identical outputs
  • MTP speculative decoding with adaptive draft depth and confidence cuts
  • MoE on 24GB via resident-if-fits expert residency and AUTO-KQUANT
  • Hopper wgmma/TMA kernels and FA3-class prefill attention
  • CUDA-graph decode with minimal host traffic
  • OpenAI-compatible server with batched decode and cross-request prefill batching
  • m-invariant prefill router ensuring isolated-identical concurrent serving

Yêu cầu, cài đặt và bắt đầu nhanh

Prebuilt Linux x86_64 binaries (sm_120a) are available on the GitHub releases page. To build from source, use `cargo build --release`. The build probes the GPU's compute capability and selects the arch automatically; `MEMRA_CUDA_ARCH` can be used to override. At startup, the engine verifies the binary matches the device and fails early with a rebuild hint if there is a mismatch (`MEMRA_ARCH_CHECK=0` bypasses).

Cách sử dụng

Run `./target/release/kernel-check` to verify every kernel against the CPU reference (must end with `ALL GREEN`). For generation, use `MEMRA_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."`. For speculative decoding, use `MEMRA_SPEC_K=3 ./target/release/run-spec /path/to/model.gguf`. To auto-download and run from Hugging Face, use `./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"`. To start the OpenAI-compatible server, run `./target/release/memra-server`.

Khả năng tương thích của mô hình và trường hợp sử dụng

Supported models include Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma-4 12B, Gemma-4 26B-A4B MoE, Gemma-4 31B, Gemma-4 E4B, Ornith-1.0-9B, and Ornith-1.0-35B. Hy3 Layer103.5 is supported but under tuning. KAT-Coder-V2.5 and Qwen-AgentWorld-35B-A3B are in bring-up. MiniMax-M3 REAP50 is in progress.

Ghi chú về giấy phép và rủi ro

Repository metadata declares MIT. Before publishing or commercial use, review the repository LICENSE file, copyright notices and dependency terms.

Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: MIT. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

Phát hành và bảo trì

Not stated in the repository metadata

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython