추론, 배포 및 런타임
vllm
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
★ 89.5K⑂ 21KPython
ToolAI.io · GitHub 채널
GitHub에서 인기 있는 AI 프로젝트
LLM, 에이전트, MCP, RAG 및 코딩 분야의 활발한 오픈 소스 AI 리포지토리를 추적하고, 스타 수와 검증된 프로젝트 정보를 제공합니다.
공개 리포지토리 데이터
프로젝트 색인
추론, 배포 및 런타임
TensorRT-LLM — Optimized LLM inference
NVIDIA/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way
★ 14.4K⑂ 2.7KC++
추론, 배포 및 런타임
hal0
hal0ai/hal0
An open-source, self-hosted home AI inference platform designed to turn a Linux box into an OpenAI-compatible inference appliance, with native optimization for AMD Strix Halo hardware.
★ 68⑂ 7Python
추론, 배포 및 런타임
xLLM: High-Performance Inference Engine for Diverse AI Accelerators
xllm-ai/xllm
xLLM is a C++ inference framework for LLM, VLM, DiT and REC models. It targets high-throughput, low-latency deployment on several AI accelerator families and separates service-layer scheduling and availability from engine-layer computation.
★ 1.5K⑂ 279C++
추론, 배포 및 런타임
vLLM Ascend
vllm-project/vllm-ascend
A community-maintained hardware plugin that enables vLLM to run model inference and serving workloads on supported Ascend NPUs.
★ 2.7K⑂ 2.1KC++
에이전트 및 멀티 에이전트
dstack
dstackai/dstack
A vendor-agnostic control plane for provisioning and orchestrating training, inference, development, and agentic workloads across GPU clouds, Kubernetes, and on-premises infrastructure.
★ 2.2K⑂ 249Python
추론, 배포 및 런타임
memra
avifenesh/memra
A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.
★ 314⑂ 35Rust
추론, 배포 및 런타임
any-llm: A Unified Python Interface for LLM Providers
mozilla-ai/any-llm
any-llm is a Python SDK for communicating with multiple LLM providers through one interface. It supports switching providers and models with minimal code changes while using official provider SDKs when available.
★ 2.2K⑂ 211Python
추론, 배포 및 런타임
tiktoken — Fast tokenization
openai/tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
★ 19K⑂ 1.6KRust
추론, 배포 및 런타임
Xorbits Inference (Xinference)
xorbitsai/inference
A powerful and versatile library designed to serve language, speech recognition, and multimodal models. It allows users to swap GPT for any LLM by changing a single line of code and run models on cloud, on-prem, or locally via a unified, production-ready inference API.
★ 9.5K⑂ 853Python
에이전트 및 멀티 에이전트
Plano: AI-Native Proxy Server and Data Plane for Agentic Apps
katanemo/plano
Plano is an AI-native proxy server and data plane built in Rust that centralizes LLM routing, agent orchestration, observability, and guardrails, allowing developers to focus on core agent logic rather than infrastructure plumbing.
★ 7K⑂ 484Rust
추론, 배포 및 런타임
Hugging Face Transformers
huggingface/transformers
A model-definition framework for state-of-the-art machine learning models across text, vision, audio, and multimodal domains, supporting both inference and training.
★ 163.3K⑂ 34.1KPython
2 / 4페이지 · 프로젝트 46개