ToolAI.io · GitHub Channel

Trending AI Projects on GitHub

Track active open-source AI repositories across LLM, agent, MCP, RAG and coding topics, with stars and verified project facts.

Public repository data

Project index

48 projects
Inference, Deployment & Runtime

sglang

sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

★ 32.2K⑂ 8.1KPython
Apache-2.0Q98
Fine-tuning, Training & Data

DeepSpeed

deepspeedai/deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

★ 43K⑂ 4.9KPython
Apache-2.0Q94
Inference, Deployment & Runtime

vllm

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

★ 89.5K⑂ 21KPython
Apache-2.0Q98
Screenshot of TensorRT-LLM — Optimized LLM inference
Inference, Deployment & Runtime

TensorRT-LLM — Optimized LLM inference

NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way

★ 14.4K⑂ 2.7KC++
NOASSERTIONQ82
Screenshot of hal0
Inference, Deployment & Runtime

hal0

hal0ai/hal0

An open-source, self-hosted home AI inference platform designed to turn a Linux box into an OpenAI-compatible inference appliance, with native optimization for AMD Strix Halo hardware.

★ 68⑂ 7Python
Apache-2.0Q89
Inference, Deployment & Runtime

xLLM: High-Performance Inference Engine for Diverse AI Accelerators

xllm-ai/xllm

xLLM is a C++ inference framework for LLM, VLM, DiT and REC models. It targets high-throughput, low-latency deployment on several AI accelerator families and separates service-layer scheduling and availability from engine-layer computation.

★ 1.5K⑂ 279C++
Apache-2.0Q98
Inference, Deployment & Runtime

vLLM Ascend

vllm-project/vllm-ascend

A community-maintained hardware plugin that enables vLLM to run model inference and serving workloads on supported Ascend NPUs.

★ 2.7K⑂ 2.1KC++
Apache-2.0Q98
Agents & Multi-Agent

dstack

dstackai/dstack

A vendor-agnostic control plane for provisioning and orchestrating training, inference, development, and agentic workloads across GPU clouds, Kubernetes, and on-premises infrastructure.

★ 2.2K⑂ 249Python
MPL-2.0Q98
Screenshot of memra
Inference, Deployment & Runtime

memra

avifenesh/memra

A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.

★ 314⑂ 35Rust
MITQ92
Inference, Deployment & Runtime

any-llm: A Unified Python Interface for LLM Providers

mozilla-ai/any-llm

any-llm is a Python SDK for communicating with multiple LLM providers through one interface. It supports switching providers and models with minimal code changes while using official provider SDKs when available.

★ 2.2K⑂ 211Python
Apache-2.0Q98
Screenshot of tiktoken — Fast tokenization
Inference, Deployment & Runtime

tiktoken — Fast tokenization

openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models

★ 19K⑂ 1.6KRust
MITQ86
Screenshot of Xorbits Inference (Xinference)
Inference, Deployment & Runtime

Xorbits Inference (Xinference)

xorbitsai/inference

A powerful and versatile library designed to serve language, speech recognition, and multimodal models. It allows users to swap GPT for any LLM by changing a single line of code and run models on cloud, on-prem, or locally via a unified, production-ready inference API.

★ 9.5K⑂ 853Python
Apache-2.0Q98

Page 2 / 4 · 48 projects

Recently updated

Craft Anythingstorytold/craft-anything★ 27.4K elephant-goldfish-modelgoogle/elephant-goldfish-model★ 15.6K DeepInferdeepseek-ai/DeepInfer★ 25.7K ComfyUIComfy-Org/ComfyUI★ 136.7K CrewAICrewAIInc/crewAI★ 59.5K Microsoft Agent Frameworkmicrosoft/agent-framework★ 14K

Most starred

ECCaffaan-m/ECC★ 248.8K Hermes Agentnousresearch/hermes-agent★ 227.1K tensorflowtensorflow/tensorflow★ 198.8K AutoGPTSignificant-Gravitas/AutoGPT★ 187.1K ollamaollama/ollama★ 180.2K markitdown — Document conversion and extractionmicrosoft/markitdown★ 174.9K