ToolAI.io · GitHub Channel
A fact-based index of LLM, agent, MCP, RAG and AI development projects, with license, setup, download, screenshots and related resources for each entry.
Public repository data
xllm-ai/xllm
xLLM is a C++ inference framework for LLM, VLM, DiT and REC models. It targets high-throughput, low-latency deployment on several AI accelerator families and separates service-layer scheduling and availability from engine-layer computation.
NVIDIA/TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
avifenesh/memra
A from-scratch LLM inference engine built in Rust and CUDA, specifically optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures, delivering exactness-gated performance without relying on frameworks like ggml.
PaddlePaddle/Paddle-Lite
PaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge (飞桨高性能深度学习端侧推理引擎）
ollama/ollama
A tool for running and managing large language models locally.
hal0ai/hal0
An open-source, self-hosted home AI inference platform designed to turn a Linux box into an OpenAI-compatible inference appliance, with native optimization for AMD Strix Halo hardware.
NVIDIA/Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
mudler/LocalAI
A local inference engine for self-hosting models and AI services.
ray-project/ray
A distributed computing runtime for machine learning training, tuning and model serving.
MoonshotAI/checkpoint-engine
Checkpoint-engine is a simple middleware to update model weights in LLM inference engines
janhq/jan
A local AI chat application for running models on a personal computer.
Page 2 / 2 · 23 projects
