Inférence, déploiement et exécution

Xorbits Inference (Xinference)

xorbitsai/inference

A powerful and versatile library designed to serve language, speech recognition, and multimodal models. It allows users to swap GPT for any LLM by changing a single line of code and run models on cloud, on-prem, or locally via a unified, production-ready inference API.

Ouvrir le dépôt officiel ↗ Page d'accueil du projet ↗
★ 9,5KÉtoiles
⑂ 853Forks
60Problèmes ouverts
PythonLangue
Apache-2.0Licence
Q@project.QualityScoreScore éditorial

Captures d'écran du projet

Capture d’écran de Xorbits Inference (Xinference) Capture d’écran de Xorbits Inference (Xinference) Capture d’écran de Xorbits Inference (Xinference) Capture d’écran de Xorbits Inference (Xinference) Capture d’écran de Xorbits Inference (Xinference) Capture d’écran de Xorbits Inference (Xinference)

Vue d’ensemble

Xinference simplifies the deployment and serving of state-of-the-art built-in or custom AI models using a single command. It supports distributed inference across multiple devices or machines and integrates with various inference engines like vLLM and ggml. The framework provides an OpenAI-compatible RESTful API, enabling seamless migration and interoperability for applications built on the OpenAI API standard.

Fonctionnalités clés

  • Model Serving Made Easy: Deploy models with a single command
  • State-of-the-Art Built-in Models: Access cutting-edge open-source models effortlessly
  • Heterogeneous Hardware Utilization: Intelligently use GPUs and CPUs via ggml
  • Flexible API and Interfaces: OpenAI-compatible RESTful API, RPC, CLI, and WebUI
  • Distributed Deployment: Distribute inference across multiple devices or machines
  • Built-in Integration with Third-Party Libraries: LangChain, LlamaIndex, Dify, Chatbox, Xagent
  • Auto batching for improved throughput
  • Agent-native Serving via Xagent integration

Prérequis, installation et démarrage rapide

Install via pip: `pip install "xinference[all]"`. For Docker, run: `docker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0`. For Kubernetes, add the helm repo: `helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts` and install via helm.

Utilisation

Start a local instance by running: `xinference-local`. Once running, models can be interacted with via the Web UI, cURL, command line, or the Xinference Python client.

Compatibilité des modèles et cas d’usage

Supports a wide range of built-in models including Llama3, ChatGLM, GLM4, Flan-T5, Gemma, Mistral, Qwen, Whisper, WizardLM, MiniMax-M3, VibeThinker, Nex-N2, Unlimited-OCR, Ornith-1.0-35B, MiniCPM5-1B, jina-embeddings-v5, and MiniCPM-V-4.6. Also supports custom models.

Licence et notes sur les risques

Licensed under Apache-2.0.

Editorial verification 2026-08-09: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.

Publication et maintenance

Xinference 3.0.0 is available with migration notes and breaking changes. Enhancements include Agent-native Serving with Xagent, Auto batch for concurrent requests, Xllamacpp for continuous batching, distributed inference across workers, and VLLM shared KV cache across multiple replicas.

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython