Overview
The project can run as an HTTP gateway, be embedded through its Go SDK, or serve as a replacement endpoint for supported provider SDKs. A built-in web interface supports configuration, monitoring, and analytics. The repository describes both open-source capabilities and additional enterprise deployment features.
Key features
- Single OpenAI-compatible interface for 23+ providers
- Automatic retries, provider or model fallbacks, and load balancing
- Text, image, audio, and streaming request support
- Semantic response caching based on similarity
- MCP gateway capabilities for external tool access
- Virtual keys, usage tracking, budgets, rate limits, and fine-grained access control
- Prometheus metrics, distributed tracing, logging, monitoring, and analytics
- Web, API-driven, and file-based configuration
- Extensible plugin architecture
- HTTP gateway, Go SDK, and replacement endpoints for supported SDKs
- Enterprise options described for clustering, adaptive load balancing, guardrails, private networking, OIDC provisioning, custom controls, and governance
Requirements, installation and quick start
Usage
Model compatibility and use cases
The README states support for 1,000+ models across 23+ providers. Named providers include OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Cerebras, Cohere, Mistral, Ollama, and Groq. It also documents integrations for OpenAI, Anthropic, AWS Bedrock, Google GenAI, LiteLLM, and LangChain SDKs. A complete model list is not included in the supplied repository record.
License and risk notes
The repository metadata identifies the license as Apache-2.0, and the README states that the project is licensed under the Apache 2.0 License. Review the repository LICENSE file before redistribution or deployment.
Release and maintenance
Not stated in the repository metadata.