Project screenshots
Overview
LLMix acts as an intermediate harness between an application and its LLM provider SDKs. Instead of replacing existing OpenAI, Anthropic, or LiteLLM client code, it wraps the calls to manage the surrounding infrastructure. It shifts model behavior from hard-coded strings into MDA (config) presets, allowing teams to hot-swap models without redeploying applications. The layer introduces cross-runtime parity for operational concerns like response caching, circuit breaking, key pooling, and singleflight deduplication.
Key features
- Config-driven MDA presets for dynamic model switching
- Signed and compiled configuration registry with tamper-rejection
- Two-tier response cache (L1 memory, L2 Redis) with cross-runtime canonical keys
- Key pools with round-robin selection, 429 rotation, and 401/403 dead-key eviction
- Jittered exponential backoff retries honoring Retry-After headers
- Circuit breakers scoped by provider and effective base URL
- Singleflight deduplication for concurrent identical requests
- AIMD adaptive concurrency control driven by rate-limit feedback
- Cross-runtime parity for Python, TypeScript, and Rust
- Dispatch helpers for OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU
Requirements, installation and quick start
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
Optional dependencies:
TypeScript OpenAI helpers: `npm install ai @ai-sdk/openai`
Python Redis cache: `pip install "sno-llmix[redis]"`
Rust OpenAI helper and Redis cache: `cargo add llmix-rs --features providers-openai,redis`
Usage
TypeScript Example:
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python Example:
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
Model compatibility and use cases
Compatible with OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU models via dedicated dispatch helpers. OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base URL handling.
License and risk notes
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Release and maintenance
Not stated in the repository metadata