Capturas de pantalla del proyecto
Resumen
LLMix acts as an intermediate harness between an application and its LLM provider SDKs. Instead of replacing existing OpenAI, Anthropic, or LiteLLM client code, it wraps the calls to manage the surrounding infrastructure. It shifts model behavior from hard-coded strings into MDA (config) presets, allowing teams to hot-swap models without redeploying applications. The layer introduces cross-runtime parity for operational concerns like response caching, circuit breaking, key pooling, and singleflight deduplication.
Características principales
- Config-driven MDA presets for dynamic model switching
- Signed and compiled configuration registry with tamper-rejection
- Two-tier response cache (L1 memory, L2 Redis) with cross-runtime canonical keys
- Key pools with round-robin selection, 429 rotation, and 401/403 dead-key eviction
- Jittered exponential backoff retries honoring Retry-After headers
- Circuit breakers scoped by provider and effective base URL
- Singleflight deduplication for concurrent identical requests
- AIMD adaptive concurrency control driven by rate-limit feedback
- Cross-runtime parity for Python, TypeScript, and Rust
- Dispatch helpers for OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU
Requisitos, instalación y guía rápida
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
Optional dependencies:
TypeScript OpenAI helpers: `npm install ai @ai-sdk/openai`
Python Redis cache: `pip install "sno-llmix[redis]"`
Rust OpenAI helper and Redis cache: `cargo add llmix-rs --features providers-openai,redis`
Uso
TypeScript Example:
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python Example:
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
Compatibilidad de modelos y casos de uso
Compatible with OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU models via dedicated dispatch helpers. OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base URL handling.
Notas sobre la licencia y los riesgos
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Lanzamiento y mantenimiento
Not stated in the repository metadata