Ảnh chụp màn hình dự án
Tổng quan
LLMix acts as an intermediate harness between an application and its LLM provider SDKs. Instead of replacing existing OpenAI, Anthropic, or LiteLLM client code, it wraps the calls to manage the surrounding infrastructure. It shifts model behavior from hard-coded strings into MDA (config) presets, allowing teams to hot-swap models without redeploying applications. The layer introduces cross-runtime parity for operational concerns like response caching, circuit breaking, key pooling, and singleflight deduplication.
Tính năng chính
- Config-driven MDA presets for dynamic model switching
- Signed and compiled configuration registry with tamper-rejection
- Two-tier response cache (L1 memory, L2 Redis) with cross-runtime canonical keys
- Key pools with round-robin selection, 429 rotation, and 401/403 dead-key eviction
- Jittered exponential backoff retries honoring Retry-After headers
- Circuit breakers scoped by provider and effective base URL
- Singleflight deduplication for concurrent identical requests
- AIMD adaptive concurrency control driven by rate-limit feedback
- Cross-runtime parity for Python, TypeScript, and Rust
- Dispatch helpers for OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU
Yêu cầu, cài đặt và bắt đầu nhanh
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
Optional dependencies:
TypeScript OpenAI helpers: `npm install ai @ai-sdk/openai`
Python Redis cache: `pip install "sno-llmix[redis]"`
Rust OpenAI helper and Redis cache: `cargo add llmix-rs --features providers-openai,redis`
Cách sử dụng
TypeScript Example:
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python Example:
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
Khả năng tương thích của mô hình và trường hợp sử dụng
Compatible with OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU models via dedicated dispatch helpers. OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base URL handling.
Ghi chú về giấy phép và rủi ro
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Phát hành và bảo trì
Not stated in the repository metadata