プロジェクトのスクリーンショット
概要
LLMix acts as an intermediate harness between an application and its LLM provider SDKs. Instead of replacing existing OpenAI, Anthropic, or LiteLLM client code, it wraps the calls to manage the surrounding infrastructure. It shifts model behavior from hard-coded strings into MDA (config) presets, allowing teams to hot-swap models without redeploying applications. The layer introduces cross-runtime parity for operational concerns like response caching, circuit breaking, key pooling, and singleflight deduplication.
主な機能
- Config-driven MDA presets for dynamic model switching
- Signed and compiled configuration registry with tamper-rejection
- Two-tier response cache (L1 memory, L2 Redis) with cross-runtime canonical keys
- Key pools with round-robin selection, 429 rotation, and 401/403 dead-key eviction
- Jittered exponential backoff retries honoring Retry-After headers
- Circuit breakers scoped by provider and effective base URL
- Singleflight deduplication for concurrent identical requests
- AIMD adaptive concurrency control driven by rate-limit feedback
- Cross-runtime parity for Python, TypeScript, and Rust
- Dispatch helpers for OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU
要件、インストール、クイックスタート
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
Optional dependencies:
TypeScript OpenAI helpers: `npm install ai @ai-sdk/openai`
Python Redis cache: `pip install "sno-llmix[redis]"`
Rust OpenAI helper and Redis cache: `cargo add llmix-rs --features providers-openai,redis`
使用方法
TypeScript Example:
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python Example:
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
モデルの互換性とユースケース
Compatible with OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU models via dedicated dispatch helpers. OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base URL handling.
ライセンスとリスクに関する注意事項
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
リリースとメンテナンス
Not stated in the repository metadata