프로젝트 스크린샷
개요
LLMix acts as an intermediate harness between an application and its LLM provider SDKs. Instead of replacing existing OpenAI, Anthropic, or LiteLLM client code, it wraps the calls to manage the surrounding infrastructure. It shifts model behavior from hard-coded strings into MDA (config) presets, allowing teams to hot-swap models without redeploying applications. The layer introduces cross-runtime parity for operational concerns like response caching, circuit breaking, key pooling, and singleflight deduplication.
주요 기능
- Config-driven MDA presets for dynamic model switching
- Signed and compiled configuration registry with tamper-rejection
- Two-tier response cache (L1 memory, L2 Redis) with cross-runtime canonical keys
- Key pools with round-robin selection, 429 rotation, and 401/403 dead-key eviction
- Jittered exponential backoff retries honoring Retry-After headers
- Circuit breakers scoped by provider and effective base URL
- Singleflight deduplication for concurrent identical requests
- AIMD adaptive concurrency control driven by rate-limit feedback
- Cross-runtime parity for Python, TypeScript, and Rust
- Dispatch helpers for OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU
요구 사항, 설치 및 빠른 시작
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
Optional dependencies:
TypeScript OpenAI helpers: `npm install ai @ai-sdk/openai`
Python Redis cache: `pip install "sno-llmix[redis]"`
Rust OpenAI helper and Redis cache: `cargo add llmix-rs --features providers-openai,redis`
사용 정보
TypeScript Example:
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python Example:
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
모델 호환성 및 사용 사례
Compatible with OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU models via dedicated dispatch helpers. OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base URL handling.
라이선스 및 위험 참고 사항
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
릴리스 및 유지 관리
Not stated in the repository metadata