项目截图
概览
LLMix 充当应用程序与其 LLM 提供商 SDK 之间的中间层。它并不替代现有的 OpenAI、Anthropic 或 LiteLLM 客户端代码，而是封装这些调用以管理周边的基础设施。它将模型行为从硬编码字符串转变为 MDA（配置）预设，允许团队在不重新部署应用程序的情况下热替换模型。该层为响应缓存、熔断、密钥池和单次飞行去重等运维关注点引入了跨运行时的一致性。
主要功能
- 配置驱动的 MDA 预设，支持动态模型切换
- 带有防篡改拒绝机制的签名编译配置注册表
- 双层响应缓存（L1 内存，L2 Redis），支持跨运行时的规范键
- 密钥池支持轮询选择、429 轮换以及 401/403 失效密钥剔除
- 遵循 Retry-After 头部的带抖动指数退避重试
- 按提供商和有效基础 URL 划分作用域的熔断器
- 针对并发相同请求的单次飞行去重
- 由速率限制反馈驱动的 AIMD 自适应并发控制
- Python、TypeScript 和 Rust 的跨运行时一致性
- 针对 OpenAI、Anthropic、Gemini、OpenRouter、DeepInfra、Novita、Together 和 Sno GPU 的分派辅助库
要求、安装和快速入门
TypeScript: `npm install @snoai/llmix`
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
可选依赖：
TypeScript OpenAI 辅助库: `npm install ai @ai-sdk/openai`
Python Redis 缓存: `pip install "sno-llmix[redis]"`
Rust OpenAI 辅助库和 Redis 缓存: `cargo add llmix-rs --features providers-openai,redis`
Python: `pip install sno-llmix`
Rust: `cargo add llmix-rs`
可选依赖：
TypeScript OpenAI 辅助库: `npm install ai @ai-sdk/openai`
Python Redis 缓存: `pip install "sno-llmix[redis]"`
Rust OpenAI 辅助库和 Redis 缓存: `cargo add llmix-rs --features providers-openai,redis`
用法
使用提供商分派辅助库和可选的响应缓存初始化 `CallPipeline`。为提供商设置 API 密钥池。通过传递配置对象（提供商、模型、参数、缓存策略）和消息负载来执行调用。最后，关闭管道以释放资源。
TypeScript 示例：
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python 示例：
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
TypeScript 示例：
typescript
import { CallPipeline, KeyPool, TwoTierCache, openaiDispatch } from "@snoai/llmix";
const pipeline = new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") });
pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));
const response = await pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [{ role: "user", content: "Explain LLMix." }] });
console.log(response.content);
await pipeline.close();
Python 示例：
python
import asyncio, os
from llmix import CallInput, CallPipeline, KeyPool, PipelineConfig, TwoTierCache, openai_dispatch
async def main():
pipeline = CallPipeline(PipelineConfig(dispatch=openai_dispatch(), response_cache=TwoTierCache("memory")))
pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))
response = await pipeline.call(CallInput(config={"provider": "openai", "model": "gpt-4o-mini", "common": {"temperature": 0.2, "max_output_tokens": 512}, "caching": {"strategy": "memory"}}, messages=[{"role": "user", "content": "Explain LLMix."}]))
print(response.content)
await pipeline.close()
asyncio.run(main())
模型兼容性与使用场景
通过专用的分派辅助库兼容 OpenAI、Anthropic、Gemini、OpenRouter、DeepInfra、Novita、Together 和 Sno GPU 模型。兼容 OpenAI 的提供商会重用 OpenAI 请求结构，并处理特定于提供商的基础 URL。
许可证与风险说明
Apache-2.0
Editorial verification 2026-08-02: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
发布与维护
Not stated in the repository metadata