local.ai

Main Features & Characteristics

• Powerful Native App: Built with a Rust backend, local.ai is memory efficient and compact (<10MB on Mac M2, Windows, and Linux .deb).
• CPU Inferencing: Supports CPU inferencing, adapts to available threads, and supports GGML quantization (q4, 5.1, 8, f16).
• Model Management: Keep track of AI models in one centralized location, directory agnostic; features a resumable, concurrent downloader and usage-based sorting.
• Digest Verification: Ensure model integrity with robust BLAKE3 and SHA256 digest compute features; includes a known-good model API, license and usage chips, BLAKE3 quick check, and model info card.
• Inferencing Server: Start a local streaming server for AI inferencing in 2 clicks (Load model, then start server); includes a quick inference UI, writes to .mdx, inference params, and remote vocabulary.

Workflow

Start an inference session (e.g., with WizardLM 7B model) in 2 clicks. Powers any AI app, offline or online (e.g., in tandem with window.ai).

Supported Platforms

Available for Windows (.MSI, .EXE), macOS (M1/M2, Intel), and Linux (AppImage, .deb).

Pricing

Free and open-source, source codes are licensed under GPLv3.

Upcoming Features

GPU Inferencing, Parallel session, Nested directory, Custom Sorting and Searching, Model Explorer, Model Search, Model Recommendation, Server Manager, /audio, /image.