한국어

inference runtime

TensorRT-LLM — Optimized LLM inference

NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way

Open official repository ↗ Project homepage ↗
★ 14.2KStars
⑂ 2.6KForks
1562Open issues
C++Language
NOASSERTIONLicense
Q@project.QualityScoreEditorial score

Project screenshots

TensorRT-LLM — Optimized LLM inference screenshot TensorRT-LLM — Optimized LLM inference screenshot

Overview

NVIDIA/TensorRT-LLM is an open-source project in ToolAI's Inference and runtime category. TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way The page is intended to help technical readers decide whether the project is worth a closer review; it is not an endorsement. The C++ implementation and the repository owner NVIDIA are recorded from the catalog metadata.

Key features

  • Official repository maintained by NVIDIA; the repository description identifies it as optimized llm inference.
  • Primary language recorded for this editorial entry: C++.
  • Repository statistics and links are shown as a dated metadata snapshot, so readers can re-check them before adoption.
  • The official README, releases and issue tracker remain the source of truth for installation and compatibility details.

Requirements, installation and quick start

Requirements depend on the current official documentation. Start with the repository's README and pin a reviewed release before production use.

git clone https://github.com/NVIDIA/TensorRT-LLM
cd TensorRT-LLM

Use the project's documented environment manager, model weights and optional accelerators. Do not copy commands from an unverified mirror; check the release notes and lock compatible dependencies first.

Usage

This page is an objective orientation for optimized llm inference. Read the official quick-start example, run it in an isolated environment, and record the exact commit or release used. For an agent or model workflow, keep tool permissions narrow, validate external inputs, and measure latency and quality on a representative test set. The repository's README and examples should be consulted for API details that may change.

Model compatibility and use cases

The repository metadata describes the project scope but does not guarantee compatibility with every model, provider, checkpoint or accelerator. Confirm supported model identifiers, tokenizer versions, hardware requirements and license terms in the official documentation before connecting it to an LLM, agent, MCP server or RAG pipeline.

License and risk notes

GitHub metadata reports the repository license as NOASSERTION. Review the repository LICENSE and any separate model, dataset, weight or third-party dependency terms before redistribution or commercial use. A missing or NOASSERTION license is not a grant of permission.

Editorial verification 2026-08-02: official repository URL, owner, description, license field and repository statistics were reviewed. License metadata: NOASSERTION. Re-check README, releases and dependency/model terms before production use.

Release and maintenance

ToolAI verified the repository page and metadata on 2026-08-02. Snapshot: 14,211 stars, 2,605 forks and 1,562 open issues; repository creation is not stated and last pushed date is 2026-07-25 when supplied by the catalog. Check the official releases page for the current version rather than treating these numbers as a guarantee of maintenance or quality.