Welcome to TensorRT LLM’s Documentation!#
Getting Started
Deployment Guide
- LLM Examples
- VisualGen Examples
- Online Serving Examples
- Aiperf Client
- Aiperf Client For Multimodal
- Curl Chat Client
- Curl Chat Client For Multimodal
- Curl Completion Client
- Curl Responses Client
- Deepseek R1 Reasoning Parser
- OpenAI Chat Client
- OpenAI Chat Client for Multimodal
- OpenAI Completion Client
- Openai Completion Client For Lora
- OpenAI Completion Client with JSON Schema
- OpenAI Responses Client
- Prometheus Metrics
- Dynamo K8s Example
- Model Recipes
- CPU Affinity configuration in TensorRT LLM
Models
CLI Reference
API Reference
- LLM API Introduction
- API Reference
LLMAsyncLLMMultimodalEncoderCompletionOutputRequestOutputGuidedDecodingParamsSamplingParamsDisaggregatedParamsConversationParamsDisaggScheduleStyleBlockReuseConfigKvCacheConfigMambaStateConfigKvCacheRetentionConfigCudaGraphConfigDecodeCudaGraphConfigEncodeCudaGraphConfigMoeConfigLookaheadDecodingConfigMedusaDecodingConfigEagleDecodingConfigEagle3DecodingConfigMTPDecodingConfigSchedulerConfigCapacitySchedulerPolicyQuantConfigQuantAlgoCalibConfigRequestErrorMpiCommSessionExtendedRuntimePerfKnobConfigBatchingTypeContextChunkingPolicyDynamicBatchConfigCacheTransceiverConfigNGramDecodingConfigPARDDecodingConfigDFlashDecodingConfigDSparkDecodingConfigSADecodingConfigSAEnhancerConfigUserProvidedDecodingConfigTorchCompileConfigDraftTargetDecodingConfigLlmArgsTorchLlmArgsAutoDecodingConfigAttentionDpConfigLoRARequestSaveHiddenStatesDecodingConfigRocketSparseAttentionConfigReorderRequestPolicyConfigDeepSeekSparseAttentionConfigDeepSeekV4SparseAttentionConfigMiniMaxM3SparseAttentionConfigSchedulingParamsSkipSoftmaxAttentionConfigTriAttentionKvCacheCompressionConfigPrometheusMetricsConfigPrefillCudaGraphBackendThinkingBudgetLogitsProcessoradd_thinking_budget_logits_processorMultimodalConfig
Features
- Feature Combination Matrix
- Multi-Head, Multi-Query, and Group-Query Attention
- Disaggregated Serving
- Embeddings (Encoder-Only Models)
- KV Cache System
- Long Sequences
- LoRA (Low-Rank Adaptation)
- Multimodal Support in TensorRT LLM
- Overlap Scheduler
- Paged Attention, IFB, and Request Scheduling
- Parallelism in TensorRT LLM
- Quantization
- Sampling
- Additional Outputs
- Post-Processing Hook
- Guided Decoding
- Speculative Decoding
- Checkpoint Loading
- ModelExpress (MX) Checkpoint Loading
- AutoDeploy (Beta)
- AutoDeploy Transforms
- Ray Orchestrator (Prototype)
- Torch Compile & Prefill CUDA Graph
- Helix Parallelism
- KV Cache Connector
- Sparse Attention
Developer Guide
Blogs
- Evaluating Agentic Serving with Trace Replay and Job-Level Metrics
- DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM
- Scaling Video Generation Across NVL72 rack with TensorRT-LLM
- Abstract
- Table of Contents
- The Scaling Challenge in Video Generation
- CFG Parallelism
- Ulysses Parallelism
- Context Parallelism
- Scaling the VAE Decoder
- The Parallelism Mesh
- NVL72 Scaling Results
- Serving with trtllm-serve
- Quality Evaluation
- Conclusion
- Picking the Performant Configuration
- Limitations and Future Work
- References
- Acknowledgement
- MoE as Dense GEMM: Optimizing Low-Latency MoE Inference on NVIDIA Blackwell
- Joint Optimization of Agent Applications and TensorRT-LLM
- Helix Parallelism in TensorRT-LLM: Scaling Multi-Million-Token Decoding with KV Cache Sharding
- Temporal Correlation Meets Sparse Attention: Guess-Verify-Refine Top-K for Blackwell
- Tuning CUDA Graph Batch Sizes for Higher Output Throughput
- DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
- Optimizing MoE Communication with One-Sided AlltoAll Over NVLink
- Sparse Attention in TensorRT LLM
- Accelerating Long-Context Inference with Skip Softmax Attention
- Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs
- Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary)
- Inference Time Compute Implementation in TensorRT LLM
- Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly
- Running GPT-OSS-120B with Eagle3 Speculative Decoding on GB200/B200 (TensorRT LLM)
- ADP Balance Strategy
- Running a High Performance GPT-OSS-120B Inference Server with TensorRT LLM
- Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization)
- N-Gram Speculative Decoding in TensorRT LLM
- How to launch Llama4 Maverick + Eagle3 TensorRT LLM server
- Disaggregated Serving in TensorRT LLM
- Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP)
- Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers
- DeepSeek R1 MTP Implementation and Optimization
- Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
- How to get best performance on DeepSeek-R1 in TensorRT LLM
- H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT LLM
- New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget
- H100 has 4.6x A100 Performance in TensorRT LLM, achieving 10,000 tok/s at 100ms to first token
- Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100
- Speed up inference with SOTA quantization techniques in TRT-LLM
Quick Links