GPU Programming Insights

LET'S EXPLORE THE NEW FRONTIER OF EFFICIENT LANGUAGE MODELS The Problem: Matrix Multiplication Bottleneck Matrix multiplications (MatMul) are the computational bottleneck in modern Large Language Models. Every transformer layer relies on expensive MatMul operations for self-attention and feed-forward networks. This creates three major problems: 1. Computational Expense: Massive memory bandwidth and energy consumption at scale 2. Environmental Impact: Training and running LLMs creates enormous carbon footprints 3. Economic Barriers: High compute costs limit accessibility and create unfavorable unit economics The research paper "Scalable MatMul-free Language Modeling" tackles this fundamental challenge head-on. Core Innovation: Ternary Weights The breakthrough lies in simplifying the most expensive operation. Instead of using full-precision weights (32-bit floating-point numbers), the model restricts all weights to just three values: {-1, 0, +1}. How it works: - Weight = 1: Add the input value - Weight = -1: Subtract the input value - Weight = 0: Skip it entirely This transforms expensive multiplications into simple additions and subtractions. No multiplication hardware needed, dramatically reducing computational cost. Beyond weights, activations are quantized to 8-bit precision and combined with RMSNorm (Root Mean Square Normalization) for numerical stability during training. Architecture: MLGRU Replaces Self-Attention Traditional self-attention requires expensive matrix multiplications and has quadratic complexity (n²) as sequence length grows. The paper introduces MLGRU (MatMul-free Linear Gated Recurrent Unit) as a replacement. The secret: Most computations are linear and can be parallelized. Non-linearity is only added at final steps for expressivity. This is the same principle behind modern efficient models like RWKV and Mamba. Memory Optimization: Fused BitLinear Layer Traditional approaches waste time moving data between GPU memory levels (HBM to SRAM and back). The Fused BitLinear Layer combines RMSNorm and quantization into a single operation executed directly in the faster SRAM. Benefits: - Eliminates multiple data transfers between memory levels - Up to 61% memory reduction during training - More than 10x memory reduction during inference - Significantly faster training speeds For more details on this research and a deeper breakdown of the results read the following--https://lnkd.in/guyNSV86

⚙️ 𝐯𝐋𝐋𝐌, 𝐒𝐆𝐋𝐚𝐧𝐠, 𝐥𝐥𝐚𝐦𝐚.𝐜𝐩𝐩, 𝐓𝐞𝐧𝐬𝐨𝐫𝐑𝐓-𝐋𝐋𝐌 — 𝐭𝐡𝐞𝐲 𝐚𝐥𝐥 𝐬𝐨𝐥𝐯𝐞 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐩𝐫𝐨𝐛𝐥𝐞𝐦 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭𝐥𝐲.   Understanding ℎ𝑜𝑤 inference engines work internally changed how I design production systems.   𝐓𝐡𝐞 𝐜𝐨𝐫𝐞 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞 𝐞𝐯𝐞𝐫𝐲 𝐞𝐧𝐠𝐢𝐧𝐞 𝐬𝐨𝐥𝐯𝐞𝐬:   Given limited GPU memory, serve as many concurrent requests as possible at acceptable latency.   𝐓𝐡𝐞 𝐤𝐞𝐲 𝐭𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬:   𝐏𝐚𝐠𝐞𝐝𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧 (𝐯𝐋𝐋𝐌'𝐬 𝐢𝐧𝐧𝐨𝐯𝐚𝐭𝐢𝐨𝐧) → Manages KV cache like OS virtual memory → No more pre-allocating contiguous memory blocks → Dramatically improves memory utilization → Enables higher batch sizes   𝐂𝐨𝐧𝐭𝐢𝐧𝐮𝐨𝐮𝐬 𝐁𝐚𝐭𝐜𝐡𝐢𝐧𝐠 → Don't wait for a batch to complete before starting new requests → As one request finishes decoding, slot in the next one → Keeps GPU utilization consistently high   𝐏𝐫𝐞𝐟𝐢𝐱 𝐂𝐚𝐜𝐡𝐢𝐧𝐠 → Reuse KV cache for common prompt prefixes → If 100 requests share the same system prompt, compute it once → Massive speedup for multi-turn conversations   𝐒𝐩𝐞𝐜𝐮𝐥𝐚𝐭𝐢𝐯𝐞 𝐃𝐞𝐜𝐨𝐝𝐢𝐧𝐠 → Use a small "draft" model to predict multiple tokens → Verify with the large model in parallel → Accept if correct, reject and regenerate if wrong → Can achieve 2-3x speedup without quality loss   𝐌𝐲 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐚𝐝𝐯𝐢𝐜𝐞:   Don't just pick an engine and stick with it.   Different engines excel at different workloads: → 𝐯𝐋𝐋𝐌 — Best general-purpose, excellent for high-throughput serving → 𝐒𝐆𝐋𝐚𝐧𝐠 — Great for structured generation and complex prompts → 𝐥𝐥𝐚𝐦𝐚.𝐜𝐩𝐩 — Best for edge deployment and CPU inference → 𝐓𝐞𝐧𝐬𝐨𝐫𝐑𝐓-𝐋𝐋𝐌 — Best raw performance on NVIDIA hardware   Mix and match based on your traffic patterns.

Excited to share our latest research paper, Kitsune, which tackles a fundamental challenge in GPU architecture. I’ve worked on dataflow architectures in various forms throughout my career, so it is deeply satisfying to demonstrate a method for orchestrating dataflow on one of the world's most ubiquitous silicon solutions: the GPU. The research asks a critical question: "Can modest adjustments to the current GPU architecture enable efficient dataflow execution, thereby circumventing the constraints of vertical fusion without necessitating a clean-slate architecture design?" The answer, we found, lies in a surprisingly elegant solution. The heart of our idea is an ultra-fast producer/consumer queue. By implementing this via a software-only ring queue and a modest grid scheduler adjustment, we can unlock efficient dataflow execution without abandoning the hardware and established software codebase we already have. Check out the full paper here: https://lnkd.in/gnQFvhup Great working with Michael Davies and Neal Crago on this!

nvidia now releases its most optimized inference kernels through a PhD student's open-source project. here's a breakdown of FlashInfer: FlashInfer is a GPU kernel library built specifically for LLM serving. it won Best Paper at MLSys 2025, powers both SGLang and vLLM, and NVIDIA now actively ships TensorRT-LLM kernels through it. the creator, Zihao Ye, built it during his PhD at UW and now works at NVIDIA full-time. LLM serving has a combinatorial explosion of attention kernels. every combination of KV-cache layout (paged, radix tree, tree masks), attention variant (GQA, MLA, RoPE-fused, sliding window), and batch mode (prefill, decode, append, shared prefix) needs a different kernel. FlashInfer's insight was: all KV-cache layouts are special cases of block-sparse matrices. paged attention is just block-sparse with page_size as block width. radix tree? block-sparse. tree attention for speculative decoding? block-sparse. one abstraction can replace what used to be separate kernel implementations. then you get JIT compilation to handle the variant explosion, in the form of CUDA/CUTLASS templates that get specialized at runtime there's two other major innovations built on top of FlashInfer: 1. cascade attention when multiple requests share a prefix (document QA, system prompts), FlashInfer decomposes attention into two stages: a multi-query kernel for the shared prefix (loaded once into SMEM, reused across all queries) and a batch decode kernel for unique suffixes. results merge using an associative operator on partial attention states. 31x speedup over vLLM's PagedAttention for 32K-token shared prefixes at batch size 256. 2. plan/run scheduling for CUDAGraph LLM serving has dynamic sequence lengths. CUDAGraphs need static configurations. FlashInfer solves this with a two-phase pattern: plan() inspects request shapes and computes balanced scheduling metadata, run() launches kernels. you plan once per decode step, then replay across all transformer layers. FlashInfer is an amazing project that i deeply respect, so also want to share some links for anyone that wants to go deeper: - paper (MLSys 2025 Best Paper): https://lnkd.in/gc_CTbnf - github: https://lnkd.in/gwfQ8B72 - NVIDIA blog: https://lnkd.in/gzs_uquk - cascade attention deep dive: https://lnkd.in/gHGqdNTV - docs: https://docs.flashinfer.ai

AI just delivered a computation breakthrough: Translating PyTorch to CUDA isn’t just a human problem anymore. Modern AI relies on GPU-optimized CUDA kernels, but handcrafting these requires rare expertise spanning algorithms, hardware, and memory hierarchies. This bottleneck now has a scalable solution: The AI CUDA Engineer. Sakana AI’s new framework uses Large Language Models (LLMs) to convert PyTorch operations into correct CUDA kernels and evolutionary optimization to iteratively maximize runtime efficiency. Key innovations: 1. Automatic translation (91% success rate) via error feedback loops 2. LLM-guided evolution combining model-generated variants with profiling data 2. Innovation Archive—a repository of 17K optimized kernels that seed future optimizations via RAG The results? A median 1.52x speedup over native PyTorch, with extreme gains like 54x faster diagonal matrix multiplications. Their system even translated and optimized full ResNet architectures into CUDA, achieving 1.44x speedups via fused shared-memory kernels. Why this matters: LLMs are moving beyond code generation to optimization—mastering hardware-specific constraints without human priors. With models writing code for 72% of PyTorch operations faster than torch.compile, democratizing GPU programming is no longer hypothetical. It's open for everyone: you can explore their open-sourced kernels or probe limitations 𝘳𝘪𝘨𝘩𝘵 𝘯𝘰𝘸. For industries like agriculture seeking location-specific AI—or anyone battling CUDA complexity—automating kernel engineering might just be the compute multiplier you need. Fore more on the AI CUDA Engineer and other AI highlights, check out this week's LLM Watch: https://lnkd.in/dfPZhpt6

Last week in PyTorch Conference 2025, we announced we are open sourcing torchcomms and NCCLX/CTran. Today, we share more details about NCCLX/CTran design and how we used them in production GenAI training and inference. Check out our white paper on this topic - https://lnkd.in/gySuXi6Y Some features we covered in this paper - * Host-driven collectives * Zero-copy data transfer * CTran/Network co-design (DQPLB) * Zero-copy and SM-free Send/Receive for PP * RMA Put for TP * Fault tolerant AllReduce * GPU-resident collectives for EP * Low-latency optimization * Scalable Initialization in Training * GPU Memory Management for comms * Fault localization and Performance Observability * CPU emulation This paper covers years of innovation and production experience from GPU communication teams at Meta, in supporting generations of LLAMA models training and inference. Hope you enjoy reading it!

Calling on AI to make us more productive ninjas—I'm thrilled to share a recent publication from our lab (https://lnkd.in/dWP7nf5t) introducing Xe-Forge, our LLM-powered pipeline for automating Triton kernel optimization on Intel GPUs—the first effort of its kind specifically targeting Intel GPUs. Given a functionally correct Triton kernel, Xe-Forge applies up to nine optimization stages, ranging from algorithmic restructuring and operator fusion to block pointer modernization, GPU-specific tuning, and open-ended discovery. Each stage is driven by a Chain-of-Verification-and-Refinement (CoVeR) agent that generates candidates, validates them on real hardware, and iterates on failures. The results are compelling: up to 82× speedups on individual kernels, a 1.17× geometric mean improvement across 97 KernelBench kernels, and 2–13.3× speedups on Flash Attention on the Intel Arc Pro B70. Marcin Spoczynski Alexander Heinecke Adam Siemieniuk Weilin Xu

Modal reverse-engineered Flash Attention 4. Here's what you need to know. FA4 is ~20% faster than Nvidia's own closed-source kernels. No official technical report yet. Modal read the source code and broke it down. The biggest change isn't the math. It's the architecture. FA4 runs an async pipeline with 5 specialized warp types: → Load warp: moves Q, K, V tiles from GPU RAM into shared memory → MMA warp: runs the actual matmuls on Tensor Cores → 8 Softmax warps: normalize attention scores → 4 Correction warps: rescale outputs as normalization shifts → 1-2 Epilogue warps: write final outputs back to GPU RAM Each warp type handles one stage. They run concurrently. Producer-consumer model with barrier synchronization. Modal's words: "it vaguely resembles a microservices diagram." The two math tricks: 1️⃣ Fast approximate exponentials Replace hardware SFU exponentiation with a cubic polynomial. Same bf16 precision. Avoids SFU queueing bottleneck. Based on a 1999 paper by Schraudolph. 2️⃣ Smarter online softmax Old: rescale every time a new max appears. New: only rescale when numerical stability is actually threatened. 10x fewer rescaling operations. Why this matters beyond FA4: Triton's team gave up writing Blackwell attention and built Gluon at a lower level instead. GPU programming is shifting from "write a kernel" to "architect an async pipeline across specialized hardware." 💾 Save this. The full breakdown is worth the read → https://lnkd.in/eRKBFk5c