T riton B ench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (2025)

Abstract: Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and accessibility. However, programming and parallel optimization still require considerable trial and error from Triton developers. Despite advances in large language models (LLMs) for conventional code generation, these models struggle to generate accurate, performance-optimized Triton code, as they lack awareness of its specifications and the complexities of GPU programming. More critically, there is an urgent need for systematic evaluations tailored to Triton. In this work, we introduce TritonBench, the first comprehensive benchmark for Triton operator generation. TritonBench features two evaluation channels: a curated set of 184 real-world operators from GitHub and a collection of operators aligned with PyTorch interfaces. Unlike conventional code benchmarks prioritizing functional correctness, TritonBench also profiles efficiency performance on widely deployed GPUs aligned with industry applications. Our study reveals that current state-of-the-art code LLMs struggle to generate efficient Triton operators, highlighting a significant gap in high-performance code generation.

## introduction triton @xcite language, a high-level python-like programming language designed for implementing efficient gpu kernels, is playing an increasingly pivotal role in the ever-scaling deep learning ecosystems @xcite . due to the superior portability, flexibility, lightweight design, and accessibility to less proficient programmers, triton is prevalently adopted in modern large language model (llm) frameworks such as vllm @xcite , lightllm (modeltc, 2025), liger-kernel @xcite and unsloth @xcite . however, crafting high-performance operators remains challenging, especially for the intricate balance between memory hierarchy management, parallel thread coordination, and hardwarespecific optimizations. even though triton abstracts away many complexities of low-level programming architectures like cuda, it still requires developers to manually handle critical aspects such as pointer arithmetic and memory access patterns, making performance tuning a labor-intensive process that often involves extensive trial and error. current research in ai-assisted coding has reached a human-competitive level @xcite ), yet it is primarily restricted to general-purpose languages like python. however, llms still face challenges in generating domain specific language (dsl) code. specifically for triton, current models might be unfamiliar with triton specification and the intricacies of gpu programming @xcite . most importantly, the ability of these models to produce high-quality triton code remains unassessed. therefore, a highquality benchmark paired with performance-aware metrics is urgently required. in this study, we present tritonbench, a performance-aware benchmark framework for triton generation, which contains two channels, namely tritonbench-g and tritonbench-t. specifically, tritonbench-g contains 184 carefully curated operators from existing github repositories, reflecting the realistic demand for triton operator development. as a complement, tritonbench-t is composed of operator development tasks aligned with pytorch interfaces, covering operators under-represented by public sources. moreover, unlike the majority of code benchmarks merely prioritizing functional correctness @xcite , tri-similarity @triton.jit def add_kernel @xcite ... def add_wrapper @xcite ... tonbench emphasizes efficiency performance profiling against reference programs on nvidia gpus, better aligning industrial demands. as shown in figure 1 , for tritonbench-g, we follow three steps: 1) scrape and collect high-quality operators, 2) generate instructions via prompts, and 3) annotate test code with llms. moreover, hpc experts evaluate gpu performance for all triton codes. for tritonbench-t, we provide operator generation tasks aligned with py-torch. to construct these tasks, we first perform a frequency analysis to select torch operators, combine them into diverse sets, and provide paired instructions and test code. our evaluation metrics include similarity, call and execution accuracy, speed up, and gpu efficiency. we conduct extensive experiments across a broad range of llms. overall, the difficulty of tritonbench-g is greater than that of tritonbench-t. the highest execution accuracy on tritonbench-g can reach 23.91%, while on tritonbench-t, it can reach 53.01%. for all correctly executed operators generated by the models, the best speed up on tritonbench-g is 1.56×, whereas, on tritonbench-t, it is 1.91×. additionally, we perform in-depth analyses of llms' behavior on tritonbench and summarize the challenges in triton generation. the results reveal that current llms are not yet fully capable of handling tritonbench, underscoring the challenge of enabling llms to generate triton code effectively. we hope this work initiates evaluation in this under-explored area and fosters advancements in llm-driven operator development. ## tritonbench-g triton @xcite is a dsl that abstracts away low-level complexities to simplify gpu programming for computation-intensive tasks, with flexibility for specialized applications like machine learning. typically, a triton operator includes at least a kernel and a wrapper. the kernel comprises code executed on the gpu, focusing on tensor element addressing and thread parallel coordination. meanwhile, the wrapper offers a python function that encapsulates the kernel call. figure 2 shows an example of triton operator. we create tritonbench-g by curating highquality human-authored triton operators from github, which reflects triton's currently actual requirements. the following sections will explain data collection ( § 3.1), data statistics ( § 3.2), operator quality rating ( § 3.3), test code design ( § 3.4), and evaluation metrics ( § 3.5). ## tritonbench-t the real-world triton operators introduced in § 3 primarily focus on highly frequent operations. as a complement, we propose tritonbench-t, which aligns the triton wrapper with interfaces of the pytorch library @xcite . together, tritonbench-g and tritonbench-t form a complementary evaluation framework. the following sections elaborate on the data construction ( § 4.1), data statistics ( § 4.2), test code and metrics ( § 4.3), and benchmark comparisons ( § 4.4). ## experiments we conduct an extensive set of experiments on tri-tonbench to rigorously evaluate the performance and capabilities of current llms. ## analysis in this section, we examine the distribution of correct and incorrect operators across difficulty levels (d1-d5) for the top-performing models, deepseek-r1 and gpt-o1, as shown in figure 4 and figure 5 . additionally, we analyze the error patterns of incorrect operators and summarize the main challenges for each benchmark as detailed in ## conclusion in this work, we present tritonbench, a dualchannel benchmark specifically designed for evaluating llms' generation for triton operators. tritonbench-g integrates real-world triton operator samples from open repositories, while tritonbench-t introduces complementary tasks that align with pytorch interfaces. our evaluation framework addresses both functional accuracy and the performance on nvidia gpus. we also conduct extensive experiments and detailed analysis on our benchmark, and find that current llms struggle to generate high-quality triton operators, un-derscoring the necessity for further advancement in generating accurate as well as performance-aware triton code. we anticipate tritonbench will serve as an essential framework for advancing automated operator generation for triton.