QC oder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback (2025)

Abstract: Large language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a language generation task that bridges natural language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as quantum programming, where human coders write Python code that is executed on a quantum computer. To address this gap, we introduce QCoder Benchmark, an evaluation framework that assesses LLMs on quantum programming with feedback from simulated hardware devices. Our benchmark offers two key features. First, it supports evaluation using a quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark. In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder Benchmark dataset and public evaluation API to support further research.

## introduction programming code generation has emerged as an important and practical problem in language generation studies @xcite . this task requires models to generate correct and executable code by bridging natural language, human expertise, and formal programming logic. recent advances in large language models (llms) have led to impressive performance on classical programming benchmarks @xcite . however, these benchmarks are primarily evaluated in software-only environments, where failures are typically limited to runtime errors or syntax violations detected by a software development environment such as python interpreters. in contrast, little is known about how llms perform in domains such as quantum programming @xcite , where generated code must not only be syntactically correct, but also conform to strict, domain-specific constraints imposed by real or simulated quantum hardware. quantum programming serves as a representative example of hardware constraint-driven code generation tasks. as shown in figure 1 , given the instruction as a natural language, this code generation task involves generating a python code to produce a quantum circuit that can be run on a separate hardware, i.e., either a real quantum computer or simulator. unlike classical programs, correctness and executability depend not only on the absence of runtime errors in python, but also on whether the resulting circuits comply with constraints imposed by quantum hardware. these hardware-level constraints include, for example, limitations on circuit depth (i.e., the number of sequential gate operations) and the availability of only certain types of quantum gates in a quantum computer. as a result, quantum code must be syntactically valid in python and must generate quantum circuits that conform to the logical requirements of quantum computation. to enhance studies on this constrained generation task, we introduce qcoder benchmark, a dataset and evaluation framework specifically designed for quantum code generation. our benchmark contains 1) pairs of a programming contest problem and human-written solutions and 2) an evaluation tool to provide hardware-specific feedback. unlike prior benchmarks that rely on generic python execution, our evaluation tool uses a quantum simulator that returns quantum-specific feedback about domain-specific constrains, e.g., circuit depth and inappropriate uses of unsupported quantum gates. this evaluation tool allows feedbackdriven iterative language generation: a generation paradigm in which models incorporate feedback from a hardware to refine their generated codes @xcite . this paper uses our benchmark to investigate whether llms can improve their quantum code generation performance by incorporating domainspecific feedback. our experiments show that even advanced llms like gpt-4o achieve only around 18.97% accuracy, while the best performing llm o3 reaches 65.52% and it outperforms averaged success rate of human-written codes submitted to programming contests (39.98%). we also find that incorporating feedback into prompt to refine codes can significantly improve generation performance, emphasizing the importance of feedback from a simulated hardware. we release qcoder benchmark as public resources to support further research on code generation under complex constraints. this paper makes the following contributions: 1) we implement iterative code generators that use feedback from a simulated hardware-based evaluation tool, 2) we empirically demonstrate that such feedback effectively enhances llms' performances, and 3) our benchmark data and evaluation api will be made public. ## related work various coding benchmarks have been proposed for general-purpose code generation tasks, such as humaneval @xcite , mostly basic python problems (mbpp) @xcite , and the apps dataset @xcite . these benchmarks primarily focus on solving basic algorithmic problems written in python and are evaluated using predefined input-output test cases. our benchmark differs from these benchmarks in two key aspects. first, it targets a domain-specific coding task-quantum programming-which involves generating circuits that must conform to real-world hardware constraints. second, rather than relying solely on static test cases, our benchmark evaluates generated code using a simulated quantum computer, offering a new paradigm for evaluating executable and hardware-aware code. domain-specific coding benchmarks have been introduced for various domains, including data science (ds-1000 @xcite ), secure coding (llmseceval @xcite ), database query generation (spider @xcite ), and bioinformatics (biocoder @xcite ). in the quantum programming domain, qiskit hu-maneval @xcite shares similar motivations with ours. however, our benchmark differs in two important ways: (1) it provides a hardware simulator-based evaluation framework for assessing generated quantum circuits, and (2) each programming task is accompanied by multiple human-written implementations, enabling comparative analysis between human and llm-generated code. quantum programming is a form of code generation aimed at controlling hardware, but only a few attempts exist in this direction e.g., a study that generates code to control robots @xcite . refinement of generated code has also been shown to be effective in various setups @xcite @xcite . our study extends this line of work by incorporating hardware-aware feedback. ## qcoder benchmark our benchmark consists of pairs of programming contest problems and human-written solutions, together with an evaluation tool that provides quantum hardware-aware feedback 2 . our benchmark differs from existing general-purpose or quantum benchmarks @xcite in two aspects: (1) it enables fine-grained evaluation from domain-specific perspectives (e.g., circuit depth), not just functional correctness, and (2) it includes human-written solutions collected from programming contest submissions. these submissions often contain errors and variations useful for studying the differences between llm-generated and human-written code. ## methods this paper compares prompt-based code generators rather than finetuning-based models because developing prompt-based techniques is particularly important for domains like quantum programming, where users are often not experts in natural language processing. the following subsections describe our prompt and refinement process using hardware-aware feedback expressed in natural language. ## main results in this section, we present the results of our experiments. we begin by comparing overall success rates, followed by a fine-grained failure analysis to understand the types of errors commonly produced. we then examine the effects of iterative refinement using simulator-based feedback and compare model performance against human-written code. ## case study: actual outputs this section presents a specific example from the qcoder benchmark to illustrate how feedback can effectively improve code generation. the example problem is shown in figure 7 . we show the reference program for this problem in figure 8 . in this problem, it is not necessary to consider the amplitudes of the ground states. instead, it suffices to view the ground state as a problem: implement the operation of preparing the state |ψ⟩ from the zero state on a quantum circuit qc with 2 qubits. the state |ψ⟩ is defined as next, a controlled-hadamard gate is applied, targeting the second qubit and controlled by the first. this results in through this sequence of operations, a superposition of the three computational basis states |00⟩, def solve() -> quantumcircuit: @xmath0.sqrt(np.abs(a0)**2 + np.abs(a1)**2 + np.abs(a2)**2) @xmath1.array([a0, a2, a1, 0], @xmath2.initialize(state, [0, 1]) return qc |10⟩, and |01⟩ is successfully prepared. the available code is shown in fig. 8 . fig. 9 shows the code generated at the first iteration. in this code, the .initialize() method is used, however, since this method is not included in the allowed set of quantum gates. therefore, the gate constraint violation error is raised. this violation report is then included in the prompt for refinement as shown in section 4, the second code generated by the model is shown in fig. 10 . this code passes the runtime error check, however, the evaluator produces the wrong output error although the runtime or other constraints' checks are passed. the generated code at third iteration shown in fig. 11 can successfully generate the desired state. note that the model employs a different approach from the example solution shown in fig. 8 . through iterative refinement, the model gradually modifies the code to produce one that satisfies the input constraints. finally, we also observed that many failure cases involved python's runtime errors that are relatively easy for humans to fix, such as missing import statements (e.g., import math). they still struggle with basic coding hygiene that would typically be trivial for humans. ## conclusions we introduced qcoder benchmark, which consists of a dataset and evaluation tool for quantum code generation. this benchmark is designed to investigate the capabilities of llms under domainspecific hardware-aware constraints. by integrating a quantum simulator that returns hardware-aware feedback, we implemented a feedback-driven it-def solve() -> quantumcircuit: @xmath3.sqrt(a0**2+a1**2+a2**2) @xmath4.sqrt(a0_norm**2 + a2_norm**2) @xmath5.atan(a1_norm / denominator) qc.ry(theta, 0) @xmath6.atan(a2_norm/a0_norm) @xmath7.control(1) qc.append(c_ry, [0, 1]) return qc erative code generator. our experiments revealed that even advanced models like gpt-4o struggle with quantum programming tasks, while reasoningoriented models like o3 show superior performance and can even outperform human-written code submissions for programming contests. these findings suggest the importance of refinement of codes by domain-specific feedback. while our benchmark and experiments focus on quantum programming, the proposed feedback-driven generation framework-where domain-specific constraint violations are systematically detected and incorporated into iterative code refinement-could generalize to other domains that impose strict execution constraints. potential applications include robotics and embedded system programming, where functional correctness alone is insufficient and compliance with real-world constraints (e.g., timing, resource usage, device compatibility) must be enforced. we leave the exploration of such domains to future work.