T e X pert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLM s (2025)

Abstract: LaTeX’s precision and flexibility in typesetting have made it the gold standard for the preparation of scientific documentation. Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material using LaTeX with natural language instructions, yet current benchmarks completely lack evaluation of this ability. By introducing TeXpert, our benchmark dataset with natural language prompts for generating LaTeX code focused on components of scientific documents across multiple difficulty levels, we conduct an in-depth analysis of LLM performance in this regard and identify frequent error types. Our evaluation across open and closed-source LLMs highlights multiple key findings: LLMs excelling on standard benchmarks perform poorly in LaTeX generation with a significant accuracy drop-off as the complexity of tasks increases; open-source models like DeepSeek v3 and DeepSeek Coder strongly rival closed-source counterparts in LaTeX tasks; and formatting and package errors are unexpectedly prevalent, suggesting a lack of diverse LaTeX examples in the training datasets of most LLMs. Our dataset, code, and model evaluations are available on GitHub at https://github.com/knowledge-verse-ai/TeXpert.

## introduction latex is a highly versatile and widely adopted document preparation system built over the tex typesetting program @xcite . with research-specific advantages including robust handling of mathematical equations, simple formatting commands, and straightforward management of references, it is a popular choice to produce publication-ready scientific material @xcite . the recent emergence of llms across various applications @xcite @xcite coupled with improved instruction-following ability @xcite prompts an essential research question: "can llms generate publication-ready latex code for components of scientific documents from natural language instructions?". through this research, we aim to evaluate the capability of llms in generating syntactically and logically accurate latex code (which we refer to as accurate latex code generation or simply latex generation) and analyse the main types of errors they encounter. while certain aspects of latex code generation with llms, especially for mathematical content @xcite , have been significantly studied, a comprehensive study of llms' latex generation ability for various components commonly used in scientific documents (such as tables, figures, bibliography, etc.) remains unexplored. we believe a comprehensive benchmark for evaluating llms on latex generation offers two key benefits: analysing common errors llms make in generating latex code can provide format and error-based hints for flagging ai-generated research material @xcite , and delineating the complexity of latex tasks that llms can reliably perform can greatly reduce researchers' effort on formatting and typesetting. in this work, we evaluate a diverse range of closed-source and open-source llms on their la-tex generation capabilities. the main contributions of this paper can be stated as follows: existing works on the evaluation of llms treat latex-based tasks only as a peripheral component or limit their scope to specific output formats. the ability of llms to generate mathematical latex equations from various sources has been explored in datasets like math @xcite , mathbridge @xcite and stem-pom @xcite . similarly, the struc-bench dataset @xcite ) contains natural language inputs to test llms' latex generation ability specific only to tabular content. the im2latex-100k dataset @xcite also focuses on the narrow aspect of testing the ability of llms to convert images of mathematical formulae into latex code, while image2struct (roberts et al., 2024) includes testing vision-language models in extracting structured latex information from images. a straightforward idea to evaluate the natural language to latex ability of llms would be to generate free-to-use latex templates 2 representing various document styles and formats using textual queries. however, these templates are often too large to be directly generated by large language models (llms) and are constrained only to a standard set of basic commands, limiting their applicability in this research. several instructionfollowing benchmarks for llms evaluate their ability to follow natural language commands @xcite ; however, there is a notable absence of datasets specifically designed to assess models in latex code generation for scientific material. identifying and acting upon this need, we present texpert, an organised dataset designed to evaluate llms' capability to generate syntactically and logically correct latex code from textual descriptions, focused on scientific document components. ## dataset construction to assess llms' capability to convert unstructured textual descriptions to latex code, we build a benchmark dataset by following the process described in figure 1 . the process involves two major steps: collecting atomic latex commands: we begin by systematically analyzing a range of data sources and scientific document templates to collect atomic latex commands (details of sources 2 https://www.overleaf.com/latex/templates and methodology are provided in appendix a.1). these atomic commands, representing the minimal functional units commonly used in scientific writing and typically consisting of a backslash followed by a keyword and optional arguments, were extracted to form the basis of our dataset. the commands were then classified into 5 categories based on their purpose, as shown in ## experimental setup we utilise a systematic evaluation framework to assess llms' ability to generate syntactically correct latex code from natural language prompts using the texpert dataset. we experiment with a for each sample across the three difficulty levels in texpert, we provide the llm with a prompt containing task instructions for latex code generation (provided in figure 5 in appendix b). during generation, model parameters were set to predetermined values to ensure deterministic outputs, as detailed in we then evaluate each llm's response with gpt-4o as a judge, using a predefined evaluation prompt (refer to figure 4 in appendix b) to compute success rates and classify error types (described in ## result discussion the accuracy of latex generation for scientific documents across difficulty classes is presented in ## error analysis in this section, we provide a brief analysis of the most common error types and probable sources during latex generation by llms. from our perspective, most powerful llms still struggle to provide error-free code due to basic oversights like missing packages and unfaithful instruction following. it is encouraging to see minimal capability errors and syntax errors. we leave an in-depth analysis of the root cause of errors to the future scope. logical errors dominate: logical errors consistently account for the majority of issues across llms, highlighting struggles to fully satisfy task requirements. in all the cases we analysed, the most pronounced errors across all model variants were focused on missed instructions and wrong structural placement, especially in gpt-4o-mini and all open-source models. similarly, error clustering in multiple equation and table generation tasks indicates that llms like deepseek v3 and mistral large struggle with maintaining long-range consistency. we believe these errors likely arise from weak structural understanding inherent in llms, limited exposure to latex context, and misalignment between pretraining tasks and formal document generation. frequent formatting lapses: notably, formatting errors occur far more frequently than we anticipated in all the llms we experimented with. analysis of the evaluations reveals that these errors primarily involve incorrect environment selection and malformed tables or captions accompanying large tables or figures. such issues indicate limited structural understanding and inadequate grounding in latex syntax, even in larger models like deepseek v3 and gpt-4o, showing that scale alone is not the solution. we speculate that these errors stem from a scarcity of training data and examples specifically addressing table formatting and related constructs. package errors are concerning: package errors are prominently caused by improper or incomplete inclusion and configuration of essential latex packages, especially bibliography-related ones, most prominent in claude 3.5 sonnet. gpt-4o has the lowest share of missing packages, showing encouraging signs that more inclusive training data might mitigate this issue, although codestral's minimal package error rate also suggests potential for alternative approaches to reduce them further. additionally, the use of non-standard or incompatible packages, especially in deepseek and mistral models, is concerning and may point to llms hallucinating or making up packages to fill reasoning gaps. overall, package issues suggest a fundamental gap in dependency management and environment consistency within latex code generated by llms. ## conclusion we curate texpert, a comprehensive benchmark designed to challenge llms to evaluate their la-tex code generation capability from natural language prompts. our dataset consists of a total of 440 high-quality samples, organised by difficulty. our findings reveal that latex generation is still an underperforming skill in llms and that there is a need to include latex package details and complex layouts in the training data for llms to improve their capability in this task. by making the code and dataset for texpert publicly available, we hope to support and encourage further research within the community.