Math Natural Language Inference: this should be easy! (2025)

Abstract: We ask whether contemporary LLMs are able to perform natural language inference (NLI) tasks on mathematical texts. We call this the Math NLI problem. We construct a corpus of Math NLI pairs whose premises are from extant mathematical text and whose hypotheses and gold labels were provided by people with experience in both research-level mathematics and also in the NLI field. We also investigate the quality of corpora using the same premises but whose hypotheses are provided by LLMs themselves. We not only investigate the performance but also the inter-group consistency of the diverse group of LLMs. We have both positive and negative findings. Among our positive findings: in some settings, using a majority vote of LLMs is approximately equivalent to using human-labeled data in the Math NLI area. On the negative side: LLMs still struggle with mathematical language. They occasionally fail at even basic inferences. Current models are not as prone to hypothesis-only “inference” in our data the way the previous generation had been. In addition to our findings, we also provide our corpora as data to support future work on Math NLI.

## introduction we study natural language inference (nli) tasks in the area of research-level mathematics. one might think that llms would do extremely well on this task. after all, what counts as an entailment or contradiction in everyday-language texts is often taken as a complicated version of what happens with mathematics. so we might expect purported mathematical inferences to be easier to evaluate than those in everyday language. and unlike language in the wild, the domain of mathematics is fairly well-defined. facts, definitions, and logical reasoning play a large role in mathematical writing. sentences ought to be precise and unambiguous. however, there are complications with mathematical text from the start. the vocabulary may be unfamiliar to a generic audience: mathematical parlance can use daily words with new, unfamiliar meanings, e.g. 'ring', 'field', or even 'folklore'. on top of this, the use of visual elements such as symbols, equations, and diagrams, almost changes the very language of the text from plain text to a richer, multimodal language. the field lacks open-source resources such as dictionaries and glossaries for mathematical concepts. it is much harder to find a "person on the street" annotator of mathematics than of more common forms of text. when confronted with the incredible solutions to mathematical-like problems that deep learning systems can offer nowadays (e.g., alphageometry @xcite ), it is difficult to believe that these systems cannot understand the basics of causality or of propositional reasoning used throughout mathematics. nonetheless, when tested on these basics, the llm-based systems still make very surprising (to humans) mistakes. further, the fact that llms do not have a notion of selfconsistency has been documented in many recent papers @xcite @xcite . but mathematics, as usually practiced, needs self-consistency. in a sense, it seems that sometimes the deep learning systems deserve an a+ in advanced problem solving but a b in the basics. for all of these reasons, we could conclude, perhaps surprisingly, that the nli task is not much easier when using llms to deal with mathematical text after all. in this paper, we shall see how precisely correct math nli using llms can be. we decided to experiment and build a corpus of nli inference pairs, comparing the output of several llms on mathematical text. ## evaluating llms on the seed corpus in our first experiment, we harness llms to evaluate the seed corpus. ## evaluating llms on gpt-generated mathnli corpus next, we had the 4 models in group 1 and 5 models in group 2 label the 89 pairs. the results are shown in the evaluation results on the gpt-generated corpus using the gpt-generated label as the true label are shown in our analysis reveals that while the e and c pairs generated by gpt show a certain level of consistency relative to our seed pairs, n pairs are frequently misclassified as e. (this finding echoes what we saw in our previous experiment, but there the pairs were human-generated.) surprisingly, llama 2 classifies 75.9% of n pairs as e. among all evaluated models, mixtral showed the least susceptibility to this issue, maintaining the highest accuracy of 76.0%. although its performance on the seed corpus was not outstanding, mixtral achieved the highest overall accuracy of 86.5% on the gptgenerated corpus. furthermore, it was observed that llms tend to identify c pairs within the gpt-generated corpus more accurately than they do within the seed corpus; 8 out of 9 models achieved an f1 score of over 90%. notably, gemma2 successfully detected all the c samples in the gpt-generated corpus. ## sanity checks for previous nli datasets such as snli @xcite and mnli @xcite , studies have shown that some of the systems which detect "entailment" were working on artifacts of the datasets, instead of doing any logical reasoning (e.g., @xcite . in this section we provide some sanity checks on our corpus. ## final remarks we find it useful to discuss our work by seeing how it aligns with the perceptive conclusions drawn by @xcite . 3 we agree that evaluating models on nli tasks is still relevant. for math nli, we do not find models to be saturated. this contrasts with ordinary language nli (onli). we also confirm their finding that "while the similarity of model distributions with human label distributions increases with scale, it is still much higher