LCHAIM - Investigating Long Context Reasoning in H ebrew (2025)

Abstract: Natural Language Inference (NLI) has gained significant attention recently due to its importance in understanding how machines comprehend and reason about language. While English has received tremendous interest, Morphologically Rich Languages (MRLs) like Hebrew, require more research. In this paper, we address the evaluation of Hebrew NLI models by introducing LCHAIM, a dataset designed to evaluate these models on tasks involving long premises and complex reasoning. The dataset, created by translating and validating the English ConTRoL dataset, consists of 8,325 context-hypothesis pairs that require coreferential, temporal, logical and analytical reasoning. Our experiments show the difficulty of contextual reasoning in Hebrew, as evidenced by the performance of different models. Fine-tuning the LongHero model on both the shorter premise Hebrew NLI and the LCHAIM datasets yielded a mean accuracy of 52%, that is 35% less than human performance. Similarly, Large language Models (LLMs) like Gemma-9B, Dicta-LM-2.0-7B, and GPT-4o achieved a top mean accuracy of 60.12% in few-shot setting.

## introduction nli, also called textual entailment (te) is a key task in natural language processing @xcite . it involves determining whether a given hypothesis can be logically inferred from a premise. this task has shown to be very helpful in various applications, including text classification, event extraction, and summarization evaluation, @xcite @xcite . nli has gained significant attention recently due to its importance in understanding how machines comprehend and reason about language. this is largely because almost any task can be generalized to determining entailment or contradiction in context between texts @xcite . although many datasets have been developed to train and evaluate nli models @xcite @xcite @xcite , these datasets primarily focus on sentence-level examples, which do not fully capture the complexity of real-world language understanding. to address this issue, @xcite proposed control, a long context nli dataset that enabled investigating contextual reasoning for nli. 2 the example in figure 1 demonstrates mathematical reasoning, which requires an understanding of calculus to solve it. however, despite large advances in nlp, research and datasets predominantly focus on english, leaving low-resource languages like hebrew underrepresented. furthermore, the rich morphological structure of hebrew presents significant challenges for natural language understanding (nlu) tasks, leading for example to inconsistencies and errors in evaluation metrics @xcite . recent studies have increased interest in hebrew @xcite , but the emphasis remains largely on morpho-syntactic tasks. despite the existence of a few nli datasets for morphologically rich languages @xcite @xcite , to the best of our knowledge, no existing dataset specifically addresses the challenges associated with long premises that require complex reasoning. furthermore, none of the aforementioned datasets is in hebrew. to address these gaps, we present in this paper a first investigation of long context reasoning in hebrew. as part of this investigation, we compile lchaim -long context hebrew with advanced reasoning inference model benchmark -a hebrew version of control @xcite . lchaim contains 8,325 premise-hypothesis pairs in hebrew, labeled for contradiction, entailment, or neutral relationship between premise p and hypothesis h, and is obtained from the control dataset via automatic translation and human validation. then, we provide an evaluation of hebrew language models for the task of nli, using the resulting dataset. we show that the task of nli with contextual reasoning is challenging even for larger language models, which shows the need for the construction of complex nlu benchmarks in hebrew. the research contribution is three-fold: ## lchaim dataset the lchaim dataset is a translated version of control @xcite . it is compiled from publicly available online verbal reasoning tests, including those used in police initial recruitment exams, medical college admissions, and university clinical aptitude tests, as well as corporate verbal aptitude assessments. these tests are structurally similar to nli tasks, presenting a premise and a hypothesis with three answer choices: true, false, or cannot say. these choices correspond to the nli labels entailment, contradiction, and neutral. the data format of lchaim aligns with existing nli benchmarks @xcite . a notable difference from existing datasets is that lchaim features longer premises, often spanning one or more passages. additionally, each premise is paired with three or more hypotheses. the verbal reasoning tests require candidates to comprehend meaning, evaluate logical strength, make valid inferences, and identify appropriate conclusions. the passages cover various topics, including current affairs, business, science, the environment, economics, history, meteorology, health, and education. these questions are of high quality and are used in rigorous assessments, reflecting a high level of difficulty. after removing duplicates, the control and lchaim datasets consisted of 8,325 context-hypothesis pairs. lexical overlap analysis, calculated using jaccard similarity between premises and hypotheses in each class, shows only 4.87% overlap for entail-ment pairs and 5.49% for contradiction pairs, indicating that the dataset presents significant challenges for simple lexical matching techniques @xcite . lchaim is created as an analog to the original control dataset, and with the hypothesis that translation from english to hebrew does not affect the reasoning types present in the dataset at large. as such lchaim contains various reasoning tasks such as coreferential reasoning @xcite , verbal logical reasoning @xcite , temporal and mathematical reasoning @xcite , information integration over paragraphs @xcite , and analytical reasoning @xcite . ## results fine-tuning only using lchaim, without the hebnli step, resulted in a mean accuracy of 38.4% and an f1 score of 32.4, which indicates an absolute difference of 1%, and 3 respectively. continuous fine-tuning on both hebnli and lchaim resulted in a mean accuracy of 42.6% and an f1 score of 41.3, or an overall 3%, and 5 absolute increase respectively. this means that for alephbert the best setting is fine-tuning on both the short premise hebnli and the long premise lchaim datasets. these settings also showed better calibration than the original pre-trained version, with relatively balanced class prediction. longhero, which has a long enough context to be able to accommodate the long premises, was able to achieve a mean accuracy and an f1-score of 34.5% and 33.7 respectively without fine-tuning. the model displayed some imbalance towards the entailment class. fine-tuning longhero using the shorter premise hebnli resulted in an accuracy of 32% and an f1 score of 25, or an absolute difference of 2% and 8 compared to the non-fine-tuned version. fine-tuning only using the lchaim training set resulted in an accuracy of 41% and an f1 score of 34, or an absolute difference of 7% and 1 respectively. furthermore, fine-tuning longhero on hebnli and then on lchaim resulted in the best overall performance of 52% mean accuracy and f1 score, an absolute performance difference of 18% and 19 respectively. this represents a substantial relative improvement of 35% in both mean accuracy and f1 score, highlighting a significant boost in performance. this model also showed the overall best calibration between the 3 predicted classes. the best model for the task was a finetuned longhero using both hebnli and lchaim. human performance is displayed in results for the shorter premise hebnli test set are presented in fine-tuning using first the lchaim and then hebnli datasets resulted in a mean accuracy and an f1 score of 78.28% and 78.24% for aleph-bert, offering similar results to fine-tuning only with hebnli . the same approach resulted in a mean accuracy and an f1 score of 83.93% and 83.78% respectively for the longhero model, a 5.5% increase in performance compared to finetuning only with hebnli. the best model for the hebnli test set (longhero-all), achieved a 31.76% absolute higher mean accuracy compared to the best model for the lchaim test set (also longhero). both models showed good calibration over both test sets. for comparison, sota performance for hebnli is reported in the hebrew llm leaderboard, with a mean accuracy of 95.48% for qwen2.5-72b (qwen et al., 2025). 5 5 error analysis ## discussion the results of our evaluation of the lchaim dataset shed light on the challenges and performance of models dealing with long-premise nli tasks in hebrew. to the best of our knowledge, lchaim is the first long premise, complex reasoning, nli dataset available in hebrew. model performance the alephbert model struggled with long-premise nli tasks. it achieved a mean accuracy of 39.50% and an f1-score of 35.94, and tended to over-predict entailment. finetuning alephbert on hebnli dataset or the lchaim train set individually showed no improvement in performance. however, fine-tuning with hebnli followed by lchaim showed performance gains. this resulted in a mean accuracy and f1-score of 42.60% and 41.32 respectively. this suggests the shorter, more simple task of hebnli does not contribute towards the harder, longer-premise lchaim test set performance on its own, but can provide a foundation for further fine-tuning. therefore, including the long-premise data from lchaim is essential for better performance in the longer premise tasks. with that said, performance was still considerably lower then the 84.97% mean accuracy human performance. longhero, which is designed to handle longer contexts, achieved a mean accuracy and an f1-score of 34.53% and 33.74 respectively. this model had also calibration issues similar to ale-phbert. it is worth noting the english equivalent models (bert-base, and longformer) did not experience these calibration issues when considering the english version of this dataset @xcite . the best results were seen with the fine-tuned longhero using both hebnli and then lchaim, with a mean accuracy and f1 score of 52.17% and 52.20 respectively. for comparison, performance over the shorter hebnli dataset were significantly higher than the performance over the lchaim test set considering the sequentially finetuned alephbert and longhero. these models showed a 25-30% higher mean accuracy and f1 score on the hebnli test set. with caution, we hypothesize this is an indicator of the complexity of the reasoning in the lchaim dataset. gpt-4o performed best with a mean accuracy of 60.12% in 2-shot learning, but still did not significantly surpass the smaller gemma-9b. both gemma-9b and dicta-lm-2.0-7b showed no improvement with few-shot learning compared to zero-shot learning. complex reasoning and morphological richness we hypothesize that the lchaim dataset is a long reach to hebrew models because, unlike previous nli benchmarks, it contains tasks that require complex reasoning (for example temporal reasoning shown in figure 3 ). furthermore, we show that morphological richness of samples in the test set is inversely correlated with performance. this finding aligns with our hypothesis, and previous research claiming morphologically rich lan-guages like hebrew are more difficult for language models than non morphologically rich languages like english. this conception of their mission is fairly recent, dating back to the 18th and early 19th century development of scientific history, and has been largely fostered by professional historians who have adopted the assumption that the study of natural and inevitable human activity. before the end of the 18th century, history was not taught in almost any schools, and it did not attempt to provide an interpretation of human life as a whole. this is more suited to the role of religion, of philosophy, or perhaps even of poetry. ## conclusion this study introduced the lchaim dataset, a benchmark for long-premise nli task in hebrew, that focuses on complex reasoning types like coref-erential, temporal, and analytical reasoning. our experiments show that current state-of-the-art models perform far below human levels. the lchaim dataset provides a tough challenge for future research, underscoring the need for better models and methods. by making lchaim and our code available, we hope to encourage further exploration and advancements in hebrew nlu research.