H e S um: a Novel Dataset for Abstractive Text Summarization in H ebrew (2024)

Abstract: While large language models (LLMs) excel in various natural language tasks in English, their performance in low-resource languages like Hebrew, especially for generative tasks such as abstractive summarization, remains unclear. The high morphological richness in Hebrew adds further challenges due to the ambiguity in sentence comprehension and the complexities in meaning construction. In this paper, we address this evaluation and resource gap by introducing HeSum, a novel benchmark dataset specifically designed for Hebrew abstractive text summarization. HeSum consists of 10,000 article-summary pairs sourced from Hebrew news websites written by professionals. Linguistic analysis confirms HeSum’s high abstractness and unique morphological challenges. We show that HeSum presents distinct difficulties even for state-of-the-art LLMs, establishing it as a valuable testbed for advancing generative language technology in Hebrew, and MRLs generative challenges in general.

## introduction recent advances with large language models (llms, @xcite demonstrate impressive capabilities, encompassing diverse tasks such as natural language (nl) understanding and reasoning, including classification tasks such as commonsense reasoning @xcite and sentiment analysis @xcite , as well as generative tasks like summarization and dialogue systems @xcite . however, these impressive achievements are primarily demonstrated for the english language. our understanding of how these models perform on low-resource languages is limited, as current evaluations are primarily focused on languages with abundant data @xcite . this concern is particularly relevant for morphologically rich languages (mrls) such as hebrew, which is known for their word complexity and ambiguity, leading to processing difficulty @xcite . despite advances in natural language processing for hebrew, which so far covered tasks as reading comprehension @xcite , named entity recognition @xcite , sentiment analysis @xcite , and text-based geolocation @xcite ; a crucial gap persists in the ability to evaluate novel, human-like generated text, as in abstractive text generation. abstractive text-generation requires both natural language understanding and reasoning over the input, and the ability to generate grammatically, and in particular morpho-syntactically, correct text, as well as semantically and morpho-semantically coherent, fluent text that conveys consistent meanings. notably, text-generation models are also prone to 'hallucinations' -generating factually incorrect content. these challenges are further amplified in hebrew due to its morphological richness which leads to a complex realization of sentence structure and meaning. in order to enable empirically quantified assessment of these aspects of text generation in mrls, we present a novel benchmark dataset for hebrew abstractive text summarization (hesum). hesum consists of 10,000 articles paired with their corresponding summaries, all of which have been sourced from three different hebrew news websites, all written by professional journalists. this curated collection offers several key advantages: (i) high abstractness -extensive linguistic analysis validates hesum's summaries as demonstrably more abstractive even when compared to english benchmarks. (ii) unique hebrew challenges -meticulous linguistic analysis pinpoints the inherent complexities specific to hebrew summarization, offering valuable insights into the nuanced characteristics that differentiate it from its english counterpart. and (iii) thorough llm evaluation -we conducted a comprehensive empirical analysis using state-of-the-art llms, demonstrating that hesum presents unique challenges even for these contemporary models. by combining high abstractness, nuanced morphological complexities, and a rigorous llm evaluation, hesum establishes itself as a valuable resource for advancing the frontiers of abstractive text summarization in mrl settings. ## the challenge linguistic challenges in hebrew morphologically rich languages (mrls) pose distinct challenges for generative tasks, above and beyond morphologically impoverished ones such as english. in mrls, each input token can be composed of multiple lexical and functional elements, each contributing to the overall structure and semantic meanings of the generated text. for instance, the hebrew word ' ‫'וכשמביתנו‬ is composed of seven morphemes: ‫'ו'‬ ('and'), ' ‫'כש‬ ('when'), ‫מ'‬ ' ('from'), ' ‫'ה‬ ('the'), ' ‫'בית‬ ('house'), ' ‫'של‬ ('of'), and ' ‫'אנחנו‬ ('us'). this has ramifications for both the understanding of mrl texts, a process that necessitates morphological segmentation, and for generating mrl texts, requiring morphological fusion. at comprehension, hebrew poses an additional challenge due to its inherent ambiguity, with many tokens admitting multiple valid segmentations, e.g., ‫'הקפה'‬ could be interpreted as ‫'+'ה'‬ ‫'קפה‬ ('the'+'coffee'); as ‫'הקפה'‬ ('orbit'); or as ' ‫'הק‬ + ' ‫'של‬ + ' ‫'היא‬ ('perimeter'+'of'+'her'). during generation, the emergence of unseen morphological compositions, where unfamiliar morphemes combine in familiar ways, poses an additional challenge @xcite . these challenges, coupled with inherent linguistic features like morphological inflections, construct-state nouns (smixut), and flexible word order, create a multifaceted challenge for llms in processing and generating hebrew texts. the hesum task we aim to unlock the comprehension-and-generation challenge in mrl settings by first tackling the abstractive text summarization task @xcite , here focusing on modern hebrew. given an input document in hebrew, specifically a news article, our goal is to generate a short, clear, hebrew summary of the key information in the article. in contrast to extractive summarization, here novel morphosyntactic structures need to be generated to communicate the summary. ## conclusion this research seeks to fill a critical gap in the field of llms assessment for generative creative tasks in mrls, by presenting hesum, a new dataset for hebrew abstractive summarization, that includes 10k article-summary pairs sourced from professional journalists on hebrew news websites. hesum offers three key advantages: high level of abstractness in summarization, distinct challenges specific to the hebrew language, and a thorough empirical assessment of llms using this dataset. by integrating these aspects, hesum establishes itself as a valuable resource for researchers striving to push the boundaries of generative tasks, and specifically abstractive text summarization in hebrew.