Irony Detection in H ebrew Documents: A Novel Dataset and an Evaluation of Neural Classification Methods (2025)

Abstract: This paper focuses on the use of single words in quotation marks in Hebrew, which may or may not be an indication of irony. Because no annotated dataset yet exists for such cases, we annotate a new dataset consisting of over 4000 cases of words within quotation marks from Hebrew newspapers. On the basis of this dataset, we train and evaluate a series of seven BERT-based classifiers for irony detection, identifying the features and configurations that most effectively contribute the irony detection task. We release this novel dataset to the NLP community to promote future research and benchmarking regarding irony detection in Hebrew.

## introduction irony understanding involves a complex interpretation process. although irony is inherently indirect, its interpretation may be enhanced by textual markers. this paper focuses on the use of one of the most prevalent irony markers -quotation marks enclosing single words. the analysis combines a theory-based, pragmatically oriented textual analysis of the pattern under study with experiments aiming to train a neural network to automatically identify ironic quotation marks and differentiate them from similar non-ironic quotes, used for naming and marking peculiar lexical choices. whereas ironic quotation marks received some theoretical and experimental attention in pragmatics and in computational linguistics, we are not aware of studies which compare systematically ironic quotes with their non-ironic counterparts. the paper is structured as follows: following a concise overview of related studies (section 2), we illustrate the three aforementioned categories (section 3), and report on experiments conducted to train neural networks to classify any given instance of a word in quotation marks as one of them. the assumption underlying these experiments is the following: if the distinctions that we have identified are in fact sufficiently indicated within the text, then we would expect a neural network to be able to pick up on these indications and correctly classify these categories. we present the classifier's pitfalls in (7), while the results and their implications for irony detection are discussed in the closing section (8). ## related work within the large body of research on verbal irony in pragmatics, it is widely accepted that verbal irony has two defining features: it is inherently indirect, and it necessarily conveys the speaker's attitude, mostly some degree of dissociation and criticism. most researchers agree that the interpretation of indirectness in general and irony in particular involves several levels of meaning and context-based identification of some incongruity between them. we rely on grice's three-level distinction @xcite between sentence meaning, utterance meaning and speaker's meaning, whereby speaker's meaning is what the speaker means to convey by uttering a given utterance in a given situation. in irony interpretation, contextual information is exploited for two different purposes: as a cue, when it indicates that the utterance meaning is not a plausible candidate for being the speaker's meaning, and as a clue, when it is used to compute an alternative, ironic speaker's meaning which, under the circumstances, may be intended by the speaker @xcite . full interpretation of the speaker's meaning includes the detection of the ironic criticism, as well as the identification of the victim of irony (towards whom the criticism is addressed) and its locus (towards what it is directed) @xcite . in everyday discourse, indirect speaker's meanings in general and ironic criticism in particular may be missed or reconstructed partially. competing pragmatic accounts provide us with insights into the nature of cues which trigger an ironic interpretation, the major ones being: a blatant flouting of the maxim of quality, ("try to make your contribution on that is true") @xcite , related to the classic, aristotelian view of irony as conveying the opposite meaning; a blatant flouting of other gricean maxims, i.e., the expectations underlying cooperative communication @xcite ; the reversal of evaluation @xcite @xcite ; as a pretense @xcite ; and irony as a non-attributive, echoic metarepresentation @xcite @xcite . the latter is specifically related to the use of ironic quotation marks. in this view, a necessary condition for irony comprehension is the recognition that the speaker implicitly mentions, or echoically metarepresents, a true or imagined proposition, thought, belief, opinion, norm or an interpretation thereof, without explicitly attributing it to its source, be it real or imagined. by so doing, she expresses a derogatory attitude towards the echoed utterance, thought, opinion or their interpretation and implicitly criticizes its source @xcite @xcite @xcite . accordingly, in ironic utterances the literal meaning is not substituted for by an indirect, opposite meaning. rather, "the speaker mentions a proposition in such a way as to make clear that she rejects it as ludicrously false, inappropriate, or irrelevant" @xcite . viewing the pattern under discussion as a case of non-attributive metarepresentation explains why the use of quotation marks is non-arbitrary: it might be considered as "borrowed" from typically attributive metarepresentations such as direct speech. studies indicate that quotation marks are associated with irony @xcite and play a beneficial role in its recognition and processing @xcite . the more partial the quotation is vis-à-vis its presumed source, the more likely it is to convey irony @xcite . we examine single words in quotation marks since they are manifestly partial in this respect @xcite . longer units in quotes will be explored at a later stage. from a pragmatic viewpoint, the indirect nature of irony presupposes that textual markers are nonobligatory. however, when they do exist they are mostly equivocal, as they may be used for other pur-poses as well. studies of irony markers in written discourse shed light on phonological, morphological and non-verbal patterns, including exclamation marks, emoticons and quotation marks in written discourse @xcite @xcite . an interesting marker is "multiple uses of irony" @xcite or "redundancy" @xcite @xcite , whereby various cues for irony or multiple occurrences of irony markers in a given co-text support each other and enhance the identification of irony. this may also apply to numerous uses of quotation marks in the same text @xcite . in pragmatics, the interplay between ironic quotation marks and their co-textual environments in mediated political discourse have received special attention @xcite @xcite @xcite @xcite highlighting their evaluative and attitudinal functions. weizman (2020) considers john has been "successful" these last years as a case of non-attributive echoic metarepresentation, whereby an ironic reading relies on the identification of quotation marks as a marker of echoic mention, which, in turn, is a cue for the detection of a mismatch between the proposition in quotes and contextual information. over the past decades, relevant studies in computational linguistics have evolved significantly in their approach to irony detection. initially, researchers focused on lexical and syntactic features, punctuation marks, and positive/negative polarity. in addition to these linguistic features, scholars have particularly emphasized the role of non-verbal elements in social media contexts, such as emoticons and hashtags (e.g., @xcite @xcite @xcite . in terms of computational modeling, early approaches primarily relied on statistical methods, specifically utilizing features like bag-of-words @xcite and pattern-based analysis @xcite . building upon these foundations, researchers then developed rule-based approaches, examining elements such as sentiment disparity between hashtags and text content on twitter @xcite . despite their contributions to the field, these methods proved to be time-and labor-intensive @xcite . consequently, the field has witnessed a shift toward more sophisticated approaches, particularly deeplearning techniques. for instance, the use of sim-ilarity between word embeddings as features for sarcasm detection @xcite . throughout this evolution, quotation marks have consistently been included with other markers of irony in several multi-variant studies of irony detection in computational linguistics and neighboring approaches (e.g., @xcite @xcite @xcite . furthermore, while these various approaches have advanced our understanding, most models continue to treat irony as a rhetorical device or figure of speech rather than a pragmatic phenomenon, often employing binary classifications (ironic vs. non-ironic). moreover, their cotextual environments and various functions have not received specific consideration. in our data, quotation marks enclosing single words are used for three purposes -conveying irony, naming and marking the journalist's awareness of a peculiar lexical choice (henceforth lexical peculiarity). we proceed to illustrate the distinction between them. ## analysis the textual realizations of all three functions are identical: each consists of a single word in quotation marks. furthermore, since in hebrew there are no capital letters, the category of naming is not formally differentiated from the two other categories in any way. the following utterances represent the three categories: 1. they are very particular about saying "halel" every day. in example (1), the quotes encolsing halel mark a proper name -the name of a jewish prayer. in (2), the quotes indicate that the journalist is aware of the non-normative use of an interrogative adverb as a noun. in (3), the word in quotes, right, echoically metarepresents the belief that religions may be perceived as either right or wrong, and convey the journalist's ironic criticism of this simplistic and harmful perception. hence, whereas in example (3) the quotation marks are metarepresentational and typically judgmental, in example (1) they are referential and in example (2) they are meta-linguistic since they convey the speaker's linguistics awareness. additionally, whereas in (1) and ( 2 ) the quotes are local, in the sense that they pertain to the meaning or the form of the word they enclose, in (3) the conveyed stance touches upon a larger co-textual environment since the ironic criticism is directed also at the belief that prescriptive judgments of religion may justify murders on its behalf. ## annotated dataset for hebrew irony our dataset is annotated to distinguish ironic uses of quotation marks from other uses. it is the first of its kind in hebrew, since we are not aware of any other datasets comprised to address the phenomenon of ironic quotes. the dataset consists of op-eds from major and popular israeli news platforms. we collected the data using two methods: (a) automated crawling of op-ed articles from the opinion sections in the platforms in 2019-2020. this data was collected by a social media monitoring and analysis company. (b) manual collection of op-ed articles published in 2020. the data was annotated by three pragmatics experts, who annotated each instance of a single word enclosed in quotation marks in the context of the entire article, distinguishing between naming/lexical peculiarity/irony. in case of disagreement, two labels were assigned to the disputed word, such that the label assigned by two annotators preceded the label assigned by a single annotator. the classifier considered only the first label for the target word. on the whole, we have 59 cases (1.4%) of double annotation.the vast majority of these (56) are related to the distinction between irony and lexical peculiarity. we are pleased to release this new annotated dataset to the nlp community. 1 ## experimental setup we train neural networks to classify any given instance of a word enclosed within quotation marks (henceforth: "target word") as one of the aforementioned classes: "naming", "lexical peculiarity" or "irony". we run each of our sentences through dictabert in order to produce a contextual embedding for each instance of a word within quotation marks. we then aim to train a multi-layer perceptron (mlp) to classify each instance of these contextual embeddings into one of our three categories. as we describe in detail below, we experiment with multiple trains of such an mlp, each time progressively providing the classifier with more information about the word, the sentence, and the surrounding context, in order to determine how much information is truly needed to correctly assess the presence or absence of irony within the quoted word. all mlps are trained 10 epochs, with a learning rate of 0.0001, a hidden layer of size 100, with the adam optimizer, and a batch size of 32. we evaluate the performance of each mlp using 10-fold cross validation; we calculate separate recall, precision, and f1 scores for each of the classes. ## discussion and conclusions starting with the premise that irony is necessarily indirect, this paper aims to delve into the nature of irony detection, by combining pragmatic analyses with experimentation purporting to train neural networks to identify ironic speaker's meaning. through this experiments we can learn about the validity of our predictions and improve them where necessary. with this purpose in mind, we focused on single words enclosed in quotation marks, conceptualized as textual realizations of non-attributive echoic metarepresentation which, in turn, is a possible cue for the detection of a mismatch between the proposition in quotes and contextual information. the analysis of ironic quotation marks shows that a full interpretation of the speaker's ironic meaning requires the detection of echoic mention, somewhat facilitated by the quotation marks, and the identification of the victim of irony (who is being criticized) and its target (what is being criticized. since the textual pattern under study fulfills two additional functions -naming and marking the speaker's awareness of a peculiar lexical choice, we proposed a distinction between these three polysemous patterns, foregrounding the pragmatic differences between them. to our knowledge, no such comparison has been made before. drawing on the pragmatic distinction, we proceeded to examine to what extent the three patterns are distinguished by a neural network, with the underlying assumption that if the distinctions identified through pragmatic analysis are sufficiently indicated in the text, then we would expect a neural network to be able to pick up on these indications and correctly classify these categories. all in all, the experiments yielded good results concerning our primary goal, i.e. the classifier's ability to identify cases of irony (f1 score of .825, as per as for the other two categories, we obtained very good results regarding its ability to distinguish 'naming' from the two other categories (f1 score of .906, as per to conclude, we adopt gibbs and colston's (2023:9) view: we typically believe that irony is a completely human affair, but there have been interesting attempts to create computational models of irony use and understanding. [. . . ] one of the beauties, and major challenges of computer modeling is that it forces researchers to make concrete decisions on how best to implement some linguistic observation or theoretical idea (e.g., how to create a workable model of echoic mention, pretense, or what is meant by incongruity). this statement introduces veale's (2023) discussion of computational models designed to detect irony and produce it. veale compares various computational models and proposes his epic model, combining a theoretical approach with computational expertise, and concludes: "a computational approach to irony is no substitute for an actual theory of irony". the two sides of the mirror are illuminated: @xcite highlight the potential contribution of computational studies to pragmatics, whereas veale (2023) manifestly foregrounds the indispensable contribution of theoretical thinking to a computational approach. the belief in this mutual contribution has been underlying the study we describe in this paper.