Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century S panish (2025)

Abstract: This study explores the use of large language models (LLMs) to enhance datasets and improve irony detection in 19th-century Latin American newspapers. Two strategies were employed to evaluate the efficacy of BERT and GPT models in capturing the subtle nuances nature of irony, through both multi-class and binary classification tasks. First, we implemented dataset enhancements focused on enriching emotional and contextual cues; however, these showed limited impact on historical language analysis. The second strategy, a semi-automated annotation process, effectively addressed class imbalance and augmented the dataset with high-quality annotations. Despite the challenges posed by the complexity of irony, this work contributes to the advancement of sentiment analysis through two key contributions: introducing a new historical Spanish dataset tagged for sentiment analysis and irony detection, and proposing a semi-automated annotation methodology where human expertise is crucial for refining LLMs results, enriched by incorporating historical and cultural contexts as core features.

## introduction irony is a nuanced and often subtle form of communication, especially in historical texts, where cultural context is crucial in understanding the intended meaning. detecting irony in written language has long been a challenge for natural language processing (nlp) @xcite , as it is a trope whose actual meaning differs from what is literally enunciated @xcite . hence, irony detection involves identifying contradictions between what is said and the underlying meaning or context. this challenge becomes even more complex when dealing with historical texts, where linguistic expressions, cultural references, and societal norms differ significantly from modern times. this research focuses on irony detection in 19thcentury latin american newspapers, utilizing large language models (llms) and feed-forward neural networks to experiment with various strategies. the dataset, annotated by experts, was processed for multi-class and binary classification. bert-like models were used for text encoding and transfer learning, while gpt-4o models were applied for sentiment classification with customized prompts. a bert encoder with a feed-forward neural network was used as a comparative baseline to examine the enhancement of text analysis specific to irony detection. a semi-automatic annotation approach was also developed to incorporate new, untagged data. by employing gpt-4o models with tailored prompts, initial classifications were machine-generated and verified by human experts, reducing annotation time and effort while maintaining high-quality standards. integrating human expertise with machinegenerated results enhanced the dataset and allowed for more effective model training. consequently, the study contributes to research on sentiment analysis in historical texts and demonstrates llms' capabilities in enhancing text classification in specialized contexts. ## related work philosophers and linguists have yet to reach a definitive agreement on defining certain figurative tropes including irony, sarcasm, satire, hyperbole, analogy, restatement, paradox, and parody. there have been arguments regarding subtle differences, such as the humorous intention in irony versus the explicitly offensive intention in verbal sarcasm. however, there is a broader consensus on considering irony as the overarching category @xcite . in nlp, irony, sarcasm, and satire have often been used interchangeably, focusing on linguistic and sentiment features. early work suggested satire exploits human tendencies like gullibility and confirmation bias @xcite . other approaches, achieving moderate success, explored linguistic patterns, including slang and profanity, using techniques such as bi-normal separation (bns) @xcite . traditional rule-based methods relied on linguistic indicators like interjections and hyperbole @xcite . however, deep learning models such as elmo and bert have revolutionized the field by incorporating embeddings that capture contextual information across sentences. rajadesingan et al. emphasized the importance of user behavior and historical tweet data in improving sarcasm classification, demonstrating how conversational context can enhance detection accuracy @xcite . both satire and sarcasm detection have significantly benefited from integrating contextual features and deep learning, showing superior performance compared to traditional methods. nonetheless, challenges remain due to the reliance on cultural and societal context, which complicates creating highly accurate models. irony detection has similarly grown in importance as researchers address the challenges of figurative language. early approaches to irony detection primarily utilized static linguistic features and manually annotated datasets but struggled to capture irony's nuanced and dynamic nature, particularly when dealing with context-specific language, such as that found in social media and historical texts. the introduction of deep learning models using transformer-based architectures for llms like bert and gpt has significantly improved the capacity for irony detection. huang et al. explored the application of deep learning techniques such as recurrent neural networks (rnn), revealing that models with attention mechanisms outperformed others on irony detection tasks using social media data @xcite . these models excel by capturing both syntactic and semantic relationships within sentences, enabling comprehensive irony analysis. similarly, ren et al. proposed a knowledge-enhanced neural network that incorporates contextual information from external knowledge sources like wikipedia to enhance irony detection performance @xcite . recent studies have focused on improving irony detection by integrating emotional and contextual cues. lin et al. introduced a newly developed method combining llms with emotion-centric text enhancement to improve the detection of irony @xcite . their approach highlighted the significance of subtle emotional cues often overlooked in traditional models. by using gpt-4 to expand original texts with additional content, the researchers significantly enhanced irony detection in benchmark datasets. ozturk et al. introduced a de-biasing approach for irony, satire, and sarcasm detection, utilizing generative llms to reduce stylistic biases produced by single-source corpus training datasets @xcite . their findings indicate that such stylistic bias impacts model robustness and that llms-based enhancement can mitigate these biases. however, its effect on causal language models like llama-3.1 remains limited. this approach aligns with recent trends in dataset enhancement and bias reduction to improve the detection of figurative language. recent methodologies introduced a human-llms collaborative annotation framework that addresses the limitations of llms-generated labels by combining automated annotation with human expertise @xcite . their method incorporates three major steps: llms predict labels and generate explanations; a verifier assesses label quality; and human annotators review and reannotate labels flagged as low quality. this research shows the necessity for hybrid approaches that merge llms scalability with human precision, particularly for complex tasks such as irony detection. despite advancements, challenges persist in handling class imbalance, stylistic bias, and the nuanced nature of figurative language, especially in historical texts. prior work on deep learning models and llms-driven enhancements often focuses on contemporary datasets, neglecting historical linguistic and cultural context variations. moreover, research on semi-automated annotation strategies leveraging human-llms collaboration is limited. our study addresses these gaps by integrating a structured semi-automated annotation process, improving dataset balance, and fine-tuning domainspecific models for 19th-century spanish. by combining llms-powered augmentation with human verification, we propose a scalable method for enriching training data while maintaining historical linguistic authenticity. this work refines irony detection in historical texts and provides a broader framework for augmenting and improving figurative language classification in underrepresented domains. ## data the dataset used in this research initially comprised a tagged corpus of 2,734 entries. each entry contains text samples randomly extracted from 19th-century latin american newspapersthe latamxix dataset @xcite . tags were manually assigned by three human experts to each entry, falling into one of the following categories: "ironía" (irony), "posi-tivo" (positive), "negativo" (negative), and "neutro" (neutral). these tags represent the predominant sentiment of the extracted text. in our dataset, irony involves multilayered expressions of emotions, including criticism, humor, and sarcasm, as well as the use of poetic language -a distinctive feature of the era. the primary dataset was used to conduct experiments on text enhancement and fine-tune bertlike models for classifying irony. as mentioned previously, the dataset comprises four distinct classes. to extend the scope of experimentation, a copy of the dataset was created where the "positivo," "negativo," and "neutro" classes were merged into a single class labeled "no ironía" (not irony). this transformation converts the task from multi-class classification to binary classification. for better computational processing, a new column named "category_encoded" was created in both datasets to contain the same tags encoded as numerical values, facilitating interpretation. in the second phase of the research, the primary dataset with the original text (without enhancement) was augmented with 1,016 additional entries. these new fragments also originated from the latamxix dataset and were used to implement the semi-automated methodology for annotation. the final dataset consists of 3,750 annotated entries, resulting in a more balanced collection that corrected the initial underrepresentation of the "ironía" class, preserves the historical linguistic value of the original texts, and improves the accuracy of the llms bert-like models fine-tuned for the historical irony classification task foot_0 . ## methodology the experimentation encompasses two primary aspects: dataset enhancement and augmentation and the construction of the classification pipeline. gpt-4o was employed in conjunction with prompt engineering to enhance the text and establish a baseline for measuring improvements in the classification task. subsequently, bert-like models were used to classify the dataset. detailed explanations of these components are provided in the following subsections. ## discussion as expected, the baseline using gpt-4o exclusively for classification demonstrated the weakest performance among the models. its accuracy and recall metrics were significantly lower than those based on our bert-based pipeline and data augmentation techniques, with accuracy values of 0.39 for multi-class and 0.72 for binary classification. this was particularly evident in multi-class classification, where irony detection proved especially challenging. these results highlight the limitations of relying solely on gpt-4o for nuanced text analysis tasks. the binary classification approach delivered superior overall performance, with accuracy values approaching one hundred percent. simplifying the task to binary classification (ironic vs. non-ironic) improved generalization and precision across most configurations. however, the setup consistently struggled with recall for the irony class, which ranged from 0.09 to a maximum of 0.34 in baseline configurations. this indicates that while the model accurately predicted non-ironic cases, it frequently failed to recognize ironic instances. the results of the semi-automated annotation process emphasize the valuable role of human verification in supporting automated annotation methods. the evaluation through human inspection of gpt-4o suggestions for tagging irony revealed some differences between machine-generated tags and human evaluations, as shown in the disparity in positive sentiment detection highlights gpt-4o's limitation in contextualizing historical nuances. for instance, the model struggled to discern an author's potentially positive intent mostly because, during that period, poetic language was often employed to praise individuals or concepts, which was not indicative of irony. however, in modern contexts, such excessive praise, particularly concerning politics, might typically be interpreted as ironic, highlighting the differing interpretations across eras. these findings further illustrate gpt-4o's cultural and historical biases, suggesting that while these models provide substantial assistance, human expertise remains essential for accurate sentiment analysis in complex historical datasets. the process facilitated the generation of classifications by processing a set of previously untagged entries with a higher likelihood of irony using a tailored gpt-4o prompt. the gpt-4o automatic classifications and the detailed justifications were reviewed by human experts for accuracy. confirmed annotations were integrated into the pri-mary dataset, enriching and balancing it. this effectively addressed class imbalance as shown in for multi-class classification, the augmented dataset led to higher recall values for irony detection, suggesting that the semi-automated annotation strategy contributed positively. however, the performance of non-ironic classes showed slight reductions, highlighting the trade-offs involved in emphasizing irony-related signals. in binary classification, combining the semi-automated annotation process with the dccuchile/bert-base-spanish-wwmcased model yielded strong results, with precision and recall values of 0.70 and 0.93, respectively. notably, recall improved significantly, increasing by up to 0.50 compared to previous experiments. these results indicate that the semi-automated process helped expand the dataset and mitigate imbalance, enhancing irony detection foot_1 . interestingly, the enhanced dataset, designed for emotional and contextual enrichment, while effectively sharpening certain sentiment cues, did not result in significant gains in irony detection over the original dataset. this finding indicates that enhancing emotional intensity alone does not necessarily capture the subtleties of historical irony. for example, gpt-4o struggled to identify irony when applying enhancement techniques to a sentence expressing political contradictions during the porfiriato in mexico foot_2 , primarily due to a lack of historical context. as depicted in figure 3 , porfirio díaz, who initially opposed re-elections citing constitutional violations, ironically remained in power for 35 years. although enhancing has been successful in previous works @xcite , it did not effectively capture the specific cultural and historical features present in latin-american historical texts. this highlights the cultural bias embedded in commercial models such as gpt-4o, emphasizing the need for more tailored approaches when working with historical and culturally nuanced materials. overall, the semi-automated annotation process, especially in binary classification, achieved the best performance for irony detection. this approach's ability to expand the dataset and address class im-balance marks the best approach we found for this task. despite these advances, irony detection continues to present significant challenges due to its inherent complexity. historical texts bring additional layers of difficulty with their unique linguistic and cultural references. future research should focus on refining domain-specific prompts, evaluating alternative models, and developing increasingly automated architectures, including agent-based workflows that systematically incorporate historical context to enhance irony detection accuracy. continued efforts to expand and enrich historical datasets will contribute to more reliable and generalized methods for irony detection and sentiment analysis across diverse cultural and historical contexts, ultimately enriching both humanities scholarship and the capabilities of large llms.