Hengam: An Adversarially Trained Transformer for P ersian Temporal Tagging (2022)

Abstract: Many NLP main tasks benefit from an accurate understanding of temporal expressions, e.g., text summarization, question answering, and information retrieval. This paper introduces Hengam, an adversarially trained transformer for Persian temporal tagging outperforming state-of-the-art approaches on a diverse and manually created dataset. We create Hengam in the following concrete steps: (1) we develop HengamTagger, an extensible rule-based tool that can extract temporal expressions from a set of diverse language-specific patterns for any language of interest. (2) We apply HengamTagger to annotate temporal tags in a large and diverse Persian text collection (covering both formal and informal contexts) to be used as weakly labeled data. (3) We introduce an adversarially trained transformer model on HengamCorpus that can generalize over the HengamTagger’s rules. We create HengamGold, the first high-quality gold standard for Persian temporal tagging. Our trained adversarial HengamTransformer not only achieves the best performance in terms of the F1-score (a type F1-Score of 95.42 and a partial F1-Score of 91.60) but also successfully deals with language ambiguities and incorrect spellings. Our code, data, and models are publicly available at https://github.com/kargaranamir/Hengam .

## introduction a wide array of natural language processing (nlp) applications relies on accurately identifying of events and their respective occurrence times. text summarization @xcite @xcite , question answering @xcite @xcite , and information retrieval tasks requiring to classify information in a chronological order @xcite are all examples of such applications. in order to address these needs in the last decades, there has been an increased interest in temporal information extraction systems and developing their appropriate corpora and evaluation frameworks. tempeval challenges are, for instance, great examples of such efforts held as a part of semeval workshops focusing on temporal information extraction @xcite @xcite . the study of temporal expressions in english and other languages has been an ongoing research track in the last decade, spanning renowned rule-based efforts such as heideltime @xcite and @xcite to learning-based approaches, e.g., a transformer-based "bert got a date" @xcite . the majority of efforts in this area have been rule-based, which is suffering from (i) a relatively low recall, as finite rules are usually insufficient to deal with all forms of temporal expressions, and (ii) a relatively low precision, as solely relying on the surface form would lead to a high false positive rate. on the other hand, training on a limited set of examples imposes a challenge for learning-based approaches, as this way, they can hardly see a diverse set of time patterns, even in the presence of large and high-quality datasets @xcite . thus, an approach combining the strength of both rule-based approaches and learning-based approaches in temporal tagging would be extremely beneficial. similar to many other languages, both rule-based approaches @xcite and learningbased approaches @xcite @xcite are developed for the persian language. parstime @xcite is probably the first and the most popular attempt to identify and normalize persian temporal expressions, which also uses the timeml scheme @xcite . parstime being purely rule-based has several limitations: (i) inability to handle ambiguities in the language, (ii) incapability to deal with a wide range of temporal terms, and (iii) failing to generalize. the other studies in persian time tagging have attempted to recognize time and date entities as a subset of named entity recognition (ner) tasks, such as mor-phobert @xcite , beheshti-ner @xcite and parsbert @xcite . these studies all tackle this problem using transfer learning by training a supervised ner model on variations of a pretrained transformer language model, in particular, a bert @xcite model. time and date tags are included in persian ner datasets, such as peyma @xcite , a'laam @xcite , persian-ner (text-mining.ir, 2018), and nsurl'19 @xcite . however, training models based on these datasets do not lead to a high-performance temporal tagging model, as they contain a limited number of temporal tags and do not cover all forms of possible temporal expressions in persian. for instance, peyma, which is used in several studies, including morphobert, beheshti-ner, and parsbert, only contains 2126 sentences containing temporal expressions. in addition to the small number of training examples, these datasets are far from being an appropriate temporal dataset that must cover most types of temporal expressions and consider language-specific constraints. some of the language-specific challenges in persian are: (i) the difference between formal and informal writing styles, (ii) lexical ambiguity (homographs), and (iii) the use of three calendar systems in persian: the gregorian, hijri, and jalali calendars, unlike most of languages, referring mostly to only one or two calendars in their texts. this paper aims to bridge the gap between rulebased and transformer-based approaches by creating an unbiased temporal tagged corpus using a rule-based approach and then adversarial training of a state-of-the-art transformer model. training begins by fine-tuning a pre-trained model on a created corpus, followed by adversarial fine-tuning with a smaller, strongly labeled corpus using projected gradient descent (pgd). the following are the main contributions of this paper: ## related work the approaches for the identification of temporal expression fall within two main categories: (i) rulebased and (ii) learning-based methods. rule-based methods. rule-based methods identify temporal expressions by constructing deterministic rules. here we summarize the main instances of such works, namely gutime @xcite , heideltime (strötgen and gertz, 2010), sutime (chang and manning, 2012), and syntime @xcite . gutime is a part of the tarsqi toolkit to enhance questionanswering systems in temporally-related queries. gutime extends tempex @xcite with machine-learned rules to resolve temporal expressions based on the timeml timex 3 standard. heideltime employs knowledge resources and linguistic clues to normalize extracted temporal expression rules. sutime is another renowned system built on tokensregex (chang and manning, 2014) mapping regular expressions defined over text and tokens to semantic objects. syntime proposes general type-based heuristic rules detecting time mentions based on the similar syntactic behavior of temporal words. syntime identifies temporal tokens in raw text, searches for other specified types in their surroundings, and then merges these segments into temporal expressions. parstime @xcite is the only previous attempt to develop a rule-based temporal tagger capable of identifying and normalizing persian temporal expressions. however, some challenges are not addressed by parstime, such as homographs, common spelling mistakes, and informal variations in temporal expressions in persian. unfortunately, due to a lack of documentation and feedback from the authors, we were not able to run the parstime, but by reviewing the parstime code, we ensured that all predefined patterns are reflected in hengamtagger. furthermore, hengam-tagger resolves some of the parstime challenges by defining exclusion patterns and covering a far broader range of temporal expressions described in the §3.1. learning-based methods. a majority of learningbased methods were introduced at the tempeval challenge of semeval @xcite @xcite . such models traditionally use textual features, such as characters, words, syntactic, and semantic features. these studies have utilized statistical models such as conditional random fields (crfs), markov logic networks, and support vector machines (svms) to model temporal expressions (uzzaman and allen, 2010; @xcite . with the recent advances in nlp, models built on top of pre-trained language models, such as bert @xcite , are introduced @xcite @xcite . these models are trained on several datasets supporting temporal pattern units @xcite @xcite . for the persian language, the learning-based approaches are mainly trained over the general persian ner datasets, and there is no public annotated dataset in standard time schemes, such as timeml. examples of these datasets are peyma @xcite , persian-ner (text-mining.ir, 2018), and nsurl'19 @xcite . there have also been a couple of studies discussing the creation of a dataset of temporal pattern units. however, we were unable to access their data by contacting the authors @xcite . existing persian temporal taggers are created using the above-mentioned ner datasets utilizing a variation of bert transformers @xcite , such as morphobert @xcite , beheshti-ner @xcite and pars-bert @xcite . morphobert (using a persian morphological analyzer combined with bert) and beheshti-ner (utilizing a crf model on top of the bert network) are ner approaches presented at the nsurl'19 workshop @xcite and ranked first and second respectively. previous studies @xcite have noted that, due to the lack of time and date examples in the nsurl'19 and peyma datasets, the worst results of the seven different ner classes were associated with time and date categories. ## materials and methods in this section, we present the workflow of hengam shown in figure 1 . we firstly (i) start with a rulebased tagger (hengamtagger), which is then used in (ii) creating a weakly labeled dataset (hengam-corpus). (iii) ultimately, we present our hengam adversarial transformer model (hengamtransformer) trained over a strongly labeled dataset. we also describe how we develop a gold standard for this task and evaluate hengam variations against the state-of-the-art approaches. ## conclusions in this paper, we proposed hengam, an accurate adversarially trained transformer for persian temporal tagging outperforming state-of-the-art approaches on a diverse and manually created dataset. we achieved this system in the following concrete steps: (1) we developed hengamtagger, a fast and extensible rule-based tool that can extract temporal expressions from any language by creating language-specific patterns 3 . (2) we used hengam-tagger to annotate a large and diverse persian text collection (covering both formal and informal contexts) for temporal tags. this way, we made hengamcorpus and used it as weakly labeled data for subsequent learning-based temporal tagging. (3) we introduced an adversarially trained transformer model on hengamcorpus that can generalize over the hengamtagger's rules evaluated over a set of challenging examples named hengamchallengeset. we studied available persian temporal datasets and found that the current datasets are inadequate for developing a system to identify temporal expressions. we created the first high-quality gold standard for persian temporal tagging called hengamgold. the adversarial hengamtransformer not only achieved the best performance in terms of the f1-score but also successfully dealt with language ambiguities and incorrect spellings.