Towards Open Domain Event Trigger Identification using Adversarial Domain Adaptation (2020)

Abstract: We tackle the task of building supervised event trigger identification models which can generalize better across domains. Our work leverages the adversarial domain adaptation (ADA) framework to introduce domain-invariance. ADA uses adversarial training to construct representations that are predictive for trigger identification, but not predictive of the example’s domain. It requires no labeled data from the target domain, making it completely unsupervised. Experiments with two domains (English literature and news) show that ADA leads to an average F1 score improvement of 3.9 on out-of-domain data. Our best performing model (BERT-A) reaches 44-49 F1 across both domains, using no labeled target data. Preliminary experiments reveal that finetuning on 1% labeled data, followed by self-training leads to substantial improvement, reaching 51.5 and 67.2 F1 on literature and news respectively.

## introduction events are a key semantic phenomenon in natural language understanding. they embody a basic function of language: the ability to report happenings. events are a basic building block for narratives across multiple domains such as news articles, stories and scientific abstracts, and are important for many downstream tasks such as question answering @xcite and summarization @xcite . despite their utility, event extraction remains an onerous task. a major reason for this is that the notion of what counts as an "event" depends heavily on the domain and task at hand. for example, should a system which extracts events from doctor notes only focus on medical events (eg: symptoms, treatments), or also annotate lifestyle events (eg: dietary changes, ex-ercise habits) which may have bearing on the patient's illness? to circumvent this, prior work has mainly focused on annotating specific categories of events @xcite @xcite or narratives from specific domains @xcite . this has an important implication for supervised event extractors: they do not generalize to data from a different domain or containing different event types @xcite . conversely, event extractors that incorporate syntactic rule-based modules @xcite tend to overgenerate, labeling most verbs and nouns as events. achieving a balance between these extremes will help in building generalizable event extractors, a crucial problem since annotated training data may be expensive to obtain for every new domain. prior work has explored unsupervised @xcite , distantly supervised @xcite @xcite and semisupervised approaches @xcite @xcite , which largely focus on automatically generating in-domain training data. in our work, we try to leverage annotated training data from other domains. motivated by the hypothesis that events, despite being domain/ task-specific, often occur in similar contextual patterns, we try to inject lexical domain-invariance into supervised models, improving generalization, while not overpredicting events. concretely, we focus on event trigger identification, which aims to identify triggers (words) that instantiate an event. for example, in "john was born in sussex", born is a trigger, invoking a birth event. to introduce domain-invariance, we adopt the adversarial domain adaptation (ada) framework @xcite which constructs representations that are predictive for trigger identification, but not predictive of the example's domain, using adversarial training. this framework requires no labeled target domain data, making it completely unsupervised. our experiments with two domains (english literature and news) show that ada makes supervised models more robust on out-of-domain data, with an average f1 score improvement of 3.9, at no loss of in-domain performance. our best performing model (bert-a) reaches 44-49 f1 across both domains using no labeled data from the target domain. further, preliminary experiments demonstrate that finetuning on 1% labeled data, followed by self-training leads to substantial improvement, reaching 51.5 and 67.2 f1 on literature and news respectively. ## approaching open domain event trigger identification throughout this work, we treat the task of event trigger identification as a token-level classification task. for each token in a sequence, we predict whether it is an event trigger. to ensure that our trigger identification model can transfer across domains, we leverage the adversarial domain adaptation (ada) framework @xcite , which has been used in several nlp tasks @xcite @xcite @xcite . ## incorporating minimal labeled data finetuning on labeled data: we run finetuning experiments to study improvement in model performance on incorporating small amounts of labeled target domain data. for both domains, we finetune bert-a, slowly increasing the percentage of labeled data used from 1%-5%. 5 we compare bert-a with two other models. the first model is naive bert with no domain adaptation (bert-noda). the second model is a bert model trained via supervised domain adaptation (bert-feda), which we use as an indicator of ceiling performance. the supervised domain adaptation method we use is the neural modification of frustratingly easy domain adaptation developed in @xcite . frustratingly easy domain adaptation @xcite uses a feature augmentation strategy to improve performance when annotated data from both source and target domains is available. this algorithm simply duplicates input features 3 times, dataset p r f1 timebank 68.9 65.5 67.2 litbank 40.3 71.5 51.5 figures 2 and 3 present the results of these experiments. performance of all models steadily improves with more data, but bert-a starts with a much higher f1 score than bert-noda, demonstrating that ada boosts performance when little annotated training data is available. performance increase of bert-noda is suprisingly rapid, especially on litbank. however, it is worth noting that 5% of the litbank training set is ∼10,000 tokens, which is a substantial amount to annotate. therefore, bert-a beats bert-noda on sample efficiency. we can also see that bert-a does not do much worse than bert-feda, which performs supervised adaptation. ## conclusion in this work, we tackled the task of building generalizable supervised event trigger identification models using adversarial domain adaptation (ada) algorithm 1 selftrain(d l , d u , t ) input: teacher model t , labeled data d @xmath0, e l 1 ), ..., (x l m , x l m )}, unlabeled data d @xmath1, ...x u n }, output: trained student model s 1: finetune the teacher model t by minimizing cross-entropy loss on labeled data 1 m m @xmath2, e l i ) 2: generate labels {e u 1 , ..., e u n } for unlabeled data d u using t 3: train a student model s by minimizing crossentropy loss on both datasets d l , d u 1 m m @xmath3, e l i ) + 1 n n @xmath4, e u i ) 4: iterative training: repeat step 2 using updated student model s to introduce domain-invariance. our experiments with two domains (english literature and news)