We introduce the task of predicting adverbial presupposition triggers such as also and again . Solving such a task requires detecting recurring or similar events in the discourse context , and has applications in natural language generation tasks such as summarization and dialogue systems . We create two new datasets for the task , derived from the Penn Treebank and the Annotated English Gigaword corpora , as well as a novel attention mechanism tailored to this task . Our attention mechanism augments a baseline recurrent neural network without the need for additional trainable parameters , minimizing the added computational cost of our mechanism . We demonstrate that our model statistically outperforms a number of baselines , including an LSTM-based language model . In pragmatics , presuppositions are assumptions or beliefs in the common ground between discourse participants when an utterance is made ( Frege , 1892 ; Strawson , 1950 ; Stalnaker , 1973 Stalnaker , , 1998 ) ) , and are ubiquitous in naturally occurring discourses ( Beaver and Geurts , 2014 ) . Presuppositions underly spoken statements and written sentences and understanding them facilitates smooth communication . We refer to expressions that indicate the presence of presuppositions as presupposition triggers . These include definite descriptions , factive verbs and certain adverbs , among others . For example , consider the following statements : ( 1 ) John is going to the restaurant again . * Authors ( listed in alphabetical order ) contributed equally . ( 2 ) John has been to the restaurant . ( 1 ) is only appropriate in the context where ( 2 ) is held to be true because of the presence of the presupposition trigger again . One distinguishing characteristic of presupposition is that it is unaffected by negation of the presupposing context , unlike other semantic phenomena such as entailment and implicature . The negation of ( 1 ) , John is not going to the restaurant again . , also presupposes ( 2 ) . Our focus in this paper is on adverbial presupposition triggers such as again , also and still . Adverbial presupposition triggers indicate the recurrence , continuation , or termination of an event in the discourse context , or the presence of a similar event . In one study of presuppositional triggers in English journalistic texts ( Khaleel , 2010 ) , adverbial triggers were found to be the most commonly occurring presupposition triggers after existential triggers . 1 Despite their frequency , there has been little work on these triggers in the computational literature from a statistical , corpus-driven perspective . As a first step towards language technology systems capable of understanding and using presuppositions , we propose to investigate the detection of contexts in which these triggers can be used . This task constitutes an interesting testing ground for pragmatic reasoning , because the cues that are indicative of contexts containing recurring or similar events are complex and often span more than one sentence , as illustrated in Sentences ( 1 ) and ( 2 ) . Moreover , such a task has immediate practical consequences . For example , in language generation applications such as summarization and dialogue systems , adding presuppositional triggers in contextually appropriate loca-1 Presupposition of existence are triggered by possessive constructions , names or definite noun phrases . tions can improve the readability and coherence of the generated output . We create two datasets based on the Penn Treebank corpus ( Marcus et al . , 1993 ) and the English Gigaword corpus ( Graff et al . , 2007 ) , extracting contexts that include presupposition triggers as well as other similar contexts that do not , in order to form a binary classification task . In creating our datasets , we consider a set of five target adverbs : too , again , also , still , and yet . We focus on these adverbs in our investigation because these triggers are well known in the existing linguistic literature and commonly triggering presuppositions . We control for a number of potential confounding factors , such as class balance , and the syntactic governor of the triggering adverb , so that models can not exploit these correlating factors without any actual understanding of the presuppositional properties of the context . We test a number of standard baseline classifiers on these datasets , including a logistic regression model and deep learning methods based on recurrent neural networks ( RNN ) and convolutional neural networks ( CNN ) . In addition , we investigate the potential of attention-based deep learning models for detecting adverbial triggers . Attention is a promising approach to this task because it allows a model to weigh information from multiple points in the previous context and infer long-range dependencies in the data ( Bahdanau et al . , 2015 ) . For example , the model could learn to detect multiple instances involving John and restaurants , which would be a good indication that again is appropriate in that context . Also , an attention-based RNN has achieved success in predicting article definiteness , which involves another class of presupposition triggers ( Kabbara et al . , 2016 ) . As another contribution , we introduce a new weighted pooling attention mechanism designed for predicting adverbial presupposition triggers . Our attention mechanism allows for a weighted averaging of our RNN hidden states where the weights are informed by the inputs , as opposed to a simple unweighted averaging . Our model uses a form of self-attention ( Paulus et al . , 2018 ; Vaswani et al . , 2017 ) , where the input sequence acts as both the attention mechanism 's query and key / value . Unlike other attention models , instead of simply averaging the scores to be weighted , our approach aggregates ( learned ) attention scores by learning a reweighting scheme of those scores through another level ( dimension ) of attention . Additionally , our mechanism does not introduce any new parameters when compared to our LSTM baseline , reducing its computational impact . We compare our model using the novel attention mechanism against the baseline classifiers in terms of prediction accuracy . Our model outperforms these baselines for most of the triggers on the two datasets , achieving 82.42 % accuracy on predicting the adverb " also " on the Gigaword dataset . The contributions of this work are as follows : 1 . We introduce the task of predicting adverbial presupposition triggers . 2 . We present new datasets for the task of detecting adverbial presupposition triggers , with a data extraction method that can be applied to other similar pre-processing tasks . 3 . We develop a new attention mechanism in an RNN architecture that is appropriate for the prediction of adverbial presupposition triggers , and show that its use results in better prediction performance over a number of baselines without introducing additional parameters . 2 Related Work In this work , we have investigated the task of predicting adverbial presupposition triggers and introduced several datasets for the task . Additionally , we have presented a novel weighted-pooling attention mechanism which is incorporated into a recurrent neural network model for predicting the presence of an adverbial presuppositional trigger . Our results show that the model outperforms the CNN and LSTM , and does not add any additional parameters over the standard LSTM model . This shows its promise in classification tasks involving capturing and combining relevant information from multiple points in the previous context . In future work , we would like to focus more on designing models that can deal with and be optimized for scenarios with severe data imbalance . We would like to also explore various applications of presupposition trigger prediction in language generation applications , as well as additional attention-based neural network architectures .

Challenge: Adverbaial triggers indicate the event recurrence, continuation, or termination in the discourse context and are frequently found in English but there are few related works.
Approach: They introduce an adverbial presupposition trigger prediction task and datasets and propose an attention mechanism that augments a recurrent neural network without additional trainable parameters.
Outcome: The proposed model outperforms baselines including an LSTM-based language model on most of the triggers on the two datasets.