Exploring Statistical and Neural Models for Noun Ellipsis Detection and Resolution in E nglish (2020)

Abstract: Computational approaches to noun ellipsis resolution has been sparse, with only a naive rule-based approach that uses syntactic feature constraints for marking noun ellipsis licensors and selecting their antecedents. In this paper, we further the ellipsis research by exploring several statistical and neural models for both the subtasks involved in the ellipsis resolution process and addressing the representation and contribution of manual features proposed in previous research. Using the best performing models, we build an end-to-end supervised Machine Learning (ML) framework for this task that improves the existing F1 score by 16.55% for the detection and 14.97% for the resolution subtask. Our experiments demonstrate robust scores through pretrained BERT (Bidirectional Encoder Representations from Transformers) embeddings for word representation, and more so the importance of manual features– once again highlighting the syntactic and semantic characteristics of the ellipsis phenomenon. For the classification decision, we notice that a simple Multilayar Perceptron (MLP) works well for the detection of ellipsis; however, Recurrent Neural Networks (RNN) are a better choice for the much harder resolution step.

## introduction noun ellipsis is a linguistic phenomenon where the head noun of a noun phrase gets deleted, without making the sentence ungrammatical. for example in the sentence in (1) from @xcite , the noun presentation is elided at "[e]". 1. john's presentation on urban development was virtually ignored because @xcite ] was so much more interesting. the elided information can be retrieved from the previous context as in (1) or with the knowledge of idiomatic usage of language as in i will be back in two @xcite . where two means two minutes. it is also possible that the reference of the elided information comes from extra-linguistic, situational context. for example, consider a speaker pointing towards the roses in a shop and saying an utterance as in i will take two @xcite . while human interlocutors resolve any such elided information by disambiguating from context, cognitive commonsense extension and reasoning @xcite , ellipsis resolution can be a hard task for natural language processing (nlp) systems @xcite . resolution of ellipsis comprises two tasks -detection of the elided material and antecedent selection @xcite . ellipses occur in the environment of certain syntactical structures or trigger words, also known as licensors or triggers of ellipses. they are useful syntactic cues for the detection of noun ellipsis. see figure 1 for an example of the noun ellipsis resolution process. ## related work nominal ellipsis has been a topic of interest in theoretical linguistics for a very long time @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . computational approaches to the ellipsis phenomenon majorly focus on the verb phrase ellipsis (vpe) along with a few related phenomenon such as gapping, sluicing and do-so anaphora, for instance, the detection of vpe in the penn treebank using pattern match @xcite , a transformation learning-based approach to generated patterns for vpe resolution @xcite , the domain independent vpe detection and resolution using machine learning @xcite , automatically parsed text @xcite , sentence trimming methods @xcite , linguistic principles (mcshane and babkin, 2016), improved parsing techniques that encode elided material dependencies for reconstruction of sentences containing gapping @xcite , discriminative and margin infused algorithms @xcite , multilayer perceptrons (mlp) and transformers @xcite . in recent times, there has been a surge in the computational research on nominal ellipsis and closely related phenomena @xcite @xcite @xcite . for the resolution process, we previously proposed a rule based system @xcite that detects noun ellipsis using syntactic constraints on licensors of ellipsis and resolves them by matching part-of-speech (pos) tag similarity between the licensor of ellipsis and the modifier of the antecedent. it later fine tunes these syntactic rules on a small curated dataset that contains 234 instances of noun ellipsis along with some negative samples @xcite . for the present paper, we further the research on noun ellipses by using the noel corpus annotated by us previously @xcite to experiment with state-ofthe-art ml models. ## the proposed approach following the vpe resolution framework presented by @xcite , we investigate a similar framework for noun ellipsis resolution in english and present alternative choices of the models at each step as shown in figure 2 . we use the noel corpus @xcite ) that marks noun ellipsis instances as a separate layer (using the stand-off annotation scheme) on the cornell movie dialogs corpus @xcite . the corpus marks a total of 946 annotations, of which 438 are described as endophoric, i.e. with a textual antecedent, and 508 exophoric, i.e. without a textual antecedent. ## experiments we choose naive bayes and linear svm as our statistical models, and multilayer perceprton (mlp) and bidirectional long short term memory (bi-lstm) network as our neural models. detection task, we take the annotated 946 positive samples (exophoric) and randomly choose 946 negative samples. similarly, for the resolution task, we take 438 positive samples (endophoric) and 438 randomly chosen negative samples. we perform a standard 70-10-20 split to obtain the train, development and test set respectively, and follow the 5-fold cross validation procedure to capture both classes properly in each case. for mlp, we take a simple, two-layer feedforward network (ffnn) or two layers of multiple computational units interconnected in a feed-forward way without loops. we have a single hidden layer with 768 neurons and a sigmoid function. a unidirectional weight connection exists between the two successive layers. the classification decision is made by turning the input vector representations of a word with its context into a score. the network has a softmax output layer. for bi-lstm, we have embedding layer, time-distributed translate layer, bi-lstm (rnn) layer, batch normalization layer, dropout layer and prediction layer. the activation used is softmax. the loss function is calculated with cross entropy. we train in batch sizes of 16 and early stopping with max epochs of 100. in early stopping the patience is kept to be 10 and the optimizer used is adam. we use default values for the learning rate. we use keras @xcite for coding the models. ## results we evaluate the performance of our models in terms of f1-score, computed by taking an average f1-scores obtained from the 5-folds results. we experiment with sixteen models each for the noun ellipsis detection and resolution. the results on the testset for precision, recall and f1-score values are presented in table .2. as expected, the neural models perform significantly better than the statistical ones for both the subtasks. our experiments show that for the detection task, bert embeddings with a simple mlp gives best scores. this is expected because, bert currently provides the most powerful contextual word representations, using 12 separate attention mechanism for each layer, where, at each layer, each token can focus on 12 distinct aspects of other tokens. since transformers @xcite use many distinct attention heads (12*12=144 for the base bert model), each head can focus on a different kind of constituent combinations, making bert broadly attending over the whole sentence. in our task, the subtask classification representation precision recall f1-score ft 0.2992 0.5112 0.3775 ft+f 0.3379 0.5750 0.4257 naive bayes bert 0.3919 0.5503 0.4578 bert+f 0.4558 0.6230 0.5264 ft 0.4001 0.4960 0.4429 ft+f 0.5024 0.6040 0.5485 svm bert 0.4966 0.6003 0.5435 bert+f 0.5372 0.6611 0.5927 resolution ft 0.5901 0.6577 0.6221 ft+f 0.7005 0.8122 0.7522 mlp bert 0.6951 0.7798 0.7350 bert+f 0.8015 0.8531 0.8265 ft 0.6009 0.6321 0.6161 ft+f 0.7029 0.8044 0.7502 bi-lstm bert 0.7195 0.7383 0.7288 bert+f 0.8652 0.8166 0.8401 ## conclusion we explored statistical and neural models for noun ellipsis detection and resolution, presenting a strong results for this task. as expected, neural classifiers perform significantly better than the statistical with the same input representation. as with several other nlp tasks, the contextual nature of bert is useful for noun ellipsis resolution too, making robust predictions with simple neural classifiers. finally, addition of manual features boosts the performance of all classifiers including those that use bert, highlighting that ellipsis is a syntactically constrained phenomenon.