A C omprehensive C omparison of W ord E mbeddings in E vent & E ntity C oreference R esolution. (2021)

Abstract: Coreference Resolution is an important NLP task and most state-of-the-art methods rely on word embeddings for word representation. However, one issue that has been largely overlooked in literature is that of comparing the performance of different embeddings across and within families. Therefore, we frame our study in the context of Event and Entity Coreference Resolution (EvCR & EnCR), and address two questions : 1) Is there a trade-off between performance (predictive and run-time) and embedding size? 2) How do the embeddings’ performance compare within and across families? Our experiments reveal several interesting findings. First, we observe diminishing returns in performance with respect to embedding size. E.g. a model using solely a character embedding achieves 86% of the performance of the largest model (Elmo, GloVe, Character) while being 1.2% of its size. Second, the larger models using multiple embeddings learns faster despite being slower per epoch. However, it is still slower at test time. Finally, Elmo performs best on both EvCR and EnCR, while GloVe and FastText perform best in EvCR and EnCR respectively.

## introduction coreference resolution (cr) is an important nlp task. it can be subdivided into event and entity coreference resolution (evcr and encr). these tasks serves as the basis for several downstream applications such as information extraction, text summarization, machine translation and text mining @xcite @xcite . state-of-the-art methods for cr @xcite @xcite rely on various word embeddings for word representation. these embeddings are organized into three families: static, contextual and character embeddings @xcite @xcite , each differing in size. contextual embeddings are larger (1024) compared to the other families (usually 300 for static and 50 for character). they also tend to outperform the other families in most tasks but lead to larger and heavier models @xcite . we are thus confronted with a trade-off of performance (predictive & run-time) vs. dimensionality. moreover, embeddings also differ within families which also leads to differences in predictive performance. several studies investigated how different embeddings influence the predictive performance in different tasks @xcite @xcite . however, the two aforementioned issues of the performance vs. dimensionality trade-off and performance variations within and across embedding families have been overlooked to a large extent, especially in coreference resolution. literature is still unclear about which embeddings perform best in which tasks, and whether larger, more expressive embeddings should also be preferred or whether some predictive performance can be compromised for improved run time. thus, we seek to address two questions in the context of cr: 1) is there a trade-off between performance @xcite and embedding size? 2) how do the embeddings' performance compare within and across families? the current state-of-the-art in evcr @xcite rely on three families of embeddings for word representation, and thus provides a suitable frameworks for addressing our research questions. starting from the original model of @xcite , we performed various experiments and ablative studies across and within each family of embeddings, resulting in 16 different models. 1 . we compared their predictive performance, size (number of parameters) , run-time and memory usage. we discovered high level of diminishing returns in term of predictive performance per embedding. the smallest model (using solely a character embedding (dos santos and zadrozny, 2014)) achieves 86% of the performance of the largest model (glove @xcite , elmo @xcite , character embedding) with 1.2% of its size. hence, incorporating additional embeddings leads to diminishing returns in terms of predictive performance. in addition, we found that size and run-time are weakly correlated: larger (more complex) models can converge faster (number of epochs and total training time) than smaller ones . in terms of predictive performance, we found glove and fasttext perform best in evcr and encr respectively in their family with elmo being the best overall. moreover, we found that the smallest aforementioned model outperforms word2vec (∼+10 f1), yielding predictive performance close to the previous state-of-the-art @xcite in evcr (68.43 vs 69 f1). our results can have important implications for practitioners in implementing cr and other nlp models in real-life applications. ## conclusion we used the state-of-the-art in evcr @xcite as a framework to investigate the complexity-performance trade-off and compare the predictive performance of word embeddings across and within the three families. we observed that the smallest model using solely a character embedding yielded 86% of the performance of the original (largest) model (using elmo, glove and a character embeddings) despite being only 1.2% of its size. in fact, that smallest model achieves similar performance (68.43 f1) to the previous state-of-the-art in evcr (69 f1) @xcite . paradoxically, we found that the largest model converged faster during training (by 21% in overall run-time) as it took only 14 epochs vs 24 for the character model. overall, we found size and runtime to be weakly correlated. in addition, our experiments revealed that augmenting the model with additional embeddings does not substantially improve the performance, leading to diminishing returns in term of predictive performance per embedding. concerning predictive performance, one of our most interesting result is that the model using solely a character embedding significantly outperformed (∼+10 f1) a larger model using solely a static embedding (word2vec) while being radically smaller (4% of its size). hence, while character embeddings have often been used as supplementary embeddings, they can actually compete with other embeddings' families in terms of predictive performance per size. finally, our experiments lead us to conclude that for the task of event and entity coreference resolution, glove, fasttext and elmo yielded the best predictive performance. glove and fasttext performed best in evcr and encr respectively in their family while elmo performs best overall. future directions include working on other comprehensive study of embeddings in other tasks and experimenting with cr models using different embeddings for different tasks to improve performance. e.g. glove and fasttext in evcr and encr respectively.