D -Coref: A Fast and Lightweight Coreference Resolution Model using D istil BERT (2020)

Abstract: Smart devices are often deployed in some edge-devices, which require quality solutions in limited amount of memory usage. In most of the user-interaction based smart devices, coreference resolution is often required. Keeping this in view, we have developed a fast and lightweight coreference resolution model which meets the minimum memory requirement and converges faster. In order to generate the embeddings for solving the task of coreference resolution, DistilBERT, a light weight BERT module is utilized. DistilBERT consumes less memory (only 60% of memory in comparison to BERT-based heavy model) and it is suitable for deployment in edge devices. DistilBERT embedding helps in 60% faster convergence with an accuracy compromise of 2.59%, and 6.49% with respect to its base model and current state-of-the-art, respectively.

## introduction edge devices require natural language processing (nlp) for understanding users' input foot_0 . whenever it comes to user interaction, coreference resolution becomes an important task for analysing user's input. it involves determining all referring expressions that point to the same real-world entity. a grouping of referring expressions with the same referent is called a coreference chain or cluster. the goal of a coreference resolution system is to output all the coreference chains of a given text @xcite . several works on coreference resolution are available in the literature having very high accuracy @xcite @xcite @xcite . these models use elmo @xcite and bert @xcite for learning the semantic space of the input. because of the use of such heavy transformers with millions of parameters, these models require a lot of memory. however, smart devices like smartphones should be responsive, light-weight, and energy-efficient models. this motivates us to design a light-weighted coreference resolution model which is suitable for the deployment in smart-devices. we contributed in word context representation of c2f-model @xcite , by forming it from embedding generated by distilbert instead of elmo. we use c2f-model as our baseline model, since this is used as base model in all the recent works @xcite , @xcite ). distilbert is a smaller, faster, cheaper, and light-weight distilled version of bert, which is approx. 97% efficient in comparison to bert. it is 40% smaller in size, and 60% faster @xcite . the embeddings are generated from the distilbert for learning the semantic space of the sentences. after the generation of embeddings, they are passed to the bidirectional lstm, followed by span head for calculation of mention scores. these mention scores are used for forming the coreference chain using hierarchical clustering as defined in @xcite . the standard @xcite dataset is utilized for the performance evaluation of our proposed model. experimental results show that, the developed system requires only 60% memory for execution, in comparison to the bertbased heavy models, while remaining 97% efficient, and 60% faster converging too. we have also shown that 768 embedding dimension is sufficient for word context embedding generation from distilbert. ## the proposed approach in order to utilize the embeddings generated by distilbert for extracting the coreference chains, we have integrated the recently proposed higherorder coreference model proposed in @xcite in our system. we refer to this work as c2fmodel. ## dataset used and experimental set-up conll-2012 shared task corpus is a standard coreference resolution corpus @xcite . we have used the english-based corpus for evaluating the performance of our proposed approach. our experimental setup is almost similar to that of c2f-model and we have modified some parts of their code to generate word context representation from distilbert embeddings, which are: 1) the elmo embeddings are replaced by the dis-tilbert embeddings which are lighter and faster. ## results and analysis in this section, we have discussed the performance of our model on the standard conll-2012 dataset, along with different features of the model. ## conclusion and future work we have devised a fast and light-weight coreference resolution model using distilbert. in order to generate a faster and light-weight model, the accuracy gets compromised. word context representation in reasonable lower dimension can work like representation in higher dimension with proper tuning. our developed system requires only 60% memory for execution, in comparison to the bert-based heavy models, while remaining 97% efficient too. thus, it is suitable for edge devices. in future we will try to come up with a model having better performance with same or lesser space requirement.