Anaphora Resolution in Dialogue: Description of the DFKI - T alking R obots System for the CODI - CRAC 2021 Shared-Task (2021)

Abstract: We describe the system developed by the DFKI-TalkingRobots Team for the CODI-CRAC 2021 Shared-Task on anaphora resolution in dialogue. Our system consists of three subsystems: (1) the Workspace Coreference System (WCS) incrementally clusters mentions using semantic similarity based on embeddings combined with lexical feature heuristics; (2) the Mention-to-Mention (M2M) coreference resolution system pairs same entity mentions; (3) the Discourse Deixis Resolution (DDR) system employs a Siamese Network to detect discourse anaphor-antecedent pairs. WCS achieved F1-score of 55.6% averaged across the evaluation test sets, M2M achieved 57.2% and DDR achieved 21.5%.

## introduction this paper describes the anaphora resolution system developed by the dfki-talkingrobots team for the codi-crac 2021 shared-task (ccst) 1 @xcite . anaphora resolution (ar) is important for many natural language processing tasks, such as information extraction, summarization, question answering, etc. most existing systems have been developed for ar in text and focus on coreference resolution. ccst was set up to address ar in dialogue, including not only coreference but also bridging and discourse deixis. we were inspired to join ccst, although we did not have an existing ar system to start with. initial small tests of publicly available pre-trained ar models from the allennlp and huggingface libraries on our dialogue data in the domain of disaster response team communication @xcite ) yielded discouraging results. participating in ccst was a way to kick start the development of an ar system of our own. our long-term aim is to develop a domain-independent ar system to be used in several projects of the talking robots group at dfki foot_0 , including for the interpretation of team communication in disaster response @xcite . our system for the ccst is the first step in this direction, and should be considered work in progress. it consists of two independent ar subsystems: workspace coreference resolution system (wcs, sec. 2.1); mention-to-mention coreference resolution system (m2m, sec. 2.2); and the discourse deixis resolution system (ddr, sec. 2.4).the subsystems were developed using the data provided for ccst and evaluated in the ccst tracks eval-ar and eval-dd (pred), respectively. see tab. 1 for a summary. sec. 2 describes the subsystems one by one, including the respective approach, mention detection method and how we used the ccst data. we also highlight the differences between our approaches and recent related work in the respective subsections. in sec. 3 we provide the results and error analysis for each subsystem. we conclude and point out future work in sec. 4. ## conclusions and outlook we described the three anaphora resolution subsystems we developed from scratch for ccst using only the datasets provided by the ccst organizers. wcs is a coreference resolution approach that incrementally clusters mentions using semantic similarity based on embeddings combined with lexical feature heuristics. it employs spacy for markable detection. m2m is another coreference resolution approach which pairs the mentions for the same entities. for markable detection it uses our own bilstm-crf model trained on the ccst data. wcs achieved f1-score of 55.6% averaged across the ccst evaluation test sets, using predicted markables; the m2m system achieved 57.2%. finally, ddr employs a siamese network to distinguish true discourse anaphor-antecedent pairs. its markable detection is also based on spacy. it has average f1-score 21.5% using predicted mentions. our models employ novel ideas, but are still in the early stages of development. ccst gave us a boost and we see many options for future work regarding both markable detection as well as the ar resolution models themselves. for example, we would like to improve the feature extraction for wcs; enhance m2m by clustering; combine wcs and m2m; and improve the detection of discourse deixis mentions that need to be resolved. we also plan to apply the systems on other datasets, which we are currently preparing. combined head and span embeddings (3) further improve the performance by 1.2%. interestingly, the trend is different for bert: bert head (4) achieves 64.82% and bert span (5) 66.43%. note that in our case bert span refers to an average over all bert embeddings of the tokens within the mention span (this is different from spanbert). we observe a very small increase in performance when bert span is combined with bert head (6), 66.59% conll score; bert and glove combined together (7) achieve 66.96%. we also found that wcs performs better when we add numberbatch embeddings, e.g., bert with numberbatch (8) has 68.08% conll score. we found that the combination of glove, bert, numberbatch, masked lm, distance and speaker embeddings (9) gives us 68.13% on the test data. furthermore, when the position embedding is added (10) the score increases to 70.09 that indicates that the positions of mentions within the document foot_4 play a very important role in resolving coreference. although we tested various combinations of embeddings, our experiments were limited to a subset of the data (we used only 50 documents for training). another limitation of our approach is that we defined the dimensionality of each embedding separately, i.e., when passing an embedding through several linear layers we defined the output dimensionality independently for each embedding type. it would be interesting to see how interactions between different embeddings affect the dimensionality. for example, it might be that 100 dimensions work well for glove when it is used alone but when it is combined with bert the dimensionality should be increased (or decreased) to achieve better performance. we would also like to test whether ablation studies show similar trends with different types of data (e.g., dialogue vs. non-dialogue, different genres) and perform a proper statistical analysis of the results. we see our present results as a starting point of an investigation of interactions between the embedding types in dialogue, to determine what embeddings work best in which case and what dimensionality they should have.