Anaphora Resolution in Dialogue: Cross-Team Analysis of the DFKI - T alking R obots Team Submissions for the CODI - CRAC 2021 Shared-Task (2021)

Abstract: We compare our team’s systems to others submitted for the CODI-CRAC 2021 Shared-Task on anaphora resolution in dialogue. We analyse the architectures and performance, report some problematic cases in gold annotations, and suggest possible improvements of the systems, their evaluation, data annotation, and the organization of the shared task.

## introduction the goal of this paper is to compare the results obtained by the anaphora resolution systems developed by the dfki-talkingrobots team for the codi-crac 2021 shared-task (ccst) @xcite to the results of the other systems in ccst. 1 we submitted two systems for the anaphora resolution task (ar), and one system for discourse deixis resolution (dd). track ar ar dd system wcs m2m ddr setting pred. pred. pred. baselines ---learning framework workspace clustering mention to mention pairing heuristics siamese net markable identification spacy bilstm-crf spacy train. data ccst ccst ccst dev. data ccst ccst ccst wcs employs a clustering approach explained in @xcite . wcs differs from other models that participated in the competition in several ways. both winner models (utd and lxucs) are based on the implementation described in @xcite ) that is an adaptation of the endto-end c2f-coref model introduced by @xcite . the model which achieved the third best result (ku_nlp) uses pointer networks and models pairwise relations between mentions. unlike these models wcs compares each mention to the clusters available in the workspace and decides whether it should be assigned to one of the existing clusters or initiates a new cluster. also, wcs did not use any additional data for training and relied only on the annotations provided by the ccst organizers. wcs achieved an average f1-score of 64.99% on the light data, 43.93% on ami, 59.93% on persuasion and 53.55% on switchboard. when evaluated on the gold mentions, wcs showed the following results: 73.24% f-score on light, 56.48% on ami, 72.06% on persuasion and 62.85% on switchboard. having gold mentions as input gives an improvement of up to 12% on ami and persuasion, up to 9% on switchboard and 6% on light. wcs ranked number four on the leaderboard. we see this as an encouraging result for the first prototype of this coreference system. ## discourse deixic resolution (ddr) next we present a cross-team comparison of the discourse deixis resolution systems submitted to ccst, as well as some difficulties we had with the gold discourse deixis annotations. we discuss the eval-dd(pred) track, as we did not participate in eval-dd(gold). there were only two submissions, from us and from the utd_nlp team. ## conclusions and ccst outlook we compared our systems and their results to the systems and results by other teams. we also presented some ideas that this analysis gave us for the further development of our systems. besides improvements of mention detection we also plan to combine wcs and m2m to benefit from their complementary strengths. as the analysis showed, m2m did well on resolving personal pronoun coref-erence, whereas wcs' clustering approach is better than m2m at noun coreference cases. another interesting challenge to consider is to also account for discourse deixis in one integrated system or even by the same approach, similarly to the utd_nlp system. admittedly, the difficult examples that we have pointed out are ones that every state-of-the-art system finds hard. in the scope of the analysis that we were able to carry out given the time and resources, we could not identify properties of examples that different system configurations are able to handle versus not, in order to draw generalizations on what kinds of examples are handled well by what approaches, and why. such analysis is also complicated by the fact that the systems differ in markable detection. a systematic deeper analysis is an interesting topic for future research. to conclude, we would like to make a few suggestions for the future of the shared task. we think that the shared task is very useful and it would be good to continue running it, because anaphora resolution is far from being a solved task. first, we suggest to have separate tracks for the evaluation of systems trained only on the provided data vs. systems (pre-)trained on external/additional data. this would help not only to achieve a fair comparison but also to see whether improvements are due to (pre-)training with more data and transfer or due to better feature engineering and modeling. in order to study what specifically was learned from the external data that could not be learned by the training data provided in ccst it would be useful to compare the results of models trained with the same approach on only the external data vs. only the ccst data vs. using all data. second, we would like to see a separate comparison of mention detection and anaphor identification results for the different systems. our experience showed that finding correct mentions is crucial for the overall performance and we believe that this stage should be evaluated separately. it would also be interesting to compare results on both dialogues and narrative texts. third, future competitions might consider using more fine-grained evaluation metrics because different cases of coreference have different difficulty levels. e.g., 'a black cat' and 'the black cat' are much easier to resolve compared to 'a black cat' and 'the dark and furry creature'. we believe that a more differentiated evaluation could be done (at least partially) in an automated way. for instance, similarly to the sieve-based approach @xcite one could compare a referent and its true antecedent using exact string matching vs. head matching, compute similarity between the corresponding mention embeddings or count the number of candidates (valid markables) between the anaphor and its antecedent. our analysis showed that some tricky cases of coreference, e.g., 'this sword' and 'your gift to us, the soldiers' were missed by all participating systems and we expect that weighting each pair w.r.t. the difficulty level and/or distinguishing difficulty levels in the presentation of the results would help to shift the focus towards resolving 'harder' cases. the evaluation could also report results on pronominal and nominal anaphora separately, and distinguish the different syntactic types of antecedents. finally, we have some suggestions for the organization of the shared task. namely, it would be helpful to have a more detailed description of the task, its tracks, test data format (with a couple of input-output examples). for instance, it was unclear that for the eval-dd(gold) track we would only get anaphors in the identity column and would need to distinguish between discourse deixis and 'standard' anaphora markables. more information regarding the submission procedure would also be useful, for example for the case that a team has more than one system for one track. quality control of the annotations should be improved, or gold vs. silver (vs. bronze) data should be distinguished. data quality is very important, because we are making conclusions about the correctness of models based on the annotations. one possibility could be to ask participating teams to provide annotations or additional data sets annotated according to uniform guidelines. as for new challenges to include in the shared task in the future, there is no shortage of smaller steps and giant leaps. the easiest ones were already mentioned above, i.e., evaluation of the contribution of external training data, separate evaluation of mention detection and anaphor identification, as well as a more finegrained evaluation scheme, in order to obtain more insight into the capabilities of the different models. other potential extensions depend on the availability of annotated data, and even agreed annotation standards. thanks to existing datasets it would probably be most feasible to extend the task, on the one hand, to zero pronouns and other forms of implicit entity reference and/or event references, and on the other hand, to other languages. multimodal reference would also be interesting, but more challenging, from reference to visual objects embedded in text to reference to physical situations, for example in cooking or physical activity videos, which are used in the languageand-vision community. finally, non-verbal forms or reference, such as pointing, are in our view the most challenging, due to lack of data as well as annotation standards, and the difficulty of annotation.