B en C oref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference Annotations (2023)

Abstract: Coreference Resolution is a well studied problem in NLP. While widely studied for English and other resource-rich languages, research on coreference resolution in Bengali largely remains unexplored due to the absence of relevant datasets. Bengali, being a low-resource language, exhibits greater morphological richness compared to English. In this article, we introduce a new dataset, BenCoref, comprising coreference annotations for Bengali texts gathered from four distinct domains. This relatively small dataset contains 5200 mention annotations forming 502 mention clusters within 48,569 tokens. We describe the process of creating this dataset and report performance of multiple models trained using BenCoref. We anticipate that our work sheds some light on the variations in coreference phenomena across multiple domains in Bengali and encourages the development of additional resources for Bengali. Furthermore, we found poor crosslingual performance at zero-shot setting from English, highlighting the need for more language-specific resources for this task.

## introduction coreference resolution is the task of identifying all references to the same entity in a document. this task originally started as a sub-task of information extraction. the message understanding conferences @xcite first introduced three tasks, collectively referred to as semeval, designed to measure the deeper understanding of any information extraction (ie) system. one of these three tasks proposed in the event was coreferencial noun phrase identification. the automatic content extraction (ace) program @xcite was the first major 1 codes used to generate the results along with data is available at: https://github.com/shadmanrohan/ bencoref initiative that created a large dataset with entity, event and relation annotations. this project revealed some major complexities behind creating such dataset. some of the significant challenges reported by the annotators include the coreference of generic entities, use of metonymy, characterization of geo-political entity, distinguishing certain complex relations, and recognizing implicit vs. explicit relations. since then coreference resolution, anaphoric & cataphoric relation identification, event reference detection has been studied widely. as a result, large datasets like ace @xcite , ontonotes @xcite , wikicoref (ghaddar and langlais, 2016), and litbank @xcite were made public. some datasets, like ace @xcite , and ontonotes, expanded this task beyond english to include more languages, like arabic, and chinese. this coreference resolution task has shown potential in improving many downstream nlp tasks like machine translation @xcite , literary analysis @xcite , question anwering @xcite ), text summarization @xcite , etc. however, bengali, despite being a popular lanaguage, has seen very little work is this direction due to lack of public datasets. figure 1 shows a sample from our dataset with each color representing an unique entity. the main contributions of this work are: ## challenges in bengali one of the main challenges we faced was the absence of preexisting coreference annotation guidelines tailored for the bengali language. to overcome this obstacle, we adapted the ontonotes coreference annotation guideline to suit our objectives. this highlighted several distinctive linguistic characteristics of bengali, such as zero anaphora, non-anaphoric pronouns, and case-marking, that needs to be carefully considered when preforming co-reference annotation in bengali. each of this is discussed with more details and examples in figure 9 in the appendix. moreover, we discovered that existing annotation software is ill-equipped to manage bengali text, occasionally leading to inaccurate rendering and unstable character display. this underscores the importance of advancing normalization techniques and standardization of bengali digital representation. ## data domain description the bengali language can be braoadly categorized into two primary literary dialects, namely "shadhubhasha" and "choltibhasha." "shadhubhasha" was commonly used by bangla writers and individuals in the 19th and early 20th centuries, while "choltibhasha" is currently the more prevalent and colloquial dialect. this dataset contains both domains of bengali text, with story and novel texts sourced from copyright-free books of the 19th and 20th centuries, and biography and descriptive texts obtained from modern sources, primarily in "choltibhasha." a brief description of each domain is given below: ## domain specific coreference behaviour characterization in this section, some statistics is presented to better understand the coreference phenomenon across each domain. each coreference cluster may refer to different type of entities, like an object, people, location or event. an arbitrary design choice was made to not explicitly mark the type of entity. we start by analyzing the mean and standard deviation between mentions across the domains. the majority of texts in bencoref belong to the stories domain, while the biography domain has the smallest contribution. the distribution of mentions, clusters, and tokens across the categories in bencoref is presented in figure 4 . figure 2 depicts the distribution of cluster size across each domain. the cluster size refers to the total number of mentions in each coreference chain. it is worth noting that singletons were not annotated in bencoref. the story domain has the highest number of coreference chains with two mentions only. since the story domain contributes the most data to the dataset, this may be a contributing factor to its high frequency in each cluster size. besides story, the descriptive domain also seems to have more larger coreference chains. figure 3 compares the spread of coreference chains in each domain, where the spread refers to the token-level distance between the beginning and end of a coreference chain. a general trend can be observed that as the size of the coreference chain increases, its corresponding frequency decreases in each domain. an additional "index.csv" file is included with in the dataset, which serves as an index to all the documents included, organized by title and author. a partial view of this file is presented in appendix figure 6 . ## methodology we used bnwikisource foot_0 and banglapedia @xcite as sources of copyright-free bengali text for our dataset. banglapedia was used for biographies and wiki-like descriptive texts. the dataset creation process is discussed in detail in the following paragraphs. ## experiments we took an end-to-end neural network based modeling approach. the following section discusses the algorithm, followed by the experimental setup, evaluation strategy and analysis of results. we used the 300-dimensional fasttext and glove embeddings @xcite as words representations. to generate contexual representations the embeddings were passed through a bi-directional lstm @xcite for some experiments and a variation of the popular transformer-based @xcite pretrained model, bert @xcite , for other experiments. for the task of coreference resolution, the contextual representations from these base models were passed on to a span ranking model-head, originally proposed in @xcite . for the crosslingual experiment, a multilingual bert was finetuned on the ontonotes dataset. for hyperparameter optimization, we tuned the maximum number of words in a span(s), maximum number of antecedents per span(a), and coref layer depth(cl). ## conclusion this paper presented bencoref, the first publicly available dataset of coreference annotations in bengali. the creation process and annotation guidelines were described in detail to facilitate future work in this area. we then used the dataset to develop an end-to-end coreference resolution system and reported its performance across different domains. our findings indicate that a lower mean and standard deviation of token-distance between mentions may lead to better results, but further experiments on other datasets are needed to confirm this hypothesis. we also observed a higher tendency for breakage in longer coreference chains. our zero-shot cross-lingual experiment demonstrated that coreference knowledge does not easily transfer through multilingual training, highlighting the importance of language-specific datasets. while some studies @xcite have reported success in developing projection techniques to improve cross-lingual coreference resolution, further research is required to explore this area. figure 8: bert-base model's prediction on a biography document