Incorporating Subjectivity into Gendered Ambiguous Pronoun ( GAP ) Resolution using Style Transfer (2022)

Abstract: The GAP dataset is a Wikipedia-based evaluation dataset for gender bias detection in coreference resolution, containing mostly objective sentences. Since subjectivity is ubiquitous in our daily texts, it becomes necessary to evaluate models for both subjective and objective instances. In this work, we present a new evaluation dataset for gender bias in coreference resolution, GAP-Subjective, which increases the coverage of the original GAP dataset by including subjective sentences. We outline the methodology used to create this dataset. Firstly, we detect objective sentences and transfer them into their subjective variants using a sequence-to-sequence model. Secondly, we outline the thresholding techniques based on fluency and content preservation to maintain the quality of the sentences. Thirdly, we perform automated and human-based analysis of the style transfer and infer that the transferred sentences are of high quality. Finally, we benchmark both GAP and GAP-Subjective datasets using a BERT-based model and analyze its predictive performance and gender bias.

## introduction in natural language, subjectivity refers to the aspects of communication used to express opinions, evaluations, and speculations, often influenced by one's emotional state and viewpoints. it is introduced in natural language by using inflammatory words and phrases, casting doubt over a fact, or presupposing the truth. writers and editors of texts like newspapers, journals, and textbooks try to avoid subjectivity, yet it is pervasive in these texts. hence, many nlp applications, including information retrieval, question answering systems, recommender systems, and coreference resolution, would benefit from being able to model subjectivity in natural language @xcite . ## conclusion in this work, we analyzed the addition of the subjectivity attribute in gap, a widely used evaluation corpus for the detection of gender bias in coreference resolution. we utilized sentence-level supervised style transfer using sequence-to-sequence models to transfer the objective sentences in gap to their subjective variants. we outlined the efficacy of our proposed style transfer approach using suitable metrics for content preservation and fluency and a human evaluation of the transferred sentences. we proposed a new evaluation corpus, gap-subjective, which consists of the reconstructed texts along with their new entity offsets. we benchmarked and analyzed the predictive performance and gender bias of bert-based models in both gap and gap-subjective. future work may include increasing coverage of objective-heavy datasets for other downstream tasks and increas-ing the coverage of gap using other attributes.