pwc-archive/papers-with-abstracts huggingface dataset:
Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019
Authors: Kellie Webster, Marta R. Costa-juss{\`a}, Christian Hardmeier, Will Radford
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

The 1st ACL workshop on Gender Bias in Natural Language Processing included a shared task on gendered ambiguous pronoun (GAP) resolution. This task was based on the coreference challenge defined in Webster et al. (2018), designed to benchmark the ability of systems to resolve pronouns in real-world contexts in a gender-fair way. 263 teams competed via a Kaggle competition, with the winning system achieving logloss of 0.13667 and near gender parity. We review the approaches of eleven systems with accepted description papers, noting their effective use of BERT (Devlin et al., 2018), both via fine-tuning and for feature extraction, as well as ensembling.
Proposed Taxonomy for Gender Bias in Text; A Filtering Methodology for the Gender Generalization Subtype
Authors: Yasmeen Hitti, Eunbee Jang, Ines Moreno, Carolyne Pelletier
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Sentence

The purpose of this paper is to present an empirical study on gender bias in text. Current research in this field is focused on detecting and correcting for gender bias in existing machine learning models rather than approaching the issue at the dataset level. The underlying motivation is to create a dataset which could enable machines to learn to differentiate bias writing from non-bias writing. A taxonomy is proposed for structural and contextual gender biases which can manifest themselves in text. A methodology is proposed to fetch one type of structural gender bias, Gender Generalization. We explore the IMDB movie review dataset and 9 different corpora from Project Gutenberg. By filtering out irrelevant sentences, the remaining pool of candidate sentences are sent for human validation. A total of 6123 judgments are made on 1627 sentences and after a quality check on randomly selected sentences we obtain an accuracy of 75{\%}. Out of the 1627 sentences, 808 sentence were labeled as Gender Generalizations. The inter-rater reliability amongst labelers was of 61.14{\%}.
Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis
Authors: Scott Friedman, Sonja Schmer-Galunder, Anthony Chen, Jeffrey Rye
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Cultural Vocal Bursts Intensity Prediction, Word Embeddings

Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, rooting from inherent bias in their training text. These biases are often sub-optimal and recent work poses methods to rectify them; however, these biases may shed light on actual racial or gender gaps in the culture(s) that produced the training text, thereby helping us understand cultural context through big data. This paper presents an approach for quantifying gender bias in word embeddings, and then using them to characterize statistical gender gaps in education, politics, economics, and health. We validate these metrics on 2018 Twitter data spanning 51 U.S. regions and 99 countries. We correlate state and country word embedding biases with 18 international and 5 U.S.-based statistical gender gaps, characterizing regularities and predictive strength.
Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories
Authors: Kaytlin Chaloner, Alfredo Maldonado
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Bias Detection, Clustering, Gender Bias Detection, Two-sample testing, Word Embeddings

Prior work has shown that word embeddings capture human stereotypes, including gender bias. However, there is a lack of studies testing the presence of specific gender bias categories in word embeddings across diverse domains. This paper aims to fill this gap by applying the WEAT bias detection method to four sets of word embeddings trained on corpora from four different domains: news, social networking, biomedical and a gender-balanced corpus extracted from Wikipedia (GAP). We find that some domains are definitely more prone to gender bias than others, and that the categories of gender bias present also vary for each set of word embeddings. We detect some gender bias in GAP. We also propose a simple but novel method for discovering new bias categories by clustering word embeddings. We validate this method through WEAT{'}s hypothesis testing mechanism and find it useful for expanding the relatively small set of well-known gender bias word categories commonly used in the literature.
Filling Gender \& Number Gaps in Neural Machine Translation with Black-box Context Injection
Authors: Amit Moryossef, Roee Aharoni, Yoav Goldberg
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

When translating from a language that does not morphologically mark information such as gender and number into a language that does, translation systems must {``}guess{''} this missing information, often leading to incorrect translations in the given context. We propose a black-box approach for injecting the missing information to a pre-trained neural machine translation system, allowing to control the morphological variations in the generated translations without changing the underlying model or training data. We evaluate our method on an English to Hebrew translation task, and show that it is effective in injecting the gender and number information and that supplying the correct information improves the translation accuracy in up to 2.3 BLEU on a female-speaker test set for a state-of-the-art online black-box system. Finally, we perform a fine-grained syntactic analysis of the generated translations that shows the effectiveness of our method.
The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations
Authors: Jo{\~a}o Sedoc, Lyle Ungar
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Word Embeddings

Systemic bias in word embeddings has been widely reported and studied, and efforts made to debias them; however, new contextualized embeddings such as ELMo and BERT are only now being similarly studied. Standard debiasing methods require heterogeneous lists of target words to identify the {``}bias subspace{''}. We show show that using new contextualized word embeddings in conceptor debiasing allows us to more accurately debias word embeddings by breaking target word lists into more homogeneous subsets and then combining ({''}Or{'}ing{''}) the debiasing conceptors of the different subsets.
BERT Masked Language Modeling for Co-reference Resolution
Authors: Felipe Alfaro, Marta R. Costa-juss{\`a}, Jos{\'e} A. R. Fonollosa
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: General Classification, Language Modeling, Language Modelling, Masked Language Modeling

This paper explains the TALP-UPC participation for the Gendered Pronoun Resolution shared-task of the 1st ACL Workshop on Gender Bias for Natural Language Processing. We have implemented two models for mask language modeling using pre-trained BERT adjusted to work for a classification problem. The proposed solutions are based on the word probabilities of the original BERT model, but using common English names to replace the original test names.
Transfer Learning from Pre-trained BERT for Pronoun Resolution
Authors: Xingce Bao, Qianqian Qiao
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: General Classification, Transfer Learning

The paper describes the submission of the team {``}We used bert!{''} to the shared task Gendered Pronoun Resolution (Pair pronouns to their correct entities). Our final submission model based on the fine-tuned BERT (Bidirectional Encoder Representations from Transformers) ranks 14th among 838 teams with a multi-class logarithmic loss of 0.208. In this work, contribution of transfer learning technique to pronoun resolution systems is investigated and the gender bias contained in classification models is evaluated.
Fill the GAP: Exploiting BERT for Pronoun Resolution
Authors: Kai-Chou Yang, Timothy Niven, Tzu Hsuan Chou, Hung-Yu Kao
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Coreference Resolution, Data Augmentation, Natural Language Inference

In this paper, we describe our entry in the gendered pronoun resolution competition which achieved fourth place without data augmentation. Our method is an ensemble system of BERTs which resolves co-reference in an interaction space. We report four insights from our work: BERT{'}s representations involve significant redundancy; modeling interaction effects similar to natural language inference models is useful for this task; there is an optimal BERT layer to extract representations for pronoun resolution; and the difference between the attention weights from the pronoun to the candidate entities was highly correlated with the correct label, with interesting implications for future work.
On GAP Coreference Resolution Shared Task: Insights from the 3rd Place Solution
Authors: Artem Abzaliev
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: coreference-resolution, Coreference Resolution

This paper presents the 3rd-place-winning solution to the GAP coreference resolution shared task. The approach adopted consists of two key components: fine-tuning the BERT language representation model (Devlin et al., 2018) and the usage of external datasets during the training process. The model uses hidden states from the intermediate BERT layers instead of the last layer. The resulting system almost eliminates the difference in log loss per gender during the cross-validation, while providing high performance.
Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling
Authors: S Attree, eep
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: coreference-resolution, Coreference Resolution

This paper presents a strong set of results for resolving gendered ambiguous pronouns on the Gendered Ambiguous Pronouns shared task. The model presented here draws upon the strengths of state-of-the-art language and coreference resolution models, and introduces a novel evidence-based deep learning architecture. Injecting evidence from the coreference models compliments the base architecture, and analysis shows that the model is not hindered by their weaknesses, specifically gender bias. The modularity and simplicity of the architecture make it very easy to extend for further improvement and applicable to other NLP problems. Evaluation on GAP test data results in a state-of-the-art performance at 92.5{\%} F1 (gender bias of 0.97), edging closer to the human performance of 96.6{\%}. The end-to-end solution presented here placed 1st in the Kaggle competition, winning by a significant lead.
Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques
Authors: Joel Escud{\'e} Font, Marta R. Costa-juss{\`a}
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Fairness, Machine Translation, Translation, Word Embeddings

Neural machine translation has significantly pushed forward the quality of the field. However, there are remaining big issues with the output translations and one of them is fairness. Neural models are trained on large text corpora which contain biases and stereotypes. As a consequence, models inherit these social biases. Recent methods have shown results in reducing gender bias in other natural language processing tools such as word embeddings. We take advantage of the fact that word embeddings are used in neural machine translation to propose a method to equalize gender biases in neural machine translation using these representations. Specifically, we propose, experiment and analyze the integration of two debiasing techniques over GloVe embeddings in the Transformer translation architecture. We evaluate our proposed system on the WMT English-Spanish benchmark task, showing gains up to one BLEU point. As for the gender bias evaluation, we generate a test set of occupations and we show that our proposed system learns to equalize existing biases from the baseline system.
Automatic Gender Identification and Reinflection in Arabic
Authors: Nizar Habash, Houda Bouamor, Christine Chung
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

The impressive progress in many Natural Language Processing (NLP) applications has increased the awareness of some of the biases these NLP systems have with regards to gender identities. In this paper, we propose an approach to extend biased single-output gender-blind NLP systems with gender-specific alternative reinflections. We focus on Arabic, a gender-marking morphologically rich language, in the context of machine translation (MT) from English, and for first-person-singular constructions only. Our contributions are the development of a system-independent gender-awareness wrapper, and the building of a corpus for training and evaluating first-person-singular gender identification and reinflection in Arabic. Our results successfully demonstrate the viability of this approach with 8{\%} relative increase in Bleu score for first-person-singular feminine, and 5.3{\%} comparable increase for first-person-singular masculine on top of a state-of-the-art gender-blind MT system on a held-out test set.
Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling
Authors: Farhana Ferdousi Liza, Marek Grzes
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We analyse Recurrent Neural Networks (RNNs) to understand the significance of multiple LSTM layers. We argue that the Weighted Finite-state Automata (WFA) trained using a spectral learning algorithm are helpful to analyse RNNs. Our results suggest that multiple LSTM layers in RNNs help learning distributed hidden states, but have a smaller impact on the ability to learn long-term dependencies. The analysis is based on the empirical results, however relevant theory (whenever possible) was discussed to justify and support our conclusions.
Crowdsourced Hedge Term Disambiguation
Authors: Morgan Ulinski, Julia Hirschberg
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Word Sense Disambiguation

We address the issue of acquiring quality annotations of hedging words and phrases, linguistic phenomenona in which words, sounds, or other constructions are used to express ambiguity or uncertainty. Due to the limited availability of existing corpora annotated for hedging, linguists and other language scientists have been constrained as to the extent they can study this phenomenon. In this paper, we introduce a new method of acquiring hedging annotations via crowdsourcing, based on reformulating the task of labeling hedges as a simple word sense disambiguation task. We also introduce a new hedging corpus we have constructed by applying this method, a collection of forum posts annotated using Amazon Mechanical Turk. We found that the crowdsourced judgments we obtained had an inter-annotator agreement of 92.89{\%} (Fleiss{'} Kappa=0.751) and, when comparing a subset of these annotations to an expert-annotated gold standard, an accuracy of 96.65{\%}.
Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task
Authors: Frances Yung, Vera Demberg, Merel Scholman
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Relation, Vocal Bursts Valence Prediction

The perspective of being able to crowd-source coherence relations bears the promise of acquiring annotations for new texts quickly, which could then increase the size and variety of discourse-annotated corpora. It would also open the avenue to answering new research questions: Collecting annotations from a larger number of individuals per instance would allow to investigate the distribution of inferred relations, and to study individual differences in coherence relation interpretation. However, annotating coherence relations with untrained workers is not trivial. We here propose a novel two-step annotation procedure, which extends an earlier method by Scholman and Demberg (2017a). In our approach, coherence relation labels are inferred from connectives that workers insert into the text. We show that the proposed method leads to replicable coherence annotations, and analyse the agreement between the obtained relation labels and annotations from PDTB and RSTDT on the same texts.
Annotating and analyzing the interactions between meaning relations
Authors: Darina Gold, Venelin Kovatchev, Torsten Zesch
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Natural Language Inference, Semantic Similarity, Semantic Textual Similarity, Sentence, Specificity

Pairs of sentences, phrases, or other text pieces can hold semantic relations such as paraphrasing, textual entailment, contradiction, specificity, and semantic similarity. These relations are usually studied in isolation and no dataset exists where they can be compared empirically. Here we present a corpus annotated with these relations and the analysis of these results. The corpus contains 520 sentence pairs, annotated with these relations. We measure the annotation reliability of each individual relation and we examine their interactions and correlations. Among the unexpected results revealed by our analysis is that the traditionally considered direct relationship between paraphrasing and bi-directional entailment does not hold in our data.
CCGweb: a New Annotation Tool and a First Quadrilingual CCG Treebank
Authors: Kilian Evang, Lasha Abzianidze, Johan Bos
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We present the first open-source graphical annotation tool for combinatory categorial grammar (CCG), and the first set of detailed guidelines for syntactic annotation with CCG, for four languages: English, German, Italian, and Dutch. We also release a parallel pilot CCG treebank based on these guidelines, with 4x100 adjudicated sentences, 10K single-annotator fully corrected sentences, and 82K single-annotator partially corrected sentences.
The making of the Litkey Corpus, a richly annotated longitudinal corpus of German texts written by primary school children
Authors: Ronja Laarmann-Quante, Stefanie Dipper, Eva Belke
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Language Acquisition, POS

To date, corpus and computational linguistic work on written language acquisition has mostly dealt with second language learners who have usually already mastered orthography acquisition in their first language. In this paper, we present the Litkey Corpus, a richly-annotated longitudinal corpus of written texts produced by primary school children in Germany from grades 2 to 4. The paper focuses on the (semi-)automatic annotation procedure at various linguistic levels, which include POS tags, features of the word-internal structure (phonemes, syllables, morphemes) and key orthographic features of the target words as well as a categorization of spelling errors. Comprehensive evaluations show that high accuracy was achieved on all levels, making the Litkey Corpus a useful resource for corpus-based research on literacy acquisition of German primary school children and for developing NLP tools for educational purposes. The corpus is freely available under https://www.linguistics.rub.de/litkeycorpus/.
Tagging modality in Oceanic languages of Melanesia
Authors: Annika Tjuka, Lena Wei{\ss}mann, Kilu von Prince
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: counterfactual, TAG

Primary data from small, low-resource languages of Oceania have only recently become available through language documentation. In our study, we explore corpus data of five Oceanic languages of Melanesia which are known to be mood-prominent (in the sense of Bhat, 1999). In order to find out more about tense, aspect, modality, and polarity, we tagged these categories in a subset of our corpora. For the category of modality, we developed a novel tag set (MelaTAMP, 2017), which categorizes clauses into factual, possible, and counterfactual. Based on an analysis of the inter-annotator consistency, we argue that our tag set for the modal domain is efficient for our subject languages and might be useful for other languages and purposes.
Harmonizing Different Lemmatization Strategies for Building a Knowledge Base of Linguistic Resources for Latin
Authors: Francesco Mambrini, Marco Passarotti
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: LEMMA, Lemmatization

The interoperability between lemmatized corpora of Latin and other resources that use the lemma as indexing key is hampered by the multiple lemmatization strategies that different projects adopt. In this paper we discuss how we tackle the challenges raised by harmonizing different lemmatization criteria in the context of a project that aims to connect linguistic resources for Latin using the Linked Data paradigm. The paper introduces the architecture supporting an open-ended, lemma-based Knowledge Base, built to make textual and lexical resources for Latin interoperable. Particularly, the paper describes the inclusion into the Knowledge Base of its lexical basis, of a word formation lexicon and of a lemmatized and syntactically annotated corpus.
Assessing Back-Translation as a Corpus Generation Strategy for non-English Tasks: A Study in Reading Comprehension and Word Sense Disambiguation
Authors: Fabricio Monsalve, Kervy Rivas Rojas, Marco Antonio Sobrevilla Cabezudo, Arturo Oncevay
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Reading Comprehension, Translation, Word Sense Disambiguation

Corpora curated by experts have sustained Natural Language Processing mainly in English, but the expensiveness of corpora creation is a barrier for the development in further languages. Thus, we propose a corpus generation strategy that only requires a machine translation system between English and the target language in both directions, where we filter the best translations by computing automatic translation metrics and the task performance score. By studying Reading Comprehension in Spanish and Word Sense Disambiguation in Portuguese, we identified that a more quality-oriented metric has high potential in the corpora selection without degrading the task performance. We conclude that it is possible to systematise the building of quality corpora using machine translation and automatic metrics, besides some prior effort to clean and process the data.
A Framework for Annotating `Related Works' to Support Feedback to Novice Writers
Authors: Arlene Casey, Bonnie Webber, Dorota Glowacka
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Understanding what is expected of academic writing can be difficult for novice writers to assimilate, and recent years have seen several automated tools become available to support academic writing. Our work presents a framework for annotating features of the Related Work section of academic writing, that supports writer feedback.
An Online Annotation Assistant for Argument Schemes
Authors: John Lawrence, Jacky Visser, Chris Reed
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Understanding the inferential principles underpinning an argument is essential to the proper interpretation and evaluation of persuasive discourse. Argument schemes capture the conventional patterns of reasoning appealed to in persuasion. The empirical study of these patterns relies on the availability of data about the actual use of argumentation in communicative practice. Annotated corpora of argument schemes, however, are scarce, small, and unrepresentative. Aiming to address this issue, we present one step in the development of improved datasets by integrating the Argument Scheme Key {--} a novel annotation method based on one of the most popular typologies of argument schemes {--} into the widely used OVA software for argument analysis.