pwc-archive/papers-with-abstracts huggingface dataset:
DEFT 2021: Évaluation automatique de réponses courtes, une approche basée sur la sélection de traits lexicaux et augmentation de données (DEFT 2021 : Automatic short answer grading, a lexical features selection and data augmentation based approach)
Authors: Timothée Poulain, Victor Connes
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6
Tasks: automatic short answer grading, Data Augmentation

Cet article présente la participation de l’équipe Proofreaders du LS2N au DÉfi Fouille de Textes 2021 (DEFT 2021). La tâche proposée consiste en la poursuite automatique de l’évaluation de réponses courtes d’étudiants (EAQRC) à partir de quelques réponses déjà corrigées par l’enseignant pour chaque énoncé. Une étude comparative de différents traits lexicaux, ainsi qu’une augmentation artificielle de données et de différents modèles de régression pour la notation des réponses courtes est réalisée. Les méthodes sont évaluées en termes de précision, d’erreur quadratique moyenne et de score de corrélation de Spearman. Notre erreur quadratique moyenne varie entre 0.090 et 0.101 et notre précision entre 0.147 et 0.17. Le code source est disponible à l’adresse suivante : https://github.com/poulain-tim/DEFT

_2021
DOING@DEFT : utilisation de lexiques pour une classification efficace de cas cliniques (In this paper, we present our participation to the DEFT 2021 task 1)
Authors: Nicolas Hiot, Anne-Lyse Minard, Flora Badin
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6

Nous présentons dans cet article notre participation à la tâche 1 de la campagne d’évaluation francophone DEFT 2021, sur l’identification du profil clinique du patient. Nous proposons une méthode évolutive et efficace en temps et en ressources pour la classification de documents médicaux pouvant être facilement adaptée à d’autres domaines de recherche. Notre système a obtenu les meilleures performances sur cette tâche avec une F-mesure de 0,814.
Identification de profil clinique du patient: Une approche de classification de séquences utilisant des modèles de langage français contextualisés (Identification of patient clinical profiles : A sequence classification approach using contextualised French language models )
Authors: Aidan Mannion, Thierry Chevalier, Didier Schwab, Lorraine Goeuriot
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6
Tasks: Language Modelling

Cet article présente un résumé de notre soumission pour Tâche 1 de DEFT 2021. Cette tâche consiste à identifier le profil clinique d’un patient à partir d’une description textuelle de son cas clinique en identifiant les types de pathologie mentionnés dans le texte. Ce travail étudie des approches de classification de texte utilisant des plongements de mots contextualisés en français. À partir d’une base de référence d’un modèle constitué pour la compréhension générale de la langue française, nous utilisons des modèles pré-entraînés avec masked language modelling et affinés à la tâche d’identification, en utilisant un corpus externe de textes cliniques fourni par SOS Médecins, pour développer des ensembles de classifieurs binaires associant les textes cliniques à des catégories de pathologies.
Mesure de similarité textuelle pour l’évaluation automatique de copies d’étudiants (Textual similarity measurement for automatic evaluation of students’ answers)
Authors: Xiaoou Wang, Xingyu Liu, Yimei Yue
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6
Tasks: Sentence, Sentence Embeddings

Cet article décrit la participation de l’équipe Nantalco à la tâche 2 du Défi Fouille de Textes 2021 (DEFT) : évaluation automatique de copies d’après une référence existante. Nous avons utilisé principalement des traits basés sur la similarité cosinus des deux vecteurs représentant la similarité textuelle entre des réponses d’étudiant et la référence. Plusieurs types de vecteurs ont été utilisés (vecteur d’occurrences de mots, vecteur tf-idf, embeddings non contextualisés de fastText, embeddings contextualisés de CamemBERT et enfin Sentence Embeddings Multilingues ajustés sur des corpus multilingues). La meilleure performance du concours sur cette tâche a été de 0.682 (précision) et celle de notre équipe 0.639. Cette performance a été obtenue avec les Sentence Embeddings Multilingues alors que celle des embeddings non ajustés ne s’est élevée qu’à 0.55, suggérant que de récents modèles de langues pré-entraînés doivent être fine-tunés afin d’avoir des embeddings adéquats au niveau phrastique.
Participation d’EDF R&D à DEFT 2021 (EDF R&D Participation to DEFT 2021)
Authors: Philippe Suignard, Alexandra Benamar, Nazim Messous, Clément Christophe, Marie Jubault, Meryl Bothua
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6

Ce papier présente la participation d’EDF R&D à la campagne d’évaluation DEFT 2021. Notre équipe a participé aux deux dernières tâches proposées (T2 et T3), deux tâches sur le calcul de similarité sémantique entre textes courts, et s’est classée 1ère sur ces deux tâches. Cette édition proposait deux nouvelles tâches pour l’évaluation automatique de réponses d’étudiants à des questions d’enseignants. Le corpus se composait d’une centaine d’énoncés en informatique avec la correction de l’enseignant et les réponses d’une cinquantaine d’étudiants en moyenne par question, sur 2 ans. La tâche 2 consistait à évaluer les réponses des étudiants en prenant pour référence la correction produite par l’enseignant et la tâche 3 à évaluer les réponses d’étudiants à partir d’un ensemble composé d’un énoncé et de plusieurs réponses d’étudiants déjà corrigées par l’enseignant.e.
Participation de Berger-Levrault (BL.Research) à DEFT 2021 : de l’apprentissage des seuils de validation à la classification multi-labels de documents (Berger-Levrault (BL)
Authors: Mokhtar Boumedyen Billami, Lina Nicolaieff, Camille Gosset, Christophe Bortolaso
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6

Cet article présente notre participation à l’édition 2021 du DÉfi Fouille de Textes (DEFT) et plus précisément à la première tâche liée à l’identification du profil clinique du patient. Cette tâche consiste à sélectionner, pour un document décrivant l’état d’un patient, les différents types de maladies rencontrées correspondant aux entrées génériques des chapitres du MeSH (Medical Subject Headings). Dans notre travail, nous nous sommes intéressés aux questions suivantes : (1) Comment améliorer les représentations vectorielles de documents, voire de classes ? (2) Comment apprendre des seuils de validation de classes ? Et (3) Une approche combinant apprentissage supervisé et similarité sémantique peut-elle apporter une meilleure performance à un système de classification multi-labels ?
QUEER@DEFT2021 : Identification du Profil Clinique de Patients et Notation Automatique de Copies d’Étudiants (QUEER@DEFT2021 : Patients Clinical Profile Identification and Automatic Student Grading )
Authors: Yoann Dupont, Carlos-Emiliano González-Gallardo, Gaël Lejeune, Alice Millour, Jean-Baptiste Tanguy
Date: 2021-06-01 00:00:00
Proceeding: JEP/TALN/RECITAL 2021 6

Nous présentons dans cet article notre contribution aux 3 tâches de la campagne d’évaluation du défi Fouille de Texte 2021. Dans la tâche d’identification de de profil clinique (tâche 1) nous présentons une méthode de recherche d’information basé sur un index dérivé du MeSH. Pour la tâche de notation automatique à partir d’une correction (tâche 2), nous avons expérimenté une méthode de similarité de vecteurs de chaînes de caractères. Pour la tâche de notation à partir de copies déjà notées (tâche 3) nous avons entraîné un réseau de neurones LSTM.
Unmasking Contextual Stereotypes: Measuring and Mitigating BERT’s Gender Bias
Authors: Marion Bartl, Malvina Nissim, Albert Gatt
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: counterfactual, Word Embeddings

Contextualized word embeddings have been replacing standard embeddings as the representational knowledge source of choice in NLP systems. Since a variety of biases have previously been found in standard word embeddings, it is crucial to assess biases encoded in their replacements as well. Focusing on BERT (Devlin et al., 2018), we measure gender bias by studying associations between gender-denoting target words and names of professions in English and German, comparing the findings with real-world workforce statistics. We mitigate bias by fine-tuning BERT on the GAP corpus (Webster et al., 2018), after applying Counterfactual Data Substitution (CDS) (Maudslay et al., 2019). We show that our method of measuring bias is appropriate for languages such as English, but not for languages with a rich morphology and gender-marking, such as German. Our results highlight the importance of investigating bias and mitigation techniques cross-linguistically,especially in view of the current emphasis on large-scale, multilingual language models.
Interdependencies of Gender and Race in Contextualized Word Embeddings
Authors: May Jiang, Christiane Fellbaum
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Word Embeddings

Recent years have seen a surge in research on the biases in word embeddings with respect to gender and, to a lesser extent, race. Few of these studies, however, have given attention to the critical intersection of race and gender. In this case study, we analyze the dimensions of gender and race in contextualized word embeddings of given names, taken from BERT, and investigate the nature and nuance of their interaction. We find that these demographic axes, though typically treated as physically and conceptually separate, are in fact interdependent and thus inadvisable to consider in isolation. Further, we show that demographic dimensions predicated on default settings in language, such as in pronouns, may risk rendering groups with multiple marginalized identities invisible. We conclude by discussing the importance and implications of intersectionality for future studies on bias and debiasing in NLP.
Fine-tuning Neural Machine Translation on Gender-Balanced Datasets
Authors: Marta R. Costa-jussà, Adrià de Jorge
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Machine Translation, Translation

Misrepresentation of certain communities in datasets is causing big disruptions in artificial intelligence applications. In this paper, we propose using an automatically extracted gender-balanced dataset parallel corpus from Wikipedia. This balanced set is used to perform fine-tuning techniques from a bigger model trained on unbalanced datasets to mitigate gender biases in neural machine translation.
Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It
Authors: Danielle Saunders, Rosie Sallis, Bill Byrne
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Machine Translation, NMT, Sentence, Translation

Neural Machine Translation (NMT) has been shown to struggle with grammatical gender that is dependent on the gender of human referents, which can cause gender bias effects. Many existing approaches to this problem seek to control gender inflection in the target language by explicitly or implicitly adding a gender feature to the source sentence, usually at the sentence level. In this paper we propose schemes for incorporating explicit word-level gender inflection tags into NMT. We explore the potential of this gender-inflection controlled translation when the gender feature can be determined from a human reference, or when a test sentence can be automatically gender-tagged, assessing on English-to-Spanish and English-to-German translation. We find that simple existing approaches can over-generalize a gender-feature to multiple entities in a sentence, and suggest effective alternatives in the form of tagged coreference adaptation data. We also propose an extension to assess translations of gender-neutral entities from English given a corresponding linguistic convention, such as a non-binary inflection, in the target language.
Can Existing Methods Debias Languages Other than English? First Attempt to Analyze and Mitigate Japanese Word Embeddings
Authors: Masashi Takeshita, Yuki Katsumata, Rafal Rzepka, Kenji Araki
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Word Embeddings

It is known that word embeddings exhibit biases inherited from the corpus, and those biases reflect social stereotypes. Recently, many studies have been conducted to analyze and mitigate biases in word embeddings. Unsupervised Bias Enumeration (UBE) (Swinger et al., 2019) is one of approach to analyze biases for English, and Hard Debias (Bolukbasi et al., 2016) is the common technique to mitigate gender bias. These methods focused on English, or, in smaller extent, on Indo-European languages. However, it is not clear whether these methods can be generalized to other languages. In this paper, we apply these analyzing and mitigating methods, UBE and Hard Debias, to Japanese word embeddings. Additionally, we examine whether these methods can be used for Japanese. We experimentally show that UBE and Hard Debias cannot be sufficiently adapted to Japanese embeddings.
Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas
Authors: Amanda Cercas Curry, Judy Robertson, Verena Rieser
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Multiple-choice

Conversational voice assistants are rapidly developing from purely transactional systems to social companions with “personality”. UNESCO recently stated that the female and submissive personality of current digital assistants gives rise for concern as it reinforces gender stereotypes. In this work, we present results from a participatory design workshop, where we invite people to submit their preferences for a what their ideal persona might look like, both in drawings as well as in a multiple choice questionnaire. We find no clear consensus which suggests that one possible solution is to let people configure/personalise their assistants. We then outline a multi-disciplinary project of how we plan to address the complex question of gender and stereotyping in digital assistants.
Semi-Supervised Topic Modeling for Gender Bias Discovery in English and Swedish
Authors: Hannah Devinney, Jenny Björklund, Henrik Björklund
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Articles, Topic Models

Gender bias has been identified in many models for Natural Language Processing, stemming from implicit biases in the text corpora used to train the models. Such corpora are too large to closely analyze for biased or stereotypical content. Thus, we argue for a combination of quantitative and qualitative methods, where the quantitative part produces a view of the data of a size suitable for qualitative analysis. We investigate the usefulness of semi-supervised topic modeling for the detection and analysis of gender bias in three corpora (mainstream news articles in English and Swedish, and LGBTQ+ web content in English). We compare differences in topic models for three gender categories (masculine, feminine, and nonbinary or neutral) in each corpus. We find that in all corpora, genders are treated differently and that these differences tend to correspond to hegemonic ideas of gender.
Gender and sentiment, critics and authors: a dataset of Norwegian book reviews
Authors: Samia Touileb, Lilja Øvrelid, Erik Velldal
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12

Gender bias in models and datasets is widely studied in NLP. The focus has usually been on analysing how females and males express themselves, or how females and males are described. However, a less studied aspect is the combination of these two perspectives, how female and male describe the same or opposite gender. In this paper, we present a new gender annotated sentiment dataset of critics reviewing the works of female and male authors. We investigate if this newly annotated dataset contains differences in how the works of male and female authors are critiqued, in particular in terms of positive and negative sentiment. We also explore the differences in how this is done by male and female critics. We show that there are differences in how critics assess the works of authors of the same or opposite gender. For example, male critics rate crime novels written by females, and romantic and sentimental works written by males, more negatively.
Gender-Aware Reinflection using Linguistically Enhanced Neural Models
Authors: Bashar Alhafni, Nizar Habash, Houda Bouamor
Date: 2020-12-01 00:00:00
Proceeding: GeBNLP (COLING) 2020 12
Tasks: Decoder, Grammatical Error Correction, Sentence

In this paper, we present an approach for sentence-level gender reinflection using linguistically enhanced sequence-to-sequence models. Our system takes an Arabic sentence and a given target gender as input and generates a gender-reinflected sentence based on the target gender. We formulate the problem as a user-aware grammatical error correction task and build an encoder-decoder architecture to jointly model reinflection for both masculine and feminine grammatical genders. We also show that adding linguistic features to our model leads to better reinflection results. The results on a blind test set using our best system show improvements over previous work, with a 3.6% absolute increase in M2 F0.5.
Impact of COVID-19 in Natural Language Processing Publications: a Disaggregated Study in Gender, Contribution and Experience
Authors: Christine Basta, Marta R. Costa-Jussa
Date: 2021-04-01 00:00:00
Proceeding: EACL (LTEDI) 2021 4
Tasks: Position

This study sheds light on the effects of COVID-19 in the particular field of Computational Linguistics and Natural Language Processing within Artificial Intelligence. We provide an inter-sectional study on gender, contribution, and experience that considers one school year (from August 2019 to August 2020) as a pandemic year. August is included twice for the purpose of an inter-annual comparison. While the trend in publications increased with the crisis, the results show that the ratio between female and male publications decreased. This only helps to reduce the importance of the female role in the scientific contributions of computational linguistics (it is now far below its peak of 0.24). The pandemic has a particularly negative effect on the production of female senior researchers in the first position of authors (maximum work), followed by the female junior researchers in the last position of authors (supervision or collaborative work).
Adapting the Portuguese Braille System to Formal Semantics
Authors: Luís Filipe Cunha
Date: 2021-04-01 00:00:00
Proceeding: EACL (LTEDI) 2021 4

Since the seminal work of Richard Montague in the 1970s, mathematical and logic tools have successfully been used to model several aspects of the meaning of natural language. However, visually impaired people continue to face serious difficulties in getting full access to this important instrument. Our paper aims to present a work in progress whose main goal is to provide blind students and researchers with an adequate method to deal with the different resources that are used in formal semantics. In particular, we intend to adapt the Portuguese Braille system in order to accommodate the most common symbols and formulas used in this kind of approach and to develop pedagogical procedures to facilitate its learnability. By making this formalization compatible with the Braille coding (either traditional and electronic), we hope to help blind people to learn and use this notation, essential to acquire a better understanding of a great number of semantic properties displayed by natural language.
Cross-Lingual Transfer Learning for Hate Speech Detection
Authors: Irina Bigoulaeva, Viktor Hangya, Alexander Fraser
Date: 2021-04-01 00:00:00
Proceeding: EACL (LTEDI) 2021 4
Tasks: Cross-Lingual Transfer, Hate Speech Detection, Transfer Learning, Word Embeddings

We address the task of automatic hate speech detection for low-resource languages. Rather than collecting and annotating new hate speech data, we show how to use cross-lingual transfer learning to leverage already existing data from higher-resource languages. Using bilingual word embeddings based classifiers we achieve good performance on the target language by training only on the source dataset. Using our transferred system we bootstrap on unlabeled target language data, improving the performance of standard cross-lingual transfer approaches. We use English as a high resource language and German as the target language for which only a small amount of annotated corpora are available. Our results indicate that cross-lingual transfer learning together with our approach to leverage additional unlabeled data is an effective way of achieving good performance on low-resource target languages without the need for any target-language annotations.
hBERT + BiasCorp - Fighting Racism on the Web
Authors: Olawale Onabola, Zhuang Ma, Xie Yang, Benjamin Akera, Ibraheem Abdulrahman, Jia Xue, Dianbo Liu, Yoshua Bengio
Date: 2021-04-01 00:00:00
Proceeding: EACL (LTEDI) 2021 4

Subtle and overt racism is still present both in physical and online communities today and has impacted many lives in different segments of the society. In this short piece of work, we present how we’re tackling this societal issue with Natural Language Processing. We are releasing BiasCorp, a dataset containing 139,090 comments and news segment from three specific sources - Fox News, BreitbartNews and YouTube. The first batch (45,000 manually annotated) is ready for publication. We are currently in the final phase of manually labeling the remaining dataset using Amazon Mechanical Turk. BERT has been used widely in several downstream tasks. In this work, we present hBERT, where we modify certain layers of the pretrained BERT model with the new Hopfield Layer. hBert generalizes well across different distributions with the added advantage of a reduced model complexity. We are also releasing a JavaScript library 3 and a Chrome Extension Application, to help developers make use of our trained model in web applications (say chat application) and for users to identify and report racially biased contents on the web respectively
An Overview of Fairness in Data – Illuminating the Bias in Data Pipeline
Authors: Senthil Kumar B, Aravindan Chandrabose, Bharathi Raja Chakravarthi
Date: 2021-04-01 00:00:00
Proceeding: EACL (LTEDI) 2021 4
Tasks: Diversity, Fairness

Data in general encodes human biases by default; being aware of this is a good start, and the research around how to handle it is ongoing. The term ‘bias’ is extensively used in various contexts in NLP systems. In our research the focus is specific to biases such as gender, racism, religion, demographic and other intersectional views on biases that prevail in text processing systems responsible for systematically discriminating specific population, which is not ethical in NLP. These biases exacerbate the lack of equality, diversity and inclusion of specific population while utilizing the NLP applications. The tools and technology at the intermediate level utilize biased data, and transfer or amplify this bias to the downstream applications. However, it is not enough to be colourblind, gender-neutral alone when designing a unbiased technology – instead, we should take a conscious effort by designing a unified framework to measure and benchmark the bias. In this paper, we recommend six measures and one augment measure based on the observations of the bias in data, annotations, text representations and debiasing techniques.