pwc-archive/papers-with-abstracts huggingface dataset:
Identifying the Importance of Content Overlap for Better Cross-lingual Embedding Mappings
Authors: Réka Cserháti, Gábor Berend
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11

In this work, we analyze the performance and properties of cross-lingual word embedding models created by mapping-based alignment methods. We use several measures of corpus and embedding similarity to predict BLI scores of cross-lingual embedding mappings over three types of corpora, three embedding methods and 55 language pairs. Our experimental results corroborate that instead of mere size, the amount of common content in the training corpora is essential. This phenomenon manifests in that i) despite of the smaller corpus sizes, using only the comparable parts of Wikipedia for training the monolingual embedding spaces to be mapped is often more efficient than relying on all the contents of Wikipedia, ii) the smaller, in return less diversified Spanish Wikipedia works almost always much better as a training corpus for bilingual mappings than the ubiquitously used English Wikipedia.
On the Cross-lingual Transferability of Contextualized Sense Embeddings
Authors: Kiamehr Rezaee, Daniel Loureiro, Jose Camacho-Collados, Mohammad Taher Pilehvar
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Word Embeddings, Word Sense Disambiguation

In this paper we analyze the extent to which contextualized sense embeddings, i.e., sense embeddings that are computed based on contextualized word embeddings, are transferable across languages.To this end, we compiled a unified cross-lingual benchmark for Word Sense Disambiguation. We then propose two simple strategies to transfer sense-specific knowledge across languages and test them on the benchmark.Experimental results show that this contextualized knowledge can be effectively transferred to similar languages through pre-trained multilingual language models, to the extent that they can out-perform monolingual representations learnednfrom existing language-specific data.
Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages
Authors: Kelechi Ogueji, Yuxin Zhu, Jimmy Lin
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Language Modeling, Language Modelling, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Pretrained Multilingual Language Models, text-classification, Text Classification

Pretrained multilingual language models have been shown to work well on many languages for a variety of downstream NLP tasks. However, these models are known to require a lot of training data. This consequently leaves out a huge percentage of the world’s languages as they are under-resourced. Furthermore, a major motivation behind these models is that lower-resource languages benefit from joint training with higher-resource languages. In this work, we challenge this assumption and present the first attempt at training a multilingual language model on only low-resource languages. We show that it is possible to train competitive multilingual language models on less than 1 GB of text. Our model, named AfriBERTa, covers 11 African languages, including the first language model for 4 of these languages. Evaluations on named entity recognition and text classification spanning 10 languages show that our model outperforms mBERT and XLM-Rin several languages and is very competitive overall. Results suggest that our “small data” approach based on similar languages may sometimes work better than joint training on large datasets with high-resource languages. Code, data and models are released at https://github.com/keleog/afriberta.
Analysis of Zero-Shot Crosslingual Learning between English and Korean for Named Entity Recognition
Authors: Jongin Kim, Nayoung Choi, Seunghyun Lim, Jungwhan Kim, Soojin Chung, Hyunsoo Woo, Min Song, Jinho D. Choi
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Articles, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

This paper presents a English-Korean parallel dataset that collects 381K news articles where 1,400 of them, comprising 10K sentences, are manually labeled for crosslingual named entity recognition (NER). The annotation guidelines for the two languages are developed in parallel, that yield the inter-annotator agreement scores of 91 and 88% for English and Korean respectively, indicating sublime quality annotation in our dataset. Three types of crosslingual learning approaches, direct model transfer, embedding projection, and annotation projection, are used to develop zero-shot Korean NER models. Our best model gives the F1-score of 51% that is very encouraging, considering the extremely distinct natures of these two languages. This is pioneering work that explores zero-shot cross-lingual learning between English and Korean and provides rich parallel annotation for a core NLP task such as named entity recognition.
Regularising Fisher Information Improves Cross-lingual Generalisation
Authors: Asa Cooper Stickland, Iain Murray
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Memorization

Many recent works use ‘consistency regularisation’ to improve the generalisation of fine-tuned pre-trained models, both multilingual and English-only. These works encourage model outputs to be similar between a perturbed and normal version of the input, usually via penalising the Kullback–Leibler (KL) divergence between the probability distribution of the perturbed and normal model. We believe that consistency losses may be implicitly regularizing the loss landscape. In particular, we build on work hypothesising that implicitly or explicitly regularizing trace of the Fisher Information Matrix (FIM), amplifies the implicit bias of SGD to avoid memorization. Our initial results show both empirically and theoretically that consistency losses are related to the FIM, and show that the flat minima implied by a small trace of the FIM improves performance when fine-tuning a multilingual model on additional languages. We aim to confirm these initial results on more datasets, and use our insights to develop better multilingual fine-tuning techniques.
DMix: Distance Constrained Interpolative Mixup
Authors: Ramit Sawhney, Megh Thakkar, Shrey Pandit, Debdoot Mukherjee, Lucie Flek
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Classification, Data Augmentation, Sentence, Sentence Classification

Interpolation-based regularisation methods have proven to be effective for various tasks and modalities. Mixup is a data augmentation method that generates virtual training samples from convex combinations of individual inputs and labels. We extend Mixup and propose DMix, distance-constrained interpolative Mixup for sentence classification leveraging the hyperbolic space. DMix achieves state-of-the-art results on sentence classification over existing data augmentation methods across datasets in four languages.
Sequence Mixup for Zero-Shot Cross-Lingual Part-Of-Speech Tagging
Authors: Megh Thakkar, Vishwa Shah, Ramit Sawhney, Debdoot Mukherjee
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Cross-Lingual Transfer, Data Augmentation, Part-Of-Speech Tagging, Transfer Learning

There have been efforts in cross-lingual transfer learning for various tasks. We present an approach utilizing an interpolative data augmentation method, Mixup, to improve the generalizability of models for part-of-speech tagging trained on a source language, improving its performance on unseen target languages. Through experiments on ten languages with diverse structures and language roots, we put forward its applicability for downstream zero-shot cross-lingual tasks.
Well-Defined Morphology is Sentence-Level Morphology
Authors: Omer Goldman, Reut Tsarfaty
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Morphological Analysis, Morphological Inflection, Sentence

Morphological tasks have gained decent popularity within the NLP community in the recent years, with large multi-lingual datasets providing morphological analysis of words, either in or out of context. However, the lack of a clear linguistic definition for words destines the annotative work to be incomplete and mired in inconsistencies, especially cross-linguistically. In this work we expand morphological inflection of words to inflection of sentences to provide true universality disconnected from orthographic traditions of white-space usage. To allow annotation for sentence-inflection we define a morphological annotation scheme by a fixed set of inflectional features. We present a small cross-linguistic dataset including semi-manually generated simple sentences in 4 typologically diverse languages annotated according to our suggested scheme, and show that the task of reinflection gets substantially more difficult but that the change of scope from words to well-defined sentences allows interface with contextualized language models.
Cross-Lingual Training of Dense Retrievers for Document Retrieval
Authors: Peng Shi, Rui Zhang, He Bai, Jimmy Lin
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Document Ranking, Passage Ranking, Retrieval, Translation

Dense retrieval has shown great success for passage ranking in English. However, its effectiveness for non-English languages remains unexplored due to limitation in training resources. In this work, we explore different transfer techniques for document ranking from English annotations to non-English languages. Our experiments reveal that zero-shot model-based transfer using mBERT improves search quality. We find that weakly-supervised target language transfer is competitive compared to generation-based target language transfer, which requires translation models.
What Determines the Order of Verbal Dependents in Hindi? Effects of Efficiency in Comprehension and Production
Authors: Kartik Sharma, Richard Futrell, Samar Husain
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11

Word order flexibility is one of the distinctive features of SOV languages. In this work, we investigate whether the order and relative distance of preverbal dependents in Hindi, an SOV language, is affected by factors motivated by efficiency considerations during comprehension/production. We investigate the influence of Head–Dependent Mutual Information (HDMI), similarity-based interference, accessibility and case-marking. Results show that preverbal dependents remain close to the verbal head when the HDMI between the verb and its dependent is high. This demonstrates the influence of locality constraints on dependency distance and word order in an SOV language. Additionally, dependency distance were found to be longer when the dependent was animate, when it was case-marked and when it was semantically similar to other preverbal dependents. Together the results highlight the crosslinguistic generalizability of these factors and provide evidence for a functionally motivated account of word order in SOV languages such as Hindi.
Images and Imagination: Automated Analysis of Priming Effects Related to Autism Spectrum Disorder and Developmental Language Disorder
Authors: Michaela Regneri, Diane King, Fahreen Walji, Olympia Palikara
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11

Different aspects of language processing have been shown to be sensitive to priming but the findings of studies examining priming effects in adolescents with Autism Spectrum Disorder (ASD) and Developmental Language Disorder (DLD) have been inconclusive. We present a study analysing visual and implicit semantic priming in adolescents with ASD and DLD. Based on a dataset of fictional and script-like narratives, we evaluate how often and how extensively, content of two different priming sources is used by the participants. The first priming source was visual, consisting of images shown to the participants to assist them with their storytelling. The second priming source originated from commonsense knowledge, using crowdsourced data containing prototypical script elements. Our results show that individuals with ASD are less sensitive to both types of priming, but show typical usage of primed cues when they use them at all. In contrast, children with DLD show mostly average priming sensitivity, but exhibit an over-proportional use of the priming cues.
Production-based Cognitive Models as a Test Suite for Reinforcement Learning Algorithms
Authors: Adrian Brasoveanu, Jakub Dotlacil
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: reinforcement-learning, Reinforcement Learning (RL)

We introduce a framework in which production-rule based computational cognitive modeling and Reinforcement Learning can systematically interact and inform each other. We focus on linguistic applications because the sophisticated rule-based cognitive models needed to capture linguistic behavioral data promise to provide a stringent test suite for RL algorithms, connecting RL algorithms to both accuracy and reaction-time experimental data. Thus, we open a path towards assembling an experimentally rigorous and cognitively realistic benchmark for RL algorithms. We extend our previous work on lexical decision tasks and tabular RL algorithms (Brasoveanu and Dotlačil, 2020b) with a discussion of neural-network based approaches, and a discussion of how parsing can be formalized as an RL problem.
Evaluating Word Embeddings for Language Acquisition
Authors: Raquel G. Alhama, Caroline Rowland, Evan Kidd
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: Language Acquisition, Word Embeddings

Continuous vector word representations (or word embeddings) have shown success in capturing semantic relations between words, as evidenced with evaluation against behavioral data of adult performance on semantic tasks (Pereira et al. 2016). Adult semantic knowledge is the endpoint of a language acquisition process; thus, a relevant question is whether these models can also capture emerging word representations of young language learners. However, the data of semantic knowledge of children is scarce or non-existent for some age groups. In this paper, we propose to bridge this gap by using Age of Acquisition norms to evaluate word embeddings learnt from child-directed input. We present two methods that evaluate word embeddings in terms of (a) the semantic neighbourhood density of learnt words, and (b) the convergence to adult word associations. We apply our methods to bag-of-words models, and we find that (1) children acquire words with fewer semantic neighbours earlier, and (2) young learners only attend to very local context. These findings provide converging evidence for validity of our methods in understanding the prerequisite features for a distributional model of word learning.
Guessing the Age of Acquisition of Italian Lemmas through Linear Regression
Authors: Irene Russo
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: regression

The age of acquisition of a word is a psycholinguistic variable concerning the age at which a word is typically learned. It correlates with other psycholinguistic variables such as familiarity, concreteness, and imageability. Existing datasets for multiple languages also include linguistic variables such as the length and the frequency of lemmas in different corpora. There are substantial sets of normative values for English, but for other languages, such as Italian, the coverage is scarce. In this paper,a set of regression experiments investigates whether it is possible to guess the age of acquisition of Italian lemmas that have not been previously rated by humans. An intrinsic evaluation is proposed, correlating estimated Italian lemmas’ AoA with English lemmas’ AoA. An extrinsic evaluation - using AoA values as features for the classification of literary excerpts labeled by age appropriateness - shows how es-sential is lexical coverage for this task.
Word Co-occurrence in Child-directed Speech Predicts Children’s Free Word Associations
Authors: Abdellah Fourtassi
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11

The free association task has been very influential both in cognitive science and in computational linguistics. However, little research has been done to study how free associations develop in childhood. The current work focuses on the developmental hypothesis according to which free word associations emerge by mirroring the co-occurrence distribution of children’s linguistic environment. I trained a distributional semantic model on a large corpus of child language and I tested if it could predict children’s responses. The results largely supported the hypothesis: Co-occurrence-based similarity was a strong predictor of children’s associative behavior even controlling for other possible predictors such as phonological similarity, word frequency, and word length. I discuss the findings in the light of theories of conceptual development.
Development of Multi-level Linguistic Alignment in Child-adult Conversations
Authors: Thomas Misiek, Benoit Favre, Abdellah Fourtassi
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: Diversity

Interactive alignment is a major mechanism of linguistic coordination. Here we study the way this mechanism emerges in development across the lexical, syntactic, and conceptual levels. We leverage NLP tools to analyze a large-scale corpus of child-adult conversations between 2 and 5 years old. We found that, across development, children align consistently to adults above chance and that adults align consistently more to children than vice versa (even controlling for language production abilities). Besides these consistencies, we found a diversity of developmental trajectories across linguistic levels. These corpus-based findings provide strong support for an early onset of multi-level linguistic alignment in children and invites new experimental work.
Conditioning, but on Which Distribution? Grammatical Gender in German Plural Inflection
Authors: Kate McCurdy, Adam Lopez, Sharon Goldwater
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: Decoder

Grammatical gender is a consistent and informative cue to the plural class of German nouns. We find that neural encoder-decoder models learn to rely on this cue to predict plural class, but adult speakers are relatively insensitive to it. This suggests that the neural models are not an effective cognitive model of German plural formation.
Learning Pronoun Case from Distributional Cues: Flexible Frames for Case Acquisition
Authors: Xiaomeng Ma, Martin Chodorow, Virginia Valian
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (CMCL) 2020 11
Tasks: Sentence

Case is an abstract grammatical feature that indicates argument relationship in a sentence. In English, cases are expressed on pronouns, as nominative case (e.g. I, he), accusative case (e.g. me, him) and genitive case (e.g. my, his). Children correctly use cased pronouns at a very young age. How do they acquire abstract case in the first place, when different cases are not associated with different meanings? This paper proposes that the distributional patterns in parents’ input could be used to distinguish grammatical cases in English.
Individual Differences in the Movement-Mood Relationship in Digital Life Data
Authors: Glen Coppersmith, Alex Fine, Patrick Crutchley, Joshua Carroll
Date: 2021-06-01 00:00:00
Proceeding: NAACL (CLPsych) 2021 6

Our increasingly digitized lives generate troves of data that reflect our behavior, beliefs, mood, and wellbeing. Such “digital life data” provides crucial insight into the lives of patients outside the healthcare setting that has long been lacking, from a better understanding of mundane patterns of exercise and sleep routines to harbingers of emotional crisis. Moreover, information about individual differences and personalities is encoded in digital life data. In this paper we examine the relationship between mood and movement using linguistic and biometric data, respectively. Does increased physical activity (movement) have an effect on a person’s mood (or vice-versa)? We find that weak group-level relationships between movement and mood mask interesting and often strong relationships between the two for individuals within the group. We describe these individual differences, and argue that individual variability in the relationship between movement and mood is one of many such factors that ought be taken into account in wellbeing-focused apps and AI systems.
Dissociating Semantic and Phonemic Search Strategies in the Phonemic Verbal Fluency Task in early Dementia
Authors: Hali Lindsay, Philipp Müller, Nicklas Linz, Radia Zeghari, Mario Magued Mina, Alexandra Konig, Johannes Tröger
Date: 2021-06-01 00:00:00
Proceeding: NAACL (CLPsych) 2021 6
Tasks: Management, Semantic Similarity, Semantic Textual Similarity

Effective management of dementia hinges on timely detection and precise diagnosis of the underlying cause of the syndrome at an early mild cognitive impairment (MCI) stage. Verbal fluency tasks are among the most often applied tests for early dementia detection due to their efficiency and ease of use. In these tasks, participants are asked to produce as many words as possible belonging to either a semantic category (SVF task) or a phonemic category (PVF task). Even though both SVF and PVF share neurocognitive function profiles, the PVF is typically believed to be less sensitive to measure MCI-related cognitive impairment and recent research on fine-grained automatic evaluation of VF tasks has mainly focused on the SVF. Contrary to this belief, we show that by applying state-of-the-art semantic and phonemic distance metrics in automatic analysis of PVF word productions, in-depth conclusions about production strategy of MCI patients are possible. Our results reveal a dissociation between semantically- and phonemically-guided search processes in the PVF. Specifically, we show that subjects with MCI rely less on semantic- and more on phonemic processes to guide their word production as compared to healthy controls (HC). We further show that semantic similarity-based features improve automatic MCI versus HC classification by 29% over previous approaches for the PVF. As such, these results point towards the yet underexplored utility of the PVF for in-depth assessment of cognition in MCI.
Demonstrating the Reliability of Self-Annotated Emotion Data
Authors: Anton Malko, Cecile Paris, Andreas Duenser, Maria Kangas, Diego Molla, Ross Sparks, Stephen Wan
Date: 2021-06-01 00:00:00
Proceeding: NAACL (CLPsych) 2021 6

Vent is a specialised iOS/Android social media platform with the stated goal to encourage people to post about their feelings and explicitly label them. In this paper, we study a snapshot of more than 100 million messages obtained from the developers of Vent, together with the labels assigned by the authors of the messages. We establish the quality of the self-annotated data by conducting a qualitative analysis, a vocabulary based analysis, and by training and testing an emotion classifier. We conclude that the self-annotated labels of our corpus are indeed indicative of the emotional contents expressed in the text and thus can support more detailed analyses of emotion expression on social media, such as emotion trajectories and factors influencing them.