pwc-archive/papers-with-abstracts huggingface dataset:
Grammaticality and Language Modelling
Authors: Jingcheng Niu, Gerald Penn
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: CoLA, Experimental Design, Language Modeling, Language Modelling, Linguistic Acceptability

Ever since Pereira (2000) provided evidence against Chomsky’s (1957) conjecture that statistical language modelling is incommensurable with the aims of grammaticality prediction as a research enterprise, a new area of research has emerged that regards statistical language models as “psycholinguistic subjects” and probes their ability to acquire syntactic knowledge. The advent of The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) has earned a spot on the leaderboard for acceptability judgements, and the polemic between Lau et al. (2017) and Sprouse et al. (2018) has raised fundamental questions about the nature of grammaticality and how acceptability judgements should be elicited. All the while, we are told that neural language models continue to improve. That is not an easy claim to test at present, however, because there is almost no agreement on how to measure their improvement when it comes to grammaticality and acceptability judgements. The GLUE leaderboard bundles CoLA together with a Matthews correlation coefficient (MCC), although probably because CoLA’s seminal publication was using it to compute inter-rater reliabilities. Researchers working in this area have used other accuracy and correlation scores, often driven by a need to reconcile and compare various discrete and continuous variables with each other. The score that we will advocate for in this paper, the point biserial correlation, in fact compares a discrete variable (for us, acceptability judgements) to a continuous variable (for us, neural language model probabilities). The only previous work in this area to choose the PBC that we are aware of is Sprouse et al. (2018a), and that paper actually applied it backwards (with some justification) so that the language model probability was treated as the discrete binary variable by setting a threshold. With the PBC in mind, we will first reappraise some recent work in syntactically targeted linguistic evaluations (Hu et al., 2020), arguing that while their experimental design sets a new high watermark for this topic, their results may not prove what they have claimed. We then turn to the task-independent assessment of language models as grammaticality classifiers. Prior to the introduction of the GLUE leaderboard, the vast majority of this assessment was essentially anecdotal, and we find the use of the MCC in this regard to be problematic. We conduct several studies with PBCs to compare several popular language models. We also study the effects of several variables such as normalization and data homogeneity on PBC.
One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations
Authors: Jesper Brink Andersen, Mikkel Bak Bertelsen, Mikkel Hørby Schou, Manuel R. Ciosici, Ira Assent
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: Word Embeddings

Word embeddings are an active topic in the NLP research community. State-of-the-art neural models achieve high performance on downstream tasks, albeit at the cost of computationally expensive training. Cost aware solutions require cheaper models that still achieve good performance. We present several reproduction studies of intrinsic evaluation tasks that evaluate non-contextual word representations in multiple languages. Furthermore, we present 50-8-8, a new data set for the outlier identification task, which avoids limitations of the original data set, such as ambiguous words, infrequent words, and multi-word tokens, while increasing the number of test cases. The data set is expanded to contain semantic and syntactic tests and is multilingual (English, German, and Italian). We provide an in-depth analysis of word embedding models with a range of hyper-parameters. Our analysis shows the suitability of different models and hyper-parameters for different tasks and the greater difficulty of representing German and Italian languages.
On Aligning OpenIE Extractions with Knowledge Bases: A Case Study
Authors: Kiril Gashteovski, Rainer Gemulla, Bhushan Kotnis, Sven Hertling, Christian Meilicke
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: Open Information Extraction

Open information extraction (OIE) is the task of extracting relations and their corresponding arguments from a natural language text in un- supervised manner. Outputs of such systems are used for downstream tasks such as ques- tion answering and automatic knowledge base (KB) construction. Many of these downstream tasks rely on aligning OIE triples with refer- ence KBs. Such alignments are usually eval- uated w.r.t. a specific downstream task and, to date, no direct manual evaluation of such alignments has been performed. In this paper, we directly evaluate how OIE triples from the OPIEC corpus are related to the DBpedia KB w.r.t. information content. First, we investigate OPIEC triples and DBpedia facts having the same arguments by comparing the information on the OIE surface relation with the KB rela- tion. Second, we evaluate the expressibility of general OPIEC triples in DBpedia. We in- vestigate whether—and, if so, how—a given OIE triple can be mapped to a single KB fact. We found that such mappings are not always possible because the information in the OIE triples tends to be more specific. Our evalua- tion suggests, however, that significant part of OIE triples can be expressed by means of KB formulas instead of individual facts.
ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation
Authors: Hanna Wecker, Annemarie Friedrich, Heike Adel
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: Clustering, Patent classification, Sentiment Analysis

This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation
Authors: Neslihan Iskender, Tim Polzehl, Sebastian Möller
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: Informativeness

One of the main challenges in the development of summarization tools is summarization quality evaluation. On the one hand, the human assessment of summarization quality conducted by linguistic experts is slow, expensive, and still not a standardized procedure. On the other hand, the automatic assessment metrics are reported not to correlate high enough with human quality ratings. As a solution, we propose crowdsourcing as a fast, scalable, and cost-effective alternative to expert evaluations to assess the intrinsic and extrinsic quality of summarization by comparing crowd ratings with expert ratings and automatic metrics such as ROUGE, BLEU, or BertScore on a German summarization data set. Our results provide a basis for best practices for crowd-based summarization evaluation regarding major influential factors such as the best annotation aggregation method, the influence of readability and reading effort on summarization evaluation, and the optimal number of crowd workers to achieve comparable results to experts, especially when determining factors such as overall quality, grammaticality, referential clarity, focus, structure & coherence, summary usefulness, and summary informativeness.
Evaluating Word Embeddings on Low-Resource Languages
Authors: Nathan Stringham, Mike Izbicki
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (Eval4NLP) 2020 11
Tasks: Model Selection, Word Embeddings

The analogy task introduced by Mikolov et al. (2013) has become the standard metric for tuning the hyperparameters of word embedding models. In this paper, however, we argue that the analogy task is unsuitable for low-resource languages for two reasons: (1) it requires that word embeddings be trained on large amounts of text, and (2) analogies may not be well-defined in some low-resource settings. We solve these problems by introducing the OddOneOut and Topk tasks, which are specifically designed for model selection in the low-resource setting. We use these metrics to successfully tune hyperparameters for a low-resource emoji embedding task and word embeddings on 16 extinct languages. The largest of these languages (Ancient Hebrew) has a 41 million token dataset, and the smallest (Old Gujarati) has only a 1813 token dataset.
‘Practical’, if that’s the word
Authors: Eimear Maguire
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Certain conditionals have something other than a clause as their consequent: their antecedent if-clauses are ‘adverbial clauses’ without a verb. We argue that they function in a way already seen for those with clausal consequents, despite lacking the content we might expect for the formation of a conditional. The use of the if-clause with sub-clausal consequents is feasible thanks to the fact that this function does not depend on the consequent content, and so is not impeded when the consequent does not provide a proposition, question or imperative. To support this we provide meaning rules for conditionals in terms of information state updates, letting the same construction play out in different ways depending on context and content.
Personae under uncertainty: The case of topoi
Authors: Bill Noble, Ellen Breitholtz, Robin Cooper
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

In this paper, we propose a probabilistic model of social signalling which adopts a persona-based account of social meaning. We use this model to develop a socio-semantic theory of conventionalised reasoning patterns, known as topoi. On this account the social meaning of a topos, as conveyed in a argument, is based on the set of idealogically-related topoi it indicates in context. We draw a connection between the role of personae in social meaning and the category adjustment effect, a well-known psychological phenomenon in which the representation of a stimulus is biased in the direction of the category in which it falls. Finally, we situate the interpretation of social signals as an update to the information state of an agent in a formal TTR model of dialogue.
Dogwhistles as Identity-based interpretative variation
Authors: Quentin Dénigot, Heather Burnett
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

The following paper presents a formal model for the description of dogwhistles. Dogwhistles are a class of terms or expressions often used in political discourse that are used with the goal of being interpreted in different ways by different communities. The model presented here describes this phenomenon using a variation on the Social Meaning Games framework that uses probability distributions over possible interpretation functions as well as RSA/IBR reasoning.
Conditional answers and the role of probabilistic epistemic representations
Authors: Jos Tellings
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Conditional utterances can be used in discourse as answers to regular, non-conditional questions in situations of partial knowledge of the answerer. We claim that the probabilities assigned to possible epistemic states of A are a measure of the utility of conditional answers. A second criterion that makes a conditional answer ‘if p, then q’ relevant has to do with the dependency between p and q that is conveyed in the statement. A conditional answer counts as relevant when this dependency leads the question asker to shift from a decision problem about q to an alternative, easier, decision problem about p.
Linguistic interpretation as inference under argument system uncertainty: the case of epistemic must
Authors: Brandon Waldon
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Modern semantic analyses of epistemic language (incl. the modals must and might) can be characterized by the following ‘credence assumption’: speakers have full certainty regarding the propositions that structure their epistemic state. Intuitively, however: a) speakers have graded, rather than categorical, commitment to these propositions, which are often never fully and explicitly articulated; b) listeners have higher-order uncertainty about this speaker uncertainty; c) must p is used to communicate speaker commitment to some conclusion p and to indicate speaker commitment to the premises that condition the conclusion. I explore the consequences of relaxing the credence assumption by extending the argument system semantic framework first proposed by Stone (1994) to a Bayesian probabilistic framework of modeling pragmatic interpretation (Goodman and Frank, 2016). The analysis makes desirable predictions regarding the behavior and interpretation of must, and it suggests a new way of considering the nature of context and communicative exchange.
Fast visual grounding in interaction: bringing few-shot learning with neural networks to an interactive robot
Authors: José Miguel Cano Santín, Simon Dobnik, Mehdi Ghanimifard
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: Few-Shot Learning, Transfer Learning, Visual Grounding

The major shortcomings of using neural networks with situated agents are that in incremental interaction very few learning examples are available and that their visual sensory representations are quite different from image caption datasets. In this work we adapt and evaluate a few-shot learning approach, Matching Networks (Vinyals et al., 2016), to conversational strategies of a robot interacting with a human tutor in order to efficiently learn to categorise objects that are presented to it and also investigate to what degree transfer learning from pre-trained models on images from different contexts can improve its performance. We discuss the implications of such learning on the nature of semantic representations the system has learned.
Discrete and Probabilistic Classifier-based Semantics
Authors: Staffan Larsson
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: Classification

We present a formal semantics (a version of Type Theory with Records) which places classifiers of perceptual information at the core of semantics. Using this framework, we present an account of the interpretation and classification of utterances referring to perceptually available situations (such as visual scenes). The account improves on previous work by clarifying the role of classifiers in a hybrid semantics combining statistical/neural classifiers with logical/inferential aspects of meaning. The account covers both discrete and probabilistic classification, thereby enabling learning, vagueness and other non-discrete linguistic phenomena.
Social Meaning in Repeated Interactions
Authors: Elin McCready, Robert Henderson
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Judgements about communicative agents evolve over the course of interactions both in how individuals are judged for testimonial reliability and for (ideological) trustworthiness. This paper combines a theory of social meaning and persona with a theory of reliability within a game-theoretic view of communication, giving a formal model involving interactional histories, repeated game models and ways of evaluating social meaning and trustworthiness.
Towards functional, agent-based models of dogwhistle communication
Authors: Robert Henderson, Elin McCready
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Henderson and McCready 2017, 2018, 2019 build a novel theory of so-called ‘dogwhistle’ communication by extending the social meaning games of Burnett 2017. This work reports on an ongoing project to build systems to model the evolution of dogwhistle communication in a population based on probability monads (Erwig and Kollmansberger, 2006; Kidd, 2007). The ultimate results will be useful not just for dogwhistles, but modeling the diffusion and evolution of social meaning in populations in general. The initial results presented here is a computational implementation of Henderson and McCready 2018, which will serve as the basis for models with multiple speakers and repeated interactions.
Stochastic Frames
Authors: Annika Schuster, Corina Stroessner, Peter Sutton, Henk Zeevat
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

In the frame hypothesis (CITATION), human concepts are equated with frames, which extend feature lists by a functional structure consisting of attributes and values. For example, a bachelor is represented by the attributes gender and marital status and their values ‘male’ and ‘unwed’. This paper makes the point that for many applications of concepts in cognition, including for concepts to be associated with lexemes in natural languages, the right structures to assume are not merely frames but stochastic frames in which attributes are associated with probability distributions over values. The paper introduces the idea of stochastic frames and suggests three applications: vagueness, ambiguity, and typicality.
A toy distributional model for fuzzy generalised quantifiers
Authors: Mehrnoosh Sadrzadeh, Gijs Wijnholds
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: model

Recent work in compositional distributional semantics showed how bialgebras model generalised quantifiers of natural language. That technique requires working with vector space over power sets of bases, and therefore is computationally costly. It is possible to overcome the computational hurdles by working with fuzzy generalised quantifiers. In this paper, we show that the compositional notion of semantics of natural language, guided by a grammar, extends from a binary to a many valued setting and instantiate in it the fuzzy computations. We import vector representations of words and predicates, learnt from large scale compositional distributional semantics, interpret them as fuzzy sets, and analyse their performance on a toy inference dataset.
Generating Lexical Representations of Frames using Lexical Substitution
Authors: Saba Anwar, Artem Shelmanov, Alexander Panchenko, Chris Biemann
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6

Semantic frames are formal linguistic structures describing situations/actions/events, e.g. Commercial transfer of goods. Each frame provides a set of roles corresponding to the situation participants, e.g. Buyer and Goods, and lexical units (LUs) – words and phrases that can evoke this particular frame in texts, e.g. Sell. The scarcity of annotated resources hinders wider adoption of frame semantics across languages and domains. We investigate a simple yet effective method, lexical substitution with word representation models, to automatically expand a small set of frame-annotated sentences with new words for their respective roles and LUs. We evaluate the expansion quality using FrameNet. Contextualized models demonstrate overall superior performance compared to the non-contextualized ones on roles. However, the latter show comparable performance on the task of LU expansion.
Informativity in Image Captions vs. Referring Expressions
Authors: Elizabeth Coppock, Danielle Dionne, Nathanial Graham, Elias Ganem, Shijie Zhao, Shawn Lin, Wenxing Liu, Derry Wijaya
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: Image Captioning, Object, Referring Expression, Referring expression generation, Text Generation

At the intersection between computer vision and natural language processing, there has been recent progress on two natural language generation tasks: Dense Image Captioning and Referring Expression Generation for objects in complex scenes. The former aims to provide a caption for a specified object in a complex scene for the benefit of an interlocutor who may not be able to see it. The latter aims to produce a referring expression that will serve to identify a given object in a scene that the interlocutor can see. The two tasks are designed for different assumptions about the common ground between the interlocutors, and serve very different purposes, although they both associate a linguistic description with an object in a complex scene. Despite these fundamental differences, the distinction between these two tasks is sometimes overlooked. Here, we undertake a side-by-side comparison between image captioning and reference game human datasets and show that they differ systematically with respect to informativity. We hope that an understanding of the systematic differences among these human datasets will ultimately allow them to be leveraged more effectively in the associated engineering tasks.
How does Punctuation Affect Neural Models in Natural Language Inference
Authors: Adam Ek, Jean-Philippe Bernardy, Stergios Chatzikyriakidis
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: Natural Language Inference, Sensitivity

Natural Language Inference models have reached almost human-level performance but their generalisation capabilities have not been yet fully characterized. In particular, sensitivity to small changes in the data is a current area of investigation. In this paper, we focus on the effect of punctuation on such models. Our findings can be broadly summarized as follows: (1) irrelevant changes in punctuation are correctly ignored by the recent transformer models (BERT) while older RNN-based models were sensitive to them. (2) All models, both transformers and RNN-based models, are incapable of taking into account small relevant changes in the punctuation.
Building a Swedish Question-Answering Model
Authors: Hannes von Essen, Daniel Hesslow
Date: 2020-06-01 00:00:00
Proceeding: PaM 2020 6
Tasks: Machine Translation, model, Question Answering

High quality datasets for question answering exist in a few languages, but far from all. Producing such datasets for new languages requires extensive manual labour. In this work we look at different methods for using existing datasets to train question-answering models in languages lacking such datasets. We show that machine translation followed by cross-lingual projection is a viable way to create a full question-answering dataset in a new language. We introduce new methods both for bitext alignment, using optimal transport, and for direct cross-lingual projection, utilizing multilingual BERT. We show that our methods produce good Swedish question-answering models without any manual work. Finally, we apply our proposed methods on Spanish and evaluate it on the XQuAD and MLQA benchmarks where we achieve new state-of-the-art values of 80.4 F1 and 62.9 Exact Match (EM) points on the Spanish XQuAD corpus and 70.8 F1 and 53.0 EM on the Spanish MLQA corpus, showing that the technique is readily applicable to other languages.