pwc-archive/papers-with-abstracts huggingface dataset:
Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels
Authors: Nicolò Felicioni, Maurizio Ferrari Dacrema, Paolo Cremonesi
Date: 2021-05-14 00:00:00
Tasks: Position

It is common for video-on-demand and music streaming services to adopt a user interface composed of several recommendation lists, i.e. widgets or swipeable carousels, each generated according to a specific criterion or algorithm (e.g. most recent, top popular, recommended for you, editors' choice, etc.). Selecting the appropriate combination of carousel has significant impact on user satisfaction. A crucial aspect of this user interface is that to measure the relevance a new carousel for the user it is not sufficient to account solely for its individual quality. Instead, it should be considered that other carousels will already be present in the interface. This is not considered by traditional evaluation protocols for recommenders systems, in which each carousel is evaluated in isolation, regardless of (i) which other carousels are displayed to the user and (ii) the relative position of the carousel with respect to other carousels. Hence, we propose a two-dimensional evaluation protocol for a carousel setting that will measure the quality of a recommendation carousel based on how much it improves upon the quality of an already available set of carousels. Our evaluation protocol takes into account also the position bias, i.e. users do not explore the carousels sequentially, but rather concentrate on the top-left corner of the screen. We report experiments on the movie domain and notice that under a carousel setting the definition of which criteria has to be preferred to generate a list of recommended items changes with respect to what is commonly understood.
Post-processing Multi-Model Medium-Term Precipitation Forecasts Using Convolutional Neural Networks
Authors: Bob de Ruiter
Date: 2021-05-14 00:00:00
Tasks: BIG-bench Machine Learning, regression

The goal of this study was to improve the post-processing of precipitation forecasts using convolutional neural networks (CNNs). Instead of post-processing forecasts on a per-pixel basis, as is usually done when employing machine learning in meteorological post-processing, input forecast images were combined and transformed into probabilistic output forecast images using fully convolutional neural networks. CNNs did not outperform regularized logistic regression. Additionally, an ablation analysis was performed. Combining input forecasts from a global low-resolution weather model and a regional high-resolution weather model improved performance over either one.
Cause and Effect: Hierarchical Concept-based Explanation of Neural Networks
Authors: Mohammad Nokhbeh Zaeem, Majid Komeili
Date: 2021-05-14 00:00:00
Tasks: Negation

In many scenarios, human decisions are explained based on some high-level concepts. In this work, we take a step in the interpretability of neural networks by examining their internal representation or neuron's activations against concepts. A concept is characterized by a set of samples that have specific features in common. We propose a framework to check the existence of a causal relationship between a concept (or its negation) and task classes. While the previous methods focus on the importance of a concept to a task class, we go further and introduce four measures to quantitatively determine the order of causality. Moreover, we propose a method for constructing a hierarchy of concepts in the form of a concept-based decision tree which can shed light on how various concepts interact inside a neural network towards predicting output classes. Through experiments, we demonstrate the effectiveness of the proposed method in explaining the causal relationship between a concept and the predictive behaviour of a neural network as well as determining the interactions between different concepts through constructing a concept hierarchy.
A hyperparameter-tuning approach to automated inverse planning
Authors: Kelsey Maass, Aleksandr Aravkin, Minsun Kim
Date: 2021-05-14 00:00:00
Tasks: Bayesian Optimization, Dimensionality Reduction

Radiotherapy inverse planning often requires planners to modify parameters in the treatment planning system's objective function to produce clinically acceptable plans. Due to the manual steps in this process, plan quality can vary depending on the planning time available and the planner's skills. This study investigates two hyperparameter-tuning methods for automated inverse planning. Because this framework does not train a model on previously-optimized plans, it can be readily adapted to practice pattern changes, and plan quality is not limited by that of a training cohort. We selected 10 patients who received lung SBRT using manually-generated clinical plans. We used random sampling (RS) and Bayesian optimization (BO) to tune parameters using linear-quadratic utility functions based on 11 clinical goals. Normalizing all plans to have PTV D95 equal to 48 Gy, we compared plan quality for the automatically-generated and manually-generated plans. We also investigated the impact of iteration count on the automatically-generated plans, comparing planning time and plan utility for RS and BO plans with and without stopping criteria. Without stopping criteria, the median planning time was 1.9 and 2.3 hours for RS and BO plans. The OAR doses in the RS and BO plans had a median percent difference (MPD) of 48.7% and 60.4% below clinical dose limits and an MPD of 2.8% and 3.3% below clinical plan doses. With stopping criteria, the utility decreased by an MPD of 5.3% and 3.9% for RS and BO plans, but the median planning time was reduced to 0.5 and 0.7 hours, and the OAR doses still had an MPD of 42.9% and 49.7% below clinical dose limits and an MPD of 0.3% and 1.8% below clinical plan doses. This study demonstrates that hyperparameter-tuning approaches to automated inverse planning can reduce active planning time with plan quality that is similar to or better than manually-generated plans.
A causal learning framework for the analysis and interpretation of COVID-19 clinical data
Authors: Elisa Ferrari, Luna Gargani, Greta Barbieri, Lorenzo Ghiadoni, Francesco Faita, Davide Bacciu
Date: 2021-05-14 00:00:00

We present a workflow for clinical data analysis that relies on Bayesian Structure Learning (BSL), an unsupervised learning approach, robust to noise and biases, that allows to incorporate prior medical knowledge into the learning process and that provides explainable results in the form of a graph showing the causal connections among the analyzed features. The workflow consists in a multi-step approach that goes from identifying the main causes of patient's outcome through BSL, to the realization of a tool suitable for clinical practice, based on a Binary Decision Tree (BDT), to recognize patients at high-risk with information available already at hospital admission time. We evaluate our approach on a feature-rich COVID-19 dataset, showing that the proposed framework provides a schematic overview of the multi-factorial processes that jointly contribute to the outcome. We discuss how these computational findings are confirmed by current understanding of the COVID-19 pathogenesis. Further, our approach yields to a highly interpretable tool correctly predicting the outcome of 85% of subjects based exclusively on 3 features: age, a previous history of chronic obstructive pulmonary disease and the PaO2/FiO2 ratio at the time of arrival to the hospital. The inclusion of additional information from 4 routine blood tests (Creatinine, Glucose, pO2 and Sodium) increases predictive accuracy to 94.5%.
Fine-grained Interpretation and Causation Analysis in Deep NLP Models
Authors: Hassan Sajjad, Narine Kokhlikyan, Fahim Dalvi, Nadir Durrani
Date: 2021-05-17 00:00:00
Proceeding: NAACL 2021 4
Tasks: Domain Adaptation

This paper is a write-up for the tutorial on "Fine-grained Interpretation and Causation Analysis in Deep NLP Models" that we are presenting at NAACL 2021. We present and discuss the research work on interpreting fine-grained components of a model from two perspectives, i) fine-grained interpretation, ii) causation analysis. The former introduces methods to analyze individual neurons and a group of neurons with respect to a language property or a task. The latter studies the role of neurons and input features in explaining decisions made by the model. We also discuss application of neuron analysis such as network manipulation and domain adaptation. Moreover, we present two toolkits namely NeuroX and Captum, that support functionalities discussed in this tutorial.
Room to Grow: Understanding Personal Characteristics Behind Self Improvement Using Social Media
Authors: MeiXing Dong, Xueming Xu, Yiwei Zhang, Ian Stewart, Rada Mihalcea
Date: 2021-05-17 00:00:00
Proceeding: NAACL (SocialNLP) 2021 6

Many people aim for change, but not everyone succeeds. While there are a number of social psychology theories that propose motivation-related characteristics of those who persist with change, few computational studies have explored the motivational stage of personal change. In this paper, we investigate a new dataset consisting of the writings of people who manifest intention to change, some of whom persist while others do not. Using a variety of linguistic analysis techniques, we first examine the writing patterns that distinguish the two groups of people. Persistent people tend to reference more topics related to long-term self-improvement and use a more complicated writing style. Drawing on these consistent differences, we build a classifier that can reliably identify the people more likely to persist, based on their language. Our experiments provide new insights into the motivation-related behavior of people who persist with their intention to change.
SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising
Authors: Kuan Xu, Yongbo Wang, Yongliang Wang, Zujie Wen, Yang Dong
Date: 2021-05-17 00:00:00
Proceeding: Findings (NAACL) 2022 7
Tasks: Decoder, Denoising, slot-filling, Slot Filling, Text to SQL, Text-To-SQL

In text-to-SQL task, seq-to-seq models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq model to robust text-to-SQL generation. Instead of inducing constraint to decoder or reformat the task as slot-filling, we propose to train seq-to-seq model with Schema aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives acts as the auxiliary tasks for better modeling the structural data in S2S generation. In addition, we improve and propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of seq-to-seq model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. The results indicate that the capacity of vanilla seq-to-seq architecture for text-to-SQL may have been under-estimated.
Factoring Statutory Reasoning as Language Understanding Challenges
Authors: Nils Holzenberger, Benjamin Van Durme
Date: 2021-05-17 00:00:00
Proceeding: ACL 2021 5
Tasks: Natural Language Inference

Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that approached statutory reasoning as a monolithic textual entailment problem, with neural baselines performing nearly at-chance. To address this challenge, we decompose statutory reasoning into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog programs. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statutory reasoning are shown to benefit from the additional structure, improving on prior baselines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress.
Studying the association of online brand importance with museum visitors: An application of the semantic brand score
Authors: A. Fronzetti Colladon, F. Grippa, R. Innarella
Date: 2021-05-17 00:00:00
Tasks: Diversity, regression

This paper explores the association between brand importance and growth in museum visitors. We analyzed 10 years of online forum discussions and applied the Semantic Brand Score (SBS) to assess the brand importance of five European Museums. Our Naive Bayes and regression models indicate that variations in the combined dimensions of the SBS (prevalence, diversity and connectivity) are aligned with changes in museum visitors. Results suggest that, in order to attract more visitors, museum brand managers should focus on increasing the volume of online posting and the richness of information generated by users around the brand, rather than controlling for the posts' overall positivity or negativity.
Using social network and semantic analysis to analyze online travel forums and forecast tourism demand
Authors: A Fronzetti Colladon, B Guardabascio, R Innarella
Date: 2021-05-17 00:00:00

Forecasting tourism demand has important implications for both policy makers and companies operating in the tourism industry. In this research, we applied methods and tools of social network and semantic analysis to study user-generated content retrieved from online communities which interacted on the TripAdvisor travel forum. We analyzed the forums of 7 major European capital cities, over a period of 10 years, collecting more than 2,660,000 posts, written by about 147,000 users. We present a new methodology of analysis of tourism-related big data and a set of variables which could be integrated into traditional forecasting models. We implemented Factor Augmented Autoregressive and Bridge models with social network and semantic variables which often led to a better forecasting performance than univariate models and models based on Google Trend data. Forum language complexity and the centralization of the communication network, i.e. the presence of eminent contributors, were the variables that contributed more to the forecasting of international airport arrivals.
A CCG-Based Version of the DisCoCat Framework
Authors: Richie Yeung, Dimitri Kartsaklis
Date: 2021-05-17 00:00:00
Proceeding: ACL (SemSpace, IWCS) 2021 6

While the DisCoCat model (Coecke et al., 2010) has been proved a valuable tool for studying compositional aspects of language at the level of semantics, its strong dependency on pregroup grammars poses important restrictions: first, it prevents large-scale experimentation due to the absence of a pregroup parser; and second, it limits the expressibility of the model to context-free grammars. In this paper we solve these problems by reformulating DisCoCat as a passage from Combinatory Categorial Grammar (CCG) to a category of semantics. We start by showing that standard categorial grammars can be expressed as a biclosed category, where all rules emerge as currying/uncurrying the identity; we then proceed to model permutation-inducing rules by exploiting the symmetry of the compact closed category encoding the word meaning. We provide a proof of concept for our method, converting "Alice in Wonderland" into DisCoCat form, a corpus that we make available to the community.
Sentence Similarity Based on Contexts
Authors: Xiaofei Sun, Yuxian Meng, Xiang Ao, Fei Wu, Tianwei Zhang, Jiwei Li, Chun Fan
Date: 2021-05-17 00:00:00
Tasks: Language Modeling, Language Modelling, Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Similarity

Existing methods to measure sentence similarity are faced with two challenges: (1) labeled datasets are usually limited in size, making them insufficient to train supervised neural models; (2) there is a training-test gap for unsupervised language modeling (LM) based models to compute semantic scores between sentences, since sentence-level semantics are not explicitly modeled at training. This results in inferior performances in this task. In this work, we propose a new framework to address these two issues. The proposed framework is based on the core idea that the meaning of a sentence should be defined by its contexts, and that sentence similarity can be measured by comparing the probabilities of generating two sentences given the same context. The proposed framework is able to generate high-quality, large-scale dataset with semantic similarity scores between two sentences in an unsupervised manner, with which the train-test gap can be largely bridged. Extensive experiments show that the proposed framework achieves significant performance boosts over existing baselines under both the supervised and unsupervised settings across different datasets.
Ensemble-based Transfer Learning for Low-resource Machine Translation Quality Estimation
Authors: Ting-Wei Wu, Yung-An Hsieh, Yi-Chieh Liu
Date: 2021-05-17 00:00:00
Tasks: Machine Translation, Miscellaneous, Sentence, Transfer Learning, Translation

Quality Estimation (QE) of Machine Translation (MT) is a task to estimate the quality scores for given translation outputs from an unknown MT system. However, QE scores for low-resource languages are usually intractable and hard to collect. In this paper, we focus on the Sentence-Level QE Shared Task of the Fifth Conference on Machine Translation (WMT20), but in a more challenging setting. We aim to predict QE scores of given translation outputs when barely none of QE scores of that paired languages are given during training. We propose an ensemble-based predictor-estimator QE model with transfer learning to overcome such QE data scarcity challenge by leveraging QE scores from other miscellaneous languages and translation results of targeted languages. Based on the evaluation results, we provide a detailed analysis of how each of our extension affects QE models on the reliability and the generalization ability to perform transfer learning under multilingual tasks. Finally, we achieve the best performance on the ensemble model combining the models pretrained by individual languages as well as different levels of parallel trained corpus with a Pearson's correlation of 0.298, which is 2.54 times higher than baselines.
Classifying Argumentative Relations Using Logical Mechanisms and Argumentation Schemes
Authors: Yohan Jo, Seojin Bang, Chris Reed, Eduard Hovy
Date: 2021-05-17 00:00:00
Tasks: Argument Mining, Relation, Representation Learning

While argument mining has achieved significant success in classifying argumentative relations between statements (support, attack, and neutral), we have a limited computational understanding of logical mechanisms that constitute those relations. Most recent studies rely on black-box models, which are not as linguistically insightful as desired. On the other hand, earlier studies use rather simple lexical features, missing logical relations between statements. To overcome these limitations, our work classifies argumentative relations based on four logical and theory-informed mechanisms between two statements, namely (i) factual consistency, (ii) sentiment coherence, (iii) causal relation, and (iv) normative relation. We demonstrate that our operationalization of these logical mechanisms classifies argumentative relations without directly training on data labeled with the relations, significantly better than several unsupervised baselines. We further demonstrate that these mechanisms also improve supervised classifiers through representation learning.
Data Augmentation for Sign Language Gloss Translation
Authors: Amit Moryossef, Kayo Yin, Graham Neubig, Yoav Goldberg
Date: 2021-05-16 00:00:00
Proceeding: MTSummit 2021 8
Tasks: Data Augmentation, Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Low Resource NMT, Machine Translation, NMT, Sign Language Translation, Translation

Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss-to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low-resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergence by proposing two rule-based heuristics that generate pseudo-parallel gloss-text pairs from monolingual spoken language text. By pre-training on the thus obtained synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.
How is BERT surprised? Layerwise detection of linguistic anomalies
Authors: Bai Li, Zining Zhu, Guillaume Thomas, Yang Xu, Frank Rudzicz
Date: 2021-05-16 00:00:00
Proceeding: ACL 2021 5
Tasks: Density Estimation

Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is morphosyntactic than when it is semantic, while commonsense anomalies do not exhibit surprisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of linguistic anomalies.
The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus
Authors: Samia Touileb, Jeremy Barnes
Date: 2021-05-16 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Cross-Lingual Transfer, Part-Of-Speech Tagging, Sentiment Analysis

Recent years have seen a rise in interest for cross-lingual transfer between languages with similar typology, and between languages of various scripts. However, the interplay between language similarity and difference in script on cross-lingual transfer is a less studied problem. We explore this interplay on cross-lingual transfer for two supervised tasks, namely part-of-speech tagging and sentiment analysis. We introduce a newly annotated corpus of Algerian user-generated comments comprising parallel annotations of Algerian written in Latin, Arabic, and code-switched scripts, as well as annotations for sentiment and topic categories. We perform baseline experiments by fine-tuning multi-lingual language models. We further explore the effect of script vs. language similarity in cross-lingual transfer by fine-tuning multi-lingual models on languages which are a) typologically distinct, but use the same script, b) typologically similar, but use a distinct script, or c) are typologically similar and use the same script. We find there is a delicate relationship between script and typology for part-of-speech, while sentiment analysis is less sensitive.