pwc-archive/papers-with-abstracts huggingface dataset:
Using four different online media sources to forecast the crude oil price
Authors: M. Elshendy, A. Fronzetti Colladon, E. Battistoni, P. A. Gloor
Date: 2021-05-19 00:00:00
Tasks: Articles

This study looks for signals of economic awareness on online social media and tests their significance in economic predictions. The study analyses, over a period of two years, the relationship between the West Texas Intermediate daily crude oil price and multiple predictors extracted from Twitter, Google Trends, Wikipedia, and the Global Data on Events, Language, and Tone database (GDELT). Semantic analysis is applied to study the sentiment, emotionality and complexity of the language used. Autoregressive Integrated Moving Average with Explanatory Variable (ARIMAX) models are used to make predictions and to confirm the value of the study variables. Results show that the combined analysis of the four media platforms carries valuable information in making financial forecasting. Twitter language complexity, GDELT number of articles and Wikipedia page reads have the highest predictive power. This study also allows a comparison of the different fore-sighting abilities of each platform, in terms of how many days ahead a platform can predict a price movement before it happens. In comparison with previous work, more media sources and more dimensions of the interaction and of the language used are combined in a joint analysis.
Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?
Authors: Maxime Peyrard, Beatriz Borges, Kristina Gligorić, Robert West
Date: 2021-05-19 00:00:00
Tasks: Benchmarking, Sentence

The automatic detection of humor poses a grand challenge for natural language processing. Transformer-based systems have recently achieved remarkable results on this task, but they usually (1)~were evaluated in setups where serious vs humorous texts came from entirely different sources, and (2)~focused on benchmarking performance without providing insights into how the models work. We make progress in both respects by training and analyzing transformer-based humor recognition models on a recently introduced dataset consisting of minimal pairs of aligned sentences, one serious, the other humorous. We find that, although our aligned dataset is much harder than previous datasets, transformer-based models recognize the humorous sentence in an aligned pair with high accuracy (78%). In a careful error analysis, we characterize easy vs hard instances. Finally, by analyzing attention weights, we obtain important insights into the mechanisms by which transformers recognize humor. Most remarkably, we find clear evidence that one single attention head learns to recognize the words that make a test sentence humorous, even without access to this information at training time.
Robustness and stability of enterprise intranet social networks: The impact of moderators
Authors: A. Fronzetti Colladon, F. Vagaggini
Date: 2021-05-19 00:00:00

In this study, we tested the robustness of three communication networks extracted from the online forums included in the intranet platforms of three large companies. For each company we analyzed the communication among employees both in terms of network structure and content (language used). Over a period of eight months, we analyzed more than 52,000 messages posted by approximately 12,000 employees. Specifically, we tested the network robustness and the stability of a set of structural and semantic metrics, while applying several different node removal strategies. We removed the forum moderators, the spammers, the overly connected nodes and the nodes lying at the network periphery, also testing different combinations of these selections. Results indicate that removing spammers and very peripheral nodes can be a relatively low impact strategy in this context; accordingly, it could be used to clean the noise generated by these types of social actor and to reduce the computation complexity of the analysis. On the other hand, the removal of moderators seems to have a significant impact on the network connectivity and the shared content. The most affected variables are closeness centrality and contribution index. We also found that the removal of overly connected nodes can significantly change the network structure. Lastly, we compared the behavior of moderators with the other users, finding distinctive characteristics by which moderators can be identified when their list is unknown. Our findings can help online community managers to understand the role of moderators within intranet forums and can be useful for social network analysts who are interested in evaluating the effects of graph simplification techniques.
Combining GCN and Transformer for Chinese Grammatical Error Detection
Authors: Jinhong Zhang
Date: 2021-05-19 00:00:00
Tasks: Grammatical Error Detection, Graph Neural Network

This paper describes our system at NLPTEA-2020 Task: Chinese Grammatical Error Diagnosis (CGED). The goal of CGED is to diagnose four types of grammatical errors: word selection (S), redundant words (R), missing words (M), and disordered words (W). The automatic CGED system contains two parts including error detection and error correction and our system is designed to solve the error detection problem. Our system is built on three models: 1) a BERT-based model leveraging syntactic information; 2) a BERT-based model leveraging contextual embeddings; 3) a lexicon-based graph neural network leveraging lexical information. We also design an ensemble mechanism to improve the single model's performance. Finally, our system achieves the highest F1 scores at detection level and identification level among all teams participating in the CGED 2020 task.
Essay-BR: a Brazilian Corpus of Essays
Authors: Jeziel C. Marinho, Rafael T. Anchieta, Raimundo S. Moura
Date: 2021-05-19 00:00:00

Automatic Essay Scoring (AES) is defined as the computer technology that evaluates and scores the written essays, aiming to provide computational models to grade essays either automatically or with minimal human involvement. While there are several AES studies in a variety of languages, few of them are focused on the Portuguese language. The main reason is the lack of a corpus with manually graded essays. In order to bridge this gap, we create a large corpus with several essays written by Brazilian high school students on an online platform. All of the essays are argumentative and were scored across five competencies by experts. Moreover, we conducted an experiment on the created corpus and showed challenges posed by the Portuguese language. Our corpus is publicly available at https://github.com/rafaelanchieta/essay.
Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots
Authors: Jia-Chen Gu, Hui Liu, Zhen-Hua Ling, Quan Liu, Zhigang Chen, Xiaodan Zhu
Date: 2021-05-19 00:00:00
Tasks: Retrieval

Persona can function as the prior knowledge for maintaining the consistency of dialogue systems. Most of previous studies adopted the self persona in dialogue whose response was about to be selected from a set of candidates or directly generated, but few have noticed the role of partner in dialogue. This paper makes an attempt to thoroughly explore the impact of utilizing personas that describe either self or partner speakers on the task of response selection in retrieval-based chatbots. Four persona fusion strategies are designed, which assume personas interact with contexts or responses in different ways. These strategies are implemented into three representative models for response selection, which are based on the Hierarchical Recurrent Encoder (HRE), Interactive Matching Network (IMN) and Bidirectional Encoder Representations from Transformers (BERT) respectively. Empirical studies on the Persona-Chat dataset show that the partner personas neglected in previous studies can improve the accuracy of response selection in the IMN- and BERT-based models. Besides, our BERT-based model implemented with the context-response-aware persona fusion strategy outperforms previous methods by margins larger than 2.7% on original personas and 4.6% on revised personas in terms of hits@1 (top-1 accuracy), achieving a new state-of-the-art performance on the Persona-Chat dataset.
Do Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition
Authors: Shengfei Lyu, Xingyu Wu, Jinlong Li, Qiuju Chen, Huanhuan Chen
Date: 2021-05-19 00:00:00
Tasks: Relation, Relation Classification

Deep neural networks such as BERT have made great progress in relation classification. Although they can achieve good performance, it is still a question of concern whether these models recognize the directionality of relations, especially when they may lack interpretability. To explore the question, a novel evaluation task, called Relation Direction Recognition (RDR), is proposed to explore whether models learn the directionality of relations. Three metrics for RDR are introduced to measure the degree to which models recognize the directionality of relations. Several state-of-the-art models are evaluated on RDR. Experimental results on a real-world dataset indicate that there are clear gaps among them in recognizing the directionality of relations, even though these models obtain similar performance in the traditional metric (e.g. Macro-F1). Finally, some suggestions are discussed to enhance models to recognize the directionality of relations from the perspective of model design or training.
Sentence Extraction-Based Machine Reading Comprehension for Vietnamese
Authors: Phong Nguyen-Thuan Do, Nhat Duy Nguyen, Tin Van Huynh, Kiet Van Nguyen, Anh Gia-Tuan Nguyen, Ngan Luu-Thuy Nguyen
Date: 2021-05-19 00:00:00
Tasks: Articles, Machine Reading Comprehension, Question Answering, Reading Comprehension, Sentence, Vietnamese Datasets

The development of natural language processing (NLP) in general and machine reading comprehension in particular has attracted the great attention of the research community. In recent years, there are a few datasets for machine reading comprehension tasks in Vietnamese with large sizes, such as UIT-ViQuAD and UIT-ViNewsQA. However, the datasets are not diverse in answers to serve the research. In this paper, we introduce UIT-ViWikiQA, the first dataset for evaluating sentence extraction-based machine reading comprehension in the Vietnamese language. The UIT-ViWikiQA dataset is converted from the UIT-ViQuAD dataset, consisting of comprises 23.074 question-answers based on 5.109 passages of 174 Wikipedia Vietnamese articles. We propose a conversion algorithm to create the dataset for sentence extraction-based machine reading comprehension and three types of approaches for sentence extraction-based machine reading comprehension in Vietnamese. Our experiments show that the best machine model is XLM-R_Large, which achieves an exact match (EM) of 85.97% and an F1-score of 88.77% on our dataset. Besides, we analyze experimental results in terms of the question type in Vietnamese and the effect of context on the performance of the MRC models, thereby showing the challenges from the UIT-ViWikiQA dataset that we propose to the language processing community.
QuatDE: Dynamic Quaternion Embedding for Knowledge Graph Completion
Authors: Haipeng Gao, Kun Yang, Yuxue Yang, Rufai Yusuf Zakari, Jim Wilson Owusu, Ke Qin
Date: 2021-05-19 00:00:00
Tasks: Graph Embedding, Knowledge Base Completion, Knowledge Graph Completion, Knowledge Graph Embedding, Link Prediction, Relation, Triplet

Knowledge graph embedding has been an active research topic for knowledge base completion (KGC), with progressive improvement from the initial TransE, TransH, RotatE et al to the current state-of-the-art QuatE. However, QuatE ignores the multi-faceted nature of the entity and the complexity of the relation, only using rigorous operation on quaternion space to capture the interaction between entitiy pair and relation, leaving opportunities for better knowledge representation which will finally help KGC. In this paper, we propose a novel model, QuatDE, with a dynamic mapping strategy to explicitly capture the variety of relational patterns and separate different semantic information of the entity, using transition vectors to adjust the point position of the entity embedding vectors in the quaternion space via Hamilton product, enhancing the feature interaction capability between elements of the triplet. Experiment results show QuatDE achieves state-of-the-art performance on three well-established knowledge graph completion benchmarks. In particular, the MR evaluation has relatively increased by 26% on WN18 and 15% on WN18RR, which proves the generalization of QuatDE.
Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
Authors: Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, Minlie Huang
Date: 2021-05-19 00:00:00
Proceeding: ACL 2021 5
Tasks: Decoder, Semantic Similarity, Semantic Textual Similarity, Sentence, Story Generation, Text Generation

Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing generation models (e.g., BART) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the decoder to capture the high-level semantics and discourse structures in the context beyond token-level co-occurrence. In this paper, we propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. To this end, we propose two pretraining objectives to learn the representations by predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Extensive experiments show that our model can generate more coherent texts than state-of-the-art baselines.
Answering Product-Questions by Utilizing Questions from Other Contextually Similar Products
Authors: Ohad Rozen, David Carmel, Avihai Mejer, Vitaly Mirkis, Yftah Ziser
Date: 2021-05-19 00:00:00
Proceeding: NAACL 2021 4

Predicting the answer to a product-related question is an emerging field of research that recently attracted a lot of attention. Answering subjective and opinion-based questions is most challenging due to the dependency on customer-generated content. Previous works mostly focused on review-aware answer prediction; however, these approaches fail for new or unpopular products, having no (or only a few) reviews at hand. In this work, we propose a novel and complementary approach for predicting the answer for such questions, based on the answers for similar questions asked on similar products. We measure the contextual similarity between products based on the answers they provide for the same question. A mixture-of-expert framework is used to predict the answer by aggregating the answers from contextually similar products. Empirical results demonstrate that our model outperforms strong baselines on some segments of questions, namely those that have roughly ten or more similar resolved questions in the corpus. We additionally publish two large-scale datasets used in this work, one is of similar product question pairs, and the second is of product question-answer pairs.
OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics
Authors: Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, Minlie Huang
Date: 2021-05-19 00:00:00
Proceeding: ACL 2021 5
Tasks: Story Generation, Text Generation

Automatic metrics are essential for developing natural language generation (NLG) models, particularly for open-ended language generation tasks such as story generation. However, existing automatic metrics are observed to correlate poorly with human evaluation. The lack of standardized benchmark datasets makes it difficult to fully evaluate the capabilities of a metric and fairly compare different metrics. Therefore, we propose OpenMEVA, a benchmark for evaluating open-ended story generation metrics. OpenMEVA provides a comprehensive test suite to assess the capabilities of metrics, including (a) the correlation with human judgments, (b) the generalization to different model outputs and datasets, (c) the ability to judge story coherence, and (d) the robustness to perturbations. To this end, OpenMEVA includes both manually annotated stories and auto-constructed test examples. We evaluate existing metrics on OpenMEVA and observe that they have poor correlation with human judgments, fail to recognize discourse-level incoherence, and lack inferential knowledge (e.g., causal order between events), the generalization ability and robustness. Our study presents insights for developing NLG models and metrics in further research.
A Sequence-to-Set Network for Nested Named Entity Recognition
Authors: Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, Yueting Zhuang
Date: 2021-05-19 00:00:00
Tasks: Decoder, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Nested Named Entity Recognition

Named entity recognition (NER) is a widely studied task in natural language processing. Recently, a growing number of studies have focused on the nested NER. The span-based methods, considering the entity recognition as a span classification task, can deal with nested entities naturally. But they suffer from the huge search space and the lack of interactions between entities. To address these issues, we propose a novel sequence-to-set neural network for nested NER. Instead of specifying candidate spans in advance, we provide a fixed set of learnable vectors to learn the patterns of the valuable spans. We utilize a non-autoregressive decoder to predict the final set of entities in one pass, in which we are able to capture dependencies between entities. Compared with the sequence-to-sequence method, our model is more suitable for such unordered recognition task as it is insensitive to the label order. In addition, we utilize the loss function based on bipartite matching to compute the overall training loss. Experimental results show that our proposed model achieves state-of-the-art on three nested NER corpora: ACE 2004, ACE 2005 and KBP 2017. The code is available at https://github.com/zqtan1024/sequence-to-set.
Effective Attention Sheds Light On Interpretability
Authors: Kaiser Sun, Ana Marasović
Date: 2021-05-18 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Language Modeling, Language Modelling

An attention matrix of a transformer self-attention sublayer can provably be decomposed into two components and only one of them (effective attention) contributes to the model output. This leads us to ask whether visualizing effective attention gives different conclusions than interpretation of standard attention. Using a subset of the GLUE tasks and BERT, we carry out an analysis to compare the two attention matrices, and show that their interpretations differ. Effective attention is less associated with the features related to the language modeling pretraining such as the separator token, and it has more potential to illustrate linguistic features captured by the model for solving the end-task. Given the found differences, we recommend using effective attention for studying a transformer's behavior since it is more pertinent to the model output by design.
Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing
Authors: Ganesh Jawahar, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan
Date: 2021-05-18 00:00:00
Proceeding: NAACL (CALCS) 2021 6
Tasks: Decoder, Language Modelling, Machine Translation, Translation

We describe models focused at the understudied problem of translating between monolingual and code-mixed language pairs. More specifically, we offer a wide range of models that convert monolingual English text into Hinglish (code-mixed Hindi and English). Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models (i.e., mT5 and mBART) on the task finding both to work well. Given the paucity of training data for code-mixing, we also propose a dependency-free method for generating code-mixed texts from bilingual distributed representations that we exploit for improving language model performance. In particular, armed with this additional data, we adopt a curriculum learning approach where we first finetune the language models on synthetic data then on gold code-mixed data. We find that, although simple, our synthetic code-mixing method is competitive with (and in some cases is even superior to) several standard methods (backtranslation, method based on equivalence constraint theory) under a diverse set of conditions. Our work shows that the mT5 model, finetuned following the curriculum learning procedure, achieves best translation performance (12.67 BLEU). Our models place first in the overall ranking of the English-Hinglish official shared task.
LCP-RIT at SemEval-2021 Task 1: Exploring Linguistic Features for Lexical Complexity Prediction
Authors: Abhinandan Desai, Kai North, Marcos Zampieri, Christopher M. Homan
Date: 2021-05-18 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Lexical Complexity Prediction, POS, regression

This paper describes team LCP-RIT's submission to the SemEval-2021 Task 1: Lexical Complexity Prediction (LCP). The task organizers provided participants with an augmented version of CompLex (Shardlow et al., 2020), an English multi-domain dataset in which words in context were annotated with respect to their complexity using a five point Likert scale. Our system uses logistic regression and a wide range of linguistic features (e.g. psycholinguistic features, n-grams, word frequency, POS tags) to predict the complexity of single words in this dataset. We analyze the impact of different linguistic features in the classification performance and we evaluate the results in terms of mean absolute error, mean squared error, Pearson correlation, and Spearman correlation.
POINTREC: A Test Collection for Narrative-driven Point of Interest Recommendation
Authors: Jafar Afzali, Aleksander Mark Drzewiecki, Krisztian Balog
Date: 2021-05-19 00:00:00

This paper presents a test collection for contextual point of interest (POI) recommendation in a narrative-driven scenario. There, user history is not available, instead, user requests are described in natural language. The requests in our collection are manually collected from social sharing websites, and are annotated with various types of metadata, including location, categories, constraints, and example POIs. These requests are to be resolved from a dataset of POIs, which are collected from a popular online directory, and are further linked to a geographical knowledge base and enriched with relevant web snippets. Graded relevance assessments are collected using crowdsourcing, by pooling both manual and automatic recommendations, where the latter serve as baselines for future performance comparison. This resource supports the development of novel approaches for end-to-end POI recommendation as well as for specific semantic annotation tasks on natural language requests.