pwc-archive/papers-with-abstracts huggingface dataset:
Latent Tree Decomposition Parsers for AMR-to-Text Generation
Authors: Lisa Jin, Daniel Gildea
Date: 2021-08-27 00:00:00
Tasks: AMR-to-Text Generation, Clustering, Molecular Property Prediction, Property Prediction, Text Generation, Tree Decomposition

Graph encoders in AMR-to-text generation models often rely on neighborhood convolutions or global vertex attention. While these approaches apply to general graphs, AMRs may be amenable to encoders that target their tree-like structure. By clustering edges into a hierarchy, a tree decomposition summarizes graph structure. Our model encodes a derivation forest of tree decompositions and extracts an expected tree. From tree node embeddings, it builds graph edge features used in vertex attention of the graph encoder. Encoding TD forests instead of shortest-pairwise paths in a self-attentive baseline raises BLEU by 0.7 and chrF++ by 0.3. The forest encoder also surpasses a convolutional baseline for molecular property prediction by 1.92% ROC-AUC.
Tree Decomposition Attention for AMR-to-Text Generation
Authors: Lisa Jin, Daniel Gildea
Date: 2021-08-27 00:00:00
Tasks: AMR-to-Text Generation, Text Generation

Text generation from AMR requires mapping a semantic graph to a string that it annotates. Transformer-based graph encoders, however, poorly capture vertex dependencies that may benefit sequence prediction. To impose order on an encoder, we locally constrain vertex self-attention using a graph's tree decomposition. Instead of forming a full query-key bipartite graph, we restrict attention to vertices in parent, subtree, and same-depth bags of a vertex. This hierarchical context lends both sparsity and structure to vertex state updates. We apply dynamic programming to derive a forest of tree decompositions, choosing the most structurally similar tree to the AMR. Our system outperforms a self-attentive baseline by 1.6 BLEU and 1.8 chrF++.
ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection
Authors: Iftitahu Ni'mah, Meng Fang, Vlado Menkovski, Mykola Pechenizkiy
Date: 2021-08-27 00:00:00
Proceeding: Findings (EMNLP) 2021 11
Tasks: Few-Shot Text Classification, intent-classification, Intent Classification, Out of Distribution (OOD) Detection, text-classification, Text Classification, Zero-Shot Out-of-Domain Detection

The ability to detect Out-of-Domain (OOD) inputs has been a critical requirement in many real-world NLP applications. For example, intent classification in dialogue systems. The reason is that the inclusion of unsupported OOD inputs may lead to catastrophic failure of systems. However, it remains an empirical question whether current methods can tackle such problems reliably in a realistic scenario where zero OOD training data is available. In this study, we propose ProtoInfoMax, a new architecture that extends Prototypical Networks to simultaneously process in-domain and OOD sentences via Mutual Information Maximization (InfoMax) objective. Experimental results show that our proposed method can substantially improve performance up to 20% for OOD detection in low resource settings of text classification. We also show that ProtoInfoMax is less prone to typical overconfidence errors of Neural Networks, leading to more reliable prediction results.
Injecting Text in Self-Supervised Speech Pretraining
Authors: Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, Pedro Moreno
Date: 2021-08-27 00:00:00
Tasks: Contrastive Learning, Language Modelling, speech-recognition, Speech Recognition

Self-supervised pretraining for Automated Speech Recognition (ASR) has shown varied degrees of success. In this paper, we propose to jointly learn representations during pretraining from two different modalities: speech and text. The proposed method, tts4pretrain complements the power of contrastive learning in self-supervision with linguistic/lexical representations derived from synthesized speech, effectively learning from untranscribed speech and unspoken text. Lexical learning in the speech encoder is enforced through an additional sequence loss term that is coupled with contrastive loss during pretraining. We demonstrate that this novel pretraining method yields Word Error Rate (WER) reductions of 10% relative on the well-benchmarked, Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0 only. The proposed method also serves as an effective strategy to compensate for the lack of transcribed speech, effectively matching the performance of 5000 hours of transcribed speech with just 100 hours of transcribed speech on the AMI meeting transcription task. Finally, we demonstrate WER reductions of up to 15% on an in-house Voice Search task over traditional pretraining. Incorporating text into encoder pretraining is complimentary to rescoring with a larger or in-domain language model, resulting in additional 6% relative reduction in WER.
Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors
Authors: Ryo Nagata, Manabu Kimura, Kazuaki Hanawa
Date: 2021-08-27 00:00:00
Proceeding: Findings (ACL) 2022 5
Tasks: Grammatical Error Detection, Language Modeling, Language Modelling

In this paper, we explore the capacity of a language model-based method for grammatical error detection in detail. We first show that 5 to 10% of training data are enough for a BERT-based error detection method to achieve performance equivalent to a non-language model-based method can achieve with the full training data; recall improves much faster with respect to training data size in the BERT-based method than in the non-language model method while precision behaves similarly. These suggest that (i) the BERT-based method should have a good knowledge of grammar required to recognize certain types of error and that (ii) it can transform the knowledge into error detection rules by fine-tuning with a few training samples, which explains its high generalization ability in grammatical error detection. We further show with pseudo error data that it actually exhibits such nice properties in learning rules for recognizing various types of error. Finally, based on these findings, we explore a cost-effective method for detecting grammatical errors with feedback comments explaining relevant grammatical rules to learners.
Translation Error Detection as Rationale Extraction
Authors: Marina Fomicheva, Lucia Specia, Nikolaos Aletras
Date: 2021-08-27 00:00:00
Proceeding: Findings (ACL) 2022 5
Tasks: Sentence, Translation

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.
Query-Focused Extractive Summarisation for Finding Ideal Answers to Biomedical and COVID-19 Questions
Authors: Diego Mollá, Urvashi Khanna, Dima Galat, Vincent Nguyen, Maciej Rybinski
Date: 2021-08-27 00:00:00
Tasks: Question Answering, Sentence, Sentence Classification

This paper presents Macquarie University's participation to the BioASQ Synergy Task, and BioASQ9b Phase B. In each of these tasks, our participation focused on the use of query-focused extractive summarisation to obtain the ideal answers to medical questions. The Synergy Task is an end-to-end question answering task on COVID-19 where systems are required to return relevant documents, snippets, and answers to a given question. Given the absence of training data, we used a query-focused summarisation system that was trained with the BioASQ8b training data set and we experimented with methods to retrieve the documents and snippets. Considering the poor quality of the documents and snippets retrieved by our system, we observed reasonably good quality in the answers returned. For phase B of the BioASQ9b task, the relevant documents and snippets were already included in the test data. Our system split the snippets into candidate sentences and used BERT variants under a sentence classification setup. The system used the question and candidate sentence as input and was trained to predict the likelihood of the candidate sentence being part of the ideal answer. The runs obtained either the best or second best ROUGE-F1 results of all participants to all batches of BioASQ9b. This shows that using BERT in a classification setup is a very strong baseline for the identification of ideal answers.
Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling
Authors: Adeep Hande, Karthik Puranik, Konthala Yasaswini, Ruba Priyadharshini, Sajeetha Thavareesan, Anbukkarasi Sampath, Kogilavani Shanmugavadivel, Durairaj Thenmozhi, Bharathi Raja Chakravarthi
Date: 2021-08-27 00:00:00
Tasks: Language Identification, Marketing

Social media has effectively become the prime hub of communication and digital marketing. As these platforms enable the free manifestation of thoughts and facts in text, images and video, there is an extensive need to screen them to protect individuals and groups from offensive content targeted at them. Our work intends to classify codemixed social media comments/posts in the Dravidian languages of Tamil, Kannada, and Malayalam. We intend to improve offensive language identification by generating pseudo-labels on the dataset. A custom dataset is constructed by transliterating all the code-mixed texts into the respective Dravidian language, either Kannada, Malayalam, or Tamil and then generating pseudo-labels for the transliterated dataset. The two datasets are combined using the generated pseudo-labels to create a custom dataset called CMTRA. As Dravidian languages are under-resourced, our approach increases the amount of training data for the language models. We fine-tune several recent pretrained language models on the newly constructed dataset. We extract the pretrained language embeddings and pass them onto recurrent neural networks. We observe that fine-tuning ULMFiT on the custom dataset yields the best results on the code-mixed test sets of all three languages. Our approach yields the best results among the benchmarked models on Tamil-English, achieving a weighted F1-Score of 0.7934 while scoring competitive weighted F1-Scores of 0.9624 and 0.7306 on the code-mixed test sets of Malayalam-English and Kannada-English, respectively.
Secoco: Self-Correcting Encoding for Neural Machine Translation
Authors: Tao Wang, Chengqi Zhao, Mingxuan Wang, Lei LI, Hang Li, Deyi Xiong
Date: 2021-08-27 00:00:00
Proceeding: Findings (EMNLP) 2021 11
Tasks: Machine Translation, NMT, Translation

This paper presents Self-correcting Encoding (Secoco), a framework that effectively deals with input noise for robust neural machine translation by introducing self-correcting predictors. Different from previous robust approaches, Secoco enables NMT to explicitly correct noisy inputs and delete specific errors simultaneously with the translation decoding process. Secoco is able to achieve significant improvements over strong baselines on two real-world test sets and a benchmark WMT dataset with good interpretability. We will make our code and dataset publicly available soon.
Automated Generation of Accurate \& Fluent Medical X-ray Reports
Authors: Hoang T. N. Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, Li Cheng
Date: 2021-08-27 00:00:00

Our paper focuses on automating the generation of medical reports from chest X-ray image inputs, a critical yet time-consuming task for radiologists. Unlike existing medical re-port generation efforts that tend to produce human-readable reports, we aim to generate medical reports that are both fluent and clinically accurate. This is achieved by our fully differentiable and end-to-end paradigm containing three complementary modules: taking the chest X-ray images and clinical his-tory document of patients as inputs, our classification module produces an internal check-list of disease-related topics, referred to as enriched disease embedding; the embedding representation is then passed to our transformer-based generator, giving rise to the medical reports; meanwhile, our generator also pro-duces the weighted embedding representation, which is fed to our interpreter to ensure consistency with respect to disease-related topics.Our approach achieved promising results on commonly-used metrics concerning language fluency and clinical accuracy. Moreover, noticeable performance gains are consistently ob-served when additional input information is available, such as the clinical document and extra scans of different views.
Lingxi: A Diversity-aware Chinese Modern Poetry Generation System
Authors: Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li
Date: 2021-08-27 00:00:00
Tasks: Diversity, Semantic Similarity, Semantic Textual Similarity, Sentence, Text Generation

Poetry generation has been a difficult task in natural language processing. Unlike plain neural text generation tasks, poetry has a high requirement for novelty, since an easily-understood sentence with too many high frequency words might not be considered as poetic, while adequately ambiguous sentences with low frequency words can possibly be novel and creative. Inspired by this, we present Lingxi, a diversity-aware Chinese modern poetry generation system. We propose nucleus sampling with randomized head (NS-RH) algorithm, which randomizes the high frequency part ("head") of the predicted distribution, in order to emphasize on the "comparatively low frequency" words. The proposed algorithm can significantly increase the novelty of generated poetry compared with traditional sampling methods. The permutation of distribution is controllable by tuning the filtering parameter that determines the "head" to permutate, achieving diversity-aware sampling. We find that even when a large portion of filtered vocabulary is randomized, it can actually generate fluent poetry but with notably higher novelty. We also propose a semantic-similarity-based rejection sampling algorithm, which creates longer and more informative context on the basis of the short input poetry title while maintaining high semantic similarity to the title, alleviating the off-topic issue.
EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa
Authors: Taewoon Kim, Piek Vossen
Date: 2021-08-26 00:00:00
Tasks: Emotion Recognition, Emotion Recognition in Conversation

We present EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa, a simple yet expressive scheme of solving the ERC (emotion recognition in conversation) task. By simply prepending speaker names to utterances and inserting separation tokens between the utterances in a dialogue, EmoBERTa can learn intra- and inter- speaker states and context to predict the emotion of a current speaker, in an end-to-end manner. Our experiments show that we reach a new state of the art on the two popular ERC datasets using a basic and straight-forward approach. We've open sourced our code and models at https://github.com/tae898/erc.
Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization
Authors: Chujie Zheng, Kunpeng Zhang, Harry Jiannan Wang, Ling Fan, Zhe Wang
Date: 2021-08-26 00:00:00
Tasks: Abstractive Text Summarization, Contrastive Learning, Decoder, Denoising, Sentence, Text Summarization

In this paper, we present a denoising sequence-to-sequence (seq2seq) autoencoder via contrastive learning for abstractive text summarization. Our model adopts a standard Transformer-based architecture with a multi-layer bi-directional encoder and an auto-regressive decoder. To enhance its denoising ability, we incorporate self-supervised contrastive learning along with various sentence-level document augmentation. These two components, seq2seq autoencoder and contrastive learning, are jointly trained through fine-tuning, which improves the performance of text summarization with regard to ROUGE scores and human evaluation. We conduct experiments on two datasets and demonstrate that our model outperforms many existing benchmarks and even achieves comparable performance to the state-of-the-art abstractive systems trained with more complex architecture and extensive computation resources.
Artificial Neural Networks Based Analysis of BLDC Motor Speed Control
Authors: Porselvi T, Sai Ganesh CS, Aouthithiye Barathwaj SR Y
Date: 2021-08-27 00:00:00

Artificial Neural Network (ANN) is a simple network that has an input, an output, and numerous hidden layers with a set of nodes. Implementation of ANN algorithms in electrical, and electronics engineering always satisfies with the expected results as ANN handles binary data more accurately. Brushless Direct Current motor (BLDC motor) uses electronic closed-loop controllers to switch DC current to the motor windings and produces the magnetic fields. The BLDC motor finds various applications owing to its high speed, low maintenance and adequate torque capability. They are highly preferred than the other motors because of their better performance and it is easy to control their speed by Power Converters. This article presents a method of speed control of BLDC motors where speed is controlled by changing the DC input voltage of the bridge converter that feeds the motor winding. The control is done by using a PI based speed controller. The motor is modeled in the MATLAB/Simulink and the speed control is obtained with a PI controller. EMF signals, rotor speed, electromagnetic torque, Hall Effect signals, PWM and EMF signals simulations are then obtained. This acquired data is then fed into binary artificial neural networks and as a result, the ANN model predicts the corresponding parameters close to the simulation results. Both the mathematical based simulation and data based prediction gives satisfactory results
Recent Developments in Program Synthesis with Evolutionary Algorithms
Authors: Dominik Sobania, Dirk Schweim, Franz Rothlauf
Date: 2021-08-27 00:00:00
Tasks: Evolutionary Algorithms, Program Synthesis

The automatic generation of computer programs is one of the main applications with practical relevance in the field of evolutionary computation. With program synthesis techniques not only software developers could be supported in their everyday work but even users without any programming knowledge could be empowered to automate repetitive tasks and implement their own new functionality. In recent years, many novel program synthesis approaches based on evolutionary algorithms have been proposed and evaluated on common benchmark problems. Therefore, we identify in this work the relevant evolutionary program synthesis approaches and provide an in-depth analysis of their performance. The most influential approaches we identify are stack-based, grammar-guided, as well as linear genetic programming. Further, we find that these approaches perform well on benchmark problems if there is a simple mapping from the given input to the correct output. On problems where this mapping is complex, e.g., if the problem consists of several sub-problems or requires iteration/recursion for a correct solution, results tend to be worse. Consequently, for future work, we encourage researchers not only to use a program's output for assessing the quality of a solution but also the way towards a solution (e.g., correctly solved sub-problems).
Quantum Sub-Gaussian Mean Estimator
Authors: Yassine Hamoudi
Date: 2021-08-27 00:00:00

We present a new quantum algorithm for estimating the mean of a real-valued random variable obtained as the output of a quantum computation. Our estimator achieves a nearly-optimal quadratic speedup over the number of classical i.i.d. samples needed to estimate the mean of a heavy-tailed distribution with a sub-Gaussian error rate. This result subsumes (up to logarithmic factors) earlier works on the mean estimation problem that were not optimal for heavy-tailed distributions [BHMT02,BDGT11], or that require prior information on the variance [Hein02,Mon15,HM19]. As an application, we obtain new quantum algorithms for the $(\epsilon,\delta)$-approximation problem with an optimal dependence on the coefficient of variation of the input random variable.
Dealing with Typos for BERT-based Passage Retrieval and Ranking
Authors: Shengyao Zhuang, Guido Zuccon
Date: 2021-08-27 00:00:00
Proceeding: EMNLP 2021 11
Tasks: Information Retrieval, Language Modeling, Language Modelling, Open-Domain Question Answering, Passage Ranking, Passage Re-Ranking, Passage Retrieval, Question Answering, Re-Ranking, Retrieval

Passage retrieval and ranking is a key task in open-domain question answering and information retrieval. Current effective approaches mostly rely on pre-trained deep language model-based retrievers and rankers. These methods have been shown to effectively model the semantic matching between queries and passages, also in presence of keyword mismatch, i.e. passages that are relevant to a query but do not contain important query keywords. In this paper we consider the Dense Retriever (DR), a passage retrieval method, and the BERT re-ranker, a popular passage re-ranking method. In this context, we formally investigate how these models respond and adapt to a specific type of keyword mismatch -- that caused by keyword typos occurring in queries. Through empirical investigation, we find that typos can lead to a significant drop in retrieval and ranking effectiveness. We then propose a simple typos-aware training framework for DR and BERT re-ranker to address this issue. Our experimental results on the MS MARCO passage ranking dataset show that, with our proposed typos-aware training, DR and BERT re-ranker can become robust to typos in queries, resulting in significantly improved effectiveness compared to models trained without appropriately accounting for typos.
Detecting Propaganda on the Sentence Level during the COVID-19 Pandemic
Authors: Rong-Ching Chang, Chu-Hsing Lin
Date: 2021-07-31 00:00:00
Tasks: Misinformation, Sentence

The spread of misinformation, conspiracy, and questionable content and information manipulation by foreign adversaries on social media has surged along with the COVID-19 pandemic. Such malicious cyber-enabled actions may cause increasing social polarization, health crises, and property loss. In this paper, using fine-tuned contextualized embedding trained on Reddit, we tackle the detection of the propaganda of such user accounts and their targeted issues on Twitter during March 2020 when the COVID-19 epidemic became recognized as a pandemic. Our result shows that the pro-China group appeared to be tweeting 35 to 115 times more than the neutral group. At the same time, neutral groups were tweeting more positive-attitude content and voicing alarm for the COVID-19 situation. The pro-China group was also using more call-for-action words on political issues not necessarily China-related.