pwc-archive/papers-with-abstracts huggingface dataset:
EduQG: A Multi-format Multiple Choice Dataset for the Educational Domain
Authors: Amir Hadifar, Semere Kiros Bitew, Johannes Deleu, Chris Develder, Thomas Demeester
Date: 2022-10-12 00:00:00
Tasks: Distractor Generation, Multiple-choice, Question Generation, Question-Generation, Sentence

We introduce a high-quality dataset that contains 3,397 samples comprising (i) multiple choice questions, (ii) answers (including distractors), and (iii) their source documents, from the educational domain. Each question is phrased in two forms, normal and close. Correct answers are linked to source documents with sentence-level annotations. Thus, our versatile dataset can be used for both question and distractor generation, as well as to explore new challenges such as question format conversion. Furthermore, 903 questions are accompanied by their cognitive complexity level as per Bloom's taxonomy. All questions have been generated by educational experts rather than crowd workers to ensure they are maintaining educational and learning standards. Our analysis and experiments suggest distinguishable differences between our dataset and commonly used ones for question generation for educational purposes. We believe this new dataset can serve as a valuable resource for research and evaluation in the educational domain. The dataset and baselines will be released to support further research in question generation.
Summary on the ISCSLP 2022 Chinese-English Code-Switching ASR Challenge
Authors: Shuhao Deng, Chengfei Li, Jinfeng Bai, Qingqing Zhang, Wei-Qiang Zhang, Runyan Yang, Gaofeng Cheng, Pengyuan Zhang, Yonghong Yan
Date: 2022-10-12 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Code-switching automatic speech recognition becomes one of the most challenging and the most valuable scenarios of automatic speech recognition, due to the code-switching phenomenon between multilingual language and the frequent occurrence of code-switching phenomenon in daily life. The ISCSLP 2022 Chinese-English Code-Switching Automatic Speech Recognition (CSASR) Challenge aims to promote the development of code-switching automatic speech recognition. The ISCSLP 2022 CSASR challenge provided two training sets, TAL_CSASR corpus and MagicData-RAMC corpus, a development and a test set for participants, which are used for CSASR model training and evaluation. Along with the challenge, we also provide the baseline system performance for reference. As a result, more than 40 teams participated in this challenge, and the winner team achieved 16.70% Mixture Error Rate (MER) performance on the test set and has achieved 9.8% MER absolute improvement compared with the baseline system. In this paper, we will describe the datasets, the associated baselines system and the requirements, and summarize the CSASR challenge results and major techniques and tricks used in the submitted systems.
Integrating Translation Memories into Non-Autoregressive Machine Translation
Authors: Jitao Xu, Josep Crego, François Yvon
Date: 2022-10-12 00:00:00
Tasks: Knowledge Distillation, Machine Translation, Translation

Non-autoregressive machine translation (NAT) has recently made great progress. However, most works to date have focused on standard translation tasks, even though some edit-based NAT models, such as the Levenshtein Transformer (LevT), seem well suited to translate with a Translation Memory (TM). This is the scenario considered here. We first analyze the vanilla LevT model and explain why it does not do well in this setting. We then propose a new variant, TM-LevT, and show how to effectively train this model. By modifying the data presentation and introducing an extra deletion operation, we obtain performance that are on par with an autoregressive approach, while reducing the decoding load. We also show that incorporating TMs during training dispenses to use knowledge distillation, a well-known trick used to mitigate the multimodality issue.
Improving Graph-Based Text Representations with Character and Word Level N-grams
Authors: Wenzhe Li, Nikolaos Aletras
Date: 2022-10-12 00:00:00
Tasks: Graph Representation Learning, Representation Learning, text-classification, Text Classification, Text Summarization

Graph-based text representation focuses on how text documents are represented as graphs for exploiting dependency information between tokens and documents within a corpus. Despite the increasing interest in graph representation learning, there is limited research in exploring new ways for graph-based text representation, which is important in downstream natural language processing tasks. In this paper, we first propose a new heterogeneous word-character text graph that combines word and character n-gram nodes together with document nodes, allowing us to better learn dependencies among these entities. Additionally, we propose two new graph-based neural models, WCTextGCN and WCTextGAT, for modeling our proposed text graph. Extensive experiments in text classification and automatic text summarization benchmarks demonstrate that our proposed models consistently outperform competitive baselines and state-of-the-art graph-based models.
Step out of KG: Knowledge Graph Completion via Knowledgeable Retrieval and Reading Comprehension
Authors: Xin Lv, Yankai Lin, Zijun Yao, Kaisheng Zeng, Jiajie Zhang, Lei Hou, Juanzi Li
Date: 2022-10-12 00:00:00
Tasks: Information Retrieval, Knowledge Graph Completion, Knowledge Graphs, Reading Comprehension, Retrieval

Knowledge graphs, as the cornerstone of many AI applications, usually face serious incompleteness problems. In recent years, there have been many efforts to study automatic knowledge graph completion (KGC), most of which use existing knowledge to infer new knowledge. However, in our experiments, we find that not all relations can be obtained by inference, which constrains the performance of existing models. To alleviate this problem, we propose a new model based on information retrieval and reading comprehension, namely IR4KGC. Specifically, we pre-train a knowledge-based information retrieval module that can retrieve documents related to the triples to be completed. Then, the retrieved documents are handed over to the reading comprehension module to generate the predicted answers. In experiments, we find that our model can well solve relations that cannot be inferred from existing knowledge, and achieve good results on KGC datasets.
Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion
Authors: Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srinivas, Greg Durrett, Junyi Jessy Li
Date: 2022-10-12 00:00:00
Tasks: Dependency Parsing, Question Answering, Sentence

Automatic discourse processing is bottlenecked by data: current discourse formalisms pose highly demanding annotation tasks involving large taxonomies of discourse relations, making them inaccessible to lay annotators. This work instead adopts the linguistic framework of Questions Under Discussion (QUD) for discourse analysis and seeks to derive QUD structures automatically. QUD views each sentence as an answer to a question triggered in prior context; thus, we characterize relationships between sentences as free-form questions, in contrast to exhaustive fine-grained taxonomies. We develop the first-of-its-kind QUD parser that derives a dependency structure of questions over full documents, trained using a large, crowdsourced question-answering dataset DCQA (Ko et al., 2022). Human evaluation results show that QUD dependency parsing is possible for language models trained with this crowdsourced, generalizable annotation scheme. We illustrate how our QUD structure is distinct from RST trees, and demonstrate the utility of QUD analysis in the context of document simplification. Our findings show that QUD parsing is an appealing alternative for automatic discourse processing.
Zero-Shot Prompting for Implicit Intent Prediction and Recommendation with Commonsense Reasoning
Authors: Hui-Chi Kuo, Yun-Nung Chen
Date: 2022-10-12 00:00:00
Tasks: Language Modeling, Language Modelling

Intelligent virtual assistants are currently designed to perform tasks or services explicitly mentioned by users, so multiple related domains or tasks need to be performed one by one through a long conversation with many explicit intents. Instead, human assistants are capable of reasoning (multiple) implicit intents based on user utterances via commonsense knowledge, reducing complex interactions and improving practicality. Therefore, this paper proposes a framework of multi-domain dialogue systems, which can automatically infer implicit intents based on user utterances and then perform zero-shot prompting using a large pre-trained language model to trigger suitable single task-oriented bots. The proposed framework is demonstrated effective to realize implicit intents and recommend associated bots in a zero-shot manner.
AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning
Authors: Tao Yang, Jinghao Deng, Xiaojun Quan, Qifan Wang, Shaoliang Nie
Date: 2022-10-12 00:00:00
Tasks: Language Modeling, Language Modelling

Fine-tuning large pre-trained language models on downstream tasks is apt to suffer from overfitting when limited training data is available. While dropout proves to be an effective antidote by randomly dropping a proportion of units, existing research has not examined its effect on the self-attention mechanism. In this paper, we investigate this problem through self-attention attribution and find that dropping attention positions with low attribution scores can accelerate training and increase the risk of overfitting. Motivated by this observation, we propose Attribution-Driven Dropout (AD-DROP), which randomly discards some high-attribution positions to encourage the model to make predictions by relying more on low-attribution positions to reduce overfitting. We also develop a cross-tuning strategy to alternate fine-tuning and AD-DROP to avoid dropping high-attribution positions excessively. Extensive experiments on various benchmarks show that AD-DROP yields consistent improvements over baselines. Analysis further confirms that AD-DROP serves as a strategic regularizer to prevent overfitting during fine-tuning.
MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score
Authors: Sunjae Kwon, Zonghai Yao, Harmon S. Jordan, David A. Levy, Brian Corner, Hong Yu
Date: 2022-10-12 00:00:00
Tasks: Articles, Language Modeling, Language Modelling, model, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

This paper proposes a new natural language processing (NLP) application for identifying medical jargon terms potentially difficult for patients to comprehend from electronic health record (EHR) notes. We first present a novel and publicly available dataset with expert-annotated medical jargon terms from 18K+ EHR note sentences ($MedJ$). Then, we introduce a novel medical jargon extraction ($MedJEx$) model which has been shown to outperform existing state-of-the-art NLP models. First, MedJEx improved the overall performance when it was trained on an auxiliary Wikipedia hyperlink span dataset, where hyperlink spans provide additional Wikipedia articles to explain the spans (or terms), and then fine-tuned on the annotated MedJ data. Secondly, we found that a contextualized masked language model score was beneficial for detecting domain-specific unfamiliar jargon terms. Moreover, our results show that training on the auxiliary Wikipedia hyperlink span datasets improved six out of eight biomedical named entity recognition benchmark datasets. Both MedJ and MedJEx are publicly available.
SEAL : Interactive Tool for Systematic Error Analysis and Labeling
Authors: Nazneen Rajani, Weixin Liang, Lingjiao Chen, Meg Mitchell, James Zou
Date: 2022-10-11 00:00:00

With the advent of Transformers, large language models (LLMs) have saturated well-known NLP benchmarks and leaderboards with high aggregate performance. However, many times these models systematically fail on tail data or rare groups not obvious in aggregate evaluation. Identifying such problematic data groups is even more challenging when there are no explicit labels (e.g., ethnicity, gender, etc.) and further compounded for NLP datasets due to the lack of visual features to characterize failure modes (e.g., Asian males, animals indoors, waterbirds on land, etc.). This paper introduces an interactive Systematic Error Analysis and Labeling (\seal) tool that uses a two-step approach to first identify high error slices of data and then, in the second step, introduce methods to give human-understandable semantics to those underperforming slices. We explore a variety of methods for coming up with coherent semantics for the error groups using language models for semantic labeling and a text-to-image model for generating visual features. SEAL toolkit and demo screencast is available at https://huggingface.co/spaces/nazneen/seal.
CLIP also Understands Text: Prompting CLIP for Phrase Understanding
Authors: An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, Julian McAuley
Date: 2022-10-11 00:00:00
Tasks: Clustering, Transfer Learning

Contrastive Language-Image Pretraining (CLIP) efficiently learns visual concepts by pre-training with natural language supervision. CLIP and its visual encoder have been explored on various vision and language tasks and achieve strong zero-shot or transfer learning performance. However, the application of its text encoder solely for text understanding has been less explored. In this paper, we find that the text encoder of CLIP actually demonstrates strong ability for phrase understanding, and can even significantly outperform popular language models such as BERT with a properly designed prompt. Extensive experiments validate the effectiveness of our method across different datasets and domains on entity clustering and entity set expansion tasks.
Scaling Up Deliberation for Multilingual ASR
Authors: Ke Hu, Bo Li, Tara N. Sainath
Date: 2022-10-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Decoder, speech-recognition, Speech Recognition

Multilingual end-to-end automatic speech recognition models are attractive due to its simplicity in training and deployment. Recent work on large-scale training of such models has shown promising results compared to monolingual models. However, the work often focuses on multilingual models themselves in a single-pass setup. In this work, we investigate second-pass deliberation for multilingual speech recognition. Our proposed deliberation is multilingual, i.e., the text encoder encodes hypothesis text from multiple languages, and the decoder attends to multilingual text and audio. We investigate scaling the deliberation text encoder and decoder, and compare scaling the deliberation decoder and the first-pass cascaded encoder. We show that deliberation improves the average WER on 9 languages by 4% relative compared to the single-pass model. By increasing the size of the deliberation up to 1B parameters, the average WER improvement increases to 9%, with up to 14% for certain languages. Our deliberation rescorer is based on transformer layers and can be parallelized during rescoring.
Cross-Lingual Speaker Identification Using Distant Supervision
Authors: Ben Zhou, Dian Yu, Dong Yu, Dan Roth
Date: 2022-10-11 00:00:00
Tasks: Language Modeling, Language Modelling, Speaker Identification

Speaker identification, determining which character said each utterance in literary text, benefits many downstream tasks. Most existing approaches use expert-defined rules or rule-based features to directly approach this task, but these approaches come with significant drawbacks, such as lack of contextual reasoning and poor cross-lingual generalization. In this work, we propose a speaker identification framework that addresses these issues. We first extract large-scale distant supervision signals in English via general-purpose tools and heuristics, and then apply these weakly-labeled instances with a focus on encouraging contextual reasoning to train a cross-lingual language model. We show that the resulting model outperforms previous state-of-the-art methods on two English speaker identification benchmarks by up to 9% in accuracy and 5% with only distant supervision, as well as two Chinese speaker identification datasets by up to 4.7%.
Detecting Propagators of Disinformation on Twitter Using Quantitative Discursive Analysis
Authors: Mark M. Bailey
Date: 2022-10-11 00:00:00
Tasks: Community Detection

Efforts by foreign actors to influence public opinion have gained considerable attention because of their potential to impact democratic elections. Thus, the ability to identify and counter sources of disinformation is increasingly becoming a top priority for government entities in order to protect the integrity of democratic processes. This study presents a method of identifying Russian disinformation bots on Twitter using centering resonance analysis and Clauset-Newman-Moore community detection. The data reflect a significant degree of discursive dissimilarity between known Russian disinformation bots and a control set of Twitter users during the timeframe of the 2016 U.S. Presidential Election. The data also demonstrate statistically significant classification capabilities (MCC = 0.9070) based on community clustering. The prediction algorithm is very effective at identifying true positives (bots), but is not able to resolve true negatives (non-bots) because of the lack of discursive similarity between control users. This leads to a highly sensitive means of identifying propagators of disinformation with a high degree of discursive similarity on Twitter, with implications for limiting the spread of disinformation that could impact democratic processes.
Streaming Punctuation for Long-form Dictation with Transformers
Authors: Piyush Behre, Sharman Tan, Padma Varadharajan, Shuangyu Chang
Date: 2022-10-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Form, Machine Translation, Segmentation, speech-recognition, Speech Recognition

While speech recognition Word Error Rate (WER) has reached human parity for English, long-form dictation scenarios still suffer from segmentation and punctuation problems resulting from irregular pausing patterns or slow speakers. Transformer sequence tagging models are effective at capturing long bi-directional context, which is crucial for automatic punctuation. Automatic Speech Recognition (ASR) production systems, however, are constrained by real-time requirements, making it hard to incorporate the right context when making punctuation decisions. In this paper, we propose a streaming approach for punctuation or re-punctuation of ASR output using dynamic decoding windows and measure its impact on punctuation and segmentation accuracy across scenarios. The new system tackles over-segmentation issues, improving segmentation F0.5-score by 13.9%. Streaming punctuation achieves an average BLEU-score improvement of 0.66 for the downstream task of Machine Translation (MT).
Automatic Speech Recognition of Low-Resource Languages Based on Chukchi
Authors: Anastasia Safonova, Tatiana Yudina, Emil Nadimanov, Cydnie Davenport
Date: 2022-10-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

The following paper presents a project focused on the research and creation of a new Automatic Speech Recognition (ASR) based in the Chukchi language. There is no one complete corpus of the Chukchi language, so most of the work consisted in collecting audio and texts in the Chukchi language from open sources and processing them. We managed to collect 21:34:23 hours of audio recordings and 112,719 sentences (or 2,068,273 words) of text in the Chukchi language. The XLSR model was trained on the obtained data, which showed good results even with a small amount of data. Besides the fact that the Chukchi language is a low-resource language, it is also polysynthetic, which significantly complicates any automatic processing. Thus, the usual WER metric for evaluating ASR becomes less indicative for a polysynthetic language. However, the CER metric showed good results. The question of metrics for polysynthetic languages remains open.
Measuring and Improving Semantic Diversity of Dialogue Generation
Authors: Seungju Han, Beomsu Kim, Buru Chang
Date: 2022-10-11 00:00:00
Tasks: Dialogue Generation, Diversity

Response diversity has become an important criterion for evaluating the quality of open-domain dialogue generation models. However, current evaluation metrics for response diversity often fail to capture the semantic diversity of generated responses, as they mainly consider lexical aspects of the generated responses. In this paper, we introduce a new automatic evaluation metric to measure the semantic diversity of generated responses. Through human evaluation, we demonstrate that our proposed metric captures human judgments on response diversity better than existing lexical-level diversity metrics. Furthermore, motivated by analyzing an existing dialogue dataset, we propose a simple yet effective learning method that improves the semantic diversity of generated responses. Our learning method weights training samples based on the semantic distribution of the training set. We show that our learning method improves response diversity and coherency better than other baseline methods through automatic and human evaluation.
Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment
Authors: César González-Gutiérrez, Audi Primadhanty, Francesco Cazzaro, Ariadna Quattoni
Date: 2022-10-11 00:00:00
Tasks: text-classification, Text Classification

Annotating large collections of textual data can be time consuming and expensive. That is why the ability to train models with limited annotation budgets is of great importance. In this context, it has been shown that under tight annotation budgets the choice of data representation is key. The goal of this paper is to better understand why this is so. With this goal in mind, we propose a metric that measures the extent to which a given representation is structurally aligned with a task. We conduct experiments on several text classification datasets testing a variety of models and representations. Using our proposed metric we show that an efficient representation for a task (i.e. one that enables learning from few samples) is a representation that induces a good alignment between latent input structure and class structure.
Exploring the optimality of approximate state preparation quantum circuits with a genetic algorithm
Authors: Tom Rindell, Berat Yenilen, Niklas Halonen, Arttu Pönni, Ilkka Tittonen, Matti Raasakka
Date: 2022-10-12 00:00:00

We study the approximate state preparation problem on noisy intermediate-scale quantum (NISQ) computers by applying a genetic algorithm to generate quantum circuits for state preparation. The algorithm can account for the specific characteristics of the physical machine in the evaluation of circuits, such as the native gate set and qubit connectivity. We use our genetic algorithm to optimize the circuits provided by the low-rank state preparation algorithm introduced by Araujo et al., and find substantial improvements to the fidelity in preparing Haar random states with a limited number of CNOT gates. Moreover, we observe that already for a 5-qubit quantum processor with limited qubit connectivity and significant noise levels (IBM Falcon 5T), the maximal fidelity for Haar random states is achieved by a short approximate state preparation circuit instead of the exact preparation circuit. We also present a theoretical analysis of approximate state preparation circuit complexity to motivate our findings. Our genetic algorithm for quantum circuit discovery is freely available at https://github.com/beratyenilen/qc-ga .