pwc-archive/papers-with-abstracts huggingface dataset:
The Cross-lingual Conversation Summarization Challenge
Authors: Yulong Chen, Ming Zhong, Xuefeng Bai, Naihao Deng, Jing Li, Xianchao Zhu, Yue Zhang
Date: 2022-05-01 00:00:00
Tasks: Abstractive Dialogue Summarization, Conversation Summarization, Cross-Lingual Abstractive Summarization, Machine Translation, Text Summarization, Translation

We propose the shared task of cross-lingual conversation summarization, \emph{ConvSumX Challenge}, opening new avenues for researchers to investigate solutions that integrate conversation summarization and machine translation. This task can be particularly useful due to the emergence of online meetings and conferences. We construct a new benchmark, covering 2 real-world scenarios and 3 language directions, including a low-resource language. We hope that \emph{ConvSumX} can motivate researches to go beyond English and break the barrier for non-English speakers to benefit from recent advances of conversation summarization.
Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs
Authors: Xu Wang, Simin Fan, Jessica Houghton, Lu Wang
Date: 2022-04-30 00:00:00
Proceeding: NAACL 2022 7
Tasks: Misconceptions, Question Generation, Question-Generation

NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators' time and benefiting student learning. Yet, QG systems have not been widely adopted in classrooms to date. In this work, we aim to pinpoint key impediments and investigate how to improve the usability of automatic QG techniques for educational purposes by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models. We perform an in-depth need finding study with 11 instructors across 7 different universities, and summarize their thought processes and needs when creating questions. While instructors show great interests in using NLP systems to support question design, none of them has used such tools in practice. They resort to multiple sources of information, ranging from domain knowledge to students' misconceptions, all of which missing from today's QG systems. We argue that building effective human-NLP collaborative QG systems that emphasize instructor control and explainability is imperative for real-world adoption. We call for QG systems to provide process-oriented support, use modular design, and handle diverse sources of input.
HateCheckHIn: Evaluating Hindi Hate Speech Detection Models
Authors: Mithun Das, Punyajoy Saha, Binny Mathew, Animesh Mukherjee
Date: 2022-04-30 00:00:00
Proceeding: LREC 2022 6
Tasks: Diagnostic, Hate Speech Detection

Due to the sheer volume of online hate, the AI and NLP communities have started building models to detect such hateful content. Recently, multilingual hate is a major emerging challenge for automated detection where code-mixing or more than one language have been used for conversation in social media. Typically, hate speech detection models are evaluated by measuring their performance on the held-out test data using metrics such as accuracy and F1-score. While these metrics are useful, it becomes difficult to identify using them where the model is failing, and how to resolve it. To enable more targeted diagnostic insights of such multilingual hate speech models, we introduce a set of functionalities for the purpose of evaluation. We have been inspired to design this kind of functionalities based on real-world conversation on social media. Considering Hindi as a base language, we craft test cases for each functionality. We name our evaluation dataset HateCheckHIn. To illustrate the utility of these functionalities , we test state-of-the-art transformer based m-BERT model and the Perspective API.
Detoxifying Language Models with a Toxic Corpus
Authors: Yoon A Park, Frank Rudzicz
Date: 2022-04-30 00:00:00
Proceeding: LTEDI (ACL) 2022 5
Tasks: Text Generation

Existing studies have investigated the tendency of autoregressive language models to generate contexts that exhibit undesired biases and toxicity. Various debiasing approaches have been proposed, which are primarily categorized into data-based and decoding-based. In our study, we investigate the ensemble of the two debiasing paradigms, proposing to use toxic corpus as an additional resource to reduce the toxicity. Our result shows that toxic corpus can indeed help to reduce the toxicity of the language generation process substantially, complementing the existing debiasing methods.
Clues Before Answers: Generation-Enhanced Multiple-Choice QA
Authors: Zixian Huang, Ao Wu, Jiaying Zhou, Yu Gu, Yue Zhao, Gong Cheng
Date: 2022-04-30 00:00:00
Proceeding: NAACL 2022 7
Tasks: Decoder, Multiple-choice, Multiple Choice Question Answering (MCQA), Question Answering

A trending paradigm for multiple-choice question answering (MCQA) is using a text-to-text framework. By unifying data in different tasks into a single text-to-text format, it trains a generative encoder-decoder model which is both powerful and universal. However, a side effect of twisting a generation target to fit the classification nature of MCQA is the under-utilization of the decoder and the knowledge that can be decoded. To exploit the generation capability and underlying knowledge of a pre-trained encoder-decoder model, in this paper, we propose a generation-enhanced MCQA model named GenMC. It generates a clue from the question and then leverages the clue to enhance a reader for MCQA. It outperforms text-to-text models on multiple MCQA datasets.
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Authors: Ivan Vulić, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, Anna Korhonen
Date: 2022-04-30 00:00:00
Tasks: Contrastive Learning, Cross-Lingual Entity Linking, Entity Linking, Pretrained Multilingual Language Models, Semantic Similarity, Semantic Textual Similarity, Sentence, Word Translation

Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be 'rewired' into effective cross-lingual lexical encoders via the contrastive learning procedure, and that they contain more cross-lingual lexical knowledge than what 'meets the eye' when they are used as off-the-shelf SEs. This way, we also provide an effective tool for harnessing 'covert' multilingual lexical knowledge hidden in multilingual sentence encoders.
EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing
Authors: Chengyu Wang, Minghui Qiu, Chen Shi, Taolin Zhang, Tingting Liu, Lei LI, Jianing Wang, Ming Wang, Jun Huang, Wei Lin
Date: 2022-04-30 00:00:00
Tasks: Few-Shot Learning, Knowledge Distillation

The success of Pre-Trained Models (PTMs) has reshaped the development of Natural Language Processing (NLP). Yet, it is not easy to obtain high-performing models and deploy them online for industrial practitioners. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale PTMs, and provides a unified framework of model training, inference and deployment for real-world applications. Currently, EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. The source code of our EasyNLP toolkit is released at GitHub (https://github.com/alibaba/EasyNLP).
A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammar Error Correction
Authors: Nankai Lin, Xiaotian Lin, Ziyu Yang, Shengyi Jiang
Date: 2022-04-30 00:00:00
Tasks: Chinese Word Segmentation, Grammatical Error Correction, Segmentation, Sentence

As a fundamental task in natural language processing, Chinese Grammatical Error Correction (CGEC) has gradually received widespread attention and become a research hotspot. However, one obvious deficiency for the existing CGEC evaluation system is that the evaluation values are significantly influenced by the Chinese word segmentation results or different language models. The evaluation values of the same error correction model can vary considerably under different word segmentation systems or different language models. However, it is expected that these metrics should be independent of the word segmentation results and language models, as they may lead to a lack of uniqueness and comparability in the evaluation of different methods. To this end, we propose three novel evaluation metrics for CGEC in two dimensions: reference-based and reference-less. In terms of the reference-based metric, we introduce sentence-level accuracy and char-level BLEU to evaluate the corrected sentences. Besides, in terms of the reference-less metric, we adopt char-level meaning preservation to measure the semantic preservation degree of the corrected sentences. We deeply evaluate and analyze the reasonableness and validity of the three proposed metrics, and we expect them to become a new standard for CGEC.
Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers
Authors: Vivek Kumar, Rishabh Maheshwary, Vikram Pudi
Date: 2022-04-30 00:00:00
Proceeding: NAACL 2022 7
Tasks: Data Augmentation, Diversity, Math

Existing Math Word Problem (MWP) solvers have achieved high accuracy on benchmark datasets. However, prior works have shown that such solvers do not generalize well and rely on superficial cues to achieve high performance. In this paper, we first conduct experiments to showcase that this behaviour is mainly associated with the limited size and diversity present in existing MWP datasets. Next, we propose several data augmentation techniques broadly categorized into Substitution and Paraphrasing based methods. By deploying these methods we increase the size of existing datasets by five folds. Extensive experiments on two benchmark datasets across three state-of-the-art MWP solvers show that proposed methods increase the generalization and robustness of existing solvers. On average, proposed methods significantly increase the state-of-the-art results by over five percentage points on benchmark datasets. Further, the solvers trained on the augmented dataset perform comparatively better on the challenge test set. We also show the effectiveness of proposed techniques through ablation studies and verify the quality of augmented samples through human evaluation.
Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
Authors: Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang, Sang-Woo Lee, WooMyoung Park
Date: 2022-04-30 00:00:00
Proceeding: NAACL 2022 7
Tasks: Few-Shot Learning

Recent open-domain dialogue models have brought numerous breakthroughs. However, building a chat system is not scalable since it often requires a considerable volume of human-human dialogue data, especially when enforcing features such as persona, style, or safety. In this work, we study the challenge of imposing roles on open-domain dialogue systems, with the goal of making the systems maintain consistent roles while conversing naturally with humans. To accomplish this, the system must satisfy a role specification that includes certain conditions on the stated features as well as a system policy on whether or not certain types of utterances are allowed. For this, we propose an efficient data collection framework leveraging in-context few-shot learning of large-scale language models for building role-satisfying dialogue dataset from scratch. We then compare various architectures for open-domain dialogue systems in terms of meeting role specifications while maintaining conversational abilities. Automatic and human evaluations show that our models return few out-of-bounds utterances, keeping competitive performance on general metrics. We release a Korean dialogue dataset we built for further research.
To Know by the Company Words Keep and What Else Lies in the Vicinity
Authors: Jake Ryland Williams, Hunter Scott Heidenreich
Date: 2022-04-30 00:00:00

The development of state-of-the-art (SOTA) Natural Language Processing (NLP) systems has steadily been establishing new techniques to absorb the statistics of linguistic data. These techniques often trace well-known constructs from traditional theories, and we study these connections to close gaps around key NLP methods as a means to orient future work. For this, we introduce an analytic model of the statistics learned by seminal algorithms (including GloVe and Word2Vec), and derive insights for systems that use these algorithms and the statistics of co-occurrence, in general. In this work, we derive -- to the best of our knowledge -- the first known solution to Word2Vec's softmax-optimized, skip-gram algorithm. This result presents exciting potential for future development as a direct solution to a deep learning (DL) language model's (LM's) matrix factorization. However, we use the solution to demonstrate a seemingly-universal existence of a property that word vectors exhibit and which allows for the prophylactic discernment of biases in data -- prior to their absorption by DL models. To qualify our work, we conduct an analysis of independence, i.e., on the density of statistical dependencies in co-occurrence models, which in turn renders insights on the distributional hypothesis' partial fulfillment by co-occurrence statistics.
Answer Consolidation: Formulation and Benchmarking
Authors: Wenxuan Zhou, Qiang Ning, Heba Elfardy, Kevin Small, Muhao Chen
Date: 2022-04-29 00:00:00
Proceeding: NAACL 2022 7
Tasks: Benchmarking, Question Answering

Current question answering (QA) systems primarily consider the single-answer scenario, where each question is assumed to be paired with one correct answer. However, in many real-world QA applications, multiple answer scenarios arise where consolidating answers into a comprehensive and non-redundant set of answers is a more efficient user interface. In this paper, we formulate the problem of answer consolidation, where answers are partitioned into multiple groups, each representing different aspects of the answer set. Then, given this partitioning, a comprehensive and non-redundant set of answers can be constructed by picking one answer from each group. To initiate research on answer consolidation, we construct a dataset consisting of 4,699 questions and 24,006 sentences and evaluate multiple models. Despite a promising performance achieved by the best-performing supervised models, we still believe this task has room for further improvements.
Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
Authors: Han Xiang Choong, Yew-Soon Ong, Abhishek Gupta, Caishun Chen, Ray Lim
Date: 2022-05-02 00:00:00
Tasks: All, Deep Learning

For deep learning, size is power. Massive neural nets trained on broad data for a spectrum of tasks are at the forefront of artificial intelligence. These large pre-trained models or Jacks of All Trades (JATs), when fine-tuned for downstream tasks, are gaining importance in driving deep learning advancements. However, environments with tight resource constraints, changing objectives and intentions, or varied task requirements, could limit the real-world utility of a singular JAT. Hence, in tandem with current trends towards building increasingly large JATs, this paper conducts an initial exploration into concepts underlying the creation of a diverse set of compact machine learning model sets. Composed of many smaller and specialized models, the Set of Sets is formulated to simultaneously fulfil many task settings and environmental conditions. A means to arrive at such a set tractably in one pass of a neuroevolutionary multitasking algorithm is presented for the first time, bringing us closer to models that are collectively Masters of All Trades.
Sequence Learning and Consolidation on Loihi using On-chip Plasticity
Authors: Jack Lindsey, James B Aimone
Date: 2022-05-02 00:00:00

In this work we develop a model of predictive learning on neuromorphic hardware. Our model uses the on-chip plasticity capabilities of the Loihi chip to remember observed sequences of events and use this memory to generate predictions of future events in real time. Given the locality constraints of on-chip plasticity rules, generating predictions without interfering with the ongoing learning process is nontrivial. We address this challenge with a memory consolidation approach inspired by hippocampal replay. Sequence memory is stored in an initial memory module using spike-timing dependent plasticity. Later, during an offline period, memories are consolidated into a distinct prediction module. This second module is then able to represent predicted future events without interfering with the activity, and plasticity, in the first module, enabling online comparison between predictions and ground-truth observations. Our model serves as a proof-of-concept that online predictive learning models can be deployed on neuromorphic hardware with on-chip plasticity.
Budgeted Classification with Rejection: An Evolutionary Method with Multiple Objectives
Authors: Nolan H. Hamilton, Errin Fulp
Date: 2022-05-01 00:00:00

Classification systems are often deployed in resource-constrained settings where labels must be assigned to inputs on a budget of time, memory, etc. Budgeted, sequential classifiers (BSCs) address these scenarios by processing inputs through a sequence of partial feature acquisition and evaluation steps with early-exit options. This allows for an efficient evaluation of inputs that prevents unneeded feature acquisition. To approximate an intractable combinatorial problem, current approaches to budgeted classification rely on well-behaved loss functions that account for two primary objectives (processing cost and error). These approaches offer improved efficiency over traditional classifiers but are limited by analytic constraints in formulation and do not manage additional performance objectives. Notably, such methods do not explicitly account for an important aspect of real-time detection systems -- the fraction of "accepted" predictions satisfying a confidence criterion imposed by a risk-averse monitor. We propose a problem-specific genetic algorithm to build budgeted, sequential classifiers with confidence-based reject options. Three objectives -- accuracy, processing time/cost, and coverage -- are considered. The algorithm emphasizes Pareto efficiency while accounting for a notion of aggregate performance via a unique scalarization. Experiments show our method can quickly find globally Pareto optimal solutions in very large search spaces and is competitive with existing approaches while offering advantages for selective, budgeted deployment scenarios.
Predicting and Optimizing for Energy Efficient ACMV Systems: Computational Intelligence Approaches
Authors: Deqing Zhai, Yeng Chai Soh
Date: 2022-04-19 00:00:00

In this study, a novel application of neural networks that predict thermal comfort states of occupants is proposed with accuracy over 95%, and two optimization algorithms are proposed and evaluated under two real cases (general offices and lecture theatres/conference rooms scenarios) in Singapore. The two optimization algorithms are Bayesian Gaussian process optimization (BGPO) and augmented firefly algorithm (AFA). Based on our earlier studies, the models of energy consumption were developed and well-trained through neural networks. This study focuses on using novel active approaches to evaluate thermal comfort of occupants and so as to solves a multiple-objective problem that aims to balance energy-efficiency of centralized air-conditioning systems and thermal comfort of occupants. The study results show that both BGPO and AFA are feasible to resolve this no prior knowledge-based optimization problem effectively. However, the optimal solutions of AFA are more consistent than those of BGPO at given sample sizes. The best energy saving rates (ESR) of BGPO and AFA are around -21% and -10% respectively at energy-efficient user preference for both Case 1 and Case 2. As a result, an potential benefit of S$1219.1 can be achieved annually for this experimental laboratory level in Singapore.
A Heteroskedasticity-Robust Overidentifying Restriction Test with High-Dimensional Covariates
Authors: Qingliang Fan, Zijian Guo, Ziwei Mei
Date: 2022-04-30 00:00:00
Tasks: Vocal Bursts Intensity Prediction

This paper proposes an overidentifying restriction test for high-dimensional linear instrumental variable models. The novelty of the proposed test is that it allows the number of covariates and instruments to be larger than the sample size. The test is scale-invariant and is robust to heteroskedastic errors. To construct the final test statistic, we first introduce a test based on the maximum norm of multiple parameters that could be high-dimensional. The theoretical power based on the maximum norm is higher than that in the modified Cragg-Donald test (Koles\'{a}r, 2018), the only existing test allowing for large-dimensional covariates. Second, following the principle of power enhancement (Fan et al., 2015), we introduce the power-enhanced test, with an asymptotically zero component used to enhance the power to detect some extreme alternatives with many locally invalid instruments. Finally, an empirical example of the trade and economic growth nexus demonstrates the usefulness of the proposed test.