pwc-archive/papers-with-abstracts huggingface dataset:
Retrieval Based Response Letter Generation For a Customer Care Setting
Authors: Biplob Biswas, Renhao Cui, Rajiv Ramnath
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Data Augmentation, Management, Response Generation, Retrieval, Text Generation

Letter-like communications (such as email) are a major means of customer relationship management within customer-facing organizations. These communications are initiated on a channel by requests from customers and then responded to by the organization on the same channel. For decades, the job has almost entirely been conducted by human agents who attempt to provide the most appropriate reaction to the request. Rules have been made to standardize the overall customer service process and make sure the customers receive professional responses. Recent progress in natural language processing has made it possible to automate response generation. However, the diversity and open nature of customer queries and the lack of structured knowledge bases make this task even more challenging than typical task-oriented language generation tasks. Keeping those obstacles in mind, we propose a deep-learning based response letter generation framework that attempts to retrieve knowledge from historical responses and utilize it to generate an appropriate reply. Our model uses data augmentation to address the insufficiency of query-response pairs and employs a ranking mechanism to choose the best response from multiple potential options. We show that our technique outperforms the baselines by significant margins while producing consistent and informative responses.
Knowledge extraction from aeronautical messages (NOTAMs) with self-supervised language models for aircraft pilots
Authors: Alexandre Arnold, Fares Ernez, Catherine Kobus, Marion-Cécile Martin
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

During their pre-flight briefings, aircraft pilots must analyse a long list of NoTAMs (NOtice To AirMen) indicating potential hazards along the flight route, sometimes up to pages for long-haul flights. NOTAM free-text fields typically have a very special phrasing, with lots of acronyms and domain-specific vocabulary, which makes it differ significantly from standard English. In this paper, we pretrain language models derived from BERT on circa 1 million unlabeled NOTAMs and reuse the learnt representations on three downstream tasks valuable for pilots: criticality prediction, named entity recognition and translation into a structured language called Airlang. This self-supervised approach, where smaller amounts of labeled data are enough for task-specific fine-tuning, is well suited in the aeronautical context since expert annotations are expensive and time-consuming. We present evaluation scores across the tasks showing a high potential for an operational usability of such models (by pilots, airlines or service providers), which is a first to the best of our knowledge.
Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining
Authors: Minhua Chen, Badrinath Jayakumar, Michael Johnston, S. Eman Mahmoodi, Daniel Pressel
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Intent Discovery, Language Modeling, Language Modelling, Multi-Task Learning, Sentence, Sentence Embedding, Sentence-Embedding

A key challenge in the creation and refinement of virtual assistants is the ability to mine unlabeled utterance data to discover common intents. We develop an approach to this problem that combines large-scale pre-training and multi-task learning to derive a semantic embedding that can be leveraged to identify clusters of utterances that correspond to unhandled intents. An utterance encoder is first trained with a language modeling objective and subsequently adapted to predict intent labels from a large collection of cross-domain enterprise virtual assistant data using a multi-task cosine softmax loss. Experimental evaluation shows significant advantages for this multi-step pre-training approach, with large gains in downstream clustering accuracy on new applications compared to standard sentence embedding approaches. The approach has been incorporated into an interactive discovery tool that enables visualization and exploration of intents by system analysts and builders.
Lightweight Transformers for Conversational AI
Authors: Daniel Pressel, Wenshuo Liu, Michael Johnston, Minhua Chen
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: GPU, Intent Detection, Natural Language Understanding

To understand how training on conversational language impacts performance of pre-trained models on downstream dialogue tasks, we build compact Transformer-based Language Models from scratch on several large corpora of conversational data. We compare the performance and characteristics of these models against BERT and other strong baselines on dialogue probing tasks. Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, with an ever-increasing number of parameters, resulting in difficulties in model deployment. We focus instead on training fast, lightweight models that excel at natural language understanding (NLU) and can replace existing lower-capacity conversational AI models with similar size and speed. In the process, we develop a simple but unique curriculum-based approach that moves from general-purpose to dialogue-targeted both in terms of data and objective. Our resultant models have around 1/3 the number of parameters of BERT-base and produce better representations for a wide array of intent detection datasets using linear and Mutual-Information probing techniques. Additionally, the models can be easily fine-tuned on a single consumer GPU card and deployed in near real-time production environments.
What Do Users Care About? Detecting Actionable Insights from User Feedback
Authors: Kasturi Bhattacharjee, Rashmi Gangadharaiah, Kathleen McKeown, Dan Roth
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7

Users often leave feedback on a myriad of aspects of a product which, if leveraged successfully, can help yield useful insights that can lead to further improvements down the line. Detecting actionable insights can be challenging owing to large amounts of data as well as the absence of labels in real-world scenarios. In this work, we present an aggregation and graph-based ranking strategy for unsupervised detection of these insights from real-world, noisy, user-generated feedback. Our proposed approach significantly outperforms strong baselines on two real-world user feedback datasets and one academic dataset.
CTM - A Model for Large-Scale Multi-View Tweet Topic Classification
Authors: Vivek Kulkarni, Kenny Leung, Aria Haghighi
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Classification, Topic Classification

Automatically associating social media posts with topics is an important prerequisite for effective search and recommendation on many social media platforms. However, topic classification of such posts is quite challenging because of (a) a large topic space (b) short text with weak topical cues, and (c) multiple topic associations per post. In contrast to most prior work which only focuses on post-classification into a small number of topics (10-20), we consider the task of large-scale topic classification in the context of Twitter where the topic space is 10 times larger with potentially multiple topic associations per Tweet. We address the challenges above and propose a novel neural model,  that (a) supports a large topic space of 300 topics (b) takes a holistic approach to tweet content modeling – leveraging multi-modal content, author context, and deeper semantic cues in the Tweet. Our method offers an effective way to classify Tweets into topics at scale by yielding superior performance to other approaches (a relative lift of \mathbf{20}\% in median average precision score) and has been successfully deployed in production at Twitter.
Constraint-based Multi-hop Question Answering with Knowledge Graph
Authors: Sayantan Mitra, Roshni Ramnani, Shubhashis Sengupta
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Link Prediction, Multi-hop Question Answering, Natural Language Queries, Question Answering

The objective of a Question-Answering system over Knowledge Graph (KGQA) is to respond to natural language queries presented over the KG. A complex question answering system typically addresses one of the two categories of complexity: questions with constraints and questions involving multiple hops of relations. Most of the previous works have addressed these complexities separately. Multi-hop KGQA necessitates reasoning across numerous edges of the KG in order to arrive at the correct answer. Because KGs are frequently sparse, multi-hop KGQA presents extra complications. Recent works have developed KG embedding approaches to reduce KG sparsity by performing missing link prediction. In this paper, we tried to address multi-hop constrained-based queries using KG embeddings to generate more flexible query graphs. Empirical results indicate that the proposed methodology produces state-of-the-art outcomes on three KGQA datasets.
Fast Bilingual Grapheme-To-Phoneme Conversion
Authors: Hwa-Yeon Kim, Jong-Hwan Kim, Jae-Min Kim
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Data Augmentation, Grapheme-to-Phoneme Conversion, Sentence, text-to-speech, Text to Speech

Autoregressive transformer (ART)-based grapheme-to-phoneme (G2P) models have been proposed for bi/multilingual text-to-speech systems. Although they have achieved great success, they suffer from high inference latency in real-time industrial applications, especially processing long sentence. In this paper, we propose a fast and high-performance bilingual G2P model. For fast and exact decoding, we used a non-autoregressive structured transformer-based architecture and data augmentation for predicting output length. Our model achieved better performance than that of the previous autoregressive model and about 2700% faster inference speed.
Knowledge Extraction From Texts Based on Wikidata
Authors: Anastasia Shimorina, Johannes Heinecke, Frédéric Herledan
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: coreference-resolution, Coreference Resolution, Relation, Relation Extraction

This paper presents an effort within our company of developing knowledge extraction pipeline for English, which can be further used for constructing an entreprise-specific knowledge base. We present a system consisting of entity detection and linking, coreference resolution, and relation extraction based on the Wikidata schema. We highlight existing challenges of knowledge extraction by evaluating the deployed pipeline on real-world data. We also make available a database, which can serve as a new resource for sentential relation extraction, and we underline the importance of having balanced data for training classification models.
Parameter-efficient Continual Learning Framework in Industrial Real-time Text Classification System
Authors: Tao Zhu, Zhe Zhao, Weijie Liu, Jiachi Liu, Yiren Chen, Weiquan Mao, Haoyan Liu, Kunbo Ding, Yudong Li, Xuefeng Yang
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Continual Learning, text-classification, Text Classification

Catastrophic forgetting is a challenge for model deployment in industrial real-time systems, which requires the model to quickly master a new task without forgetting the old one. Continual learning aims to solve this problem; however, it usually updates all the model parameters, resulting in extensive training times and the inability to deploy quickly. To address this challenge, we propose a parameter-efficient continual learning framework, in which efficient parameters are selected through an offline parameter selection strategy and then trained using an online regularization method. In our framework, only a few parameters need to be updated, which not only alleviates catastrophic forgetting, but also allows the model to be saved with the changed parameters instead of all parameters. Extensive experiments are conducted to examine the effectiveness of our proposal. We believe this paper will provide useful insights and experiences on developing deep learning-based online real-time systems.
Q2R: A Query-to-Resolution System for Natural-Language Queries
Authors: Shiau Hong Lim, Laura Wynter
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Natural Language Queries, Retrieval

We present a system for document retrieval that combines direct classification with standard content-based retrieval approaches to significantly improve the relevance of the retrieved documents. Our system exploits the availability of an imperfect but sizable amount of labeled data from past queries. For domains such as technical support, the proposed approach enhances the system’s ability to retrieve documents that are otherwise ranked very low based on content alone. The system is easy to implement and can make use of existing text ranking methods, augmenting them through the novel Q2R orchestration framework. Q2R has been extensively tested and is in use at IBM.
Identifying Corporate Credit Risk Sentiments from Financial News
Authors: Noujoud Ahbali, Xinyuan Liu, Albert Nanda, Jamie Stark, Ashit Talukder, Rupinder Paul Khandpur
Date: 2022-07-01 00:00:00
Proceeding: NAACL (ACL) 2022 7
Tasks: Management, text-classification, Text Classification

Credit risk management is one central practice for financial institutions, and such practice helps them measure and understand the inherent risk within their portfolios. Historically, firms relied on the assessment of default probabilities and used the press as one tool to gather insights on the latest credit event developments of an entity. However, due to the deluge of the current news coverage for companies, analyzing news manually by financial experts is considered a highly laborious task. To this end, we propose a novel deep learning-powered approach to automate news analysis and credit adverse events detection to score the credit sentiment associated with a company. This paper showcases a complete system that leverages news extraction and data enrichment with targeted sentiment entity recognition to detect companies and text classification to identify credit events. We developed a custom scoring mechanism to provide the company’s credit sentiment score (CSS^{TM}) based on these detected events. Additionally, using case studies, we illustrate how this score helps understand the company’s credit profile and discriminates between defaulters and non-defaulters.
Multilingualism Encourages Recursion: a Transfer Study with mBERT
Authors: Andrea De Varda, Roberto Zamparelli
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7

The present work constitutes an attempt to investigate the relational structures learnt by mBERT, a multilingual transformer-based network, with respect to different cross-linguistic regularities proposed in the fields of theoretical and quantitative linguistics. We pursued this objective by relying on a zero-shot transfer experiment, evaluating the model’s ability to generalize its native task to artificial languages that could either respect or violate some proposed language universal, and comparing its performance to the output of BERT, a monolingual model with an identical configuration. We created four artificial corpora through a Probabilistic Context-Free Grammar by manipulating the distribution of tokens and the structure of their dependency relations. We showed that while both models were favoured by a Zipfian distribution of the tokens and by the presence of head-dependency type structures, the multilingual transformer network exhibited a stronger reliance on hierarchical cues compared to its monolingual counterpart.
Typological Word Order Correlations with Logistic Brownian Motion
Authors: Kai Hartung, Gerhard Jäger, Sören Gröttrup, Munir Georges
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: All

In this study we address the question to what extent syntactic word-order traits of different languages have evolved under correlation and whether such dependencies can be found universally across all languages or restricted to specific language families.To do so, we use logistic Brownian Motion under a Bayesian framework to model the trait evolution for 768 languages from 34 language families. We test for trait correlations both in single families and universally over all families.Separate models reveal no universal correlation patterns and Bayes Factor analysis of models over all covered families also strongly indicate lineage specific correlation patters instead of universal dependencies.
Cross-linguistic Comparison of Linguistic Feature Encoding in BERT Models for Typologically Different Languages
Authors: Yulia Otmakhova, Karin Verspoor, Jey Han Lau
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7

Though recently there have been an increased interest in how pre-trained language models encode different linguistic features, there is still a lack of systematic comparison between languages with different morphology and syntax. In this paper, using BERT as an example of a pre-trained model, we compare how three typologically different languages (English, Korean, and Russian) encode morphology and syntax features across different layers. In particular, we contrast languages which differ in a particular aspect, such as flexibility of word order, head directionality, morphological type, presence of grammatical gender, and morphological richness, across four different tasks.
Tweaking UD Annotations to Investigate the Placement of Determiners, Quantifiers and Numerals in the Noun Phrase
Authors: Luigi Talamo
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: LEMMA

We describe a methodology to extract with finer accuracy word order patterns from texts automatically annotated with Universal Dependency (UD) trained parsers. We use the methodology to quantify the word order entropy of determiners, quantifiers and numerals in ten Indo-European languages, using UD-parsed texts from a parallel corpus of prosaic texts. Our results suggest that the combinations of different UD annotation layers, such as UD Relations, Universal Parts of Speech and lemma, and the introduction of language-specific lists of closed-category lemmata has the two-fold effect of improving the quality of analysis and unveiling hidden areas of variability in word order patterns.
A Database for Modal Semantic Typology
Authors: Qingxia Guo, Nathaniel Imel, Shane Steinert-Threlkeld
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7

This paper introduces a database for crosslinguistic modal semantics. The purpose of this database is to (1) enable ongoing consolidation of modal semantic typological knowledge into a repository according to uniform data standards and to (2) provide data for investigations in crosslinguistic modal semantic theory and experiments explaining such theories. We describe the kind of semantic variation that the database aims to record, the format of the data, and a current snapshot of the database, emphasizing access and contribution to the database in light of the goals above. We release the database at https://clmbr.shane.st/modal-typology.
The SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes
Authors: Johann-Mattis List, Ekaterina Vylomova, Robert Forkel, Nathan Hill, Ryan Cotterell
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: Image Restoration

This study describes the structure and the results of the SIGTYP 2022 shared task on the prediction of cognate reflexes from multilingual wordlists. We asked participants to submit systems that would predict words in individual languages with the help of cognate words from related languages. Training and surprise data were based on standardized multilingual wordlists from several language families. Four teams submitted a total of eight systems, including both neural and non-neural systems, as well as systems adjusted to the task and systems using more general settings. While all systems showed a rather promising performance, reflecting the overwhelming regularity of sound change, the best performance throughout was achieved by a system based on convolutional networks originally designed for image restoration.
Bayesian Phylogenetic Cognate Prediction
Authors: Gerhard Jäger
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: Cognate Prediction, Prediction

In Jäger (2019) a computational framework was defined to start from parallel word lists of related languages and infer the corresponding vocabulary of the shared proto-language. The SIGTYP 2022 Shared Task is closely related. The main difference is that what is to be reconstructed is not the proto-form but an unknown word from an extant language. The system described here is a re-implementation of the tools used in the mentioned paper, adapted to the current task.
Mockingbird at the SIGTYP 2022 Shared Task: Two Types of Models forthe Prediction of Cognate Reflexes
Authors: Christo Kirov, Richard Sproat, Alexander Gutkin
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: Decoder, Image Restoration

The SIGTYP 2022 shared task concerns the problem of word reflex generation in a target language, given cognate words from a subset of related languages. We present two systems to tackle this problem, covering two very different modeling approaches. The first model extends transformer-based encoder-decoder sequence-to-sequence modeling, by encoding all available input cognates in parallel, and having the decoder attend to the resulting joint representation during inference. The second approach takes inspiration from the field of image restoration, where models are tasked with recovering pixels in an image that have been masked out. For reflex generation, the missing reflexes are treated as “masked pixels” in an “image” which is a representation of an entire cognate set across a language family. As in the image restoration case, cognate restoration is performed with a convolutional network.
A Transformer Architecture for the Prediction of Cognate Reflexes
Authors: Giuseppe Celano
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7
Tasks: Decoder, Prediction

This paper presents the transformer model built to participate in the SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes. It consists of an encoder-decoder architecture with multi-head attention mechanism. Its output is concatenated with the one hot encoding of the language label of an input character sequence to predict a target character sequence. The results show that the transformer outperforms the baseline rule-based system only partially.
Investigating Information-Theoretic Properties of the Typology of Spatial Demonstratives
Authors: Sihan Chen, Richard Futrell, Kyle Mahowald
Date: 2022-07-01 00:00:00
Proceeding: NAACL (SIGTYP) 2022 7

Using data from Nintemann et al. (2020), we explore the variability in complexity and informativity across spatial demonstrative systems using spatial deictic lexicons from 223 languages. We argue from an information-theoretic perspective (Shannon, 1948) that spatial deictic lexicons are efficient in communication, balancing informativity and complexity. Specifically, we find that under an appropriate choice of cost function and need probability over meanings, among all the 21146 theoretically possible spatial deictic lexicons, those adopted by real languages lie near an efficient frontier. Moreover, we find that the conditions that the need probability and the cost function need to satisfy are consistent with the cognitive science literature regarding the source-goal asymmetry. We also show that the data are better explained by introducing a notion of systematicity, which is not currently accounted for in Information Bottleneck approaches to linguistic efficiency.