pwc-archive/papers-with-abstracts huggingface dataset:
What we need to learn if we want to do and not just talk
Authors: Rashmi Gangadharaiah, Balakrishnan Narayanaswamy, Charles Elkan
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Chatbot, Machine Translation

In task-oriented dialog, agents need to generate both fluent natural language responses and correct external actions like database queries and updates. Our paper makes the first attempt at evaluating state of the art models on a large real world task with human users. We show that methods that achieve state of the art performance on synthetic datasets, perform poorly in real world dialog tasks. We propose a hybrid model, where nearest neighbor is used to generate fluent responses and Seq2Seq type models ensure dialogue coherency and generate accurate external actions. The hybrid model on the customer support data achieves a 78{\%} relative improvement in fluency, and a 200{\%} improvement in accuracy of external calls.
Data Collection for Dialogue System: A Startup Perspective
Authors: Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, Jason Mars
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: General Classification, intent-classification, Intent Classification, Text Classification

Industrial dialogue systems such as Apple Siri and Google Now rely on large scale diverse and robust training data to enable their sophisticated conversation capability. Crowdsourcing provides a scalable and inexpensive way of data collection but collecting high quality data efficiently requires thoughtful orchestration of the crowdsourcing jobs. Prior study of this topic have focused on tasks only in the academia settings with limited scope or only provide intrinsic dataset analysis, lacking indication on how it affects the trained model performance. In this paper, we present a study of crowdsourcing methods for a user intent classification task in our deployed dialogue system. Our task requires classification of 47 possible user intents and contains many intent pairs with subtle differences. We consider different crowdsourcing job types and job prompts and analyze quantitatively the quality of the collected data and the downstream model performance on a test set of real user queries from production logs. Our observation provides insights into designing efficient crowdsourcing jobs and provide recommendations for future dialogue system data collection process.
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning
Authors: Pararth Shah, Dilek Hakkani-T{\"u}r, Bing Liu, Gokhan T{\"u}r
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Reinforcement Learning, Reinforcement Learning (RL)

End-to-end neural models show great promise towards building conversational agents that are trained from data and on-line experience using supervised and reinforcement learning. However, these models require a large corpus of dialogues to learn effectively. For goal-oriented dialogues, such datasets are expensive to collect and annotate, since each task involves a separate schema and database of entities. Further, the Wizard-of-Oz approach commonly used for dialogue collection does not provide sufficient coverage of salient dialogue flows, which is critical for guaranteeing an acceptable task completion rate in consumer-facing conversational agents. In this paper, we study a recently proposed approach for building an agent for arbitrary tasks by combining dialogue self-play and crowd-sourcing to generate fully-annotated dialogues with diverse and natural utterances. We discuss the advantages of this approach for industry applications of conversational agents, wherein an agent can be rapidly bootstrapped to deploy in front of users and further optimized via interactive learning from actual users of the system.
Quality Estimation for Automatically Generated Titles of eCommerce Browse Pages
Authors: Nicola Ueffing, Jos{\'e} G. C. de Souza, Gregor Leusch
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, Translation

At eBay, we are automatically generating a large amount of natural language titles for eCommerce browse pages using machine translation (MT) technology. While automatic approaches can generate millions of titles very fast, they are prone to errors. We therefore develop quality estimation (QE) methods which can automatically detect titles with low quality in order to prevent them from going live. In this paper, we present different approaches: The first one is a Random Forest (RF) model that explores hand-crafted, robust features, which are a mix of established features commonly used in Machine Translation Quality Estimation (MTQE) and new features developed specifically for our task. The second model is based on Siamese Networks (SNs) which embed the metadata input sequence and the generated title in the same space and do not require hand-crafted features at all. We thoroughly evaluate and compare those approaches on in-house data. While the RF models are competitive for scenarios with smaller amounts of training data and somewhat more robust, they are clearly outperformed by the SN models when the amount of training data is larger.
Atypical Inputs in Educational Applications
Authors: Su-Youn Yoon, Aoife Cahill, Anastassia Loukina, Klaus Zechner, Brian Riordan, Nitin Madnani
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Speech Recognition

In large-scale educational assessments, the use of automated scoring has recently become quite common. While the majority of student responses can be processed and scored without difficulty, there are a small number of responses that have atypical characteristics that make it difficult for an automated scoring system to assign a correct score. We describe a pipeline that detects and processes these kinds of responses at run-time. We present the most frequent kinds of what are called non-scorable responses along with effective filtering models based on various NLP and speech processing technologies. We give an overview of two operational automated scoring systems {---}one for essay scoring and one for speech scoring{---} and describe the filtering models they use. Finally, we present an evaluation and analysis of filtering models used for spoken responses in an assessment of language proficiency.
SystemT: Declarative Text Understanding for Enterprise
Authors: Laura Chiticariu, Marina Danilevsky, Yunyao Li, Frederick Reiss, Huaiyu Zhu
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Document Classification, Entity Extraction using GAN, Relation Extraction, Semantic Parsing, Sentiment Analysis

The rise of enterprise applications over unstructured and semi-structured documents poses new challenges to text understanding systems across multiple dimensions. We present SystemT, a declarative text understanding system that addresses these challenges and has been deployed in a wide range of enterprise applications. We highlight the design considerations and decisions behind SystemT in addressing the needs of the enterprise setting. We also summarize the impact of SystemT on business and education.
From dictations to clinical reports using machine translation
Authors: Gregory Finley, Wael Salloum, Najmeh Sadoughi, Erik Edwards, Am Robinson, a, Nico Axtmann, Michael Brenndoerfer, Mark Miller, David Suendermann-Oeft
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, Punctuation Restoration, speech-recognition, Speech Recognition, Translation

A typical workflow to document clinical encounters entails dictating a summary, running speech recognition, and post-processing the resulting text into a formatted letter. Post-processing entails a host of transformations including punctuation restoration, truecasing, marking sections and headers, converting dates and numerical expressions, parsing lists, etc. In conventional implementations, most of these tasks are accomplished by individual modules. We introduce a novel holistic approach to post-processing that relies on machine callytranslation. We show how this technique outperforms an alternative conventional system{---}even learning to correct speech recognition errors during post-processing{---}while being much simpler to maintain.
Benchmarks and models for entity-oriented polarity detection
Authors: Lidia Pivovarova, Arto Klami, Roman Yangarber
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Sentiment Analysis, Transfer Learning

We address the problem of determining entity-oriented polarity in business news. This can be viewed as classifying the polarity of the sentiment expressed toward a given mention of a company in a news article. We present a complete, end-to-end approach to the problem. We introduce a new dataset of over 17,000 manually labeled documents, which is substantially larger than any currently available resources. We propose a benchmark solution based on convolutional neural networks for classifying entity-oriented polarity. Although our dataset is much larger than those currently available, it is small on the scale of datasets commonly used for training robust neural network models. To compensate for this, we use transfer learning{---}pre-train the model on a much larger dataset, annotated for a related but different classification task, in order to learn a good representation for business text, and then fine-tune it on the smaller polarity dataset.
Bag of Experts Architectures for Model Reuse in Conversational Language Understanding
Authors: Rahul Jha, Alex Marin, Suvamsh Shivaprasad, Imed Zitouni
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Domain Adaptation, Natural Language Understanding

Slot tagging, the task of detecting entities in input user utterances, is a key component of natural language understanding systems for personal digital assistants. Since each new domain requires a different set of slots, the annotation costs for labeling data for training slot tagging models increases rapidly as the number of domains grow. To tackle this, we describe Bag of Experts (BoE) architectures for model reuse for both LSTM and CRF based models. Extensive experimentation over a dataset of 10 domains drawn from data relevant to our commercial personal digital assistant shows that our BoE models outperform the baseline models with a statistically significant average margin of 5.06{\%} in absolute F1-score when training with 2000 instances per domain, and achieve an even higher improvement of 12.16{\%} when only 25{\%} of the training data is used.
A Novel Approach to Part Name Discovery in Noisy Text
Authors: Nobal Bikram Niraula, Daniel Whyatt, Anne Kao
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Management

As a specialized example of information extraction, part name extraction is an area that presents unique challenges. Part names are typically multi-word terms longer than two words. There is little consistency in how terms are described in noisy free text, with variations spawned by typos, ad hoc abbreviations, acronyms, and incomplete names. This makes search and analyses of parts in these data extremely challenging. In this paper, we present our algorithm, PANDA (Part Name Discovery Analytics), based on a unique method that exploits statistical, linguistic and machine learning techniques to discover part names in noisy text such as that in manufacturing quality documentation, supply chain management records, service communication logs, and maintenance reports. Experiments show that PANDA is scalable and outperforms existing techniques significantly.
The Alexa Meaning Representation Language
Authors: Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, Spyros Matsoukas
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Spoken Language Understanding

This paper introduces a meaning representation for spoken language understanding. The Alexa meaning representation language (AMRL), unlike previous approaches, which factor spoken utterances into domains, provides a common representation for how people communicate in spoken language. AMRL is a rooted graph, links to a large-scale ontology, supports cross-domain queries, fine-grained types, complex utterances and composition. A spoken language dataset has been collected for Alexa, which contains âˆ¼20k examples across eight domains. A version of this meaning representation was released to developers at a trade show in 2016.
Practical Application of Domain Dependent Confidence Measurement for Spoken Language Understanding Systems
Authors: Mahnoosh Mehrabani, David Thomson, Benjamin Stern
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Automatic Speech Recognition (ASR), Feature Engineering, intent-classification, Intent Classification, Machine Translation, speech-recognition, Speech Recognition, Spoken Dialogue Systems, Spoken Language Understanding

Spoken Language Understanding (SLU), which extracts semantic information from speech, is not flawless, specially in practical applications. The reliability of the output of an SLU system can be evaluated using a semantic confidence measure. Confidence measures are a solution to improve the quality of spoken dialogue systems, by rejecting low-confidence SLU results. In this study we discuss real-world applications of confidence scoring in a customer service scenario. We build confidence models for three major types of dialogue states that are considered as different domains: how may I help you, number capture, and confirmation. Practical challenges to train domain-dependent confidence models, including data limitations, are discussed, and it is shown that feature engineering plays an important role to improve performance. We explore a wide variety of predictor features based on speech recognition, intent classification, and high-level domain knowledge, and find the combined feature set with the best rejection performance for each application.
Prediction for the Newsroom: Which Articles Will Get the Most Comments?
Authors: Carl Ambroselli, Julian Risch, Ralf Krestel, Andreas Loos
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Articles, regression

The overwhelming success of the Web and mobile technologies has enabled millions to share their opinions publicly at any time. But the same success also endangers this freedom of speech due to closing down of participatory sites misused by individuals or interest groups. We propose to support manual moderation by proactively drawing the attention of our moderators to article discussions that most likely need their intervention. To this end, we predict which articles will receive a high number of comments. In contrast to existing work, we enrich the article with metadata, extract semantic and linguistic features, and exploit annotated data from a foreign language corpus. Our logistic regression model improves F1-scores by over 80{\%} in comparison to state-of-the-art approaches.
Document-based Recommender System for Job Postings using Dense Representations
Authors: Ahmed Elsafty, Martin Riedl, Chris Biemann
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Document Embedding, Recommendation Systems

Job boards and professional social networks heavily use recommender systems in order to better support users in exploring job advertisements. Detecting the similarity between job advertisements is important for job recommendation systems as it allows, for example, the application of item-to-item based recommendations. In this work, we research the usage of dense vector representations to enhance a large-scale job recommendation system and to rank German job advertisements regarding their similarity. We follow a two-folded evaluation scheme: (1) we exploit historic user interactions to automatically create a dataset of similar jobs that enables an offline evaluation. (2) In addition, we conduct an online A/B test and evaluate the best performing method on our platform reaching more than 1 million users. We achieve the best results by combining job titles with full-text job descriptions. In particular, this method builds dense document representation using words of the titles to weigh the importance of words of the full-text description. In the online evaluation, this approach allows us to increase the click-through rate on job recommendations for active users by 8.0{\%}.
Alignment, Acceptance, and Rejection of Group Identities in Online Political Discourse
Authors: Hagyeong Shin, Gabriel Doyle
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6

Conversation is a joint social process, with participants cooperating to exchange information. This process is helped along through linguistic alignment: participants{'} adoption of each other{'}s word use. This alignment is robust, appearing many settings, and is nearly always positive. We create an alignment model for examining alignment in Twitter conversations across antagonistic groups. This model finds that some word categories, specifically pronouns used to establish group identity and common ground, are negatively aligned. This negative alignment is observed despite other categories, which are less related to the group dynamics, showing the standard positive alignment. This suggests that alignment is strongly biased toward cooperative alignment, but that different linguistic features can show substantially different behaviors.
Combining Abstractness and Language-specific Theoretical Indicators for Detecting Non-Literal Usage of Estonian Particle Verbs
Authors: Eleri Aedmaa, Maximilian K{\"o}per, Sabine Schulte im Walde
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6

This paper presents two novel datasets and a random-forest classifier to automatically predict literal vs. non-literal language usage for a highly frequent type of multi-word expression in a low-resource language, i.e., Estonian. We demonstrate the value of language-specific indicators induced from theoretical linguistic research, which outperform a high majority baseline when combined with language-independent features of non-literal language (such as abstractness).
Verb Alternations and Their Impact on Frame Induction
Authors: Esther Seyffarth
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Question Answering

Frame induction is the automatic creation of frame-semantic resources similar to FrameNet or PropBank, which map lexical units of a language to frame representations of each lexical unit{'}s semantics. For verbs, these representations usually include a specification of their argument slots and of the selectional restrictions that apply to each slot. Verbs that participate in diathesis alternations have different syntactic realizations whose semantics are closely related, but not identical. We discuss the influence that such alternations have on frame induction, compare several possible frame structures for verbs in the causative alternation, and propose a systematic analysis of alternating verbs that encodes their similarities as well as their differences.
A Generalized Knowledge Hunting Framework for the Winograd Schema Challenge
Authors: Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Common Sense Reasoning, Coreference Resolution, Information Retrieval

We introduce an automatic system that performs well on two common-sense reasoning tasks, the Winograd Schema Challenge (WSC) and the Choice of Plausible Alternatives (COPA). Problem instances from these tasks require diverse, complex forms of inference and knowledge to solve. Our method uses a knowledge-hunting module to gather text from the web, which serves as evidence for candidate problem resolutions. Given an input problem, our system generates relevant queries to send to a search engine. It extracts and classifies knowledge from the returned results and weighs it to make a resolution. Our approach improves F1 performance on the WSC by 0.16 over the previous best and is competitive with the state-of-the-art on COPA, demonstrating its general applicability.
Towards Qualitative Word Embeddings Evaluation: Measuring Neighbors Variation
Authors: B{\'e}n{\'e}dicte Pierrejean, Ludovic Tanguy
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Embeddings Evaluation, Named Entity Recognition (NER), Part-Of-Speech Tagging, Word Embeddings

We propose a method to study the variation lying between different word embeddings models trained with different parameters. We explore the variation between models trained with only one varying parameter by observing the distributional neighbors variation and show how changing only one parameter can have a massive impact on a given semantic space. We show that the variation is not affecting all words of the semantic space equally. Variation is influenced by parameters such as setting a parameter to its minimum or maximum value but it also depends on the corpus intrinsic features such as the frequency of a word. We identify semantic classes of words remaining stable across the models trained and specific words having high variation.
Learning Word Embeddings for Data Sparse and Sentiment Rich Data Sets
Authors: Prathusha Kameswara Sarma
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: General Classification, Learning Word Embeddings, Sentiment Analysis, Sentiment Classification, Specificity, Word Embeddings

This research proposal describes two algorithms that are aimed at learning word embeddings for data sparse and sentiment rich data sets. The goal is to use word embeddings adapted for domain specific data sets in downstream applications such as sentiment classification. The first approach learns word embeddings in a supervised fashion via SWESA (Supervised Word Embeddings for Sentiment Analysis), an algorithm for sentiment analysis on data sets that are of modest size. SWESA leverages document labels to jointly learn polarity-aware word embeddings and a classifier to classify unseen documents. In the second approach domain adapted (DA) word embeddings are learned by exploiting the specificity of domain specific data sets and the breadth of generic word embeddings. The new embeddings are formed by aligning corresponding word vectors using Canonical Correlation Analysis (CCA) or the related nonlinear Kernel CCA. Experimental results on binary sentiment classification tasks using both approaches for standard data sets are presented.
Igbo Diacritic Restoration using Embedding Models
Authors: Ignatius Ezeani, Mark Hepple, Ikechukwu Onyenwe, Enemouh Chioma
Date: 2018-06-01 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, Word Embeddings

Igbo is a low-resource language spoken by approximately 30 million people worldwide. It is the native language of the Igbo people of south-eastern Nigeria. In Igbo language, diacritics - orthographic and tonal - play a huge role in the distinguishing the meaning and pronunciation of words. Omitting diacritics in texts often leads to lexical ambiguity. Diacritic restoration is a pre-processing task that replaces missing diacritics on words from which they have been removed. In this work, we applied embedding models to the diacritic restoration task and compared their performances to those of n-gram models. Although word embedding models have been successfully applied to various NLP tasks, it has not been used, to our knowledge, for diacritic restoration. Two classes of word embeddings models were used: those projected from the English embedding space; and those trained with Igbo bible corpus ({\mbox{$\approx$}} 1m). Our best result, 82.49{\%}, is an improvement on the baseline n-gram models.