pwc-archive/papers-with-abstracts huggingface dataset:
Auto-completion for Te Reo Māori
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cultural Vocal Bursts Intensity Prediction

Typing assistance in online environments is a feature that English-speaking users take for granted. Gmail, as an example, is an email provider for over 1.8 billion users worldwide. However, it fails to support lower resourced languages, providing only 5 languages with auto completion support.
The Māori language is the medium for revitalizing Māori culture. The Māori language of Aotearoa New Zealand, as an endangered language, is under threat. To ensure its ongoing survival, Māori language tools must be created and the language must be supported in technological environments.
This project is focused on developing and implementing an auto-completion tool in a web environment for Te Reo Māori. The following report
will elaborate on the highly optimized search trees and the various ranking systems used to produce quick and accurate suggestions. Further
areas are investigated, including Google’s Smart Compose, data structure performance, and the use of Māori data.
Dual-space Hierarchical Learning for Goal-guided Conversational Recommendation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Conversational Recommendation, Recommendation Systems, Representation Learning

Proactively and naturally guiding the dialog from the non-recommendation context~(e.g., Chit-chat) to the recommendation scenario is crucial for the Conversational Recommender System~(CRS). Prior studies mainly focus on planning the next dialog goal~(e.g., chat on a movie star) conditioned on the previous dialog. However, we find the dialog goals can be simultaneously observed at different levels, which can be utilized to improve CRS.In this paper, we propose the \textit{\textbf{D}ual-space \textbf{H}ierarchical \textbf{L}earning}~(\textbf{DHL}) to leverage multi-level goal sequences and their hierarchical relationships for conversational recommendation. Specifically, we exploit multi-level goal sequences from both the representation space and the optimization space. In the representation space, we propose the hierarchical representation learning where a cross attention module derives mutually enhanced multi-level goal representations. Additionally, we propose a soft labeling strategy to gradually guide the optimization direction. Experiments on two real-world datasets verify the effectiveness of our approach.
Impact of Tokenization on Language Models: An Analysis for Turkish
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Tokenization is an important text preprocessing step to prepare input tokens for language models. WordPiece and BPE are de-facto methods employed by large language models, such as BERT and GPT. However, the impact of tokenization can be different for the agglutinative languages having words with prefixes and suffixes, such as Turkic languages. We compare five tokenization methods, including a morphological-level tokenization that takes agglutinative language structure into account. We train tokenizers, and pre-train mini language models using RoBERTa pre-training procedure on Turkish OSCAR corpus. We then fine-tune our models on six downstream tasks. There are two main outcomes: (i) Morphological and word-level tokenizers outperform de-facto tokenizers in particular cases. (ii) Mini models can be competitive to larger state-of-the-art models, such that a 14-times smaller model can recover 94\% of the performance of a larger model.
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Language Modeling, Language Modelling, Re-Ranking

State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking. To this end, models generally utilize an encoder-only (like BERT) paradigm or an encoder-decoder (like T5) approach. These paradigms, however, are not without flaws, i.e., running the model on all query-document pairs at inference-time incurs a significant computational cost. This paper proposes a new training and inference paradigm for re-ranking. We propose to finetune a pretrained encoder-decoder model using in the form of document to query generation. Subsequently, we show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference. This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference. Our experiments show that this new paradigm achieves results that are comparable to the more expensive cross-attention ranking approaches while being up to 6.8X faster. We believe this work paves the way for more efficient neural rankers that leverage large pretrained models.
A Primer in NMTology: What we have understood about NMT
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, NMT, Translation

Neural Machine Translation (NMT) has been through great revolutions in recent years. Accompanied with improvements in translation quality are works that attempted to understand the working mechanism of various aspects of the NMT framework. In our paper, we survey those efforts on unveiling the \textit{black box} of the standard NMT framework. To begin with, we briefly introduce the three critical components of the holistic NNT framework; nextly, we deliver a clear \textit{component-centric} categorization and clean summary of these specific works \textit{guided} by \textit{frequently-asked} questions (FAQs) that aim at making up \textit{lack} of understanding; finally, we discuss several limitations, future directions and inspirations. We believe this paper could facilitate the community to weave a holistic and clear picture of our current understandings of the standard NMT framework and shed light on its future improvements and developments. Please check this website https://nmtology.github.io/ for a visual guidance of the FAQs.
Pre-Training with Syntactic Structure Prediction for Chinese Semantic Error Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Existing Chinese text error detection mainly focuses on spelling errors and simple grammatical errors. These errors have been studied extensively and are relatively simple for humans. Chinese Semantic Error Recognition (CSER) pays attention to more complex semantic errors that humans cannot easily recognize compared with Chinese text error detection. Considering the complex syntactic relation between words, we find that syntactic structure from the syntax tree can help identify semantic errors. In this paper, we consider adopting the pre-trained models to solve the task of CSER. To make the model learn syntactic structure in the pre-training stage, we designed a novel pre-training task to predict the syntactic structure from the syntax tree between different words. Due to the lack of a published dataset for CSER, we build a high-quality dataset for CSER for the first time named Corpus of Chinese Linguistic Semantic Acceptability (CoCLSA), which is extracted from the high school examinations. The experimental results on the CoCLSA show that our pre-trained model based on the new pre-training task has a positive performance compared with existing pre-trained models.
Multi-Stage Prompting for Knowledgeable Dialogue Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dialogue Generation, Language Modelling, Retrieval

Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model (LM) and large-scale knowledge bases. These models typically fail to generalize on topics outside of the knowledge base, and require maintaining separate potentially large checkpoints each time finetuning is needed. In this paper, we aim to address these limitations by leveraging the inherent knowledge stored in the pretrained LM as well as its powerful generation ability. We propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM. We first prompt the LM to generate knowledge based on the dialogue context. Then, we further prompt it to generate responses based on the dialogue context and the previously generated knowledge. Results show that our knowledge generator outperforms the state-of-the-art retrieval-based model by 5.8\% when combining knowledge relevance and correctness. In addition, our multi-stage prompting outperforms the finetuning-based dialogue model in terms of response knowledgeability and engagement by up to 10% and 5%, respectively. Furthermore, we scale our model up to 530 billion parameters and demonstrate that larger LMs improve the generation correctness score by up to 10%, and response relevance, knowledgeability and engagement by up to 10%.
Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectors
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Abuse Detection, Abusive Language

Robustness of machine learning models on ever-changing real-world data is critical, especially for applications affecting human well-being such as content moderation. New kinds of abusive language continually emerge in online discussions in response to current events (e.g., COVID-19), and the deployed abuse detection systems should be updated regularly to remain accurate. In this paper, we show that general abusive language classifiers tend to be fairly reliable in detecting out-of-domain explicitly abusive utterances but fail to detect new types of more subtle, implicit abuse. Next, we propose an interpretability technique, based on the Testing Concept Activation Vector (TCAV) method from computer vision, to quantify the sensitivity of a trained model to the human-defined concepts of explicit and implicit abusive language, and use that to explain the generalizability of the model on new data, in this case, COVID-related anti-Asian hate speech. Extending this technique, we introduce a novel metric, Degree of Explicitness, for a single instance and show that the new metric is beneficial in suggesting out-of-domain unlabeled examples to effectively enrich the training data with informative, implicitly abusive texts.
Knowledge-guided Transformer for Joint Theme and Emotion Classification of Chinese Classical Poetry
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Emotion Classification

The classifications of the theme and emotion are essential for understanding and organizing  Chinese classical poetry. Existing works fail to consider the lexical knowledge mined from poem  annotations, which intuitively reflects the theme and emotion. In addition, they just treat them as two separate tasks without considering that the emotion is usually related with the theme. In this paper, we propose a Knowledge-guided Transformer Model (KTM) for joint theme and emotion classification of  Chinese classical poetry. Specifically, we first respectively construct two lexical dictionaries for the theme and emotion based on the poem annotations. Then we take full advantage of the lexical dictionaries with a knowledge-based mask-transformer to represent poems. Furthermore, considering the correlations between the theme and emotion, our model jointly classifies the theme and emotion for Chinese classical poetry by stacking the two subtasks. Extensive experiments demonstrate that our model achieves state-of-the-art performance on both theme and emotion classifications, especially on tail labels.
Incorporating Multiple Knowledge Sources for Targeted Aspect-based Financial Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Domain Adaptation, Language Modeling, Language Modelling, Sentiment Analysis

Combining symbolic and subsymbolic methods has become a promising strategy as research tasks in AI grow increasingly complicated and require a higher levels of understanding. Targeted Aspect-based Financial Sentiment Analysis (TABFSA) is one of such complicated tasks, as it involves information extraction, specification, and domain adaptation. External knowledge has been proven useful for general-purpose sentiment analysis, but not yet for the finance domain. Current state-of-the-art Financial Sentiment Analysis (FSA) models, however, have overlooked the importance of external knowledge. To fill this gap, we propose using attentive CNN and LSTM to strategically integrate multiple external knowledge sources into the pre-trained language model fine-tuning process for TABFSA. Experiments on the FiQA Task 1 and SemEval 2017 Task 5 datasets show that the knowledge-enabled models systematically improve upon their plain deep learning counterparts, and some outperform the state-of-the-art results reported in terms of aspect sentiment analysis error.
Overcoming a Theoretical Limitation of Self-Attention
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: LEMMA, Machine Translation, Open-Ended Question Answering

Although transformers are remarkably effective for many tasks, there are some surprisingly easy-looking regular languages that they struggle with. Hahn shows that for languages where acceptance depends on a single input symbol, a transformer's classification decisions get closer and closer to random guessing (that is, a cross-entropy of 1) as input strings get longer and longer. We examine this limitation using two languages: PARITY, the language of bit strings with an odd number of 1s, and FIRST, the language of bit strings starting with a 1. We demonstrate three ways of overcoming the limitation implied by Hahn's lemma. First, we settle an open question by constructing a transformer that recognizes PARITY with perfect accuracy, and similarly for FIRST. Second, we use layer normalization to bring the cross-entropy of both models arbitrarily close to zero. Third, when transformers need to focus on a single position, as for FIRST, we find that they can fail to generalize to longer strings; we offer a simple remedy to this problem that also improves length generalization in machine translation.
Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstractive Dialogue Summarization, Contrastive Learning, Text Summarization

Pre-trained models have brought remarkable success on the text summarization task. For dialogue summarization, the subdomain of text summarization, utterances are concatenated to flat text before being processed. As a result, existing summarization systems based on pre-trained models are unable to recognize the unique format of the speaker-utterance pair well in the dialogue. To investigate this issue, we conduct probing tests and manual analysis, and find that the powerful pre-trained model can not identify different speakers well in the conversation, which leads to various factual errors. Moreover, we propose three speaker-aware supervised contrastive learning (SCL) tasks: Token-level SCL, Turn-level SCL, and Global-level SCL. Comprehensive experiments demonstrate that our methods achieve significant performance improvement on two mainstream dialogue summarization datasets. According to detailed human evaluations, pre-trained models equipped with SCL tasks effectively generate summaries with better factual consistency.
Speaking Rationally by Gestures: Information Theoretic Insights from Multi-Modal Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The multi-modality nature of human communication can be utilized to enhance the performance of computational language models. However, few studies have explored the non-verbal channels with finer theoretical lens. We use multi-modal language models trained against monologue video data to study how the non-verbal expression contributes to communication, by examining two aspects: first, whether incorporating gesture representations can improve the language model's performance (perplexity), and second, whether the gesture channel demonstrates the similar pattern of entropy rate constancy (ERC) found in verbal language, which is governed by Information Theory. We have positive results to support both assumptions. The conclusion is that speakers indeed use simple gestures to convey information that enhances verbal communication, and how this information is organized is a rational process.
Heterogeneous-Graph Reasoning and Fine-Grained Aggregation for Fact Checking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fact Checking, Graph Attention

Fact checking is a challenging task that requires corresponding evidences to verify the property of a claim based on reasoning. Previous studies generally i) construct the graph by treating each evidence-claim pair as node which is a simple way that ignores to exploit their implicit interaction, or building a fully-connected graph among claim and evidences where the entailment relationship between claim and evidence would be considered equally to the semantic relationship among evidences; ii) aggregate evidences equally without considering their different stances towards the verification of fact. Towards the above issues, we propose a novel heterogeneous-graph reasoning and fine-grained aggregation model, with two following modules: 1) a heterogeneous graph attention network module to distinguish different types of relationships within the constructed graph; 2) fine-grained aggregation module which learns the implicit stance of evidences towards the prediction result in details. Extensive experiments on the benchmark dataset demonstrate that our proposed model achieves much better performance than state-of-the-art methods.
MMNet: Multimodal Fusion via Mutual Learning Network for Fake News Detection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fake News Detection

The rapid development of social media provides a hotbed for the dissemination of fake news, which misleads readers and causes negative effects on society. We observe that a large amount of news contains images in addition to texts. Existing works on multimodal fake news detection conduct multimodal fusion by simple concatenating or co-attention networks, which is insufficient to fuse the multimodal information effectively. Considering that people judge news by alternatively exploring text-oriented and vision-oriented views, we propose a novel mutual learning based model MMNet to enhance the multimodal fusion for improving fake news detection. Additionally, in MMNet, we design a new multimodal fusion block to not only capture inter-modality interactions but also extract critical features for fake news detection. Extensive experiments on two public real datasets demonstrate that MMNet better captures the inter-dependencies of multimodal information, and outperforms state-of-the-art methods.
A Novel Efficient and Effective Preprocessing Strategy for Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, text-classification, Text Classification

Text classification is an essential task of natural language processing. Preprocessing, which determines the representation of text features, is one of the key steps of text classification architecture. This paper proposes a novel efficient and effective preprocessing strategy with three methods for text classification using OMP algorithm to complete the classification. The main idea of our new preprocessing strategy is that we combine regular filtering and/or stopwords removal with tokenization and lowcase convertion, which can effectively reduce  the feature dimension and improve the quality of text feature matrix to some extent. Simulation tests on 20Newsgroups dataset show compared with the existing state-of-the-art method, our new best method reduces the number of features by 19.85$\%$, 34.35$\%$, 26.25$\%$, and 38.67$\%$, and increase the speed of text classification by 17.38\%, 25.64\%, 23.76\%, and 33.38\% with similar classification accuracy on religion, computer, science and sport data, respectively.
KETOD: Knowledge-Enriched Task-Oriented Dialogue
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Response Generation

Existing studies in dialogue system research mostly treat task-oriented dialogue and chit-chat separately. Towards building a human-like assistant that can converse naturally and seamlessly with users, the system needs to be able to conduct both types of conversations effectively. In this work, we investigate how task-oriented dialogue and knowledge-grounded chit-chat can be effectively integrated into a single model. To this end, we create a new dataset, KETOD (Knowledge-Enriched Task-Oriented Dialogue), where we naturally enrich task-oriented dialogue with chit-chats based on relevant entity knowledge. We also propose two new models, SimpleToDPlus and Combiner, for the proposed task. Experimental results on both automatic and human evaluations show that the proposed methods can significantly improve the performance in knowledge-enriched response generation while maintaining a competitive task-oriented dialog performance. We believe our new dataset will be a valuable resource for future studies. We will make the code and the dataset publicly available upon acceptance.
Persian Natural Language Inference: A Meta-learning approach
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Meta-Learning, Natural Language Inference, Transfer Learning

Incorporating information from other languages can improve the results of tasks in low-resource languages. A powerful method of building functional natural language processing systems for low-resource languages is to combine multilingual pre-trained representations with cross-lingual transfer learning. In general, however, shared representations are learned separately, either across tasks or across languages. This paper proposes a meta-learning approach for inferring natural language in Persian. Alternately, meta-learning uses different task information (such as QA in Persian) or other language information (such as natural language inference in English). Also, we investigate the role of task augmentation strategy for forming additional high-quality tasks. We evaluate the proposed method using four languages and an auxiliary task. Compared to the baseline approach, the proposed model consistently outperforms it, improving accuracy by roughly six percent. We also examine the effect of finding appropriate initial parameters using zero-shot evaluation and CCA similarity.
On Event Detection in Scientific Papers: A Multi-Domain Dataset
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Event Detection

Given the growing number of scientific papers, automatic information extraction in scientific documents is important for efficient knowledge update and discovery. A key component in scientific papers involves rhetorical activities/events to convey new knowledge and convince readers of the correctness. This work explores a new information extraction problem for scientific documents, aiming to identify event trigger words of rhetorical events/activities, i.e., event detection (ED). To promote future research in this area, we present SciEvent, the first and new dataset for event detection in scientific documents. SciEvent annotates scientific papers of four different domains (i.e., computer science, biology, physics, and mathematics) using 8 popular event types. Our experiments on SciEvent demonstrate the challenges of scientific ED for existing models and call for further research effort in this area. We will publicly release SciEvent to facilitate future research.