pwc-archive/papers-with-abstracts huggingface dataset:
Text Simplification with Reinforcement Learning Using Supervised Rewards on Grammaticality, Meaning Preservation, and Simplicity
Authors: Akifumi Nakamachi, Tomoyuki Kajiwara, Yuki Arase
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Text Generation, Text Simplification

We optimize rewards of reinforcement learning in text simplification using metrics that are highly correlated with human-perspectives. To address problems of exposure bias and loss-evaluation mismatch, text-to-text generation tasks employ reinforcement learning that rewards task-specific metrics. Previous studies in text simplification employ the weighted sum of sub-rewards from three perspectives: grammaticality, meaning preservation, and simplicity. However, the previous rewards do not align with human-perspectives for these perspectives. In this study, we propose to use BERT regressors fine-tuned for grammaticality, meaning preservation, and simplicity as reward estimators to achieve text simplification conforming to human-perspectives. Experimental results show that reinforcement learning with our rewards balances meaning preservation and simplicity. Additionally, human evaluation confirmed that simplified texts by our method are preferred by humans compared to previous studies.
Label Representations in Modeling Classification as Text Generation
Authors: Xinyi Chen, Jingxian Xu, Alex Wang
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Classification, text-classification, Text Classification, Text Generation, Transfer Learning

Several recent state-of-the-art transfer learning methods model classification tasks as text generation, where labels are represented as strings for the model to generate. We investigate the effect that the choice of strings used to represent labels has on how effectively the model learns the task. For four standard text classification tasks, we design a diverse set of possible string representations for labels, ranging from canonical label definitions to random strings. We experiment with T5 on these tasks, varying the label representations as well as the amount of training data. We find that, in the low data setting, label representation impacts task performance on some tasks, with task-related labels being most effective, but fails to have an impact on others. In the full data setting, our results are largely negative: Different label representations do not affect overall task performance.
Generating Inflectional Errors for Grammatical Error Correction in Hindi
Authors: Ankur Sonawane, Sujeet Kumar Vishwakarma, Bhavana Srivastava, Anil Kumar Singh
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Grammatical Error Correction

Automated grammatical error correction has been explored as an important research problem within NLP, with the majority of the work being done on English and similar resource-rich languages. Grammar correction using neural networks is a data-heavy task, with the recent state of the art models requiring datasets with millions of annotated sentences for proper training. It is difficult to find such resources for Indic languages due to their relative lack of digitized content and complex morphology, compared to English. We address this problem by generating a large corpus of artificial inflectional errors for training GEC models. Moreover, to evaluate the performance of models trained on this dataset, we create a corpus of real Hindi errors extracted from Wikipedia edits. Analyzing this dataset with a modified version of the ERRANT error annotation toolkit, we find that inflectional errors are very common in this language. Finally, we produce the initial baseline results using state of the art methods developed for English.
Touch Editing: A Flexible One-Time Interaction Approach for Translation
Authors: Qian Wang, Jiajun Zhang, Lemao Liu, Guoping Huang, Chengqing Zong
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Translation

We propose a touch-based editing method for translation, which is more flexible than traditional keyboard-mouse-based translation postediting. This approach relies on touch actions that users perform to indicate translation errors. We present a dual-encoder model to handle the actions and generate refined translations. To mimic the user feedback, we adopt the TER algorithm comparing between draft translations and references to automatically extract the simulated actions for training data construction. Experiments on translation datasets with simulated editing actions show that our method significantly improves original translation of Transformer (up to 25.31 BLEU) and outperforms existing interactive translation methods (up to 16.64 BLEU). We also conduct experiments on post-editing dataset to further prove the robustness and effectiveness of our method.
Graph Attention Network with Memory Fusion for Aspect-level Sentiment Analysis
Authors: Li Yuan, Jin Wang, Liang-Chih Yu, Xuejie Zhang
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Graph Attention, Sentiment Analysis

Aspect-level sentiment analysis(ASC) predicts each specific aspect term{'}s sentiment polarity in a given text or review. Recent studies used attention-based methods that can effectively improve the performance of aspect-level sentiment analysis. These methods ignored the syntactic relationship between the aspect and its corresponding context words, leading the model to focus on syntactically unrelated words mistakenly. One proposed solution, the graph convolutional network (GCN), cannot completely avoid the problem. While it does incorporate useful information about syntax, it assigns equal weight to all the edges between connected words. It may still incorrectly associate unrelated words to the target aspect through the iterations of graph convolutional propagation. In this study, a graph attention network with memory fusion is proposed to extend GCN{'}s idea by assigning different weights to edges. Syntactic constraints can be imposed to block the graph convolutional propagation of unrelated words. A convolutional layer and a memory fusion were applied to learn and exploit multiword relations and draw different weights of words to improve performance further. Experimental results on five datasets show that the proposed method yields better performance than existing methods.
FERNet: Fine-grained Extraction and Reasoning Network for Emotion Recognition in Dialogues
Authors: YingMei Guo, Zhiyong Wu, Mingxing Xu
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Emotion Recognition

Unlike non-conversation scenes, emotion recognition in dialogues (ERD) poses more complicated challenges due to its interactive nature and intricate contextual information. All present methods model historical utterances without considering the content of the target utterance. However, different parts of a historical utterance may contribute differently to emotion inference of different target utterances. Therefore we propose Fine-grained Extraction and Reasoning Network (FERNet) to generate target-specific historical utterance representations. The reasoning module effectively handles both local and global sequential dependencies to reason over context, and updates target utterance representations to more informed vectors. Experiments on two benchmarks show that our method achieves competitive performance compared with previous methods.
SentiRec: Sentiment Diversity-aware Neural News Recommendation
Authors: Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Diversity, News Recommendation

Personalized news recommendation is important for online news services. Many news recommendation methods recommend news based on their relevance to users{'} historical browsed news, and the recommended news usually have similar sentiment with browsed news. However, if browsed news is dominated by certain kinds of sentiment, the model may intensively recommend news with the same sentiment orientation, making it difficult for users to receive diverse opinions and news events. In this paper, we propose a sentiment diversity-aware neural news recommendation approach, which can recommend news with more diverse sentiment. In our approach, we propose a sentiment-aware news encoder, which is jointly trained with an auxiliary sentiment prediction task, to learn sentiment-aware news representations. We learn user representations from browsed news representations, and compute click scores based on user and candidate news representations. In addition, we propose a sentiment diversity regularization method to penalize the model by combining the overall sentiment orientation of browsed news as well as the click and sentiment scores of candidate news. Extensive experiments on real-world dataset show that our approach can effectively improve the sentiment diversity in news recommendation without performance sacrifice.
BCTH: A Novel Text Hashing Approach via Bayesian Clustering
Authors: Ying Wenjie, Yuquan Le, Hantao Xiong
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Clustering, Information Retrieval, Retrieval

Similarity search is to find the most similar items for a certain target item. The ability of similarity search at large scale plays a significant role in many information retrieval applications, and thus has received much attention. Text hashing is a promising strategy, which utilizes binary encoding to represent documents, obtaining attractive performance. This paper makes the first attempt to utilize Bayesian Clustering for Text Hashing, dubbed as BCTH. Specifically, BCTH is able to map documents to binary codes by utilizing multiple Bayesian Clusterings in parallel, where each Bayesian Clustering is responsible for one bit. Our approach employs the bit-balanced constraint to maximize the amount of information in each bit. Meanwhile, the bit-uncorrected constraint is adopted to keep the independence among all bits. The time complexity of BCTH is linear, where the hash codes and hash function are jointly learned. The experimental results, based on four widely-used datasets, demonstrate that BCTH is competitive, compared with currently competitive baselines in the perspective of both precision and training speed.
Lightweight Text Classifier using Sinusoidal Positional Encoding
Authors: Byoung-Doo Oh, Yu-Seop Kim
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Position, text-classification, Text Classification

Large and complex models have recently been developed that require many parameters and much time to solve various problems in natural language processing. This paper explores an efficient way to avoid models being too complicated and ensure nearly equal performance to models showing the state-of-the-art. We propose a single convolutional neural network (CNN) using the sinusoidal positional encoding (SPE) in text classification. The SPE provides useful position information of a word and can construct a more efficient model architecture than before in a CNN-based approach. Our model can significantly reduce the parameter size (at least 67{\%}) and training time (up to 85{\%}) while maintaining similar performance to the CNN-based approach on multiple benchmark datasets.
A Simple and Effective Usage of Word Clusters for CBOW Model
Authors: Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Word Embeddings

We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.
Investigating Learning Dynamics of BERT Fine-Tuning
Authors: Yaru Hao, Li Dong, Furu Wei, Ke Xu
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Language Modeling, Language Modelling

The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks. In this paper, we inspect the learning dynamics of BERT fine-tuning with two indicators. We use JS divergence to detect the change of the attention mode and use SVCCA distance to examine the change to the feature extraction mode during BERT fine-tuning. We conclude that BERT fine-tuning mainly changes the attention mode of the last layers and modifies the feature extraction mode of the intermediate and last layers. Moreover, we analyze the consistency of BERT fine-tuning between different random seeds and different datasets. In summary, we provide a distinctive understanding of the learning dynamics of BERT fine-tuning, which sheds some light on improving the fine-tuning results.
Exploiting WordNet Synset and Hypernym Representations for Answer Selection
Authors: Weikang Li, Yunfang Wu
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Answer Selection, Question Answering, Sentence

Answer selection (AS) is an important subtask of document-based question answering (DQA). In this task, the candidate answers come from the same document, and each answer sentence is semantically related to the given question, which makes it more challenging to select the true answer. WordNet provides powerful knowledge about concepts and their semantic relations so we employ WordNet to enrich the abilities of paraphrasing and reasoning of the network-based question answering model. Specifically, we exploit the synset and hypernym concepts to enrich the word representation and incorporate the similarity scores of two concepts that share the synset or hypernym relations into the attention mechanism. The proposed WordNet-enhanced hierarchical model (WEHM) consists of four modules, including WordNet-enhanced word representation, sentence encoding, WordNet-enhanced attention mechanism, and hierarchical document encoding. Extensive experiments on the public WikiQA and SelQA datasets demonstrate that our proposed model significantly improves the baseline system and outperforms all existing state-of-the-art methods by a large margin.
A Simple Text-based Relevant Location Prediction Method using Knowledge Base
Authors: Mei Sasaki, Shumpei Okura, Shingo Ono
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Articles

In this paper, we propose a simple method to predict salient locations from news article text using a knowledge base (KB). The proposed method uses a dictionary of locations created from the KB to identify occurrences of locations in the text and uses the hierarchical information between entities in the KB for assigning appropriate saliency scores to regions. It allows prediction at arbitrary region units and has only a few hyperparameters that need to be tuned. We show using manually annotated news articles that the proposed method improves the f-measure by {\textgreater} 0.12 compared to multiple baselines.
BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation
Authors: Reinald Adrian Pugoy, Hung-Yu Kao
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Collaborative Filtering, Recommendation Systems, Word Embeddings

We propose a novel, accurate, and explainable recommender model (BENEFICT) that addresses two drawbacks that most review-based recommender systems face. First is their utilization of traditional word embeddings that could influence prediction performance due to their inability to model the word semantics{'} dynamic characteristic. Second is their black-box nature that makes the explanations behind every prediction obscure. Our model uniquely integrates three key elements: BERT, multilayer perceptron, and maximum subarray problem to derive contextualized review features, model user-item interactions, and generate explanations, respectively. Our experiments show that BENEFICT consistently outperforms other state-of-the-art models by an average improvement gain of nearly 7{\%}. Based on the human judges{'} assessment, the BENEFICT-produced explanations can capture the essence of the customer{'}s preference and help future customers make purchasing decisions. To the best of our knowledge, our model is one of the first recommender models to utilize BERT for neural collaborative filtering.
Transformer-based Approach for Predicting Chemical Compound Structures
Authors: Yutaro Omote, Kyoumoto Matsushita, Tomoya Iwakura, Akihiro Tamura, Takashi Ninomiya
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Multi-Task Learning

By predicting chemical compound structures from their names, we can better comprehend chemical compounds written in text and identify the same chemical compound given different notations for database creation. Previous methods have predicted the chemical compound structures from their names and represented them by Simplified Molecular Input Line Entry System (SMILES) strings. However, these methods mainly apply handcrafted rules, and cannot predict the structures of chemical compound names not covered by the rules. Instead of handcrafted rules, we propose Transformer-based models that predict SMILES strings from chemical compound names. We improve the conventional Transformer-based model by introducing two features: (1) a loss function that constrains the number of atoms of each element in the structure, and (2) a multi-task learning approach that predicts both SMILES strings and InChI strings (another string representation of chemical compound structures). In evaluation experiments, our methods achieved higher F-measures than previous rule-based approaches (Open Parser for Systematic IUPAC Nomenclature and two commercially used products), and the conventional Transformer-based model. We release the dataset used in this paper as a benchmark for the future research.
Neural Gibbs Sampling for Joint Event Argument Extraction
Authors: Xiaozhi Wang, Shengyu Jia, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Jie zhou
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Event Argument Extraction, Event Extraction

Event Argument Extraction (EAE) aims at predicting event argument roles of entities in text, which is a crucial subtask and bottleneck of event extraction. Existing EAE methods either extract each event argument roles independently or sequentially, which cannot adequately model the joint probability distribution among event arguments and their roles. In this paper, we propose a Bayesian model named Neural Gibbs Sampling (NGS) to jointly extract event arguments. Specifically, we train two neural networks to model the prior distribution and conditional distribution over event arguments respectively and then use Gibbs sampling to approximate the joint distribution with the learned distributions. For overcoming the shortcoming of the high complexity of the original Gibbs sampling algorithm, we further apply simulated annealing to efficiently estimate the joint probability distribution over event arguments and make predictions. We conduct experiments on the two widely-used benchmark datasets ACE 2005 and TAC KBP 2016. The Experimental results show that our NGS model can achieve comparable results to existing state-of-the-art EAE methods. The source code can be obtained from https://github.com/THU-KEG/NGS.
Named Entity Recognition in Multi-level Contexts
Authors: Yubo Chen, Chuhan Wu, Tao Qi, Zhigang Yuan, Yongfeng Huang
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Multi-Task Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Sentence

Named entity recognition is a critical task in the natural language processing field. Most existing methods for this task can only exploit contextual information within a sentence. However, their performance on recognizing entities in limited or ambiguous sentence-level contexts is usually unsatisfactory. Fortunately, other sentences in the same document can provide supplementary document-level contexts to help recognize these entities. In addition, words themselves contain word-level contextual information since they usually have different preferences of entity type and relative position from named entities. In this paper, we propose a unified framework to incorporate multi-level contexts for named entity recognition. We use TagLM as our basic model to capture sentence-level contexts. To incorporate document-level contexts, we propose to capture interactions between sentences via a multi-head self attention network. To mine word-level contexts, we propose an auxiliary task to predict the type of each word to capture its type preference. We jointly train our model in entity recognition and the auxiliary classification task via multi-task learning. The experimental results on several benchmark datasets validate the effectiveness of our method.
UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database
Authors: Canwen Xu, Tao Ge, Chenliang Li, Furu Wei
Date: 2020-12-01 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Language Modeling, Language Modelling

Chinese and Japanese share many characters with similar surface morphology. To better utilize the shared knowledge across the languages, we propose UnihanLM, a self-supervised Chinese-Japanese pretrained masked language model (MLM) with a novel two-stage coarse-to-fine training approach. We exploit Unihan, a ready-made database constructed by linguistic experts to first merge morphologically similar characters into clusters. The resulting clusters are used to replace the original characters in sentences for the coarse-grained pretraining of the MLM. Then, we restore the clusters back to the original characters in sentences for the fine-grained pretraining to learn the representation of the specific characters. We conduct extensive experiments on a variety of Chinese and Japanese NLP benchmarks, showing that our proposed UnihanLM is effective on both mono- and cross-lingual Chinese and Japanese tasks, shedding light on a new path to exploit the homology of languages.