pwc-archive/papers-with-abstracts huggingface dataset:
Alignment based Sequence Ensemble with Multiple Results from a Single Neural Model Architecture
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: POS

Sequence labeling is a fundamental framework that provides the elemental structure and content information for additional natural language processing. However, existing proposed ensemble approaches do not focus on sequence alignment during post-processing. Here, we present a weighted ensemble technique using a sequence alignment approach for a sequence labeling task. Our proposed technique addresses two problems of ensemble technique. First, an ensemble technique requires a multiple model system, and we used a model with multiple random seeds and an additional dropout layer to build a simple ensemble system. Second, we focused on a sequence label list that increased accuracy using alignment for the ensemble technique during post-processing. We evaluated our approach on a representative sequence labeling tasks, part-of-speech and dependency parser. Most results obtained by ensemble sequence alignment approach with various sub-sequence units showed an increase in the F1-score over a single neural network result in the sequence labeling task. Comparing with the hard voting result on the Penn-treebank\cite{marcus1993PTB}, the F1-scores increased up to 0.45 at POS-tagged dataset, and up to 0.12 at DP-tagged dataset.
Semantic-Oriented Unlabeled Priming for Large-Scale Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: In-Context Learning

Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning. Unfortunately, for in-context learning there is currently no way to leverage unlabeled data, which is often much easier to obtain in large quantities than labeled examples. In this work, we therefore investigate ways to make use of unlabeled examples to improve the zero-shot performance of pretrained language models without any finetuning: We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples, assigning labels to them in a zero-shot fashion, and then using them for in-context learning. We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.
Assessing the Coherence Modeling Capabilities of Pretrained Transformer-based Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity, Sentence, Sentence Ordering

The task of ordering a shuffled set of sentences into a coherent text is used to evaluate the capacity of a model to understand causal and temporal relations between entities and events. Recent approaches rely on pretrained Transformer-based models, but it remains unknown whether the differences between them, such as size, pretraining data and objectives, affect their coherence modeling capacity. We present a simple architecture for sentence ordering that relies exclusively on pretrained Transformer-based encoder-only models. This allows us to compare the coherence modeling capabilities of the monolingual and multilingual versions of BERT, RoBERTa, and DistilBERT. We show that RoBERTa-based models outperform BERT-based models and are more robust when ordering longer documents with more than 10 sentences. Thus, the intuitive advantage offered by sentence-based objectives such as Next Sentence Prediction used in BERT is effectively compensated by the higher amount and diversity of the training data used in RoBERTa. However, the difference between multilingual versions of BERT and RoBERTa is narrower. This suggests that exposure to  different languages partially makes up for the benefits of larger and more diverse training data.
What Makes Machine Reading Comprehension Questions Difficult? Investigating Variation in Passage Sources and Question Types
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Logical Reasoning, Machine Reading Comprehension, Multiple-choice, Natural Language Understanding, Reading Comprehension

For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems.
However, we do not yet know how best to select passages to collect a variety of challenging examples.
In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples.
To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty.
Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for technical passages.
These results suggest that when creating a new benchmark dataset, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority.
A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: SSTOD

A slot value might be provided segment by segment over multiple-turn interactions in a dialog, especially for some important information such as phone numbers and names. It is a common phenomenon in daily life, but little attention has been paid to it in previous work. To fill the gap, this paper defines a new task named Sub-Slot based Task-Oriented Dialog (SSTOD) and builds a Chinese dialog dataset SSD for boosting research on SSTOD. The dataset includes total 40K dialogs and 500K utterances from four different domains: Chinese names, phone numbers, ID numbers and license plate numbers. The data is well annotated with sub-slot values, slot values, dialog states and actions. We find some new linguistic phenomena and interactive manners in SSTOD which raise some new challenges of building agents for the task. We test three state-of-the-art models on SSTOD and find they cannot handle the new task well on any of the four domains. We also investigate an improved model by involving slot knowledge in a plug-in manner. More work should be done to meet the new challenges raised from SSTOD which widely exists in real-life applications.
Multi-Party Empathetic Dialogue Generation: A New Task for Dialog Systems
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dialogue Generation, Response Generation

Empathetic dialogue assembles emotion understanding, feeling projection, and appropriate response generation. Existing work for empathetic dialogue generation concentrates on the two-party conversation scenario. Multi-party dialogues, however, are pervasive in reality. Furthermore, emotion and sensibility are typically confused; a refined empathy analysis is needed for comprehending fragile and nuanced human feelings. We address these issues by proposing a novel task called Multi-Party Empathetic Dialogue Generation in this study. A new dataset MPED with 130k multi-party dialogues is correspondingly presented for this task, which makes up for the absence of a large-scale benchmark in this field. Additionally, a Static-Dynamic model for Multi-Party Empathetic Dialogue Generation, SDMPED, is introduced as a baseline by exploring the static sensibility and dynamic emotion for the multi-party empathetic dialogue learning, the aspects that help SDMPED achieve the state-of-the-art performance on MPED.
An Isotropy Analysis in the Multilingual BERT Embedding Space
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Several studies have explored various advantages of multilingual pre-trained models (e.g., multilingual BERT) in capturing shared linguistic knowledge. However, their limitations have not been paid enough attention to. In this paper, we investigate the representation degeneration problem and outlier dimensions in multilingual contextual word representations (CWRs) of BERT. We show that though mBERT exhibits no outliers among its representations, its multilingual embedding space is highly anisotropic. Furthermore, our experimental results demonstrate that similarly to their monolingual counterparts, increasing the isotropy of multilingual embedding spaces can significantly improve their representation power and performance. Our analysis indicates that, although the degenerated directions vary in different languages, they encode similar linguistic knowledge, suggesting a shared linguistic space among languages.
Is Cross-lingual Evaluation Only About Cross-lingual?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval, Sentence, Sentence Retrieval

Multilingual pre-trained language models (mPLMs) have achieved great success on various cross-lingual tasks. However, we find that the higher performance on these tasks cannot be regarded as the better cross-lingual ability because models’ task-specific abilities can also influence the performance. In this work, we do a comprehensive study on two representative cross-lingual evaluation protocols: sentence retrieval and zero-shot transfer. We find that current cross-lingual evaluation results strongly depend on mPLMs' task-specific abilities so that the performance can be improved without any improvement in models' cross-lingual ability. To have more accurate comparisons of cross-lingual ability between mPLMs, we propose two new indexes based on the two evaluation protocols: calibrated sentence retrieval performance and transfer rate, and experimentally show that our proposed indexes effectively eliminate the effects of task-specific abilities on the cross-lingual evaluation.
HADE: Hierarchical Affective Dialog Encoder for Personality Recognition in Conversation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Personality Recognition in Conversation

Personality recognition in conversation aims to determine the personality traits of speakers through the dialogue content, which is of great importance in designing personalized conversational AI. Existing methods that use only linguistic patterns in utterances limit their performance. To fill in the gap, we investigate the effectiveness of incorporating affective information and modeling the interactions among speakers in conversations for personality recognition. However, available corpus with personality and explicit affective annotations is rare. Besides, modeling the dialog flow with multiple speakers is difficult. Faced with the issues, we proposed Hierarchical Affective Dialog Encoder (HADE) for effective personality recognition in conversation. HADE utilizes manual annotated Valance-Arousal-Dominance (VAD) vectors of single words and implicitly extracts affective information from utterances. Then, it introduces a hierarchical architecture with the dialog state embeddings to identify the speakers and encode the whole dialog flow. Finally, the affective information is integrated by an auxiliary VAD regression task to enhance personality recognition. Extensive experiments on a well-known dataset, \textbf{FriendsPersona}, demonstrate the effectiveness of our method compared with state-of-the-art models. Besides, we conduct an ablation study to discuss different approaches for integrating affective information and dialog flow modeling; the design of both parts in HADE is also verified to be effective for personality recognition in conversation.
A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Early exiting allows instances to exit at different layers according to the estimation of difficulty.
Previous works usually adopt heuristic metrics such as the entropy of internal outputs to measure instance difficulty, which suffers from generalization and threshold-tuning. In contrast, learning to exit, or learning to predict instance difficulty is a more appealing way. Though some effort has been devoted to employing such "learn-to-exit" modules, it is still unknown whether and how well the instance difficulty can be learned. As a response, we first conduct experiments on the learnability of instance difficulty, which demonstrates that modern neural models perform poorly on predicting instance difficulty. Based on this observation, we propose a simple-yet-effective Hash-based Early Exiting approach HashEE) that replaces the learn-to-exit modules with hash functions to assign each token to a fixed exiting layer. Different from previous methods, HashEE requires no internal classifiers nor extra parameters, and therefore is more efficient.HashEE can be used in various tasks (including language understanding and generation) and model architectures such as seq2seq models. Experimental results on classification, regression, and generation tasks demonstrate that HashEE can achieve higher performance with fewer FLOPs and inference time compared with previous state-of-the-art early exiting methods.
PREME: Preference-based Meeting Exploration through an Interactive Questionnaire
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The recent increase in the volume of online meetings necessitates automated tools for managing and organizing the material,  especially when an attendee has missed the discussion and needs assistance in quickly exploring it.  In  this work,   we propose a  novel end-to-end framework for generating interactive questionnaires for preference-based meeting exploration.  As a result, users are supplied with a list of suggested questions reflecting their preferences.  Since the task is new,  we introduce an automatic evaluation strategy.   Namely,  it measures how much the generated questions via questionnaire are answerable to ensure factual correctness and covers the source meeting for the depth of possible exploration.
CaM-Gen: Causally-aware Guided Text Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Causal Inference, Text Generation

Content is created for a well-defined purpose, often described by a metric or signal represented in the form of structured information. The relationship between the goal (metrics) of target content and the content itself is non-trivial. While large-scale language models show promising text generation capabilities, guiding the generated text with external metrics is challenging.
These metrics and content tend to have inherent relationships and not all of them may be of consequence. We introduce CaM-Gen: Causally-aware Generative Networks guided by user-defined target metrics incorporating the causal relationships between the metric and content features. We leverage causal inference techniques to identify causally significant aspects of a text that lead to the target metric and then explicitly guide generative models towards these by a feedback mechanism. We propose this mechanism for variational autoencoder and Transformer-based generative models. The proposed models beat baselines in terms of the target metric control while maintaining fluency and language quality of the generated text. To the best of our knowledge, this is one of the early attempts at controlled generation incorporating a metric guide using causal inference.
Description-Driven Task-Oriented Dialog Modeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: dialog state tracking

Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained in task-specific ontology or schemata. Since these schemata are designed by system developers, the naming convention for slots and intents is not uniform across tasks, and may not convey their semantics effectively. This can lead to models memorizing arbitrary patterns in data, resulting in suboptimal performance and generalization. In this paper, we propose that schemata should be modified by replacing names or notations entirely with natural language descriptions. We show that a language description-driven system exhibits better understanding of task specifications, higher performance on state tracking, improved data efficiency, and effective zero-shot transfer to unseen tasks. Following this paradigm, we present a simple yet effective Description-Driven Dialog State Tracking (D3ST) model, which relies purely on schema descriptions and an "index-picking'' mechanism. We demonstrate the superiority in quality, data efficiency and robustness of our approach as measured on the MultiWOZ (Budzianowski et al.,2018), SGD (Rastogi et al., 2020), and the recent SGD-X (Lee et al., 2021) benchmarks.
Metadata Shaping: Natural Language Annotations for the Long Tail
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Language models (LMs) struggle to capture knowledge about rare entities. To better capture entity knowledge, a common procedure in prior work is to start with a base LM such as BERT and to modify the LM architecture or objective function to produce a knowledge-aware LM. Proposed knowledge-aware LMs perform well compared to base LMs on entity-rich tasks; however deploying, understanding, and maintaining many different specialized architectures is challenging, and they also often introduce additional computational costs. Thus we ask to what extent we can match the quality of these architectures using a base LM and only changing the data. We propose metadata shaping, a method which inserts readily available entity metadata, such as descriptions and categorical tags, into examples at train and inference time based on mutual information. Intuitively, if metadata corresponding to popular entities overlap with metadata for rare entities, the LM may be able to better reason about the rare entities using patterns learned from similar popular entities. On standard entity-rich tasks (TACRED, FewRel, OpenEntity), metadata shaping exceeds the BERT-baseline by an average of 4.3 F1 points and achieves state-of-the-art results. We further show the gains are on average 4.4x larger for the slice of examples containing tail vs. popular entities.
Packed Levitated Marker for Entity and Relation Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: NER, Relation, Relation Extraction

Recent entity and relation extraction works focus on investigating how to obtain a better span representation from the pre-trained encoder. However, a major limitation of existing works is that they ignore the interrelation between spans (pairs). In this work, we propose a novel span representation approach, named Packed Levitated Markers (PL-Marker),  to consider the interrelation between the spans (pairs) by strategically packing the markers in the encoder. In particular, we propose a neighborhood-oriented packing strategy, which  considers the neighbor spans integrally to better model the entity boundary information. Furthermore, for those more complicated span pair classification tasks, we  design a subject-oriented packing strategy, which packs each subject and all its objects to model the interrelation between the same-subject span pairs. The experimental results show that, with the enhanced marker feature, our model advances baselines on six NER benchmarks, and obtains a 3.5%-3.6% strict relation F1 improvement with higher speed over previous state-of-the-art models on ACE04 and ACE05. All the code and data of this paper will be made publicly available.
What does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Transfer Learning

Procedural text contains rich anaphoric phenomena yet has not received much attention in NLP. To fill this gap, we investigate the textual properties of two types of procedural text, recipes and chemical patents, and generalize an anaphora annotation framework developed for the chemical domain for modelling anaphoric phenomena in recipes. We apply this framework to annotate the RecipeRef corpus with both bridging and coreference relations. Through comparison to chemical patents, we show the complexity of anaphora resolution in recipes. We demonstrate empirically that transfer learning from the chemical domain improves resolution of anaphora in recipes, suggesting transferability of general procedural knowledge.  The corpus is made available at \url{withheld\_for\_review}.
Searching for fingerspelled content in American Sign Language
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval, Translation

Natural language processing for sign language video---including tasks like recognition, translation, and search---is crucial for making artificial intelligence technologies accessible to deaf individuals, and is gaining research interest in recent years.  In this paper, we address the problem of searching for fingerspelled keywords or key phrases in raw sign language videos.  This is an important task since significant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before.  We propose an end-to-end model for this task, FSS-Net, that jointly detects fingerspelling and matches it to a text sequence. Our experiments, done on a large public dataset of ASL fingerspelling in the wild, show the importance of fingerspelling detection as a component of a search and retrieval model.  Our model significantly outperforms baseline methods adapted from prior work on related tasks.
Measuring Fairness of Text Classifiers via Prediction Sensitivity
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Attribute, counterfactual, Fairness, Prediction, Sensitivity, text-classification, Text Classification

With the rapid growth in language processing applications, fairness has emerged as an important consideration in data-driven solutions.  Although various fairness definitions have been explored in the recent literature, there is lack of consensus on which metrics most accurately reflect the fairness of a system. 
       In this work, we propose a new formulation -- accumulated prediction sensitivity, which measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. The metric attempts to quantify the extent to which a single prediction depends on a protected attribute, where the protected attribute encodes the membership status of an individual in a protected group.  
       We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness. It also correlates well with humans' perception of fairness. We conduct experiments on two text classification datasets -- Jigsaw Toxicity, and Bias in Bios, and evaluate the correlations between metrics and manual annotations on whether the model produced a fair outcome. We observe that the proposed fairness metric based on prediction sensitivity is statistically significantly more correlated with human annotation than the existing counterfactual fairness metric.