pwc-archive/papers-with-abstracts huggingface dataset:
Let the Model Decide its Curriculum for Multitask Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Multi-Task Learning

Curriculum learning strategies in prior multi-task learning approaches arrange datasets in a difficulty hierarchy either based on human perception or by exhaustively searching the optimal arrangement. However, human perception of difficulty may not always correlate well with machine interpretation leading to poor performance and exhaustive search is computationally expensive. Addressing these concerns, we propose two classes of techniques to arrange training instances into a learning curriculum based on difficulty scores computed via model-based approaches. The two classes i.e Dataset-level and Instance-level differ in the granularity of arrangement. We conduct comprehensive experiments with $12$ datasets and show that instance-level and dataset-level techniques lead to an average performance improvement of $4.17\%$ and $3.15\%$ over their respective baseline methods. Furthermore, we find that most of this improvement comes from correctly answering the difficult instances, implying a greater efficacy of our techniques on difficult tasks.
STT: Soft Template Tuning for Few-Shot Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Few-Shot Learning, Language Modeling, Language Modelling, Masked Language Modeling, Sentiment Analysis, Sentiment Classification

With the rapid expansion of large pre-trained language models, fine-tuning all the model parameters for downstream tasks is becoming computationally prohibitive. The recently developed prompt-based methods freeze the entire model parameters and only update the so-called prompt parameters appended to the inputs, significantly reducing the burden of fully fine-tuning. However, standard prompt-based methods mainly consider the case where sufficient data of downstream tasks are available. It is still unclear whether the advantage can be transferred to the few-shot regime, where only limited data are available for each downstream task. Our empirical studies suggest there is still a gap between prompt tuning and fully fine-tuning for few-shot learning. We propose a new prompt-tuning framework, called Soft Template Tuning (STT), to bridge the gap. STT combines manual prompts and auto-prompts, and treats downstream classification tasks as a masked language modeling task. STT can close the gap between fine-tuning and prompt-based methods without introducing additional parameters. Importantly, it can even outperform the time- and resource-consuming fine-tuning method on sentiment classification tasks.
Extreme Zero-Shot Learning for Extreme Text Classification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification, Zero-Shot Learning

The eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set. However, the XMC setup faces two challenges: (1) it is not generalizable to predict unseen labels in dynamic environments, and (2) it requires a large amount of supervised (instance, label) pairs, which can be difficult to obtain for emerging domains. In this paper, we consider a more practical scenario called Extreme Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text of instances and labels are accessible. Few-Shot XMC (FS-XMC), an extension to EZ-XMC with limited supervision is also investigated. To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based encoders with self-supervised contrastive losses. Specifically, we develop a pre-training method $\textbf{MACLR}$, which thoroughly leverages the raw text with techniques including $\textbf{M}$ulti-scale $\textbf{A}$daptive $\textbf{C}$lustering, $\textbf{L}$abel $\textbf{R}$egularization, and self-training with pseudo positive pairs. Experimental results on four public EZ-XMC datasets demonstrate that MACLR achieves superior performance compared to all other leading baseline methods, in particular with approximately 5-10% improvement in precision and recall on average. Moreover, we show that our pre-trained encoder can be further improved on FS-XMC when there are a limited number of ground-truth positive pairs in training.
Learning the Ordering of Coordinate Compounds and Elaborate Expressions in Hmong, Lahu, and Chinese
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Coordinate compounds (CCs) and elaborate expressions (EEs) are coordinate constructions common in languages of East and Southeast Asia. Mortensen (2006) claims that (1) the linear ordering of EEs and CCs in Hmong, Lahu, and Chinese can be predicted via phonological hierarchies and (2) that these phonological hierarchies lack a clear phonetic rationale. These claims are significant because morphosyntax has often been seen as in a feed-forward relationship with phonology, and phonological generalizations have often been assumed to be phonetically "natural". We investigate whether the ordering of CCs and EEs can be learned empirically and whether computational models (classifiers and sequence-labeling models) learn unnatural hierarchies similar to those posited by Mortensen (2006). We find that decision trees and SVMs learn to predict the order of CCs/EEs on the basis of phonology, beating strong baselines for all three languages, with DTs learning hierarchies strikingly similar to those proposed by Mortensen. However, we also find that a neural sequence labeling model is able to learn the ordering of elaborate expressions in Hmong very effectively without using any phonological information. We argue that EE ordering can be learned through two independent routes: phonology and lexical distribution, presenting a more nuanced picture than previous work.
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Few-Shot Learning, Natural Language Inference

Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. Such success can give the impression that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompts manually written for natural language inference (NLI). We find that models learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively “good” prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021; Wei et al., 2021). Despite some success, instruction-tuned models are capable of producing good predictions with misleading prompts even at zero shots. In sum, notwithstanding prompt-based models’ impressive improvement, we find evidence of serious limitations that question the degree to which language models really understand the meaning of prompts in the way humans do.
That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Pretraining multimodal models on Electronic Health Records (EHRs) provides a means to learn rich representations that might transfer to downstream tasks with minimal supervision. Recent multimodal models induce soft local alignments between modalities (image regions and sentences). This is of particular interest in the medical domain, where alignments could serve to highlight regions in an image relevant to specific phenomena described in free-text. Past work has presented example “heatmaps” as qualitative evidence that cross-modal soft alignments can be interpreted in this manner. However, there has been little quantitative evaluation of such alignments. Here we compare alignments from a state-of-the-art multimodal (image and text) model for EHR with human annotations that associate image regions with sentences. Our main finding is that the text has surprisingly little influence on the attention; alignments do not consistently reflect basic anatomical information. Moreover, synthetic modifications, such as substituting "left" for "right," do not substantially influence attention. We find that simple techniques such as masking out entity names during training show promise in terms of their ability to improve alignments without additional supervision.
Learning Monolingual Sentence Embeddings with Large-scale Parallel Translation Datasets
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Contrastive Learning, Semantic Textual Similarity, Sentence, Sentence Embeddings, STS, Translation

Although contrastive learning has greatly improved sentence representation, its performance is still limited by the size of monolingual sentence-pair datasets. Meanwhile, there exist large-scale parallel translation pairs (100x larger than monolingual pairs) that are highly correlated in semantic, but have not been utilized for learning  sentence representation. Furthermore, given parallel translation pairs, previous contrastive learning frameworks can not well balance the monolingual embeddings’ alignment and uniformity which represent the quality of embeddings. In this paper, we build on the top of dual encoder and propose to freeze the source language encoder, utilizing its consistent embeddings to supervise the target language encoder via contrastive learning, where source-target translation pairs are regarded as positives. We provide the first exploration of utilizing parallel translation sentence pairs to learn monolingual sentence embeddings and show superior performance to balance the alignment and uniformity. We achieve a new state-of-the-art performance on the average score of standard semantic textual similarity (STS), outperforming both SimCSE and Sentence-T5, and the best performance in corresponding tracks on transfer tasks.
Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Entity Linking, Profile Generation, Retrieval

There is little work on entity linking (EL) over Wikidata, even though it is the most extensive crowdsourced knowledge base. The scale of Wikidata can open up many new real-world applications, but its massive number of entities also makes EL challenging. To effectively narrow down the search space, we propose a novel candidate retrieval paradigm based on entity profiling. Wikidata entities and their textual fields are first indexed into a text search engine (e.g., Elasticsearch). During inference, given a mention and its context, we use a sequence-to-sequence (seq2seq) model to generate the profile of the target entity, which consists of its title and description. We use the profile to query the indexed search engine to retrieve candidate entities. Our approach complements the traditional approach of using a Wikipedia anchor-text dictionary, enabling us to further design a highly effective hybrid method for candidate retrieval. Combined with a simple cross-attention reranker, our complete EL framework achieves state-of-the-art results on three Wikidata-based datasets and strong performance on TACKBP-2010.
Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Constituency Parsing, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), POS

Unsupervised constituency parsing has been explored much but is still far from being solved as currently mainstream unsupervised constituency parser only captures the unlabeled structure of sentences. Properties in the substitution of constituents make it possible to detect constituents in a particular label. We propose an unsupervised and training-free labeling procedure by leveraging a newly introduced metric, Neighboring Distribution Divergence (NDD), which evaluates semantic changes caused by editions. We develop NDD into Dual POS-NDD (DP-NDD) and build templates called "molds" to extract labeled constituents from sentences. We show that DP-NDD labels constituents precisely and inducts more accurate unlabeled constituency trees than all previous unsupervised methods. Following two frameworks for labeled constituency trees inference, we set the new state-of-the-art for unlabeled F1 and labeled F1. Further studies show our approach can be scaled to other span labeling problems, i.e., named entity recognition.
Multi-Aspect co-Attentional Collaborative Filtering for Extreme Multi-label Text Classification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Collaborative Filtering, Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification

This work proposes a general and effective architecture for the extreme multi-label text classification (XMTC), and reformate the learning task to an interaction function between document and label. Recently, there are many studies trying to enhance text representation or reduce the number of labels to optimize the problem of lack of information in a text or the sparsity of the possibility vector. In the field of recommendation, a similar problem is already defined and studied for a quite long time. It is worthy to learn methods from recommendation to XMTC for finding matching relations in large size of dataset accurately. With co- attention mechanism and neural collaborative filtering, we not only learn informative label representation enhanced by document-specific label group vector and label-specific text feature vector but also build an effective interaction function to get matching score. After ex- tensive comparison experiments with various models, results demonstrate the architecture we proposed outperforms most of the methods and achieves significant improvement on basic document encoders.
Focus-Driven Contrastive Learning for Medical Question Summarization
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Contrastive Learning, Decoder, Sentence

Automatic medical question summarization can significantly help the system to understand consumer health questions and retrieve correct answers. The Seq2Seq model based on maximum likelihood estimation (MLE) has been applied in this task, which faces two general problems: the model can not capture well question focus and and the traditional MLE strategy lacks the ability to understand sentence-level semantics. To alleviate these problems, we propose a novel question focus-driven contrastive learning framework (QFCL). Specially, we propose an easy and effective approach to generate hard negative samples based on the question focus, and exploit contrastive learning at both encoder and decoder to obtain better sentence level representations. On three medical benchmark datasets, our proposed model achieves new state-of-the-art results, and obtains a performance gain of 12.2%, 28.7% and 9.6% over the baseline BART model on three datasets respectively. Further human judgement and detailed analysis prove that our QFCL model learns better sentence representations with the ability to distinguish different sentence meanings, and generates high-quality summaries by capturing question focus.
Uncovering Surprising Event Boundaries in Narratives
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Sentence

When reading stories, people can naturally identify sentences in which a new event starts, i.e., \textit{event boundaries}, using their knowledge of how events typically unfold, but a computational model to detect event boundaries is not yet available.We characterize and detect sentences with expected or surprising event boundaries in an annotated corpus of short diary-like stories, using a model that combines commonsense knowledge and narrative flow features with a RoBERTa classifier. Our results show that, while commonsense and narrative features can help improve performance overall, detecting event boundaries that are more subjective remains challenging for our model.We also find that sentences marking surprising event boundaries are less likely to be causally related to the preceding sentence, but are more likely to express emotional reactions of story characters, compared to sentences with no event boundary.
Context-Aware Prompt: Customize A Unique Prompt For Each Input
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Prompt Learning

After the proposal of BERT, pre-trained language models have become the dominant approach for solving many NLP tasks. Typically, a linear classifier is added to the head of the model for fine-tuning to fit downstream tasks, while a more recent approach, also known as prompt-based learning or prompt-learning, using prompts to perform various downstream tasks, is considered to be able to uncover the potential of the language model.Prior study, however, attempted to find a universal prompt for a certain task across all samples. Therefore, we propose a novel method, Context-Aware Prompt (CAP), which provides a unique continuous prompt for each sample input by combining contextual information to further investigate the potential capabilities of the language models. On the SuperGlue benchmark, our method outperforms multiple models with vanilla fine-tuning. Furthermore, we extend the use of prompts to include Replaced Token Detection (RTD) type prompts, allowing models like ELECTRA and DeBERTaV3 that employ RTD as a training objective to use prompts for downstream tasks.
ProcessBERT: Towards Equivalence Judgment of Variable Definitions among Multiple Engineering Documents
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling

Physical models play an important role in the process industry. However, conventional physical model building requires a survey on a huge amount of literature and trial-and-error to improve the model performance. We aim to develop an automated physical model builder (AutoPMoB), which automatically collects documents about a target process from literature databases, extracts necessary information from them, and builds a desired physical model by reorganizing the information. In this study, we proposed a method of judging equivalence of variable definitions, which is one of the fundamental technologies to realize AutoPMoB. We built a large-scale corpus specialized in chemical engineering and developed ProcessBERT, which is a domain-specific language model pre-trained on our corpus. We created datasets from papers related to chemical processes and evaluated the performance of ProcessBERT in the equivalence judgment task. We found that ProcessBERT outperformed the other language models in the similarity-based method.
HCL-MTC Hierarchical Contrastive Learning for Multi-label Text Classification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, Contrastive Learning, Multi Label Text Classification, Multi-Label Text Classification, Multi-Task Learning, text-classification, Text Classification

Multi-label text classification is a big challenging subtask in text classification, where labels generally form a tree structure. Existing solutions learn the label tree structure in a shallow manner and ignore the distinctive information between labels. To address this problem, we propose a Hierarchical Contrastive Learning for Multi-label Text Classification (HCL-MTC), which constructs the graph based on the contrastive knowledge between labels. Specifically, we formulate the MTC as a multi-task learning by introducing a sampling hierarchical contrastive loss, which learns both the correlative and distinctive label information and is beneficial in learning deep label hierarchy. The experimental results show that the proposed model can achieve considerable improvements on both public datasets (i.e., RCV1-v2 and WoS).
On Transferability of Prompt Tuning for Natural Language Processing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Transfer Learning

Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks and other PLMs with a trained projector on similar tasks; (2) as initialization, trained soft prompts and projected prompts can significantly accelerate training and also improve performance of PT in similar tasks and other PLMs respectively. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essential for transferability. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code will be publicly released.
Representation Learning for Resource-Constrained Keyphrase Generation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Denoising, Keyphrase Generation, Language Modeling, Language Modelling, Representation Learning

State-of-the-art keyphrase generation methods generally depend on large annotated datasets, limiting their performance in domains with constrained resources. To overcome this challenge, we investigate pre-training strategies to learn an intermediate representation suitable for the keyphrase generation task. We introduce salient span recovery and salient span prediction as guided denoising language modeling objectives that condense the domain-specific knowledge essential for keyphrase generation. Through experiments on benchmarks spanning multiple domains, we show the effectiveness of the proposed approaches for facilitating low resource and zero-shot keyphrase generation.
What do models learn from training on more than text? Measuring visual commonsense knowledge
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models\footnote{A link to a GitHub repo with the evaluation tasks and code necessary for reproducing our results will be placed here. For reviewing purposes, we add it as supplementary material.} and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.