pwc-archive/papers-with-abstracts huggingface dataset:
In-the-Wild Video Question Answering
Authors: Santiago Castro, Naihao Deng, Pingxuan Huang, Mihai Burzo, Rada Mihalcea
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Evidence Selection, Question Answering, Video Question Answering, Video Understanding

Existing video understanding datasets mostly focus on human interactions, with little attention being paid to the “in the wild” settings, where the videos are recorded outdoors. We propose WILDQA, a video understanding dataset of videos recorded in outside settings. In addition to video question answering (Video QA), we also introduce the new task of identifying visual support for a given question and answer (Video Evidence Selection). Through evaluations using a wide range of baseline models, we show that WILDQA poses new challenges to the vision and language research communities. The dataset is available at https: //lit.eecs.umich.edu/wildqa/.
Dual Capsule Attention Mask Network with Mutual Learning for Visual Question Answering
Authors: Weidong Tian, Haodong Li, Zhong-Qiu Zhao
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

A Visual Question Answering (VQA) model processes images and questions simultaneously with rich semantic information. The attention mechanism can highlight fine-grained features with critical information, thus ensuring that feature extraction emphasizes the objects related to the questions. However, unattended coarse-grained information is also essential for questions involving global elements. We believe that global coarse-grained information and local fine-grained information can complement each other to provide richer comprehensive information. In this paper, we propose a dual capsule attention mask network with mutual learning for VQA. Specifically, it contains two branches processing coarse-grained features and fine-grained features, respectively. We also design a novel stackable dual capsule attention module to fuse features and locate evidence. The two branches are combined to make final predictions for VQA. Experimental results show that our method outperforms the baselines in terms of VQA performance and interpretability and achieves new SOTA performance on the VQA-v2 dataset.
Efficient Multilingual Multi-modal Pre-training through Triple Contrastive Loss
Authors: Youhan Lee, Kyungtae Lim, Woonhyuk Baek, Byungseok Roh, Saehoon Kim
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: image-classification, Image Classification, Image-text Retrieval, Retrieval, Sentence, Sentence Embedding, Sentence-Embedding, Text Retrieval, Translation, Triplet, Zero-Shot Image Classification

Learning visual and textual representations in the shared space from web-scale image-text pairs improves the performance of diverse vision-and-language tasks, as well as modality-specific tasks. Many attempts in this framework have been made to connect English-only texts and images, and only a few works have been proposed to extend this framework in multilingual settings with the help of many translation pairs. In this multilingual approach, a typical setup is to use pairs of (image and English-text) and translation pairs. The major limitation of this approach is that the learning signal of aligning visual representation with under-resourced language representation is not strong, achieving a sub-optimal performance of vision-and-language tasks. In this work, we propose a simple yet effective enhancement scheme for previous multilingual multi-modal representation methods by using a limited number of pairs of images and non-English texts. In specific, our scheme fine-tunes a pre-trained multilingual model by minimizing a triplet contrastive loss on triplets of image and two different language texts with the same meaning, improving the connection between images and non-English texts. Experiments confirm that our enhancement strategy achieves performance gains in image-text retrieval, zero-shot image classification, and sentence embedding tasks.
Content Type Profiling of Data-to-Text Generation Datasets
Authors: Ashish Upadhyay, Stewart Massie
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Data-to-Text Generation, Text Generation, Vocal Bursts Type Prediction

Data-to-Text Generation (D2T) problems can be considered as a stream of time-stamped events with a text summary being produced for each. The problem becomes more challenging when event summaries contain complex insights derived from multiple records either within an event, or across several events from the event stream. It is important to understand the different types of content present in the summary to help us better define the system requirements so that we can build better systems. In this paper, we propose a novel typology of content types, that we use to classify the contents of event summaries. Using the typology, a profile of a dataset is generated as the distribution of the aggregated content types which captures the specific characteristics of the dataset and gives a measure of the complexity present in the problem. Extensive experimentation on different D2T datasets is performed and these demonstrate that neural systems struggle in generating contents of complex types.
A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model
Authors: Dongyuan Li, Jingyi You, Kotaro Funakoshi, Manabu Okumura
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Ancient Text Restoration, Attribute, Language Modeling, Language Modelling, Text Infilling

Text infilling aims to restore incomplete texts by filling in blanks, which has attracted more attention recently because of its wide application in ancient text restoration and text rewriting. However, attribute- aware text infilling is yet to be explored, and existing methods seldom focus on the infilling length of each blank or the number/location of blanks. In this paper, we propose an Attribute-aware Text Infilling method via a Pre-trained language model (A-TIP), which contains a text infilling component and a plug- and-play discriminator. Specifically, we first design a unified text infilling component with modified attention mechanisms and intra- and inter-blank positional encoding to better perceive the number of blanks and the infilling length for each blank. Then, we propose a plug-and-play discriminator to guide generation towards the direction of improving attribute relevance without decreasing text fluency. Finally, automatic and human evaluations on three open-source datasets indicate that A-TIP achieves state-of- the-art performance compared with all baselines.
Multi Graph Neural Network for Extractive Long Document Summarization
Authors: Xuan-Dung Doan, Le-Minh Nguyen, Khac-Hoai Nam Bui
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Document Summarization, Graph Neural Network, Sentence

Heterogeneous Graph Neural Networks (HeterGNN) have been recently introduced as an emergent approach for extracting document summarization (EDS) by exploiting the cross-relations between words and sentences. However, applying HeterGNN for long documents is still an open research issue. One of the main majors is the lacking of inter-sentence connections. In this regard, this paper exploits how to apply HeterGNN for long documents by building a graph on sentence-level nodes (homogeneous graph) and combine with HeterGNN for capturing the semantic information in terms of both inter and intra-sentence connections. Experiments on two benchmark datasets of long documents such as PubMed and ArXiv show that our method is able to achieve state-of-the-art results in this research field.
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation
Authors: Ernie Chang, Alex Marin, Vera Demberg
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Knowledge Distillation, Text Generation

The demand for multilingual dialogue systems often requires a costly labeling process, where human translators derive utterances in low resource languages from resource rich language annotation. To this end, we explore leveraging the inductive biases for target languages learned by numerous pretrained teacher models by transferring them to student models via sequence-level knowledge distillation. By assuming no target language text, the both the teacher and student models need to learn from the target distribution in a few/zero-shot manner. On the MultiATIS++ benchmark, we explore the effectiveness of our proposed technique to derive the multilingual text for 6 languages, using only the monolingual English data and the pretrained models. We show that training on the synthetic multilingual generation outputs yields close performance to training on human annotations in both slot F1 and intent accuracy; the synthetic text also scores high in naturalness and correctness based on human evaluation.
Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation
Authors: Fei-Tzin Lee, Miguel Ballesteros, Feng Nan, Kathleen McKeown
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Abstract Meaning Representation, Language Modeling, Language Modelling, Sentence, Text Generation

Large pretrained language models offer powerful generation capabilities, but cannot be reliably controlled at a sub-sentential level. We propose to make such fine-grained control possible in pretrained LMs by generating text directly from a semantic representation, Abstract Meaning Representation (AMR), which is augmented at the node level with syntactic control tags. We experiment with English-language generation of three modes of syntax relevant to the framing of a sentence - verb voice, verb tense, and realization of human entities - and demonstrate that they can be reliably controlled, even in settings that diverge drastically from the training distribution. These syntactic aspects contribute to how information is framed in text, something that is important for applications such as summarization which aim to highlight salient information.
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment
Authors: Ge Luo, Hebi Li, Youbiao He, Forrest Sheng Bao
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10

Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. Inspired by preference labeling in existing work of summarization evaluation, we propose to judge summary quality by learning the preference rank of summaries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experiments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings.
Multi-Attribute Controlled Text Generation with Contrastive-Generator and External-Discriminator
Authors: Guisheng Liu, Yi Li, Yanqing Guo, Xiangyang Luo, Bo wang
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Attribute, Comment Generation, Text Generation

Though existing researches have achieved impressive results in controlled text generation, they focus mainly on single-attribute control. However, in applications like automatic comments, the topic and sentiment need to be controlled simultaneously. In this work, we propose a new framework for multi-attribute controlled text generation. To achieve this, we design a contrastive-generator that can effectively generate texts with more attributes. In order to increase the convergence of the text on the desired attributes, we adopt an external-discriminator to distinguish whether the generated text holds the desired attributes. Moreover, we propose top-n weighted decoding to further improve the relevance of texts to attributes. Automated evaluations and human evaluations show that our framework achieves remarkable controllability in multi-attribute generation while keeping the text fluent and diverse. It also yields promising performance on zero-shot generation.
Coordination Generation via Synchronized Text-Infilling
Authors: Hiroki Teranishi, Yuji Matsumoto
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Data Augmentation, Sentence, Sentence Classification, Text Infilling

Generating synthetic data for supervised learning from large-scale pre-trained language models has enhanced performances across several NLP tasks, especially in low-resource scenarios. In particular, many studies of data augmentation employ masked language models to replace words with other words in a sentence. However, most of them are evaluated on sentence classification tasks and cannot immediately be applied to tasks related to the sentence structure. In this paper, we propose a simple yet effective approach to generating sentences with a coordinate structure in which the boundaries of its conjuncts are explicitly specified. For a given span in a sentence, our method embeds a mask with a coordinating conjunction in two ways (”X and [mask]”, ”[mask] and X”) and forces masked language models to fill the two blanks with an identical text. To achieve this, we introduce decoding methods for BERT and T5 models with the constraint that predictions for different masks are synchronized. Furthermore, we develop a training framework that effectively selects synthetic examples for the supervised coordination disambiguation task. We demonstrate that our method produces promising coordination instances that provide gains for the task in low-resource settings.
KHANQ: A Dataset for Generating Deep Questions in Education
Authors: Huanli Gong, Liangming Pan, Hengchang Hu
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Question Generation, Question-Generation

Designing in-depth educational questions is a time-consuming and cognitively demanding task. Therefore, it is intriguing to study how to build Question Generation (QG) models to automate the question creation process. However, existing QG datasets are not suitable for educational question generation because the questions are not real questions asked by humans during learning and can be solved by simply searching for information. To bridge this gap, we present KHANQ, a challenging dataset for educational question generation, containing 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy. Each data sample is carefully paraphrased and annotated as a triple of 1) Context: an independent paragraph on which the question is based; 2) Prompt: a text prompt for the question (e.g., the learner’s background knowledge); 3) Question: a deep question based on Context and coherent with Prompt. By conducting a human evaluation on the aspects of appropriateness, coverage, coherence, and complexity, we show that state-of-the-art QG models which perform well on shallow question generation datasets have difficulty in generating useful educational questions. This makes KHANQ a challenging testbed for educational question generation.
Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation
Authors: Husam Quteineh, Spyridon Samothrakis, Richard Sutcliffe
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Data Augmentation, MRPC, SST-2, Text Generation

Large-scale pretrained language models have led to significant improvements in Natural Language Processing. Unfortunately, they come at the cost of high computational and storage requirements that complicate their deployment on low-resource devices. This issue can be addressed by distilling knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. However, this can be difficult for tasks with very limited data. To overcome this challenge, we present a novel approach where knowledge can be distilled from a teacher model to a student model through the generation of synthetic data. For this to be done, we first fine-tune the teacher and student models, as well as a Natural Language Generation (NLG) model, on the target task dataset. We then let both student and teacher work together to condition the NLG model to generate examples that can enhance the performance of the student. We tested our approach on two data generation methods: a) Targeted generation using the Monte Carlo Tree Search (MCTS) algorithm, and b) A Non-Targeted Text Generation (NTTG) method. We evaluate the effectiveness of our approaches against a baseline that uses the BERT model for data augmentation through random word replacement. By testing this approach on the SST-2, MRPC, YELP-2, DBpedia, and TREC-6 datasets, we consistently witnessed considerable improvements over the word-replacement baseline.
Boosting Code Summarization by Embedding Code Structures
Authors: Jikyoeng Son, Joonghyuk Hahn, HyeonTae Seo, Yo-Sub Han
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Code Summarization, Graph Embedding

Recent research on code summarization relies on the structural information from the abstract syntax tree (AST) of source codes. It is, however, questionable whether it is the most effective to use AST for expressing the structural information. We find that a program dependency graph (PDG) can represent the structure of a code more effectively. We propose PDG Boosting Module (PBM) that encodes PDG into graph embedding and the framework to implement the proposed PBM with the existing models. PBM achieves improvements of 6.67% (BLEU) and 7.47% (ROUGE) on average. We then analyze the experimental results, and examine how PBM helps the training of baseline models and its performance robustness. For the validation of robustness, we measure the performance of an out-of-domain benchmark dataset, and confirm its robustness. In addition, we apply a new evaluation measure, SBERT score, to evaluate the semantic performance. The models implemented with PBM improve the performance of SBERT score. This implies that they generate summaries that are semantically more similar to the reference summary.
Comparative Graph-based Summarization of Scientific Papers Guided by Comparative Citations
Authors: Jingqiang Chen, Chaoxiang Cai, Xiaorui Jiang, KeJia Chen
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Articles, Sentence

With the rapid growth of scientific papers, understanding the changes and trends in a research area is rather time-consuming. The first challenge is to find related and comparable articles for the research. Comparative citations compare co-cited papers in a citation sentence and can serve as good guidance for researchers to track a research area. We thus go through comparative citations to find comparable objects and build a comparative scientific summarization corpus (CSSC). And then, we propose the comparative graph-based summarization (CGSUM) method to create comparative summaries using citations as guidance. The comparative graph is constructed using sentences as nodes and three different relationships of sentences as edges. The relationship that sentences occur in the same paper is used to calculate the salience of sentences, the relationship that sentences occur in two different papers is used to calculate the difference between sentences, and the relationship that sentences are related to citations is used to calculate the commonality of sentences. Experiments show that CGSUM outperforms comparative baselines on CSSC and performs well on DUC2006 and DUC2007.
JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation
Authors: Jingyi You, Dongyuan Li, Manabu Okumura, Kenji Suzuki
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: cross-modal alignment, Disease Prediction, Image Captioning, Medical Report Generation, Prediction

Automated radiology report generation aims to generate paragraphs that describe fine-grained visual differences among cases, especially those between the normal and the diseased. Existing methods seldom consider the cross-modal alignment between textual and visual features and tend to ignore disease tags as an auxiliary for report generation. To bridge the gap between textual and visual information, in this study, we propose a “Jointly learning framework for automated disease Prediction and radiology report Generation (JPG)” to improve the quality of reports through the interaction between the main task (report generation) and two auxiliary tasks (feature alignment and disease prediction). The feature alignment and disease prediction help the model learn text-correlated visual features and record diseases as keywords so that it can output high-quality reports. Besides, the improved reports in turn provide additional harder samples for feature alignment and disease prediction to learn more precise visual and textual representations and improve prediction accuracy. All components are jointly trained in a manner that helps improve them iteratively and progressively. Experimental results demonstrate the effectiveness of JPG on the most commonly used IU X-RAY dataset, showing its superior performance over multiple state-of-the-art image captioning and medical report generation methods with regard to BLEU, METEOR, and ROUGE metrics.
Automatic Nominalization of Clauses
Authors: John S. Y. Lee, Ho Hung Lim, Carol Webster, Anton Melser
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Language Modeling, Language Modelling, Natural Language Inference

Nominalization re-writes a clause as a noun phrase. It requires the transformation of the head verb of the clause into a deverbal noun, and the verb’s modifiers into nominal modifiers. Past research has focused on the selection of deverbal nouns, but has paid less attention to predicting the word positions and word forms for the nominal modifiers. We propose the use of a textual entailment model for clause nominalization. We obtained the best performance by fine-tuning a textual entailment model on this task, outperforming a number of unsupervised approaches using language model scores from a state-of-the-art neural language model.
Source-summary Entity Aggregation in Abstractive Summarization
Authors: José Ángel González, Annie Louis, Jackie Chi Kit Cheung
Date: 2022-10-01 00:00:00
Proceeding: COLING 2022 10
Tasks: Abstractive Text Summarization

In a text, entities mentioned earlier can be referred to in later discourse by a more general description. For example, Celine Dion and Justin Bieber can be referred to by Canadian singers or celebrities. In this work, we study this phenomenon in the context of summarization, where entities from a source text are generalized in the summary. We call such instances source-summary entity aggregations. We categorize these aggregations into two types and analyze them in the Cnn/Dailymail corpus, showing that they are reasonably frequent. We then examine how well three state-of-the-art summarization systems can generate such aggregations within summaries. We also develop techniques to encourage them to generate more aggregations. Our results show that there is significant room for improvement in producing semantically correct aggregations.