pwc-archive/papers-with-abstracts huggingface dataset:
Frequency effects in Linear Discriminative Learning
Authors: Maria Heitmeier, Yu-Ying Chuang, Seth D. Axen, R. Harald Baayen
Date: 2023-06-19 00:00:00
Tasks: Incremental Learning

Word frequency is a strong predictor in most lexical processing tasks. Thus, any model of word recognition needs to account for how word frequency effects arise. The Discriminative Lexicon Model (DLM; Baayen et al., 2018a, 2019) models lexical processing with linear mappings between words' forms and their meanings. So far, the mappings can either be obtained incrementally via error-driven learning, a computationally expensive process able to capture frequency effects, or in an efficient, but frequency-agnostic solution modelling the theoretical endstate of learning (EL) where all words are learned optimally. In this study we show how an efficient, yet frequency-informed mapping between form and meaning can be obtained (Frequency-informed learning; FIL). We find that FIL well approximates an incremental solution while being computationally much cheaper. FIL shows a relatively low type- and high token-accuracy, demonstrating that the model is able to process most word tokens encountered by speakers in daily life correctly. We use FIL to model reaction times in the Dutch Lexicon Project (Keuleers et al., 2010) and find that FIL predicts well the S-shaped relationship between frequency and the mean of reaction times but underestimates the variance of reaction times for low frequency words. FIL is also better able to account for priming effects in an auditory lexical decision task in Mandarin Chinese (Lee, 2007), compared to EL. Finally, we used ordered data from CHILDES (Brown, 1973; Demuth et al., 2006) to compare mappings obtained with FIL and incremental learning. The mappings are highly correlated, but with FIL some nuances based on word ordering effects are lost. Our results show how frequency effects in a learning model can be simulated efficiently, and raise questions about how to best account for low-frequency words in cognitive models.
Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction
Authors: Qian Li, Shu Guo, Cheng Ji, Xutan Peng, Shiyao Cui, JianXin Li
Date: 2023-06-19 00:00:00
Tasks: Decoder, Relation, Relation Extraction

Multi-Modal Relation Extraction (MMRE) aims at identifying the relation between two entities in texts that contain visual clues. Rich visual content is valuable for the MMRE task, but existing works cannot well model finer associations among different modalities, failing to capture the truly helpful visual information and thus limiting relation extraction performance. In this paper, we propose a novel MMRE framework to better capture the deeper correlations of text, entity pair, and image/objects, so as to mine more helpful information for the task, termed as DGF-PT. We first propose a prompt-based autoregressive encoder, which builds the associations of intra-modal and inter-modal features related to the task, respectively by entity-oriented and object-oriented prefixes. To better integrate helpful visual information, we design a dual-gated fusion module to distinguish the importance of image/objects and further enrich text representations. In addition, a generative decoder is introduced with entity type restriction on relations, better filtering out candidates. Extensive experiments conducted on the benchmark dataset show that our approach achieves excellent performance compared to strong competitors, even in the few-shot situation.
Fine-Tuning Language Models for Scientific Writing Support
Authors: Justin Mücke, Daria Waldow, Luise Metzger, Philipp Schauz, Marcel Hoffman, Nicolas Lell, Ansgar Scherp
Date: 2023-06-19 00:00:00
Tasks: Sentence

We support scientific writers in determining whether a written sentence is scientific, to which section it belongs, and suggest paraphrasings to improve the sentence. Firstly, we propose a regression model trained on a corpus of scientific sentences extracted from peer-reviewed scientific papers and non-scientific text to assign a score that indicates the scientificness of a sentence. We investigate the effect of equations and citations on this score to test the model for potential biases. Secondly, we create a mapping of section titles to a standard paper layout in AI and machine learning to classify a sentence to its most likely section. We study the impact of context, i.e., surrounding sentences, on the section classification performance. Finally, we propose a paraphraser, which suggests an alternative for a given sentence that includes word substitutions, additions to the sentence, and structural changes to improve the writing style. We train various large language models on sentences extracted from arXiv papers that were peer reviewed and published at A*, A, B, and C ranked conferences. On the scientificness task, all models achieve an MSE smaller than $2\%$. For the section classification, BERT outperforms WideMLP and SciBERT in most cases. We demonstrate that using context enhances the classification of a sentence, achieving up to a $90\%$ F1-score. Although the paraphrasing models make comparatively few alterations, they produce output sentences close to the gold standard. Large fine-tuned models such as T5 Large perform best in experiments considering various measures of difference between input sentence and gold standard. Code is provided under https://github.com/JustinMuecke/SciSen.
Quantitative dynamics of design thinking and creativity perspectives in company context
Authors: Georgi V. Georgiev, Danko D. Georgiev
Date: 2023-06-19 00:00:00
Tasks: Word Similarity

This study is intended to provide in-depth insights into how design thinking and creativity issues are understood and possibly evolve in the course of design discussions in a company context. For that purpose, we use the seminar transcripts of the Design Thinking Research Symposium 12 (DTRS12) dataset "Tech-centred Design Thinking: Perspectives from a Rising Asia," which are primarily concerned with how Korean companies implement design thinking and what role designers currently play. We employed a novel method of information processing based on constructed dynamic semantic networks to investigate the seminar discussions according to company representatives and company size. We compared the quantitative dynamics in two seminars: the first involved managerial representatives of four companies, and the second involved specialized designers and management of a design center of single company. On the basis of dynamic semantic networks, we quantified the changes in four semantic measures -- abstraction, polysemy, information content, and pairwise word similarity -- in chronologically reconstructed individual design-thinking processes. Statistical analyses show that design thinking in the seminar with four companies, exhibits significant differences in the dynamics of abstraction, polysemy, and information content, compared to the seminar with the design center of single company. Both the decrease in polysemy and abstraction and the increase in information content in the individual design-thinking processes in the seminar with four companies indicate that design managers are focused on more concrete design issues, with more information and less ambiguous content to the final design product. By contrast, specialized designers manifest more abstract thinking and appear to exhibit a slightly higher level of divergence in their design processes.
Multilingual Few-Shot Learning via Language Model Retrieval
Authors: Genta Indra Winata, Liang-Kang Huang, Soumya Vadlamannati, Yash Chandarana
Date: 2023-06-19 00:00:00
Tasks: Few-Shot Learning, In-Context Learning, Intent Detection, Language Modeling, Language Modelling, model, Natural Language Understanding, Retrieval, Sentiment Analysis, Topic Classification

Transformer-based language models have achieved remarkable success in few-shot in-context learning and drawn a lot of research interest. However, these models' performance greatly depends on the choice of the example prompts and also has high variability depending on how samples are chosen. In this paper, we conduct a comprehensive study of retrieving semantically similar few-shot samples and using them as the context, as it helps the model decide the correct label without any gradient update in the multilingual and cross-lingual settings. We evaluate the proposed method on five natural language understanding datasets related to intent detection, question classification, sentiment analysis, and topic classification. The proposed method consistently outperforms random sampling in monolingual and cross-lingual tasks in non-English languages.
Comparison of L2 Korean pronunciation error patterns from five L1 backgrounds by using automatic phonetic transcription
Authors: Eun Jung Yeo, Hyungshin Ryu, Jooyoung Lee, Sunhee Kim, Minhwa Chung
Date: 2023-06-19 00:00:00

This paper presents a large-scale analysis of L2 Korean pronunciation error patterns from five different language backgrounds, Chinese, Vietnamese, Japanese, Thai, and English, by using automatic phonetic transcription. For the analysis, confusion matrices are generated for each L1, by aligning canonical phone sequences and automatically transcribed phone sequences obtained from fine-tuned Wav2Vec2 XLS-R phone recognizer. Each value in the confusion matrices is compared to capture frequent common error patterns and to specify patterns unique to a certain language background. Using the Foreign Speakers' Voice Data of Korean for Artificial Intelligence Learning dataset, common error pattern types are found to be (1) substitutions of aspirated or tense consonants with plain consonants, (2) deletions of syllable-final consonants, and (3) substitutions of diphthongs with monophthongs. On the other hand, thirty-nine patterns including (1) syllable-final /l/ substitutions with /n/ for Vietnamese and (2) /\textturnm/ insertions for Japanese are discovered as language-dependent.
Adaptive Ordered Information Extraction with Deep Reinforcement Learning
Authors: Wenhao Huang, Jiaqing Liang, Zhixu Li, Yanghua Xiao, Chuanjun Ji
Date: 2023-06-19 00:00:00
Tasks: Deep Reinforcement Learning, Event Extraction, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Information extraction (IE) has been studied extensively. The existing methods always follow a fixed extraction order for complex IE tasks with multiple elements to be extracted in one instance such as event extraction. However, we conduct experiments on several complex IE datasets and observe that different extraction orders can significantly affect the extraction results for a great portion of instances, and the ratio of sentences that are sensitive to extraction orders increases dramatically with the complexity of the IE task. Therefore, this paper proposes a novel adaptive ordered IE paradigm to find the optimal element extraction order for different instances, so as to achieve the best extraction results. We also propose an reinforcement learning (RL) based framework to generate optimal extraction order for each instance dynamically. Additionally, we propose a co-training framework adapted to RL to mitigate the exposure bias during the extractor training phase. Extensive experiments conducted on several public datasets demonstrate that our proposed method can beat previous methods and effectively improve the performance of various IE tasks, especially for complex ones.
Gender Differences in Abuse: The Case of Dutch Politicians on Twitter
Authors: Isabelle van der Vegt
Date: 2023-06-19 00:00:00

Online abuse and threats towards politicians have become a significant concern in the Netherlands, like in many other countries across the world. This paper analyses gender differences in abuse received by Dutch politicians on Twitter, while taking into account the possible additional impact of ethnic minority status. All tweets directed at party leaders throughout the entire year of 2022 were collected. The effect of gender and ethnic minority status were estimated for six different linguistic measures of abuse, namely, toxicity, severe toxicity, identity attacks, profanity, insults, and threats. Contrary to expectations, male politicians received higher levels of all forms of abuse, with the exception of threats, for which no significant gender difference was found. Significant interaction effects between gender and ethnic minority status were found for a number of abuse measures. In the case of severe toxicity, identity attacks, and profanity, female ethnic minority politicians were more severely impacted than their ethnic majority female colleagues, but not worse than male politicians. Finally, female ethnic minority politicians received the highest levels of threats compared to all groups. Given that online abuse and threats are reported to have a negative effect on political participation and retention, these results are particularly worrying.
Unsupervised Open-domain Keyphrase Generation
Authors: Lam Thanh Do, Pritom Saha Akash, Kevin Chen-Chuan Chang
Date: 2023-06-19 00:00:00
Tasks: Informativeness, Keyphrase Generation

In this work, we study the problem of unsupervised open-domain keyphrase generation, where the objective is a keyphrase generation model that can be built without using human-labeled data and can perform consistently across domains. To solve this problem, we propose a seq2seq model that consists of two modules, namely \textit{phraseness} and \textit{informativeness} module, both of which can be built in an unsupervised and open-domain fashion. The phraseness module generates phrases, while the informativeness module guides the generation towards those that represent the core concepts of the text. We thoroughly evaluate our proposed method using eight benchmark datasets from different domains. Results on in-domain datasets show that our approach achieves state-of-the-art results compared with existing unsupervised models, and overall narrows the gap between supervised and unsupervised methods down to about 16\%. Furthermore, we demonstrate that our model performs consistently across domains, as it overall surpasses the baselines on out-of-domain datasets.
Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models
Authors: Tomoki Sugimoto, Yasumasa Onoe, Hitomi Yanaka
Date: 2023-06-19 00:00:00
Tasks: Natural Language Inference

Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. It is unclear whether current LMs realize the generalization capacity for temporal inference across languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites. We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns). Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.
Fine-tuning Large Enterprise Language Models via Ontological Reasoning
Authors: Teodoro Baldazzi, Luigi Bellomarini, Stefano Ceri, Andrea Colombo, Andrea Gentili, Emanuel Sallinger
Date: 2023-06-19 00:00:00
Tasks: Knowledge Graphs, Specificity

Large Language Models (LLMs) exploit fine-tuning as a technique to adapt to diverse goals, thanks to task-specific training data. Task specificity should go hand in hand with domain orientation, that is, the specialization of an LLM to accurately address the tasks of a given realm of interest. However, models are usually fine-tuned over publicly available data or, at most, over ground data from databases, ignoring business-level definitions and domain experience. On the other hand, Enterprise Knowledge Graphs (EKGs) are able to capture and augment such domain knowledge via ontological reasoning. With the goal of combining LLM flexibility with the domain orientation of EKGs, we propose a novel neurosymbolic architecture that leverages the power of ontological reasoning to build task- and domain-specific corpora for LLM fine-tuning.
Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations
Authors: Dongyu Ru, Lin Qiu, Xipeng Qiu, Yue Zhang, Zheng Zhang
Date: 2023-06-19 00:00:00
Tasks: Sentence

Discourse analysis is an important task because it models intrinsic semantic structures between sentences in a document. Discourse markers are natural representations of discourse in our daily language. One challenge is that the markers as well as pre-defined and human-labeled discourse relations can be ambiguous when describing the semantics between sentences. We believe that a better approach is to use a contextual-dependent distribution over the markers to express discourse information. In this work, we propose to learn a Distributed Marker Representation (DMR) by utilizing the (potentially) unlimited discourse marker data with a latent discourse sense, thereby bridging markers with sentence pairs. Such representations can be learned automatically from data without supervision, and in turn provide insights into the data itself. Experiments show the SOTA performance of our DMR on the implicit discourse relation recognition task and strong interpretability. Our method also offers a valuable tool to understand complex ambiguity and entanglement among discourse markers and manually defined discourse relations.
Summarization from Leaderboards to Practice: Choosing A Representation Backbone and Ensuring Robustness
Authors: David Demeter, Oshin Agarwal, Simon Ben Igeri, Marko Sterbentz, Neil Molino, John M. Conroy, Ani Nenkova
Date: 2023-06-18 00:00:00

Academic literature does not give much guidance on how to build the best possible customer-facing summarization system from existing research components. Here we present analyses to inform the selection of a system backbone from popular models; we find that in both automatic and human evaluation, BART performs better than PEGASUS and T5. We also find that when applied cross-domain, summarizers exhibit considerably worse performance. At the same time, a system fine-tuned on heterogeneous domains performs well on all domains and will be most suitable for a broad-domain summarizer. Our work highlights the need for heterogeneous domain summarization benchmarks. We find considerable variation in system output that can be captured only with human evaluation and are thus unlikely to be reflected in standard leaderboards with only automatic evaluation.
UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning
Authors: Kang Zhao, Wei Liu, Jian Luan, Minglei Gao, Li Qian, Hanlin Teng, Bin Wang
Date: 2023-06-18 00:00:00
Tasks: Conversation Summarization, Decoder, Representation Learning, Retrieval

Open-domain long-term memory conversation can establish long-term intimacy with humans, and the key is the ability to understand and memorize long-term dialogue history information. Existing works integrate multiple models for modelling through a pipeline, which ignores the coupling between different stages. In this paper, we propose a Unified framework for Long-term Memory Conversations (UniMC), which increases the connection between different stages by learning relevance representation. Specifically, we decompose the main task into three subtasks based on probability graphs: 1) conversation summarization, 2) memory retrieval, 3) memory-augmented generation. Each subtask involves learning a representation for calculating the relevance between the query and memory, which is modelled by inserting a special token at the beginning of the decoder input. The relevance representation learning strengthens the connection across subtasks through parameter sharing and joint training. Extensive experimental results show that the proposed method consistently improves over strong baselines and yields better dialogue consistency and engagingness.
Evolutionary Verbalizer Search for Prompt-based Few Shot Text Classification
Authors: Tongtao Ling, Lei Chen, Yutao Lai, Hai-Lin Liu
Date: 2023-06-18 00:00:00
Tasks: Few-Shot Text Classification, Language Modeling, Language Modelling, text-classification, Text Classification

Recent advances for few-shot text classification aim to wrap textual inputs with task-specific prompts to cloze questions. By processing them with a masked language model to predict the masked tokens and using a verbalizer that constructs the mapping between predicted words and target labels. This approach of using pre-trained language models is called prompt-based tuning, which could remarkably outperform conventional fine-tuning approach in the low-data scenario. As the core of prompt-based tuning, the verbalizer is usually handcrafted with human efforts or suboptimally searched by gradient descent. In this paper, we focus on automatically constructing the optimal verbalizer and propose a novel evolutionary verbalizer search (EVS) algorithm, to improve prompt-based tuning with the high-performance verbalizer. Specifically, inspired by evolutionary algorithm (EA), we utilize it to automatically evolve various verbalizers during the evolutionary procedure and select the best one after several iterations. Extensive few-shot experiments on five text classification datasets show the effectiveness of our method.
Position: AI Evaluation Should Learn from How We Test Humans
Authors: Yan Zhuang, Qi Liu, Zachary A. Pardos, Patrick C. Kyllonen, Jiyun Zu, Zhenya Huang, Shijin Wang, Enhong Chen
Date: 2023-06-18 00:00:00
Tasks: Mathematical Reasoning, Position

As AI systems continue to evolve, their rigorous evaluation becomes crucial for their development and deployment. Researchers have constructed various large-scale benchmarks to determine their capabilities, typically against a gold-standard test set and report metrics averaged across all items. However, this static evaluation paradigm increasingly shows its limitations, including high evaluation costs, data contamination, and the impact of low-quality or erroneous items on evaluation reliability and efficiency. In this Position, drawing from human psychometrics, we discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics or value of each test item in the benchmark, and tailoring each model's evaluation instead of relying on a fixed test set. This paradigm provides robust ability estimation, uncovering the latent traits underlying a model's observed scores. This position paper analyze the current possibilities, prospects, and reasons for adopting psychometrics in AI evaluation. We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.