pwc-archive/papers-with-abstracts huggingface dataset:
DISL: Fueling Research with A Large Dataset of Solidity Smart Contracts
Authors: Gabriele Morello, Mojtaba Eshghie, Sofia Bobadilla, Martin Monperrus
Date: 2024-03-25 00:00:00
Tasks: Benchmarking

The DISL dataset features a collection of $514,506$ unique Solidity files that have been deployed to Ethereum mainnet. It caters to the need for a large and diverse dataset of real-world smart contracts. DISL serves as a resource for developing machine learning systems and for benchmarking software engineering tools designed for smart contracts. By aggregating every verified smart contract from Etherscan up to January 15, 2024, DISL surpasses existing datasets in size and recency.
State Space Models as Foundation Models: A Control Theoretic Overview
Authors: Carmen Amo Alonso, Jerome Sieber, Melanie N. Zeilinger
Date: 2024-03-25 00:00:00
Tasks: Mamba, State Space Models

In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by the recent success of Mamba, showing better performance than the state-of-the-art Transformer architectures in language tasks. Foundation models, like e.g. GPT-4, aim to encode sequential data into a latent space in order to learn a compressed representation of the data. The same goal has been pursued by control theorists using SSMs to efficiently model dynamical systems. Therefore, SSMs can be naturally connected to deep sequence modeling, offering the opportunity to create synergies between the corresponding research areas. This paper is intended as a gentle introduction to SSM-based architectures for control theorists and summarizes the latest research developments. It provides a systematic review of the most successful SSM proposals and highlights their main features from a control theoretic perspective. Additionally, we present a comparative analysis of these models, evaluating their performance on a standardized benchmark designed for assessing a model's efficiency at learning long sequences.
Dynamic Relative Representations for Goal-Oriented Semantic Communications
Authors: Simone Fiorellino, Claudio Battiloro, Emilio Calvanese Strinati, Paolo Di Lorenzo
Date: 2024-03-25 00:00:00
Tasks: Semantic Communication

In future 6G wireless networks, semantic and effectiveness aspects of communications will play a fundamental role, incorporating meaning and relevance into transmissions. However, obstacles arise when devices employ diverse languages, logic, or internal representations, leading to semantic mismatches that might jeopardize understanding. In latent space communication, this challenge manifests as misalignment within high-dimensional representations where deep neural networks encode data. This paper presents a novel framework for goal-oriented semantic communication, leveraging relative representations to mitigate semantic mismatches via latent space alignment. We propose a dynamic optimization strategy that adapts relative representations, communication parameters, and computation resources for energy-efficient, low-latency, goal-oriented semantic communications. Numerical results demonstrate our methodology's effectiveness in mitigating mismatches among devices, while optimizing energy consumption, delay, and effectiveness.
CapsF: Capsule Fusion for Extracting psychiatric stressors for suicide from twitter
Authors: Mohammad Ali Dadgostarnia, Ramin Mousa, Saba Hesaraki
Date: 2024-02-07 00:00:00
Tasks: Binary Classification

Along with factors such as cancer, blood pressure, street accidents and stroke, suicide has been one of Iran main causes of death. One of the main reasons for suicide is psychological stressors. Identifying psychological stressors in an at risk population can help in the early prevention of suicidal and suicidal behaviours. In recent years, the widespread popularity and flow of real time information sharing of social media have allowed for potential early intervention in large scale and even small scale populations. However, some automated approaches to extract psychiatric stressors from Twitter have been presented, but most of this research has been for non Persian languages. This study aims to investigate the techniques of detecting psychological stress related to suicide from Persian tweets using learning based methods. The proposed capsule based approach achieved a binary classification accuracy of 0.83.
Distilling Named Entity Recognition Models for Endangered Species from Large Language Models
Authors: Jesse Atuhurra, Seiveright Cargill Dujohn, Hidetaka Kamigaito, Hiroyuki Shindo, Taro Watanabe
Date: 2024-03-13 00:00:00
Tasks: In-Context Learning, Knowledge Distillation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Relation Extraction, Transfer Learning

Natural language processing (NLP) practitioners are leveraging large language models (LLM) to create structured datasets from semi-structured and unstructured data sources such as patents, papers, and theses, without having domain-specific knowledge. At the same time, ecological experts are searching for a variety of means to preserve biodiversity. To contribute to these efforts, we focused on endangered species and through in-context learning, we distilled knowledge from GPT-4. In effect, we created datasets for both named entity recognition (NER) and relation extraction (RE) via a two-stage process: 1) we generated synthetic data from GPT-4 of four classes of endangered species, 2) humans verified the factual accuracy of the synthetic data, resulting in gold data. Eventually, our novel dataset contains a total of 3.6K sentences, evenly divided between 1.8K NER and 1.8K RE sentences. The constructed dataset was then used to fine-tune both general BERT and domain-specific BERT variants, completing the knowledge distillation process from GPT-4 to BERT, because GPT-4 is resource intensive. Experiments show that our knowledge transfer approach is effective at creating a NER model suitable for detecting endangered species from texts.
Using Contextual Information for Sentence-level Morpheme Segmentation
Authors: Prabin Bhandari, Abhishek Paudel
Date: 2024-03-15 00:00:00
Tasks: Segmentation, Sentence

Recent advancements in morpheme segmentation primarily emphasize word-level segmentation, often neglecting the contextual relevance within the sentence. In this study, we redefine the morpheme segmentation task as a sequence-to-sequence problem, treating the entire sentence as input rather than isolating individual words. Our findings reveal that the multilingual model consistently exhibits superior performance compared to monolingual counterparts. While our model did not surpass the performance of the current state-of-the-art, it demonstrated comparable efficacy with high-resource languages while revealing limitations in low-resource language scenarios.
Linguistics from a topological viewpoint
Authors: Rui Dong
Date: 2024-03-16 00:00:00
Tasks: Topological Data Analysis

Typological databases in linguistics are usually categorical-valued. As a result, it is difficult to have a clear visualization of the data. In this paper, we describe a workflow to analyze the topological shapes of South American languages by applying multiple correspondence analysis technique and topological data analysis methods.
Towards Enabling FAIR Dataspaces Using Large Language Models
Authors: Benedikt T. Arnold, Johannes Theissen-Lipp, Diego Collarana, Christoph Lange, Sandra Geisler, Edward Curry, Stefan Decker
Date: 2024-03-18 00:00:00

Dataspaces have recently gained adoption across various sectors, including traditionally less digitized domains such as culture. Leveraging Semantic Web technologies helps to make dataspaces FAIR, but their complexity poses a significant challenge to the adoption of dataspaces and increases their cost. The advent of Large Language Models (LLMs) raises the question of how these models can support the adoption of FAIR dataspaces. In this work, we demonstrate the potential of LLMs in dataspaces with a concrete example. We also derive a research agenda for exploring this emerging field.
Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?
Authors: Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann
Date: 2024-03-25 00:00:00
Tasks: Cross-Lingual Transfer, Machine Translation, Natural Language Understanding, Representation Learning, Sentence, Transfer Learning, Translation

Multilingual pretraining and fine-tuning have remarkably succeeded in various natural language processing tasks. Transferring representations from one language to another is especially crucial for cross-lingual learning. One can expect machine translation objectives to be well suited to fostering such capabilities, as they involve the explicit alignment of semantically equivalent sentences from different languages. This paper investigates the potential benefits of employing machine translation as a continued training objective to enhance language representation learning, bridging multilingual pretraining and cross-lingual applications. We study this question through two lenses: a quantitative evaluation of the performance of existing models and an analysis of their latent representations. Our results show that, contrary to expectations, machine translation as the continued training fails to enhance cross-lingual representation learning in multiple cross-lingual natural language understanding tasks. We conclude that explicit sentence-level alignment in the cross-lingual scenario is detrimental to cross-lingual transfer pretraining, which has important implications for future cross-lingual transfer studies. We furthermore provide evidence through similarity measures and investigation of parameters that this lack of positive influence is due to output separability -- which we argue is of use for machine translation but detrimental elsewhere.
Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
Authors: Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, Xuanhua Shi
Date: 2024-03-25 00:00:00
Tasks: Code Generation, Retrieval

Large Language Models (LLMs) have shown remarkable progress in automated code generation. Yet, LLM-generated code may contain errors in API usage, class, data structure, or missing project-specific information. As much of this project-specific context cannot fit into the prompts of LLMs, we must find ways to allow the model to explore the project-level code context. We present CoCoGen, a new code generation approach that uses compiler feedback to improve the LLM-generated code. CoCoGen first leverages static analysis to identify mismatches between the generated code and the project's context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository. We integrate CoCoGen with two representative LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), and apply it to Python code generation. Experimental results show that CoCoGen significantly improves the vanilla LLMs by over 80% in generating code dependent on the project context and consistently outperforms the existing retrieval-based code generation baselines.
TEI2GO: A Multilingual Approach for Fast Temporal Expression Identification
Authors: Hugo Sousa, Ricardo Campos, Alípio Jorge
Date: 2024-03-25 00:00:00

Temporal expression identification is crucial for understanding texts written in natural language. Although highly effective systems such as HeidelTime exist, their limited runtime performance hampers adoption in large-scale applications and production environments. In this paper, we introduce the TEI2GO models, matching HeidelTime's effectiveness but with significantly improved runtime, supporting six languages, and achieving state-of-the-art results in four of them. To train the TEI2GO models, we used a combination of manually annotated reference corpus and developed ``Professor HeidelTime'', a comprehensive weakly labeled corpus of news texts annotated with HeidelTime. This corpus comprises a total of $138,069$ documents (over six languages) with $1,050,921$ temporal expressions, the largest open-source annotated dataset for temporal expression identification to date. By describing how the models were produced, we aim to encourage the research community to further explore, refine, and extend the set of models to additional languages and domains. Code, annotations, and models are openly available for community exploration and use. The models are conveniently on HuggingFace for seamless integration and application.
Cross-lingual Contextualized Phrase Retrieval
Authors: Huayang Li, Deng Cai, Zhi Qu, Qu Cui, Hidetaka Kamigaito, Lemao Liu, Taro Watanabe
Date: 2024-03-25 00:00:00
Tasks: Contrastive Learning, Language Modelling, Large Language Model, Machine Translation, Retrieval, Word Alignment

Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks by leveraging the fine-grained information that phrases offer. In our work, we propose a new task formulation of dense retrieval, cross-lingual contextualized phrase retrieval, which aims to augment cross-lingual applications by addressing polysemy using context information. However, the lack of specific training data and models are the primary challenges to achieve our goal. As a result, we extract pairs of cross-lingual phrases using word alignment information automatically induced from parallel sentences. Subsequently, we train our Cross-lingual Contextualized Phrase Retriever (CCPR) using contrastive learning, which encourages the hidden representations of phrases with similar contexts and semantics to align closely. Comprehensive experiments on both the cross-lingual phrase retrieval task and a downstream task, i.e, machine translation, demonstrate the effectiveness of CCPR. On the phrase retrieval task, CCPR surpasses baselines by a significant margin, achieving a top-1 accuracy that is at least 13 points higher. When utilizing CCPR to augment the large-language-model-based translator, it achieves average gains of 0.7 and 1.5 in BERTScore for translations from X=>En and vice versa, respectively, on WMT16 dataset. Our code and data are available at \url{https://github.com/ghrua/ccpr_release}.
Encoding of lexical tone in self-supervised models of spoken language
Authors: Gaofei Shen, Michaela Watkins, Afra Alishahi, Arianna Bisazza, Grzegorz Chrupała
Date: 2024-03-25 00:00:00

Interpretability research has shown that self-supervised Spoken Language Models (SLMs) encode a wide variety of features in human speech from the acoustic, phonetic, phonological, syntactic and semantic levels, to speaker characteristics. The bulk of prior research on representations of phonology has focused on segmental features such as phonemes; the encoding of suprasegmental phonology (such as tone and stress patterns) in SLMs is not yet well understood. Tone is a suprasegmental feature that is present in more than half of the world's languages. This paper aims to analyze the tone encoding capabilities of SLMs, using Mandarin and Vietnamese as case studies. We show that SLMs encode lexical tone to a significant degree even when they are trained on data from non-tonal languages. We further find that SLMs behave similarly to native and non-native human participants in tone and consonant perception studies, but they do not follow the same developmental trajectory.
New Intent Discovery with Attracting and Dispersing Prototype
Authors: Shun Zhang, Jian Yang, Jiaqi Bai, Chaoran Yan, Tongliang Li, Zhao Yan, Zhoujun Li
Date: 2024-03-25 00:00:00
Tasks: Intent Discovery, Language Modeling, Language Modelling, Large Language Model

New Intent Discovery (NID) aims to recognize known and infer new intent categories with the help of limited labeled and large-scale unlabeled data. The task is addressed as a feature-clustering problem and recent studies augment instance representation. However, existing methods fail to capture cluster-friendly representations, since they show less capability to effectively control and coordinate within-cluster and between-cluster distances. Tailored to the NID problem, we propose a Robust and Adaptive Prototypical learning (RAP) framework for globally distinct decision boundaries for both known and new intent categories. Specifically, a robust prototypical attracting learning (RPAL) method is designed to compel instances to gravitate toward their corresponding prototype, achieving greater within-cluster compactness. To attain larger between-cluster separation, another adaptive prototypical dispersing learning (APDL) method is devised to maximize the between-cluster distance from the prototype-to-prototype perspective. Experimental results evaluated on three challenging benchmarks (CLINC, BANKING, and StackOverflow) of our method with better cluster-friendly representation demonstrate that RAP brings in substantial improvements over the current state-of-the-art methods (even large language model) by a large margin (average +5.5% improvement).
Evaluating Shortest Edit Script Methods for Contextual Lemmatization
Authors: Olia Toporkov, Rodrigo Agerri
Date: 2024-03-25 00:00:00
Tasks: LEMMA, Lemmatization, token-classification, Token Classification

Modern contextual lemmatizers often rely on automatically induced Shortest Edit Scripts (SES), namely, the number of edit operations to transform a word form into its lemma. In fact, different methods of computing SES have been proposed as an integral component in the architecture of several state-of-the-art contextual lemmatizers currently available. However, previous work has not investigated the direct impact of SES in the final lemmatization performance. In this paper we address this issue by focusing on lemmatization as a token classification task where the only input that the model receives is the word-label pairs in context, where the labels correspond to previously induced SES. Thus, by modifying in our lemmatization system only the SES labels that the model needs to learn, we may then objectively conclude which SES representation produces the best lemmatization results. We experiment with seven languages of different morphological complexity, namely, English, Spanish, Basque, Russian, Czech, Turkish and Polish, using multilingual and language-specific pre-trained masked language encoder-only models as a backbone to build our lemmatizers. Comprehensive experimental results, both in- and out-of-domain, indicate that computing the casing and edit operations separately is beneficial overall, but much more clearly for languages with high-inflected morphology. Notably, multilingual pre-trained language models consistently outperform their language-specific counterparts in every evaluation setting.
A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course
Authors: Will Yeadon, Alex Peach, Craig P. Testrow
Date: 2024-03-25 00:00:00
Tasks: Prompt Engineering

This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $n = 300$ data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 \times 10^{-10}$). Prompt engineering significantly improved scores for both GPT-4 (p = $1.661 \times 10^{-4}$) and GPT-3.5 (p = $4.967 \times 10^{-9}$). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from `Definitely AI' to `Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary `AI' or `Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.
Explainable Transformer-Based Anomaly Detection for Internet of Things Security Check for updates
Authors: A Saghir, H Beniwal, KD Tran, A Raza, L Koehl, X Zeng, KP Tran
Date: 2024-01-05 00:00:00
Proceeding: EAI/Springer Innovations in Communication and Computing 2024 1
Tasks: Anomaly Detection, energy management, Management

The IoT finds extensive applications across various domains, including healthcare, transportation systems, energy management, and manufacturing, as evidenced by prior research [1-4]. This technology facilitates seamless communication and data exchange among numerous interconnected devices, often requiring minimal human intervention. This connectivity offers innumerable advantages for service providers and end-users, as highlighted in the literature [3]. IoT applications have ushered in a transformative era, rendering buildings, vehicles, healthcare systems, and even entire urban environments' smart'and interconnected.
Dirichlet process mixture of Gaussian process functional regressions and its variational EM algorithm
Authors: Tao Li，Jinwen Ma
Date: 2023-02-01 00:00:00
Proceeding: Pattern Recognition 模式识别 2023 2
Tasks: Model Selection

Gaussian Process Functional Regression (GPFR) is a powerful tool in functional data analysis. In practical applications, functional data may be generated from different signal sources, and a single GPFR is not flexible enough to accurately model the data. To tackle the heterogeneity problem, a finite mixture of Gaussian Process Functional Regressions (mix-GPFR) was suggested. However, the number of components in mix-GPFR needs to be specified a priori, which is difficult to determine in practice. In this paper, we propose a Dirichlet Process Mixture of Gaussian Process Functional Regressions (DPM-GPFR), in which there are potentially infinite many GPFR components dominated by a Dirichlet process. Thus, DPM-GPFR is far more flexible than a single GPFR, and sidestep the model selection problem in mix-GPFR. We further develop a fully Bayesian treatment for learning DPM-GPFR based on the Variational Expectation-Maximization (VEM) algorithm. Experimental results on both synthetic datasets and real-world datasets demonstrate the effectiveness of our proposed method.
Towards Better Dynamic Graph Learning: New Architecture and Unified Library
Authors: Le Yu, Leilei Sun, Bowen Du, Weifeng Lv
Date: 2023-03-23 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Dynamic Link Prediction, Dynamic Node Classification, Graph Learning, Link Prediction, Node Classification

We propose DyGFormer, a new Transformer-based architecture for dynamic graph learning. DyGFormer is conceptually simple and only needs to learn from nodes' historical first-hop interactions by: (1) a neighbor co-occurrence encoding scheme that explores the correlations of the source node and destination node based on their historical sequences; (2) a patching technique that divides each sequence into multiple patches and feeds them to Transformer, allowing the model to effectively and efficiently benefit from longer histories. We also introduce DyGLib, a unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating protocols to promote reproducible, scalable, and credible dynamic graph learning research. By performing exhaustive experiments on thirteen datasets for dynamic link prediction and dynamic node classification tasks, we find that DyGFormer achieves state-of-the-art performance on most of the datasets, demonstrating its effectiveness in capturing nodes' correlations and long-term temporal dependencies. Moreover, some results of baselines are inconsistent with previous reports, which may be caused by their diverse but less rigorous implementations, showing the importance of DyGLib. All the used resources are publicly available at https://github.com/yule-BUAA/DyGLib.