pwc-archive/papers-with-abstracts huggingface dataset:
Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval
Authors: Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, Nan Duan
Date: 2023-02-03 00:00:00
Tasks: Relation, Representation Learning, Retrieval, Sentence, XLM-R

Recently multi-lingual pre-trained language models (PLM) such as mBERT and XLM-R have achieved impressive strides in cross-lingual dense retrieval. Despite its successes, they are general-purpose PLM while the multilingual PLM tailored for cross-lingual retrieval is still unexplored. Motivated by an observation that the sentences in parallel documents are approximately in the same order, which is universal across languages, we propose to model this sequential sentence relation to facilitate cross-lingual representation learning. Specifically, we propose a multilingual PLM called masked sentence model (MSM), which consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document. The document encoder is shared for all languages to model the universal sequential sentence relation across languages. To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives. Comprehensive experiments on four cross-lingual retrieval tasks show MSM significantly outperforms existing advanced pre-training models, demonstrating the effectiveness and stronger cross-lingual retrieval capabilities of our approach. Code and model will be available.
Around the world in 60 words: A generative vocabulary test for online research
Authors: Pol van Rijn, Yue Sun, Harin Lee, Raja Marjieh, Ilia Sucholutsky, Francesca Lanzarini, Elisabeth André, Nori Jacoby
Date: 2023-02-03 00:00:00
Tasks: Cultural Vocal Bursts Intensity Prediction

Conducting experiments with diverse participants in their native languages can uncover insights into culture, cognition, and language that may not be revealed otherwise. However, conducting these experiments online makes it difficult to validate self-reported language proficiency. Furthermore, existing proficiency tests are small and cover only a few languages. We present an automated pipeline to generate vocabulary tests using text from Wikipedia. Our pipeline samples rare nouns and creates pseudowords with the same low-level statistics. Six behavioral experiments (N=236) in six countries and eight languages show that (a) our test can distinguish between native speakers of closely related languages, (b) the test is reliable ($r=0.82$), and (c) performance strongly correlates with existing tests (LexTale) and self-reports. We further show that test accuracy is negatively correlated with the linguistic distance between the tested and the native language. Our test, available in eight languages, can easily be extended to other languages.
Bioformer: an efficient transformer language model for biomedical text mining
Authors: Li Fang, Qingyu Chen, Chih-Hsuan Wei, Zhiyong Lu, Kai Wang
Date: 2023-02-03 00:00:00
Tasks: Articles, Document Classification, Language Modeling, Language Modelling, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Question Answering, Relation Extraction

Pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) have achieved state-of-the-art performance in natural language processing (NLP) tasks. Recently, BERT has been adapted to the biomedical domain. Despite the effectiveness, these models have hundreds of millions of parameters and are computationally expensive when applied to large-scale NLP applications. We hypothesized that the number of parameters of the original BERT can be dramatically reduced with minor impact on performance. In this study, we present Bioformer, a compact BERT model for biomedical text mining. We pretrained two Bioformer models (named Bioformer8L and Bioformer16L) which reduced the model size by 60% compared to BERTBase. Bioformer uses a biomedical vocabulary and was pre-trained from scratch on PubMed abstracts and PubMed Central full-text articles. We thoroughly evaluated the performance of Bioformer as well as existing biomedical BERT models including BioBERT and PubMedBERT on 15 benchmark datasets of four different biomedical NLP tasks: named entity recognition, relation extraction, question answering and document classification. The results show that with 60% fewer parameters, Bioformer16L is only 0.1% less accurate than PubMedBERT while Bioformer8L is 0.9% less accurate than PubMedBERT. Both Bioformer16L and Bioformer8L outperformed BioBERTBase-v1.1. In addition, Bioformer16L and Bioformer8L are 2-3 fold as fast as PubMedBERT/BioBERTBase-v1.1. Bioformer has been successfully deployed to PubTator Central providing gene annotations over 35 million PubMed abstracts and 5 million PubMed Central full-text articles. We make Bioformer publicly available via https://github.com/WGLab/bioformer, including pre-trained models, datasets, and instructions for downstream use.
Controlling for Stereotypes in Multimodal Language Model Evaluation
Authors: Manuj Malik, Richard Johansson
Date: 2023-02-03 00:00:00
Tasks: Language Model Evaluation, Language Modeling, Language Modelling, model

We propose a methodology and design two benchmark sets for measuring to what extent language-and-vision language models use the visual signal in the presence or absence of stereotypes. The first benchmark is designed to test for stereotypical colors of common objects, while the second benchmark considers gender stereotypes. The key idea is to compare predictions when the image conforms to the stereotype to predictions when it does not. Our results show that there is significant variation among multimodal models: the recent Transformer-based FLAVA seems to be more sensitive to the choice of image and less affected by stereotypes than older CNN-based models such as VisualBERT and LXMERT. This effect is more discernible in this type of controlled setting than in traditional evaluations where we do not know whether the model relied on the stereotype or the visual signal.
The unreasonable effectiveness of few-shot learning for machine translation
Authors: Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, Orhan Firat
Date: 2023-02-02 00:00:00
Tasks: Decoder, Few-Shot Learning, Machine Translation, Self-Supervised Learning, Translation

We demonstrate the potential of few-shot translation systems, trained with unpaired language data, for both high and low-resource language pairs. We show that with only 5 examples of high-quality translation data shown at inference, a transformer decoder-only model trained solely with self-supervised learning, is able to match specialized supervised state-of-the-art models as well as more general commercial translation systems. In particular, we outperform the best performing system on the WMT'21 English - Chinese news translation task by only using five examples of English - Chinese parallel data at inference. Moreover, our approach in building these models does not necessitate joint multilingual training or back-translation, is conceptually simple and shows the potential to extend to the multilingual setting. Furthermore, the resulting models are two orders of magnitude smaller than state-of-the-art language models. We then analyze the factors which impact the performance of few-shot translation systems, and highlight that the quality of the few-shot demonstrations heavily determines the quality of the translations generated by our models. Finally, we show that the few-shot paradigm also provides a way to control certain attributes of the translation -- we show that we are able to control for regional varieties and formality using only a five examples at inference, paving the way towards controllable machine translation systems.
Curriculum-Guided Abstractive Summarization
Authors: Sajad Sotudeh, Hanieh Deilamsalehy, Franck Dernoncourt, Nazli Goharian
Date: 2023-02-02 00:00:00
Tasks: Abstractive Text Summarization, Decoder, Extreme Summarization, Informativeness, Sentence

Recent Transformer-based summarization models have provided a promising approach to abstractive summarization. They go beyond sentence selection and extractive strategies to deal with more complicated tasks such as novel word generation and sentence paraphrasing. Nonetheless, these models have two shortcomings: (1) they often perform poorly in content selection, and (2) their training strategy is not quite efficient, which restricts model performance. In this paper, we explore two orthogonal ways to compensate for these pitfalls. First, we augment the Transformer network with a sentence cross-attention module in the decoder, encouraging more abstraction of salient content. Second, we include a curriculum learning approach to reweight the training samples, bringing about an efficient learning procedure. Our second approach to enhance the training strategy of Transformers networks makes stronger gains as compared to the first approach. We apply our model on extreme summarization dataset of Reddit TIFU posts. We further look into three cross-domain summarization datasets (Webis-TLDR-17, CNN/DM, and XSum), measuring the efficacy of curriculum learning when applied in summarization. Moreover, a human evaluation is conducted to show the efficacy of the proposed method in terms of qualitative criteria, namely, fluency, informativeness, and overall quality.
A Hybrid Training Algorithm for Continuum Deep Learning Neuro-Skin Neural Network
Authors: Mehrdad Shafiei Dizaji
Date: 2023-02-03 00:00:00

In this brief paper, a learning algorithm is developed for Deep Learning NeuroSkin Neural Network to improve their learning properties. Neuroskin is a new type of neural network presented recently by the authors. It is comprised of a cellular membrane which has a neuron attached to each cell. The neuron is the cells nucleus. A neuroskin is modelled using finite elements. Each element of the finite element represents a cell. Each cells neuron has dendritic fibers which connects it to the nodes of the cell. On the other hand, its axon is connected to the nodes of a number of different neurons. The neuroskin is trained to contract upon receiving an input. The learning takes place during updating iterations using sensitivity analysis. It is shown that while the neuroskin cannot present the desirable response, it improves gradually to the desired level.
Coevolving Boolean and Multi-Valued Regulatory Networks
Authors: Larry Bull
Date: 2023-02-03 00:00:00

Random Boolean networks have been used widely to explore aspects of gene regulatory networks. A modified form of the model through which to systematically explore the effects of increasing the number of gene states has previously been introduced. In this paper, these discrete dynamical networks are coevolved within coupled, rugged fitness landscapes to explore their behaviour. Results suggest the general properties of the Boolean model remain with higher valued logic regardless of the update scheme or fitness sampling method. Introducing topological asymmetry in the coevolving networks is seen to alter behaviour.
Symbiosis of an artificial neural network and models of biological neurons: training and testing
Authors: Tatyana Bogatenko, Konstantin Sergeev, Andrei Slepnev, Jürgen Kurths, Nadezhda Semenova
Date: 2023-02-03 00:00:00

In this paper we show the possibility of creating and identifying the features of an artificial neural network (ANN) which consists of mathematical models of biological neurons. The FitzHugh--Nagumo (FHN) system is used as an example of model demonstrating simplified neuron activity. First, in order to reveal how biological neurons can be embedded within an ANN, we train the ANN with nonlinear neurons to solve a a basic image recognition problem with MNIST database; and next, we describe how FHN systems can be introduced into this trained ANN. After all, we show that an ANN with FHN systems inside can be successfully trained and its accuracy becomes larger. What has been done above opens up great opportunities in terms of the direction of analog neural networks, in which artificial neurons can be replaced by biological ones. \end{abstract}
Sample Complexity of Probability Divergences under Group Symmetry
Authors: Ziyu Chen, Markos A. Katsoulakis, Luc Rey-Bellet, Wei Zhu
Date: 2023-02-03 00:00:00

We rigorously quantify the improvement in the sample complexity of variational divergence estimations for group-invariant distributions. In the cases of the Wasserstein-1 metric and the Lipschitz-regularized $\alpha$-divergences, the reduction of sample complexity is proportional to the group size if the group is finite. In addition to the published version at ICML 2023, our proof indeed has included the case when the group is infinite such as compact Lie groups, the convergence rate can be further improved and depends on the intrinsic dimension of the fundamental domain characterized by the scaling of its covering number. Our approach is different from that in [Tahmasebi & Jegelka, ICML 2024] and our work also applies to asymmetric divergences, such as the Lipschitz-regularized $\alpha$-divergences. For the maximum mean discrepancy (MMD), the improvement of sample complexity is more nuanced, as it depends on not only the group size but also the choice of kernel. Numerical simulations verify our theories.
Failure-informed adaptive sampling for PINNs, Part II: combining with re-sampling and subset simulation
Authors: Zhiwei Gao, Tao Tang, Liang Yan, Tao Zhou
Date: 2023-02-03 00:00:00

This is the second part of our series works on failure-informed adaptive sampling for physic-informed neural networks (FI-PINNs). In our previous work \cite{gao2022failure}, we have presented an adaptive sampling framework by using the failure probability as the posterior error indicator, where the truncated Gaussian model has been adopted for estimating the indicator. In this work, we present two novel extensions to FI-PINNs. The first extension consist in combining with a re-sampling technique, so that the new algorithm can maintain a constant training size. This is achieved through a cosine-annealing, which gradually transforms the sampling of collocation points from uniform to adaptive via training progress. The second extension is to present the subset simulation algorithm as the posterior model (instead of the truncated Gaussian model) for estimating the error indicator, which can more effectively estimate the failure probability and generate new effective training points in the failure region. We investigate the performance of the new approach using several challenging problems, and numerical experiments demonstrate a significant improvement over the original algorithm.
Committed Private Information Retrieval
Authors: Quang Cao, Hong Yen Tran, Son Hoang Dau, Xun Yi, Emanuele Viterbo, Chen Feng, Yu-Chih Huang, Jingge Zhu, Stanislav Kruglik, Han Mao Kiah
Date: 2023-02-03 00:00:00
Tasks: Information Retrieval, Retrieval

A private information retrieval (PIR) scheme allows a client to retrieve a data item $x_i$ among $n$ items $x_1,x_2,\ldots,x_n$ from $k$ servers, without revealing what $i$ is even when $t < k$ servers collude and try to learn $i$. Such a PIR scheme is said to be $t$-private. A PIR scheme is $v$-verifiable if the client can verify the correctness of the retrieved $x_i$ even when $v \leq k$ servers collude and try to fool the client by sending manipulated data. Most of the previous works in the literature on PIR assumed that $v < k$, leaving the case of all-colluding servers open. We propose a generic construction that combines a linear map commitment (LMC) and an arbitrary linear PIR scheme to produce a $k$-verifiable PIR scheme, termed a committed PIR scheme. Such a scheme guarantees that even in the worst scenario, when all servers are under the control of an attacker, although the privacy is unavoidably lost, the client won't be fooled into accepting an incorrect $x_i$. We demonstrate the practicality of our proposal by implementing the committed PIR schemes based on the Lai-Malavolta LMC and three well-known PIR schemes using the GMP library and blst, the current fastest C library for elliptic curve pairings.
DAFA: Diversity-Aware Feature Aggregation for Attention-Based Video Object Detection
Authors: Si-Dong Roh, Ki-Seok Chung
Date: 2022-09-01 00:00:00
Proceeding: IEEE Access 2022 9
Tasks: Diversity, Management, object-detection, Object Detection, Video Object Detection

We present a framework for attention-based video object detection using a simple yet effective external memory management algorithm. An attention mechanism has been adopted in video object detection task to enrich the features of key frames using adjacent frames. Although several recent studies utilized frame-level first-in-first-out (FIFO) memory to collect global video information, such a memory structure suffers from collection inefficiency, which results in low attention performance and high computational cost. To address this issue, we developed a novel scheme called diversity-aware feature aggregation (DAFA). Whereas other methods do not store sufficient feature information without expanding memory capacity, DAFA efficiently collects diverse features while avoiding redundancy using a simple Euclidean distance-based metric. Experimental results on the ImageNet VID dataset demonstrate that our lightweight model with global attention achieves 83.5 mAP on the ResNet-101 backbone, which exceeds the accuracy levels of most existing methods with a minimum runtime. Our method with global and local attention stages obtains 84.5 and 85.9 mAP on ResNet-101 and ResNeXt-101, respectively, thus achieving state-of-the-art performance without requiring additional post-processing methods.
Patient-friendly Clinical Notes: Towards a new Text Simplification Dataset
Authors: Jan Trienes, Jörg Schlötterer, Hans-Ulrich Schildhaus, Christin Seifert
Date: 2022-12-07 00:00:00
Proceeding: Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR) 2022 12
Tasks: Text Simplification

Automatic text simplification can help patients to better understand their own clinical notes. A major hurdle for the development of clinical text simplification methods is the lack of high quality resources. We report ongoing efforts in creating a parallel dataset of professionally simplified clinical notes. Currently, this corpus consists of 851 document-level simplifications of German pathology reports. We highlight characteristics of this dataset and establish first baselines for paragraph-level simplification.
Motion ID: Human Authentication Approach
Authors: Aleksei Gavron, Konstantin Belev, Konstantin Kudelkin, Vladislav Shikhov, Andrey Akushevich, Alexey Fartukov, Vladimir Paramonov, Dmitry Syromolotov, Artem Makoyan
Date: 2023-01-25 00:00:00
Tasks: Time Series, Time Series Analysis

We introduce a novel approach to user authentication called Motion ID. The method employs motion sensing provided by inertial measurement units (IMUs), using it to verify the persons identity via short time series of IMU data captured by the mobile device. The paper presents two labeled datasets with unlock events: the first features IMU measurements, provided by six users who continuously collected data on six different smartphones for a period of 12 weeks. The second one contains 50 hours of IMU data for one specific motion pattern, provided by 101 users. Moreover, we present a two-stage user authentication process that employs motion pattern identification and user verification and is based on data preprocessing and machine learning. The Results section details the assessment of the method proposed, comparing it with existing biometric authentication methods and the Android biometric standard. The method has demonstrated high accuracy, indicating that it could be successfully used in combination with existing methods. Furthermore, the method exhibits significant promise as a standalone solution. We provide the datasets to the scholarly community and share our project code.
Command Line Interface Risk Modeling
Authors: Dr Anthony L. Faulds
Date: 2023-01-17 00:00:00
Tasks: Cloud Computing

Protecting sensitive data is an essential part of security in cloud computing. However, only specific privileged individuals have access to view or interact with this data; therefore, it is unscalable to depend on these individuals also to maintain the software. A solution to this is to allow non-privileged individuals access to maintain these systems but mask sensitive information from egressing. To this end, we have created a machine-learning model to predict and redact fields with sensitive data. This work concentrates on Azure PowerShell, showing how it applies to other command-line interfaces and APIs. Using the F5-score as a weighted metric, we demonstrate different transformation techniques to map this problem from an unknown field to the well-researched area of natural language processing.
Enabling Trade-offs in Privacy and Utility in Genomic Data Beacons and Summary Statistics
Authors: Rajagopal Venkatesaramani, Zhiyu Wan, Bradley A. Malin, Yevgeniy Vorobeychik
Date: 2023-01-11 00:00:00

The collection and sharing of genomic data are becoming increasingly commonplace in research, clinical, and direct-to-consumer settings. The computational protocols typically adopted to protect individual privacy include sharing summary statistics, such as allele frequencies, or limiting query responses to the presence/absence of alleles of interest using web-services called Beacons. However, even such limited releases are susceptible to likelihood-ratio-based membership-inference attacks. Several approaches have been proposed to preserve privacy, which either suppress a subset of genomic variants or modify query responses for specific variants (e.g., adding noise, as in differential privacy). However, many of these approaches result in a significant utility loss, either suppressing many variants or adding a substantial amount of noise. In this paper, we introduce optimization-based approaches to explicitly trade off the utility of summary data or Beacon responses and privacy with respect to membership-inference attacks based on likelihood-ratios, combining variant suppression and modification. We consider two attack models. In the first, an attacker applies a likelihood-ratio test to make membership-inference claims. In the second model, an attacker uses a threshold that accounts for the effect of the data release on the separation in scores between individuals in the dataset and those who are not. We further introduce highly scalable approaches for approximately solving the privacy-utility tradeoff problem when information is either in the form of summary statistics or presence/absence queries. Finally, we show that the proposed approaches outperform the state of the art in both utility and privacy through an extensive evaluation with public datasets.