pwc-archive/papers-with-abstracts huggingface dataset:
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Authors: Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
Date: 2022-12-01 00:00:00

Obtaining labeled data to train a model for a task of interest is often expensive. Prior work shows training models on multitask data augmented with task descriptions (prompts) effectively transfers knowledge to new tasks. Towards efficiently building task-specific models, we assume access to a small number (32-1000) of unlabeled target-task examples and use those to retrieve the most similar labeled examples from a large pool of multitask data augmented with prompts. Compared to the current practice of finetuning models on uniformly sampled prompted multitask data (e.g.: FLAN, T0), our approach of finetuning on cross-task nearest neighbors is significantly more data-efficient. Using only 2% of the data from the P3 pool without any labeled target-task data, our models outperform strong baselines trained on all available data by 3-30% on 12 out of 14 datasets representing held-out tasks including legal and scientific document QA. Similarly, models trained on cross-task nearest neighbors from SuperNaturalInstructions, representing about 5% of the pool, obtain comparable performance to state-of-the-art models on 12 held-out tasks from that pool. Moreover, the models produced by our approach also provide a better initialization than single multitask finetuned models for few-shot finetuning on target-task data, as shown by a 2-23% relative improvement over few-shot finetuned T0-3B models on 8 datasets.
Open Relation and Event Type Discovery with Type Abstraction
Authors: Sha Li, Heng Ji, Jiawei Han
Date: 2022-11-30 00:00:00
Tasks: Event Extraction, Relation, Relation Extraction, Vocal Bursts Type Prediction

Conventional closed-world information extraction (IE) approaches rely on human ontologies to define the scope for extraction. As a result, such approaches fall short when applied to new domains. This calls for systems that can automatically infer new types from given corpora, a task which we refer to as type discovery. To tackle this problem, we introduce the idea of type abstraction, where the model is prompted to generalize and name the type. Then we use the similarity between inferred names to induce clusters. Observing that this abstraction-based representation is often complementary to the entity/trigger token representation, we set up these two representations as two views and design our model as a co-training framework. Our experiments on multiple relation extraction and event extraction datasets consistently show the advantage of our type abstraction approach. Code available at https://github.com/raspberryice/type-discovery-abs.
Word Alignment in the Era of Deep Learning: A Tutorial
Authors: Bryan Li
Date: 2022-11-30 00:00:00
Tasks: Deep Learning, Machine Translation, NMT, Translation, Word Alignment

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.
Task-Specific Embeddings for Ante-Hoc Explainable Text Classification
Authors: Kishaloy Halder, Josip Krapac, Alan Akbik, Anthony Brew, Matti Lyra
Date: 2022-11-30 00:00:00
Tasks: Classification, Incremental Learning, text-classification, Text Classification

Current state-of-the-art approaches to text classification typically leverage BERT-style Transformer models with a softmax classifier, jointly fine-tuned to predict class labels of a target task. In this paper, we instead propose an alternative training objective in which we learn task-specific embeddings of text: our proposed objective learns embeddings such that all texts that share the same target class label should be close together in the embedding space, while all others should be far apart. This allows us to replace the softmax classifier with a more interpretable k-nearest-neighbor classification approach. In a series of experiments, we show that this yields a number of interesting benefits: (1) The resulting order induced by distances in the embedding space can be used to directly explain classification decisions. (2) This facilitates qualitative inspection of the training data, helping us to better understand the problem space and identify labelling quality issues. (3) The learned distances to some degree generalize to unseen classes, allowing us to incrementally add new classes without retraining the model. We present extensive experiments which show that the benefits of ante-hoc explainability and incremental learning come at no cost in overall classification accuracy, thus pointing to practical applicability of our proposed approach.
Improving astroBERT using Semantic Textual Similarity
Authors: Felix Grezes, Thomas Allen, Sergi Blanco-Cuaresma, Alberto Accomazzi, Michael J. Kurtz, Golnaz Shapurian, Edwin Henneken, Carolyn S. Grant, Donna M. Thompson, Timothy W. Hostetler, Matthew R. Templeton, Kelly E. Lockhart, Shinyi Chen, Jennifer Koch, Taylor Jacovich, Pavlos Protopapas
Date: 2022-11-29 00:00:00
Tasks: Astronomy, Language Modeling, Language Modelling, Semantic Textual Similarity

The NASA Astrophysics Data System (ADS) is an essential tool for researchers that allows them to explore the astronomy and astrophysics scientific literature, but it has yet to exploit recent advances in natural language processing. At ADASS 2021, we introduced astroBERT, a machine learning language model tailored to the text used in astronomy papers in ADS. In this work we: - announce the first public release of the astroBERT language model; - show how astroBERT improves over existing public language models on astrophysics specific tasks; - and detail how ADS plans to harness the unique structure of scientific papers, the citation graph and citation context, to further improve astroBERT.
Scalable and adaptive variational Bayes methods for Hawkes processes
Authors: Deborah Sulem, Vincent Rivoirard, Judith Rousseau
Date: 2022-12-01 00:00:00
Tasks: Data Augmentation, Point Processes

Hawkes processes are often applied to model dependence and interaction phenomena in multivariate event data sets, such as neuronal spike trains, social interactions, and financial transactions. In the nonparametric setting, learning the temporal dependence structure of Hawkes processes is generally a computationally expensive task, all the more with Bayesian estimation methods. In particular, for generalised nonlinear Hawkes processes, Monte-Carlo Markov Chain methods applied to compute the doubly intractable posterior distribution are not scalable to high-dimensional processes in practice. Recently, efficient algorithms targeting a mean-field variational approximation of the posterior distribution have been proposed. In this work, we first unify existing variational Bayes approaches under a general nonparametric inference framework, and analyse the asymptotic properties of these methods under easily verifiable conditions on the prior, the variational class, and the nonlinear model. Secondly, we propose a novel sparsity-inducing procedure, and derive an adaptive mean-field variational algorithm for the popular sigmoid Hawkes processes. Our algorithm is parallelisable and therefore computationally efficient in high-dimensional setting. Through an extensive set of numerical simulations, we also demonstrate that our procedure is able to adapt to the dimensionality of the parameter of the Hawkes process, and is partially robust to some type of model mis-specification.
Dense Hebbian neural networks: a replica symmetric picture of supervised learning
Authors: Elena Agliari, Linda Albanese, Francesco Alemanno, Andrea Alessandrelli, Adriano Barra, Fosca Giannotti, Daniele Lotito, Dino Pedreschi
Date: 2022-11-25 00:00:00
Tasks: Retrieval, valid

We consider dense, associative neural-networks trained by a teacher (i.e., with supervision) and we investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as quality and quantity of the training dataset, network storage and noise, that is valid in the limit of large network size and structureless datasets: these networks may work in a ultra-storage regime (where they can handle a huge amount of patterns, if compared with shallow neural networks) or in a ultra-detection regime (where they can perform pattern recognition at prohibitive signal-to-noise ratios, if compared with shallow neural networks). Guided by the random theory as a reference framework, we also test numerically learning, storing and retrieval capabilities shown by these networks on structured datasets as MNist and Fashion MNist. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate supervised learning in neural networks, beyond the shallow limit, in general.
Quadratically Regularized Optimal Transport: nearly optimal potentials and convergence of discrete Laplace operators
Authors: Gilles Mordant, Stephen Zhang
Date: 2022-11-20 00:00:00

We consider the conjecture proposed in Matsumoto, Zhang and Schiebinger (2022) suggesting that optimal transport with quadratic regularisation can be used to construct a graph whose discrete Laplace operator converges to the Laplace--Beltrami operator. We derive first order optimal potentials for the problem under consideration and find that the resulting solutions exhibit a surprising resemblance to the well-known Barenblatt--Prattle solution of the porous medium equation. Then, relying on these first order optimal potentials, we derive the pointwise $L^2$-limit of such discrete operators built from an i.i.d. random sample on a smooth compact manifold. Simulation results complementing the limiting distribution results are also presented.
Assessing the Impact of Music Recommendation Diversity on Listeners: A Longitudinal Study
Authors: Lorenzo Porcaro, Emilia Gómez, Carlos Castillo
Date: 2022-12-01 00:00:00
Tasks: Diversity, Music Recommendation

We present the results of a 12-week longitudinal user study wherein the participants, 110 subjects from Southern Europe, received on a daily basis Electronic Music (EM) diversified recommendations. By analyzing their explicit and implicit feedback, we show that exposure to specific levels of music recommendation diversity may be responsible for long-term impacts on listeners' attitudes. In particular, we highlight the function of diversity in increasing the openness in listening to EM, a music genre not particularly known or liked by the participants previous to their participation in the study. Moreover, we demonstrate that recommendations may help listeners in removing positive and negative attachments towards EM, deconstructing pre-existing implicit associations but also stereotypes associated with this music. In addition, our results show the significant clout that recommendation diversity has in generating curiosity in listeners.
Principled Multi-Aspect Evaluation Measures of Rankings
Authors: Maria Maistro, Lucas Chaves Lima, Jakob Grue Simonsen, Christina Lioma
Date: 2022-12-01 00:00:00
Tasks: Document Ranking, Information Retrieval, Retrieval

Information Retrieval evaluation has traditionally focused on defining principled ways of assessing the relevance of a ranked list of documents with respect to a query. Several methods extend this type of evaluation beyond relevance, making it possible to evaluate different aspects of a document ranking (e.g., relevance, usefulness, or credibility) using a single measure (multi-aspect evaluation). However, these methods either are (i) tailor-made for specific aspects and do not extend to other types or numbers of aspects, or (ii) have theoretical anomalies, e.g. assign maximum score to a ranking where all documents are labelled with the lowest grade with respect to all aspects (e.g., not relevant, not credible, etc.). We present a theoretically principled multi-aspect evaluation method that can be used for any number, and any type, of aspects. A thorough empirical evaluation using up to 5 aspects and a total of 425 runs officially submitted to 10 TREC tracks shows that our method is more discriminative than the state-of-the-art and overcomes theoretical limitations of the state-of-the-art.
NIR-Prompt: A Multi-task Generalized Neural Information Retrieval Training Framework
Authors: Shicheng Xu, Liang Pang, HuaWei Shen, Xueqi Cheng
Date: 2022-12-01 00:00:00
Tasks: Information Retrieval, Open-Domain Question Answering, Question Answering, Reranking, Retrieval

Information retrieval aims to find information that meets users' needs from the corpus. Different needs correspond to different IR tasks such as document retrieval, open-domain question answering, retrieval-based dialogue, etc., while they share the same schema to estimate the relationship between texts. It indicates that a good IR model can generalize to different tasks and domains. However, previous studies indicate that state-of-the-art neural information retrieval (NIR) models, e.g, pre-trained language models (PLMs) are hard to generalize. Mainly because the end-to-end fine-tuning paradigm makes the model overemphasize task-specific signals and domain biases but loses the ability to capture generalized essential signals. To address this problem, we propose a novel NIR training framework named NIR-Prompt for retrieval and reranking stages based on the idea of decoupling signal capturing and combination. NIR-Prompt exploits Essential Matching Module (EMM) to capture the essential matching signals and gets the description of tasks by Matching Description Module (MDM). The description is used as task-adaptation information to combine the essential matching signals to adapt to different tasks. Experiments under in-domain multi-task, out-of-domain multi-task, and new task adaptation settings show that NIR-Prompt can improve the generalization of PLMs in NIR for both retrieval and reranking stages compared with baselines.
Movie Recommendation System using Composite Ranking
Authors: Irish Mehta, Aashal Kamdar
Date: 2022-11-30 00:00:00
Tasks: Articles, Collaborative Filtering, Movie Recommendation, Recommendation Systems

In today's world, abundant digital content like e-books, movies, videos and articles are available for consumption. It is daunting to review everything accessible and decide what to watch next. Consequently, digital media providers want to capitalise on this confusion and tackle it to increase user engagement, eventually leading to higher revenues. Content providers often utilise recommendation systems as an efficacious approach for combating such information overload. This paper concentrates on developing a synthetic approach for recommending movies. Traditionally, movie recommendation systems use either collaborative filtering, which utilises user interaction with the media, or content-based filtering, which makes use of the movie's available metadata. Technological advancements have also introduced a hybrid technique that integrates both systems. However, our approach deals solely with content-based recommendations, further enhancing it with a ranking algorithm based on content similarity metrics. The three metrics contributing to the ranking are similarity in metadata, visual content, and user reviews of the movies. We use text vectorization followed by cosine similarity for metadata, feature extraction by a pre-trained VGG19 followed by K-means clustering for visual content, and a comparison of sentiments for user reviews. Such a system allows viewers to know movies that "feel" the same.
Electric vehicle demand estimation and charging station allocation using urban informatics
Authors: Zhiyan Yi, Xiaoyue Cathy Liu, Ran Wei
Date: 2022-04-01 00:00:00
Proceeding: Transportation Research Part D: Transport and Environment 2022 4

This paper performs a novel data-driven approach to optimize electric vehicle (EV) public
charging. We translate the study area into a directed graph by partitioning it into discrete grids. A
modified geographical PageRank (MGPR) model is developed to estimate EV charging demand,
built upon trip origin–destination (OD) and social dimension features, and validated against real-
world charging data. The results are fed into the capacitated maximal coverage location problem
(CMCLP) model to optimize the spatial layout of public charging stations by maximizing their
utilization. It is shown that MGPR can effectively quantify the EV charging demand with satis-
factory accuracy. Optimized EV charging stations based on the CMCLP model can remedy the
spatial mismatch between the EV demand and the existing charging station allocations. The
developed methodological framework is highly generalizable and can be extended to other re-
gions for EV charging demand estimation and optimal charging infrastructure siting.
Exploring the Relationship between Center and Neighborhoods: Central Vector oriented Self-Similarity Network for Hyperspectral Image Classification
Authors: Mingsong Li, Yikun Liu, Guangkuo Xue, Yuwen Huang, and Gongping Yang
Date: 2022-10-31 00:00:00
Proceeding: IEEE Transactions on Circuits and Systems for Video Technology 2022 10
Tasks: Hyperspectral image analysis, Hyperspectral Image Classification, Hyperspectral Image Segmentation, image-classification

To mine the spectral-spatial information of target pixel in hyperspectral image classification (HSIC), convolutional neural network (CNN)-based models widely adopt patch-based input pattern, where a patch represents its central pixel and the neighbor pixels play auxiliary roles in the classification process. However, compared to the central pixel, its neighbor pixels often have different contributions for classification. Although many existing patch-based CNNs could adaptively emphasize the spatial neighbor information, most of them ignore the latent relationship between the center pixel and its neighbor pixels. Moreover, efficient spectral-spatial feature extraction has been a difficult yet vital topic for HSIC. To address the mentioned problems, a central vector oriented self-similarity network (CVSSN) is proposed for HSIC. Specifically, based on two similarity measures, we firstly design an adaptive weight addition based spectral vector self-similarity module (AWA-SVSS) in input space and a Euclidean distance based feature vector self-similarity module (ED-FVSS) in feature space to fully mine the central vector oriented spatial relationships. Besides, a spectral-spatial information fusion module (SSIF) is formulated as a new pattern to fuse the central 1D spectral vector and the corresponding 3D patch for efficient spectral-spatial feature learning of the subsequent modules. Moreover, we implement a channel spatial separation convolution module (CSS-Conv) and a scale information complementary convolution module (SIC-Conv) for efficient spectral-spatial feature learning. Extensive experimental results on four popular HSI data sets demonstrate the effectiveness and efficiency of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/CVSSN.
Improved Infilling of Missing Metadata from Expendable Bathythermographs (XBTs) Using Multiple Machine Learning Methods
Authors: Stephen Haddad, Rachel E. Killick, Matthew D. Palmer, Mark J. Webb, Rachel Prudden, Francesco Capponi, and Samantha V. Adams
Date: 2022-09-01 00:00:00
Proceeding: Journal of Atmospheric and Oceanic Technology 2022 9
Tasks: Vocal Bursts Type Prediction

Historical in situ ocean temperature profile measurements are important for a wide range of ocean and

climate research activities. A large proportion of the profile observations have been recorded using expendable bathyther-
mographs (XBTs), and required bias corrections for use in climate change studies. It is generally accepted that the bias,

and therefore bias correction, depends on the type of XBT used. However, poor historical metadata collection practices

mean the XBT probe type information is often missing, for 59% of profiles between 1967 and 2000, limiting the develop-
ment of reliable bias corrections. We develop a process of estimating missing instrument type metadata (the combination

of both model and manufacturer) systematically, constructing a machine learning pipeline based on thorough data explo-
ration to inform these choices. The predicted instrument type, where missing, will facilitate improved XBT bias correc-
tions. The new approach improves the accuracy of the XBT type classification compared to previous approaches from a

recall value of 0.75–0.94. We also develop an approach to account for the uncertainty associated with metadata assign-
ments using ensembles of decision trees, which could feed into an ensemble approach to creating ocean temperature data-
sets. We describe the challenges arising from the nature of the dataset in applying standard machine learning techniques

to the problem. We have implemented this in a portable, reproducible way using standard data science tools, with a view
to these techniques being applied to other similar problems in climate science.
Economics of NFTs: The Value of Creator Royalties
Authors: Brett Hemenway Falk, Gerry Tsoukalas, Niuniu Zhang
Date: 2022-12-01 00:00:00

Non-Fungible Tokens (NFTs) promise to revolutionize how content creators (e.g., artists) price and sell their work. One core feature of NFTs is the option to embed creator royalties which earmark a percentage of future sale proceeds to creators, each time their NFTs change hands. As popular as this feature is in practice, its utility is often questioned because buyers, the argument goes, simply ``price it in at the time of purchase''. As intuitive as this argument sounds, it is incomplete. We find royalties can add value to creators in at least three distinct ways. (i) Risk sharing: when creators and buyers are risk sensitive, royalties can improve trade by splitting the risks associated with future price volatility; (ii) Dynamic pricing: in the presence of information asymmetry, royalties can extract more revenues from better-informed speculators over time, mimicking the benefits of ``dynamic pricing''; (iii) Price discrimination: when creators sell multi-unit NFT collections, royalties can better capture value from heterogeneous buyers. Our results suggest creator royalties play an important and sometimes overlooked role in the economics of NFTs.