pwc-archive/papers-with-abstracts huggingface dataset:
Genomic regulatory architecture of human embryo retroviral LTR elements affecting evolution, development, and pathophysiology of Modern Humans
Authors: Gennadi Glinsky
Date: 2023-10-25 00:00:00

Two distinct families of pan-primate endogenous retroviruses, namely HERVL and HERVH, infected primates germline, colonized host genomes, and evolved into the global retroviral genomic regulatory dominion (GRD) operating during human embryogenesis (HE). HE retroviral GRD constitutes 8839 highly conserved fixed LTR elements linked to 5444 down-stream target genes forged by evolution into a functionally-consonant constellation of 26 genome-wide multimodular genomic regulatory networks (GRNs), each of which is defined by significant enrichment of numerous single gene ontology (GO)-specific traits. Locations of GRNs appear scattered across chromosomes to occupy from 5.5%-15.09% of human genome. Each GRN harbors from 529-1486 retroviral LTRs derived from LTR7, MLT2A1, and MLT2A2 sequences that are quantitatively balanced according to their genome-wide abundance. GRNs integrate activities from 199-805 down-stream target genes, including transcription factors, chromatin-state remodelers, signal-sensing and signal-transduction mediators, enzymatic and receptor binding effectors, intracellular complexes and extracellular matrix elements, and cell-cell adhesion molecules. GRNs compositions consist of several hundred to thousands smaller GO enrichment-defined genomic regulatory modules (GRMs) combining from a dozen to hundreds LTRs and down-stream target genes, which appear to operate on individuals life-span timescale along specific phenotypic avenues to exert profound effects on patterns of transcription, protein-protein interactions, developmental phenotypes, physiological traits, and pathological conditions of Modern Humans. Overall, this study identifies 69,573 statistically significant retroviral LTR-linked GRMs (Binominal FDR q-value threshold of 0.001), including 27,601 GRMs validated by the single GO-specific directed acyclic graph (DAG) analyses across six GO annotations.
Influenza Hospitalisations in England during the 2022/23 Season: do different data sources drive divergence in modelled waves? A comparison of surveillance and administrative data
Authors: Jonathon Mellor, Rachel Christie, James Guilder, Robert S Paton, Suzanne Elgohari, Conall Watson, Sarah Deeny, Thomas Ward
Date: 2023-10-16 00:00:00
Tasks: Additive models

Accurate and representative data is vital for precisely reporting the impact of influenza in healthcare systems. Northern hemisphere winter 2022/23 experienced the most substantial influenza wave since the COVID-19 pandemic began in 2020. Simultaneously, new data streams become available within health services because of the pandemic. Comparing these data, surveillance and administrative, supports the accurate monitoring of population level disease trends. We analysed admissions rates per capita from four different collection mechanisms covering National Health Service hospital Trusts in England over the winter 2022/23 wave. We adjust for difference in reporting and extracted key epidemic characteristics including the maximum admission rate, peak timing, cumulative season admissions and growth rates by fitting generalised additive models at national and regional levels. By modelling the admission rates per capita across surveillance and administrative data systems we show that different data measuring the epidemic produce different estimates of key quantities. Nationally and in most regions the data correspond well for the maximum admission rate, date of peak and growth rate, however, in subnational analysis discrepancies in estimates arose, particularly for the cumulative admission rate. This research shows that the choice of data used to measure seasonal influenza epidemics can influence analysis substantially at sub-national levels. For the admission rate per capita there is comparability in the sentinel surveillance approach (which has other important functions), rapid situational reports, operational databases and time lagged administrative data giving assurance in their combined value. Utilising multiple sources of data aids understanding of the impact of seasonal influenza epidemics in the population.
The rejection that defies anti-rejection drugs. Chronic vascular rejection (allograft vasculopathy): The role of terminology and linguistic relativity
Authors: Vladimir M. Subbotin, Michael V. Subotin
Date: 2023-10-24 00:00:00
Tasks: Medical Procedure

Solid organ transplantation has by now become a common medical procedure. Owing to the introduction of new immunosuppressive drugs, the allograft loss due to acute rejection has been reduced significantly over time. Tragically, the number of donor organs lost due to allograft vasculopathy (AV), generally named chronic vascular rejection or chronic rejection, has remained significant and unchanged for decades. We argue that designation of AV as chronic rejection, and its classification as a delayed long-lasting reaction of recipient immune effectors against donor alloantigens have given us the wrong impression that we have identified the necessary cause of the disease. However, whatever treatment options we have in the anti-rejection toolbox, despite their success in treating acute rejection, do not work for AV. Yet, the scientific community has continued to conceptualize and approach AV within the alloimmunity and rejection model. Due to unproductive research from the alloimmunity perspective, the number of transplanted hearts lost due to this pathology today is almost the same as it was fifty years ago. We believe that this phenomenon falls under the rubric of linguistic relativity, and that the language we chose to name the disease has restricted our cognitive ability to solve the problem. While the initial perception of the AV as chronic rejection was logical and scientific, the subsequent experience revealed that such perception and approach have been fruitless, and likely are incorrect. Considering our tragic failure to prevent and treat allograft vasculopathy using all available knowledge on alloimmunity and rejection, we must finally disassociate the former from the latter. A good way to start this process is to change the words we are using; particularly, the words we chose to name the disease. We have to step out of the alloimmunity rejection box
A Sparse Bayesian Learning for Diagnosis of Nonstationary and Spatially Correlated Faults with Application to Multistation Assembly Systems
Authors: Jihoon Chung, Zhenyu Kong
Date: 2023-10-20 00:00:00
Tasks: Fault Diagnosis

Sensor technology developments provide a basis for effective fault diagnosis in manufacturing systems. However, the limited number of sensors due to physical constraints or undue costs hinders the accurate diagnosis in the actual process. In addition, time-varying operational conditions that generate nonstationary process faults and the correlation information in the process require to consider for accurate fault diagnosis in the manufacturing systems. This article proposes a novel fault diagnosis method: clustering spatially correlated sparse Bayesian learning (CSSBL), and explicitly demonstrates its applicability in a multistation assembly system that is vulnerable to the above challenges. Specifically, the method is based on a practical assumption that it will likely have a few process faults (sparse). In addition, the hierarchical structure of CSSBL has several parameterized prior distributions to address the above challenges. As posterior distributions of process faults do not have closed form, this paper derives approximate posterior distributions through Variational Bayes inference. The proposed method's efficacy is provided through numerical and real-world case studies utilizing an actual autobody assembly system. The generalizability of the proposed method allows the technique to be applied in fault diagnosis in other domains, including communication and healthcare systems.
Test-time Augmentation for Factual Probing
Authors: Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi, Kentaro Inui
Date: 2023-10-26 00:00:00
Tasks: Language Modeling, Language Modelling, Relation, World Knowledge

Factual probing is a method that uses prompts to test if a language model "knows" certain world knowledge facts. A problem in factual probing is that small changes to the prompt can lead to large changes in model output. Previous work aimed to alleviate this problem by optimizing prompts via text mining or fine-tuning. However, such approaches are relation-specific and do not generalize to unseen relation types. Here, we propose to use test-time augmentation (TTA) as a relation-agnostic method for reducing sensitivity to prompt variations by automatically augmenting and ensembling prompts at test time. Experiments show improved model calibration, i.e., with TTA, model confidence better reflects prediction accuracy. Improvements in prediction accuracy are observed for some models, but for other models, TTA leads to degradation. Error analysis identifies the difficulty of producing high-quality prompt variations as the main challenge for TTA.
PubDef: Defending Against Transfer Attacks From Public Models
Authors: Chawin Sitawarin, Jaewon Chang, David Huang, Wesson Altoyan, David Wagner
Date: 2023-10-26 00:00:00

Adversarial attacks have been a looming and unaddressed threat in the industry. However, through a decade-long history of the robustness evaluation literature, we have learned that mounting a strong or optimal attack is challenging. It requires both machine learning and domain expertise. In other words, the white-box threat model, religiously assumed by a large majority of the past literature, is unrealistic. In this paper, we propose a new practical threat model where the adversary relies on transfer attacks through publicly available surrogate models. We argue that this setting will become the most prevalent for security-sensitive applications in the future. We evaluate the transfer attacks in this setting and propose a specialized defense method based on a game-theoretic perspective. The defenses are evaluated under 24 public models and 11 attack algorithms across three datasets (CIFAR-10, CIFAR-100, and ImageNet). Under this threat model, our defense, PubDef, outperforms the state-of-the-art white-box adversarial training by a large margin with almost no loss in the normal accuracy. For instance, on ImageNet, our defense achieves 62% accuracy under the strongest transfer attack vs only 36% of the best adversarially trained model. Its accuracy when not under attack is only 2% lower than that of an undefended model (78% vs 80%). We release our code at https://github.com/wagner-group/pubdef.
In-Context Learning Dynamics with Random Binary Sequences
Authors: Eric J. Bigelow, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tomer D. Ullman
Date: 2023-10-26 00:00:00
Tasks: In-Context Learning

Large language models (LLMs) trained on huge corpora of text datasets demonstrate intriguing capabilities, achieving state-of-the-art performance on tasks they were not explicitly trained for. The precise nature of LLM capabilities is often mysterious, and different prompts can elicit different capabilities through in-context learning. We propose a framework that enables us to analyze in-context learning dynamics to understand latent concepts underlying LLMs' behavioral patterns. This provides a more nuanced understanding than success-or-failure evaluation benchmarks, but does not require observing internal activations as a mechanistic interpretation of circuits would. Inspired by the cognitive science of human randomness perception, we use random binary sequences as context and study dynamics of in-context learning by manipulating properties of context data, such as sequence length. In the latest GPT-3.5+ models, we find emergent abilities to generate seemingly random numbers and learn basic formal languages, with striking in-context learning dynamics where model outputs transition sharply from seemingly random behaviors to deterministic repetition.
Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion
Authors: Laura Smith, YunHao Cao, Sergey Levine
Date: 2023-10-26 00:00:00
Tasks: Deep Reinforcement Learning, Efficient Exploration, Reinforcement Learning (RL)

Deep reinforcement learning (RL) can enable robots to autonomously acquire complex behaviors, such as legged locomotion. However, RL in the real world is complicated by constraints on efficiency, safety, and overall training stability, which limits its practical applicability. We present APRL, a policy regularization framework that modulates the robot's exploration over the course of training, striking a balance between flexible improvement potential and focused, efficient exploration. APRL enables a quadrupedal robot to efficiently learn to walk entirely in the real world within minutes and continue to improve with more training where prior work saturates in performance. We demonstrate that continued training with APRL results in a policy that is substantially more capable of navigating challenging situations and is able to adapt to changes in dynamics with continued training.
JudgeLM: Fine-tuned Large Language Models are Scalable Judges
Authors: Lianghui Zhu, Xinggang Wang, Xinlong Wang
Date: 2023-10-26 00:00:00

Evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively. To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges. We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias. To address these issues, JudgeLM introduces a bag of techniques including swap augmentation, reference support, and reference drop, which clearly enhance the judge's performance. JudgeLM obtains the state-of-the-art judge performance on both the existing PandaLM benchmark and our proposed new benchmark. Our JudgeLM is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8 A100 GPUs. JudgeLM obtains high agreement with the teacher judge, achieving an agreement exceeding 90% that even surpasses human-to-human agreement. JudgeLM also demonstrates extended capabilities in being judges of the single answer, multimodal models, multiple answers, and multi-turn chat.
Using State-of-the-Art Speech Models to Evaluate Oral Reading Fluency in Ghana
Authors: Owen Henkel, Hannah Horne-Robinson, Libby Hills, Bill Roberts, Joshua McGrane
Date: 2023-10-26 00:00:00

This paper reports on a set of three recent experiments utilizing large-scale speech models to evaluate the oral reading fluency (ORF) of students in Ghana. While ORF is a well-established measure of foundational literacy, assessing it typically requires one-on-one sessions between a student and a trained evaluator, a process that is time-consuming and costly. Automating the evaluation of ORF could support better literacy instruction, particularly in education contexts where formative assessment is uncommon due to large class sizes and limited resources. To our knowledge, this research is among the first to examine the use of the most recent versions of large-scale speech models (Whisper V2 wav2vec2.0) for ORF assessment in the Global South. We find that Whisper V2 produces transcriptions of Ghanaian students reading aloud with a Word Error Rate of 13.5. This is close to the model's average WER on adult speech (12.8) and would have been considered state-of-the-art for children's speech transcription only a few years ago. We also find that when these transcriptions are used to produce fully automated ORF scores, they closely align with scores generated by expert human graders, with a correlation coefficient of 0.96. Importantly, these results were achieved on a representative dataset (i.e., students with regional accents, recordings taken in actual classrooms), using a free and publicly available speech model out of the box (i.e., no fine-tuning). This suggests that using large-scale speech models to assess ORF may be feasible to implement and scale in lower-resource, linguistically diverse educational contexts.
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Authors: Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox
Date: 2023-10-26 00:00:00
Tasks: Imitation Learning

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .
SPA: A Graph Spectral Alignment Perspective for Domain Adaptation
Date: 2023-10-26 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Domain Adaptation, Single Particle Analysis, Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) is a pivotal form in machine learning to extend the in-domain model to the distinctive target domains where the data distributions differ. Most prior works focus on capturing the inter-domain transferability but largely overlook rich intra-domain structures, which empirically results in even worse discriminability. In this work, we introduce a novel graph SPectral Alignment (SPA) framework to tackle the tradeoff. The core of our method is briefly condensed as follows: (i)-by casting the DA problem to graph primitives, SPA composes a coarse graph alignment mechanism with a novel spectral regularizer towards aligning the domain graphs in eigenspaces; (ii)-we further develop a fine-grained message propagation module -- upon a novel neighbor-aware self-training mechanism -- in order for enhanced discriminability in the target domain. On standardized benchmarks, the extensive experiments of SPA demonstrate that its performance has surpassed the existing cutting-edge DA methods. Coupled with dense model analysis, we conclude that our approach indeed possesses superior efficacy, robustness, discriminability, and transferability. Code and data are available at: https://github.com/CrownX/SPA.
An Open Source Data Contamination Report for Large Language Models
Authors: Yucheng Li, Frank Guerin, Chenghua Lin
Date: 2023-10-26 00:00:00
Tasks: HellaSwag, Language Modeling, Language Modelling, Large Language Model, MMLU, Multiple-choice

Data contamination in model evaluation has become increasingly prevalent with the growing popularity of large language models. It allows models to "cheat" via memorisation instead of displaying true capabilities. Therefore, contamination analysis has become an crucial part of reliable model evaluation to validate results. However, existing contamination analysis is usually conducted internally by large language model developers and often lacks transparency and completeness. This paper presents an extensive data contamination report for over 15 popular large language models across six popular multiple-choice QA benchmarks. We also introduce an open-source pipeline that enables the community to perform contamination analysis on customised data and models. Our experiments reveal varying contamination levels ranging from 1\% to 45\% across benchmarks, with the contamination degree increasing rapidly over time. Performance analysis of large language models indicates that data contamination does not necessarily lead to increased model metrics: while significant accuracy boosts of up to 14\% and 7\% are observed on contaminated C-Eval and Hellaswag benchmarks, only a minimal increase is noted on contaminated MMLU. We also find larger models seem able to gain more advantages than smaller models on contaminated test sets.
Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models
Authors: Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, Sanjeev Arora
Date: 2023-10-26 00:00:00
Tasks: AI Agent

With LLMs shifting their role from statistical modeling of language to serving as general-purpose AI agents, how should LLM evaluations change? Arguably, a key ability of an AI agent is to flexibly combine, as needed, the basic skills it has learned. The capability to combine skills plays an important role in (human) pedagogy and also in a paper on emergence phenomena (Arora & Goyal, 2023). This work introduces Skill-Mix, a new evaluation to measure ability to combine skills. Using a list of $N$ skills the evaluator repeatedly picks random subsets of $k$ skills and asks the LLM to produce text combining that subset of skills. Since the number of subsets grows like $N^k$, for even modest $k$ this evaluation will, with high probability, require the LLM to produce text significantly different from any text in the training set. The paper develops a methodology for (a) designing and administering such an evaluation, and (b) automatic grading (plus spot-checking by humans) of the results using GPT-4 as well as the open LLaMA-2 70B model. Administering a version of to popular chatbots gave results that, while generally in line with prior expectations, contained surprises. Sizeable differences exist among model capabilities that are not captured by their ranking on popular LLM leaderboards ("cramming for the leaderboard"). Furthermore, simple probability calculations indicate that GPT-4's reasonable performance on $k=5$ is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training. We sketch how the methodology can lead to a Skill-Mix based eco-system of open evaluations for AI capabilities of future models.
Instability of computer vision models is a necessary result of the task itself
Authors: Oliver Turnbull, George Cevora
Date: 2023-10-26 00:00:00

Adversarial examples resulting from instability of current computer vision models are an extremely important topic due to their potential to compromise any application. In this paper we demonstrate that instability is inevitable due to a) symmetries (translational invariance) of the data, b) the categorical nature of the classification task, and c) the fundamental discrepancy of classifying images as objects themselves. The issue is further exacerbated by non-exhaustive labelling of the training data. Therefore we conclude that instability is a necessary result of how the problem of computer vision is currently formulated. While the problem cannot be eliminated, through the analysis of the causes, we have arrived at ways how it can be partially alleviated. These include i) increasing the resolution of images, ii) providing contextual information for the image, iii) exhaustive labelling of training data, and iv) preventing attackers from frequent access to the computer vision system.