trentmkelly/LessWrong-43k huggingface dataset:
z is not the cause of x
by hrbigelow — 2023-10-23T17:43:59.563Z

Introduction

In this note, I argue that the interpretation of representations (mental or machine learning) as causes of observed effects, is wrong by any sensible definition of causality. This interpretation seems to be somewhat common, although not pervasive. I propose a new interpretation based on dualism. In the new interpretation, representations are instead causes of subsequent representations. The gist of it is shown in the figure, with time or causality flowing along the x-axis, and the level of representation along the y-axis. This re-interpretation has implications for the broader notion of "truth", in both the scientific and common-sense senses.

I partly arrived at this conclusion when contemplating large language models. They made me question more and more what it means to understand, reason, or know something. Actually, I had asked this question years before in slightly different form: what is the foundation of machine learning, and is there such a thing as "the true model" which we are trying to discover through SGD and regularization? What makes a "good" representation of the data?

One answer that kept coming up in some form or another is that a good representation zshould discover the causal factors of the observed data, x. The question became especially acute for me when I was studying the original variational autoencoder paper by Kingma and Welling. As an aside, a variational autoencoder (VAE) is a pair of models called generative (mapping z to x) and recognition (mapping x to z) which usually co-train each other on a dataset from x. After studying the paper for a long time, it seemed to me one could train a VAE and fit a data distribution arbitrarily well without the hidden variables z meaning anything at all. Indeed, nowhere in the paper do they mention that z are causal factors of x. And, as much as I tried, I could not find anything in the math where z might take on emergent meaning.

But why should z take on "meaning"? The model was just using these hidden variables as a mathematical tool to create a flexible marginal distribution over x. On reflection, the only reason I was surprised was because I thought of the VAE as a model for human cognition, and we know humans build meaningful representations, and somehow do it in an unsupervised fashion kin...
Some of my predictable updates on AI
by Aaron_Scher — 2023-10-23T17:24:34.720Z

Introduction

Author note: I’m struggling to write this in a way I’m happy with. Rather than having it sit in my drafts, I’m going to post in an unfinished state now. I don’t think I endorse sharing/upvoting this much, but if you find the content particularly compelling feel free to overrule me. Epistemic status: speculative and imprecise forecasts

Joe Carlsmith has a lengthy blog post about

predictably updating on AI risk

. I skimmed it and found it interesting. This post includes some of my predictions about AI and AI risk in the next year or so, including what I expect to happen and how I think I should update if my predictions are wrong. Partially I’m writing this to force myself to make predictions, partially I’m writing to get feedback from others about my predictions, and partially I’m writing to try and spread these predictions and their associated updates with others concerned about AI existential safety. This is a suspicious activity partially because it’s really hard to currently know how future events should affect my beliefs. i.e., I currently expect a thing like election interference to not change my beliefs much, but there may be unpredicted-by-me circumstances which actually make election interference evidence very important; doing this exercise might make it more difficult to change my beliefs properly later.

For each item, I’ll note:

What’s the uncertainty
What do I expect and how much do I expect it
How I should update if something I don’t expect happens, and how much I should care

tldr

In short, here are some things I expect:

Some big policy stuff will happen, and it will seem neutral or positive
AI labs will agree on some safety standards which will seem positive
AI labs might compete on some safety things, but I expect a mediocre outcome
Task-oriented / agentic LLMs will be a bigger deal, and this will be scary
Misuse threats will be a big deal, and many more people will care about this
AI will enable interference in the US 2024 elections, but the policy effects are the main part I care about
We’ll make significant progress on alignment for current AIs which will look optimistic

Overall, I expect the most significant updates on the table in the next year are related to how seriously AGI labs seem to be taking existential safety (where I expect some positive signs), and how AI alignment research is going (where I expect pretty good result...
Programmatic backdoors: DNNs can use SGD to run arbitrary stateful computation
by Fabien Roger — 2023-10-23T16:37:45.611Z

Thanks to Kshitij Sachan for helpful feedback on the draft of this post.

If you train a neural network with SGD, you can embed within the weights of the network any state machine: the network encodes the state st in its weights, uses it and the current input to computes the new state st+1, and then uses SGD to encode the new state in its weights.

We present an algorithm to embed a state machine into a pytorch module made out of neural network primitives (+, x, sigmoid, ...) but using a non-standard architecture. We use it for several toy experiments, such as this experiment where we translate a simple state machine into a neural network, and make the output of the network vary when we reach an intermediate state:

In this post, we explain how to embed a state machine into a neural network, discuss some of the limitations of our algorithm and how these limitations could be overcome, and present some experimental results.

Our embedding is quite efficient. If you’re allowed to use stopgrad (a function that stops the gradient from passing through), we only need one parameter to store one real number, and storing only requires a small constant number of floating point operations. If you aren’t allowed to use stop grad, we can’t encode as efficiently.

We did this work for two reasons:

It was fun and easy, and gave us some new intuition for how SGD works. Also, as computer scientists, we’re professionally obligated to take opportunities to demonstrate

that surprising things allow implementation of general computation

.
Part of our motivation in constructing this was to shed some light on

gradient-hacking

concerns. This particular construction seems very unlikely to be constructible by early transformative AI, and in general we suspect gradient hacking won’t be a big safety concern for early transformative AI (ctrl-f “gradient hacking”

here

for more of Buck’s unconfident opinion on this). But it does allow us to construct a lower bound on the maximum possible efficiency of a system that piggybacks on SGD in order to store information; further research on the tractability of gradient hacking might want to use this as a starting point.

Simple notebooks with our experimental results can be found here:

https://github.com/redwoodresearch/Gradient-Machine

The embedding algorithm

The idea

As a simple example, suppose you’re training a model N on...
Machine Unlearning Evaluations as Interpretability Benchmarks
by NickyP — 2023-10-23T16:33:04.878Z

Interpreting Models by Ablation. Image generated by DALL-E 3.

Introduction

Interpretability in machine learning, especially in language models, is an area with a large number of contributions. While this can be quite useful for improving our understanding of models, one issue is that there is the lack of robust benchmarks to evaluate the efficacy of different interpretability techniques. Drawing comparisons and determining their true effectiveness in real-world scenarios becomes a difficult task.

Interestingly, there exists a parallel in the realm of non-language models under the research umbrella of Machine Unlearning. In this field, the objective is twofold: firstly, to deliberately diminish the model's performance on specified "unlearned" tasks, and secondly, to ensure that the model's proficiency is maintained or even enhanced on certain "retained" tasks. The inherent challenge here is achieving a balance between these seemingly opposing goals, and thus comes with a range of metrics for measuring the effectiveness of the techniques.

Drawing inspiration from Machine Unlearning, I believe that the metrics developed in this space could potentially serve as a litmus test for interpretability techniques in language models. By applying interpretability techniques as unlearning strategies, we can better test the effectiveness of interpretability methods, essentially setting benchmarks for how well these techniques can steer language models in desired directions.

If we aspire to have truly interpretable models, we must not only develop sophisticated techniques, but also robust benchmarks against which these techniques can be validated. Machine Unlearning might just offer the rigorous testing ground we need.

The rest of this post will: 1) Give a brief overview of Machine Unlearning, 2) Give a brief list of Machine Unlearning metrics, and how they may be applicable, 3) Give a deeper dive on each of the metrics, 4) Discuss how these fit in with existing metrics in Interpretability.

Machine Unlearning

Many papers in the subfield of Machine Unlearning are motivated by privacy preservation, and pose the question: "If we trained on someone's information that is now retracted, how can we remove that information without needing to retrain the whole model?"

There are multiple ways you might achieve unlearning. The "ideal/standard" is often to train the model again, but witho...
VLM-RM: Specifying Rewards with Natural Language
by ChengCheng — 2023-10-23T14:11:34.493Z
Link: https://far.ai/post/2023-10-vlm-rm/

tl;dr

We show how to use Vision-Language Models (VLM), and specifically CLIP models, as reward models (RM) for RL agents. Instead of manually specifying a reward function, we only need to provide text prompts like “a humanoid robot kneeling” to instruct and provide feedback to the agent. Importantly, we find that larger VLMs provide more accurate reward signals, so we expect this method to work even better with future models.

Figure 1: We use CLIP as a reward model to train a MuJoCo humanoid robot to (1) stand with raised arms, (2) sit in a lotus position, (3) do the splits, and (4) kneel on the ground (from left to right). Each task is specified by a single sentence text prompt like ‘a humanoid robot kneeling’ without additional prompt engineering. For more videos of our trained agents, see sites.google.com/view/vlm-rm

Motivation

Reinforcement Learning (RL) relies on either manually specifying reward functions, which can be challenging to define, or learning reward models from a large amount of human feedback, which is expensive to provide.

To address these challenges, we explore the potential of using pretrained Vision-Language Models (VLMs) to provide a reward signal instead. VLMs are pretrained on a large amount of data connecting text and images. CLIP models are a prime example of VLMs, and we show how to leverage them to specify tasks for RL agents using natural language.

Using pretrained models to oversee other models has been a recent trend in the alignment community. Methods such as Constitutional AI leverage capable language models to supervise other language models, taking only a small amount of human feedback as input (e.g., in the form of a “constitution”). This approach is more sample efficient and potentially more scalable than using pairwise comparison. However, using pretrained models to supervise other agents has been mostly studied in language-only tasks. Our work opens up a new domain in which we can evaluate related approaches: vision-based RL tasks. For further details, please refer to our research paper Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning.

Vision Language Models as Reward Models
(VLM-RMs)

VLMs, like CLIP, are trained to understand both visual and textual information. Previous work showed that these models can successfully solve downstream tasks they have not been trained for, such as classifying images ...
Contra Dance Dialect Survey
by jefftk — 2023-10-23T13:40:08.294Z

Fifteen years ago I travelled to a bunch of dances and wrote something
up (

pdf)
on how people tended to dance in different areas. I'm curious how
this has changed over the years, and while I have my observations I'd
like to get some better data. So! I'm running a survey, and would
find your response very helpful:

survey.

Comment via: facebook, mastodon
What is an "anti-Occamian prior"?
by Zane — 2023-10-23T02:26:10.851Z

I've seen references to "anti-Occamian priors" in the Sequences, where Eliezer was talking about how not all possible minds would agree with Occam's Razor. I'm not sure how such a prior could consistently exist.

It seems like Occam's Razor just logically follows from the basic premises of probability theory. Assume the "complexity" of a hypothesis is how many bits it takes to specify under a particular method of specifying hypotheses, and that hypotheses can be of any length. Then for any prior that assigns nonzero probability to any finite hypothesis H, there must exist some level of complexity L such that any hypothesis more complex than L is less likely than H.

(That is to say, if a particular 13-bit hypothesis is 0.01% likely, then there are at most 9,999 other hypotheses with >= 0.01% probability mass. If the most complicated of these <10,000 hypotheses is 27 bits, then every hypothesis that takes 28 bits or more to specify is less likely than the 13-bit hypothesis. You can change around the numbers 13 and 0.01% and 27 as much as you want, but as long as there's any hypothesis whatsoever with non-infinitesimal probability, then there's some level where everything more complex than that level is less likely than that hypothesis.)

This seems to prove that an "anti-Occamian prior" - that is to say, a hypothesis that always assigns more probability to more complex hypotheses and less to the less complex - is impossible. Or at least, that it assigns zero probability to every finite hypothesis. (You could, I suppose, construct a prior such that {sum of probability mass from all 1-bit hypotheses} is 1/3 of {sum of probability mass from all 2-bit hypotheses}, which is then itself 1/3 of {sum of probability mass from all 3-bit hypotheses}, and on and on forever, and that would indeed be anti-Occamian - but it would also assign zero probability to every finite hypothesis, which would make it essentially meaningless.)

Am I missing something about what "anti-Occamian prior" is really supposed to mean here, or how it could really be consistent?
AI Safety is Dropping the Ball on Clown Attacks
by trevor — 2023-10-22T20:09:31.810Z

Epistemic status: High confidence (~>70%) that clown attacks are prevalent, and deliberately weaponized by governments and/or intelligence agencies in particular. Very high confidence (~>90%) that the human brain is highly vulnerable to clown attacks, and that a lack of awareness of clown attacks is a security risk, like using the word “password” as your password, except with control of your own mind at stake rather than control over your computer’s operating system and/or your files. This has been steelmanned; the 10 years ago/10 years from now error bars seem appropriately wide.

These concepts are complicated, and I have done my best to make it as easy as possible for most people in AI safety to understand it, even people without a quant background (e.g. AI governance).

Clown attacks

The core dynamic of clown attacks is that perception of social status affects what thoughts the human brain is and isn’t willing to think. This can be used by intelligence agencies and governments to completely deny access to specific lines of thought. Generally, there’s a lot of ways to socially engineer someone’s world model by taking a target concept and having the wrong people say it at specific times and in specific ways. Clown attacks include having right-wing clowns being the main people who are seen talking about the Snowden revelations, or degrowthers being the main people talking about the possibility that technological advancement/human capabilities will end it’s net-positive trend. These are only specific examples of cost-efficient ways to use a specific circumstance (clowns) to change the way that someone feels about a targeted concept.

With clown attacks, the exploit/zero day is the human tendency to associate specific lines of thought with low social status or low-status people, which will consistently inhibit the human brain from pursuing that targeted line of thought.

Although clown attacks may seem mundane on their own, they are a case study proving that powerful human thought steering technologies have probably already been invented, deployed, and tested at scale by AI companies, and are reasonably likely to end up being weaponized against the entire AI safety community at some point in the next 10 years.

AI safety is dropping the ball on clown attacks, at minimum

AI safety is basically a community of nerds who each chanced upon the engineer...
The Drowning Child
by Tomás B. — 2023-10-22T16:39:53.016Z

Epistemic status: possibly very evil devil's advocacy, philospohicly unsophisticard ruminations

We are all familiar with Singer's drowning child thought experiment. I have often found it very compelling, or at least felt I ought to find it very compelling. My revealed preferences, as with those of nearly everyone else, tell a different story.

But it has always struck me that Singer's story can be read in two directions, and social desirability bias would likely prevent one from reading it aloud in the less-common way.

If a thought experiment is to drastically alter one's life, it seems worth a little devil's advocacy.

In this spirit, I ask if the fact that one does not donate all their energy to ameliorating the plight of the world's most impoverished provide some evidence that one does not, in fact, care as much about the near-by drowning child as they think they do?

What are some purely-selfish reasons one might have for saving the child. I can think of a few.

You gain some prestige as a hero.
You demonstrate your strength and competence publicly in the event someone witnesses your heroism.
It is fun to imagine heroic acts in your head that you would not, in fact, do in real life.
The gratefulness of the child and/or their parents isn't without value.
Saving a child in this way is a novel, interesting experience.

If we alter Singer's story such that a hero-for-hire offers to save the near-by child, and he asks for a sum equivalent to the price of your suit (and a guarantee of that you will keep the transaction confidential) we can remove the weight of these motivations. Supposing you cannot swim and the hero-for-hire is the only person around, would you take this deal?

I think I would take the deal, but if I'm honest with myself it does seem very slightly less compelling.

If we extend this scenario and imagine that accepting this deal every week would mean sacrificing half your annual salary, the decision becomes even more complex. Would the majority still take the deal? Or would they perhaps find a path to their office where large bodies of water are not in view?
Announcing Timaeus
by Jesse Hoogland — 2023-10-22T11:59:03.938Z

Timaeus is a new AI safety research organization dedicated to making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. Currently, we are working on singular learning theory and developmental interpretability. Over time we expect to work on a broader research agenda, and to create understanding-based evals informed by our research.

Let sleeping gods (not) lie.

Activities

Our primary focus is research. For now, we're a remote-first organization. We collaborate primarily through online seminars and the DevInterp Discord, with regular in-person meetings at workshops and conferences (see below). We're also investing time in academic outreach to increase the general capacity for work in technical AI alignment.

Research

We believe singular learning theory, a mathematical subject founded by Sumio Watanabe, will lead to a better fundamental understanding of large-scale learning machines and the computational structures that they learn to represent. It has already given us concepts like the learning coefficient and insights into phase transitions in Bayesian learning. We expect significant advances in the theory to be possible, and that these advances can inform new tools for alignment.

Developmental interpretability is an approach to understanding the emergence of structure in neural networks, which is informed by singular learning theory but also draws on mechanistic interpretability and ideas from statistical physics and developmental biology. The key idea is that phase transitions organize learning and that detecting, locating, and understanding these transitions could pave a road to evaluation tools that prevent the development of dangerous capabilities, values, and behaviors. We're engaged in a research sprint to test the assumptions of this approach.

We see these as two particularly promising research directions, and they are our focus for now. Like any ambitious research, they are not guaranteed to succeed, but there's plenty more water in the well. Broadly speaking, the research agenda of Timaeus is oriented towards solving problems in technical AI alignment using deep ideas from across many areas of mathematics and the sciences, with a "full stack" approach that integrates work from pure mathematics through to machine learning experiments.

The outputs we have contributed to so far:

Hoogland & Van W

...
Into AI Safety - Episode 0
by jacobhaimes — 2023-10-22T03:30:57.865Z
Link: https://into-ai-safety.github.io/episode/episode-0/

As I mentioned in a previous post, I am starting a podcast which will follow me as I begin my career in AI safety. Today I published episode 0, an intro which covers the reasons that I am making this series, what I plan to focus on, and some context for my current projects.

Initially, I am publishing episodes on the Into AI Safety website and Spotify (show link). For further logistics details and information, visit the Into AI Safety About page.

If you have any questions, recommendations, concerns, etc. please reach out. I would greatly appreciate any guidance or advice you have to offer.