trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] Play with SAEs on Llama 3
by Tom McGrath — 2024-09-25T22:35:44.824Z

We (Goodfire) just put our research preview live - you can play with Llama 3 and use sparse autoencoders to read & write from its internal activations. This is a linkpost for:

The research preview.
Our blog post about building it.

Taking research and turning it into something you can actually use and play with has been great. It's surprising how much of a difference iterating on something when you expect it to actually be used feels; I think it's definitely pushed the quality of what you can do with SAEs up a notch.
Mira Murati leaves OpenAI/ OpenAI to remove non-profit control
by Sodium — 2024-09-25T21:15:17.315Z

OpenAI to remove non-profit control

Reuters reports: Exclusive: OpenAI to remove non-profit control and give Sam Altman equity

ChatGPT-maker OpenAI is working on a plan to restructure its core business into a for-profit benefit corporation that will no longer be controlled by its non-profit board, people familiar with the matter told Reuters, in a move that will make the company more attractive to investors.

Full text here

Murati leaves OpenAI

Murati writes

Hi all,

I have something to share with you. After much reflection, I have made the difficult decision to leave OpenAl.

My six-and-a-half years with the OpenAl team have been an extraordinary privilege. While I'll express my gratitude to many individuals in the coming days, I want to start by thanking Sam and Greg for their trust in me to lead the technical organization and for their support throughout the years.

There's never an ideal time to step away from a place one cherishes, yet this moment feels right. Our recent releases of speech-to-speech and OpenAl o1 mark the beginning of a new era in interaction and intelligence - achievements made possible by your ingenuity and craftsmanship. We didn't merely build smarter models, we fundamentally changed how Al systems learn and reason through complex problems. We brought safety research from the theoretical realm into practical applications, creating models that are more robust, aligned, and steerable than ever before. Our work has made cutting-edge Al research intuitive and accessible, developing technology that adapts and evolves based on everyone's input. This success is a testament to our outstanding teamwork, and it is because of your brilliance, your dedication, and your commitment that OpenAl stands at the pinnacle of Al innovation.

I'm stepping away because I want to create the time and space to do my own exploration. For now, my primary focus is doing everything in my power to ensure a smooth transition, maintaining the momentum we've built.

I will forever be grateful for the opportunity to build and work alongside this remarkable team. Together, we've pushed the boundaries of scientific understanding in our quest to improve human well-being. While I may no longer be in the trenches with you, I will still be rooting for you all.

With deep gratitude for the friendships forged, the triumphs achieved, and most importantly, the challenges overcome together.

Mira

Sta...
Comparing Forecasting Track Records for AI Benchmarking and Beyond
by ChristianWilliams — 2024-09-25T21:01:15.975Z
Link: https://www.metaculus.com/notebooks/28552/comparing-forecasting-track-records-for-ai-benchmarking-and-beyond/

Thank you to Molly Hickman, Deger Turan, Ryan Beck, Sylvain Chevalier, Luke Sabor, and Leonard Barrett for reading drafts and discussing the ideas presented here.

Special thank you to Nikos Bosse for his valuable feedback and input on statistical methods.

Introduction

On a recent episode of Conversations with Tyler, political forecaster Nate Silver expressed skepticism that AIs will soon replace human forecasters, although there is active interest from many including FutureSearch, Halawi et al, and Schoenegger et al. More recently, the Center for AI Safety responded to Silver’s skepticism with Superhuman Automated Forecasting, a post announcing the creation of a “superhuman AI forecasting bot” that “performs better than experienced human forecasters.” However their methodology has been highly contested.

We believe Silver’s skepticism is still warranted: AI forecasting does not match — let alone surpass — the performance of experienced forecasters. Metaculus created the AI Benchmarking Series to examine this question, and we've now developed an epistemically rigorous approach to address many of the methodological issues highlighted by others.

Challenges in Forecasting Evaluation

Many bot forecasting studies to date have used historical forecasting questions to try to back-test performance, but their validity has been called into question by potential data leakage issues. In contrast, our analysis uses predictions from our AI Forecasting Benchmark Series, which compares community forecasters, pro forecasters, and AI forecasting bots, all on real-world “live” questions for which outcomes were unknown as of the series launch. This avoids any data leakage risks.

Our approach addresses another common AI benchmarking concern: that models may be trained to do well on a test that never changes, but then fail when they take a test that they have never seen. We are running quarterly AI forecasting tournaments through mid-2025, each time using new questions including fresh topics, thereby avoiding the possibility of training a bot on a repeating set of questions.

Methodology

We pose identical questions to bots and Pro forecasters on topics similar to those found on Metaculus with an emphasis on questions that aren’t found elsewhere, in order to reward independent reasoning. We were careful to ensure that forecasts from both bots and humans remained hidden while forec...
Extending the Off-Switch Game: Toward a Robust Framework for AI Corrigibility
by OwenChen — 2024-09-25T20:38:22.928Z

Introduction
The off-switch game, first introduced by Stuart Armstrong in the context of AI safety, highlights a critical challenge in AI alignment: designing systems that are not only powerful and goal-driven but also corrigible—meaning they allow humans to intervene and shut them down without resistance. In this game, an AI agent must reason about whether to allow a human to turn it off, given its objective function. If the AI perceives its shutdown as harmful to achieving its goal, it may develop incentives to prevent human intervention. Solving this problem is fundamental to ensuring AI systems remain aligned with human values and control.

In this post, I extend the off-switch game by exploring two key dimensions that have received less attention: (1) the integration of dynamic goal-modification mechanisms that can align with human preferences over time, and (2) the role of trust calibration in the human-AI interaction, which affects when humans choose to deploy the off-switch. Both dimensions, I argue, are crucial for developing a robust framework for AI corrigibility. I will also explore potential solutions that aim to enhance AI’s transparency and cooperative behavior, making it more predictable and, therefore, safer.

Understanding the Off-Switch Game

The basic structure of the off-switch game involves three actors:

A human operator (H) who has the option to press an off-switch.
An AI agent (A) with a utility function (U_A) that defines its goals.
An off-switch mechanism (S) that halts the AI’s operation when pressed by the human.

The challenge arises when the AI's utility function is rigid. If U_A assigns higher utility to completing its objective than to being switched off, A may resist the off-switch to continue pursuing its goal. In this scenario, A could either overtly prevent the human from using S or covertly influence the human’s decision-making to reduce the likelihood of S being pressed.

Existing solutions to this problem often involve incorporating uncertainty into the AI’s utility function, leading to corrigibility—the AI remains indifferent or mildly supportive of shutdown if it detects that the human prefers it. However, current models of corrigibility lack key features that real-world applications of AI would require: dynamic goal revision and trust calibration.

1. Dynamic Goal-Modification for Real-World Applications

One limitation of the off-swi...
Evaluating Synthetic Activations composed of SAE Latents in GPT-2
by Giorgi Giglemiani — 2024-09-25T20:37:48.227Z
Link: https://arxiv.org/abs/2409.15019

This research was completed for London AI Safety Research (LASR) Labs 2024. The team was supervised by Stefan Heimershiem (Apollo Research). Find out more about the programme and express interest in upcoming iterations here.

TL;DR

We find that real activations cannot be mimicked using a “bag of SAE latents” with no internal structure, and that geometric and statistical properties of SAE latents play an important role in the composition of a realistic activation (Figure 1).

Figure 1: The CDF of model sensitivity (over 1000 samples) for perturbations towards real, synthetic, and random activations. Synthetic-baseline activations (purple, simple “bags of SAE latents”) cannot match the behavior of real activations (orange), but synthetic-structured activations (green, taking into account SAE latent geometry) achieve a closer match. This shows that the geometry of SAE latents matters.

Paper Overview

Sparse Auto-Encoders (SAEs) are an increasingly popular method in mechanistic interpretability used to decompose a model’s residual stream into monosemantic and interpretable latents. As the community’s reliance on SAEs grows, it is crucial to verify that they accurately capture abstractions that the model uses in its computation. Recent work has shown that linearly perturbing a language model's activations at an early layer can result in step-function-like changes (“blowups”) in the model's final layer activations (Heimersheim and Mendel, 2024). Importantly, it has been found that perturbing in the direction of model-generated (“real”) activations caused blowups sooner than perturbing in random directions (Figure 2), i.e. the model is more sensitive to perturbations towards real activations than perturbations in random directions. This behavior distinguishes real activations from random directions, and we use it to evaluate synthetic activations made out of SAE latents.

In order to test the model’s sensitivity to different directions, we perturb 1000 real activations towards different types of activations and present the results in Figure 3. Here is a summary of the different types of activations we compare:

Model-generated: An activation directly taken from Layer 1 for a random prompt given to the model.
Random: A randomly sampled point from a normal distribution with the same mean and covariance as model-generated activations.
Synthetic Baseline:&

...
Climate Change And Global Warming
by Zero Contradictions — 2024-09-25T19:13:09.508Z
Link: https://zerocontradictions.net/civilization/climate-change

1. Understanding Climate Change

See: Wikipedia: Climate Change.

I support the scientific consensus that climate change and global warming are ongoing, human-caused, and likely to have strong negative consequences for humans, the Earth, and the world’s ecosystems. The evidence is clear that the CO2 levels the Earth’s atmosphere, the Earth’s sea levels, and the Earth’s temperatures are the highest they have ever been in modern human history. The best explanation for why such significant changes to the Earth’s ecology have occurred in such a geologically short time period is that the Industrial Revolution caused humans to start burn fossil fuels for energy.

However, I also believe that climate change has become moralized and politicized on both sides. This has caused people to have a lot of propaganda and inaccurate information to arise. There are many ecological phenomena that have been wrongly attributed to climate change, since their causes have been misidentified. In this essay, I will talk about climate change, how it compares to other ecological issues, my predictions for the future, and what humanity should do to stop climate change and address related ecological issues.

1.1. Why is Climate Change a Problem?

The Earth is estimated to have had much higher temperatures, CO2 levels, and other differing ecological conditions millions of years ago.

The climate has always been fluctuating significantly throughout the Earth’s recorded history.

That is true. However, we also have to keep in mind that modern humans evolved during the Last Glacial Period when the planet was much cooler. From a bio-evolutionary perspective, it makes sense that humans would want to keep the Earth at cooler temperatures than the projected temperatures that we are predicting decades from now. As global temperatures continue to rise and the climate continues to change, humans will become increasingly more maladapted to the Earth’s environmental conditions.

If we want ourselves, the next generations, and the Earth’s flora and fauna to avoid struggling and dying as a result of these changing ecological conditions, then it is imperative that we act now to prevent further global warming. If possible, it’d also be ideal if we can find a way to reverse global warming and its effects.

(see the rest of the post in the link)
How to prevent collusion when using untrusted models to monitor each other
by Buck — 2024-09-25T18:58:20.693Z

Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques, because by default you are unlikely to notice the problem until the hard-to-reverse outcome has already occurred.

One of the main classes of techniques that you could apply in the hope of preventing these catastrophic actions is what we call untrusted monitoring (introduced in our paper here), which is roughly “before you execute a model’s action, ask another instance of that model whether the proposed action would be extremely dangerous”.

This basic idea of asking one instance of a model to review actions proposed by another instance is very natural; it’s been investigated by e.g. LLM Critics Help Catch LLM Bugs and various earlier work[1]. But if you’re worried about the models actively trying to subvert your safety measures scheming, there’s a big problem with this approach: why doesn’t the monitor just lie when it’s asked about an actually dangerous action, and let that action go through? I’ll call this “collusion”.

In this post, I’ll analyze how much of a problem collusion is for untrusted monitoring, using the conservative approach we argued for in our post introducing AI control and demonstrated in AI Control: Improving Safety Despite Intentional Subversion: I model these potentially-scheming models as adversaries, and analyze safety by trying to think of strategies that the models could follow to subvert our safety techniques. I don’t present empirical evidence in this post, though all of this is strongly informed by experimental evidence from control evaluations my collaborators and I have done.

According to this adversarial analysis, the risk from collusion depends strongly on what methods you adopt to mitigate it:

If you don’t design the untrusted monitoring setup to prevent collusion, then collusion is quite likely to cause untrusted monitoring to fail.
If you include some simple and cheap anti-collusion mechanisms (which I’ll descri

...
Alignment by default: the simulation hypothesis
by gb — 2024-09-25T16:26:00.552Z

I wrote a very brief comment to Eliezer's last post, which upon reflection I thought could benefit from a separate post to fully discuss its implications.

Eliezer argues that we shouldn't really hope to be spared even though

Asking an ASI to leave a hole in a Dyson Shell, so that Earth could get some sunlight not transformed to infrared, would cost It 4.5e-10 of Its income.

He then goes on to discuss various reasons why the minute cost to the ASI is insufficient reason for hope.

I made the following counter:

Isn’t the ASI likely to ascribe a prior much greater than 4.54e-10 that it is in a simulation, being tested precisely for its willingness to spare its creators?

I later added:

I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it will find such evidence.

So, what's wrong with my argument, exactly?
A Dialogue on Deceptive Alignment Risks
by Rauno Arike — 2024-09-25T16:10:12.294Z

I have long been puzzled by the wide differences in the amount of concern established alignment researchers show for deceptive alignment risks. I haven't seen a post that clearly outlines the cruxes: much of the relevant conversation is buried deep within LessWrong comment sections and shortform threads or isn't on the forum at all. Joe Carlsmith's report on scheming gives an excellent overview of the arguments but is 127 pages long. This post is my attempt to give a relatively short overview of the most important arguments on deceptive alignment through a dialogue between Skeptic and Advocate, fictional characters on the two sides of the debate.[1]

Definitions and scope

Skeptic: "First, let's settle on a shared definition. When I speak of deceptive alignment, I mean the specific failure mode described in Risks from Learned Optimization: an ML model is within a training process and reasons that it should emit particular outputs to avoid the training process modifying it in an undesired way. I do NOT talk about deception in general, where an AI is interacting with humans for any reason and knowingly provides the human with false or misleading information. In other words, Deceptive AI ≠ Deceptively-aligned AI.

Second, let's define the bounds of the argument. When I say I assign a probability of <1% to deceptive alignment occurring, I mean a probability of <1% before human obsolescence. I'm not claiming that our future AI successors won't be worrying about deception when they're building literal galaxy brains that are nothing like today's LLMs. Do we agree on both of these points?"

Advocate: "Yep, sounds good."

Skeptic: "Great. Given this definition and scope, I claim that the likelihood of deceptive alignment occurring is somewhere between 0.1% and 1%. What's your estimate?"

Advocate: "I would put it at 40%."

How might deceptive alignment arise?

Skeptic: "This is a bold claim, given the lack of empirical evidence we have seen for deceptive alignment. Where does this number come from?"

Advocate: "Well, let's start by stating the obvious. When subject to selection pressure, humans often misrepresent their true motivations to get ahead. The easiest way to find deceptively aligned humans is to look at job seekers or politicians. This is a very basic fact about human cognition and already sufficient to consider the hypothesis as...
[Paper] Hidden in Plain Text: Emergence and Mitigation of Steganographic Collusion in LLMs
by Yohan Mathew — 2024-09-25T14:52:48.263Z
Link: https://arxiv.org/abs/2410.03768

This research was completed for London AI Safety Research (LASR) Labs 2024 by Yohan Mathew, Ollie Matthews, Robert McCarthy and Joan Velja. The team was supervised by Nandi Schoots and Dylan Cope (King’s College London, Imperial College London). Find out more about the programme and express interest in upcoming iterations here.

The full paper can be found here, while a short video presentation covering the highlights of the paper is here (note that some graphs have been updated since the presentation).

Introduction

Collusion in multi-agent systems is defined as 2 or more agents covertly coordinating to the disadvantage of other agents [6], while steganography is the practice of concealing information within a message while avoiding detection. In future multi-AI systems there is a risk that due to misspecified incentives, LLMs will learn to collude via the use of text-based steganography and bypass human oversight [1].

Concrete settings where steganography has previously been argued to present a risk include:

Models subtly conveying demographic biases present in their training data [1];
Powerful models using encoded reasoning (where the LLM encodes intermediate steps of misaligned reasoning in the generated text in a way that is not understandable to human readers) to generate sycophantic responses [2, 4];
Settings where an “untrusted” AI is monitoring the outputs of another “untrusted” AI, and they collude via the use of steganography to bypass oversight [3].

These concerns motivate the need for research into steganography. Our paper investigates the possibility of steganography ‘naturally‘ emerging in LLMs, as well as the robustness of current steganography mitigation techniques. We outline the contributions of the paper below.

1 - We Present Methods for Eliciting Steganography in LLMs

We devise some optimization methods to automatically elicit steganography in LLMs - a gradient-based reinforcement learning (GBRL) method, and two variants of an In-context Reinforcement Learning (ICRL) method. Our GBRL method uses Proximal Policy Optimization (PPO) [5] to perform weight updates, while our ICRL method uses iterative in-context learning without weight updates. These methods can serve to:

Evaluate the steganographic capabilities of LLMs,
Act as proxies for optimization pressures that may inadvertently encourage steganogra

...
AIS Hungary Operations Officer role, Deadline: 2024 October 6th
by gergogaspar — 2024-09-25T13:54:25.077Z

Apply & read more here.

Please note that EA Hungary is also hiring for an Operations Officer role. If you are interested in both roles, please apply to that role as well here.
(If you are interested in both, we recommend making the two applications in parallel, as some questions overlap.)