trentmkelly/LessWrong-43k huggingface dataset:
Training AI agents to solve hard problems could lead to Scheming
by Marius Hobbhahn — 2024-11-19T00:10:55.522Z

TLDR: We want to describe a concrete and plausible story for how AI models could become schemers. We aim to base this story on what seems like a plausible continuation of the current paradigm. Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed, self-guided, outcome-based, online learning procedure, which we call the “science loop”, where the AI makes incremental progress toward its high-level goal. We think this “science loop” encourages goal-directedness, instrumental reasoning, instrumental goals, beyond-episode goals, operational non-myopia, and indifference to stated preferences, which we jointly call “Consequentialism”. We then argue that consequentialist agents that are situationally aware are likely to become schemers (absent countermeasures) and sketch three concrete example scenarios. We are uncertain about how hard it is to stop such agents from scheming. We can both imagine worlds where preventing scheming is incredibly difficult and worlds where simple techniques are sufficient. Finally, we provide concrete research questions that would allow us to gather more empirical evidence on scheming.
We’re not confident in the core arguments and stories of this post. You can see it more like a best guess attempt after thinking about this for a few months on the side. Our primary goal is to start having more discussions about how catastrophic scheming could realistically develop.
This is a personal post and may diverge from the opinions of other members of Apollo. We would like to thank Mikita Balesni for detailed feedback.

Introduction

We think it’s plausible that AIs that are substantially more capable than human experts at a large range of long-term, economically valuable tasks could exist in the next

ten years

. Such AIs could be utopian for humanity, or they could be catastrophically bad. One of the main factors for whether we expect the future to be good or bad is whether these AI systems end up aligned or whether they are scheming, i.e. appear aligned for instrumental reasons while being secretly misaligned. In this post, we want to outline what we think is a plausible story of how an AI system could become a schemer in a realistic setting.

For this post, we will define scheming as:

AI systems covertly and strategically pursuing misaligned goals

A schemer acts ...
Proactive 'If-Then' Safety Cases
by Nathan Helm-Burger — 2024-11-18T21:16:37.237Z

Holden proposed the idea of if-then planning for AI safety. I think this is potentially a very good idea, depending on the implementation details.

I've heard criticisms of the If-Then style of planning that it is inherently reactive, rather than proactive. I think this is not necessarily true, and want to make a case for the value of proactive if-then planning.

Reactive

First, let's look at reactive if-then planning. I'll lay out reactive versions of three critical questions facing us about our near term future about which we might want to do if-then planning.

Reactive triggers:

- AI Biorisk: If we see empirical proof of substantial harm resulting from bioweapons which provably required AI-assistance to be created, then we put governance in place to address this risk going forwards. We will seek to reduce the likelihood of this harm happening again.

- Rapid recursive self-improvement: If we see empirical proof that some group was able to implement an AI system which provably self-improved more rapidly than if the group had directly worked on it, and the result of this self-improvement was an advance upon SotA frontier algorithms/models, then we will put governance in place to regulate such processes. We will seek to prevent any such processes occurring outside of carefully secured sandbox environments with government oversight.

- Unauthorized autonomous replication: if we see empirical proof that an AI system not under the control of a human or group is self-sustainingly persisting in an uncontained manner (e.g. has access to the internet or robotic actuators) and this AI system has caused significant harms in the world (e.g. breaking laws in pursuit of resources/survival), then we will put governance in place to empower enforcement agencies to halt this AI system and prevent the existence of such systems in the future.

On the plus side, I think these reactive if-then cases would be a really easy sell for most people and most governments. On the downside, the reactive nature of the chosen triggers means that we wait until harm has occurred before taking action. In all three of these cases, I argue that there is a substantial probability that the first case of such a harm would be an unacceptably large catastrophe.

Proactive

Now let's consider what proactive if-then planning might look like for each of these three cases. The co...
Will Orion/Gemini 2/Llama-4 outperform o1
by LuigiPagani — 2024-11-18T21:15:55.953Z

What's your bet on the next frontier models (Orion, Gemini 2, Llama-4) vs o1 in coding, math and logical reasoning benchmarks?

Will it have:

Better performance
Similar performance
Worse performance

Curious to hear your answers...

For OpenAI the question is if the increase in size and training on synthetic data will beat the teaching model, without test time compute.

In the comments there is some clarifications related to what I intend for "next-frontier" models.
How to use bright light to improve your life. 
by Nat Martin — 2024-11-18T19:32:10.667Z

You may have heard that you 'shouldn't use screens late in the evening' and maybe even that 'it's good for you to get exposure to sunshine as soon as possible after waking'. For the majority of people, these are generally beneficial heuristics. They are also the extent of most people's knowledge about how light affects their wellbeing.

The multiple mechanisms through which light affects our physiology make it hard to provide generalisable guidance. Among other things, the time of day, your genetics, your age, your mood and the brightness, frequency and duration of exposure to light all interrelate in determining how it affects us.

This document will explain some of the basic mechanisms through which light affects our physiology, with the goal of providing a framework to enable you to make informed decisions around your light exposure. After reading this, at any time on any given day, you should have a sense as to what type of light exposure you need right now. These decisions should lead to noticeable improvements in mood and productivity, whilst also improving sleep and reducing the risk of various long-term diseases.

Addressing SAD

Although SAD (Seasonal Affective Disorder) is a common framing used when describing the effect of light on health, I am going to largely avoid using the term here. Let me explain why...

Officially, SAD is a form of Major Depressive Disorder that comes and goes with seasonal patterns. Typically, this is characterised by depressive symptoms that occur in autumn and winter and resolve in spring and summer. [Confusingly, technically people who find themselves experiencing depression only in Summer also fall under the diagnosis of SAD].

One reason SAD is a challenging category is that in common parlance and in pop science, it is used to describe people with broader and milder symptoms. Indeed, one survey carried out by the reputable UK polling company YouGov declared that 29% of people in the UK are suffering from SAD[1]. This definitional ambiguity leads to a lot of confusion in the debate around SAD.

The second challenge is that research suggests that mechanisms underlying SAD vary between individuals [2]. With genetic, hormonal changes, disruptions in circadian rhythms and environmental influences all having been implicated, some question whether it is a conceptually useful grouping of independent conditions.

The only thing that is t...
Social events with plausible deniability
by Chris Lakin — 2024-11-18T18:25:17.339Z
Link: https://chrislakin.blog/p/plausible-deniability

We wanted to run an event where controversial opinions could be shared without restraint: “a safe space for what you can't say in ‘safe spaces’”

So a mechanism was devised:

Examples of what may or may have been on the slips:

“Only white men can save us.”
“Intelligent people are worth more.”
“Bestiality is good, actually.”

We also advertised the event so only a certain kind of person showed up:

One attendee said:

It was life-giving to be in an environment where I could say what I think without second guessing myself

Has anyone else run social events with plausible deniability?

[To be invited my events in SF, Hit “Subscribe” on this page.]
How likely is brain preservation to work?
by Andy_McKenzie — 2024-11-18T16:58:54.632Z

People often ask me this. It’s a good question. It seems potentially important for anyone who doesn’t want to die, or doesn’t want other people to have to die. I don’t feel like I have a good answer, which is one reason I want to do more research. But recently I think my answer has gotten a little better and I want to explain why.

A superficially similar question, “Does brain preservation work?” is also often asked, and I can see why people ask this, too — but it’s a question that can only be answered in the future.

Brain preservation is not suspended animation

. It’s about attempting to preserve the molecular structure of the brain — the physical substrate that encodes our memories and identity — in hopes that future technology might be able to extract and use that information. So people who say “brain preservation [or cryonics] doesn't work” are either confused or making a very strong claim about neuroscience (that molecular structure doesn’t encode identity/memories) and/or future technology (that we can know with certainty what future technology won’t be able to do), which as far as I know is not justified.

The more incisive question is not “Does it work?” but rather “What is the probability that it will work, for any given person?” This, in turn, can be

broken down

into several more specific probabilities. Here is one way to do so:

1: The probability that the preservation procedure in the ideal case retains sufficient information for survival, including the structures encoding long-term memories and personality.

2: The probability that the way in which any one particular case differs from the ideal is not so damaging as to destroy the information (e.g., it not taking too long from legal death until the preservation procedure begins, and the initial preservation procedure being performed in a sufficiently high-quality manner).

3: The probability that the preserved brain remains preserved long-term (e.g. sufficient organizational robustness, legal protections, and maintenance performed).

4: The probability that civilization survives for a long enough time (i.e. no extinction events, and technological civilization being maintained to a sufficient degree). Obviously, people’s opinions wildly differ here.

5: The probability that civilization develops revival technology (i.e. there is a sufficient degree of advancement over time in e.g. molecular nanotechnology...
Why imperfect adversarial robustness doesn't doom AI control
by Buck — 2024-11-18T16:05:06.763Z

(thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments)

Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would definitely help with control, but I think this argument misses several important points; my overall take is that current levels of adversarial robustness don’t come close to dooming AI control.

The first and most important point is that adversarial robustness is normally researched in a setting that’s much easier for the adversary. In most discussions, if an adversary can ever get a model to do something bad—even after many attempts and with a low success rate—this counts as a failure. But in real control settings, an AI that tries and fails to jailbreak might get caught, which is bad for its chances of success.

In addition, the AI might not even be able to get direct evidence of whether its attempts worked, or what its attempts even were. As a concrete example, consider a simple control setup we call "defer-to-trusted" (see here for more discussion):

Generate an action using the untrusted model
Get a suspiciousness score from the trusted model
If the score is in the top 1% of suspiciousness, audit the action (e.g. get a human contractor to investigate whether it was an attempt at causing a safety failure; if so, shut down the deployment, otherwise let the action execute and continue the execution of the agent).
If the score is in the top 10% of suspiciousness, replace the action with one from the trusted model.

(When I say "trusted model", I just mean one we're sure isn't scheming; the simplest argument for trusting a model is that it’s too incompetent to scheme effectively.)

In this setup, suppose the model tries to generate a jailbreak.

If that action is replaced, the untrusted model will be asked to take another action
It can probably tell when an action was generated by the trusted model, and so will know that a replacement happened. But it won’t know what it actually tried, unless it is able to guess very accurately what it would have said. This might make developing successful attacks much harder. (It’s like the inverse of the “use a time loop to iterate on strategies to manipulate som

...
Ethical Implications of the Quantum Multiverse
by Jonah Wilberg — 2024-11-18T16:00:20.645Z

What kinds of ethical implications should we expect from the Many-Worlds Interpretation of Quantum Mechanics (MWI)? I’ll argue that we shouldn’t expect decision-making to change. The implications are more about how we should think or feel about events in our lives, and the virtues of taking a cosmic perspective.

According the

many-worlds interpretation (MWI)

of quantum mechanics, the universe is constantly splitting into a staggeringly large number of decoherent branches containing galaxies, civilizations, and people exactly like you and me[1].

You might think such a metaphysically radical theory should have pretty radical implications for how one should live. The quantum physicist John Bell once wrote “if such a theory were taken seriously, it would hardly be possible to take anything else seriously”.

But proponents of MWI - including both Eliezer and David Wallace - have concluded the opposite. Eliezer

quotes

Egan's Law: It all adds up to normality. There are no major ethical implications at all[2].

I’ll argue that it's important to distinguish between two kinds of ethical implications we might expect MWI to have, which I’ll call ‘decision-theoretic’ and ‘virtue-theoretic’ (I’ll explain what I mean by these names).

Eliezer and Wallace are right that MWI doesn’t have decision-theoretic implications. But they overlook the fact that MWI plausibly has implications for virtue theory.

Decision theory in the multiverse

The main reason for thinking MWI has no significant implications for ethics comes from decision theory.

In standard decision theory you try to calculate the expected value for a particular course of action by multiplying the utility of each possible consequence by the probability of that consequence occurring.

The main difference in MWI is that each possible consequence corresponds to a world that actually ends up existing, rather than being just hypothetical.

But this makes no difference to the expected value that the calculation spits out. As Eliezer puts it:

“Your decision theory should (almost always) be the same, whether you suppose that there is a 90% probability of something happening, or if it will happen in 9 out of 10 worlds. Now, because people have trouble handling probabilities, it may be helpful to visualize something happening in 9 out of 10 worlds. But this just helps you use normal decision theory.”

So it...
Reducing x-risk might be actively harmful
by MountainPath — 2024-11-18T14:25:07.127Z

Great. Another crucial consideration I missed. I was convinced that working on reducing the existential risk for humanity should be a global priority.

Upholding our potential and ensuring that we can create a truly just future seems so wonderful.

Well, recently I was introduced to the idea that this might actually not be the case.

The argument is rooted in suffering-focused ethics and the concept of complex cluelessness. If we step back and think critically though, what predicts suffering more than the mere existence of sentient beings—humans in particular? Our history is littered with pain and exploitation: factory farming, systemic injustices, and wars, to name just a few examples. Even with our best intentions, humanity has perpetuated vast amounts of suffering.

So here’s the kicker: what if reducing existential risks isn’t inherently good? What if keeping humanity alive and flourishing actually risks spreading suffering further and faster—through advanced technologies, colonization of space, or systems we can’t yet foresee? And what if our very efforts to safeguard the future have unintended consequences that exacerbate suffering in ways we can't predict?

I was also struck by the critique of the “time of perils” assumption. The idea that now is a uniquely critical juncture in history, where we can reduce existential risks significantly and set humanity on a stable trajectory, sounds compelling. But the evidence supporting this claim is shaky at best. Why should we believe that reducing risks now will have lasting, positive effects over millennia—or even that we can reduce these risks at all, given the vast uncertainties?

This isn’t to say existential risk reduction is definitively bad—just that our confidence in it being good might be misplaced. A truly suffering-focused view might lean toward seeing existential risk reduction as neutral at best, and possibly harmful at worst.

It’s humbling, honestly. And frustrating. Because I want to believe that by focusing on existential risks, we’re steering humanity toward a better future. But the more I dig, the more I realize how little we truly understand about the long-term consequences of our actions.

So, what now? I’m not sure.

I am sick of missing crucial considerations. All I want to do is to make a positive impact. But no. Radical uncertainty it is.

I know that this will potentially cost me hundreds of h...
Monthly Roundup #24: November 2024
by Zvi — 2024-11-18T13:20:06.086Z

This is your monthly roundup. Let’s get right to it.

Young People are Young and Stupid

As a reminder that yes college students are often young and stupid and wrong about everything, remember the time they were behind a ban on paid public toilets? This is a central case of the kind of logic that often gets applied by college students.

No One Voted for This

HR and Title IX training seems like it’s going a lot of compelled speech in the form of ‘agree with us or you can’t complete your training and the training is required for your job,’ and also a lot of that compelled speech is outright lying because it’s confirmation of statements that are universally recognized to be insane?

Robin Hanson: Scenario: 2 women talking. X, married to woman, announces is pregnant. Y asks how they got pregnant, was it friend, donor, or IVF? 3rd person overhears, wonders if they should immediately intervene in convo to tell Y they are discriminating. Should they?

Context: This is example given in my workplace harassment/discrimination training, & one can’t move on unless one agrees that 3rd person should intervene.

My training says “Those questions are a little invasive!”

Training by Vector Solutions.

I do realize Robin’s followers can be odd, but yeah, not this time, and this is 87-1.

They also forced people to affirm the ‘affirmative specific consent’ rule, which voters disapproved of by 11-1.

Discrimination

Hard to pronounce names constitute 10%-50% of ethnic penalties among economics PhD job candidates, says new AEJ piece.

Qi Ge and Stephen Wu: The results are primarily driven by candidates with weaker résumés, suggesting that cognitive biases may contribute to the penalty of having a difficult-to-pronounce name.

Given this was not a controlled experiment, I’d ask if choosing an unpronounceable name is correlated to other parental characteristics that matter here.

The good news is you can solve for this – you can change your name.

A paper via MR says that across seven studies ‘attractiveness discrimination’ goes undetected because people lack the ability to do so, not because they think it is fine, and warn that interventions to increase salience of the issue would likely decrease detection of gender and race discrimination.

My read is that the people saying they disapprove of this type of discrimination are mostly lying, or at least answe...
A Straightforward Explanation of the Good Regulator Theorem
by Alfred Harwood — 2024-11-18T12:45:48.568Z

This post was written during the agent foundations fellowship with Alex Altair funded by the LTFF. Thanks to Alex, Jose, Daniel, Cole, and Einar for reading and commenting on a draft.

The Good Regulator Theorem, as published by Conant and Ashby in their 1970 paper (cited over 1700 times!) claims to show that 'every good regulator of a system must be a model of that system', though it is a subject of debate as to whether this is actually what the paper shows. It is a fairly simple mathematical result which is worth knowing about for people who care about agent foundations and selection theorems. You might have heard about the Good Regulator Theorem in the context of John Wentworth's 'Gooder Regulator' theorem and his other improvements on the result.

Unfortunately, the original 1970 paper is notoriously unfriendly to readers. It makes misleading claims, doesn't clearly state what exactly it shows and uses strange non-standard notation and cybernetics jargon ('coenetic variables' anyone?). If you want to understand the theorem without reading the paper, there are a few options. John Wentworth's post has a nice high-level summary but refers to the original paper for the proof. John Baez's blogpost is quite good but is very much written in the spirit of trying to work out what the paper is saying, rather than explaining it intuitively. I couldn't find an explanation in any control theory textbooks (admittedly my search was not exhaustive). A five year-old stackexchange question, asking for a rigorous proof, goes unanswered. The best explainer I could find was Daniel L. Scholten's 'A Primer for Conant and Ashby's Good-Regulator Theorem' from the mysterious, now-defunct 'GoodRegulatorProject.org' (link to archived website). This primer is nice, but really verbose (44 pages!). It is also aimed at approximately high-school (?) level, spending the first 15 pages explaining the concept of 'mappings' and conditional probability.

Partly to test my understanding of the theorem and partly to attempt to fill this gap in the market for a medium-length, entry-level explainer of the original Good Regulator Theorem, I decided to write this post.

Despite all the criticism, the actual result is pretty neat and the math is not complicated. If you have a very basic familiarity with Shannon entropy and conditional probability, you should be able to understand the Good Regulator Theorem...