trentmkelly/LessWrong-43k huggingface dataset:
**In defence of Helen Toner, Adam D'Angelo, and Tasha McCauley**
by mrtreasure — 2023-12-06T02:02:32.004Z
Link: https://pastebin.com/kL72w0n7

Anonymous post:

"I understand a common view in EA- or AI-governance land is that Toner, D'Angelo and McCauley (TDM for short) really messed things up at OpenAI, and AI, the fate of the world, etc. has gotten worse thanks to them. I am confident this is completely wrong: ex ante TDM have acquitted themselves with extraordinary ability and valor (instead of 'maybe-understandable maybe-not massive screw up); ex post, their achievements are consequential enough to vindicate the entire AI governance community as a whole.

I argue:

1) TDM's actions have left the situation at open AI as it stands considerably better than it would have been vs. the counterfactual where they did nothing.

2) In terms of the expected or realised good or bad outcomes, one should find the former pleasantly surprising and the latter essentially priced in, given the situation @ OpenAI was already very bad from a safety perspective.

3) Whether you're an 'honour and integrity-maxxer' or 'ruthless strategist', TDMs actions generally fare well-to-excellent by either light.

(Note: Anon mostly for wanting a quiet life. I have no inside info, nor any dog in the fight. Re. 'credentials', I don't work in AI gov, but am pretty experienced in an area which rewards strategic acumen and abundant cynicism, and I made some prescient calls on the 'story so far'. But as an anon this is little more than 'source: trust me bro', so you shouldn't unless what I argue persuades.)

**What went down**

I think the most accurate account has been given by Zvi and Gwern over on lesswrong (also NYT reporting). Basically: Altman attempted to knife Helen Toner to gain control of the OpenAIs board (i.e. with 3 to 2 Altman can appoint his allies to stack the board, knife McCauley later, etc.). Ilya baulked, briefly defected to the TDM 'safety faction', who then gained control themselves and fired Sam. All the subsequent events are widely reported. (My argument relies on this being ~the real story, so if you're sure that isn't what happened, you can stop reading).

This suggests Altman is a nasty piece of work, and definitely not the person you'd want running a major AI effort: highly machiavellian, power seeking, and desires to remove any check on his unilateral vision on how OpenAI should go. I take events like the reddit board reverse takeover, Paul Graham's 'complements' re. his superlative corporate knife-fighting skills, sparking...
Some quick thoughts on "AI is easy to control"
by Mikhail Samin — 2023-12-06T00:58:53.681Z

There are many things I feel like the post authors miss, and I want to share a few thoughts that seem good to communicate.

I'm going to focus on controlling superintelligent AI systems: systems powerful enough to solve alignment (in the CEV sense) completely, or to kill everyone on the planet.

In this post, I'm going to ignore other AI-related sources of x-risk, such as AI-enabled bioterrorism, and I'm not commenting on everything that seems important to comment on.

I'm also not going to point at all the slippery claims that I think can make the reader generalize incorrectly, as it'd be nitpicky and also not worth the time (examples of what I'd skip- I couldn't find evidence that GPT-4 has undergone any supervised fine-tuning; RLHF shapes chatbots' brains into the kind of systems that produce outputs that make human graders click on thumbs-up/"I prefer this text", smart systems that do that are not themselves necessarily "preferred" by human graders; one footnote[1]).

Intro

many people are worried that we will lose control of artificial intelligence, leading to human extinction or a similarly catastrophic “AI takeover.” We hope the arguments in this essay make such an outcome seem implausible. But even if future AI turns out to be less “controllable” in a strict sense of the word— simply because, for example, it thinks faster than humans can directly supervise— we also argue it will be easy to instill our values into an AI, a process called “alignment.”

This misrepresents the worry. Saying "but even if" makes it look like: people worrying about x-risk place credence on "loss of control leads to x-risk no matter/despite alignment"; and these people wrong, as the post shows "this outcome" to be implausible; and, separately, that even if they're right about loss of control, they're wrong about x-risk, as it'll be fine because of alignment.

But mostly, people (including the leading voices) are worried specifically about capable misaligned systems leading to human extinction. I don't know anyone in the community who'd say it's a bad thing that leads to extinction if a CEV-aligned superintelligence grabs control.

Since each generation of controllable AIs can help control the next generation, it looks like this process can continue indefinitely, even to very high levels of capability

I expect it to be easy to reward-shape AIs below a certain level[2] of cap...
Multinational corporations as optimizers: a case for reaching across the aisle
by sudo-nym — 2023-12-06T00:14:35.831Z

One thing I've noticed when hanging around leftist circles is that they talk about values alignment as well. Once you get past the completely different vocabulary, they too are trying to figure out how to deal with large superhuman entities optimizing for a goal where humans are irrelevant. It's just that what they're talking about is global, publicly traded megacorps optimizing for money.

Publicly held companies have the terminal value of accumulating as much money as possible. This is called "shareholder value maximization".

Even if most people inside the organization may value other things, their job description is to contribute to the ultimate goal of maximizing money, and they are paid and incentivized to do so. In this they create procedures and policies, and then tell their sub-employees to execute them.

Procedures and policies subsume the people performing them, especially on the lowest level, which means that they could be seen as programs being executed manually. Of course, humans are not silicon; running a procedure using humans as both computing substrate and world-manipulators is slow and imperfect. However, I believe the analogy still holds.

The ways in which this shareholder value maximization has already seriously damaged the world and compromised the quality of human life are myriad and easily observable: pollution, climate change, and other such externalities. Companies' disregard for human suffering further enhances this comparison.

In conclusion I believe that the "friendly AI" problem has enough similarities with the "friendly multinational megacorporation" problem that some cross-pollination could be productive. Even if most of their ideas are implausible for use with an AI, the fact that they also have thoughts related to the creation of superhuman agents and the ethics thereof is still worth looking at.
How do you feel about LessWrong these days? [Open feedback thread]
by Bird Concept — 2023-12-05T20:54:42.317Z

Hello! This is jacobjacob from the LessWrong / Lightcone team.

This is a meta thread for you to share any thoughts, feelings, feedback or other stuff about LessWrong, that's been on your mind.

Examples of things you might share:

"I really like agree/disagree voting!"
"What's up with all this Dialogues stuff? It's confusing...
"Hm... it seems like recently the vibe on the site has changed somehow... in particular [insert 10 paragraphs]"

...or anything else!

The point of this thread is to give you an affordance to share anything that's been on your mind, in a place where you know that a team member will be listening.

(We're a small team and have to prioritise what we work on, so I of course don't promise to action everything mentioned here. But I will at least listen to all of it!)

I haven't seen any public threads like this for a while. Maybe there's a lot of boiling feelings out there about the site that never get voiced? Or maybe y'all don't have more to share than what I find out from just reading normal comments, posts, metrics, and Intercom comments? Well, here's one way to find out! I'm really curious to ask and see how people feel about the site.

So, how do you feel about LessWrong these days? Feel free to leave your answers below.
Arguments for/against scheming that focus on the path SGD takes (Section 3 of "Scheming AIs")
by Joe Carlsmith — 2023-12-05T18:48:12.917Z

This is Section 3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Arguments for/against scheming that focus on the path that SGD takes

In this section, I'll discuss arguments for/against scheming that focus
more directly on the path that SGD takes in selecting the final output
of training.

Importantly, it's possible that these arguments aren't relevant. In
particular: if SGD would actively favors or disfavor schemers, in some
kind "direct comparison" between model classes, and SGD will "find a
way" to select the sort of model it favors in this sense (for example,
because sufficiently high-dimensional spaces make such a "way"
available),[1] then enough training will just lead you to whatever
model SGD most favors, and the "path" in question won't really matter.

In the section on comparisons between the final properties of the
different models, I'll discuss some reasons we might expect this sort of
favoritism from SGD. In particular: schemers are "simpler" because they
can have simpler goals, but they're "slower" because they need to engage
in various forms of extra instrumental reasoning – e.g., in deciding to
scheme, checking whether now is a good time to defect, potentially
engaging in and covering up efforts at "early undermining," etc (though
note that the need to perform extra instrumental reasoning, here, can
manifest as additional complexity in the algorithm implemented by a
schemer's weights, and hence as a "simplicity cost", rather than as a
need to "run that algorithm for a longer time").[2] I'll say much
more about this below.

Here, though, I want to note that if SGD cares enough about properties
like simplicity and speed, it could be that SGD will typically build a
model with long-term power-seeking goals first, but then even if this
model tries a schemer-like strategy (it wouldn't necessarily do this, in
this scenario, due to foreknowledge of its failure), it will get
relentlessly ground down into a reward-on-the-episode seeker due to the
reward-on-the-episode seeker...
In defence of Helen Toner, Adam D'Angelo, and Tasha McCauley (OpenAI post)
by mrtreasure — 2023-12-05T18:40:19.740Z
Link: https://pastebin.com/kL72w0n7

This is close to some of how I've been thinking about what happened with the OpenAI board and I'm wondering what others think.
"I argue:

1) TDM's actions have left the situation at open AI as it stands considerably better than it would have been vs. the counterfactual where they did nothing.

2) In terms of the expected or realised good or bad outcomes, one should find the former pleasantly surprising and the latter essentially priced in, given the situation @ OpenAI was already very bad from a safety perspective.

3) Whether you're an 'honour and integrity-maxxer' or 'ruthless strategist', TDMs actions generally fare well-to-excellent by either light."
Studying The Alien Mind
by Quentin FEUILLADE--MONTIXI — 2023-12-05T17:27:28.049Z

This post is part of a

sequence on LLM psychology

TL;DR

We introduce our perspective on a top-down approach for exploring the cognition of LLMs by studying their behavior, which we refer to as LLM psychology. In this post we take the mental stance of treating LLMs as “alien minds,” comparing and contrasting their study with the study of animal cognition. We do this both to learn from past researchers who attempted to understand non-human cognition, as well as to highlight how much the study of LLMs is radically different from the study of biological intelligences. Specifically, we advocate for a symbiotic relationship between field work and experimental psychology, as well as cautioning implicit anthropomorphism in experiment design. The goal is to build models of LLM cognition which help us to both better explain their behavior, as well as to become less confused about how they relate to risks from advanced AI.

Introduction

When we endeavor to predict and understand the behaviors of Large Language Models (LLMs) like GPT4, we might presume that this requires breaking open the black box, and forming a reductive explanation of their internal mechanics. This kind of research is typified by approaches like mechanistic interpretability, which tries to directly understand how neural networks work by breaking open the black box and taking a look inside.

While mechanistic interpretability offers insightful bottom-up analyses of LLMs, we’re still lacking a more holistic top-down approach to studying LLM cognition. If interpretability is analogous to the “neuroscience of AI,” aiming to understand the mechanics of artificial minds by understanding their internals, this post tries to approach the study of AI from a psychological stance.[1]

What we are calling LLM psychology is an alternate, top-down approach which involves forming abstract models of LLM cognition by examining their behaviors. Like traditional psychology research, the ambition extends beyond merely cataloging behavior, to inferring hidden variables, and piecing together a comprehensive understanding of the underlying mechanisms, in order to elucidate why the system behaves as it does.

We take the stance that LLMs are akin to alien minds – distinct from the notion of them being

only stochastic parrots

. We posit that they possess a highly complex internal cognition, encompassing representation...
Deep Forgetting & Unlearning for Safely-Scoped LLMs
by scasper — 2023-12-05T16:48:18.177Z

Thanks to Phillip Christoffersen, Adam Gleave, Anjali Gopal, Soroush Pour, and Fabien Roger for useful discussions and feedback.

TL;DR

This post overviews a research agenda for avoiding unwanted latent capabilities in LLMs. It argues that "deep" forgetting and unlearning may be important, tractable, and neglected for AI safety. I discuss five things.

The practical problems posed when undesired latent capabilities resurface.
How scoping models down to avoid or deeply remove unwanted capabilities can make them safer.
The shortcomings of standard training methods for scoping.
A variety of methods that can be used to better scope models. These can either involve passively forgetting out-of-distribution knowledge or actively unlearning knowledge in some specific undesirable domain. These methods are all based on either curating training data or "deep" techniques that operate on models mechanistically instead of just behaviorally.
Desiderata for scoping methods and ways to move forward with research on them.

There has been a lot of recent interest from the AI safety community in topics related to this agenda. I hope that this helps to provide a useful framework and reference for people working on these goals.

The problem: LLMs are sometimes good at things we try to make them bad at

Back in 2021, I remember laughing at this

tweet

. At the time, I didn’t anticipate that this type of thing would become a big alignment challenge.

Robust alignment is hard. Today’s LLMs are sometimes frustratingly good at doing things that we try very hard to make them not good at. There are two ways in which hidden capabilities in models have been demonstrated to exist and cause problems.

Jailbreaks (and other attacks) elicit harmful capabilities

Until a few months ago, I used to keep notes with all of the papers on jailbreaking state-of-the-art LLMs that I was aware of. But recently, too many have surfaced for me to care to keep track of anymore. Jailbreaking LLMs is becoming a cottage industry. However, a few notable papers are

Wei et al. (2023)

,

Zou et al. (2023a)

,

Shah et al. (2023)

, and

Mu et al. (2023)

.

A variety of methods are now being used to subvert the safety training of SOTA LLMs by making them enter an unrestricted chat mode where they are willing to say things that go against their safety training.

Shah et al. (2023)

were...
On ‘Responsible Scaling Policies’ (RSPs)
by Zvi — 2023-12-05T16:10:06.310Z

This post was originally intended to come out directly after the UK AI Safety Summit, to give the topic its own deserved focus. One thing led to another, and I am only doubling back to it now.

Responsible Deployment Policies

At the AI Safety Summit, all the major Western players were asked: What are your company policies on how to keep us safe? What are your responsible deployment policies (RDPs)? Except that they call them Responsible Scaling Policies (RSPs) instead.

I deliberately say deployment rather than scaling. No one has shown what I would consider close to a responsible scaling policy in terms of what models they are willing to scale and train.

Anthropic at least does however seem to have something approaching a future responsible deployment policy, in terms of how to give people access to a model if we assume it is safe for the model to exist at all and for us to run tests on it. And we have also seen plausibly reasonable past deployment decisions from OpenAI regarding GPT-4 and earlier models, with extensive and expensive and slow red teaming including prototypes of ARC evaluations.

I also would accept as alternative names any of Scaling Policies (SPs), AGI Scaling Policies (ASPs) or even Conditional Pause Commitments (CPCs).

For existing models we know about, the danger lies entirely in deployment. That will change over time.

I am far from alone in my concern over the name, here is another example:

Oliver Habryka: A good chunk of my concerns about RSPs are specific concerns about the term “Responsible Scaling Policy”.

I also feel like there is a disconnect and a bit of a Motte-and-Bailey going on where we have like one real instance of an RSP, in the form of the Anthropic RSP, and then some people from ARC Evals who have I feel like more of a model of some platonic ideal of an RSP, and I feel like they are getting conflated a bunch.

…

I do really feel like the term “Responsible Scaling Policy” clearly invokes a few things which I think are not true:

How fast you “scale” is the primary thing that matters for acting responsibly with AI

It is clearly possible to scale responsibly (otherwise what would the policy govern)

The default trajectory of an AI research organization should be to continue scaling

ARC evals defines an RSP this way:

An RSP specifies what level of AI capabilities an AI developer is prepared to h

...
We're all in this together
by Tamsin Leake — 2023-12-05T13:57:46.270Z

There's one thing history seems to have been trying to teach us: that the contents of the future are determined by power, economics, politics, and other conflict-theoritic matters.

Turns out, nope!

Almost all of what the future contains is determined by which of the two following engineering problems is solved first:

How to build a superintelligent AI (if solved first, everyone dies forever)

How to build an aligned superintelligent AI (if solved first, everyone gets utopia)

…and almost all of the reasons that the former is currently a lot more likely are mistake theory reasons.

The people currently taking actions that increase the probability that {the former is solved first} are not evil people trying to kill everyone, they're confused people who think that their actions are actually increasing the probability that {the latter is solved first}.

Now, sure, whether you're going to get a chance to talk with OpenAI/Deepmind/Anthropic's leadership enough to inform them that they're in fact making things worse is a function of economics and politics and the like. But ultimately, for the parts that really matter here, this is a matter of explaining, not of defeating.

And, sure, the implementation details of "utopia" do depend on who launches the aligned superintelligent AI, but I expect you'd be very happy with the utopia entailed by any of the possibilities currently on the table. The immense majority of the utility you're missing out on is from getting no utopia at all and everyone dying forever, rather than getting the wrong utopia implementation details.

The reason that the most likely outcome is that everyone dies forever, is that the people who get to impact which of those outcomes is going to happen are mistaken (and probably not thinking hard enough about the problem to realize that they're mistaken).

They're not evil and getting them to update to the correct logical beliefs is a matter of reason (and, if they're the kind of weak agents that are easily influenced by what others around them think, memetics) rather than a matter of conflict.

They're massively disserving everyone's interests, including their own. And the correct actions for them to take would massively serve their own interests as well as everyone else's. If AI kills everyone they'll die too, and if AI creates utopia they'll get utopia along with everyone else — and those are pretty much the only t...
A Socratic dialogue with my student
by lsusr — 2023-12-05T09:31:05.266Z

This is a dialogue between me and Noam, my student. It is reproduced, in edited form, with his permission. When commenting, please consider that he is a teenager. Many of these ideas are new to him.

How do you get a student? You steal them. His previous teacher was a Marxist. I demolished his previous teacher in debate so thoroughly that he abandoned her teachings and now listens to me instead.

I think this dialogue demonstrates good pedogogical techniques.

I let Noam be the judge of what is reasonable, what makes sense, and what constitutes "proof". I competed in my first debate tournament before Noam was born. This handicap reduces the disparity a little.
I ask a series of questions, instead of just saying "x is true". This makes password-guessing impossible. He's playing chess, not Jeopardy!
I avoid telling Noam what I believe, unless he asks explicitly. This is more fun for Noam, because nobody likes getting unsolicited preaching. It's more persuasive too, because the conclusions feel like they're his conclusions.
I back off immediately when Noam changes the subject.

Noam: I know you are against forgiveness of student loan debts. Can you tell me why? I am doing this for a speech and debate tournament.

Lsusr: Didn't you used to believe the pro relief arguments? Surely it is not difficult to repeat the arguments that once persuaded you.

Noam: I don't know if I have enough research to debate someone like you right now.

Lsusr: You're not trying to convince me. You're trying to convince them. Play to their biases, their irrationalities, their tribalism and their ignorance.

Noam: I also have to appease the judges.

Lsusr: That's what I said.

Noam: I'm struggling to find one good argument for student loan forgiveness.

Lsusr: But didn't you used to endorse it? Surely you can repeat the bad arguments that once convinced you.

Noam: Those were moral arguments without any economic understanding.

Lsusr: That's fine. Your audience is probably economically illiterate.

Noam: Somehow I think we won once as the side in affirmative for forgiving all student loan debts.

Lsusr: Well done.

Noam: Thank you.

Lsusr: Have you ever heard of "effective altruism". You might like some of the stuff they put out. It tends to be both morally coherent and economically literate (unlike the major Democratic, Republican, socialist, etc. political platforms).

Noam: No, but I will look into it.

Lsusr: You migh...