trentmkelly/LessWrong-43k huggingface dataset:
AI #63: Introducing Alpha Fold 3
by Zvi — 2024-05-09T14:20:03.176Z

It was a remarkably quiet announcement. We now have Alpha Fold 3, it does a much improved job predicting all of life’s molecules and their interactions. It feels like everyone including me then shrugged and went back to thinking about other things. No cool new toy for most of us to personally play with, no existential risk impact, no big trades to make, ho hum.

But yes, when we look back at this week, I expect what we remember will be Alpha Fold 3.

Unless it turns out that it is Sophon, a Chinese technique to potentially make it harder to fine tune an open model in ways the developer wants to prevent. I do not expect this to get the job done that needs doing, but it is an intriguing proposal.

We also have 95 theses to evaluate in a distinct post, OpenAI sharing the first draft of their model spec, Apple making a world class anti-AI and anti-iPad ad that they released thinking it was a pro-iPad ad, more fun with the mysterious gpt2, and more.

The model spec from OpenAI seems worth pondering in detail, so I am going to deal with that on its own some time in the coming week.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Agents, simple and complex.

Language Models Don’t Offer Mundane Utility. No gadgets, no NPCs.

GPT-2 Soon to Tell. Does your current model suck? In some senses.

Fun With Image Generation. Why pick the LoRa yourself?

Deepfaketown and Botpocalypse Soon. It’s not exactly going great.

Automation Illustrated. A look inside perhaps the premiere slop mill.

They Took Our Jobs. Or are we pretending this to help the stock price?

Apple of Technically Not AI. Mistakes were made. All the feels.

Get Involved. Dan Hendrycks has a safety textbook and free online course.

Introducing. Alpha Fold 3. Seems like a big deal.

In Other AI News. IBM, Meta and Microsoft in the model game.

Quiet Speculations. Can we all agree that a lot of intelligence matters a lot?

The Quest for Sane Regulation. Major labs fail to honor their commitments.

The Week in Audio. Jack Clark on Politico Tech.

Rhetorical Innovation. The good things in life are good.

Open Weights are Unsafe and Nothing Can Fix This. Unless, maybe? Hmm.

The Lighter Side. Mmm, garlic bread. It’s been too long.

Language Models Offer Mundane Utility

How much utility for how much cost? Kapoor and Narayanan argue that with the rise of agent-based systems, you have to evaluate different m...
I Got 95 Theses But a Glitch Ain’t One
by Zvi — 2024-05-09T14:10:02.677Z

Or rather Samuel Hammond does. Tyler Cowen finds it interesting but not his view.

I put up a market, and then started looking. Click through to his post for the theses. I will be quoting a few of them in full, but not most of them.

I am not trying to be exact with these probabilities when the question calls for them, nor am I being super careful to make them consistent, so errors and adjustments are inevitable.

Section 1 is Oversight of AGI labs is prudent.

I do tend to say that.

There are few things more important to U.S. national interest than close monitoring of frontier model capabilities, and also the ability to intervene.

Indeed, I believe one should be at best skeptical or ambivalent about most potential forms of regulation of anything, AI included. Yet I think the case for ‘oversight of the frontier labs’ is overwhelming.

Shout it from the rooftops: “As a temporary measure, using compute thresholds to pick out the AGI labs for safety-testing and disclosures is as light-touch and well-targeted as it gets.” It would be so helpful if more people understood this, and more others stopped pretending they did not understand it.

This as well. When you regulate ‘use’ or ‘risk’ you need to check on everyone’s ‘use’ of everything, and you make a lot of detailed micro interventions, and everyone has to file lots of paperwork and do lots of dumb things, and the natural end result is universal surveillance and a full ‘that which is not compulsory is forbidden’ regime across much of existence. Whereas a technology-focused approach can be entirely handled by the lab or manufacturer, then you are free.

Exactly. Compute is an imperfect proxy, but it is remarkably simple and robust. When it makes mistakes, they are false positives, where someone uses compute poorly and gets poor results. That is a small (measurement) mistake. Certainly compute is vastly better than all proposed alternative metrics.

It is highly reasonable to invoke the Defense Production Act regarding frontier AI as an actual bone fide national security situation where defense is a key concern. It is a far better justification than the median invocation of the act. The better reason to use the DPA is that it is currently the only mechanism available to the executive and our Congress is for now incapable of legislative action.

It does not require AGI or ASI to be near f

...
The Human's Role in Mesa Optimization
by silentbob — 2024-05-09T12:07:06.617Z

When it comes to mesa optimization, there’s usually two optimizers mentioned: the “base optimizer”, such as SGD, the process used for training the model. And the “mesa optimizer”, meaning roughly “the optimizer beneath the optimizer”. One key question in this context is when and under which circumstances such mesa optimizers may occur. This question is relevant for the AI safety field and alignment research, because optimizer AIs are of particular interest: firstly, they're potentially much more capable than non-optimizing AIs, and secondly, dangerous properties such as instrumental convergence may apply to them, leading to risks of e.g. runaway optimization that could escape human control.

In

Risks from Learned Optimization

, the authors

explain

:

Conventionally, the base optimizer in a machine learning setup is some sort of gradient descent process with the goal of creating a model designed to accomplish some specific task.

Sometimes, this process will also involve some degree of meta-optimization wherein a meta-optimizer is tasked with producing a base optimizer that is itself good at optimizing systems to achieve particular goals. Specifically, we will think of a meta-optimizer as any system whose task is optimization. For example, we might design a meta-learning system to help tune our gradient descent process. Though the model found by meta-optimization can be thought of as a kind of learned optimizer, it is not the form of learned optimization that we are interested in for this sequence. Rather, we are concerned with a different form of learned optimization which we call mesa-optimization.

This sounds pretty sensible to me, but I think it’s missing an important piece: the (usually human) AI engineers who are behind the whole training process. Even if there’s no automated meta optimizer, the humans are still there, basically performing the same task: they are selecting model architectures, tweaking hyperparameters, and running all kinds of experiments, usually in order to get the emerging AI to be as capable as possible. So ultimately we (potentially) end up with this state of affairs:

picture: https://simpleaisafety.org/en/posts/mesa-optimizers/

The meta optimizer (AI engineer) tweaks the base optimizer (SGD) so that it trains a mesa optimizer (model) to interact with the world once deployed.

Of course an AI engineer who highly values safe...
Visualizing neural network planning
by Nevan Wichers — 2024-05-09T06:40:46.582Z

TLDR

We develop a technique to try and detect if a NN is doing planning internally. We apply the decoder to the intermediate representations of the network to see if it’s representing the states it’s planning through internally. We successfully reveal intermediate states in a simple Game of Life model, but find no evidence of planning in an AlphaZero chess model. We think the idea won’t work in its current state for real world NNs because they use higher-level, abstract representations for planning that our current technique cannot decode. Please comment if you have ideas that may work for detecting more abstract ways the NN could be planning.

Idea and motivation

To make safe ML, it’s important to know if the network is performing mesa optimization, and if so, what optimization process it’s using. In this post, I'll focus on a particular form of mesa optimization: internal planning. This involves the model searching through possible future states and selecting the ones that best satisfy an internal goal. If the network is doing internal planning, then it’s important the goal it’s planning for is aligned with human values. An interpretability technique which could identify what states it’s searching through would be very useful for safety. If the NN is doing planning it might represent the states it’s considering in that plan. For example, if predicting the next move in chess, it may represent possible moves it’s considering in its hidden representations.

We assume that NN is given the representation of the environment as input and that the first layer of the NN encodes the information into a hidden representation. Then the network has hidden layers and finally a decoder to compute the final output. The encoder and decoder are trained as an autoencoder, so the decoder can reconstruct the environment state from the encoder output. Language models are an example of this where the encoder is the embedding lookup.

Our hypothesis is that the NN may use the same representation format for states it’s considering in its plan as it does for the encoder's output. Our idea is to apply the decoder to the hidden representations at different layers to decode them. If our hypothesis is correct, this will recover the states it considers in its plan. This is similar to the Logit Lens for LLMs, but we’re applying it here to investigate mesa-optimization.

A potential pitfa...
Forecasting: the way I think about it
by Molly — 2024-05-09T00:49:01.768Z
Link: https://cuttyshark.substack.com/p/forecasting-the-way-i-think-about

This is the first post in a little series I'm slowly writing on how I see forecasting, particularly conditional forecasting; what it's good for; and whether we should expect people to agree if they just talk to each other enough.

Views are my own. I work at the

Forecasting Research Institute

(FRI), I forecast with the Samotsvety group, and to the extent that I have formal training in this stuff, it's mostly from studying and collaborating with Leonard Smith, a chaos specialist.

My current plan is:

Forecasting: the way I think about it [this post]
The promise of conditional forecasting / cruxing for parameterizing our models of the world
What we're looking at and what we're paying attention to (Or: why we shouldn't expect people to agree today (Or: there is no "true" probability))

What do I do when I forecast? Let's say I'm forecasting an arbitrary bad outcome U that we're going to resolve in/by 2100 (e.g. AI-related catastrophe). I ask myself:

What are all the worlds I can imagine for 2100?
What's my P(U) in each of these possible worlds?
Take the value for P(U) that has half the probability mass below and half above – that may not be the

modal

world – it's the

median

world, and it's where my expected log score is maximized

Fig. 1

Imagining all the worlds is impossible, so I wind up decomposing the probability mass function into a few types of worlds and thinking about how being in each world would affect P(U) — i.e. for worlds A, B, C, … I have P(U|A), P(U|B), P(U|C) etc (Fig. 2). And I have ideas about how likely we are to wind up in each of A, B, C, etc. Here, B is my "modal world" and my "expectation" world is somewhere between C and D on the P(U) scale.

Fig. 2

If you want to get really fancy, you can factor in uncertainty about U in each of these worlds, treat them all as distributions (some are pointier, some are more uncertain), and think about your all-things-considered P(U) as a mixed distribution of all of your worlds. This can always be distilled into a point estimate by taking that center of mass (dotted line in Fig. 1). You can use tools like

squiggle

for this.

Side-note: I think some people just think about the modal world B by default. It's probably the first world you think of. It's the world you most think will come to pass. But you don't maximize your log score by forecasting P(U|B) when you're asked for P(U).

In our projects at F...
some thoughts on LessOnline
by Raemon — 2024-05-08T23:17:41.372Z

I mostly wrote this for facebook, but it ended up being a whole-ass post so I figured I'd put it here too.

I'm helping run "LessOnline: A Festival of Writers Who Are Wrong On the Internet (But Striving To Be Less So)".

I'm incentivized to say nice things about the event. So, grain of salt and all. But, some thoughts, which roughly breakdown into:

– The vibe: preserving cozy/spaciousness of a small retreat at a larger festival

– The audience: "Reunion for the The Extended Family Blogosphere, both readers and writers."

– Manifest, and Summer Camp

...

I. The Vibe

I've been trying to explain the vibe I expect and it's tricksy. I think the vibe will be something like "CFAR Reunion meets Manifest."

But a lot of people haven't been to a CFAR Reunion or to Manifest.

I might also describe it like "the thing the very first EA Summit (before EA Global) was like, before it became EA Global and got big." But very few people went to that either.

Basically: I think this will do a pretty decent job of having the feel of a smaller (~60 person), cozy retreat, but while being more like 200 - 400 people. Lightcone has run several ~60 person private retreats, which succeeded being a really spacious intellectual environment, with a pretty high hit rate for meeting new people who you might want to end up having a several hour conversation with.

Realistically, with a larger event there'll be at least some loss of "cozy/spaciousness", and a somewhat lower hit rate for people you want to talk to with the open invites. But, I think Lightcone has learned a lot about how to create a really nice vibe. We've built our venue, Lighthaven, with "warm, delightful, focused intellectual conversation" as a primary priority. Whiteboards everywhere, lots of nooks and a fractal layout that makes it often feel like you're in a seclude private conversation by a firepit, even though hundreds of other people are nearby (often at another secluded private conversation with _their_ own firepit!)

(It's sort of weird that this kind of venue is extremely rare. Many events are hotels, which feel vaguely stifling and corporate. And the nice spacious retreat centers we've used don't score well on the whiteboard front, and surprisingly not even that well on "lots of nooks")

...

Large events tend to use "Swap Card" for causing people to meet each other. I do find Swap Card really good for nailing down a lot of short meeti...
Zero-Sum Defeats Nash Equilibrium
by Richard Henage — 2024-05-08T19:49:02.798Z

Bike locks are a good example. There are twenty bikes at the bike rack in the parking lot. Some bike riders decide to buy bike locks for their bikes to protect them from theft. But this just means that the thieves steal other, unprotected bikes from the rack. The problem wasn't solved, just moved around. The wary bike riders acted in self-interest in a zero-sum game and the result was net 0, but the wary bike riders came out ahead. As long as some bikes remain unprotected, the crime rate will remain the same. All of the bike riders would have to go to the effort of buying bike locks before any progress is made in solving the problem. This would be a Nash Equilibrium...

However, if all the bike riders act in self-interest by selfishly protecting their own bikes at the expense of the others, they will all buy bike locks and the thieves will have no bikes to steal. So in some systems, a Zero-Sum environment actually defeats the Nash Equilibrium.

The same concept would apply easily to cybersecurity: hackers will go after the easiest target, and if everyone tries to not be the easiest target the global level of cyber defense will rise and hacking will become less common.

So if there's a seemingly Zero-Sum system or a seeming Nash Equilibrium, adding the other component might solve the problem. Can this be applied to other problems?

AI safety: Nash Equilibrium[1]. It's in everyone's selfish interest to ignore safe practices and try to make their AI as powerful as possible. But if everyone does this, it's likely that at least one AI will be both powerful enough and malicious enough to cause serious damage and make everyone lose. Is there a way to add a Zero-Sum component to this? Maybe a task force could be created that punishes the least-safe AI systems or rewards the most-safe, based not on their absolute safety level but on their safety level relative to other systems. This system could be a fine/grant or it could just be a label: "World's safest AI system as voted by experts". This is a Zero-Sum component because no matter how hard developers try, the same total number of rewards and punishments will be doled out.
Environmental damage: Nash Equilibrium. Since important parts of the environment are shared between countries and even the entire world, harm done to it is spread out among multiple countries. This means that a selfish country wouldn't place enough negativ

...
Semantic Disagreement of Sleeping Beauty Problem
by Ape in the coat — 2024-05-08T19:09:55.023Z

This is the tenth post in my series on Anthropics. The previous one is Beauty and the Bets.

Introduction

In my previous posts I've been talking about the actual object-level disagreement between halfers and thirders - which of the answers formally is correct and which is not. I've shown that there is one correct model for the Sleeping Beauty problem, that describes it instead of something else, successfully passes the statistical test, has sound mathematical properties and deals with every betting scheme.

But before we can conclude that the issue is fully resolved, there is still a notable semantic disagreement left, as well as several interesting questions. If the thirder answer isn't the correct "probability", then what is it? What are the properties of this entity? And why are people so eager to confuse it with the actual probability?

In this post I'm going to answer these questions and dissolve the last, purely semantic part of the disagreement.

A Limitation of Probability Theory

Probability theory has an curious limitation. It is unable to talk about events which happen multiple times during the same iteration of an experiment. Instead, when the conditions of the experiment state that something happens multiple times, we are supposed to define "happening multiple times" as an elementary outcome in itself.

This is, actually, a very useful property, that allows probability theory to work as intended, so that the measure of an event properly corresponds to the credence that rational agents are suppose to have in that event. But it may lead to conflicts with our intuitions about the matter.

Let's consider an example:

A coin is tossed. On Heads you are given a blue ball, on Tails you are given a blue ball and then you are given a red ball.

Here we say that there are two well-defined mutually exclusive outcomes. Exactly one of them happens at every iteration of the experiment.

Ω={Heads&Blue, Tails&Both}={Heads&Blue&!Red, Tails&Blue&Red}

We can go into a bit more details and define separate sample spaces for getting a blue ball and getting a red ball:

{Blue}

P(Blue)=1

{Red, !Red}

P(Red)=P(!Red)=1/2

We can coherently talk about the probability to get only one ball. Such event happens when the red ball isn't given

P(OnlyOne)=P(Heads&Blue)=P(!Red)=1/2

And the probability to get any ball at all. Such an event happens on every it...
Is There a Power Play Overhang?
by crispweed — 2024-05-08T17:39:47.821Z
Link: https://upcoder.com/21/is-there-a-power-play-overhang

This post is about risks in the development of increasingly capable AI, in particular the risk of losing control to AI and extinction risk. I'll suggest that a key question is, "When do we need to take this kind of risk seriously?"

We'll look at the issue of 'agency overhang', which suggests that adding agent-like abilities to AI could result in a sudden and surprising increase in these kinds of risks.

I'll draw on intuitions about humans taking administrative and political control (with reference to the 'Dictator Book Club') and rephrase agency overhang as 'power play overhang'.

I'll finish by suggesting that a lot of people may be making a subtle but important mistake in imagining just one fairly specific path to dangerous AI.
Experience Switching to Right Shoulder Round
by jefftk — 2024-05-08T17:30:02.095Z

Contra dance has a figure where two people walk a small circle looking
at each other. When it was introduced into contra in the 1970s as a
borrowing from ECD, it had the name "gypsy", originally

from
Morris dancing, but many communities now use "right shoulder
round".

In many dance communities the debate over whether and how to switch
functioned as a highly acrimonious culture war outlet. I really
didn't want our group going through that, but talking publicly about
how I didn't want that at the time would have been counterproductive.
Now that it's been ~5y since switching to "right shoulder round" and
~10y from the first big online discussions, I think this is probably
something I can share some history on.

While I'm sure people had occasionally talked about being
uncomfortable with the term, I think the first big online discussion
started in January 2014 with a since-deleted post in a Facebook group:

From: Elio Lewis

To: Stuff Contra Dancers Say

Date: 2014-01-20 9:48am

Hey, contra dance callers! I totally just figured out the ideal
substitution for that racist-named move! It should be called a
"hippie". It sounds similar enough to the offensive term that people
will still cue off of it (though I'd note my substitution during the
walkthrough), it appeals to a sense of silliness, and it's unlikely to
offend anyone. If you like the idea, please spread it around!

The discussion was long and heated, properties it shared with later
iterations on other platforms (ex: October 2015, January 2016, April 2016,
etc on Shared Weight). There were two main questions, the same ones
as in the role terms debate:

Should we switch away from the traditional term?

If we do switch, what should we switch to?

There were a lot of candidate terms, with a variety of issues, and
"right shoulder round" was quite a late addition. The first place I
find it written down is March
2018, four years into the debates. (That thread also gives a good
flavor of how these discussions tended to go, with a lot of
frustration, anger, and people talking past each other.) Looking back
at emails, it took about six months for the term to go from unknown to
the favorite.

The approach our dance organization took was mostly waiting: I knew
that this was something that had the potential to be divisive and
that there wasn't a widely supported replacement, and I s...