trentmkelly/LessWrong-43k huggingface dataset:
On Anthropic’s Sleeper Agents Paper
by Zvi — 2024-01-17T16:10:05.145Z

The recent paper from Anthropic is getting unusually high praise, much of it I think deserved.

The title is: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.

Scott Alexander also covers this, offering an excellent high level explanation, of both the result and the arguments about whether it is meaningful. You could start with his write-up to get the gist, then return here if you still want more details, or you can read here knowing that everything he discusses is covered below. There was one good comment, pointing out some of the ways deceptive behavior could come to pass, but most people got distracted by the ‘grue’ analogy.

Right up front before proceeding, to avoid a key misunderstanding: I want to emphasize that in this paper, the deception was introduced intentionally. The paper deals with attempts to remove it.

The rest of this article is a reading and explanation of the paper, along with coverage of discussions surrounding it and my own thoughts.

Abstract and Basics

Paper Abstract: Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques?

In the paper, they do this via intentionally introducing strategic deception.

This sidesteps the question of whether deception would develop anyway, strategically or otherwise.

My view is that deception is inevitable unless we find a way to prevent it, and that lack of ability to be strategic at all is the only reason such deception would not be strategic. More on that later.

Abstract continues: To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024.

We find that such backdoored behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it).

The backdoored behavior is most persistent in the largest models

...
A Pedagogical Guide to Corrigibility
by A.H. — 2024-01-17T11:45:06.075Z

This post is about ideas in the 2015 paper Corrigibility by Soares et al. The paper is not too hard to follow but is written in a fairly dry academic style (which is reasonable as it is an academic paper!). It also uses what I think is a clunky notation which makes some fairly simple results less intuitive.

The point of this post is not to provide rigorous proofs for every claim in the paper, but to focus more on intuitions, examples, and explanations. I'm especially trying to emphasise that the results of the paper are actually just straightforward consequences of expected utility maximization. The results not rely on any particularly esoteric ideas about agents or any unique features of the 'shutdown problem'. So don't be surprised if it seems simple -it mostly is! - but someone reading the original paper might miss this fact (I did).

This post is aimed to stand alone and you don't need to have read the original paper to follow along. The only pre-requisite for reading this post is basic familiarity with expected value reasoning.

Part 1: A Money Maximizer

Suppose you are a gambler trying to maximize the expected value of your financial gains. You value every dollar equally and do not have a decreasing marginal utility in money. We'll use this frame to understand some central claims of the corrigibility paper.

The First Claim

You enter a casino which is famously in financial ruin after it keeps offering visitors generous gambles. You are offered the following gamble:

With probability p, you gain $100 and probability (1−p), you gain $10.

Regardless of the value of p, it sounds like a good gamble! After all, it has an expected value of p×$100+(1−p)×$10 which will always be positive.

Now, suppose you are playing this gamble at the casino with p=16 (eg. a dice roll where you win when a six comes up) and the casino offers you an upgrade. You can now play the gamble with p=12 (eg. a coin flip). Do you accept?

This isn't much of a choice! Obviously, the gamble where p=12is a better deal: it gives you a higher probability of winning the larger amount, and a lower probability of winning the smaller amount. This gamble has an expected value of $55, whereas the p=16 gamble has an expected value of $25.

Now suppose you are playing the gamble with p=12. The casino offers you another upgrade "You can incr...
An Introduction To The Mandelbrot Set That Doesn't Mention Complex Numbers
by Yitz — 2024-01-17T09:48:07.930Z

Note: This post assumes you've heard of the Mandelbrot set before, and you want to know more about it, but that you find imaginary and complex numbers (e.g. the square root of negative one) a bit mystifying and counterintuitive. Instead of helping you understand the relevant math like a reasonable person would, I'm just going to pretend the concept doesn't exist, and try to explain how to generate the Mandelbrot set anyway.

My goal is for this post to (theoretically) be acceptable to the historical René Descartes, who coined the term "Imaginary number" because he did not believe such things could possibly exist.

I hereby formally invite you to a dance.

Pretty please with a cherry on top??

Since we're (presumably) both cool, hip people, let's go with a somewhat avant-garde dance that's popular with the kids these days.

I call this dance the Mandelbrot Waltz, but you can call it whatever you'd like. This dance follows very simple rules, with the quirk that your starting location will influence your part in the dance. You will unfortunately be cursed to dance forever (there's always a catch to these dance invitations!), but if you ever touch the edges of the dance floor, the curse will be lifted and your part in the dance ends, so it's really not all that bad...

In case you don't already know the moves, I'll describe how to do the dance yourself (if given an arbitrary starting point on the dance floor) step-by-step.

How To Perform The Mandelbrot Waltz: A Step-By-Step Guide

Preparation:

You will need: Yourself, an empty room, and a drawing tool (like chalk or tape).
Setup: Draw a line from the center of the room to the nearest part of the wall, like so:

This line is to help orient the dancers, so we'll refer to it as the "orienting line" moving forward.

Now, draw a circle around the room's center, such that it intersects the "orienting line" halfway through. It should look something like this:

A bird's-eye view of a dance hall, all decked out for a Mandelbrot Waltz. Fancy!
Note that drawing on the floor is totally optional; if you can keep this all in your head, good for you.

Starting Position:

Choose a starting point anywhere you want in the room. Remember this position—or jot it down on a notepad if your memory is bad—for later.

The red dot is you!

Step 1 - Rotation Doubling:

Imagine a line connecting your current position to the center of the circle:

Congrats on your new imagina

...
Vote in the LessWrong review! (LW 2022 Review voting phase)
by habryka — 2024-01-17T07:22:17.921Z

Click

here

to begin voting. Click here for a general overview of the 2022 review.

The final phase of the 2022 review is upon us! We’ve spent 2 weeks nominating posts, and a month reviewing them. And today, we begin the final vote!

The LessWrong Review has two major goals.

Improve the LessWrong community's longterm feedback and reward cycle.
Build common knowledge about the best ideas we've discovered on LessWrong.

Voting sends a signal about which posts were most important. This then feeds into various compilation and distillation efforts. By default we will create a sequence of top posts, but we are also looking into other curation efforts (like ebooks and audiobooks and maybe print-on-demand books).

Voting

started yesterday (January 15th) and continues through the end of January.

You can see the results of previous votes on the Best Of page: https://www.lesswrong.com/bestoflesswrong

Who Can Vote?

All users registered before 2021 can vote. The LessWrong curation team will weight the votes of users with 1000+ karma more highly when assembling sequences, books, or other projects.

How do I vote?

Go to the Review Voting page. There, you'll see posts that you haven't yet voted on sorted first, with posts you've previously given a karma-vote to sorted to the very top (they have a green stripe along the left side. Strong upvotes have a darker green).

The posts will include the reviews that got written about that post. This is intended to help you make an informed vote. You can read reviews that look interesting or highly upvoted to get a better sense of how the post held up.

You vote by clicking buttons that assign a post a score. A score of 1 means roughly "this post was good." A score of 4 means "this post was quite important". A score of 9 means "this post was extremely important." A vote of 0 means "I don't have a strong opinion."

(A vote of "4" is 4x as strong as a vote of "1", a vote of "9" is 9x)

If you intuitively mark posts as "good/important/extremely-important", you'll probably do fine expressing your votes.

Okay, but I am a nerdy LessWronger and I like to know how things work in detail. How does this work under the hood?

The system is built on top of a quadratic voting system. In quadratic voting, you have a limited number of points to spend. You can vote things more strongly, but the cost of a vote increases at a quadratic rate. So:

A vote of "1" cost

...
Coalescer Models
by DaemonicSigil — 2024-01-17T06:39:30.102Z

DaemonicSigil

So we're here today to discuss this post on an improvement to diffuser networks: https://www.lesswrong.com/posts/6Cdusp5xzrHEoHz9n/faster-latent-diffusion

Just as some general background for readers, there are various kinds of model one can train to imitate some empirical (i.e. you have a lot of examples) distribution. The main ones are autoencoders, GANs, and diffusers. (At least among architectures general enough to handle images. For text there's also the option of just predicting a probability distribution over the next token, since it has a very linear causal structure to it, and the set of tokens is not too large. IIRC people have tried this to for images too by flattening them to a string of tokens and it works surprisingly okay.) So far it seems like diffusers give the best results, but they're also slow, because to generate an image from the diffuser you have to run the image through a denoising network multiple times and then add back more noise at each step.

The distribution of images is clustered and clumped together in a very complicated hierarchical way. So if we're the denoising network and we're given an image with a small amount of noise to correct, we'll be able to guess which cluster it came from pretty well. If the image instead has a large amount of noise, then it has travelled ("diffused") pretty far through space and there are a lot of different clusters it could have come from. We can't say for sure which cluster is the correct one, so we just output an estimate that is the average of the centers of all the plausible clusters, so our loss isn't too big in the case of any one outcome.

That's why diffusers need so many steps. They need to gradually anneal down from high noise levels to low noise levels, and at each step they're able to become a little more decisive about which clusters and sub-clusters the image is in. GANs don't have this problem because they can just generate an entire image in one shot, but they have other drawbacks such as being a huge pain to train and suffering from mode collapse.

You suggested that I should take a look at Consistency Models as background to your idea. Rather than training a network to predict the average denoised image directly, we train it to be consistent in its predictions for different amounts of noise. The following loss function is Eq. 10 in the paper:

L=Ez∼N(0,I),x∼dataset[|fθ(x+tn

...
Maybe talking isn't the best way to communicate with LLMs
by mnvr — 2024-01-17T06:24:18.458Z
Link: https://mrmr.io/chatgpt

There are 3 reasons why not all interaction with LLMs is going to be using natural language:

SaSSy interfaces are better for focused tasks
Some people don’t like talking
Talking to a non-sentient intelligence is creepy

#1 is already known and acted upon, this post covers the other two.
D&D.Sci Hypersphere Analysis Part 3: Beat it with Linear Algebra
by aphyer — 2024-01-16T22:44:52.424Z

As before, this is composed entirely of spoilers for abstractapplic's D&D.Sci scenario, don't read if you want to solve on your own!

Last time, we left off with a solution that could reliably get us ~95% performance on all our generators. Sadly, in the Glorious and Righteous Interstellar Empire, 95% performance still means Just and Reasonable Beheading (praise our Eternal Empress for Her leniency)!

Quality Score

We plotted Performance in our first writeup, and saw this:

But we've seen a few things since then. In particular, looking at how different effects interacted makes them look multiplicative rather than additive. So now let's try instead getting a frequency chart of Log(Performance):

That looks somewhat familiar!

So! Our new theory is as follows:

There are a variety of different things that can hurt performance.
All of these things are multiplicative. So one 50% reduction and one 20% reduction make a 60% reduction.
All or almost all of these things are independent, we don't have to worry about interactions.

Given that, we're going to try out linear regression:

We are going to try to predict Penalty (negative log of performance).
We're going to use as predictors some existing variables:

Four boolean columns for air tasting of Apples/Burning/Copper/Mint (all four being 0 means air tastes of nothing).
Four boolean columns for Skittering/Buzzing/Squelching/Humming.
Feng Shui (as two separate booleans for Exceptional/Disharmonious, with the default being Adequate)
Strange Smell (as two separate booleans for None/Extremely, with the default being Somewhat)

And some derivatives of existing variables:

Latitude: rather than including it directly, we include a 0 or 1 'Is In Death Zone?'
Longitude: again rather than including it directly, we include a sine wave that seems to match the 'sunlight effect': Cosine(Longitude-60).
Value of Pi: since High and Low values look different, and high values less than ~3.16 look like they might not do anything at all, we include 'amount pi is below 3.14159' and 'amount pi is above 3.14159' as separate cols so that we can derive e.g. 'being too low is 4x worse than being too high'.
Murphy's Constant: based on trying to fit a curve to what we saw in part 2, I'm including all of MC, MC^2,(2 ^ MC), (3^MC), and (4^MC) as indicator cols.

This produces something that does a pretty good job of predicting score:

but still has more...
The weak-to-strong generalization (WTSG) paper in 60 seconds
by sudo — 2024-01-16T22:44:49.078Z
Link: https://arxiv.org/abs/2312.09390

There's probably nothing new here if you've read the paper.

I am not affiliated with the paper authors. I took some figures from the paper.

You should really go read the paper.

Thanks to everyone who helped me with this post! Thanks to Justis for providing feedback on a draft!

See also: https://www.lesswrong.com/posts/9W8roCAeEccSa3Chz/weak-to-strong-generalization-eliciting-strong-capabilities

The headline result of WTSG is that, across a variety of supervised tasks, when you fine-tune a strong model using "pseudo-labels" (i.e. labels generated by a model, rather than ground-truth labels) from a weak model, the strong model surpasses the performance of the weak model.

The methodology used to determine this result is approximately as follows:

Fine-tune a small, weak model on your ground truth labels;
Take a large, strong model. Train it on pseudo-labels produced by the small, weak model;
Take an identical large, strong model. Train it directly on the ground truth labels.
Evaluate and compare the performance of the strong model trained on pseudolabels, the strong model trained on ground truth labels, and the weak model trained on ground truth labels.

Q: Is the headline result because the strong model couldn't simulate the pseudo-labels well, so resorted to the next best thing for (predicting the truth)?

A: Maybe.

It doesn't seem like the strong model in the WTSG paper is capable of perfectly simulating the weak model.
When the authors tried making simulating the weak model very easy (by appending "I think this is {weak_label}. What do you think?" to every prompt), the WTSG effect became substantially worse. (If this seems somewhat toy, I feel this way too. I'm currently working on improving on this experiment!)
When the authors tried using completely unsimulable label noise (instead of the more simulable mistakes of the weak models), performance predictably increased.

Q: What is this "aux loss" about?

A: That's the "auxiliary confidence loss" that the authors discovered improved weak-to-strong generalization in many domains. It tells the model to be more confident, and "reduces imitation of supervisor mistakes." (pg. 13)

Q: Did they try replacing the fine-tuning with linear probing or prompting?

A: Yes. They seem to work about as well.

Q: Should I still read the paper?
A: Yes! There's a bunch of interesting stuff that I haven't included here. (Inverse sca...
Social media alignment test
by amayhew — 2024-01-16T20:56:16.618Z
Link: https://naiveskepticblog.wordpress.com/2024/01/16/social-media-alignment-test/

Epistemic Status: Probably not necessary, certainly not sufficient.

Think you have a good AI alignment plan? Prove it!

Build some lower stakes mass influence system, like a social media platform, which implements your plan. If it is more engaging and more pro-social, then your plan will be worth looking at. No powerful AI should be deployed until the alignment plan can pass this basic test.

Alignment plans need to deal with the fundamental divide between machines doing what we tell them to do and machines doing what we want them to do. Any proposal that can address this problem can not be limited to a particular machine architecture. Thus, it should be able to be translated into social media.

The aligned social media system would connect people who might do good together, facilitate cooperation, and promote ideas and messages that encourage the public good. The meta-debate about what counts as the public good seem likely to be a particularly important conversation to promote and referee.

Perhaps establish a moral parliament to facilitate the role of referee. There may need to be a moral landscape to allow multiple different ways of flourishing. All of this should probably be transparent.

Should the system create text of its own? Probably not.

If your proposal can not make social media less toxic and more pro-social it is not ready for primetime.
Medical Roundup #1
by Zvi — 2024-01-16T20:30:35.802Z

Saving up medical and health related stories from several months allowed for much better organizing of them, so I am happy I split these off. I will still post anything more urgent on a faster basis. There’s lots of things here that are fascinating and potentially very important, but I’ve had to prioritize and focus elsewhere, so I hope others pick up various torches.

Vaccination Ho!

We have a new malaria vaccine. That’s great. WHO thinks this is not an especially urgent opportunity, or any kind of ‘emergency’ and so wants to wait for months before actually putting shots into arms. So what if we also see reports like ‘cuts infant deaths by 13%’? WHO doing WHO things, WHO Delenda Est and all that. What can we do about this?

Also, EA and everyone else who works in global health needs to do a complete post-mortem of how this was allowed to take so long, and why they couldn’t or didn’t do more to speed things along. There are in particular claims that the 2015-2019 delay was due to lack of funding, despite a malaria vaccine being an Open Phil priority. Saloni Dattani, Rachel Glennerster and Siddhartha Haria write about the long road for Works in Progress. They recommend future use of advance market commitments, which seems like a no brainer first step.

We also have an FDA approved vaccine for chikungunya.

Oh, and also we invented a vaccine for cancer, a huge boost to melanoma treatment.

Katalin Kariko and Drew Weissman win the Nobel Prize for mRNA vaccine technology. Rarely are such decisions this easy. Worth remembering that, in addition to denying me admission despite my status as a legacy, the University of Pennsylvania also refused to allow Kariko a tenure track position, calling her ‘not of faculty quality,’ and laughed at her leaving for BioNTech, especially when they refer to this as ‘Penn’s historic research team.’

Did you also know that Katalin’s advisor threatened to have her deported if she switched labs, and attempted to follow through on that threat?

I also need to note the deep disappointment in Elon Musk, who even a few months ago was continuing to throw shade on the Covid vaccines.

And what do we do more generally about the fact that there are quite a lot of takes that one has reason to be nervous to say out loud, seem likely to be true, and also are endorsed by the majority of the population?

When we discovered all the vaccines. Progress conti...
Being nicer than Clippy
by Joe Carlsmith — 2024-01-16T19:44:23.893Z

(Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series I'm calling "Otherness and control in
the age of AGI." I'm hoping that the individual essays can be read
fairly well on their own, but see
here
for a summary of the essays that have been released thus far.)

In my last
essay,
I discussed a certain kind of momentum, in some of the philosophical
vibes underlying the AI risk discourse,[1] towards deeming more and
more agents – including: human agents – "misaligned" in the sense of:
not-to-be-trusted to optimize the universe hard according to their
values-on-reflection. We can debate exactly how much mistrust to have in
different cases, here, but I think the sense in which AI risk issues can
extend to humans, too, can remind us of the sense in which AI risk is
substantially (though, not entirely) a generalization and
intensification of the sort of "balance of power between agents with
different values" problem we already deal with in the context of the
human world. And I think it may point us towards guidance from our
existing ethical and political traditions, in navigating this problem,
that we might otherwise neglect.

In this essay, I try to gesture at a part of these traditions that I see
as particularly important: namely, the part that advises us to be "nicer
than Clippy" – not just in what we do with spare matter and energy, but
in how we relate to agents-with-different-values more generally. Let me
say more about what I mean.

Utilitarian vices

As many have noted, Yudkowsky's paperclip maximizer looks a lot like
total utilitarian. In particular, its sole aim is to "tile the universe"
with a specific sort of hyper-optimized pattern. Yes, in principle, the
alignment worry applies to goals that don't fit this schema (for
example: "cure cancer" or "do god-knows-whatever kludge of weird
gradient-descent-implanted proxy stuff"). But somehow, especially in
Yudkowskian discussions of AI risk, the misaligned AIs often end up
looking pretty utilitarian-y, and a universe tiled with something – and
in particular, "tiny-molecular-blahs" – often ends seeming like a
notably common sort of superintelligent Utopia.

What's more, while Yudkowsky doesn't think human values are utilitarian,
he thinks of us (or at least, himself) as sufficiently galaxy-eating
that it's easy to round off his "battle ...