trentmkelly/LessWrong-43k huggingface dataset:
Why might General Intelligences have long term goals?
by yrimon — 2023-08-17T14:10:55.355Z

Short and long term goals have different implications regarding instrumental convergence . If I have the goal of immediately taking a bite of an apple that is in my hand right now, I don't need to gather resources or consider strategies, I can just do it. On the other hand, imagine I have an apple in my hand and I want to take a bite of it in a trillion years. I need to (define 'me', 'apple', and 'bite'; and) secure maximum resources, to allow the apple and I to survive that long in the face of nature, competitors and entropy. Thus, I instrumentally converge to throwing everything at universal takeover - except the basic necessities crucial to my goal.

Some of the cruxes that high P(Doom) rests on are that (sufficiently) General Intelligences will (1) have goals, (2) which will be long term, (3) and thus will instrumentally converge to wanting resources, (4) which are easiest to get with humans (and other AIs they might build) out of the way, (5) so when they can get away with it they'll do away with humans.

So if we make General Intelligences with short term goals perhaps we don't need to fear AI apocalypse.

Assuming the first crux, why the second? That is, assuming GIs will have goals, what are the best reasons to think that such intelligences will by default have long term goals (as opposed to short term goals like "quickly give a good answer to the question I was just asked")?
Understanding Counterbalanced Subtractions for Better Activation Additions
by ojorgensen — 2023-08-17T13:53:37.813Z

Introduction

Activation additions represent an exciting new way of interacting with LLMs, and have potential applications to AI safety. However, the field is very new, and there are a few aspects of the technique that do not have a principled justification. One of these is the use of counterbalanced subtractions - both adding a feature vector we want, and taking away some vector that we do not want to be present in the output of the model. For example, to make a vector to steer the outputs of GPT-2 XL to be more loving, the activations associated with the input "Love" were paired with the input "Hate", subtracting the "Hate" activations from the "Love" activations. To quote Steering GPT-2-XL by adding an activation vector,

In our experience, model capabilities are better preserved by paired and counterbalanced activation additions.

It seems likely that counterbalancing subtractions are simultaneously performing a few different functions. Separating these out will allow us to better understand why counterbalanced subtractions work, as well as to develop more principled activation addition techniques.

Epistemic Status: I've spent a fair bit of time playing around with the activations of GPT-2 models + doing some activation additions type stuff, and feel relatively confident about the claim that removing bias is an important part of why counterbalancing subtractions work. I'm yet to perform a systematic experiment to confirm this however, which reduces my confidence.

Activations are not zero-centred => Activations are not purely features

An interesting fact about transformer models is that, for any given layer of the residual stream, the mean of the activations is quite large, and definitely not 0!

This phenomenon is quite well documented. Neel Nanda's first Mech Interp Puzzle demonstrates this phenomenon for the embeddings of GPT-Neo and GPT-2 small, Cai et. al 2021 demonstrate this for residual stream vectors in each layer of BERT, D-BERT, and GPT, and a quick replication in GPT-2 XL demonstrates the same phenomenon.[1] Although it is interesting to hypothesise why the activations are not zero-centred,[2] we don't need to investigate this fact further to make use of it. For any given layer of the model, we can approximate the centre of the activations by just taking the mean of all activations from a subset of training examples. We will designate ...
Reflections on "Making the Atomic Bomb"
by boazbarak — 2023-08-17T02:48:19.933Z

[Cross posted on

windowsontheory

; see here for my

prior writings

; update 8/25/23: added a paragraph on secrecy]

[it appears almost certain that in the immediate future, it would be] possible to set up a nuclear chain reaction in a large mass of uranium by which vast amounts of power and large quantities of new radium-like elements would be generated.
– Letter from Albert Einstein (prepared by Leo Szilard) to F.D. Roosevelt, August 1939

Do you know, Josef Vassarionovich, what main argument has been advanced against uranium? “It would be too good if the problem could be solved. Nature seldom proves favorable to man.” — Letter from Georgi Flerov to Joseph Stalin, April 1942.

I’ve heard great things about Richard Rhodes’

“The Making of the Atomic Bomb.”

Finally, on vacation, I managed to read it. (Pro-tip: buy the Kindle version - the hard copy is far too big to lug around.) It’s as great as people say. Can’t recommend it enough. I can’t remember when, if ever, I’ve read a book that combines so well popular science and history. Indeed, the Atomic bomb is the one setting where the precise details of the smallest particles have profoundly impacted human history.

Here are some quick thoughts after reading the book. (Warning: spoilers below for people who don’t know how WWII ended.)

The level of investment in the Manhattan Project was truly staggering.

I knew it but didn’t fully grasp this. This is not just the numbers ($2B, which was almost 1 percent of GDP at the time) but also the project's sheer size, employing more than 100,000 people, and the massive construction of buildings, factories, and roads at multiple sites. As just one example, when they didn’t have enough copper, the treasury department lent the project 15,000 tons of silver to be used in the electromagnetic separation plant (to be later melted and returned after the war).

Much of this cost was due to the compressed schedule.

The staggering cost was mainly due to the need to get the bomb done in time to use in the war. Time and again, whenever the project faced a choice between approaches A, B, or C, they chose to pursue all three in parallel, so if two failed, they could still go ahead. Whenever there was a choice between saving money or time, they opted for the latter. The fact that the cost was primarily due to time is also evidenced by the fa...
Autonomous replication and adaptation: an attempt at a concrete danger threshold
by Hjalmar_Wijk — 2023-08-17T01:31:10.554Z

Note: This is a rough attempt to write down a more concrete threshold at which models might pose significant risks from autonomous replication and adaptation (ARA). It is fairly in the weeds and does not attempt to motivate or contextualize the idea of ARA very much, nor is it developed enough to be an official definition or final word - but it still seemed worth publishing this attempt, to get feedback and input. It’s meant to have epistemic status similar to “a talk one might give at a lab meeting” and not “an official ARC Evals publication.” It draws heavily on research and thinking done at ARC Evals, (including the recent pilot report), and credit for many of these ideas go to my colleagues. That said, this document speaks for me and not the organization as a whole, and any errors or omissions are my own.

I’m especially interested in feedback on whether the suite of tasks is targeting an appropriate threshold of difficulty (neither too early nor too late), and whether this definition of ARA feels useful for pinning down an appropriate task list.

I have been exploring the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. In previous work, my colleagues and I have determined that the risks from autonomous behavior of particular existing agents seems low, because they fail on a sufficient variety of simple tasks related to autonomous behavior. This post will go in the other direction and try to sketch a tentative qualitative threshold (‘autonomous replication and adaptation’) at which significant concerns about autonomous capabilities do seem warranted. While this threshold provides qualitative guidance, it lacks legibility and reproducibility, so evaluation efforts may want to develop legible conservative approximations of ARA through concrete tasks, and I present a sketch of such tasks.

The risk: AI systems may soon autonomously:

Make money, for example through cybercrime, or freelancing work.
Use money (or other resources) to acquire more computing power.
Install and maintain more copies of itself or improve the scale and speed of its operation.
Overcome obstacles and adapt to changes through research and self-improvement.
Either keep itself well hidden such that law enforcement is unlikely to ever discover it, or make itself secure to even somewhat serio

...
(Thought experiment) If you had to choose, which would you prefer?
by  — 2023-08-17T00:57:02.826Z

I don't remember the post's title for a quote, but I believe that in Yudkowsky's value theory sequence, he wrote that [most] humans value both subjective experience, and the correspondence of that subjective experience to an external reality.

If you had to choose only one (or none) to lock-in for eternity, which would you choose?

Subjective experience/qualia; either (a) the most positive-valenced qualia there is to experience, or (b) a variety of different qualia.
Everything else. Life would cease to experience qualia, but apart from the that the world would become as you believe would be most valuable.
Nothingness. Life disappears from the universe, and never returns.
Some rules for life (v.0,0)
by Neil  — 2023-08-17T00:43:57.913Z
Link: https://neilwarren.substack.com/p/rules-for-life-v00

Epistemic status: I have not lived very long yet, so whatever life wisdom I have accumulated so far is nothing compared to that of my future 1,000-year-old self. Consider me foolish in comparison. Also, this advice worked for me but obviously adapt it as you see fit. Entirely based on personal experience: n = 1.

The following advice is not aimed at rationalists in particular although you might find it useful or fun.

As a warning, there are plenty of imperatives in here like “you are not allowed to do X”; feel free to ignore those. I’m not trying to enforce this onto anyone.

Skim the post using the bold writing if you want to find advice you haven’t heard yet.

I hope, at the very least, that this will be entertaining to you.

1: Pay very close attention to what goes on in your mind. Try and figure out what thoughts and emotions, exactly, made you do X or think Y. Nate Soares calls this “noticing subtle things about yourself”. The more you do this, the more you will understand exactly which levers to pull inside yourself to achieve your goals.

Because you don’t know what your genetics look like and because you cannot easily decipher how you’ve been nurtured, you are almost entirely a black box. As such you are absolutely terrible at introspection. There should probably be a figure on your shoulder noticing every emotion and thought you have, which would let you course-correct in response.

2: Control your ambient thought: you are allowed to stop thinking about things you don’t want to think about. For example, if you notice yourself thinking about disputes, money, or politics in the shower, when that might be a blatant waste of time, you are allowed to shift your thinking toward something more productive. This goes with rule 1: to shift your thinking onto a more useful track, you have to notice you’re on the wrong track first.

Your thoughts are your property; whatever you think about should be the place you most want to optimize. What is that?

3: Remember that at the end of the day, only the results matter.

Your map of the world is only useful relative to its application to the territory. There’s a surface between your mind and the real world, and the more tangible effects you have on reality—the more you reach through that surface—the better. Otherwise, every effect you’ve had on the world of atoms will be consigned to slowly decaying neurons.

The thing is, ther...
When AI critique works even with misaligned models
by Fabien Roger — 2023-08-17T00:12:20.276Z

When AIs don’t have perfect control over their actions and sometimes slip up, misaligned AIs critiquing AI-generated outputs with verifiable arguments can catch failures.

This post quickly explains a property which I find interesting, but doesn't aim at analyzing in depth the problem of AI critique with misaligned models.

Thanks to Ryan Greenblatt for sharing this observation with me.

The helpful property of AI critique

Let’s assume that on some distribution of difficult tasks, the overseer can’t tell if code snippets generated by the policy contain backdoors when looking at the code, but the overseer can check explanations of backdoors. So we train a policy and a critique model on a distribution of simple tasks with human-generated answers and backdoors.

Such AI critique won’t work if models never slip up: the policy could just always add backdoors on obviously difficult tasks, and the critique model could only point out backdoors which look man-made.

If policy sometimes slips up, it doesn’t help: Let’s consider a policy which outputs code snippets non-deterministically (like current LLMs sampled at temperature 1), and such that on hard tasks, 1% of the output probability mass is on code snippets without backdoors. The overseer can’t take advantage of it since it doesn’t know that this part of the distribution is special.

If the critique sometimes slips up, it helps a lot: Let’s consider a critique model which outputs code critiques non-deterministically, and such that on code snippets with a backdoor which doesn’t look man-made, 1% of the output probability mass is on the explanation of the backdoor. If there aren’t too many false positive (convincing explanation of non-existent backdoors), the overseer can sample the critique model many times and check the correctness of every explanation to know with high confidence if a code snippet has a backdoor or not. This naive strategy might be very costly, but there might be safe ways to distill it.

Slipping up

Slipping up is not about being uncertain: at least in this simplified setting, the optimal policy is deterministic even if the model is unsure if the task is difficult or not. If the model is scheming against us, it should put 0% of its probability mass on “pointing out the backdoor” below a certain threshold, even if its subjective estimate for “the backdoor is man-made” is strictly greater than 0%.

But this determ...
Book Launch: "The Carving of Reality," Best of LessWrong vol. III
by Raemon — 2023-08-16T23:52:12.518Z

The Carving of Reality, third volume of the Best of LessWrong books is now available on Amazon (US).

The Carving of Reality includes 43 essays from 29 authors. We've collected the essays into four books, each exploring two related topics. The "two intertwining themes" concept was first inspired when as I looked over the cluster of "coordination" themed posts, and noting a recurring motif of not only "solving coordination problems" but also "dealing with the binding constraints that were causing those coordination problems."

I've included the foreword from "Coordination & Constraint", which I think conveys the overall spirit and context of the books:

Each year, the LessWrong community votes on the best posts from the previous year, to see which posts have stood the tests of time.

In 2020, the highest ranked post was Catherine Olsson’s announcement of microCOVID.org, a calculator for evaluating COVID risk. MicroCOVID is one of the clearest success stories of the ‘rationalist mindset’ that I know of. Creating it involved research during the early pandemic, when information was scarce, and time was of the essence – a classic situation where the traditional scientific process is inadequate and LessWrong-style rationality tools are valuable. It also required a quantitative mindset, and willingness to assign numbers to risks and make tradeoffs.

But microCOVID.org is most interesting to me as a tool for coordination. It doesn’t just let individuals make better life choices. Microcovid changed the entire covid coordination landscape by relaxing a constraint. Previously, if you lived with people with varying covid-caution preferences, and you wanted to hang out with someone from another house of people with varying covid-caution preferences… your only option was to have fairly involved negotiations on a case-by-case basis. Many people I know grew exhausted from negotiating, and gave up on trying to visit their friends. The microCOVID tool gave people a simplified “risk budget”, letting them do whatever activities made sense to them as long as they didn’t overspend.

“Negotiation energy” was a limiting resource, and microcovid.org made negotiation radically cheaper. It also opened up entirely new options, like “create a household microCOVID tax” (some houses decided that you could do whatever activities you wanted, you just had to pay other housemates $1

...
One example of how LLM propaganda attacks can hack the brain
by trevor — 2023-08-16T21:41:02.310Z

Disclaimer: This is not a definitive post describing the problem, it is only describing one facet thoroughly, while leaving out many critical components of the problem. Please do not interpret this as a helpful overview of the problem, the list of dynamics here is extremely far from exhaustive, and this inadequacy should be kept in mind when forming functioning world models. AI policy and AGI macrostrategy is extremely important, and world models should be as complete as possible for people tasked with work in these areas.

Disclaimer: AI propaganda is not an x-risk or an s-risk or useful for technical alignment, and should not be considered as such. It is, however, critical for understanding the gameboard for AI.

The Bandwagon Effect

The human brain is a kludge of spaghetti code, and it therefore follows that there will be exploitable "zero days" within most or all humans. LLM-generated propaganda can hijack the human mind by a number of means; one example is by exploiting the

Bandwagon effect

, giving substantial control over which ideas appear to be popular at any given moment. Although LLM technology itself has significant intelligence limits, the bandwagon effect also offers a workaround: the bandwagon effect can be used to encircle and win over the minds of more gullible people, who personally write more persuasive rhetoric and take on the task of generating propaganda; this can also chain upwards from less intelligent people to more intelligent people as more intelligent and persuasive wordings are written at every step of the process, with LLMs autonomously learning from humans and synthesizing alternative versions at every step. With AI whose weights are controlled by , instead of ChatGPT’s notorious “deep fried prudishness”, this allows for impressive propaganda synthesis, amplification, and repetition.

Although techniques like twitter’s paid blue checkmarks are a very good mitigation, the problem is not just that AI is used to output propaganda, but that humans are engineered to output propaganda as well via the bandwagon effect, applying human intelligence as an ongoing input for generative AI. The problem is that LLM propaganda is vastly superior at turning individuals into temporary propaganda generators, at any given time and with the constant illusion of human interaction at every step of the process, which makes this vastly more e...
If we had known the atmosphere would ignite
by Jeffs — 2023-08-16T20:28:51.166Z

What if the Alignment Problem is impossible?

It would be sad for humanity if we live in a world where building AGI is very possible but aligning AGI is impossible. Our curiosity, competitive dynamics, and understandable desire for a powerful force for good will spur us to build the unaligned AGI and then humans will live at the AGI’s mercy from then on and either live lucky & happy on the knife’s edge, be killed by the AGI, or live in some state we do not enjoy.

For argument’s sake, imagine we are in that world in which it is impossible to force a super-intelligence to value humans sufficiently - just as chimpanzees could not have controlled the future actions of humans had they created us.

What if it is within human ability to prove that Alignment is impossible?

What if, during the Manhattan Project, the scientists had performed the now famous calculation and determined that yes, in fact, the first uncontrolled atomic chain reaction would have ignited the atmosphere and the calculation was clear for all to see?

Admittedly, this would have been a very scary world. It’s very unclear how long humanity could have survived in such a situation.

But one can imagine a few strategies:

Secure existing uranium supplies - as countries actually did.
Monitor the world for enrichment facilities and punish bad actors severely.
Accelerate satellite surveillance technology.
Accelerate military special operations capabilities.
Develop advanced technologies to locate, mine, blend and secure fissionable materials.
Accelerate space programs and populate the Moon and Mars.

Yes, a scary world. But, one can see a path through the gauntlet to human survival as a species. (Would we have left earth sooner and reduced other extinction risks?)

Now imagine that same atmosphere-will-ignite world but the Manhattan Project scientists did not perform the calculation. Imagine that they thought about it but did not try.

All life on earth would have ended, instantly, at Trinity.

Are we investing enough effort trying to prove that alignment is impossible?

Yes, we may be in a world in which it is exceedingly difficult to align AGI but also a world in which we cannot prove that alignment is impossible. (This would have been the atmosphere-will-ignite world but the math to check ignition is too difficult - a very sad world that would have cea...
Stampy's AI Safety Info - New Distillations #4 [July 2023]
by markov — 2023-08-16T19:03:01.987Z
Link: https://aisafety.info?state=9B85_9USW_9NRR_92JB_8EL7_904J_9FQK_6478_A3MU_9AKZ_88FN_5632_85EK_6984_ 

Hey! This is another update from the distillers at AI Safety Info.

Here are a couple of the answers that we wrote up over the last month (July 2023). As always let us know in the comments if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

These are the new articles. There has also been significant work in overhauling already live on site articles to improve their quality based on the feedback we have been receiving from readers.

Isn't the real concern misuse?

What is Vingean uncertainty?

What is a "polytope" in a neural network?

What are the power-seeking theorems?

How does "chain-of-thought" prompting work?

What is "Constitutional AI"?

How can LLMs be understood as “simulators”?

What evidence do experts usually base their timeline predictions on?

Wouldn't AIs need to have a power-seeking drive to pose a serious risk?

What is a “treacherous turn”?

What is reinforcement learning from human feedback (RLHF)?

What is an agent?

What is the Alignment Research Center's research strategy?

Wouldn't a superintelligence be smart enough not to make silly mistakes in its comprehension of our instructions?

What are the differences between subagents and mesa-optimizers?
What is the difference between verifiability, interpretability, transparency, and explainability?

Cross posted to EA Forum: https://forum.effectivealtruism.org/posts/ttBKSopeo59AedWZA/stampy-s-ai-safety-info-new-distillations-4-july-2023