trentmkelly/LessWrong-43k huggingface dataset:
Poorly-Aimed Death Rays
by Thane Ruthenis — 2022-06-11T18:29:55.430Z

Alternate framing: Optimality is the tiger, and agents are its teeth.

Tonally relevant: Godzilla Strategies.

It's a problem when people think that a superintelligent AI will be just a volitionless tool that will do as told. But it's also a problem when people focus overly much on the story of "agency". When they imagine that all of the problems come from the AI "wanting" things, "thinking" things, and consequentializing all over the place about it. If only we could make it more of a volitionless tool! Then all of our problems would be solved. Because the problem is the AI using its power in clever ways with the deliberate intent to hurt us, right?

This, I feel, fails entirely to appreciate the sheer power of optimization, and how even the slightest failure to aim it properly, the slightest leakage of its energy in the wrong direction, for the briefest of moments, will be sufficient to wash us all away.

The problem isn't making a superintelligent system that wouldn't positively want to kill us. Accidentally killing us all is a natural property of superintelligence. The problem is making an AI that will deliberately spend a lot of effort on ensuring it's

not

killing us.

I find planet-destroying Death Rays to be a good analogy. Think the Death Star. Think—

Imagine that you're an engineer employed by an... eccentric fellow. The guy has a volcano lair, weird aesthetic tastes, and a tendency to put words like "world" and "domination" one after another. You know the type.

One of his latest schemes is to blow up Jupiter. To that end, he'd had excavated a giant cavern underneath his volcano lair, dug a long cylindrical tunnel from that cavern to the surface, and ordered your team to build a beam weapon in that cavern and shoot it through the tunnel at Jupiter.

You're getting paid literal tons of money, so you don't complain (except about the payment logistics). You have a pretty good idea of how to do that project, too. There are these weird crystal things your team found lying around. If you poke one in a particular way, it releases a narrow energy beam which blows up anything it touches. The power of the beam scales superexponentially with the strength of the poke; you're pretty sure shooting one with a rifle will do the Jupiter-vanishing trick.

There's just one problem: aim. You can never quite predict which part of the crystal will emit the beam. It depends on where you p...
AGI Safety Communications Initiative
by ines — 2022-06-11T17:34:31.415Z

Crossposted from the EA forum

A small group of AGI existential safety field-builders and I are starting research exploring a potential initiative about informing the public and/or important stakeholders about the risks of misaligned AI and the difficulties of aligning it.

We are aware that a public communication initiative like this carries risks (including of harming the AGI x-safety community’s reputation, of sparking animosity and misunderstandings between communities, or drawing attention to ways to misuse or irresponsibly develop scaleable ML architectures). We are still in the stage of evaluating whether/how this initiative would be good to pursue.

We are posting this on the forum to avoid the scenario where someone else starts a project about this at the same time and we end up doing duplicate work.

How you can get involved:

If you are currently undertaking work similar to this or are interested in doing so, message me your email address along with a bit of context about yourself/what you are doing.
We are drafting a longer post to share our current considerations and open questions. Message me if you would like to review the draft.
We are looking for one or two individuals who are excited to facilitate a research space for visiting researchers. The space will run in Oxford (one week in Sep ’22) and in Prague (9-16 Oct ’22) with accommodation and meals provided for. If you take on the role as facilitator, you will receive a monthly income of $2-3K gross for 3 months and actually get to spend most of that time on your own research in the area (of finding ways to clarify unresolved risks of transformative AI to/with other stakeholders). If you are interested, please message me and briefly describe your research background (as relevant to testing approaches for effective intergroup communication, conflict-resolution and/or consensus-building).
A gaming group for rationality-aware people
by dhatas — 2022-06-11T16:04:29.791Z

Hi everyone.

It was suggested 5 years ago to make a gaming group for rationalists.
However, I haven't found one.

So I made a Discord gaming-server for people who know about rationality (invite link).

Below, you will find the actions I took and the reasoning behind them.

Many papers were published on the negative and positive impact of gaming, so I will skip the whole debate of whether making a gaming server is good or bad.

The target audience is the people from the LessWrong community. However, there are no input filters. If somebody wants to chill with us, I see no reason to reject.

The rules are kept to the minimum: No calling names, no politics/religion/drama/nsfw.

Discord was chosen as a hosting platform because the solutions that don't collect data are unpopular and will stunt the community growth. Just don't write anything compromising when using Discord.

There are many things that need to be settled, such as moderators, automatic role assignment, and so on, but I think we will handle them as they come.
Why don't you introduce really impressive people you personally know to AI alignment (more often)?
by Verden — 2022-06-11T15:59:03.454Z

I have a few questions to the subset of readers who:

Believe technical AI alignment research is both important and hard to make significant progress in
Have a personal connection with a person who doesn't know much about AI alignment, but who you think would have a real chance to make valuable contributions to the field if they entered it (or perhaps you know someone who cares about AI risk and have such a personal connection, and you have enough knowledge to talk on their behalf). It may be your friend, colleague, supervisor, etc.

I would love to hear your thoughts on some of the following questions:

What reasons prevent you from introducing them to AI alignment by e.g. scheduling time with them and talking about some of the motivations and open problems in the field?
If you've tried something like this, how did it go?
What factors do you think would increase your willingness to bring AI alignment to their attention and/or the potential value resulting from it? Bonus points for reasonably low-hanging fruit here.
Godzilla Strategies
by johnswentworth — 2022-06-11T15:44:16.385Z

Clutching a bottle of whiskey in one hand and a shotgun in the other, John scoured the research literature for ideas... He discovered several papers that described software-assisted hardware recovery. The basic idea was simple: if hardware suffers more transient failures as it gets smaller, why not allow software to detect erroneous computations and re-execute them? This idea seemed promising until John realized THAT IT WAS THE WORST IDEA EVER. Modern software barely works when the hardware is correct, so relying on software to correct hardware errors is like asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. THIS DOES NOT LEAD TO RISING PROPERTY VALUES IN TOKYO. It’s better to stop scaling your transistors and avoid playing with monsters in the first place, instead of devising an elaborate series of monster checks-and-balances and then hoping that the monsters don’t do what monsters are always going to do because if they didn’t do those things, they’d be called dandelions or puppy hugs.

- James Mickens,

The Slow Winter

There’s a lot of AI alignment strategies which can reasonably be described as “ask Godzilla to prevent Mega-Godzilla from terrorizing Japan”. Use one AI to oversee another AI. Have two AIs debate each other. Use one maybe-somewhat-aligned AI to help design another. Etc.

Alignment researchers discuss various failure modes of asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. Maybe one of the two ends up much more powerful than the other. Maybe the two make an acausal agreement. Maybe the Nash Equilibrium between Godzilla and Mega-Godzilla just isn’t very good for humans in the first place. Etc. These failure modes are useful for guiding technical research.

… but I worry that talking about the known failure modes misleads people about the strategic viability of Godzilla strategies. It makes people think (whether consciously/intentionally or not) “well, if we could handle these particular failure modes, maybe asking Godzilla to prevent Mega-Godzilla from terrorizing Japan would work”.

What I like about the Godzilla analogy is that it gives a strategic intuition which much better matches the real world. When someone claims that their elaborate clever scheme will allow us to safely summon Godzilla in order to fight Mega-Godzilla, the intuitively-obviously-correct response is “THIS DOES NOT LEAD TO RISING PROPERTY VALUES...
Steganography and the CycleGAN - alignment failure case study
by Jan Czechowski — 2022-06-11T09:41:28.841Z

1. This is a (lightly edited) transcript of a lightning talk I gave during the Virtual AI Safety Camp 2022. The original talk in video format can be found here (can also by listened to)

2. Many thanks to Remmelt Ellen for preparing the initial version of the transcript and motivating me to publish this

3. I could probably improve this post a lot, but I decided to publish it as is because otherwise there's a chance I'd have never published it.

Just to start – recall the story of the PaperClip Maximiser - a story about a model that is doing things that we don't want it to do (like destroying humanity) as an unintended side effect of fulfilling a goal it was explicitly programmed to pursue.

The story that I wanted to tell today is one where such a situation occurred in real life, which is quite simple, well-documented, and not really widely known.

So coming back to 2017, I just started working on my first position that had NLP in the title, so I was very excited about artificial intelligence and learning more about it. Somewhere around the same time, the CycleGAN paper came out.

For those not familiar: what is CycleGAN about? In short, it was a clever way to put two GANs (Generative Adversarial Networks - how exactly these work is not important here) together that allowed training a model to translate pictures between two domains without any paired training examples. For example, in the middle, we see a translation from zebras to horses. You don't have datasets of paired examples of photos of zebras and photos of horses in the same postures.

Source: [1]

So how does this system work more or less? The important part here is that basically, you train two translators. Here they are called G and F. One is translating data from domain X to Y, and the other one is translating the other way around.

Source: [1]

In training, to the given training sample you first apply G, then F. So, in the horses-to-zebras example: you take a picture of a horse and first you apply G to land in the zebra domain, then you apply G to land back in the horse domain. And you require the first and the third picture to be similar to each other. The so-called cycle-consistency loss. At the same time, a discriminator (D_y) check if the second image Y isn't obviously different from known photos of zebras, giving the model an incentive to generate realistic images of zebras.

One of the datasets that were not ...
The Mountain Troll
by lsusr — 2022-06-11T09:14:01.479Z

It was a sane world. A Rational world. A world where every developmentally normal teenager was taught Bayesian probability.

Saundra's math class was dressed in their finest robes. Her teacher, Mr Waze, had invited the monk Ryokan to come speak. It was supposed to be a formality. Monks rarely came down from their mountain hermitages. The purpose of inviting monks to speak was to show respect for how much one does not know. And yet, monk Ryokan had come down to teach a regular high school class of students.

Saundra ran to grab monk Ryokan a chair. All the chairs were the same—even Mr Waze's. How could she show respect to the mountain monk? Saundra's eyes darted from chair to chair, looking for the cleanest or least worn chair. While she hesitated, Ryokan sat on the floor in front of the classroom. The students pushed their chairs and desks to the walls of the classroom so they could sit in a circle with Ryokan.

"The students have just completed their course on Bayesian probability," said Mr Waze.

"I see[1]," said Ryokan.

"The students also learned the history of Bayesian probability," said Mr Waze.

"I see," said Ryokan.

There was an awkward pause. The students waited for the monk to speak. The monk did not speak.

"What do you think of Bayesian probability?" said Saundra.

"I am a Frequentist," said Ryokan.

Mr Waze stumbled. The class gasped. A few students screamed.

"It is true that trolling is a core virtue of rationality," said Mr Waze, "but one must be careful not to go too far."

Ryokan shrugged.

Saundra raised her hand.

"You may speak. You need not raise your hand. Rationalism does not privilege one voice above all others," said Ryokan.

Saundra's voice quivered. "Why are you a Frequentist?" she said.

"Why are you a Bayesian?" said Ryokan. Ryokan kept his face still but he failed to conceal the twinkle in his eye.

Saundra glanced at Mr Waze. She forced herself to look away.

"May I ask you a question?" said Ryokan.

Saundra nodded.

"With what probability do you believe in Bayesianism?" said Ryokan.

Saundra thought about the question. Obviously not 1 because no Bayesian believes anything with a confidence of 1. But her confidence was still high.

"Ninety-nine percent," said Saundra, "Zero point nine nine."

"Why?" said Ryokan, "Did you use Bayes' Equation? What was your prior probability before your teacher taught you Bayesianism?"

"I notice I am confused," said Saundra.

"The most import...
How fast can we perform a forward pass?
by jsteinhardt — 2022-06-10T23:30:10.341Z

Thanks to Hao Zhang, Kayvon Fatahalian, and Jean-Stanislas Denain for helpful discussions and comments.

Over the last month, I’ve spent a lot of time trying to answer the following question:

How quickly can we perform one forward pass in a transformer model?

By a transformer model, I mean BERT, GPT-3, T5, Chinchilla, or other large language models that use a transformer architecture. By a forward pass, I mean the computation needed to generate the next token given all the tokens so far.[1] By “how quickly”, I mean how much wall clock time elapses between the call to the forward pass and its completion. So, even if I can run 1,000 forward passes in parallel, if each takes 1 second to complete, the answer is 1 second (not 1 millisecond).

One way to attempt answering this is to take the total number of operations in a forward pass and divide by the speed of your favorite GPU in FLOPS (floating-point operations/second). But this is wrong, because you would do better by parallelizing across multiple GPUs.[2]

The question then is really “how effectively can I parallelize a forward pass?” It turns out that this has different answers based on how “wasteful” we’re willing to be, in terms of GPU utilization. If we are willing to utilize only 5% of the GPU (but parallelize across many GPUs), we can perform the forward pass more quickly. So I’ll actually answer two questions:

How quickly can we perform a forward pass, assuming we require each GPU to have at least 40% utilization relative to roofline FLOPS?[3]

If we are willing to decrease utilization by a factor of k, how much faster can we perform a forward pass?

To simplify the analysis, I’ll make several assumptions (this is mainly targeted at people who are very familiar with GPU nuts and bolts; don’t worry if you don’t understand them yet):

[A] Parallelization, both within and across GPUs, is done via matrix tiling, as discussed in the NVIDIA User’s Guide.

[B] All bottleneck operations can be run at high arithmetic intensity for large batch sizes.

[C] The constraining resource is always either compute capacity, on-chip memory bandwidth, or network bandwidth of a single machine. In particular, it is not the L2 cache speed, the cross-GPU interconnect, or the ability to route information between far-away nodes in the network.

Assumption A holds for most systems as implemented today, but it’s possible that algori...
Summary of "AGI Ruin: A List of Lethalities"
by Stephen McAleese — 2022-06-10T22:35:48.500Z

Overview

Introduction

This post is a summary of Eliezer Yudkowsky’s post “AGI Ruin: A List of Lethalities”. I wrote it because I think the original post is longer and less organized than I would like it to be. The purpose of this post is to summarize the main points in the original post and structure the points in a new layout that I hope will make it easier to quickly read and reference each point.

The summary contains the following sections:

Overview
Why we need to solve the AI alignment problem
Why AI alignment is hard
Alignment techniques
The AI safety field

An AI is aligned if it doesn’t cause an existential catastrophe

When Eliezer says that AI alignment is lethally difficult, he is not talking about the challenge of creating perfect or ‘provable’ alignment. He says that even creating an outcome where there are any survivors is unlikely given the incomplete alignment methods we have today.

"if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty percent chance of killing more than one billion people, I'll take it."

Why we need to solve the AI alignment problem

AI can be much smarter than humans

AI can be much smarter than humans and use information much more efficiently than humans when making decisions or forming beliefs about the world. For example, AlphaZero learned to be superhuman at Go in only a few days.

AI alone could be very dangerous for humans

A misaligned AI smarter than humans could cause human extinction. The AI would not need a robot body to be dangerous because it could convince one or more humans to carry out its plans or hack into human infrastructure to make use of it. The AI could invent dangerous technologies such as advanced nanotechnology that we would not be able to defend against.

"Losing a conflict with a high-powered cognitive system looks at least as deadly as everybody on the face of the Earth suddenly falls over dead within the same second."

We can’t just decide not to build AGI

There is a lot of computer hardware in the world such as GPUs and many people have access to it. AI software and hardware are continually getting better and it would be very difficult to completely halt progress in these areas because everyone in the world would have to agree not to make progress in these areas.

Many actors are working on AGI research and even if one or more of them refrain from making progress, ...
How dangerous is human-level AI?
by Alex_Altair — 2022-06-10T17:38:27.643Z

I don't expect anything here to be original; this is just me thinking things through, and perhaps this post will be useful as a relatively stand-alone argument.

Typically, you can envision many many more future states than there are future states you could bring about. You can imagine going grocery shopping at any location on earth, but realistically you can only go grocery shopping at places where there are already grocery stores. This is also true for worldstates which would be good for you. You can imagine running every specific business in the world (and thereby capturing the profit) but you can probably only run a very small fraction of these businesses. If you were more capable you could run all of them; but even at lower capability, you can imagine running all of them.

I don't see why this wouldn't be true for advanced AIs, ones that make plans to attain future states according to some criterion. So it seems very likely to me that AIs will realize that taking over the world[1] would be instrumentally valuable[2] long before they could attain it. That is, the AI will be able to formulate the plan of taking over the world, and will evaluate it as high-utility, but will also evaluate it as unattainable. What does this look like as we turn the capabilities knob? What might these AIs do just on the cusp of being able to attain world domination?

To get a better handle on this, imagine an AI that is human-level, but still an AI in the sense that it is software implemented on a modern computer. Of course, "human-level" is not one thing; advanced AI systems will likely be far above human-level at some tasks, even if they are below human-level at others. But for the sake of making it easier to think about, assume they are basically human-level at being able to understand how the world works and think up plans. We are not assuming the AI has anything like human values or goals; we are not assuming anything about its goals, other than that it has some, and is approximately an expected utility maximizer. For the sake of tradition, we can use paperclips as the proxy goal. Given that, what factors would make it easier or harder to take over the world?

To generate answers to this question, we can substitute the question, why don't humans take over the world? Throughout history it has occurred to a lot of individual humans that taking over the world might be adv...
Another plausible scenario of AI risk: AI builds military infrastructure while collaborating with humans, defects later.
by avturchin — 2022-06-10T17:24:19.444Z

As was said before: "A lot of the AI risk arguments seem to come... with a very particular transhumanist aesthetic about the future (nanotech, ... etc.). I find these things (especially the transhumanist stuff) to not be very convincing…

I here suggest a plausible scenario where AI can get its own infrastructure and kill all living beings without the use of nanotech and biotech. A similar plan was described in the book “A for Andromeda” by Hoyl, but for Alien AI from SETI.

I assume that no nanotech or biotech will be used in this scenario. I also assume that AI has a subgoal “kill all humans”.

In a nutshell, AI’s plan is:

1.AI promises a group of people (likely a country, probably one of the nuclear powers) a military advantage over all its rivals, but in exchange, the group should help AI to get out of the box and collaborate in building autonomous military infrastructure as well as human-independent AI-infrastructure (data centers, electricity generation, chip fabs, general-purpose robots). Military infrastructure will be based around autonomous weapons. The question about their size and aesthetics is second: it could be missiles, drones, nanobots or even robots, or some combination of them.

2. After the needed AI-infrastructure is created and the required weapons are deployed, the AI kills not only rivals of his group of people, but also the group itself.

Some properties of this plan:

There is no need for AI to hide its recursive self-improvement from humans, so it will outperform other AIs which have to do RSI in secret.

Any AI-takeover plan assumes that AI will have to build an independent computational infrastructure which is capable to function without humans, or at least with the help of an enslaved group of humans.

Any AI-infrastructure-building plan converges to robots-building-other-robots, or robots self-replicating.

Any robots-building-robots plan converges to smaller and smaller “self-replicating” robots, which become eventually as close to nanotech as possible.