trentmkelly/LessWrong-43k huggingface dataset:
New roles on my team: come build Open Phil's technical AI safety program with me!
by Ajeya Cotra — 2023-10-19T16:47:59.701Z

Open Phil

announced

two weeks ago that we’re hiring for over 20 roles across our teams working on global catastrophic risk reduction — and we’ll answer questions at

our AMA

starting tomorrow. Ahead of that, I wanted to share some information about the roles I’m hiring for on

my team

(Technical AI Safety). This team is aiming to think through what technical research could most help us understand and reduce AI x-risk, and build thriving fields in high priority research areas by making grants to great projects and research groups.

First of all — since we initially listed roles on Sep 29, we’ve added three new roles in Technical AI Safety that you might not have seen yet if you only saw the original announcement! In addition to the (Senior) Program Associate role that was there originally, we added an Executive Assistant role last week — and yesterday we added a (Senior) Research Associate role and a role for a Senior Program Associate specializing in a particular subfield of AI safety research (e.g. interpretability, alignment theory, etc). Check those out if they seem interesting! The Executive Assistant role in particular requires a very different, less technical skill set.

Secondly, before starting to answer AMA questions, I wanted to highlight that our technical AI safety giving is far away from where it should be at equilibrium, there is considerable room to grow, and hiring more people is likely to lead quickly to more and better grants. My estimate is that last year, we recommended around ~$25M in grants to technical AI safety,[1] and so far this year I’ve recommended a similar amount. With more capacity for grant evaluation, research, and operations, we think this could pretty readily double or more.

All of our GCR teams (Technical AI Safety led by me, Capacity Building led by Claire Zabel, AI Governance and Policy led by Luke Muehlhauser, and Biosecurity led by Andrew Snyder-Beattie) are heavily capacity constrained right now — especially the teams that do work related to AI, given the recent boom in interest and activity in that area. I think my team currently faces even more severe constraints than other program teams. Compared to other teams, my team:

Is much smaller: Until literally last week, it was just me focusing primarily on technical AI safety (although Claire’s team sometimes funds technical AI s

...
Infinite tower of meta-probability
by fryolysis — 2023-10-19T16:44:31.496Z

Suppose that I have a coin with probability of heads p. I certainly know that p is fixed and does not change as I toss the coin. I would like to express my degree of belief in p and then update it as I toss the coin.

Using a constant pdf to model my initial belief, the problem becomes a classic one and it turns out that my belief in p should be expressed with the pdf f(x)=(nh)xh(1−x)n−h after observing h heads out of n tosses. That's fine.

But let's say I'm a super-skeptic guy that avoids accepting any statement with certainty, and I am aware of the issue of parametrization dependence too. So I dislike this solution and instead choose to attach beliefs to statements of the form S(f)= "my initial degree of belief is represented with probability density function f."

Well this is not quite possible since the set of all such f is uncountable. However something similar to the probability density trick we use for continuous variables should do the job here as well. After observing some heads and tails, each initial belief function will be updated just as we did before, which will create a new uneven "density" distribution over S(f). When I want to express my belief that p is in between numbers a and b, now I have a probability density function instead of a definite number, which is a collection of all definite numbers from each (updated) prior. Now I can use the mean of this function to express my guess and I can even be skeptic about my own belief!

This first meta level is still somewhat manageable, as I computed the Var(μ) = 1/12 for the initial uniform density over S(f) where μ is the mean of a particular f. I am not sure whether my approach is correct, though. Since the domain of each f is finite, I discretize this domain and represent the uniform density over S(f) as a finite collection of continuous random variables whose joint density is constant. Then taking the limit to infinity.

The whole thing may not make sense at all. I'm just curious what would happen if we use even deeper meta levels, with the outermost level being the uniform "thing". Is there any math literature anybody knows that already explored something similar to this idea? Like maybe use of probability theory in highe...
A NotKillEveryoneIsm Argument for Accelerating Deep Learning Research
by Logan Zoellner — 2023-10-19T16:28:52.218Z
Link: https://midwitalignment.substack.com/p/a-notkilleveryoneism-argument-for

TLDR: This is really just a longer version of this comment.

A metaphor

You are Rocket McRocket-Face, the CEO of Rockets Inc, the world’s largest and most reputable rocket company. Rockets Inc isn’t the only rocket company in the world, but it is by far the biggest, richest, and most powerful rocket company on Earth. No other rocket company holds a candle to Rocket Inc. Nor are they likely to in the next 3-5 years.

Rocket McRocket-Face, CEO of Rockets Inc.

As the CEO of Rockets Inc, you dream of one day reaching the moon. Your reasons for dreaming are twofold. The first reason is that reaching the moon is a dream that men have had since ancient times. It is an achievement truly worthy to behold. The second reason is a bit more practical. It is widely agreed in the world of rocket science that the first person to reach the moon will hold a commanding advantage. From the heights of the moon the first person to reach it will be virtually unconquerable, able to hurl moon rocks down to punish any of their enemies.

Picture of Rocket McRocket-Face looking at a map of the moon

One day, two scientists come to you with research proposals for new types of rockets.

The first scientist is Mathy McEngineer. Mathy is one of your best engineers. He is well known for the reliability of his rockets. His design is comforting, simple-to-understand, and a natural extension of currently known rocket technologies. The design is so simple it can be explained to anyone with a degree in rocket-engineering in a few minutes. It involves taking the current well-known and trustworthy rocket designs and adding a few more parts: more engines, more fuel. Nothing out of the ordinary.

Mathy’s plan may not be brilliant, but it’s trustworthy and safe. And there’s a good chance that it will reach the moon (although it’s unlikely to reach even the nearest star).

Mathy McEngineer

After your meeting with Mathy, you are feeling good about your chances of winning the race to the moon and decide to leave work early that day. On your way out of work, you are suddenly stopped by Subtle McGenius. Immediately the good feeling you had vanishes as you feel a ball of anxiety in the pit of your stomach. You hate meeting with Subtle McGenius. He doesn’t even work for Rockets Inc (you fired him years ago), but somehow he always manages to show back up at the worst possible time. Times like right no...
Knowledge Base 5: Business model
by iwis — 2023-10-19T16:06:24.008Z

The system described in the posts Knowledge Base 2: The structure and the method of building, Knowledge Base 3: Shopping advisor and other uses of knowledge base about products, and Knowledge Base 4: General applications can have the following sources of revenue:

simple ads;
fees for placing a product on a search results list;
fees for redirecting to websites of stores;
percentage of a product sale/rent price.

Why should users add information to the system?

If a user adds information that will be used to generate some revenue, we will share a part of this revenue with him. Usefulness (and credibility) of the information will be taken into account. An example of useful information are all pieces of information displayed on a product overview page on which a user clicks the Buy button.
Adding information in the initial phase of knowledge base development can be particularly profitable since users can choose the most popular information that will bring the largest income in the future. This can be compared to buying Bitcoins in the initial phase.
AI #34: Chipping Away at Chip Exports
by Zvi — 2023-10-19T15:00:03.055Z

It did not get the bulk of the attention, but the actual biggest story this week was that America tightened the rules on its chip exports, closing the loophole Nvidia was using to create the A800 and H800. Perhaps the new restrictions will actually have teeth.

Also new capabilities continue to come in based on the recent GPT upgrades, along with the first signs of adversarial attacks.

Also a lot of rhetoric, including, yes, that manifesto. Yes, I do cover it.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. A model made of gears.

Dalle-3 Complete Prompt. Here you go.

GPT-4 Real This Time. Begun, the adversarial vision attacks have.

Fun With Image Generation. Upscale the Mona Lisa, perhaps?

Deepfaketown and Botpocalypse Soon. Hi, I’m NYC Mayor Eric Adams.

People Genuinely Against Genuine People Personalities. They warned us.

They Took Our Jobs. The stack doth no longer overflow.

Get Involved. Open Philanthropy and Survival and Flourishing are hiring.

Introducing. Here, have a new model.

In Other AI News. The State of AI report is here.

Quiet Speculations. SEC chair predicts unavoidable financial crash. No.

Man With a Plan. Are responsible scaling plans responsible? Are they plans?

China. Draft guidance on how their companies can comply with the rules.

The Quest for Sane Regulation. A proposed international treaty.

The Chips are Down. The biggest loophole on chip exports to China is closing.

The Week in Audio. Justin Wolfers on GPT and homework, me on a new podcast.

Yes, We Will Speak Directly Into This Microphone. We aim to convince.

Rhetorical Innovation. Critch tries twice more, an objection taxonomy, and more.

Open Source AI is Unsafe and Nothing Can Fix This. Better documentation.

No One Would Be So Stupid As To. Unprincipled exception too good to ignore.

Aligning a Smarter Than Human Intelligence is Difficult. Ask any human.

People Are Worried About AI Killing Everyone. Quite a lot of them.

New Bengio Interview. Lot of good content.

Marc Andreessen’s Techno-Optimist Manifesto. I want to like it, except, yeah.

Other People Are Not As Worried About AI Killing Everyone. Usual suspects.

Other People Wonder Whether It Would Be Moral To Not Die. Maybe stop?

The Lighter Side. America.

Language Models Offer Mundane Utility

...
Is Yann LeCun strawmanning AI x-risks?
by Chris_Leong — 2023-10-19T11:35:08.167Z

Tom Chivers expresses his frustration with Yann LeCun:

I also find his comments here frustrating, but I want to offer another possible explanation.
Even though basically no one in the AI Safety community makes this argument, unfortunately, many people in the general population think about AI this way.

Yann probably thinks that it is a higher priority for him to address this belief held by a much larger range of people than it is for him to address the AI Safety crowd's arguments.

He may think that he's in a situation where if he focused on addressing the best arguments, he would lose the political battle due to naive people believing the "strawman" arguments.

In light of this, I don't think it's completely accurate to say he's addressing a strawman. I find it frustrating as well and I wish he'd address us directly. But I also understand the incentives that lead him to do what he does.
[Video] Too much Empiricism kills you
by Johannes C. Mayer — 2023-10-19T05:08:09.149Z
Link: https://youtu.be/vqHlPb18ROE?si=vf840i97GZwgxIjD

Here is a video I made 2 months ago. It gives a mediocre at best explanation for an important foundational argument:

It is normally possible to make progress using empirical methods, as long as you can measure how good a particular change was. That holds even when you don't really understand <what you are doing/the systems you are building>. This explains why researchers can advance capabilities in ML even though they basically do not understand the internals of the current deep learning systems at all.

I also argue that any progress in understanding can be dangerous, because it often improves the frontier of things that can be effectively explored through empirical methods. A corollary of this is that mechanistic interpretability can make it easier to advance capabilities.

The argument generalizes very far.
Are humans misaligned with evolution?
by TekhneMakre — 2023-10-19T03:14:14.759Z

There is an argument that although humans evolved under pressure to maximize inclusive genetic fitness (IGF), humans don't actually try to maximize their own IGF. This, as the argument goes, shows that in the one case we have of a process creating general intelligence, it was not the case that the optimization target of the created intelligence ended up being the same as the optimization target of the process that created it. Therefore, alignment doesn't happen by default. To quote from A central AI alignment problem: capabilities generalization, and the sharp left turn:

And in the same stroke that [an AI's] capabilities leap forward, its alignment properties are revealed to be shallow, and to fail to generalize. The central analogy here is that optimizing apes for inclusive genetic fitness (IGF) doesn't make the resulting humans optimize mentally for IGF. Like, sure, the apes are eating because they have a hunger instinct and having sex because it feels good—but it's not like they could be eating/fornicating due to explicit reasoning about how those activities lead to more IGF. They can't yet perform the sort of abstract reasoning that would correctly justify those actions in terms of IGF. And then, when they start to generalize well in the way of humans, they predictably don't suddenly start eating/fornicating because of abstract reasoning about IGF, even though they now could. Instead, they invent condoms, and fight you if you try to remove their enjoyment of good food (telling them to just calculate IGF manually). The alignment properties you lauded before the capabilities started to generalize, predictably fail to generalize with the capabilities.

Jacob published Evolution Solved Alignment (what sharp left turn?), arguing that actually humans represent a great alignment success. Evolution was trying to make things that make many copies of themselves, and humans are enormously successful on that metric. To quote Jacob:

For the evolution of human intelligence, the optimizer is just evolution: biological natural selection. The utility function is something like fitness: ex gene replication count (of the human defining genes)[1]. And by any reasonable measure, it is obvious that humans are enormously successful. If we normalize so that a utility score of 1 represents a mild success - the expectation of a typical draw of a great a

...
Brains, Planes, Blimps, and Algorithms
by ai dan — 2023-10-18T21:26:20.636Z

I.

Right now there is a big debate over whether AI is like a brain, or like an algorithm. I think that this is a lot like debating whether planes are more like birds, or like blimps. I’ll be arguing pro-bird & pro-brain.

Just to ground the analogy, In the late 1800s the Wright brothers spent a lot of time studying birds. They helped develop simple models of lift to explain their flight, they built wind tunnels in their lab to test and refine their models, they created new types of gliders based on their findings, and eventually they created the plane - a flying machine with wings.

Obviously bird wings have major differences from plane wings. Bird wings have feathers, they fold in the middle, they can flap. Inside they are made of meat and bone. Early aeronauts could have come up with a new word for plane wings, but instead they borrowed the word “wing” from birds, and I think for good reason.

Imagine you had just witnessed the Wright brothers fly, and now you’re traveling around explaining what you saw. You could say they made a flying machine, however blimps had already been around for about 50 years. Maybe you could call it a faster/smaller flying machine, but people would likely get confused trying to imagine a faster/smaller blimp.

Instead, you would probably say “No, this flying machine is different! Instead of a balloon this flying machine has wings”. And immediately people would recognize that you are not talking about some new type of blimp.

II.

If you ask most smart non-neuroscientists what is going on in the brain, you will usually get an idea of a big complex interconnected web of neurons that fire into each other, creating a cascade that somehow processes information. This web of neurons continually updates itself via experience, with connections growing stronger or weaker over time as you learn.

This is also a great simplified description of how artificial neural networks work. Which shouldn't be too surprising - artificial neural networks were largely developed as a joint effort between cognitive psychologists and computer scientists in the 50s and 60s to try and model the brain.

Note that we still don’t really know how the brain works. The Wright brothers didn’t really understand aerodynamics either. It’s one thing to build something cool that works, but it takes a long time to develop a comprehensive theory of how something...
The (partial) fallacy of dumb superintelligence
by Seth Herd — 2023-10-18T21:25:16.893Z

In

her opening statement for the Munk debate on AI risk,

Melanie Mitchell addressed a proposed example of AGI risk: an AGI tasked with fixing climate change might decide to eliminate humans as the source of carbon emissions. She says:

This is an example of what's called the fallacy of dumb superintelligence.[1] That is, it's a fallacy to think a machine could be 'smarter than humans in all respects', but still lack any common sense understanding of humans, such as understanding why we made the request to fix climate change.

I think this “fallacy” is a crux of disagreement about AI x-risk, by way of alignment difficulty. I've heard this statement from other reasonably well-informed risk doubters. The intuition makes sense. But most people in alignment would dismiss this out of hand as being itself a fallacy. Understanding these two positions not only clarifies the discussion, but suggests reasons we're overlooking a promising approach to alignment.

This "fallacy" doesn’t establish that alignment is easy. Understanding what you mean doesn’t make the AGI want to do that thing. Actions are guided by goals, which are different from knowledge. But this intuition that understanding should help alignment needn’t be totally discarded. We now have proposed alignment approaches that make use of an AIs understanding for its alignment. They do this by “pointing” a motivational system at representations in a learned knowledge system, such as “human flourishing”. I discuss two alignment plans that use this approach and seem quite promising.

Early alignment thinking assumed that this type of approach was not viable, since AGIs could "go foom" (learn very quickly and unpredictably). This assumption appears not to be true of sub-human levels of training in deep networks, and that may be sufficient for initial alignment.

With a system capable of unpredictable rapid improvement, it would be madness to let it learn prior to aligning it. It might very well grow smart enough to escape before you get a chance to stop its learning to perform alignment. Thus, its goals (or a set of rewards to shape them) must be specified before it starts to learn. In that scenario, the way we specify goals cannot make use of the AI's intelligence. Mitchell’s “fallacy” is itself a fallacy under this logic. An AGI that understands what we want can easily do things we very much don't want.

Bu...
Does AI governance needs a "Federalist papers" debate?
by azsantosk — 2023-10-18T21:08:26.098Z

During the American Revolution, a federal army and government was needed to fight against the British. Many people were afraid that the powers granted to the government for that purpose would allow it to become tyrannical in the future.

If the founding fathers had decided to ignore these fears, the United States would not exist as it is today. Instead they worked alongside the best and smartest anti-federalists to build a better institution with better mechanisms and with limited powers, which allowed them to obtain the support they needed for the constitution.

Is there something like a federalist vs anti-federalist debates of today regarding AI regulation? Is there someone working on creating a new institution with better mechanisms to limit their power, therefore assuring those on the other side that it won't be used a a path to totalitarianism? If not, should we start?
Metaculus Launches Conditional Cup to Explore Linked Forecasts
by ChristianWilliams — 2023-10-18T20:41:41.191Z
Link: https://www.metaculus.com/tournament/2503/

Game out scenarios and bring greater clarity to the relationships between important events in Metaculus's new Conditional Cup. Every week, flex your conditional forecasting skills on a new conditional pair.

The tournament has launched with two conditional pairs:

Learn more about forecasting conditional pairs here.
AI Safety 101 : Reward Misspecification
by markov — 2023-10-18T20:39:34.538Z

Overview

Reinforcement Learning: The chapter starts with a reminder of some reinforcement learning concepts. This includes a quick dive into the concept of rewards and reward functions. This section lays the groundwork for explaining why reward design is extremely important.
Optimization: This section briefly introduces the concept of Goodhart's Law. It provides some motivation behind understanding why rewards are difficult to specify in a way such that they do not collapse in the face of immense optimization pressure.
Reward misspecification: With a solid grasp of the notion of rewards and optimization the readers are introduced to one of the core challenges of alignment - reward misspecification. This is also known as the Outer Alignment problem. The section begins by discussing the necessity of good reward design in addition to algorithm design. This is followed by concrete examples of reward specification failures such as reward hacking and reward tampering.
Learning by Imitation: This section focuses on some proposed solutions to reward misspecification that rely on learning reward functions through imitating human behavior. It examines proposals such as imitation learning (IL), behavioral cloning (BC), and inverse reinforcement learning (IRL). Each section also contains an examination of possible issues and limitations of these approaches as they pertain to resolving reward hacking.
Learning by Feedback: The final section investigates proposals aiming to rectify reward misspecification by providing feedback to the machine learning models. The section also provides a comprehensive insight into how current large language models (LLMs) are trained. The discussion covers reward modeling, reinforcement learning from human feedback (RLHF), reinforcement learning from artificial intelligence feedback (RLAIF), and the limitations of these approaches.

1.0: Reinforcement Learning

The section provides a succinct reminder of several concepts in reinforcement learning (RL). It also disambiguates various often conflated terms such as rewards, values and utilities. The section ends with a discussion around distinguishing the concept of objectives that a reinforcement learning system might pursue from what it is being rewarded for. Readers who are already familiar with the basics can skip directly to section 2.

1.1. Primer

Reinforcement Learning (RL) focuses on developing agen...