trentmkelly/LessWrong-43k huggingface dataset:
MIRI Conversations: Technology Forecasting & Gradualism (Distillation)
by CallumMcDougall — 2022-07-13T15:55:40.348Z

This was submitted as part of UC Berkeley's distillation competition. Many thanks to everyone who gave feedback on this post – in particular Hannah Erlebach, Jamie Bernardi, Dewi Erwan, Arthur Conmy, and Peter Barnett.

Crossposted to the EA Forum.

TL;DR

Gradualism and catastrophism represent opposite ends of a spectrum of predictions about AI takeoff speeds. Gradualists (e.g. Paul Christiano) expect continuous progress without sharp discontinuities, whereas catastrophists (e.g. Eliezer Yudkowsky) expect there to be a phase transition around the point of human-level AGI.

In this excerpt from the Late 2021 MIRI Conversations, the participants apply gradualist / catastrophist models to different domains, and see how the predictions generated by these models hold up to reality. In this post, I’ll focus on three domains: historical examples, startups, and peoples’ skill distributions.

I will conclude with a short retrospective on some of the MIRI conversations, and the lessons we can draw from them about the state of discourse in the alignment community.

Introduction

Flight by machines heavier than air is impractical and insignificant, if not utterly impossible.

Simon Newcomb, 1835 - 1909

If we worked on the assumption that what is accepted as true really is true, then there would be little hope for advance.

Orville Wright, 1871 - 1948

On September 14th, 2021, Paul Christiano and Eliezer Yudkowsky met in a Discord server. Their goal was to try and hash out some of their disagreements related to AI takeoff speeds. In particular, Paul was defending his “soft takeoff” view against Eliezer’s “hard takeoff”.

In brief, the “soft takeoff” / gradualist story for AI progress looks like a continuous ramping up of investment in and capabilities of AI as the field progresses. There are no huge jumps or discontinuities. By the time AGI rolls around, the world already looks pretty crazy, with AI already having had a major impact on society, and GDP having grown at a pretty wild rate over the last few years.

In contrast, in the “hard takeoff” / catastrophist scenario, the development of AGI will be a phase change (classically described as “FOOM”). There will come a point where AIs can propel themselves from “relatively unimpressive” to “strongly superhuman levels of intelligence”, and this could all happen inside a very short timeframe, maybe even a few hours.

Why is t...
Passing Up Pay
by jefftk — 2022-07-13T14:10:04.876Z

I recently switched from

earning money to donate to

doing
something directly useful. I think this work is very important
(or I wouldn't have switched!), and if I were still earning to give
it's the kind of work I'd be excited to fund. I also think this about
Julia's work; what should this mean for our donations?

We've been donating 50%, and say we decide to continue doing that in
future years. Let's say our employers are willing to pay us a
combined $300k, so we donate $150k and have $150k left over. Then we
pay $51k in taxes and our employers pay another $22k, for a total of
$73k in income taxes. This leaves us with $99k for ourselves:

Donate 50%

Income
$300k

Donations
$150k

Employee-paid taxes
$51k

Employer-paid taxes

$22k

Post-tax post-donation pay
$99k

Since I'm equally happy for us to donate $1 as for either of our
employers to keep $1, however, another option would be for us to ask
for lower pay instead of donating. Let's say we want to keep the same
$99k in post-tax post-donations pay; what does that look like?

Donate 50%
Reduce Pay

Income
$300k
$130k

Donations
$150k
$0

Employee-paid taxes
$51k
$31k

Employer-paid taxes
$22k

$9k

Post-tax post-donation pay
$99k
$99k

Our family is equally well off in the two cases, but in the donation
case $33k goes to taxes that would otherwise go to something much more
valuable. That's 11% of what our employers would be willing to pay
us, and 1/3 of our post-tax post-donation pay. This happens because
while donations are "tax-deductible", that's only for Federal Income
Tax (10%-37% in brackets). Social Security (12.4%), Medicare (2.9%),
and State (5% for MA) taxes are all calculated on pre-donation income.

While this is more efficient, there are some reasons why this might
not be a good idea:

This only works if there isn't some other organization you
think would do more good with your money. People may overvalue their
particular organization when they would be more objective in choosing
among organizations at large. People have floated the idea that EA
should have a norm against donating to your employer.

Your employer might say they are willing to pay the larger
amount, but if it actually came down to it they wouldn't follow
through. Perhaps they think they can get someone else to do the job
for closer to what they had b

...
How could the universe be infinitely large?
by amarai — 2022-07-13T13:45:23.715Z

I ask this question in the context of thinking about infinite ethics. It seems to me that I would need to give up at least one of three beliefs about the universe in order to coherently think that it is infinitely large. They are:

(i) at some past point in time t the universe was finitely large

(ii) since t a finite amount of time has passed

(iii) since t the universe has always expanded with finite speed

To the extent that I continue to believe all three, I would have to be convinced that the universe is finite. Right?

My very layman's understanding of the big bang theory is that it implies (i) and (ii), i.e. that the universe was initially a finite matter-blob and that the universe started to expand a finite amount of time ago (~13 billion years give or take).

So, which of these assumptions are given up by e.g. cosmologists who take seriously that the universe is infinitely large? Or, put another way, which of these assumptions is least likely given our current understanding of physics?
John von Neumann on how to safely progress with technology
by Dalton Mabery — 2022-07-13T11:07:00.904Z

From a 1955 article written by von Neumann titled, "Can We Survive Technology?"

What safeguard remains? Apparently only day-to-day — or perhaps year-to-year — opportunistic measures, along sequence of small, correct decisions. And this is not surprising. After all, the crisis is due to the rapidity of progress, to the probable further acceleration thereof, and to the reaching of certain critical relationships. Specifically, the effects that we are now beginning to produce are of the same order of magnitude as that of "the great globe itself." Indeed, they affect the earth as an entity. Hence further acceleration can no longer be absorbed as in the past by an extension of the area of operations. Under present conditions it is unreasonable to expect a novel cure-all. For progress there is no cure. Any attempt to find automatically safe channels for the present explosive variety of progress must lead to frustration. The only safety possible is relative, and it lies in an intelligent exercise of day-to-day judgment […]

The one solid fact is that the difficulties are due to an evolution that, while useful and constructive, is also dangerous. Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required:
patience, flexibility, intelligence.

This sounds an awful lot like what AI Alignment has been trying to do. Granted, as Scott Alexander notes in a book review of von Neumann's biography:

This sounds suspiciously like the smartest man in the world admitting he’s not sure what to do.
Everyone is an Imposter 
by Tharin — 2022-07-13T08:46:20.950Z
Link: https://echoesandchimes.com/2022/07/everyone-is-an-imposter/

I.

I am generally considered to be a pretty smart guy.

Now to say this directly is to invite doubt and disaster. I know that. I feel it deep in my bones. It’s been drilled into me that, outside of certain domains (like rap and sports), it’s impolite to talk directly about one’s abilities. But I’m pushing through my instinctive discomfort to write this, as I think it might be worth it.

When I say I’m a “pretty smart guy”, immediately I feel compelled to clarify what I’m not saying. I’m not saying I’m the “smartest” (whatever that might mean), or that I’m made unique or special by my above-average intelligence. I’m just saying that it seems true that above-average intelligence is something I possess, and that, despite how I feel about it, probably it doesn’t function like a fearful bird, likely to fly off as soon as I glance in its direction.

The world’s provided me with a fair amount of feedback in support of this conclusion. My grades have been high since I was in prep school, staying high through my undergrad and postgrad studies. I seem to have fairly consistently placed in the top 10-20% of my cohort, depending how you draw the boundaries.

Of course, grades aren’t the only way we measure intelligence, there are lots of reasons why smart people might get bad grades, and it’s possible to get good grades and still be rather dull. Fine. But it’s not just grades that’ve led me to think this—I continue to be complimented on my creativity and my mental abilities in the working world, sometimes by people who have nothing to gain by doing so. And most of my friends (and even some of my enemies) hold me in similar regard: just the other night I hung out with someone who, after he left, thanked me for the unusually stimulating conversation.

I’m belabouring this point, awkward as it feels, because I’m hoping you’ll believe me when I say that despite my intelligence, I am, so often, so painfully stupid. Slow, dull, uninspired, and uninteresting—like a snail that, having decided to end it all, begins sliming towards salt, only to fall asleep halfway there. If you know me personally, this will not come as a surprise. When I was 13, my scout master called me “either the smartest dumb person or the dumbest smart person” he’d ever met; more than a decade later, I don’t really disagree.

Recently, when reflecting on how my intelligence and my idiocy coexist,...
Straw-Steelmanning
by Chris van Merwijk — 2022-07-13T05:48:33.099Z

I've noticed that when people are asked to "Steelman" a position, they sometimes instead do what I would call "Straw-Steelmanning". Someone can also straw-steelman without having been asked to steelman or having said that they would do so.

What is straw-steelmanning? Assume someone makes an argument X for a claim C, and you are arguing against X.

Straw-manning (bad): You replace X with a weaker argument Y and argue against that, pretending as if you have thereby refuted X.
Steel-manning (good): You replace X with a stronger argument Y which still contains the core of X and argue against that, thereby actually refuting X. (The term can also be used in a context where you are not actually arguing against the claim C).
Straw-steelmanning (bad): You replace C with an entirely different claim D and make an argument Y for it which you consider to be stronger than X, pretending as if you no longer need to argue against C.

An example which I have noticed is something like the following:

"Can you steelman the position that future AI systems will pose an existential risk?"
"Well, while we whould not take these hollywood movie plots seriously, there are real social problems with AI that we have to deal with. AI will potentially cause massive inequality because entire industries will be automated and a small amount of corporation will own the AI tools that facillitate that. AI engineers will earn large wages, while demand for other professions stagnates. Moreover, we need to worry about biases in AI systems, because [etc, proceeds to argue more]"

This is a straw-steelman, because they have

Simply bypassed the original claim, replacing it with a different claim that they already agreed with.
Proceeded to argue for that claim, ignoring the original.
Alien Message Contest: Solution
by DaemonicSigil — 2022-07-13T04:07:06.010Z

Thanks to everyone who participated in the Alien Message contest, and congratulations on successfully being able to decode the message. I gave a time limit of 2 weeks at the start of the contest, but you completely solved it in a mere 2 days.

Solution

Here is the code that generated the message:

/*
* generate_message.cxx
*
* This code operates a continuous tag-system, and writes the resulting
* data to a file.
*
*/

#include <iostream>
using std::cout, std::endl;
#include <fstream>
using std::ofstream;
#include <vector>
using std::vector;
#include <cmath>
using std::rint;

using std::string;

ofstream& open_wb(string filename) {
return *new std::ofstream(filename, std::ios::binary);
}

double abs(double x) {
return x >= 0 ? x : -x;
}

class TagSystem {
public:
int pos{ 0 };
vector<double> data;

TagSystem(double x, double y, double z) {
data.push_back(x);
data.push_back(y);
data.push_back(z);
}

void iterate() {
uint readsize = 1;
if (data.size() - pos >= 2 && abs(data[pos]) < 2 && abs(data[pos+1]) < 2) {
readsize = 2;
}
if (data.size() - pos < readsize) {
cout << "Tag system halted at position " << pos << endl;
}
switch (readsize) {
case 1:
data.push_back(data[pos] / 2);
break;
case 2:
data.push_back(data[pos]*data[pos+1] + 1.0);
data.push_back(data[pos] - data[pos+1]);
data.push_back(data[pos+1] - data[pos]);
break;
}
pos += readsize;
}
};

void write_double_to_file(double x, ofstream& file) {
file.write((char*)&x, sizeof(double));
}

int main(int argc, char **argv)
{
cout << "Writing file..." << endl;
ofstream& fout = open_wb("./out.bin");
TagSystem ts(1.3, 2.1, -0.5);
for (int i = 0; i < 1000; i++) {
ts.iterate();
}
for (double elem : ts.data) {
write_double_to_file(elem, fout);
cout << elem << endl;
}
fout.close();
return 0;
}

The message is generated by a continuous version of a tag system (actually a slight generalization, since the number of items consumed at each step can vary). Instead of a discrete set of symbols from an alphabet, the items are actually numbers, and the relationship between the numbers consumed and the numbers written is defined by a ...
What is wrong with this approach to corrigibility?
by Rafael Cosman — 2022-07-12T22:55:22.342Z

Essentially: a button that when you press it kills the AI but instantly gives it reward equal to its expected discounted future reward. And to be clear, this is the AI's estimate of its expected discounted future reward, not some outside estimate.

(This is in the interest of asking dumb questions and learning in public. And I'm taking the specific narrow definition of corrigibility which is just an AI letting you turn it off.)

Thoughts: I've heard Dr. Christiano mention things like "precisely balanced incentives" (which I think uses to describe the weaknesses of this and similar approaches), but I don't see why this would be particularly difficult to balance given that this number is just an explicit float inside many RL models. Some issues I do see:

Such an agent would have no incentive to create corrigible child-agents
Such an agent would have no incentive to preserve this property while self-modifying

But I am probably missing a bunch of other issues with this general approach? Responses in the form of links to relevant papers are welcome as well.

Thanks,

Raf
Acceptability Verification: A Research Agenda
by David Udell — 2022-07-12T20:11:34.986Z
Link: https://docs.google.com/document/d/199Lkh78UA2uI9ljLEy_aWR8RBetQLRO6Kqo3_Omi1e4/view

This Google doc^ is a halted, formerly work-in-progress writeup of Evan Hubinger’s AI alignment research agenda, authored by Evan. It dates back to around 2020, and so Evan’s views on alignment have shifted since then.

Nevertheless, we thought it would be valuable to get this posted and available to everyone working in alignment!

In it, Evan outlines the following alignment scheme:

We should bake transparency tools into the loss function we’re training powerful models on, grading the model on its internal cognitive processes as well as on external behavior. We start by initializing a relatively dumb but non-deceptive model. We scale up the model, selecting against any model that isn’t demonstrably acceptable to a transparency-tool-assisted overseer.

While Evan doesn’t expect this approach to be robust against deceptively aligned models, the hope is that we can define a notion of an 'acceptability predicate' such that, if we start with a dumb aligned model and scale from there, grading on cognitive processes as well as behavior, no model on that trajectory in model space will ever become deceptive in the first place. That is, before a model can be updated to become deceptive in this training process, it hopefully first must be updated to become unacceptable and non-deceptive. We can therefore update away from all merely unacceptable models as they appear, and thereby never instantiate a deceptive model in the first place.

At the time of this doc’s writing, the leading candidate for an adequate acceptability predicate was 'demonstrably myopic.' One plausible account of 'myopia' here is “return the action that your model of HCH would return, if it received your inputs.”

Since writing up this agenda, some things that Evan has updated on include:

Understanding myopia carefully is less important relative to just

improving our transparency and interpretability capabilities.

Scaling interpretability via training transparency will require us to go through

a bunch of independent transparency tech levels

along the way.
Charting paths through model space is

not necessarily a good way to think about ML inductive biases.

Training stories

are the right way to think about alignment generally.
The best way to think about argmax HCH is to think of it as a human imitator in the

ELK

sense, but with the human replac

...
Progress links and tweets, 2022-07-12
by jasoncrawford — 2022-07-12T15:30:14.667Z
Link: https://rootsofprogress.org/links-and-tweets-2022-07-12

Links

Patrick Collison on universal covid vaccines: we could have them very soon, if it weren’t for bureaucratic roadblocks
“Current funding organizations cluster heavily around a few management styles, warping the research ecosystem” (Ben Reinhardt)
The Tony Blair Institute for Global Change announces their Progress Fellows Programme 2022 (h/t @benedictcooney)
“A jeremiad is a long literary work… in which the author bitterly laments the state of society and its morals in a serious tone of sustained invective, and always contains a prophecy of society’s imminent downfall.” (Sad and amusing to me that there is a word for this!)

Queries

What would you change the h-index to? (@eric_is_weird)
Who should be in a “Most Interesting People in the World” speaker series? No one has suggested me yet… (@tom_morganKCP)

Retweets

World’s Fair posters from DALL-E (@arbesman)
The thinking machines allow us to become more human (@tszzl)
Why did we stop reclaiming land for cities? (@MTabarrok)
Response to Blake Richards: AGI, generality, alignment, & loss functions
by Steven Byrnes — 2022-07-12T13:56:00.885Z

Blake Richards is a neuroscientist / AI researcher with appointments at

McGill

&

MiLA

. Much of his recent work has involved making connections between machine learning algorithms and the operating principles of the cortex and hippocampus, including theorizing about how the neocortex might accomplish something functionally similar to backprop. (Backprop itself is not biologically plausible.) I have read lots of his papers; they're always very interesting!

Anyway, Blake recently gave a nice interview on Michaël Trazzi’s podcast “The Inside View”. Episode and transcript at:

https://theinsideview.ai/blake

.

Blake said a lot of things in the interview that I wholeheartedly agree with; for example, I would echo pretty much everything Blake said about scaling. But Blake & I strongly part ways on the subject of (what I call) artificial general intelligence (AGI). (Blake doesn’t like the term “AGI”, more on which below.) So that’s what I’ll talk about in this post.

1. Two mental images for thinking about an AI. Or: My lips say “human-level”, but my heart says “a fancier version of GPT or AlphaFold”

I’ll jump right into a key issue that I think is lurking below the surface. Here are two mental images for thinking about an AI system (I’ll be returning to this table throughout the post)

Throughout the interview, even while Blake was saying things like “as general as a human”, I got a strong feeling that what he was imagining was the thing in the left column. But if it was actually as general as a human, it would look like the thing in the right column. For example, if an AI is actually as general as a (very competent) human, and you ask it to fold proteins, and it doesn’t know how, then it downloads AlphaFold. Or if AlphaFold doesn’t exist, it starts reading papers and textbooks, and goes about inventing AlphaFold!

(In Section 3 below, I’ll go over an example of one of the many things Blake said, in the interview, that gave me the strong impression that he was engaging in left-column thinking. If I’m wrong, then so much the better—we can just skip to the next section!)

Anyway, left-column thinking is extremely natural, because that’s what ML looks like today. But we know that it’s technically possible to make AIs that require right-column thinking—Blake and I are in full agreement that anything the human brain ...
Three Minimum Pivotal Acts Possible by Narrow AI
by Michael Soareverix — 2022-07-12T09:51:24.077Z

The Alignment problem appears extremely concerning, so here's my attempt to explain the most likely way we could live in an aligned future.

The Perfect Tool AI

Imagine a Tool AI, something like DALLE-2 or GPT-3. It's a simple AI relatively speaking, and it does one of three things exceptionally well:

-A Perfect Creator. Manufactures and controls nanotechnology to rearrange atoms in a given space. If you tell this tool to destroy an area, it will do so by using self-replicating nanotechnology to disassemble that area into basically whatever you want. You would basically have a perfect, unstoppable factory that can produce anything from anything. In this case, you can tell the AI to disassemble the components of every AI lab on earth, create human-life extending technologies, and you could live as an immortal god over humanity as you slowly put together a team of AI Alignment researchers to build an aligned AI over the centuries.

Likelihood of creation: Very low. Nanotech is a hard problem to solve, and controlling that nanotech is just as difficult.

-A Perfect (Computer) Virus. Imagine a worm like StuxNet, but it contains a simple script that connects it to a higher AI controller once it takes over a computer, and then creates new controllers once it has sufficient compute power. With each computer it infects, it gets smarter and more powerful, and most importantly, better at infecting computers. Human security teams cannot keep up. The Internet is basically an ecosystem, the same way Earth was before the explosion of cyanobacteria, and it is ripe for the taking by an advanced AI. Whoever is behind this virus can now control every computer on Earth. If they're focused on AI Alignment, then they can easily destroy any computers connected to an AI Lab and slowly assemble a team of alignment engineers to solve AGI Alignment as well as allowing compute power for other tasks like life-extension technology.

Likelihood: Medium. I am honestly surprised we haven't seen AI powered security breaches. It seems feasible.

-A Perfect Manipulator. GPT-3 is already surprisingly convincing, but what if you trained an AI to be perfectly convincing? I have seen incredibly charismatic people change another person's political beliefs over the span of a conversation. There is some string of text that could likely convince you of just about anything, and if an AI gets good enou...