trentmkelly/LessWrong-43k huggingface dataset:
Stop pushing the bus
by cousin_it — 2023-03-31T13:03:45.543Z

With the recent proposals about moratoriums and regulation, should we also start thinking about a strike by AI researchers and developers?

The reasoning I imagine as follows. AI capability is now growing really fast, and toward levels that will strongly affect the world. And AI safety lags behind. (A minute ago I used a ChatGPT jailbreak to get instructions for torturing a pregnant woman, that's the market leader performance for you.) And finally, I want to make the argument that working on AI capability while it is ahead of AI safety, is "pushing the bus".

Here's the metaphor, a bunch of people including you are pushing a bus full of children toward a precipice, and you're paid for each step. In this situation would you really say "oh I have to keep pushing, otherwise others will get all the money"? It's not like they'll profit from it! Their children will die, along with everyone else's! So there's no game theoretic angle, you can just make the decision alone, to stop pushing the frigging bus.

To clarify, working on AI isn't always bad. It could lead to a wonderful future for humanity. But when AI safety is behind, as now, then working on AI capability is pushing the bus. There's no good justification for it.

Hence the strike. Not by leadership but by AI researchers and developers themselves. I imagine a desk plaque saying "while AI safety lags behind AI capability, I refuse to work on AI capability". That's the start condition, and it also tells you when to stop (when safety catches up with current capability, which means not just stopping saying bad things, but for stronger AIs also safe and benevolent behavior more generally). And it's also the restart condition if safety starts lagging behind again.
Proposal on AI evaluation: false-proving
by ProgramCrafter — 2023-03-31T12:12:15.636Z

Currently, one of arguments against deploying powerful non-agentic AIs is that they can deceive you: you ask for plan for doing something, AI creates one and then unexpectedly gets power as result of that plan.

We can ask AI to provide proof or reasoning why its plan works. However, it can reason as fictional characters do (e.g. choose the outcome producing the most emotions; mostly applicable to LLMs), or it can include false statements in the speech. So there should be a way to find false statements (logic or factual errors).

Usually people (most importantly, pupils and students) are taught on true proofs but not on false ones. This can lead to thinking "if something looks like it's based on logic, then the result is true". So at the moment people are not very effective in searching reasoning flaws.

A possible solution to that is to introduce false-prove tasks. Such tasks shouldn't be long. Also, the conclusion could be true, to mix things up a bit.

Task: prove that 2 + 2 = 4.

Lemma: Any two natural numbers (a and b) are equal.

Let's prove that lemma by induction on max(a, b). The base case is a = b = 1, where the statement is true.

Now, we have max(a, b) > 1. Let's reduce both numbers by one and apply the induction hypothesis. Because a-1 = b-1, a=b follows, so the lemma is correct.
---

3 + 1 = 4, as 4 is the number following 3

3 = 2 by lemma

1 = 2 by lemma

After substituion we get 2 + 2 = 4. Q.E.D.

And students' own research should probably be rewarded, even leading to wrong results but where flaw is not obvious.
Is AI going to kill us all? FAQ reply to Yudkowsky, etc
by DavidLeoVeksler@gmail.com — 2023-03-31T12:10:24.314Z
Link: https://futureoflife.substack.com/p/is-ai-going-to-kill-us-all-a-faq

If you haven't read the letters from The Future of Life Institute and Eliezer Yudkowsky, do that first:

https://futureoflife.org/open-letter/pause-giant-ai-experiments/

https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

Is AI going to kill us all?

Quite possibly. We shouldn't panic, but even a 1% risk of ending the world deserves a massive intervention. The risk is much higher than 1%, but experts disagree, and there's no way to know who is right. More on this later.

Should we, therefore, pause large-model AI research?

Yes, if it were possible.

Is it possible?

Absolutely not. The cat is out of the bag, anyone who understands game theory should grasp that.

Should we try to pause research anyway?

A poorly executed pause may do more harm than good. If the "good guys" stop AI research, we risk letting "bad guys" have superpowers, or just very dangerous toys.

So are we doomed?

Probably not. There are huge unknowns, and the apocalypse is far from certain.

Why would a research pause not work?

1: Large models are overhyped, yet they have proven extremely valuable, and their value is likely to increase in improved versions. Imagine a machine that prints gold. If most people refrain from improving their machines, those who improve their printers in secret obtain even more value. Any person or nation which refuses to go along with the ban obtains tremendous value. Even a global police state cannot stop people from running code on every single computer. It's preferable for leading public entities to dominate research over million of unknowns

2: While GPT-4 requires massive resources, existing open-source models can be run on a consumer GPU. Improvements in AI models and hardware will continue to narrow this gap. A ban on large model AI research is a massive stimulus to optimizing models to run with fewer resources.

Are there any reasons to be optimistic?

Yes. I believe our options are either quick, global death or a (literally) unimaginable explosion in human flourishing. There is no in-between. Skynet won't be hunting us down, so at least the end, if it comes, will be quick.

While no one is suggesting that AI is likely to be malevolent, there are reasons to think that AI trained on comprehensive large language models may be benevolent. While humans have no idea how to teach AI ethics or values, the models work by rewarding AI for being "helpful." The nature of these m...
Harry Potter and the Data Centers of Doom
by RomanS — 2023-03-31T10:42:20.282Z

The atmosphere was thick with intellectual tension in a small, concealed room at Hogwarts School of Witchcraft and Wizardry. Three prominent figures, each a beacon of knowledge in their own right, were engaged in an intense discussion. Harry Potter, the young prodigy, Alastor "Mad-Eye" Moody, the battle-hardened auror, and Professor Quirrell, the enigmatic Defense Against the Dark Arts teacher, were gathered around a dusty, age-old table. The matter at hand was of utmost importance, and the urgency in their voices was palpable.

Harry's eyes gleamed with a mixture of apprehension and excitement as he elaborated on the technology behind the GPT-4, a highly advanced artificial intelligence system. He explained how it had the potential to revolutionize the world, but also posed an existential risk to humanity if it were to become a superintelligent AGI.

"And so you see," Harry concluded, "the very same advancements that make GPT-4 so promising could also lead to the end of us all. It's a fine line we're treading, and I fear that we may be approaching the point of no return."

Moody's magical eye swiveled about in its socket, scrutinizing the room for any signs of eavesdropping. His paranoia was well-founded; the stakes were incredibly high. "We need to consider every possible angle, Potter. The problem with bombing data centers is that it could send the wrong message. We'd be perceived as terrorists, and the backlash might push research underground, where it would be even harder to control."

Professor Quirrell, a cunning smile forming on his face, interjected, "On the contrary, Mr. Moody, it could be precisely the shock the world needs to understand the gravity of the situation. Researchers might think twice before pursuing such dangerous developments. We could potentially slow down or halt the progress of AI research altogether, buying us precious time to devise a proper solution."

Harry, intrigued by Quirrell's argument, nodded thoughtfully. "You have a point, Professor. Sometimes, drastic measures are needed to prevent an even greater catastrophe. But the question is: would the ends justify the means?"

Moody, ever the pragmatist, countered, "I agree that something must be done, but there's more to consider. If we do proceed with such an action, we must ensure that the collateral damage is minimal, and that we don't inadvertently accelerate the development of AGI. We...
Widening Overton Window - Open Thread
by Prometheus — 2023-03-31T10:03:59.517Z

Major news outlets have published articles about the Future of Life Institute's Open Letter. Time Magazine published an opinion piece by Elizer Yudkowsky. Lex Friendman featured EY on his podcast. Several US Members of Congress have spoken about the risks of AI. And a Fox News reporter asked a what the US President is doing to combat AI x-risk at a White House Press Conference.

Starting an Open Thread to discuss this, and how best to capitalize on this sudden attention.

Links:

WH Press Conference: https://twitter.com/therecount/status/1641526864626720774

Time Magazine: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

FLI Letter: https://futureoflife.org/open-letter/pause-giant-ai-experiments/
It Can't Be Mesa-Optimizers All The Way Down (Or Else It Can't Be Long-Term Supercoherence?)
by Austin Witte — 2023-03-31T07:21:57.054Z

Epistemic status: After a couple hours of arguing with myself, this still feels potentially important, but my thoughts are pretty raw here.

Hello LessWrong! I’m an undergraduate student studying at the University of Wisconsin-Madison, and part of the new Wisconsin AI Safety Initiative. This will be my first “idea” post here, though I’ve lurked on the forum on and off for close to half a decade by now. I’d ask you to be gentle, but I think I’d rather know how I’m wrong! I’d also like to thank my friend Ben Hayum for going over my first draft and WAISI more broadly for creating a space where I’m finally pursuing these ideas in a more serious capacity. Of course, I’m not speaking for anyone but myself here.

With that said, let’s begin!

The Claim

I think there are--among others--two pretty strong, concerning, and believable claims often made about the risks inherent in AGI.

First, optimizers searching for models that do well at some goal/function/etc. might come across a model which has an internal optimization process which seeks to optimize towards some other, distinct goal. This is, of course, entirely plausible: humans exist, we can fairly represent them as optimizers, they derived from natural selection which can be fairly represented as an optimizer, and their “goals” do not perfectly align. These are mesa-optimizers.
Second, the idea that the values/goals/utility function of sufficiently strong optimizers (i.e, a superintelligent AGI) will tend to cohere. In other words, stronger optimizers will tend towards more strongly optimizing for only one non-contradictory utility function (and they will do this via expected utility maximization or something like it, but I don't think this post depends on that). This is coherence.

Maybe putting these two claims right next to each other makes what I'm about to suggest very obvious, and maybe it's already something that has been discussed. However, I'm making this post because it wasn't obvious to me until I thought it up and these two discussions don't seem to come up alongside one another very often. While there have been a number of posts for and against the idea of coherence theorems, I haven't seen anything making this observation explicitly.

To make the claim clear: If sufficiently strong optimizers tend towards optimizing for a coherent utility function, that implies they will tend towards optimization...
Who is Slimrock Investments Pte. Ltd. and how are they related to Lightcone Infrastructure?
by adamrat — 2023-03-31T06:54:06.844Z

Slimrock Investments Pte. Ltd. is listed on the Alameda County Recorder's records as associated with Lightcone's recent purchase of the Rose Garden Inn in Berkeley.

"Assignment of Rents" implies that they are the lender who provided the capital to purchase the property. There is not much information about them on the internet. They appear to be a holding company incorporated in Singapore.

However, I was able to find them in a list of creditors in the bankruptcy proceedings for FTX:

What is Lightcone's relationship to Slimrock, and is there any specific reason that the purchase of the Rose Garden Inn was financed through them rather than a more mundane/pedestrian lender?
Against an AI Research Moratorium
by Eric Raymond — 2023-03-31T00:53:42.137Z

All the smart people agitating for a 6-month moratorium on AGI research seem to have unaccountably lost their ability to do elementary game theory. It'a a faulty idea regardless of what probability we assign to AI catastrophe.

Our planet is full of groups of power-seekers competing against each other. Each one of them could cooperate (join in the moratorium) defect (publicly refuse) or stealth-defect (proclaim that they're cooperating while stealthily defecting). The call for a moratorium amounts to saying to every one of those groups "you should choose to lose power relative to those who stealth-defect". It doesn't take much decision theory to predict that the result will be a covert arms race conducted in a climate of fear by the most secretive and paranoid among the power groups.

The actual effect of a moratorium, then, would be not be to prevent super-AGI; indeed it is doubtful development would even slow down much, because many of the power-seeking groups can sustain large research budgets due to past success. If there's some kind of threshold beyond which AGI immediately becomes an X-risk, we'll get there anyway simply due to power competition. The only effect of any moratorium will be to ensure that (a) the public has no idea what's going on in the stealth-defectors' labs, and (b) control of the most potent AIs will most likely be achieved first by the most secretive and paranoid of power-seekers.

A related problem is that we don't have a college of disinterested angels to exert monopoly control of AI, or even to just trust to write its alignment rules. Pournelle's Law. "Any bureaucracy eventually comes to serve its own interests rather than those it was created to help with," applies; monopoly controllers of AI will be, or will become, power-seekers themselves. And there is no more perfect rationale for totalitarian control of speech and action than "we must prevent anyone from ever building an AI that could destroy the world!" The entirely predictable result is that even if the monopolists can evade AGI catastrophe (and it's not clear they could) the technology becomes a boot stomping on humanity's face forever.

Moratorium won't work. Monopoly won't either. Freedom and transparency might. In this context, "Freedom" means "Nobody gets to control the process of AI development," and "transparency" means "All code and training ...
Are quantum indeterminacy and normal uncertainty meaningfully distinct?
by eapi — 2023-03-30T23:48:46.067Z

Let's say I've got an electron, either spin-up or spin-down. Let's make it easy and say its state is just |↑⟩+|↓⟩. I do an observation and then my Everett branch gets split in two: there's a version of me with exactly half the measure/amplitude/reality-fluid of my pre-observation branch which observes the electron as spin-up, and another version which observes the electron as spin-down. Standard stuff.

Compare and contrast with me flipping a coin. I'm uncertain whether it will land heads-up or tails-up, and I'm fairly sure the system is chaotic enough and symmetric enough and I know little enough about it that betting anything other than 0.50/0.50 would be a bad idea. Yet, post-flip, there's only one version of me: if I see the coin heads-up I'm pretty confident there's no version of me out there with meaningful measure who sees it tails-up.

The quantum indeterminacy situation seems lower-variance to me, because post-observation you have a built-in "hedge": your other branches. If I make a bet on the observation, gaining $1 on spin-up/heads and losing $1 on spin-down/tails, and I value my balance linearly in dollars and linearly in (relative) measure across branches, then:

In the quantum-indeterminacy electron-observing case, my "observer experience portfolio" deterministically goes from (measure 1,$0) to (0.5,$1)+(0.5,−$1).
In the normal-uncertainty coin-observing case, my portfolio goes from (1,$0) to one of (1,$1) or (1,−$1), and I just don't know which one.

The difference is clearer to me if you crank up the stakes. Let's take the old classic of "flip a coin/observe an electron, if it's heads/|↑⟩ you get an additional copy of Earth and if it's tails/|↓⟩ you lose your one existing copy". Leaving aside the details of how valuable an additional copy of Earth is, it certainly feels to me like there's a difference between:

In the quantum-indeterminacy case, half of your current measure gets evaporated and half goes on experiencing something interesting.
In the normal-uncertainty case, you run the risk of there being no measure of you anywhere.

My questions are:

Is that description of the two phenomena (quantum indeterminacy vs. normal uncertainty) roughly correct, taking as given the Everett MWI?
Is that description of the differences in variance between the two roughly correct?
I don't reca

...
Deference on AI timelines: survey results
by Sam Clarke — 2023-03-30T23:03:52.661Z

Crossposted to the EA Forum.

In October 2022, 91 EA Forum/LessWrong users answered the

AI timelines deference survey

. This post summarises the results.

Context

The survey was advertised in this

forum post

, and anyone could respond. Respondents were asked to whom they defer most, second-most and third-most, on AI timelines. You can see the survey

here

.

Results

This spreadsheet

has the raw anonymised survey results. Here are some plots which try to summarise them.[1]

Simply tallying up the number of times that each person is deferred to:

The plot only features people who were deferred to by at least two respondents.[2]

Some basic observations:

Overall, respondents defer most frequently to themselves—i.e. their “inside view” or

independent impression

—and Ajeya Cotra. These two responses were each at least twice as frequent as any other response.
Then there’s a kind of “middle cluster”—featuring Daniel Kokotajlo, Paul Christiano, Eliezer Yudkowsky and Holden Karnofsky—where, again, each of these responses were ~at least twice as frequent as any other response.
Then comes everyone else…[3] There’s probably something more fine-grained to be said here, but it doesn’t seem crucial to understanding the overall picture.

What happens if you redo the plot with a different metric? How sensitive are the results to that?

One thing we tried was computing a “weighted” score for each person, by giving them:

3 points for each respondent who defers to them the most
2 points for each respondent who defers to them second-most
1 point for each respondent who defers to them third-most.

If you redo the plot with that score, you get

this plot

. The ordering changes a bit, but I don’t think it really changes the high-level picture. In particular, the basic observations in the previous section still hold.

We think the weighted score (described in this section) and unweighted score (described in the previous section) are the two most natural metrics, so we didn’t try out any others.

Don’t some people have highly correlated views? What happens if you cluster those together?

Yeah, we do think some people have highly correlated views, in the sense that their views depend on similar assumptions or arguments. We tried plotting the results using the following basic clusters:

Open Philanthropy[4] cluster = {Ajeya Cotra, Holden Karnofsky, Paul Christiano, Bioanchors}
MIRI

...
AI #5: Level One Bard
by Zvi — 2023-03-30T23:00:00.690Z

This was quite the week.

OpenAI announced that GPT-4 was now offering plug-ins, including Wolfram Alpha and one that hooks up to all of your accounts including your email. GPT-4 had been out for over a full week, so this was a clear, safe, deliberate pace.

Microsoft Research published a paper claiming ‘sparks of artificial intelligence’ in GPT-4.

Tyler Cowen called on us to go ahead building more advanced AIs, because of radical agnosticism, and besides things are going to get disrupted anyway and what about China? Link goes to my in-line response. Scott Alexander offers a less polite one.

We had an open letter from FLI calling for a moratorium on training models more powerful than GPT-4, which I covered earlier today.

We had Eliezer Yudkowsky in Time saying that letter didn’t go far enough, which I will talk about next week. Eliezer also went on the Lex Fridman podcast, I’ll talk about that next week too.

Then there’s the tons of other stuff. That’s what the weekly post is all about.

Table of Contents

Table of Contents

Law of Undignified Failure. Can we not turn our AIs into agents? No? Well, f***.

Bard Tells Tall Tales. Google’s Bard is here, it’s fun. Why does it suck?

Welcome Back Code-DaVinci-002. OpenAI cancels then un-cancels old model.

The Art of the Jailbreak. It’s Greek to me.

Bryan Caplan Does Post-Mortem On Losing Public Bet. It is possible.

The Ancient Enemy, Reinforcement Learning From Human Feedback. It kills creativity. Did you know we use it on our own children?

They Took Our Jobs. Maybe you should have bought the insurance.

Evaluations of GPT-4. A few late ones.

Trust Me, I’m an AI. Sure, give the AI all your data and passwords. Why not?

Llama Is This So Easy. Smaller models everywhere, and you can improve them by having them check and reflect on their work.

Language Models Offer Mundane Utility. A continuing series.

It Turns Out You Can Do This. Drumming up business, GPT-4-style.

Fun With Image and Sound Generation. Easier to do every week.

A Thousand Household Uses. GPT-4 wins on the Alternative Uses Test.

Bill Gates on AI Impact. Going over his thoughts.

AI Gives Requested Information. Why did you request it?

In Other AI News. There is always other news.

Cheaters Gonna Cheat Cheat Cheat Cheat Cheat. Not like you can catch them.

The FTC Cometh. You’re on notice, buster.

I Gi

...
Eliezer's Videos and More
by Johannes C. Mayer — 2023-03-30T22:16:30.269Z

I like watching videos of Eliezer talking and explaining things. So here is a list of videos I have discovered so far. I have not tried to make this list exhaustive. If you know of any more videos, please post them in the comments.

Eliezer Yudkowsky - Less Wrong Q&A Playlist

The main Eliezer Yudkowsky youtube playlist sorted after the publication date (not quality).

Here is also a playlist with videos featuring Nate.