trentmkelly/LessWrong-43k huggingface dataset:
Convince me that humanity is as doomed by AGI as Yudkowsky et al., seems to believe
by Yitz — 2022-04-10T21:02:59.039Z

I’ve been very heavily involved in the (online) rationalist community for a few months now, and like many others, I have found myself quite freaked out by the apparent despair/lack of hope that seems to be sweeping the community. When people who are smarter than you start getting scared, it seems wise to be concerned as well, even if you don’t fully understand the danger. Nonetheless, it’s important not to get swept up in the crowd. I’ve been trying to get a grasp on why so many seem so hopeless, and these are the assumptions I believe they are making (trivial assumptions included, for completeness; there may be some overlap in this list):

AGI is possible to create.
AGI will be created within the next century or so, possibly even within the next few years.
If AGI is created by people who are not sufficiently educated (aka aware of a solution to the Alignment problem) and cautious, then it will almost certainly be unaligned.
Unaligned AGI will try to do something horrible to humans (not out of maliciousness, necessarily, we could just be collateral damage), and will not display sufficiently convergent behavior to have anything resembling our values.
We will not be able to effectively stop an unaligned AGI once it is created (due to the Corrigibility problem).
We have not yet solved the Alignment problem (of which the Corrigibility problem is merely a subset), and there does not appear to be any likely avenues to success (or at least we should not expect success within the next few decades).
Even if we solved the Alignment problem, if a non-aligned AGI arrives on the scene before we can implement ours, we are still doomed (due to first-mover advantage).
Our arguments for all of the above are not convincing or compelling enough for most AI researchers to take the threat seriously.
As such, unless some drastic action is taken soon, unaligned AGI will be created shortly, and that will be the end of the world as we know it.

First of all, is my list of seemingly necessary assumptions correct?

If so, it seems to me that most of these are far from proven statements of fact, and in fact are all heavily debated. Assumption 8 in particular seems to highlight this, as if a strong enough case could be made for each of the previous assumptions, it would be fairly easy to convince most intelligent researchers, which we don’t seem to observe.

A historical example which bears some simila...
Emotionally Confronting a Probably-Doomed World: Against Motivation Via Dignity Points
by TurnTrout — 2022-04-10T18:45:08.027Z

This article was written in ignorance of the alignment community’s reaction to Eliezer’s “Death with Dignity” post. The first part of this article responds to how I suspect some people reacted to that post, while the second part is my take on the post itself.

I write against defeatism; I write against decline; I write against that internal slumping that sneaks in on the coat-tails of bad news. I do not dispute the balance of evidence—are we doomed, or not? Let us simply assume that we live in a relatively doomed world: It’s very improbable that we solve AI alignment in time.

What next?

We have been taught what comes next, in this kind of story. Since, by assumption, we won’t receive a “happily ever after”, we infer we are in a tragedy. Realizing this, we are disappointed and sad. The fellowship breaks and scatters, its once-proud and vibrant members downtrodden. And then occurs a miracle which we could have turned to our advantage, but for our civilizational incompetence and our own stupidity—smart enough to build AI, yet too dumb to align it. And then the laugh track plays. And then we die! The end!

As AI_WAIFU said: “Fuck. That. Noise.”

We do not live in a story. We can, in fact, just assess the situation, and then do what makes the most sense, what makes us strongest and happiest. The expected future of the universe is—by assumption—sad and horrible, and yet where is the ideal-agency theorem which says I must be downtrodden and glum about it?

Against retreat

In response to how I suspect some people reacted to "Death with Dignity."

Suppose we have ten years left. Some may consider a very natural response—to retreat. Spend the remaining time with your family and your friends, working whatever job you want—just do whatever.

But... Um... This sounds like an awful plan? I’d feel like a lazy bum, like a cornered animal. And for those of us doing direct work, this plan sounds stupid. It sounds like a great way to throw away worlds where we would receive a miracle, and yet couldn’t do anything about it anymore because we all went home.

How would I like to see those ten years spent? I’m looking for plans which involve playing to win, which are fun and edifying, which would make me proud to be me and to be part of this community.

Ladies and gentlemen, let me mention a concept called instrumental convergence. It’s something I like thinking about sometimes. In many situat...
Does non-access to outputs prevent recursive self-improvement?
by Gunnar_Zarncke — 2022-04-10T18:37:54.332Z

Most existing large-scale models don't seem to have access to their outputs (except in the sense that some of their outputs show up in the inputs of the next generation of the model). Does this prevent recursive self-improvement - or at least limit the speed to the generation time between models?

Which existing models have access to their outputs? Conversational models like GTP-N do have access to their output but it goes thru a human conversational partner and are thus limited to the speed of that partner - except in the cases where two models have been hooked up to each other. Most cognitive architectures that try to model human thought have such access but simple input-output models e.g, for image generation don't.
A Brief Excursion Into Molecular Neuroscience
by Jan — 2022-04-10T17:55:10.174Z
Link: https://universalprior.substack.com/p/a-brief-excursion-into-molecular?s=w

TL;DR: Poking fun at complicated nomenclature (p̴͓̂͘á̴̻͌s̶̻̗͋͑ṯ̸̹̈́̌à̶̳), identifying a pretty important signaling factor in the brain, and arbitrating (part of) the Guzey/Mendonça debate.

Previously in this series:

How to build a mind - neuroscience edition

.

The Emperor's New Ǫ̵͎͊G̶̦̉̇l̶͉͇̝̽͆̚i̷͔̓̏͌c̷̱̙̍̂͜k̷̠͍͌l̷̢̍͗̃n̷̖͇̏̆å̴̤c̵̲̼̫͑̎̆

(Ǫ̵͎͊G̶̦̉̇l̶͉͇̝̽͆̚i̷͔̓̏͌c̷̱̙̍̂͜k̷̠͍͌l̷̢̍͗̃n̷̖͇̏̆å̴̤c̵̲̼̫͑̎̆ is an intentionally garbled version of the word O-GlcNAc. The reasons for garbling it will hopefully become clear while reading this essay!)

Flashback: It's June 7th, 2017. The transformer architecture

will drop on the arxiv

in 5 days, Hugh Hefner will die in less than 3 months[1], and the computational neuroscience community is having a field day on Twitter. It started with this tweet from

Michael Nitabach

from Yale School of Medicine complaining about opaque jargon in theoretical papers:

(33 likes is a lot on neurotwitter.)

There was a bit of goofy back and forth about the (un)importance of mathematical vocabulary in neuroscience, some attempts at explaining the term, and eventually the call for a workshop on "Manifold-splaining", which happened half a year later[2].

There are some things to be said here about how we use terminology[3] or what manifolds actually are[4], but I'm bringing this up because I distinctly[5] remember an offhand comment that someone made at that workshop. The comment was that it’s unfair that experimentalists get to write things like "

O-GlcNAc signaling entrains the circadian clock by inhibiting BMAL1/CLOCK ubiquitination,

" and nobody gives them hell for writing their papers in Klingon. Or at least

they are not getting

enough hell for it.

When I was first confronted with the

alphabet soup

that calls itself molecular neuroscience, I was awed by my colleagues, who appeared to be fluent in speaking p̴͓̂͘á̴̻͌s̶̻̗͋͑ṯ̸̹̈́̌à̶̳. It took me a year or two to figure out that they also do not know what most of the terms mean - and for most abbreviations, there are only a handful of people in the world who can decode them. But nobody likes to

acknowledge that they don't know what things mean publicly

. Therefore everybody nods along when they read Ǫ̵͎͊G̶̦̉̇l̶͉͇̝̽͆̚i̷͔̓̏͌c̷̱̙̍̂͜k̷̠͍͌l̷̢̍͗̃n̷̖͇̏̆å̴̤c̵̲̼̫͑̎̆. So it's fine if you don't know what it means. Almost nobody does[6].

Let’s talk about BDNF.

Perhaps you are the kind of person that takes the prece...
Finally Entering Alignment
by Ulisse Mini — 2022-04-10T17:01:36.898Z

Background

Reading
Death With Dignity
combined with recent advancements shortening my timeline
has finally made me understand on a gut level that
nature is allowed to kill me.
Because of this alignment has gone from "An interesting career path I might pursue after I finish studying" to "If I don't do something I'll die while there was something else I could have done."

A big part of writing this is to get tailored advice, so a bit about my current skills that could be useful to the cause

Intermediate programmer, haven't done much stuff in AI, but I'm confident I can learn quickly. I'm good with Linux and can do sysadmin-y things.

Know a good amount of math, I've read textbooks on real analysis and linear algebra. With some effort, I think I can understand technical alignment papers though I'm far away from writing them.

I'm 17 and unschooled meaning I have nearly infinite free time and no financial obligations.

Babble

Inspired by Entering at the 11th Hour here's a list of some things I can do that may contribute (not quite "babble" but close enough)

Learn ML engineering and apply to AI safety orgs. I believe I can get to level talked about in AI Safety Needs Great Engineers in a few months of focused practice.

Deal with alignment research debt by summarizing results, making exposition, quiz games, etc.

Help engineers learn math. Hold study groups, tutor etc.

Help researchers learn engineering (I'm not an amazing programmer, but a lot of research code is questionable to say the least)

Donate money to safety orgs. (will wait till I'm familiar with all the options, so I can weight based on likelihood of success)

Host AI/EA/Rationality meetup. I live in the middle of nowhere, so this would be the only one

Try to convince some young math prodigies that alignment is important (I've run into a few in math groups before)

Make a website/YT/podcast debating AGI with people in order to convince people and raise awareness. (changing one person's carrier path is worth a lot)

Lobby local politicians, see if anyone I know has connections and can put a word in

Become active on LessWrong and ElutherAI in order to find friends who'll help along the way. Hard for me because of impostor syndrome right now (you don't want to know how long writing this post took).

Reflection

I most like (1), (2-4) and (6). (7) is something I'll do next time I have the chance.

I'm going to spend...
Is Fisherian Runaway Gradient Hacking?
by Ryan Kidd — 2022-04-10T13:47:16.454Z

TL;DR: No; there is no directed agency that enforces sexual selection through an exploitable proxy. However, Fisherian runaway is an insightful example of the path-dependence of local search, where an easily acquired and apparently useful proxy goal can be so strongly favored that disadvantageous traits emerge as side effects.

Why are male peacocks so ornamented that they are at greatly increased risk of predation? How could natural selection favor such energetically expensive plumage that offers no discernible survival advantage? The answer is “sex”, or more poetically, “demons in imperfect search”.

Fisherian runaway is a natural process in which an easy-to-measure proxy for a “desired” trait is “hacked” by the optimisation pressure of evolution, leading to “undesired” traits. In the peacock example, a more ornamented tail could serve as a highly visible proxy for male fitness: peacocks that survive with larger tails are more likely to be agile and good at acquiring resources for energy. Alternatively, perhaps a preference for larger tail size is randomly acquired. In any case, once sexual selection by female peacocks has zeroed in on “plumage size” as a desirable feature, males with more plumage will likely have more children, reinforcing the trait in the population. Consequently, females are further driven to mate with large-tail men, as their male offspring will have larger tails and thus be more favored by mates. This selection process may then “run away” and produce peacocks with ever more larger tails via positive feedback, until the fitness detriment of this trait exceeds the benefit of selecting for fitter birds.

In outsourcing to sexual selection, natural selection has found an optimization demon. The overall decrease in peacock fitness is possible because the sexual selection pressure of the peahen locally exceeds the selection pressure imposed by predation and food availability. Peacocks have reached an evolutionary “dead-end”, where a maladaptive trait is dominant and persistent. If peacocks were moved “off distribution” to an environment where predation was harsher or food more scarce, they would fare significantly worse than their less ornamented, “unsexy” ancestors.

Gradient hacking is a process by which an internally acquired “mesa-optimizer” might compromise the optimization process of stochastic gradient descent (SGD) in a machine le...
Worse than an unaligned AGI
by Shmi — 2022-04-10T03:35:20.373Z

So the worst case scenario being considered here is that an unaligned superintelligent AI will emerge sooner or later and intentionally or incidentally kill off humanity on its way to paper-clip the universe one way or another. I would argue that it's not the worst possibility, since somewhere in the memory banks of that AI will be the memories of humanity, preserved forever, and possibly even simulated in some way, who knows. A much worse case would be an AI that is powerful enough to destroy humans, but not smart enough to preserve itself. And so soon after the Earthlings get wiped out, the AI stops functioning or self-destructs, and with it all memories and traces of us. I would argue that it is not an unlikely possibility. Indeed, intentional self-preservation is not guaranteed by any means, and there are plenty of examples of even human societies Easter-Islanding themselves into oblivion.

So, somewhat in the spirit of "dying with dignity", I wonder if at some point when and if a superintelligent AI appears inevitable, it would make sense to put an effort into making it at least not too dumb to die, intentionally or accidentally.

Edit: Reading the comments, it looks like my concept of what is worse and what is better is not quite aligned with that of some other humans.
Time-Time Tradeoffs
by Orpheus16 — 2022-04-10T02:33:12.929Z
Link: https://forum.effectivealtruism.org/posts/4jHqkfWpbP6Pd3aim/time-time-tradeoffs

Epistemic Status: An idea I thought of a few days ago and wrote up quickly.

I often hear EAs and rationalists making Time-Money tradeoffs. The general idea is that money can often be exchanged for time (e.g., buying an Uber to save 30 minutes), and time can be exchanged for money (e.g., spending an hour to earn $X).

For example, let’s suppose Alice values her time at $50 per hour. Alice is willing to turn her money into ways of freeing up their time (by paying other people to do work for her for less than $50 per hour, for paying for services that save her an hour for less than $50, etc.) or by knowing how much money she’d take to lose time (by only working only pay $50+, etc.).

I recently thought of another kind of tradeoff that I’ve been finding useful: Time-Time tradeoffs. The idea here is that present time can often be exchanged for future time (and vice-versa).

The idea here is that not all hours are equally valuable. A standard case might involve thinking about which hours of your day tend to be most productive. Let’s suppose that Alice “peaks” around 12PM: she tends to be 2X more productive from 12PM-1PM than any other hour of her day.

But Alice also benefits from eating lunch at some point in the afternoon. By default, Alice had been eating lunch at 12PM each day. But Alice realizes that she can push her lunch break back to 2PM. She is effectively making a trade: 12PM Alice gains an hour in exchange for an hour from 2PM Alice.

Time-time trades also relate to time-money trades. For example, if Alice values a “normal hour” of her time at $100/hour, she should be willing to pay $200 to save the hour from 12-1PM.

Additionally, time-time trades don’t have to be 1:1. I noticed this when I was at EAGxOxford. A marginal hour of my time spent on community-building (networking, coming up with ideas, etc) at EAGxOxford was at least 3X as valuable as a marginal hour of “regular” community-building. In order to maximize especially impactful time, it would be worthwhile to sacrifice a greater number of hours of less impactful time. For example, if Alice's 12-1PM is twice as effective as her usual hour, she should be willing to trade 90 minutes of “non-peak Alice time” in order to gain 60 minutes of “peak Alice time.”

After thinking about Time-Time trades, I intend to spend more time completing tasks before or after conferences, even if it tak...
Boston Contra: Fully Gender-Free
by jefftk — 2022-04-10T00:40:02.069Z

There are currently four contra dance series in Boston:

LCFD,

BIDA,

Concord Thursday,
and

Concord
Monday Fundraisers. And all four are gender-free!

This has been a long time coming. Boston's gender-free contras started
with JP's LCFD in
1988. BIDA discussions started in
2013 and we switched in
2017. Monday fundraisers have been Larks/Robins since before covid, and
the Thursday dance is going gender-free as they restart.

Since I don't know what regular Concord Monday is planning
for when they restart this may be a temporary situation. But it's an
exciting milestone regardless!
Hidden comments settings not working?
by TLW — 2022-04-09T23:15:17.973Z

(Meta/site question.)

I have all settings related to comment hiding disabled[1] (do not collapse comments / do not truncate comments in large threads / do not truncate comments on home page)[2]. However, today I noted that comments appear to be hidden some of the time regardless[3].

This appears to be a recent regression[4][5]. I do note that there have been some other QoL changes recently ( code, improved LaTeX behavior, etc).

Is there a way to turn this behaviour off?

^

That is, set. Yay negative options.

^

In general, I do not believe information hiding is a good idea[6]. I am willing to publicly discuss this elsewhere; please not on this question.

^

As an example, when opening this link: https://www.lesswrong.com/posts/F5EmBWyryhgNtqsKb/convoy-crackdown?commentId=E9KdstpkKMX59bhMh I get "Showing 3 of 5 replies (Click to show all)".

^

It is possible that I simply did not encounter said hiding before. That being said, said link is from 2m ago.

^

It is possible that I simply did not notice it before; I would be surprised if so.

^

I acknowledge there are potentially cases where all known alternatives are even worse.
Godshatter Versus Legibility: A Fundamentally Different Approach To AI Alignment
by LukeOnline — 2022-04-09T21:43:20.324Z

Disclaimers

This is a response to, IMHO, a new pessimism that's rapidly ascending here; example

one

, example

two

, example

three

This is a super complex subject, and this medium, my time and my energy are limited. This post is not perfect.
I studied to be a historian but I'm a software developer by profession. I hold a deep interest in philosophy, ethics and the developmental trajectory of our civilization, but I am not a technical expert in modern AI.

I. What the current consensus in this community seems to be

AGI will probably be very powerful, and will pursue its goals in the world.
These goals might be anything, and thus they have a very large chance of being completely different from human goals. For example: maximize the amount of paperclips in the universe, maximize profits for a certain company, maximize the input to a certain sensor
Relentlessly pursuing these narrow goals will come at the cost of everything else, taking away the space and resources humans need to survive and thrive
Thus, AGI needs to be kept on a “leash”. We must have control, we must know what it is doing, we must be able to alter its goals, we must be able to turn it off.
This is an immense, technical task that we have not completed. We must discourage further progress in the making of AGI itself, and we must strongly encourage AI safety researchers to solve the task above

It’s a fair perspective. I’m in full agreement on points one and three. But I have strong doubts about the other three. Something that… feels much closer to my position, is

this recent post.

II. The Underlying Battle

I think there is a deeper conflict going on. On one hand, we’ve got

Moloch

. Moloch is the personification of “multipolar traps”, of broken systems where perverse incentives reward behaviors that harm the system as a whole.

In some competition optimizing for X, the opportunity arises to throw some other value under the bus for improved X. Those who take it prosper. Those who don’t take it die out. Eventually, everyone’s relative status is about the same as before, but everyone’s absolute status is worse than before. The process continues until all other values that can be traded off have been – in other words, until human ingenuity cannot possibly figure out a way to make things any worse.

In a

...
A concrete bet offer to those with short AGI timelines
by Matthew Barnett — 2022-04-09T21:41:45.106Z

[Update 4 (12/23/2023): Tamay has now conceded.]

[Update 3 (3/16/2023): Matthew has now conceded.]

[Update 2 (11/4/2022): Matthew Barnett now thinks he will probably lose this bet. You can read a post about how he's updated his views here.]

[Update 1: we have taken this bet with two people, as detailed in a comment below.]

Recently, a post claimed,

it seems very possible (>30%) that we are now in the crunch-time section of a short-timelines world, and that we have 3-7 years until Moore's law and organizational prioritization put these systems at extremely dangerous levels of capability.

We (Tamay Besiroglu and I) think this claim is strongly overstated, and disagree with the suggestion that “It's time for EA leadership to pull the short-timelines fire alarm.” This post received a fair amount of attention, and we are concerned about a view of the type expounded in the post causing EA leadership to try something hasty and ill-considered.

To counterbalance this view, we express our disagreement with the post. To substantiate and make concrete our disagreement, we are offering to bet up to $1000 against the idea that we are in the “crunch-time section of a short-timelines”.

In particular, we are willing to bet at at 1:1 odds that no more than one of the following events will occur by 2026-01-01, or alternatively, 3:1 odds (in our favor) that no more than one of the following events will occur by 2030-01-01.

A model/ensemble of models achieves >80% on all tasks in the

MMLU

benchmark
A credible estimate reveals that an AI lab deployed EITHER >10^30 FLOPs OR hardware that would cost $1bn if purchased through competitive cloud computing vendors at the time on a training run to develop a single ML model (excluding autonomous driving efforts)
A model/ensemble of models will achieve >90% on the

MATH dataset

using a no-calculator rule
A model/ensemble of models achieves >80% top-1 strict accuracy on competition-level problems on the

APPS benchmark

A gold medal for the IMO Grand Challenge (conditional on it being clear that the questions were not in the training set)
A robot that can, from beginning to end, reliably wash dishes, take them out of an ordinary dishwasher and stack them into a cabinet, without breaking any dishes, and at a comparable speed to humans (<120% the average time)
Tesla’s full-self-driving capability makes fewer than on

...