trentmkelly/LessWrong-43k huggingface dataset:
How humanity would respond to slow takeoff, with takeaways from the entire COVID-19 pandemic 
by Noosphere89 — 2022-07-06T17:52:16.840Z

While there have been some posts on how a slow takeoff would occur given humanity's response to COVID, they were written during the crisis, rather than it's end to get a full account of what would likely happen.

I will be taking mostly an outside view here, which means I will mostly abstract this as "how humanity usually responds to potential crises or x-risk" as the reference class here. I will mostly ignore inside view concerns here, so specific details will be abstracted away.

I am also ignoring hard takeoff/hard singularity or Foom scenarios, where AI controls everything in hours, days or weeks.

So without further ado, here's how humanity will by default respond and how well they will respond.

The median outcome is usually what will happen.

By default, this means that people should mostly ignore the worst or best outcomes if they want to predict how a crisis or x-risk will go. This has important implications for how bad things are likely to go in slow takeoff.

Basically, it means that we should down weight scenarios that imply a lot of civilizational competence like a purely positive singularity, or the banning of AGI. Or in other words, AGI policy is useless for the attempt to align AGI unless governments get far more competent than they are today, due to the complicated thing that AGI is.

This also means that bluntly, EA/Rationalist communities are almost certainly overworried about AGI, especially suffering focused ethics people, and this is also applicable to extinction risks or existential risks. One of the biggest problems in AI Alignment and AI fields more generally is the people that get selected in AI Alignment are people that tend to buy in to the extreme worst-case scenarios, and the average AI capabilities person is far too optimistic about human level AI outcomes.

AGI, if it's built, will not go away even after millions are dead.

One of the most important things that COVID-19 forced was after a short time, it couldn't be eradicated. And while AI has a different benefit and threat profile, the differences strengthen this takeaway, as COVID was essentially pure threat. This means it is very likely that we will have to live with AGI/ASI once we build it for real.

Partial solutions to problems, and hacky attempts to align it is necessary.

The field of AI Alignment is operating under a Nirvana Fallacy, which is basically looking for perfect solutions to pro...
Should you write under a blog or your own name?
by Dalton Mabery — 2022-07-06T15:26:41.430Z

I've always wondered if it's better to write at a blog (overcomingbias.com) or write under your own name (scottaaronson.blog).

The benefits of writing under your own name seem to be easier credit to your work and that people get an instant snapshot of what you're interested in. It also shows others how much you write and how long you've been writing for, which is a display for consistency for potential employers/partners. The cons of this, though, is that sometimes one feels they can't share personal projects they're working on or other "less organizable" stuff.

If you write under a blog name, it seems to feel more credible (not sure why this is). It also allows you to use your personal website to talk about projects you're working on things you're researching, while having the option to link back to your blog to showcase your writing. If you do this, though, the consistency factor isn't there like it would be if it were on your personal website.

I currently write at daltonmabery.com, but have debated changing it to 221b.blog (a nod to Sherlock Holmes), colloquim.blog, or something of the like.
Carrying the Torch: A Response to Anna Salamon by the Guild of the Rose
by moridinamael — 2022-07-06T14:20:14.847Z

In a recent comment and

followup post

, Anna Salamon described some of the rocks upon which the Center for Applied Rationality has run aground, and invited anyone interested in the project of improving human rationality to pick up the torch.

We wanted to take this opportunity to remind you that the

Guild of the Rose is here to carry that torch onward

.

On a personal note, I figured I ought to also explicitly mention that the Guild is, in no small part, the result of many parties from all over the world reading and responding to the

bat signal that I shined into the sky in 2018

. In that post, an alien visitor with a rationality textbook exhorted us to dream a little bigger regarding the potential of the rationality movement. If that post spoke to you, then you ought to know it spoke to a lot of people, and, well, the Guild is what we're doing about it together.

And I think we're doing a pretty good job! Anna outlined a list of problems that she felt CFAR ran into, and I figured this would be a good place to describe how the Guild dealt with each of those problems.

Wait a minute – dealt with those problems? Anna just posted a few weeks ago!

When we started the Guild in 2020, we looked to CFAR as both an example to emulate and also differentiate ourselves from. We diagnosed many of the same problems that Anna describes in her post, though not necessarily in the same framing. We designed our organization to avoid those problems. We are grateful to CFAR for having pioneered this path.

Problem 1:

Differentiating effective interventions from unfalsifiable woo.

The Guild focuses on actions and habits, not psychotherapy. I think ~0% of what we teach can be called unfalsifiable woo. Even when we tread more esoteric ground (e.g. the

decision theory

course) we focus on the practical and implementable.

To sketch a perhaps not-totally-generous metaphor, imagine there are two martial arts schools you're trying to choose between:

One of these schools is esoteric, focuses on dogmatic cultivation of Warrior Spirit, and demands intensive meditation. This school promises a kind of transcendent self-mastery and hints that ki-blasts are not entirely off the table.

The other school focuses on punching bags, footwork drills, takedowns, and sparring. This school promises that if you need to throw hands, you'll probably come out of i...
Predicting Parental Emotional Changes?
by jefftk — 2022-07-06T13:50:04.387Z

Before I became a parent I cared about children in general,
intellectually, in the same way that I care about people in general: a
general sense that I want good things to happen to them. Over the
first few days of Lily's life, however, I noticed I cared for her
deeply and emotionally, where her wellbeing was incredibly important
to me, and she was far more cuddly and interesting than I expected.
Over the next few months, I realized that my disposition toward
children in general had also changed, and while I didn't feel as
strongly as toward my own, they were now similarly far more fun,
interesting, and precious than they had been. I also started having

strong emotional
reactions to reading about harm to children. Paul Graham

writes about a similar
experience, though I think one a bit more sudden than my own:

Partly, and I won't deny it, this is because of serious chemical
changes that happened almost instantly when our first child was
born. It was like someone flipped a switch. I suddenly felt protective
not just toward our child, but toward all children.

And Maia commenting
on a post of Julia's:

For me it was like a switch flipped when I gave birth. The mental
change felt similar to the mental change I went through during
puberty: just adding a completely new biological drive I didn't have
before. I actually didn't find babies very cute or interesting at all
before, but I found my baby extremely so immediately

In deciding to have kids this is a very difficult factor to consider,
because it doesn't happen to everyone. I have a vague impression that
it is a bit more common in men than women, perhaps because women
understandably have a higher threshold for deciding to have kids, but
there are definitely men who expect it to happen and are dismayed when
it doesn't and women who are surprised at how much their feelings
toward children change.

Getting better at predicting this transformation seems valuable: since
becoming a parent is a lifelong decision, the value of information is
really high. Do people tend to have similar experiences to their own
parents? Is it correlated with any other aspects of personality? Can
your friends or other people who know you well predict this? Can it
be triggered, if it's going to, by something less permanent
than becoming a parent or is the permanence a key component? Is it
something that may o...
Forecasting Through Fiction
by Yitz — 2022-07-06T05:03:55.376Z

Note: In retrospect, I think I'm making two separate points here. The first, and most important one, is the idea that "interestingness" or asking "what possible futures would make for the best story?" can provide predictive insight. I didn't spend much time on that point, despite its potential importance, so if anyone can create a steelmanned version of that position I'd appreciate it. The second point is that judging by that heuristic, the rationalist movement is terrible at placebomancy. The epistemic status of this entire post is Speculation Mode, so take everything I say with a grain of salt.

We don't live in a work of fiction. That being said, there is value in being genre-savvy; storytelling is a form of world-modeling,[1] and I think it's possible that what we find most satisfying from a storytelling perspective is more likely to reflect reality than what one would naively think.[2] As such, it may be worth modeling what we would expect to happen if reality was a story.

How would Rationalists fare in a typical story?

I posed the following scenario to three different Discord groups I'm in (none of which know much about the Rationalist movement):

Imagine you’re halfway through reading a fiction book. Here’s a summary of what’s going on so far:

The book follows a strange group of people. They call themselves “Thinkers,” and they believe they’ve discovered a set of methods that allow people to make significantly more accurate predictions of the future. And the Thinkers are worried. Using their new methods, they predict with a high degree of confidence that humanity will create a hyper-intelligent machine (so smart that it makes Einstein look like a toddler by comparison), but without human morality. This human-made machine will destroy humanity, if they don’t do something about it. The leaders of the group seem to be more-or-less in agreement—the only way to stop this apocalypse is to build a hyper-intelligent machine themselves, but one which has the same goals as humans, and will be able to prevent any other immoral hyper-intelligent machines from being created. The only problem is that the Thinkers do not yet know how to program a machine to have fully human goals.

What happens next in this story? How does it end?

I followed up by clarifying that the question is more "if you read this in a fiction book how you would expect the plot to go?", no...
Introducing the Fund for Alignment Research (We're Hiring!)
by AdamGleave — 2022-07-06T02:07:47.965Z

Cross-posted to the EA Forum

The

Fund for Alignment Research

(FAR) is hiring research engineers and communication specialists to work closely with AI safety researchers. We believe these roles are high-impact, contributing to some of the most interesting research agendas in safety. We also think they offer an excellent opportunity to build skills and connections via mentorship and working closely with researchers at a variety of labs.

We were inspired to start FAR after noticing that many academic and independent researchers wanted to work with more collaborators, but often lacked the institutional framework and hiring pipeline to do so. On the other hand, there are a large number of talented people who would like to contribute to AI safety, but lack a stepping stone into the field. Our hope is that FAR can help fill this gap, both directly accelerating valuable research and helping to address the talent pipeline.

In the remainder of this post we'll give a bit more information about FAR and our current collaborators, and then summarize our

current openings

. Please consider applying or forwarding them to a friend who might be interested! We are also actively brainstorming other ways that FAR could be useful to the community. If you have any ideas, we’d love to hear from you!

About Us

FAR is a non-profit led by

Ethan Perez

,

Adam Gleave

,

Scott Emmons

, and

Claudia Shi

: a group of AI safety researchers looking to reduce existential risk from artificial intelligence. Ethan recently graduated from the PhD program at New York University, Adam and Scott are PhD candidates at UC Berkeley, and Claudia is a PhD candidate at Columbia University.

FAR provides services to AI safety researchers to accelerate their research agendas. We are currently focused on supporting the agendas of Ethan, Adam, Scott, and Claudia. We are also trialing a collaboration with the labs of David Kruger and Jacob Steinhardt, professors at the University of Cambridge and UC Berkeley.

Our services are currently provided free-of-charge to recipients out of FAR's general support funding. In the future we plan to charge partners who use large quantities of our services on an at-cost basis. This could be paid for from a partner's existing grant, or we can also assist the partner with fundraising for this purpose.

We anticipate supporting many of the same people that BERI currently works with. However, our or...
My vision of a good future, part I
by Jeffrey Ladish — 2022-07-06T01:23:01.074Z

A future where everyone is okay

I grew up with the future feeling pretty uncertain. God was supposed to return at some point, but maybe there were going to be The End Times before that? Like, a great persecution of the Christians, and maybe something about the bottomless pit and the oceans turning to blood. I no longer believe that, but I can still relate to a lot of angst I see people express about the future, especially in regards to AI, in my own circles, and climate change, in circles a little further out.
And yes, I’m quite worried about existential risks from AI. I think the default outcome is that we build systems vastly smarter and more powerful than us, and those systems destroy us because our survival is not conducive to their divergent goals. But I do not believe our destruction is guaranteed. We might succeed at aligning the goals of these vastly more powerful intelligences with our own, and it’s that outcome that inspires me to fight for the future I believe in. We might succeed at building AI systems that possess both vast amounts of optimization power and a finely tuned concern for human well being and flourishing. If we get this right, the future could be almost unimaginably good. But not entirely unimaginable.

I want to try to imagine it, concretely, because I care about what I’m fighting for. I’m not just fighting against extinction, I’m fighting for something that could really come to pass, something amazing. So I want to know what that might look like.

In brief, the good world:

In the good future I imagine, everyone is okay. Every person has enough good food to eat and water to drink. No one has dementia, cancer, or any disease. There’s no material scarcity in the sense that there is today; anyone can live a billionaire’s lifestyle if they want to. The root causes of social problems have been treated. People still have difficulties from time to time, but everyone is set up to succeed, find romantic partners, and have a rich and fulfilling social life. People tend to be really happy most of the time.

And that’s just the beginning. Once we’ve solved the major problems, that’s where the fun begins. I have quite a list of things I want to do in the future. Countless countries, cities, and bioregions and I want to explore. I want to write science fiction novels. I want to speak new languages, meet new people, build things with my friends a...
Imperial Russia was doing fine without the Soviets
by Davis Kedrosky — 2022-07-05T22:24:30.185Z
Link: https://daviskedrosky.substack.com/p/twilight-imperium

TL;DR: I defend the “optimist” position on the economic performance of Tsarist Russia. The pre-1914 Russian economy was experiencing sustained economic growth which was interrupted by WWI and curtailed by the Bolshevik Revolution and Civil War. The “pessimist” stance ignores or explains away real signs of capitalist development in Russia in spite of its bad institutions. Main points:

Russia had cheap labor, abundant natural resources, and tons of foreign investment capital plus access to technology.
Russian political and economic institutions were not developmentalist, but were improving under the pressure of social unrest and military modernization.
Russia’s domestic market was large and prosperous enough to make protectionism work AND there was no prewar crisis of rural living standards.
Latin America is not the correct counterfactual for non-communist Russia; other primary product exporters (notably the US) successfully transitioned.
Russian heavy and light industry grew far more rapidly prior than agriculture prior to the First World War, leading to structural transformation.
The trigger for Revolution was more likely the wartime collapse of the Russian economy than its inherent weakness.

Investors of the Belle Epoque would have been surprised to hear the modern consensus that Russia was a structurally weak, doomed economy. In the decades preceding the July Crisis, the empire of the Tsars was a magnet for foreign capital, attracted by the prospects of heavy industrialization and railway construction. Russian economic growth, accompanied by rising per capita GDP and structural transformation, was rapid for the pre-1945 world and there were signs of convergence, or at least parity, with Western Europe.

Yet the Bolshevik Revolution(s) of 1917, with its crosscurrents of proletarian agitation, created the enduring impression of a stricken Russia just waiting to collapse. If the economy was running well, why did the masses revolt? Robert Allen has been among the strongest advocates for this view. His 2003 book

Farm to Factory: A Reinterpretation of the Soviet Industrial Revolution

argues that the Tsarist expansion was doomed to fail, as repressive government, moribund class structures, and declining terms of trade would have sapped growth in the absence of war.

I've been a diehard Allen fan for most of my mature existence. But in discussions with various kindly people ...
A Pattern Language For Rationality
by Vaniver — 2022-07-05T19:08:49.783Z

There's a lens to looking at the rationality project that I've been finding enlivening recently, and I think it's reached the point where more eyes and hands might be useful, while not being anywhere near complete yet. First, some background.

Christopher Alexander was a designer and architect; his thinking, and focus on patterns in particular, were

influential in programming

; wikis were first invented to facilitate the collaborative creation and modification of 'patterns' in the style he described. He wrote lots of books, but I'll focus on this trio:

The Timeless Way of Building (about the ‘quality without a name’, and good vs. bad buildings)
A Pattern Language: Towns, Buildings, Construction (about the 'patterns' they’ve identified for physical buildings and towns)
The Oregon Experiment (about the University of Oregon, and how it could replace a ‘master plan’ with principles)

The three books, while each on a different subject or layer of 'design', all work together and depend on each other. The first identifies the target, why it would even be good to pursue, and how you know whether or not you've found it. The second is a detailed description of the patterns they've found useful in approaching the target. The third is what it looks like to organize systems to deliberately and durably organize themselves according to this design.

Just like this approach was profitably translated to programming (and other areas of design), I think it's worth looking at the 'rationality' project as a way to design decisions, habits, and thinking, and attempting to deliberately incorporate Alexander's approach and strategy. This post is an attempt to get started[1] in each of the three directions, rather than fully lay them out; depending on how things go, I might turn this post into a sequence / flesh out individual sections.

The Timeless Way of Living

The Tao that can be told is not the eternal Tao; The name that can be named is not the eternal name.

[Consider also 'thinking', 'deciding', 'doing', and 'being' in place of 'living', for both this section and the next.]

Alexander talks about the “Quality Without A Name”,[2] so-called because no existing English word was a good fit, altho he could point towards it pretty effectively in a few pages; you should read the start of The Timeless Way of Building (

Amazon

,

pdf

) to get his sense of it. I think “equilibrium” come...
How to destroy the universe with a hypercomputer
by Trevor Cappallo — 2022-07-05T19:05:46.057Z

Let us stipulate the following:

The universe is fundamentally discrete / deterministic. (Seems reasonably plausible.)
There's a way to perform hypercomputation, in the sense that one can build or leverage a Turing-complete mechanism which can perform infinitely many computations in seemingly finite time, and store an infinite amount of data in seemingly finite space. (Seems wildly implausible.)

Take your hypercomputer and write some trivial code that systematically executes every possible algorithm, in particular all of the non-computable functions that require an infinite amount of steps to run. Among just a few other things, this program will eventually stumble onto a perfect simulation of our own universe, and suppose you make sure that simulation has time to run until it catches up with the current moment, so your universe and the simulation are synced up temporally (from your reference frame, perhaps, if GR is an issue).

Then take a mallet and smash your hypercomputer.

By my reasoning, it seems like it should be turtles all the way down, with infinitely many copies of you and your universe simulation if you were to drill down and examine that stack. By assertion, these are perfect simulations of the universe, such that not only will you be unable to tell them apart, but to the point where they're isomorphic and are arguably one and the same thing.

In the slightly 'weaker' version of this, there remains a 'topmost reality'. Even though you seemed to begin there, subjectively it's all the same, so you're Almost Surely in one of the infinite simulations, meaning you and your universe will wink out of existence as soon as you destroy your copy of it. Of course, the only you that will actually care is the one that will continue to exist, as well as presumably conclude that the whole thing was very anticlimactic.

In the stronger version, the simulated universe and your universe truly are the same thing, so when you destroy it, you don't leave behind even a single copy of yourself that can breathe a sigh of relief.

This is obviously grossly speculative and not especially in my wheelhouse, so I expect I got more than one thing wrong about arithmetical hierarchies or selection effects or more basic logical flaws, or what have you. By all means point them out if you're inclined. I'm especially curious if any of those mistakes immediately takes apart the whole argument, i...