trentmkelly/LessWrong-43k huggingface dataset:
Rational Animations' Script Writing Contest
by Writer — 2022-09-15T16:56:30.351Z

Cross-posted to the EA Forum.
Edit: The deadline has been removed! So you can continue to send scripts.

Introduction

I'm announcing a script writing contest for the Rational Animations YouTube channel. Write scripts of 2500 words maximum about a topic of longtermist relevance, and win:

5000 USD.
We will make an animated video based on your script to be published on the YouTube channel with due credits. It will be animated by a team of 9 animators or an animation studio and will be of much higher quality than anything we've done so far.
A potential job offer as a scriptwriter for the channel.

Bonus: you may suggest already written work. If we bring your suggestion to the channel as it is (without having to adapt it), you will get $500, but nothing if I am already considering the work.

Soft deadline: October 15th.

Topics

The topic of your script should be of longtermist relevance. Examples: existential risks of any kind, the future of humanity, philosophy related to longtermism, technologies of longtermist relevance. I don't mean to be too strict, I'll accept scripts about a broad range of topics, and I might even make exceptions and offer a prize to scripts not relevant to longtermism if I find them a lot better than the competition. As a reference, the videos already on the channel that I would classify as longtermism-relevant are:

Can we make the future a million years from now go better?
Everything might change forever this century (or we’ll go extinct)
Will we grab the universe? Grabby aliens predictions.
Humanity was born way ahead of its time. The reason is grabby aliens.
We are failing to see how much better off humanity could be
Longtermism: an idea that could save 100 billion trillion lives

Length and number of scripts

The maximum length of the script should be 2500 words. Of course, you may split a longer script into two or more parts. You may send an unlimited number of scripts.

Are there any more restrictions?

Not really.

So far, we've only published explainers on the YT channel, but you don't have to limit yourself to explainers. Do you want to write a Fable of the Dragon-Tyrant style short story? OK. Do you want to write an anime episode? OK. You can do whatever you want.

You may also adapt or send older work you've written that you think would make a good video for the YouTube channel if narrated as it is or with some tweaks. In that case...
Covid 9/15/22: Permanent Normal
by Zvi — 2022-09-15T16:00:00.607Z

I may or may not have had Covid about two weeks ago.

At the time, I knew I had come down with something. I largely lost my voice and spent about a week carefully conserving what remained. I spent over a week coughing a decent amount. I was having some trouble breathing when any pressure was applied to my chest and occasionally at other times.

I never felt the situation was that serious and thought it probably wasn’t Covid, and never got tested or took any treatment beyond the normal resting and staying home for a minor illness plus resting my voice, expecting it to get better. Which it did.

My personal trainer, however, immediately assumed that of course it was Covid. It certainly is possible. In the end, it does not much matter. I presume I will never know for sure.

I am fully recovered now, except that I lost some progress exercising, especially on the elliptical machine, and am still working to get back to where I was before, which I anticipate will take another week or two.

This is permanent normal. We get sick. We don’t know exactly what it was. Over time, we mostly feel better. There is not much Covid news, so it seems fine to have the introduction be a personal tale like this.

That also, combined with doing some longer term work on various things, is why I haven’t been putting out any additional content beyond the weekly posts for a bit. That will change. I also am considering how much I should be ‘spinning off’ various things into their own posts.

Executive Summary

Covalent boosters now available wherever vaccines are administered.

Covid cases dropped unexpectedly.

The dog that didn’t bark: Zero monkeypox news of note.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 490k cases (+9%) and 3,000 deaths (+10%)

Results: 389k cases (-14%) and 2,878 deaths (+5%).

Prediction for next week: 350k cases (-10%) and 2,625 deaths (-9%).

California normally reports cases twice a week and this week only reported once. I am going to assume this is a delay and conservatively added 20,000 additional cases and 70 additional deaths. These will be subtracted from next week’s numbers, unless I discover something weird going on that makes me do something else instead. Deaths are typically jumpy, so only making the adjustment there because of the very obvious need to adjust cases. Not going to worry too much about it all giv...
Are Human Brains Universal?
by DragonGod — 2022-09-15T15:15:21.302Z

[Previously]

Introduction

After reading and updating on the answers to my previous question, I am still left unconvinced that the human brain is qualitatively closer to chimpanzee (let alone an ant/earthworm) than it is to hypothetical superintelligences.

I suspect a reason behind my obstinacy is an intuition that human brains are "universal" in a sense that chimpanzee brains are not. So, you can't really have other engines of cognition that are more "powerful" than human brains (in the way a Turing Machine is more powerful than a Finite State Automaton), only engines of cognition that are more effective/efficient.

By "powerful" here, I'm referring to the class of "real world" problems that a given cognitive architecture can learn within a finite time.

Core Claim

Human civilisation can do useful things that chimpanzee civilisation is fundamentally incapable of:

Heavier than air flight
Launching rockets
High-fidelity long-distance communication
Etc.

There do not seem to be similarly useful things that superintelligences are capable of that humans are also fundamentally incapable of. Useful things that we could never accomplish in the expected lifetime of the universe.

Superintelligences seem like they would just be able to do the things we are already — in principle — capable of, but more effectively and/or more efficiently.

Cognitive Advantages of Artificial Intelligences

I expect a superintelligence to be superior to humans quantitatively via:

Larger working memories
Faster clock cycles (5 GHz vs 0.1 - 2 Hz)

Faster thought? [1]

Larger attention spans
Better recall
Larger long term memories

(All of the above could potentially be a several orders of magnitude difference vs homo sapiens brain given sufficient compute.)

And qualitatively via:

Parallel/multithreaded cognition

The ability to simultaneously execute:

Multiple different cognitive algorithms
Multiple instances of the same cognitive algorithm

Here too, the AI may have a several orders of magnitude difference in the number of thoughts/cognitive threads they can simultaneously maintain vs a human's "one"
This may also be a quantitative difference, but it's the closest to a qualitatively different kind of cognition exclusive to AIs that has been proposed so far

Cognitive Superiority of Artificial Intelligence

I think the aforementioned differences are potent. And would confer the AI considerable adv...
Intelligence failures and a theory of change for forecasting
by  — 2022-09-15T15:02:13.785Z

In this blog post I’m going to try to do two things. First I’m going to make the case that failures of both the US and Soviet intelligence services have substantially increased the risk of nuclear power in the atomic age, then I’m going to argue that these case studies provide potentially the best example of how Tetlock style forecasting can be used to improve the world.

Forecasting

Tetlock style forecasting is the practice of individuals or, even better, groups of individuals and having them predict the probability of events on scale from 0-1 and then scoring them on their predictions. Tetlock’s experiments have shown that some people are much much much better both than average individuals, domain experts, and relevant for this, CIA intelligence analysts with access to classified material, at predicting political events. However, as with any improvement in human power, it’s an open question about whether this tool is more likely to be used for good or ill. For instance, one could imagine authoritarian regimes using these methods to more effectively suppress protests. Furthermore, it’s not clear how many tragedies could have been averted if the revlevent actors had had access to higher quality information. For instance, the North Korean economy isn’t failing because President Kim is incorrect about whether or not his economic policies are good - the economy is failing because Kim uses the policies that are expedient to him staying in power for the longest period of time possible. If one was trying to improve the long term future as much as possible it’s not clear that developing and advocating for forecasting is better than

reducing the burden of disease in low income countries

which both have enormous benefits and we can be very confident that we do effectively.

However, I think the history of nuclear risk provides good case studies in how having better information would have reduced nuclear risk, and I’ll also sketch theoretical reasons for why one should believe that this is true.

A short history of intelligence failures

I think the role of intelligence failures during the cold war phase of the nuclear era can roughly be divided into 3 themes:

Incorrect estimates about the other sides strength
Poor understanding of the intentions of the other side
The early success of espionage and failure of counter-espionage

One recurrin...
Why deceptive alignment matters for AGI safety
by Marius Hobbhahn — 2022-09-15T13:38:53.219Z

Update: I changed the title from "Why AGI safety researchers should focus mostly on deceptive alignment" to "Why deceptive alignment matters for AGI safety". I think my original message was too strong and I'm actually much more uncertain about failure modes from AGI than the title suggests.

Comment: after I wrote the first draft of this post, Evan Hubinger published “

How likely is deceptive alignment

” in which he argues that deceptive alignment is the default outcome of NNs trained with SGD. The post is very good and I recommend everyone to read it. As a consequence, I rewrote my post to cover less of the “why is deceptive alignment likely” to the more high-level arguments of “why should we focus on deceptive alignment” and adopted Evan’s nomenclature to prevent confusion.

I’d like to thank Lee Sharkey, Richard Ngo and Evan Hubinger for providing feedback on a draft of this post.

TL;DR: No matter from which angle I look at it, I always arrive at the conclusion that deceptive alignment is either a necessary component or greatly increases the harm of bad AI scenarios. This take is not new and many(most?) people in the alignment community seem to already believe it but I think there are reasons to write this post anyway.
Firstly, newer members of the alignment community are sometimes not aware (at least I wasn’t in the beginning) that more senior people often implicitly talk about deceptive alignment when they talk about misalignment.

Secondly, there seems to be some disagreement about whether a powerful misaligned AI is deceptive by default or whether such a thing as a “corrigibly aligned” AI can even exist. I hope this post clarifies the different positions.

Epistemic status: Might have reinvented the wheel. Most of the content is probably not new for most people within the alignment community. Hope it is helpful anyway.

Update: after a discussion in the comments, I want to make some clarifications:
1. My definition of deception is a bit inconsistent throughout the post. I'm not sure what the best definition is but I think it is somewhere between "We have no clue what the model is doing" (which doesn't include active deception) to "The model is actively trying to hide something from us". Both seem like important failure modes.
2. I don't think deception is orthogonal to understanding other failure modes like getting what ...
FDT defects in a realistic Twin Prisoners' Dilemma
by SMK — 2022-09-15T08:55:52.818Z

Thanks to Caspar Oesterheld for discussion and helpful comments, as well as Tristan Cook, James Faville, Daniel Kokotajlo and Lukas Finnveden.

Summary

Updateless decision theory (UDT)/functional decision theory (FDT) can be formulated with logical conditionals as opposed to logi-causalist counterfactuals. I argue in favour of the former, on the grounds that this variant of UDT/FDT ensures robust mutual cooperation in the Twin Prisoner’s Dilemma between two realistic UDT/FDT agents, whereas the causalist variant does not. This falls out of thinking about how agents approximate decision theories and how they intervene on the outputs of different, yet similar, decision algorithms.

Introduction

Updateless decision theory does not necessarily have to be formulated with logical counterfactuals: you could also use logical conditionals. This is true of functional decision theory and timeless decision theory as well, mutatis mutandis. Specifically, we can define ‘updateless (logically) causal decision theory’ (UCDT), which is the standard formulation, and ‘updateless evidential decision theory’ (UEDT) in the UDT1.1 framework[1] respectively as:

UCDT1.1(s)=argmaxπ∈Π ∑o∈OU(o)P(┌UCDT1.1(s)=π┐ □→ o),UEDT1.1(s)=argmaxπ∈Π ∑o∈OU(o)P(o | ┌UEDT1.1(s)=π┐).

(It seems Wei Dai initially thought of UDT as something evidential; see

this

comment, for example.[2])

As I understand it,

Troll Bridge

(a logical version of the

Smoking Lesion

) is a decision problem where UCDT and UEDT come apart, and UCDT is taken to give the correct answer. I personally think it is unclear what the takeaway from Troll Bridge should be, and I think there are problems in which UEDT is clearly preferable[4] to UCDT. And the latter is what this post is about.

Mutual UDT cooperation

The causal variants of logical decision theories like UDT, FDT and TDT are understood to ensure mutual cooperation in the Twin Prisoner's Dilemma. The reasoning goes as follows: “in virtue of us being twins, we are implementing the same decision algorithm, and now the question is about what this algorithm will output. Suppose it outputs 'defect'; then both of us will defect. Suppose it outputs 'cooperate'; then both of us will cooperate. And since u(C,C)>u(D,D), I will cooperate.".

In terms of causal graphs, this is how I generally unde...
What's the longest a sentient observer could survive in the Dark Era?
by Raemon — 2022-09-15T08:43:21.923Z

Periodically I look at this graph, and I'm like, "holy shit, that is so much Nothing. Wut. Like, they converted it to log-scale so it would fit into my brain and it's still so much Nothing, Jezus Christ."

Eliezer's Dath Ilan verse has this song for people who died before the invention of cryonics:

Even if the stars should die in heaven
Our sins can never be undone
No single death wil be forgiven
When fades at last the last lit sun.
Then in the cold and silent black
As light and matter end
We’ll have ourselves a last look back
And toast an absent friend.

And I find myself wondering "What are the physical limits on 'some sentient computation surviving into the dark era?" Assuming we don't learn anything new about physics, or acausal-trade our way into another universe or whatever, what's the farthest to the right on this timeline you could possibly get, if you goal was to have some computation that, as late as possible, looks around at the universe one last time, comprehends it, and says "yep, still full of infinite blackness. Goodbye, world."

Vaguely relevant things I'm vaguely aware of include:

Something something you can somehow harvest energy from black holes. Does this plausibly get us to the end of the Black Hole Era or is that energy eventually too weaksauce?
Space is, like, really cold (especially when All Has Become Darkness), so any time-capsule robot would face an uphill battle not radiating it's energy into the void.
On long enough timescales, protons (probably?) decay, and... like, your robot would eventually... just... disintegrate?

If anyone knows enough physics to satisfy my random curiosity, uh, I'd appreciate it.
The Value of Not Being an Imposter
by sudo — 2022-09-15T08:32:54.353Z

Imagine that you are in a field filled with imposters.

Perhaps you are a doctor or nurse who truly cares about the wellbeing of your patients, and you're in a mismanaged hospital where most doctors do little to no good at all.
Perhaps you are a passionate software engineer in a company filled with ladder-climbers who care more about hacking the promotion structure than delivering a good product.
Perhaps you are a teacher who goes the extra mile to help all of your children learn, while your colleagues are busy goodhart-ing a small set of test scores.

If your field is useful, then you can do quite a lot of good by staying in your field and not becoming an imposter.

It costs the hospital/company/school the same amount of money to hire you as it costs them to hire someone else. However, by virtue of your lack of Goodharting, you are able to do more good than another employee with your pay.

When considering whether to do direct work or earn to give, it might be worth considering whether you're likely to be an imposter. If you wouldn't be an imposter, then you might just be worth much more than the next employee.
Capability and Agency as Cornerstones of AI risk ­— My current model
by wilm — 2022-09-15T08:25:05.267Z

tldr: In this post I am explaining my current model of AI risk after failing at my goal of understanding everything completely, without relying on arguments by authority. I hope this post is valuable in explaining the topic and also by giving an example of what a particular newbie found and did not find easy to understand / accessible.

This was not planned as a submission for the AI Safety Public Materials contest, but feel free to consider it.

I thank all those who helped me understand more about AI risk and AI safety and I thank benjaminalt, Leon Lang, Linda Linsefors, and Magdalena Wache for their helpful comments.

Ideas about advanced artificial intelligence (AI) have always been one of the core topics on LessWrong and in the rationality community in general. One central aspect of this is the concern that the development of artificial general intelligence (AGI) might pose an existential threat to humanity. Since a few years ago, these ideas have gained a lot of traction and the prevention of risk from A(G)I has grown into one of the central cause areas in Effective Altruism (EA). Recently, the buzz around the topic has increased even more, at least in my perception, after Eliezer's April fools post and list of fatalities.

Having first read about AGI and risk from AGI in 2016, I have gotten more and more interested about the topic until this year I finally decided to actually invest some time and effort to learn more about it. I therefore signed up for the

AGI safety fundamentals course

by EA Cambridge. Unfortunately, after completing it I still felt pretty confused about my core question of how seriously I should take existential risk (x-risk) from AI. In particular, I was feeling confused about how much of the concern for AI safety was due to the topic being a fantastic

nerd-snipe

, that might be pushed by group-think and

halo effects

around high-profile figures in the rationality and EA communities.

I therefore set out to understand the topic from the ground up, explicitly making sense of the arguments and basic assumptions without relying on arguments by authority. I did not fully succeed at this, but in this post I present my current model about x-risk from advanced AI, as well as a brief discussion of what I feel confused about. The focus lies on understanding existential risk directly caused by AI systems, as this is what I ...
General advice for transitioning into Theoretical AI Safety
by Martín Soto — 2022-09-15T05:23:06.956Z

Update 2025: This post is completely out of date, don't follow its advice.

Cross-posted from the EA Forum.

During the past months I've privately talked with +20 AI Safety researchers[1] about how to transition into Theoretical AI Safety . Here's a distillation of their advice and opinions (both general consensus and controversial takes).

Why this post?

Some great posts already exist with general advice for transitioning into AI Safety. However, these and others are mainly centered around technical Computer Science research and Machine Learning engineering. They don't delve into how more theoretical aptitudes and background (such as careers in Mathematics, theoretical Computer Science or Philosophy) can be steered into more abstract Alignment research that exploits them (except for Critch's short post). I think that's mainly because:

Most people in the Alignment community work in applied research. That is to say, there are (way) more open positions for this kind of research. Still, most people I've talked to agree we need a non-trivial percentage of the community working in Theory (and we certainly need more people working in any subfield of Alignment they want to work on[2]).
Applied career paths are more standardized, and thus easier to give advice on. As happens more broadly in academia and the job market, there are more obvious job prospects for the more applied. Theoretical careers tend to feel more like creating your own path. They often involve autonomous projects or agendas, individualized assessment of learning and evolution, and informally improving your epistemics.

This post tries to fill that gap by being a helpful first read for graduates/researchers of abstract disciplines interested in AI Alignment. I'd recommend using it as a complement to the other more general introductions and advice. The two following sections are just a summary of general community knowledge. The advice sections do include some new insights and opinions which I haven't seen comprehensively presented elsewhere.

Why Alignment research?

I presuppose familiarity with the basic arguments encouraging AI Alignment research. This is a somewhat risky and volatile area to work on for positive impact, given how little we understand the problem, and so I recommend having a good inside view of this field's theory of change (and our doubts about it) before committing hard to any path (and ...