trentmkelly/LessWrong-43k huggingface dataset:
You are probably underestimating how good self-love can be
by Charlie Rogers-Smith — 2021-11-14T00:41:35.011Z

I am very grateful to the following people, in general, and for their helpful feedback on this post: Nick Cammarata, Kaj Sotala, Miranda Dixon-Luinenburg, Sam Clarke, Mrinank Sharma, Matej Vrzala, Vlad Firoiu, Ollie Bray, Alan Taylor, Max Heitmann, Rose Hadshar, and Michelle Hutchinson.

I was on a plane to Malta when I realised I had lost something precious. I was struggling to meditate. I knew there was some disposition that made meditation easier for me in the past, something to do with internal harmony and compassion and affection. Alas, these handles failed to impact me. On a whim, I decided to read and meditate on some of my notes. 3h later, I had recovered the precious thing. It was one of the most special experiences of my life. I felt massive relief, but I was also a little scared--I knew that this state would likely pass. I made a promise to myself to not forget what I felt like, then, and to live from that place more. This post is, in part, an attempt to honour that promise. I spent most of my holiday in Malta reading about and meditating on the precious thing, and I now feel like I'm in a place where I can share something useful.

This post is about self-love. Until recently, I didn’t know that self-love was something I could aim for; that it was something worth aiming for. My guess is that I thought of self-love as something vaguely Good, a bit boring, a bit of a chore, a bit projection-loaded (I’m lovable; I love me so you can love me too), and lumped together with self-care (e.g. taking a bath). Then I found

Nick Cammarata

on Twitter and was blown away by the experiences he was describing. Nick tweeted about self-love from Sep 2020 to May 2021, and then moved on to other things. His is the main body of work related to self-love that I’m aware of, and I don't want it to be lost to time. My main intention with this post is to summarise Nick’s work and build on it with my experiences; I want to get the word out on self-love, so that you can figure out whether it’s something you want to aim for. But I'm also going to talk a little about how to cultivate it and the potential risks to doing that. One caveat to get out of the way is that I’m a beginner--I’ve been doing this stuff for under a year, for way less than 1h/day. Another is that I expect that my positive experiences with self-love are strongly linked to me being moderately depressed before I st...
Coordination Skills I Wish I Had For the Pandemic
by Raemon — 2021-11-13T23:32:11.510Z

In the previous post I noted the pandemic as a wakeup call for coordination-capabilities. There are a few different lenses to look at this through.

Today, I want to look through the lens of skills I wish I had had.

There are many pandemic-relevant skills that aren’t (especially) related to coordination. It’d have been handy to have a strong background in epidemiology, for example. But here I want to focus in skills that I think would be generally useful, and which bear on solving coordination problems in novel circumstances.

Coordination Bottlenecked On Skills

Many day-to-day coordination activities require me to have skills.

For example, I might buy groceries. This is a coordination activity – it’s only worth a farmer’s time to grow extra food if a middleman is going to buy it and transport it. It’s only worth a middleman’s time to transport it if a grocery store will stock it. It’s only worth the grocery store’s time to stock it if people will pay for it. We don’t know exactly what things people want. Me buying groceries sends a signal to the economic system to produce more apples and milk.

Participating in this system is a lot easier if I have a collection of skills, such as reading and speaking English, navigating a grocery aisle, doing basic arithmetic so I know how much my apples and milk will cost. It includes soft skills like “the social norms involved with talking to the cashier.”

Those skills didn’t come for free. Society invested in me having them. It could have invested in me having different skills (such as martial prowess), which would enable different coordination patterns.

During the 2 months of the pandemic, I found myself wishing I had several new skills that I hadn’t previously developed. Each skill would have taken me a month or so to really wrap my head around if I were trying hard. The fact that I needed all five at once felt very overwhelming. I was struggling to be functional at all and executing on the skills I already had.

Each skill was something I think would have been beneficial in single-player mode. I have a speculative sense that if multiple people around me had them, it’d have enabled compound returns. If people reliably had them, it may have been possible to build more complex systems on top of them.

Knowing what I value

When I go to the grocery store and don’t have enough money to buy both apples and milk, I have to d...
Sci-Hub sued in India
by Connor_Flexman — 2021-11-13T23:12:40.559Z

Sci-Hub is being sued by Elsevier, again, this time in India. Two other large publishers are also on the suit.

Sci-Hub has been getting sued in lots of Western countries over the past 6 years and losing default judgments because they didn’t really defend. In India, they have a better shot:

The founder Alexandra Elbakyan is actually mounting a defense this time
India has comparatively lenient copyright law
India is poor enough that the "public interest" angle is multiplied manyfold due to Elsevier’s absurd profit margin and prices

Last, it's higher-stakes than normal, because in India the plaintiffs have requested a dynamic injunction. This is a recent legal mechanism that would let the plaintiffs obtain court injunctions against ISPs serving any Sci-Hub domain with very little effort on very short notice. This might make it vastly more difficult for Indians to access Sci-Hub than e.g. US citizens after it was ruled illegal here.

I am helping with Sci-Hub’s legal defense in the small ways possible, along with Stag Lynn and formerly John Steidley. [ETA: I first heard about this case through Elizabeth Van Nostrand—thank you!] The next hearing is Tuesday, November 16th, but the case has been going for over a year and will likely continue on for at least a few weeks longer.

This post will primarily cover the larger situation, but first I wanted to state our two needs:

We would like to hear from contacts willing to write or sign onto amicus briefs on behalf of Sci-Hub. High-level academics are especially useful and likely would have to do the least work. Please reach me at connorflexman at gmail if you are interested in this possibility for yourself or another.

We’ve focused on the legal side so far, while ignoring media. If you are interested in joining the project as lead or co-lead of a media thrust for a month, email me at the same. It won’t happen unless we get another person or two. Causal impact from media is complicated and hard to cash out but we can talk about that. I will pay ~$20/hr if that helps, but don’t have actual funding for real salaries given the short notice. If you’d be great at this but need much more money, let me know too. If you do reach out, be aware the odds are very low—hiring usually takes time that we don’t quite have here, I’m risk-averse because of risk of backfire, etc.

Now, onto the case itself.

What is Sci-Hub?

Sci-Hub is a website th...
What’s the likelihood of only sub exponential growth for AGI?
by M. Y. Zuo — 2021-11-13T22:46:25.277Z

A possible scenario could be that the first confirmed AGI(s) are completely unimpressive, i.e. with equivalent or less capacities than an average 10 year child. And then tremendous effort is put into ’growing’ its capacities with the best results yielding growth only along a sub exponential curve, probably with some plateaus as well. So that any AGI may take generations to become truly superhuman.

I’m asking this as I haven’t come across any serious prior discussion other than this: https://www.lesswrong.com/posts/77xLbXs6vYQuhT8hq/why-ai-may-not-foom, though admittedly my search was pretty brief.

Is there any serious expectation for this kind of scenario?
Comments on Carlsmith's “Is power-seeking AI an existential risk?”
by So8res — 2021-11-13T04:29:30.673Z

The following are some comments I gave on Open Philanthropy Senior Research Analyst Joe Carlsmith’s Apr. 2021 “

Is power-seeking AI an existential risk?

”, published with permission and lightly edited. Joe replied; his comments are included inline. I gave a few quick replies in response, that I didn't want to worry about cleaning up; Rob Bensinger has summarized a few of them and those have also been added inline.

I think Joe Carlsmith's report is clear, extensive, and well-reasoned. I also agree with his conclusion, that there's at least a 5% chance of catastrophic risk from AI by 2070. In fact, I think that number is much too low. I'll now attempt to pinpoint areas of disagreement I have with Joe, and put forth some counterarguments to Joe's position.

Warning: this is going to be a bit quick-and-dirty, and written in a colloquial tongue.

I'll start by addressing the object-level disagreements, and then I'll give a few critiques of the argument style.

On the object level, let's look at Joe's "shorter negative" breakdown of his argument in the appendix:

Shorter negative:

By 2070:

1. It will become possible and financially feasible to build APS AI systems.

65%

2. It will much more difficult to build APS AI systems that would be practically PS-aligned if deployed than to build APS systems that would be practically PS-misaligned if deployed, but which are at least superficially attractive to deploy anyway | 1.

35%

3. Deployed, practically PS-misaligned systems will disempower humans at a scale that constitutes existential catastrophe | 1-2.

20%

Implied probability of existential catastrophe from scenarios where all three premises are true: ~5%

My odds, for contrast, are around 85%, 95%, and 95%, for an implied 77% chance of catastrophe from these three premises, with most of our survival probability coming from "we have more time than I expect". These numbers in fact seem a bit too low to me, likely because in giving these very quick-and-dirty estimates I failed to account properly for the multi-stage fallacy (more on that later), and because I have some additional probability on catastrophe from scenarios that don't quite satisfy all three of these conjuncts. But the difference between 5% and 77% is stark enough to imply signif...
A FLI postdoctoral grant application: AI alignment via causal analysis and design of agents
by PabloAMC — 2021-11-13T01:44:35.209Z

TL;DR: This is a proposal I wrote for the Future of Life postdoctoral grants, on the topic of AI Safety.
The topic of this proposal is being able to model human preferences in a causal model. This is somewhat similar to the agenda of value learning, but more related to that one of causal incentive design https://causalincentives.com/, which I think is interesting enough that more people should be aware of its existence.
I think there might be ideas worth considering, but given that I am no expert in the topic of causal models there might be important technical or strategic errors, which I would be glad to know about. In other words, this is more a structured compilation of half-baked research ideas, rather than well-grounded research projects. But I think they might be worth considering nevertheless.

Introduction

The most straightforward approaches to intent AI alignment usually consist of what is called value learning: task the agent with the problem of simultaneously figuring out what the human wants, and optimizing for it. On the other hand, providing additional causal structure to the problem might improve the robustness of the system to new environments, make it more interpretable, and able to learn from scarce human interactions.

The causal incentives agenda [1] is the AI Safety agenda based on modeling and analyzing the incentives of an agent to ensure that it is aligned with human preferences. It leverages the framework of causal reasoning and inference [2] and was originally conceived as a way to study which causal models do not incentivize reward function tampering (the agent modifying its reward function to attain higher reward) and reward function input tampering (the agent self-deceiving to get a higher reward) [3, 4]. The latter problem also includes the system manipulating a human that provides feedback to make him/her more predictable and get a higher reward. Overall, the aim of this agenda is that since we should not expect to be able to box powerful agents into constrained ways of behaving, we should pose the correct incentives for agents to work in the way we intended them to.

Later on, more formal definitions were introduced, including the Response Incentive and the Instrumental Control Incentive [5], working over Causal Influence Diagrams, which are Structural Causal Models with Decision and Utility nodes. Extension to multiple agents was ...
Is Functional Decision Theory still an active area of research?
by Grant Demaree — 2021-11-13T00:30:36.092Z

Looks like discussion of the issue died down in recent years, bit it strikes me as really important to have a good decision theory. So:

Are folks still actively working on this, and do you see it as a promising area?

It seems like the academic community was dismissive of FDT. Did anyone find a genuine reason to reject it?
Average probabilities, not log odds
by AlexMennen — 2021-11-12T21:39:19.835Z

Let's say you want to assign a probability to some proposition X. Maybe you think about what odds you'd accept bets at, and decide you'd bet on X at 1:99 odds against X, and you'd bet against X at 1:9 odds against X. This implies you think the probability of X is somewhere between 1% and 10%. If you wouldn't accept bets in either direction at intermediate odds, how should you refine this interval to a point estimate for the probability of X? Or maybe you asked two experts, and one of them told you that X has a 10% probability of being true, and another told you that X has a 1% probability of being true. If you're inclined to just trust the experts as you don't know anything about the subject yourself, and you don't know which expert to trust, how should you combine these into a point estimate for the probability of X?

One popular answer I've seen is to take the geometric mean of the odds ratios (or averaging the log odds). So in either of the above scenarios, the geometric mean of 1:9 and 1:99 is 1:√9∗99≈1:30, so you would assign a probability of about 3.2% to X. I think this is a bad answer, and that a better answer would be to average the probabilities (so, in these cases, you'd average 1% and 10% to get a probability of 5.5% for X). Here are many reasons for this:

Probabilities must add to 1. The average log odds rule doesn't do this. Let's try an example. Let's suppose you've got some event A, and you ask three experts what the probability of A is. Expert 1 tells you that A has probability 50%, while experts 2 and 3 both say that A has probability 25%. The geometric mean of 1:1, 1:3, and 1:3, is about 1:2.1, so we get an overall probability of 32.5%, just less than 1/3. But now consider two more events, B and C, such that exactly one of A, B, and C must be true. It turns out that expert 1 gives you a probability distribution 50% A, 25% B, 25% C, expert 2 gives you a probability distribution 25% A, 50% B, 25% C, and expert 3 gives you a probability distribution 25% A, 25% B, 50% C. The average log odds rule assigns a 32.5% probability to each of A, B, and C, even though you know one of them must occur. Or, put differently, the average log odds rule assigns probability 32.5% to A, 32.5% to B, and 67.5% to "A or B", violating additivity of probabilities of disjoint events. Averaging probabilities assigns probability 1/3 to each of A, B, and C, as any rul...
A Defense of Functional Decision Theory
by Heighn — 2021-11-12T20:59:18.545Z

This post is an attempt to refute an article offering critique on Functional Decision Theory (FDT). If you’re new to FDT, I recommend reading

this introductory paper

by Eliezer Yudkowsky & Nate Soares (Y&S). The critique I attempt to refute can be found here:

A Critique of Functional Decision Theory

by

wdmacaskill

. I strongly recommend reading it before reading this response post.

The article starts with descriptions of Causal Decision Theory (CDT), Evidential Decision Theory (EDT) and FDT itself. I’ll get right to the critique of FDT in this post, which is the only part I’m discussing here.

“FDT sometimes makes bizarre recommendations”

The article claims “FDT sometimes makes bizarre recommendations”, and more specifically, that FDT violates guaranteed payoffs. The following example problem, called Bomb, is given to illustrate this remark:

“Bomb.

You face two open boxes, Left and Right, and you must take one of them. In the Left box, there is a live bomb; taking this box will set off the bomb, setting you ablaze, and you certainly will burn slowly to death. The Right box is empty, but you have to pay $100 in order to be able to take it.

A long-dead predictor predicted whether you would choose Left or Right, by running a simulation of you and seeing what that simulation did. If the predictor predicted that you would choose Right, then she put a bomb in Left. If the predictor predicted that you would choose Left, then she did not put a bomb in Left, and the box is empty.

The predictor has a failure rate of only 1 in a trillion trillion. Helpfully, she left a note, explaining that she predicted that you would take Right, and therefore she put the bomb in Left.

You are the only person left in the universe. You have a happy life, but you know that you will never meet another agent again, nor face another situation where any of your actions will have been predicted by another agent. What box should you choose?”

The article answers and comments on the answer as follows:

“The right action, according to FDT, is to take Left, in the full knowledge that as a result you will slowly burn to death. Why? Because, using Y&S’s counterfactuals, if your algorithm were to output ‘Left’, then it would also have outputted ‘Left’ when the predictor made the simulation of you, and there would be no bomb in the box, and you could save yourself $100 by taking Left. In contrast, the

...
Why I'm excited about Redwood Research's current project
by paulfchristiano — 2021-11-12T19:26:26.159Z

Redwood Research's current project

is to train a model that completes short snippets of fiction without outputting text where someone gets injured. I'm excited about this direction and wanted to explain why.

(Disclaimer: I originally proposed this project, and am on Redwood’s board. So I may be biased in favor of Redwood and especially in favor of this problem being a good one to work on.)

Relevance to deceptive alignment

I think about alignment risk as having two big drivers :

Your AI is smart enough that you couldn’t even tell if it did something terrible until it’s too late, e.g. until you’ve already implemented the proposed plan and it killed everyone.
Your AI looks nice during training while humans are in control, but there is a treacherous turn once it is deployed or becomes smart enough that we can’t correct a problem.

I spend most of my time working on problem #1, but I think both problems are very important.

One of the main concrete candidate techniques for fixing problem #2 is to run the AI on tons of simulated inputs designed to look like the kind of test-time situation where your AI could kill you and take your stuff (adversarial training). But in fact these are just simulations, and if your model tries to do something really bad you just do a gradient update and move on. As long as you've solved problem #1, you can hopefully recognize if the model tries to do something bad in simulation without needing to run it in the real world.

Depending on the details you may not actually want to train on these examples---once you see a problem you may want to do something different or else run the risk of overfitting---but at any rate you definitely want to see them in order to know that there is a problem.)

I think adversarial training or evaluation look like key ingredients whether we are pursuing a principled approach to alignment (a more sophisticated version of this process are a key ingredient in my own "best guess" alignment plan) or just "winging it" (in which case I think adversarial evaluation is especially important).

Redwood’s current project

I’m discussing what

Redwood’s post

calls “step 1,” namely training a classifier that is able to recognize violent completions with an extremely low false negative rate. Ideally they would go for 0 false negatives, but in practice they are just going for as many 9s as they can get (think more like 10-30 than 99.99%---w...
Stop button: towards a causal solution
by tailcalled — 2021-11-12T19:09:45.967Z

The stop button problem is the problem that if you have an AGI, you might want to give it a stop button that humans can press, so that if it is misaligned, it can get fixed. It is a problem because it is a convergent instrumental goal to prevent yourself from being turned off. We want the AI to respect the human's choice; to allow or even help whatever the human chooses to come into fruition, while also not having any motivation to manipulate the human.

To me, the appropriate solution seems like it involves causality. Specifically, for some random variable X, define Xs to be the value of X if, counterfactually, a human tries to press the stop button, and Xf to be the value of X if, counterfactually, no human tries to press the stop button. Then define the variable B to mean that the stop button gets pressed and the AI gets shut down (within some reasonable context, e.g. maybe in less than 1 minute from now), and V to be whichever unsafe utility function you originally had. Now give the AI a policy according to the following utility function:

U=Bs+Vf

That is, humans wanting to press the button should lead to the button being pressed and the AI stopping, and humans not wanting to press the button should lead to whatever else we programmed the AI to achieve.

Properties

Does this suffer from the usual stop button problems? Let's investigate:

Will the AI try to prevent a human from pressing the button? No. While the human pressing the button would reduce V by a lot, the AI will assume that the human changes their mind and stops pressing the button when evaluating Vf, because that's what the counterfactual would involve. Since a human who is about to press the button and then decides not to will not press the button, Vf is not reduced and the AI is not worried about the button being pressed.
Will the AI try to press the button itself, regardless of whether it should be pressed? No. If the AI presses the button, then the button gets pressed regardless of whether a human tries to press it or not. But that would reduce Vf, and so it would reduce U. Meanwhile, it would fail to increase Bs, because if a human wants to press the button, then the AI won't need to do it. (It could increase Bs if the AI could see that a human wouldn't be able to press the button even

...