trentmkelly/LessWrong-43k huggingface dataset:
What is your relationship with your self?
by Elo — 2018-09-04T01:32:19.287Z

Bearlamp | Lesswrong | Greaterwrong

I had a very deep and intense conversation with a friend. We talked about “that feeling of being alone” and that relationships will come and go and really, “I am all that I can guarantee that I have” at the end of the day. My conversation partner asked me, “How do I be a friend to myself?”

And I clicked into a realisation.

Tighten your seat-belt and settle down for an epic dojo exercises that you weren’t expecting

Non-duality. Differentiation and integration.

One concept that meditation books (like: Pointing Out The Great Way, The Attention Revolution, The Mind Illuminated, Mastering The Core Teachings of the Buddha (FREE) ) talk about is the concept of non-duality (between many things, but this time specifically) between the observer of the thoughts and the one having thoughts. There’s a meditation path where the instructions are to point attention on attention itself. When I try to watch myself putting attention on somewhere else I can, but I lose a bit of attention when I “watch” myself. So I can watch myself having a thought, and feel the confusing feeling where I try to find who is having the thought, especially if I am busy watching then *oh wait now I’m watching not thought-ing*. The instructions are to repeat until [spoiler alert] you burn a hole through the idea of duality and you conclude on a sensation level of realisation there is a non-duality of self here.

(Strange and weird so far, that’s fine)

Story Time

I am my own guide.
(this story is hard to transmit and I’m scared to share it here so I’m leaving it out but it’s significant and one day I might write it up)

I have always had my own back.

I participated in a holotropic breathwork workshop. In short, the instructions are to breathe fast, deep and without breaks until you start seeing things. Then breathe into the experience. Without getting technical, there’s something about oxygenation, lack of carbon dioxide, heart rate, and lower brain access that causes interesting experiences, something about oxygenation that feels healing and something about extended duration and the right thematic music to give exciting realisations.

I followed the instructions and I found myself a little too good at breathing. I breathed so fast that I missed it. Then I worked out what had happened and decided to slow down, breathe lightly and see if I could find the right balance of where ex...
Moral realism and AI alignment
by Caspar Oesterheld — 2018-09-03T18:46:44.266Z
Link: https://casparoesterheld.com/2018/08/06/moral-realism-and-ai-alignment/

“Abstract”: Some have claimed that moral realism – roughly, the claim that moral claims can be true or false – would, if true, have implications for AI alignment research, such that moral realists might approach AI alignment differently than moral anti-realists. In this post, I briefly discuss different versions of moral realism based on what they imply about AI. I then go on to argue that pursuing moral-realism-inspired AI alignment would bypass philosophical and help resolve non-philosophical disagreements related to moral realism. Hence, even from a non-realist perspective, it is desirable that moral realists (and others who understand the relevant realist perspectives well enough) pursue moral-realism-inspired AI alignment research.
Alignment Newsletter #22
by Rohin Shah — 2018-09-03T16:10:01.116Z

Highlights

AI Governance: A Research Agenda (Allan Dafoe): A comprehensive document about the research agenda at the Governance of AI Program. This is really long and covers a lot of ground so I'm not going to summarize it, but I highly recommend it, even if you intend to work primarily on technical work.

Technical AI alignment

Agent foundations

Agents and Devices: A Relative Definition of Agency (Laurent Orseau et al): This paper considers the problem of modeling other behavior, either as an agent (trying to achieve some goal) or as a device (that reacts to its environment without any clear goal). They use Bayesian IRL to model behavior as coming from an agent optimizing a reward function, and design their own probability model to model the behavior as coming from a device. They then use Bayes rule to decide whether the behavior is better modeled as an agent or as a device. Since they have a uniform prior over agents and devices, this ends up choosing the one that better fits the data, as measured by log likelihood.

In their toy gridworld, agents are navigating towards particular locations in the gridworld, whereas devices are reacting to their local observation (the type of cell in the gridworld that they are currently facing, as well as the previous action they took). They create a few environments by hand which demonstrate that their method infers the intuitive answer given the behavior.

My opinion: In their experiments, they have two different model classes with very different inductive biases, and their method correctly switches between the two classes depending on which inductive bias works better. One of these classes is the maximization of some reward function, and so we call that the agent class. However, they also talk about using the Solomonoff prior for devices -- in that case, even if we have something we would normally call an agent, if it is even slightly suboptimal, then with enough data the device explanation will win out.

I'm not entirely sure why they are studying this problem in particular -- one reason is explained in the next post, I'll write more about it in that section.

Bottle Caps Aren't Optimisers (Daniel Filan): The previous paper detects optimizers by studying their behavior. However, if the goal is to detect an optimizer before deployment, we need to determine whether an algorithm is performing optimization by study...
Resurrection of the dead via multiverse-wide acausual cooperation
by avturchin — 2018-09-03T11:21:32.315Z

TL;DR: Measure decline in random mind creation may be prevented if we take into account very large number of random mids created in other universes.

Summary: P.Almond suggested the idea of the resurrection of the dead via a quantum random generator which creates a random mind, but such an approach has several problems: non-human beings in our world, non-necessary suffering of non-perfect copies, and measure decline.

Here I suggest three patches, which prevent most of the undesired effects:

1. Human mind matrix to prevent pure random minds appearing.

2. Digital immortality data to create a person which satisfies all known external expectations, and the use of randomness only to fill unknown information.

3. Multiverse-wide cooperation for the “cross-resurrection” of the dead between multiple worlds via quantum random minds, so the total measure of all resurrected people will not decline.

1. Introduction

Almond in “Many-Worlds Assisted Mind Uploading: A Thought Experiment” suggested the following idea about the resurrection of the dead by the use of a quantum random generator, which would create a random mind within a computer (Almond, 2006):

[A technician who lost someone’s brain scan file] writes a computer program which takes input from a physical system. The physical system, known as a quantum event generator, generates "1"s and "0"s randomly as a result of quantum events. The program will use the physical system to tell it what sequence of "1"s and "0"s will be used to try to recreate the lost scan file. The program starts with an empty scan file which will be filled with "1"s and "0"s.

If the many-worlds interpretation of quantum mechanics is correct, all possible minds will appear in separate timelines starting from the moment of random mind creation, which would mean the resurrection of everyone from his own point of view. However, this approach will a) not help an outside observer, who wants to resurrect a relative, for instance, as the observer would see only a random mind, and b) the quantum "measure" of existence of each mind will be infinitely small.

2. Problems of Almond’s approach

To illustrate the problems with quantum mind uploading, I will explore a simplified thought experiment where only names will be restored using quantum mind uploading. First, here is what Almond suggested:...
A Process for Dealing with Motivated Reasoning
by Vaughn Papenhausen — 2018-09-03T03:34:11.650Z

Epistemic status: Had a valley of bad rationality problem, did some thinking about how to solve it in my own case, thought the results might be useful for other people who have the same problem. (To be clear, I don't yet claim this solution works, I haven't really tried it yet, but it seems like the kind of thing that might work. The claim here is "this might be worth trying" rather than "this works" and the target audience is myself and people sufficiently similar to me)

Epistemic effort: Maybe 2-3 hours of writing cumulatively. Thanks to Elephantiskon for feedback.

EDIT: Here's a summary of the post, which I was told was much clearer than the original:

There's a thing people (including me) sometimes do, where they (unreflectively) assume that the conclusions of motivated reasoning are always wrong, and dismiss them out of hand. That seems like a bad plan. Instead, try going into System II mode and reexamining conclusions you think might be the result of motivated reasoning, rather than immediately dismissing them. This isn't to say that System II processes are completely immune to motivated reasoning, far from it, but "apply extra scrutiny" seems like a better strategy than "dismiss out of hand."

This habit of [automatically dismissing anything that seems like it might be the result of motivated reasoning] can lead to decision paralysis and pathological self-doubt. The point of this post is to correct for that somewhat.

It sometimes seems like a substantial fraction of my reasoning is driven by an awareness of the insidiousness of motivated reasoning, and a desire to avoid it. In particular, I sometimes have thoughts like the following:

Brain: I would like to go to philosophy graduate school.

Me: But 80,000 hours says it's usually not a good idea to go to philosophy graduate school...

Brain: But none of the other options I can come up with seem especially realistic. Combine that with the fact that grad school can be made to be a good path if you do it right, and it actually seems like a pretty good option.

Me: But I started off wanting to go to graduate school because it seemed like fun. Seems pretty suspicious that it would turn out to be my best option, despite the fact that, according to 80k, for most EAs it's not. Are you sure you're not engaging in motivated reasonin

...
Impact Measure Desiderata
by TurnTrout — 2018-09-02T22:21:19.395Z

Previously: Worrying about the Vase: Whitelisting, Overcoming Clinginess in Impact Measures

If we can penalize some quantity of "impact on the world", we can have unaligned agents whose impact - and thereby negative effect - is small.

The long-term goal of impact measure research is to find a measure which neatly captures our intuitive understanding of "impact", which doesn't have allow cheap workarounds, which doesn't fail in really weird ways, and so on. For example, when you really think through some existing approaches (like whitelisting), you see that the impact measure secretly also applies to things we do.

No approaches to date meet these standards. What do we even require of an impact measure we hope to make safe for use with arbitrarily powerful agents?

ETA 7/31/2020: I no longer endorse this way of grading impact measures. See instead Reframing Impact.

Desiderata

Goal-Agnostic

The measure should be compatible with any original goal, trading off impact with goal achievement in a principled, continuous fashion.

Example: Maximizing the reward minus impact.

Why: Constraints seem too rigid for the general case.

Value-Agnostic

The measure should be objective, and not value-laden:

"An intuitive human category, or other humanly intuitive quantity or fact, is value-laden when it passes through human goals and desires, such that an agent couldn't reliably determine this intuitive category or quantity without knowing lots of complicated information about human goals and desires (and how to apply them to arrive at the intended concept)."

Example: Measuring what portion of initially accessible states are still accessible versus a neural network which takes two state representations and outputs a scalar representing how much "bad change" occurred.

Why: Strategically, impact measures are useful insofar as we suspect that value alignment will fail. If we substantially base our impact measure on some kind of value learning - you know, the thing that maybe fails - we're gonna have a bad time. While it's possible to only rely somewhat on a vaguely correct representation of human preferences, the extent to which this representation is incorrect is the (minimal) extent to which our measure is incorrect. Let's avoid shared points of failure, shall we?

Practically, a robust value-sensitive impact measure is value-a... ,
When wishful thinking works
by AlexMennen — 2018-09-01T23:43:00.858Z

This idea is due to Scott Garrabrant.

Suppose you have propositions φ1,...,φn, and you want to form beliefs about whether they are true; specifically, you want to form a joint probability distribution P over the events φ1,...,φn. But there’s a catch: these propositions might refer to the joint probability distribution you come up with. If φ1 is the claim that P(φ1)<.5, then you have no way to assign probabilities in a well-calibrated way. But suppose these propositions depend continuously on the probabilities you assign to them. For instance, φ1 could be defined so that its “true” probability is 1−P(φ1), where P means probability that you assigned. Let f be the function from the space of joint probability distributions over φ1,...,φn to itself that sends each probability distribution μ to the true probability distribution that would result if you believed μ. In this case, you can be well-calibrated by letting P(φ1)=.5. By Brouwer’s fixed point theorem, there will always be a way to assign probabilities in a well-calibrated way.

But f could have multiple fixed points. Which one is right? You get to pick; whichever fixed point you decide to believe ends up being correct, since they are fixed points of the function determining the true probabilities from your beliefs. Cases in which there are multiple such fixed points are cases in which you actually can make something be true by believing it. So you may as well believe the fixed point according to which you have the highest expected utility.

As an example, suppose you’re suffering from an ailment that can be cured by placebo, and the placebo works even if you know it’s just a placebo, provided you believe that the placebo will work. When given a pill that you know is a placebo, you may as well believe that it will cure you, since then you’ll be right, and get better.

Related to the question of what to believe is the question of what actions to take. The traditional answer is to take the action which has the highest expected utility. Another possible answer is to act the way that you believe you will act. If we do this, then f will have lots of fixed points: for any probability distribution over actions we could take, if we believe that we will take actions according to those probabilities, then we will be correct. And picking the fixed point that maximizes expected utility recovers the original rule of picking ...
Cooperative Oracles
by Diffractor — 2018-09-01T08:05:55.899Z

The long-awaited continuation of this sequence.

Credit to Sam, Tsvi, Nisan, Scott, me, and Giles (who helped make the figures)

It may or may not become a paper, but if it doesn't, it's best for it to not be locked in a folder of drafts. This will also recap material from previous posts and papers.

Refresher on Reflective Oracles

Let M denote an arbitrary probabilistic Turing machine with oracle access, and MO refer to the same machine equipped with the oracle O. It is important to remember that M refers to a randomized algorithm, while MO refers to the algorithm equipped with a particular oracle, so it doesn't make sense to ask what the output of M is, only what the output of MO is.

Reflective oracles as originally defined here are oracles which accurately answer queries about whether a probabilistic Turing machine with access to the same oracle, MO, output a 1 with probability greater than or less than p, by outputting 1 for "yes", 0 for "no", and randomizing between the two if the probability of MO outputting 1 is exactly p. This is more expressive than it may seem at first, because it is possible to query the oracle about a probabilistic oracle machine that runs MO and checks whether the output has some arbitrary computable property.

It is also possible to use reflective oracles to make the probability distribution over the output of one algorithm vary continuously with respect to the probability distribution over the output of another algorithm, via repeated oracle calls and randomization, as demonstrated here.

Reflective oracles can be thought of as akin to halting oracles, except that they accurately answer questions about the class of probabilistic Turing machines with access to the same oracle, which halting oracles cannot do. If the diagonalization argument against the existence of a halting oracle is applied to a reflective oracle, where an algorithm invokes the oracle on itself, and outputs the bit which it is the least likely to output, a reflective oracle will randomize between outputting 0 and 1, but for any p<0.5, if the oracle is asked whether the algorithm outputs 1 with greater than probability p, it will respond "yes".

This ability to accurately answer questions about themselves makes reflective oracles well-suited for studying the unbounded computing power limit of game theory, and situations where...
On memetic weapons
by ioannes — 2018-09-01T03:25:36.489Z

[status: speculative, probably a fake framework]

July 17, 2018. Johannesburg, South Africa.

Barack Obama is giving a speech in memoriam of Nelson Mandela. 15,000 people fill a stadium, listening.

Here's a transcript.

Here's an excerpt from that transcript:

To make democracy work, Madiba [Mandela] shows us that we also have to keep teaching our children, and ourselves, to engage with people not only who look different but who hold different views. This is hard.

Most of us prefer to surround ourselves with opinions that validate what we already believe. You notice the people who you think are smart are the people who agree with you. Funny how that works. But democracy demands that we’re able also to get inside the reality of people who are different than us so we can understand their point of view. Maybe we can change their minds, but maybe they’ll change ours.

And you can’t do this if you just out of hand disregard what your opponents have to say from the start. And you can’t do it if you insist that those who aren’t like you – because they’re white, or because they’re male – that somehow there’s no way they can understand what I’m feeling, that somehow they lack standing to speak on certain matters.

Madiba lived this complexity. In prison, he studied Afrikaans so that he could better understand the people who were jailing him. And when he got out of prison he extended a hand to those who had jailed him, because he knew that they had to be a part of the democratic South Africa that he wanted to build. “To make peace with an enemy,” he wrote, “one must work with that enemy, and that enemy becomes one’s partner.”

When Barack Obama starts talking about how it's important to listen to people, even if they're white men – that's something new.

I'm worried about systemic oppression. These days, this worry manifests as concern about the war on drugs. It's a bad, bad thing, and it was explicitly architected to oppress people of color.

Even if the current agents of the drug war don't hold racist views, they are still carrying out the work of a machine that was built by racists to achieve racist ends.

I'm also worried about the state of discourse on the political Left. Discourse quality on the Right is also quite bad, but traditionally the Left has controlled our institutions of free inquiry, so degradation of the Left's discourse...
Open Thread September 2018
by Elo — 2018-08-31T21:38:19.118Z

If it’s worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

What accomplishments are you celebrating from the last month?

What are you reading?

What reflections do you have for yourself or others from the last month?

What have you tried out this month?

(Teaser for my next post) What is your relationship with yourself?
Psychology Replication Quiz
by Ben Pace — 2018-08-31T18:54:54.411Z
Link: https://80000hours.org/psychology-replication-quiz/

The great guys over at 80,000 Hours made a quiz for that recent replication study (previously discussed on LW), where you can find out if you can predict as well as the scientists or the prediction market. Description:

Can you guess which psychology experiments had correct findings and which were bogus, just from reading a brief description of their results?

Depending on how long you want to play, we'll describe 10, 15 or 21 psychology findings published in Nature and Science, and you'll have to guess whether a replication, with a much larger sample size, got the same result.

Before starting, the people who organised these 21 replications asked expert psychologists to predict which results would hold up. We'll show you how you compare to their performance at the end! (And give you links to all the papers.)
Bottle Caps Aren't Optimisers
by DanielFilan — 2018-08-31T18:30:01.108Z

Crossposted from my blog.

One thing I worry about sometimes is people writing code with optimisers in it, without realising that that's what they were doing. An example of this: suppose you were doing deep reinforcement learning, doing optimisation to select a controller (that is, a neural network that takes a percept and returns an action) that generated high reward in some environment. Alas, unknown to you, this controller actually did optimisation itself to select actions that score well according to some metric that so far has been closely related to your reward function. In such a scenario, I'd be wary about your deploying that controller, since the controller itself is doing optimisation which might steer the world into a weird and unwelcome place.

In order to avoid such scenarios, it would be nice if one could look at an algorithm and determine if it was doing optimisation. Ideally, this would involve an objective definition of optimisation that could be checked from the source code of the algorithm, rather than something like "an optimiser is a system whose behaviour can't usefully be predicted mechanically, but can be predicted by assuming it near-optimises some objective function", since such a definition breaks down when you have the algorithm's source code and can compute its behaviour mechanically.

You might think about optimisation as follows: a system is optimising some objective function to the extent that that objective function attains much higher values than would be attained if the system didn't exist, or were doing some other random thing. This type of definition includes those put forward by Yudkowsky and Oesterheld. However, I think there are crucial counterexamples to this style of definition.

Firstly, consider a lid screwed onto a bottle of water. If not for this lid, or if the lid had a hole in it or were more loose, the water would likely exit the bottle via evaporation or being knocked over, but with the lid, the water stays in the bottle much more reliably than otherwise. As a result, you might think that the lid is optimising the water remaining inside the bottle. However, I claim that this is not the case: the lid is just a rigid object designed by some optimiser that wanted water to remain inside the bottle.

This isn't an incredibly compelling counterexample, since it doesn't qualify as an optimiser according to Yudkowsky's definit

...