trentmkelly/LessWrong-43k huggingface dataset:
Increasing day to day conversational rationality
by Hazard — 2017-11-16T21:18:37.424Z

One of the things that rationality has given me is a greater appreciation for the variety of ways one can be wrong. It often seems like people who haven’t given this topic much thought have a model of “being wrong” as one-dimensional quality someone can have with varying levels of intensity. You can be completely wrong, sort of wrong, or 0% wrong, but assigning a value to one’s wrongness is all the nuance this model gives.

Given such a model, there are very few failure points in a system of two people disagreeing about something. One of them is wrong and the other is right. Or maybe they’re both sort of right. Having a one-dimensional model of being wrong makes it harder to see other failure points. What if the question is wrong? What if there isn’t actually a disagreement between you and it’s all a misunderstanding? What if you aren’t actually arguing about what you are arguing about? What if you disagree because the other person's reasoning process is setting off your alarm bells?

Because of the weirdness of words, not every english sentence can be directly translated into a formal logic proposition. Thus one needs a more nuanced understanding of being wrong in order to have more fruitful conversations and arguments.

Often in conversations, I want to address some of these issues, but I feel very clunky when bringing them in. Sometimes I’m about to bring up an idea that is common in the LW cannon, but then think, “Oh, for this to make sense to them, I’d first have to explain this, and for that to make sense I’d have to explain-- ” and by then I’ve lost the thread of the conversation and just abort the idea and try to catch up. Sometimes I do bring up the previously mentioned point, but I fail to do it in a way that communicates the essence of my point, or worse, the person I’m arguing with still feels like they’re in fight mode and reflexively begins attacking my point.

It would be incredibly useful to have worked out some “personhood interface” respecting scripts that I could call whenever I noticed a particular problem that was stymying the conversation. It’s hard enough to be rational when you can slowly think things through, and a real time conversation only makes it harder. I have a hunch that there is some decent low hanging fruit in the realm of operationalizing some scripts of the above nature.

Below I’ve worked out the particulars of one script, a...
Against Shooting Yourself in the Foot
by Eliezer Yudkowsky — 2017-11-16T20:13:35.529Z

Follow-up to: Status Regulation and Anxious Underconfidence

Somehow, someone is going to horribly misuse all the advice that is contained within this book.

Nothing I know how to say will prevent this, and all I can do is advise you not to shoot your own foot off; have some common sense; pay more attention to observation than to theory in cases where you’re lucky enough to have both and they happen to conflict; put yourself and your skills on trial in every accessible instance where you’re likely to get an answer within the next minute or the next week; and update hard on single pieces of evidence if you don’t already have twenty others.

I expect this book to be of much more use to the underconfident than the overconfident, and considered cunning plots to route printed copies of this book to only the former class of people. I’m not sure reading this book will actually harm the overconfident, since I don’t know of a single case where any previously overconfident person was actually rescued by modest epistemology and thereafter became a more effective member of society. If anything, it might give them a principled epistemology that actually makes sense by which to judge those contexts in which they are, in fact, unlikely to outperform. Insofar as I have an emotional personality type myself, it’s more disposed to iconoclasm than conformity, and inadequacy analysis is what I use to direct that impulse in productive directions.

But for those certain folk who cannot be saved, the terminology in this book will become only their next set of excuses; and this, too, is predictable.

If you were never disposed to conformity in the first place, and you read this anyway… then I won’t tell you not to think highly of yourself before you’ve already accomplished significant things. Advice like that wouldn’t have actually been of much use to myself at age 15, nor would the universe have been a better place if Eliezer-1995 had made the mistake of listening to it. But you might talk to people who have tried to reform the US medical system from within, and hear what things went wrong and why.1 You might remember the Free Energy Fallacy, and that it’s much easier to save yourself than your country. You might remember that an aspect of society can fall well short of a liquid market price, and still be far above an amateur’s reach.

I don’t have good, repeatable exercises...
Status Regulation and Anxious Underconfidence
by Eliezer Yudkowsky — 2017-11-16T19:35:00.533Z

Follow-up to: Against Modest Epistemology

I’ve now given my critique of modesty as a set of explicit doctrines. I’ve tried to give the background theory, which I believe is nothing more than conventional cynical economics, that explains why so many aspects of the world are not optimized to the limits of human intelligence in the manner of financial prices. I have argued that the essence of rationality is to adapt to whatever world you find yourself in, rather than to be “humble” or “arrogant” a priori. I’ve tried to give some preliminary examples of how we really, really don’t live in the Adequate World where constant self-questioning would be appropriate, the way it is appropriate when second-guessing equity prices. I’ve tried to systematize modest epistemology into a semiformal rule, and I’ve argued that the rule yields absurd consequences.

I was careful to say all this first, because there’s a strict order to debate. If you’re going to argue against an idea, it’s bad form to start off by arguing that the idea was generated by a flawed thought process, before you’ve explained why you think the idea itself is wrong. Even if we’re refuting geocentrism, we should first say how we know that the Sun does not orbit the Earth, and only then pontificate about what cognitive biases might have afflicted geocentrists. As a rule, an idea should initially be discussed as though it had descended from the heavens on a USB stick spontaneously generated by an evaporating black hole, before any word is said psychoanalyzing the people who believe it. Otherwise I’d be guilty of poisoning the well, also known as Bulverism.

But I’ve now said quite a few words about modest epistemology as a pure idea. I feel comfortable at this stage saying that I think modest epistemology’s popularity owes something to its emotional appeal, as opposed to being strictly derived from epistemic considerations. In particular: emotions related to social status and self-doubt.

Even if I thought modesty were the correct normative epistemology, I would caution people not to confuse the correct reasoning principle with those particular emotional impulses. You’ll observe that I’ve written one or two things above about how not to analyze inadequacy, and mistakes not to make. We hear ...
Creating Welfare Biology: A Research Proposal
by ozymandias — 2017-11-16T19:06:11.492Z

[This idea came out of an ACE research workshop. I would like to thank Zach Groff and Mark Budolfson for brainstorming this idea with me, as well as ACE for offering me the opportunity to think of it.]

[Crossposted with no modifications from my blog.]

Many people in the wild-animal suffering space think it would be a good idea to make a discipline of “welfare biology”-- that is, the scientific study of wild animal welfare, the way that animal welfare studies scientifically studies domestic animal welfare. From my perspective, there are two big benefits to creating welfare biology. First, it would probably increase the number of research dollars that go to wild-animal welfare research, while reducing the opportunity cost: welfare biology would be competing with other fields of biology for funding, not with starving Africans and tortured pigs. Second, it would give us academic credibility. In most of the developed world, terrestrial wildlife often live on government land (for example, much of the United States’s wildlife lives on the quarter of US land owned by the government), which means changing government policies towards wildlife is a promising method of improving their welfare. Even in human-inhabited areas, changing government policies may be an effective way of improving wild-animal welfare. Governments are generally more likely to listen to tenured academics than they are to bloggers.

However, it is unclear to me how one creates an academic field. It is possible that people already know how academic fields form; I have not studied the subject in depth, and would welcome links from commenters. But if there is not already an academic body of work on the subject then it seems useful to do a small research project to explore how academic fields form. I think the best method is a series of qualitative case studies exploring how various relevant scientific fields formed.

I’m aware of two similar research projects in the effective altruist community. Luke Muehlhauser has written a report on early field growth. However, his report concentrates on the role of philanthropists and only touches on what non-philanthropists can do. It also mostly examines fields relevant to artificial intelligence risk research, which has some overlap with fields relevant to welfare biology but not entirely. Animal Charity Evaluators has done several case studies of other social mov...
The Darwin Game
by Zvi — 2017-11-15T23:20:00.294Z

Epistemic Status: True story

The plan is that this post will begin the sequence Zbybpu’f Nezl.

In college I once took a class called Rational Choice. Because obviously.

Each week we got the rules for, played and discussed a game. It was awesome.

For the grand finale, and to determine the winner of the prestigious Golden Shark Award for best overall performance, we submitted computer program architectures (we’d tell the professor what our program did, within reason, and he’d code it for us) to play The Darwin Game.

The Darwin Game is a variation slash extension of the iterated prisoner’s dilemma. It works like this:

For the first round, each player gets 100 copies of their program in the pool, and the pool pairs those programs at random. You can and often will play against yourself.

Each pair now plays an iterated prisoner’s dilemma variation, as follows. Each turn, each player simultaneously submits a number from 0 to 5. If the two numbers add up to 5 or less, both players earn points equal to their number. If the two numbers add up to 6 or more, neither player gets points. This game then lasts for a large but unknown number of turns, so no one knows when the game is about to end; for us this turned out to be 102 turns.

Each pairing is independent of every other pairing. You do not know what round of the game it is, whether you are facing a copy of yourself, or any history of the game to this point. Your decision algorithm does the same thing each pairing.

At the end of the round, all of the points scored by all of your copies are combined. Your percentage of all the points scored by all programs becomes the percentage of the pool your program gets in the next round. So if you score 10% more points, you get 10% more copies next round, and over time successful programs will displace less successful programs. Hence the name, The Darwin Game.

Your goal is to have as many copies in the pool at the end of the 200th round as possible, or failing that, to survive as many rounds as possible with at least one copy.

If both players coordinate to split the pot, they will score 2.5 per round.

To create some common terminology for discussions, ‘attack’ or means to submit 3 (or a higher number) more than half the time against an opponent willing to not do that, and to ‘fold’ or ‘surrender’ is to submit 2 (or a lower number) more than half the time, with ‘full surrender’ being to alwa...
Dollars for Brains
by Jacob Falkovich — 2017-11-15T18:20:00.359Z
Link: https://putanumonit.com/2017/11/15/dollars-for-brains/

Putanumonit is now accepting donations, and donations now let you rent my brain if you have use for it.

Original: https://putanumonit.com/2017/11/15/dollars-for-brains/
Rationalising humans: another mugging, but not Pascal's
by Stuart_Armstrong — 2017-11-15T12:07:05.000Z

A putative new idea for AI control; index here.

In a previous post, I used the (p,R) model to show how a value-learning agent might pull a Pascal's mugging on humans: it would change the human reward into something much easier to maximise.

But there's a much easier mugging the agent might want to pull: it might want to make the human fully rational. And this does not depend on low-probability-high reward situations.

Planners and Rewards

In the original mugging, the agent has the ability to let the human continue (event ¬S) with πSTA, a "standard" human policy according to some criteria, or coercively change their policy (event S) into πMAX, another policy.

Thus the actual human policy is πS/M, the policy that follows πSTA given ¬S, and πMAX given S.

There are two rewards: RSTA, the standard human reward according to some criteria, and RS/M a reward that is RSTA if the agent chooses ¬S (i.e. chooses not to coerce them), and RMAX if the agent chooses S (coerce them into a new policy), where RMAX is a reward rationally maximised by πMAX.

Thus the agent has two choices: whether to coercively change the human (S), and, if so, which RMAX (and hence πMAX) to pick.

There are two planners: pr, the planner under which human behaviour is rational, given S, and p, a "reasonable" planner that maps RSTA to πS/M, the actual human policy. The planer p is assumed to note the fact that action S is overriding human rewards, since πMAX is - presumably - poor at maximising RSTA.

There are three compatible pairs: (pr,RS/M), (p,RS/M), and (p,RMAX). We'll assume the agent has learnt well, and that the probability of RMAX is some ϵ<<1.

The full equation: rationalise the human

Let V(R,π) be the expected value, according to the reward R, of the human following policy π. Let V∗(R) be the expected value of R if the human follows the optimal R-maximising policy.

Then the reward for the agent for not doing the surgery (¬S) is:

V(RSTA,πSTA. (1)

On the other hand, the reward for doing the surgery is (recall how RS/M splits):

argmaxRMAXϵV∗(RMAX)+(1−ϵ)V(RSTA,πMAX). (2)

In the previous post, I focused on the V∗(RMAX) term, wondering if there could be rewards sufficiently high to overcome the ϵ probability.

But there's another way (2) could be higher than (1). What if the term V(RSTA,πMAX) was is quite high? Indeed, if πSTA is not perfectly rational, setting RMAX=RSTA will set (2) to be

...
XOR Blackmail & Causality
by abramdemski — 2017-11-15T04:24:18.960Z

[Cross-posted from IAFF.]

I edited my

previous post

to note that I’m now much less optimistic about the direction I was going in. This post is to further elaborate the issue and my current position.

Counterfactual reasoning is something we don’t understand very well, and which has so many free parameters that it seems to explain just about any solution to a decision problem which one might want to get based on intuition. So, it would be nice to eliminate it from our ontology – to reduce the cases in which it truly captures something important to machinery which we understand, and write off the other cases as “counterfactual-of-the-gaps” in need of some other solution than counterfactuals.

My approach to this involved showing that, in many cases, EDT learns to act like CDT because its knowledge of its own typical behavior screens off the action from the correlations which are generally thought to make EDT cooperate in one-shot prisoner’s dilemma with similar agents, one-box in Newcomb’s problem, and so on. This is essentially a version of the tickle defense. I also pointed out that the same kind of self-knowledge constraint is needed to deal with some counterexamples to CDT; so, CDT can’t be justified as a way of dealing with cases of failure of self-knowledge in general. Instead, CDT seems to improve the situation in some cases of self-knowledge failure, while EDT does better in other such cases.

This suggests a view in which the self-knowledge constraint is a rationality constraint, so the tickle defense is thought of as being true for rational agents, and CDT=EDT under these conditions of rationality. I suggested that problems for which this was not true had to somehow violate the ability of the agent to perform experiments in the world; IE, the decision problem would have to be set up in such a way as to prevent the agent from decorrelating its actions from things in the environment which are not causally downstream of its actions. This seems in some sense unfair, as the environment is preventing the agent from correctly learning the causal relationships through experimentation. I called this condition the

law of logical causality

when it first occurred to me, and

mixed-strategy implementability

in the setup where I proved conditions for CDT=EDT.

In XOR Blackmail with a perfect predictor, however, mixed-strategy implementability is violated in a way which does...
Mixed-Strategy Ratifiability Implies CDT=EDT
by abramdemski — 2017-11-15T04:22:11.000Z

I provide conditions under which CDT=EDT in Bayes-net causal models.

[Epistemic status: Thanks to a discussion with Benja, I'm much less optimistic about the general thrust of this. In particular, the framework here isn't expressive enough to include XOR Blackmail. While I knew the framework would rule out some cases of interest, I was expecting to be happy to bite the bullet and say that we can't make a good story about where causality comes from in cases where EDT agents can't learn it by experimenting (IE, epsilon-exploration). But, on the contrary, XOR blackmail seems like a case where there is an intuitive notion of causality which might come from somewhere other than wishful thinking, and which differs significantly from the notion of causality which EDT agents can learn in frameworks similar to the one put forward in this post. I still have no idea where that notion of causality comes from, or whether to trust it, but XOR blackmail does seem to be a counterexample-in-spirit to the thrust of this post. More details here.]

(Cross-posted to lesserwrong.)

Previously, I discussed conditions under which LICDT=LIEDT. That case was fairly difficult to analyse, although it looks fairly difficult to get LICDT and LIEDT do differ. It's much easier to analyze the case of CDT and EDT ignoring logical uncertainty.

As I argued in that post, it seems to me that a lot of informal reasoning about the differences between CDT and EDT doesn't actually give the same problem representation to both decision theories. One can easily imagine handing a causal model to CDT and a joint probability distribution to EDT, without checking that the probability distribution could possibly be consistent with the causal model. Representing problems in Bayes nets seems like a good choice for comparing the behavior of CDT and EDT. CDT takes the network to encode causal information, while EDT ignores that and just uses the probability distribution encoded by the network.

It's easy to see that CDT=EDT if all the causal parents of an agent's decision are observed. CDT makes decisions by first cutting the links to parents, and then conditioning on alternative actions. EDT conditions on the alternatives without cutting links. So, EDT differs from CDT insofar as actions provide evidence about causal parents. If all parents are known, then it's not possible for CDT and EDT to differ.

So, any argumen

...
Towards an Axiological Approach to AI Alignment
by Gordon Seidoh Worley — 2017-11-15T02:07:47.607Z
Link: https://mapandterritory.org/towards-an-axiological-approach-to-ai-alignment-4993d044d1b8

NB: Originally posted on Map and Territory on Medium, so some of the internal series links go there.

AI alignment research currently operates primarily within the framework of decision theory and looks for ways to align or constrain utility functions such that they avoid “bad” outcomes and favor “good” ones. I think this is a reasonable approach because, as I will explain, it is a special case of the general problem of axiological alignment, and we should be able to solve the special case of aligning rational agents if we hope to solve axiological alignment in general. That said, for computational complexity reasons having a solution to aligning rational agents may not be sufficient to solve the AI alignment problem, so here I will lay out my thinking on how we might approach AI alignment more generally as a question of axiological alignment.

First, credit where credit is due. Much of this line of thinking was inspired first by Paul Christiano’s writing and later by this pseudonymous attempt to “solve” AI alignment. This led me to ask “how might we align humans?” and then “how might we align feedback processes in general?”. I don’t have solid answers for either of these questions — although I think we have some good ideas of research directions — but asking these questions has encouraged me to think about and develop a broader foundation for alignment problems that is also philosophically acceptable to me. This is a first attempt to explain those ideas.

Philosophical Acceptability

When I say I want a “philosophically acceptable” foundation for alignment problems, what I mean is I want to be able to approach alignment starting from a complete phenomenological reduction. Such a desire may not be necessary to make progress in AI alignment, but I would probably not be considering AI alignment if it were not of incidental interest to addressing the deeper issues it rests upon, so it seems reasonable for me to explore AI alignment in this way insofar as I also consider existential risk from AI important. Other people likely have other motivations and so they may not find the full depth of my approach necessary or worth pursuing. Caveat lector.

The phenomenological reduction is a method of deconstructing ontology to get as near phen...