trentmkelly/LessWrong-43k huggingface dataset:
Bayesian Reasoning with Unsong Theodicy means we shouldn't destroy the universe
by pku — 2018-07-22T01:25:07.867Z

(Contains major Unsong spoilers.)

A while ago I made the argument that if we assume God only allows the universe to exist if it contains net positive good, and we assume the goodness of a universe is normally distributed, most universes are probably kind meh (since they're clustered around the mean).

This relies, however, on assuming normal distribution of goodness. But the space of possible universes (or at least, possible universes at least somewhat like ours) seems like it might be more asymptotic: Civilization and technology advances rapidly and a lot of things change, until we either reach a positive singularity where everything is great forever, reach a negative singularity where everything is horrible forever, or destroy ourselves and/or the universe.

If we assume God doesn't allow net-negative universes to come into existence, the second case can't happen. The third case can happen, but only if the world was a net positive in the time interval before our destruction. In this case the assumption of a normal distribution applies, which means the universe is kinda meh. This has three implications:

First, our two possible scenarios are greatness or mehness. So we shouldn't try to destroy the physics, because a world in which we destroy physics is consistent with the meh scenario.

Second, we should take high-risk strategies when it comes to AI. A paperclip AI would leave the universe neutral and is consistent with a meh world. An AI that tortures everyone forever is consistent with the bad-world God wouldn't allow to exists. Therefore, if we take an AI strategy that causes either friendly AI or a paperclip AI, it's purely down to luck. But if our strategy causes a hell AI if it fails, then God Himself will ensure our success.

Third, we should try to cause as much suffering as possible before we achieve friendly AI. The more suffering we achieve now, the less likely it is that we're in a meh world, because a meh world with slightly-negative total utility would never exist. If we keep our net utility negative until we have a singularity, that singularity has to be a positive one (otherwise we wouldn't exist in the first place).
Stable Pointers to Value III: Recursive Quantilization
by abramdemski — 2018-07-21T08:06:32.287Z

This is a very loose idea.

In Stable Pointers to Value II, I pointed at a loose hierarchy of approaches in which you try to get rid of the wireheading problem by revising the feedback loop to remove the incentive to wirehead. Each revision seems to change the nature of the problem (perhaps to the point where we don't want to call it a wireheading problem, and instead would put it in a more general perverse-instantiation category), but not eliminate problems entirely.

Talking with Lawrence Chan today, he described a way of solving problems by "going meta" (a strategy which he was mostly suspicious of, in the conversation). His example was: you can't extract human values by specifying it as a learning problem, because of severe identifiability problems. However, it is not entirely implausible that we can "learn to learn human values": have humans label examples of other humans trying to do things, indicating what values are being expressed in the scenario.

If this goes wrong, you can try and iterate the operation again, learning to learn to learn...

This struck me as similar to the hierarchy I had constructed in my older post.

My interpretation of what Lawrence meant by "going meta" here is this: machine learning research "eats" other research fields by using automated learning to solve problems which were previously being solved by the process of science, IE, hand-crafting hypotheses and testing them. AI alignment research is full of cases where this doesn't seem like a very good approach. However, one attitude we can take to such cases is to do the operation again: propose to learn how humans would solve this sticky problem.

This is not at all like other learning to learn approaches which merely seek to speed up normal learning. The idea is that our object-level loss function is insufficient to point out the behavior we really want. We want new normative feedback to come in at the meta-level, telling us more about which ways of solving the object-level problem are desirable and which are undesirable.

The idea I'm about to describe seems like a fairly hopeless idea, but I'm interested in seeing how it would go regardless.

What is the fixed point of this particular "go meta" operation?

The intuition is this: any utility function we try to write down has perverse instantiations, so that we don&#...
Conceptual problems with utility functions, second attempt at explaining

by Dacyn — 2018-07-21T02:08:44.598Z

[Followup to: Conceptual problems with utility functions]

The previous post was me flailing my arms about wildly trying to gesture at something. After some conversations I think I have a somewhat better idea of what it is and can try to explain it somewhat more systematically. Let us see if it works.

The point that I want to make clear I agree with, is that there is a difference between "valuing fairness" (for example) because you want everyone to get something, versus "valuing fairness" for tactical reasons. For example, suppose that you are playing the Ultimatum Game where the point is to divide the profit from a joint project, and each person thinks they put in more effort than the other and so deserves a larger share. Then maybe you think that the "fair outcome" is for you to get 60%, while the other person thinks that the "fair outcome" is for you to get 40%.

I think a reasonable way to deal with this scenario is: each of you plays a mixed strategy with the end result that on average, both players will get 40% (meaning that 20% of the time the players will fail to reach agreement and nobody will get anything). The nice things about this strategy are (A) it gives both players at least some money and (B) it satisfies a "meta-fairness" criterion that agents with more biased notions of fairness aren't able to exploit the system to get more money out of it.

So now in our decisionmaking process there are two notions of "fairness": the initial notion of fairness that made us decide that the fair split is 60/40, and the "meta-fairness" or "inexploitability" that made us sacrifice some of the other person's money so that we prefer 40/40 to 40/60. Certainly these are different. But my point is that they are not as different as they appear, and I am not sure that there is a principled way of separating your decisionmaking process into two components "utility function" and "decision theory". Moreover, even if there is a clean way of making such a separation, it is not clear to me that "meta-fairness" is a much simpler concept than "object-level fairness". And it seems to me that the utility functions/game theory framework presumes not only that it is a simpler concept, but that it is somehow conceptually equivalent to the concept of "ma...
Can few-shot learning teach AI right from wrong?
by Charlie Steiner — 2018-07-20T07:45:01.827Z

0: Introduction

In a post a few months ago on pointing to environmental goals, Abram Demski reminded me of the appeal of defining good behavior by extensional examples. He uses the example of building bridges. The AI does a bunch of unsupervised learning to explore the simulation environment, so that when humans show it just a few labeled examples of good bridges, it will have pre-learned some high-level concepts that let it easily classify good bridges.

Unfortunately, this doesn't work, as Abram explains. But it seems like it should, in some sense - it seems like we have a basic approach that would work if only we understood some confusing details better. Maybe that's not so, but I think it's worth some effort.

One way of looking at this issue is that we're trying to understand concept learning - how the AI can emulate a human understanding of the world. Another way is as implementing an understanding of reference - giving the AI examples is an attempt to point at some "thing," and we want the AI to take this as a cue to find the concept being pointed at, not just look at the "finger" doing the pointing.

Over the last couple months I've been reading and thinking on and off about reference, and I've got about three posts worth of thoughts. This post will try to communicate what kind of value learning scheme I'm even talking about, point out some flaws, and provide a little background. The second post will start speculating about ways to get around some of these flaws and probably be the most applicable to AI, and the third post will be about humans and philosophy of reference.

1: Second Introduction

The goal, broadly, is to build an AI that satisfies human values. But no AI is going to know what human values are, or what it means to satisfy them, unless we can communicate those things, and it can learn them.

The impossible method to do this is to write down what it means to satisfy human values as a long list of program instructions. Most relevant to this post, it's impossible because nobody can write down human values by hand - we embody them, but we can't operationalize them any more than we can write down the frequency spectrum of the sounds we hear.

If we can't duplicate human values by hand, the only remaining option seems to be machine learning. The human has some complicated definition of ...
The Psychology Of Resolute Agents
by Chris_Leong — 2018-07-20T05:42:09.427Z

Epistemic Status: Exploratory

Consider Parfit's Hitchhiker from the perspective of a completely selfish agent:

"Suppose you're out in the desert, running out of water, and soon to die - when someone in a motor vehicle drives up next to you. Furthermore, the driver of the motor vehicle is a perfectly selfish ideal game-theoretic agent, and even further, so are you; and what's more, the driver is Paul Ekman, who's really, really good at reading facial microexpressions. The driver says, "Well, I'll convey you to town if it's in my interest to do so - so will you give me $100 from an ATM when we reach town?"

When you're in the desert, the deal is to your benefit, but once you're in town, you're incentivised to defect. So you should expect the driver to not believe you and leave you to die in the desert.

An agent using Updateless Decision Theory will avoid an untimely death, but I believe that someone should be able to survive without advanced decision theories (ie. just using Causal Decision Theory). And indeed, if you can successfully pre-commit to paying the driver the money, then you will survive, but this leads to the question of whether you can.

But first, I note that pre-commit has two meanings: in a broad sense, irrevocably deciding to follow a particular course of action; and in a narrow sense, as just described, but in a publicly verifiable manner. Since we are assuming that the driver has a high ability to guess your decision, these two definitions end up collapsing within this scenario.

Now, we can imagine a host of situation in which you could pre-commit: you could be trusted to pay if a court would fine you $150 if you didn't uphold your bargain, or if you were deontologist who thought that being moral was more important than a good outcome could be trusted or if God would let you into heaven if you are good. What about if we exclude outside rewards/punishments and we assume that you are completely self-interested and rational? Is pre-commitment possible is such circumstances?

The definition of "completely rational" is important here. If it means that they must make every decision rationally (according to CDT) and are forbidden to self-modify in a way that makes them lose this property, then it necessarily follows that they will defect. On the other hand, if it means that they alway...
Probability is Real, and Value is Complex
by abramdemski — 2018-07-20T05:24:49.996Z

(This post idea is due entirely to Scott Garrabrant, but it has been several years and he hasn't written it up.)

In 2009, Vladimir Nesov observed that probability can be mixed up with utility in different ways while still expressing the same preferences. The observation was conceptually similar to one made by Jeffrey and Bolker in the book The Logic of Decision, so I give them intellectual priority, and refer to the result as "Jeffrey-Bolker rotation".

Based on Nesov's post, Scott came up with a way to represent preferences as vector-valued measures, which makes the result geometrically clear and mathematically elegant.

Vector Valued Preferences

As usual, we think of a space of events E⊂Ω which form a sigma algebra. Each event has a probability P(E) and an expected utility V(E) associated with it. However, rather than dealing with V directly, we define Q(E)=P(E)V(E). Vladimir Nesov called Q "shouldness", but that's fairly meaningless. Since it is graphed on the y-axis, represents utility times probability, and is otherwise fairly meaningless, a good name for it is "up". Here is a graph of probability and upness for some events, represented as vectors:

(The post title is a pun on the fact that this looks like the complex plane: events are complex numbers with real component P and imaginary component Q. However, it is better to think of this as a generic 2D vector space rather than the complex plane specifically.)

If we assume E1 and E2 are mutually exclusive events (that is, E1∩E2=∅), then calculating the P and Q of their union E1∪E2 is simple. The probability of the union of two mutually exclusive events is just the sum:

P(E1∪E2)=P(E1)+P(E2)

The expected utility is the weighted sum of the component parts, normalized by the sum of the probabilities:

V(E1∪E2)=P(E1)V(E1)+P(E2)V(E2)P(E1)+P(E2)

The numerator is just the sum of the shouldnesses, and the denominator is just the probability of the union:

V(E1∪E2)=Q(E1)+Q(E2)P(E1∪E2)

But, we can multiply both sides by the denominator to get a relationship on shouldness alone:

V(E1∪E2)P(E1∪E2)=Q(E1)+Q(E2)

Q(E1∪E2)=Q(E1)+Q(E2)

Thus, we know that both coordinates of E1∪E2 are simply the sum of the component parts. This means union of disjoint events is vector addition in our vector space, as illustrated in my diagram earlier.

Linear Transformations

When we represent preferences in a ...
Solving the AI Race Finalists
by Gordon Seidoh Worley — 2018-07-19T21:04:49.003Z
Link: https://medium.com/goodai-news/solving-the-ai-race-finalists-15-000-of-prizes-5f57d1f6a45f

Good AI offered a prize for writing about AI races. The results are in and here are the winners:

Top scoring solutions ($3,000 each)

Kesavan Athimoolam, Solving the Artificial Intelligence Race: Mitigating the problems associated with the AI Race
Alexey Turchin and & David Denkenberger, Classification of Global Solutions for the AI Safety Problem
Ehrik L. Aldana, A Theory of International AI Coordination: Strategic implications of perceived benefits, harms,capacities, and distribution in AI development

Runners-up ($2,000 each)

David Klimek, Framework for managing risks related to emergence of AI/AGI
Gordon Worley, Avoiding AGI Races Through Self-Regulation
Morris Stuttard & Anastasia Slabukho, The AI Engineers’ Guild: proposal for an AI risk mitigation strategy
LW Update 2018-07-18 – AlignmentForum Bug Fixes
by Raemon — 2018-07-19T02:10:57.487Z

Git commit: 882df0fe53f3c2bb04fac76d5b631f32cab28b1b

AlignmentForum

If a non-AlignmentForum member's post gets moved to AlignmentForum, they can now comment on that post.
Fixed the bug on LessWrong where canceling a vote on a non-AF post briefly showed a flicker of AF karma.

Stickies

Previously we only had a single "sticky" option for posts, which meant it was slightly awkward when we wanted to sticky things in either the Meta forum, or AlignmentForum. Meta and AF now have their own sticky setting.

Form Group Styling

Form Groups (such as editing your user profile, or the moderation settings for high karma users) are a little easier to parse now.

There have been a few more performance improvements for long comment threads.
Generalized Kelly betting
by Linda Linsefors — 2018-07-19T01:38:21.311Z

[tl;dr: It's a mess, don't go there]

[Thanks to Diff for proof reading this post]

Kelly betting has the very nice property that if a gambler is betting according to a given world model, and the amount of money the gambler starts out with equals to the prior probability of that model, then after each round of bets, this gamblers money will equal the current posterior probability.

The problem with Kelly betting is that it relies on only being given one bet at a time, and that the previous bet will be evaluated before you are asked to bet on a new question. Compare this to the situation faced by the traders in a Logical Inductor, where there are always multiple bets every round and the traders don't know when any bet will be settled.

I have (almost) calculated the generalized Kelly criterion, in the case of two dual outcome simultaneous bets, with general market odds and general gambler beliefs. The only remaining part of this calculation is a cubic equation.

Send me an e-mail (linda.linsefors@gmail.com) if you want my notes. There is no guarantee that I will read all blog post comments.

Solving this last equation for general market odds, is in principle not very hard. You can find the general solution to cubic equations on Wikipedia. Except in this specific case the equation is:

a(β11)3+b(β11)2+cβ11+d=0

a=m1m′1(1−m1−m′1)b=m1m′1−p1m2m′1−p′1m1m′2+2p1p′1m1m′1c=p1p′1[−(m1+m′1)+p1m′2+p′1m2−p1p′1]d=(p1p′1)2

solve for β11.

The probability that I will get this right by hand is near zero. Maybe someone with Mathematica, or a similar tool could help me?

For the notation, there are two simultaneous and independent bets. Each bet has two outcomes, denoted by index x∈{1,2}.

px=probability of outcome x, according to gambler.mx=probability of outcome x, according to the market.

And the ' superscript denotes the second bet. By definition

p1+p2=1,p′1+p′2=1m1+m2=1,m′1+m′2=1

β11 is just a help variable I made up. It does not have a super clear interpretation, but just happens to be the key to calculating everything else.

As mentioned above, I don't have the final formula for the generalized Kelly criterion, in the case of two dual outcome simultaneous bets, with completely general market odds and gambler beliefs, not until someone solves the above equation. What I currently do have is the special case where the market probabilities for both statem...
A Step-by-step Guide to Finding a (Good!) Therapist
by squidious — 2018-07-18T01:50:00.581Z

The process of finding a good therapist is unlike the process of finding most professionals. If your dentist nags you to floss more every time you see them, it's annoying, but unlikely to really affect the quality of their work. In therapeutic work, however, most of the work comes from the bond between therapist and client. If you find yourself worried about your therapist judging you or don't trust them to bring up important parts of your life, you won't be able to get the most out of therapy. Of course, this can make the process of starting therapy intimidating! It is very important to know that you can always talk to your therapist about changing what they're doing or referring you to someone else -- something I've written about previously in my guide to termination. Here, I'll instead talk about the first in these two bookends on therapy: finding a therapist.

The first thing to remember in finding a therapist is that if you have a bad experience, it is more likely that the therapist is a bad fit than "therapy doesn't work for me". Finding a good therapist is more like dating than it is like finding a mechanic, because of the emotional trust and freedom from judgment that I mentioned above. This means you may have to try a few times before finding The One, and once you do find someone good, there is no guarantee that they will be affordable, available, or local. Part of the reason I am writing this guide is that I am hoping the advice below will reduce the number of tries it takes to find someone who works for you.

What are your needs?

In order to find someone who works for you, you have to know what your needs are. The most obvious needs here are the practical: fees, location, timing, insurance, and accessibility. If you don't have much to spend on therapy, you may be confined to clinics and whatever your insurance covers. However, it is important to know that if you're working through insurance, they are required to find you someone who meets your practical and therapeutic needs, or else they must cover someone who is otherwise out of network. So if you're looking for an OCD specialist and nobody on your insurance has availabilities, they are still required to help you out. Working with insurance can be frustrating, for both therapist and client. Fees are often so high because a therapist working ...
Simple Metaphor About Compressed Sensing
by ryan_b — 2018-07-17T15:47:17.909Z

Summary: Compressed Sensing is a subject I encountered in undergrad and thought was cool, but was far beyond my math ability. I return to it occasionally, and this is a metaphor I have developed to test my understanding.

Suppose you walked home from work every evening and passed a windowed office building without a sign or even a door. You change your route every day so as to view all sides of the building, and notice only that a different number of office windows are lit on each side. Eventually you determine to solve the mystery, and decide to peek into the windows directly. Which side of the building should you choose to look into?

We are accustomed to the idea that more information is better, so a natural answer would be "the side with the most windows lit." In the first window you see someone doing payroll - but all office buildings have people doing payroll. In the second window you see someone cleaning the floor - but all office buildings get cleaned. In the third window you see someone doing HR work - but all office buildings have people doing HR. These three windows effectively tell us nothing.

Suppose instead we chose the side of the building with the fewest windows lit. Then you might go up to the lone lit window and see schematics and models in the office. Aha! They make widgets.

And now suppose you wanted to tell all this to a friend. You could recount the whole story: walking around each side of the building; checking each window and what you found there. Or, you could just say in that building, they make widgets. This is much shorter, and more importantly has all the same information. This is because your friend already knows that all office buildings have people who do payroll, and cleaning, and HR in them; he can reconstruct those parts of the story himself.

In this metaphor what they do in the office is the signal of interest, the sides of the building are the domains of the signal, and the number of lit windows are the sparsity of the signal. I don't really understand the incoherence condition, so I'm not sure this story even gestures at it. Telling your friend is the application bit. For informed reading, see Terry Tao's blog.