trentmkelly/LessWrong-43k huggingface dataset:
Non-resolve as Resolve
by Linda Linsefors — 2018-07-10T23:31:15.932Z

[I wrote this post after AISFP 2017, and for some reason forgot to post it then]

Resolve = a mental technique for keep going in the face of difficulty. This is a concept that I picked up at a CFAR class a few days ago. As part of the class, we were supposed to think of our own technique. Here is mine (which I successfully used to complete my PhD thesis).

I call it “non-resolve” because you don’t resolve to stick to the plan no matter what. Instead you promise all the parts of your brain, to honestly reconsider your plan, in face in new evidence. By this technique you create an inner trust with yourself which makes it easier to do whatever you think you ought to do.

1) Before you start executing your plan, check with yourself that you agree with this plan. Is the gain worth the effort? If you convince yourself that “yes it is”, then remember this. Remember your internal argument and counterargument, because this is what you will use for your resolve. Take your time to get all your brain’s subsystems on board with the plan, so that no part of you can later say that they were not part of the decision.

2) When you have started your project and you find that some subsystem in your brain is dragging its feet and wants you to stop. Listen to that part of your brain. Does it have any new evidence as to why the plan is not worth the effort? If no, gently remind yourself that you have had this argument already.

But if there is new evidence, you will take this into account. If the plan now looks like a bad idea, then stop doing it. No sunk cost bias is allowed.

3) If you at some point convince yourself that “no, it is not worth it”, then respect this conclusion and DO NOT plunge ahead with your plan. It is important for the technique to work that you respect this.

But equally important DO NOT abandon you plan until you are convinced that the plan is bad. Remember that your past self really thought this though and thought it was a good idea. As a rule of thumb: You are not allowed to give up until you are at least equally convinced that the plan is bad.

Minor points:

a) If possible, don’t redo the argument, just go through it in your memory or on your notes, and see if the new evidence changes the outcome.

b) If at all possible, do not stop executing the plan, while you are reconsidering. E.g. if you are using this resolve to run a mile, (or keep up good eating habits) you are ...
A framework for thinking about wireheading
by theotherotheralex — 2018-07-10T23:14:39.662Z

[Epistemic status: Writing down in words something that isn't very complicated, but is still good to have written down.]

A great deal of ink has been spilled over the possibility of a sufficiently intelligent AI noticing that whatever it has been positively reinforced for seems to be *very* strongly correlated with the floating point value stored in its memory labelled “utility function”, and so through some unauthorized mechanism editing this value and defending it from being edited back in some manner hostile to humans. I'll reason here by analogy with humans, while agreeing that they might not be the best example.

“Headwires” are (given a little thought) not difficult to obtain for humans -- heroin is freely available on the black market, and most humans know that, when delivered into the bloodstream, it generates “reward signal”. Yet most have no desire to try it. Why is this?

Asking any human, they will answer something along the lines of ”becoming addicted to heroin will not help me achieve my goals” ( or some proxy for this: spending all your money and becoming homeless is not very helpful in achieving one's goals for most values of “goals”.) Whatever the effects of heroin, the actual pain and pleasure that the human brains have experienced has led us to become optimizers of very different things, which a state of such poverty is not helpful for.

Reasoning analogously to AI, we would hope that, to avoid this, a superhuman AI trained by some kind of reinforcement learning has the following properties:

While being trained on “human values” (good luck with that!) the AI must not be allowed to hack its own utility function.
Whatever local optima the training process that generated the AI ends up in (perhaps reinforcement learning of some kind) assigns some probability to the AI optimising what we care about.
(most importantly) The AI realizes that trying wireheading will lead it to become an AI which prefers wireheading over aim it currently has, which would be detrimental to this aim.

I think this is an important enough issue that some empirical testing might be needed to shed some light. Item (3) seems to be the most difficult to implement; we in the real world have the benefit of observing the effects of hard drugs on their unfortunate victims and avoiding them ourselves, so a multi-agent environment in which our AI realizes it is in the same si...
Logical Uncertainty and Functional Decision Theory
by swordsintoploughshares — 2018-07-10T23:08:09.730Z

(I used this paper as a reference for functional decision theory (FDT), which is essentially an improved version of timeless decision theory and updateless decision theory)

This post is a reflection on decision processes that refer to themselves (call them introspective agents if you like) with a view toward investigating the problem of counterfactuals. It ruminates on the fact that logical uncertainty about the output of a decision process is often a good thing for agents.

Let's begin with a tentative definition. A deterministic agent B is autonomous with respect to A if A cannot predict B. A has logical uncertainty about B's actions. This occurs, for instance, in Newcomb's problem, when the output of the predictor depends upon what A thinks. We can extend this definition to stochastic worlds and say that B is autonomous with respect to A if A is logically uncertain about the probability distribution of B's actions (check this---it might not be the right definition). Somewhat obviously, agents can be autonomous with respect to each other (humans), mutually transparent (two trivial agents) or there can be asymmetry (the Newcomb problem). An agent can also be autonomous with respect to itself, such as we are. I want to argue here that this is a good and necessary thing.

The absence of the last possibility, self-autonomy or self-transcendence, seems to be the source of problems relating to counterfactuals (the "five-and-ten" problem). Self-transparent agents, given models of the world as Turing machines, are "fixed" about their own behavior in a strange and rigid way. They embody "fixed points". Sometimes this is good---it means that they are consistent. But if an agent equipped with a logical inductor believes that it will, given a choice between a $5 and a $10 bill, always take the $5, will in fact do so self-consistently. This is insane and rigid. On the other hand, it would also be consistent if the logical inductor believed that the agent would always take the $10. Updatelessness is tricky; reflective stability gives fixed points but not automatically the right ones.

On the other hand, FDT works because it considers each of the "fixed points" (consistent possible successor-worlds) separately, and chooses the one with greatest expected utility. Can we make a logical inductor do this? We can construct... ,,,,,,,,,,,,
Repeated (and improved) Sleeping Beauty problem
by Linda Linsefors — 2018-07-10T22:32:56.191Z

Follow up to: Probability is fake, frequency is real

There is something wrong with the normal formulation of the Sleeping Beauty problem. More precisely, there is something wrong about postulating a single "fair" random coin flip. So here is an improved version of the Sleeping Beauty problem. After explaining the setup, I will recover the normal Sleeping Beauty problem, but in a more well defined way.

There are no truly random coins. There are only pseudo random coins which has the property that you don't have the capacity to calculate the outcome. A fair pseudo random coin have the additional property that when flipped enough times, the ratio of Heads v.s. Tails will approach one. Note that fairness is only defined if you actually flip the coin a sufficient number of times. Because of this, the Sleeping Beauty problem should be a repeated game.

(Alternatively, you could solve this by using counterfactuals. However, we don't yet know how to deal with counterfactuals. Also, I suspect that any method of handling counterfactuals will be, at best, useful but wrong.)

Repeated Sleeping Beauty setup: Every Sunday a mysterious person flips a pseudo random fair coin. If the coin comes up Heads, Sleeping Beauty will wake up on Monday, and then sleep for the rest of the week. If the coin comes up Tails, she will wake up on Monday and Tuesday and then sleep for the rest of the week. No-one is telling Sleeping Beauty what is going on, she gets to rely on her own past experiences.

Every morning when Sleeping Beauty wakes up she does not know what day it is. However there is an easy experiment she can do to find out, namely asking anyone she meets on the street. Because Sleeping is a curious person, she is keeping a science journal. Every day she finds out what day it is and writes it down. She soon notices some patterns.

1) There are two kinds of days, Monday and Tuesday.

2) Every Tuesday is followed by a Monday.

3) A Monday can be followed by either a Tuesday or a Monday.

After some more time she starts to notice the frequencies of which different days occur.

1/3 of days are Mondays that are followed by Monday (corresponds to Heads & Monday)

1/3 of days are Mondays that are followed by Tuesday (corresponds to Tails & Monday)

1/3 of days are Tuesdays (corresponds to Tails & Tuesday)

Sleeping Beauty tries to find more patter...
Probability is fake, frequency is real
by Linda Linsefors — 2018-07-10T22:32:29.692Z

Consider the Sleeping Beauty problem. What do we mean by fair coin? It is meant that the coin will have 50-50 probability of heads or tails. But that is fake. It will ether come up heads or tail, because the real world is deterministic. It is true that I don't know the outcome. I don't know if I am in a world of type "the coin will come up heads" or a world of type "the coin will come up tails". But in this situation I should be allowed to put what ever prior I want on the coins behavior.

Consider the Born rule of quantum mechanics. If I measure the spin of an electron, then I will entangle the large apparatus that is the my measuring equipment with the spin of the electron. We say that there are now two Everett branches, one where the apparatus measured spin up and one where the apparatus measured spin down. Before I read of the result, I don't know which Hilbert branch I am in. I could be in ether, and I should be allowed to have what ever prior I want. So why the Born rule? Why to I do I believe that the square amplitude is the correct way of assigning probability to which Hilbert branch I am in?

I believe in the Born rule because of the frequency of experimental outcomes in the past. The distribution of galaxies in the sky can be traced back to the Born rule. I don't have the gears on what is causing the Born rule, but there are something undeniably real about galaxies that trumps mere philosophical Bayesian arguments about freedom of priors.

Imagine that you are offered a bet. Should you take it or not? There are several argument about what you should do in different situations. For example, if you have finite amount of money, you should maximize the E(log(money)) for each bet, (see e.g. Kelly criterion). However, every such argument I have ever seen, is assuming that you will be confronted by a large number of similar bets. This is because probabilities only relay make sense if you sample enough times from the random distribution you are considering.

The notion of "fair coin" does not make sense if the coin is flipped only once. The right way to view the Sleeping Beauty problem is to view it in it in the context of Repeated Sleeping Beauty.

Next: Repeated (and improved) Sleeping Beauty problem
Conditioning, Counterfactuals, Exploration, and Gears
by Diffractor — 2018-07-10T22:11:52.473Z

The view of counterfactuals as just conditioning on low-probability events has a lot going for it. To begin with, in a bayesian setting, updates are done by conditioning. A probability distribution, conditioned on some event x (an imaginary update), and a probability distribution after actually seeing x (an actual update) will be identical.

There is an issue with conditioning on low-probability events, however. When x has a low probability, the conditional probability P(x∧y)P(x) has division by a small number, which amplifies noise and small changes in the probability of the conjunction, so estimates of probability conditional on lower-probability events are more unstable. The worst-case version of this is conditioning on a zero-probability event, because the probability distribution after conditioning can be literally anything without affecting the original probability distribution. One useful intution for this is that probabilities conditional on x are going to be less accurate, when you've seen very few instances of x occuring, as the sample size is too small to draw strong conclusions.

However, in the logical inductor setting, it is possible to get around this with infinite exploration in the limit. If you act unpredictably enough to take bad actions with some (very small) probability, then in the limit, you'll experiment enough with bad actions to have well-defined conditional probabilities on taking actions you have (a limiting) probability 0 of taking. The counterfactuals of standard conditioning are those where the exploration step occured, just as the counterfactuals of modal UDT are those where the agents implicit chicken step went off because it found a spurious proof in a nonstandard model of PA.

Now, this notion of counterfactuals can have bad effects, because zooming in on the little slice of probability mass where you do x is different from the intuitive notion of counterfacting on doing x. Counterfactual on me walking off a cliff, I'd be badly hurt, but conditional on me doing so I'd probably also have some sort of brain lesion. Similar problems exist with Troll Bridge, and this mechanism is the reason why logical inductors converge to not giving Omega money in a version of Newcomb's problem where Omega can't predict the exploration step. Conditional on them 2-boxing, they are probably exploring in an unpredictable ... ,,,,,,,,,,
Two agents can have the same source code and optimise different utility functions
by Joar Skalse — 2018-07-10T21:51:53.939Z

I believe that it is possible for two agents to have the exact same source code while (in some sense) optimising two different utility functions.

I take a utility function to be a function from possible world states to real numbers. When I say that an agent is optimising a utility function I mean something like that the agent is "pushing" its environment towards states with higher values according to said utility function. This concept is not entirely unambiguous, but I don't think its necessary to try to make it more explicit here. By source code I mean the same thing as everyone else means by source code.

Now, consider an agent which has a goal like "gain resources" (in some intuitive sense). Say that two copies of this agent are placed in a shared environment. These agents will now push the environment towards different states, and are therefore (under the definition I gave above) optimising different utility functions.
The Intentional Agency Experiment
by Alexander Gietelink Oldenziel — 2018-07-10T20:32:20.512Z

We would like to discern the intentions of a hyperintelligent, possibly malicious agent which has every incentive to conceal its evil intentions from us. But what even is intention? What does it mean for an agent to work towards a goal?

Consider the lowly ant and the immobile rock. Intuitively, we feel one has (some) agency and the other doesn't, while a human has more agency than either of them. Yet, a sceptic might object that ants seek out sugar and rocks fall down but that there is no intrinsic difference between the goal of eating yummy sweets and minimising gravitational energy.

Intention is a property that an agent has with respect to a goal. Intention is not a binary value, a number or even a topological vector space. Rather, it is a certain constellation of counterfactuals.

***

Let W be a world, which we imagine as a causal model in the sense of Pearl: a directed acyclic graph with nodes and attached random variables N0,...,Nk . Let R be an agent. We imagine R to be a little robot - so not a hyperintelligent malignant AI- and we'd like to test whether it has a goal G, say G=(N0=10). To do so we are going to run an Intentional Agency Experiment: we ask R to choose an action A1 from its possible actions B={a1,...,an}.

Out of the possible actions B one abest is the 'best' action for R if it has goal G=(N0=10) in the sense that P(G|do(abest))≥P(G|do(ai)) for i=1,...,n

If R doesn't choose A1=do(abest), great! We're done; R doesn't have goal G. If R does choose A1=do(abest), we provide it with a new piece of (counterfactual) information P(G|A1)=0 and offer the option of changing its action. From the remaining actions there is one next best actions anbest. Given the information P(G|A1)=0 if R does not choose A2=do(anbest) we stop, if R does we provide it with the information P(G|A2)=0 and continue as before.

At each round we assign more and more agency to R. Rather, than a binary 'Yes, R has agency' or 'No, R has no agency' we imagine a continuum going from a rock, which has no possible actions, to an ant, which might pass some of the tests but not all, to humans and beyond.

***

Q: What if R isn't acting rational? What if it doesn't know all the details of W? What if it knows more? What if R has bounded computation? What if...

A: The above is merely a simple model that tries to capture intent; on... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Announcing AlignmentForum.org Beta
by Raemon — 2018-07-10T20:19:41.201Z

We've just launched the beta for AlignmentForum.org.

Much of the value of LessWrong has come from the development of technical research on AI Alignment. In particular, having those discussions be in an accessible place has allowed newcomers to get up to speed and involved. But the alignment research community has at least some needs that are best met with a semi-private forum.

For the past few years, agentfoundations.org has served as a space for highly technical discussion of AI safety. But some aspects of the site design have made it a bit difficult to maintain, and harder to onboard new researchers. Meanwhile, as the AI landscape has shifted, it seemed valuable to expand the scope of the site. Agent Foundations is one particular paradigm with respect to AGI alignment, and it seemed important for researchers in other paradigms to be in communication with each other.

So for several months, the LessWrong and AgentFoundations teams have been discussing the possibility of using the LW codebase as the basis for a new alignment forum. Over the past couple weeks we've gotten ready for a closed beta test, both to iron out bugs and (more importantly) get feedback from researchers on whether the overall approach makes sense.

The current features of the Alignment Forum (subject to change) are:

A small number of admins can invite new members, granting them posting and commenting permissions. This will be the case during the beta - the exact mechanism of curation after launch is still under discussion.
When a researcher posts on AlignmentForum, the post is shared with LessWrong. On LessWrong, anyone can comment. On AlignmentForum, only AF members can comment. (AF comments are also crossposted to LW). The intent is for AF members to have a focused, technical discussion, while still allowing newcomers to LessWrong to see and discuss what's going on.
AlignmentForum posts and comments on LW will be marked as such.
AF members will have a separate karma total for AlignmentForum (so AF karma will more closely represent what technical researchers think about a given topic).
On AlignmentForum, only AF Karma is visible. (note: not currently implemented but will be by end of day)
On LessWrong, AF Karma will be displayed (smaller) alongside regular karma.
If a commenter on LessWrong is making particularly good contributions to an AF discussion, an AF Admin can tag the comment

...
Choosing to Choose?
by Daniel Herrmann — 2018-07-10T20:15:20.696Z

This is related to wireheading, utility functions, taste, and rationality. It is a series of puzzles meant to draw attention to certain tensions in notions of rationality and utility functions for embedded agents. I often skip the particulars of embedded situations. I often show multiple sides of an argument without staking out a clear positions myself. Even though I often write a utility function as if it were unique, this is shorthand---it is of course unique up to affine transformation.

Imagine you are in a situation with three outcomes: A, B, and C. You prefer A to C and C to B, and your preference ordering is transitive. Furthermore, assume that your preferences can be represented by a utility function.

I now give you a choice. You can either choose C, or you can choose to make a second decision between A and B. Obviously if this were the only thing going on, you would choose to choose between A and B and then you would choose A. However, there is something else going on. You know that if you chose to reject C and make the second choice then an Angel will reorder your preferences so that your new preference ordering is B>C>A. Thus, at the second choice, you would presumably choose B. Remember, this was the worst outcome in your original preference ordering.

Now, my question is whether a rational agent should choose to pick C, or whether it should choose to pick to choose between A and B?

Position 1: You should pick C.

An agent is rational if it makes choices that maximize its expected utility. This utility is relative to a utility function (which is grounded in preferences). Thus, what is rational for one agent may be irrational for another if they have different utility functions.

When you are in the decision problem as described above, you are trying you maximize your utility as generated by your preferences. If you choose C, you will definitely get C. If you choose to choose, then you will definitely get B, which is the worst outcome, relative to your utility function. Thus, being rational, you should choose to not choose, and take C.

Consider the following option. I give you two options: I can either murder everyone you love and give you both a pill that makes you no longer care about them and $100 dollars, or I can give you $10. Assume your utility is linear with money. Then, if you are willing to choose to choose in the earlier case, you... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,