trentmkelly/LessWrong-43k huggingface dataset:
Conditioning on Conditionals
by Scott Garrabrant — 2017-08-17T01:15:08.000Z

(From conversations with Sam, Abram, Tsvi, Marcello, and Ashwin Sah) A basic EDT agent starts with a prior, updates on a bunch of observations, and then has an choice between various actions. It conditions on each possible action it could take, and takes the action for which this conditional leads the the highest expected utility. An updateless (but non-policy selection) EDT agent has a problem here. It wants to not update on the observations, but it wants to condition on the fact that its takes a specific action given its observations. It is not obvious what this conditional should look like. In this post, I agrue for a particular way to interpret this conditioning on this conditional (of taking a specific action given a specific observation).

Formally, we have a set of observations O, a set of actions A, and a set of possible utilities U. The agent has a prior distribution P over O×A×U. For simplicity, we assume that U takes one of two possible values, 0 and 1. Our original EDT agent upon seeing O=o considers takes action
argmaxa P(U=1|O=o∧A=a).

Similarly, we would like our updateless EDT upon seeing O=o to take the action

argmaxa P(U=1|(A=a|O=o)),

but it is not clear what the "conditional distribution" Q(X)=P(X|(A=a|O=o)) should look like. We are not conditioning on a standard event. We will define Q as having a minimal KL divergence with P subject to the constraint that Q(A=a|O=o)=1. However there is still more than one way to define this. First, because KL divergence is not symmetric, and second, for on direction, the KL divergence will be infinite, so we need to take a limit, and it matters how we take this limit.

Q minimizing DKL(Q||P) subject to Q(A=a|O=o)=1.

Limit as x goes to 1 of Q minimizing DKL(P||Q) subject to Q(A=a|O=o)=x.

Limit as x goes to 0 of Q minimizing DKL(P||Q) subject to Q(O=o∧A≠a)=x.

Note that all of the above methods are natural generalizations of something that does the normal bayesian update when conditioning on an actual event.

Here, recall that DKL(P||Q)=∑iP(i)logP(i)Q(i).

As a simple example, assume that A and O are independently uniform between two options. If we condition on A=a1|O=o1, it seems most natural to preserve the fact that the probability that o1 is 1/2. Our choice of action should not change the probability of o1. However both definitions 1 and 3 give Q(o_1)=1/3. We are in effect ruling out the world in which

...
Density Zero Exploration
by AlexMennen — 2017-08-17T00:43:51.000Z

The idea here is due to Scott Garrabrant. All I did was write it.

Let's say a logical induction-based agent is making an infinite sequence
of decisions, and is using ε-exploration
on each decision. There are two desirable criteria, which are somewhat
in conflict:

First, we want there to be enough exporation that traders attempting
to bet that good strategies would have bad outcomes (and thus prevent
the good strategies from being tried, so that the bet never gets settled)
will lose arbitrarily large amounts of money if they try doing that
every time. This requires that in total, there is an infinite amount
of exploration. For example, if the agent 2−n-explores on step
n, then it is possible for a sufficiently wealthy malicious trader
to bet against a good strategy by enough that the agent will avoid
it every time, without the trader losing all its money, because the
actions it is discouraging only are taken anyway finitely many times.
But if the agent ε-explores on step n for some fixed
ε>0, then this is not possible, because each action is
taken infinitely many times no matter what any of the traders do,
so no trader can consistently make some good action appear bad without
losing all its money.

Second, we want there to be sufficiently little exploration that the
agent does not sacrifice a nontrivial amount of value to it. If actions
only have short-term effects, then it is enough for the probability
of exploration to approach 0 as n→∞, in order
for the agent to behave optimally in the limit (if actions can have
lasting consequences, then this is not enough; for instance, if there
is an action that destroys all value forever if it is ever taken,
then that action needs to never be taken; this directly conflicts
with the first criterion). For example, if the agent ε-explores
on step n for some fixed ε>0, then it never gets any
closer to acting optimally, but if it 2−n-explores on step n,
then its probability of acting optimally will approach 1 as n→∞.

Fortunately, there are sequences that converge to 0 but whose sum
diverges, like 1n, so it is possible to satisfy both of
these criteria. However, if there are important differences among
what the agent should do for different steps, then this might not
be enough. For example, if the agent 1n-explores on step
n, and it is particularly important what action the agent takes
on steps that are powers of 2, the

...
Logical Induction with incomputable sequences
by AlexMennen — 2017-08-17T00:39:01.000Z

In the definition of a logical inductor, the deductive process is
required to be computable. This, of course, does not allow the logical
inductor to use randomness, or predict uncomputable sequences. The
way traders were defined in the logical induction paper, this was
necessary, because the traders were not given access to the output
of the deductive process.

To fix this, a trading strategy for day n should be redefined as
a function that takes in the output of the deductive process on day
n−1 as its input, and outputs what the logical induction paper
defines as a trading strategy for day n; that is, an affine combination
of the form c+ξ1ϕ1+...+ξkϕk, where ϕ1,...,ϕk
are sentences, ξ1,...,ξk are expressible features of
rank ≤n, and c=−∑iξiϕ∗ni. A trader is
a function which takes in n and outputs a trading strategy for
day n. By Currying, a trader can be seen as a function that takes
in a number n and a list of sentences given by the deductive process,
and outputs an expressible feature combination as above. We can say
that a trader is efficiently computable if this function is computable
in time polynomial in n plus the total length of the sentences
output by the deductive process. The definition of exploitation would
be modified in the natural way, and there is also a natural way to
modify the logical induction algorithm, which will satisfy the logical
induction criterion.

As an example, suppose a logical inductor is given access to a sensor
that regularly produces bits based on what it observes in the environment.
We can represent the data from the sensor with an additional unary
predicate S that we add to the language, such that S(n)
is true iff the nth bit provided by the sensor is a 1 (this
assumes that we're working in a theory that can interpret arithmetic,
so that ``n'' can be expressed in the language). The deductive
process should output S(n) or ¬S(n)
on day n (and also can output consequences that it can deduce from
the values of the bits it has seen so far). Or, if the logical inductor
gets access to more empirical information or random bits as time goes
on, there could be an increasing function f such that the deductive
process outputs the truth values of S(f(n)),...,S(f(n+1)−1)
on day n. Note that in this situation, the deductive process is
computable as a function of the bitstream given by the sensor, so
the traders may as well take in as input

...
Stable Pointers to Value: An Agent Embedded in Its Own Utility Function
by abramdemski — 2017-08-17T00:22:02.000Z

(This post is largely a write-up of a conversation with Scott Garrabrant.)

Stable Pointers to Value

How do we build stable pointers to values?

As a first example, consider the wireheading problem for AIXI-like agents in the case of a fixed utility function which we know how to estimate from sense data. As discussed in Daniel Dewey's Learning What to Value and other places, if you try to implement this by putting the utility calculation in a box which rewards an AIXI-like RL agent, the agent can eventually learn to modify or remove the box, and happily does so if it can get more reward by doing so. This is because the RL agent predicts, and attempts to maximize, reward received. If it understands that it can modify the reward-giving box to get more reward, it will.

We can fix this problem by integrating the same reward box with the agent in a better way. Rather than having the RL agent learn what the output of the box will be and plan to maximize the output of the box, we use the box directly to evaluate possible futures, and have the agent plan to maximize that evaluation. Now, if the agent considers modifying the box, it evaluates that future with the current box. The box as currently configured sees no advantage to such tampering. This is called an observation-utility maximizer (to contrast it with reinforcement learning). Daniel Dewey goes on to show that we can incorporate uncertainty about the utility function into observation-utility maximizers, recovering the kind of "learning what is being rewarded" that RL agents were supposed to provide, but without the perverse incentive to try and make the utility turn out to be something easy to maximize.

This feels much like a use/mention distinction. The RL agent is maximizing "the function in the utility module", whereas the observation-utility agent (OU agent) is maximizing the function in the utility module.

The Easy vs Hard Problem

I'll call the problem which OU agents solve the easy problem of wireheading. There's also the hard problem of wireheading: how do you build a stable pointer to values if you can't build an observation-utility box? For example, how do you set things up so that the agent wants to satisfy a human, without incentivising the AI to manipulate the human to be easy to satisfy, or creating other problems in the attempt to avoid this? Daniel Dewey's approach of incorporating uncertainty abou

...
Tabooing Science + an xkcd comic about the eclipse - "Honestly, it's not that scientific."
by Voltairina — 2017-08-16T15:19:31.545Z

It occurred to me when I was reading XKCD a moment ago that given that there exists a strain of suspicion of anything 'science' among a certain crowd in this country (fundamentalists, creationists, etc), and a kind of mystique among another crowd (of the "It was in a study so it must be true" variety) that it might be helpful, given that by doing science people are more or less systematizing thinking critically and checking things to be as certain as they can about an idea, to kind of pay attention to and possibly 'play taboo' to an extent when that something-is-a-special-kind-of-a-thing-because-it-is-a-science-thing attitude comes up.

A good example being the xkcd comic I got it from:
Take my Job
by Jacob Falkovich — 2017-08-16T15:00:44.000Z

In September I am going to leave the company where I worked for last four years. I am going to get married, go on a honeymoon, and then leave my wife for two months to travel extensively in Colombia and Peru. If anyone from that part of the world reads this, holla at me!

I want to tell you about the job I’m leaving, why you should think about applying for it, and what it has taught me in the last four years about company culture, diversity, and the makings of a good workplace.

I will not mention the company by name. If you’re serious about applying you can easily figure it out, and if not, you shouldn’t care. Trust me, you haven’t heard of us. All the opinions expressed here are mine alone and not the company’s, which is the entire point of this post.

Our company does a few things, but mostly it sells regulatory reporting software to financial institutions.

Bank and financial institutions have to file a shit ton of regulatory reports. The crisis of ’08 brought some new regulations in and Trump may take a few regulations away, but there is always a shit ton of them, and more are added each year. Filing these reports requires a bank to round up a lot of data from its business units, clean the data and standardize it, run the clean numbers through a bunch of calculations, and report the results in the specific format that each of the 18 separate financial regulatory entities demands.

This work is boring and annoying, and bankers don’t want to do it. Sometimes banks have their own people do it using Excel, but then the numbers often come out wrong and the regulators get angry. Sometimes they pay professional consultants to do it for them, but those consultants charge a lot of money and still mostly use Excel. Then those banks buy our company’s software, which automates the whole thing and produces the report at the press of a button.

Automating regulatory reporting is a weird and small niche, but it’s quite lucrative. Our company has far and away the best offerings in this niche, along with a 25-year reputation of never having the numbers come out wrong. Since I joined, our New York office grew from 50 people to 150 and our revenues grew from decent to very very decent.

Does this sound like something you had wanted to do since you were a kid? I have a friend who had wanted to be a fashion designer since she was a ...
We need to think more about Terminal Values
by  — 2017-08-16T07:50:08.709Z

I just sent an email to Eliezer but it is also applicable to everyone on this website. Here it is:

Hi Eliezer,

I'm contacting you directly because I think I have an idea that could get us closer to obtaining our terminal value(s), and you are in a position of influence but also seem like someone I could possibly influence if my idea is correct. If my idea is wrong at least I'd get a lot out of hearing why.

First. I think your definition of Terminal Value is flawed. People don't ultimately want survival or health or friendship. I know that because if I asked you why you want survival, the answer would ultimately be because circuitry and chemistry make us. So it'd appear that our terminal values are actionable motivations from circuitry and biology. If we stopped our line of questioning there we'd think our terminal value was that, in which case the optimal path of action would be to find a way to hack our circuitry and biology to give us positive signals. But I don't think we should stop there.

If you model humans as optimizers of this terminal value then you'd say to people "you should try to optimize your terminal value" but it's not really a should because that's already what people do to various degrees of effectiveness. Terminal values aren't "should"s, they're "shall"s. The real question here is what do you mean when you say "you."

Let me know if any part of this seems logically unsound but I'm now going to make another argument that I'm still having a hard time processing:

"You" is poorly defined. Evidence:

- That tree over there is just as much a part of "you" as your hand is. The only difference is your conscious mind is better at finely controlling your hand than at controlling the tree, but same as how you can cause your hand to move you can also cause the tree to move by pushing it.

- The "you" some years from now is made of entirely different atoms yet you don't behave as if "you" will be dead when that happens--you still think of that as you. That means "you" is more than your physical body.

- If "you" is more than your physical body then, the first thing I was saying about chemical and electric signals being our terminal value don't make sense.

New model:

- We are a collective system of consciousnesses that is in some sense conscious. Humans are like neurons in a brain. We communicate like neurons. We communicate in v

...
Social Insight: When a Lie Is Not a Lie; When a Truth Is Not a Truth - Pt. 2
by Bound_up — 2017-08-15T00:53:19.954Z

'"If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold."' People misinterpret this as indicating that I take a policy stance in favor of regulation. It indicates no such thing. It is meant purely as guess about empirical consequences." - EY (http://lesswrong.com/lw/h2/blue_or_green_on_regulation/)

Try this a few times, and you'll stop thinking you can make "guess[es] about empirical consequences" (or say anything about empirical consequences (or say anything about empirical anything)) and have people hear anything except you showing off your policy stances. Showing off whom you associate with and what virtues you possess.

Once your eyes open to how hard it is to convince people that your sentences about how markets function are meant to describe how markets function, you give up and stop trying.

Well, if you have the time to convince people you're actually trying to say something about how the world works and not just proudly waving a verbal banner in favor of the home team, and you have the ability to make interesting a subject so much less accessible and exciting than politics (we've all seen it, haven't we, how little they care once they realize that we really are trying to describe market functions?), and the time to actually do it properly, all without alienating important people in the process...

Then, yeah, maybe. And those sets of circumstances absolutely happen and I'm glad that we do teach each other things.

But, I really, really understand why most politicians can't do anything remotely like this, and thus, say the words "If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold" only if they want people to hear "I am taking a policy stance in favor of regulation."

If you do want people to hear that, then this is a very effective way of communicating that. If you know that saying this will lead people to holding that belief about your stance, then saying it is honest, even if you don't believe that markets work that way. You're not saying/they're not hearing anything about markets, so none of your beliefs about markets can be misrepresented by saying these words. You believe something, you want to honestly communicate that belief, so you use symbols. We think of words as our symbo...
Open thread, August 14 - August 20, 2017
by Thomas — 2017-08-14T07:48:13.195Z

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top level comments on this article" and "Make this post available under..." before submitting
What Is Rationalist Berkley’s Community Culture?
by Zvi — 2017-08-12T15:21:29.000Z

Rationalist Status: Wonky (if you are not interested in the rationalist community or mission, this one is not for you)

Epistemic Status: Continuing the discussion

Response to (read this first): The Craft Is Not The Community (Otium)

Sarah’s post, The Craft Is Not The Community, was explicitly intended to start a discussion. Her central thesis is that the community has proven itself bad at doing outward-facing projects, so members should do their community-focused projects together, and form a community that can fulfill its members emotional needs, but do their outward-facing and world-saving projects in the outside world and play by the outside world’s rules.

I want to focus elsewhere, at least for this post, and accuse Sarah of burying the lead. She does not actually bury it – it’s right up at the top of the post – but she views it as background rather than the thing to be focused on.

We need to focus on it.

Her leading paragraph is super important, and I strongly, violently agree with it:

“Company culture” is not, as I’ve learned, a list of slogans on a poster. Culture consists of the empirical patterns of what’s rewarded and punished within the company. Do people win promotions and praise by hitting sales targets? By coming up with ideas? By playing nice? These patterns reveal what the company actually values.

And, so, with community cultures.

Those within a culture quickly learn what that culture truly values, and adopt their behaviors to that culture. Culture is super important. If we are building a physical community, and it seems that we are, we should think carefully about what culture we want that community to have, and work to ensure that we get what we want. Keeping what makes us unique requires constant vigilance in the best of times. We are building the community in 2017, and doing so in Berkeley, California. The pressures are and will be unending. Constant vigilance doesn’t begin to cover it.

Paul Graham once wrote a (recommended) piece called Cities and Ambition. Cities, he observes, have a culture. The great ones push you to work harder, achieve great things. They have a message about what you should be and what is good in life. The city constantly whispers that message in your ear.

He claims that New York, where I live, has the message that you should be richer. Fair enough. He says Boston wants you to be smar...
Choices Are Really Bad
by Zvi — 2017-08-12T14:19:04.000Z

Previously: Choices are Bad

Last time, I gave two (related) reasons Choices are Bad: Choices Reduce Perceived Value and Choices Force You Into Choosing Mode. Despite not looking like much, these reasons are Serious Business. They can render us unable to enjoy, think or relax even when it might seem that everything is awesome. That’s what that song is about, really: how great things are, on every axis, when we let ourselves go with the flow and not get distracted.

The rabbit hole goes deeper. It gets much worse.

Choices Cost Willpower and Create Decision Fatigue

Willpower is a controversial topic, and laying out even my simple model would be beyond scope. What matters here is that in the short term, exercising willpower is a cost (whether or not such use has long term benefits). Most of the time, you would prefer to spend less willpower to get the same result.

Often making the otherwise right choice will require the spending of willpower. When you consciously think about eating another cookie, or checking your email, not doing so costs a non-zero amount of willpower. Making that choice over and over could end up not only being distracting via being in choosing mode, but also end up sapping your willpower.

In addition to willpower, Decision Fatigue is a thing as well. Making choices (decisions) is draining something else that is related to but not identical to willpower. At some point, decision fatigue makes further choices carry an increasing cost. Some people have this problem more often than others, but there is always the risk that this will happen to you.

There’s even a version of ‘I’ve made good choices all day, it’s all right to indulge now’ that can happen, which is related but different.

You Might Choose Wrong

Simple but important.

If you give someone a choice, there is always some chance they will choose wrong. Remember the third principle of economics: People are stupid. They also are often ignorant, or distracted, or don’t care, or lots of other things. Mistakes are made. No matter how obvious the choice, people will sometimes choose… poorly.

Thus, giving me a choice between a Snickers bar that I value at $1 and M&Ms I value at 60 cents doesn’t only force me into choosing mode, lower my perceived value and help create decision fatigue (and potentially cost willpower). It also risks that if I’m no...