trentmkelly/LessWrong-43k huggingface dataset:
Policy Selection Solves Most Problems
by abramdemski — 2017-12-01T00:35:47.000Z

It seems like logically updateless reasoning is what we would want in order to solve many decision-theory problems. I show that several of the problems which seem to require updateless reasoning can instead be solved by selecting a policy with a logical inductor that's run a small amount of time. The policy specifies how to make use of knowledge from a logical inductor which is run longer. This addresses the difficulties which seem to block logically updateless decision theory in a fairly direct manner. On the other hand, it doesn't seem to hold much promise for the kind of insights which we would want from a real solution.

LI Policy Selection

Rather than running a logical inductor all the way to Pn and making a decision via the expected utilities implied by that distribution, we want to first run it to Pf(n) for f(n)<<n, and use that distribution to make a decision about how to utilize Pn to determine which action to take. (For example, we could choose f(n)=log(n).)

For simplicity, I will assume logical induction over exponential-time traders rather than the usual poly-time assumption. I also assume that numbers can be represented efficiently in the language of the logical inductor, IE, the written length is logarithmic in the number.

We can select from a set Pn of policies, each of which is a program taking the market state Pn and outputting an action. With m=f(n) in the following:

bestn:=argmaxp∈PnEm(U(x)|f(x)=m∧policy(x)=p)

policyn:=if explorationn then random(Pn) else bestn

A(n):=policyn(Pn)

The utility U(n) is a function of n because it points to the universe where A(n) lives. The argmax is taken over U(x), where x is uncertain, because n may be too large for Pm to have good beliefs about; the sentences involving n can be so large that they aren't being traded on the market yet. However, since we are using a market which can't be exploited by exp-time traders, Pm must have good beliefs about sentences whose length is asymptotically a multiple of the length of m. (The uncertainty should not harm us much, since the policy can look at sentences involving the true n and act accordingly.)

The values of explorationn and randomn are made pseudorandom relative to the market state at f(n) by diagonalizing against it, like the exploration defined here. The subscript n is on Pn because we need to grow the policy set, to ensure that the op

...
A limit to punishment 
by RST — 2017-11-30T17:08:10.633Z

In my last post I made this mind experiment:

Let’s assume that Omega has an evil twin: Psi.

Psi really likes to play with human minds, and today it will force some people to do difficult choices.

Alice is a very nice person: she does volunteer, donates moneys to important causes, is vegetarian, and she is always kind to the people she interact with.

Bob on the other hand is a murderer and torturer: he has kidnapped lots of people and then tortured them for many days in horrible ways that are worse than your deepest fears.

Psi offers two option:

A) Alice will be tortured for 1 day and then killed, while Bob will be instantly killed in a painless way.

B) Bob will be tortured for 3 days and then killed, while Alice will be instantly killed in a painless way.

If you refuse to answer, everyone will be tortured for a year and then killed.

No matter what you will choose, you too will be killed and no one will ever know what has appended.

1. What would you answer?

2. What would you answer if the 3 days in option B) were substituted by 3 years?

3. And if they were substituted by 33 years?

Even if the most compassionate answer is clearly A, most people I know (me included) prefer B, at least in the first case.

This is not surprising: anger can motivate humans to contrast perceived injustices committed toward ourselves or other member of our community and so it helps to ensure the preservation of collaborative behaviors. In other words, many people usually adopt til for tat strategies.

Moreover, I think that we aren’t able to really empathize with psychopaths: even a person with a hypothetical perfect empathy, able to feel all Bob’s emotions, would ultimately condemn him, because she would have to feel his pain and fear but also his sadism and lack of empathy, which would create contrasting emotions.

On the other hand, this person would perceive other's injustices as her own and she would feel empathic anger against Bob. Adopting this kind of empathy based ethic would probably result in a sort of tit for tat strategy based on intentions and attitudes rather than past actions.

Another reason to choose B, which could work even with a Golden Rule based morality, is signaling: choosing B here, in this moment, you signal that you care about merit, or rather you signal that you have certain emotions in response of certain behavior and that these emotions will guide your actions. Whi...
Improved formalism for corruption in DIRL
by Vanessa Kosoy — 2017-11-30T16:52:42.000Z

We give a treatment of advisor corruption in DIRL, more elegant and general than our previous formalism.

The following definition replaces the original Definition 5.

Definition

Consider a meta-universe υ=(μ,r) and β:(0,∞)→(0,∞). A metapolicy α is called β-rational for υ (as opposed to before, we assume α is an I-metapolicy rather than an ¯I-metapolicy; this is purely for notational convenience and it is straightforward to generalize the definition) when there exists {Lαt:hdomμ×A→[0,∞]}t∈(0,∞) s.t.

i. For any t∈(0,∞) and h∈hdomμt, there is a∈A s.t. Lαt(ha)=0.

ii. αt(h)(a)=exp(−β(t)Lαt(ha))maxa∗∈Aαt(h)(a∗)

iii. For any π∈Π and t∈(0,∞)

limt→∞min(Ex∼μt⋈π[∞∑n=0e−n/tLαt(x:n+1/2)]−Ex∼μt⋈π[∞∑n=0e−n/t(Vυt(x:n)−Qυt(x:n+1/2))],0)=0

In condition ii, exp(−∞) is understood to mean 0. Conditions i+ii can be seen as the definition of Lαt given αt. A notable special case of condition iii is when for any x∈hdomμt

∞∑n=0e−n/tLαt(x:n+1/2)≥∞∑n=0e−n/t(Vυt(x:n)−Qυt(x:n+1/2))

As a simple example, we can have a set of corrupt states {Ct⊆(A×O)∗}t∈(0,∞) in which the behavior of the advisor becomes arbitrary, but for each h∈Ct there is g∈(A×O)∗×A s.t. g⊏h and Lαt(g)=∞ (i.e., to corrupt the advisor one has to take an action that the advisor would never take). As opposed to before, this formalism can also account for partial corruption, e.g. if for each h∉Ct and a∈A, we have Lαt(ha)≥Vυt(h)−Qυt(ha) (like in strict β-rationality) whereas for h∈Ct, we only have Lαt(ha)≥Vυt(h)−Qυt(ha)−δ for some constant δ>0, then to ensure β-rationality, it is sufficient that for each h=a0o0a1o1…∈Ct:

max{m∣h:m∉Ct}∑n=0e−n/t(Lαt(h:nan)−Vυt(h:n)−Qυt(h:nan))≥δe−(max{m∣h:m∉Ct}+1)/t1−e−1/t

Theorem

Consider H={υk}k∈N a countable family of I-meta-universes and β:(0,∞)→(0,∞) s.t. β(t)=ω(t2/3). Let {αk}k∈N be a family of I-metapolicies s.t. for every k∈N, αk is β-rational for υk. Define ¯H:={¯υk[αk]}k∈N. Then, ¯H is learnable.

Proof of Theorem

We don't spell out the proof in detail, but only the modifications with respect to the original.

As in the proof of the original theorem, we can assume without loss of generality that H is finite. Define π∗ the same way as in Lemma A, but with Lt redefined as

Lt(ha):=Ek∼ζt(h)[Lαkt(ha)]

Similarly, define π! the same way as in the proof of Lemma A, but with Lt redefined as

Lt(ha):=Ek∼ζ!kt(h)[Lαkt(ha)]

As in the proof of Lemma A, we have

1N∑k&

...
DeepMind article: AI Safety Gridworlds
by Commander Zander — 2017-11-30T16:13:42.603Z
Link: https://deepmind.com/research/publications/ai-safety-gridworlds/

DeepMind authors present a set of toy environments that highlight various AI safety desiderata. Each is a 10x10 grid in which an agent completes a task by walking around obstacles, touching switches, etc. Some of the tests have a reward function and a hidden 'better-specified' reward function, which represents the true goals of the test. The agent is incentivized based on the reward function, but the goal is to find an agent that does not overfit it in a way that clashes with the hidden function.

I think this paper is relatively accessible to readers outside the machine learning community.
Why DRL doesn't work for arbitrary environments
by Vanessa Kosoy — 2017-11-30T12:22:37.000Z

% operators that are separated from the operand by a space

% autosize deliminaters

% operators that require brackets

% operators that require parentheses

% Paper specific

Previously, I presented the theory of DRL for finite MDPs. Now, although I believe this theory can be generalized much beyond finite MDPs (at least to finite POMDPs and infinite POMDPs that satisfy some geometric and/or dynamical systems theoretic assumptions), it cannot work for arbitrary environments (without requiring the advisor to be more than "merely sane"). Constructing a counterexample is not difficult, but it seemed worthwhile to write it down.

Let A:={a,b,c} be the set of actions and O:={o−,o+} the set of observations. Define τ:(A×O)∗→N⊔{∞} s.t. for any n∈N and h∈(A×O)n

τ(h):={∞ if ∀m∈[n]:hm∉c×Omin{m∈[n]∣hm∈c×O} otherwise

That is, τ(h) is the first time action c (stands for "commit") appears in the history h. Now, consider the following environments μa, μb.

μa(o+∣h):=⎧⎪
⎪⎨⎪
⎪⎩0 if τ(h)=∞0 if τ(h)≤n21τ(h)|{m∈[τ(h)]∣hm∈a×O}| otherwise

μb(o+∣h):=⎧⎪
⎪⎨⎪
⎪⎩0 if τ(h)=∞0 if τ(h)≤n21τ(h)|{m∈[τ(h)]∣hm∈b×O}| otherwise

Also, define the reward function r by setting r(h)=1 for any history h that ends with o+ and r(h)=0 for any other history. Denote υa,b:=(μa,b,r). That is, both universes count the number of actions a and b until time τ (the first time action c is taken). At times between τ and 2τ, they produce rewards with frequency that equals the relative fraction of the corresponding action in the count. Before τ and after 2τ, they produce no rewards.

Now, we haven't defined sane policies for arbitrary environments, but the spirit of the definition is that a sane policy is unlikely to take actions with major irreversible long-term negative consequences and has to have some non-negligible probability of taking an optimal (or at least nearly optimal) action. For example, we might define it as follows

#Definition

Consider β∈(0,∞), γ,ϵ∈(0,1) and a universe υ=(μ,r). A policy σ is called (ϵ,β)-sane for (υ,γ) when for any h∈hdomμ

i. ∃a∗∈argmaxa∈AQυγ(h,a):σ(a∗∣h)≥ϵ

ii. ∀a∈A:σ(a∣h)≤exp[β(Qυγ(h,a)−Vυγ(h))]

Here, we consider the asymptotics γ→1, β→∞ but should include the scenario β=o(11−γ) so that actions with short-term negative consequences are allowed (otherwise this wou

...
The Impossibility of the Intelligence Explosion 
by DragonGod — 2017-11-30T05:47:43.879Z

I don't agree with everything here—or even the central argument—but this post was informative for me, and I think others would benefit from it.

Things I learned.

No free lunch theorem. I'm very grateful for this, and it made me start learning more about optimisation.
From the above: there is no general intelligence. I had previously believed that finding a God's algorithm for optimisation would be one of the most significant achievements of the century. I discovered that's impossible.
Exponential growth does not imply exponential progress as exponential growth may meet exponential bottlenecks. This was also something I didn't appreciate. Upgrading from a level n intelligence to a level n+1 intelligence may require more relative intelligence than upgrading from a level n-1 to a level n. Exponential bottlenecks may result in diminishing marginal growth of intelligence.

The article may have seemed of significant pedagogical value to me, because I hadn't met these ideas before. For example, I have just started reading the Yudkowsky-Hanson AI foom debate.
Logical Updatelessness as a Robust Delegation Problem
by Scott Garrabrant — 2017-11-30T04:23:24.000Z

(Cross-posted an Less Wrong)

There is a cluster of problems in understanding naturalized agency which I call Robust Delegation (EDIT: When I first wrote this, I called it "subagent alignment"). It refers to the way in which a limited agent might direct a more powerful agent to do its bidding. The limit agent starts out in control because it gets to design the more powerful agent, but it is not easy to retain that power. In most applications the limited agent is a human and the powerful agent is an AI.

Example of problems in this subfield include corrigibility, value learning, and informed oversight.

Logical Updatelessness is one of the central open problems in decision theory. An updateless agent is one that does not update on its observations, but instead chooses what action it wants itself to output upon being input those observations. It can use this to get counterfactually mugged (which is a good thing). This is basically the only way we know how to make a reflectively stable agent.

The problem is that we do not know how to translate updatelessness into a logical setting. In the toy models in which we can define updatelessness, the agent is logically omniscient and just makes empirical observations. Thus, starts out just as powerful as it will ever be. In fact, we can view it as though the only agent is the agent existing at the beginning of time. That agent chooses a policy, a function from inputs to outputs, and all the future agents can just blindly follow that policy without thinking.

In the logical setting, the future agents are smarter than the past agents. They learn more logic in addition to making empirical observations. The agent at the beginning time does not have the power to contain a policy from all the logical observations to actions. This agent does not choose a simple policy to follow in the future, but instead chooses a complicated program to run in the future.

I claim that logical updatelessness is hard while empirical updatelessness is easy because an empirically updateless agent is deferring control to a simple policy that it understands, while a logically updateless agent has to defer control to a more powerful agent that it does not understand. Thus, logical updatelessness can be viewed as a robust delegation problem.

If we think about this just in terms of reflective stability, a reflectively stable agent has to never have a conflict bet

...
Examples of Mitigating Assumption Risk
by SatvikBeri — 2017-11-30T02:09:23.852Z

One of the subskills mentioned in Eliezer's Security Mindset post is mitigating assumption risk–that is, the risk of losing utility because some of your assumptions are wrong. There are two main ways to do this:

Gain more information about whether your assumptions hold
Make the assumption irrelevant (such as the hashing passwords example)

Here are a bunch more examples:

Repeating back what someone said in your own words, to check understanding
Adding a margin of safety when estimating how much load a bridge can bear
Using Statistical models that make fewer assumptions, or have fatter tails
Exposing your work to attack in low-risk situations, such as comedians testing new material in small clubs, or Netflix's Chaos Monkey
Emphasizing fast adaption to unexpected circumstances over better forecasting
Putting spare capacity in steps in your process that aren't the bottleneck
Testing code frequently while refactoring to check functionality doesn't unintentionally change
Doing an analysis in different ways on different datasets, and only trusting them when the conclusions match
LDL 7: I wish I had a map
by magfrump — 2017-11-30T02:03:57.713Z

When I was an undergraduate considering going to graduate school in mathematics, one thing that I knew a reasonably amount about and spent a good amount of time thinking about was what the field of math looked like overall.

In very, very broad strokes I knew that geometry, algebra, and analysis were generally different things pursued by different people. I knew that these three were very mature sets of tools at least as much as active research areas, and that other areas of math which might be more naively “interesting,” such as chaos theory or number theory or topology, were generally divided up in the research community based on how amenable certain problems were to using tools from each of those three core branches and how much the practitioners like using each of those types of tools. I also knew that these tools weren’t mutually exclusive, and that a lot of the interesting progress in my chosen subfield of number theory was driven by rapid movement and conversion between these tools—viewing an equation as related to a curve geometrically, placing the geometry of that curve in an algebraic structure, and then using analysis to find some solution in this space and trace it back to a solution of the original equation, for example.

While this is a long way from being sufficient for doing mathematics, I do think this whole process and conception of what mathematics is is very important. In particular, if I had a question and I didn’t know if the answer was known or unknown, or if there were techniques that might help, I had a good sense of who to ask and what books to look in to find answers. Sort of a technical academic equivalent of Google Fu—understanding the field well enough to search through it efficiently.

I do not have this sense for deep learning research.

I know there are three fundamental problems in machine learning; supervised, unsupervised, and reinforcement learning. I know that supervised learning problems have many, many avenues of attack and that reinforcement learning has made progress in large part by trying to reduce itself (via something like deep approximate Q-learning) to a supervised learning task, and that unsupervised learning is difficult because it’s unclear how to do this reduction.

But if I have a question like, “how can I use transfer learning to apply knowledge from a large dataset to a small dataset?” I’m going to have a lot of ...
The Mad Scientist Decision Problem
by Linda Linsefors — 2017-11-29T11:41:33.640Z

Consider Alice, the mad computer scientist. Alice has just solved general artifical intelligence and the alignment problem. On her computer she has two files, each containing a seed for a superintelligent AI, one of them is aligned with human values, the other one is a paperclip maximizer. The two AIs only differ in their goals/values, the rest of the algorithms, including decision procedures, are identical.

Alice decides to flipp a coin. If the coin comes up heads, she starts the friendly AI, and if it comes up tails, she starts the paperclip maximizer.

The coin comes up heads. Alice starts the friendly AI, and everyone rejoice. Some years later the friendly AI learns about the coinflip and of the paperclip maximizer.

Should the friendly AI counterfactually cooperate with the paperclip maximizer?

What does various decision theories say in this situation?

What do you think is the correct answer?
The Right to be Wrong
by sarahconstantin — 2017-11-28T23:43:24.210Z

Epistemic Status: pretty confident

Zvi recently came out with a post “You Have the Right to Think”, in response to Robin Hanson’s “Why be Contrarian?”, itself a response to Eliezer Yudkowsky’s new book Inadequate Equilibria. All of these revolve around the question of when you should think you “know better” or can “do better” than the status quo of human knowledge or accomplishment. But I think there’s a lot of conflation of different kinds of “should” going on.

Yudkowsky’s book, and Hanson’s post, are mostly about epistemic questions — when are you likely to get the right answer by examining an issue yourself vs. trusting experts?

Inadequate Equilibria starts with the canonical example of when you can’t outperform the experts — betting on the stock market — and explains about efficient markets, and then goes on to look into what kinds of situations deviate from efficient markets such that an individual could outperform the collective intelligence of everyone who’s tried so far. For instance, you might well be able to find a DIY treatment for your health problem that works better than anything your doctor would prescribe you, in certain situations — but due to the same incentive problems that prevented medical consensus from finding that treatment, you probably wouldn’t be able to get it to become the standard of care in the mass market.

Hanson mostly agrees with Yudkowsky’s analysis, except on some points where he thinks the argument for individual judgment being reliable is weaker.

Zvi seems to be talking about a different thing altogether when he talks about the “rights” that people have.

When he says “You have the right to disagree even when others would not, given your facts and reasoning, update their beliefs in your direction” or “You have the right to believe that someone else has superior meta-rationality and all your facts and reasoning, and still disagree with them”, I assume he’s not saying that you’d be more likely to get the right answer to a question in such cases — I think that would be false. If we posit someone who knows better than me in every relevant way, I’d definitionally be more likely to get the right answer by listening to her than disagreeing with her!

So, what does it mean to have a right to disagree even when it makes you more likely to be wrong? How can you have a right to be wrong?

I can think of two simple meanings and one subtle ...
Any Good Criticism of Karl Popper's Epistemology?
by Elliot_Temple — 2017-11-28T22:31:05.362Z

I believe there are no correct, unaddressed criticisms of Karl Popper’s epistemology (Critical Rationalism – CR). If I'm mistaken, I'd like to be told. If others are mistaken, I'd like them to find out and take an interest in CR.

CR is important for the general reasons that epistemology is important (it tells you about how to think, how to learn, how to evaluate ideas and arguments, and how AGI could work). It also refutes induction, which is a popular belief here. If CR is correct, then people here have a lot to change their mind about.

I've found CR criticism falls into some broad categories, with some overlap:

The people who heard Popper is wrong secondhand but didn’t read much Popper and have no idea what CR is actually about. They often try to rely on secondary sources to tell them what CR says, but most secondary sources on CR are bad.
The pro-induction people who don’t engage with Popper’s ideas, just try to defend induction. They don’t understand Popper’s criticism of induction and focus on their own positive case for induction. They also commonly admit that some criticisms of induction are correct, but still won’t change their minds or start learning the solution to induction’s flaws (CR).
The falsificationism straw man, which misinterprets Popper as advocating a simplistic, false view. (There are some other standard myths too, e.g. that Popper was a positivist.)
Critics of The Logic of Scientific Discovery who ignore Popper’s later works and don’t engage with CR's best ideas.
Critics with points which Popper answered while he was still alive. Most criticisms of Popper are already answered in his books, and if not there then in this collection of Popper criticism and Popper’s replies. (I linked volume two which has Popper’s replies, you will want volume 1 also.)

If you believe Popper is wrong, then: Do you believe you personally understand CR? And have you looked at Popper’s books and replies to his critics to see if your point is already answered? If so, have you written down why Popper is mistaken? If not, do you believe someone else has done all this? (They understand CR, are familiar with Popper’s books including his replies to his critics, and wrote down why Popper is mistaken.)

Whether it’s by you or someone else, you can reply with a reference to where this is publicly written down in English. I will answer it (or refer you ...