trentmkelly/LessWrong-43k huggingface dataset:
Automated Nomic Game 2
by jefftk — 2019-02-05T22:11:13.914Z

I've been playing a game of Nomic, where rules are interpreted by a script we edit instead of by the players. This morning Chelsea won our second game of Nomic with a timing attack:

Our game was set to use random numbers to figure out if someone had won. Effectively, each point you earn should give you an additional 1/100000th chance of winning. The source of randomness was python's random.random(), which was fine.

I had written a PR (#98 to move us to using something derived from the hash of the most recent merge commit on master. I was intending for this to be a timing attack: the merge commit should be predictable entirely from its contents and the current time.

The idea is, I can test for each upcoming timestamp whether there's hash that would end on me winning. A very hacky way to get git to think it's in the future is just to set your local clock fast:

CURTIME=$(date +%s) ;
sudo date -s '@1549031338' &&
TZ=America/New_York
git merge tmp-branch
-m "$MESSAGE" &&
sudo date -s '@'$CURTIME

Unfortunately, when I tested trying to exploit this I found that GitHub signs the merge commits it makes. For example:

$ git cat-file commit 182b57a9b53fc31febd9166fc03f3d91e368b64e
tree 04d5ff9335c297c203f4ee4cb5a14006bfa6abb9
parent 0c08b506c96d42d4ad8fab5c2c1701d557aa3a97
parent 6fc74cf39d7d79e8500ef4c4162decd64b82e8be
author Todd Nelling 1549324096 -0500
committer GitHub 1549324096 -0500
gpgsig -----BEGIN PGP SIGNATURE-----

wsBcBAABCAAQBQJcWM9ACRB...yWFgRvF4Ms==2G6+
-----END PGP SIGNATURE-----

Merge pull request #99 from pavellishin/more_verbose_rule_names

Be more vocal about which rule is being run

This meant there was some information in the hash that I couldn't control, and this wasn't going to work.

At this point the smart thing to do would have been to delete the PR: if I know it's dodgy but I can't exploit it then I shouldn't let anyone else take that opportunity! But instead I figured "I guess it's safe, then -- nice to have reproducible random numbers" and went to bed.

The next morning I saw that Chelsea had won, and was surprised by the winning commit:

$ git cat-file commit 0003490f8179527499e1b0739fec6d1ac22662f3
tree cc61bc798249f7dbf4d12d139f9c9e3134efa87a
parent 182b57a9b53fc31febd9166fc03f3d91e368b64e
parent 1d6c32eea2556acd67c0b4e193bb4e2639bb487b
author Chelsea Voss 1549380160 -0800
committer Chelsea Voss 1549380

...
Philosophy as low-energy approximation
by Charlie Steiner — 2019-02-05T19:34:18.617Z

In 2015, Scott Alexander wrote a post originally titled High Energy Ethics. The idea is that when one uses an extreme thought experiment in ethics (people dying, incest, the extinction of humanity, etc.), this is like smashing protons together at the speed of light at the LHC - an unusual practice, but one designed to teach us something interesting and fundamental.

I'm inclined to think that not only is that a slight mischaracterization of what's going on, but that all philosophical theories that make strong claims about the "high energy" regime are doubtful. But first, physics:

<physics>

Particle physics is about things that are very energetic - if we converted the energy per particle into a temperature, we could say the LHC produces conditions in excess of a trillion (1,000,000,000) degrees. But there is also a very broad class of physics topics that only seem to show up when it's very cold - the superconducting magnets inside said LHC, only a few meters away from the trillion-degree quarks, need to be cooled to basically absolute zero before they superconduct.

The physics of superconductors is similarly a little backwards of particle physics. Particle physicists try to understand normal, everyday behavior in terms of weird building blocks. Superconductor physicists try to understand weird behavior in terms of normal building blocks.

The common pattern here is idea that the small building blocks (in both fields) get "hidden" at lower energies. We say that the high-energy motions of the system get "frozen out." When a soup of fundamental particles gets cold enough, talking about atoms becomes a good low-energy approximation. And when atoms get cold enough, we invent new low-energy approximations like "the superconducting order parameter" as yet more convenient descriptions of their behavior.

</physics>

Some philosophers think that they're like particle physicists, elucidating the weird and ontologically basic stuff inside the everyday human. The better philosophers, though, are like superconductor physicists, trying to understand the unusual (in a cosmic sense) state of humanity in terms of mundane building blocks.

My favorite example of a "low-energy approximation" in philosophy, and the one that prompted this post, is Dennett's intentional stance.
The intentional stance advertises itself as a useful approximation. It's a way of thinking about certai

...
When to use quantilization
by RyanCarey — 2019-02-05T17:17:12.169Z

In 2015, Jessica introduced quantilization as a countermeasure for Goodhart's Law and specification-gaming. Since these are such central problems in AI safety, I consider quantilization to be one of the best innovations in AI safety so far, but it has received little attention from the AI safety field. I think one reason for this is that researchers aren't quite clear what problem, formally, quantilization solves, that other algorithms don't. So in this piece, I define a robust reward problem, and then discuss when quantilization solves this problem well, and when it doesn't.

Definition 1 (Robust reward problem)

We can define a robust reward problem as a tuple: ⟨A,U,I,D,k⟩.

A is the action space

U:A→R is the explicit reward

I⊆{A→R} is the space of implicit rewards, a family of functions I:A→R

D is a distribution over actions

k∈R+ is the maximum implicit loss in the demonstrations

The goal of the agent is to maximize V:=U+I for any I∈I. If that was the end of the definition, the task would be too difficult, because an adversarial reward could thwart any strategy. So we need to assume that I is pretty well-behaved in the region D.

Ea∼D[I(a)]≥−k(1)

Formally, the goal is to select a strategy S that maximizes the following:

minI∈I:Ea∼D[I(a)]≥−kEa∼S[V(a)]

The intuition behind (1) is that a teacher may forget to include some possible failures in their reward function, but they ought not to leave out the same mistakes that they themselves frequently make. And at any rate, without (1), assuring good performance is impossible.

When quantilization works

You can skip this section if you're familiar with Jessica's work

If we set I={A→(−∞,0]}, we recover the original setting in which quantilizers were described. Then (as V Kosoy has argued) the ordinary quantilizer theorems mean that we get the best lower bound for V.

We will need to reuse the definitions, so to briefly recap:

Definition 2 (Quantilizer). A q-quantilizer is an agent that, when faced with a decision problem, returns a random action in the top q proportion of the base distribution D, sorted by the explicit expected utility U achieved if that action is executed.

Jessica proves three things. First, that a quantilizer does not have much worse implicit loss than the base distribution D:

Ea∼Qq[I(a)]≥Ea∼D[I(a)]q

Where Qq is the distribution over actions selected by the quantilizer.

Second, she p...
(notes on) Policy Desiderata for Superintelligent AI: A Vector Field Approach
by Ben Pace — 2019-02-04T22:08:34.337Z

Meta: I thought I'd spend a little time reading the policy papers that Nick Bostrom has written. I made notes as I went along, so I spent a little while cleaning them up into a summary post. These are my notes on Bostrom, Dafoe and Flynn's 2016 policy desiderata paper, which received significant edits in 2018. I spent 6-8 hours on this post, not a great deal of time, so I've not been maximally careful.

Context and Goals

Overall, this is not a policy proposal. Nor does it commit strongly to a particular moral or political worldview. The goal of this paper is to merely observe which policy challenges are especially important or different in the case of superintelligent AI, that most moral and political worldviews will need to deal with. The paper also makes no positive argument for the importance or likelihood or timeline of superintelligent AI - it instead assumes that this shall occur in the present century, and then explores the policy challenges that would follow.

The Vector Field Approach

Botrom, Dafoe and Flynn spend a fair amount of time explaining that they’re not going to be engaging in what (I think) Robin Hanson would call standard value talk. They’re not going to endorse a particular moral or political theory, nor are they going to adopt various moral or political theories and show how they propose different policies. They’re going to look at the details of this particular policy landscape and try to talk about the regularities that will need to be addressed by most standard moral and political frameworks, and in what direction these regularities suggest changing policy.

They call this the ‘vector field’ approach. If you don't feel like you fully grok the concept, here's the quote where they lay out the formalism (with light editing for readability).

The vector field approach might then attempt to derive directional policy change conclusions of a form that we might schematically represent as follows:

“However much emphasis X you think that states ought, under present circumstances, to give to the objective of economic equality, there are certain special circumstances Y, which can be expected to hold in the radical AI context we described above, that should make you think that in those circumstances states should instead give emphasis fY(X) to the objective of economic equality."

The idea is that f here is some relatively simp

...
(Why) Does the Basilisk Argument fail?
by Lookingforyourlogic — 2019-02-03T23:50:16.932Z

As far as I can tell, the standard rebuttal to Roko's idea is based on the strategy of simply ignoring acausal blackmail. But what if you consider the hypothetical situation that there is no blackmail and acausal deals involved, and the Basilisk scenario is how things just are? Simply put, if we accept the Computational Theory of Mind, we need to accept that our experiences could in principle be a simulation by another actor. Then, if you ever find yourself entertaining the question:

Q1: Should I do everything I can to ensure the creation/simulation of an observer that is asking itself the same question as I am now, and will suffer a horrible fate if it answers no?

Would the answer that maximizes your utility not be yes? The idea being that answering no opens up the possibility that you are an actor created by someone who answered yes, and thus, might end up suffering whatever horrible fate he might have implemented. This seems to be the argument that Roko put forth in his original post, and that I have not seen soundly refuted.

Of course, one could argue that there is no evidence for being in such a simulation instead of the real world, and thus that the commitment and ethical dilemma of answering yes is not worth it. If we assign equal probability to all worlds we might inhabit, this might indeed be convincing. However, I think a kind of anthropic argument, based on Bostroms Self-Sampling Assumption (SSA) could be constructed to show that we may indeed be more likely to be in such a simulation. Assume your your hypotheses to be

H0: "I am living in the real world, which is as it appears to be, i.e. the possible observers are the ≈ 7 billion humans on planet earth".

H1: "The real world is dominated by actors who answered yes to Q1, and (possibly with the help of AGI) filled the universe with simulations of observers pondering Q1".

Then, given the data

D: "I am an observer thinking about Q1"

And assuming that in the world as it appears to be in H0, only very few people, say, one in ten thousand, asked themselves this question Q1, while under H1, the universe is literally filled with observers thinking about Q1, we get

Pr[D|H0] = 0.0001

Pr[D|H1] ≈ 1

So, the Bayes factor would show very clear evidence in favor of H1. Of course, this would apply to any hypothetical model in which most observers would have exactly the same thought...
Constructing Goodhart
by johnswentworth — 2019-02-03T21:59:53.785Z

A recent question from Scott Garrabrant brought up the issue of formalizing Goodhart’s Law. The problem is to come up with some model system where optimizing for something which is almost-but-not-quite the thing you really want produces worse results than not optimizing at all. Considering how endemic Goodhart’s Law is in the real world, this is surprisingly non-trivial.

Let’s start simple: we have some true objective u(x), and we want to choose x to maximize it. Sadly, we don’t actually have any way to determine the true value u for a given value x — but we can determine u(x)+ϵ(x), where ϵ is some random function of x. People talked about this following Scott’s question, so I won’t math it out here, but the main answer is that more optimization of u+ϵ still improves u on average over a wide variety of assumptions. John Maxwell put it nicely in his answer to Scott’s question:

If your proxy consists of something you’re trying to maximize plus unrelated noise that’s roughly constant in magnitude, you’re still best off maximizing the heck out of that proxy, because the very highest value of the proxy will tend to be a point where the noise is high and the thing you’re trying to maximize is also high.

In short: absent some much more substantive assumptions, there is no Goodhart effect.

Rather than generic random functions, I suggest thinking about Goodhart on a causal DAG instead. As an example, I’ll use the old story about soviet nail factories evaluated on number of nails made, and producing huge numbers of tiny useless nails.

We really want to optimize something like the total economic value of nails produced. There’s some complicated causal network leading from the factory’s inputs to the economic value of its outputs (we’ll use a dramatically simplified network as an example).

If we pick a specific cross-section of that network, we find that economic value is mediated by number of nails, size, and strength — those variables are enough to determine the objective. All the inputs further up influence the objective by changing number, size, and/or strength of nails.

Now, we choose number of nails as a proxy for the objective. If we were just using this proxy to optimize machine count, that would be fine — machine count only influences our objective via number of nails produced, it doesn’t effect size or strength, so number of nails is a fine proxy for our true objecti...
Conclusion to the sequence on value learning
by Rohin Shah — 2019-02-03T21:05:11.631Z

This post summarizes the sequence on value learning. While it doesn’t introduce any new ideas, it does shed light on which parts I would emphasize most, and the takeaways I hope that readers get. I make several strong claims here; interpret these as my impressions, not my beliefs. I would guess many researchers disagree with the (strength of the) claims, though I do not know what their arguments would be.

Over the last three months we’ve covered a lot of ground. It’s easy to lose sight of the overall picture over such a long period of time, so let's do a brief recap.

The “obvious” approach

Here is an argument for the importance of AI safety:

Any agent that is much more intelligent than us should not be exploitable by us, since if we could find some way to exploit the agent, the agent could also find the exploit and patch it.
Anything that is not exploitable must be an expected utility maximizer; since we cannot exploit a superintelligent AI, it must look like an expected utility maximizer to us.
Due to Goodhart’s Law, even “slightly wrong” utility functions can lead to catastrophic outcomes when maximized.
Our utility function is complex and fragile, so getting the “right” utility function is difficult.

This argument implies that by the time we have a superintelligent AI system, there is only one part of that system that could still have been influenced by us: the utility function. Every other feature of the AI system is fixed by math. As a result, we must necessarily solve AI alignment by influencing the utility function.

So of course, the natural approach is to get the right utility function, or at least an adequate one, and have our AI system optimize that utility function. Besides fragility of value, which you might hope that machine learning could overcome, the big challenge is that even if you assume full access to the entire human policy, we cannot infer their values without making an assumption about how their preferences relate to their behavior. In addition, any misspecification can lead to bad inferences. And finally the entire project of having a single utility function that captures optimal behavior in all possible environments seems quite hard to do -- it seems necessary to have some sort of feedback from humans, or you end up extrapolating in some strange way that is not necessarily what we “would have” wanted.

So does this mean we’re doomed? Well,...
AI Safety Prerequisites Course: Revamp and New Lessons
by philip_b — 2019-02-03T21:04:16.213Z

Previous post: Fundamentals of Formalisation Level 7: Equivalence Relations and Orderings. First post: Fundamentals of Formalisation level 1: Basic Logic.

Nine months ago we, RAISE, have started creating a Math Prerequisites for AI Safety online course. It has mostly MIRI research related subjects: set theory, computability theory, and logic, but we want to add machine learning related subjects in the future. For 4 months we've been adding new lessons and announcing them on LessWrong. Then we stopped, looked back and decided to improve their usability. That's what we've been busy with since August.

News since the last post

Big update of 7 levels we had previously published, which you can see in the picture above. The lessons use textbooks, which you will need to follow along. Previously lessons looked like "read that section; now solve problems 1.2, 1.3, 1.4c from the textbook; now solve these additional problems we came up with". Now our lessons still say "read that section", but the problems (and their solutions, in contrast to many textbooks, which don't provide solutions) are included in lessons themselves. Additional problems are now optional, and we recommend that students skip them by default and do them only if they need more practice. New levels in Logic, Set Theory, and Computability tracks will be like that as well.

Level 1 was very long, consisted of 45 pages of reading, and could take 10 hours for someone unfamiliar with logic. We separated it into smaller parts.

Two new levels. Level 8.1: Proof by Induction. Level 8.2: Abacus Computability.

If you study using our course, please give us feedback. Leave a comment here or email us at raise@aisafety.camp, or through the contact form. Do you have an idea about what prerequisites are most important for AI Safety research? Do you know an optimal way to learn them? Tell us using the same methods or collaborate with us.

Can you check if a mathematical proof is correct? Do you know how to make proofs understandable and easy to remember? Would you like to help to create the prerequisites course? If yes, consider volunteering.
What are some of bizarre theories based on anthropic reasoning?
by Dr. Jamchie — 2019-02-03T18:48:17.237Z

Doomsday argument and simulation argument are quite bizarre for most people. But these are not the only strange theories one can come up with when employing anthropics.

Some examples:

It is likely to have an unusual high IQ.

Perhaps brain works in a way, that having high IQ correlates with something that also causes more observer moments. Hence there are more of high IQ experience in the world than that of low IQ.

Fragile universe

Total universe destroying physical catastrophes that expands in speed of light (say false vacuum collapse) could be very frequent. As much, as once every second. And it is only due to survivor bias that we think universe is stable and safe. How would we know?

Animals does not have consciousness

There are more animals than humans on Earth. Still we find ourselves as humans. Perhaps it is because we only can be humans, as only humans have consciousness.

We are stuck inside infinite loop

Lets assume simulation argument is correct. Then we probably exist inside simulation, run by some software. All software have bugs. One of the bugs software sometimes have is getting itself in an infinite loop. Biggest amount of experience computed by this software then could possibly be inside this infinite loop.
Rationality: What's the point?
by Hazard — 2019-02-03T16:34:33.457Z

This post is part of my Hazardous Guide To Rationality. I don't expect this to be new or exciting to frequent LW people, and I would super appreciate comments and feedback in light of intents for the sequence, as outlined in the above link.

A friend once articulated that he didn't like when things are taught, "Mr. Miygai style". A bunch of disconnected, unmotivated facts, exercises, and ideas are put before you, and it's only at the very end that it clicks and you see the hidden structure and purpose of everything you've learned.

Therefore, the very first post of this sequence is going to be a drive by of what I think some of the cool/useful/amazing things are that you can get out of The Way. I never would have become a close-up magician if I hadn't seen someone do incredible things that blew my mind.

Who Is This For?

As much as it pains me to say this, it might not really matter whether or not you follow The Way. It really depends on what you're trying to do. The guy who kicked of the contemporary rationality community, Eliezer Yudkowsky, notes that besides a natural draw based on personality, the biggest reason he's invested in rationality is because he really wants to make sure Friendly AI happens before Unfriendly AI, and turns out that's really hard.

[*add more*]

What's the Pot of Gold at the End of the Rainbow?

Things I claim you can get better at

Believing true things and not believing false things.
Arrive at true beliefs faster.
"Failing mysteriously" less often
Understanding how your own mind works.

Why some of the above things are awesome

If you have something to protect, (you really want to make certain things happen) better models, more true beliefs, update speed, and being confused by lies, all make you more likely to make the changes you want to see in the world.
If you get a kick out of more deeply grokking how the world around you works, a kick you will get.
A lot of interpersonal problems come from two gaps:

One between "How human minds work" and "How you think human minds work"
One between "Your beliefs, feelings, and emotions" and "Your self-model of your beliefs, feelings, and emotions"
Shorting those gap will result in less interpersonal problems.