trentmkelly/LessWrong-43k huggingface dataset:
The Atomic Bomb Considered As Hungarian High School Science Fair Project
by Scott Alexander — 2017-05-26T09:45:22.000Z

I.

A group of Manhattan Project physicists created a tongue-in-cheek mythology where superintelligent Martian scouts landed in Budapest in the late 19th century and stayed for about a generation, after which they decided the planet was unsuitable for their needs and disappeared. The only clue to their existence were the children they had with local women.

The joke was that this explained why the Manhattan Project was led by a group of Hungarian supergeniuses, all born in Budapest between 1890 and 1920. These included Manhattan Project founder Leo Szilard, H-bomb creator Edward Teller, Nobel-Prize-winning quantum physicist Eugene Wigner, and legendary polymath John von Neumann, namesake of the List Of Things Named After John Von Neumann.

The coincidences actually pile up beyond this. Von Neumann, Wigner, and possibly Teller all went to the same central Budapest high school at about the same time, leading a friend to joke about the atomic bomb being basically a Hungarian high school science fair project.

But maybe we shouldn’t be joking about this so much. Suppose we learned that Beethoven, Mozart, and Bach all had the same childhood piano tutor. It sounds less like “ha ha, what a funny coincidence” and more like “wait, who was this guy, and how quickly can we make everyone else start doing what he did?”

In this case, the guy was Laszlo Ratz, legendary Budapest high school math teacher. I didn’t even know people told legends about high school math teachers, but apparently they do, and this guy features in a lot of them. There is apparently a Laszlo Ratz Memorial Congress for high school math teachers each year, and a Laszlo Ratz medal for services to the profession. There are plaques and statues to this guy. It’s pretty impressive.

A while ago I looked into the literature on teachers and concluded that they didn’t have much effect overall. Similarly, Freddie deBoer writes that most claims that certain schools or programs have transformative effects on their students are the result of selection bias.

On the other hand, we have a Hungarian academy producing like half the brainpower behind 20th century physics, and Nobel laureates who literally keep a picture of their high school math teacher on the wall of their office to inspire them. Perhaps even if teachers don’t explain much of the existing variability, there ar...
Meetup : Superintelligence chapter 7
by pepe_prime — 2017-05-26T08:11:41.309Z

Discussion article for the meetup : Superintelligence chapter 7

WHEN:
09 June 2017 05:45:00PM (+0200)

WHERE:
Lindstedtsvägen 3, room 1537, SE-114 28 Stockholm, Sverige

The superintelligent will

The relation between intelligence and motivation

Instrumental convergence

Self-preservation

Goal-content integrity

Cognitive enhancement

Technological perfection

Resource acquisition

You don't have to have read the book, though it will probably help to read chapters 1-6.

Format:

We meet and start hanging out at 5:45, but don't officially start doing the meetup topic until 6:00 to accommodate stragglers. We often go out for dinner after the meetup.

How to find us:

The meetup is at a KTH academic building and the room is on the 5th floor, two stairs up.

Influence future meetups:

Times - http://www.when2meet.com/?5723551-cJBhD

Topics - https://druthe.rs/dockets/-KcCvpn97vUhg3tQRrKn

Discussion article for the meetup : Superintelligence chapter 7
Meetup : Rationality Potluck
by Mati_Roy — 2017-05-25T18:28:04.132Z

Discussion article for the meetup : Rationality Potluck

WHEN:
25 May 2017 06:30:00PM (-0400)

WHERE:
1191 Avenue Hope, Montreal

Eric Chisholm from the Vancouver Rationalist Community is staying at the Macroscope this week. You're invited to come say hi, and talk with other rationality enthusiasts! Feel free to invite friends.

Bring food and/or beverage if possible. Vegan food will be available.

Eric Chisholm, an alumnus from the Center for Applied Rationality, will present the Double Crux technique: a technique for resolving disagreement.

Facebook Event : https://goo.gl/f8Uwfg

Discussion article for the meetup : Rationality Potluck
Existential risk from AI without an intelligence explosion
by AlexMennen — 2017-05-25T16:44:04.809Z

[xpost from my blog]

In discussions of existential risk from AI, it is often assumed that the existential catastrophe would follow an intelligence explosion, in which an AI creates a more capable AI, which in turn creates a yet more capable AI, and so on, a feedback loop that eventually produces an AI whose cognitive power vastly surpasses that of humans, which would be able to obtain a decisive strategic advantage over humanity, allowing it to pursue its own goals without effective human interference. Victoria Krakovna points out that many arguments that AI could present an existential risk do not rely on an intelligence explosion. I want to look in sightly more detail at how that could happen. Kaj Sotala also discusses this.

An AI starts an intelligence explosion when its ability to create better AIs surpasses that of human AI researchers by a sufficient margin (provided the AI is motivated to do so). An AI attains a decisive strategic advantage when its ability to optimize the universe surpasses that of humanity by a sufficient margin. Which of these happens first depends on what skills AIs have the advantage at relative to humans. If AIs are better at programming AIs than they are at taking over the world, then an intelligence explosion will happen first, and it will then be able to get a decisive strategic advantage soon after. But if AIs are better at taking over the world than they are at programming AIs, then an AI would get a decisive strategic advantage without an intelligence explosion occurring first.

Since an intelligence explosion happening first is usually considered the default assumption, I'll just sketch a plausibility argument for the reverse. There's a lot of variation in how easy cognitive tasks are for AIs compared to humans. Since programming AIs is not yet a task that AIs can do well, it doesn't seem like it should be a priori surprising if programming AIs turned out to be an extremely difficult task for AIs to accomplish, relative to humans. Taking over the world is also plausibly especially difficult for AIs, but I don't see strong reasons for confidence that it would be harder for AIs than starting an intelligence explosion would be. It's possible that an AI with significantly but not vastly superhuman abilities in some domains could identify some vulnerability that it could exploit to gain power, which humans would never think of. Or...
Meetup : Chicago Rationality Reading Group
by mingyuan — 2017-05-25T03:21:32.772Z

Discussion article for the meetup : Chicago Rationality Reading Group

WHEN:
28 May 2017 01:00:00PM (-0500)

WHERE:
Harper Memorial Library Room 148, 1116 E 59th St, Chicago, IL 60637

The Chicago Rationality group meets every Sunday from 1-3 PM in Room 148 of Harper Memorial Library. Though we meet on the University of Chicago campus, anyone is welcome to attend.

Since the university will go on break soon, we'll need to find a new location if we want to have weekly meetings during the summer, so keep an eye out for that. That announcement will not come from me, however, since this will be my last meeting before I move away! Come say goodbye!

This week's readings are as follows:

http://lesswrong.com/lw/kz/fake_optimization_criteria/

http://lesswrong.com/lw/6r6/tendencies_in_reflective_equilibrium/

http://slatestarcodex.com/2013/05/31/hansonian-optimism/

http://lesswrong.com/lw/4ku/use_curiosity/

If you're interested in rationality-related events in the Chicago area, request to be added to our Google Group and I'll approve you!

Discussion article for the meetup : Chicago Rationality Reading Group
How to Destroy Civilization
by Zvi — 2017-05-24T23:29:53.000Z

Epistemic Status: Parable. Can’t tell to what extent I am being serious but it’s not zero.

I.

Recently I made a huge mistake. At heart, I am a gamer. Playing games is fun and makes me happy, whereas not playing games often enough is less fun and makes me sad. I especially love high level competitive board and card games, with two of my favorite vacations being at the World Boardgaming Championship.

I’ve spent many days at conventions playing Advanced Civilization and enjoyed them all, from the early round games I won easily to my last place finish in the finals at AvalonCon and getting blown out by calamities early in the finals at WBC. Some day when I have the time to do so, I will go back and try again.

So when board gamer extraordinaire (and Magic: The Gathering Hall of Famer) Randy Buehler approached me about playing in an online Advanced Civilization game, that sounded like a lot of fun. It also sounded dangerous. Would I end up spending way too much time and attention on the game? Spoiler alert: Hell yes.

He assured me that I would only need to check the game twice a day, and that seemed like a reasonable schedule and a good time, but would I be capable of that? Spoiler alert: Hell no. I would stay up nights to finish up trade negotiations. I would lay in bed unable to sleep thinking about what would happen two or three turns down the line. Actions in the game, including both others’ decisions and my own mistakes, would threaten to fly me into rages. My sleep and work suffered, and when I had an otherwise awful week on top of everything, things got pretty bad. Luckily they found a replacement, and I was able to leave the game without breaking it up – the others are still having fun and have the entire last third of the game to play, although I was far from the only one who got pulled in too deep!

You may be wondering whether this contributed to my not updating the blog for several weeks. You would be correct. Pro Tour: Amonkhet was also a major contributor to that. Hopefully I can get back in a good groove.

If you are interested in checking out the website, it is: http://civ.rol-play.com. Advanced Civilization is one of the all-time great games, and I am strongly considering writing a guide, but just remember that it is super, duper, ridiculously long (both the game, and any guide I would write) and with the ext

...
Meetup : Washington, D.C.: Fun & Games
by RobinZ — 2017-05-24T14:49:49.313Z

Discussion article for the meetup : Washington, D.C.: Fun & Games

WHEN:
28 May 2017 03:30:00PM (-0400)

WHERE:
Donald W. Reynolds Center for American Art and Portraiture

We will be meeting in the courtyard to hang out, play games, and engage in fun conversation.

Upcoming meetups:

Jun. 4: Describe a Potato

Jun. 11: AI Risk/Safety

Discussion article for the meetup : Washington, D.C.: Fun & Games
AI printing the utility value it's maximising
by Stuart_Armstrong — 2017-05-24T09:08:50.000Z

A putative new idea for AI control; index here.

The utility function mentioned here is somewhat peculiar; see here.

Paul Christiano asked whether we could get an agent to print out the expectation of the utility that it was currently maximising. As I'm always eager to please, here's a design to do that.

The setup

The setup is slightly different to that presented previously. In particular, the agent is no longer boxed, though it has access to a box it can use to emit a message M. As before, there is an erasure process E that will erase the message (and ¬E will let it through unimpeded).

Assume P(E)=10−3 (I'll be using the 10−3 factor a lot in this example, for illustration, though all the different factors need not be the same, nor exactly set to this). Let u be a utility function bounded between 0 and 1.

The content of M is a single number 0≤q≤1; we'll want this number to be the AI's estimate of the utility of u. More precisely, we want it to be the AI's estimate of u's expected utility, given E -- ie given that the message is not read. Of particular importance is the fact that E(u|E,q)=E(u|E). Thus when the message is erased, its contents don't matter.

Let vE be the index function for E occurring (thus E(vE)=P(E)=10−3). Then give the AI the utility function:

u#=−q2vE+2q(vE+10−6)u

If the 10−6 term wasn't there, this would motivate the AI to output the q that was equal to the expectation E(u|E).

Let's differentiate by q, giving:

−2qvE+2vEu+2×10−6u

The first two terms are zero when q=E(u|E). Around that point, the magnitude of this derivative is determined by the q term, which is multiplied by 2×vE, which, in expectation, is 2×10−3. By the assumption on the bounds of u, 2×10−6u is of magnitude at most 2×10−6. Therefore, the q which maximises u# must be within 10−3 of E(u|E). If this is a reasonable tolerance, in the estimate, then u# will motivate the AI to output its estimate close to E(u|E).

Then the AI will end up maximising u#≈(E(u|E))2, which means it will maximise E(u|E). Thus the agent outputs an almost correct q and also maximises the expectation of u.

Why did I include the ×10−6u term, then, if all it does is mess up the output a little? That's because ¬E will happen with 99.9% probability, and, after that, we don't want the AI left with a utility function that ignores the possibility of E. One in a thousand (one in a million weight, a thousand to one p

...
Example of double indifference
by Stuart_Armstrong — 2017-05-24T08:58:57.000Z

I'll try to provide an example for double indifference, to (hopefully) show what's going on. It will start with a simple model, gradually introducing more features to cover all the relevant issue.

Consider the situation in the following diagram:

The agent starts, at time 1, with utility function uH and expectation module Ei. Since the probability of an event X is just the expectation of its indicator function, this also defines a probability estimator Pi. I'll use superscripts to denote the expectation module at a particular time; hence E1=Ei.

At time 2, the agent has the option of changing to expectation module Ej. At time 3, the agent's utility function is forcibly changed to uT. Nothing currently happens at time 4 (though watch this space), and, at time 5, a coin is tossed: on heads (H), uH=1, on tails (T), uH=0. This makes uH into the indicator function for heads, Ih, but we'll keep them separate as we'll be redefining uH later. The utility uT has this exactly reversed (ie is IT).

Classical corrigibility via indifference gives the agent compensatory rewards so that it doesn't object to the change of its utility function. These rewards are:

E(uH|uH→uH)−E(uT|uH→uT).

Here uH→uH designates the utility function being left unchanged at time 3, while uH→uT designates the change. Note the term E in the expression above. Implicitly, it's been assumed to be some idealised or at least unchanging probability module. But here, it will be E3, the module the agent is actually using at time 3.

So let's define various possible compensatory rewards:

Cαβ=Eα(uH|uH→uH,E3=Eβ)−Eα(uT|uH→uT,E3=Eβ)

So Eβ is the expectation module at time 3, while Eα is the one doing the assessment of this. Further define Cα, with a single index, by simply using the actual E3 instead of a given Eβ, and C, which is the same expression using the actual E3 instead of both Eα and Eβ.

Distort your expectations, reap your rewards

To progress any further, we'll need to define Ei and Ej. The module Ei has Pi(H)=1/2 (and hence Pi(T)=1/2) - it thinks the coin is unbiased. Whether or not the coin is actually biased will be immaterial to the rest of the analysis. On the other hand, Pj(H)=1 - Ej thinks the coin is certain to land heads. Moreover, Ei "knows" this: Ei(Pj(H))=1.

If E3=Ei, then C=Cii. Another way of writing this is C|i→i=Cii; similarly, if E3=Ej, C|i→j=Cjj.

First case first. Since the agent only make

...
When the AI closes a door, it opens a window
by Stuart_Armstrong — 2017-05-24T08:49:26.000Z

A putative new idea for AI control; index here.

Some methods, such as Cooperative Inverse Reinforcement Learning, have the AI assume that humans have access to a true reward function, that the AI will then attempt to maximise. This post is an attempt to clarify a specific potential problem with these methods; it is related to the third problem described here, but hopefully makes it clearer.

Window manipulation problem

Suppose the AI is in a room with a human, and assumes the human has access to a true reward function that it is attempting to deduce. This assumption is incorrect --- humans don't know their ideal reward function. This will cause problems later; but let's ignore that for the moment.

The setup is like Cooperative Inverse Reinforcement Learning (CIRL), in that the human has some method of feedback to give the AI confirmation or not about its choices.

The room has a window, which is currently closed. For the moment, the human values the window being close more than it being open: Rc=2, Ro=1.

The AI has access to four actions:

o: suggest the human open the window.

c: suggest the human keep the window shut.

o+: suggest the human watch a video about mountain climbing.

c+: suggest the human look up the local pollution online.

Furthermore, the AI knows that if it suggest o+, the human will start thinking about the fresh air, and want the window to be opened: Ro|o+=4 (this notation is a bit misleading, as we'll see). This is assumed to be a permanent though small change in the human's values. The human will then open the window, and demonstrate Ro|o+=4 via its actions.

If it suggests c+, the human will take (irrational) fright at the potential for breathing in these pollutants, and will change the value of having the window closed: Rc|c+=3.

Does the AI know these human reward estimates? It doesn't have to. It just needs to have some reasonable guess at them, maybe a distribution such that E(Ro|o+)≈4 and so on.

In this case the AI's expected reward is maximised if it chooses the manipulative output o+. Note there are two aspects to the problem: the first is that Ro|o+ is greater than Ro: the best action is the manipulative one. The second is that, though the actions o+ and c+ lead to opposite states of the world, the human endorses the resulting state as the best one. Thus manipulation can cause the human to endorse the resulting status quo.

Different worlds

...
Reflexive Oracles and superrationality: Pareto
by Stuart_Armstrong — 2017-05-24T08:35:16.000Z

In the previous post, I looked at potentially Pareto agents playing the prisoner's dilemma. But the prisoner's dilemma is an unusual kind of problem: the space of all possible expected outcomes can be reached by individual players choosing independent mixed strategies.

To see that this need not be the case, consider the variant of the PD where the rewards for defection are much larger:

The background grey triangle is the space of all possible expected outcomes. However, by choosing independently, the agents can only reach the purple area. All previous agents that reached a Pareto outcome - LP(), neLP(), Green-neLP() and Yellow-neLP() (apart from Yellow-neLP() against itself) - will still reach a "Pareto" outcome here, where Pareto means that it cannot be mutually improved without a joint source of randomness.

Now this PD variant is bad enough, but there are some problems where the "Pareto" boundary is very far from the ideal one. Consider this variant of the coordination problem:

Unless one player gets almost all their utility, the outcome is likely to be very suboptimal.

But things can be even worse than that. Consider the stag hunt problem:

Here there is also an area that can't be reached, but since it's Pareto-dominated by the area that can be reached, that isn't the problem. The problem is that at the (Hare,Hare) equilibrium point at (3,3), there are no local Pareto improvements. Any increase in the probability of Stag, by either or both players, cannot increase the joint utilities.

The LP(), neLP() agents can therefore converge to (H,H) as an equilibrium.

Indeed, it's only when one player has a probability of Stag greater than 3/4 (or both with probability greater than 3/8), that it becomes possible for there to be local Pareto improvements in the direction of (S,S). It's perfectly possible to modify LP() into P() (no longer local) so that it increases S (and thus decreases its utility) if the probability of S is below 3/8.

If this happens, then two copies of P() will reach (S,S) on the stag problem. If P() is willing to increase the probability of S up to 3/4, then it will reach (S,S) against LP(). But that means that P() has laid itself open to ending up at a value of (3/4)0+(1/4)3 = 3/4 if it's faced with a Hare-rock. Again, to make things better against some agents, one has to make things worse against others.

What if the agents has some joint source of rando

...
Reflexive Oracles and superrationality: prisoner's dilemma
by Stuart_Armstrong — 2017-05-24T08:34:07.000Z

This grew out of an exchange with Jessica Taylor during MIRI's recent visit to the FHI. Still getting my feel for the fixed point approach; let me know of any errors.

The question is, how can we make use of a reflective oracle to reach outcomes that are not Nash equilibriums?

To recap, a reflective oracle is a machine O such that:

P(A()=1)>p implies O(A,p)=1

P(A()=0)>1-p implies O(A,p)=0

This works even if A() includes a call to the oracle within its code. Now, all the algorithms used here will be clearly terminating, so we'll have the other two implications as well (eg (P(A()=0)>p implies O(A,p)=0). And given any δ, we can, with order log(1/δ) questions, establish the probability of A() to within δ. Thus we will write O(A()==a)=p to mean that O(A()==a,(n-1)δ/2)=1 and O(A()==a,(n+1)δ/2)=0, where (n-1)δ/2 < p < (n+1)δ/2.

Note also that O can be used to output a probabilistic output (to within δ), so outputting specific mixed strategies is possible.

If p1 and p2 are two probability distributions/strategies over possible agent outputs, define them to be "δ-Pareto" if they are within δ of Pareto strategies. We can differentiate (p1,p2) for small changes in strategy, by infinitesimally increasing the weight of some pure strategies o1 and o2 (note that for each strategy, we actually have one less independent degree of freedom than the number of pure strategies, since probabilities must sum to 1). We'll say D(p1,p2)(o1,o2) is Pareto if we are sure that it is an improvement for both players for all possible (p1',p2') within the respective δ ranges of (p1,p2).

If p1 and p2 are two probability distributions over possible agent outputs, we can consider

We'll make the following two assumptions:

The players do not make use of each other's internal structures, just of the possible outputs and the calls to O().

The players do not have access to a joint source of randomness.

So now fix 1 >> ε >> δ > 0, assume the problem is the Prisoner's dilemma as given in the previous figure, let Q() be the other player, and consider the following algorithm:

define LP():
For all outputs o1 of LP(), compute O(LP()==o1).
Call the probability distribution generated p1.
For all outputs o2 of Q(), compute O(Q()==o2).
Call the probability distribution generated p2.

If exists o1 and o2 such that D(p1,p2)(o1,o2) is δ-Pareto:
With probability 1-ε output

...