trentmkelly/LessWrong-43k huggingface dataset:
Book Trilogy Review: Remembrance of Earth’s Past (The Three Body Problem)
by Zvi — 2019-01-30T01:10:00.414Z

Epistemic Status: Stuff that keeps not going away so I should write it up I suppose. Brief spoiler-free review, then main talk will be spoilerific.

Spoiler-Free Review

I read the trilogy a few months ago, on general strong reviews and lack of a better known science fiction option I hadn’t already read.

I was hoping to get a Chinese perspective, some realistic physics (as per Tyler Cowen’s notes) and a unique take on various things. To that extent I got what I came for. Book felt very Chinese, or at least very not American. Physics felt like it was trying far more than most other science fiction, and the consequences are seriously explored. Take on that and many things felt unique versus other books I’ve read, in ways I will discuss in the main section.

What I didn’t feel I got was a series that was high enough quality to justify its awards and accolades, or allow me to fully recommend reading it to others. It’s not bad, it has some great moments and ideas and I don’t mind having read it, but I was hoping for better. That’s fine. That is probably a lot about my expectations getting too high, as I can’t point to (in the limited selection of things I’ve read) recent science fiction I think is even as good. Like other genres, read mostly old books is wise advice that I follow less than I should.

It is a reasonable decision to do any of: Not read the book and not continue further, not read the book and allow it to be spoiled here, to read some and see if you like it, or to read the whole thing.

This long post is long. Very long. Also inessential. Apologies. I definitely didn’t have the time to make it shorter. Best to get it out there for those who want it, and to move on to other things.

All The Spoilers – The Plot Summary

(This unfolds everything in linear order, the books rightly keep some things mysterious at some points by telling events somewhat out of order. This is what I remember and consider important, rather than an attempt to include everything. There’s a lot that happens that’s interesting enough to be included!)

Book I – The Three Body Problem

Communist China during the cultural revolution was really bad. Reeling from a combination the cultural revolution and its murder of (among others) her father and all reasonable discourse, her forced exile to the countryside, and environmental panic raised by a combination of Silent Spring and the actual terrible environ...
Alignment Newsletter #43
by Rohin Shah — 2019-01-29T21:10:02.373Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

AlphaStar: Mastering the Real-Time Strategy Game StarCraft II (The AlphaStar team): The AlphaStar system from DeepMind has beaten top human pros at StarCraft. You can read about the particular details of the matches in many sources, such as the blog post itself, this Vox article, or Import AI. The quick summary is that while there are some reasons you might not think it is conclusively superhuman yet (notably, it only won when it didn't have to manipulate the camera, and even then it may have had short bursts of very high actions per minute that humans can't do), it is clearly extremely good at StarCraft, both at the technically precise micro level and at the strategic macro level.

I want to focus instead on the technical details of how AlphaStar works. The key ideas seem to be a) using imitation learning to get policies that do something reasonable to start with and b) training a population of agents in order to explore the full space of strategies and how to play against all of them, without any catastrophic forgetting. Specifically, they take a dataset of human games and train various agents to mimic humans. This allows them to avoid the particularly hard exploration problems that happen when you start with a random agent. Once they have these agents to start with, they begin to do population-based training, where they play agents against each other and update their weights using an RL algorithm. The population of agents evolves over time, with well-performing agents splitting into two new agents that diversify a bit more. Some agents also have auxiliary rewards that encourage them to explore different parts of the strategy space -- for example, an agent might get reward for building a specific type of unit. Once training is done, we have a final population of agents. Using their empirical win probabilities, we can construct a Nash equilibrium of these agents, which forms the final AlphaStar agent. (Note: I'm not sure if at the beginning of the game, one of the agents is chosen according to the Nash probabilities, or if at each timestep an action is chosen according to the Nash probabilities. I would expect the former, since the latter would result in one agent making ...
The Question Of Perception
by The Arkon — 2019-01-29T20:59:23.450Z

The below text is an excerpt of this article.

Once during an Indian coconut harvest, a farmer, tired from his day’s work of chopping down fruit, slumped down in the shade of a tree to enjoy a coconut, and upon splitting it open, found inside a message from God (or in this case, from Vishnu, his Hindu deity). The Brahmic writing was plainly visible for anyone to see, spelt out in the two halves of oily white meat. The implications of such an experience could only have been one of the following: the first is that the Supreme Being has no qualms about revealing his Divine Will in the contents of mere palm fruit, any more than in a whirlwind or through an oracle. The second is that the farmer’s perceptual systems produced an inaccurate representation of what he saw engraved in the fruit lining. It’s anybody’s guess as to which it really was. But regardless of whatever information was contained in the coconut, the moral of the story is that whilst miracles are known to happen, it can also be said that people regularly see things which are not there.

That we end up being misled by our senses is a widely-accepted truism, as most humans mistake the limits of their perception for the limits of the world itself. It’s not uncommon for people to push forward in their endeavors with a premature understanding of things, using inadequate standards to measure what they see in the world around them. Such errors in judgement range from being mild, such as mistakenly purchasing a rotten apple because you didn’t inspect its underside, to being detrimental, such as presuming that the oasis in the middle of the desert is real, when it’s only a hallucination. Hence, there are times when people are not so much moved by external objects as by their perception of those objects; what they claim to “know” is in fact only known conditionally, since human knowledge is always limited, thanks in part to our limited perceptual systems. Rare are those whose conception of reality is not exclusively dependent on their perception of it, who understand that there are things which you know and things which you don’t know, and in between are several doorways, of which one is called “Perception”; this door must be entered with prudence.

It is a common presumption that people see the world primarily with their eyes, but in fact, that is only true when they look at the objects that they recognize; t...
Which textbook would you recommend to learn decision theory?
by supermartingale — 2019-01-29T20:48:11.577Z

Eliezer talks a lot about decision theory in his sequences, e.g. the Aumann agreement theorem or the von Neumann-Morgenstern utility theorem. From what I've seen so far, decision theory looks extremely interesting.

Which textbook in decision theory would you recommend to start with? I'd appreciate if the book contained not only theory, but also some exercise/problem section - I have noticed that usually a lecture is not enough to fully grasp a topic. I want a book which will not shy away from mathematical side of the theory.

I have a strong background in mathematics and computer science, but I only know a little about game theory.
Towards equilibria-breaking methods
by ryan_b — 2019-01-29T16:19:57.564Z

All of this is vague intuition.

Much has been said about the problems of Nash equilibria. I've been wondering about how to quantify these sorts of problems. From Moloch's Toolbox:

3. Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically coordinated action could move to a new stable state.

My animating thought is that information and incentives aren't evenly distributed, and further there is an important time dimension to the problem. I expect it should be possible to temporarily disrupt any given equilibrium. From disruption, a better equilibrium could be reached. Alternatively, maybe it is possible to go directly to the better equilibrium.

We should be able to get knowledge of the actors.
Information, which is to say beliefs-about-the-current-equilibrium, is not perfect or even among them. How imperfect and how uneven?
With knowledge of the actors, we should be able to get a distribution of their incentives.
Now we know how many actors, and their incentives, and their information. With this knowledge, we should be able to determine how big an incentive we need to offer in order to shift them into a target equilibrium.
If we want it to be a new equilibrium, we will need to affect enough actors to make it fairly stable. How many actors will we need, a la the hundredth monkey effect?
It is probably important to be able to tailor incentives to actors.
My intuition for the threshold is that the weaker the belief in the current equilibrium, the less incentive will be required to shift them. So the target is something like (current incentives from equilibrium + incentive to overcome belief).

I think of this like surveying a rock face for blasting. Once we identify the lowest-dynamite method, anyone else with enough dynamite can come along and detonate.

My default assumption is that this will take the traditional form of getting a lot of capital and using that to directly or indirectly provide these incentives. Under this assumption there needs to be a reason for that capital to be available, which is to say a profit; maybe this is realizable through market operations directly. Alternatively this could be the vehicle for a startup's pitch; it seems like this process would qualify as superlative market research.

However, it does occur to me that actors might be more like individuals wh...
Can there be an indescribable hellworld?
by Stuart_Armstrong — 2019-01-29T15:00:54.481Z

Can there be an indescribable hellworld? What about an un-summarisable one?

By hellworld, I mean a world of very low value according to our value scales - maybe one where large number of simulations are being tortured (aka mind crimes).

A hellworld could look superficially positive, if we don't dig too deep. It could look irresistibly positive.

Could it be bad in a way that we would find indescribable? It seems that it must be possible. The set of things that can be described to us is finite; the set of things that can be described to us without fundamentally changing our values is much smaller still. If a powerful AI was motivated to build a hellworld such that the hellish parts of it were too complex to be described to us, it would seem that it could. There is no reason to suspect that the set of indescribable worlds contains only good worlds.

Can it always be summarised?

Let's change the setting a bit. We have a world W, and a powerful AI A that is giving us information about W. The A is aligned/friendly/corrigible or whatever we need to be. It's also trustworthy, in that it always speaks to us in a way that increases our understanding.

Then if W is an indescribable hellworld, can A summarise that fact for us?

It seems that it can. In the very trivial sense, it can, by just telling us "it's an indescribable hellworld". But it seems it can do more than that, in a way that's philosophically interesting.

A hellworld is ultimately a world that is against our values. However, our values are underdefined and changeable. So to have any chance of saying what these values are, we need to either extract key invariant values, synthesise our contradictory values into some complete whole, or use some extrapolation procedure (eg CEV). In any case, there is a procedure for establishing our values (or else the very concept of "hellworld" makes no sense).

Now, it is possible that our values themselves may be indescribable to us now (especially in the case of extrapolations). But A can at least tell us that W is against our values, and provide some description as to the value it is against, and what part of the procedure ended up giving us that value. This does give us some partial understanding of why the hellworld is bad - a useful summary, if you want.

On a more meta level, imagine the contrary - that W was hellworld, but the superintelligent agent A could not indicate what hum

...
How much can value learning be disentangled?
by Stuart_Armstrong — 2019-01-29T14:17:00.601Z

In the context of whether the definition of human values can disentangled from the process of approximating/implementing that definition, David asks me:

But I think it's reasonable to assume (within the bounds of a discussion) that there is a non-terrible way (in principle) to specify things like "manipulation". So do you disagree?

I think it's a really good question, and its answer is related to a lot of relevant issues, so I put this here as a top-level post. My current feeling is, contrary to my previous intuitions, that things like "manipulation" might not be possible to specify in a way that leads to useful disentanglement.

Why manipulate?

First of all, we should ask why an AI would be tempted to manipulate us in the first place. It may be that it needs us to do something for it to accomplish its goal; in that case it is trying to manipulate our actions. Or maybe its goal includes something that cashes out as out mental states; in that case, it is trying to manipulate our mental state directly.

The problem is that any reasonable friendly AI would have our mental states as part of its goal - it would at least want us to be happy rather than miserable. And (almost) any AI that wasn't perfectly indifferent to our actions would be trying to manipulate us just to get its goals accomplished.

So manipulation is to be expected by most AI designs, friendly or not.

Manipulation versus explanation

Well, since the urge to manipulate is expected to be present, could we just rule it out? The problem is that we need to define the difference between manipulation and explanation.

Suppose I am fully aligned/corrigible/nice or whatever other properties you might desire, and I want to inform you of something important and relevant. In doing so, especially if I am more intelligent than you, I will simplify, I will omit irrelevant details, I will omit arguably relevant details, I will emphasise things that help you get a better understanding of my position, and de-emphasise things that will just confuse you.

And these are exactly the same sorts of behaviours that smart manipulator would do. Nor can we define the difference as whether the AI is truthful or not. We want human understanding of the problem, not truth. It's perfectly possible to manipulate people while telling them nothing but the truth. And if the AI structures the order in which it presents the true facts, it can

...
Techniques for optimizing worst-case performance
by paulfchristiano — 2019-01-28T21:29:53.164Z

If powerful ML systems fail catastrophically, they may be able to quickly cause irreversible damage. To be safe, it’s not enough to have an average-case performance guarantee on the training distribution — we need to ensure that even if our systems fail on new distributions or with small probability, they will never fail too badly.

The difficulty of optimizing worst-case performance is one of the most likely reasons that I think prosaic AI alignment might turn out to be impossible (if combined with an unlucky empirical situation).

In this post I want to explain my view of the problem and enumerate some possible angles of attack. My goal is to communicate why I have hope that worst-case guarantees are achievable.

None of these are novel proposals. The intention of this post is to explain my view, not to make a new contribution. I don’t currently work in any of these areas, and so this post should be understood as an outsider looking in, rather than coming from the trenches.

Malign vs. benign failures and corrigibility

I want to distinguish two kinds of failures:

“Benign” failures, where our system encounters a novel situation, doesn’t know how to handle it, and so performs poorly. The resulting behavior may simply be erratic, or may serve an external attacker. Their effect is similar to physical or cybersecurity vulnerabilities — they create an opportunity for destructive conflict but don’t systematically disfavor human values. They may pose an existential risk when combined with high-stakes situations, in the same way that human incompetence may pose an existential risk. Although these failures are important, I don’t think it is necessary or possible to eliminate them in the worst case.
“Malign” failures, where our system continues to behave competently but applies its intelligence in the service of an unintended goal. These failures systematically favor whatever goals AI systems tend to pursue in failure scenarios, at the expense of human values. They constitute an existential risk independent of any other destructive technology or dangerous situation. Fortunately, they seem both less likely and potentially possible to avoid even in the worst case.

I’m most interested in malign failures, and the narrower focus is important to my optimism.

The distinction between malign and benign failures is not always crisp. For example, suppose we try to predict a human’s preferen...
[Link] Did AlphaStar just click faster?
by aog — 2019-01-28T20:23:08.196Z

This is a linkpost for: https://medium.com/@aleksipietikinen/an-analysis-on-how-deepminds-starcraft-2-ai-s-superhuman-speed-could-be-a-band-aid-fix-for-the-1702fb8344d6.

tl;dr: AlphaStar clicked at a rate of 1000+ Actions Per Minute for five second periods, and a rate 1500+ APM for fractions of a second. The fastest human player can't sustain anything above 500 APM for more than a second or two. Did AlphaStar just spam click its way to victory?
"Giftedness" and Genius,
Crucial Differences
by ArthurLidia — 2019-01-28T20:22:37.664Z

This essay by Arthur Jensen is from an old book .

The genius has limits. A simple answer, and undoubtedly true. But my
assignment here is to reflect on the much more complex difference
between intellectual giftedness and genius, using the latter term in its
original sense, as socially recognized, outstandingly creative achievement. In
this think-piece (which is just that, rather than a comprehensive review of the
literature), I will focus on factors, many intriguing in and of themselves, that
are characteristic of genius. My primary thesis is that the emergence of genius is
best described using a multiplicative model.

I will argue that exceptional achievement is a multiplicative function of a
number of different traits, each of which is normally distributed, but which in
combination are so synergistic as to skew the resulting distribution of achieve­
ment. An extremely extended upper tail is thus produced, and it is within this
tail that genius can be found. An interesting two-part question then arises: how
many different traits are involved in producing extraordinary achievement, and
what are they? The musings that follow provide some conjectures that can be
drawn on to answer this critical question.
As a subject for scientific study, the topic of genius, although immensely
fascinating, is about as far from ideal as any phenom enon one can find. The
literature on real genius can claim little besides biographical anecdotes and
speculation, with this chapter contributing only more of the same. Whether the
study of genius will ever evolve from a literary art form into a systematic science
is itself highly speculative. The most promising efforts in this direction are
those by Simonton (198 8 ) and Eysenck (1 9 9 5 ), with Eysenck’s monograph
leaving little of potential scientific value that can be added to the subject at
present, pending new empirical evidence.

Intelligence

Earlier I stated that genius has limits. But its upper limit, at least in some fields,
seems to be astronomically higher than its lower limit. Moreover, the upper
lim it of genius cannot be described as characterized by precocity, high intel­
ligence, knowledge and problem -solving skills being learned with speed and
ease, outstanding academic achievement, honors and awards, or even intellec­
tual productivity. Although such attributes are commonly found at all levels of
genius, they are not discriminating in the realm of g...
Quantum Neural Net and You
by ScrubbyBubbles — 2019-01-28T18:42:01.456Z

Been thinking about this theory for awhile and just wanted some feedback and thoughts regarding it.

Theory:

We are indeed living in a simulation, however, we are self learning AI constructs within this simulation. Simulation of this magnitude would require massive computing power and resources. Knowing this the computing power is shared / offloaded to the AI within the simulation. Every AI within the simulation would get fragmented data which would be complied and relayed to the correct target / host. The AI within the simulation would be unaware of this situation. As would the AI controlling the simulation.

Since neither AI would be aware of each other's existence it would keep both simulation's 'pure' and provide a necessary conflict to challenge the other.