trentmkelly/LessWrong-43k huggingface dataset:
Pascal's Mugging and One-shot Problems
by Mathilde — 2019-04-22T22:12:21.230Z

I've had some thoughts on Pascal's Mugging which might be worth sharing. I'm assuming some familiarity with Pascal's Mugging in this post.

Before we get really started, let's transform Pascal's Mugging into a problem that is easier to reason about but still gets at the core idea.

First, forget "utility", forget money, we're maximising paperclips. I know it's kind of silly, but using money or utility really tends to muddy up thinking. I was surprised how much easier it was to reason about the problem when I switched to paperclips[0].

Now, here's my version of the problem. Suppose there's a casino, in which there is a googolplex-sided die (10^(10^100) sides). You can pay 10 paperclips to roll the die, which are destroyed. If you roll a 1 on the die, the casino manufactures 3^^^^3 paperclips. Otherwise, nothing else happens and you lost 10 paperclips.

3^^^^3 is much, much, much larger than a googolplex, so expected utility maximisation overwhelmingly says you should play at this casino if you're trying to maximise the number of paperclips.

(From this point on, I will refer to "expected utility" as "EU")

In some cases, EU is correct. Here are 3 cases.

Case 1: If you expect to live a googolplex years, then paying to roll the die a few times per year is a good idea, because over that time frame the probability of winning at least once is very high.

Case 2: You have a normal lifespan, but there are a googolplex other paperclip maximisers on Earth. In this case, everyone plays, and the probability that at least one person wins is very high.

Case 3: You are the only paperclip maximiser in this universe, but there are a googolplex alternate universes that contain alternate "you"s who are in the same situation as you, and for whom the dice rolls are all independent. In this case the probability that at least one "you" wins is high, so you should play.

Here is a different case.

Case 0: You are alone in existence. There is no one else on earth, there are no alternate realities, you are literally alone in the entirety of all existence. This is the only decision you will ever get the chance to make. Should you pay 10 paperclips to roll the die?

I think it's clear in Case 0 that you should not pay. If you pay, you lose 10 paperclips[1] and, for all practical purposes, are certain to lose. If you don't pay, at least you get to keep your 10 paperclips. Since we're trying to maximis

...
IRL 8/8: Generative Adversarial Imitation Learning
by RAISE — 2019-04-22T15:02:09.779Z
Link: https://app.grasple.com/#/level/1587

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 8, the last. To find out about our next plans, stay tuned for new LW posts or follow our Facebook group.

Note that access to the lessons requires creating an account here.

Have a nice day!
Counterfactuals about Social Media
by Zvi — 2019-04-22T12:20:00.476Z

Response To (Marginal Revolution): Counterfactuals about Social Media

See also: Against Facebook, Against Facebook: Comparison to Alternatives and Call to Action

The idea that the primary problem with such programs is ‘they make political fights weird’ or that ‘they enable censorship’ is to miss the bigger problem. Social media is ruining our lives. Directly.

They also degenerate our politics. That’s mostly a side effect.

Social media succeeds largely because of network effects. One uses Facebook, Twitter, Instagram or the others mostly because others you want to interact with are using them.

Many people know social media to be terrible for them and for their lives. Many people know Facebook is terrible, in particular (whether or not the photo-based Snapchat and Instagram, which my circles never used, are even worse, as I suspect they are). Many of those would love a better alternative.

But coordination is hard. By the time many people figured this out, it was too late.

Shifting the equilibrium by force is a reasonable response.

Even if it wasn’t too late, it’s not clear many or even a majority of people knowing it is a trap are enough to stop the trap. If Facebook is where the discussions are, and Facebook is where your friends are, that is where you will be unless you are willing to pay a heavy price. The default outcome is for Facebook to continue optimizing for its usage and revenue in ways that make our lives worse, until most people are worse off than without social media and know it, but think they are better off than if everyone else was on social media without them.

I did manage to get a bunch of my friends and readers off of Facebook, such that the equilibrium shifted to a better one. It can be done. But it is damn hard.

These companies also are engaging in a bunch of profit-maximizing and power-maximizing actions, like not letting users have control over what they see, that force users to learn to do what Facebook wants and be who Facebook wants them to be, addicts constantly producing free content. All communication between friends becomes optimized to reinforce the platform. The system rewards and shows all communication based on its anticipated reinforcement of user addictive behavior. This declares war for coalition politics and playing popularity and social games, and war against truth or any attempt to accurately map or model the world.

And yes, at so...
On the Nature of Programming Languages
by Martin Sustrik — 2019-04-22T10:50:00.862Z

What are we doing when designing a programming language? We decide whether it's going to be imperative or declarative. We add a bunch of operators. We add some kind of objects and visibility rules. We decide to make it either strongly-typed or weakly-typed. Maybe we add generics or inheritance, or maybe multiple inheritance. And so on.

The question that interests me is whether the nature of these tools is determined by the problems we want to solve, whether they are, in some way, inherent to or, in other words, directly inferrable from the nature of the problem at hand, or whether they are rather crutches for our imagination, a purely psychological constructs that help our imperfect brains to deal with the complexity of the real world.

If you asked me ten years ago, I would probably argue for the latter.

And it's not hard to "prove" it: If two people write code to solve the same problem and one makes a terrible spaghetti monster in COBOL while the other goes for super-elegant and highly abstracted solution in Haskell, does it really matter to the computer? As long as the two are compiled to the same machine code, the machine does not care. All the clever constructs used, all the elegance, they are there only to guide our intuition about the code.

So, you see, the design of a programming language is determined by human psychology, by the quirks and foibles of the human brain. If there are inheritance hierarchies in your language it's not because things in the real world tend to arrange in neat trees — in fact, they never do, and if you want to point out the tree of life, I have bad news for you: most organisms have two parents and the tree of life is really a DAG of life — it's because human brains like to think in terms of taxonomies. They just happen to find it easier to deal with the complex world in that particular way.

But is it really so?

If you asked me today I wouldn't be so sure.

But it's a question not easy to answer. If you wanted to really tackle it you would need to take human brain out of the equation, so that you can find out whether the same kinds of "language constructs" arise even without a brain having anything to do with it.

So here's the idea: What about genetic algorithms? They don't require a programmer. Yet we can look at them and determine if a language feature, say, encapsulation does emerge from the process.

Or, rather, given the r

...
Nutritional Supplements: A Potted Guide?
by pcm50 — 2019-04-22T09:51:40.877Z

I'd be interested to find a succinct and reliable guide to nutritional supplements. Understanding that the best thing to do would be to properly understand human biology, read papers on supplements, sports science etc and make up my own mind from there. Unfortunately I don't have time at the moment to do that. Nevertheless there should be a truth of the matter here.

I'd love to know how to find out:

What has been shown to be beneficial.
How reliable that knowledge is.

My own preference is to only take a punt on supplements which have a substantial effect and a strong evidence base. Other people may slide their preferences around.

The issue is that this information is very difficult to research without the fundamentals because the space is so full of spam. Hence why I am asking this here.
AI Alignment Problem: 
“Human Values” don’t Actually Exist
by avturchin — 2019-04-22T09:23:02.408Z

Previous posts in the series: “What AI Safety Researchers Have Written About the Nature of Human Values”, Possible Dangers of the Unrestricted Value Learners. Next planned post: AI safety approaches which don’t use the idea of human values.

Summary: The main current approach to the AI safety is AI alignment, that is, the creation of AI whose preferences are aligned with “human values.” Many AI safety researchers agree that the idea of “human values” as a constant, ordered sets of preferences is at least incomplete. However, the idea that “humans have values” underlies a lot of thinking in the field; it appears again and again, sometimes popping up as an uncritically accepted truth. Thus, it deserves a thorough deconstruction, which I will do by listing and analyzing comprehensively the hidden assumptions of the idea that “humans have values.” This deconstruction of human values will be centered around the following ideas: “Human values” are useful descriptions, but not real objects; “human values” are bad predictors of behavior; the idea of a “human value system” has flaws; “human values” are not good by default; and human values cannot be separated from human minds. The method of analysis is listing hidden assumptions on which the idea of “human values” is built. I recommend that either the idea of “human values” should be replaced with something better for the goal of AI safety, or at least be used very cautiously. The approaches to AI safety which don’t use the idea of human values at all may require more attention, like the use of full brain models, boxing, and capability limiting.

Introduction

The idea of AI which learns human values is a core of the current approach to artificial general intelligence (AGI) safety. However, it is based on some assumptions about the nature of human values, including the assumption that they completely describe human motivation, are non-contradictory, are normatively good, etc. Actual data from psychology provides a rather different picture.

The literature analysis of what other AGI safety researchers have written about the nature of human values is rather large and is presented in another of my texts: What AGI Safety Researchers Have Written About the Nature of Human Values. A historical overview of the evolution of the idea of human values can be found in Clawson, Vinson, “Human values: a historical and interdisciplinary ...
Evidence other than evolution for optimization daemons?
by Liam Donovan — 2019-04-21T20:50:18.986Z

The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don't have a good mental model of the differences between a world where optimization daemons can arise and a world where they can't (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I've seen is the evolution of humans, but I find it concerning that I can't make any other predictions about the world based on the idea of ODs.

What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?

[removed discussion of religion to make the question more clear/straightforward]
Any rebuttals of Christiano and AI Impacts on takeoff speeds?
by SoerenMind — 2019-04-21T20:39:51.076Z

14 months ago, Paul Christiano and AI Impacts both published forceful and well-received take-downs of many arguments for fast (discontinuous) takeoff. I haven’t seen any rebuttals that are written by established researchers, longer than comments, or otherwise convincing. The longer there is no response, the less weight I put on the outside view that proponents of fast takeoff may be right.

Where are the rebuttals? Did I miss them? Is the debate decided? Did nobody have time or motivation to write something? Is the topic too hard to explain?

Why rebuttals would be useful:

-Give the community a sense of the extent of expert disagreement to form outside views.

-Prioritization in AI policy, and to a lesser extent safety, depends on the likelihood of discontinuous progress. We may have more leverage in such cases, but this could be overwhelmed if the probability is low.

-Motivate more people to work on MIRI’s research which seems more important to solve early if there is fast takeoff.
Passing through St. Louis
by Bae's Theorem — 2019-04-21T12:45:49.294Z

I'm Alex, the coordinator for the Kansas City Rationalists. I'll be passing through St Louis for brunch/lunch on May 17th, and would love to network with anyone who's available. My community is highly varied; we run a game night, a dojo, and an SSC meetup. I'd love to share my experiences if that would be valuable to you. Or we can just socialize.

Location is up to any interested parties.
The Stack Overflow of Factored Cognition
by rmoehn — 2019-04-21T12:19:39.262Z

Abstract

Factored cognition is a possible basis for building aligned AI. Currently Ought
runs small-scale experiments with it. In this article I sketch some benefits of
building a system for doing large-scale experiments and generating large amounts
of data for ML training. Then I estimate roughly how long it would take to build
such a system. I'm not confident of this exploration being useful at all. But at
least I wrote it down.

Benefits

If you want to know what factored cognition is, see
here.

Ought does small-scale experiments with factored cognition (cf. Ought's Progress
Update Winter 2018). I thought: wouldn't it be nice to do these experiments at
much larger scale? With enough users that one root question could be answered
within three hours any time and day of the week.

Benefits:

The feedback loop would be much tighter than with the weekly or bi-weekly
experiments that Ought runs now. A tight feedback loop is great in many ways.
For example, it would allow a researcher to test more hypotheses more
often, more quickly and more cheaply. This in turn helps her to generate more
hypotheses overall.

Note that I might be misunderstanding the goals and constraints of Ought's
experiments. In that case this benefit might be irrelevant.

It would generate a lot of data. These could be used as training data when we
want to train an ML system to do factored cognition.

Quantifying these benefits is possible, but would take some weeks of modelling
and talking with people. So far I'm not confident enough of the whole idea to
make the effort.

Feasibility

We would need three things for a large-scale factored cognition system to work:
the system itself, enough users and useful behaviour of these users. I'll use
Stack Overflow as a basis for my estimates and call large-scale factored
cognition ‘Fact Overflow’.

Building Stack Overflow took five months from start of development
㊮ to public beta
㊮.
Then they spent a lot of time tweaking the system to make it more attractive and
maintain quality. So I'd say building Fact Overflow would take five to fifteen
months with a team of two to five people.

For calculating how many users would be required, I used the following estimates
(90 % confidence interval, uniformly distributed):

variable
– 5 %
– 95 %
– explanation

nw
15
300
– average number of workspaces per tree

na
1
5
– average number of actions

...
Helen Toner on China, CSET, and AI
by Rob Bensinger — 2019-04-21T04:10:21.457Z
Link: http://rationallyspeakingpodcast.org/show/rs-231-helen-toner-on-misconceptions-about-china-and-artific.html

Miscellaneous excerpts I found interesting from Julia Galef's recent interview of Helen Toner:

Helen Toner: [...] [8:00] I feel like the West has this portrayal of the company Baidu as the "Google" of China. And when you think “Google,” you think super high tech, enormous, has many products that are really cool and really widely used. Like Google has search, obviously, but it also has Gmail. It also has Google Maps. It has a whole bunch of other things, Android...

So I feel like this term, “the Google of China,” gets applied to Baidu in all kinds of ways. And in fact, it's sort of true that Baidu is the main search engine in China, because Google withdrew from China. But kind of all the other associations we have with Google don't fit super well into Baidu. Maybe other than, it is one of China's largest tech companies, that is true.

But just in terms of my overall level of how impressed I was with Baidu as a company, or how much I expected them to do cool stuff in the future -- that went down by a lot, just based on... there's no… Baidu Maps exists, but no one really uses it. The most commonly used maps app is a totally different company. There's no Baidu Mail. There's no Baidu Docs. There's a lot of stories of management dysfunction or feudalism internally. So, that was one of the clearest updates I made.

Julia Galef: [...] [19:40] In your conversations in particular with the AI scientists who you got to meet in China, what did you notice? Did anything surprise you? Were their views different in any systematic way from the American AI scientists you'd talked to?

Helen Toner: Yeah. So, I should definitely caveat that this was a small number of conversations. It was maybe five conversations of any decent length.

Julia Galef: Oh, you also went to at least one AI conference, I know, in China.

Helen Toner: Yes, that's true. But it was much more difficult to have sort of substantive, in-depth conversations there.

I think a thing that I noticed in general among these conversations with more technical people… In the West, in similar conversations that I've been a part of, there's often been a part of the conversation that was dedicated to, "How do you think AI will affect society? What do you think are the important potential risks or benefits or whatever?" And maybe I have my own views, and I...
Generating a novelty scale
by Three-Monkey Mind — 2019-04-20T09:54:41.400Z

I was writing a post when I thought “this isn’t very novel. I should mark it as such.” It then occurred to me to make a 1–10 scale to describe how novel a piece is, similar to Gwern's importance tags. However, I’d prefer an objective scale, not a rank scale that Resorter generates. If I never write anything super-duper-new, I shouldn’t have any 10s on the novelty scale.

Currently, I have a rather coarse four-point scale with points that aren’t uniformly distributed between their neighbors:

A verbatim copy of something else.

Demonstrating how to apply a technique publicly documented elsewhere.

Slightly generalizing an idea publicly documented elsewhere.

Paradigm-shifting reconceptualization akin to Newton figuring out that the same thing causes apples to fall from trees and planets to stay in their orbits.

I’m trying to flesh out this scale as best as I’m able, but I’d like to get some help if I can. I’m also trying to separate novelty from both utility and importance. For example, “hold the banana by the handle end, pinch the other end to make a crack in it, and peel it starting at the crack” is no more than a 1.1 on the novelty scale, maybe a 4 on the utility scale, and 2 or so on the importance scale.

Are there any other useful points on a novelty scale?
Reflections on Duo Standard
by Zvi — 2019-04-18T23:20:01.037Z

Previously: Speculations on Duo Standard, Reflections on the Mythic Invitational

Companion Piece: Reflections on the Mythic Invitational

Frank Karsten (Channel-Fireball): The Mythic Invitational Wasn’t Perfect And It Was Still a Smashing Success

There are two big questions to ask about Duo Standard.

The first question is, is Duo Standard a good format? Should we continue to play it? Or should we abandon it?

Wizards has already answered that this is definitely not the final form. That still leaves the question of why, and how to improve.

The second question is, What is the right thing to do in Duo Standard, right now or in general? What game theory questions, metagame shifts and strategic issues are most important? How does this differ not only from best-of-three, but from best-of-one?

My answer to the first question is: No. Duo Standard is not a good format. We should not continue to play it. The Mythic Invitational was played and the verdict is in.

The best way to give a more complete answer to the first question is to answer the second question, then double back. So that’s how this is structured.

Duo Standard: Level Zero

The biggest errors in my Speculations on Duo Standard were errors about level zero: Understanding best-of-one Standard.

I stopped playing games of best-of-one Standard the moment I realized there was a best-of-three ladder on Magic Arena. Why would I want to play an impoverished game of Magic, unless I had severe time constraints or a deck that wasn’t ready to field a sideboard yet?

Thus, I missed some key elements and dynamics. So did the players and commentary team, although different ones, and less of them.

Duo Standard is radically different from simple best-of-one. But if you start from an incorrect base, you’ll reach wrong conclusions.

The developments I missed were: Mastermind’s Acquisition in Esper, Esper Acuity, the dismissal of Sultai and the dismissal of blue. I also underestimated people’s love of mono-red and mono-white on this level, in addition to my underestimation of it on levels one and two.

The level-zero development most players definitely missed were: Blue is still great.

The ambiguous issue is that Gruul was both (from where I sit) underplayed and badly built, and did poorly.

I also suspect Golgari/Sultai was wrongly dismissed, but I dislike and don’t play the deck, so it’s hard for me to say.

Lets talk quickly about all six.

1. Mast...