trentmkelly/LessWrong-43k huggingface dataset:
Mandatory Obsessions
by Jacob Falkovich — 2018-11-14T18:19:53.007Z

Cross-posted from Putanumonit.

The Worst is to Care a Little

At a party the other day, a woman expressed her displeasure at men who “pretend to be allies of gay rights but don’t really care about it”. Did she mean men who proclaimed their progressive attitudes on LGBTQ but voted Republican on Tuesday anyway? Yes, she told me, but not only those. She reserved special scorn for those who vote in support of immigration from Muslim countries, not realizing how Muslim immigrants make the country less safe for gays.

I remarked that she’s setting quite a high bar for these men. Gay rights hardly seem like a central issue in the upcoming elections. In fact, even my gay friends are split among many political views and party allegiances. Considering the second-order impact of every political decision on the lives of gays is a standard that few senators or political pundits meet, let alone guys who have other day jobs and other interests.

“Not good enough,” the woman replied, “if you don’t care enough about gay rights to think it all through you’re not an ally.”

But the other other day, a man at an Effective Altruism meetup expressed his disdain for people who don’t care enough about climate change to vote based on climate policy and said that he doesn’t want them in EA.

And the other other other day, a woman expressed her bafflement at people whose politics aren’t driven by a concern for women’s reproductive rights. She said she doesn’t want to be friends with people who don’t think that abortion is the most important issue today in the US today.

How about someone who is pro-choice but voted for Hillary Clinton just because she shares a concern for AI safety? “That just shows you don’t really care about women,” she answered. Not caring enough is just as bad as having the wrong position.

Since I’m an alien, I don’t get to vote at all (check your privilege, citizens, and stop microagressing against me by sharing exhortations to go out and vote on Facebook). Even if I could vote in New York, I probably wouldn’t. And if I had to, I’d probably vote for Andrew Yang even though I didn’t check what his positions are on gay rights, climate change, and abortion. I can guess what Yang’s positions on these three issues are, and I guess they’re very similar to my own positions, but I wouldn’t care enough to check.

Here’s a list of other topics I was recently criticized for not caringenoug...
Deck Guide: Burning Drakes
by Zvi — 2018-11-13T19:40:00.409Z

[Note to rationalists: This is a straight up strategy article on Magic: The Gathering that the website I write for did not have room for, so if you don’t play, skip this.]

[Note to Magic websites: You have permission to post this article on your site anywhere that is not behind a paywall, provided you are clear that I wrote it, provide a prominent link back to the original post and a note that I will respond to comments here. You can delete these three notes.]

[Note to Magic players: This is Zvi Mowshowitz’s personal blog, which focuses on a variety of topics and is mostly not about Magic: The Gathering. See here for what I consider my best older posts.]

At the Pro Tour, I played an Izzet Drakes deck without Arclight Phoenix. Like all decks it will require adjustments going forward, but I believe the deck strong and well-positioned going forward. Alas, I could not win a game of limited and I had no teammates, so the me and the deck were done for the weekend after its 3-2 start.

Here is the list I played:

Main Deck:

4 Steam Vents

6 Mountain

2 Evolving Wilds

7 Island

4 Sulfur Falls

4 Enigma Drake

4 Crackling Drake

2 Rekindling Phoenix

3 Banefire

2 Shivan Fire

4 Opt

3 Anticipate

1 Divination

1 Search for Azcanta

2 Chemister’s Insight

2 Fight With Fire

2 Ionize

4 Shock

3 Lightning Strike

Sideboard:

1 Rekindling Phoenix

3 Lava Coil

2 Dive Down

2 Inescapable Blaze

3 Negate

1 Search for Azcanta

1 Banefire

2 Fiery Cannonade

Going forward, I would want to work two or three Entrancing Melody into the sideboard, with the puzzle being doing this while retaining a strong post-sideboard configuration against control. I’ll go over the origins of the build, then talk about various matchups.

As I was exploring the Arena free play experience, one of the fifteen decks held far more promise than the others. Wrath of Mages had a bunch of terrible cards with no payoffs, but it also had the core of an Enigma Drake strategy. Taking out the bad cards and putting in good cards like counters, burn and flyers resulted in a fun deck strong enough to grind the constructed event while using minimal wild cards. I used my rare wildcards on the dual lands, some of my commons on blue cantrips, and once my mana was good enough I added Crackling Drakes. At that point, the deck was doing quite well, as it makes short work of the various creature decks people were playing. The downside was that against c...
Acknowledging Human Preference Types to Support Value Learning
by Nandi — 2018-11-13T18:57:52.965Z

We analyze the usefulness of the framework of preference types [Berridge et al. 2009] to value learning by an artificial intelligence. In the context of AI the purpose of value learning is giving an AI goals aligned with humanity. We will lay the groundwork for establishing how human preferences of different types are (descriptively) or ought to be (normatively) aggregated.

This blogpost (1) describes the framework of preference types and how these can be inferred, (2) considers how an AI could aggregate our preferences, and (3) suggests how to choose an aggregation method. Lastly, we consider potential future directions that could be taken in this area.

Motivation

The reason that the concept of multiple preference types is useful for AI is that people often have internal conflicts. Examples of internal conflicts:

You are on a diet and prefer not to eat fattening food, but still really enjoy it.
Addicts that no longer enjoy their drug.
A friend who says he doesn’t want your help, but does.
In conversation with someone racist, you may have conflicting goals. On one hand, you want them to change their mind and not be racist. On the other hand, you want to make them feel bad about their racism. Unfortunately, if you act on the latter, you are unlikely to achieve the former.
Anything you would like to do, but have trouble staying engaged with; for example, you may enjoy exercise or learning a language, and have a goal of getting fit or being fluent, but not be motivated to go for a run or open a textbook.

We think that these internal conflicts can be understood as us having preferences of different types that compete with one another. If an AI ignores the fact that we can have competing preferences, then when it considers a state it will only infer our preference for that state based on one proxy, which will often leave the AI with an incomplete picture. Examples in which taking into account only one data source for preferences leads to complications:

In the case that the AI only takes into account what humans say they want, there is the problem of people lying, leaving out details (because they haven’t thought of them), or not knowing their preferences.
In the case of inverse reinforcement learning (IRL) the AI infers preferences based on behavior only. In the diet example, if people ‘have little willpower’ then an AI using IRL may incorrectly infer that the humans

...
The Steering Problem
by paulfchristiano — 2018-11-13T17:14:56.557Z

Most AI research focuses on reproducing human abilities: to learn, infer, and reason; to perceive, plan, and predict. There is a complementary problem which (understandably) receives much less attention: if you had these abilities, what would you do with them?

The steering problem: Using black-box access to human-level cognitive abilities, can we write a program that is as useful as a well-motivated human with those abilities?

This post explains what the steering problem is and why I think it’s worth spending time on.

Introduction

A capable, well-motivated human can be extremely useful: they can work without oversight, produce results that need not be double-checked, and work towards goals that aren’t precisely defined. These capabilities are critical in domains where decisions cannot be easily supervised, whether because they are too fast, too complex, or too numerous.

In some sense “be as useful as possible” is just another task at which a machine might reach human-level performance. But it is different from the concrete capabilities normally considered in AI research.

We can say clearly what it means to "predict well," "plan well," or "reason well." If we ignored computational limits, machines could achieve any of these goals today. And before the existing vision of AI is realized, we must necessarily achieve each of these goals.

For now, "be as useful as possible" is in a different category. We can't say exactly what it means. We could not do it no matter how fast our computers could compute. And even if we resolved the most salient challenges in AI, we could remain in the dark about this one.

Consider a capable AI tasked with running an academic conference. How should it use its capabilities to make decisions?

We could try to specify exactly what makes a conference good or bad. But our requirements are complex and varied, and so specifying them exactly seems time-consuming or impossible.
We could build an AI that imitates successful conference organizers. But this approach can never do any better than the humans we are imitating. Realistically, it won’t even match human performance unless we somehow communicate what characteristics are important and why.
We could ask an AI to maximize our satisfaction with the conference. But we'll get what we measure. An extensive evaluation would greatly increase the cost of the conference, while a superficial evaluation wou

...
Post-Rationality and Rationality, A Dialogue
by agilecaveman — 2018-11-13T05:55:06.551Z

A dialogue between R (rationalist) and P (post-rationalist)

Any resemblance of these fictional characters to real people is not a coincidence because nothing is ever really a coincidence.

R: I sincerely challenge you to give a purely positive account of "post-rationality" which makes no mention of rationality concepts drawn from there which are being denied or criticized. The way that a smart atheist, so challenged for a positive account of what they do believe in, can explain purely positive ideals of probability and decision which make no mention of God.

P: One key insight that has appeared in multiple post-rationalists’ texts is the causal relationship between morality, identity and group conflict. Rationalists and other “systematic” thinkers, such as libertarians craft their identity based on a particular set of principles, be it the desire to be “less wrong” as measured by probability calibration, or a desire to shape the world according to liberty or non-aggression.

They happen to exist in conflict with other group that for some reason can’t seem to follow their obvious and common-sense principles. For rationalists, those groups might be religious people clinging to “faith,” journalists failing to cite proper statistics when reporting on outliers, politicians who “paint a picture” or “persuade” while not adhering to the facts. The rationalists occasionally wonder why the world isn’t rational, hasn’t adopted their insights or doesn’t see the extreme value of knowing the True Law of Thinking.

The post-rationalist insight is that the conflict between rationalists and non-rationalists is not an arbitrary feature of the landscape of the universe, but rather the originator of both groups. By the virtue of an In-group existing, there must be an Out-group and there must have existed an original conflict that gave birth to the division.

This conflict could have taken many forms, from ethnic, political, inter-institutional, old money vs new money to one guy simply killed in one-against all sacrifice. Whatever it was, the conflict drove the division and eventually necessitated the creation of labels. Our rational thinkers and Their irrational believers. Vs Our God-fearing people and Their heretic iconoclasts.

The labels were not created out of thin air, rather they generally had to capture some real aspects of the human psyche, common sense reasoning and...
Laughing Away the Little Miseries
by Rossin — 2018-11-13T03:31:56.666Z

Note: This falls under the category of self-help advice, the usefulness of which tends to be variable between individuals. This technique has noticeably improved my day to day feelings of well being, but people are often very different and your mileage may vary.

It is notoriously difficult to increase your long term happiness. The psychology literature has coined the term ‘happiness set point’ to describe the fact that people seem to stay around the same level of happiness despite major events in their lives that one might naively expect to make them more or less happy. The classic examples of this effect are that lottery winners aren’t nearly as happy as they thought they would be and paraplegics aren’t nearly as miserable.

Studies of identical twins raised apart show that the “heritability of the stable component of subjective well-being approaches 80%”, according to David Lykken and Auke Tallegen at the University of Minnesota Center for Twin and Family Research. Further, they claim that, “Neither socioeconomic status, educational attainment, family income, marital status, nor an indicant of religious commitment could account for more than about 3% of the variance in well being”.

If it is extremely difficult to permanently increase your general feelings of well-being in the long term, then perhaps we can do better by focusing on the short term. David Lykken recommends this sort of approach, advocating that one find ways to engage in frequent small pleasures to stay bouncing above one’s default happiness levels.

This makes a great deal of sense (and if you're interested he does have a book on the subject, though I didn't find it a particularly enlightening read apart from the points I have already mentioned) but what he does not discuss as much is avoiding the things that pull you below your happiness set point, which I believe are just as important.

It is incredibly easy to allow some utterly trivial event to significantly worsen your day. Whether it is social angst from an awkward or ambiguous interaction, anger from being cut off in traffic, or crushing despair because something you worked hard on failed, there are many things it would be better not to get upset about, but these unwanted emotions are often hard to avoid.

Just realizing you are irrationally emotional often fails to actually make you feel better, at least in my experience, and ber...
Kelly bettors
by DanielFilan — 2018-11-13T00:40:01.074Z

Updated Introduction

I wrote this post roughly two years ago, and somehow it randomly got crossposted to LessWrong today. Nevertheless, despite being poorly written, I think it's neat enough that I've polished it off and left it up. It goes through a needless amount of needlessly general algebra to prove* that:

A prediction market populated by Kelly bettors is isomorphic to a person with a bunch of hypotheses updating with Bayes' rule.

This property is maintained even in the Logical Inductor-type setting where agents don't know exactly what thing they're betting on.

For a more approchable discussion of the Kelly-Bayes connection see this blog post by Buck Shlegeris.

[*] Well, the 'proof' of equilibrium behaviour in the most general case is kind of sketchy, but at any rate the most general case is too general and that proof works fine for the case of countably many possibilities.

The Kelly Criterion

The Kelly criterion for betting tells you how much to wager when someone offers you a bet. First introduced in this paper, it deals with the situation where someone is offering you a contract that pays you €1 if the event E (for concreteness, you can imagine E as the event that the Republican candidate wins the 2020 US Presidential election) and €0 otherwise. They are selling it for €q, and your probability for E is p>q (this is equivalent to the more common formulation with odds, but it's easier for me to think about). As a result, you think that it's worth buying this contract. In fact, they will sell you a scaled-up contract of your choice: for any real number r≥0, you can buy a contract that pays you €r/q if E occurs for €r, just as if you could buy r/q copies of the original contract. The question you face is this: how much of your money should you spend on this scaled-up contract? The Kelly criterion gives you an answer: you should spend (p−q)/(1−q) of your money.

Why would you spend this exact amount? One reason would be if you were an expected utility maximiser, and your utility was the logarithm of your wealth. Note that the logarithm is important here to make you risk averse: if you simply wanted to maximise your expected wealth after the bet, you would bet all your money. To show that expected log-wealth maximisers use the Kelly criterion, note that if your initial wealth is W, you spend fW on the scaled contract, and E occurs, you then have (1−f)

...
Alignment Newsletter #32
by Rohin Shah — 2018-11-12T17:20:03.572Z

Remember, treat all of the "sequence" posts as though I had highlighted them!

Highlights

Spinning Up in Deep RL (Joshua Achiam): OpenAI has released an educational resource aimed to help software engineers become skilled at deep reinforcement learning. It includes simple implementations of many deep RL algorithms (as opposed to the relatively complex, highly optimized implementations in Baselines), educational exercises, documentation, and tutorials. OpenAI will host a workshop on the topic at their headquarters on Feb 2nd, and are also planning to hold a workshop at CHAI some time in early 2019.

Rohin's opinion: I know that a lot of effort has gone into this project, and I expect that as a result this is probably the best educational resource on deep RL out there. The main other resource I know of is the deep RL bootcamp, which probably supplements this resource nicely, especially with the lectures (though it is a year out of date).

Technical AI alignment

Embedded agency sequence

Embedded World-Models (Abram Demski and Scott Garrabrant): A few slides have been added to this post since my summary last week, going into more detail about the grain-of-truth problem. This problem is particularly hard because your learned world model must include the world model itself inside of it, even in the presence of an environment that can behave adversarially towards the world model. It is easy to construct deterministic paradoxes where the world model cannot be correct -- for example, in rock-paper-scissors, if your model predicts what the opponent will do and plays the action that wins against the prediction, the opponent will (if they can) predict that and play the action that beats your action, falsifying your model. While game theory solves these sorts of scenarios, it does so by splitting the agent away from the environment, in a way that is very reminiscent of the dualistic approach. Recently, reflective oracles were developed, that solve this problem by having probabilistic models that were robust to self-reference, but they still assume logical omniscience.

Subsystem Alignment (Abram Demski and Scott Garrabrant): Any agent is likely to be built out of multiple subsystems, that could potentially have their own goals and work at cross-purposes to each other. A simple unrealistic example would be an agent composed of two parts -- a world model and a decision...
AI development incentive gradients are not uniformly terrible
by rk — 2018-11-12T16:27:31.886Z

Much of the work for this post was done together with Nuño Sempere

Perhaps you think that your values will be best served if the AGI you (or your team, company or nation) are developing is deployed first. Would you decide that it's worth cutting a few corners, reducing your safety budget, and pushing ahead to try and get your AI out the door first?

It seems plausible, and worrying, that you might. And if your competitors reason symmetrically, we would get a "safety race to the bottom".

On the other hand, perhaps you think your values will be better served if your enemy wins than if either of you accidentally produces an unfriendly AI. Would you decide the safety costs to improving your chances aren't worth it?

In a simple two player model, you should only shift funds from safety to capabilities if (the relative₁ decrease in chance of friendliness) / (the relative₁ increase in the chance of winning) < (expected relative₂ loss of value if your enemy wins rather than you). Here, the relative₁ increases and decreases are relative to the current values. The relative₂ loss of value is relative to the expected value if you win.

The plan of this post is as follows:
1. Consider a very simple model that leads to a safety race. Identify unrealistic assumptions which are driving its results.
2. Remove some of the unrealistic assumptions and generate a different model. Derive the inequality expressed above.
3. Look at some specific example cases, and see how they affect safety considerations.

A partly discontinuous model

Let's consider a model with two players with the same amount of resources. Each player's choice is what fraction of their resources to devote to safety, rather than capabilities. Whichever player contributes more to capabilities wins the race. If you win the race, you either get a good outcome or a bad outcome. Your chance of getting a good outcome increases continuously with the amount you spent on safety. If the other player wins, you get a bad outcome. A good outcome is a finite amount better than any bad outcome.

We can envisage each player as choosing a point along the line going from 0 to 1: this is the fraction of their resources they're devoting to capabilities. Therefore, the possible pairs of choices can be visualised as the square with sides of length 1. The payoffs for each player are a surface over this square, with a cliff along the line x = y

...
Aligned AI, The Scientist
by Shmi — 2018-11-12T06:36:30.972Z

The problem with constructing an aligned AI is that any active utility function or attempt at world optimization is likely to succumb to the Goodhart's law in one of its many forms, as discussed here and elsewhere by the good people of MIRI. I wonder if a more passive approach is worth considering, or may have been considered already.

Humanity is a part of the Universe, and building and organizing accurate knowledge about the Universe is what science is. Not using the scientific knowledge to advance specific interests or goals, e.g. technological advancements or personal gain, but for the knowledge's sake. Such a scientifically-minded agent would not be interested in modifying the Universe, and would limit any effects to the minimum needed to understand it. A part of this scientific research would be to understand humanity as deeply as possible, including what we humans imagine an aligned AI would look like even though we do not fully understand it ourselves at this point.

Presumably at some point such an AI would understand the universe and the humans in it enough to basically serve as a safe DWIM (do what I mean) genie. It would be inherently safe because doing anything unsafe, or agreeing to do anything unsafe would mean that the genie does not understand the part of the Universe that is the humanity. After all, we would not want to do anything that has unsafe and unintended consequences. "Unsafe" includes doing nothing at all: an AI that would prevent humans from doing anything would not understand humans, and so would not understand the universe. In other words

Aligned AI is AI the scientist, not AI the engineer.

This is, of course, is easier said than done. Learning all about the world while actively minimizing any impact on the world is something that we humans often strive to do when trying to understand the ecosystem of the Earth, with mixed results. Still, sometimes we succeed, and, odds are, so could an agent smarter than us.