trentmkelly/LessWrong-43k huggingface dataset:
You can be wrong about what you like, and you often are
by Adam Zerner — 2018-12-17T23:49:39.935Z

Meta: I'm not saying anything new here. There has been a lot of research on the topic, and popular books like Stumbling on Happiness have been written. Furthermore, I don't think I have explained any of this particularly well, or provided particularly enlightening examples. Nevertheless, I think these things are worth saying because a) a lot of people have an "I know what I like" attitude, and b) this attitude seems pretty harmful. Just be sure to treat this as more of an exploratory post than an authoritative one.

I think that the following attitudes are very common:

I'm just not one of those people who enjoys "deeper" activities like reading a novel. I like watching TV and playing video games.
I'm just not one of those people who likes healthy foods. You may like salads and swear by them, but I am different. I like pizza and french fries.
I'm just not an intellectual person. I don't enjoy learning.
I'm just not into that early retirement stuff. I need to maintain my current lifestyle in order to be happy.
I'm just not into "good" movies/music/art. I like the Top 50 stuff.

Imagine what would happen if you responded to someone who expressed one of these attitudes by saying "I think that you're wrong." Often times, the response you'll get is something along the lines of:

Who are you to tell me what I do and don't like? How can you possibly know? I'm the one who's in my own head. I know how these things make me feel.

When I think about that response, I think about optical illusions. Consider this one:

When I think about that response, I think about the following dialog:

Me: A and B are the same shade of gray.

Person: No they're not! WTF are you talking about? How can you say that they are? I can see with my eyes that they're not!

I understand the frustration. It feels like they're different shades. It feels like it is stupidly obvious that they're different shades.

And if feels like you know what you like.

But sometimes, sometimes your brain lies to you.

The image of the squares is an optical illusion. Neuroscientists and psychologists study them. And they write books like this about them.

The question of knowing what you like can be a hedonic illusion (that's what I'll decide to call it anyway). Neuroscientists and psychologists study these il...
Alignment Newsletter #37
by Rohin Shah — 2018-12-17T19:10:01.774Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

Three AI Safety Related Ideas and Two Neglected Problems in Human-AI Safety (Wei Dai): If any particular human got a lot of power, or was able to think a lot faster, then they might do something that we would consider bad. Perhaps power corrupts them, or perhaps they get so excited about the potential technologies they can develop that they do so without thinking seriously about the consequences. We now have both an opportunity and an obligation to design AI systems that operate more cautiously, that aren't prone to the same biases of reasoning and heuristics that we are, such that the future actually goes better than it would if we magically made humans more intelligent.

If it's too hard to make AI systems in this way and we need to have them learn goals from humans, we could at least have them learn from idealized humans rather than real ones. Human values don't extrapolate well -- just look at the myriad answers that people give to the various hypotheticals like the trolley problem. So, it's better to learn from humans that are kept in safe, familiar environment with all their basic needs taken care of. These are our idealized humans. In practice the AI system would learn a lot from the preferences of real humans, since that should be a very good indicator of the preferences of idealized humans. But if the idealized humans begin to have different preferences from real humans, then the AI system should ignore the "corrupted" values of the real humans.

More generally, it seems important for our AI systems to help us figure out what we care about before we make drastic and irreversible changes to our environment, especially changes that prevent us from figuring out what we care about. For example, if we create a hedonic paradise where everyone is on side-effect-free recreational drugs all the time, it seems unlikely that we check whether this is actually what we wanted. This suggests that we need to work on AI systems that differentially advance our philosophical capabilities relative to other capabilities, such as technological ones.

One particular way that "aligned" AI systems could make things worse is if they accidentally "corrupt" our...
A simple approach to 5-and-10
by Gurkenglas — 2018-12-17T18:33:46.735Z

To recap, A is trying to deduce enough about U to maximize it:

A := Find some f with a proof of "U = f(A)", then return argmax f.

If U is A, A can fail by proving f = {(5,5),(10,0)}: If it could prove this, it would be true, for only 5 is checked. Then by Löb's theorem, it can prove this.

I've thrown this problem at some people in my university and a fellow student's idea led to:

A' := Find all f with a proof of "U = f(A')", then return argmax (their pointwise maximum).

The true f is among the found f. Fooling ourselves into a suboptimal action would require us to believe it at least as good as the true maximum, which the single check refutes.

Note that A' can't prove that some f is not found, due to Gödel. Therefore, it can never prove what it is going to do!
In Defense of Finance
by Jacob Falkovich — 2018-12-17T16:59:11.149Z
Link: https://putanumonit.com/2018/12/14/defense-of-finance/

I'm sharing the link instead of a full cross-post because this essay has:

5,300 words
2 footnotes, with links that actually work
1 GIF, 2 charts, 5 expanding brains
30+ lively comments

This is a "much more than you wanted to know" type post on the financial industry:

Why everyone hates it, except for people in positions of power.
The value of finance in coordinating trade across space and time, told via the parable of Banksy and the corn cobs.
Are mortgage-backed securities a scam or a brilliant innovation?
The stupid arguments about bailouts, and the smarter arguments for/against bailouts.
Economists writing books about bank equity ratios without googling what those equity ratios actually are.
What is up with equity ratios? Is making banks hold 50% capital the solution to financial crises?
What banking regulations actually make banks do.
The inextricable romance between finance and government.
A final request for incrementalism and humility.
Fifteen Things I Learned From Watching a Game of Secret Hitler
by Zvi — 2018-12-17T13:40:01.047Z

Epistemic Status: Not likely to be true things. Right?

Liberals know nothing, fascists know everything.

Most of the policies democratic governments could pass are fascist policies that expand government power.

The remaining policies are liberal policies. There is no such thing as a conservative policy.

Liberal policies do nothing.

If the liberals do nothing enough times, they win and can congratulate themselves, no matter how much more fascist things got in the meantime.

Governments must always be passing new policies, and never take away old policies. Thus, government inevitably gets more powerful over time.

The more liberal policies you pass, the more likely it is any future policy will be fascist.

The more fascist policies you pass, the more likely it is any future policy will be fascist.

When the time comes to pass a policy, the government will choose from whatever proposals are lying around, even if all of them are fascist and everyone choosing is a liberal. There is almost never an option to just not do that, as such bold action requires a mostly fascist policy already be in place.

If the government fails to agree to pass one of the things lying around, that’s even worse, because it will then choose a new policy completely at random from what is lying around, which will probably be fascist.

Liberals spend most of their time being paranoid over which people claiming to be liberals are secretly fascists, or even secretly actual literal Hitler, as opposed to attempting to write or choose good policies.

Someone enacting liberal policies, but not in a position to assume dictatorial power, is providing strong evidence they are probably secretly Hitler.

When good people often have no choice but to do bad things, but there is no way to verify this, the default is for no one who is good to have any idea who is good and who is bad.

Despite this, good people think they know who is good and who is bad.

Introducing a random element to the play is good for veteran players, because the ‘good guys’ are no longer able to (and thus forced to as in Resistance/Avalon) fall back purely on an announced, deterministic strategy, as the ‘bad guys’ could know the rules and game the system.
Two Neglected Problems in Human-AI Safety
by Wei Dai — 2018-12-16T22:13:29.196Z

In this post I describe a couple of human-AI safety problems in more detail. These helped motivate my proposed hybrid approach, and I think need to be addressed by other AI safety approaches that currently do not take them into account.

1. How to prevent "aligned" AIs from unintentionally corrupting human values?

We know that ML systems tend to have problems with adversarial examples and distributional shifts in general. There seems to be no reason not to expect that human value functions have similar problems, which even "aligned" AIs could trigger unless they are somehow designed not to. For example, such AIs could give humans so much power so quickly or put them in such novel situations that their moral development can't keep up, and their value systems no longer apply or give essentially random answers. AIs could give us new options that are irresistible to some parts of our motivational systems, like more powerful versions of video game and social media addiction. In the course of trying to figure out what we most want or like, they could in effect be searching for adversarial examples on our value functions. At our own request or in a sincere attempt to help us, they could generate philosophical or moral arguments that are wrong but extremely persuasive.

(Some of these issues, like the invention of new addictions and new technologies in general, would happen even without AI, but I think AIs would likely, by default, strongly exacerbate the problem by differentially accelerating such technologies faster than progress in understanding how to safely handle them.)

2. How to defend against intentional attempts by AIs to corrupt human values?

It looks like we may be headed towards a world of multiple AIs, some of which are either unaligned, or aligned to other owners or users. In such a world there's a strong incentive to use one's own AIs to manipulate other people's values in a direction that benefits oneself (even if the resulting loss to others are greater than gains to oneself).

There is an apparent asymmetry between attack and defense in this arena, because manipulating a human is a straightforward optimization problem with an objective that is easy to test/measure (just check if the target has accepted the values you're trying to instill, or has started doing things that are more beneficial to you), and hence relatively easy for AIs to learn how to do, but ...
Babble, Learning, and the Typical Mind Fallacy
by  — 2018-12-16T16:51:53.827Z
Link: http://an1lam.github.io/post/2018-12-16-babble-learning/

This is an essay I wrote originally just for myself about Babble and decided to post on my personal blog at a whim. The post is messy and digresses more than I'd like, but writing my thoughts down clarified my own thinking on the topic enough that I figured they might help someone else some day.
What are some concrete problems about logical counterfactuals?
by Chris_Leong — 2018-12-16T10:20:26.618Z

Logical counterfactuals are key to Functional Decision Theory and last I heard still an unsolved problem. Unfortunately, I am still rather confused about what exactly we are trying to solve. The only concrete problem I know of in this space is the 5-and-10 problem. But as far as I know, this is solved by writing programs that immediately cause a paradox if they ever discover their output. So presumably there are some unsolved concrete problems that relate to logical counterfactuals?

Edit: I should mention my post on the Cooperation Game as an example. Plus the further work section of this slideshow.
The E-Coli Test for AI Alignment
by johnswentworth — 2018-12-16T08:10:50.502Z

Let’s say you have an idea in mind for how to align an AI with human values.

Go prep a slide with some e-coli, put it under a microscope, and zoom in until you can see four or five cells. Your mission: satisfy the values of those particular e-coli. In particular, walk through whatever method you have in mind for AI alignment. You get to play the role of the AI; with your sophisticated brain, massive computing power, and large-scale resources, hopefully you can satisfy the values of a few simple e-coli cells.

Perhaps you say “this is simple, they just want to maximize reproduction rate.” Ah, but that’s not quite right. That’s optimizing for the goals of the process of evolution, not optimizing for the goals of the godshatter itself. The e-coli has some frozen-in values which have evolved to approximate evolutionary fitness maximization in some environments; your job is optimize for the frozen-in approximation, even in new environments. After all, we don’t want a strong AI optimizing for the reproductive fitness of humans - we want it optimizing for humans’ own values.

On the other hand, perhaps you say “these cells don’t have any consistent values, they’re just executing a few simple hardcoded algorithms.” Well, you know what else doesn’t have consistent values? Humans. Better be able to deal with that somehow.

Perhaps you say “these cells are too simple, they can’t learn/reflect/etc.” Well, chances are humans will have the same issue once the computational burden gets large enough.

This is the problem of AI alignment: we need to both define and optimize for the values of things with limited computational resources and inconsistent values. To see the problem from the AI’s point of view, look through a microscope.
on wellunderstoodness
by Quinn — 2018-12-16T07:22:19.250Z

epistemic status: nothing new, informal, may or may not provide a novel compression which gives at least one person more clarity. Also, not an alignment post, just uses arguments about alignment as a motivating example, but the higher-level takeaway is extremely similar to the Rocket Alignment dialogue and to the definition of "deconfusion" that MIRI offered recently (even though I'm making a very low-level point)

notation: please read a <- x as "a replaced with x", e.g. (fx) x<-5 => (mx+b) x<-5 => m5+b. (typically the sequents literature uses fx[5/x] but I find this much harder to read).

motivation

A younger version of me once said "What's the big deal with sorceror's apprentice / paperclip maximizer? you would obviously just give an ϵ and say that you want to be satisfied up to a threshold". But ultimately it isn't hard to believe that software developers don't want to be worrying about whether some hapless fool somewhere forgot to give the ϵ. Not to mention that we don't expect the need for such thresholds to be finitely enumerable. Not to mention the need to pick the right epsilons.

The aspiration of value alignment just tells you "we want to really understand goal description, we want to understand it so hard that we make epsilons (chasing after the perfectly calibrated thresholds) obsolete". Well, is it clear to everyone what "really understand" even means? It's definitely not obvious, and I don't think everyone's on the same page about it.

core claim

I want to focus on the following idea;

to be mathematically understood is to be reduced to substitution instances.

Consider; linear algebra. If I want to teach someone the meaning and the importance of ax+by+c=0 I only need to make sure (skipping over addition, multiplication, and equality) that they can distinguish between a scalar and a variable and then sculpt one form from another. Maybe it generalizes to arbitrary length aixi and to arbitrary number of equations as nicely as it does because the universe rolled us some dumb luck, it's unclear to me, but I know that every time I see a flat thing in the world I can say "grab the normal forms from the sacred texts and show me by substitution how this data is an instance of that". Yes, if I had coffee I'd derive it from something more primitive and less sacred, but I've been switching to tea.

Substitution isn't at the root of the "all the ques

...
Sabine "Bee" Hossenfelder (and Robin Hanson) on How to fix Academia with Prediction Markets
by Shmi — 2018-12-16T06:37:13.623Z
Link: http://backreaction.blogspot.com/2018/12/dont-ask-what-science-can-do-for-you.html

"Don’t ask what science can do for you." is the clever title.

As I understand it, the prediction markets in science would not be easily influenced by the scientists and they would take into account a group of people currently completely neglected in scientific evaluations: those who left academia dissatisfied. Not a new approach in the industry, but certainly not something that has been tried in a research-oriented environment, as far as I know.
New edition of "Rationality: From AI to Zombies"
by Rob Bensinger — 2018-12-15T21:33:56.713Z

MIRI is releasing a new edition of Rationality: From AI to Zombies, including the first set of R:AZ print books. As of this morning, print versions of Map and Territory (volume 1) and How to Actually Change Your Mind (volume 2) are now available on Amazon (1, 2), and we'll be rolling out the other four volumes of R:AZ over the coming months.

R:AZ is a book version of Eliezer Yudkowsky's original sequences, collecting a bit under half of his Overcoming Bias and LessWrong writing from November 2006 to September 2009. Map and Territory is the canonical place to start, but we've tried to make How to Actually Change Your Mind a good jumping-on point too, since some people might prefer to dive right into HACYM.

The price for the print books is $6.50 for Map and Territory, and $8 for How to Actually Change Your Mind. The new edition is also available electronically (in EPUB, MOBI, and PDF versions) on a pay-what-you-want basis: 1, 2. The HACYM ebook is currently available for preorders, and should be delivered in the next day.

The previous edition of R:AZ was a single sprawling 1800-page ebook. I announced at the time that we were also going to release a paper version divided into six more manageable chunks; but this ended up taking a lot longer than I expected, and involved more substantive revisions to the text.

Changes going into the new edition include:

The first sequence in Map and Territory, "Predictably Wrong," has been heavily revised, with a goal of making it a much better experience for new readers.
More generally, R:AZ is now more optimized for new readers, and less focused on extreme fidelity to the original blog posts, since this was one of the biggest requests from LessWrongers in response to the previous edition of R:AZ. This isn't a huge change, but it was an update about which option to pick in quite a few textual tradeoffs.
A bunch of posts have been added or removed. E.g., The Robbers Cave Experiment was removed because while it's still a cool and interesting study, the researchers' methods and motives have turned out to be pretty bad, and it isn't particularly essential to HACYM.
The "Against Doublethink" sequence in How to Actually Change Your Mind has been removed, to reduce HACYM's page count (and therefore its price and its potential to intimidate readers) and improve the book's focus.

...
Gwern about centaurs: there is no chance that any useful man+machine combination will work together for more than 10 years, as humans soon will be only a liability
by avturchin — 2018-12-15T21:32:55.180Z
Link: https://www.reddit.com/r/reinforcementlearning/comments/a3rbm3/alphazero_shedding_new_light_on_the_grand_games/

This is a quote from the discussion in Reddit (responding to another commenter):

"I remember and I kind of subscribe to this idea of "man+machines as a future of work". I think Tyler Cowen implanted this idea into my head"

Yes, Cowen was big on that too in stuff like Average is Over.

Very irresponsible of them to try to foster complacency like that. It should have been beyond obvious that there was no reason chess engines wouldn't keep improving and that at some point very quickly, far from representing a new stable paradigm and a reason to not worry about technological unemployment, the 'centaur' would be a net liability. As far as I can tell, in chess, the centaur era lasted barely a decade, and would've been shorter still had anyone been seriously researching computer chess rather than disbanding research after Deep Blue or the centaur tournaments kept running instead of stopping a while ago. In Go, it lasted a year at best (if we assume that world champs like Lee Sedol could spot 'delusions' like made it lose a game to Lee Sedol and contribute at all, but then by the Ke Jie tournament with Master, between Master's performance and the various match settings, it looked like humans were way far behind and liabilities when paired with Master, and even if we doubt that, Zero then came out and superseded Master entirely). Not very comforting precedents... The idea was nice but it doesn't work.
Argue Politics* With Your Best Friends
by sarahconstantin — 2018-12-15T19:00:00.549Z

Epistemic Status: I endorse this strongly but don’t think I’m being original or clever at all.

Until recently — yesterday, in fact — I was seriously wrong about something.

I thought that it was silly when I saw people spending lots of energy arguing with their closest friends who almost completely agreed with them, but not quite.

That’s some People’s Front Of Judaea shit, I thought. Don’t you know that guy you’re arguing with so vehemently is your friend? He likes you! He’s a pretty good guy! He even shares your values and models, almost completely! He’s only wrong about this one, itty bitty, relatively abstract thing!

Meanwhile, there are people out there in the world who don’t share your values. And there are people out there who are actually evil and do awful things.

It’s like “ok, saying mean things about Muslims can be bad, but being a Muslim terrorist is a hell of a lot worse! Why do the people who are so quick to penalize Islamophobic speech never have anything bad to say about actual mass murder? C’mon, get a sense of proportion!”

I still think, obviously, that really bad actions are worse than slightly bad actions.

But I was seriously misunderstanding why people argue with their close friends.

Have you noticed my mistake yet? Give it a moment.

. . .

. . .

. . .

Ok, here it is.

Arguing is not a punishment.

Again.

Arguing is not a punishment.

Sure, serious wrongdoing should be penalized, and socially disapproved of, more than mild wrongdoing. (Murder is worse than prejudiced speech.)

Also, fixing big problems should take priority over fixing little problems. (Saving money on rent is worth more of your attention than saving money on apples.)

But let’s frame it differently.

Cooperation is really valuable. Stable cooperation, that is; when even in the future, when you know each other better, and you’ve had more time to think, you’ll still want to cooperate.

Trust is really valuable, and scarce. Justified trust, that is; when you can rely on what somebody says to be true and base your decisions on information you get from them.

Having “true friends” — people you can cooperate with and trust, stably, to a high degree — is valuable.

Yeah, you can get along and even thrive in a low-trust environment if you have the right skills for it. Havamal, the medieval Icelandic wisdom literature, attributed to the god Odin, is my favorite advice for how to be a savvy customer in a...