trentmkelly/LessWrong-43k huggingface dataset:
Notes on notes on virtues
by David Gross — 2020-12-30T17:47:04.102Z

“The characteristic feature of all ethics is to consider human life as a game that can be won or lost and to teach man the means of winning.” ―Simone de Beauvoir[1]

Summary: The Notes on Virtues sequence shares my research into various virtues. This introductory post explains what I’m trying to accomplish.

I think the key to being a better, more satisfied, and more effective person is to become more skillful and well-rounded in practicing the virtues.[2] Alas, childhood training in the virtues is haphazard, and remedial training (or self-help) as adults is also spotty. I hope to help fix this by assembling information about the virtues, with a focus on practical ways we can improve in them.

Why I think this is important

“The branch of philosophy on which we are at present engaged differs from the others in not being a subject of merely intellectual interest—I mean we are not concerned to know what goodness essentially is, but how we are to become good people, for this alone gives the study its practical value.” ―Aristotle[3]

Life is complex. We are constantly confronted by a variety of challenges. To address them well, we need to have learned a variety of basic life skills such that they are second-nature to us. “The virtues” are skills that apply to challenges common to human lives.

If you have a better command of the virtues, this helps you thrive and improves your effect on those around you. Society at large benefits from a higher level of competence in the virtues of those in it. But our culture is not good at teaching or encouraging the virtues.

Our institutions of formal childhood education are patchy. You’ll get reading, writing, and ’rithmetic, but will you get resourcefulness, resilience, restraint, responsibility, rectification, or reputability? Institutions like scouting or religion pick up some of the slack, but not nearly enough. Parents have little guidance on how to convey virtue to their children effectively, and also have blind spots from their own spotty educations. There have been some gestures toward formal “character education” of children, which is probably a good sign. But children will learn most from the example of their elders: if we don’t value virtues enough to pursue them in our own lives, that will make more of an impression on the up-and-coming generation than any “do as we say, not as we do” education will.

A virtue gym?

For...
My unbundling of morality
by Rudi C — 2020-12-30T15:19:10.073Z

Inspired by seeing Morality as "Coordination", vs "Altruism".

Coordination: a) To do more win-win stuff. b) To band against some outgroup (win-win-lose).
Selfish genes: E.g., your genes make you care about your family directly, as they are carrying your genes.
Empathy: E.g., just thinking about how being tortured sucks so much makes someone want to stop others from being tortured.
Signaling and reputation: E.g., having a reputation for being fair can give you power and status. Inversely, having a reputation for dishonesty can close a lot of opportunities.
Insurance: Reducing variance of outcomes (e.g., bats sharing food).
Hardwired assumptions: E.g., incest leads to flawed children.
Fear of punishment: E.g., murder probably won't go unanswered.
Rewards of subserviency: E.g., respecting high-status people is not without benefits.
Power/status games: Enforcing norms can increase your own status and decrease others'.
Optimizing trade-offs for personal benefits: E.g., net-neutrality is good for middle-class people, bad for poor people. "Bravery debates" might fall under this umbrella as well.
Instinctual game-theoretic strategies: E.g., people like having more control and agency ("freedom"). Note that this is different from coordination (coordination is a subset of this); A lot of these strategies are win-lose.
Abdicating responsibility: E.g., the current fiasco of Covid vaccines. People prefer passive harm to risky interventions because any activity brings responsibility.

What more can you think of? (Of course, a lot of these have some overlap.)
I object (in theory)
by Sunny from QAD — 2020-12-30T12:50:56.420Z
Link: https://questionsanddaylight.com/?p=603

In middle school, I had a really good teacher for social studies. (I don’t think I realized how good she was at her job until much later.) On one occasion, she wanted to demonstrate to us the difference between capitalist ideology and socialist ideology.

Her demonstration came in two parts: first, she handed out pieces of candy corn in random quantities to all the students, and told us we could play games of rock-paper-scissors with one another to increase our lot. Every time you won a game against someone, you got to take one of their pieces of candy. Games could only be initiated with consent from both parties, if I remember correctly. We played games for maybe five or ten minutes, and much candy corn traded hands. This was meant to represent capitalism.

Then, she collected all the candy and handed it out once again, this time giving exactly two pieces to each student. There was no trading phase in this round, so we just sat there for a few seconds considering the even piles of candy in front of us. This was meant to represent socialism.

Finally, with the second round of candies still on our desks, she asked us for our thoughts about which system was better. My own opinion, if I remember right, was that I favored the socialist way of handing out candy corn. (This is not an endorsement of establishing socialism as the arbiter of other resource conflicts.) I don’t remember most of what my classmates said, but the general consensus was that the capitalist system was better, because in that system, we had a hand in our own fate. If we weren’t happy with the amount of candy we got, we could, by golly, get up and do something about that.

(Interestingly, they also described capitalism as being “more fair”. I’m pretty sure I raised my hand at that point and said that, regardless of which option you think is better, the latter was clearly more fair. But at least one person persisted and said that fairness means having agency. At the time I thought that was dumb, but now I think it’s just invoking sense two of the word “fair”, where it means not that everybody gets the same treatment but rather that the outcome each person gets depends predictably on their actions. This is not to be confused with the equality/equity distinction, which looks at the first sense of the word “fair” and asks whether “same treatment” means “same opportunity” or “same outcome”. But this is a t...
Review Voting Thread
by Ben Pace — 2020-12-30T03:23:06.075Z

We're half way through the second annual review, and 121 posts have been nominated

We've had more than double the number of individual nominations than last year, but on reviews we're still playing catchup. Last year we had 118 reviews, yet this year we've only had 51 so far.

When there's so many posts, it can be daunting to figure out which ones to review, so to help out, I'm making this thread. Every comment on this thread will be a post, and you should vote on which ones you would like to read a review of.

A review is something ideally that puts it in context of a broader conversation, describes its key contributions, its strengths and flaws, and where more work can be done.

(Or something else. Many people who write self-reviews often give a different flavor of review. And I've read many great short reviews, e.g. Jameson Quinn and Zvi last year did a lot of short reviews that communicated their impression of the post quite clearly.)

So I'm going to leave 122 comments on this post. 121 comments will just be a post title, and the other one will be for thread meta. (Search "Meta Thread".) I will remove my own votes from them, so they all start at zero.

Please vote on the comments to show how much you'd like to see reviews of different posts! Feel free to add a comment about what sort of review you'd like to see.

(Yes, I will probably get a lot of karma from this thread. Mwahaha you have fallen for my evil trap.)

(Also, my thanks to reviewers magfrump and Zvi with 5 each, johnswentworth with 6 reviews, and to fiddler with 10 (!), all thoughtful and valuable.)
Collider bias as a cognitive blindspot?
by TurnTrout — 2020-12-30T02:39:35.700Z

Zack M. Davis summarizes collider bias as follows:

The explaining-away effect (or, collider bias; or, Berkson's paradox) is a statistical phenomenon in which statistically independent causes with a common effect become anticorrelated when conditioning on the effect.

In the language of d-separation, if you have a causal graph X → Z ← Y, then conditioning on Z unblocks the path between X and Y.

... if you have a sore throat and cough, and aren't sure whether you have the flu or mono, you should be relieved to find out it's "just" a flu, because that decreases the probability that you have mono. You could be inflected with both the influenza and mononucleosis viruses, but if the flu is completely sufficient to explain your symptoms, there's no additional reason to expect mono.[1]

Wikipedia gives a further example:

Suppose Alex will only date a man if his niceness plus his handsomeness exceeds some threshold. Then nicer men do not have to be as handsome to qualify for Alex's dating pool. So, among the men that Alex dates, Alex may observe that the nicer ones are less handsome on average (and vice versa), even if these traits are uncorrelated in the general population. Note that this does not mean that men in the dating pool compare unfavorably with men in the population. On the contrary, Alex's selection criterion means that Alex has high standards. The average nice man that Alex dates is actually more handsome than the average man in the population (since even among nice men, the ugliest portion of the population is skipped). Berkson's negative correlation is an effect that arises within the dating pool: the rude men that Alex dates must have been even more handsome to qualify.

No crazy psychoanalysis, just a simple statistical artifact. (On a meta level, perhaps attractive people are meaner for some reason, but a priori, doesn't collider bias explain away the need for other explanations?)

This seems like it could be everywhere. Most things have more than one causal parent; if it has many parents, there's probably a pair which is independent. Then some degree of collider bias will occur for almost all probability distributions represented by the causal diagram, since collider bias will exist if P(¬A∧¬B)>0 (in the linked formalism). And if we don't notice it unless we make a serious effort to reason about the causal structure of a problem, then we ...
Desires as film posters
by KatjaGrace — 2020-12-30T00:30:16.884Z

Sometimes I like to think of desires as like film posters. You come across them, and they urge you to do something, and present it in a certain way, and induce some inclination to do it. But film posters are totally different from films. If you like a film poster, you don’t have to try to see the film. There is no metaphysical connection between the beauty of a film poster and the correctness of you seeing the film. It’s some evidence, but you have other evidence, and you get to choose. A film poster can be genuinely the most beautiful film poster you’ve ever seen, without the film being a worthwhile use of two hours. That’s largely an orthogonal question. If you put up the poster on your wall and look at it lovingly every day, and never see the film, that doesn’t need to be disappointing—it might be the best choice, and you might be satisfied in choosing it.
What Are Some Alternative Approaches to Understanding Agency/Intelligence?
by interstice — 2020-12-29T23:21:05.779Z

Apparently, MIRI has given up on their current mainline approach to understanding agency and are trying to figure out what to do next. It seems like it might be worthwhile to collect some alternative approaches to the problem -- after all, intelligence and agency feature in pretty much all areas of human thought and action, so the space of possible ways to make progress should be pretty vast. By no means is it exhausted by the mathematical analysis of thought experiments! What are people's best ideas?

(By 'understanding agency' I mean research that is attempting to establish a better understanding of how agency works, not alignment research in general. So IDA would not be considered agent foundations, since it takes ML capabilities as a black-box. )

ETA: I originally wrote 'agent foundations' in place of 'understanding agency' in the above, which was ambiguous between a broad sense of the term(any research aimed at obtaining a foundational understanding of agency) and a narrow sense(the set of research directions outlined in the agent foundations agenda document). See this comment by Rob re: MIRI's ongoing work on agent foundations(narrow sense).
Debate Minus Factored Cognition
by abramdemski — 2020-12-29T22:59:19.641Z

AI safety via debate has been, so far, associated with Factored Cognition. There are good reasons for this. For one thing, Factored Cognition gives us a potential gold standard for amplification -- what it means to give very, very good answers to questions. Namely, HCH. To the extent that we buy HCH as a gold standard, proving that debate approximates HCH in some sense would give us some assurances about what it is accomplishing.

I'm personally uncertain about HCH as a gold standard, and uncertain about debate as a way to approximate HCH. However, I think there is another argument in favor of debate. The aim of the present essay is to explicate that argument.

As a consequence of my argument, I'll propose an alternate system of payoffs for the debate game, which is not zero sum.

No Indescribable Hellworlds Hypothesis

Stuart Armstrong described the Siren Worlds problem, which is a variation of Goodhart's Law, in order to describe the dangers of over-optimizing imperfect human evaluations. This is a particularly severe version of Goodhart, in that we can assume that we have access to a perfect human model to evaluate options -- so in a loose sense we could say we have complete knowledge of human values. The problem is that a human (or a perfect model of a human) can't perfectly evaluate options, so the option which is judged best may still be terrible.

Stuart later articulated the No Indescribable Hellworld hypothesis, which asserts that there would always be a way to explain to the human (/human model) why an option was bad. Let's call this a "defeater" -- an explanation which defeats the proposal. This assumption implies that if we combine human (/human model) evaluation with some way of finding defeaters, we could safely optimize based on the resulting judgements -- at least, nothing could go too wrong. (We might only get a guarantee that we avoid sufficiently bad options, depending on the form of our "no indescribable hellworld" assumption.)

The hypothesis isn't clearly true or false. However, it does make some sense to conjecture that violations of our values should be explicable to us -- what else would it mean to violate "our values", after all?

Stuart himself mentions that the assumption implies "trustworthy debate" would avoid hellworlds. My goal is mostly to investigate this argument a bit further.

It turns out my argument here is also very similar to o...
AXRP Episode 3 - Negotiable Reinforcement Learning with Andrew Critch
by DanielFilan — 2020-12-29T20:45:23.435Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it's been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity's future potential.

In this episode, I talk with Andrew Critch about negotiable reinforcement learning: what happens when two people (or organizations, or what have you) who have different beliefs and preferences jointly build some agent that will take actions in the real world. In the paper we discuss, it's proven that the only way to make such an agent Pareto optimal - that is, have it not be the case that there's a different agent that both people would prefer to use instead - is to have it preferentially optimize the preferences of whoever's beliefs were more accurate. We discuss his motivations for working on the problem and what he thinks about it.

Daniel Filan:
Hello, everybody. Today, we're going to be talking to Andrew Critch. Andrew Critch got his Ph.D. in algebraic geometry at UC Berkeley. He's worked at Jane Street as an algorithmic trader, the Machine Intelligence Research Institute as a researcher. And he also co-founded the Center for Applied Rationality, where he was a curriculum designer. But currently he's a research scientist at UC Berkeley's Center for Human Compatible A.I. Today, the paper we're going to be talking about is Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making. The authors are Nishant Desai, Andrew Critch and Stuart Russell. Hello Andrew.

Andrew Critch:
Hi. Nice to be here.

Daniel Filan:
Nice to have you here. So I guess my first question about this paper is, what problem is it solving?

Andrew Critch:
Right. So when I quote, unquote solve a research problem, I'm usually trying to do two different things. One is that I'm answering an actual formally specifiable math question or maybe computer science question. The other thing is, I'm trying to draw attention to an area. So for me, the purpose of this paper was to draw attention to an area that I felt neglected. And so that's a meta research problem that it's trying to solve. And then the research problem that it's solving, or the object level proble...
AXRP Episode 2 - Learning Human Biases with Rohin Shah
by DanielFilan — 2020-12-29T20:43:28.190Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it's been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity's future potential.

One approach to creating useful AI systems is to watch humans doing a task, infer what they're trying to do, and then try to do that well. The simplest way to infer what the humans are trying to do is to assume there's one goal that they share, and that they're optimally achieving the goal. This has the problem that humans aren't actually optimal at achieving the goals they pursue. We could instead code in the exact way in which humans behave suboptimally, except that we don't know that either. In this episode, I talk with Rohin Shah about his paper about learning the ways in which humans are suboptimal at the same time as learning what goals they pursue: why it's hard, how he tried to do it, how well he did, and why it matters.

Daniel Filan:
Today, we have Rohin Shah. Rohin is a graduate student here at UC Berkeley's Center for Human Compatable A.I., or CHAI. He's co-authored quite a few different papers and he's soon to be a research scientist at DeepMind. Today, we'll be talking about his paper "On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference". This appeared at ICML 2019 and the co-authors were Noah Gundotra, Pieter Abbeel and Anca Dragan. Welcome to the show.

Rohin Shah:
Yeah, thanks for having me Daniel. I'm excited to be here.

Daniel Filan:
All right. So I guess my first question is, what's the point of this paper? Why did you write it?

Rohin Shah:
Yeah. So I think this was one of the first - this was the first piece of research that I did after joining CHAI. And at the time - I wouldn't necessarily, I just wouldn't agree with this now - but at the time, the motivation was, well, when we have a superintelligent system, it's going to look like an expected utility maximiser. So that determines everything about it except, you know, what utility function it's maximising. It seems like the thing we want to do is give it the right utility function. A natural way to do this is inverse reinforcement lear...
AXRP Episode 1 - Adversarial Policies with Adam Gleave
by DanielFilan — 2020-12-29T20:41:51.578Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it's been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity's future potential.

In this episode, Adam Gleave and I talk about adversarial policies. Basically, in current reinforcement learning, people train agents that act in some kind of environment, sometimes an environment that contains other agents. For instance, you might train agents that play sumo with each other, with the objective of making them generally good at sumo. Adam's research looks at the case where all you're trying to do is make an agent that defeats one specific other agents: how easy is it, and what happens? He discovers that often, you can do it pretty easily, and your agent can behave in a very silly-seeming way that nevertheless happens to exploit some 'bug' in the opponent. We talk about the experiments he ran, the results, and what they say about how we do reinforcement learning.

Daniel Filan:
Hello everybody, today I'll be speaking with Adam Gleave. Adam is a grad student at UC Berkeley. He works with the Center for Human Compatible AI, and he's advised by Professor Stuart Russell. Today, Adam and I are going to be talking about the paper he wrote, Adversarial policies: Attacking deep reinforcement learning. This was presented at ICLR 2020, and the co-authors are Michael Dennis, Cody Wild, Neel Kant, Sergey Levine and Stuart Russell. So, welcome Adam.

Adam Gleave:
Yeah, thanks for having me on the show Daniel.

Daniel Filan:
Okay, so I guess my first question is, could you summarize the paper? What did you do, what did you find?

Adam Gleave:
Sure. So, the basic premise of the paper is that we're really concerned about adversarial attacks in machine learning systems, and most adversarial attacks people have talked about have assumed this kind of Lp-norm threat model, where you take some existing input and you add a small amount of perturbation to that input, and then something like an image classifier drastically changes its classification accuracy. And probably people have seen examples of this where you add some white noise to a panda and thi...