trentmkelly/LessWrong-43k huggingface dataset:
Alignment Newsletter #35
by Rohin Shah — 2018-12-04T01:10:01.209Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

This week we don't have any explicit highlights, but remember to treat the sequences as though they were highlighted!

Technical AI alignment

Iterated amplification sequence

Corrigibility (Paul Christiano): A corrigible agent is one which helps its operator, even with tasks that would change the agent itself, such as correcting mistakes in AI design. Consider a good act-based agent, which chooses actions according to our preferences over that action. Since we have a short-term preference for corrigibility, the act-based agent should be corrigible. For example, if we are trying to turn off the agent, the agent will turn off because that's what we would prefer -- it is easy to infer that the overseer would not prefer that agents stop the overseer from shutting them down. Typically we only believe that the agent would stop us from shutting it down if it makes long-term plans, in which case being operational is instrumentally useful, but with act-based agents the agent only optimizes for its overseer's short term preferences. One potential objection is that the notion of corrigibility is not easy to learn, but it seems not that hard to answer the question "Is the operator being misled", and in any case we can try this with simple systems, and the results should improve with more capable systems, since as you get smarter you are more capable of predicting the overseer.

In addition, even if an agent has a slightly wrong notion of the overseer's values, it seems like it will improve over time. It is not hard to infer that the overseer wants the agent to make its approximation of the overseer's values more accurate. So, as long as the agent has enough of the overseer's preferences to be corrigible, it will try to learn about the preferences it is wrong about and will become more and more aligned over time. In addition, any slight value drifts caused by eg. amplification will tend to be fixed over time, at least on average.

Rohin's opinion: I really like this formulation of corrigibility, which I find quite different from MIRI's paper. This seems a lot more in line with the kind of reasoning that I want from an AI system, and it seems like iterated amplificatio...
Peanut Butter
by Jacob Falkovich — 2018-12-03T19:30:56.907Z

This is part 2 of my interview with deluks917. In part 1 we discussed online weirdos, free thinkers, and Uyghurs. Stay tuned for part 3 on archetypes, the will to knowledge and Moana.

We talk about materialism, Buddhism, how to deal with the inevitable suffering that comes from attachment, and how peanut butter ties everything together. I have a lot of thoughts on all of the above!

The original post on Putanumonit has internal links you can use to jump from the interview to my thoughts and back. On LW you'll have to read through the interview and get to my thoughts at the bottom.

Jacob: Before I get your view on materialism, I want to share a story from last week.

It was my wife’s birthday, and she wanted to go to a fancy restaurant. A very fancy restaurant. We both wore nice outfits, and the meal cost more than everything we wore, including shoes.

At the start of the meal, we had large blue plates with golden trim in front of us. Then they put another round plate on top of that, and a smaller round plate, a square white plate, an even smaller round plate, and a tiny elongated plate on top of that. And on that pedestal of gleaming porcelain was an inch-long Italian pastry of some sort. Then they brought a cloth napkin wrapped around a slice of lemon and a twig of rosemary and poured water from a silver pitcher on the napkin so that it would be freshly wet.

I found the entire thing absurd, excess for the sake of excess. And I thought about you – as skeptical as I am of this sort of thing, you are far more skeptical. I remember coming over with a bottle of wine and asking if you had any wine glasses, and you said: “no, I’m not really into materialism” as if wine glasses were obviously superfluous.

deluks917: It seems that there is a path that almost everyone walks down. You get new things, and then you hedonically adapt. You start to feel like you can’t give up certain material comforts. Suddenly you’re willing to compromise your morals to get the money you feel you need.

People constantly want new cars, new clothes. Things they would’ve been very happy with in the past no longer make them happy. If I looked at my current life through the eyes of a subsistence farmer, shouldn’t I just be filled with joy at this material abundance?

It’s not just material things. You see people who are successful professors at good universities who are very upset because they won’...
Book Review - Other Minds: The Octopus, the Sea, and the Deep Origins of Consciousness
by Vaughn Papenhausen — 2018-12-03T08:00:00.000Z

In this post:

A true-false test about octopuses

What is it like to be an octopus?

An exercise in updating on surprising facts

Experiments related to animal suffering and consciousness

The evolution of aging

Should you read Other Minds: The Octopus, the Sea, and the Deep Origins of Consciousness?

I. Introduction

Peter Godfrey-Smith's Other Minds: the Octopus, the Sea, and the Deep Origins of Consciousness is a phenomenal mishmash of octopus- and consciousness-related topics. It deals with everything from the evolution of octopuses, to their social life, to animal consciousness (including octopus consciousness), to evolutionary theories of aging, and more. All of this is tied together by a palpable fascination with octopuses, which manifests itself in rich descriptions of Godfrey-Smith's own experiences scuba-diving off the coast of Australia to observe them.

The book attempts to fit discussion of an impressive amount of interesting topics all into one slim volume. On the one hand, this is great, as each topic is fascinating in its own right, and several are relevant to EA/rationality. On the other hand, fitting in so many topics is a difficult task which the book only halfway pulls off. There wasn’t enough room to discuss each topic in as much depth as they deserved, and the breadth of topics meant that the book felt somewhat unorganized and disunified. The book as a whole didn’t seem to have any central claim; it was simply a collection of interesting facts, observations, musings, and theories that somehow relate to either octopuses, consciousness, or both, plus a bunch of fascinating first-hand descriptions of octopus behavior.

Do I recommend the book? Yes and no. For general interest, definitely--it’s an interesting, enjoyable read; but for rationalists and EAs, there are probably better things to read on each topic the book discusses that would go into more depth, so it may not be the most effective investment of time for learning about, say, theories of animal consciousness. So in the rest of this review I’ve tried to 80:20 the book a bit, pulling out the insights that I found most interesting and relevant to EA/rationalism (as well as adding my own musings here and there). Because of this, the review is both quite long, and unavoidably reflects a bit of the disjointedness of the book itself--the sections can largely be read independently of each other.

...
Coherence arguments do not entail goal-directed behavior
by Rohin Shah — 2018-12-03T03:26:03.563Z

One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.

We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent, since if we could notice these failures, so could the agent. So we should expect that powerful agents appear coherent to us. (Note that it is possible that the agent doesn’t fix the failures because it would not be worth it -- in this case, the argument says that we will not be able to notice any exploitable failures.)

Taken together, these arguments suggest that we should model an agent much smarter than us as an expected utility (EU) maximizer. And many people agree that EU maximizers are dangerous. So does this mean we’re doomed? I don’t think so: it seems to me that the problems about EU maximizers that we’ve identified are actually about goal-directed behavior or explicit reward maximizers. The coherence theorems say nothing about whether an AI system must look like one of these categories. This suggests that we could try building an AI system that can be modeled as an EU maximizer, yet doesn’t fall into one of these two categories, and so doesn’t have all of the problems that we worry about.

Note that there are two different flavors of arguments that the AI systems we build will be goal-directed agents (which are dangerous if the goal is even slightly wrong):

Simply knowing that an agent is intelligent lets us infer that it is goal-directed. (EDIT: See these comments for more details on this argument.)
Humans are particularly likely to build goal-directed agents.

I will only be arguing against the first claim in this post, and will talk about the second claim in the next post.

All behavior can be rationalized as EU maximization

Suppose we have access to the entire policy of an agent, that is, given any universe-history, we know what action the agent will take. Can we tell whether the agent is an EU maximizer?

Actually, no matter what the policy is, we can view the agent ...
Benign model-free RL
by paulfchristiano — 2018-12-02T04:10:45.205Z

In my last post, I described three research areas in AI control that I see as central: reward learning, robustness, and deliberation.

In this post I argue that these three pieces may be sufficient to get a benign and competitive version of model-free reinforcement learning. I think this is an important intermediate goal of solving AI control.

This post doesn’t discuss benign model-based RL at all, which I think is another key obstacle for prosaic AI control.

(This post overlaps extensively with my post on ALBA, but I hope this one will be much clearer. Technically, ALBA is an implementation of the general strategy outlined in this post. I think the general strategy is much more important than that particular implementation.)

Ingredients

Reward learning and robustness

Given a benign agent H, reward learning allows us to construct a reward function r that can be used to train a weaker benign agent A. If our training process is robust, the resulting agent A will remain benign off of the training distribution (though it may be incompetent off of the training distribution).

Schematically, we can think of reward learning + robustness as a widget which takes a slow, benign process H and produces a fast, benign process A

A’s capabilities should be roughly the “intersection” of H’s capabilities and our RL algorithms’ competence. That is, A should be able to perform a task whenever both H can perform that task and our RL algorithms can learn to perform that task.

In these pictures, the vertical axis corresponds intuitively to “capability,” with higher agents being more capable. But in reality I’m thinking of the possible capabilities as forming a complete lattice. That is, a generic pair of levels of capabilities is incomparable, with neither strictly dominating the other.

Amplification

If we iteratively apply reward learning and robustness, we will obtain a sequence of weaker and weaker agents. To get anywhere, we need some mechanism that lets us produce a stronger agent.

The capability amplification problem is to start with a weak agent A and a human expert H, and to produce a significantly more capable agent Hᴬ. The more capable agent can take a lot longer to think, all we care about is that it eventually arrives at better decisions than A. The key challenge is ensuring that Hᴬ remains benign, i.e. that the system doesn’t acquire new preferences as it becomes more capable.

An e...
Anyone use the "read time" on Post Items?
by Raemon — 2018-12-01T23:16:23.249Z

Someone recently mentioned that they found the "read time" on home page post-items to be more of annoying clutter than a helpful indicator of whether to read something.

It occurred to me that I don't personally use them, and in general it's important for each UI element to be pulling it's weight. I'm curious how many people actively like/use them?

[Update: enough people have chimed in that I'm reasonably confident it's a good feature to have]
Intuitions about goal-directed behavior
by Rohin Shah — 2018-12-01T04:25:46.560Z

One broad argument for AI risk is the Misspecified Goal argument:

The Misspecified Goal Argument for AI Risk: Very intelligent AI systems will be able to make long-term plans in order to achieve their goals, and if their goals are even slightly misspecified then the AI system will become adversarial and work against us.

My main goal in this post is to make conceptual clarifications and suggest how they affect the Misspecified Goal argument, without making any recommendations about what we should actually do. Future posts will argue more directly for a particular position. As a result, I will not be considering other arguments for focusing on AI risk even though I find some of them more compelling.

I think of this as a concern about long-term goal-directed behavior. Unfortunately, it’s not clear how to categorize behavior as goal-directed vs. not. Intuitively, any agent that searches over actions and chooses the one that best achieves some measure of “goodness” is goal-directed (though there are exceptions, such as the agent that selects actions that begin with the letter “A”). (ETA: I also think that agents that show goal-directed behavior because they are looking at some other agent are not goal-directed themselves -- see this comment.) However, this is not a necessary condition: many humans are goal-directed, but there is no goal baked into the brain that they are using to choose actions.

This is related to the concept of optimization, though with intuitions around optimization we typically assume that we know the agent’s preference ordering, which I don’t want to assume here. (In fact, I don’t want to assume that the agent even has a preference ordering.)

One potential formalization is to say that goal-directed behavior is any behavior that can be modelled as maximizing expected utility for some utility function; in the next post I will argue that this does not properly capture the behaviors we are worried about. In this post I’ll give some intuitions about what “goal-directed behavior” means, and how these intuitions relate to the Misspecified Goal argument.

Generalization to novel circumstances

Consider two possible agents for playing some game, let’s say TicTacToe. The first agent looks at the state and the rules of the game, and uses the minimax algorithm to find the optima...
Taking vitamin D3 with K2 in the morning
by ChristianKl — 2018-11-30T19:00:59.563Z

Epistemological status: I studied bioinformatics but I'm no domain expert and layout my current view on the issue based on the facts I found in the literature. This is not health advice.

tl,dr:

There’s a strong evolutionary selection for producing enough Vitamin D. Vitamin D works together with Vitamin K2, and as a result, high Vitamin D supplementation should come with Vitamin K2 supplementation.

Personally, I decided to consume every morning 10 000 IU of Vitamin D3 and 200 µg of K2 (all-trans MK7) and believe that it’s likely beneficial for many fellow rationalists who don’t spend a lot of time exposing a lot of skin to the sun to take the same supplements.

The importance of Vitamin D

Gwern's

meta-analysis on Vitamin D

suggests that the existing academic publications suggest that this costs 0.33 years of life. Later in this article, I will make the case why I believe that the literature is likely biased to underrate the effect for systematic reasons.

Which Vitamin D should be supplemented?

Humans produce Vitamin D3. While Vitamin D2 that gets produced by mushrooms can also be used by humans, it makes sense to supplement Vitamin D3 as it’s the form of Vitamin D around which evolution optimized us given that it’s what’s available in the natural environment. Besides the evolutionary argument,

The case against ergocalciferol (vitamin D2) as a vitamin supplement

lists a few other reasons as well.

How much Vitamin D3 should be taken?

According to the

Evaluation, Treatment, and Prevention of Vitamin D Deficiency: an Endocrine Society Clinical Practice Guideline

:

For clinical care, it appears that all current methodologies are adequate if one targets a 25(OH)D value higher than current cut points; for example, a value of 40 ng/ml (100 nmol/L) is without toxicity and virtually ensures that the individual's “true” value is greater than 30 ng/ml (75 nmol/L). A clinical approach of targeting a higher 25(OH)D value seems prudent in that improving vitamin D status should reduce multiple adverse consequences of vitamin D deficiency at an extremely low cost with minimal toxicity risk.

In a

German study

they found median 25(OH)D levels of 19.8 ng/ml. While this isn't a meta-review it illustrates that plenty of people is strongly under the recommendation of the Endocrine Society. I'm German and I used to take 5000 IU vitamin D3 per day and got tested by my doctor and got a value of ...
Overconfident talking down, humble or hostile talking up
by ozziegooen — 2018-11-30T12:41:54.980Z

A Distribution of Knowlege

If one were to make a distribution of the amount of knowledge different people have about, say, macroeconomics, I would suspect the distribution to be somewhat lognormal; they would have tails to both ends, but be very skewed to the right. Most people have almost no knowledge of macroeconomics, some have a bit, then there is a long tail of fewer and fewer people who make up the experts.

The above graph doesn’t exactly resemble what I’d expect for macroeconomics but acts as a rough heuristic. The large numbers represent halvings of the remaining percentiles (3/4th, 7/8th, 15/16th, etc).[1]

I’m going to posit the following claims:[2]

Claim 1: It’s easy to judge where on the curve people are who are lower than you.

Claim 2: It’s difficult to judge where on the curve people are who are higher than you, absent of sophisticated systems to support this.

Given these, let’s imagine a few situations:

Say you’re the local economist for a state government. You have some actions you really would like the government to take, even though your colleagues wouldn’t typically approve. You’re around a +4 on the macroeconomic scale, and your most knowledgeable colleagues are around a +2. Could you get away with pretending that macroeconomics has a very confident stance that happens to align with what you want to see happen? How would you do so?
You’re a local radio intellectual who’s a +3 on the macroeconomic scale. Almost all of your listeners are below a +1.5. You’ve been essentially lying to them for some time about macroeconomic theory because it helps your political message. A professor who’s a +5 starts writing a few articles that call you out on your lies. How do you respond?
You’re a college student who’s a +3 on the macroeconomic scale. Your professors are all a good deal higher and will be the primary ones evaluating your studies. You want to legitimately learn macroeconomics. How do you treat your professors?

Overconfident talking down, humble or hostile talking up

I think the answers I’d expect from these questions can be summarized in the phrase “Overconfident talking down, humble or hostile talking up.”

When you’re communicating with people who know less than you, and you have little accountability from people who know more, then you generally have the option of claiming to be more knowledgeable than you are, and lying in ways that are useful to you....
Tentatively considering emotional stories (IFS and “getting into Self”)
by Kaj_Sotala — 2018-11-30T07:40:02.710Z

I’ve recently been getting a lot out of the psychotherapy model of Internal Family Systems, as described in this book. I just wrote a comment on Slate Star Codex describing some of its basics and what I’ve gotten out of it, and thought that I might as well repost it here:

I recommend this book, though with the note that I often don’t need to follow the full process outlined there. Sometimes it’s definitely necessary, but what I’ve found even more commonly useful is something that it discusses at the beginning of the book, which it calls “getting into self”.

Here’s the basic idea. Suppose that a part of your mind is really angry at someone, and telling a story (which might not be true) about how that person is a horrible person with no redeeming qualities. Internal Family Systems says that there are three modes in which you might react to that part:

First, you may be entirely blended with it (for those familiar, this corresponds to what Acceptance and Commitment Therapy calls cognitive fusion). This means that you are experiencing everything in terms of the story that it is telling, and have forgotten that this is an emotional reaction. So you feel that it’s just objectively true that the other person is horrible and with no redeeming qualities.

Or you might be partially blended with it. In this case, you realize that you are experiencing an emotional reaction, and that your thoughts and feelings might not be entirely justified, but you still feel them and might not be able to stop yourself from behaving according to them anyway.

Finally, you might be “in Self”, meaning entirely unblended. Here you are still aware of the emotions and thoughts, but your subjective experience is that they’re not your emotions, they’re someone else’s – they’re coming from a part of your mind which is experienced as separate from “you”. In this mode, you do not feel threatened or overwhelmed by them, and you can maintain a state of open curiosity towards whether or not they are actually true.

My experience is that usually if I have an unpleasant emotion, I will try to do one of two things: either reject it entirely and push it out of my mind, or buy into the story that it’s telling and act accordingly. Once I learned the techniques for getting into Self, I got the ability to sort of… just hang out with the emotion, neither believing it to be absolutely true nor needing to show it to ...