trentmkelly/LessWrong-43k huggingface dataset:
Corrigible but misaligned: 
 a superintelligent messiah
by zhukeepa — 2018-04-01T06:20:50.577Z

If we build an AGI, we'd really like it to be corrigible. Some ways Paul Christiano has described corrigibility: "[The AI should help me] figure out whether I built the right AI and correct any mistakes I made, remain informed about the AI’s behavior and avoid unpleasant surprises, make better decisions and clarify my preferences, acquire resources and remain in effective control of them, ensure that my AI systems continue to do all of these nice things..."

I don't think corrigibility is anything close to sufficient for alignment. I'll argue that "messianic" agents are corrigible, illustrate how a superintelligence could be messianic but catastrophically misaligned, and explore my intuitions about when corrigible superintelligences are actually aligned.

Messiahs are corrigible

If someone extraordinarily wise and charismatic—let's call him a messiah—comes into contact with a group of people, those people are likely to consider him to be corrigible. In his heart of hearts, the messiah would be trying to help them, and everyone would know that. He'd listen carefully to their criticisms of him, and make earnest efforts to improve accordingly. He'd be transparent about his intentions and visions of the future. He'd help them understand who they are and what they want, much better than they'd be able to themselves, and guide their lives in directions they consider to be genuinely superior. He'd protect them, and help them gain the resources they desire. He'd be an effortless leader—he'd never have to restrict anyone's actions, because they'd just wish so strongly to follow his word.

He might also think it's a good idea for his followers to all drink cyanide together, or murder some pregnant actresses, and his followers might happily comply.

I don't think a corrigible superintelligence would guide us down such an insidious path. I even think it would substantially improve the human condition, and would manage to avoid killing us all. But I think it might still lead us to astronomical moral waste.

A corrigible, catastrophically misaligned superintelligence

The world's in total chaos, and we're on the brink of self-annihilation. It's looking like we're doomed, but a ragtag team of hippie-philosopher-AI-researchers manages to build a corrigible AGI in the ni...
LW Update 3/31 - Post Highlights and Bug Fixes
by Raemon — 2018-04-01T04:01:01.326Z

In the last few weeks we implemented several bug fixes, including:

Login form gives more useful error messages
Fixed most broken post links from the Lesswrong migration. (There are still some issues surrounding linked comments, and old images)
Notifications don't appear twice if you reply to your own comment or someone replies to a comment of yours on a post that you created
Sequence navigation links at the bottom of the sequence post now properly keep you "in the sequence" (before, they'd send you to the next post, but then the next post wouldn't be in the sequence, just a standalone post). [There are still occasional issues with sequence links at the end of sequence, where the behavior is a bit more complicated because sometimes you're supposed to go to the next sequence and sometimes you're not]
Some backend fixes to improve the search functionality.

Usernames

Usernames on the frontpage are clickable again. (We disabled this awhile ago because it was leading to a lot of misclicks. We've changed the Post List Items quite a bit since then, and had heard some complaints about not being able to click on usernames to lead to a user's profile. We'll see how it goes this time)

Highlights

Whenever you publish or edit a post, it now generates a highlight based on the first 2000 characters. In the Recent Discussion section, posts that you haven't read yet will display the highlight so you can get some context before reading comments (the post will be marked with a green dot if you haven't read the post yet, which will disappear as soon as you've expanded the highlight once)

New posts will appear in the recent discussion section (to reduce a weird incentive to comment on your own post to get it onto the frontpage if it was a meta or personal blogpost. The recent discussion section will essentially serve as a chronological list of which posts most recently had some kind of activity)
Schelling Shifts During AI Self-Modification
by MikailKhan — 2018-04-01T01:58:10.835Z

Introduction

In this essay, I name and describe a mechanism that might break alignment in any system that features an originally aligned AI improving its capabilities through self-modification. I doubt that this idea is new, but I haven't yet seen it named and described in these terms. This post discusses this mechanism within the context of agents that iteratively improve themselves to surpass human intelligence, but the general idea should be applicable to most self-modification schemes.

This idea directly draws inspiration from Scott Alexander’s Schelling Fences on Slippery Slopes.

Schelling Shifts

A Schelling shift occurs when an agent fails to properly anticipate that modifying a parameter will increase its next version’s willingness to further modify the same parameter. This causes future iterations of the agent to modify the parameter further than the initial agent would be comfortable with.

Let A[t] refer to the state of agent A at the end of modification iteration t, and let X[t] refer to the value of parameter X at the end of modification iteration t.

For the clarity of this explanation, let’s say that X represents adherence to a certain constraint. The properly aligned initial agent A[0] predicts that any reduction to X will improve performance for a certain objective, and that alignment will be preserved as long as the reduction of X is of a sufficiently small magnitude. A[0] is not exactly sure how far it can safely reduce X, but is entirely confident that if X[1] is at least 95% of X[0], then A[1] will still adhere to the constraint with perfect consistency.

In this case, A[0] establishes a Schelling point at 95% of X[0], and is only comfortable reducing X up to this point, even though further reduction might lead to greater performance for certain objectives.

During the next modification iteration, A[1] once again predicts that reducing X will improve performance. Because X[1] < X[0], A[1] reasons about X differently than A[0], and decides that it is willing to reduce X even further. A[1] establishes a new Schelling Point at 95% of X[1], which is 90.25% of X[0]. After the next iteration, X[2] is lower than A[0] would have been comfortable with, introducing the possibility that A[2] is no longer properly aligned with A[0].

As a general rule, a Schelling shift occurs when A[t + c] and A[t] establish different Schelling points for modification to pa...
Reframing misaligned AGI's: well-intentioned non-neurotypical assistants
by zhukeepa — 2018-04-01T01:22:36.993Z

I think when people imagine misaligned AGI's, they tend to imagine a superintelligent agent optimizing for something other than human values (e.g. paperclips, or a generic reward signal), and mentally picture them as adversarial or malevolent. I think this visualization isn't as applicable for AGI's trained to optimize for human approval, like act-based agents, and I'd like to present one that is.

If you've ever employed someone or had a personal assistant, you might know that the following things are consistent:

The employee or assistant is genuinely trying their hardest to optimize for your values. They're trying to understand what you want as much as they can, asking you for help when things are unclear, not taking action until they feel like their understanding is adequate, etc.
They follow your instructions literally, under a sensible-to-them-seeming interpretation completely different from your own, and screw up the task entirely.

Suppose you were considering hiring a personal assistant, and you knew a few things about it:

Your assistant was raised in a culture completely different from your own.
Your assistant is extremely non-neurotypical. It doesn't have an innate sense of pain or empathy or love, it's a savant at abstract reasoning, and it learned everything it knows about the world (including human values) from Wikipedia.
Your assistant is in a position where it has access to enormous amounts of resources, and could easily fool you or overpower you if it decided to.

You might consider hiring this assistant and trying really, really hard to communicate to it exactly what you want. It seems like a way better idea to just not hire this assistant. Actually, you’d probably want to run for the hills if you were forced to. Some specific failure modes you might envision:

Your assistant's understanding of your values will be weird and off, perhaps in ways that are hard to communicate or even pin down.
Your assistant might reason in a way that looks convoluted and obviously wrong to you, while looking natural and obviously correct to it, leading it to happily take actions you'd consider catastrophic.

As an illustration of the above, imagine giving an eager, brilliant, extremely non-neurotypical friend free rein to help you find a romantic partner (e.g. helping you write your OKCupid profile and setting you up on ...
The Regularizing-Reducing Model
by RyenKrusinga — 2018-04-01T01:16:06.644Z
Link: https://drive.google.com/file/d/1bRUsAGBGS4w3suImAc5ikPT9GFI6LYTv/view?usp=sharing

The linked document is my (last-minute) submission for the AI alignment prize. Feedback is appreciated, even though it cannot change my submission entry.
Metaphilosophical competence can't be disentangled from alignment
by zhukeepa — 2018-04-01T00:38:11.533Z

Having human values is insufficient for alignment

Suppose there's a button where if you push it and name a human, that human becomes 1,000,000,000,000,000x more powerful. (What I mean by that isn't precisely specified—imagine some combination of being able to think much faster, becoming more intelligent, and having far more resources, to the point that they could easily overpower the rest of the world.)

Try running some thought experiments where you push the button to amplify:

Jesus
Buddha
Adolf Hitler
Donald Trump
Kim Jong-un
King Salman of Saudi Arabia
Ayn Rand
Elon Musk
Ray Kurzweil
Eliezer Yudkowsky
Yourself

My intuition is that some of these people are catastrophic to amplify, and some might be OK to amplify. It's interesting to me that amplifying some of these people might be catastrophic, given that they're fellow human beings, raised in human societies, born with human genomes, who almost certainly care about the future well-being of humanity.

One reason I’d feel queasy amplifying anyone is that they might fall into an epistemic pit, where they arrive at some critically wrong conclusion and take either huge or infinite amounts of time to update away from it. If someone’s reasoning process gets amplified, I wouldn’t generally trust them to be good at arriving at true beliefs—intelligence needn’t go hand-in-hand with rationality or philosophical competence.

In particular, it’s very unclear to me whether people would quickly update away from ideologies. In practice, humanity as a whole has not obviously fallen into any permanent epistemic pits, but I think this is because no single ideology has clearly dominated the world. If you have indefinite decisive power over the world, you have far less incentive to consider perspectives very different from your own, and unless you both care about and are good at seeking true beliefs, you wouldn’t do a good job learning from the people around you.

Another reason I’d feel queasy amplifying anyone is that they might take irreversible catastrophic actions (perhaps unknowingly). Genocides would be one example. Restructuring society such that it gets forever stuck in an epistemic pit would be another. Building a superintelligence without appreciating the risks is yet another (and clearly the most disastrous, and also the least obviously disastrous).

I consider these all failures in something I’ll term metaphilos...
Belief alignment
by hnowak — 2018-04-01T00:13:21.026Z

The following are some non-technical ideas about AI alignment based on human beliefs, rather than our true reward function. My impression is that the role of beliefs is often implied in passing, but I haven‘t found any elaborations on the topic. I‘d be grateful for relevant references, if someone knows any.

Do as I say, not as I do

I believe we should not aim to make AIs imitate our behaviour or even just our preferences in every regard. An AI with the same reward function as an average human would want what we want, e.g. to be in control of their own lives, to make lots of money, to eat icecream, etc. At best, having such human goals would make them inefficient, at worst dangerous, but mostly, it would make no sense. Why should a robot – or even worse, a disembodied AI – want to eat icecream? But most humans like icecream and delight in eating it sometimes and there is no logical reason why an AI could not want the same, even if it made no practical sense. An AI with a copy of a human reward function would also be undesirable in that the latter would be adapted to human biases, selfishness, laziness, and the general limitations of humans, which are not easily overcome (by humans) even with ample knowledge and time to think.

Still, we want the behaviour of an AI to match what we want, and simply spelling it out is unfeasible, since we are unable to take into account every potentially relevant variable in every possible situation. I believe a solution is to use human beliefs. These include all that we think we know about our own values and preferences and everything that we want an AI to do. They are also something that we can communicate, and an Artificial General Intelligence would by definition be capable of understanding us. Communicating all our beliefs would still be unfeasible, but an AGI should also be capable of reasoning about our beliefs, fill in the gaps, and in case of doubt ask us to clarify. Once it has a clear understanding of our moral beliefs, and provided it is intelligent enough, it should be able to surmise our theoretical viewpoint about the moral aspects of any situation as well as us.

Modelling the beliefs of a human might seem more far-fetched than learning their true reward function, as the latter can in theory be deduced from observing an agent‘s behaviour, whereas the former presumes to intrude into the black box of that is a human min...
A Sketch of Good Communication
by Ben Pace — 2018-03-31T22:48:59.652Z

"Often I compare my own Fermi estimates with those of other people, and that’s sort of cool, but what’s way more interesting is when they share what variables and models they used to get to the estimate."

– Oliver Habryka, at a model building workshop at FHI in 2016

One question that people in the AI x-risk community often ask is

"By what year do you assign a 50% probability of human-level AGI?"

We go back and forth with statements like "Well, I think you're not updating enough on AlphaGo Zero." "But did you know that person X has 50% in 30 years? You should weigh that heavily in your calculations."

However, 'timelines' is not the interesting question. The interesting parts are in the causal models behind the estimates. Some possibilities:

Do you have a story about how the brain in fact implements back-propagation, and thus whether current ML techniques have all the key insights?
Do you have a story about the reference class of human brains and monkey brains and evolution, that gives a forecast for how hard intelligence is and as such whether it’s achievable this century?
Do you have a story about the amount of resources flowing into the problem, that uses factors like 'Number of PhDs in ML handed out each year' and 'Amount of GPU available to the average PhD'?

Timelines is an area where many people discuss one variable all the time, where in fact the interesting disagreement is much deeper. Regardless of whether our 50% dates are close, when you and I have different models we will often recommend contradictory strategies for reducing x-risk.

For example, Eliezer Yudkowsky, Robin Hanson, and Nick Bostrom all have different timelines, but their models tell such different stories about what’s happening in the world that focusing on timelines instead of the broad differences in their overall pictures is a red herring.

(If in fact two very different models converge in many places, this is indeed evidence of them both capturing the same thing - and the more different the two models are, the more likely this factor is ‘truth’. But if two models significantly disagree on strategy and outcome yet hit the same 50% confidence date, and we should not count this as agreement.)

Let me sketch a general model of communication.

A Sketch

Step 1: You each have a different model that predicts a different probability for a certain event.

“I see your probability of reaching human level AGI in

...
Harry Potter and the Method of Entropy 1 [LessWrong version]
by habryka — 2018-03-31T20:38:45.125Z

[I copied over Alkjash's post, which linked to a PDF to LessWrong, to test our editor and generally show that it's not super hard to copy over things from existing LaTeX material. It took me about 8 minutes to convert the PDF to our formatting]

Introduction

This is the start of a sequence on the entropy method in combinatorics. As with any method in combinatorics, it's best explained by solving a series of problems. The best references on the subject are Chapter 14 of Alon and Spencer [1] and Radhakrishnan's expository paper [2].

Today, we solve three toy problems to motivate the definition of entropy.

1. Sorting

At his summer job at Ollivander's, Harry is given n wands of unknown weight w1,…,wn which he is to sort in increasing order. All Harry has to work with is a balance scale which determines the relative masses of any two wands wi and wj. How many weighings does Harry need?

Take a minute to think about this problem if you haven't seen it before.

1.1 Solution

In the beginning, Harry is agnostic about which of n! possible orderings is the true answer. In his mind there are n! possible worlds, one for each ordering of the wands.

After some number of weighings, suppose Harry sees a set S of possible worlds before he weighs wi and wj against each other. The answer lets him distinguish between the set of worlds S< in which wi<wj and the set of worlds S> in which wi>wj. Either way, he gets to eliminate one or the other.

But Harry may be unlucky and only eliminate the smaller of S< and S> at each step. He will be left with the larger of S< and S>, which is at least half the size of S. That is, each weighing decreases the total number of possible worlds by at most a factor of two.

Thus, he needs log2n! steps to halve the total number of possible worlds down to a unique one.

1.2 Exercises

1. Check that log2n!=Θ(nlogn), so this lower bound matches the best possible sorting algorithm runtime.

2. Check that being allowed to weigh any set of weights against any other set doesn't help - Harry still needs log2n! weighing operations.

3. What is the exact number of weighings needed to sort 5 wands?

4. The solution suggests that a good sorting algorithm balances the sizes of S< and S> as closely as possible at each step. How does your favorite sorting algorithm do in this regard?

2. Twenty Questio

... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Harry Potter and the Method of Entropy
by alkjash — 2018-03-31T20:10:00.448Z

I did some Aversion Factoring and found that my main aversion to writing math is the terrible LaTeX support on WordPress. For now, I’ll try writing things up offline and posting PDFs.

HPMoE is a sequence on the entropy method in combinatorics. It should be fun to think about for anyone interested in information theory and probability as well.

HPMoE1

Please post solutions to the exercises in ROT13.
Salience
by Tueskes — 2018-03-31T19:52:42.178Z

Epistemic Status: Speculative, a theory that works well for the experiences I've had. I'm interested in how it overlays onto other people's experiences.

When I come across a lesson or a theory about the world that I've run into before, particularly after a long period of time, I often experience a feeling of increased understanding; that THIS time I've grasped the concept and its relevance.

For example, there's a quote attributed to Plato (although apparently not his words) that goes "The first and greatest victory is to conquer yourself". Recently I've been noticing a lot of issues around self-discipline in myself and so have been quite conscious of things along this axis. It feels like now, when I read this, I'm truly getting the essence of it. But when I read it as a teenager I thought I did too, even though, now, it doesn't seem like I had as much of a handle on its significance then as I do today, and I'm sure every time I encounter it in the future it will feel like I've finally fully grokked the idea that the words are trying to convey.

This feeling used to scare me. If you could be exposed to the same idea and it still appears novel, then how can you make sure you're improving your understanding of the world and not just caught up flitting from concept A to concept B back to concept A like a philosophical dory? And it has implications for interpersonal relationships, too. A friend will tell us something, or try to teach us, and perhaps it will come across as a meaningless platitude, even though this is a person who we hold in high regard for their wisdom. Or you'll try to explain a thought you've had that seems so significant to you, but others won't be as enamored with it as you. How do you ever know to what degree you are talking about the same thing as someone else?

The root of all of this is simply that our brains are trying their best to model the world around us given the evidence they have so far. So when we hear Plato/not-Plato talking about conquering ourselves as a teenager, we just don't have the same breadth of life experience and self examination as we do today. The feeling of increased significance comes from a weightier evidence base supporting that idea. I've been calling this salience - the property of a thought being noticeable or striking - a...