trentmkelly/LessWrong-43k huggingface dataset:
Formal Metaethics and Metasemantics for AI Alignment
by June Ku — 2019-10-08T05:37:20.826Z

A Brief Introduction to MetaEthical.AI

tl;dr: AIXI for Friendliness

[Crossposted from my new blog.]

Abstract: We construct a fully technical ethical goal function for AI by directly tackling the philosophical problems of metaethics and mental content. To simplify our reduction of these philosophical challenges into “merely” engineering ones, we suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available.

Given such a model, the AI attributes beliefs and values to a brain in two stages. First, it identifies the syntax of a brain’s mental content by selecting a decision algorithm which is i) isomorphic to the brain’s causal processes and ii) best compresses its behavior while iii) maximizing charity. The semantics of that content then consists first in sense data that primitively refer to their own occurrence and then in logical and causal structural combinations of such content.

The resulting decision algorithm can capture how we decide what to do, but it can also identify the ethical factors that we seek to determine when we decide what to value or even how to decide. Unfolding the implications of those factors, we arrive at what we should do. All together, this allows us to imbue the AI with the necessary concepts to determine and do what we should program it to do.

See the open source code and commentary at www.metaethical.ai

Motivation

Researchers like Eliezer Yudkowsky and Nick Bostrom, among others, have argued for the urgent need to develop a rigorous framework for ensuring that smarter-than-human intelligence will be beneficial for humanity. I think of them and the community around their respective nonprofits, the Machine Intelligence Research Institute and the Oxford Future of Humanity Institute, as tending to share a cluster of views, including the following:

We should have a wide probability distribution over when human-level AI will be developed. If we go by expert predictions, most predict it will arrive within this century. A cautionary approach should prepare for even more rapid timelines.
Once AI that is at least as intelligent as its human creators has been developed, there is a positive feedback loop in which it can take over the task of improving its own intelligence, quickly resulting in a superintelligence vastly greater than that of humans.

...
AI Alignment Writing Day Roundup #2
by Ben Pace — 2019-10-07T23:36:36.307Z

Here are some of the posts from the AI Alignment Forum writing day. Due to the participants writing 34 posts in less than 24 hours (!), I'm re-airing them to let people have a proper chance to read (and comment) on them, in roughly chronological order.

1) Computational Model: Causal Diagrams with Symmetry by Johns Wentworth

This post is about representing logic, mathematics, and functions with causal models.

For our purposes, the central idea of embedded agency is to take these black-box systems which we call “agents”, and break open the black boxes to see what’s going on inside.

Causal DAGs with symmetry are how we do this for Turing-computable functions in general. They show the actual cause-and-effect process which computes the result; conceptually they represent the computation rather than a black-box function.

I'm new to a lot of this, but to me this seemed like a weird and surprising way to think about math (e.g. the notion that the input n is something that causes the next input n−1). Seems like a very interesting set of ideas to explore.

2) Towards a mechanistic understanding of corrigibility by Evan Hubinger

This post builds off of Paul Christiano's post on Worst Case Guarantees. That post claims:

Even if we are very careful about how we deploy ML, we may reach the point where a small number of correlated failures could quickly become catastrophic... I think the long-term safety of ML systems requires being able to rule out this kind of behavior, which I’ll call unacceptable, even for inputs which are extremely rare on the input distribution.

Paul then proposes a procedure built around adversarial search, where one part of the system searches for inputs that produce unacceptable outputs in the trained agent, and talks more about how one might build such a system.

Evan's post tries to make progress on finding a good notion of acceptable behaviour from an ML system. Paul's post offers two conditions about the ease of choosing an acceptable action (in particular, that it should not stop the agent achieving a high average reward and that is shouldn't make hard problems much harder), but Evan's conditions are about the ease of training an acceptable model. His two conditions are:

It must be not that hard for an amplified overseer to verify that a model is acceptable.
It must be not that hard to f

...
Bets and updating
by Eigil Rischel — 2019-10-07T23:06:18.778Z

Suppose the US presidential election is tomorrow.
You currently assign a probability of 50% to each outcome. (We are ignoring the small possibility that neither of the main party candidates will win).

A man approaches you, and offers you a bet of $10, at 2-1 odds. In other words, if candidate one wins, he pays you $20, if candidate two wins, you pay him $10.

Should you accept this bet? What if the bet was for $10000 instead?
Assume that your utility is linear in dollars (or assume that the bet is for utilons instead, whatever). If not, why not? Try to think about this before reading on.

The answer is that it depends on your priors - in particular, it depends on how you interpret the evidence of being offered the bet. In general, if someone offers you a large bet on some outcome, it's probably safe to assume they have access to a reasonable amount of information about the outcome. Depending on how much information your own probability estimate is based on, you should update towards the odds they offered you.

If you update too little, and accept too many bets, you will lose a lot of money to people with better information than you. On the other hand, you can also go too far in the other direction. If your response to being offered a five-cent bet is to immediately update to accept their probabilities (and refuse the bet), you will be very easy to fool (although hard to exploit by betting).

Now suppose there are two superintelligences, Omega and Omicron.
They are both excellent at modelling both you and the presidential election.
Omega has a strong preference for money, and a weak preference for having you believe false things about the presidential election.
Omicron has this swapped - it wants you to believe that actual outcome of the election (which it has predicted) is extremely unlikely, and has a weak preference for money.

Omega executes the following plan: It looks through a large number of possible bets, looking for ones that will give it a lot of money (according to its predicted outcome of the election). For each of them, it predicts whether you will, if offered, take the bet, or simply update your belief to be more accurate (if the bet is such that Omega wins money, it must be "in the correct direction"). It finds the best bet you will take (if any), and offers you this bet.

Omicron does a similar thing, but instead looks for bets that you won't take - be...
Occam's Razor May Be Sufficient to Infer the Preferences of Irrational Agents: A reply to Armstrong & Mindermann
by Daniel Kokotajlo — 2019-10-07T19:52:19.266Z

[Epistemic Status: My inside view feels confident, but I’ve only discussed this with one other person so far, so I won't be surprised if it turns out to be confused.]

Armstrong and Mindermann (A&M) argue

"that even with a reasonable simplicity prior/Occam’s razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple ‘normative’ assumptions, which cannot be deduced exclusively from observations."

I explain why I think their argument is faulty, concluding that maybe Occam's Razor is sufficient to do the job after all.

In what follows I assume the reader is familiar with the paper already or at least with the concepts within it.

Brief summary of A&M's argument:

(This is merely a brief sketch of A&M’s argument; I’ll engage with it in more detail below. For the full story, read

their paper.

)

Take a human policy pi = P(R) that we are trying to represent in the planner-reward formalism. R is the human’s reward function, which encodes their desires/preferences/values/goals. P() is the human’s planner function, which encodes how they take their experiences as input and try to choose outputs that achieve their reward. Pi, then, encodes the overall behavior of the human in question.

Step 1: In any reasonable language, for any plausible policy, you can construct “degenerate” planner-reward pairs that are almost as simple as the simplest possible way to generate the policy, yet yield high regret (i.e. have a reward component which is very different from the "true"/"Intended" one.)

Example: The planner deontologically follows the policy, despite a buddha-like empty utility function
Example: The planner greedily maximizes the reward function "obedience-to-the-policy."
Example: Double-negated version of example 2.

It’s easy to see that these examples, being constructed from the policy, are at most slightly more complex than the simplest possible way to generate the policy, since they could make use of that way.

Step 2: The "intended" planner-reward pair--the one that humans would judge to be a reasonable decomposition of the human policy in question--is likely to be significantly more complex than the simplest possible planner-reward pa...
The sentence structure of mathematics
by countedblessings — 2019-10-07T18:58:46.630Z

"Alice pushes Bob."

"Cat drinks milk."

"Comment hurts feelings."

These are all different sentences that describe wildly different things. People are very different from cats, and cats are very different from comments. Bob, milk, and feelings don't have much to do with each other. Pushing, drinking, and (emotionally) hurting are also really different things.

But I bet these sentences all feel really similar to you.

They should feel similar. They all have the same structure. Specifically, that structure is

Nounverbnoun.

Because these sentences all share the same fundamental underlying structure, they all feel quite similar even though they are very different on the surface. (The mathematical term for "fundamentally the same but different on the surface" is isomorphic.)

When you studied sentence structure back in grammar school (it wasn't just me, right?) you learned to break down sentences into their parts of speech. You learn that nouns are persons, places, or things, and verbs are the activities that nouns do. Adjectives describe nouns, and adverbs describe pretty much anything. Prepositions tell you where nouns go. Etc.

Parts of speech are really abstract and really general. When you look at the surface, the sentence

the ant crawls on the ground

and the sentence

the spaceship flies through space

could not possibly be more different. But when you look at the sentence structure, they're nearly identical.

The concept of "parts of speech" emerge when we notice certain general patterns arising in the way we speak. We notice that whether we're talking about ants or spaceships, we're always talking about things. And whether we're talking about crawling or flying, we're always talking about actions.

And so on for adjectives, adverbs, conjunctions, etc., which always seem to relate back to nouns and verbs—adjectives modify nouns, for example.

Next we simply give things and actions, descriptors and relational terms some confusing names to make sure the peons can't catch on—nouns and verbs, adjectives and prepositions—and we have a way of breaking down any English sentence into its fundamental parts.

That is to say, if you know the abstract rules governing sentence structure—the types of pieces and their connections—you can come up with structures that any English sentence is but a particular example of.

Like how "Alice pushes Bob" is but a particular example of "Noun verb noun."

At th...
[Link] Book Review: ‘The AI Does Not Hate You’ by Tom Chivers (Scott Aaronson)
by eigen — 2019-10-07T18:16:15.850Z

Scott Aaronson uploaded a review about The AI Does Not Hate You, a book by Tom Chivers.

The book:

This is a book about AI and AI risk. But it's also more importantly about a community of people who are trying to think rationally about intelligence, and the places that these thoughts are taking them, and what insight they can and can't give us about the future of the human race over the next few years.

The book has a dual purpose, it gives an account of the most important events that happened on the rationalist community while informing on the current state of the AI risk field. There's a Lesswrong discussion here.

This is to talk about Scott A. post. In the review he gives his opinions about the book, including his relationship with the rationalist community and his somewhat changing views on AI risk.

Reading Chivers’s book prompted me to reflect on my own relationship to the rationalist community.

The astounding progress in deep learning and reinforcement learning and GANs, which caused me (like everyone else, perhaps) to update in the direction of human-level AI in our lifetimes being an actual live possibility.
Welcome to Memphis LW/SSC Meetup [Edit With Your Details]
by SGCleveland — 2019-10-07T17:51:36.211Z

We are a small meetup that gets together to discuss rationalist and adjacent topics.

We've been meeting every first Saturday of the month at Crosstown Concourse for the past several months

We welcome anyone with interests across effective altruism, rationalism, or AI alignment. Most of us have read at least a couple posts of Slate Star Codex, but otherwise have had a few different paths to find this community.
[AN #67]: Creating environments in which to study inner alignment failures
by Rohin Shah — 2019-10-07T17:10:01.269Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Towards an empirical investigation of inner alignment

(Evan Hubinger) (summarized by Rohin): Last week, we saw that the worrying thing about

mesa optimizers

(

AN #58

) was that they could have

robust capabilities, but not robust alignment

(

AN#66

). This leads to an inner alignment failure: the agent will take competent, highly-optimized actions in pursuit of a goal that you didn't want.

This post proposes that we empirically investigate what kinds of mesa objective functions are likely to be learned, by trying to construct mesa optimizers. To do this, we need two ingredients: first, an environment in which there are many distinct proxies that lead to good behavior on the training environment, and second, an architecture that will actually learn a model that is itself performing search, so that it has robust capabilities. Then, the experiment is simple: train the model using deep RL, and investigate its behavior off distribution to distinguish between the various possible proxy reward functions it could have learned. (The next summary has an example.)

Some desirable properties:

- The proxies should not be identical on the training distribution.

- There shouldn't be too many reasonable proxies, since then it would be hard to identify which proxy was learned by the neural net.

- Proxies should differ on "interesting" properties, such as how hard the proxy is to compute from the model's observations, so that we can figure out how a particular property influences whether the proxy will be learned by the model.

Rohin's opinion: I'm very excited by this general line of research: in fact, I developed my own

proposal

along the same lines. As a result, I have a lot of opinions, many of which I wrote up in

this comment

, but I'll give a summary here.

I agree pretty strongly with the high level details (focusing on robust capabilities without robust alignment, identifying multiple proxies as the key issue, and focusing on environment design and architecture choice as the hard problems). I do differ in the details though. I'm more interes...
The Gears of Impact
by TurnTrout — 2019-10-07T14:44:51.212Z

​

​

Scheduling

: The remainder of the sequence will be released after some delay.

Exercise: Why does instrumental convergence happen? Would it be coherent to imagine a reality without it?

Notes

Here, our descriptive theory relies on our ability to have reasonable beliefs about what we'll do, and how things in the world will affect our later decision-making process. No one knows how to formalize that kind of reasoning, so I'm leaving it a black box: we somehow have these reasonable beliefs which are apparently used to calculate AU.
In technical terms, AU calculated with the "could" criterion would be closer to an optimal value function, while actual AU seems to be an on-policy prediction, whatever that means in the embedded context. Felt impact corresponds to TD error.

This is one major reason I'm disambiguating between AU and EU; in the non-embedded context. In reinforcement learning, AU is a very particular kind of EU: V∗(s), the expected return under the optimal policy.

Framed as a kind of EU, we plausibly use AU to make decisions.
I'm not claiming normatively that "embedded agentic" EU should be AU; I'm simply using "embedded agentic" as an adjective.
Hybrid Lottery Admission
by jefftk — 2019-10-07T13:20:01.496Z

Last year we organized a new contra dance weekend,

Beantown Stomp (retrospective). It sold
out, and some people were frustrated that they didn't hear about it
before we went to a waitlist. Since we're going to have a second one
(March 20-22nd, put it in your calendar!) several people have
suggested that we do a lottery this year. Looking at last years
applications, however, I'm not so sure:

We admitted people first-come first-served and went to waitlist at the
end of February for a weekend in late March. Everyone who applied
before the weekend did end up getting in off the waitlist for at least
part of it, though there were also people who would have liked to come
who didn't try registering because they knew the weekend was sold out.

This year we're going to allow more people to attend, 250 instead of
200: there was no time last year where the hall was too full, and my
worries about not being able to ventilate well enough were unfounded.
So there should be less pressure on tickets, with more of them to go
around.

So instead of jumping right in with a lottery, here's what I'm
thinking: when we open registration the first 125 applications are
accepted. Then if we get application #126 before registration has
been open for a month, we pause registration and hold a lottery for
the remaining 125 spots. This keeps the simplicity and immediacy of
first-come first-served, unless demand is high enough that we need a
lottery.

Thoughts? I don't know of other events that have done this: most have
stayed first-come first-served until they start seeing all their
tickets selling out right away, and then switched to a lottery. The
hybrid proposal here is a way to get most of the benefits of both when
you don't know which situation you're in yet.
Replace judges with Keynesian beauty contests?
by John_Maxwell — 2019-10-07T04:00:37.906Z

The job of the judicial system is to interpret the law. It's valuable for society that the law is interpreted consistently--for example, so we can make legal agreements and trust that judges will enforce those agreements in a way that's similar to how they enforced such agreements in the past. This is why legal precedents exist.

I think one possible problem facing democracy, at least here in the US, is that incentives for judges to follow precedents aren't strong enough. Because judges have so much flexibility in how they interpret the law, the appointment of judges has become a high-stakes political contest that our democracy wasn't designed to do robustly. Because judges have significant leeway in sentencing, their apparent capriciousness leads people to distrust the judicial system. Here's a proposal for fixing these problems and reducing noise in the judicial system--which could be useful, but is probably flawed and mainly just fun to think about.

A Keynesian beauty contest is a scenario John Maynard Keynes used to describe the behavior of financial markets. The idea is that a newspaper prints photographs of 100 faces and challenges readers to pick the 6 faces who will be picked most frequently by the readership as a whole. The interesting thing is that choosing the faces you personally find attractive isn't a good strategy. You're better off choosing faces which are attractive to the population at large--or, if you believe other newspaper readers are savvy, you might instead choose faces people think people think people are attracted to (and so on).

Let's suppose that the newspaper editor makes a mistake and happens to run the same 100 faces a second time. You'd be a fool not to pick the six winning faces from last time! Not only does the data show that those faces are likely to win--everyone else knows that the data shows that those faces are likely to win, and they're going to make their choices accordingly.

In other words, the Keynesian beauty contest is structured to incentivize predictability and conformism in one's judgments.

Now for fixing the judicial system.

Imagine if anyone who passed the bar was allowed to sign up to be a "Keynesian beauty contest judge" (KBCJ hereafter). Every month, they'd be sent some legal cases to make decisions on. Perhaps there'd be a transcript of some kind of court investigation to review, with identifying inf...
What do the baby eaters tell us about ethics?
by spookyuser — 2019-10-06T22:27:47.656Z

I just finished the baby eater sequence ( https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worlds-collide-0-8 ) and aside from it being an incredibly engaging story I feel like there is a deeper message about ethical systems I don't fully understand yet.

In the sequence introduction Eliezer says it makes points about "naturalistic metaethics" but I wonder what points are these specifically, since after reading the SEP page on moral naturalism https://plato.stanford.edu/entries/naturalism-moral/ I can't really figure out what the mind-independent moral facts are in the story.

Another thought I've had since I read the story is that it seems like a lot of human-human interactions are really human-babyeater interactions. If you're religious and talking to an atheist about God, both of you will look like baby eaters to the other. Likewise if you watch Fox News everyone on CNN or MSNBC will look like baby eaters but the same is true in reverse, everyone watching CNN will think Fox News are the baby eaters.

I have to say, this feels like some kind of ethical nihilism, but I would be curious to know if there are any canonical meta-ethical or ethical theories that correspond to the _message_ of the baby eater sequence, because if there is one, I think I agree with it.