trentmkelly/LessWrong-43k huggingface dataset:
Boeing 737 MAX MCAS as an agent corrigibility failure
by Shmi — 2019-03-16T01:46:44.455Z

The Boeing Maneuvering Characteristics Augmentation System (MCAS) can be thought of, if reaching a bit, as a specialized AI: it performs a function normally reserved for a human pilot: pitching the nose down when it deems the angle of attack to be dangerously high. This is not, by itself, a problem. There are pilots in the cockpit who can take control when needed.

Only in this case they couldn't. Simply manually pitching the nose up when the MCAS pitching it down too much would not disengage the system, it would activate again, and again. One has to manually disengage the autopilot (this information was not in the pilot training). For comparison, think of the cruise control system in a car: the moment you press the brake pedal, it disengages; if you push the gas pedal, then release, it return to the preset speed. At no time it tries to override your actions. Unlike MCAS.

MCAS disregards critical human input and even fights the human for control in order to reach its goal of "nominal flight parameters". From the Corrigibility paper:

We say that an agent is “corrigible” if it tolerates or assists many forms of outside correction, including atleast the following: (1) A corrigible reasoner must at least tolerate and preferably assist the programmers in their attempts to alter or turn off the system...

In this case the "agent" actively fought its human handlers instead of assisting them. Granted, the definition above is about programmers, not pilots, and the existing MCAS probably would not fight a software update, being a dumb specialized agent.

But we are not that far off: a lot of systems include built-in security checks for the remote updates. If one of those checks were to examine the algorithm the updated code uses and reject it when it deems it unacceptable because it fails its internal checks, the corrigibility failure would be complete! In a life-critical always-on system this would produce a mini-Skynet. I don't know whether something like that has happened yet, but I would not be surprised if it has, and resulted in catastrophic consequences.
Humans aren't agents - what then for value learning?
by Charlie Steiner — 2019-03-15T22:01:38.839Z

Humans aren't agents in the same way a thermostat isn't an agent.

A specimen of Temperatus moderatu humilis

Consider a truly humble thermostat. One temperature sensor, one set point, one output to turn the furnace on while the sensed temperature is below the set point. If we're being generous, we might construct an intentional stance model of the world in which this thermostat is an agent that wants the house to be at the set point.

But if the environment were a little bit different - say we block the vents leading to the upstairs - the thermostat doesn't try to unblock the vents. In the changed environment, it just acts as if it wants only the downstairs to be at the set point.

It is not that one of these is what the thermostat Really Wants and the other isn't. This entire model in which the thermostat wants things is a convenient fiction we're using to help think about the world. To ask what the thermostat Really Wants even after you know all the physical facts is an error, driven by the human tendency to mix up properties of our models with properties of objects.
You can fix this, and model the thermostat in a way that correctly predicts its behavior in more environments, but every time you make such an expansion of the space of environments, you make your model of the thermostat more concrete and less agenty. Eventually you end up with something like "It wants to increase the output signal when the input voltage is smaller than the voltage controlled by the dial on the front," at which point you might as well strip off the veneer about it "wanting" anything and predict it using physics.
This is what humans are like. In the ancestral environment, I would behave like someone who wants to eat fresh fruits and vegetables. Introduce Doritos to the environment, and I'll eat those instead. To expand the space of environments to include Doritos, you had to make your model of me more concrete (i.e. "Charlie wants to eat things that taste good"). If you pump heroin into my brain, I'll behave like someone who wants more heroin - which you can predict if you stop modeling me in terms of tastes and start modeling me in terms of anatomy and chemistry.

The model of me as someone who wants to eat fresh fruits and vegetables didn't fail because I have True Values and eating Doritos fulfills my True Values better than eating wild berries, but because the environment ...
Privacy
by Zvi — 2019-03-15T20:20:00.269Z

Follow-up to: Blackmail

[Note on Compass Rose response: This is not a response to the recent Compass Rose response, it was written before that, but with my post on Hacker News I need to get this out now. It has been edited in light of what was said. His first section is a new counter-argument against a particular point that I made – it is interesting, and I have a response but it is beyond scope here. It does not fall into either main category, because it is addressing a particular argument of mine rather than being a general argument for blackmail. The second counter-argument is a form of #1 below, combined with #2, #3 and #4 (they do tend to go together) so it is addressed somewhat below, especially the difference between ‘information tends to be good’ and ‘information chosen, engineered and shared so to be maximally harmful tends to be bad.’ My model and Ben’s of practical results also greatly differ. We intend to hash all this out in detail in conversations, and I hope to have a write-up at some point. Anyway, on to the post at hand.]

There are two main categories of objection to my explicit thesis that blackmail should remain illegal.

Today we will not address what I consider the more challenging category. Claims that while blackmail is bad, making it illegal does not improve matters. Mainly because we can’t or won’t enforce laws, so it is unclear what the point is. Or costs of enforcement exceed benefits.

The category I address here claims blackmail is good. We want more.

Key arguments in this category:

Information is good.*

Blackmail reveals bad behavior.

Blackmail provides incentive to uncover bad behavior.

Blackmail provides a disincentive to bad behavior.

Only bad, rich or elite people are vulnerable to blackmail.

We should strongly enforce all norms on everyone, without context dependence not explicitly written into the norm, and fix or discard any norms we don’t want to enforce in this way.

A key assumption is that blackmail mostly targets existing true bad behavior. I do not think this is true. For true or bad or for existing. For details, see the previous post.

Such arguments also centrally argue against privacy. Blackmail advocates often claim privacy is unnecessary or even toxic.

It’s one thing to give up on privacy in practice, for yourself, in the age of Facebook. I get that. It’s another to argue that privacy is bad. That it is bad to not revea...
Active Curiosity vs Open Curiosity
by Unreal — 2019-03-15T16:54:45.389Z

I think the word ‘curiosity’ is used to describe two distinct things that I will now differentiate as active curiosity and open curiosity.

Active curiosity is driven & purposeful. Like thirst, it seeks to be quenched.

When you see a blurry object among distant waves and it looks like it might be a humpback whale, and you want to know.

When you are asked a trivia question like, “How many people have seen the broadway show Hamilton more than once?” or “What’s the life expectancy of people my age in the US, in 2019?” And you find yourself wanting to go to Google.

When you watch a YouTube video of someone doing something crazy, and you’re like, How did they DO that?

When you hear someone mention your name from across the room, and you become anxious to know what they’re saying about you.

Active curiosity activates the part of your brain that anticipates a reward, and it can enhance learning, making it easier to remember surprising results. [1, 2]

//

There’s another kind of curiosity that is often referred to by therapy books and practitioners. It is phenomenologically different, and it seems good to be able to distinguish the two types.

This type of curiosity, which I’ll refer to as open curiosity, is best achieved when you feel safe, relaxed, and peaceful. In my experience, it basically requires parasympathetic nervous system activation.

I’m aware of at least one person who can’t recall experiencing this type of curiosity. So I don’t expect this to be a common or universal experience, but I think it’s achievable by all human minds in theory.

This type of curiosity isn’t very driven. It doesn’t need satisfaction or answers. It is open to any possibility and can look without judgment, evaluation, worry, or anxiety.

It is evoked by the Litany of Gendlin and the Litany of Tarski. It is related to original seeing / boggling / seeing with fresh eyes.

When I have open curiosity, I do have things I’m curious about! So it isn’t a totally passive experience. I often use open curiosity to get curious about myself or another person. It’s a very useful state for doing therapy-related work, as all emotions and thoughts feel acceptable and manageable, rather than overwhelming or undesirable.

Perhaps strangely, this type of curiosity is open to knowing, in addition to not knowing. It is open to understanding, in addition to not understanding. It doesn’t need to know or understand things...
IDA 5-8/14: Approval Directed Agents
by RAISE — 2019-03-14T23:58:12.079Z
Link: https://app.grasple.com/#/level/1519

Every Thursday for 4 weeks, we will be posting lessons about Iterated Distillation and Amplification. They're largely based on Paul Christiano's sequence here on LW. He graciously allowed us to use his work.

This is the second section, containing four lessons, which introduces Approval Directed Agents.

Note that access to the lessons requires creating an account here.

Have a nice day!
Risk of Mass Human Suffering / Extinction due to Climate Emergency 
by willfranks — 2019-03-14T18:32:51.096Z

The good folks at Extinction Rebellion are working crazy hard to promote the scientific case for impending societal and economic collapse due to climate change.

This is their talk presenting that case: Heading for Extinction and What to Do About It

I'm curious as to whether this (lesswrong /EA) community finds the case scientifically convincing, and further, whether their strategy for nonviolent direct action (i.e. taking to the streets and not moving until the government respond to demands) is the best plan of action (given the apparently mounting urgency of action on carbon emissions)?
Speculations on Duo Standard
by Zvi — 2019-03-14T14:30:00.343Z

Epistemic Status: Wild Mass Guessing

Tomorrow’s invitational tournament at PAX East will feature the new Duo Standard format. Each player will have two decks, play one in each of the first two games in a random order, then choose which deck they want for game three. There will be no sideboards.

Reading Wyatt Darby’s recent otherwise excellent article about preparing for the tournament, I noticed a lack of strategic thinking about the format. Wyatt was thinking well about best of one as opposed to best of three, in terms of lack of sideboards and the ban on Nexus of Fate. What he wasn’t thinking about at all, or was wisely keeping to himself, was the question of how choosing one of two decks for game three changes things.

We saw this once with the Player of the Year tiebreaker. The format there required the winner to win one game with each of their decks in a best of one format. Both players responded by selecting good best of one decks, but without thinking much about the broader structure. It was easy to understand this, as both players also had a Pro Tour to prepare for, so spending lots of time on playoff was hard to justify.

While we were watching the playoff, Kai Budde suggested a great idea. Players should submit four dedicated anti-creature decks, configured to crush decks like mono-white, mono-red and mono-blue. There are enough distinct configurations for this to be viable. Then, if your opponent has even one of these creature decks, that deck has to keep playing until it wins a game. Deck diversity is a disadvantage in this format! If you are diverse, the decks that are good against your opponents’ slate will win, leaving you with decks that are bad playing more games. You want your opponent doing that, not you.

Twin Standard instead lets you play two out of three games with the deck you prefer. You see what your opponents’ two decks are, then you both choose. You want your two decks to present different challenges.

What are the ways to respond to this?

Option 1: Two Good Different Decks

A reasonable first-level option is to select two decks with different match-ups. One could for example choose two of Esper/Jeskai Control, Sultai Midrange and Mono-White/Red/Blue Aggro. If you know what your opponents’ deck choice for game three is, you’ll always have at least a solid response. Note that if one of your decks is Sultai Midrange, this becomes far less true...
Combining individual preference utility functions
by Stuart_Armstrong — 2019-03-14T14:14:38.772Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

I've been working on a way of synthesising the preferences of a single given human.

After this is done, there is the business of combining these utilities into a single utility function for the whole human population. One obvious way of doing this is to weigh the different utilities by some individual measure of intensity, then add them together, and maximise the sum.

But there are some arguments against this. For example, a lot of people's preferences is over the behaviour or experiences of other people. This includes things like promoting religious or cultural beliefs (and this can cover preferences like wanting others to have basic human rights or to be free from oppressive situations).

Apart from simple summing, which may result in an excessive strength to a majority, there are two other obvious solutions: the first is to remove preferences over other people entirely. Note that this would also remove most preferences over social and cultural systems. The other one is to remove any anti-altruistic components to the system: preferences over the suffering of others are no longer valid (note this would allow punishment-as-a-deterrence, or punishment-as-retribution-to-a-victim, but not punishment-as-abstract-justice). This may be a bit tricky to define - what's the clear difference between wanting to be of higher status and wanting others to be of lower status? - but might be a desirable compromise. It might, indeed, be the kind of compromise that different people would "negotiate to" in some sort of "moral parliament" example, since people often tend to prefer their own individual desires over desires to control other people, so this might be a solution that would have majority support.
Mysteries, identity, and preferences over non-rewards
by Stuart_Armstrong — 2019-03-14T13:52:40.170Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

I've given examples of preferences over non-rewards.

But those examples are easy to dismiss as irrational or imposing on others, as they involved religion, or things like preferences over other people knowing the truth.

But if we think in terms of preferences over personal identity, we have far more widespread and natural examples of this.

One small example: in most murder-mysteries, I'd prefer not to know the identity of the killer at the beginning of the show. If I'm taking part in a murder mystery dinner, I'd definitely not want too. If I'm playing games of hidden information for fun, then I would not want to know all the hidden information - or else that makes this pointless. Lots of people enjoy watching sport events live; much less people re-watch random past sporting events whose result they already know. Eliezer talks about the joy in scientific discovery.

So preferences over non-rewards are quite natural and common.

Hedonism solution?

Now, you could say you don't have a "preferences over knowing/not knowing something", but that you "enjoy the experience of learning", and thus reduce the whole thing to hedonism. Eliezer argues against this, claiming to have preferences that don't reduce to hedonism (preferences I share, incidentally).

But even fully giving in to hedonism only partially solves the problem; yes, we can model someone's faith as saying they enjoy the experience of belief, and murder-mystery watchers enjoy the experience of seeing intricate puzzles solved. But this doesn't solve the problem that, to maximise hedonic experience, we still have to control what knowledge the human has.
Blackmailers are privateers in the war on hypocrisy
by Benquo — 2019-03-14T08:13:12.824Z
Link: http://benjaminrosshoffman.com/blackmailers-are-privateers-in-the-war-on-hypocrisy/

Allowing blackmail seems prima facie good to me, since it's a tax on covert illicit bejhavior. Zvi seems to think, to the contrary, that it's prima facie bad.
Robin Hanson argued: If there exists some information about someone that, if revealed, would cause people to coordinate to punish them, then it's good for this information to be revealed because on average it's good for such people to be punished. Blackmail rewards people for investigating covert illicit behavior that would otherwise remain undetected, and correspondingly punishes the people engaging in that behavior.
Zvi offered two interesting arguments against this, which I'll address one at a time.

The argument against leverage

First, Zvi responded that blackmail is obviously bad because it creates the incentive to pressure people into covertly behaving in ways that would get them in trouble if revealed, in order to to have leverage over them - which can then be used to force more covert bad behavior for yet more leverage. When done at scale, this can lead to both large amounts of bad behavior which would not have otherwise occurred, and large levels of extraction.
People can get away with subtle and indirect blackmail already. But so long as it's technically illegal, repeated leveraged blackmail at scale is not a feasible strategy; a large firm specializing in blackmail would quickly become unpopular, and the target of regulatory scrutiny.
This argument assumes a prior state where some agents already have enough leverage over most people to force them to engage in mildly illicit behavior. But any agent in a position to do that could easily use their leverage to force their victim to extract some sort of perfectly legal further leverage, such as borrowing at a high rate of interest, or making their lifestyle depend increasingly on some core bottleneck controlled by the extortionist. This too constrains the victim to do what the extortionist says, lest they default on their obligation and lose everything.<br/>This argument against blackmail is not specific to blackmail, but seems instead to be a general consideration against capitalism, privatization, and doing business at scale - since these empower positive feedback loops of rent-seeking behavior.

The argument against information

Zvi also offers an argument that is more purely targeted against blackmail. Even without leveraged schemes to man

...
AI Safety Prerequisites Course: Basic abstract representations of computation
by RAISE — 2019-03-13T19:38:56.301Z

Followup to AI Safety Prerequisites Course: Revamp and New Lessons. First post.

These are three new lessons of our online course on math formalizations required for AI safety research:

Level 10: Recursive Functions

Level 11: Set Theoretic Recursion

Level 12: The Equivalence of Different Notions of Computability

With these lessons, the student now should:

Understand the basic abstract representations of computation.

Know some of what we can expect from computers and also what we can’t expect from them.

Know a lot more set theoretic tools like equivalence relations and orderings.

Have seen the construction of the natural numbers from the perspective of set theory.

Know about mathematical induction, and have used it!

Know about recursion, and have used it!

If you study using our course, please give us feedback. Leave a comment here or email us at raise@aisafety.info, or through the contact form. Do you have an idea about what prerequisites are most important for AI Safety research? Do you know an optimal way to learn them? Tell us using the same methods or collaborate with us.

Can you check if a mathematical proof is correct? Do you know how to make proofs understandable and easy to remember? Would you like to help to create the prerequisites course? If yes, consider volunteering.
Question: MIRI Corrigbility Agenda
by algon33 — 2019-03-13T19:38:05.729Z

MIRI's reading list on corrigbility seems out dated, and I can't find a centralised list Does anyone have, or know of, one?

As a side note, has MIRI stopped updating their reading list? It seems like that's the case.

EDIT:

Links given in the comment section to do with corrigibility. I'll try and update this with some summaries as I read them.

https://www.greaterwrong.com/posts/5bd75cc58225bf0670375041/a-first-look-at-the-hard-problem-of-corrigibility

https://arbital.com/p/corrigibility/

https://arbital.com/p/updated_deference/

https://arxiv.org/pdf/1611.08219.pdf
A theory of human values
by Stuart_Armstrong — 2019-03-13T15:22:44.845Z

At the end of my post on needing a theory of human values, I stated that the three components of such a theory were:

A way of defining the basic preferences (and basic meta-preferences) of a given human, even if these are under-defined or situational.

A method for synthesising such basic preferences into a single utility function or similar object.

A guarantee we won't end up in a terrible place, due to noise or different choices in the two definitions above.

To summarise this post, I sketch out methods for 1. and 2., and look at what 3. might look like, and what we can expect from such a guarantee, and some of the issues with it.

Basic human preferences

For the first point, I'm defining a basic preference as existing within the mental models of a human.

Any preference judgement within that model - that some outcome was better than another, that some action was a mistake, that some behaviour was foolish, that someone is to be feared - is defined to be a basic preference.

Basic meta-preferences work in the same way, with meta-preferences just defined to be preferences over preferences (or over methods of synthesising preferences). Also include odd meta-preferences here - such as preferences over beliefs. I'll try to transform these odd preferences in "identity preferences": preferences over the kind of person you want to be.

"Reasonable" situations

To define that, we need to define the class of "reasonable" situations in which to have these mental models. These could be real situations (Mrs X thought that she'd like some sushi as she went past the restaurant) or counterfactual (if Mr Y had gone past that restaurant, he would have wanted sushi). The "one-step hypotheticals post" is about defining these reasonable situations.

Anything that occurs outside of a reasonable situation is discarded as not indicative of genuine basic human preference; this is due to the fact that humans can be persuaded to endorse/unendorse almost anything in the right situation (eg by drugs or brain surgery, if all else fails).

We can have preferences and meta-preferences over non-reasonable situations (what to do in a world where plants were conscious?), as long as these preferences and meta-preferences were expressed in reasonable situations. We can have a CEV style meta-preference ("I wish my preferences were more like what a CEV would generate"), but, apart from that, the preferen...