trentmkelly/LessWrong-43k huggingface dataset:
[Link] Truth-telling is aggression in zero-sum frames (Jessica Taylor)
by ioannes — 2019-09-11T05:53:59.188Z

https://unstableontology.com/2019/09/10/truth-telling-is-aggression-in-zero-sum-frames/

Excerpt:

If we adopt a frame that says that unusual social plots are actions that are against someone (which is a zero-sum frame), this leads to the conclusion that truth-telling is aggression, as it is necessarily part of an unusual social plot.

Non-zero-sum frames, of course, usually interpret truth-telling positively: it contributes to a shared information commons, which helps just about everyone, with few exceptions. People are often capable of switching to non-zero-sum frames in natural emergency situations, but such situations are rare.

To transition from a zero-sum frame to a non-zero-sum frame, from normalized lying to normalized truth-telling, requires a special social plot involving unusual truth-telling. Because it almost never happens by default.
Rationality Exercises Prize of September 2019 ($1,000)
by Ben Pace — 2019-09-11T00:19:51.488Z

Added: Prizewinners announced in this comment below.

This post is an announcement of a prize for the best exercises submitted in the next two weeks on a topic of your choice, that are of interest to the LW community. We're planning to distribute $1,000, where $500 of that will go to the first place.

To submit some exercises, leave a comment here linking to your exercises by midnight at the end of Friday 20th September PDT (San Francisco time). You can PM one of us with it if you want to, but we'll be publishing all the entries that win a prize.

Why exercises?

I want to talk about why exercises are valuable, but my thinking is so downstream of reading the book Thinking Physics, that I'd rather just let its author (Lewis Carroll Epstein) speak instead. (All formatting is original.)

The best way to use this book is NOT to simply read it or study it, but to read a question and STOP. Even close the book. Even put it away and THINK about the question. Only after you have formed a reasoned opinion should you read the solution. Why torture yourself thinking? Why jog? Why do push-ups?

If you are given a hammer with which to drive nails at the age of three you may think to yourself, "OK, nice." But if you are given a hard rock with which to drive nails at the age of three, and at the age of four you are given a hammer, you think to yourself, "What a marvellous invention!" You see, you can't really appreciate the solution until you first appreciate the problem.

What are the problem of physics? How to calculate things? Yes - but much more. The most important problem in physics is perception, how to conjure mental images, how to separate the non-essentials from the essentials and get to the hear of a problem, HOW TO ASK YOURSELF QUESTION. Very often these questions have little to do with calculations and have simple yes or no answers: Does a heavy object dropped at the same time and from the same height as a light object strike the earth first? Does the observed speed of a moving object depend on the observer's speed? Does a particle exist or not? Does a fringe pattern exist or not? These qualitative questions are the most vital questions in physics.

You must guard against letting the quantitative superstructure of physics obscure its qualitative foundation. It has been said by more than one wise old physicist that you really unders

...
Relaxed adversarial training for inner alignment
by evhub — 2019-09-10T23:03:07.746Z

This post is part of research I did at OpenAI with mentoring and guidance from Paul Christiano. It also represents my current agenda regarding what I believe looks like the most promising approach for addressing inner alignment.

One particularly concerning failure mode for any sort of advanced AI system is for it to have good performance on the training distribution, but nevertheless still have some input on which it behaves highly “unacceptably” during deployment. For the purposes of this post, we will follow Paul Christiano in thinking about acceptability as some concept that satisfies the following two conditions:

“As long as the model always behaves acceptably, and achieves a high reward on average, we can be happy.”

“Requiring a model to always behave acceptably wouldn't make a hard problem too much harder.”

Thus, acceptability is a different condition than asking for good performance during deployment—an honest mistake is considered acceptable, as is just failing to do anything interesting at all. That being said, some of the sorts of things you might want from acceptability could include corrigibility, non-deception, or myopia.[1] The hope for this style of approach is that it is competitive to only apply advanced machine learning to those situations which have clear notions of acceptability.

The tension, however, rests in being able to achieve good performance while maintaining an acceptability guarantee. Thus, the ideal solution to this problem would be a way of converting a training procedure which can achieve arbitrary performance into a training procedure which is not much harder to run, and which can still achieve arbitrary performance, but which also enforces an acceptability guarantee.

One obvious way to attempt to maintain an acceptability guarantee is to construct a loss function which incentivizes acceptability as part of performance. Something of this form is almost certainly going to be a component of any safe system. That being said, there are some reasons why approaches of this form might be insufficient if used in isolation.

First, one important thing to note about folding acceptability into performance is that the resulting acceptability guarantee is fairly weak: you get an average-case acceptability guarantee that holds only in the case of no distributional shift, with no guarantee when your deployment distribution is different than yo...
[AN #63] How architecture search, meta learning, and environment design could lead to general intelligence
by Rohin Shah — 2019-09-10T19:10:01.174Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

(Jeff Clune) (summarized by Yuxi Liu and Rohin): Historically, the

bitter lesson

(

AN #49

) has been that approaches that leverage increasing computation for learning outperform ones that build in a lot of knowledge. The current ethos towards AGI seems to be that we will come up with a bunch of building blocks (e.g. convolutions, transformers, trust regions, GANs, active learning, curricula) that we will somehow manually combine into one complex powerful AI system. Rather than require this manual approach, we could instead apply learning once more, giving the paradigm of AI-generating algorithms, or AI-GA.

AI-GA has three pillars. The first is to learn architectures: this is analogous to a superpowered neural architecture search that can discover convolutions, recurrence and attention without any hardcoding. The second is to learn the learning algorithms, i.e. meta-learning. The third and most underexplored pillar is to learn to generate complex and diverse environments within which to train our agents. This is a natural extension of meta-learning: with meta-learning, you have to specify the distribution of tasks the agent should perform well on; AI-GA simply says to learn this distribution as well.

POET

(

AN #41

) is an example of recent work in this area.

A strong reason for optimism about the AI-GA paradigm is that it mimics the way that humans arose: natural selection was a very simple algorithm that with a lot of compute and a very complex and diverse environment was able to produce a general intelligence: us. Since it would need fewer building blocks (since it aims to learn everything), it could succeed faster than the manual approach, at least if the required amount of compute is not too high. It is also much more neglected than the "manual" approach.

However, there are safety concerns. Any powerful AI that comes from an AI-GA will be harder to understand, since it's produced by this vast computation where everything is learned, and so it would ...
Free-to-Play Games: Three Key Trade-Offs
by Zvi — 2019-09-10T12:10:00.440Z

Most free-to-play games, especially mobile free-to-play games, use similar models to extract revenue and keep players coming back. These models work by creating toxic trade-offs that are central to their functioning. This post highlights the main three, and will act as reference for future gaming posts. Note that some games that can be played for free, such as the excellent Path of Exile, mostly do not do these things and are not fundamentally ‘free-to-play’ games, rather they are games that are free. But they are the exception, the extreme end of a continuum from ‘so friendly we just give you this great experience in the hopes you’ll want to throw us money’ to ‘made a pact with the Canadian devil.’

For an analysis of Magic: The Gathering Arena in particular, this remains the central point of my analysis, centered on what I here call the third trade-off. Arena counts as free-to-play in bad ways, but is on the less toxic end of the spectrum, especially with regard to the second trade-off, which I am seeing over time as more central. However, while Arena remains relatively friendly, it is moving towards being more toxic, with things like the second wildcard for historic card creation and with the mastery track.

The Three Trade-offs

The first key trade-off is to make games less fun in order to cause you to pay them huge amounts of money to make them less not fun, aiming at extracting extremely large payments from a few whales as the primary revenue stream. This prevents them from giving you the version of their game that would be the most fun, even if you are paying a lot of money, because they still want you to pay more. If you are not paying any money, and aren’t going to bring in new players, they almost want to drive you away.

The second key trade-off is between success in the game and your life, focus and time. The games use timers and delays to force you to constantly pay attention to the game if you want to accomplish the maximum. Not doing so lets valuable resources sit idle, whether they are free resources or paid for resources. Thus, the back of your mind becomes infected by the need to constantly check and perform tasks, and there is a sense of panic that you are delaying starting something because that delays it finishing which delays starting the next thing, and so on. This is not healthy. The game is telling you when and how to play, rather than you...
Hackable Rewards as a Safety Valve?
by Davidmanheim — 2019-09-10T10:33:40.238Z

Reading Deepmind's latest research and accompanying blogpost, I wanted to highlight an under-appreciated aspect of safety. As a bit of background, Carlos Perez points out Josha Bach's "Lebowski theorem," which states that "no superintelligent AI is going to bother with a task that is harder than hacking its reward function." Given that, I see a potential perverse effect of some types of alignment research - especially research into embedded agency and robust alignment which makes AI uninterested in reward tampering. (Epistemic Status: my confidence in the argument is moderate, and I am more confident in the earlier claims.)

In general, unsafe AI is far more likely to tamper with its reward function than to find more distant (and arguably more problematic) ways to tamper with the world to maximize its objective. (epistemic status: fairly high confidence) Once an AI is smart enough to spend its time reward hacking, then wasting time on developing greater intelligence is unneeded. For that reason, this theorem seems likely to function as at least a mild safety valve. It's only if we close this valve too tightly that we would plausibly see ML that reached human-level intelligence. At that point, of course, we should expect that the AI will begin to munchkin the system, just as a moderately clever human would. And anti-munchkin-ing is a narrow instance of security more generally.

Security generally is like cryptography narrowly in an importance sense; it's easy to build a system that you yourself can't break, but very challenging to build one that others cannot exploit. (Epistemic status: more speculative) This means that even if our best efforts go towards safety, an AI seems very unlikely to need more than "mild" superintelligence to break it - unless it's been so well aligned that it doesn't want to hack its objective function.

This logic implies (Epistemic status: most speculative, still with some confidence) that moderate progress in AI safety is potentially far more dangerous than very little progress - and raises critical questions of how close to this unsafe uncanny valley we currently are, and how wide the valley is.
Counterfactual Oracles = online supervised learning with random selection of training episodes
by Wei Dai — 2019-09-10T08:29:08.143Z

Most people here probably already understand this by now, so this is more to prevent new people from getting confused about the point of Counterfactual Oracles (in the ML setting) because there's not a top-level post that explains it clearly at a conceptual level. Paul Christiano does have a blog post titled Counterfactual oversight vs. training data, which talks about the same thing as this post except that he uses the term "counterfactual oversight", which is just Counterfactual Oracles applied to human imitation (which he proposes to use to "oversee" some larger AI system). But the fact he doesn't mention "Counterfactual Oracle" makes it hard for people to find that post or see the connection between it and Counterfactual Oracles. And as long as I'm writing a new top level post, I might as well try to explain it with my own words. The second part of this post lists some remaining problems with oracles/predictors that are not solved by Counterfactual Oracles.

Without further ado, I think when the Counterfactual Oracle is translated into the ML setting, it has three essential characteristics:

Supervised training - This is safer than reinforcement learning because we don't have to worry about reward hacking (i.e., reward gaming and reward tampering), and it eliminates the problem of self-confirming predictions (which can be seen as a form of reward hacking). In other words, if the only thing that ever sees the Oracle's output during a training episode is an automated system that computes the Oracle's reward/loss, and that system is secure because it's just computing a simple distance metric (comparing the Oracle's output to the training label), then reward hacking and self-confirming predictions can't happen. (ETA: See this comment for how I've changed my mind about this.)

Independent labeling of data - This is usually taken for granted in supervised learning but perhaps should be explicitly emphasized here. To prevent self-confirming predictions, the labeling of data has to be done without causal influence from the Oracle. That is, the Oracle's output should be isolated in some protected system until it's time to compute the reward/loss so that it can't affect how the training label is generated. But note that it's perfectly fine for humans or other systems to look at the question/input that is given to the Oracle, in order generate the training label.

O

...
Is my result wrong? Maths vs intuition vs evolution in learning human preferences
by Stuart_Armstrong — 2019-09-10T00:46:25.356Z

The mathematical result is clear: you cannot deduce human preferences merely by observing human behaviour (even with simplicity priors).

Yet many people instinctively reject this result; even I found it initially counter-intuitive. And you can make a very strong argument that it's wrong. It would go something like this:

"I, a human H, can estimate what human K wants, just by observing their behaviour. And these estimations have evidence behind them: K will often agree that I've got their values right, and I can use this estimation to predict K's behaviour. Therefore, it seems I've done the impossible: go from behaviour to preferences."

Evolution and empathy modules

This is how I interpret what's going on here. Humans (roughly) have empathy modules E which allow them to estimate the preferences of other humans, and prediction modules P which use the outcome of E to predict their behaviour. Since evolution is colossally lazy, these modules don't vary much from person to person.

So, for hK a history of human K's behaviour in typical circumstances, the modules for two humans H and J will give similar answers:

EH(hK)≈EJ(hK).

Moreover, when humans turn their modules to their own behaviour, they get similar result. The human K will have a privileged access to their own deliberations; so define ˇhK as the internal history of K. Thus:

EH(hK)≈EJ(hK)≈EK(ˇhK).

This idea connects with partial preferences/partial models in the following way: EK(ˇhK) gives K access to their own internal models and preferences; so the approximately equal symbols above means that, by observing the behaviour of other humans, we have approximate access to their own internal models.

Then P just takes the results of E to predict future behaviour; since E and P have co-evolved, it's no surprise that P would have a good predictive record.

So, given E, it is true that a human can estimate the preferences of another human, and, given P, it is true they can use this knowledge to predict behaviour.

The problems

So, what are the problems here? There are three:

E and P only function well in typical situations. If we allow humans to self-modify arbitrarily or create strange other beings (such as AIs themselves, or merged human-AIs), then our empathy and predictions will start to fail[1].

It needs E and P to be given; but defining these for AIs is very tricky. Time and time again, we've found that tasks t

...
Shallow Review of Consistency in Statement Evaluation
by Elizabeth — 2019-09-09T23:21:40.303Z

Overview

Most existing forecasting or evaluation platform questions are for very clearly verifiable questions:

"Who will win the next election?"
"How many cars will Tesla sell in 2030?"
“How many jelly beans are in this jar?”

But many of the questions we care about are do not look like this. They might…

Be severely underspecified, e.g. “How much should we charge this customer for this vague feature request?”
Involve value judgements, e.g. “What is the optimum prison sentence for this convict?”, “How much does this plaintiff deserve for pain and suffering?”
Not have a clear stopping point, e.g. "What is the relative effectiveness of AI safety research vs. bio risk research?"
Require multiple steps instead of a yes/no or numerical answer, e.g. “What treatment is appropriate for this patient with precancerous cells?”
Not have good referrents, e.g. “What is the market size for this completely new tech?”

An entity who could answer these questions well would be a very valuable asset. But what does well even mean here? We want people to be accurate, of course, but in many cases we also need their predictions/evaluations to be consistent to be actionable. This is especially true when fairness norms are in play, such as pricing[1] and prison sentencing.

There is a lot of research showing that people make inconsistent evaluations (with each other and themselves across time) across a wide variety of fields, even those that more closely resemble the “easy” questions above (valuing stocks, appraising real estate, sentencing criminals, evaluating job performance, auditing financial statements)[2]. It is even more difficult to consistently evaluate or predict novel questions or low-frequency events, like “Will India use a nuclear weapon on Pakistan by 1/1/20” or “How much counterfactual value has this organization created?”.

This paper is a shallow review of the literature around how to get entities to make consistent judgements. I want to note up front that a major limitation of this write-up and of shallow reviews in general is that I mostly relied on authors’ descriptions of their work and conclusions, rather than verifying their experimental design and conclusions for myself, or looking up others’ opinions of papers. As such, this post should be taken as a description of the state of the literature, not the state of the world.

Speaking of underspecifie...
Simple and composite partial preferences
by Stuart_Armstrong — 2019-09-09T23:07:26.358Z

Partial preferences are preferences that exist within a human's internal model, eg "this painting is better than that one", "I don't like getting punched", "that was embarrassing", "I'd prefer to fund this charity than that one", and so on.

In order to elicit these partial preferences, we can use one-step hypotheticals: brief descriptions of a hypothetical situation where causing the human to model it and reach a preference.

But sometimes our reaction to a hypothetical are more complex. Consider:

During the Notre-Dame Cathedral fire, a relic said to be part of the True Cross was threatened by the fire. Ten firefighters could go in and save it; however, there is a 10% that all of them might die. Is it better that they go in or not?

When confronted with something like that, I might reason thusly:

Well, it's certainly not part of the True Cross, even if that existed. However, it is an important medieval artefact with probably a lot of history (I know that because it was in the most important cathedral in France). I value things like that quite highly, and don't want them to be destroyed*. On the other hand, beware status quo bias**: I don't particularly miss historical items that were destroyed in the past*; but this artefact would be appreciated by probably millions in the future, and that matters to me*. Lots of religious people, French people, and people with a respect for history would not want it destroyed, and I value their satisfaction to some extent*. A 10% chance of ten deaths should be considered similar, in my estimation, to 1 death with certainty** (however, without the connotations of "this person specifically must die"**). Firefighters are professionals, who specifically accept to take risks in these kinds of situations, so they're not like "innocent victims" who I would give extra weight to*. They are likely pretty young, and I care about the amount of life they could lose*.

So, would I pay one life for a very-but-not-fantastically valuable artefact that would be valued and enjoyed by millions? But those people would enjoy the cathedral and its art even without this particular relic. Estimating how much value such an artefact could create, compared with the value remaining in a human life (this is a relevant comparison for me**), I'd guess that it's not worth saving in this circumstance* (but I would change my mind if the risk of death were lower

...
Alternative name for a Bug List
by Bae's Theorem — 2019-09-09T15:14:20.618Z

At our Dojo, we performed an FMEA on all the various CFAR techniques we have gone over.
The biggest failure mode of the Bug List was "discounting the positive".
Additionally, if we shift our focus to a more positive mindset, it tends to be more motivating.

So basically, instead of being a list of problems in your life, it becomes a list of features that the Best Version of You has, as a list of things you should work on.
This also includes features you already share with this Best Version of You! That way, you can continue to cultivate these features and not lose them, while validating yourself.

I'm just trying to think of a good pithy name for this list. "Feature List" seems pretty obvious, but can you think of any better ones?