trentmkelly/LessWrong-43k huggingface dataset:
The Inner Alignment Problem
by evhub — 2019-06-04T01:20:35.538Z

This is the third of five posts in the Risks from Learned Optimization Sequence based on the paper “Risks from Learned Optimization in Advanced Machine Learning Systems” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Each post in the sequence corresponds to a different section of the paper.

In this post, we outline reasons to think that a mesa-optimizer may not optimize the same objective function as its base optimizer. Machine learning practitioners have direct control over the base objective function—either by specifying the loss function directly or training a model for it—but cannot directly specify the mesa-objective developed by a mesa-optimizer. We refer to this problem of aligning mesa-optimizers with the base objective as the inner alignment problem. This is distinct from the outer alignment problem, which is the traditional problem of ensuring that the base objective captures the intended goal of the programmers.

Current machine learning methods select learned algorithms by empirically evaluating their performance on a set of training data according to the base objective function. Thus, ML base optimizers select mesa-optimizers according to the output they produce rather than directly selecting for a particular mesa-objective. Moreover, the selected mesa-optimizer's policy only has to perform well (as scored by the base objective) on the training data. If we adopt the assumption that the mesa-optimizer computes an optimal policy given its objective function, then we can summarize the relationship between the base and mesa- objectives as follows:(17)
θ∗=argmaxθ E(Obase(πθ)), whereπθ=argmaxπ E(Omesa(π|θ))
That is, the base optimizer maximizes its objective Obase by choosing a mesa-optimizer with parameterization θ based on the mesa-optimizer's policy πθ, but not based on the objective function Omesa that the mesa-optimizer uses to compute this policy. Depending on the base optimizer, we will think of Obase as the negative of the loss, the future discounted reward, or simply some fitness function by which learned algorithms are being selected.

An interesting approach to analyzing this connection is presented in Ibarz et al, where empirical samples of the true reward and a learned reward on the same trajectories are used to create a scatter-plot visualization of the alignment between the two.

...
Our plan for 2019-2020: consulting for AI Safety education
by RAISE — 2019-06-03T16:51:27.534Z

UPDATE: this plan received sizable criticism. We are reflecting on it, and working on a revision.

Tl;dr: a conversation with a grantmaker made us drop our long-held assumption that outputs needed to be concrete to be recognized. We decided to take a step back and approach the improvement of the AI Safety pipeline on a more abstract level, doing consulting and research to develop expertise in the area. This will be our focus in the next year.

Trial results

We have tested our course in April. We didn’t get a positive result. It looks like this was due to bad test design, with a high variance and low number of participants clouding any pattern that could have emerged. In hindsight, we should clearly have tested knowledge before the intervention as well as after it, though arguably this would have been nearly impossible given the one-month deadline that our funder imposed.

What we did learn is that we are greatly unaware of the extent to which our course is being used. This is mostly due to using software that is not yet mature enough to give this kind of data. If we want to continue building the course, we feel that our first priority ought to be to set up a feedback mechanism that gives us precise insights into how students are journeying through.
However, other developments have pointed our attention away from developing the course, and towards developing the question that the course is an answer to.

If funding wasn’t a problem

During the existence of RAISE, it’s runway has never been longer than about 2 months. This did cripple our ability to make long term decisions, in favor of dishing out some quick results to show value. Seen from a "quick feedback loops" paradigm, this may have been a healthy dynamic. It did also lead to sacrifices that we didn’t actually want to make.

Had we been tasked with our particular niche without any funding constraints, our first move would have been to do extensive study into what the field needs. We feel that EA is missing a management layer. There is a lot that a community-focused management consultant could do, simply by connecting all the dots and coordinating the many projects and initiatives that exist in the LTF space. We have identified 30 (!) small and large organisations that are involved in AI Safety. Not all of them are talking to each other, or even aware of each other.

Our niche being AI Safety education, we wo...
To first order, moral realism and moral anti-realism are the same thing
by Stuart_Armstrong — 2019-06-03T15:04:56.363Z

I've taken a somewhat caricaturist view of moral realism[1], describing it, essentially, as the random walk of a process defined by its "stopping" properties.

In this view, people start improving their morality according to certain criteria (self-consistency, simplicity, what they would believe if they were smarter, etc...) and continue on this until the criteria are finally met. Because there is no way of knowing how "far" this process can continue until the criteria are met, this can drift very far indeed from its starting point.

Now I would like to be able to argue, from a very anti-realist perspective, that:

Argument A: I want to be able to judge that morality α is better than morality β, based on some personal intuition or judgement of correctness. I want to be able to judge that β is alien and evil, even if it is fully self-consistent according to formal criteria, while α is not fully self-consistent.

Moral realists look like moral anti-realists

Now, I maintain that this "random walk to stopping point" is an accurate description of many (most?) moral realist systems. But it's a terrible description of moral realists. In practice, most moral realists allow for the possibility of moral uncertainty, and hence that their preference approach might have a small chance of being wrong.

And how would they identify that wrongness? By looking outside the formal process, and checking if the path that the moral "self-improvement" is taking is plausible, and doesn't lead to obviously terrible outcomes.

So, to pick one example from Wei Dai (similar examples can be found in this post on self-deception, and in the "Senator Cruz" section of Scott Alexander's "debate questions" post):

I’m envisioning that in the future there will also be systems where you can input any conclusion that you want to argue (including moral conclusions) and the target audience, and the system will give you the most convincing arguments for it. At that point people won’t be able to participate in any online (or offline for that matter) discussions without risking their object-level values being hijacked.

If the moral realist approach included getting into conversations with such systems and thus getting randomly subverted, then the moral realists I know would agree that the approach had failed, no matter how internally consistent it seems. Thus, they allow, in practice, some considerations aki

...
Conditional meta-preferences
by Stuart_Armstrong — 2019-06-03T14:09:54.357Z

I'd just want to make the brief point that many human meta-preferences are conditional.

Sure, we have "I'd want to be more generous", or "I'd want my preferences to be more consistent". But there are many variations of "I'd want to believe in a philosophical position if someone brings me a very convincing argument for it" and, to various degrees of implicitness or explicitness, "I'd want to stop believing in cause X if implementing it leads to disasters".

Some are a mix of conditional and anti-conditional: "I'd want to believe in X even if there was strong evidence against it, but if most of my social group turns against X, then I would want to too".

The reason for this stub of a post is that when I think of meta-preferences, I generally think of them as conditional; yet I've read some comments by people that imply that they think that I think of meta-preferences in an un-conditional way[1]. So I made this post to have a brief reference point.

Indeed, in a sense, every attempt to come up with normative assumptions to bridge the is-ought gap in value learning, is an attempt to explicitly define the conditional dependence of preferences upon the facts of the physical world.

Defining meta-preferences that way is not a problem, and bringing the definition into the statement of the meta-preference is not a problem either. In many cases, whether we label something conditional or non-conditional is a matter of taste, or whether we'd done the updating ahead of time or not. Contrast "I love chocolate", with "I love delicious things" with the observation "I find chocolate delicious", with "conditional on it being delicious, I would love chocolate" (and "I find chocolate delicious").

This sentence does actually make sense. ↩︎
Does Bayes Beat Goodhart?
by abramdemski — 2019-06-03T02:31:23.417Z

Stuart Armstrong has claimed to beat Goodhart with Bayesian uncertainty -- rather than assuming some particular objective function (which you try to make as correct as possible), you represent some uncertainty. A similar claim was made in The Optimizer's Curse and How to Beat It, the essay which introduced a lot of us to ... well, not Goodhart's Law itself (the post doesn't make mention of Goodhart), but, that kind of failure. I myself claimed that Bayes beats regressional Goodhart, in Robust Delegation:

I now think this isn't true -- Bayes' Law doesn't beat Goodhart fully. It doesn't even beat regressional Goodhart fully. (I'll probably edit Robust Delegation to change the claim at some point.)

(Stuart makes some more detailed claims about AI and the nearest-unblocked-strategy problem which aren't exactly claims about Goodhart, at least according to him. I don't fully understand Stuart's perspective, and don't claim to directly address it here. I am mostly only addressing the question of the title of my post: does Bayes beat Goodhart?)

If approximate solutions are concerning, why would mixtures of them be unconcerning?

My first argument is a loose intuition: Goodhartian phenomena suggest that somewhat-correct-but-not-quite-right proxy functions are not safe to optimize (and in some sense, the more optimization pressure is applied, the less safe we expect it to be). Assigning weights to a bunch of somewhat-but-not-quite-right possibilities just gets us another somewhat-but-not-quite-right possibility. Why would we expect this to fundamentally solve the problem?

Perhaps the Bayesian mixture across hypotheses is closer to being correct, and therefore, gives us an approximation which is able to stand up to more optimization pressure before it breaks down. But this is a quantitative distinction, not a qualitative one. How big of a difference do we expect that to make? Wouldn't it still break down about as badly when put under tremendous optimization pressure?
Perhaps the point of the Bayesian mixture is that, by quantifying uncertainty about the various hypotheses, it encourages strategies which hedge their bets -- satisfying a broad range of possible utility functions, by avoiding doing something terrible for one utility function in order to get a few more points for another. But this incentive to hedge

...
What is the evidence for productivity benefits of weightlifting?
by jp — 2019-06-02T19:17:35.883Z

[Mod Note: This question received an answer that seemed worth curating. See the answer by LW user hereisonehand for the curation notice]

I've been weightlifting for a while, and I've heard vaguely good things about it's effect on productivity, like a general increase in energy levels. A recent quick google search session came up empty. If someone looks into the literature and finds something interesting I'll pay a $50 prize.*

Assume the time horizon is <5 years. I'd prefer answers focus predominantly on productivity benefits. Effects on cardiovascular could be part of an analysis, but would not qualify on their own. If the evidence is for something clearly linked to productivity, like sleep, I'd count that. Introspective evidence will also not qualify. Comparisons to other forms of exercise would be especially interesting. Assume a healthy individual, although I'm at least somewhat interested in effects on individuals with depression or anxiety given their prevalence.

*Prize to go to best answer, as judged by me, if there are any that meet some minimal threshold of rigor, also as judged by me.
On alien science
by Richard_Ngo — 2019-06-02T14:50:01.437Z

In his book The Fabric of Reality, David Deutsch makes the case that science is about coming up with good and true explanations, with all other considerations being secondary. This clashes with the more conventional view that the goal of science is to allow us to make accurate predictions - see for example this quote from the Nobel prize-winning physicist Steven Weinberg:

“The important thing is to be able to make predictions about images on the astronomers’ photographic plates, frequencies of spectral lines, and so on, and it simply doesn’t matter whether we ascribe these predictions to the physical effects of gravitational fields on the motion of planets and photons [as in pre-Einsteinian physics] or to a curvature of space and time.”

It’s true that a key trait of good explanations is that they can be used to make accurate predictions, but I think that taking prediction to be the point of doing science is misguided in a few ways.
Firstly, on a historical basis, many of the greatest scientists were clearly aiming for explanation not prediction. Astronomers like Copernicus and Kepler knew what to expect when they looked at the sky, but spent their lives searching for the reason why it appeared that way. Darwin knew a lot about the rich diversity of life on earth, but wanted to know how it had come about. Einstein was trying to reconcile Maxwell’s equations, the Michelson-Morley experiment, and classical mechanics. Predictions are often useful to verify explanations, but they’re rarely the main motivating force for scientists. And often they’re not the main reason why a theory should be accepted, either. Consider three of the greatest theories of all time: Darwinian evolution, Newtonian mechanics and Einsteinian relativity. In all three cases, the most compelling evidence for them was their ability to cleanly explain existing observations that had previously baffled scientists.
We can further clarify the case for explanation as the end goal of science by considering a thought experiment from Deutsch’s book. Suppose we had an “experiment oracle” that could predict the result of any experiment, but couldn’t tell us why it would turn out that way. In that case, I think experimental science would probably fade away, but the theorists would flourish, because it’d be more important than ever to figure out what questions to ask! Deutsch’s take on this:

“If we gave it the

...
Moral Mazes and Short Termism
by Zvi — 2019-06-02T11:30:00.348Z

Previously: Short Termism and Quotes from Moral Mazes

Epistemic Status: Long term

My list of quotes from moral mazes has a section of twenty devoted to short term thinking. It fits with, and gives internal gears and color to, my previous understanding of of the problem of short termism.

Much of what we think of as a Short Term vs. Long Term issue is actually an adversarial Goodhart’s Law problem, or a legibility vs. illegibility problem, at the object level, that then becomes a short vs. long term issue at higher levels. When a manager milks a plant (see quotes 72, 73, 78 and 79) they are not primarily trading long term assets for short term assets. Rather, they are trading unmeasured assets for measured assets (see 67 and 69).

This is why you can have companies like Amazon, Uber or Tesla get high valuations. They hit legible short-term metrics that represent long-term growth. A start-up gets rewarded for their own sort of legible short-term indicators of progress and success, and of the quality of team and therefore potential for future success. Whereas other companies, that are not based on growth, report huge pressure to hit profit numbers.

The overwhelming object level pressure towards legible short-term success, whatever that means in context, comes from being judged in the short term on one’s success, and having that judgment being more important than object-level long term success.

The easiest way for this to be true is not to care about object-level long term success. If you’re gone before the long term, and no one traces the long term back to you, why do you care what happens? That is exactly the situation the managers face in Moral Mazes (see 64, 65, 70, 71, 74 and 83, and for a non-manager very clean example see 77). In particular:

74. We’re judged on the short-term because everybody changes their jobs so frequently.

And:

64. The ideal situation, of course, is to end up in a position where one can fire one’s successors for one’s own previous mistakes.

Almost as good as having a designated scapegoat is to have already sold the company or found employment elsewhere, rendering your problems someone else’s problems.

The other way to not care is for the short-term evaluation of one’s success or failure to impact long-term success. If not hitting a short-term number gets you fired, or prevents your company from getting acceptable terms on financing or gets you...
Selection vs Control
by abramdemski — 2019-06-02T07:01:39.626Z

This is something which has bothered me for a while, but, I'm writing it specifically in response to the recent post on mesa-optimizers.

I feel strongly that the notion of 'optimization process' or 'optimizer' which people use -- partly derived from Eliezer's notion in the sequences -- should be split into two clusters. I call these two clusters 'selection' vs 'control'. I don't have precise formal statements of the distinction I'm pointing at; I'll give several examples.

Before going into it, several reasons why this sort of thing may be important:

It could help refine the discussion of mesa-optimization. The article restricted its discussion to the type of optimization I'll call 'selection', explicitly ruling out 'control'. This choice isn't obviously right. (More on this later.)
Refining 'agency-like' concepts like this seems important for embedded agency -- what we eventually want is a story about how agents can be in the world. I think almost any discussion of the relationship between agency and optimization which isn't aware of the distinction I'm drawing here (at least as a hypothesis) will be confused.
Generally, I feel like I see people making mistakes by not distinguishing between the two (whether or not they've derived their notion of optimizer from Eliezer). I judge an algorithm differently if it is intended as one or the other.

(See also Stuart Armstrong's summary of other problems with the notion of optimization power Eliezer proposed -- those are unrelated to my discussion here, and strike me more as technical issues which call for refined formulae, rather than conceptual problems which call for revised ontology.)

The Basic Idea

Eliezer quantified optimization power by asking how small a target an optimization process hits, out of a space of possibilities. The type of 'space of possibilities' is what I want to poke at here.

Selection

First, consider a typical optimization algorithm, such as simulated annealing. The algorithm constructs an element of the search space (such as a specific combination of weights for a neural network), gets feedback on how good that element is, and then tries again. Over many iterations of this process, it finds better and better elements. Eventually, it outputs a single choice.

This is the p...
FB/Discord Style Reacts
by Raemon — 2019-06-01T21:34:27.167Z

For the past year I've wanted LessWrong to include something like Discord, Facebook or Slashdot style reactions.

Facebook Style means "there's a few key reactions that people use"

Discord Style means "there's nigh-infinite reactions and you can add more, but there still end up being a few commonly used defaults."

Slashdot Style means "after upvoting or downvoting, you have the option of clicking a button that clarifies why you upvoted or downvoted."

Of these, I'm most excited for Discord-Style. But I think any of them would be improvements (if done well)

Habryka recently wrote a shortform comment on this subject. My own thoughts come in a few different frames.

Separating Enthusiasm from Approval

Boos/Yays vs 'approve/disapprove'

Empirically, people want to cheer for their causes, boo causes they dislike, signal their social allegiance and try to ensure the overton window moves in the direction they want. I don't think you can really fight this. But you can nudge people to disentangle this from "what gets attentional allocation on a site about rationality."

I think it's important that when you see a comment you like, and you feel the impulse to go "yeah! good point! go team!" the first impulse you have, the first button available and exciting to click, is a button that doesn't send any signals about how that comment should be sorted, and doesn't aggregate into an overall user-score you can check (that, for good or for ill, people will tend to associate with social status)

Other things vs 'approve/disapprove'

Boos/yays aren't the only thing I'm worried about. Ideally, I want LessWrong to reward good thinking over things like being funny, or exciting. (Being funny and exciting should still get rewarded, but no amount of clever injokes should add up to something greater than "wrote an actually useful, insightful point.")

"Viscerally Fun but Low Signal Buttons" should be easy to access. "Higher Signal" buttons should require more effort and thought.

With both of the above in mind, I think it's important that "Yay", or "Funny" buttons should be the first, most obvious thing to click on. They should feel satisfying to click, and you shouldn't feel motivated to click more things if that's the only...