trentmkelly/LessWrong-43k huggingface dataset:
Conditions for Mesa-Optimization
by evhub — 2019-06-01T20:52:19.461Z

This is the second of five posts in the Risks from Learned Optimization Sequence based on the paper “Risks from Learned Optimization in Advanced Machine Learning Systems” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Each post in the sequence corresponds to a different section of the paper.

In this post, we consider how the following two components of a particular machine learning system might influence whether it will produce a mesa-optimizer:

The task: The training distribution and base objective function.

The base optimizer: The machine learning algorithm and model architecture.

We deliberately choose to present theoretical considerations for why mesa-optimization may or may not occur rather than provide concrete examples. Mesa-optimization is a phenomenon that we believe will occur mainly in machine learning systems that are more advanced than those that exist today.[1] Thus, an attempt to induce mesa-optimization in a current machine learning system would likely require us to use an artificial setup specifically designed to induce mesa-optimization. Moreover, the limited interpretability of neural networks, combined with the fact that there is no general and precise definition of “optimizer,” means that it would be hard to evaluate whether a given model is a mesa-optimizer.

2.1. The task

Some tasks benefit from mesa-optimizers more than others. For example, tic-tac-toe can be perfectly solved by simple rules. Thus, a base optimizer has no need to generate a mesa-optimizer to solve tic-tac-toe, since a simple learned algorithm implementing the rules for perfect play will do. Human survival in the savanna, by contrast, did seem to benefit from mesa-optimization. Below, we discuss the properties of tasks that may influence the likelihood of mesa-optimization.

Better generalization through search. To be able to consistently achieve a certain level of performance in an environment, we hypothesize that there will always have to be some minimum amount of optimization power that must be applied to find a policy that performs that well.

To see this, we can think of optimization power as being measured in terms of the number of times the optimizer is able to divide the search space in half—that is, the number of bits of information provided.(9) After these divisions, there will be some remaining space of poli

...
The Fundamental Theorem of Asset Pricing: Missing Link of the Dutch Book Arguments
by johnswentworth — 2019-06-01T20:34:06.924Z

Assumed background:

Acyclic preferences, Dutch Book theorems

There are fairly elementary arguments that, in the absence of uncertainty, any preferences not described by a utility function are problematic - this is the circular preferences argument. There are also fairly elementary arguments that, if we handle uncertainty by taking weighted sums of utilities of different outcomes, then the weights should follow the usual rules of probability - these are the Dutch Book arguments. But in the middle there’s a jump: we need to assume that taking weighted sums of utilities makes sense for some reason. There are some high-powered theorems which make that jump (specifically the complete class theorem), but they’re not very mathematically accessible.

(If any of that sounds new, you should read

Yudkowsky’s excellent intro to this stuff

before reading this post.)

It turns out that there is a relatively simple theorem which bridges the gap between deterministic utility and Dutch Book arguments. But rather than hanging out in decision theory textbooks, it’s been living it up in finance. It’s called the Fundamental Theorem of Asset Pricing (FTAP).

Here’s the setup. Just like the Dutch Book arguments, we have a bunch of tradable assets - i.e. betting contracts, like stock options or horse race bets. We have a bunch of possible outcomes - i.e. possible prices of an underlying stock at expiry, or possible winners of the horse race. Each asset's final value will depend on the outcome. Then the FTAP states that either:

There exists some portfolio of assets which costs $0 to buy (can include short sales) and is guaranteed a positive payout (i.e. arbitrage), or
There exists a probability distribution such that the price of each asset is the expected value of its payout (i.e. price is a weighted sum of possible outcome values).

Note that this is exactly what we need to round out the Dutch Book arguments: either there exists an arbitrage opportunity, or we compare assets using a weighted sum of possible outcome values.

Let’s prove it. First, we’ll name some variables:

Vij: a big matrix which contains the value of each asset i under each possible outcome j.
Si: current price of asset i (we need P for probability, so S represents price).
pj: probability distribution over outcomes j (which may or may not exist)
qi: arbitrage portfolio (which may or may not exist)

FTAP says that either:

A

...
I translated 'Twelve Virtues of Rationality' into Hebrew.
by Yoav Ravid — 2019-06-01T18:52:02.436Z

Here it is - if you know Hebrew and have feedback, do give it, either in the comments here or in the document (it's not fully edited).

I don't really know what am i supposed to do with it now, though. where should i put it? can/should i put it on my future website? (of course i don't own the translation anymore than the original piece)

Regarding the translation, the thing I'm most uncertain of is (ironically) which word to use for 'virtue', if you have a better idea, please share.
Welcome and Open Thread June 2019
by roland — 2019-06-01T13:44:38.655Z

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, and seeing if there are any meetups in your area.

The Open Thread sequence is here.
"But It Doesn't Matter"
by Zack_M_Davis — 2019-06-01T02:06:30.624Z

If you ever find yourself saying, "Even if Hypothesis H is true, it doesn't have any decision-relevant implications," you are rationalizing! The fact that H is interesting enough for you to be considering the question at all (it's not some arbitrary trivium like the 1923th binary digit of π, or the low temperature in São Paulo on September 17, 1978) means that it must have some relevance to the things you care about. It is vanishingly improbable that your optimal decisions are going to be the same in worlds where H is true and worlds where H is false. The fact that you're tempted to say they're the same is probably because some part of you is afraid of some of the imagined consequences of H being true. But H is already true or already false! If you happen to live in a world where H is true, and you make decisions as if you lived in a world where H is false, you are thereby missing out on all the extra utility you would get if you made the H-optimal decisions instead! If you can figure out exactly what you're afraid of, maybe that will help you work out what the H-optimal decisions are. Then you'll be a better position to successfully notice which world you actually live in.
Feedback Requested! Draft of a New About/Welcome Page for LessWrong
by Ruby — 2019-06-01T00:44:58.977Z

Context for Draft / Request for Feedback

The LessWrong team is hoping to soon display a new About/Welcome page which does an improved job of conveying what LessWrong.com is about and how community members can productively use the site.

However, LessWrong is a community site and I (plus the team) feel it's not appropriately for us to unilaterally declare what LessWrong is about. So here's our in-progress draft of a new About/Welcome page. Please let us know what you think in the comments. Please especially let us know if you think LessWrong is actually about something else. Or even just what it means to you.

Thanks!

<3 Ruby

---------------------------------------------------------------------------------

Related:

Team / Who are we?
A Brief History of LessWrong

The tl;dr

LessWrong is a community blog devoted to the art of human rationality.

We invite you to use this site for any number of reasons, including, but not limited to: learning valuable things, being entertained, sharing and getting feedback on your ideas, and participating in a community you like. However, fundamentally, this site is designed for two main uses:

As a place to level-up your rationality
As a place to apply your rationality to important real-world problems

Primary things to do on LessWrong are:

Read LessWrong’s repository of rationality materials
Join a local rationality meetup
Join in a discussion
Ask or answer a question
Write a post

Leveling up your rationality

First off, what is rationality?

Rationality is a term which can have different connotations to different people. On LessWrong, we mean something like the following:

Rationality is thinking in ways which systematically arrive at truth.
Rationality is thinking in ways which cause you to achieve your goals.
Rationality is trying to do better on purpose.
Rationality is reasoning well even in the face of massive uncertainty.
Rationality is making good decisions even when it’s hard.
Rationality is being self-aware, understanding how your own mind works, and applying this knowledge to thinking better.

What rationality is not:

Forsaking all human emotion and intuition to embrace Cold Hard Logic.

Why should I care about rationality?

One reason to care about rationality is because you intrinsically care about having true beliefs. You might also care about rationality because you care about anything at all. Our ability to achieve our goals depends on 1...
A Brief History of LessWrong

by Ruby — 2019-06-01T00:43:59.408Z

In 2006,

Eliezer Yudkowsky

,

Robin Hanson

, and others began writing on

Overcoming Bias

, a group blog with the general theme of how to move one’s beliefs closer to reality despite biases such as overconfidence and wishful thinking. In 2009, after the topics drifted more widely, Eliezer moved to a new community blog, LessWrong.

LessWrong was seeded with series of daily blog posts written by Eliezer, originally known as The Sequences, and more recently compiled into an edited volume,

Rationality: A-Z

. These writings attracted a large community of readers and writers interested in the art of human rationality.

In 2015-2016 the site underwent a steady decline of activity leading some to declare the site dead. In 2017, a team led by Oliver Habryka took over the administration and development of the site, relaunching it on an

entirely new codebase

later that year.

The new project, dubbed LessWrong 2.0, was the first time LessWrong had a full-time dedicated development team behind it instead of only volunteer hours. Site activity recovered from the 2015-2016 decline and

has remained at steady levels

since the launch.

The team behind LessWrong 2.0 has ambitions not limited to maintaining the original LessWrong community blog and forum. The LessWrong 2.0 team conceives of itself more broadly as an organization attempting to build community, culture, and technology which will drive intellectual progress on the world’s most pressing problems.
The LessWrong Team
by Ruby — 2019-06-01T00:43:31.545Z

LessWrong Core Team

Oliver Habryka / habryka

Oliver Habryka is the current project lead for LessWrong.com, where he tries to build infrastructure for making intellectual progress on global catastrophic risks, cause prioritization and the art of rationality. He used to work at the Centre for Effective Altruism US as strategic director, ran the EA Global conferences for 2015 and 2016 and is an instructor for the Center for Applied Rationality. He has generally been involved with community organizing for the Effective Altruism and Rationality communities in a large variety of ways. He studied Computer Science and Mathematics at UC Berkeley, and his primary interests are centered around understanding how to develop communities and systems that can make scalable progress on difficult philosophical and scientific problems.

Ruben Bloom / Ruby

I found LessWrong in 2012 after googling "singularity" and finding facingthesingularity.com (since renamed). Among the first posts I read was Philosophy: A Diseased Discipline, which pissed me off because I was philosophy major. But then I read it realized I agreed with all of it. Since then, I've been deep in the LW community. I became a local organizer in Melbourne, mentored at a dozen CFAR workshops, and eventually made my way to the Bay Area. Before joining the LessWrong team in 2019, I studied electrical engineering and philosophy, and worked as a Data Scientist and Product Manager.

Robert Mushkatblat / RobertM

I found LessWrong through HPMOR in 2012, briefly interned at MIRI in 2013 (which didn't accomplish much except convince me to not enter academia as a post-grad, which was good), and started attending meetups in Los Angeles shortly thereafter. In 2017 I took over running those meetups. In 2022 Eliezer posted MIRI announces new "Death With Dignity" strategy, which was the sharp poke I needed to go do something about the whole x-risk thing.

By trade, I'm a software engineer. I'm responsible for technical leadership at LessWrong.

Raymond Arnold / Raemon

I've been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I've been interested in improving my own epistemic standards and helpi

...
Risks from Learned Optimization: Introduction
by evhub — 2019-05-31T23:44:53.703Z

This is the first of five posts in the Risks from Learned Optimization Sequence based on the paper “Risks from Learned Optimization in Advanced Machine Learning Systems” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Each post in the sequence corresponds to a different section of the paper.

Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, and Joar Skalse contributed equally to this sequence. With special thanks to Paul Christiano, Eric Drexler, Rob Bensinger, Jan Leike, Rohin Shah, William Saunders, Buck Shlegeris, David Dalrymple, Abram Demski, Stuart Armstrong, Linda Linsefors, Carl Shulman, Toby Ord, Kate Woolverton, and everyone else who provided feedback on earlier versions of this sequence.

Motivation

The goal of this sequence is to analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization, a neologism we introduce in this sequence. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned?

We believe that this sequence presents the most thorough analysis of these questions that has been conducted to date. In particular, we present not only an introduction to the basic concerns surrounding mesa-optimizers, but also an analysis of the particular aspects of an AI system that we believe are likely to make the problems related to mesa-optimization relatively easier or harder to solve. By providing a framework for understanding the degree to which different AI systems are likely to be robust to misaligned mesa-optimization, we hope to start a discussion about the best ways of structuring machine learning systems to solve these problems. Furthermore, in the fourth post we will provide what we think is the most detailed analysis yet of a problem we refer as deceptive alignment which we posit may present one of the largest—though not necessarily insurmountable—current obstacles to producing safe advanced machine learning systems using techniques sim...
Site Guide: Personal Blogposts vs Frontpage Posts

by Ruby — 2019-05-31T23:08:07.363Z

Posts on practically any topic are welcomed on LessWrong [1]. I (and others on the team) feel it is important that members are able to “bring their entire selves” to LessWrong and are able to share all their thoughts, ideas, and experiences without fearing whether they are “on topic” for LessWrong. Rationality is not restricted to only specific domains of one’s life and neither should LessWrong be.

However, to maintain its overall focus while still allowing posts on any topic, LessWrong classifies posts as either Personal blogposts or as Frontpage posts.

The two classifications

Personal blogposts

Is the default classification for all posts.
Are not displayed by default on the homepage.
Can be on any topic [1] and in any format: nothing is “off topic”
Suitable for personal interests, blogging, and general ramblings

e.g. your thoughts on Magic the Gathering, a poem, or a short story you wrote

Suitable for discussion of current events
Suitable for discussion of specific social and community issues
Suitable for discussion of highly divisive topics
Suitable for discussion of the LessWrong website*

Frontpage posts

Are displayed by default to all users.
Authors can allow moderators to give their post Frontpage status if the moderator judges the post to be:

Useful, novel, and relevant to many LessWrong members
“Timeless”, i.e. minimizes references to current events and is likely to remain useful even after a few years
The post attempts to explain rather than persuade

This system allows LessWrong members to write about whatever is of interest to them while ensuring that only members who wish to see “off topic” content see that content.

*The LessWrong team will make some exceptions in the case of announcement posts we believe should have maximally broad visibility, e.g. the FAQ, and give these Frontpage status.

How to view personal blogposts

Personal blogposts, with their laxer restrictions, are not shown by default on LessWrong’s homepage. To view Personal blogposts, you can:

Click the “Include Personal blogposts” checkbox beneath the Latest Posts section.
Visit the

All Posts

page and ensure "Filtered by" is set to “All Posts”
Find a user’s Personal blogposts by visiting their user profile page
Personal blogposts and their comments appear in the Recent Discussion f

...
When Observation Beats Experiment
by johnswentworth — 2019-05-31T22:58:57.986Z

Suppose we have a strain of lab rats which are colored purple, and we want to know why. We suspect that chemical X is responsible, so we run an experiment:

We genetically modify our purple rats to repress X production, and find that their purple coloration disappears.
We genetically modify ordinary rats to produce X, and find that their coats turn purple.

We conclude that chemical X is both necessary and sufficient to turn rats’ coats purple. Case closed!

… or maybe not.

Suppose that rats are purple-colored if-and-only-if they express Purple Pigment (PP) above some threshold level. Purple Pigment, in turn, is chemically produced from X and Y:

X+Y⇌PP

High levels of PP could result from high levels of X, or from high levels of Y. Either way, increasing X enough will always turn a rat purple, and decreasing X enough will always turn a rat not-purple. So our experiment doesn’t tell us whether our particular rats are purple due to high X or high Y - it could be either. In order to tell the difference, we need to go measure X and Y levels in our rats - not an experiment, but an observation.

(Warning: technical details not relevant to the main point were brushed under the rug there.)

Generalizing: experiments are really good for figuring out the structure of the underlying causal graph. How can we tell that Purple Pigment is produced from X and Y in the first place? Experiment: we try various levels of X and Y and see which rats are purple.

But if we want to know the state of the causal graph, in some real-world system, then observation beats experiment. To find out whether our particular rats are purple because of high X or high Y, we should measure their X and Y levels, without any experimental intervention. Of course, this only works if we’ve already done the experiments to figure out the structure of the system.
How to determine if my sympathetic or my parasympathetic nervous system is currently dominant?
by philip_b — 2019-05-31T20:40:30.664Z

I've been reading CFAR Handbook's chapter about Againstness. The chapter's idea is that when your sympathetic nervous system (SNS) is dominant over parasympathetic nervous system (PSNS), your introspection is impaired and you tend to be less rational, hence you should learn how to know which system is currently dominant and also learn to switch to PSNS dominance.

They provide the following table of how your body, mind, and behaviour change depending on which system is dominant

In order to learn to know where you are on SNS-PSNS spectrum they recommend observing yourself in different situations, determine which system is dominant, and try to find patterns (e.g., maybe after doing physical exercise your SNS is usually dominant).

Well, I tried observing myself and figuring out which nervous system is active, and about 70% of the times I can't determine it. I ask myself questions like "Do I feel joy?", "Am I at peace?", "Are my muscles tense?", "Does my skin feel like flushed?", "Is my breathing shallow, or slow and in belly?". More often than not I can't answer these questions. And it doesn't help that there's no easy way to calibrate by observing the ground truth.

If there are people here who can determine which of your two nervous systems is dominant, how do you do it? Any tips for me?

Also, sometimes I get signals from my body that don't fit in this framework. When it's time for me to go to sleep but I don't do it, after a while my awareness and introspective clarity become impaired, I feel both energized and lacking energy at the same time, I fill like my system 2 (as in Kahneman's two systems) doesn't turn on a lot, and I get impatient. What happens with SNS and PSNS at this time? I don't know.
What is the best online community for questions about AI capabilities?
by Ben Goldhaber — 2019-05-31T15:38:11.678Z

I'm interested in learning more about existing ML model capabilities, and in particular how easily a model that is used on one task could be used for another, similar task.

For example, if I want to know and/or estimate whether an existing AI system like AlphaZero could beat Angry Birds with current capabilities, what's the best place to post that type question?