trentmkelly/LessWrong-43k huggingface dataset:
Announcement: AI alignment prize winners and next round
by cousin_it — 2018-01-15T14:33:59.892Z

We (Zvi Mowshowitz, Vladimir Slepnev and Paul Christiano) are happy to announce that the

AI Alignment Prize

is a success. From November 3 to December 31 we received over 40 entries representing an incredible amount of work and insight. That's much more than we dared to hope for, in both quantity and quality.

In this post we name six winners who will receive $15,000 in total, an increase from the originally planned $5,000.

We're also kicking off the next round of the prize, which will run from today until March 31, under the same rules as before.

The winners

First prize of $5,000 goes to Scott Garrabrant (MIRI) for his post

Goodhart Taxonomy

, an excellent write-up detailing the possible failures that can arise when optimizing for a proxy instead of the actual goal. Goodhart’s Law is simple to understand, impossible to forget once learned, and applies equally to AI alignment and everyday life. While Goodhart’s Law is widely known, breaking it down in this new way seems very valuable.

Five more participants receive $2,000 each:

Tobias Baumann (FRI) for his post

Using Surrogate Goals to Deflect Threats

. Adding failsafes to the AI's utility function is a promising idea and we're happy to see more detailed treatments of it.
Vanessa Kosoy (MIRI) for her work on Delegative Reinforcement Learning (

1

,

2

,

3

). Proving performance bounds for agents that learn goals from each other is obviously important for AI alignment.
John Maxwell (unaffiliated) for his post

Friendly AI through Ontology Autogeneration

. We aren't fans of John's overall proposal, but the accompanying philosophical ideas are intriguing on their own.
Alex Mennen (unaffiliated) for his posts on

legibility to other agents

and

learning goals of simple agents

. The first is a neat way of thinking about some decision theory problems, and the second is a potentially good step for real world AI alignment.
Caspar Oesterheld (FRI) for his

post

and

paper

studying which decision theories would arise from environments like reinforcement learning or futarchy. Caspar's angle of attack is new and leads to interesting results.

We'll be contacting each winner by email to arrange transfer of money.

We would also like to thank everyone who participated. Even if you didn't get one of the prizes today, please don't let that discourage you!

The next round

We are now announcing the next round ...
What strange and ancient things might we find beneath the ice?
by Benquo — 2018-01-15T10:10:01.010Z

If I am not for myself, who will be for me?
If I am only for myself, what am I?
If not now, when?
-Hillel

Nature is not good, only proto-good.

-Paolo Soleri

Epistemic status: literally a dream

I awaken.

I am in the desert, alone.

I see the ribcage of a long-since-dead animal. I see a long row of such bones, twisted in a way that reminds me of - but is definitely not - the double helix.

I know what this means.

Evolution, on the margin, always eats free energy, to make more energy-eaters. It is a race with no upper bound, and there will be no victory. Instead, the cosmic commons will be exhausted by resource-claimers with no plan to do anything with the resources. Not that there will be anything left when the race is over.

If I do not take the next step in the dance of life, my line ends. I am just a corpse along the way.

If I do take the next step in the dance of life, then I do no better than life. As Paolo Soleri said, nature is not good; it is merely proto-good.

And if not now ...

Then I look again. Instead of bones I see a railroad. For longitudinal bones, metal tracks laid by the hand of industrial humanity, stretching in a straight line towards infinity. For ribs, the crossties that allow the uneven ground to support those even rails.

I know what this means.

The economic logic of global capitalism. Another race towards infinity, leaving nothing. Quarterly returns. Goodhart's law. Accountability and automation replacing judgment wherever they can, culminating in a Disneyland with no children. The symbol of this, the train, no natural place for the line of tracks to end, stations but no terminals, stretching on forever.

I look again. I see a road.

I awaken.

My body is warm. My head is warm. I need ice. I call for ice. A bowl of ice is brought to me. I hold the cool cubes against my temples, melt them against my forehead to cool my brain.

I awaken.

I am the Pharaoh in Egypt. My household is mighty. The world of the narrow land, the only land worth having, is my household, centered in my palace. Every decision is mine. In this warm, entropic land, it is hard to put in the mental work to run a great kingdom, but my kingdom has vast resources. I can sent servants to the icy north, to mountaintops, to bring back ice to cool me.

The resources expended on cognition increase the cognitive overhead of running my empire. I delegate. We conquer the adjacent lands, sending ships to ca...
why do trivial inconveniences matter?
by lex — 2018-01-15T06:32:29.151Z

[epistemic status: messy. probably approximately one of the major processes for task carrythrough failure.]

explorer: why do i care about trivial inconveniences? those sound, uh, trivial

speaker: well, most of the time, youre not gonna be thinking shit like this, youre just gonna be running routines

explorer: what? i havent seen a routine

speaker: yeah because a routine is to a first approximation an alternative to you.

explorer: wait i thought i was just, like, me

speaker: nooo, youre a reflective process. youre optimized for figuring out long term shit and whatever. you got other processes for when you just needa walk to the train station or something.

explorer: oh ok hi everyone

everyone: hi

explorer: so can Doer Optimizers do the thing even if its inconvenient?

speaker: technically, yes.

explorer: ...?

speaker: a typical doer process will be pretty low-energy, itll just try to do the thing the way it usually does it and give up if it doesnt work

explorer: why does it do that?

speaker: if it doesnt do that, it will try to load a webpage and encounter something bad happening and wake up six hours later with a dehydration headache and a third of a new linux distro and—as flashbacks to things involving the word 'firmware' appear before your eyes—you will observe that the webpage is still not loaded

explorer: why did it do that?

speaker: well, it got told to do the thing, and then, like, to do the thing it had to do this other thing and...

explorer: oh. whoa.

speaker: yeah.

explorer: what if i want a Doer Process thats high power but doesnt do that

speaker: well, uh, you can have doer processes that are really closely tied to the task. this is hard, because they require you to both Act, Continuously (so, something thats happy doing the same thing over and over) and Check If Youve Gone Out Of Bounds (consider what youre doing and see if its what you wanna be doing)

explorer: okay but how do i do that

speaker: i guess youd need, like, a routine to do the thing and a routine to check if youre doing the thing, where the checker is lighter weight than your full conciousness.

explorer: can i not be in an environment where things are constantly tryign to entrap you into running a doer process paying attention to them?

speaker: well, yes, but if you want to access information youll have to go out into the internet where everyone wants you to pay attention to them and will try to...
introducing: target stress
by lex — 2018-01-14T22:07:54.996Z

note: this concept is running on a predictive processing paradigm, approximately, but a fairly generalized version of said paradigm which seems obviously true.

target stress is the expectation of how much stress one is going to experience at a given time. a low target stress level is, for example, lying in bed before falling asleep: this is a low-activity state, the body is relaxed, etc. an example of a high target stress level would be, say, doing physical activity in the cold. low target stress is generally correlated with reflectivity and openness to change of direction; flow states are usually associated with high target stress. i claim that target stress is a parameter which you can alter explicitly if you so choose and one which you likely should be tracking.

this parameter heavily regulates pain tolerance and change tolerance, since it changes how surprising they are. hence, one of the avenues of dealing with pain, sensory issues, and the like is upregulating target stress so that the level of discomfort is no longer important/relevant to perception. chronic low target stress is basically the ever-present expectation that it is not worth it to spend energy on persisting through adversity because it is time to rest. the extreme form thereof can lead to inability to stand for long periods of time in seemingly physically healthy people.

chronic high target stress, particularly emotionally, is, i think, roughly the same thing as c-ptsd: it means constantly living in fear and eventually adaption to care a lot about avoiding pain and inability to lower stress (this is still better than having low target stress and being wrong). it is good to avoid that. it appears that in order to be healthy, people need to regularly cycle their target stress level. i do not yet fully understand this phenomenon. “self care” seems to basically describe dropping target stress really low and then taking care of all the needs that appear once they are not being filtered out, to keep them from accumulating (“self care” is also some other things).

when target stress is higher than experienced stress, you get twitchiness, anticipation, the need to exercise, and sensory underloading. the need to stim is sometimes caused by this. experiences stress higher than target stress is confusing, very unpleasant, and difficult to deal with. unexpected negative events like having a drink spille...
What Is "About" About?
by Bound_up — 2018-01-14T18:54:48.179Z

It has seemed to me that saying that something is "about" something else is probably the vaguest, least useful way possible to say that the two ideas are connected. I've used "about" myself in exactly those instances when I perceived a connection of some kind between two ideas, but I was incapable of articulating what that connection was, incapable even of seeing it myself.

"Well," I might say. "It seems like X is...about Y in some way...It's not exactly that X causes it...maybe X is a subsection of Y? No, that's not quite right, either. I'm not really sure, honestly, but to my eye, they seem linked to each other in some way. X is...about...Y."

Well, what did that mean, that it was "about" Y? Did they have a similar conceptual structure, were they causally related in one way or another, were they both discussed by the same groups, did they have similar effects on the world...like I said, it's vague. It could be any or none or all of these. Saying X is "about" Y tells you nothing very meaningful about them, so as a word, it's next-to-useless. Or, is it?

Now, I think I see what "about" is about; now I see it quite clearly. Saying X is "about" Y means, to say it very precisely, that you want people to give Y's connotations to X. For some reasons, X's connotations are undesirable, and you'd like Y's instead. Now, X is X, it is itself, so it is only natural that it will evoke its own connotations. If you're going to give it the more desirable connotations of Y, we must wrench the natural perspective off its natural path and force it down another.

Think of "it's not about who wins, it's about how you play the game," ie, please use the conceptual connections around this "how you play the game" idea to think about who won here today. Please don't use the connotations, do not use the subconscious connections linked to the "winning idea."

Now, what does "how you play the game" mean, exactly? No, that's not the point; the point isn't the point is to use those hard-to-define connotations, use the feelings around the "how you play the game" concept, the point is to take the aura around "how you play the game" and surround the "it" in question with...
Epistemic self-sufficiency on the edge of knowledge
by  — 2018-01-14T18:44:39.861Z

Winning an argument does not give you evidence that you are correct. Passing a security audit does not give you evidence you are secure. Inventing something doesn't mean a culture is great.

You are just a bag of mostly water. Your default expectation should be that you are not even wrong, but don't have the right concepts to be right or wrong about.

Other people are not obliged to correct you, for they are also bags of mostly water that also might not have the right concepts to be right or wrong about them.

You must be trying to correct yourself. Figure out when you are likely to be wrong or just not know enough, and somehow find the information you need to continue.

It is nice when they people can correct you, but do not expect or rely on it from other people. The further you go from the well travelled locations the more you will need to check your own knowledge and look for travelling partners that do the same.

You will need to consult with people that have different knowledge to you. Pull in botanists/meteorologists/geologists. Anyone that might have generic knowledge about the regularities of the world. But that knowledge may or may not apply to the new locations you want to go to.

If you are not drawing the information you acquire on your travels on the public map, no one will be able to correct your bad information. If you draw bad information on the public map, in order to get people to correct you, people will not have faith in any of the information you try to draw.

Look at your knowledge with a skeptical eye.
No, Seriously. Just Try It: TAPs
by Hazard — 2018-01-14T15:24:38.692Z

This next semester (I'm in university, so that's how I measure time) I'm working on developing my ability to better integrate arbitrary habits into my behavior. Trigger-action-planning (a more detailed explanation as well) is LW's most concrete strategy for doing such a thing, so I've decided to just try it. Starting 3 weeks ago and proceeding for the next 6 months I'm working on my personal approach to Trigger-action-planning (TAP for short).

My basic structure for this:

Each week (Sunday morning) design a new TAP, or redesign an old one.
Aim to follow through on the TAP in all applicable situations throughout the week.
At the end of each day (as part of an already existing review process) note if I did or did not follow through.
At the end of each week, look back and think on if the TAP was useful, what worked well, what was hard, and all that jazz.

The biggest things that's jumped out at me so far has been that not all TAPs are created equal. I think conceptual similarity does not imply actionable similarity applies very heavily to TAPs. In light of this, I'm approaching each week's TAP from a very implementation specific perspective, and then afterwards I'm going to think about what connections and universal principles might apply.

Here are some things I've noticed so far:

It's seems like there's a "mindfulness bootstrapping" problem, in that it often feels like my TAPs are only activated because of preexisting mindfulness triggers.
I've been putting the handle for each week's TAP on my phone lock screen. This was a very effective reminder originally, though it's already lost most of its power three weeks in. Mayhaps having a randomly generated background image would help prevent my mind from filtering out expected reminders. One google search and perusing the first page of results did not produce an app to do this.
I notice that sometimes there's a glimmer of noticing I'm in a context to activate the TAP, yet I don't. Part of this might be friction related to stopping whatever train of thought/awareness I'm currently on. Giving myself explicit permission to derail thoughts to execute a TAP could help.

I'll be posting updates on this project every so often. If you have had trouble with attempting TAPs or similar situational habit practices in the past...
[Paper] Global Catastrophic and Existential Risks Communication Scale, similar to Torino scale
by avturchin — 2018-01-14T10:29:49.926Z

We (Alexey Turchin and David Denkenberger) have a new paper out where we suggest a scale to communicate the size of global catastrophic and existential risks.

For impact risks, we have the Torino scale of asteroid danger which has five color-coded levels. For hurricanes we have the Saffir-Simpson scale of five categories. Here we present similar scale for communicating the size of the global catastrophic and existential risks.

Typically, some vague claims about probability are used as a communication tool for existential risks, for example, some may say, “There is a 10 per cent chance that humanity will be exterminated by nuclear war”. But the probability of the most serious global risks is difficult to measure, and the probability estimate doesn’t take into account other aspects of risks, for example, preventability, uncertainty, timing, relation to other risks etc. As a result, claims about probability could be misleading or produce reasonable skepticism.

To escape these difficulties, we suggested creating a scale to communicate existential risks, similar to the Torino scale of asteroid danger.

In our scale, there are six color codes, from white to purple. If hard probabilities are known, the color corresponds to probability intervals for a fixed timeframe of 100 years, which helps to solve uncertainty and timing.

However, for most serious risks, like AI, their probabilities are not known, but the required levels of prevention action are known. For these cases, the scale communicates the risk’s size through the required level of prevention action. In some sense, it is similar to Updateless Decision Theory, where an event’s significance is measured, not by observable probabilities, but by the utility of corresponding actions. The system would work, because in many cases of x-risks, the required prevention actions are not very sensitive to the probability.

How should the scale be implemented in practice? If probabilities are not known, a group of experts should aggregate available information and communicate it to the public and policymakers, saying something like: "We think that AI is a red risk, a pandemic is a yellow risk and asteroid danger is a green risk.” It would help to bring some order to the public perception of each risk—where, currently, asteroid danger is clearly overestimated compared to the risks of AI risk—without making unsustainable cla...
Syllable
by alkjash — 2018-01-14T00:29:41.589Z
Link: https://radimentary.wordpress.com/2018/01/13/syllable/

Status: fun to write, very derivative, ambivalent about content.

This post responds to Why and How to Name Things.

In the evolution of language, new words are constantly produced, bubbling in and out of primordial concept-space like quantum foam. Most of them are awkward, unevocative, and noncentral, falling out of fashion faster than you can say "Planck time." A few gems cut out unexplored territory and grow in popularity until they take their rightful place in the common parlance. As they do so, however, they fall prey to concept creep, being diluted to incorporate such a plethora of valences as to become useless. Essential words like "love" have been dragged through the mud or mouthed insincerely so often that it feels affected to even utter them.

As old concept handles die or creep, a constant creative drive is required into invent new words for old and new phenomena. This places a heavy burden on writer and reader alike. Improper names are like programmer debt - let a weak name stick early on and you'll pay for it your whole life.

This babble outlines the shape of this problem and what we might do about it.

First, I argue that as far as names go, only length matters. Shorter words are exponentially more important. I call this the Inverse Teenage Boy Model (ITBM) of Names.

Having built an ITBM, we need a nuclear weapon for it to deliver. I propose a new usage of the word syllable: a verb, meaning "to rename with fewer syllables." I give examples of savvy syllabling in math, computer science, and rationality, and try to syllable some names I hate.

The Inverse Teenage Boy Model

The Teenage Boy Model (TBM) is a universal theory of value developed in the most advanced locker rooms in the history of our civilization. Its thesis: only size matters. Carl
Sagan was one of the champions of the TBM:

Look again at that dot. That's here. That's home. That's us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, e

...
Field-Building and Deep Models
by Ben Pace — 2018-01-13T21:16:14.523Z

What is important in hiring/field-building in x-risk and AI alignment communities and orgs? I had a few conversations on this recently, and I'm trying to publicly write up key ideas more regularly.

I had in mind the mantra 'better written quickly than not written at all', so you can expect some failures in enjoyability and clarity. No character represents any individual, but is an amalgam of thoughts I’ve had and that others have raised.

albert cares deeply about x-risk from AI, and wants to grow the field of alignment quickly; he also worries that people in x-risk community errs too much on the side of hiring people similar to themselves.

ben cares deeply about x-risk from AI, and thinks that we should grow the AI safety community slowly and carefully; he feels it’s important to ensure new members of the community understand what’s already been learned, and avoid the eternal september effect.

albert: So, I understand you care about picking individuals and teams that agree with your framing of the problem.

ben: That sounds about right - a team or community must share deep models of their problems to make progress together.

albert: Concretely on the research side, what research seems valuable to you?

ben: If you’re asking what I think is most likely to push the needle forward on alignment, then I’d point to MIRI’s and Paul’s respective research paths, and also some of the safety work being done at DeepMind and FHI.

albert: Right. I think there are also valuable teams being funded by FLI and Open Phil who think about safety while doing more mainstream capabilities research. More generally, I think you don't need to hire people that think very similarly to you in your organisations. Do you disagree?

ben: That's an interesting question. On the non-research side, my first thought is to ask what Y Combinator says about organisations. One thing we learn from YC are that the first 10-20 hires of your organisation will make or break it, especially the co-founders. Picking even a slightly suboptimal co-founder - someone who doesn't perfectly fit your team culture, understand the product, and work well with you - is the easiest way to kill your company. This suggests to me a high prior on selectivity (though I haven't looked in detail into the other research groups you mention).

albert: So you're saying that if the x-risk community ...