trentmkelly/LessWrong-43k huggingface dataset:
The Alignment Problem
by lsusr — 2022-07-11T03:03:03.271Z

Last month Eliezer Yudkowsky wrote "a poorly organized list of individual rants" about how AI is going to kill us all. In this post, I attempt to summarize the rants in my own words.

These are not my personal opinions. This post is not the place for my personal opinions. Instead, this post is my attempt to write my understanding of Yudkowsky's opinions.

I am much more optimistic about our future than Yudkowsky is. But that is not the topic of this post.

Humanity is going to build an AGI as fast as we can.

Humanity is probably going to build an AGI, and soon.

But if an AGI is going to kill us all can't we choose to just not build an AGI?

Nope! If humanity had the coordination ability to "just not build an AGI because an AGI is an existential threat" then we wouldn't have built doomsday weapons whose intended purpose is to be an existential threat.

The first nuclear weapon was a proof of concept. The second and third nuclear bombs were detonated on civilian targets. "Choosing not to build an AGI" is much, much harder than choosing not to build nuclear weapons because:

Nukes are physical. Software is digital. It is very hard to regulate information.

Nukes are expensive. Only nation-states can afford them. This limits the number of actors who are required to coordinate.

Nukes require either plutonium or enriched uranium, both of which are rare and which have a small number of legitimate uses. Datacenters meet none of those criteria.

Uranium centrifuges are difficult to hide.

Nuclear bombs and nuclear reactors are very different technologies. It is easy to build a nuclear reactor that cannot be easily converted into weapon use.

A nuclear reactor will never turn into a nuclear weapon by accident. A nuclear weapon is not just a nuclear reactor that accidentally melted down.

Nuclear weapons are the easiest thing in the world to coordinate on and (TPNW aside) humanity has mostly failed to do so.

Maybe people will build narrow AIs instead of AGIs.

We [humanity] will build the most powerful AIs we can. An AGI that combines two narrow AIs will be more powerful than both of the narrow AIs. This is because the hardest part of building an AGI is figuring out what representation to use. An AGI can do everything a narrow AI can do plus it gets transfer learning on top of that.

Maybe building an AGI is really hard—so hard we won't build it this century.

Maybe. But recent developmen...
Immanuel Kant and the Decision Theory App Store
by Daniel Kokotajlo — 2022-07-10T16:04:04.248Z

[Epistemic status: About as silly as it sounds.]

Prepare to be astounded by this rationalist reconstruction of Kant, drawn out of an unbelievably tiny parcel of Kant literature![1]

Kant argues that all rational agents will:

“Act only according to that maxim whereby you can at the same time will that it should become a universal law.” (421)[2][3]
“Act in such a way that you treat humanity, whether in your own person or in the person of another, always at the same time as an end and never simply as a means.” (429)[2]

Kant clarifies that treating someone as an end means striving to further their ends, i.e. goals/values. (430)[2]
Kant clarifies that strictly speaking it’s not just humans that should be treated this way, but all rational beings. He specifically says that this does not extend to non-rational beings. (428)[2]

“Act in accordance with the maxims of a member legislating universal laws for a merely possible kingdom of ends.” (439)[2]

Not only are all of these claims allegedly derivable from the concept of instrumental rationality, they are supposedly equivalent!

Bold claims, lol. What is he smoking?

Well, listen up…

Taboo “morality.” We are interested in functions that map [epistemic state, preferences, set of available actions] to [action].

Suppose there is an "optimal" function. Call this "instrumental rationality," a.k.a. “Systematized Winning.”

Kant asks: Obviously what the optimal function tells you to do depends heavily on your goals and credences; the best way to systematically win depends on what the victory conditions are. Is there anything interesting we can say about what the optimal function recommends that isn’t like this? Any non-trivial things that it tells everyone to do regardless of what their goals are?[4]

Kant answers: Yes! Consider the twin Prisoner's Dilemma--a version of the PD in which it is common knowledge that both players implement the same algorithm and thus will make the same choice. Suppose (for contradiction) that the optimal function defects. We can now construct a new function, Optimal+, that seems superior to the optimal function:

IF in twin PD against someone who you know runs Optimal+: Cooperate

ELSE: Do whatever the optimal function will do.

Optimal+ is superior to the optimal function because it is exactly the same except that it gets better results in the twin PD (because the opponent will cooperate too, because they are run...
Metaculus is seeking experienced leaders, researchers & operators for high-impact roles
by ChristianWilliams — 2022-07-10T14:27:38.197Z
Link: https://apply.workable.com/metaculus/

Metaculus's mission is to build epistemic infrastructure that enables the global community to model, understand, predict, and navigate the world’s most important and complex challenges.​

We’re thrilled to share the first round of new high-impact open roles as we expand our operations over the next year.

Open roles include:

a Chief of Staff to lead mission critical internal initiatives
a CTO role ideal for someone with management experience who can help guide our technology team
an AI Forecasting Lead to drive the development of a comprehensive AI forecasting agenda
a Forecasting Program Director — ideal for someone who has a strong understanding of the forecasting field and a penchant for relationship-building
and a Forecasting Program Analyst who will manage projects for Metaculus’s partners in government

Learn more about these and our other open positions (more coming soon!)

here

.
Avoid the abbreviation "FLOPs" – use "FLOP" or "FLOP/s" instead
by Daniel_Eth — 2022-07-10T10:44:38.046Z

Especially in discussions about AI, the abbreviation "FLOPs" is being used for both "floating point operations per second" (a measure of computational power) and "floating point operations" (a measure of total computations, and equivalent to the previous term times seconds). This is ambiguous and confusing. For clarity, I propose people avoid this specific abbreviation and instead use the alternatives of "FLOP" (for floating point operations) and "FLOP/s" (for floating point operations per second).
My Opportunity Costs
by abstractapplic — 2022-07-10T10:14:58.827Z

When I was younger, I had time and energy to spare on weird side-projects, and a comparative lack of ideas worth spending them on. Now I have some ideas, but no time and energy to spare. I share these ideas below, in case anyone is in the same situation I was a decade ago[1].

Editing Weird Worthwhile Writers

Lsusr has a lot of interesting things to say. He also makes a lot of trivial grammar/spelling/style mistakes. It occurred to me about a year ago that someone could pleasantly do a modest amount of good very easily by volunteering to edit for him; it also occurred to me that doing this for his business writing might be a good way to learn about running a company and make connections that could be useful later. I then proceeded to not do that, because editing work isn’t my comparative advantage, and because I’d feel like I was stealing a role which could be filled by someone much younger, more ambitious and less experienced than me. Recently, that role has been filled by someone much younger, more ambitious and less experienced than me; it seems to be going pretty well for everyone involved so far.

Dominic Cummings has a lot of interesting things to say. He also has an idiosyncratic writing style which is heavily informed by his time in the Civil Service, where everyone he had to communicate with was some combination of a) captive audiences, b) people whose hectic schedules leave them with negligible attention spans c) people who use all the same jargon he does and d) people he actively despises. A teenager on Twitter has accumulated a >10k following by translating his tweets from Dom-ese to English. I suspect that rewriting some of his essays would be a good way to get appreciation and attention from the internet in general and LessWrong in particular.

[Redacted because I don’t know if sharing this idea would be legal]

Accumulating Life Stories

I used to have a hobby of showing up at train stations after midnight and asking people who had missed the last train to tell me their life story. Faced with the choice between spending eight hours scrolling on their phone and spending that time talking about themselves, almost all of them said yes.

I did this because I wanted to hear about the world from a source without the usual filters. The people I know in real life are few in number and mostly my age. Media in general and the internet in particular are disproport...
Why Portland
by Adam Zerner — 2022-07-10T07:20:39.785Z

Meta: This is a personal post. Almost like a journal entry, but also loosely intended to be enjoyable and/or useful to other people.

(Cross-posted on my personal blog.)

Why do people live where they live? Everyone's got a different answer.

I lived in Vegas for about five years. Why did I live there? My answer was pretty simple. My girlfriend went to school there. So she needed to be there for school. And I wanted to be with her. So I lived in Vegas.

But she graduated at the end of 2021. I work remotely, and she's open to whatever, so we were no longer tied to Vegas, and it was time to decide where we want to live.

Well, there are a lot of cool options on the table. So at first our plan was to live for something like three months at four or five different places to try em out and see what we think. But then, instead, we decided to just move to Portland, the top place on our list. Well, we're currently about five months into a year-long lease and are probably going to buy something in the next year or two.

Why did we decide to do this? To start, let's go over what we were looking for.

The big thing is walkability. There are so many reasons for this.

Cars are expensive.

Having to walk places is makes it easy to be active, which is good for both health and happiness.

Walkable areas are awesome for a bunch of reasons.

Cars are one of the most risky things when it comes to dying early, and I am weird and prefer to avoid that risk.

Unfortunately, walkable places tend to be quite expensive. Think: big cities like New York, Boston and SF. Places that expensive would really make it harder to retire early, something I'm pursuing, so I'd like to avoid them. But when you filter for cities that are both walkable and not crazy expensive, well there actually aren't a lot left! Here is the list I came up with:

Portland

Philly

Denver

Miami

And then as a ~second tier:

DC

Austin

Pittsburgh

San Diego

Boston

Smaller towns

Then there are also places outside of America, like Mexico, Costa Rica, Amsterdam, Berlin and Thailand. Perhaps we'll revisit that in the future. It'd make seeing friends and family really difficult, and my job requires me to be in America. Plus language barriers would be annoying.

Oh, and my girlfriend likes to smoke weed, so we want to be somewhere where she can do that.

Working backwards from the end of tier two...

There are some smaller towns (as opposed to real...
Hessian and Basin volume
by Vivek Hebbar — 2022-07-10T06:59:33.702Z

Thanks to Thomas Kwa for the question which prompted this post.

Note: This is mostly a primer / introductory reference, not a research post. However, the details should be interesting even to those familiar with the area.

When discussing “broad basins” in the loss landscape of a DNN, the Hessian of loss is often referenced. This post explains a simple theoretical approximation of basin volume which uses the Hessian of loss.[1]

Suppose our minimum has loss=0. Define the basin as the region of parameter space draining to our minimum where loss < threshold T.[2]

Simplest model

If all eigenvalues of the Hessian are positive and non trivial,[3] we can approximate the loss as a paraboloid centered on our minimum:

The part of the paraboloid we see on the graph is precisely our basin, since the vertical axis is cut off at a loss threshold.

The vertical axis is loss, and the horizontal plane is parameter space. The shape of the basin in parameter space is the shadow of this paraboloid, which is an ellipsoid.

The principal directions of curvature of the paraboloid are given by the eigenvectors of the Hessian. The curvatures (second derivative) in each of those directions is given by the corresponding eigenvalue.

Radii of the ellipsoid: If we start at our minimum and walk away in a principal direction, the loss as a function of distance traveled is L(x)=12λix2, where λi is the Hessian eigenvalue for that direction. So given our loss threshold T, we will hit that threshold at a distance of x=√2Tλi. This is the radius of the loss-basin ellipsoid in that direction.

The volume of the ellipsoid is Vbasin=Vn∏i√2Tλi, where the constant Vn is the volume of the unit n-ball in n dimensions. Since the product of the eigenvalues is the determinant of the Hessian, we can write this as:

Vbasin=Vn(2T)n/2√det[Hessian]

So the basin volume is inversely proportional to the square root of the determinant of the Hessian. Everything in the numerator is a constant, so only the determinant of the Hessian matters in this model.

The problem with this model is that the determinant of the Hessian is usually zero, due to zero eigenvalues.

Fixing the model

If we don't include a regularization term in the loss, the basin as we defined it earlier can actually be infinitely big ...
Taste & Shaping
by CFAR!Duncan — 2022-07-10T05:50:14.416Z

Author's note: while it does not belong in the handbook proper, Duncan Sabien's essay Goodhart's Imperius is very close to (and partially derived from) the content in this chapter, and is a good extension or follow-up.

Epistemic status: Mixed

Many of the concepts presented in Taste & Shaping (such as operant conditioning, hyperbolic discounting, and theories of intrinsic vs. extrinsic motivation) are all well-known and well-researched. Similarly, the problem the unit seeks to address (of experiencing an emotional disconnect between actions and goals) is widely known and discussed. There is some academic support for the efficacy of priming and narrative reframings, and strong anecdotal support (but no formal research) for the overall conceptual framework.

English uses the word “want” to mean both the declarative, persistent desire to achieve a long-term goal, and also the immediate, visceral desire to satisfy an in-the-moment urge. That’s how we end up saying sentences like “I want to exercise, but I don’t want to exercise...know what I mean?”

Part of the reason that we don’t want to exercise, in the moment (or that we don’t want to work on page 37 of our dissertation, or that we don’t want to call up our parents even though it’s been a while, or that we don’t want to look at our work email backlog, or any number of other things) has to do with emotional valence.

Objects, actions, and experiences often carry with them a sort of aggregated positive-negative or approach-avoid rating, which we at CFAR casually refer to as a “yuck” or a “yum.” Consider, for instance:

The sound that a laptop makes when it falls on a hard, concrete floor
The feelings evoked by the “pinwheel of death” on an Apple computer
Your emotional reaction to the sound you typically use as your alarm
The sight of flashing lights in your rear view mirror

For many people, the sound of a falling laptop is actually painful in some hard-to-define way—it’s as though the sound is somehow intrinsically bad, just as the pinwheel of death is fundamentally frustrating. On an empirical, logical level, it’s clear that these associations were picked up after the fact— many people grow to hate their alarm clock sound over time; few people choose an unpleasant sound to begin with.

But on an emotional level, the yucks and yums can feel inherent and essential. We want to exercise, because exercise (and health an...
Comment on "Propositions Concerning Digital Minds and Society"
by Zack_M_Davis — 2022-07-10T05:48:51.013Z

I will do my best to teach them

About life and what it's worth

I just hope that I can keep them

From destroying the Earth

—Jonathan Coulton, "The Future Soon"

In a recent paper, Nick Bostrom and Carl Shulman present "Propositions Concerning Digital Minds and Society", a tentative bullet-list outline of claims about how advanced AI could be integrated into Society.

I want to like this list. I like the kind of thing this list is trying to do. But something about some of the points just feels—off. Too conservative, too anthropomorphic—like the list is trying to adapt the spirit of the Universal Declaration of Human Rights to changed circumstances, without noticing that the whole ontology that the Declaration is written in isn't going to survive the intelligence explosion—and probably never really worked as a description of our own world, either.

This feels like a weird criticism to make of Nick Bostrom and Carl Shulman, who probably already know any particular fact or observation I might include in my commentary. (Bostrom literally wrote the book on superintelligence.) "Too anthropomorphic", I claim? The list explicitly names many ways in which AI minds could differ from our own—in overall intelligence, specific capabilities, motivations, substrate, quality and quantity (!) of consciousness, subjective speed—and goes into some detail about how this could change the game theory of Society. What more can I expect of our authors?

It just doesn't seem like the implications of the differences have fully propagated into some of the recommendations?—as if an attempt to write in a way that's comprehensible to Shock Level 2 tech executives and policymakers has failed to elicit all of the latent knowledge that Bostrom and Shulman actually possess. It's understandable that our reasoning about the future often ends up relying on analogies to phenomena we already understand, but ultimately, making sense of a radically different future is going to require new concepts that won't permit reasoning by analogy.

After an introductory sub-list of claims about consciousness and the philosophy of mind (just the basics: physicalism; reductionism on personal identity; some non-human animals are probably conscious and AIs could be, too), we get a sub-list about respecting AI interests. This is an important topic: if most our civilization's thinking is soon to be done inside of machines, t...
Heaven: The last part of dystopia
by Existism — 2022-07-09T22:36:34.943Z

What does that mean?

A century ago, Albert Einstein taught us: everything is relative. The way you see the world depends on your point of view and context. What is hot for you might be mild for me, what is a success for me might be a failure to you. There’s no black and white, there’s only the perspective you choose to look at things. Everything can change in a snap when you provide context.

It’s impossible to be born into heaven. You cannot understand light without understanding darkness, you cannot know what hot is unless there is something colder, you cannot be born into heaven. Therefore to be born into utopia you would need to be born into dystopia and experience it before understanding utopia. Therefore to be born into heaven would be to be born into the last portion of dystopia, it couldn’t be perfect, and the perfect thing would be able to help create it. Billionaires are far happier when they earn it than when they are born into it, and I believe the same for heaven.

Everything in the World Is Relative

You can have anything you can imagine if you are willing to pay the price. But you’ll never be satisfied with life if you just mindlessly try to control everything. Life is like a river, you can’t control it fully, it flows where it wants, it was flowing before you and will flow long after you. Choosing the right perspective on life matters more than you think and only you can choose one that makes your life meaningful, happy and complete.

To him, the new house is beautiful and grandiose in appearance; to me, it's ugly and pretentious. It's all relative, then, isn't it?

You think you're poor? Then take a look at the skid row bums living in the sewers downtown. Everything is relative.

To start with, if you are VERY SURE about what life is, and how it operates I push you to question that…

“I am the wisest man alive, for I know one thing, and that is that I know nothing” -Socrates

While your viewpoints may not fully align with my own, I at least expect you to have an open-mind, as you should expect me to have an open-mind towards your viewpoints. At the end of the day no 2 minds, or people are the same.

We all grew up with our different experiences and hold different beliefs and values for different reasons. While I do not ask anyone to share my beliefs, I of course want to outline where they stem from, why I think they are rational, and how they could help ...
Hope Can = Heaven 
by Existism — 2022-07-09T22:35:12.740Z

When we listen to stories of humans, of life and of death, we fail to fathom the complexity of the stories of all those who have lived before us. We tend to forget the hundreds of generations of humans that lived, worried, loved, and died prior to you being where you are now. You forget you are a miracle of chance experiencing a sliver of infinite possibility on a rock with all these other carbon units, a vast majority you've yet to meet.

Sometimes I think about humanity in the same way I think of pointillism, which is the artistic technique of painting thousands of microdots on a canvas, that contrast and compliment each other in a multitude of colors. From up close these dots seem nonsensical, even plain wrong- a blue dot next to a yellow dot, that from further away appears white. From a distant viewpoint these individual dots combine to make an intricate scene. Each life simply a tiny colorful dot, complimenting and contrasting with those neighboring, blending to contribute towards a bigger picture, a bigger goal.

We realize our planet and everything we care about is a speck in this vast space-time spectrum. A vast space-time spectrum that we have a chance to give great meaning to, to explore, a chance to create from.

How many of us actually daydream about the goal of humanity in the universe?

How many of us allow ourselves to feel the specialness of being here right now, to have such an opportunity, to have the ability to shape the planet to a manifestation that represents the human race.

We’ve come a long way, and we can go a long way too. We could quite literally inherit the universe, to make it OURS, not for the sake of possession, but for the sake of protection, of observation of its infinite beauty. For the sake of UNIVERSAL flourishing.

So far, every human has died a martyr for evolution. What is the end goal for all this creation and loss? Are we really going to let it all go to waste?

Let’s take a moment to imagine what our future could be. Imagine a future world where we are free from our current limitations. Free from biological time. Free from aging, death, disease and biological fragility. Free from spatial limits. From being constrained to one planet, even though we barely have time to appreciate it already.

We can spread through the universe, inhabiting and stabilizing planets in the way that our ancestors did with Earth’s countries....