trentmkelly/LessWrong-43k huggingface dataset:
I’m no longer sure that I buy dutch book arguments and this makes me skeptical of the "utility function" abstraction
by Eli Tyre — 2021-06-22T03:53:33.868Z

[Epistemic status: half-baked, elucidating an intuition. Possibly what I’m saying here is just wrong, and someone will helpfully explain why.]

Thesis: I now think that utility functions might be a bad abstraction for thinking about the behavior of agents in general, including highly capable agents.

Over the past years, in thinking about agency and AI, I’ve taken the concept of a “utility function” for granted as the natural way to express an entity's goals or preferences.

Of course, we know that humans don’t have well defined utility functions (they’re inconsistent, and subject to all kinds of framing effects), but that’s only because humans are irrational. According to my prior view, to the extent that a thing acts like an agent, it’s behavior corresponds to some utility function. That utility function might or might not be explicitly represented, but if an agent is rational, there’s some utility function that reflects it’s preferences.

Given this, I might be inclined to scoff at people who scoff at “blindly maximizing” AGIs. “They just don’t get it”, I might think. “They don’t understand why agency has to conform to some utility function, and an AI would try to maximize expected utility.”

Currently, I’m not so sure. I think that using "my utility function" as a stand in for "my preferences" is biting a philosophical bullet, importing some unacknowledged assumptions. Rather than being the natural way to conceive of preferences and agency, I think utility functions might be only one possible abstraction, and one that emphasizes the wrong features, giving a distorted impression of what agents are actually like.

I want to explore that possibility in this post.

Before I begin, I want to make two notes.

First, all of this is going to be hand-wavy intuition. I don’t have crisp knock-down arguments, only a vague discontent. But it seems like more progress will follow if I write up my current, tentative, stance even without formal arguments.

Second, I don’t think utility functions being a poor abstraction for agency in the real world has much bearing on whether there is AI risk. It might change the shape and tenor of the problem, but highly capable agents with alien seed preferences are still likely to be catastrophic to human civilization and human values. I mention this because the sentiments expressed in this essay are casually downstream of conversatio...
On the limits of idealized values
by Joe Carlsmith — 2021-06-22T02:10:50.073Z

(Cross-posted from Hands and Cities)

On a popular view about meta-ethics, what you should value is determined by what an idealized version of you would value. Call this view “idealizing subjectivism.”

Idealizing subjectivism has been something like my best-guess meta-ethics. And lots of people I know take it for granted. But I also feel nagged by various problems with it — in particular, problems related to (a) circularity, (b) indeterminacy, and (c) “passivity.” This post reflects on such problems.

My current overall take is that especially absent certain strong empirical assumptions, idealizing subjectivism is ill-suited to the role some hope it can play: namely, providing a privileged and authoritative (even if subjective) standard of value. Rather, the version of the view I favor mostly reduces to the following (mundane) observations:

If you already value X, it’s possible to make instrumental mistakes relative to X.
You can choose to treat the outputs of various processes, and the attitudes of various hypothetical beings, as authoritative to different degrees.

This isn’t necessarily a problem. To me, though, it speaks against treating your “idealized values” the way a robust meta-ethical realist treats the “true values.” That is, you cannot forever aim to approximate the self you “would become”; you must actively create yourself, often in the here and now. Just as the world can’t tell you what to value, neither can your various hypothetical selves — unless you choose to let them. Ultimately, it’s on you.

I. Clarifying the view

Let’s define the view I have in mind a little more precisely:

Idealizing subjectivism: X is intrinsically valuable, relative to an agent A, if and only if, and because, A would have some set of evaluative attitudes towards X, if A had undergone some sort of idealization procedure.

By evaluative attitudes, I mean things like judgments, endorsements, commitments, cares, desires, intentions, plans, and so on. Versions of the view differ in which they focus on.

Example types of idealization might include: full access to all relevant information; vivid imaginative acquaintance with the relevant facts; the limiting culmination of some sort of process of reflection, argument, and/or negotiation/voting/betting between representatives of different perspectives; the elimination of “biases”; the elimination of evaluat...
An animated introduction to longtermism (feat. Robert Miles)
by Writer — 2021-06-21T19:24:42.595Z
Link: https://youtu.be/vvehj0KvzK8

EA Forum cross-post
This article is the script of the YT video linked above. It is an animated introduction to the idea of longtermism. The video also briefly covers "The Case for Strong Longtermism" by William MacAskill and then goes over Nick Bostrom's "Astronomical Waste: The Opportunity Cost of Delayed Technological Development". If you would like a refresher about Longtermism, you may read this script, and if you are curious about the animations, you can head to the video. This time the narrator is not me, but Robert Miles.
Consider this: scientific progress and the collective well-being of humanity have been on an upward trajectory for centuries, and if it continues, then humanity has a potentially vast future ahead in which we might inhabit countless star systems and create trillions upon trillions of worthwhile lives.

This is an observation that has profound ethical implications, because the actions we take today have the potential to impact our vast future, and therefore influence an astronomically large number of future lives.

Hilary Greaves and William MacAskill, in their paper “

The Case for Strong Longtermism

” define strong longtermism as the thesis that says, simplifying a little bit: “in a wide class of decision situations, the best action to take is the one that has the most positive effects on the far future”. It’s easy to guess why in light of what I just said. This is a consequence of the fact that the far future contains an astronomically large number of lives.
The question is: what are the actions that have the most positive effect on the far future? There are a few positions one could take, and if you want a deep dive into all the details I suggest reading the paper that I mentioned [note: see section 3 of "The Case for Strong Longtermism] .

In this video, I will consider two main ways in which we could most positively affect the far future. They have been brought forward by Nick Bostrom in his paper “

Astronomical Waste: The Opportunity Cost of Delayed Technological Development

”.
Bostrom writes: “suppose that 10 billion humans could be sustained around an average star. The Virgo supercluster could contain 10^23 humans”.

For reference, 10^23 is 10 billion humans multiplied by 10 trillion stars! And remember that one trillion is one thousand times a billion.

In addition, you need to also consider that future humans would lead vastly better liv...
To the Robobassinet and Progress
by jefftk — 2021-06-21T18:20:04.620Z

Most parents of small babies, if they could, would be happy to have
someone gently cuddle, shush, and rock their baby back to sleep when
they needed settling at night. Babies can't comfortably soothe
themselves to sleep yet, but taking care of them is exhausting work.

Wealthy people have historically handled this problem by hiring nurses,
but paying someone to spend the night looking after your baby can't
ever be within reach for most people. On the other hand, these simple
actions are a very good fit for automation. We can't automate
everything: diaper changing and feeding are much more complicated, but
soothing a baby to sleep is actually a good fit for a machine!

With our first two babies we coslept, swaddling them in an annex off the bed. Once they were old enough
to roll over we stopped swaddling and moved them to a crib. When this
worked well, the baby would wake, quietly snuffle, wake Julia, nurse,
and go back to sleep. When it didn't work well I was doing a lot of
walking, bouncing, patting, and shushing. And so much frustration
when a baby who had fallen asleep in your arms would wake when you set
them down. And Julia says she couldn't fully relax while sleeping
because she needed to pay a little bit of continuing attention to not
rolling onto the baby and keeping the covers clear. We were both very
tired all the time.

With our newborn, we have an automatic "Snoo"
bassinet, and works very well. You swaddle the baby and clip them into
the bassinet so they can't roll over. When you turn it on, it plays
white noise and gently swivels to rock the baby to sleep. It listens
to the noises the baby makes, and if they're not settling it swivels a
bit more vigorously and plays louder white noise. If they still don't
settle after a few escalations, though, it gives up and you come and
figure out whether the baby needs food, a new diaper, or something
else. We only have five nights of experience with it, but it seems to
work very well and the ideas behind it make sense.

Note: the 3:05-8:15 block on the fourth night is wrong: there was a
feed in the middle, we just forgot to turn off the Snoo.

It doesn't look like anything like this existed when our previous
child was born; I can't find specific dates, but the initial round of
publicity announcing it dates to late
October
2016.

One downside is that it makes the early parenting ex...
Idea: Give the Nobel peace prize to institution designers
by Tim Liptrot — 2021-06-21T17:22:07.289Z

The Nobel committee has a problem. They keep giving the peace prize to people to war criminals.

In their most recent debacle, they have the 2019 prize to Abey Ahmed of Ethiopia for making peace with Eritrea. It seemed sensible, a popular young presidents buried the hatchet with their former territory to focus on development. Ahmed seemed like a good man, a great candidate.

Unfortunately, being a good man is easily faked. We now know Ahmed’s real motivation for peace was expanding his coalition relative to internal rivals that might stop his rise to dictatorship. Shortly after the peace deal Ahmed canceled federal elections to destroy subnational rivals. The Tigrayans resisted, and now Ethiopian and Eritrean troops together are busily destroying Tigray.

The problem is that politicians can and do pretend to be “good” to get power. If you are a potential ally, then whatever you define as good can easily be impersonated; committed democrat, liberal, communist, developmental state. After all, any politician that can’t pretend to care never reaches the national stage. The Nobel Peace prize is one more feather in their cap, which only requires a bit more pretending. But in weak democracies the charade only lasts as long as the leader is vulnerable. Once the leader gets enough mercenaries, expectation and appointees, eliminating rivals is worth more than Norwegian admiration.

What are the Norwegian bigwigs to do? They want to give prizes to high-status people that further peace. Sure, they can give grants to NGO’s and activists, but these political lightweights are lower-status.

How about constitutional designers? The real problem of politics is not finding someone who isn’t a sociopath, it’s designing a system that makes the sociopaths serve the people. Thomas Jefferson sold pardons after he lost re-election, but for decades before that he worked hard for the people to build his career.

The lawyers and politicians who write them are lower status than presidents, but they are higher status than NGO leaders or street activists. And they are not in power, so they can’t commit atrocities later.

More importantly, the institutional design, coalition building and negotiating that produces peace and democracy don’t get enough publicity. It’s a long hard slog, slow boring of hard beams. The us didn’t protect democracy by picking good people, they distributed violence capacity betw...
Self-Responsibility
by hath — 2021-06-21T16:17:43.803Z

Note: This is meant to be legible to high school students who are not LessWrong regulars.

If you'd asked me in high school what the difference was between high school kids and adults, I'd have said it was that adults had to earn a living. Wrong. It's that adults take responsibility for themselves. Making a living is only a small part of it. Far more important is to take intellectual responsibility for oneself.

―Paul Graham, What You'll Wish You'd Known

Intro

I go to a public high school. I've observed the same things Paul Graham has. Students typically allow themselves to follow the system wherever it takes them, without pausing to think about whether their goals are best achieved by exclusively putting energy into a college application for four years.[1] Because of this, it is rare to see people pursue self-responsibility. This post is meant to sketch out the details of what being self-responsible looks like, why highschoolers should start being self-responsible, and what to do once you've taken responsibility for yourself.

Self-responsibility

Self-responsibility is understanding that you are the only person who can give you the life you want. Self-responsibility looks like spending time writing down what you want to fix in your life, and then brainstorming how you can make positive changes. Self-responsibility looks like having the audacity to ask people for what you want. Self-responsibility looks like realizing that it takes effort to improve yourself, and creating a plan to learn skills and develop character traits you want to have.
Tony Stark in a cave. Mark Watney trapped on Mars. This is the type of self-responsibility I'm talking about. Unlike those two, failing to become self-responsible probably won't kill you. There just won't be anyone trying to help you reach your goals in life.

Self-awareness

When I ask people what they want to change in their life, they usually have to think about it; it's not a question people ask themselves. We don't often consider how our routines or situations may fall short of what we want. Self-awareness is crucial to self-responsibility. When I started keeping a journal to write about how I felt about my life, I ended up becoming much more driven to make changes. I highly recommend journaling as a tool for self-awareness. Being aware of what you want is a necessary prerequisite to getting what you want. This is als...
How can there be a godless moral world ?
by momom2 — 2021-06-21T12:34:13.770Z

When searching for ideas and arguments for a post I intend to write on theism, I came upon a comment by Scott Alexander who claimed that many theists would change their mind if you could convince them on a gut-level that there could exist a godless moral world.

I introspected and realized that not only did I not believe on a gut-level that there could be such a world, I also couldn't think of any convincing argument to imagine it intellectually at least. I also realized that even though it may not be sufficient to convince me of the inexistence of God, it would be a strong blow to my faith nonetheless, and it's almost definitely a necessary argument to atheism.

I'm pretty sure such an argument exists because there are so many atheists on this forum who seem convinced via reason that the world is godless and moral.

Therefore, I would be grateful if someone could give me not only reasons why and how there can be a morality without God, but also arguments that could speak to my gut-level.

Disclaimer : please do not post arguments against the existence of God. This is not the subject of this question, and I intend to give a better opportunity for that later. I fancy myself a rationalist, but I am also a devout theist, so please don't make appeals to obviousness or general trustworthiness of a certain category of people (in my experience, it's pretty obvious that God exists, and devout theists are the most trustworthy people I know --on an intelligence basis).
Edit (to take into consideration the first comments) :
By "a moral world", I mean that many actions or states are categorized as good or evil, and that this is a good measure to evaluate whether we should do these actions of reach these states, regardless of other measures such as expected utility or pleasure.
For example, I can understand why societies that discourage murder will probably fare better than societies that promote it. I don't understand why murder is bad, if not because God said so.
(Argumenting that my understanding of morality is flawed is welcome too.)

Edit : This question did not have quite the effect I expected it to have... In fact, it probably reassured me in my faith. For everyone who would have preferred to find more convincing arguments, I refer you to this post.
Reinforcing Habits
by lynettebye — 2021-06-21T10:41:35.727Z

Epistemic status: hypothesis based on >10 anecdotal examples

Every month or so, I’ll get a client asking why their attempts to start a habit failed. They want to have an automatic action requiring minimal willpower. The client is usually familiar with at least one habit-building model. Most commonly Charles Duhig’s

Cue, Routine, Reward loop

(in “The Power of Habit”) or CFAR’s

Trigger-Action Plans (TAPs)

.

Their model may go like Duhig’s story: he wanted to change his habit of eating a cookie each afternoon (motivated by watching the scale creep up). So he identified his cue (the time of day around 3pm), planned a new routine to replace the old one (talk to colleagues for ten minutes), and had a new habit. My client usually wants to know what thwarts their attempts to do likewise.

What’s missing from that example?

I read The Power of Habit many years ago, and honestly didn’t remember Duhig’s cookie story any better than my clients often do. So I was surprised when I revisited it and found Duhig actually did a series of experiments to find what the reward was. Something sweet? Nope, eating a candy bar at his desk didn’t feel great. Just taking a break? Nope, taking a walk outside didn’t cut it. Talking with friends? Yeah, that felt rewarding. Based on his experiments, we can guess that getting a cookie each afternoon was a means for getting him to talk to his colleagues while eating it. Because the reward here was socializing, he could build a new habit that didn’t use the cookie as an intermediate step.

This reward step is often neglected by my habit-struggling clients. They want the low-effort, automatic aspects of habits. They lack, however, anything to make those behaviors sticky.

I think it might help to reframe habits as repeatedly-reinforced behaviors. Our brains, often subconsciously, have tied a particular action to some cue after repeatedly having that action rewarded. Simple patterns of cue, action, and reward in close proximity get reinforced, such as “it’s mid afternoon -> I’ll get a cookie -> rewarded by social connection.”

Intentionally designing good habits is hard. By default, our unconscious habits are selected for rewarding behaviors. For example, when I’m feeling blue, eating chocolate and hugs make me happier. I don’t need to train myself to eat chocolate when I feel blue - this happens quite easily!

On ...
Four Components of Audacity 
by lsusr — 2021-06-21T04:49:06.968Z

For a long time I've wondered how to measure nonconformity. To measure nonconformity I needed to define "nonconformity". But no matter how I defined "nonconformity" my definitions felt so subjective they could apply to anybody, from a certain point of view. If everybody is nonconformist then nobody is nonconformist because the word "nonconformist" isn't meaningful.

Today I realized that the opposite of conformity is audacity.

audacity

noun, plural au·dac·i·ties.

boldness or daring, especially with confident or arrogant disregard for personal safety, conventional thought, or other restrictions.

effrontery or insolence; shameless boldness:
His questioner's audacity shocked the lecturer.

Usually audacities . audacious or particularly bold or daring acts or statements.

―Dictionary.com

Audacity is bold, daring, shameless and impertinent. Cultivate these qualities and you will cultivate nonconformity.

My favorite technique of boldness is to simply tell the truth. One trick is to never prefix statements with "I believe". Don't say "I believe x". If x is true then just say "x". (If x is untrue then don't say x and don't believe x.) The unqualified statement is bolder. Crocker's rules encode boldness into a social norm.

Daring comes from doing things that scare you.

Shamelessness comes from not caring what other people think on short time horizons.

The most impressive people I know care a lot about what people think, even people whose opinions they really shouldn’t value (a surprising numbers of them do something like keeping a folder of screenshots of tweets from haters). But what makes them unusual is that they generally care about other people’s opinions on a very long time horizon—as long as the history books get it right, they take some pride in letting the newspapers get it wrong.

―The Strength of Being Misunderstood by Sam Altman

Impertinence comes from treating superiors as equals. I don't know how to cultivate impertinence because I'm status-blind to begin with. Impertinence is the opposite of submission; it is dangerous to be impertinent when your livelihood is on the line.